NFS client performance slow with ls

[owen@illinois.edu]

Hi all,

  I've come across a problem with either Debian or recent Linux
kernels with NFS that I am really confused by.  I have an NFS server
(Debian Squeeze) exporting to a large number of users.  While setting
up a new computer (Squeeze) as an NFS client, I noticed that it was
taking ~10 seconds to run 'ls -l' on a directory with ~900 entries.
This seemed a bit slow.  I ran 'ls' a few times with 'time':

client1 # time ls 
<output of ls>

real    0m0.362s
user    0m0.007s
sys     0m0.042s

client1 # time ls
<output of ls>

real    0m0.349s
user    0m0.007s
sys     0m0.043s
client1 #

The times were all relatively close.

  As a shot in the dark to troubleshoot, I ran 'time strace ls -l' to
see if any system calls were timing out.  When I ran it this way, the
performance was back to normal:

client1 # time strace ls
<lots of output>

real    0m0.044s
user    0m0.015s
sys     0m0.029s
client1 # time strace ls
<lots of output>

real    0m0.045s
user    0m0.012s
sys     0m0.033s
client1 #

This is absolutely repeatable.  This behavior appears consistent with
several Debian Squeeze machines.  I've used the 2.6.32-5-amd64 kernel,
plus a custom Xen kernel and a custom SCST iSCSI kernel.  The behavior
is the same.  The behavior is also shown on Xen guests running on the
Xen box.  I have one Etch guest and one Squeeze guest, and both show
the behavior, so it seems to be backend-network related.  But then why
would 'strace' change things?

I have a Debian Etch workstation that performs as expected -- it takes
a similar amount of time to run 'ls' or 'strace ls' and the times are
~.1 seconds.

The only other common element to this is that the servers are all
using the BNX2 network driver while the workstation is using the
forcedeth network driver.

Anyone have any ideas?

--
Brynnen Owen
University of Illinois.