--- Begin Message ---
- To: Debian Bug Tracking System <submit@bugs.debian.org>
- Subject: nfs: server not responding
- From: Jerome Walters <jeronimo@fixity.net>
- Date: Sat, 16 May 2009 17:45:47 +0300
- Message-id: <20090516144547.27099.6651.reportbug@lila.open.bg>
Package: nfs-kernel-server
Version: 1:1.1.4-1
Severity: normal
Description of problem:
Periodically, and with no obvious cause, all NFS connections between our
Debian Testing (_Squeeze_) x86 client (a diskless node which uses nfsroot
and boots from the server) and our Debian Testing (_Squeeze_) x86 server
hang ang and dmesg on the client side informs that the server is "not
responding".
The server is responding to everyone else's requests.
Restarting the nfsd on the server doesn't appear to solve the problem.
At first I wasnt able to capture some debug information since /var/log was
mounted over the nfs, so I have installed a hard drive where I mounted
only /var/log to be able to capture debug logs from the client as well.
Debug Logs:
http://fixity.net/tmp/client.log.gz - Kernel RPC Debug Log from the client
http://fixity.net/tmp/server.log.gz - Kernel RPC Debug Log from the server
How reproducible:
Happens from 10 to 90 minutes after booting the diskless node.
Actual results:
NFS connections stop responding, system hangs or becomes very slow and
unresponsive (it doesnt respond to Ctrl+Alt+Del as well). 60 to 90 minutes
after the first server time out client says server OK but the client is
still unresponsive. Immediately after that the client logs server connection
loss again which leads to continues loop. Client is still unresponsive.
Sometimes client resumes normal operation for couple of hours but then the
problem repeats.
Connectivity info:
Both the client and the server are connected to Gigabit Ethernet Cisco Metro
series managable switch. Both of them use Intel Pro 82545GM Gigabit Ethernet
Server Controllers. Neither one of them log any Ethernet errors and none are
logged by the switch.
Expected results:
NFS connections continue to function and don't fail like clockwork when
every other client on the network has no issues.
Client & Server Load:
For the purposes of testing both machines were only running needed daemons
and weren't loaded at all.
Client & Server Kernel:
On both the client and server custom compiled linux 2.6.29.3 kernel was used.
Configuration file @ http://fixity.net/tmp/config-2.6.29.3.gz
Client & Server Network interface fragmented packet queue length:
net.ipv4.ipfrag_high_thresh = 524288
net.ipv4.ipfrag_low_thresh = 393216
Client Versions:
libnfsidmap2/squeeze uptodate 0.21-2
nfs-common/squeeze uptodate 1:1.1.4-1
Client Mount (cat /proc/mounts | grep nfsroot):
10.11.11.1:/nfsroot / nfs rw,vers=3,rsize=524288,wsize=524288,namlen=255,
hard,nointr,nolock,proto=tcp,timeo=7,retrans=10,sec=sys,addr=10.11.11.1 0 0
Client fstab:
proc /proc proc defaults 0 0
/dev/nfs / nfs defaults 1 1
none /tmp tmpfs defaults 0 0
none /var/run tmpfs defaults 0 0
none /var/lock tmpfs defaults 0 0
none /var/tmp tmpfs defaults 0 0
Client Daemons:
portmap, rpc.statd, rpc.idmapd
Server Daemons:
portmap, rpc.statd, rpc.idmapd, rpc.mountd --manage-gids
Server Versions:
libnfsidmap2/squeeze uptodate 0.21-2
nfs-common/squeeze uptodate 1:1.1.4-1
nfs-kernel-server/testing uptodate 1:1.1.4-1
Server Export:
/nfsroot 10.11.11.*(rw,no_root_squash,async,no_subtree_check)
Server Options:
RPCNFSDCOUNT=16
RPCNFSDPRIORITY=0
RPCMOUNTDOPTS=--manage-gids
NEED_SVCGSSD=no
RPCSVCGSSDOPTS=no
Additional Info:
Since I have read that tweaking the nfsroot mount options could improve the
situation a have tested with different options as follows:
rsize/wsize=1024|2048|4096|8192|32768|524288
timeo=15|60|600
retrans=3|10|20
None resulted in solving the problem.
Any help or suggestions on fixing the problem would be highly appreciated. I
have been messing with that problem for the last couple of weeks and ran out
of ideas.
Best Regards,
Jerome Walters
-- System Information:
Debian Release: squeeze/sid
APT prefers old-stable
APT policy: (500, 'old-stable'), (500, 'testing')
Architecture: i386 (i686)
Kernel: Linux 2.6.29.3 (SMP w/2 CPU cores)
Locale: LANG=en_US, LC_CTYPE=en_US (charmap=ISO-8859-1)
Shell: /bin/sh linked to /bin/bash
Versions of packages nfs-kernel-server depends on:
ii libblkid1 1.41.3-1 block device id library
ii libc6 2.9-4 GNU C Library: Shared libraries
ii libcomerr2 1.41.3-1 common error description library
ii libgssglue1 0.1-2 mechanism-switch gssapi library
ii libkrb53 1.6.dfsg.4~beta1-13 Transitional library package/krb4
ii libnfsidmap2 0.21-2 An nfs idmapping library
ii librpcsecgss3 0.18-1 allows secure rpc communication us
ii libwrap0 7.6.q-16 Wietse Venema's TCP wrappers libra
ii lsb-base 3.2-22 Linux Standard Base 3.2 init scrip
ii nfs-common 1:1.1.4-1 NFS support files common to client
ii ucf 3.0018 Update Configuration File: preserv
nfs-kernel-server recommends no packages.
nfs-kernel-server suggests no packages.
-- no debconf information
--- End Message ---
--- Begin Message ---
Source: nfs-utils
Source-Version: 1:1.2.3-1
We believe that the bug you reported is fixed in the latest version of
nfs-utils, which is due to be installed in the Debian FTP archive:
nfs-common_1.2.3-1_i386.deb
to main/n/nfs-utils/nfs-common_1.2.3-1_i386.deb
nfs-kernel-server_1.2.3-1_i386.deb
to main/n/nfs-utils/nfs-kernel-server_1.2.3-1_i386.deb
nfs-utils_1.2.3-1.debian.tar.bz2
to main/n/nfs-utils/nfs-utils_1.2.3-1.debian.tar.bz2
nfs-utils_1.2.3-1.dsc
to main/n/nfs-utils/nfs-utils_1.2.3-1.dsc
nfs-utils_1.2.3.orig.tar.bz2
to main/n/nfs-utils/nfs-utils_1.2.3.orig.tar.bz2
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 528939@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Luk Claes <luk@debian.org> (supplier of updated nfs-utils package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Format: 1.8
Date: Sun, 27 Mar 2011 18:54:45 +0200
Source: nfs-utils
Binary: nfs-kernel-server nfs-common
Architecture: source i386
Version: 1:1.2.3-1
Distribution: unstable
Urgency: low
Maintainer: Debian kernel team <debian-kernel@lists.debian.org>
Changed-By: Luk Claes <luk@debian.org>
Description:
nfs-common - NFS support files common to client and server
nfs-kernel-server - support for NFS kernel server
Closes: 441361 474037 528939 560388 585085 610363 612933
Changes:
nfs-utils (1:1.2.3-1) unstable; urgency=low
.
* New upstream release
- 7-remove-duplicate-exports-paragraphs : removed
- 12-svcgssd-document-n-option: updated
- 14-allow-address-without-name : removed
- 15-mountd-fix-path-comparison-for-v4-crossmnt : removed
- 16-mount.nfs.man-nfs.man-update-distinction-between-fstypes:
part about nfs.man removed and
renamed to 16-mount.nfs.man-update-distinction-between-fstypes
- mountd: fix --manage-gids hang due to int/uint bug (0f05c8a)
(Closes: #528939,585085)
- mount.nfs: Don't do anything fancy if this is a remount (f11547f)
(Closes: #612933)
- mount: Mount should retry unreachable hosts (5a355f4)
(Closes: #560388)
- Try to use kernel function to determine supported Kerberos
enctypes (258f10f) (Closes: #474037)
- nfs-common: Add Recommends python for mountstats and nfsiostat
* Make sure everything is shipped (inspired by #594933)
* nfs-common.init: Enable idmapd by default (Closes: #610363)
* Install bug scripts to ease debuging
* Build depend on libtirpc-dev and enable IPv6 (Closes: #441361)
Checksums-Sha1:
73c798e75ea2383ff52b35d1f4a126eff4f75096 1486 nfs-utils_1.2.3-1.dsc
da70a29191b07056d71b6e427a87d5cfd8628523 672759 nfs-utils_1.2.3.orig.tar.bz2
6250529f8d059b1a8ac41c6af07994869108bb74 34766 nfs-utils_1.2.3-1.debian.tar.bz2
6f28acf65cd986b44f2b2d8636317cc3053b943b 153014 nfs-kernel-server_1.2.3-1_i386.deb
56b5a8ab3eca6f10c779c4708a787e4b2fadac78 238680 nfs-common_1.2.3-1_i386.deb
Checksums-Sha256:
9f073630b852082d8a8a9fcc44d9862453b25f21837ffd9d88c207bc8cf4049f 1486 nfs-utils_1.2.3-1.dsc
5575ece941097cbfa67fbe0d220dfa11b73f5e6d991e7939c9339bd72259ff19 672759 nfs-utils_1.2.3.orig.tar.bz2
3e5d5fa173ecf8597d996565ca6a2932a7b0a4ea9ef3197e46bbb3d527a3ccdd 34766 nfs-utils_1.2.3-1.debian.tar.bz2
5126b4374a789d9ad3f9d817eb1e558931914edfe177660cf981d760d04d016d 153014 nfs-kernel-server_1.2.3-1_i386.deb
07b68d201ef9d8668a839d8c533bef4205d04f6215c84a45f1f102cba060cd76 238680 nfs-common_1.2.3-1_i386.deb
Files:
72abcfe642257f0579d14c54590ae8f8 1486 net standard nfs-utils_1.2.3-1.dsc
1131dc5f27c4f3905a6e7ee0d594fd4d 672759 net standard nfs-utils_1.2.3.orig.tar.bz2
d9de93233d39669b2207ea9694911630 34766 net standard nfs-utils_1.2.3-1.debian.tar.bz2
aa9e1a2f770d8a9855793535d89f59c3 153014 net optional nfs-kernel-server_1.2.3-1_i386.deb
81790c7ffec2ccc3ad713d23802f1d96 238680 net standard nfs-common_1.2.3-1_i386.deb
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
iEYEARECAAYFAk2PcJMACgkQ5UTeB5t8Mo3DxACePOeosewmAxKKFFn3zNz27xMe
nS8AnRfvi7XjUIgmeJxvc7WAKvK7Q6Fp
=fo5P
-----END PGP SIGNATURE-----
--- End Message ---