Bug#632074: linux-2.6: NFS4 client sends NULL calls in the TCP session
Package: linux-2.6
Severity: important
After kernel upgrade, our debian machines connected through NFSv4 to a NetApp filer cause the log of warning messages by the NAS :
Client 1XX.1XX.2XX.73 has an authentication error 2
Client 1XX.1XX.2XX.73 is sending bad rpc requests with error: RPC version mismatch or authentication error(73)
This does not prevent the use of the file system, but seems to be a violation of the NFS 4 protocol (see below)
This is known to happen with the following version of linux-image :
2.6.32-5-amd64
2.6.32-5-686-bigmem
2.6.29-xs5.5.0.17
and is known not to happen with the follwing versions :
2.6.18.xs4.0.1.900.5799
2.6.26-2-686
Some of the system listed above are virtual machines on XenServer 5.5 or 5.6 hosts, but some are physical machine, so I discard the possibility of a problem on the xen
version of the kernel.
A ticket has been filed bus us to the NetApp support, and after an analysis of the tcp trace they sent back the problem to the client side.
Below see the analysis by NetApp of this issue, and the description of the protocl exchange creating this issue.
-------------------------------
Tue May 24 10:05:59 MEST [vcid@s-jrciprna004p: nfsd.rpc.request.bad:warning]: Client 1XX.1XX.2XX.73 is sending bad rpc requests with error: RPC version mismatch or
authentication error(73)
We looked in the code and the (73) has no significance here and is simply the error code number for "RPC version mismatch or authentication error".
What we see is that the following occurs at the time of these errors:
- The client has an established TCP session on which it does NFSv4.
- The NFSv4 calls uses Kerberos.
- On that TCP session, the client occasionally does a NULL call.
- The filer rejects it with an authentication error (auth state 2, client must begin new session)
- The client does a new NULL call on a separate TCP session without a GSS context.
- The filer responds and a new context is established.
- The client continues on the original TCP session with the new context.
This explains why no side effect is seen: the client simply establishes a new context and continues as if nothing had happened.
We have checked through the trace for vlan 240 and the pattern is the same throughout and the error always happens for NULL calls only (occasionally two replies may be
sent in the same TCP payload, but the error is always on the NULL reply, then).
We know that some Debian kernels do not exhibit this problem at any time, but others do. This (along with the problem being tied to NULL calls only)
suggest to us that this is due to client side behaviour.
Anyway, we tried to check for the first occurrence of the error, which warrants some chronology. We'll do references per clock second for ease.
- The first client call is at 10:04:16 in an established NFS mount.
- The initial part of the trace, the client only uses TCP port 1006.
- The client uses the same GSS context, with the exception of a SETCLIENTID and a SETCLIENTID_CONFIRM call.
- At 10:05:00 the client tears down four GSS sessions (used for Kerberos) using RPCSEC_GSS_DESTROY in an NFSv4 NULL call. This is done from TCP port 1006 but for four
different contexts. None of these have been used in the trace at that point.
- The client continues with more cals on port 1006 using the the same GSS context.
- Still at 10:05:00 (frame 1982792), the client uses an NFSv4 call to do a RPCSEC_GSS_INIT to establish a new GSS context.
- The client continues using the new GSS context and does not reuse the old context.
- The sequence described above on the NULL calls start.
Looking closer at these steps, we notice something important in the NULL calls.
Above, the client destroyed four GSS contexts that were not used during the trace. However, it did not destroy the GSS context it was using for a while there.
However, we now note the client actually does a RPCSEC_GSS_DESTROY in each of the NFSv4 NULL calls where we respond with an authentication error. As the error
indicates that the client has to begin a new session, this seems like a reasonable response to the call.
So to summarise:
- The filer logs these errors when the client destroys a GSS context.
- The error message is a logical response.
The decission to tear down the GSS context is with the client. So this would seem to be a client side issue after all, which just happens to get logged on the filer.
--------------------------
-- System Information:
Debian Release: 5.0.8
APT prefers oldstable
APT policy: (500, 'oldstable')
Architecture: i386 (i686)
Kernel: Linux 2.6.32-5-686-bigmem (SMP w/2 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/bash
Reply to: