Hi there,
On 07/10/2015 01:02 PM, Jonas Meurer wrote:
> Am 2015-07-08 15:34, schrieb Jonas Meurer:
>> I've another annoying issue with my new Kerberos-secured NFSv4 setup.
>> Sometimes when Exim4 writes to the mounted NFS share, it fails to set
>> owner and permissions on the written file. Exim4 runs as local user
>> Debian-exim:Debian-exim but tries to set owner of created files on
>> the NFS share to 'mail:mail'. Both the local user Debian-exim and
>> the local user mail are authenticated against the Kerberos server and
>> principals 'Debian-exim@DOMAIN.ORG' as well as 'mail@DOMAIN.ORG' do
>> exist.
>>
>> Obviously, not time Exim4 creates a file and sets owner on the NFS
>> share, the error is produced. Most of the time, this just works and
>> new files are owned by 'mail:mail'. But sometimes, it fails. In
>> these cases, Exim4 gives the following error:
>
> In the meantime, I did some further debugging. I still have no clue
> what triggers the error. It happens serveral times every hour, but
> I didn't find a pattern yet. Interestingly, the error vanishs by
> itself after a few minutes.
>
> Below's some debugging output of rpc.idmapd on the Kerberos-/NFS-
> Server.
Ok, these long entries explain what's happening from the server
perspective - and the server does everything correct here. First of
all, this has nothing do with Kerberos, but with NFSv4's idmapping.
Let me back up a little. If you look at the internet standard covering
NFSv4 (RFC 3530) [0], there's a nice little section called
5.8. Interpreting owner and owner_group
which describes the attributes used to transmit user and group
ownership information between server and client (both for what to show
with 'ls' and for what to do with 'chown/chmod'). To quote parts of
that section:
[...] It is expected that the client and
server will have their own local representation of owner and
owner_group that is used for local storage or presentation to the end
user. Therefore, it is expected that when these attributes are
transferred between the client and server that the local
representation is translated to a syntax of the form
"user@dns_domain". This will allow for a client and server that do
not use the same local representation the ability to translate to a
common syntax that can be interpreted by both. [...]
The translation used to interpret owner and group strings is not
specified as part of the protocol. This allows various solutions to
be employed. For example, a local translation table may be consulted
that maps between a numeric id to the user@dns_domain syntax. [...]
In the case where there is no translation available to the client or
server, the attribute value must be constructed without the "@".
Therefore, the absence of the @ from the owner or owner_group
attribute signifies that no translation was available at the sender
and that the receiver of the attribute should not use that string as
a basis for translation into its own internal format. [...]
Note that Linux always uses uids internally in the kernel to store user
credentials, so what happens is that the sender of a NFSv4 packet takes
the uid, translates it into a string representation containing an @,
and then sends that over the netowrk - the receiver translates it back
into a uid and uses that in its own data structures. (Btw. if you run
ls to show a directory listing, ls will translate it back into a string
representation, possibly a different one, so you may see a username
instead of just a number. So on NFS three different translations will
happen if you type ls -l somewhere: 1. server: uid -> nfs-name,
2. client (kernel): nfs-name -> uid and 3. client (ls): uid -> name.
nfs-name is typically the same as name with just an @nfs4-domain at
the end, but doesn't have to be.)
This is what the idmapping on both the server and the client is about:
to tell the NFSv4 server and client implementations how to do this type
of translation.
(Side note: if you are using sec=sys and a recent enough kernel on both
server and client, IIRC 3.4 or newer - but I may be mistaken about the
version that was properly implemented - idmapping is not required at
all, since the kernel supports just sending the uids in ASCII as
numbers in that field, see the RFC. But that doesn't work for
sec=krb5*, because the security model depends on requests being
authenticated with the principal of the user on the client, which
implies that there needs to be idmapping anyway to map the ids to a
principal and back. On the other hand, sec=sys doesn't have anything
you'd want to call security model at all. sec=sys is just "let me trust
the client completely" or at best "let me trust the client with
everything but root itself".)
Now let's get back to those log messages:
> Jul 10 10:46:59 nfs1 rpc.idmapd[4946]: nfsdcb: authbuf=gss/krb5i authtype=user
> Jul 10 10:46:59 nfs1 rpc.idmapd[4946]: nfs4_name_to_uid: calling nsswitch->name_to_uid
> Jul 10 10:46:59 nfs1 rpc.idmapd[4946]: nss_getpwnam: name '8' domain 'freesources.org': resulting localname '(null)'
Here we have to look at two source codes: idmapd (part of nfs-utils)
and libnfsidmap (its own package). idmapd is just a wrapper around that
library and implements a simple event loop around a couple of pipes it
uses to communicate with the kernel. The nfsdcb function is the handler
that processes idmapping requests and replies to them (set up in [1],
function body in [2]). Note that it just reads the request, parses it
and then calls a function to do the actual mapping (imconv), which
itself then calls into the libnfsidmap library.
Since you can setup multiple different idmappings, those are
implemented in a similar manner to plugins in libnfsidmap. In your
case, you are using the 'nss' plugin (name service switch, the libc
mechanism normal programs use to lookup users and groups, typically
in /etc/passwd and /etc/group). That contains the function nss_getpwnam
that is called (as you can see from the debug message), whose body is
found in [3].
There you can see where the log message comes from:
IDMAP_LOG(4, ("nss_getpwnam: name '%s' domain '%s': "
"resulting localname '%s'\n", name, domain, localname));
If you backtrace everything through the call stack of both libnfsidmap
and idmapd, the 'name' veriable contains the data idmapd got from the
kernel, and if you went digging into the kernel source, you'd see that
it just takes the raw data it got from the NFS client and passes it on
to the idmapper.
So that means that if you look at your log message, the NFS client
transmitted the string '8' (it's a 1-byte UTF-8 string with a single
digit, it's not a binary representation of that number!) to the server;
the server passed that string on to the idmapper, the idmapper then
notices "oh dear, it doesn't contain an @", and so it says "nope,
sorry, can't translate that". See the quoted RFC as to why that
behavior is correct.
Compare that to the case where everything worked:
> Jul 10 10:50:34 nfs1 rpc.idmapd[4946]: nss_getpwnam: name 'mail@freesources.org' domain 'freesources.org': resulting localname 'mail'
Here you can see that the owner name that was transmitted by the NFS
client was 'mail@freesources.org' (and not simply '8'), so that does
contain an @; nss_getpwname can see that the domain name matches
and just strips it, resulting in a user name 'mail', which it looks
up in /etc/passwd, returns the user id (in this case, 8, because it's
the same on client and server) and the server is perfectly happy.
So to summarize your problem from this perspective so far:
- Nothing to to with Kerberos, since that's only for authenticating
the packets.
- The NFS server appears to work properly throughout the whole thing.
- The NFS client sends just the uid converted to a string in some
cases instead of the properly translated NFS username, which the
server then rejects.
So why does the client send the wrong username?
Taking a look at the current kernel source code, there's a function in
the NFS client called 'nfs_map_uid_to_name' that maps the kernel's uids
into NFS names. It's defined in fs/nfs/idmap.c [4] and contains the
following lines (on Jessie's kernel):
if (!(server->caps & NFS_CAP_UIDGID_NOMAP))
ret = nfs_idmap_lookup_name(id, "user", buf, buflen, idmap);
if (ret < 0)
ret = nfs_map_numeric_to_string(id, buf, buflen);
Ignore the NFS_CAP_UIDGID_NOMAP part, that flag is never set when using
sec=krb5* (it's for the sec=sys direct numeric id case, see above).
So what happens here is the following:
1. nfs_idmap_lookup_name() is called
2. It fails
3. So the kernel says "let's put in the numeric id, so the server's
logs will at least show some useful information", so it just
converts the number to a string and sends that along.
So that's what causes your problem: every once in a while, idmapping
will fail, so the kernel will just send a number. But that number will
cause the chown command to fail, since the server won't translate it
back.
Now for the million dollar question: why does idmapping fail on the
client sometimes?
Short answer: I have no idea.
Longer answer: I have been chasing the same bug (or at least a VERY
similar one) for a LONG time, but I haven't made headway on it yet.
(Mostly because I don't understand a lot of the kernel code involved
in this to the degree required and I didn't have time to sit down and
do the necessary homework.)
My problem was kind of the other way around: sometimes files would
appear to belong to nobody [5] instead of their proper user on the
client. Direct access would still work (because the actual proper
checking is done on the server), but 'ls' would display the
permissions wrong. Even worse, with home directories on NFS, the ssh
client would sometimes complain, because it has checks in it that the
.ssh directory is only writable by the user themselves - and if the
.ssh directory appears to belong to somebody else, ssh will not work.
(From reports from other people on the same system, vim also appears to
dislike this situation, possibly other programs as well.) After a
couple of minutes, everything would be back to normal and the files
would then have their proper owner again.
So in that case, the mapping NFS name -> uid doesn't quite work, which
is very similar to your problem, where uid -> NFS name (we're talking
about the client here!) doesn't work.
Anyway: idmapping on the client is actually quite complicated if you
look into the precise details. There are actually two possbile ways of
using idmapping on the client:
- Using idmapd running as a daemon
- Using nfsidmap via the request-key upcall
In case of idmapd, it tries to open pipes provided by the kernel and
uses them to communicate with it (similar to the NFS server, but it
doesn't use the same pipes).
The other possibility is to use the 'nfsidmap' helper (also part of
nfs-utils). To explain that I need to step back a bit again: the kernel
provides a generic key-value cache interface [8]. This interface is
used to store NFS name <-> id mappings on modern Linux kernels
(starting somewhere between 3.3 and 3.5 IIRC). What the NFS client code
in the kernel does is that once it needs to acquire a mapping, it will
query the generic cache. If the mapping has been cached already, it
will just use the cached value immediately. If the mapping is not
present in the cache, the kernel will call a helper program
/sbin/request-key (if it exists) to populate that mapping.
/sbin/request-key is very generic, not for NFS specifically; it's
called when ANY entry in the kernel's key cache is not present, not
just for NFS mappings. There's a package called keyutils that contains
this binary. It is configured via files in /etc/request-key.d and
nfs-utils installs a file id_resolver.conf there to make sure that
/usr/sbin/nfsidmap is called if keys (i.e. cache entries) for type
id_resolver are requested.
So that means the following:
- NFS client needs a mapping (in either direction)
- asks via the kernel's key interface [8] for the result of that
mapping
- kernel calls /sbin/request-key with type 'id_resolver'
- /sbin/request-key calls /usr/sbin/nfsidmap (becausee of its
configuration)
- /usr/sbin/nfsidmap tries to resolv the entry and stores it in the
kernel keyring (i.e. the cache)
- the NFS client sees the cache entry and uses that mapping
Now it gets even more complicated: data from idmapd is ALSO stored in
that cache nowadays - since newer kernel versions were not supposed to
break older setups with idmapd. But that means that idmapd doesn't know
about that cache - so what happens here is the following:
- NFS client needs a mapping
- asks via the kernel's key interface [8] for the result of that
mapping
- /sbin/request-key doesn't exist, so fails
- NFS client sees that failure and falls back to using its pipe to
talk to idmapd
- NFS client (kernel code!) puts the result of that request directly
into the keyring - but instead of using type 'id_resolver' it uses
type 'id_legacy'.
If you want to see that in action, do the following on the client as
root:
cat /proc/keys
That should have a couple of entries in there that show how the id
mapping is stored in principle.
(Note that this is just on the NFS client - on the NFS server idmapd is
still used and ONLY that can be used - and the kernel's key cache is
NOT used there.)
So where's the bug?
- I don't know yet. On a Wheezy kernel I didn't experience the
mapping failure I described (or at least not even remotely as
often), whereas the same idmapd/libnfsidmap version have this
issue with a wheezy-backports kernel (i.e. Jessie kernel), and
the slightly newer Jessie versions of idmapd/libnfsidmap have
the same trouble under Jessie. So my guess is that there's
something in the kernel that in some circumstances fails and
then caches the "doesn't have a proper mapping" value that only
expires after some time.
- Since you are using just regular nsswitch via /etc/passwd,
nss_getpwnam should *never* fail in your case, unless you do
some weird stuff with /etc/passwd at the same time. (In my case,
users are stored in LDAP, so that introduces a potential further
problem here, because there could be some problem with the
network connection.)
- I had mixed results when it comes to comparing nfsidmap and idmapd.
The problem in my case is that the error occurs only sporadically
and was therefore quite hard to pin down for me. I had the
impression that it kind of depends on the kernel version which of
these mechanisms works better, but that may only be anecdotal,
because of the rarity of the problem in my case.
(I really need to find an easy way to reproduce and specifically
trigger this issue in a set of VMs - that would really help with
debugging.)
So where does that leave you?
- You can try to use the nfsidmap mechanism instead of idmapd. It's
actually really trivial to do that: install the keyutils package
on the client and that's it. (You may stop idmapd or leave it
running, shouldn't make a difference if /sbin/request-key exists
and it supports upcalls via /usr/sbin/nfsidmap.) To disable it
again, remove the package. (Maybe it's sufficient to comment out
the line in /etc/request-key.d/id_resolver.conf to disable it but
keep keyutils installed, but I'm not sure. The package is small
enough that removing / reinstalling it should be easy enough.)
Note that /etc/idmapd.conf is used by both tools, so you don't need
to configure anything.
Might help, might make the problem worse. As I said above: Since
my (related) problem wan't so easy to trigger for me, I'm not
completely sure.
- If it consistently occurs in your case, maybe you could increase
the debugging level of the client's idmapd and/or nfsidmap
(Verbosity in the configuration file is honored by both.) Then
you can compare the log entries from server and client and try
to correlate the lookups to see if you find something interesting
on the client just before the server complaining. (But if it's
really a kernel bug, debugging the userspace side will not likely
yield results, but you never know - and maybe the bug is really
not in the kernel.)
- You could try a newer kernel (e.g. 4.0.8-1 from Debian testing) on
the client and see if the problem persists with that - maybe
somebody fixed it in the mean time (possibly accidentally while
cleaning up code or so - or fixing another issue that seemed
unrelated or something). Haven't done that myself yet.
Sorry that I couldn't give you a better answer so far, I'd also really
like to find that bug and finally squash it.
Slight tangent:
>> But more testing reveiled that even a chown to '8:8' works on the
>> NFS share. So using UID instead of username doesn't seem to be the
>> problem here.
Note that what you use on the command line is completely irrelevant in
this case: the kernel only provides an API that takes numbers (so the
command chown mail:mail would translate that itself before calling the
kernel, even on local filesystems) - and the kernel itself translates
it to NFS names (see above). So anytime you do a chown on something, as
long as chown doesn't complain about 'invalid user' (try that as root:
touch /tmp/a; chown blablabla /tmp/a), the kernel will always only see
numbers from that command - regardless of the filesystem used. (The
kernel doesn't care about usernames, that's why you can actually have
users in LDAP, NIS, MySQL, whatever without modifying the kernel: the
kernel is only interested in UIDs for operations and only in special
cases like NFS where the standard requires translation to names will it
care about names - but even then it will call a userspace program such
as idmapd to do the actual translation, so that the kernel remains
completely agnostic as to how the users are managed on a given system.)
(Just FYI.)
Regards,
Christian
[0] https://tools.ietf.org/html/rfc3530#section-5.8
[1] http://sources.debian.net/src/nfs-utils/1:1.2.8-9/utils/idmapd/idmapd.c/#L757
[2] http://sources.debian.net/src/nfs-utils/1:1.2.8-9/utils/idmapd/idmapd.c/#L499
[3] http://sources.debian.net/src/libnfsidmap/0.25-5/nss.c/#L163
[4] http://sources.debian.net/src/linux/3.16.7-ckt11-1/fs/nfs/idmap.c/#L760
[5] Or rather, the numerical value of 2^32-2, since there's a bug in
libnfsidmap that I reported a year ago [6], where the fix went into
version 0.26 of libnfsidmap, which was only released in October of
2014 [7], being too late for Jessie. But regardless, even if it
said 'nobody' instead of that huge number in Jessie, that wouldn't
solve anything.
[6] http://www.spinics.net/lists/linux-nfs/msg43825.html
[7] http://git.linux-nfs.org/?p=steved/libnfsidmap.git;a=summary
[8] https://www.kernel.org/doc/Documentation/security/keys.txt
Attachment:
signature.asc
Description: OpenPGP digital signature