Bug#1120598: nfs-common: ls input/output error ("NFS: readdir(/) returns -5") on krb5 NFSv4 client using SHA2
Package: nfs-common
Version: 1:2.8.4-1+b1
Severity: important
X-Debbugs-Cc: twr+debbugs@tylerwross.com
When the session key of a kerberos ticket uses a SHA2 cipher (aes256-cts-hmac-sha384-192 and aes128-cts-hmac-sha256-128 tested), readdir requests fail.
SHA1 ciphers (aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 tested) work as expected.
ls reports the following:
ls: reading directory '/mnt/example/': Input/output error
stat and touch of files and directories is working, and cat'ing a file works (see also: later note about cat with NFSv4.1 and 4.0).
Example of a non-working ticket, as reported by klist -e:
11/12/25 18:37:30 11/13/25 17:49:03 nfs/nfssrv.ipa.twrlab.net@IPA.TWRLAB.NET
Etype (skey, tkt): aes256-cts-hmac-sha384-192, aes256-cts-hmac-sha384-192
Example of a working ticket:
11/12/25 19:01:46 11/13/25 18:27:33 nfs/nfssrv.ipa.twrlab.net@IPA.TWRLAB.NET
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha384-192
If rpcdebug is enabled for nfs and rpc modules, the following is logged to dmesg:
[332376.797836] NFS: nfs_weak_revalidate: inode 262146 is valid
[332376.798512] NFS: revalidating (0:58/262146)
[332376.799169] --> nfs41_call_sync_prepare data->seq_server 00000000e22b1bd9
[332376.799916] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=64
[332376.800764] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[332376.801507] RPC: gss_krb5_get_mic_v2
[332376.802009] encode_sequence: sessionid=1762048597:1479457708:22:0 seqid=27 slotid=0 max_slotid=0 cache_this=0
[332376.803204] RPC: gss_krb5_get_mic_v2
[332376.803726] RPC: xs_tcp_send_request(260) = 0
[332376.804536] RPC: gss_krb5_verify_mic_v2
[332376.805093] RPC: gss_krb5_verify_mic_v2
[332376.805643] decode_attr_type: type=040000
[332376.806149] decode_attr_change: change attribute=22
[332376.806866] decode_attr_size: file size=4096
[332376.807398] decode_attr_fsid: fsid=(0xfdcb5a40986843e0/0xa4fc6c44ad8345ad)
[332376.808154] decode_attr_fileid: fileid=262146
[332376.808742] decode_attr_fs_locations: fs_locations done, error = 0
[332376.809495] decode_attr_mode: file mode=0777
[332376.810042] decode_attr_nlink: nlink=3
[332376.810695] decode_attr_owner: uid=591200000
[332376.811229] decode_attr_group: gid=591200004
[332376.811761] decode_attr_rdev: rdev=(0x0:0x0)
[332376.812291] decode_attr_space_used: space used=4096
[332376.812878] decode_attr_time_access: atime=1762383044
[332376.813487] decode_attr_time_create: btime=1761952933
[332376.814098] decode_attr_time_metadata: ctime=1762055558
[332376.814895] decode_attr_time_modify: mtime=1762055558
[332376.815578] decode_attr_mounted_on_fileid: fileid=262146
[332376.816225] decode_getfattr_attrs: xdr returned 0
[332376.816796] decode_getfattr_generic: xdr returned 0
[332376.817374] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=64
[332376.818135] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[332376.818873] nfs4_free_slot: slotid 1 highest_used_slotid 0
[332376.819604] nfs41_sequence_process: Error 0 free the slot
[332376.820228] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[332376.820930] NFS: nfs_update_inode(0:58/262146 fh_crc=0xad8c294c ct=2 info=0x4427e7f)
[332376.821767] NFS: (0:58/262146) revalidation complete
[332376.822342] NFS: nfs_weak_revalidate: inode 262146 is valid
[332376.823056] NFS: permission(0:58/262146), mask=0x24, res=0
[332376.823684] NFS: open dir(/)
[332376.824087] NFS: readdir(/) starting at cookie 0
[332376.824641] _nfs4_proc_readdir: dentry = /, cookie = 0
[332376.825229] --> nfs41_call_sync_prepare data->seq_server 00000000e22b1bd9
[332376.825967] --> nfs4_alloc_slot used_slots=0000 highest_used=4294967295 max_slots=64
[332376.826814] <-- nfs4_alloc_slot used_slots=0001 highest_used=0 slotid=0
[332376.827616] RPC: gss_krb5_get_mic_v2
[332376.828114] encode_sequence: sessionid=1762048597:1479457708:22:0 seqid=28 slotid=0 max_slotid=0 cache_this=0
[332376.829146] encode_readdir: cookie = 0, verifier = 00000000:00000000, bitmap = 0018091a:00b4a23a:00000000
[332376.830144] RPC: gss_krb5_get_mic_v2
[332376.830720] RPC: xs_tcp_send_request(284) = 0
[332376.831431] RPC: gss_krb5_verify_mic_v2
[332376.831967] RPC: gss_krb5_verify_mic_v2
[332376.832498] --> nfs4_alloc_slot used_slots=0001 highest_used=0 max_slots=64
[332376.833254] <-- nfs4_alloc_slot used_slots=0003 highest_used=1 slotid=1
[332376.833994] nfs4_free_slot: slotid 1 highest_used_slotid 0
[332376.834695] nfs41_sequence_process: Error 0 free the slot
[332376.835318] nfs4_free_slot: slotid 0 highest_used_slotid 4294967295
[332376.836016] _nfs4_proc_readdir: returns -5
[332376.836519] NFS: readdir(/) returns -5
Environment/Supporting Systems:
- The NFS server is a fresh Debian 13 cloud image. freeipa-client, gssproxy, nfs-kernel-server, and qemu-guest-agent have been installed. Joined to FreeIPA via ipa-client-install.
- Kerberos is provided by a newly installed FreeIPA instance on Fedora 43.
Failing NFS client configurations:
1. Freshly deployed and updated Debian 13 official cloud image (debian-13-genericcloud-amd64). freeipa-client, gssproxy, nfs-common, and qemu-guest-agent have been installed. Joined to FreeIPA via ipa-client-install.
2. Freshly installed Debian sid via mini ISO (2025-11-01). Same configuration as 1/above.
3. Minimal replication config: freshly installed Debian 13 via debian-13.1.0-amd64-netinst.iso . Installed nfs-common, krb5-config and krb5-user. Manually installed keytab: no additional krb5 configuration done (realm was automatically configured from hostname by krb5-config).
Working NFS client configuration:
- Fedora 43 installation configured via ipa-client-install .
This issue was escalated to me by someone with a matching production environment (FreeIPA on Fedora 43, and Debian 13 NFS client(s) and server). This original reporter also found that a Fedora 43 client worked as-expected with SHA2.
Miscellaneous observations:
- Testing was primarily conducted with NFS v4.2. Error occurs with krb5, krb5i and krb5p on 4.2. Also confirmed with krb5i on 4.1 and 4.0 (other combinations of krb5/krb5p and vers 4.1/4.0 not tested).
- readdir failure observed when client is mounted with NFS v4.2, 4.1, and 4.0. ls reports "input/output error" and dmesg reports "readdir(/) returns -5" in all 3 versions.
- When mounted with v4.1 and 4.0, cat'ing a file also fails with SHA2. There is no obvious (to me) error in dmesg. stat/touch of files and directories remains working.
- Failing state is cached on the client: if a user runs ls with a SHA2 session key, then acquires a new SHA1 session key ticket, the "input/output error" persists unless the NFS share is remounted. Setting noac, actimeo=0, and lookupcache=none mount options do not affect this behavior: the error persists until a remount. Error persisted when left overnight (about 13 hours).
- Cursory examination of a packet capture shows an apparently normal NFSv4 readdir call and reply. The reply contains the expected directory listing.
Attempted file/directory operations with SHA2 session key and sec=krb5i:
(all are successful/OK with SHA1 session key)
ls directory:
4.2: "Input/output error"
4.1: "Input/output error"
4.0: "Input/output error"
stat file and directory:
4.2: OK
4.1: OK
4.0: OK
touch file and directory:
4.2: OK
4.1: OK
4.0: OK
cat file:
4.2: OK
4.1: "Input/output error"
4.0: "Input/output error"
-- Package-specific info:
-- rpcinfo --
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
-- /etc/default/nfs-common --
NEED_STATD=
NEED_IDMAPD=
NEED_GSSD=
-- /etc/nfs.conf --
[general]
pipefs-directory=/run/rpc_pipefs
[nfsrahead]
[exports]
[exportfs]
[gssd]
use-gss-proxy=1
[lockd]
[exportd]
[mountd]
manage-gids=y
[nfsdcld]
[nfsd]
[statd]
[sm-notify]
[svcgssd]
-- /etc/nfs.conf.d/*.conf --
-- System Information:
Debian Release: forky/sid
APT prefers unstable
APT policy: (500, 'unstable')
Architecture: amd64 (x86_64)
Kernel: Linux 6.17.7+deb14+1-amd64 (SMP w/4 CPU threads; PREEMPT)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages nfs-common depends on:
ii adduser 3.153
ii init-system-helpers 1.69
ii keyutils 1.6.3-6
ii libc6 2.41-12
ii libcap2 1:2.75-10+b1
ii libcom-err2 1.47.2-3+b3
ii libdevmapper1.02.1 2:1.02.205-2
ii libevent-core-2.1-7t64 2.1.12-stable-10+b1
ii libgssapi-krb5-2 1.21.3-5
ii libk5crypto3 1.21.3-5
ii libkeyutils1 1.6.3-6
ii libkrb5-3 1.21.3-5
ii libmount1 2.41.2-4
ii libnfsidmap1 1:2.8.4-1+b1
ii libtirpc3t64 1.3.6+ds-1
ii libwrap0 7.6.q-36
ii rpcbind 1.2.7-1
ii ucf 3.0052
Versions of packages nfs-common recommends:
ii python3 3.13.7-1
Versions of packages nfs-common suggests:
pn open-iscsi <none>
pn watchdog <none>
-- no debconf information
Reply to: