--- Begin Message ---
Package: libc6
Version: 2.31-1
Severity: critical
Justification: breaks unrelated software; breaks remote access
TL;DR: sshd privsep child dies with SIGSYS in clock_nanosleep() (libc6 2.31-1)
while it succeeded using nanosleep() under libc6 2.30-8
The machine in question is running buster with some selected packages
(mainly compilers and development stuff) from bullseye (and is located
at a remote location).
The running kernel is 4.19.0-9-amd64 4.19.118-2.
openssh-server 1:7.9p1-10+deb10u2 is running.
After upgrading libc6 from 2.30-8 to 2.31-1 (which caused sshd to restart),
sshd was running, but dropped incoming connections during authentication.
Luckily I still had a terminal open and could downgrade again to 2.30-8
which restored accessibility.
Thanks to the people trying to guess usernames and passwords, I noticed this
difference in /var/log/auth.log:
with 2.31-1:
Jul 20 21:52:11 hostname sshd[25603]: Invalid user ping from 139.219.0.102 port 39588
Jul 20 21:52:11 hostname sshd[25603]: pam_unix(sshd:auth): check pass; user unknown
Jul 20 21:52:11 hostname sshd[25603]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=139.219.0.102
Jul 20 21:52:13 hostname sshd[25603]: Failed password for invalid user ping from 139.219.0.102 port 39588 ssh2
after downgrading to 2.30-8:
Jul 20 21:54:33 hostname sshd[26824]: Invalid user mickey from 51.83.131.123 port 32822
Jul 20 21:54:33 hostname sshd[26824]: pam_unix(sshd:auth): check pass; user unknown
Jul 20 21:54:33 hostname sshd[26824]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=51.83.131.123
Jul 20 21:54:35 hostname sshd[26824]: Failed password for invalid user mickey from 51.83.131.123 port 32822 ssh2
Jul 20 21:54:35 hostname sshd[26824]: Received disconnect from 51.83.131.123 port 32822:11: Bye Bye [preauth]
Jul 20 21:54:35 hostname sshd[26824]: Disconnected from invalid user mickey 51.83.131.123 port 32822 [preauth]
I can reproduce this by running sshd in a mininmal buster chroot and
upgrading libc6 (+ libgcc-s1 libcrypto1 libc-bin).
(There is even no need to restart sshd (which was started under 2.31-1) after
downgrading libc6 again to 2.30-8 to get it functional again.)
I haven't tried sshd/bullseye. I haven't tried booting with 2.31-1.
$ ssh -vvv foo@localhost -p 9922
[...]
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:/07awyZSdCd9QgaTWi1dn3kEg9rbZtYC+ejHd6ZFi2w
debug3: put_host_port: [127.0.0.1]:9922
debug3: put_host_port: [localhost]:9922
debug3: hostkeys_foreach: reading file "/home/beckmann/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/beckmann/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys from [localhost]:9922
debug1: Host '[localhost]:9922' is known and matches the ECDSA host key.
debug1: Found key in /home/beckmann/.ssh/known_hosts:4
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /home/beckmann/.ssh/id_dsa
debug1: Will attempt key: /home/beckmann/.ssh/id_ecdsa
debug1: Will attempt key: /home/beckmann/.ssh/id_ed25519
debug1: Will attempt key: /home/beckmann/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
Connection closed by 127.0.0.1 port 9922
# /usr/sbin/sshd -ddd
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 328
debug2: parse_server_config: config /etc/ssh/sshd_config len 328
debug3: /etc/ssh/sshd_config:13 setting Port 9922
debug3: /etc/ssh/sshd_config:26 setting SyslogFacility LOCAL7
debug3: /etc/ssh/sshd_config:27 setting LogLevel DEBUG3
debug3: /etc/ssh/sshd_config:56 setting PasswordAuthentication yes
debug3: /etc/ssh/sshd_config:61 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:84 setting UsePAM yes
debug3: /etc/ssh/sshd_config:89 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:93 setting PrintMotd no
debug3: /etc/ssh/sshd_config:111 setting AcceptEnv LANG LC_*
debug3: /etc/ssh/sshd_config:114 setting Subsystem sftp /usr/lib/openssh/sftp-server
debug1: sshd version OpenSSH_7.9, OpenSSL 1.1.1d 10 Sep 2019
debug1: private host key #0: ssh-rsa SHA256:Ny3efyKdYNkwzXduBRO9Fzl0k2K505paO5QFGGw0o1s
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:/07awyZSdCd9QgaTWi1dn3kEg9rbZtYC+ejHd6ZFi2w
debug1: private host key #2: ssh-ed25519 SHA256:COeHE8usWY7gfl1+F5DqGx8pptr/4duLPiaPai3J5uo
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-ddd'
debug3: oom_adjust_setup
debug1: Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 9922 on 0.0.0.0.
Server listening on 0.0.0.0 port 9922.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 9922 on ::.
Server listening on :: port 9922.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 328
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 127.0.0.1 port 42960 on 127.0.0.1 port 9922
debug1: Client protocol version 2.0; client software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug2: fd 3 setting O_NONBLOCK
debug3: ssh_sandbox_init: preparing seccomp filter sandbox
debug2: Network child is on pid 3021
debug3: preauth child monitor started
debug3: privsep user:group 101:65534 [preauth]
debug1: permanently_set_uid: 101/65534 [preauth]
debug3: ssh_sandbox_child: setting PR_SET_NO_NEW_PRIVS [preauth]
debug3: ssh_sandbox_child: attaching seccomp filter program [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug3: send packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug3: receive packet: type 20 [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1 [preauth]
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com [preauth]
debug2: compression stoc: none,zlib@openssh.com [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c [preauth]
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa [preauth]
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com,zlib [preauth]
debug2: compression stoc: none,zlib@openssh.com,zlib [preauth]
debug2: languages ctos: [preauth]
debug2: languages stoc: [preauth]
debug2: first_kex_follows 0 [preauth]
debug2: reserved 0 [preauth]
debug1: kex: algorithm: curve25519-sha256 [preauth]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256 [preauth]
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
debug3: receive packet: type 30 [preauth]
debug3: mm_sshkey_sign entering [preauth]
debug3: mm_request_send entering: type 6 [preauth]
debug3: mm_sshkey_sign: waiting for MONITOR_ANS_SIGN [preauth]
debug3: mm_request_receive_expect entering: type 7 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 6
debug3: mm_answer_sign
debug3: mm_answer_sign: hostkey proof signature 0x557e074790c0(100)
debug3: mm_request_send entering: type 7
debug2: monitor_read: 6 used once, disabling now
debug3: send packet: type 31 [preauth]
debug3: send packet: type 21 [preauth]
debug2: set_newkeys: mode 1 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug3: send packet: type 7 [preauth]
debug3: receive packet: type 21 [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug2: set_newkeys: mode 0 [preauth]
debug1: rekey after 134217728 blocks [preauth]
debug1: KEX done [preauth]
debug3: receive packet: type 5 [preauth]
debug3: send packet: type 6 [preauth]
debug3: receive packet: type 50 [preauth]
debug1: userauth-request for user foo service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug3: mm_getpwnamallow entering [preauth]
debug3: mm_request_send entering: type 8 [preauth]
debug3: mm_getpwnamallow: waiting for MONITOR_ANS_PWNAM [preauth]
debug3: mm_request_receive_expect entering: type 9 [preauth]
debug3: mm_request_receive entering [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 8
debug3: mm_answer_pwnamallow
debug2: parse_server_config: config reprocess config len 328
debug3: mm_answer_pwnamallow: sending MONITOR_ANS_PWNAM: 1
debug3: mm_request_send entering: type 9
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for foo [preauth]
debug3: mm_start_pam entering [preauth]
debug3: mm_request_send entering: type 100 [preauth]
debug3: mm_inform_authserv entering [preauth]
debug3: mm_request_send entering: type 4 [preauth]
debug2: input_userauth_request: try method none [preauth]
debug3: mm_request_receive entering
debug3: monitor_read: checking request 100
debug1: PAM: initializing for "foo"
debug1: PAM: setting PAM_RHOST to "127.0.0.1"
debug1: PAM: setting PAM_TTY to "ssh"
debug2: monitor_read: 100 used once, disabling now
debug3: user_specific_delay: user specific delay 0.000ms [preauth]
debug3: ensure_minimum_time_since: elapsed 1.381ms, delaying 4.141ms (requested 5.522ms) [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug3: monitor_read: checking request 4
debug3: mm_answer_authserv: service=ssh-connection, style=, role=
debug2: monitor_read: 4 used once, disabling now
debug3: mm_request_receive entering
debug1: do_cleanup
debug1: PAM: cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug1: Killing privsep child 3021
debug1: audit_event: unhandled event 12
This debug output does not look very helpful ... trying strace on sshd
This might be something interesting: the first child forked from sshd
dies with SIGSYS when running under 2.31-1:
(diff is from 2.30-8 (successful run) to 2.31-1 (failed run))
[...]
write(4, "\0\0\0\16ssh-connection\0\0\0\0\0\0\0\0", 26) = 26
write(7, "\0\0\0/\0\0\0\6\0\0\0'input_userauth_reque"..., 51) = 51
write(7, "\0\0\08\0\0\0\7\0\0\0000user_specific_delay:"..., 60) = 60
-clock_gettime(CLOCK_BOOTTIME, {tv_sec=4151042, tv_nsec=403970688}) = 0
+clock_gettime(CLOCK_BOOTTIME, {tv_sec=4150710, tv_nsec=544415415}) = 0
write(7, "\0\0\0X\0\0\0\7\0\0\0Pensure_minimum_time_"..., 92) = 92
-nanosleep({tv_sec=0, tv_nsec=2340526}, NULL) = 0
+clock_nanosleep(CLOCK_REALTIME, 0, {tv_sec=0, tv_nsec=2426619}, <unfinished ...>) = ?
++++ killed by SIGSYS +++
If you need me to try something else to get more information, don't hesitate to ask.
Andreas
--- End Message ---