can't ssh after an upgrade
Hi,
After the last few (at least) upgrades I haven't been able to login to
remote systems. ssh hangs before I get a password prompt, and since
rebooting the system I'm ssh-ing from fixes things it appears to be a
local problem.
bsass@onegee:~$ ssh -v -v -v bsass@smokie
OpenSSH_4.7p1 Debian-13, OpenSSL 0.9.8g 19 Oct 2007
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to smokie [192.168.0.100] port 22.
debug1: Connection established.
debug1: identity file /home/bsass/.ssh/identity type -1
debug1: identity file /home/bsass/.ssh/id_rsa type -1
debug1: identity file /home/bsass/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version
OpenSSH_4.7p1 Debian-13
debug1: match: OpenSSH_4.7p1 Debian-13 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.7p1 Debian-13
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit:
aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit:
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit:
aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit:
aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit:
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 116/256
debug2: bits set: 518/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /home/bsass/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug3: check_host_in_hostfile: filename /home/bsass/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'smokie' is known and matches the RSA host key.
debug1: Found key in /home/bsass/.ssh/known_hosts:1
debug2: bits set: 537/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
...and there it hangs until I get bored and ctrl-c it.
Detailed use case 1 (# of dashes indicates a particular term session):
- start dselect in a konsole session
- perform update
--- execute "ssh -X smokie" via the konsole session menu (opens new tab)
--- enter password at prompt
--- `su - dselect'
--- perform update
----- execute "ssh -X bms" via the konsole session menu (opens new tab)
----- enter password at prompt
----- `su - dselect'
----- perform update
[i.e., I'm updating 3 boxes simultaneously; 1 local, 2 remote]
- select packages
- install packages
- quit (closes tab)
--- select packages (on smokie)
--- install packages
--- quit
--- logout from smokie (closes tab)
----- select packages (on bms)
----- install packages
----- quit
----- logout from bms (closes tab)
[i.e., upgrade boxes serially
(because they share a /var/cache/apt/archives directory])
- try ssh to smokie|bms again --> hung ssh
[i.e., something broke during the upgrade but it is unclear whether it
is local or remote]
Detailed use case 2 (# of dashes indicates a particular term session):
- start dselect in a konsole session
- perform update
- select packages
- install packages
--- execute "ssh -X smokie" via the konsole session menu (opens new tab)
--- [hangs before password prompt received]
<reboot the local box>
- execute "ssh -X smokie" via the konsole session menu
- [works as expected]
[i.e., this appears to indicate that something is going wrong on the
local box independently of whether the remote systems have been
upgraded]
I can ssh from both bms and smokie (the "remote" boxes) to onegee.
[i.e., it is only the client on onegee which is messed up]
There is nothing in any logs (local or remote) when ssh hangs.
All three boxes are currently up to date Unstable systems.
Onegee is a little strange in that /usr is an NFS mount exported from
smokie over a 100mb LAN; which I mention only because it tends to
manifest start-stop-daemon problems which the other two do not. To make
matters even worse, I have not been able to reproduce these s-s-d
related problems at will, they only seem to happen during a system
upgrade (i.e., `apt-get --reinstall', `dpkg -i', etc., doesn't trigger
them). I suspect that this box (onegee, the one with the problem) may
be more sensitive to race conditions than a self contained system.
Any ideas on what is happenning or how I can track the problem
down/determine if it is a bug?
- Bruce
p.s. probably an unrelated cosmetic issue, but...
/var/run contains both an empty sshd directory and a sshd.pid
file. /var/run/sshd is (re)created when sshd starts but doesn't appear
to ever get used; /var/run/sshd.pid behaves as expected. It looks like
either a superfluous sshd dir is being created, or sshd.pid is
misplaced.
Reply to: