passwordless ssh root logins stopped working after testing dist-upgrade

["Russell L. Carter" <rcarter@pinyon.org>]

Hi folks,
New to this list but been running mostly current testing for almost
ten years now.  I've got a problem with ssh root logins after I
dist-upgraded yesterday.  The following is derived from an email I
sent today to debian-user.

Ryan Manikowski wrote:
> On 4/6/2010 4:37 PM, Russell L. Carter wrote:

>
> What you're trying to do here is login to the 'root' account using your
> non-root account to initiate the ssh connection. It is reading the
> 'id_rsa.pub' pubkey file from /home/<user>/.ssh/ and this is why it is
> failing. The non-root account on the remote side (in this case, your
> localhost) does not have access to ANY files in /root/ so it will never
> work.
>
>
>
>      Ryan Manikowski


Ok, if that is the correct explanation, why does ssh to another
regular user account work?  Why does ssh root@<some_other_older_system>
just work?  I just performed the following steps:

On my main system I have two user accounts, 'rcarter' and 'sardine'.  I
remove the .ssh directories from 'rcarter', 'sardine', and 'root'.  I
create a new rsa key for rcarter (creates ~rcarter/.ssh) and then
ssh-copy-id -i the new key to sardine@localhost and root@localhost,
which creates a new .ssh directory with authorized_keys for each.
Then I ssh-add the new key to the agent as rcarter.

1.  $ ssh sardine@localhost logs in w/o password
2.  $ ssh root@localhost asks for password

This is reproducible on two 'testing' systems that have worked
flawlessly for at least two years each, but were both dist-upgraded
yesterday, and they now exhibit this same behavior.

However, if I ssh-copy-id the new key created by rcarter to root on
two systems that I haven't dist-upgraded in several
weeks and then ssh root@<systemname> works fine, as it always
has.  I diffed the ssh_config and sshd_configs and the only
difference were comments. So the problem would seem to be in
sshd.  I believe my sshd_config is stock.  It doesn't matter
whether the key is rsa or dsa.  This breaks quite a lot of
things in a cluster so obviously I'm motivated to help any
way I can.  And I shouldn't rule out the possibility of
massive failure between the keyboard and the chair.  Maybe
something changed in PAM interaction with SSH?

Appended is a transcript of the actions I took to achieve the
above, output from ssh -vvvv root@localhost.

transcript:  (I removed root and sardine's .ssh dirs before)

rcarter@feyerabend> pwd
/home/rcarter/.ssh
rcarter@feyerabend> cd ..
rcarter@feyerabend> mv .ssh dot.ssh
rcarter@feyerabend> ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/rcarter/.ssh/id_rsa):
Created directory '/home/rcarter/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/rcarter/.ssh/id_rsa.
Your public key has been saved in /home/rcarter/.ssh/id_rsa.pub.
The key fingerprint is:
54:06:d2:08:a4:6d:26:9e:e0:0f:01:1a:1f:67:ff:91 rcarter@feyerabend
The key's randomart image is:
+--[ RSA 2048]----+
|o ..=..o..o      |
|oo * ....+       |
|o.+ + . E        |
|.o.=   o .       |
| oo     S        |
|  o              |
|   .             |
|                 |
|                 |
+-----------------+
rcarter@feyerabend> ssh-copy-id -i sardine@localhost
sardine@localhost's password:
Now try logging into the machine, with "ssh 'sardine@localhost'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

rcarter@feyerabend> ssh-copy-id -i root@localhost
root@localhost's password:
Now try logging into the machine, with "ssh 'root@localhost'", and check in:

  .ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

rcarter@feyerabend> slogin sardine@localhost
Enter passphrase for key '/home/rcarter/.ssh/id_rsa':

rcarter@feyerabend> ssh-add
Enter passphrase for /home/rcarter/.ssh/id_rsa:
Identity added: /home/rcarter/.ssh/id_rsa (/home/rcarter/.ssh/id_rsa)
rcarter@feyerabend> slogin sardine@localhost
Linux feyerabend 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Apr  6 16:36:06 2010 from localhost
sardine@feyerabend> exit
logout
Connection to localhost closed.
rcarter@feyerabend> slogin root@localhost
root@localhost's password:

rcarter@feyerabend>

rcarter@feyerabend> ssh -vvvv root@localhost
OpenSSH_5.3p1 Debian-3, OpenSSL 0.9.8n 24 Mar 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to localhost [::1] port 22.
debug1: Connection established.
debug1: identity file /home/rcarter/.ssh/identity type -1
debug3: Not a RSA1 key file /home/rcarter/.ssh/id_rsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'Proc-Type:'
debug3: key_read: missing keytype
debug2: key_type_from_name: unknown key type 'DEK-Info:'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /home/rcarter/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /home/rcarter/.ssh/id_dsa type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3p1 Debian-3
debug1: match: OpenSSH_5.3p1 Debian-3 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3p1 Debian-3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 792 bytes for a total of 824
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: 
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: 
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: 
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: 
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: 
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: 
diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: 
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: 
aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: 
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: 
hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 848
debug2: dh_gen_key: priv key bits set: 121/256
debug2: bits set: 495/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 144 bytes for a total of 992
debug3: check_host_in_hostfile: filename /home/rcarter/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 1
debug1: Host 'localhost' is known and matches the RSA host key.
debug1: Found key in /home/rcarter/.ssh/known_hosts:1
debug2: bits set: 516/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1008
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1056
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/rcarter/.ssh/id_rsa (0x7f1f7f206ba0)
debug2: key: /home/rcarter/.ssh/identity ((nil))
debug2: key: /home/rcarter/.ssh/id_dsa ((nil))
debug3: Wrote 64 bytes for a total of 1120
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred 
gssapi-keyex,gssapi-with-mic,gssapi,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/rcarter/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1488
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/rcarter/.ssh/identity
debug3: no such identity: /home/rcarter/.ssh/identity
debug1: Trying private key: /home/rcarter/.ssh/id_dsa
debug3: no such identity: /home/rcarter/.ssh/id_dsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
root@localhost's password:

rcarter@feyerabend>




>
>
> ]] Devision Media Services LLC [[
>      www.devision.us
>  ryan@devision.us | 716.771.2282
>
>
>>
>>>      Ryan Manikowski
>>>
>>>
>>> ]] Devision Media Services LLC [[
>>>      www.devision.us
>>>  ryan@devision.us | 716.771.2282
>>>
>>>
>>> On 4/6/2010 4:06 PM, d.sastre.medina@gmail.com wrote:
>>>> On Tue, Apr 06, 2010 at 03:24:04PM -0400, Tony Nelson wrote:
>>>>
>>>>> On 10-04-06 14:12:19, Russell L. Carter wrote:
>>>>>
>>>>>> root@feyerabend> diff -u ssh_config ssh_config.dpkg-dist
>>>>>> --- ssh_config  2010-04-05 21:14:26.172871668 -0700
>>>>>> +++ ssh_config.dpkg-dist        2010-01-04 09:05:12.000000000 -0700
>>>>>> @@ -17,8 +17,8 @@
>>>>>>   # ssh_config(5) man page.
>>>>>>
>>>>>>   Host *
>>>>>> -ForwardAgent yes
>>>>>> -ForwardX11 yes
>>>>>> +#   ForwardAgent no
>>>>>> +#   ForwardX11 no
>>>>>>   #   ForwardX11Trusted yes
>>>>>>   #   RhostsRSAAuthentication no
>>>>>>   #   RSAAuthentication yes
>>>>>>
>>>>> I don't see any "PermitRootLogin without-password" line in your diff.
>>>>>
>>>> Hello,
>>>>
>>>> That would disable password login for root, but does not enable per-se
>>>> pubkey auth (AFAIK).
>>>>
>>>> man sshd_config explain this: PermitRootLogin, PubkeyAuthentication
>>>> and AuthorizedKeysFile entries.
>>>>
>>>> Regards.
>>>>
>>>>
>>>
>>
>
>