Bug#1109742: upgrade-reports: No new SSH connections possible during large part of upgrade to Debian Trixie
Control: affects -1 openssh-server
[TL;DR: I think it may not be possible to properly solve this without a
bookworm update as well as a change to trixie.]
On Thu, Jul 24, 2025 at 01:19:40PM +0100, Colin Watson wrote:
On Tue, Jul 22, 2025 at 07:42:07PM +0200, Manfred Stock wrote:
Further Comments/Problems: I've upgraded several Bookworm systems to
Trixie so far, which went pretty smooth. But there's one thing I keep
noticing, and which I observed a bit more closely while upgrading the
system I'm sending this report from: Starting at roughly the time when
dpkg says something like
Unpacking openssh-server (1:10.0p1-5) over (1:9.2p1-2+deb12u6) ...
I'm not able anymore to open new SSH connections to the system I'm
upgrading. The SSH daemon is still running, and the existing connections
also still work, but new connections fail with
kex_exchange_identification: read: Connection reset by peer
Connection reset by fd... port 22
on the client.
[...]
Thanks for the report. This will be due to the split of sshd-session
from the main sshd binary; the old sshd re-executed itself with
different arguments, but the new sshd executes sshd-session instead
and has removed support for the parameters that it used to rely on
during re-execution.
I'll have to set up a suitable environment to test this, but my best
idea for now is to have openssh-server.preinst take a copy of the old
sshd binary before dpkg unpacks the new files, and patch sshd to
re-exec that copy if it exists and it receives the -R option. The
postinst can then remove the copy after it's restarted the new sshd.
This approach failed in my first test. To control the order of
operations, I just ran "dpkg --unpack" on the new .deb, and then the new
/usr/sbin/sshd failed before it got as far as re-execing the temporary
copy because it needed a newer libc. apt might not do that in normal
situations, but we clearly want to avoid this.
My next approach was to try a temporary diversion of /usr/sbin/sshd.
This works, although it involves a slightly odd invocation for
openssh-server to be able to divert one of its own files. See the
"openssh_10.0p1-6.debdiff" attachment. Would the release team accept
something like this for trixie?
However, this isn't the whole story. Once the new libssl3t64 is
unpacked, new connections fail with "OpenSSL version mismatch. Built
against 30000100, you have 30500010". This part of the problem can't be
fixed by a change in trixie, because the problem is that the _old_ sshd,
before restarting, fails to tolerate newer minor versions of OpenSSL.
This was fixed upstream in OpenSSH 9.4, and if I'd noticed previously
that this would be an upgrade problem I'd already have included it in a
bookworm update.
So, I think we also need to fix that in bookworm. See the
"openssh_9.2p1-2+deb12u7.debdiff" attachment (for brevity I've pruned
some noise from git-dpm that just updates some commit IDs in patches).
Timing-wise, this is tricky. IMO we really need to get this out before
trixie releases to minimize the chance of users running into this if
they rush to upgrade. Would the security team be willing to consider
pushing this out via -security? Failing that, we'd have to wait until
the next point release of bookworm, which I think would be unfortunate
given that the consequences of sshd being broken between unpack and
configure can include a failed remote upgrade with no way to access the
system (if you forget to maintain a separate ssh connection, or if your
network connection is interrupted).
Thanks,
--
Colin Watson (he/him) [cjwatson@debian.org]
diff -Nru openssh-10.0p1/debian/changelog openssh-10.0p1/debian/changelog
--- openssh-10.0p1/debian/changelog 2025-05-09 13:40:49.000000000 +0100
+++ openssh-10.0p1/debian/changelog 2025-06-05 02:53:41.000000000 +0100
@@ -1,3 +1,11 @@
+openssh (1:10.0p1-6) UNRELEASED; urgency=medium
+
+ * Temporarily divert /usr/sbin/sshd during upgrades from before
+ 1:9.8p1-1~, to avoid new connections failing between unpack and
+ configure (closes: #1109742).
+
+ -- Colin Watson <cjwatson@debian.org> Thu, 05 Jun 2025 02:53:41 +0100
+
openssh (1:10.0p1-5) unstable; urgency=medium
* Ensure that configure knows the path to passwd; fixes reproducibility of
diff -Nru openssh-10.0p1/debian/openssh-server.postinst openssh-10.0p1/debian/openssh-server.postinst
--- openssh-10.0p1/debian/openssh-server.postinst 2025-05-09 13:40:49.000000000 +0100
+++ openssh-10.0p1/debian/openssh-server.postinst 2025-06-05 02:53:41.000000000 +0100
@@ -11,10 +11,16 @@
get_config_option() {
option="$1"
+ sshd_path=/usr/sbin/sshd
[ -f /etc/ssh/sshd_config ] || return
- /usr/sbin/sshd -G | sed -n "s/^$option //Ip"
+ # begin-remove-after: released:forky
+ if [ -e /usr/sbin/sshd.session-split ]; then
+ sshd_path=/usr/sbin/sshd.session-split
+ fi
+ # end-remove-after
+ "$sshd_path" -G | sed -n "s/^$option //Ip"
}
@@ -109,6 +115,24 @@
systemctl unmask ssh.service
systemctl disable ssh.service
fi
+ # begin-remove-after: released:forky
+ if dpkg --compare-versions "$2" lt-nl 1:9.8p1-1~; then
+ # We're ready to restart the listener process so that it
+ # executes sshd-session rather than sshd for new
+ # connections, so we can remove this diversion now. This
+ # starts a brief window where new connections will fail
+ # (ending when the service is restarted), but at least it's
+ # all contained within this postinst.
+ #
+ # See openssh-server.preinst for why we use this odd package
+ # name.
+ dpkg-divert --package openssh-client --remove --no-rename \
+ --divert /usr/sbin/sshd.session-split /usr/sbin/sshd
+ if [ -e /usr/sbin/sshd.session-split ]; then
+ mv -f /usr/sbin/sshd.session-split /usr/sbin/sshd
+ fi
+ fi
+ # end-remove-after
fi
#DEBHELPER#
diff -Nru openssh-10.0p1/debian/openssh-server.preinst openssh-10.0p1/debian/openssh-server.preinst
--- openssh-10.0p1/debian/openssh-server.preinst 1970-01-01 01:00:00.000000000 +0100
+++ openssh-10.0p1/debian/openssh-server.preinst 2025-06-05 02:53:41.000000000 +0100
@@ -0,0 +1,21 @@
+#!/bin/sh
+set -e
+
+# begin-remove-after: released:forky
+if [ "$1" = upgrade ] && dpkg --compare-versions "$2" lt-nl 1:9.8p1-1~; then
+ # Temporarily divert the new sshd binary, since OpenSSH 9.8 moved
+ # part of its responsibilities to sshd-session, and unpacking the
+ # new sshd to its normal path would break new connections. We'll
+ # remove this diversion when we're ready to restart the listener
+ # process.
+ #
+ # Since we're trying to divert a file shipped in this package, we
+ # use a package name that we know doesn't ship /usr/sbin/sshd.
+ dpkg-divert --package openssh-client --add --no-rename \
+ --divert /usr/sbin/sshd.session-split /usr/sbin/sshd
+fi
+# end-remove-after
+
+#DEBHELPER#
+
+exit 0
diff -Nru openssh-9.2p1/debian/.git-dpm openssh-9.2p1/debian/.git-dpm
--- openssh-9.2p1/debian/.git-dpm 2025-05-08 11:54:24.000000000 +0100
+++ openssh-9.2p1/debian/.git-dpm 2025-07-24 15:16:43.000000000 +0100
@@ -1,6 +1,6 @@
# see git-dpm(1) from git-dpm package
-cf9b65754f0e54de11d075fc7317ae90a1ae4389
-cf9b65754f0e54de11d075fc7317ae90a1ae4389
+35433578d3aa9f7d3c8ac1752954ce53bb2d0dce
+35433578d3aa9f7d3c8ac1752954ce53bb2d0dce
cf3c3acb2b8f74eeca7fcee269b1d33ac83f1188
cf3c3acb2b8f74eeca7fcee269b1d33ac83f1188
openssh_9.2p1.orig.tar.gz
diff -Nru openssh-9.2p1/debian/changelog openssh-9.2p1/debian/changelog
--- openssh-9.2p1/debian/changelog 2025-05-08 11:54:24.000000000 +0100
+++ openssh-9.2p1/debian/changelog 2025-07-24 15:16:43.000000000 +0100
@@ -1,3 +1,10 @@
+openssh (1:9.2p1-2+deb12u7) UNRELEASED; urgency=medium
+
+ * Handle OpenSSL >=3 ABI compatibility; this helps to avoid new ssh
+ connections failing during upgrades to trixie (#1109742).
+
+ -- Colin Watson <cjwatson@debian.org> Thu, 24 Jul 2025 15:16:43 +0100
+
openssh (1:9.2p1-2+deb12u6) bookworm; urgency=medium
* CVE-2025-32728: sshd(8): fix the DisableForwarding directive, which was
diff -Nru openssh-9.2p1/debian/patches/no-openssl-version-status.patch openssh-9.2p1/debian/patches/no-openssl-version-status.patch
--- openssh-9.2p1/debian/patches/no-openssl-version-status.patch 2025-05-08 11:54:24.000000000 +0100
+++ openssh-9.2p1/debian/patches/no-openssl-version-status.patch 2025-07-24 15:16:43.000000000 +0100
@@ -1,4 +1,4 @@
-From 5d1c32cb181d5b4392210ddbf2ff84fcda79a89c Mon Sep 17 00:00:00 2001
+From 2e1efa038b6e5ca9127eb21eb3773dc8e8f0ab33 Mon Sep 17 00:00:00 2001
From: Kurt Roeckx <kurt@roeckx.be>
Date: Sun, 9 Feb 2014 16:10:14 +0000
Subject: Don't check the status field of the OpenSSL version
@@ -14,34 +14,35 @@
Bug-Debian: https://bugs.debian.org/664383
Bug-Debian: https://bugs.debian.org/732940
Forwarded: not-needed
-Last-Update: 2014-10-07
+Last-Update: 2025-07-24
Patch-Name: no-openssl-version-status.patch
---
- openbsd-compat/openssl-compat.c | 6 +++---
- openbsd-compat/regress/opensslvertest.c | 1 +
- 2 files changed, 4 insertions(+), 3 deletions(-)
+ openbsd-compat/openssl-compat.c | 8 ++++----
+ openbsd-compat/regress/opensslvertest.c | 2 ++
+ 2 files changed, 6 insertions(+), 4 deletions(-)
diff --git a/openbsd-compat/openssl-compat.c b/openbsd-compat/openssl-compat.c
-index a37ca61bf..c1749210d 100644
+index 6c65003f2..30e97c464 100644
--- a/openbsd-compat/openssl-compat.c
+++ b/openbsd-compat/openssl-compat.c
-@@ -34,7 +34,7 @@
- /*
- * OpenSSL version numbers: MNNFFPPS: major minor fix patch status
- * We match major, minor, fix and status (not patch) for <1.0.0.
-- * After that, we acceptable compatible fix versions (so we
-+ * After that, we accept compatible fix and status versions (so we
- * allow 1.0.1 to work with 1.0.0). Going backwards is only allowed
- * within a patch series.
- */
-@@ -55,10 +55,10 @@ ssh_compatible_openssl(long headerver, long libver)
+@@ -49,18 +49,18 @@ ssh_compatible_openssl(long headerver, long libver)
+ return 1;
+
+ /*
+- * For versions >= 3.0, only the major and status must match.
++ * For versions >= 3.0, only the major must match.
+ */
+ if (headerver >= 0x3000000f) {
+- mask = 0xf000000fL; /* major,status */
++ mask = 0xf0000000L; /* major */
+ return (headerver & mask) == (libver & mask);
}
/*
-- * For versions >= 1.0.0, major,minor,status must match and library
-+ * For versions >= 1.0.0, major,minor must match and library
- * fix version must be equal to or newer than the header.
+- * For versions >= 1.0.0, but <3, major,minor,status must match and
++ * For versions >= 1.0.0, but <3, major,minor must match and
+ * library fix version must be equal to or newer than the header.
*/
- mask = 0xfff0000fL; /* major,minor,status */
+ mask = 0xfff00000L; /* major,minor */
@@ -49,14 +50,22 @@
lfix = (libver & 0x000ff000) >> 12;
if ( (headerver & mask) == (libver & mask) && lfix >= hfix)
diff --git a/openbsd-compat/regress/opensslvertest.c b/openbsd-compat/regress/opensslvertest.c
-index d50066609..aeccc107b 100644
+index 99c894418..351df4374 100644
--- a/openbsd-compat/regress/opensslvertest.c
+++ b/openbsd-compat/regress/opensslvertest.c
-@@ -37,6 +37,7 @@ struct version_test {
-
+@@ -28,6 +28,7 @@ struct version_test {
+ } version_tests[] = {
/* built with 1.0.1b release headers */
{ 0x1000101fL, 0x1000101fL, 1},/* exact match */
+ { 0x1000101fL, 0x10001010L, 1}, /* different status: ok */
{ 0x1000101fL, 0x1000102fL, 1}, /* newer library patch version: ok */
{ 0x1000101fL, 0x1000100fL, 1}, /* older library patch version: ok */
{ 0x1000101fL, 0x1000201fL, 1}, /* newer library fix version: ok */
+@@ -48,6 +49,7 @@ struct version_test {
+
+ /* built with 3.0.1 release headers */
+ { 0x3010101fL, 0x3010101fL, 1},/* exact match */
++ { 0x3010101fL, 0x30101010L, 1}, /* different status: ok */
+ { 0x3010101fL, 0x3010102fL, 1}, /* newer library patch version: ok */
+ { 0x3010101fL, 0x3010100fL, 1}, /* older library patch version: ok */
+ { 0x3010101fL, 0x3010201fL, 1}, /* newer library fix version: ok */
diff -Nru openssh-9.2p1/debian/patches/openssl-3-abi-compatibility-test.patch openssh-9.2p1/debian/patches/openssl-3-abi-compatibility-test.patch
--- openssh-9.2p1/debian/patches/openssl-3-abi-compatibility-test.patch 1970-01-01 01:00:00.000000000 +0100
+++ openssh-9.2p1/debian/patches/openssl-3-abi-compatibility-test.patch 2025-07-24 15:16:43.000000000 +0100
@@ -0,0 +1,61 @@
+From a8d557cac510e68ca2a27f691044fd588b8ef32a Mon Sep 17 00:00:00 2001
+From: Darren Tucker <dtucker@dtucker.net>
+Date: Tue, 9 May 2023 17:12:50 +1000
+Subject: Update OpenSSL compat test for 3.x.
+
+Origin: upstream, https://anongit.mindrot.org/openssh.git/commit/?id=47742c513e4e045ecc985c6483fc5c8b050acda2
+Bug-Debian: https://bugs.debian.org/1109742
+Last-Update: 2025-07-24
+
+Patch-Name: openssl-3-abi-compatibility-test.patch
+---
+ openbsd-compat/regress/opensslvertest.c | 29 +++++++++++++++++--------
+ 1 file changed, 20 insertions(+), 9 deletions(-)
+
+diff --git a/openbsd-compat/regress/opensslvertest.c b/openbsd-compat/regress/opensslvertest.c
+index d50066609..99c894418 100644
+--- a/openbsd-compat/regress/opensslvertest.c
++++ b/openbsd-compat/regress/opensslvertest.c
+@@ -26,15 +26,6 @@ struct version_test {
+ long libver;
+ int result;
+ } version_tests[] = {
+- /* built with 0.9.8b release headers */
+- { 0x0090802fL, 0x0090802fL, 1}, /* exact match */
+- { 0x0090802fL, 0x0090804fL, 1}, /* newer library fix version: ok */
+- { 0x0090802fL, 0x0090801fL, 1}, /* older library fix version: ok */
+- { 0x0090802fL, 0x0090702fL, 0}, /* older library minor version: NO */
+- { 0x0090802fL, 0x0090902fL, 0}, /* newer library minor version: NO */
+- { 0x0090802fL, 0x0080802fL, 0}, /* older library major version: NO */
+- { 0x0090802fL, 0x1000100fL, 0}, /* newer library major version: NO */
+-
+ /* built with 1.0.1b release headers */
+ { 0x1000101fL, 0x1000101fL, 1},/* exact match */
+ { 0x1000101fL, 0x1000102fL, 1}, /* newer library patch version: ok */
+@@ -44,6 +35,26 @@ struct version_test {
+ { 0x1000101fL, 0x1010101fL, 0}, /* newer library minor version: NO */
+ { 0x1000101fL, 0x0000101fL, 0}, /* older library major version: NO */
+ { 0x1000101fL, 0x2000101fL, 0}, /* newer library major version: NO */
++
++ /* built with 1.1.1b release headers */
++ { 0x1010101fL, 0x1010101fL, 1},/* exact match */
++ { 0x1010101fL, 0x1010102fL, 1}, /* newer library patch version: ok */
++ { 0x1010101fL, 0x1010100fL, 1}, /* older library patch version: ok */
++ { 0x1010101fL, 0x1010201fL, 1}, /* newer library fix version: ok */
++ { 0x1010101fL, 0x1010001fL, 0}, /* older library fix version: NO */
++ { 0x1010101fL, 0x1020001fL, 0}, /* newer library minor version: NO */
++ { 0x1010101fL, 0x0010101fL, 0}, /* older library major version: NO */
++ { 0x1010101fL, 0x2010101fL, 0}, /* newer library major version: NO */
++
++ /* built with 3.0.1 release headers */
++ { 0x3010101fL, 0x3010101fL, 1},/* exact match */
++ { 0x3010101fL, 0x3010102fL, 1}, /* newer library patch version: ok */
++ { 0x3010101fL, 0x3010100fL, 1}, /* older library patch version: ok */
++ { 0x3010101fL, 0x3010201fL, 1}, /* newer library fix version: ok */
++ { 0x3010101fL, 0x3010001fL, 1}, /* older library fix version: ok */
++ { 0x3010101fL, 0x3020001fL, 1}, /* newer library minor version: ok */
++ { 0x3010101fL, 0x1010101fL, 0}, /* older library major version: NO */
++ { 0x3010101fL, 0x4010101fL, 0}, /* newer library major version: NO */
+ };
+
+ void
diff -Nru openssh-9.2p1/debian/patches/openssl-3-abi-compatibility.patch openssh-9.2p1/debian/patches/openssl-3-abi-compatibility.patch
--- openssh-9.2p1/debian/patches/openssl-3-abi-compatibility.patch 1970-01-01 01:00:00.000000000 +0100
+++ openssh-9.2p1/debian/patches/openssl-3-abi-compatibility.patch 2025-07-24 15:16:43.000000000 +0100
@@ -0,0 +1,60 @@
+From 12494bc6c4da1fc4a29b0f026f83d8c030d8ac35 Mon Sep 17 00:00:00 2001
+From: Darren Tucker <dtucker@dtucker.net>
+Date: Mon, 8 May 2023 20:12:59 +1000
+Subject: Handle OpenSSL >=3 ABI compatibility.
+
+Beyond OpenSSL 3.0, the ABI compatibility guarantees are wider (only
+major must match instead of major and minor in earlier versions).
+bz#3548, ok djm@
+
+Origin: upstream, https://anongit.mindrot.org/openssh.git/commit/?id=b7afd8a4ecaca8afd3179b55e9db79c0ff210237
+Bug-Debian: https://bugs.debian.org/1109742
+Last-Update: 2025-07-24
+
+Patch-Name: openssl-3-abi-compatibility.patch
+---
+ openbsd-compat/openssl-compat.c | 20 +++++++++++---------
+ 1 file changed, 11 insertions(+), 9 deletions(-)
+
+diff --git a/openbsd-compat/openssl-compat.c b/openbsd-compat/openssl-compat.c
+index a37ca61bf..6c65003f2 100644
+--- a/openbsd-compat/openssl-compat.c
++++ b/openbsd-compat/openssl-compat.c
+@@ -33,10 +33,10 @@
+
+ /*
+ * OpenSSL version numbers: MNNFFPPS: major minor fix patch status
+- * We match major, minor, fix and status (not patch) for <1.0.0.
+- * After that, we acceptable compatible fix versions (so we
+- * allow 1.0.1 to work with 1.0.0). Going backwards is only allowed
+- * within a patch series.
++ * Versions >=3 require only major versions to match.
++ * For versions <3, we accept compatible fix versions (so we allow 1.0.1
++ * to work with 1.0.0). Going backwards is only allowed within a patch series.
++ * See https://www.openssl.org/policies/releasestrat.html
+ */
+
+ int
+@@ -48,15 +48,17 @@ ssh_compatible_openssl(long headerver, long libver)
+ if (headerver == libver)
+ return 1;
+
+- /* for versions < 1.0.0, major,minor,fix,status must match */
+- if (headerver < 0x1000000f) {
+- mask = 0xfffff00fL; /* major,minor,fix,status */
++ /*
++ * For versions >= 3.0, only the major and status must match.
++ */
++ if (headerver >= 0x3000000f) {
++ mask = 0xf000000fL; /* major,status */
+ return (headerver & mask) == (libver & mask);
+ }
+
+ /*
+- * For versions >= 1.0.0, major,minor,status must match and library
+- * fix version must be equal to or newer than the header.
++ * For versions >= 1.0.0, but <3, major,minor,status must match and
++ * library fix version must be equal to or newer than the header.
+ */
+ mask = 0xfff0000fL; /* major,minor,status */
+ hfix = (headerver & 0x000ff000) >> 12;
diff -Nru openssh-9.2p1/debian/patches/series openssh-9.2p1/debian/patches/series
--- openssh-9.2p1/debian/patches/series 2025-05-08 11:54:24.000000000 +0100
+++ openssh-9.2p1/debian/patches/series 2025-07-24 15:16:43.000000000 +0100
@@ -16,6 +16,8 @@
ssh-argv0.patch
doc-hash-tab-completion.patch
ssh-agent-setgid.patch
+openssl-3-abi-compatibility.patch
+openssl-3-abi-compatibility-test.patch
no-openssl-version-status.patch
gnome-ssh-askpass2-icon.patch
systemd-readiness.patch
Reply to: