Bug#241496: Fw: Undelivered Mail Returned to Sender
On 30/03/11 20:15 +0100, Colin Watson wrote:
> On Wed, Mar 30, 2011 at 06:31:53PM +0300, Phil Carmody wrote:
> > On 30/03/11 12:23 +0100, Colin Watson wrote:
> > > On Wed, Mar 30, 2011 at 11:49:48AM +0300, Phil Carmody wrote:
> > > > I performed the upgrade to 1:5.5p1-6 months ago, I don't know what
> > > > version I was coming from. All I know is that my customised
> > > > sshd_config was overwritten during the upgrade. As this 'upgrade'
> > > > resulted in the opening of world-facing ports against my will, I
> > > > consider it to not just be a grave error as it's not just a loss
> > > > of data, but also a security issue.
> > >
> > > The previous comments from Matthew and myself in this bug still stand.
> > > Please provide:
> > >
> > > * /var/log/dpkg.log* (which may show what version you were coming
> > > from)
> >
> > -- 8< --
> > # zgrep openssh /var/log/dpkg.log*
> > /var/log/dpkg.log.1:2011-02-09 03:06:06 upgrade openssh-server 1:5.1p1-5 1:5.5p1-6
> [...]
> > > * /etc/ssh/sshd_config
> >
> > Well, right now it's the default from the package, that's the bug, see.
>
> I'd really like an exact copy, just to make sure. The default text can
> change from version to version, and it would be helpful to be able to
> double-check it against my revision control repository.
The 'before' is alas lost in history. However, this maybe turns it
into a reverse chess puzzle - what must the file have been like in
order for it to be modified but not backed up?
> > Subsystem sftp /usr/lib/openssh/sftp-server
> >
> > UsePAM yes
>
> (For example, the lack of a comment here indicates that the creation of
> the original version of this file predated 1:5.2p1-1.)
Entirely believale that it's several sub-sub-versions back.
My g/f and I have pieced together a memory that suggests the file was
over-written before Nov 20 2010. Alas I don't have any provably
gap-less logs going back that far. However, the oldest dpkg.log does
go all the way to 2010-11-10. I update/upgrade irregularly, I am
prepared to believe that my dpkg.logs are contiguous. (But I had had
logging issues due to massive hack/spam attacks filling /var/log, and
forcably deleted hundreds of megabytes of logs several times.)
I did write a webpage documenting what I was changing in various files
including sshd_config, and it looks like that was last edited on
2010-08-19, and I don't think I revisited it after its initial edit.
Therefore it seems like the version that apparently did the overwrite
was pulled some time between August and November last year. Alas that
gives us no idea about which version was (modified by me and)
overwritten.
...
> Now, your dpkg.log indicates that the previous version was 1:5.1p1-5.
> Can you confirm that all was well when you had that version?
It's been wrong for at least 4 months. It's just that every time we
ssh'ed in on the wrong (=default) port we said "must fix that", but we
were on holiday or in the pub, or basically anywhere apart from a good
place to do system administration. One time I decided to have a quick
look I accidentally disabled ssh on all ports, and also accidentally
logged off. Ooops. That caused my g/f to have to read her mail via an
unusual route, and hence we managed to work out that date via logs on
other machines, the Nov 20 above.
> Working through the logic: your report (and the other similar reports in
> this bug log) states that you previously had /etc/ssh/sshd_config, so
> you must have entered the first if block in create_sshdconfig. You
> don't have /etc/ssh/sshd_config.dpkg-old, which rules out the block that
> upgrades from pre-3.7 versions. dpkg.log indicates that $oldversion
> should have been 1:5.1p1-5,
Or earlier.
> so the only other piece of code in
> create_sshdconfig that you should have hit was the one to handle
> LogLevel SILENT. That calls this function:
>
> fix_loglevel_silent() {
> if [ "$(get_config_option LogLevel)" = SILENT ]; then
> set_config_option LogLevel QUIET
> fi
> }
>
> (Do you remember having set LogLevel SILENT?)
Wouldn't have touched that, I'm pretty sure. I like defaults wherever possible.
> get_config_option is read-only. set_config_option does:
>
> set_config_option() {
> option="$1"
> value="$2"
>
> perl -le '[... code to set $option to $value ...]' \
> "$option" "$value" \
> < /etc/ssh/sshd_config > /etc/ssh/sshd_config.dpkg-new
> chown --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
> chmod --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
> mv /etc/ssh/sshd_config.dpkg-new /etc/ssh/sshd_config
> }
>
> This seems straightforward enough; it's running under 'set -e', so any
> failure would cause the whole script to exit. Even if that protection
> failed somehow, the worst case I can imagine would be ending up with no
> sshd_config at all - certainly not good, but not the symptom you
> reported.
Unless that is the problem. Ran once and failed. Then a "let's see if
it works if I run it again" mentality leads to a second run where it
doesn't do the [ -e ] block any more, and writes a fresh one?
If it's possible at all, it's very possible on my system. I've had
out-of-space failures on that machine a few times when upgrading (I
partitioned it about 5 years ago, things have grown a lot since then),
so I'm in the habit of doing a little tidy up and just trying again.
> So far, I haven't been able to think of any logical failure in any of
> this that might cause local changes to be erased in such a way that the
> default file would be put back in place. Can you confirm that the file
> you ended up with is I'm happy to install
> code to ensure that there's a backup kept before calling any of the
> targeted-upgrade functions,
I do prefer the idea of the -old being /exactly/ what the file was before
the upgrade, certainly.
> but I don't think that will really fix the
> problem here. That leaves me still rather stumped.
I'm in part a bugfixing firefighter in ${DAYJOB}. I know all about
being stumped. Nothing beats being able to reproduce the error. All
we're doing now is forensics, which is a poor substitute.
However, I'm now hyper-sensitive to the packages that will be
upgraded, and what I might try to do is sneak in a quick set -x into
the postinst and see what happens, next time openssh-server is
available.
> I don't suppose you have /var/log/apt/term.log*? The output from dpkg
> when it performed this upgrade might still be in one of those files.
Alas that file's only 5 days old.
> I wonder, also, if some other package that you have locally installed is
> interfering with things: another package fiddling with
> /etc/ssh/sshd_config could explain this bug when looking at
> openssh-server alone doesn't, and perhaps explain why I've only had a
> few reports of this when it seems as though it should happen to either
> everybody or nobody. Could you run 'dpkg -S sshd_config' and 'grep -r
> sshd_config /var/lib/dpkg/info' to look for this kind of thing?
I run a very minimal install, and everything's debian apart from a few
programs that I've written myself. dpkg -S finds nothing, thank goodness,
and the grep only picks out the obvious:
-- 8< --
/var/lib/dpkg/info/openssh-server.postrm: rm -f /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.list:/usr/share/doc/openssh-client/examples/sshd_config
/var/lib/dpkg/info/openssh-server.list:/usr/share/man/man5/sshd_config.5.gz
/var/lib/dpkg/info/openssh-server.config: [ -f /etc/ssh/sshd_config ] || return
/var/lib/dpkg/info/openssh-server.config: /etc/ssh/sshd_config 2>/dev/null
/var/lib/dpkg/info/openssh-server.config:if [ -e /etc/ssh/sshd_config ]
/var/lib/dpkg/info/openssh-server.postinst: [ -f /etc/ssh/sshd_config ] || return
/var/lib/dpkg/info/openssh-server.postinst: /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.postinst: < /etc/ssh/sshd_config > /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chown --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chmod --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: mv /etc/ssh/sshd_config.dpkg-new /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.postinst: < /etc/ssh/sshd_config > /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chown --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chmod --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: mv /etc/ssh/sshd_config.dpkg-new /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.postinst: < /etc/ssh/sshd_config > /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chown --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chmod --reference /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: mv /etc/ssh/sshd_config.dpkg-new /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.postinst: if [ -e /etc/ssh/sshd_config ] ; then
/var/lib/dpkg/info/openssh-server.postinst: ! grep -iq ^UsePAM /etc/ssh/sshd_config) || \
/var/lib/dpkg/info/openssh-server.postinst: /etc/ssh/sshd_config ; then
/var/lib/dpkg/info/openssh-server.postinst: echo -n 'Upgrading sshd_config (old version in .dpkg-old) ...'
/var/lib/dpkg/info/openssh-server.postinst: cp -a /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-old
/var/lib/dpkg/info/openssh-server.postinst: /etc/ssh/sshd_config > /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: echo >> /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: echo 'UsePAM yes' >> /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chown --reference /etc/ssh/sshd_config \
/var/lib/dpkg/info/openssh-server.postinst: /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: chmod --reference /etc/ssh/sshd_config \
/var/lib/dpkg/info/openssh-server.postinst: /etc/ssh/sshd_config.dpkg-new
/var/lib/dpkg/info/openssh-server.postinst: mv /etc/ssh/sshd_config.dpkg-new /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.postinst: #Preserve old sshd_config before generating a new one
/var/lib/dpkg/info/openssh-server.postinst: if [ -e /etc/ssh/sshd_config ] ; then
/var/lib/dpkg/info/openssh-server.postinst: mv /etc/ssh/sshd_config /etc/ssh/sshd_config.dpkg-old
/var/lib/dpkg/info/openssh-server.postinst: cat <<EOF > /etc/ssh/sshd_config
/var/lib/dpkg/info/openssh-server.postinst:# See the sshd_config(5) manpage for details
/var/lib/dpkg/info/openssh-server.preinst:# access limits that are hard to express in sshd_config.
/var/lib/dpkg/info/openssh-server.md5sums:a8f85459802674a416b903c8be7774d6 usr/share/doc/openssh-client/examples/sshd_config
/var/lib/dpkg/info/openssh-server.md5sums:24e6a2d6f56d5fd52651db030a4124bb usr/share/man/man5/sshd_config.5.gz
-- 8< --
Phil
--
I find the easiest thing to do is to k/f myself and just troll away
-- David Melville on r.a.s.f1
Reply to: