merged-/usr-via-symlinks damage control (was Re: usrmerge -- plan B?)

[Guillem Jover <guillem@debian.org>]

Hi!

On Tue, 2018-11-20 at 22:16:17 +0100, Adam Borowski wrote:
> Thus, it seems to me that the plan A for usrmerge has serious downsides for
> dubious benefits.  What about the plan B I described above?

So, people still seem to be conflating merged-/usr (the filesystem
layout) with the different ways to deploy it. Personally I do agree
that merged-/usr has benefits, and that's why I've f.ex. been switching
the shared library packages I maintain to do a proper and correct switch
by installing their files in /usr/lib instead of /lib.

The underlaying problem with the merged-/usr-via-symlinks is not
really that it has generated bugs, many transitions we perform incur
those, for the better. The problem is that it has generated those when
it was really not necessary in the first place, has made things
unnecessarily more complicated for everyone, and it's a terrible hack
trying to reap "quick" benefits at the cost of everyhing else, with
long lasting consequences even after a full migration to /usr would
have been finished. And here's why:

1) The merged-/usr-via-symlinks deployment method used by both
   debootstrap and usrmerge, means that those systems will get
   permanent filesystem aliasing problems, forever. Even when all
   files might have been moved to their /usr counterparts in the
   packages! This is due to the fact that different pathnames can
   end up pointing (before any canonicalization!) to the same dentry.

   This does not only affect dpkg (dpkg, dpkg-divert, dpkg-trigger,
   etc), and update-alternatives (in a worse way), but any program
   that uses pathanmes in the filesystem as keys in databases f.ex.

2) It bypasses the packaging system understanding of what is on the
   filesystem and creates mismatches between what's on binary packages
   and what's on disk.

3) Switching packages to the merged-/usr layout could have been
   accomplished automatically via debhelper for a coverage of around
   99% (?) of the archive. With something along the lines of:

     ,---
     D=debian/tmp
     for d in /bin /sbin /lib; do
       for p in $(find $D/$d ! -type d); do
         b="${p##$D/}"
         m="$D/usr$b"
         if [ ! -e "$m" ]; then
           mkdir -p "$(dirname $m)"
           mv "$p" "$m"
           ln -s "${m##$D}" "$p"
         fi
       done
     done
     `---

   With the property that it would handle gracefully all cases were the
   maintainer has checked that no compat symlinks are required and has
   then progressively moved the pathname installation to their final
   destination under /usr.

4) Due to having to support the broken merged-/usr-via-symlinks
   deployment, when we want to move the contents of the binary
   packages to the merged-/usr layout, we require now to include tons
   of logic in (probably new) maintainer scripts, when we have been
   trying to remove them altogether. :( With even more files untracked
   by dpkg itself, bypassing the packaging system even more, when the
   compat symlinks could have been shipped in the binary packages.

5) Most new installations will not even benefit from this hacky and
   rushed deployment, as long as they do not use a separate parition
   for /usr!

6) The merged-/usr-via-symlinks deployment hack is irreversible right
   now.

This is all really terrible engineering. :(


So, as part of damage control I'm going to:

  - include the Build-Tainted-By patches into dpkg 1.19.5.
  - include a bug-script in dpkg for reportbug to report whether the
    system has been broken by the merged-/usr-via-symlinks hack.
  - file a bug on reportbug to request reporting that for all packages.
  - file a bug on base-installer to request an option to install
    non-broken systems due to merged-/usr-via-symlinks.

I'm also pondering whether to implement checks for the broken
merged-/usr-via-symlinks within «dpkg --audit», even though I don't
really want to, as that would imply hardcoding Debian-specific
pathnames in there. :(

And I'm probably going to end up writing a unmerge-usr-via-symlinks
script so that people with damaged systems can go back somewhat to a
sane state, and to open the possibility for a fully automated migration
to a proper and correct merged-/usr w/o all the problems above.

Thanks,
Guillem