Bug#759491: Defining pseudo-essential

[Guillem Jover <guillem@debian.org>]

[ Only found time to finish up the reply I started weeks ago now, so I
  might lost my train of thought from then. :/ ]

Hi!

On Wed, 2014-08-27 at 18:02:29 -0700, Russ Allbery wrote:
> Guillem Jover <guillem@debian.org> writes:
> > On Wed, 2014-08-27 at 09:22:42 -0700, Russ Allbery wrote:
> 
> >> @@ -1321,6 +1323,76 @@ zope.
> >>           mailing list and a consensus about doing that has been
> >>           reached.
> >>         </p>
> >> +
> >> +       <sect1 id="pseudo-essential">
> >> +         <heading>Pseudo-essential packages</heading>
> >> +
> >> +         <p>
> >> +           Sometimes, specific libraries, commands, or services are
> >> +           essential in the sense that they must be usable at all times,
> >> +           even if not configured, but the package that provides these
> >> +           interfaces may change.  The two most common cases are
> >> +           essential shared libraries, which must not be explicitly
> >> +           declared essential so that they can be removed during
> >> +           transitions, or commands and services that can be provided by
> >> +           the user's choice of multiple packages.  A typical example of
> >> +           the latter is <prgn>awk</prgn>: the presence of a
> >> +           working <prgn>awk</prgn> command is part of the Essential set,
> >> +           but there are multiple acceptable implementations
> >> +           of <prgn>awk</prgn> that can provide this required command.
> >> +         </p>
> 
> > The way I've always thought and understood pseudo-essential has been any
> > packages in the set of packages pulled by Essential:yes ones but not
> > marked as Essential:yes. Because they cannot be removed either, even
> > with alternatives, you always need at least one.
> 
> Hm.  One of the points of this wording is to actually agree on a
> definition of pseudo-essential.  I think it would be more useful if we
> could reserve "pseudo-essential" for packages that provide essential
> functionality but cannot be marked as essential due to one of the various
> limitations in how that field works.

Sure, the problem is that because Essential has different facets,
reserving pseudo-essential for packages that honor all of them makes
it only appliable to awk, which seems a bit pointless to me.

The way I've always thought about pseudo-essential, seems to be more
unambiguous to me, as it has no exceptions, it just gives a name to
the other packages pulled in by the Essential:yes set (its complement),
and it makes it somewhat clear that they are not entirely essential
(pseudo-). But I'm not sure if others might find that convincing or
even useful. :)

> For the other packages that are dependencies of essential packages, maybe
> we should just use that: "dependencies of essential packages."  Although
> I'm not sure why we shouldn't declare some of those packages
> pseudo-essential and keep their functionality in the essential set.

As mentioned above, because I don't think many of the pseudo-essential
packages honor all Essential:yes facets, I don't think it makes sense
to promote them to be so. Most are just implementation details that
might change over time.

> > The second prominent case of pseudo-essential are private dependencies
> > of an Essential:yes or another pseudo-essential package. The awk case is
> > actually an exception rather than a common case.
> 
> The problem, of course, is that we have good no mechanism to detect the
> missing dependencies on these packages, so in practice the dependencies
> tend to get dropped, and they end up as pseudo-essential.

I think this is more out of misconceptions or lack of documentation
than accidental dependencies being dropped. And I'm talking from
recent experience with the xz-utils package (for example #582706)

But I agree that the fact that they are difficult or “impossible” to
uninstall makes it also difficult to detect any possible accidental
removal. (I'm not sure how we could go about trying to detect those.)

> > Not all pseudo-essential packages should be considered to be the same,
> > awk is an almost Essential pseudo-essential, but only by convention.
> 
> I was considering whether we should explicitly list the packages, like
> awk, that can be assumed and don't have to be depended on, but I'm not
> sure that makes sense to do.  Although if it's only awk, we could just do
> that.

Yeah, it should only really be awk, and I think it makes sense to list
it explicitly.

> > Because virtual packages cannot be Essential:yes it just gets pulled in
> > by base-files, so its functionality is expected to be available to any
> > other package (w/o an explicit dependency), but most (if not all) of the
> > rest of pseudo-essential packages are just private implementation
> > details, and should only be relied on by their direct dependers, like
> > xz-utils was for a while from dpkg, before it got switched to liblzma.
> 
> Do you think we should just say that all pseudo-essential packages another
> package uses must have a declared explicit dependency except awk, which is
> a special case?  I could do that.

Definitely (as I think I mentioned somehow in a cut-off paragraph :).

> > The problem is that the Essential field conveys several conflated
> > meanings and functions.
> 
> >  * It marks packages that are necessary for the system, at installation
> >    _and_ boot time, that's why it's made very hard to remove them.
> >  * It marks packages that are necessary for dpkg to be functional all
> >    the time, even when unpacked during upgrades, but this also applies
> >    for booting even on sudden system crashes and similar.
> >  * It's also used to avoid dependency loops; and because these are
> >    always going to be present, packages can actually get by with not
> >    listing them as direct dependencies (execept when versioned).
> 
> > The distinction on when a pseudo-essential needs to function as an
> > Essential:yes package is determined by its depender, so if it's
> > pulled through Pre-Depends then it should behave as an Essential:yes,
> > otherwise it'st just a required dependency.
> 
> Ah, that's a good distinction.  Is that reliable?

Well it should be! :) But that only defines what should behave how
from the depender PoV. As always, when such annotations are held in
the depender instead of the dependee, there might be a coordination
problem if both maintainers have not agreed explicitly on what is
required of the packages.

> Can I say that only
> Pre-Depends from an essential package creates a pseudo-essential package,
> but not a straight Depends?  (I think that makes libc6 not
> pseudo-essential, since I think it's just Depends, but I'm not positive.)

(libc6 is always pulled by Pre-Depends due to at least dpkg.)

Ok, let's check the actual list of Essential:yes + pseudo-essential on
amd64 (as of two weeks ago more or less):

Essential:yes («aptitude search '~E !apt (~ramd64 |~rall)'»)
=============

base-files
base-passwd
bash
bsdutils
coreutils
dash
debianutils
diffutils
dpkg
e2fsprogs
findutils
grep
gzip
hostname
init
libc-bin
login
mount
ncurses-base
ncurses-bin
perl-base
sed
sysvinit-utils
tar
util-linux

pseudo-essential (+- «aptitude search '~prequired !~E (~ramd64 |~rall)'»)
================

This is a quick approximation, plus manually added packages, from those
I only consider pseudo-essential the ones being pulled by Pre-Depends
and Depends, the rest just need their priorities updated to make the
set self-consistent, I'm ignoring proposals to trim Essential:yes here.

Pre-Depended
------------
debconf
e2fslibs
libacl1
libattr1
libblkid1
libc6
libcomerr2
liblzma5
libmount1
libncurses5
libpam-modules
libpam-modules-bin
libpam-runtime
libpam0g
libpcre3
libselinux1
libsmartcols1
libss2
libtinfo5
libuuid1
mawk
multiarch-support
zlib1g

Pre-Depended (need priority bumped)
------------
libaudit1
libdb5.3
libsystemd-id128-0
libsystemd-journal0

Pre-Depended (by way of systemd-sysv 214/exp, need priority bumped)
------------
systemd

Depended
--------
gcc-4.9-base
initscripts
insserv
libgcc1
libsepol1
lsb-base
sensible-utils
startpar
sysv-rc
tzdata

Depended (need priority bumped)
--------
libaudit-common
libcap2
libdebconfclient0
libgcrypt11
libgpg-error0

Depended (by way of systemd-sysv 214/exp, need priority bumped)
--------
acl
adduser
libcap2-bin
libcryptsetup4
libkmod2
libncursesw5
libprocps3
libsystemd0
passwd
procps
udev

Recommended (need priority lowered)
-----------
debconf-i18n
liblocale-gettext-perl
libtext-charwidth-perl
libtext-iconv-perl
libtext-wrapi18n-perl

Legacy (need priority lowered)
------
gcc-4.7-base
gcc-4.8-base

> > Packages that divert, provide alternatives, etc might need to behave
> > like an Essential:yes package (to not break the system's expectations),
> > but I'd not consider them to be pseudo-essential, as long as the
> > dependency is not initiated by an Essential:yes or a pseudo-essential
> > package.
> 
> Well, the reason why I wanted to include them in the pseudo-essential set
> is that they're required to have that same "works properly even if not
> configured" behavior.  That is pretty much the defining characteristic of
> pseudo-essential, isn't it?  (Particularly if we say that packages are
> required to depend on pseudo-essential packages they use.)

Not in my mind, no. Precisely because Essential packages are not
depended upon, the depending packages cannot specify how much they
depend on them, or at what time they need to be functional (Depends
vs Pre-Depends). But as pseudo-essential get explicitly depended,
the dependers can specify what are their expectations, and those
might even change over time.

Thanks,
Guillem