Hi Santiago,
Quoting Santiago Vila (2019-03-12 23:43:02)
> > > Do any of them still don't know that base-passwd should be configured
> > > first because otherwise any other package using root (be it base-files or
> > > any other) will fail? I think this was already settled in the last
> > > discussion we had about this several years ago.
> > multistrap doesn't take care of this and you can provoke a
> > base-files.postinst failure this way.
> In such case I would say that as a bootstrapping tool it's not doing its job
> properly.
>
> The first rule of a bootstrapping tool is that it has to work.
> (And there are actually no other rules. As far as it does its job,
> you are allowed to do all sorts of dirty hacks).
>
> Bootstrapping tools exist so that we don't have to worry about
> dependencies on essential packages. It has always been my opinion that
> if we start to do hacks here and there so that bootstrapping tools
> work properly, we are already doing it wrong.
>
> > [,,,] Just because debootstrap encodes a ton of hacks to make things barely
> > work (and break every so often) doesn't mean we have to maintain them until
> > eternity.
> I would say: Just because people writing new bootstrapping tools seem to
> forget the lessons learned from previous bootstrapping tools, we have to
> learn again what bootstrapping really means: It's not adding hacks to the
> normal packages, it's concentrating all the hacks in the bootstrapping tools,
> so that we can keep ordinary packages clean of hacks.
>
> (Or at least that's what I think it was the idea behind the essential
> definition).
I'm writing as the author of another "bootstrapping tool", namely mmdebstrap
which currently (really accidentally only) does not suffer from this bug but
might happen to be affected by it in the future.
I would especially like to challenge your view that (if I understood it
correctly) all the magic for getting from nothing to a Debian chroot should be
encoded in a bootstrapping script or program. You are right that currently this
is how it's done but I'd like to convince you that maybe it would be worth
reevaluating how we are currently doing this because maybe there is a better
way to do it.
Let me also point out here, that debootstrap has a place even in a world where
all the magic (the things you call hacks twice above) were moved into the
individual packages instead of being encoded in a central bootstrapping script.
This is because we will still need something that would be considered a hack
from the Debian perspective when we create a Debian chroot on systems that are
not Debian. My tool mmdebstrap for example will likely only work on Debian or
derivatives because it requires apt to be installed.
Lets look at the bigger picture: we have an archive with multiple software
components we call packages on some remote server and locally we have a fresh
empty directory which we somehow want to populate with the functionality that
these software components provide. The question is: what would be the best way
to do it? Or what would be a good engineering solution to this problem?
I claim (and that's where I want to challenge your view) that a better
engineering solution than the status quo in Debian would be to let each of
these software components (or packages) encode their requirements in a way that
lets a bootstrapping tool solve the installation task without involving the
other components and thus solve the chicken-and-egg problem of bootstrapping.
If this were possible, we can achieve a number of benefits which I'll be
getting to in a bit. That this *is* possible is shown by [guillem]'s mail to
this bug report. Another approach would be to use functionality of packages
installed on the outside of the chroot to fulfill requirements of the packages
inside the new chroot [detached chroot].
[guillem] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=924401#35
[detached chroot] https://wiki.debian.org/Teams/Dpkg/Spec/InstallBootstrap
So if you can accept that it is technically achievable to create a component
based operating system with only very little "magic" in the bootstrapping tools
but all required information about interaction of even essential packages in
the components itself, then we can achieve multiple benefits:
- solving dependencies is a complex matter [sudoku] and implementing a
software that can handle NP-complete [np-complete] problem instances is not
trivial (see code of apt, aptitude, dose, aspcud...). Existing tools like
(c)debootstrap only understand just enough about package dependencies that
they happen to be able to install a very minimal system. The status quo is
to not add many packages via the --include option but install them via apt
inside the chroot after debootstrap is done (#768062, #878961). If we could
use existing tools that have a better understanding of Debian dependencies
(like apt, aptitude, dose...) for bootstrapping a Debian chroot then we
could:
* create chroots with multiple architectures, honoring multi-arch
* create chroots with packages from multiple mirrors
* add all desired packages in the initial program invocation instead of
requiring a multi-step process that requires some guess work about
which dependency situations might be too hard
Especially supporting multiple mirrors would be a great benefit because it
would allow to create a chroot of a Debian stable release that includes the
updates and security mirrors *and* is up-to-date without requiring extra
steps (#543819, #762222) Or how about multiple suites? (#690210)
[sudoku] https://lists.gforge.inria.fr/pipermail/dose-devel/2013-December/000059.html
[np-complete] https://xavierleroy.org/publi/edos-frcss06.pdf
- The idea behind component based software distributions like Debian is, that
each package declares how it relates to other packages in the distribution.
To solve dependency problems and perform package installations we do not
maintain a central registry or script but it is up to each individual
package of how it can be installed via the dependencies, conflicts and
maintainer scripts it provides. We would generally think it wrong if we
would solve a dependency bug by implementing that apt now encodes that to
install a specific package XY it has to do some special thing. But so far,
this kind of argument about good engineering practice is not applied to the
debootstrap level. There, we (currently) accepted that it is okay that
debootstrap employs all sorts of hacks to be able to populate a directory
with a minimal chroot and we even accept that sometimes individual packages
have to change just because debootstrap would otherwise not be able to
handle it. This kind of design has numerous downsides:
* debootstrap has to carry workarounds even long after they are not
anymore required by packages in the current stable release so that
old-(old-)stable releases can still be debootstrapped
* package maintainers have to go through debootstrap before an issue
can be fixed instead of just modifying their package
* debootstrap has to implement these workarounds for each derivative
and each derivative has to go to the debootstrap maintainer before
their distribution is supported
* workarounds can be architecture specific, so the set of supported
architectures is (again) not limited by what is implemented in the
packages but in debootstrap
* all these workarounds must be implemented by all debootstrap-like
software. So in essence, debootstrap, cdebootstrap, multistrap, and
mmdebstrap will all have to implement all workarounds for all
architectures and derivatives and try to all keep up to date
But what if we would extend the clean design choice that we made for normal
package installation to the debootstrap-phase? We would then be able to reap
the benefits of this approach to developing a component-based software
distribution even at the very early debootstrap-phase:
* debootstrapping Debian and derivatives that choose to follow the
design that we yet have to decide upon could create a chroot of Debian
or a derivative just by supplying one (or more) mirror urls to the
debootstrap tool. All distribution-specific magic would be handled
by *the packages* and their metadata alone.
* workarounds would live in the packages themselves and would only
live there for as long as they are required for the distribution
they are in and for the architecture they are for
* not one single point that de-facto determines how the debootstrap
process has to look like and thus no argument like: "debootstrap
cannot do this and that, so we have to add this workaround to that
package"
* process is defined by declarative metadata and not by turing-complete
scripts
* downstream derivatives do not need to touch debootstrap if they
want to change how their system is set up but only change the
affected packages -- this keeps the code local to the place
that is affected
* multiple debootstrap-like tools do not all have to implement the same
magic multiple times and all have to be kept up-to-date when
anything in any derivative or architecture changes
* easier bootstrapping of new architectures from zero as we get rid
of all Essential:yes packages have to already exist
What I'm listing here are just the benefits of component based software
design and we currently apply it with great success to normal package
installation for many, many reasons that we all accept and like. My point
is: why not extend all these good practices of component based software
distributions to the debootstrap phase? It is certainly technically
possible.
I'm not advocating for doing "hacks here and there so that bootstrapping tools
work properly" as you put it but that we think about the question of whether
maybe there is a better way to populate an empty directory with software
components that does not require as much magic as currently has to be supplied
by tools like debootstrap. The result would not be "hacks here and there" but a
cleaner and more robust setup procedure.
So what I want you to take away from this long mail is: maybe we should
re-think the way we are currently creating a Debian chroot because maybe there
exists a different approach that would give us benefits that are actually
important to our users and make maintaining the universal operating system
easier?
Thanks!
cheers, josch
Attachment:
signature.asc
Description: signature