Re: Is running dpkg-buildpackage manually from the command line forbidden?
On Thu, 16 Jan 2020 at 08:50:25 -0800, Daniel Schepler wrote:
> I've been running a manual test bootstrap of Debian (starting with
> cross-compiled packages amd64 -> i386 up to the point I was able to
> install debhelper), and posting a few bugs I've found along the way.
> These are where I found that having extra packages installed during
> the dpkg-buildpackage run either failed or resulted in broken
> packages. (Some examples of the type of thing I mean: #948522,
> #887902.)
>
> However, I've been getting push back on some of these, with
> maintainers of the opinion that it isn't actually a bug. So, I
> thought I'd consult here to get more opinions on whether these are
> true bugs, or whether I'm at fault for trying to run dpkg-buildpackage
> manually instead of using it through pbuilder or sbuild.
I would say it's often a bug, but usually (always?) a non-RC bug, because
there's a straightforward workaround (build in a minimal environment).
Opening bugs as RC tends to put maintainers on the defensive (if nothing
else because RC bugs have automatic consequences, like autoremovals), so
it's often better to err on the side of opening bugs with a lower severity
(and escalating if there's later consensus that this should happen)
rather than starting with a severity that could be seen as unjustified.
FTBFS and broken packages when built in a non-minimal environment are
special cases of the more general problem of getting a different package
when built in a non-minimal environment, which is sometimes trivial
to solve, but somewhat intractable to detect and solve in general,
because there are so many different non-minimal environments - as Ansgar
mentioned, not just packages that we can capture in Build-Depends and
Build-Conflicts, but also the contents of directories like /usr/local,
/opt and ~/bin, and various properties of your system such as whether it
has undergone the /usr merge, whether alternatives have been reconfigured,
the filesystem you're running on, whether an LSM is in use, ...
I think it's best-practice for package maintainers to explicitly disable
optional features that are autodetected but aren't wanted in the package
(./configure --disable-weasels --without-libstoat, or equivalent), but
this isn't always possible without relatively extensive patches to the
upstream build system, which themselves carry a cost.
> The mere existence of the Build-Conflicts field [implies that we are
> meant to use it]
Yes to a point, but here's a somewhat devil's-advocate response to that:
Suppose we lived in a parallel timeline where there is a serious QA effort
put into making sure packages produce the same results, regardless of
how they are built, as long as their Build-Depends and Build-Conflicts
(and their -Arch, -Indep flavours) are satisfied.
In that world, one rational thing for maintainers of packages with
optional features to do would be to add an ever-increasing number of
Build-Conflicts (every time someone notices a package that can alter or
breaks the build). I could easily imagine it becoming no longer feasible
to build large sets of packages unless you use sbuild or equivalent,
because at least a few of those packages will have a Build-Conflicts on
something you needed for another of the packages - which might not even
be necessary any more, but as an individual package maintainer, there's
little incentive to *remove* Build-Conflicts on things that you yourself
don't use, particularly if insufficient Build-Conflicts are considered RC.
So, pragmatically, it might be a good idea to optimize away the steps
where we put lots of effort into QA, report lots of bugs, get them
fixed by addition of Build-Conflicts, and find ourselves having to use
a minimal build environment to get your desired set of packages built -
by using a minimal build environment to start with. And we're back to
the advice to use pbuilder or sbuild.
I don't think it's coincidence that there has been so much parallel
evolution between Debian's pbuilder and sbuild (among others), Fedora's
mock, openSUSE's obs-build, Flatpak apps' flatpak-builder, and lots of
projects' Docker-based build systems - getting a reliable and reproducible
build of a complex package/module in minimal environments is already quite
difficult, which makes the harder task of making it reliably reproducible
in arbitrary non-minimal environments look rather unappealing.
> It would be very difficult to impossible to test for every possible
> combination of packages that satisfy the Build-Depends. (Though I
> would think a vast majority of such bugs would be detected by a
> reproducible-builds type setup with one build being the standard
> minimal chroot, and the other build using a chroot with as many
> packages as possible installed.)
I think I dimly remember someone setting up "the buildd from hell"
which deliberately did this as a QA mechanism, but it doesn't seem to
have continued in any systematic way.
> requiring use of pbuilder or sbuild would place a
> burden on users who previously would have made local patches by a
> sequence of "apt-get source package; cd package-*/; edit;
> dpkg-buildpackage -b -uc; sudo apt-get install ../*.deb
I think in an ideal world, we'd have better tools for those users to build
modified packages in a way that more closely resembles what happens on
the production Debian infrastructure - which might for instance mean CI
services, or my vectis[1] tool, or sbuild-debian-developer-setup, or even
autopkgtest (which is really for testing packages, but as a side-effect,
it knows how to build packages in an environment that in practice is going
to be close-to-minimal).
pbuilder and the usual sbuild+schroot setup have the disadvantage
of requiring root privileges and crossing privilege boundaries,
but vectis uses virtual machines (in practice this means kvm group
membership or udev/logind uaccess, to get write access to /dev/kvm)
and as namespace/container stuff gets more powerful and more trusted, I'd
hope that we'll get a better ability to install build-dependencies and do
builds in unprivileged containers.
A justification for non-minimal build environments that I'm personally
more interested in is debugging, whether by users developing local patches
or by a package's maintainer. For successful, one-off builds, throwing
a .dsc or source directory at a build service and waiting for a .deb to
fall out is sufficient - but when the package isn't building correctly,
it's often useful to be able to try it interactively and iteratively
(without doing a full rebuild every time), in a non-minimal environment
that contains debugging tools. In this situation, you'll often end up
with a package that you are not entirely confident is reproducible,
and the next thing to do after your patch seems ready to propose is to
rebuild the package "properly" to confirm that it still works.
Re-bootstrapping seems a lot more like the "one-off build" use-case to me,
so I would recommend something sbuild-like.
smcv
[1] https://salsa.debian.org/smcv/vectis
Reply to: