packages that FTBFS twice in a row ...

[Andreas Beckmann <anbe@debian.org>]

Hi,

here are my observations after analyzing a few pbuilder --twice failures
in experimental. I only did full (source+arch+indep) builds.


The command sequence being tested is roughly:

1. debian/rules clean
2. dpkg-source
3. debian/rules build binary
4. debian/rules clean
5. dpkg-source
6. debian/rules build binary

1.-3. are utilized by regular builds and tested by normal archive-wide
rebuilds, therefore I'll only consider packages passing these steps.

Possible failure scenarios for the second build:

4. debian/rules clean
   There are bugs related to make distclean and debian/rules clean, they
   usually don't show up during 1. since the source tree is not in a
   configured state (no Makefile?) and the debian/ tree is not in a
   built state.

   a) make distclean may fail, probably an upstream bug
      * #884898, icu/experimental, parallel distclean race

   b) passes, but leaves new/modified files in the source tree
      * This is the majority of bugs, usually generated filed don't get
        deleted properly. The subsequent error from dpkg-source is
        pretty obvious.

   c) passes, but leaves new/modified in the debian/ tree
      * #884419, #884815: override_dh_clean does not run dh_clean
        resulting in debian/$PKG/ package build directories and
        debhelper logfiles being left in the tree. Subsequent
        dpkg-source will choke on unknown binary files under debian/
        There is a lintian check coming: #884817 "lintian: check for
        override_dh_clean target missing call to dh_clean".
      * there may be text-only changes in debian/ that don't trigger
        errors, but cause differing source package to be built in 2./5.

   d) passes, but deletes files that cannot be regenerated
      * #884706, #884813, #884819, #884889
        This passes the subsequent dpkg-source (since it will just warn
        on deleted files by default, which is very convient to get rid
        of generated files that are shipped in the upstream tarball),
        but will fail during the following build

5. dpkg-source

   a) may fail as a consequence of 4.b), 4.c)

   b) may build a source package that differs from the one built in 2.
      Testing for these bugs easily may need some new features in
      pbuilder, e.g. a --one-and-a-half option :-)
      A possible candidate is git/experimental 1:2.15.1+next.20171214-1
      which should add a 'debian/git-core/usr/share/doc/git-core -> git'
      symlink (#884890).

6. debian/rules build binary

   a) may fail as a consequence of 4.d)


How could we check for these bugs?

I think the interesting failures to look for are 4.a), 5.b), 6.a)
(leaving out the probabe majority 5.a)), since they may make working
with the packages difficult by failing in non-obvious ways. I would also
consider these as RC.

We would probably need some enhancements for pbuilder to allow testing
points 4. and 5. without doing a full --twice run. Also I don't know
whether it is easily possible to classify pbuilder --twice failures to
the command that failed and whether it was first or second build.

Given proper pbuilder support (deliver build result from the first build
even if 4. debian/rules clean failed), testing for 4.a) should be
trivially possible to do for reproducible builds with only marginal
computation time requirements.

Does the reproducible build effort currently test for reproducibility of
source packages? In that case testing for 5.b) could hopefully be done
with reproducible builds as well, it would just require another
dpkg-source call. Failures for 5.a) could be filtered out unless someone
volunteers for the bug filing.

Testing for 6.a) effectively doubles the compute power needed for a
rebuild test, so maybe that should be rather left to infrequent
specialized archive-wide rebuilds.


Andreas