[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Wanted: system/volunteer to do whole-archive rebuild (×3) to test sbuild



On Mon, Feb 14, 2011 at 11:36:55PM +0000, Roger Leigh wrote:
> sbuild, which does the job of building binary packages on our buildds,
> uses a built-in build-dependency resolver ("internal") to work out
> what packages need installing/removing in order to satisfy a package's
> Build-Depends and Build-Conflicts.  Unfortunately, this has a number
> of bugs, e.g. #403246, as well as serious maintainability issues which
> make it less than desirable.  Last year, we introduced two new
> resolvers, "apt" and "aptitude" which delegate the somewhat complex
> build dependency resolving to the tools which are best at it.
> 
> Now that squeeze is out, it would be great if we could revisit the
> discussion about which build dependency resolver we want to use on
> our buildds.  The main concern here is that switching resolvers will
> change exactly which packages are installed in some situations, mainly
> in the case of packages depending on alternatives, which could lead to
> different packages being installed, inconsistent selection of packages
> being installed, or broken package builds.  The intenal resolver was
> dumb: it just always chose the first dependency even if it was
> unsatisfiable.  aptitude is far cleverer; apt somewhere in between,
> though we've tweaked aptitude to behave as close to stupid as we can.

The results of this are below.  Many thanks to Tollef Fog Heen for
his kind donation of time on one of his big machines to run three
archive rebuilds.  A better quality PDF version is attached.

Due to the size limits of the lists, the full version is at
http://people.debian.org/~rleigh/squeeze-rebuild/report.pdf

Regards,
Roger


        sbuild Build Dependency Resolver Comparison


1.  Introduction

     sbuild  has three different build dependency resolvers,
internal, apt  and  aptitude.   The  historical  default  is
internal, while apt and aptitude are much newer.  The inter‐
nal resolver  is  a  build‐dependency  resolver  implemented
directly  in  Perl,  and  which has a number of deficiencies
mainly relating to newer additions to the dependency format,
and  also  in  resolving  of  complex  versioned and virtual
dependencies.   The  apt  and  aptitude  resolvers  delegate
almost  all  dependency  resolution  to the apt and aptitude
tools, whose dependency resolution is far more  capable  and
better  tested.   However,  the  strict, simple, predictable
dependency resolution provided by internal must also be pro‐
vided  by  the alternative resolvers in order for them to be
suitable for use on our build dæmons.

     A goal for the wheezy release is to replace the  inter‐
nal  resolver  with  one  of  the apt or aptitude resolvers.
However, we had no hard information upon  which  to  base  a
decision.  I have consequently rebuilt the entire Debian ar‐
chive three times  using  the  internal,  apt  and  aptitude
resolvers,  using  the squeeze release since it’s stable and
unchanging between the runs.  The build logs have been  pro‐
cessed  and  the  build  status  and  dependency information
inserted into a PostgreSQL database in order to allow direct
comparison of the behaviour of the three resolvers.

     These tests all used the current version of sbuild from
unstable (0.60.9‐1), with a one line patch to  the  aptitude
resolver  to  make it use the same dpkg options as the other
resolvers when installing packages.  The  internal  resolver
has  a  bugfix the version on the buildds does not yet have,
allowing it to delegate  virtual  dependency  resolution  to
apt‐get  rather than failing outright (since older versions’
handling of virtual packages was completely broken).

     Many thanks to Tollef Fog Heen and freedesktop.org  for
the use of a big machine on which to do the builds.  This 24
core beast was able to rebuild  the  entire  archive  in  14
hours, taking four days in total for the three builds.

     The  build  logs, database schema and database dump are
all available  at  http://people.debian.org/~rleigh/squeeze‐;
rebuild/.


2.  Build summary

     These are the summary statistics for all the builds:


       ┌─────────┬────────────┬──────────────┬───────┐
       │Resolver │ Status     │ Fail stage   │ Count │
       ├─────────┼────────────┼──────────────┼───────┤
       │apt      │ attempted  │ build        │    38 │
       │apt      │ given‐back │ install‐deps │     5 │
       │apt      │ skipped    │ arch‐check   │  6195 │
       │apt      │ successful │              │  8372 │
       ├─────────┼────────────┼──────────────┼───────┤
       │aptitude │ attempted  │ build        │    41 │
       │aptitude │ skipped    │ arch‐check   │  6195 │
       │aptitude │ successful │              │  8374 │
       ├─────────┼────────────┼──────────────┼───────┤
       │internal │ attempted  │ build        │    37 │
       │internal │ given‐back │ install‐deps │    16 │
       │internal │ skipped    │ arch‐check   │  6195 │
       │internal │ successful │              │  8362 │
       └─────────┴────────────┴──────────────┴───────┘

     Skipped  packages  failing  the  architecture check are
Architecture: all packages; these were all  skipped  by  the
different resolvers prior to installation of build dependen‐
cies, and will not be  considered  further.   Packages  were
given back due to errors during installation of build depen‐
dencies, and comprised:


   ┌─────────┬───────────────────────────────────────────┐
   │Resolver │ Build                                     │
   ├─────────┼───────────────────────────────────────────┤
   │apt      │ circlepack_5.1‐7                          │
   │apt      │ libcomplearn‐mod‐ppmd_1.0.7‐2             │
   │apt      │ libcomplearn‐mod‐ppmdx_1.0.7‐2            │
   │apt      │ linux‐modules‐di‐amd64‐2.6_1.23           │
   │apt      │ xserver‐xorg‐video‐glide_1.0.3‐2+squeeze1 │
   ├─────────┼───────────────────────────────────────────┤
   │internal │ allegro4.2_2:4.2.2‐2.2                    │
   │internal │ arts_1.5.9‐3                              │
   │internal │ cheesetracker_0.9.15.3‐4                  │
   │internal │ circlepack_5.1‐7                          │
   │internal │ djplay_0.5.0‐3.1                          │
   │internal │ dovecot_1:1.2.15‐4                        │
   │internal │ ecasound2.2_2.7.0‐1.1                     │
   │internal │ galan_0.3.0+beta4‐2                       │
   │internal │ libcomplearn‐mod‐ppmd_1.0.7‐2             │
   │internal │ libcomplearn‐mod‐ppmdx_1.0.7‐2            │
   │internal │ libvisual‐plugins_0.4.0.dfsg.1‐2          │
   │internal │ linux‐modules‐di‐amd64‐2.6_1.23           │
   │internal │ specimen_0.5.2rc3‐1.1                     │
   │internal │ t38modem_1.2.0‐1                          │
   │internal │ timemachine_0.3.0‐3                       │
   │internal │ xserver‐xorg‐video‐glide_1.0.3‐2+squeeze1 │
   └─────────┴───────────────────────────────────────────┘

     circlepack,  libcomplearn‐mod‐ppmd,   libcomplearn‐mod‐
ppmdx,  linux‐modules‐di‐amd64  and xserver‐xorg‐video‐glide
were not built by  either  the  apt  or  internal  resolvers
because  the  needed  build  dependencies  were unavailable.
They did build not build with aptitude either, but the apti‐
tude failure did not terminate the build; these are reported
as attempted builds because failure occurs when  dpkg‐build‐
package  notices  the  build dependencies are not satisfied.
This will require fixing in the aptitude resolver.

     allegro, arts, cheesetracker, djplay, ecasound2, galan,
libvisual‐plugins,  specimen  and timemachine all have build
dependencies upon libjack_0.100.0‐dev  which  is  a  virtual
package.   The  internal  resolver  can  not resolve virtual
dependencies, while the other resolvers can.  These packages
should be fixed to use libjack‐dev.

     dovecot  (bug filed), steghide (already fixed in unsta‐
ble) and t38modem (bug filed) have a  versioned  build  con‐
flict  upon  linux‐kernel‐headers  which causes the internal
resolver to purge build‐essential and the  whole  toolchain,
which  causes  build failure.  This isn’t a problem with the
build‐conflict being incorrect, though a conflict on  linux‐
kernel‐headers (<< 2.5.999‐test7‐bk‐14) is certainly useless
and outdated, this is a bug in the internal resolver.   How‐
ever,  the  build  conflict  is outdated; given that we only
support linux >= 2.6, so it can be safely removed.  The more
advanced apt and aptitude resolvers could cope with the ver‐
sioned virtual conflict.

     Attempted (failed) builds failed during  building  with
dpkg‐buildpackage,  and  were  the  same  for all resolvers,
allowing for differences in dependency resolution failure.

3.  Methodology

     In order to allow for  meaningful  comparisons  between
resolvers,  only builds which were successful with all three
resolvers will be compared in the following  sections.   The
apt  and  aptitude  resolvers install additional packages in
order to function, which require excluding from the  compar‐
isons.   Both  the  apt and aptitude resolvers install dummy
dependency packages in order to delegate  resolving  to  the
apt  or  aptitude  programs, which should not be included in
the differences between the  resolvers.   Additionally,  the
aptitude  resolver requires aptitude, and its package depen‐
dencies, to be installed, and these also require  exclusion.
By  removing  these known differences, only the unknown dif‐
ferences will show up.

     Using  SQL  set  difference  (EXCEPT),   the   packages
installed  only  when  using  the internal resolver, or only
when using the apt (or apt resolver) can be compared;  pack‐
ages  installed  in both situations will be ignored.  NOT IN
will be used to  remove  unwanted  resolver  and  dependency
packages.   The  database  schema, views and queries used to
analyse the data, together with the scripts used to  perform
the  builds  and  parse  the  build  logs to get the data to
insert into the database are in the appendix at the  end  of
this document.


4.  Differences between internal and apt

     A  total  of  54  out of 8345 successful builds (0.65%)
showed discrepancies in the installed package set.

[Detail removed]

5.  Differences between internal and aptitude

     A total of 55 out of  8345  successful  builds  (0.66%)
showed discrepancies in the installed package set.

[Detail removed]

6.  Differences between apt and aptitude

     The  package  sets  installed  by apt and aptitude were
almost identical.  Only two packages out of 8345  successful
builds (0.02%) showed discrepancies in the installed package
set.

     Packages installed with internal  and  apt  (not  apti‐
tude):


       ┌────────────────┬───────────────────────────┐
       │Build           │ Package                   │
       └────────────────┴───────────────────────────┘
       │tulip_3.1.2‐2.3 │ libdrm2_2.4.21‐1~squeeze3 │
       │                │ libxdamage1_1:1.1.3‐1     │
       │                │ libxfixes3_1:4.0.5‐1      │
       │                │ libxxf86vm1_1:1.1.0‐2     │
       └────────────────┴───────────────────────────┘

     Packages  installed  with  internal  and  aptitude (not
apt):


          ┌────────────────┬──────────────────────┐
          │Build           │ Package              │
          └────────────────┴──────────────────────┘
          │piklab_0.15.7‐1 │ libsqlite3‐0_3.7.3‐1 │
          └────────────────┴──────────────────────┘


7.  Analysis of resolver differences

alex_2.3.3‐1

     apt and aptitude install docbook_4.5‐4.

     Depends on docbook‐utils, docbook‐xsl and  docbook‐xml.
docbook  is  only  suggested,  and  there  is an alternative
dependency in docbook‐dsssl.  apt‐get  and  aptitude  behave
differently when installing a dependency via the dummy pack‐
age than they do when invoked  directly  as  internal  does.
When  invoked with a list of packages to install, docbook is
not installed, presumably because the alternative dependency
is already satisfied.

avogadro_1.0.1‐3

     internal installs sip4_4.10.2‐1.

     sip4  is  a transitional package.  apt and aptitude are
intelligent enough to  realise  that  this  is  provided  by
another package and not install it, whereas internal is not.

boxbackup_0.11~rc2‐7squeeze1

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

calibre_0.7.7+dfsg‐1squeeze1

     internal installs sip4_4.10.2‐1.

     Same reason as for avogadro.

coinor‐cbc_2.5.0‐2

     apt and aptitude install libatlas‐base‐dev_3.8.3‐27 and
libatlas‐dev_3.8.3‐27.

     This is an alternative dependency in liblapack‐dev.  It
can  be satisfied by libblas‐3gf.so provided by libblas‐dev.
internal doesn’t  pick  both  alternatives,  while  apt  and
internal do.

ecj_3.5.1‐1

     internal  installs  ca‐certificates_20090814+nmu2,  ca‐
certificates‐java_20100412,   default‐jre‐headless_1:1.6‐40,
liblcms1_1.18.dfsg‐1.2+b3,     libnspr4‐0d_4.8.6‐1,     lib‐
nss3‐1d_3.12.8‐1, openjdk‐6‐jre‐headless_6b18‐1.8.3‐2, open‐
jdk‐6‐jre‐lib_6b18‐1.8.3‐2,   openssl_0.9.8o‐4  and  tzdata‐
java_2010o‐1

     This is a redundant set of packages pulled in  via  the
ant build dependency, which has a complex set of four possi‐
ble alternatives.  apt and aptitude  realise  that  this  is
satisfied by gcj‐4.4‐jdk, and do not install them.

elinks_0.12~pre5‐2

     internal installs docbook_4.5‐4.

     The  build  dependency  docbook‐utils depends upon doc‐
book‐dsssl which in turn depends upon either docbook or doc‐
book‐xml.   When  run via apt‐get or aptitude directly, doc‐
book is installed, and when installed via a dependency pack‐
age, it is not.

enblend‐enfuse_4.0+dfsg‐1

     apt and aptitude install ttf‐dejavu‐core_2.31‐1.

     The fontconfig‐config dependency has alternative depen‐
dencies  upon  four  different  font   packages.    internal
realises  that  they  are already satisfied, whereas apt and
aptitude do not.

f‐spot_0.6.2‐2

     apt and aptitude install automake_1:1.11.1‐1.

     The build dependency intltool has an alternative depen‐
dency on automake or automaken.  Consequently, apt and apti‐
tude install two versions of  automake,  and  internal  only
one.

gdcm_2.0.16‐2


     apt    and   aptitude   install   gcj‐4.4‐base_4.4.5‐2,
gcj‐4.4‐jre_4.4.5‐2,   gcj‐4.4‐jre‐headless_4.4.5‐2,    gcj‐
jre_4:4.4.5‐1, gcj‐jre‐headless_4:4.4.5‐1, libgcj10_4.4.5‐2,
libgcj10‐awt_4.4.5‐2 and libgcj‐common_1:4.4.5‐1.

     Possible alternatives affecting  this  are  default‐jdk
and  libvtk‐java.  This is again a case of choosing multiple
alternatives.

gjiten_2.6‐2.1

     internal installs docbook_4.5‐4.

     Same reason as for elinks.

gmime2.2_2.2.25‐2

     internal           installs            libosp5_1.5.2‐8,
libostyle1c2_1.4devel1‐19 and openjade_1.4devel1‐19.

     Multiple docbook possibilities via alternatives in doc‐
book‐dsssl and gtk‐doc‐tools.

gmime2.4_2.4.14‐1+nmu1

     internal           installs            libosp5_1.5.2‐8,
libostyle1c2_1.4devel1‐19 and openjade_1.4devel1‐19.

     Same reasons as for gmime2.2.

gnome‐color‐manager_2.30.2‐1

     internal            installs           libosp5_1.5.2‐8,
libostyle1c2_1.4devel1‐19 and openjade_1.4devel1‐19.

     Same reasons as for gmime2.2.

gridengine_6.2u5‐1

     internal installs default‐jre‐headless_1:1.6‐40.

     Same reasons as for ecj.

happy_1.18.4‐2

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

hardware‐monitor_1.4.2‐1.1

     apt and aptitude install makedev_2.3.1‐89.

     Alternative dependency in libsensors4 build dependency.

hplip_3.10.6‐2

     apt and aptitude  install  udev_164‐3,  while  internal
installs makedev_2.3.1‐89.

     Same reason as for hardware‐monitor.

iceweasel_3.5.16‐4

     apt and aptitude install ttf‐dejavu‐core_2.31‐1.

     Same reason as for enblend‐enfuse.

idzebra_2.0.44‐2

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

java3d_1.5.2+dfsg‐5

     internal installs default‐jre‐headless_1:1.6‐40.

     Same reasons as for ecj.

jblas_1.1.1‐2

     internal installs default‐jre‐headless_1:1.6‐40.

     Same reasons as for ecj.

k3d_0.8.0.2‐6

     apt and aptitude install ttf‐dejavu‐core_2.31‐1.

     Same reason as for enblend‐enfuse.

libjdic‐java_0.9.5‐7

     apt and aptitude install default‐jre‐headless_1:1.6‐40.

     Same reasons as for ecj.

libjogl‐java_1.1.1+dak1‐9

     internal installs default‐jre‐headless_1:1.6‐40.

     Same reasons as for ecj.

libraw1394_2.0.5‐2

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

nut_2.4.3‐1.1squeeze1

     apt and aptitude install makedev_2.3.1‐89.

     Same reasons as for hardware‐monitor.

openmeeg_2.0.0.dfsg‐2

     apt and aptitude install liblapack3gf_3.2.1‐8.

     Same reasons as for coinor‐cbc.

openoffice.org_1:3.2.1‐11+squeeze2

     internal installs default‐jre‐headless_1:1.6‐40.

     Same reasons as for ecj.

opensp_1.5.2‐8

     internal installs docbook_4.5‐4.

     Same reason as for elinks.

openturns_0.13.2‐8

     apt  and  aptitude  install  libatlas3gf‐base_3.8.3‐27,
libatlas‐base‐dev_3.8.3‐27 and libatlas‐dev_3.8.3‐27.

     Same reasons as for coinor‐cbc.

ounit_1.0.3‐5

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

pauker_1.8+dfsg‐4

     apt and aptitude install default‐jre_1:1.6‐40.

     Same reasons as for ecj.

phaseshift_0.40‐13.2

     internal installs xlibmesa‐gl‐dev_1:7.5+8.

     This package is transitional.   apt  and  aptitude  are
clever  enough  to  skip  it,  while internal uses the first
dependency.

php5_5.3.3‐7

     internal installs libdb‐dev_4.8.

     Another transitional‐like package where apt  and  apti‐
tude  pick  the  direct dependency rather than using a meta‐
package.

piklab_0.15.7‐1

     internal and aptitude install libsqlite3‐0_3.7.3‐1.

     internal     installs     libasyncns0_0.3‐1.1,     lib‐
cap2_1:2.19‐3,                          libflac8_1.2.1‐2+b1,
libphonon4_4:4.6.0really4.4.2‐1,      libpulse0_0.9.21‐3+b1,
libpulse‐mainloop‐glib0_0.9.21‐3+b1,           libqt4‐assis‐
tant_4:4.6.3‐4,                       libqt4‐dbus_4:4.6.3‐4,
libqt4‐designer_4:4.6.3‐4,             libqt4‐dev_4:4.6.3‐4,
libqt4‐help_4:4.6.3‐4,          libqt4‐multimedia_4:4.6.3‐4,
libqt4‐network_4:4.6.3‐4,       libqt4‐qt3support_4:4.6.3‐4,
libqt4‐script_4:4.6.3‐4,       libqt4‐scripttools_4:4.6.3‐4,
libqt4‐sql_4:4.6.3‐4,                  libqt4‐svg_4:4.6.3‐4,
libqt4‐test_4:4.6.3‐4,              libqt4‐webkit_4:4.6.3‐4,
libqt4‐xml_4:4.6.3‐4,  libqt4‐xmlpatterns_4:4.6.3‐4,  libqt‐
core4_4:4.6.3‐4, libqtgui4_4:4.6.3‐4,  libsndfile1_1.0.21‐3,
libwrap0_7.6.q‐19,           libxtst6_2:1.1.0‐3          and
qt4‐qmake_4:4.6.3‐4

     The package has an (arguably buggy)  alternative  build
dependency  on  both  libqt4‐dev or libqt3‐mt‐dev.  internal
installs both sets; apt and aptitude choose to only  install
a single set.  libsqlite3‐0 is pulled in via a very indirect
dependency, probably somewhere within the Qt libraries.

plplot_5.9.5‐4

     apt and aptitude install ttf‐dejavu‐core_2.31‐1.

     Same reason as for enblend‐enfuse.

poker‐network_1.7.7‐3.2

     internal        installs        apache2.2‐bin_2.2.16‐6,
apache2.2‐common_2.2.16‐6,     apache2‐mpm‐prefork_2.2.16‐6,
apache2‐utils_2.2.16‐6,         libapache2‐mod‐php5_5.3.3‐7,
libapr1_1.4.2‐6,  libaprutil1_1.3.9+dfsg‐5, libaprutil1‐dbd‐
sqlite3_1.3.9+dfsg‐5,  libaprutil1‐ldap_1.3.9+dfsg‐5,   lib‐
cap2_1:2.19‐3,          libldap‐2.4‐2_2.4.23‐7,         lib‐
sasl2‐2_2.1.23.dfsg1‐7 and procps_1:3.2.8‐9

     Incredibly, these are all pulled in via libapache2‐mod‐
php5  through  one  of  the php5 build dependencies (I still
haven’t fathomed which).

qmtest_2.4.1‐1

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

qtiplot_0.9.8‐1

     internal      installs      docbook_4.5‐4,      python‐
sip4‐dev_4.10.2‐1 and sip4_4.10.2‐1.

     Same  reason as for elinks for docbook, and as for avo‐
gadro for sip4.

rhythmbox_0.12.8‐3

     libosp5_1.5.2‐8,  libostyle1c2_1.4devel1‐19  and  open‐
jade_1.4devel1‐19.

     Same reasons as for gmime2.2.

rss‐glx_0.9.1‐3

     internal    installs    libdrm2_2.4.21‐1~squeeze3   and
libxxf86vm1_1:1.1.0‐2.

     This package has a number of alternative  OpenGL  build
dependencies.   internal  will  only  ever pick libgl1‐mesa‐
swx11‐dev, while apt and aptitude may pick one of the others
which does not include these packages.

scalapack_1.8.0‐6

     apt  and  aptitude  install  libatlas3gf‐base_3.8.3‐27,
libatlas‐base‐dev_3.8.3‐27 and libatlas‐dev_3.8.3‐27.

     Same reasons as for coinor‐cbc.

scidavis_0.2.4‐1

     internal installs python‐sip4‐dev_4.10.2‐1.

     Same reason as for avogadro.

shared‐mime‐info_0.71‐4

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

subversion_1.6.12dfsg‐4


     internal    installs     ca‐certificates‐java_20100412,
default‐jre_1:1.6‐40,  default‐jre‐headless_1:1.6‐40, libac‐
cess‐bridge‐java_1.26.2‐5,            libaccess‐bridge‐java‐
jni_1.26.2‐5,    libnspr4‐0d_4.8.6‐1,   libnss3‐1d_3.12.8‐1,
openjdk‐6‐jre_6b18‐1.8.3‐2,              openjdk‐6‐jre‐head‐
less_6b18‐1.8.3‐2,     openjdk‐6‐jre‐lib_6b18‐1.8.3‐2    and
tzdata‐java_2010o‐1.

     Multiple alternatives via junit.

taskjuggler_2.4.3‐2

     apt and aptitude install opensp_1.5.2‐8.

     There are a number of DocBook build  dependencies  con‐
taining  alternative  dependencies leading to multiple solu‐
tions.

thuban_1.2.2‐2

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.

tig_0.16‐2

     internal installs docbook_4.5‐4.

     Same reason as for elinks.

trackballs_1.1.4‐4

     internal installs xlibmesa‐gl‐dev_1:7.5+8.

     Package contains an alternative build  dependency  upon
xlibmesa‐gl‐dev and libgl‐dev.

ttt_1.7‐3.2

     internal  installs  fontconfig‐config_2.8.0‐2.1, libex‐
pat1_2.0.1‐7,                      libfontconfig1_2.8.0‐2.1,
libfreetype6_2.4.2‐2.1,                  libxext6_2:1.1.2‐1,
libxft2_2.1.14‐2, libxrender1_1:0.9.6‐1,  libxss1_1:1.2.0‐2,
tcl8.5_8.5.8‐2,  tk8.5_8.5.8‐1,  ttf‐dejavu‐core_2.31‐1  and
ucf_3.0025+nmu1.

     The build dependency blt‐dev depends on  both  versions
8.4 and 8.5 of tcl and tk.  The internal resolver decides to
install both versions, whereas apt and aptitude do not.

tulip_3.1.2‐2.3

     internal  and  apt  install  libdrm2_2.4.21‐1~squeeze3,
libxdamage1_1:1.1.3‐1,        libxfixes3_1:4.0.5‐1       and
libxxf86vm1_1:1.1.0‐2.

     Multiple alternative dependencies in  build  dependency
libqt4‐opengl‐dev and libosmesa6‐dev.

xmakemol_5.16‐5

     internal  installs  libdrm2_2.4.21‐1~squeeze3, libxdam‐
age1_1:1.1.3‐1,           libxfixes3_1:4.0.5‐1           and
libxxf86vm1_1:1.1.0‐2.

     Multiple  possible  OpenGL  solutions,  as for previous
examples.

xplanet_1.2.1‐3  apt  and   aptitude   install   ttf‐dejavu‐
core_2.31‐1.

     Same reason as for enblend‐enfuse.

yaz_4.0.11‐1

     apt and aptitude install docbook_4.5‐4.

     Same reason as for alex.


8.  Identified differences between resolvers

     The  differences observed in all of the packages in the
previous section have the same root cause.   The  method  of
installing build dependencies

     internal  invokes apt‐get with the argument install and
then a list of dependencies to install,  then  repeats  this
with  remove  with  a  list of conflicts to remove.  apt and
aptitude invoke apt‐get or aptitude, respectively,  with  an
install  argument  and a dummy dependency package containing
the complete set of depends and conflicts.  Both apt‐get and
aptitude  behave slightly differently if the package list is
provided on the command line or  as  a  dependency  package.
The  build  dependencies  requested  will  be installed, but
depending upon indirect alternative dependencies, there  may
be additional packages installed (or not) depending upon the
solution found.  The alex and elinks packages, above, demon‐
strates  this with each having two alternative solutions for
the DocBook dependencies.  Note  that  neither  solution  is
incorrect; both are technically correct solutions.

     The  same  different behaviour when faced with multiple
alternatives applies to all the other packages.  Only a cou‐
ple  of  cases  are directly due to alternative dependencies
directly in the build  dependencies;  all  other  cases  are
indirect, in the dependencies of dependencies.

·    The  OpenGL  library  dependencies  are overly complex;
     these could use some simplification.

·    The same situation exists with the multiple Java alter‐
     natives,  and differing ordering in different packages.
     Again, some simplification is in order.

·    And the same situation exists for  DocBook;  there  are
     just too many solutions to give consistent results.

     The  apt  and  aptitude resolvers, where they do differ
from the internal resolver, are not installing any different
build‐dependencies than which the maintainer requested.  The
variation lies in leaf packages; the packages the maintainer
wanted are always installed.


9.  Next steps

     Is  resolving  virtual  dependencies bad?  Well, we are
already resolving them anyway.  Any indirect virtual  depen‐
dencies  (virtual  dependencies  of  build dependencies) are
being  satisfied  by  apt‐get  using  the  current  internal
resolver.   Supporting  them in sbuild merely makes this be‐
haviour consistent, and as the above analysis  demonstrates,
most  of  the alternatives are not in the build dependencies
themselves, they are indirect.

     Are alternative dependencies bad?  Almost unequivocally
yes.   The  entire point of build dependencies is to specify
exactly the packages required for building; by allowing mul‐
tiple  possibilities, the reproducibility and consistency of
builds is compromised.  Almost every single existing use  of
alternative dependencies is buggy.  There are only a limited
number of  legitimate  uses  for  alternative  dependencies,
examples  of  which include using different packages on dif‐
ferent architectures (e.g.  linux  vs.  kfreebsd  vs.  hurd)
where  consistency on a single architecture is still guaran‐
teed, or to permit easier backporting when the packages have
been  renamed.   Every  example  seen  here,  the Qt 3 and 4
alternative being the worst, was buggy.  However, the  hand‐
ful of packages misusing alternatives are a tiny minority.

     Whether  or  not  alternative  dependencies  should  be
allowed in build dependencies is a  long‐standing  point  of
contention.   The  question  of  whether they are allowed is
completely orthogonal to whether they are possible.  The apt
and  aptitude  resolvers  make  the use of alternative build
dependencies possible, as a side effect  of  using  a  fully
functional  resolver.   This does not imply that they should
be used.  We should not enforce an unwritten policy  through
long‐standing  defects in our tools; the restrictions should
be defined in Policy, and compliance with  Policy  validated
with tools such as lintian.

     To  be  completely clear about this, the purpose of the
apt and aptitude resolvers is  not  to  enable  the  use  of
alternative build dependencies.  This is just a side effect.
The purpose is to replace a long obsolete and buggy resolver
with  one  which  works under all circumstances, and doesn’t
break on a multitude of corner cases, and with new additions
to  the  dependency syntax.  It will mean new syntax will be
usable when support is added to apt or aptitude, rather than
waiting  for  years, because the internal resolver is mostly
unmaintainable.  The main gain is removing a big  source  of
unnecessary buggy complexity and replacing it with something
simple, understandable, maintainable and vastly more  power‐
ful.

     Is  it safe to switch to apt or aptitude as the default
resolver?    Almost   certainly.    This   exhaustive   test
demonstrates  that  99.34–99.35%  of  the  existing  archive
builds entirely unchanged.  Of the  0.65–0.66%  which  built
with an altered installed package set, zero failed to build.
The differences are explained by the different ways apt  and
aptitude  compute  the  dependencies depending upon how they
are invoked, and are almost entirely benign.  There are some
packages  with  currently buggy dependencies which should be
fixed irrespective of the resolver used.  The apt and  apti‐
tude  build  resolvers  are  largely equivalent, with only a
0.02% difference.

     lintian should be able to check for misuse of  alterna‐
tive  and  virtual  dependencies in Build‐Depends and Build‐
Conflicts.

10.  Work items

·    Aptitude resolver must terminate build on  install‐deps
     failure.   Not a serious bug, but it means the cause of
     failure is reported correctly.

·    Should the internal resolver  delegate  virtual  depen‐
     dency resolution to apt‐get (current release behaviour)
     or revert to aborting all builds using  virtual  depen‐
     dencies?

·    apt and aptitude resolvers don’t print the parsed pack‐
     age  build  dependencies,  so  it’s  not  possible   to
     directly  check  the installed package set matches what
     was requested.  Print the list as for internal.


-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.

Attachment: signature.asc
Description: Digital signature


Reply to: