[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Summary of discussion regarding improvements needed in autopkgtest and britney

Hi all,

The last couple of days I have been discussing with you via e-mail and
IRC (much thanks to those involved) about current issues of autopkgtest
running tests for britney. I'll summarize my view of it in this e-mail
with the pro's and con's of different solutions and my proposed way
forward (for now).

======== ISSUES ========

There are a couple of issues that were the main topic of the discussion.
I'll summarize here so it is clear how I see the issue;

0. autopkgtest needs too much assistance (both in Debian and in Ubuntu)
for cases it should, IMHO, handle automatically. Part of this problem is
in dependency issues that albeit technically present, most of the time
aren't practical problems. Fixing those is my preferred solution. This
is even more true in Debian, as Ubuntu didn't want to bother Debian
maintainers with that and didn't want to carry those patches either.

1. britney triggers tests for packages that are at that moment
non-installable from unstable (especially, or maybe only, during

2. autopkgtest enables testing migration candidates by using all
packages from testing except for the binary packages of the candidate
which come from unstable. It achieves that via apt pinning. However,
sometimes this candidate has dependencies that need to come from
unstable. The current solution for that situation is that autopkgtest
contains a fallback which removes all the pinning. This has the drawback
that all packages from unstable are allowed to be installed, including
potentially broken packages or broken combinations.

3. autopkgtest has a needs-recommends restriction which is documented to
"Enable installation of recommended packages in apt for the test
dependencies. This does not affect build dependencies." The way this is
currently implemented is via apt's APT::Install-Recommends option. This
implementation leads to more packages installed than I believe is
desirable for this option. On top of that, installing recommends may
hide real dependency issues.

4. apt doesn't install recommends if an external solver is used. #900989

5. autopkgtest (or apt) is silent when recommends can't be installed.
Typically, tests that need the recommends will just fail, but it will
not be obvious why this is. The failure is correct, but the error
message will waste peoples time in debugging. So in my opinion it is a
must to improve the error handling of this situation. The FAIL will
remain a FAIL, but avoids wasting peoples time. #896698 [no-fail]

======== SOLUTIONS ========

1.a. As already discussed with nthykier some months ago, britney could
do some *recursive* installability checks before requesting a test. This
will drive down the number of false negatives, especially during
transitions. What is more, it will reduce the stress on autopkgtest in
combination with apt to come up with the right solution while there is none.

- less failures that just need a redo later-> less time spent by humans
  to check
- (some) duplication of logic, as the real installability check of
  britney happens in phase 2, so after the policy checks.
- doesn't need to be perfect, just better than now

2.a. If autopkgtest is told which packages are needed from unstable
(even with the exact version) to have a coherent set, it doesn't need to
guess what is a reasonable solution for britney. Therefore britney could
output a set of packages from unstable per test, instead of just the
current trigger.

- autopkgtest doesn't need to guess what britney wants
- apt will double check installability of the solution as it is not free
  anymore to do something else.
- with versions autopkgtest could fail if it doesn't test the version
  that britney expected it would find,
- new additional logic in britney (albeit probably similar to the
  installability logic, haven't checked)

2.b. the fallback in autopkgtest could be removed. autopkgtest should
rely on it being told what packages are allowed.

- removes a hack
- easier to understand and track for manual debuggers what is going on
- less code (it already exists, so not a big one)
- guarantees there is only one entity (britney) that decides what to
- maybe exposes unknown issues currently "hidden" behind this fallback

2.c. apt's internal solver only considers so called candidate versions.
External solvers like aspcud exist, that can optionally search for any
solution as long as installability is guaranteed.

- use a solver that is better suited for the non-standard use case we
  have at hand. I.e. one that can find solutions for the request:
  "install the following packages a, b, .. z from unstable and testing,
   while taking *as much as possible* from testing and while taking
   binary packages from source Q from unstable if they are to be
- solves LP: #1760810 [worker-crash] (worker crashes due to removal of
  packages needed to function)
- non-standard resolver
- apt doesn't install recommends [no-recommends] so extra code
  needed if needs-recommends restriction remains supported
- unsure if it works on oldest supported Ubuntu release, so potentially
  fallback to old code required

3.a. & 4.a. autopkgtest could deprecate the needs-recommends restriction

- removes a rather ambiguous restriction (see josch's e-mail [josch])
- make test dependencies more explicit may expose missing real
- installing recommends is the default, so why not enable testing that
- it is legitimate for tests to cover non-standard functionality
  provided by recommends. Removing this restriction makes duplication of
  the recommends line from d/control required.
- needs adaptation of autodep8 for perl
- ~150 packages are using this explicitly.

3.b. & 4.b. autopkgtest could install the recommends of the installed
test dependencies (so after installing the listed test dependencies).

- probably more in line with the intentions of the restriction
- similar to 5.b. below
- more (hackish) code

3.c. & 4.c. autopkgtest could change the behavior of needs-recommends to
only add the recommends of the binary packages from the package that is
being tested.

- it already has the source available for that, so adding the recommends
  is trivial
- less intrusive than removal, probably closer to original intent
- change of the meaning of the existing restriction
- susceptible for bug #896023 [mismatch] (installed version of the
  package doesn't always match the version of its test), potentially
  making that bug worse.

5.a. apt could warn or fail via an option (#901444 [apt-option]) and
autopkgtest could use that.

- little logic needed for autopkgtest
- depends on apt solution
- requires different temporary solution as it needs a long time before
  available in stable and Ubuntu's oldest supported version

5.b. autopkgtest could check if recommends are installed and create a
clear error message (e.g. grep for the "Recommended packages" in apt's
output and use dpkg-query to see if each package is installed)

- can be done now
- similar to 4.b. above
- more code for no change on pass/fail criteria

======== DISCARDED SOLUTIONS ========

1.b. Reordering of logic in britney to first calculate which package
sets need to go in lock step for installability purposes and only ask
tests per set.

Why discarded: britney currently doesn't calculate these disjoint sets,
but calculates only one set: the one that is allowed to migrate.
Therefore it would be too intrusive to change that on top of the
reordering of policy vs full installability. Also, the policy currently
influences if a package is candidate (not so in Ubuntu by the way).

2.d. autopkgtest could figure out which packages of the candidate are
going to be installed and request them with the <package>/unstable syntax.

Why discarded: autopkgtest doesn't have (and shouldn't be) a resolver.
Until apt finds the solution, it isn't known which packages are going to
be installed, so autopkgtest doesn't know to which packages it should
apply that syntax.

4.d. & 5.c. autopkgtest could add the recommends to the requested packages

Why discarded: similar to 2.d. autopkgtest doesn't have (and shouldn't
be) a resolver. Until apt finds the solution, it isn't known which
packages are going to be installed, so autopkgtest doesn't know from
which packages *and which version* to take the recommends.

======== PROPOSAL ========

Although I personally like the aspcud solver solution, I feel people are
reluctant to use the non-default apt resolver if it isn't needed.
Therefore I put that solution to plan b. Without an external solver used
by apt, the urgency of the needs-recommends problem is reduced to
autopkgtest not failing if recommends aren't installable. So I'll not
plan to deprecate it now.

Unless people have remarks, objections or other ideas not mentioned
above, it is my proposal that I work on (in this order)
1.a. britney recursive installability
2.a. britney producing coherent sets
2.b. autopkgtest removal (well, maybe make it optional) of fallback
and in parallel
5.b. autopkgtest check if recommends are installed

======== FINALLY ========

Thanks for reading so far. I am looking forward to any remark regarding
these issues. Also if you believe other issues regarding autopkgtest use
by britney are more important than these, feel free to comment.


[apt-option]    https://bugs.debian.org/901444
[josch]         https://lists.debian.org/deity/2018/06/msg00027.html
[mismatch]      https://bugs.debian.org/896023
[no-fail]       https://bugs.debian.org/896698
[no-recommends] https://bugs.debian.org/900989
[worker-crash]  https://bugs.launchpad.net/bugs/1760810

Attachment: signature.asc
Description: OpenPGP digital signature

Reply to: