Hi all, The last couple of days I have been discussing with you via e-mail and IRC (much thanks to those involved) about current issues of autopkgtest running tests for britney. I'll summarize my view of it in this e-mail with the pro's and con's of different solutions and my proposed way forward (for now). ======== ISSUES ======== There are a couple of issues that were the main topic of the discussion. I'll summarize here so it is clear how I see the issue; 0. autopkgtest needs too much assistance (both in Debian and in Ubuntu) for cases it should, IMHO, handle automatically. Part of this problem is in dependency issues that albeit technically present, most of the time aren't practical problems. Fixing those is my preferred solution. This is even more true in Debian, as Ubuntu didn't want to bother Debian maintainers with that and didn't want to carry those patches either. 1. britney triggers tests for packages that are at that moment non-installable from unstable (especially, or maybe only, during transitions). 2. autopkgtest enables testing migration candidates by using all packages from testing except for the binary packages of the candidate which come from unstable. It achieves that via apt pinning. However, sometimes this candidate has dependencies that need to come from unstable. The current solution for that situation is that autopkgtest contains a fallback which removes all the pinning. This has the drawback that all packages from unstable are allowed to be installed, including potentially broken packages or broken combinations. 3. autopkgtest has a needs-recommends restriction which is documented to "Enable installation of recommended packages in apt for the test dependencies. This does not affect build dependencies." The way this is currently implemented is via apt's APT::Install-Recommends option. This implementation leads to more packages installed than I believe is desirable for this option. On top of that, installing recommends may hide real dependency issues. 4. apt doesn't install recommends if an external solver is used. #900989 [no-recommends] 5. autopkgtest (or apt) is silent when recommends can't be installed. Typically, tests that need the recommends will just fail, but it will not be obvious why this is. The failure is correct, but the error message will waste peoples time in debugging. So in my opinion it is a must to improve the error handling of this situation. The FAIL will remain a FAIL, but avoids wasting peoples time. #896698 [no-fail] ======== SOLUTIONS ======== 1.a. As already discussed with nthykier some months ago, britney could do some *recursive* installability checks before requesting a test. This will drive down the number of false negatives, especially during transitions. What is more, it will reduce the stress on autopkgtest in combination with apt to come up with the right solution while there is none. pro: - less failures that just need a redo later-> less time spent by humans to check con: - (some) duplication of logic, as the real installability check of britney happens in phase 2, so after the policy checks. note: - doesn't need to be perfect, just better than now 2.a. If autopkgtest is told which packages are needed from unstable (even with the exact version) to have a coherent set, it doesn't need to guess what is a reasonable solution for britney. Therefore britney could output a set of packages from unstable per test, instead of just the current trigger. pro: - autopkgtest doesn't need to guess what britney wants - apt will double check installability of the solution as it is not free anymore to do something else. - with versions autopkgtest could fail if it doesn't test the version that britney expected it would find, con: - new additional logic in britney (albeit probably similar to the installability logic, haven't checked) 2.b. the fallback in autopkgtest could be removed. autopkgtest should rely on it being told what packages are allowed. pro: - removes a hack - easier to understand and track for manual debuggers what is going on - less code (it already exists, so not a big one) - guarantees there is only one entity (britney) that decides what to test con: - maybe exposes unknown issues currently "hidden" behind this fallback 2.c. apt's internal solver only considers so called candidate versions. External solvers like aspcud exist, that can optionally search for any solution as long as installability is guaranteed. pro: - use a solver that is better suited for the non-standard use case we have at hand. I.e. one that can find solutions for the request: "install the following packages a, b, .. z from unstable and testing, while taking *as much as possible* from testing and while taking binary packages from source Q from unstable if they are to be installed." - solves LP: #1760810 [worker-crash] (worker crashes due to removal of packages needed to function) con: - non-standard resolver - apt doesn't install recommends [no-recommends] so extra code needed if needs-recommends restriction remains supported - unsure if it works on oldest supported Ubuntu release, so potentially fallback to old code required 3.a. & 4.a. autopkgtest could deprecate the needs-recommends restriction pro: - removes a rather ambiguous restriction (see josch's e-mail [josch]) - make test dependencies more explicit may expose missing real dependencies con: - installing recommends is the default, so why not enable testing that - it is legitimate for tests to cover non-standard functionality provided by recommends. Removing this restriction makes duplication of the recommends line from d/control required. - needs adaptation of autodep8 for perl - ~150 packages are using this explicitly. 3.b. & 4.b. autopkgtest could install the recommends of the installed test dependencies (so after installing the listed test dependencies). pro: - probably more in line with the intentions of the restriction - similar to 5.b. below con: - more (hackish) code 3.c. & 4.c. autopkgtest could change the behavior of needs-recommends to only add the recommends of the binary packages from the package that is being tested. pro: - it already has the source available for that, so adding the recommends is trivial - less intrusive than removal, probably closer to original intent con: - change of the meaning of the existing restriction - susceptible for bug #896023 [mismatch] (installed version of the package doesn't always match the version of its test), potentially making that bug worse. 5.a. apt could warn or fail via an option (#901444 [apt-option]) and autopkgtest could use that. pro: - little logic needed for autopkgtest con: - depends on apt solution - requires different temporary solution as it needs a long time before available in stable and Ubuntu's oldest supported version 5.b. autopkgtest could check if recommends are installed and create a clear error message (e.g. grep for the "Recommended packages" in apt's output and use dpkg-query to see if each package is installed) pro: - can be done now - similar to 4.b. above con: - more code for no change on pass/fail criteria ======== DISCARDED SOLUTIONS ======== 1.b. Reordering of logic in britney to first calculate which package sets need to go in lock step for installability purposes and only ask tests per set. Why discarded: britney currently doesn't calculate these disjoint sets, but calculates only one set: the one that is allowed to migrate. Therefore it would be too intrusive to change that on top of the reordering of policy vs full installability. Also, the policy currently influences if a package is candidate (not so in Ubuntu by the way). 2.d. autopkgtest could figure out which packages of the candidate are going to be installed and request them with the <package>/unstable syntax. Why discarded: autopkgtest doesn't have (and shouldn't be) a resolver. Until apt finds the solution, it isn't known which packages are going to be installed, so autopkgtest doesn't know to which packages it should apply that syntax. 4.d. & 5.c. autopkgtest could add the recommends to the requested packages Why discarded: similar to 2.d. autopkgtest doesn't have (and shouldn't be) a resolver. Until apt finds the solution, it isn't known which packages are going to be installed, so autopkgtest doesn't know from which packages *and which version* to take the recommends. ======== PROPOSAL ======== Although I personally like the aspcud solver solution, I feel people are reluctant to use the non-default apt resolver if it isn't needed. Therefore I put that solution to plan b. Without an external solver used by apt, the urgency of the needs-recommends problem is reduced to autopkgtest not failing if recommends aren't installable. So I'll not plan to deprecate it now. Unless people have remarks, objections or other ideas not mentioned above, it is my proposal that I work on (in this order) 1.a. britney recursive installability 2.a. britney producing coherent sets 2.b. autopkgtest removal (well, maybe make it optional) of fallback and in parallel 5.b. autopkgtest check if recommends are installed ======== FINALLY ======== Thanks for reading so far. I am looking forward to any remark regarding these issues. Also if you believe other issues regarding autopkgtest use by britney are more important than these, feel free to comment. Paul [apt-option] https://bugs.debian.org/901444 [josch] https://lists.debian.org/deity/2018/06/msg00027.html [mismatch] https://bugs.debian.org/896023 [no-fail] https://bugs.debian.org/896698 [no-recommends] https://bugs.debian.org/900989 [worker-crash] https://bugs.launchpad.net/bugs/1760810
Attachment:
signature.asc
Description: OpenPGP digital signature