Dear release team, Since Dec 2016⁰ I have been working together with multiple people (including nthykier) to prepare the integration of britney with autopkgtest results. We believe the work has reached a state where full deployment can be considered. Let me describe how it is designed to work and what I request from you. Design ====== We started with autopkgtest policy in the britney2 code as created by Ubuntu. Ubuntu fully blocks migration of a package from the *-proposed pocket to the full suite when the package introduces a regression in the autopkgtest test suite of the package or any of its reverse dependencies. During autopkgtesting, the new version of the package is installed in the full suite. Of course hints can be used to ignore failure or bad tests. Although I believe we want such a gating behavior in Debian as well for unstable to testing migration, we worked on implementing an alternative scheme in parallel. Instead of allowing or blocking migration, this scheme is decreasing or increasing the time a package has to age. If a package has an autopkgtest and causes no regression, the package is rewarded by having to spend less time in unstable before it is considered a valid candidate. If a package causes regression (whether or not it has an autopkgtest of its own), its required age is increased. The additional time can be used to investigate the situation and fix it (after all, the problem may or may not be with the new version of the package). Obviously we didn't teach britney how to run an autopkgtest. Instead, we adapted the current framework on https://ci.debian.net (which until recently only ran tests in unstable) to be able to take input from britney, install the right package from unstable into testing and run the required tests. (The required changes landed in the debci and autopkgtest packages.) Because packages that cause regression can migrate in this scheme, we implemented a mechanism to reset the baseline of a package's autopkgtest result. Every package has a special migration-reference trigger that runs a package autopkgtest in testing (no packages from unstable). Every DD (or actually, everybody that can use Debian's SSO), can request retries of any test run, thus the baseline can easily be reset by maintainers. Ideally, this should be done automatically (I'll work on something, but I don't consider this blocking for deployment), but to prevent the complexity of combining the right peaces of information in britney, we didn't want to implement this in britney itself. When we move to the gating scheme, this issue becomes less relevant as only hinted packages get into this situation. (Ubuntu has a bug¹ open on baseline resetting, I think what we know have is already an improvement over their situation). Request ======= First of all, I'd like to request review of the changes I made to britney to facilitate the integration with autopkgtest results. My work lives on the autopkgtest branch of the github repository of britney². I would appreciate it when you could have a look at the new code and provide feedback. Secondly, assuming we can come to deployment of the new britney policy, I started a gobby document³ to draft an announcement to d-d-a. The document has already been commented on by nthykier and terceiro (CI), but I'd like it to be as good as possible. Shadow britney ============== I have started a shadow cron job of britney on respighi to see how everything is behaving. I'll start filing bugs on regressions such that the current code already has value, but the drawback of a shadow run is that the information is very poorly discover-able for interested parties. Also, I expect people to start noticing that ci.d.n is generating more results, I think it would be great if we can explain this in one go after britney has the autopkgtest code deployed. Side note 1 =========== In order to convince myself (and others) of the proper functioning of the code, I fixed the integration of the github repository with Travis⁴ as the default Travis framework was too old and testing was broken since some changes last year. Side note 2 =========== Because I had the hope, which is confirmed by now, that Ubuntu wants to merge the changes in Debian back into their britney version, I mostly kept around the code to support their workflow. Some changes are still needed to limit (or ideally fully mitigate) the delta between Debian and Ubuntu, as I wasn't always motivated to find a way before the confirmation. Looking forward to your responses, Paul ⁰ https://lists.debian.org/debian-release/2016/12/msg00310.html ¹ https://bugs.launchpad.net/britney/+bug/1700668 ² https://github.com/Debian/britney2/tree/autopkgtest ³ https://gobby.debian.org/export/Teams/Release/using-autopkgtest ⁴ https://travis-ci.org/Debian/britney2/
Attachment:
signature.asc
Description: OpenPGP digital signature