Hey Paul, Thanks for the email! On Fri, Aug 03, 2018 at 11:36:48AM +0200, Paul Gevers wrote: > I think your question was something like "issues are only found late by > indirect reverse dependencies, by which time there is not much to do > that accept the situation. Does Debian also experience that?" That's right. The way I put it I think was "we are bad at pinning the blame for test regressions at the right place". > First, I do find issues like that and I create bugs (hopefully with the > right severity) as I create bugs for most regressions that aren't > related to unsolved bugs in our infrastructure. That sounds like really valuable work, thanks for doing that. Am I right in thinking that this is mostly through manual examination of each particular failure? > Second, because how Debian works with a different baseline (our baseline > is the current situation in testing, rather than all past results for a > package) than Ubuntu, the bug report is there but the baseline is > updated automatically so the problem for gating "goes away". Indeed. I think this is part of the situation that I don't find 100% comfortable. I can see the justification: we missed when the regression came in, and it's not fair to impose penalties on maintainers/packages that didn't cause the problem. That is arguably a sensible way to deal with the current reality. But I think it should be considered a workaround for the problem that I'm specifying. Ideally we would have 0 flaky tests (which mess everything else up) and every test regression would exactly pin the blame on the change(s?) that caused it. If we get better at catching things at the right time, the testing-as-baseline thing won't be so necessary - because it will be really abnormal for something to go from green to red in testing: we'll have noticed it in unstable when it actually happened. I know that achieving this perfectly is probably not possible, so steps towards this state are what I would be looking for. > I think a real solution may be to test not only the direct reverse > dependencies, but also indirect reverse dependencies. If I am correct > ci.debian.net (with 10 workers for unstable and testing) is doing that > for the unstable archive for years already so this is probably less > problematic than it sounds, although I am a bit worried about the time > it takes sometimes. (We could add more workers of course). I think that's the right idea, but I'm a bit worried about whether it scales. There are some extreme cases like glibc and perl - just testing their direct dependencies on Ubuntu for us results in more than 1,000 test requests (we test on 6 arches) and if more than one of these big packages is uploaded close together in time we already end up with a multi-day backlog of pending tests. I can't exactly remember what our capacity is, but on x86 I'm sure it's more than 10 instances - more like 40 (shared between i386 and amd64). I'd need to do some analysis to figure out what kind of numbers we'd be talking about here. > Related note, we test all packages in a pure testing environment at > least once a week, so if needed, figuring out what changed is easier > than for you (you noted that there may have been quite some time between > tests in Ubuntu), although we aren't doing that yet. We discussed this between a small group after the BoF had finished, and had the idea to do the same in Ubuntu too. Something like this: keep a low priority queue constantly filled with all packages that have tests in 'testing' (which we call 'the release') which is consumed from whenever the other queues are empty. If we do this, then there should be a reasonably fresh baseline available to look at whenever something flips to red. Hope these comments have been helpful. Cheers, -- Iain Lane [ iain@orangesquash.org.uk ] Debian Developer [ laney@debian.org ] Ubuntu Developer [ laney@ubuntu.com ]
Attachment:
signature.asc
Description: PGP signature