[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: autopkgtesting and regressions that slip into the baseline

Hi Iain,

On 08-08-18 14:06, Iain Lane wrote:
> On Fri, Aug 03, 2018 at 11:36:48AM +0200, Paul Gevers wrote:
>> I think your question was something like "issues are only found late by
>> indirect reverse dependencies, by which time there is not much to do
>> that accept the situation. Does Debian also experience that?"
> That's right. The way I put it I think was "we are bad at pinning the
> blame for test regressions at the right place".

Often (but by far not always) I leave that up to the packages involved.
There have been numerous bugs I have filed against both the package that
starts to fail as well as to the package that happens to trigger the
failure first.

>> First, I do find issues like that and I create bugs (hopefully with the
>> right severity) as I create bugs for most regressions that aren't
>> related to unsolved bugs in our infrastructure.
> That sounds like really valuable work, thanks for doing that. Am I right
> in thinking that this is mostly through manual examination of each
> particular failure?

See also [1]. That is correct. But in nearly all cases, what I do is
quickly try to spot the error, see if it already makes sense (in quite a
bit of the cases it does, although I don't think it is above 50%). Then
I check if the apt fallback triggered, the changelog of the new package
and the status of the tests in unstable (can all be done in a couple of
minutes). Typically submitting the bug correctly costs most time (I have
to improve my scripts to generate a proper template with the info I
want). I spend most time actually communicating with people like about
the right technical solution; versioned breaks and versioned depends are
often not fully on the radar and currently not 100% supported by the
autopkgtest framework. Bug 896023 is a nasty one in that respect, which
should be fixed by britney; I am working on a patch to add multiple
packages from unstable at the same time (I want the apt fallback to be
disabled for Debian testing when that lands).

>> Second, because how Debian works with a different baseline (our baseline
>> is the current situation in testing, rather than all past results for a
>> package) than Ubuntu, the bug report is there but the baseline is
>> updated automatically so the problem for gating "goes away".
> Indeed. I think this is part of the situation that I don't find 100%
> comfortable.
> I can see the justification: we missed when the regression came in, and
> it's not fair to impose penalties on maintainers/packages that didn't
> cause the problem. That is arguably a sensible way to deal with the
> current reality. But I think it should be considered a workaround for
> the problem that I'm specifying.

Well, although I agree with your statement, it's also a consequence of
the current way Debian works (albeit we all agree that we want to go to
gating, much like Ubuntu). As we are only delaying, eventually
regressions are *allowed* to enter, unless the bug has RC status.

> Ideally we would have 0 flaky tests (which mess everything else up)

I have automatic retries in place, so there are not such a big problem
in Debian. Also, all flaky tests with bugs filed are force-badtest by me.

> and
> every test regression would exactly pin the blame on the change(s?) that
> caused it.

If we didn't have that stupid (but currently needed) apt fallback. Also,
because of dependencies, you may only be able to migrate a group of
packages in lock-step and then any package in the group can actually
cause the regression. I don't think that without rebuilding (and manual
work) you'll be able to spot for every regression which package actually
caused the issue. I agree we can do better than now.

> If we get better at catching things at the right time, the
> testing-as-baseline thing won't be so necessary - because it will be
> really abnormal for something to go from green to red in testing: we'll
> have noticed it in unstable when it actually happened.

No. Unstable is broken on a regular basis. Transitions are often the
cause for that, especially in combination with apt's behavior of only
considering candidates. I.e. from the Depends and availability view
point you would be able to find a solution, except apt doesn't want to
install it.

> I know that achieving this perfectly is probably not possible, so steps
> towards this state are what I would be looking for.

Full ack on that.

>> I think a real solution may be to test not only the direct reverse
>> dependencies, but also indirect reverse dependencies. If I am correct
>> ci.debian.net (with 10 workers for unstable and testing) is doing that
>> for the unstable archive for years already so this is probably less
>> problematic than it sounds, although I am a bit worried about the time
>> it takes sometimes. (We could add more workers of course).
> I think that's the right idea, but I'm a bit worried about whether it
> scales. There are some extreme cases like glibc and perl - just testing
> their direct dependencies on Ubuntu for us results in more than 1,000
> test requests (we test on 6 arches) and if more than one of these big
> packages is uploaded close together in time we already end up with a
> multi-day backlog of pending tests. I can't exactly remember what our
> capacity is, but on x86 I'm sure it's more than 10 instances - more like
> 40 (shared between i386 and amd64).
> I'd need to do some analysis to figure out what kind of numbers we'd be
> talking about here.

Maybe Antonio can comment on our unstable situation. I am pretty sure
that we sometimes are behind, yes, and I also know we have 10 workers in
total, only amd64. Since we prioritized submissions over the debci
(unstable) tests I haven't seen much backlog, except indeed in the glibc

>> Related note, we test all packages in a pure testing environment at
>> least once a week, so if needed, figuring out what changed is easier
>> than for you (you noted that there may have been quite some time between
>> tests in Ubuntu), although we aren't doing that yet.
> We discussed this between a small group after the BoF had finished, and
> had the idea to do the same in Ubuntu too. Something like this: keep a
> low priority queue constantly filled with all packages that have tests
> in 'testing' (which we call 'the release') which is consumed from
> whenever the other queues are empty. If we do this, then there should be
> a reasonably fresh baseline available to look at whenever something
> flips to red.

Indeed. There are currently about 8600 packages with tests in testing.
That means about 1 test per minute when looping is done in a week
(that's what I do). (@Antonio) It would be even greater if I could do
that will lower prio than the current gating testing, but at the moment
this isn't really an issue.

Did you further discuss the idea's of Ian about bisecting a regression?



Attachment: signature.asc
Description: OpenPGP digital signature

Reply to: