[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

autopkgtesting and regressions that slip into the baseline

Hi Iain,

Unfortunately your question in the CI BoF yesterday came late and my
stream by that time was minutes behind. I also needed some time to
process your question. I don't have all the answers yet that I think are
relevant, but let's start discussing.

I think your question was something like "issues are only found late by
indirect reverse dependencies, by which time there is not much to do
that accept the situation. Does Debian also experience that?"

First, I do find issues like that and I create bugs (hopefully with the
right severity) as I create bugs for most regressions that aren't
related to unsolved bugs in our infrastructure.

Second, because how Debian works with a different baseline (our baseline
is the current situation in testing, rather than all past results for a
package) than Ubuntu, the bug report is there but the baseline is
updated automatically so the problem for gating "goes away".

What is missing in Debian (but can be added relatively easily added once
somebody sets his/her mind to it by looking at the database), is finding
regressions in the baseline that weren't spotted during migration. In
Debian this indeed is an issue right now in the sense that this isn't
noticed (as it isn't noticed in Ubuntu until after the culprit
migrated). I think a real solution may be to test not only the direct
reverse dependencies, but also indirect reverse dependencies. If I am
correct ci.debian.net (with 10 workers for unstable and testing) is
doing that for the unstable archive for years already so this is
probably less problematic than it sounds, although I am a bit worried
about the time it takes sometimes. (We could add more workers of course).

Related note, we test all packages in a pure testing environment at
least once a week, so if needed, figuring out what changed is easier
than for you (you noted that there may have been quite some time between
tests in Ubuntu), although we aren't doing that yet.

What do you (and others) think?


Attachment: signature.asc
Description: OpenPGP digital signature

Reply to: