Re: Dealing with ci.d.n for package regressions

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Paul Gevers writes ("Re: Dealing with ci.d.n for package regressions"):
> On 03-05-18 14:12, Ian Jackson wrote:
> > 3. "Required age increased by 10 days because of autopkgtest"
> > seems to appear when either (i) when there are tests that should be
> > run but which haven't completed and (ii) when some tests newly failed ?
> > I wasn't able to see any examples of the latter.
> 
> I gave an example link to python3.6 which worked at the time of writing,
> but of course (that how it goes) changed by an new upload. python2.7
> seems to show one: libgnatcoll (bug filed: 895235)

Thanks.  That is quite a clear report with a helpful link, indeed.

> > 4. Can we have a way to trigger tests from updates of non-direct
> > rdepends ?  At some point in the future maybe we will run tests of
> > whole batches of updates and then have some algorithm to chop out
> > what the failures are caused by, but for now it would be useful to
> > be able to declare a specific indirect dependency for test trigger.
> > Maybe an XS- header field ?
> 
> Just add it as a test dependency in one of your tests?

I hadn't realissed that _test_ dependencies would trigger retests, as
well as actual package dependencies.

Doing as you suggest for a real test feels wrong, since it involves
denormalising (in the relational database sense) the dependency graph.

But I guess I could introduce a test which does nothing, but which has
as direct dependencies the indirect dependencies I want to be retested
for.  It's a bit of a bodge but if we invented a feature name for this
test it would even give us an upgrade path:

  Tests: some-empty-shell-script
  Depends: indirect-dep-1, indirect-dep-2
  Features: hint-indirect-dependencies-retest

And then if we later add a more `proper' way of saying the same thing,
it can understand this old way of writing it.  Or we can ignore it if
we have a better way of doing the same thing later.

What do people think ?

> > 5. AIUI there is no automatic way for the maintainers of the
> > rdependency to be notified of a test failure which is blocking
> > migration of one of their dependencies.  Is that right ?  The result
> > is probably that if the maintainers of the dependency don't follow it
> > up, the regression will migrate and the rdepenency maintainers will be
> > left to fix it up.
> 
> No, it's all manual and currently I am doing most triaging (bunk and
> ginggs have contributed multiple bugs as well). The last couple of weeks
> I was able to file most bugs before the short expiry of 5 days, now with
> 15 days the task gets easier. If error messages and output are clean,
> this isn't so difficult. However, quite often output is hopeless for a
> bystander and difficult to judge the root cause and the severity. I hope
> we can improve this in the future by pointing people to the right tools
> (do they exist (for all languages)?) such that output gets standardized
> a bit more than currently.

For my part I'm sorry that the output from autopkgtest itself is not
always easy to navigate.

But, anyway, thanks for your effort, but it obviously doesn't scale to
have the central infrastructure team triage things.  How easy would it
be to have the CI automatically send an email to the maintainers of
the rdependency and the dependency ?

I think we need to get into the habit of the maintainers talking to
each other about these kind of things, before we start increasing the
blocking time.  Otherwise we risk developing a culture where the
dependency's maintainers usually do some kind of workaround, which the
rdependency's maintainers may find out about much later if at all.

Ian.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.