Bug#926409: lintian: autopkgtest takes very long to finish

[Balint Reczey <balint.reczey@canonical.com>]

Hi Chris,

On Thu, Apr 4, 2019 at 7:30 PM Chris Lamb <lamby@debian.org> wrote:
>
> tags 926409 + moreinfo
> thanks
>
> Hi Balint,
>
> > Lintian has a lot of tests which is great for coverage, but maybe some
> > of them could be skipped in autopkgtest runs.
>
> Interesting. I guess I would have three follow-up questions here:
>
>  * On what criterion or criteria could we include or exclude tests
>    from the autopkgtest runs? Whilst we could skip the unit tests (as
>    these are "just" Perl that is unlikely to vary) the most
>    interesting ones to run in terms of detecting regressions in an
>    real-world environment (the entire point of autopkgtests from my
>    point of view) would be the tests of the checks themselves and
>    these likely constitute the vast majority of the total time.

One criterion that came to my mind is filtering by severity, including
errors for sure, but not pedantic ones.
The full suite can run during the build thus we don't loose a lot of coverage.
I hoped to rely on Lintian maintainer's judgement about what to omit.

>  * I'm not sure *how* we can speed up the tests. I mean, they all
>    essentially involve building Debian packages with all the usual
>    debhelper calls, etc. Speeding *this* up is somewhat out-of-scope
>    of this Lintian wishlist issue, alas.

A profiling round with perf would point out a few things IMO, but as a
start I did a timestamped run here to find slowest tests:
https://objectstorage.prodstack4-5.canonical.com/v1/AUTH_77e2ada1e7a84929a74ba3b87153c0ac/autopkgtest-disco-rbalint-scratch/disco/arm64/l/lintian/20190404_002129_96f0c@/log.gz

If there many package builds in the tests just changing the faster
compression helps, a lot with little effort (until zstd becomes the
default ;-)).

>  * Why not simply increase Ubuntu's timeout? I would concede this is
>    not the best use of CI resources, but the trade-off with "human"
>    time would appear to be worth it here.

I agree, and we may increase the timeout, but the running the tests
seems to take longer in general than seems reasonable.

Cheers,
Balint

> However, perhaps Felix has some input here as he has been doing a lot
> of work on the test suite recently?
>
>
> Best wishes,
>
> --
>       ,''`.
>      : :'  :     Chris Lamb
>      `. `'`      lamby@debian.org chris-lamb.co.uk
>        `-



-- 
Balint Reczey
Ubuntu & Debian Developer