Re: CI pipeline timeout
Hi Russ,
first thanks for merging a lot of MRs! I today went through a few
more.
Russ Allbery wrote:
> It looks like the Lintian autopkgtests take too long to run most of the
> time, which is making it fairly hard to handle merge requests since the
> pipeline fails. The project timeout was already set to 1d but the
> autopkgtests job is timing out after one hour, so I assume that this is a
> runner timeout that we can't increase.
I think this has been cut down since then. I currently only see
autopkgtest runtimes between 1 and 2 hours. The samples I looked at
needed 75 and 95 minutes.
> Or... hm. I just realized that Salsa shows the pipeline as running
> on the project of the MR submitter. Maybe the problem is that it's
> using their timeout and not ours?
Exactly, I noticed that, too.
But there also seem different timeouts for the main repo and other's
repos. Our current time limit in the main repo is 3 hours, said to be
defined by the "runner" — whatever that means. But for merge requests
(i.e. lintian repos of forks) it seems to be 1 hour. So autopkgtests
from merge requests seem to run into timeouts way more often, but also
not always.
> That still poses roughly the same problem, though.
Ack.
JFTR: We also currently have problems with the i386 builds on Salsa
CI, but that's because binutils is missing its i386 binary packages
for about a day. A rebuild of them is currently running, so this
hopefully should be solved tomorrow. I currently ignore them and the
tests are now all passing again.
> This seems to be the only CI job that runs the test suite, so clearly we
> need it to confirm that MRs haven't broken anything.
>
> I'm wondering what the options are here.
Pushing the parts from an MR into a feature branch so far helped, but
is a lot of unnecessary work.
> Maybe we can break the test suite up into multiple parts that can
> run separately? That will take as long but should avoid the timeout.
I'm not sure if its really possible to break up autopkgtests so that
it runs in more than one runner.
> Obviously ideal would be to speed up testing.
Full ack.
> It looks like just building all of the test packages takes a
> half-hour, and then running Lintian on all of them takes longer than
> that, at least some of the time. We do build a *lot* of test
> packages (1,437),
That's a result of Felix breaking up and duplicating test packages to
have specific packages for each test. But I'm not sure if this is
really slower than before when we always ran all lintian checks
against all (but fewer) test packages.
> but of course it's nice to have one test package per conceptual test
> and not to have to think about merging multiple tests together into
> a single package.
IMHO both have adavantages and I'm not 100% sure which variant I'd
prefer.
> If this were GitHub Actions, I'd be tempted to try to cache all of the
> test packages and only rebuild them if something has changed, which I
> suspect would save some time. I'm not sure how to do that with GitLab.
Maybe the Salsa CI Team has some ideas here.
Anyway, I think with the reintroduction of sliding windows for
regexp-based checks by Bastien as well as with blacklisting file
suffixes/types for the very-long-line-… tag, we already gained a lot
of speed again.
This not only speeds up the test suite, but especially also checking
packages with a lot of files like PostgreSQL, Firefox, Linux, etc.
And I suspect that there are more of that kind of optimization
possibilities than it the test suite itself.
Regards, Axel
--
,''`. | Axel Beckert <abe@debian.org>, https://people.debian.org/~abe/
: :' : | Debian Developer, ftp.ch.debian.org Admin
`. `' | 4096R: 2517 B724 C5F6 CA99 5329 6E61 2FF9 CD59 6126 16B5
`- | 1024D: F067 EA27 26B9 C3FC 1486 202E C09E 1D89 9593 0EDE
Reply to: