Re: salsa.debian.org partially down
On Tue, 13 Aug 2019, Hector Oron wrote:
> Hello,
>
> Missatge de Bastian Blank <waldi@debian.org> del dia dt., 13 d’ag.
> 2019 a les 11:51:
> >
> > Hi folks
> >
> > salsa.debian.org is partially down for now. Especially everything that
> > concerns Salsa CI.
> >
> > Someone decided to inject a few thousand CI jobs using Salsa CI
> > yesterday evening. Since then the system is not longer at 50% load, but
> > at a 100%. It turns out that the configured amount of concurrency in CI
> > builds can't be handled by the current available system resources.
> >
> > This now affects all user access to salsa.debian.org. So we disabled
> > access to all the Salsa CI stuff for now, which will make everything
> > using it fail.
> >
> > We have to discuss first how we can go forward.
>
> Those are very bad news. I hope that can be recovered soon. If you
> need more resources to be able to handle more jobs or longer timeouts,
> please let DPL know so more resources can be added.
It is already recovered. We will investigate where we can extend the
ressources. But some misusages (like requesting >1300 merge requests via API
on a big project, that in consequence run >1300 ci jobs, that...) can't be
solved regardless on how many resources we add.
Alex
Reply to: