Re: Salsa CI overload
Hi Denis,
> >> I feel like making people wait around for CI to run isn't a great use
> >> of our collective time.
> >
> > +1 to more Salsa CI runner resources.
>
> Is there a way for companies or institutes to contribute by supplying
> compute cycles from their clusters or data centres? I'm working for a
> research lab that has ample compute power and we would like to help out,
> but there may be some conditions that make this more complicated.
Do you represent and organization willing to donate capacity?
Some potential donors have expressed their willingness to contribute
resources tickets at https://salsa.debian.org/salsa/support, see e.g.
https://salsa.debian.org/salsa/support/-/issues/301, but based on what
I have understood from Salsa Admins is that it is easier for them to
simply expand the existing fleet at Google Cloud than to support hosts
across multiple cloud providers / datacenters. They have also been
increasing the runner fleet size in past years, and based on latest
email seems the salsa-wide CI system has up to 64 parallel runners
available at the moment.
Looking at the recently published https://salsa-status.debian.net/ on
the 30-day view, the spike we had 10 days ago due to Ruby team mass
changes is over and the base load seems to hover around 300 pipelines
per day.
We could further get this to drop to around 200 pipelines per day if
we promoted a culture that projects where Salsa CI is constantly
failing would turn it off, as CI as a regression testing system is
kind of moot if it constantly failing in a project and none of the
maintainers is fixing it, so it would be better to turn off to avoid
wasting resources. We have actually been doing a small email campaign
notifying Salsa projects that we have in Salsa CI stats seen been
running either scheduled pipelines that always fail or just very large
pipelines (build 1h+) with debian/latest failing asking them to turn
off Salsa CI until they can later re-enable it being in a passing
(green) state.
In the Salsa CI team we are also aware of some teams running excessive
amount of reverse builds on every git commit. We are soon announcing
https://salsa.debian.org/salsa-ci-team/pipeline/-/merge_requests/613
run reverse builds in a more controlled way hopefully helping to avoid
excessive number of builds.
- Otto
/Member of Salsa CI team, maintaining the pipeline code (but not in
Salsa Admin team and not maintaining the Salsa CI runners)
Reply to: