Re: salsa.debian.org partially down

[Ian Jackson <ijackson@chiark.greenend.org.uk>]

Alexander Wirt writes ("Re: salsa.debian.org partially down"):
> It is already recovered. We will investigate where we can extend the
> ressources. But some misusages (like requesting >1300 merge requests via API
> on a big project, that in consequence run >1300 ci jobs, that...) can't be
> solved regardless on how many resources we add. 

Thanks for the reports from you and Bastian.  Thanks also for having
the energy and effort to deal with this kind of thing.  It's annoying
when a thing you're responsible for breaks because of foolish user
action, and then you have to scramble to fix it.

Maybe I'm teaching my grandmother to such eggs, but your message made
me want to suggest possible solutions/mitigations for the problem you
mention above.  Please feel free to disregard what follows.


I think the problem can be summarised/generalised as "someone makes
more requests to salsa than it has capacity to fulfil".

Traditional approaches to this include (mentioning all that I can
think of, even inappropriate or already-done ones; and, not knowing
what features gitlab has for this):

 * Per-user quotas.  (The kind of user who submits 1300 MRs might well
   react to a limit by creating more guest accounts...)

 * Per-project quotas.  (This avoids the above problem.  It
   ring-fences problems with poor contributor behaviour to the
   projects whose contributors are behaving poorly.)

 * Queuing jobs, so that the effect is contained (eg to the CI
   subsystem) until an administrator can cancel some jobs.  I think
   maybe earlier when Bastian wrote "It turns out that the configured
   amount of concurrency in CI builds can't be handled by the current
   available system resources" he was referring to a tuneable which
   would have the effect of queueing things, next time.  I guess
   you've adjusted this already.

 * Restricting resource-intensive actions to certain users.
   In our context this would seem to involve asking project
   maintainers to manually trigger CI on MRs.  That seems like it
   would be annoying and best avoided if we can.

 * Balkanising the system into multiple instances (perhaps with
   different configurations) so that each instance is exposed to a
   much smaller userbase.  I doubt we have the effort for this even if
   we could come up with a sensible division, and liked the idea.
   (One way to test the waters in this direction would be for someone
   to set up a competitor to salsa based on an entirely different
   management stack.)

 * Documentation, deterrence and punishment.  I mention this for
   completeness; given that we have so many users, and also offer
   guest accounts, this is not an appropriate strategy for salsa.

I hope that you find this message useful, rather than just a statement
of things which are mostly obvious and/or irrelevant.

Regards,
Ian.

-- 
Ian Jackson <ijackson@chiark.greenend.org.uk>   These opinions are my own.

If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
a private address which bypasses my fierce spamfilter.