On Mon, Aug 25, 2025 at 09:05:14PM +0100, Ian Jackson wrote: > Hi. > > (I wasn't able to find another mailing list for this. I found a bit > of discussion on debian-devel about one specific overload incident.) > > Since sid opened, people have naturally been hard at work. Great! > > But I notice that Salsa CI is now frequently very overloaded. Even > when it's not half a day behind (!), I often see that my package is > able to run only about one job at a time. I think this is slowing a > lot of us down quite considerably. I took some statistics. Those are only for our instance wide runner. Since beginning of the month we had roughly three times the amount of jobs. Today one project (node-glob?!?) alone took 25% of all jobs (2700?!? I really have to check those numbers afer some sleep, but all others numbers look sane). salsa wasn't built for that. That is not only about more runners, the results and the artifacts have to be processed by godard too. So just waiving with _some_ money doesn't help. Sure, a complete overhaul would make sense (2 recent servers, one just for postgresql, recent storage and so on). But that is not just some money, that would mean a lot money. Today (UTC): 2025-08-26T00:00:00+00:00 .. 2025-08-27T00:00:00+00:00 Jobs processed today (instance-wide): 11142 Queue time today (s): avg=2008.9, median=185.1, p95=8466.9 Status distribution today: success: 9248 failed: 1809 canceled: 55 running: 30 Top 5 projects today: js-team/node-glob: 2751 aquilamacedo/pipeline: 305 salsa-ci-team/pipeline: 293 freexian-team/debusine: 288 pochu/lts-pipeline: 237 Jobs by day (all scanned, not just today): 2025-07-28: 3970 2025-07-29: 3264 2025-07-30: 3066 2025-07-31: 5084 2025-08-01: 4058 2025-08-02: 5619 2025-08-03: 3950 2025-08-04: 3217 2025-08-05: 3251 2025-08-06: 3157 2025-08-07: 4091 2025-08-08: 3210 2025-08-09: 4478 2025-08-10: 6444 2025-08-11: 7653 2025-08-12: 8361 2025-08-13: 5407 2025-08-14: 4956 2025-08-15: 5128 2025-08-16: 5763 2025-08-17: 5372 2025-08-18: 5398 2025-08-19: 5076 2025-08-20: 5317 2025-08-21: 5453 2025-08-22: 5219 2025-08-23: 5702 2025-08-24: 6613 2025-08-25: 14768 2025-08-26: 11142 Alex
Attachment:
signature.asc
Description: PGP signature