Hi, On 13-02-2023 15:59, Dipak Zope1 wrote:
There is some issue with 5.10.0-21 kernel and we are working on it. This can cause performance impact on CI servers.
I have rebooted to the old kernel yesterday. That helps a bit indeed, although most of the issues I reported predate that kernel upgrade.
As Paul mentioned we have upgraded CI servers to better capacity in May last year. Is today’s performance worse than what we observed right after upgrade?
As you can see e.g. here [1,2] it comes and goes (albeit sometimes the queue was empty). I don't think its very different, I just never got out of the s390x host what I was expecting. Long time I blamed it on the "stealing" that happens on a shared host, but I think there's more.
[1] https://ci.debian.net/munin/ci-worker-s390x-01/ci-worker-s390x-01/debci_packages_processed.html
[2] https://ci.debian.net/munin/ci-worker-s390x-01/ci-worker-s390x-01/cpu.html
Is the performance deteriorated consistently over period of time or suddenly observed? Is there any incidence – like change/upgrade in software or hardware component which is coinciding with it if it is sudden change?
James Addison suggested in [3] to increase a prefetch counter in amqp (although its the same on all hosts); I have done so on the s390x host and at least initially it seems to help keeping the host busier.
[3] https://salsa.debian.org/ci-team/debci/-/issues/92#note_381306 Paul
Attachment:
OpenPGP_signature
Description: OpenPGP digital signature