Hi, On 04-01-2024 17:28, Chris Hofstaedtler wrote:
On Thu, Jan 04, 2024 at 03:37:21PM +0100, Paul Gevers wrote:Hi, On 04-01-2024 15:08, Chris Hofstaedtler wrote:It would seem that the host runs out of IPC space?What is IPC space?https://manpages.debian.org/bookworm/manpages/sysvipc.7.en.html https://manpages.debian.org/bookworm/manpages/ipc_namespaces.7.en.htmlAnd when does a host run out of it? As I said, this is one of our most powerful hosts, so I would expect it to run out of things last.Does it run more tests in parallel than other workers, or so?Yes, this host (like most of our host, but a bit more) runs multiple lxc based debci workers.My guess: the default limits are static, and if LXC doesn't do anything special, the limits are probably shared with the whole host. kernel.shmmax, kernel.msgmax are I think the limits (but I'm not entirely sure).
Can you figure out decent numbers for these? Below I printed the output of lsipc and AFAICT SHMMAX is already pretty big ;) (and the same on all our hosts, which is also true for MSGMAX).
On the other hand, $(ipcs -a) doesn't show anything on the host, not even if I let it run in a while-loop (1 second interval) while I schedule the test of pdns. So, could this be a bug in systemd (which you claim below should be handeling this) or is this just not really supported in lxc and do you need a full VM. Because it works elsewhere, I feel more like a bug, and it would not be the first instance where code fails to properly handle 64 cores or 256GB or RAM.
I wouldn't know what to do about this, its not really under the control of src:pdns.Well, maybe check for it and fail gracefully?But how? systemd sets up the IPC namespace.
exit with 77 when you detect problems and add the skippable restriction.
Or, since a couple of days, if qemu VM don't run out of IPC space, we could run them in qemu always.I imagine a fully separated VM would not run out of IPC space, indeed.
I just ran the test in qemu on ci-worker13 and it PASSed. Paul root@ci-worker13:~# lsipcRESOURCE DESCRIPTION LIMIT USED USE% MSGMNI Number of message queues 32000 0 0.00% MSGMAX Max size of message (bytes) 8K - - MSGMNB Default max size of queue (bytes) 16K - - SHMMNI Shared memory segments 4096 0 0.00% SHMALL Shared memory pages 18446744073692774399 0 0.00% SHMMAX Max size of shared memory segment (bytes) 16E - - SHMMIN Min size of shared memory segment (bytes) 1B - - SEMMNI Number of semaphore identifiers 32000 0 0.00% SEMMNS Total number of semaphores 1024000000 0 0.00% SEMMSL Max semaphores per semaphore set. 32000 - - SEMOPM Max number of operations per semop(2) 500 - - SEMVMX Semaphore max value 32767 - -
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature