Re: help needed to manage s390x host for ci.debian.net
Hi,
On 17.02.23 17:04, Antonio Terceiro wrote:
So there is for sure something wrong with the client-server connection
there. Reworking the client for robustness is on my TODO list for a
while.
There's a lot of these:
Feb 14 08:56:25 ci-worker-s390x-01 debci[1155941]: waiting for header frame: a SSL error occurred
But alas, the worker will fail and immediately restart. But what's more
concerning is the context:
Feb 14 08:39:50 ci-worker-s390x-01 debci[1355790]: bacula testing/s390x tmpfail
Feb 14 08:56:25 ci-worker-s390x-01 debci[1155941]: waiting for header frame: a SSL error occurred
This looks pretty common:
Feb 14 00:45:12 ci-worker-s390x-01 debci[2652291]: libgd2 testing/s390x fail
Feb 14 01:01:48 ci-worker-s390x-01 debci[546227]: waiting for header frame: a SSL error occurred
Feb 14 02:45:30 ci-worker-s390x-01 debci[1209706]: mmdebstrap testing/s390x pass
Feb 14 03:02:05 ci-worker-s390x-01 debci[3642098]: waiting for header frame: a SSL error occurred
Feb 14 04:40:10 ci-worker-s390x-01 debci[12655]: cacti testing/s390x tmpfail
Feb 14 04:56:51 ci-worker-s390x-01 debci[3015158]: waiting for header frame: a SSL error occurred
So we seem to lose at least 15 minutes of worker time when that happens.
The failures are sometimes but not necessarily correlated:
Feb 17 01:07:17 ci-worker-s390x-01 debci[1149352]: waiting for header frame: a SSL error occurred
Feb 17 01:13:46 ci-worker-s390x-01 debci[552417]: waiting for header frame: a SSL error occurred
Feb 17 01:16:19 ci-worker-s390x-01 debci[1261598]: waiting for header frame: a SSL error occurred
Feb 17 01:21:02 ci-worker-s390x-01 debci[1487252]: waiting for header frame: a SSL error occurred
Feb 17 01:53:30 ci-worker-s390x-01 debci[3589185]: waiting for header frame: a SSL error occurred
Feb 17 02:03:24 ci-worker-s390x-01 debci[4184831]: waiting for header frame: a SSL error occurred
Feb 17 02:18:31 ci-worker-s390x-01 debci[3986861]: waiting for header frame: a SSL error occurred
Feb 17 02:41:11 ci-worker-s390x-01 debci[4167140]: waiting for header frame: a SSL error occurred
Feb 17 05:44:55 ci-worker-s390x-01 debci[1543385]: waiting for header frame: a SSL error occurred
Feb 17 05:47:10 ci-worker-s390x-01 debci[2598734]: waiting for header frame: a SSL error occurred
Feb 17 06:24:39 ci-worker-s390x-01 debci[1275755]: waiting for header frame: a SSL error occurred
Feb 17 06:50:05 ci-worker-s390x-01 debci[3680449]: waiting for header frame: a SSL error occurred
Feb 17 07:33:09 ci-worker-s390x-01 debci[107515]: waiting for header frame: a SSL error occurred
Feb 17 07:48:04 ci-worker-s390x-01 debci[2816244]: waiting for header frame: a SSL error occurred
Feb 17 07:54:07 ci-worker-s390x-01 debci[2284573]: waiting for header frame: a SSL error occurred
Feb 17 12:40:38 ci-worker-s390x-01 debci[4069122]: waiting for header frame: a SSL error occurred
Feb 17 15:39:40 ci-worker-s390x-01 debci[3343838]: waiting for header frame: a SSL error occurred
Feb 17 20:23:33 ci-worker-s390x-01 debci[3531969]: waiting for header frame: a SSL error occurred
Feb 17 21:21:28 ci-worker-s390x-01 debci[1815008]: waiting for header frame: a SSL error occurred
Feb 17 23:28:02 ci-worker-s390x-01 debci[2830093]: waiting for header frame: a SSL error occurred
Feb 18 01:38:13 ci-worker-s390x-01 debci[3999976]: waiting for header frame: a SSL error occurred
Feb 18 04:21:49 ci-worker-s390x-01 debci[1774710]: waiting for header frame: a SSL error occurred
Feb 18 04:21:53 ci-worker-s390x-01 debci[1530267]: waiting for header frame: a SSL error occurred
Feb 18 04:43:09 ci-worker-s390x-01 debci[2484158]: waiting for header frame: a SSL error occurred
Feb 18 04:54:21 ci-worker-s390x-01 debci[3870455]: waiting for header frame: a SSL error occurred
Feb 18 06:46:27 ci-worker-s390x-01 debci[632005]: waiting for header frame: a SSL error occurred
Feb 18 06:52:56 ci-worker-s390x-01 debci[516286]: waiting for header frame: a SSL error occurred
Feb 18 09:41:23 ci-worker-s390x-01 debci[57375]: waiting for header frame: a SSL error occurred
It doesn't look like amqp-consume has a lot of options in this space. I
do wonder if a Wireguard tunnel would help, if only to move this from a
firewall-mediated TCP stream to a couple of UDP packets that are less
likely to be filtered. But I don't know how amenable the firewall is to
these either.
I'm personally not a friend of munin because it makes math on the graphs
hard. Do you have an idea how many packages the s390x manages to process
per day and how that compares to the other workers? PubSub queues are
not the easiest to introspect and I'd like to know how far we are off in
intake into the queue per day vs. what we can process.
Kind regards and thanks
Philipp Kern
Reply to: