Bug#1070792: flatpak: flaky autopkgtest on amd64: test-summaries: File 'httpd-log' doesn't match regexp
Source: flatpak
Version: 1.14.8-1
Severity: important
Tags: help upstream
User: debian-ci@lists.debian.org
Usertags: flaky
X-Debbugs-Cc: debian-ci@lists.debian.org
Control: affects -1 + src:glib2.0
Control: found -1 1.4.6-1
One of the tests in Flatpak's extensive test-suite appears to be "flaky":
that is, it usually passes, but is not completely reliable.
This means that when packages that are depended on by the test suite, such
as the glib2.0 security fix currently in unstable, are trying to migrate
to testing, the test will randomly pass or fail, causing those packages
to be detected as having caused a regression when in fact they have not.
For example see
https://ci.debian.net/packages/f/flatpak/unstable/amd64/46118904/
(while testing a new systemd from experimental),
https://ci.debian.net/packages/f/flatpak/testing/amd64/46456176/
(while testing a new glib2.0) and
https://ci.debian.net/packages/f/flatpak/unstable/amd64/45863529/
(while testing a new glib2.0 from experimental).
I suspect this is a race condition in the test, rather than a result of
a change immediately before one of those runs. This appears to happen
rather rarely.
The failure mode is that one of several instantiations of test-summaries.sh
(which is run multiple times with different options, and I have never seen
more than one of its runs per test fail) will fail like this:
378s ++ assert_not_file_has_content httpd-log summaries/cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.gz
378s ++ assert_file_has_content httpd-log summaries/7d82d86befacc5608749cb851a2deaa04156345146dbfd8970b6ff97a321db23-cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.delta
378s File 'httpd-log' doesn't match regexp 'summaries/7d82d86befacc5608749cb851a2deaa04156345146dbfd8970b6ff97a321db23-cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.delta' at test-summaries.sh:258
378s OK closing connection
378s fusermount: entry for /tmp/test-flatpak-0Wl0GX/runtime/doc not found in /etc/mtab
378s 127.0.0.1 - - [09/May/2024 00:24:18] "GET /test/summaries/7d82d86befacc5608749cb851a2deaa04156345146dbfd8970b6ff97a321db23-cb09c0ee49abd2a14a6a556463d4cf696e5f58a2341112c612ea533ac18531f9.delta HTTP/1.1" 200 -
378s FAIL: Flatpak/test-summaries@system.wrap.test (Child process exited with code 1)
This is fairly obviously a race condition: the desired request does get
written to the web server log, but not until after the test script has
checked for it, not found it, and made the test fail. The test probably
needs to loop until the desired output is seen in the log, or make some
other distinctive request as a marker and loop until that request is
seen in the log.
I am aware that, in practice, I am this package's only uploader, so this
is in some sense my fault, and the community will presumably now demand
that I drop everything and fix it. I do not have the bandwidth to be
the single point of failure for the various things that I have ended up
responsible for, and I would very much appreciate it if other contributors
could help to improve the reliability of this test suite upstream.
smcv
Reply to: