[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#1015272: liburing autopkgtest started to hang containers in Debian and Ubuntu since ~2022-07-11



Control: fixed -1 5.10.162-1 6.1.8-1

Hi,

On Sun, 21 Aug 2022 21:35:58 +0200 Bastian Blank <waldi@debian.org> wrote:
On Sun, Aug 21, 2022 at 07:42:10PM +0200, Guillem Jover wrote:
> It seems like there was a regression with the latest stable update
> that affects the autopkgtest for liburing. Reassigning.

Please provide enough information to make isolating the problem
possible.

https://ci.debian.net/packages/libu/liburing/ is completely silent as
there are not results for any of the failed runs.

I decided to try again to see if I could collect more information. The test now passes on amd64, arm64, i386 and ppc64el, all running 5.10.162-1 and on riscv64 running unstable. However, on armhf, armel (amd64 kernel) and s390x (all running 5.10.158-2), it seems that the observation of brian is still true, some test in test-unit test segfaults, the test exits and hangs. @Guillem, do you see something more in the output below (armhf log) that may be of interest? And maybe spot something to run in isolation?

When I try to destroy the lxc, that fails and in ps output I see this:
root 3053528 0.0 0.0 5388 3072 ? Ss 03:34 0:00 [lxc monitor] /var/lib/lxc ci-061-8c60e21c root 3061512 0.0 0.0 0 0 ? Ss 03:35 0:00 \_ [systemd] debian 3110684 0.0 0.0 2140 192 ? DL 03:37 0:00 \_ ./iopoll-leak.t

Note the "D" state.

Reading the changelog of 5.10.162-1 I see io_uring mentioned a couple of times. Therefor I assume this bug is fixed in that version. Is it worth pursuing the real issue here?

Paul

root@ci-061-705317d0:/tmp/autopkgtest-lxc.v8gx_5j5/downtmp# cat test-unit-stdout
+ [ -n  ]
+ CC=gcc
+ ./configure --cc=gcc
prefix                        /usr
includedir                    /usr/include
libdir                        /usr/lib
libdevdir                     /usr/lib
relativelibdir
mandir                        /usr/man
datadir                       /usr/share
stringop_overflow             yes
array_bounds                  yes
__kernel_rwf_t                yes
__kernel_timespec             yes
open_how                      yes
statx                         yes
glibc_statx                   yes
C++                           yes
has_ucontext                  yes
NVMe uring command support    yes
liburing_nolibc               no
CC                            gcc
CXX                           g++
+ make runtests
make[1]: Entering directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/src'
     CC setup.ol
     CC queue.ol
     CC register.ol
     CC syscall.ol
     AR liburing.a
ar: creating liburing.a
 RANLIB liburing.a
     CC setup.os
     CC queue.os
     CC register.os
     CC syscall.os
     CC liburing.so.2.3
make[1]: Leaving directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/src' make[1]: Entering directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/test'
     CC helpers.o
     CC 232c93d07b74.t
     CC 35fa71a030ca.t
     CC 500f9fbadef8.t
     CC 7ad0e4b2f83c.t
     CC 8a9973408177.t
     CC 917257daa0fe.t
     CC a0908ae19763.t
     CC a4c0b3decb33.t
     CC accept.t
     CC accept-link.t
     CC accept-reuse.t
     CC accept-test.t
     CC across-fork.t
     CC b19062a56726.t
     CC b5837bd5311d.t
     CC buf-ring.t
     CC ce593a6c480a.t
     CC close-opath.t
     CC connect.t
     CC cq-full.t
     CC cq-overflow.t
     CC cq-peek-batch.t
     CC cq-ready.t
     CC cq-size.t
     CC d4ae271dfaae.t
     CC d77a67ed5f27.t
     CC defer.t
     CC defer-taskrun.t
     CC double-poll-crash.t
     CC drop-submit.t
     CC eeed8b54e0df.t
     CC empty-eownerdead.t
     CC eventfd.t
     CC eventfd-disable.t
     CC eventfd-reg.t
     CC eventfd-ring.t
     CC exec-target.t
     CC exit-no-cleanup.t
     CC fadvise.t
     CC fallocate.t
     CC fc2a85cb02ef.t
     CC fd-pass.t
     CC file-register.t
     CC files-exit-hang-poll.t
     CC files-exit-hang-timeout.t
     CC file-update.t
     CC file-verify.t
     CC fixed-buf-iter.t
     CC fixed-link.t
     CC fixed-reuse.t
     CC fpos.t
     CC fsync.t
     CC hardlink.t
     CC io-cancel.t
     CC iopoll.t
     CC iopoll-leak.t
     CC io_uring_enter.t
     CC io_uring_passthrough.t
     CC io_uring_register.t
     CC io_uring_setup.t
     CC lfs-openat.t
     CC lfs-openat-write.t
     CC link.t
     CC link_drain.t
     CC link-timeout.t
     CC madvise.t
     CC mkdir.t
     CC msg-ring.t
     CC multicqes_drain.t
     CC nolibc.t
     CC nop-all-sizes.t
     CC nop.t
     CC openat2.t
     CC open-close.t
     CC open-direct-link.t
     CC open-direct-pick.t
     CC personality.t
     CC pipe-eof.t
     CC pipe-reuse.t
     CC poll.t
     CC poll-cancel.t
     CC poll-cancel-all.t
     CC poll-cancel-ton.t
     CC poll-link.t
     CC poll-many.t
     CC poll-mshot-update.t
     CC poll-mshot-overflow.t
     CC poll-ring.t
     CC poll-v-poll.t
     CC pollfree.t
     CC probe.t
     CC read-before-exit.t
     CC read-write.t
     CC recv-msgall.t
     CC recv-msgall-stream.t
     CC recv-multishot.t
     CC register-restrictions.t
     CC rename.t
     CC ringbuf-read.t
     CC ring-leak2.t
     CC ring-leak.t
     CC rsrc_tags.t
     CC rw_merge_test.t
     CC self.t
     CC sendmsg_fs_cve.t
     CC send_recv.t
     CC send_recvmsg.t
     CC shared-wq.t
     CC short-read.t
     CC shutdown.t
     CC sigfd-deadlock.t
     CC skip-cqe.t
     CC socket.t
     CC socket-rw.t
     CC socket-rw-eagain.t
     CC socket-rw-offset.t
     CC splice.t
     CC sq-full.t
     CXX sq-full-cpp.t
     CC sqpoll-cancel-hang.t
     CC sqpoll-disable-exit.t
     CC sq-poll-dup.t
     CC sqpoll-exit-hang.t
     CC sq-poll-kthread.t
     CC sq-poll-share.t
     CC sqpoll-sleep.t
     CC sq-space_left.t
     CC stdout.t
     CC submit-and-wait.t
     CC submit-link-fail.t
     CC submit-reuse.t
     CC sync-cancel.t
     CC symlink.t
     CC teardowns.t
     CC thread-exit.t
     CC timeout.t
     CC timeout-new.t
     CC timeout-overflow.t
     CC tty-write-dpoll.t
     CC unlink.t
     CC wakeup-hang.t
     CC xattr.t
     CC single-issuer.t
     CC send-zerocopy.t
     CC statx.t
make[1]: Leaving directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/test' make[1]: Entering directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/examples'
     CC io_uring-cp
     CC io_uring-test
     CC io_uring-udp
     CC link-cp
     CC poll-bench
     CC send-zerocopy
     CC ucontext-cp
make[1]: Leaving directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/examples' make[1]: Entering directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/test'
Running test 232c93d07b74.t                                         4 sec
Running test 35fa71a030ca.t                                         5 sec
Running test 500f9fbadef8.t                                         14 sec
Running test 7ad0e4b2f83c.t                                         1 sec
Running test 8a9973408177.t                                         0 sec
Running test 917257daa0fe.t                                         0 sec
Running test a0908ae19763.t                                         0 sec
Running test a4c0b3decb33.t                                         11 sec
Running test accept.t Accept[0] got -11
test_accept_nonblock(before, 1) failed
Test accept.t failed with ret 1
Running test accept-link.t                                          0 sec
Running test accept-reuse.t                                         1 sec
Running test accept-test.t accept() failed to use addr & addrlen parameters!
Test accept-test.t failed with ret 1
Running test across-fork.t                                          0 sec
Running test b19062a56726.t                                         0 sec
Running test b5837bd5311d.t                                         0 sec
Running test buf-ring.t                                             0 sec
Running test ce593a6c480a.t                                         1 sec
Running test close-opath.t                                          0 sec
Running test connect.t                                              0 sec
Running test cq-full.t                                              0 sec
Running test cq-overflow.t cq-overflow.t: cq-overflow.c:298: submit_one_nop: Assertion `ret == 1' failed. ./runtests.sh: line 66: 7389 Aborted timeout -s INT -k $TIMEOUT $TIMEOUT "${test_exec[@]}"
Test cq-overflow.t failed with ret 134
Running test cq-peek-batch.t                                        0 sec
Running test cq-ready.t                                             0 sec
Running test cq-size.t                                              0 sec
Running test d4ae271dfaae.t SQPOLL skipped for regular user
Skipped
Running test d77a67ed5f27.t SQPOLL skipped for regular user
Skipped
Running test defer.t SQPOLL skipped for regular user
Skipped
Running test defer-taskrun.t                                        Skipped
Running test double-poll-crash.t                                    Skipped
Running test drop-submit.t                                          0 sec
Running test eeed8b54e0df.t                                         0 sec
Running test empty-eownerdead.t SQPOLL skipped for regular user
Skipped
Running test eventfd.t                                              0 sec
Running test eventfd-disable.t                                      0 sec
Running test eventfd-reg.t                                          6 sec
Running test eventfd-ring.t                                         0 sec
Running test exec-target.t                                          0 sec
Running test exit-no-cleanup.t                                      0 sec
Running test fadvise.t                                              0 sec
Running test fallocate.t                                            0 sec
Running test fc2a85cb02ef.t Test needs failslab/fail_futex/fail_page_alloc enabled, skipped
Skipped
Running test fd-pass.t                                              Skipped
Running test file-register.t Skipping files not supported
file alloc ranges are not supported, skip
4 sec
Running test files-exit-hang-poll.t                                 1 sec
Running test files-exit-hang-timeout.t                              1 sec
Running test file-update.t                                          1 sec
Running test file-verify.t                                          4 sec
Running test fixed-buf-iter.t                                       0 sec
Running test fixed-link.t                                           0 sec
Running test fixed-reuse.t                                          Skipped
Running test fpos.t inconsistent reads, got 0s:8192 1s:6144
f_pos incorrect, expected 14336 have 7
failed read async=0 blocksize=7
Test fpos.t failed with ret 255
Running test fsync.t                                                0 sec
Running test hardlink.t linkat not supported, skipping
Skipped
Running test io-cancel.t read got cancelled or wait failed
test_dont_cancel_another_ring() failed
Test io-cancel.t failed with ret 1
Running test iopoll.t                                               1 sec
Running test iopoll-leak.t Test iopoll-leak.t timed out (may not be a failure)
Running test io_uring_enter.t                                       0 sec
Running test io_uring_passthrough.t                                 Skipped
Running test io_uring_register.t Unable to map a huge page. Try increasing /proc/sys/vm/nr_hugepages by at least 1.
Skipping the hugepage test
0 sec
Running test io_uring_setup.t expected -22, got -1
FAIL
Test io_uring_setup.t failed with ret 1
Running test lfs-openat.t                                           0 sec
Running test lfs-openat-write.t                                     0 sec
Running test link.t                                                 0 sec
Running test link_drain.t                                           1 sec
Running test link-timeout.t                                         1 sec
Running test madvise.t                                              0 sec
Running test mkdir.t mkdirat not supported, skipping
Skipped
Running test msg-ring.t                                             Skipped
Skipped
Running test multicqes_drain.t Test multicqes_drain.t timed out (may not be a failure)
Running test nolibc.t                                               Skipped
Running test nop-all-sizes.t                                        0 sec
Running test nop.t                                                  0 sec
Running test openat2.t fixed open isn't supported
0 sec
Running test open-close.t close for fixed files is not supported
0 sec
Running test open-direct-link.t                                     0 sec
Running test open-direct-pick.t                                     0 sec
Running test personality.t Not root, skipping
0 sec
Running test pipe-eof.t                                             0 sec
Running test pipe-reuse.t                                           0 sec
Running test poll.t                                                 0 sec
Running test poll-cancel.t                                          0 sec
Running test poll-cancel-all.t                                      0 sec
Running test poll-cancel-ton.t                                      0 sec
Running test poll-link.t                                            1 sec
Running test poll-many.t                                            6 sec
Running test poll-mshot-update.t no poll update, skip
0 sec
Running test poll-mshot-overflow.t ./poll-mshot-overflow.t: test(false) failed
Skipped
Running test poll-ring.t                                            0 sec
Running test poll-v-poll.t                                          0 sec
Running test pollfree.t                                             0 sec
Running test probe.t                                                0 sec
Running test read-before-exit.t ./runtests.sh: line 66: 7853 Killed timeout -s INT -k $TIMEOUT $TIMEOUT "${test_exec[@]}"
Test read-before-exit.t failed with ret 137
Running test read-write.t SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
SQPOLL skipped for regular user
Not root, skipping test_write_efbig
SQPOLL skipped for regular user
SQPOLL skipped for regular user
2 sec
Running test recv-msgall.t                                          0 sec
Running test recv-msgall-stream.t got wrong length: 256
test recv failed
Test recv-msgall-stream.t failed with ret 1
Running test recv-multishot.t                                       Skipped
Running test register-restrictions.t                                1 sec
Running test rename.t Rename not supported, skipping
0 sec
Running test ringbuf-read.t                                         0 sec
Running test ring-leak2.t                                           1 sec
Running test ring-leak.t Test ring-leak.t timed out (may not be a failure) Running test rsrc_tags.t doesn't support rsrc tags, skip
0 sec
Running test rw_merge_test.t                                        0 sec
Running test self.t                                                 0 sec
Running test sendmsg_fs_cve.t chroot not allowed, skip
0 sec
Running test send_recv.t SQPOLL skipped for regular user
SQPOLL skipped for regular user
0 sec
Running test send_recvmsg.t                                         0 sec
Running test shared-wq.t                                            0 sec
Running test short-read.t                                           0 sec
Running test shutdown.t Shutdown not supported, skipping
0 sec
Running test sigfd-deadlock.t                                       0 sec
Running test skip-cqe.t IOSQE_CQE_SKIP_SUCCESS is not supported, skip
1 sec
Running test socket.t No socket support, skipping
0 sec
Running test socket-rw.t                                            0 sec
Running test socket-rw-eagain.t                                     0 sec
Running test socket-rw-offset.t Got -29, expected 128
Test socket-rw-offset.t failed with ret 1
Running test splice.t                                               0 sec
Running test sq-full.t                                              0 sec
Running test sq-full-cpp.t                                          0 sec
Running test sqpoll-cancel-hang.t                                   1 sec
Running test sqpoll-disable-exit.t                                  1 sec
Running test sq-poll-dup.t queue_init: -1/0
test 0 0 failed
Test sq-poll-dup.t failed with ret 1
Running test sqpoll-exit-hang.t ./sqpoll-exit-hang.t: skipped, not root
0 sec
Running test sq-poll-kthread.t SQPOLL skipped for regular user
test_sq_poll_kthread_stopped_exit failed
Test sq-poll-kthread.t failed with ret 2
Running test sq-poll-share.t queue_init: -1/0
Test sq-poll-share.t failed with ret 1
Running test sqpoll-sleep.t ./sqpoll-sleep.t: skipped, not root
0 sec
Running test sq-space_left.t                                        0 sec
Running test stdout.t This is a pipe test
This is a fixed pipe test
0 sec
Running test submit-and-wait.t                                      1 sec
Running test submit-link-fail.t cancel failed 0, ud 0
failed 1, hard 0, drain 0,link_last 0, size 2, idx 1
Test submit-link-fail.t failed with ret 1
Running test submit-reuse.t                                         4 sec
Running test sync-cancel.t Test sync-cancel.t timed out (may not be a failure) Running test symlink.t symlinkat not supported, skipping
0 sec
Running test teardowns.t                                            0 sec
Running test thread-exit.t cqe->res=-125, Expected 512
Test thread-exit.t failed with ret 1
Running test timeout.t ./runtests.sh: line 66: 8367 Segmentation fault timeout -s INT -k $TIMEOUT $TIMEOUT "${test_exec[@]}"
Test timeout.t failed with ret 139
Running test timeout-new.t feature IORING_FEAT_EXT_ARG not supported, skipping.
0 sec
Running test timeout-overflow.t                                     Skipped
Running test tty-write-dpoll.t                                      0 sec
Running test unlink.t Unlink not supported, skipping
0 sec
Running test wakeup-hang.t                                          2 sec
Running test xattr.t No xattr support, skipping
0 sec
Running test skip-cqe.t IOSQE_CQE_SKIP_SUCCESS is not supported, skip
0 sec [1]
Running test single-issuer.t SETUP_SINGLE_ISSUER is not supported, skip
Skipped
Running test send-zerocopy.t                                        Skipped
Running test statx.t                                                0 sec
Running test sq-full-cpp.t 0 sec [0] Tests timed out (4): <iopoll-leak.t> <multicqes_drain.t> <ring-leak.t> <sync-cancel.t> Tests failed (15): <accept.t> <accept-test.t> <cq-overflow.t> <fpos.t> <io-cancel.t> <io_uring_setup.t> <read-before-exit.t> <recv-msgall-stream.t> <socket-rw-offset.t> <sq-poll-dup.t> <sq-poll-kthread.t> <sq-poll-share.t> <submit-link-fail.t> <thread-exit.t> <timeout.t>
make[1]: *** [Makefile:231: runtests] Error 1
make[1]: Leaving directory '/tmp/autopkgtest-lxc.v8gx_5j5/downtmp/build.ksh/src/test'
make: *** [Makefile:21: runtests] Error 2


s390x has the following ending:
Tests timed out (4): <iopoll-leak.t> <multicqes_drain.t> <ring-leak.t> <sync-cancel.t> Tests failed (18): <accept.t> <accept-test.t> <cq-overflow.t> <fpos.t> <io-cancel.t> <io_uring_setup.t> <read-before-exit.t> <recv-msgall-stream.t> <recv-multishot.t> <rw_merge_test.t> <send_recv.t> <socket-rw-offset.t> <sq-poll-dup.t> <sq-poll-kthread.t> <sq-poll-share.t> <submit-link-fail.t> <thread-exit.t> <timeout.t>

Attachment: OpenPGP_signature
Description: OpenPGP digital signature


Reply to: