Bug#996951: linux-image-5.14.0-3-amd64: iouring looses requests
Package: src:linux
Version: 5.14.12-1
Severity: grave
Justification: causes non-serious data loss
X-Debbugs-Cc: daniel@mariadb.org
Dear Maintainer,
MariaDB has been investigating a 10.6+ related problem for a while
https://jira.mariadb.org/browse/MDEV-26674
https://jira.mariadb.org/browse/MDEV-26555
The result of this investigation is that between 5.11 and fixed in 5.15
is a uring kernel related fault that results in a write request getting
lost.
The result of this is that MariaDB-10.6 users, and perhaps other
applications using the iouring kernel interface will loose either
availablity or data.
The good news is I've validated that the linux mainline 5.14.14 build
from https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.14.14/ has
actually fixed this problem.
As such this affects bullseye-backports, bookworm and side currently.
This can be validated by installing mariadb-test-10.6 from MariaDB's
repo.
https://mariadb.org/download/#mariadb-repositories
To test run:
cd /usr/share/mysql/mysql-test
./mtr --vardir=/tmp/var --parallel=4 encryption.innochecksum{,,,,,}
./mtr --vardir=/tmp/var --parallel=4 stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb stress.ddl_innodb
A test failure (after a large timeout 10 mins I think) results in the
mariadb error:
2021-10-21 9:08:43 0 [ERROR] [FATAL] InnoDB: innodb_fatal_semaphore_wait_threshold was exceeded for dict_sys.latch. Please refer to https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/
Marko (MariaDB innodb maintainer) in https://jira.mariadb.org/browse/MDEV-26674?focusedCommentId=202674&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-202674
already validated the problem on sid.
This is reported on an ubuntu impish machine with the Debian kernel
installed to eliminate any other userspace effects that may have caused
this.
-- Package-specific info:
** Version:
Linux version 5.14.0-3-amd64 (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.3.0-11) 10.3.0, GNU ld (GNU Binutils for Debian) 2.37) #1 SMP Debian 5.14.12-1 (2021-10-14)
** Command line:
BOOT_IMAGE=/vmlinuz-5.14.0-3-amd64 root=/dev/mapper/ubuntu--vg-ubuntu--lv ro
** Not tainted
** Kernel log:
[ 1.866883] ata4: SATA link down (SStatus 0 SControl 300)
[ 1.869434] scsi 0:0:0:0: CD-ROM QEMU QEMU DVD-ROM 2.5+ PQ: 0 ANSI: 5
[ 1.878673] ata6: SATA link down (SStatus 0 SControl 300)
[ 1.880335] ata5: SATA link down (SStatus 0 SControl 300)
[ 1.915393] sr 0:0:0:0: [sr0] scsi3-mmc drive: 4x/4x cd/rw xa/form2 tray
[ 1.916496] cdrom: Uniform CD-ROM driver Revision: 3.20
[ 1.958195] sr 0:0:0:0: Attached scsi CD-ROM sr0
[ 2.217095] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input3
[ 2.225457] pcieport 0000:00:02.6: pciehp: Slot(0-6): No device found
[ 2.525250] raid6: avx2x4 gen() 29809 MB/s
[ 2.593249] raid6: avx2x4 xor() 13304 MB/s
[ 2.661249] raid6: avx2x2 gen() 34952 MB/s
[ 2.729248] raid6: avx2x2 xor() 20847 MB/s
[ 2.797248] raid6: avx2x1 gen() 29982 MB/s
[ 2.865248] raid6: avx2x1 xor() 16322 MB/s
[ 2.933259] raid6: sse2x4 gen() 14256 MB/s
[ 3.001247] raid6: sse2x4 xor() 8371 MB/s
[ 3.069248] raid6: sse2x2 gen() 14546 MB/s
[ 3.137248] raid6: sse2x2 xor() 8631 MB/s
[ 3.205249] raid6: sse2x1 gen() 12159 MB/s
[ 3.273248] raid6: sse2x1 xor() 6207 MB/s
[ 3.273649] raid6: using algorithm avx2x2 gen() 34952 MB/s
[ 3.274147] raid6: .... xor() 20847 MB/s, rmw enabled
[ 3.274552] raid6: using avx2x2 recovery algorithm
[ 3.278389] xor: automatically using best checksumming function avx
[ 3.282251] async_tx: api initialized (async)
[ 3.433164] Btrfs loaded, crc32c=crc32c-intel, zoned=yes
[ 3.667862] EXT4-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 3.759716] Not activating Mandatory Access Control as /sbin/tomoyo-init does not exist.
[ 3.853765] systemd[1]: Inserted module 'autofs4'
[ 3.893289] systemd[1]: systemd 248.3-1ubuntu8 running in system mode. (+PAM +AUDIT +SELINUX +APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS -OPENSSL +ACL +BLKID +CURL +ELFUTILS -FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP -LIBFDISK +PCRE2 -PWQUALITY -P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[ 3.895610] systemd[1]: Detected virtualization kvm.
[ 3.896183] systemd[1]: Detected architecture x86-64.
[ 3.907380] systemd[1]: Hostname set to <impish>.
[ 4.275407] systemd[1]: Queued start job for default target Graphical Interface.
[ 4.278640] systemd[1]: Created slice system-modprobe.slice.
[ 4.284509] systemd[1]: Created slice system-systemd\x2dfsck.slice.
[ 4.287754] systemd[1]: Created slice User and Session Slice.
[ 4.289457] systemd[1]: Started Forward Password Requests to Wall Directory Watch.
[ 4.291130] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[ 4.293130] systemd[1]: Reached target Slices.
[ 4.295232] systemd[1]: Reached target Local Verity Integrity Protected Volumes.
[ 4.297851] systemd[1]: Listening on Device-mapper event daemon FIFOs.
[ 4.299603] systemd[1]: Listening on LVM2 poll daemon socket.
[ 4.301182] systemd[1]: Listening on multipathd control socket.
[ 4.303065] systemd[1]: Listening on fsck to fsckd communication Socket.
[ 4.304524] systemd[1]: Listening on initctl Compatibility Named Pipe.
[ 4.306292] systemd[1]: Listening on Journal Audit Socket.
[ 4.307941] systemd[1]: Listening on Journal Socket (/dev/log).
[ 4.310054] systemd[1]: Listening on Journal Socket.
[ 4.311913] systemd[1]: Listening on Network Service Netlink Socket.
[ 4.313937] systemd[1]: Listening on udev Control Socket.
[ 4.315747] systemd[1]: Listening on udev Kernel Socket.
[ 4.318290] systemd[1]: Mounting Huge Pages File System...
[ 4.320763] systemd[1]: Mounting POSIX Message Queue File System...
[ 4.323184] systemd[1]: Mounting Kernel Debug File System...
[ 4.328055] systemd[1]: Mounting Kernel Trace File System...
[ 4.332963] systemd[1]: Starting Journal Service...
[ 4.335422] systemd[1]: Starting Set the console keyboard layout...
[ 4.337771] systemd[1]: Starting Create list of static device nodes for the current kernel...
[ 4.340979] systemd[1]: Starting Monitoring of LVM2 mirrors, snapshots etc. using dmeventd or progress polling...
[ 4.346661] systemd[1]: Starting Load Kernel Module configfs...
[ 4.369499] systemd[1]: Starting Load Kernel Module drm...
[ 4.377752] systemd[1]: Starting Load Kernel Module fuse...
[ 4.379529] systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
[ 4.383612] systemd[1]: Condition check resulted in Set Up Additional Binary Formats being skipped.
[ 4.385537] systemd[1]: Condition check resulted in File System Check on Root Device being skipped.
[ 4.390332] systemd[1]: Starting Load Kernel Modules...
[ 4.395567] systemd[1]: Starting Remount Root and Kernel File Systems...
[ 4.401664] systemd[1]: Starting Coldplug All udev Devices...
[ 4.408033] EXT4-fs (dm-0): re-mounted. Opts: (null). Quota mode: none.
[ 4.413624] fuse: init (API version 7.34)
[ 4.415296] systemd[1]: Started Journal Service.
[ 4.476521] Adding 4019196k swap on /swap.img. Priority:-2 extents:7 across:4846588k FS
[ 4.498562] systemd-journald[375]: Received client request to flush runtime journal.
[ 4.779184] sr 0:0:0:0: Attached scsi generic sg0 type 5
[ 4.905351] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[ 5.043460] snd_hda_codec_generic hdaudioC0D0: autoconfig for Generic: line_outs=1 (0x3/0x0/0x0/0x0/0x0) type:line
[ 5.043465] snd_hda_codec_generic hdaudioC0D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[ 5.043467] snd_hda_codec_generic hdaudioC0D0: hp_outs=0 (0x0/0x0/0x0/0x0/0x0)
[ 5.043468] snd_hda_codec_generic hdaudioC0D0: mono: mono_out=0x0
[ 5.043469] snd_hda_codec_generic hdaudioC0D0: inputs:
[ 5.043470] snd_hda_codec_generic hdaudioC0D0: Line=0x5
[ 5.135361] intel_pmc_core intel_pmc_core.0: initialized
[ 5.256674] alua: device handler registered
[ 5.259674] emc: device handler registered
[ 5.262224] rdac: device handler registered
[ 5.364044] EXT4-fs (vda2): mounted filesystem with ordered data mode. Opts: (null). Quota mode: none.
[ 5.450722] audit: type=1400 audit(1634806008.296:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=581 comm="apparmor_parser"
[ 5.451038] audit: type=1400 audit(1634806008.296:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=582 comm="apparmor_parser"
[ 5.451042] audit: type=1400 audit(1634806008.296:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=582 comm="apparmor_parser"
[ 5.468431] audit: type=1400 audit(1634806008.312:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine" pid=586 comm="apparmor_parser"
[ 5.468444] audit: type=1400 audit(1634806008.312:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/snapd/snap-confine//mount-namespace-capture-helper" pid=586 comm="apparmor_parser"
[ 5.481385] audit: type=1400 audit(1634806008.328:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=585 comm="apparmor_parser"
[ 5.481391] audit: type=1400 audit(1634806008.328:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=585 comm="apparmor_parser"
[ 5.481393] audit: type=1400 audit(1634806008.328:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/connman/scripts/dhclient-script" pid=585 comm="apparmor_parser"
[ 5.481395] audit: type=1400 audit(1634806008.328:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/{,usr/}sbin/dhclient" pid=585 comm="apparmor_parser"
[ 8.437751] loop: module loaded
[ 8.438109] loop0: detected capacity change from 0 to 8
[ 8.458870] squashfs: version 4.0 (2009/01/31) Phillip Lougher
** Model information
sys_vendor: QEMU
product_name: Standard PC (Q35 + ICH9, 2009)
product_version: pc-q35-5.2
chassis_vendor: QEMU
chassis_version: pc-q35-5.2
bios_vendor: SeaBIOS
bios_version: 1.14.0-4.fc34
** Loaded modules:
squashfs
loop
dm_multipath
scsi_dh_rdac
scsi_dh_emc
scsi_dh_alua
intel_rapl_msr
intel_rapl_common
intel_pmc_core_pltdrv
intel_pmc_core
kvm_intel
snd_hda_codec_generic
ledtrig_audio
kvm
snd_hda_intel
irqbypass
snd_intel_dspcfg
snd_intel_sdw_acpi
rapl
snd_hda_codec
snd_hda_core
snd_hwdep
snd_pcm
snd_timer
serio_raw
snd
virtio_balloon
sg
soundcore
virtio_console
joydev
evdev
qemu_fw_cfg
sch_fq_codel
fuse
configfs
virtio_rng
rng_core
ip_tables
x_tables
autofs4
ext4
crc16
mbcache
jbd2
btrfs
blake2b_generic
zstd_compress
raid10
raid456
async_raid6_recov
async_memcpy
async_pq
async_xor
async_tx
xor
raid6_pq
libcrc32c
crc32c_generic
raid1
raid0
multipath
linear
md_mod
sr_mod
cdrom
hid_generic
usbhid
hid
crct10dif_pclmul
crct10dif_common
crc32_pclmul
crc32c_intel
ghash_clmulni_intel
dm_mod
qxl
drm_ttm_helper
ttm
drm_kms_helper
ahci
libahci
virtio_net
virtio_blk
net_failover
failover
cec
rc_core
libata
drm
aesni_intel
crypto_simd
scsi_mod
psmouse
cryptd
xhci_pci
xhci_hcd
usbcore
i2c_i801
i2c_smbus
lpc_ich
usb_common
virtio_pci
virtio_pci_modern_dev
virtio_ring
virtio
button
** PCI devices:
not available
** USB devices:
not available
-- System Information:
Distributor ID: Ubuntu
Description: Ubuntu 21.10
Release: 21.10
Codename: impish
Architecture: amd64 (x86_64)
Kernel: Linux 5.14.0-3-amd64 (SMP w/2 CPU threads)
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages linux-image-5.14.0-3-amd64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.140ubuntu6
ii kmod 28-1ubuntu4
ii linux-base 4.5ubuntu9
Versions of packages linux-image-5.14.0-3-amd64 recommends:
ii apparmor 3.0.3-0ubuntu1
pn firmware-linux-free <none>
Versions of packages linux-image-5.14.0-3-amd64 suggests:
pn debian-kernel-handbook <none>
ii grub-pc 2.04-1ubuntu47
pn linux-doc-5.14 <none>
Versions of packages linux-image-5.14.0-3-amd64 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
pn firmware-misc-nonfree <none>
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>
-- no debconf information
Reply to: