Bug#1116358: linux-image-6.12.48+deb13-amd64: LVM snapshots causing I/O errors in KVM guest with aio=io_uring set
Control: tags -1 + moreinfo
On Fri, Sep 26, 2025 at 10:34:22AM +0300, Kevin Lumik wrote:
> Package: src:linux
> Version: 6.12.48-1
> Severity: normal
>
> Dear Maintainer,
>
> After upgrading from Debian Bookworm to Trixie, an issue within KVM guests when creating an LVM snapshot of its volume
> has surfaced. When a LVM snapshot is taken from the host, the guest starts to get I/O errors. The issue seems to only
> appear when aio=io_uring is specified in the KVM drive parameters and also seems to resolve when downgrading the kernel
> package down to 6.1. The issue is also not reproducible when using aio=native.
>
> KVM args for the drive: "-drive id=drive-
> virtio0,format=raw,file=/dev/dom/vps_testsql,cache=none,aio=io_uring,index=0,media=disk,if=virtio"
>
> An IO workload is being created in the VM using "fio --randrepeat=1 --ioengine=io_uring --direct=1 --gtod_reduce=1 --
> name=randwrite --filename=/root/test.bin --bs=4k --iodepth=64 --runtime=60 --numjobs=32 --readwrite=randwrite --size=1G
> --rwmixread=75 --group_reporting"
>
> After running "lvcreate -s /dev/dom/vps_testsql -n test -L 1G" on the host we can observe fio erroring out:
>
> ...
> fio: io_u error on file /root/test.bin: Input/output error: write offset=46977024, buflen=4096
> fio: pid=7367, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: io_u error on file /root/test.bin: Input/output error: write offset=261505024, buflen=4096
> fio: pid=7374, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: io_u error on file /root/test.bin: Input/output error: write offset=973840384, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=9637888, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=159965184, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=857505792, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=90787840, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=26427392, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=955621376, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=96169984, buflen=4096
> fio: pid=7372, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: pid=7362, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: io_u error on file /root/test.bin: Input/output error: write offset=203702272, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=814649344, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=91467776, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=948256768, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=105295872, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=75247616, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=1062293504, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=111955968, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=942563328, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=117354496, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=1050402816, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=125419520, buflen=4096
> fio: io_u error on file /root/test.bin: Input/output error: write offset=129044480, buflen=4096
> fio: pid=7369, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: pid=7373, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: pid=7348, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: pid=7375, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: pid=7358, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
> fio: pid=7359, err=5/file:io_u.c:1876, func=io_u error, error=Input/output error
>
> randwrite: (groupid=0, jobs=32): err= 5 (file:io_u.c:1876, func=io_u error, error=Input/output error): pid=7344: Thu Sep
> 25 16:28:30 2025
> write: IOPS=194k, BW=758MiB/s (795MB/s)(3976MiB/5244msec); 0 zone resets
> bw ( KiB/s): min=657104, max=1009816, per=100.00%, avg=785345.80, stdev=4331.73, samples=320
> iops : min=164276, max=252454, avg=196336.40, stdev=1082.93, samples=320
> cpu : usr=0.33%, sys=1.07%, ctx=18912, majf=0, minf=455
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
> issued rwts: total=0,1019955,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=64
>
> Run status group 0 (all jobs):
> WRITE: bw=758MiB/s (795MB/s), 758MiB/s-758MiB/s (795MB/s-795MB/s), io=3976MiB (4169MB), run=5244-5244msec
>
> Disk stats (read/write):
> vda: ios=1/1001361, merge=0/2, ticks=0/10314531, in_queue=10314549, util=97.82%
> Bus error
>
> ---
>
> And the error is also visible in the kernel error log of the VM:
> [ 361.962970] I/O error, dev vda, sector 83277192 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.963945] I/O error, dev vda, sector 83067480 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964489] I/O error, dev vda, sector 82881208 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964499] I/O error, dev vda, sector 83031976 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964501] I/O error, dev vda, sector 83089832 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964503] I/O error, dev vda, sector 83156184 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964522] I/O error, dev vda, sector 83183384 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964524] I/O error, dev vda, sector 83310704 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964525] I/O error, dev vda, sector 83311144 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
> [ 361.964532] I/O error, dev vda, sector 83315272 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 2
>
> The host does not generate any erros, they only seem to occur within the VM.
Thanks for your report.
Two questions:
- Can you temporarily test as well the upper kernels from unstable
(6.16.8-1 right now, but soonish 6.16.9-1 as well) to see if the
problem is triggerable there as well?
- Depending on the above question, would you be able to bisect
upstream changes to determine either the introducing commit, or if
we find it is fixed in 6.16.y to determine the fixing commit, so we
might assess why it is missing in 6.12.y?
Note: If we find the problem is prosent in 6.16.y then we need to find
the breaking change between 6.1.y and 6.12.y. It might be asier here
if we first make the range of potential upstream versions more narrow,
so you might first do a "manual bisect" on Debian revisions (For that
use the snapshot.debian.org service to retrieve linux-images uploaded
to Debian. E.g. go down from 6.12 in "major version" steps until we
have close enough range of broken/fixed upstream major versions. Then
perform a normal bisect between the upstream versions.
Let me know if you need more details on the bisection steps.
Regards,
Salvatore
Reply to: