Bug#1116358: linux-image-6.12.48+deb13-amd64: LVM snapshots causing I/O errors in KVM guest with aio=io_uring set

[Salvatore Bonaccorso <carnil@debian.org>]

Control: tags -1 + moreinfo

Hi Kevin,

On Thu, Oct 02, 2025 at 02:28:48PM +0300, Kevin Lumik wrote:
> On Sat, 27 Sep 2025 16:09:08 +0200 Salvatore Bonaccorso
> <carnil@debian.org> wrote:
> > - Can you temporarily test as well the upper kernels from unstable
> >   (6.16.8-1 right now, but soonish 6.16.9-1 as well) to see if the
> >   problem is triggerable there as well?
> 
> I tested the upper kernels and was still able to reproduce the issue on
> both 6.16.8-1 and 6.16.9-1.

Ok thanks for doing so, have just updated as well the metadata to
reflect that.

> > Note: If we find the problem is prosent in 6.16.y then we need to find
> > the breaking change between 6.1.y and 6.12.y. It might be asier here
> > if we first make the range of potential upstream versions more narrow,
> > so you might first do a "manual bisect" on Debian revisions (For that
> > use the snapshot.debian.org service to retrieve linux-images uploaded
> > to Debian. E.g. go down from 6.12 in "major version" steps until we
> > have close enough range of broken/fixed upstream major versions. Then
> > perform a normal bisect between the upstream versions.
> 
> Using snapshot.debian.org I have determined that the breaking change was
> introduced between 6.11.10 (tested 6.11.10-1) and 6.12 (tested 6.12.3-1
> and 6.12~rc6-1~exp1).
> 
> Thank you for your assistance. Let me know what would be the next step.

Okay so we have a good range to look at. The next steps would be to
bisect the changes. Actually it would be good to first test 6.11
itself upstream. So the steps to do would be roughly the following,
which needs compiling and testing a few kernels:

    git clone https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
    cd linux-stable
    git checkout v6.11
    cp /boot/config-$(uname -r) .config
    yes '' | make localmodconfig
    make savedefconfig
    mv defconfig arch/x86/configs/my_defconfig

    # test 6.11 to ensure this is "good"
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm it successfully boots / problem does not exist

    # test 6.12-rc6 to ensure this is "bad"
    git checkout v6.12-rc6
    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install the resulting .deb package and confirm it fails to boot / problem exists

With that confirmed, the bisection can start:

    git bisect start
    git bisect good v6.11
    git bisect bad v6.12-rc6

In each bisection step git checks out a state between the oldest
known-bad and the newest known-good commit. In each step test using:

    make my_defconfig
    make -j $(nproc) bindeb-pkg
    ... install, try to boot / verify if problem exists

and if the problem is hit run:

    git bisect bad

and if the problem doesn't trigger run:

    git bisect good

. Please pay attention to always select the just built kernel for
booting, it won't always be the default kernel picked up by grub.

Iterate until git announces to have identified the first bad commit.

Then provide the output of

    git bisect log

In the course of the bisection you might have to uninstall previous
kernels again to not exhaust the disk space in /boot. Also in the end
uninstall all self-built kernels again.

Hope this helps so far?

Regards,
Salvatore