Bug#1116358: linux-image-6.12.48+deb13-amd64: LVM snapshots causing I/O errors in KVM guest with aio=io_uring set
Hello,
I've identified the first bad commit using git bisect:
90bfb28d5fa8127a113a140c9791ea0b40ab156a is the first bad commit
commit 90bfb28d5fa8127a113a140c9791ea0b40ab156a
Author: Jens Axboe <axboe@kernel.dk>
Date: Tue Sep 10 08:57:04 2024 -0600
io_uring/rw: drop -EOPNOTSUPP check in __io_complete_rw_common()
A recent change ensured that the necessary -EOPNOTSUPP -> -EAGAIN
transformation happens inline on both the reader and writer side,
and hence there's no need to check for both of these anymore on
the completion handler side.
Signed-off-by: Jens Axboe <axboe@kernel.dk>
io_uring/rw.c | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
Here is the git bisect log as well:
git bisect start
# status: waiting for both good and bad commits
# good: [98f7e32f20d28ec452afb208f9cffc08448a2652] Linux 6.11
git bisect good 98f7e32f20d28ec452afb208f9cffc08448a2652
# status: waiting for bad commit, 1 good commit known
# bad: [59b723cd2adbac2a34fc8e12c74ae26ae45bf230] Linux 6.12-rc6
git bisect bad 59b723cd2adbac2a34fc8e12c74ae26ae45bf230
# bad: [de848da12f752170c2ebe114804a985314fd5a6a] Merge tag 'drm-next-
2024-09-19' of https://gitlab.freedesktop.org/drm/kernel
git bisect bad de848da12f752170c2ebe114804a985314fd5a6a
# bad: [7b17f5ebd5fc5e9275eaa5af3d0771f2a7b01bbf] Merge tag 'soc-dt-
6.12' of git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc
git bisect bad 7b17f5ebd5fc5e9275eaa5af3d0771f2a7b01bbf
# good: [64dd3b6a79f0907d36de481b0f15fab323a53e5a] Merge tag 'for-
linus-non-x86' of git://git.kernel.org/pub/scm/virt/kvm/kvm
git bisect good 64dd3b6a79f0907d36de481b0f15fab323a53e5a
# bad: [daa394f0f9d3cb002c72e2d3db99972e2ee42862] Merge tag 'core-
debugobjects-2024-09-16' of
git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
git bisect bad daa394f0f9d3cb002c72e2d3db99972e2ee42862
# good: [effdcd5275ed645f6e0f8e8ce690b97795722197] Merge tag 'affs-for-
6.12-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
git bisect good effdcd5275ed645f6e0f8e8ce690b97795722197
# bad: [26bb0d3f38a764b743a3ad5c8b6e5b5044d7ceb4] Merge tag 'for-
6.12/block-20240913' of git://git.kernel.dk/linux
git bisect bad 26bb0d3f38a764b743a3ad5c8b6e5b5044d7ceb4
# bad: [3a4d319a8fb5a9bbdf5b31ef32841eb286b1dcc2] Merge tag 'for-
6.12/io_uring-20240913' of git://git.kernel.dk/linux
git bisect bad 3a4d319a8fb5a9bbdf5b31ef32841eb286b1dcc2
# good: [df2825e98507d10cb037a308087ecd7cb3f6688d] btrfs: always pass
readahead state to defrag
git bisect good df2825e98507d10cb037a308087ecd7cb3f6688d
# good: [69a3a0a45a2f72412c2ba31761cc9193bb746fef] Merge tag 'erofs-
for-6.12-rc1' of
git://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs
git bisect good 69a3a0a45a2f72412c2ba31761cc9193bb746fef
# good: [ecd5c9b29643f383d39320e30d21b8615bd893da] io_uring/kbuf: add
io_kbuf_commit() helper
git bisect good ecd5c9b29643f383d39320e30d21b8615bd893da
# good: [f011c9cf04c06f16b24f583d313d3c012e589e50] io_uring/sqpoll: do
not allow pinning outside of cpuset
git bisect good f011c9cf04c06f16b24f583d313d3c012e589e50
# bad: [84eacf177faa605853c58e5b1c0d9544b88c16fd] io_uring/io-wq:
inherit cpuset of cgroup in io worker
git bisect bad 84eacf177faa605853c58e5b1c0d9544b88c16fd
# bad: [90bfb28d5fa8127a113a140c9791ea0b40ab156a] io_uring/rw: drop -
EOPNOTSUPP check in __io_complete_rw_common()
git bisect bad 90bfb28d5fa8127a113a140c9791ea0b40ab156a
# good: [c0a9d496e0fece67db777bd48550376cf2960c47] io_uring/rw: treat -
EOPNOTSUPP for IOCB_NOWAIT like -EAGAIN
git bisect good c0a9d496e0fece67db777bd48550376cf2960c47
# first bad commit: [90bfb28d5fa8127a113a140c9791ea0b40ab156a]
io_uring/rw: drop -EOPNOTSUPP check in __io_complete_rw_common()
On Thu, 2025-10-02 at 16:25 +0200, Salvatore Bonaccorso wrote:
> Control: tags -1 + moreinfo
>
> Hi Kevin,
>
> On Thu, Oct 02, 2025 at 02:28:48PM +0300, Kevin Lumik wrote:
> > On Sat, 27 Sep 2025 16:09:08 +0200 Salvatore Bonaccorso
> > <carnil@debian.org> wrote:
> > > - Can you temporarily test as well the upper kernels from
> > > unstable
> > > (6.16.8-1 right now, but soonish 6.16.9-1 as well) to see if
> > > the
> > > problem is triggerable there as well?
> >
> > I tested the upper kernels and was still able to reproduce the
> > issue on
> > both 6.16.8-1 and 6.16.9-1.
>
> Ok thanks for doing so, have just updated as well the metadata to
> reflect that.
>
> > > Note: If we find the problem is prosent in 6.16.y then we need to
> > > find
> > > the breaking change between 6.1.y and 6.12.y. It might be asier
> > > here
> > > if we first make the range of potential upstream versions more
> > > narrow,
> > > so you might first do a "manual bisect" on Debian revisions (For
> > > that
> > > use the snapshot.debian.org service to retrieve linux-images
> > > uploaded
> > > to Debian. E.g. go down from 6.12 in "major version" steps until
> > > we
> > > have close enough range of broken/fixed upstream major versions.
> > > Then
> > > perform a normal bisect between the upstream versions.
> >
> > Using snapshot.debian.org I have determined that the breaking
> > change was
> > introduced between 6.11.10 (tested 6.11.10-1) and 6.12 (tested
> > 6.12.3-1
> > and 6.12~rc6-1~exp1).
> >
> > Thank you for your assistance. Let me know what would be the next
> > step.
>
> Okay so we have a good range to look at. The next steps would be to
> bisect the changes. Actually it would be good to first test 6.11
> itself upstream. So the steps to do would be roughly the following,
> which needs compiling and testing a few kernels:
>
> git clone
> https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
> cd linux-stable
> git checkout v6.11
> cp /boot/config-$(uname -r) .config
> yes '' | make localmodconfig
> make savedefconfig
> mv defconfig arch/x86/configs/my_defconfig
>
> # test 6.11 to ensure this is "good"
> make my_defconfig
> make -j $(nproc) bindeb-pkg
> ... install the resulting .deb package and confirm it
> successfully boots / problem does not exist
>
> # test 6.12-rc6 to ensure this is "bad"
> git checkout v6.12-rc6
> make my_defconfig
> make -j $(nproc) bindeb-pkg
> ... install the resulting .deb package and confirm it fails to
> boot / problem exists
>
> With that confirmed, the bisection can start:
>
> git bisect start
> git bisect good v6.11
> git bisect bad v6.12-rc6
>
> In each bisection step git checks out a state between the oldest
> known-bad and the newest known-good commit. In each step test using:
>
> make my_defconfig
> make -j $(nproc) bindeb-pkg
> ... install, try to boot / verify if problem exists
>
> and if the problem is hit run:
>
> git bisect bad
>
> and if the problem doesn't trigger run:
>
> git bisect good
>
> . Please pay attention to always select the just built kernel for
> booting, it won't always be the default kernel picked up by grub.
>
> Iterate until git announces to have identified the first bad commit.
>
> Then provide the output of
>
> git bisect log
>
> In the course of the bisection you might have to uninstall previous
> kernels again to not exhaust the disk space in /boot. Also in the end
> uninstall all self-built kernels again.
>
> Hope this helps so far?
>
> Regards,
> Salvatore
Reply to: