Bug#604469: linux-image-2.6.26-2-openvz-amd64: openvz - deadlock during RAID rebuild with container backing store on LVM+snapshot
Package: linux-image-2.6.26-2-openvz-amd64
Version: 2.6.26-25lenny1
Severity: normal
On Lenny, I have observed the following behaviour:
An I/O deadlock occurs under the following conditions:
. OpenVZ container data stored on an LVM for which the PV is an md RAID1
. RAID1 md undergoing a rebuild or check
. An LVM snapshot is active for the logical volume (in order to take a
backup)
. I/O occurs within the container
The RAID resync then stops, as does all other I/O to the filesystem
which is mounted upon the LVM.
Adding various debug to the md raid1 driver shows that when the system
gets to this state, there are a number of bio requests which are still
pending (or at least their callback never gets executed).
http://marc.info/?t=128473541100001&r=1&w=2
Adding the debug printks and atomic counters appears to make the deadlock
occur more readily (within seconds rather than minutes of starting the
openvz container).
My guess was that this is some sort of a priority inversion deadlock, I/O
in the container is triggering I/O outside of the container (via LVM
snapshot) which must complete first (because of the OpenVZ scheduling
rules or otherwise), and possibly the md barrier code is enforcing the
opposite ordering.
.. but when I tried:
echo 0 > /sys/block/sda/queue/iosched/virt_mode
echo 0 > /sys/block/sdb/queue/iosched/virt_mode
prior to starting the container, didn't seem to change things. So maybe
that's not the problem.
Simply using the container private directory as a chroot, and placing
reasonably heavy I/O load does not seem to cause the same deadlock to
occur.
I haven't seen the deadlock occur on 2.6.32, but haven't tried to insert
the same debugging statements.
-- System Information:
Debian Release: squeeze/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-5-openvz-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash
Versions of packages linux-image-2.6.26-2-openvz-amd64 depends on:
ii debconf [debconf-2.0] 1.5.36 Debian configuration management sy
ii initramfs-tools [linux-initra 0.98.5 tools for generating an initramfs
ii module-init-tools 3.12-1 tools for managing Linux kernel mo
ii vzctl 3.0.24-10 server virtualization solution - c
linux-image-2.6.26-2-openvz-amd64 recommends no packages.
Versions of packages linux-image-2.6.26-2-openvz-amd64 suggests:
pn grub | lilo <none> (no description available)
pn linux-doc-2.6.26 <none> (no description available)
Reply to: