[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#604469: linux-image-2.6.26-2-openvz-amd64: openvz - deadlock during RAID rebuild with container backing store on LVM+snapshot



Package: linux-image-2.6.26-2-openvz-amd64
Version: 2.6.26-25lenny1
Severity: normal

On Lenny, I have observed the following behaviour:

An I/O deadlock occurs under the following conditions:

. OpenVZ container data stored on an LVM for which the PV is an md RAID1
. RAID1 md undergoing a rebuild or check
. An LVM snapshot is active for the logical volume (in order to take a
backup)
. I/O occurs within the container

The RAID resync then stops, as does all other I/O to the filesystem
which is mounted upon the LVM.

Adding various debug to the md raid1 driver shows that when the system
gets to this state, there are a number of bio requests which are still
pending (or at least their callback never gets executed).

http://marc.info/?t=128473541100001&r=1&w=2

Adding the debug printks and atomic counters appears to make the deadlock
occur more readily (within seconds rather than minutes of starting the
openvz container).

My guess was that this is some sort of a priority inversion deadlock, I/O
in the container is triggering I/O outside of the container (via LVM
snapshot) which must complete first (because of the OpenVZ scheduling
rules or otherwise), and possibly the md barrier code is enforcing the
opposite ordering.

.. but when I tried:

echo 0 > /sys/block/sda/queue/iosched/virt_mode
echo 0 > /sys/block/sdb/queue/iosched/virt_mode

prior to starting the container, didn't seem to change things.  So maybe
that's not the problem.

Simply using the container private directory as a chroot, and placing
reasonably heavy I/O load does not seem to cause the same deadlock to
occur.

I haven't seen the deadlock occur on 2.6.32, but haven't tried to insert
the same debugging statements.




-- System Information:
Debian Release: squeeze/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-openvz-amd64 (SMP w/8 CPU cores)
Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
Shell: /bin/sh linked to /bin/dash

Versions of packages linux-image-2.6.26-2-openvz-amd64 depends on:
ii  debconf [debconf-2.0]         1.5.36     Debian configuration management sy
ii  initramfs-tools [linux-initra 0.98.5     tools for generating an initramfs
ii  module-init-tools             3.12-1     tools for managing Linux kernel mo
ii  vzctl                         3.0.24-10  server virtualization solution - c

linux-image-2.6.26-2-openvz-amd64 recommends no packages.

Versions of packages linux-image-2.6.26-2-openvz-amd64 suggests:
pn  grub | lilo                   <none>     (no description available)
pn  linux-doc-2.6.26              <none>     (no description available)



Reply to: