Bug#667434: lvcreate / lvremove snapshot under Xen causes Kernel OOPs
On Thu, 2012-04-05 at 09:00 +1200, Quintin Russ wrote:
> Hi Ian,
>
> On 05/04/12 01:00, Ian Campbell wrote:
> > Hi Quintin,
> >
> > Thanks for your report.
> >
> > On Wed, 2012-04-04 at 13:54 +1200, Quintin Russ wrote:
> >> Package: linux-image-2.6.32-5-xen-amd64
> >> Version: 2.6.32-39
> >> Severity: important
> >>
> >> We have observed an issue when a Xen dom0 is removing a snapshot for a
> >> logical volume and another process comes along to create a snapshot
> >> for that same device (different names) causing the server to Kernel
> >> Ooops. According to my logs sometimes removing of the snapshot can
> >> pause or take a while contributing to the issue. Attempts to add
> >> locking code (using dotlockfile) have not so far been successful in
> >> mitigating this bug, but we are still exploring this option.
> >>
> >> The nodes that are affected intermittently& we have been unable to
> >> reproduce this issue in the lab (on either the same model of hardware
> >> or hardware that has crashed in production). From our logs we can see
> >> that every time this issue occurs one process has been removing the
> >> snapshot while another has been creating a snapshot shortly after
> >> (seconds normally). We are currently seeing about a 5% chance of a
> >> crash per month (assuming our nodes are equal).
> >>
> >> This bug looks similar to a number of bugs that have already been
> >> filed related to this
> >> issue:http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=614400 A quick
> >> Google search shows many more (which have mostly been merged):
> >> https://www.google.co.nz/webhp?q=site%3Abugs.debian.org%20xen%
> >> 20snapshot%20kernel%20oops%20squeeze
> > Those issues were believed to be fixed in 2.6.32-34 and you are running
> > 2.6.32-39 so either this is a different issue (perhaps with similar
> > symptoms) or the issue isn't really fixed. Either way I think we need to
> > see your kernel logs containing the actual oops in order to make any
> > progress.
>
> Yes, we have been having this problem since before 2.6.32-34 and were
> very hopeful that change would fix it. This sadly was not the case.
> Unfortunately there isn't anything in the logs for this, but I have a
> screenshot from the console, which I have attached.
Thanks.
Googling around for issues with sync_super threw up
https://bugzilla.redhat.com/show_bug.cgi?id=587265 and
https://bugzilla.redhat.com/show_bug.cgi?id=550724. Comment 81 of the
second one mentioned issues with IRQ handling which reminded me that a
bunch of those were fixed 2.6.32-40 whereas you are running -39 (which
is fair enough since that is the version currently in stable). Could you
try the kernel from stable-proposed-updates (now 2.6.32-43)?
Also referenced was https://lkml.org/lkml/2010/9/1/178 which supports
the interrupt problem theory.
If there's any chance of setting up a serial console to catch this issue
should it happen again then that would be very useful too.
Ian.
>
> I also had an idle shell at the time the server crashed and this is what
> I saw:
>
> Message from syslogd@dom0 at Apr 4 01:37:22 ...
> kernel:[4805213.000629] Oops: 0000 [#1] SMP
>
> Message from syslogd@dom0 at Apr 4 01:37:22 ...
> kernel:[4805213.000661] last sysfs file:
> /sys/devices/virtual/block/dm-49/removable
>
> Message from syslogd@dom0 at Apr 4 01:37:22 ...
> kernel:[4805213.001891] Stack:
>
> Message from syslogd@dom0 at Apr 4 01:37:22 ...
> kernel:[4805213.002101] Call Trace:
>
> Message from syslogd@dom0 at Apr 4 01:37:22 ...
> kernel:[4805213.002540] Code: 66 ff 05 c9 83 58 00 48 89 ef e8 db 7a
> f7 ff 48 89 df e8 7f fe ff ff e8 51 b0 21 00 48 c7 c7 e0 99 67 81 e8 3b
> c0 21 00 48 8b 1b <48> 8b 03 48 81 fb 90 d1 48 81 0f 18 08 0f 85 64 ff
> ff ff 66 ff
>
> Message from syslogd@dom0 at Apr 4 01:37:22 ...
> kernel:[4805213.002901] CR2: 0000000000000000
>
> Please let me know if there is anything further I can provide.
--
Ian Campbell
Current Noise: Crippled Black Phoenix - The Heart Of Every Country
Dealer prices may vary.
Reply to: