Bug#705124: Info received (base: Filesystem corruption issue)

On Tue, Apr 16, 2013 at 5:59 AM, Ian Campbell <ijc@hellion.org.uk> wrote:

On Mon, 2013-04-15 at 15:02 -0400, Anthony Sheetz wrote:
> Current status.
> New laptop hard drive purchased and installed.

Thanks!

> Experiment 1
[....]

> Result: 4.9Gb file transferred fine!

> Experiment 2

[...]
> Result: successful transfer.

Great news.

> Experiment 3
> Change from above (2):
> LVM with Full Disk Encryption
> Install package linux-image-3.2.0-0.bpo.4-amd64 from
> backports.debian.org
> data transferred to root filesystem.
> Result: md5sum fails, failure.
>
>
> Experiment 4
> Change from above (3):
> experiment trying to disable barriers, with no luck. Tried booting
> with rootflags=barrier=0, putting barrier=0 in /etc/fstab. None of
> these prevented seeing blkfront: xvda2: barriers enabled in the
> kern.log. Couldn't proceed further.

Silly question: kern.log isn't rotated/cleared on boot -- you sure you
are looking at the lines for the running kernel? Looking at the output
of dmesg might be more reliable from that PoV.

Was checking the log time, and was certain it was from the most recent boot each time.

Under Wheezy at least (not sure about Squeeze) /proc/mounts should also
indicate whether or not barriers are enabled for each device.

Checked here after doing more experiments, and never see information about barriers - more below.

> Discussion:
>
> file sizes are always correct, which is interesting.
> Seems that full disk encryption is the culprit.

I agree.

Google suggests that at least at one point in time (~2010) barriers in
dm-crypt didn't work or possibly were even disabled and that LVM
historically was also a bit patchy in this area. I can't find much
authoritative about this but
http://www.saout.de/pipermail/dm-crypt/2012-April/002441.html suggests
this is no longer the case (the Wheezy kernel should be plenty new
enough).

Your dom0 rootfs is on the same crypt+LVM driver, right?

Yes.

What does dom0 dmesg say about barriers on that device/filesystem?

Nothing in dmesg about barriers on any file system. 'dmesg|grep barrier' returns nothing.

Is it ext3 also?

yes.

Perhaps you could post the dom0 dmesg for completeness.

Attached.

I've had a stooge around on my laptop (which also runs encrypted root)
but I don't see anything about enabling/disabling barriers anywhere
(dmesg, etc, crypttab etc). /proc/mount says barriers are on for me
(although I see no messages one way or the other in dmesg or kern.log).
I don't run Xen on this system and have no spare room in the LV to try
unfortunately.

> What's next?

It'd be useful to gather the information above and also to revisit the
barrier=0 thing. As well as adding it to your kernel command line it
might be worth adding it to /etc/fstab and running "update-initramfs
-u", just in case its the initrd which is dealing with it in this case
(I doubt it though).

Tried all of that, and still can't get any evidence that any volume can be mounted with barriers.

Or you could try the barrier option with the separate data device, which
should be a lot easier to control?

Tried that as well. Attached the /etc/fstab from domU in case I got something wrong.

Googling around suggests you aren't the first to see this (e.g. #688441)
but in those cases the barrier fix helped.

May have fixed that specific case (though I'm skeptical), but it did not fix the case that case referred to.

Failing all that I think the next step would be to take this to the xen
blkif guys on xen-devel@lists.xen.org and see if they can offer any
hints as to why domU barriers aren't turning into barriers on the
underlying device or further debugging ideas etc.

Getting in touch with them immediately after I send this message.

Ian.