Control: tags -1 -unreproducible
Hi Russell,
Thank you for providing more info. Now I see where you're running
into known limitations with btrfs (all versions). Reply follows inline.
BTW, you're not using SMR and/or USB disks, right?
On Mon, Feb 25, 2019 at 12:33:51PM -0600, Russell Mosemann wrote:
> Steps to reproduce
>
> Simply copying a file into the file system can cause things to lock up. In
> this case, the files will usually be thin-provisioned qcow2 disks for kvm
> vm's. There is no detailed formula to force the lockup to occur, but it
> happens regularly, sometimes multiple times in one day.
>
Have you read https://wiki.debian.org/Btrfs ? Specifically "COW on
COW: Don't do it!" ? If you did read it, maybe the document needs to
be more firm about this... eg: "take care to use raw images" should
be "under no circumstances use non-raw images". P.S. Yes, I know that
page would benefit from a reorganisation... Sorry about it's current
state.
> Files are often copied from a master by reference (cp --reflink), one per
> day to perform a daily backup for up to 45 days. Removing older files is a
> painfully slow process, even though there are only 45 files in the
> directory. Doing a scrub is almost a sure way to lock up the system,
> especially if a copy or delete operation is in progress. On two systems,
> crashes occur with 4.18 and 4.19 but not 4.17. On the other systems that
> crash, it does not seem to matter if it is 4.17, 4.18 or 4.19.
>
It might be that >4.17 fixed some corner-case corruption issue, for
example by adding an additional check during each step of a backref
walk, and that this makes the timeout more frequent and severe. eg:
4.17 works because it is less strict.
By the way, is it your VM host that locks up, or your VM guests? Do[es]
they[it] recover if you leave it alone for many hours? I didn't see
any oopses or panics in your kernel logs.
Reflinked file are like snapshots, any I/O on a file must walk every branch
of the backref tree that is relevant to a file. For more info see:
https://btrfs.wiki.kernel.org/index.php/Resolving_Extent_Backrefs
As the tree grows and becomes more complex, a COW fs will get slower.
You've hit the >120sec threshold, due to one or more of the issues
discussed in this email. eg: a scrub, even during a file copy/delete
should never cause this timeout. I haven't experienced one since
linux-4.4.x or 4.9.x...
To get a figure that will provide a sense of scale to how many
operations it takes to do anything other than reflink or snapshot you
can consult the output of:
filefrag each_live_copy_vm_image
I expect the number of extends will exceed tens of thousands. BTW,
you can use btrfsmaintenance to periodically defrag the source (and
only the source) images. Note that this will break reflinks between
SOURCE and each of the 45 REFLINKED-COPIES, but not between
REFLINK-COPY1 and REFLINK-COPY2. Defragging weekly strikes a nice
balance between lost space efficiency (due to fewer shared references
between today's backup and yesterday's) and avoiding the performance
issue you've encountered. Mounting with autodefrag is the least space
efficient. (P.S. Also, I don't trust autodefrag)
IIRC btrfs-debug-tree can accurately count references.
> Unless otherwise indicated
>
> using qgroups: No
>
Whew, thank you for not! :-) Qgroups make this kind of issue worse.
> using compression: Yes, compress-force=zstd
>
If userspace CPU usage is already high then compression may introduce
additional latency and contribute to the >120sec warning.
> number of snapshots: Zero
>
But 45 reflinked copies per VM.
> number of subvolumes: top level subvolume only
>
I believe it was Chris Murphy who wrote (on linux-btrfs) about how
segmenting different functions/datasets into different
non-hierarchically structured (eg: flat layout) subvolumes reduces
lock contention during backref walks. This is a performance tuning
tip that needs to be investigated and integrated into the wiki
article. Eg:
_____id:5 top-level_____ <-either unmounted, or
/ | | | \ mounted somewhere like
/ | | | \ /.btrfs-admin, /.volume, etc.
/ | | | \
host_rootfs VM0 VM1 VM2 data_shared_between_VMs
> raid profile: None
>
> using bcache: No
>
Thanks.
> layered on md or lvm: No
>
But layered on hardware raid?
> vhost002
>
> # grep btrfs /etc/mtab
> /dev/sdc1 /usr/local/data/datastore2 btrfs
> rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
[1] Thank you for using noatime. Please explain this configuration.
eg: does each VM have three virtual disks (containing btrfs volumes),
backed by qcow2 images, backed by a btrfs volume on the VM host? I
thought you were using qcow2 images, but this looks like passthrough
of some kind.
If the former:
Every write causes a COW operation in the inner btrfs, and the qcow2,
and the outer btrfs volume. The inner btrfs volume compresses once,
and then the outer btrfs volume will try to compress again.
> # smartctl -l scterc /dev/sdc
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control:
> Read: Disabled
> Write: Disabled
>
[2] Have you seen any SATA resets in your logs? The default kernel
timeout is 30sec, and drives without SCR ERT can sometimes take an
undefined (though generally under 180sec) amount of time to reattempt
to read a block...and if it's an SMR drive with writing I/O the delay
to successful read can be even worse.
> vhost003
>
> # grep btrfs /etc/mtab
> /dev/sdb4 /usr/local/data/datastore2 btrfs
> rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
[1] Also, why aren't you using noatime here too?
> (RAID controller)
>
> # smartctl -l scterc /dev/sdb
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.17.0-0.bpo.1-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
[4] Ok, so sdb is the raid controller on the host? And sdb is passed
through, and you shutdown one VM before mounting the same
btrfs-on-hardware_RAID partition in another VM?
> vhost004
>
> # grep btrfs /etc/mtab
> /dev/sdb4 /usr/local/data/datastore2 btrfs
> rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
[3] [1]. Also, why aren't you using noatime here too?
> (RAID controller)
>
> # smartctl -l scterc /dev/sdb
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.17.0-0.bpo.1-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
>
>
> # btrfs dev stats /usr/local/data/datastore2
> [/dev/sdb4].write_io_errs 0
> [/dev/sdb4].read_io_errs 0
> [/dev/sdb4].flush_io_errs 0
> [/dev/sdb4].corruption_errs 0
> [/dev/sdb4].generation_errs 0
>
[4] So vhost03 and vhost04 mount the same partition from the raid
controller on the host via passthrough? At the same time?
> vhost031
>
> # grep btrfs /etc/mtab
> /dev/sdc1 /usr/local/data/datastore2 btrfs
> rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
[3] [4]
> # smartctl -l scterc /dev/sdc
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control:
> Read: Disabled
> Write: Disabled
>
[2]
> vhost032
>
> # grep btrfs /etc/mtab
> /dev/sdc1 /usr/local/data/datastore2 btrfs
> rw,relatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
[3] [4]
> # smartctl -l scterc /dev/sdc
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> SCT Error Recovery Control:
> Read: Disabled
> Write: Disabled
>
Is this sdc a qcow2 image or a passed through megaraid partition?
> vhost182
>
> # grep btrfs /etc/mtab
> /dev/sdc1 /usr/local/data/datastore2 btrfs
> rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
[1]
[snip]
> lxc008
>
> number of subvolumes: 1416
That's *way* too many. This is a major contributing factor to the
timeouts...
[snip]
> # grep btrfs /etc/mtab
> /dev/sdc1 /usr/local/data2 btrfs
> rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
Is this in a container rather than a VM?
> (RAID controller)
>
> # smartctl -d megaraid,0 -l scterc /dev/sdc
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
> Write SCT (Get) Error Recovery Control Command failed: ATA return
> descriptor not supported by controller firmware
> SCT (Get) Error Recovery Control command failed
>
A different raid controller? Aiie, this is a complex setup...
> lxc009
>
> # grep btrfs /etc/mtab
> /dev/sda3 / btrfs rw,relatime,space_cache,subvolid=5,subvol=/ 0 0
> /dev/sdb1 /usr/local/data2 btrfs
> rw,noatime,compress-force=zstd,space_cache,subvolid=5,subvol=/ 0 0
>
>
>
> (RAID controller)
>
> # smartctl -l scterc /dev/sdb
> smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.19.0-0.bpo.2-amd64] (local
> build)
> Copyright (C) 2002-16, Bruce Allen, Christian Franke,
> www.smartmontools.org
>
>
>
> # btrfs dev stats /usr/local/data2
> [/dev/sdb1].write_io_errs 0
> [/dev/sdb1].read_io_errs 0
> [/dev/sdb1].flush_io_errs 0
> [/dev/sdb1].corruption_errs 0
> [/dev/sdb1].generation_errs 0
>
Ok, first decide where you want to reflink/snapshot, either inside the
VMs or outside.
If inside:
* Host your VM images on an ext4 or xfs partition.
* Use btrfs inside the VM.
- Use noatime inside the VM.
* To get backups onto the host, use the network or a
passed-through partition.
If outside:
* Host raw VM images on btrfs (noatime).
* Use ext4 on xfs inside the VM.
- Everything is already COWed, checksummed, and compressed on the
VM host, so it's absolutely not needed here.
- Also use noatime inside the VM.
* Periodically defrag the live copy of your VM images.
! Note that many on the linux-btrfs mailing list do not recommend
btrfs for this type of workload, if performance is important.
? Maybe the partition pass through is how you're getting around
this issue?
Hacky "it's too late to rethink this server": Use chattr +C on the VM
images. Note that the images will no longer be checksummed (see
wiki).
Maybe I've misunderstood, but it looks like you're running btrfs
volumes, on top of qcow2 images, on top of a btrfs host volume.
That's an easy to reproduce recipe for problems of this kind.
Sincerely,
Nicholas
Attachment:
signature.asc
Description: PGP signature