Re: ZFS performance (was: Re: deduplicating file systems: VDO with Debian?)
On 11/10/22 07:44, hw wrote:
On Wed, 2022-11-09 at 21:36 -0800, David Christensen wrote:
On 11/9/22 00:24, hw wrote:
> On Tue, 2022-11-08 at 17:30 -0800, David Christensen wrote:
Be careful that you do not confuse a ~33 GiB full backup set, and 78
snapshots over six months of that same full backup set, with a full
backup of 3.5 TiB of data.
The full backup isn't deduplicated?
"Full", "incremental", etc., occur at the backup utility level -- e.g.
on top of the ZFS filesystem. (All of my backups are full backups using
rsync.) ZFS deduplication occurs at the block level -- e.g. the bottom
of the ZFS filesystem. If your backup tool is writing to, or reading
from, a ZFS filesystem, the backup tool is oblivious to the internal
operations of ZFS (compression or none, deduplicaton or none, etc.) so
long as the filesystem "just works".
Writing to a ZFS filesystem with deduplication is much slower than
simply writing to, say, an ext4 filesystem -- because ZFS has to hash
every incoming block and see if it matches the hash of any existing
block in the destination pool. Storing the existing block hashes in a
dedicated dedup virtual device will expedite this process.
But when it needs to write almost nothing because almost everthing gets
deduplicated, can't it be faster than having to write everthing?
There are many factors that affect how fast ZFS can write files to disk.
You will get the best answers if you run benchmarks using your
hardware and data.
>> I run my backup script each night. It uses rsync to copy files and
>
> Aww, I can't really do that because my servers eats like 200-300W
because it has
> so many disks in it. Electricity is outrageously expensive here.
Perhaps platinum rated power supplies? Energy efficient HDD's/ SSD's?
If you pay for it ... :)
Running it once in a while for some hours to make backups is still possible.
Replacing the hardware is way more expensive.
My SOHO server has ~1 TiB of data. A ZFS snapshot takes a few seconds.
ZFS incremental replication to the backup server proceeds at anywhere
from 0 to 50 MB/s, depending upon how much content is new or has changed.
> Sounds like a nice setup. Does that mean you use snapshots to keep
multiple
> generations of backups and make backups by overwriting everything
after you made
> a snapshot?
Yes.
I start thinking more and more that I should make use of snapshots.
Taking snapshots is fast and easy. The challenge is deciding when to
destroy them.
zfs-auto-snapshot can do both automatically:
https://packages.debian.org/bullseye/zfs-auto-snapshot
https://manpages.debian.org/bullseye/zfs-auto-snapshot/zfs-auto-snapshot.8.en.html
Without deduplication or compression, my backup set and 78 snapshots
would require 3.5 TiB of storage. With deduplication and compression,
they require 86 GiB of storage.
Wow that's quite a difference! What makes this difference, the compression or
the deduplication?
Deduplication.
When you have snapshots, you would store only the
differences from one snapshot to the next,
and that would mean that there aren't
so many duplicates that could be deduplicated.
I do not know -- I have not crawled the ZFS code; I just use it.
Users can recover their own files without needing help from a system
administrator.
You have users who know how to get files out of snapshots?
Not really; but the feature is there.
>>>> For compressed and/or encrypted archives, image, etc., I do not use
>>>> compression or de-duplication
>>>
>>> Yeah, they wouldn't compress. Why no deduplication?
>>
>>
>> Because I very much doubt that there will be duplicate blocks in
such files.
>
> Hm, would it hurt?
Yes. ZFS deduplication is resource intensive.
But you're using it already.
I have learned the hard way to only use deduplication when it makes sense.
What were the makes and models of the 6 disks? Of the SSD's? If you
have a 'zpool status' console session from then, please post it.
They were (and still are) 6x4TB WD Red (though one or two have failed over time)
and two Samsung 850 PRO, IIRC. I don't have an old session anymore.
These WD Red are slow to begin with. IIRC, both SDDs failed and I removed them.
The other instance didn't use SSDs but 6x2TB HGST Ultrastar. Those aren't
exactly slow but ZFS is slow.
Those HDD's should be fine with ZFS; but those SSD's are desktop drives,
not cache devices. That said, I am making the same mistake with Intel
SSD 520 Series. I have considered switching to one Intel Optane Memory
Series and a PCIe 4x adapter card in each server.
MySQL appears to have the ability to use raw disks. Tuned correctly,
this should give the best results:
https://dev.mysql.com/doc/refman/8.0/en/innodb-system-tablespace.html#innodb-raw-devices
Could mysql 5.6 already do that? I'll have to see if mariadb can do that now
...
I do not know -- I do not run MySQL or Maria.
Please run 'zpool status' and post the console session (prompt, command
entered, output displayed). Please correlate the vdev's to disk drive
makes and models.
See above ... The pool is a raidz1-0 with the 6x4TB Red drives, and no SSDs are
left.
Please run and post the relevant command for LVM, btrfs, whatever.
On 11/9/22 03:41, hw wrote:
What is the make and model of your server?
I put it together myself. The backup server uses a MSI mainboard with the
designation S0121 C204 SKU in a Chenbro case that has a 16xLFF backplane. It
has only 16GB RAM and would max out at 32GB.
... the backup server is currently using btrfs.
Okay.
What is the make and model of your controller cards?
They're HP smart array P410. FreeBSD doesn't seem to support those.
I use the LSI 9207-8i with "IT Mode" firmware (e.g. host bus adapter,
not RAID):
https://www.ebay.com/sch/i.html?_from=R40&_trksid=p2380057.m570.l1313&_nkw=lsi+9207&_sacat=0
... the data to back up is mostly (or even all) on btrfs. ... copy the
files over with rsync. ...
the data comes from different machines and all backs up to one volume.
I suggest creating a ZFS pool with a mirror vdev of two HDD's. If you
can get past your dislike of SSD's, add a mirror of two SSD's as a
dedicated dedup vdev. (These will not see the hard usage that cache
devices get.) Create a filesystem 'backup'. Create child filesystems,
one for each host. Create grandchild filesystems, one for the root
filesystem on each host. Set up daily rsync backups of the root
filesystems on the various hosts to the ZFS grandchild filesystems. Set
up zfs-auto-snapshot to take daily snapshots of everything, and retain
10 snapshots. Then watch what happens.
David
Reply to: