Re: HDD long-term data storage with ensured integrity
- To: debian-user@lists.debian.org
- Subject: Re: HDD long-term data storage with ensured integrity
- From: Marc SCHAEFER <schaefer@alphanet.ch>
- Date: Fri, 3 May 2024 13:26:30 +0200
- Message-id: <[🔎] 20240503112630.GA2462114@alphanet.ch>
- In-reply-to: <20240408200401.GA3027064@alphanet.ch>
- References: <d9875885-9eeb-4dad-9b0c-7698bb0806a9@holgerdanske.com> <20240408093826.GA2893822@alphanet.ch> <2eaa986b-5beb-44dd-8d86-5e1f6f654fa2@holgerdanske.com> <20240408200401.GA3027064@alphanet.ch>
On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote:
> For off-site long-term offline archiving, no, I am not using RAID.
Now, as I had to think a bit about ONLINE integrity, I found this
comparison:
https://github.com/t13a/dm-integrity-benchmarks
Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid
I tend to have a biais favoring UNIX layered solutions against
"all-into-one" solutions, and it seems that performance-wise,
it's also quite good.
I wrote this script to convince myself of auto-correction
of the ext4+dm-integrity+dm-raid layered approach.
It gives:
[ ... ]
[ 390.249699] md/raid1:mdX: read error corrected (8 sectors at 21064 on dm-11)
[ 390.249701] md/raid1:mdX: redirecting sector 20488 to other mirror: dm-7
[ 390.293807] md/raid1:mdX: dm-11: rescheduling sector 262168
[ 390.293988] md/raid1:mdX: read error corrected (8 sectors at 262320 on dm-11)
[ 390.294040] md/raid1:mdX: read error corrected (8 sectors at 262368 on dm-11)
[ 390.294125] md/raid1:mdX: read error corrected (8 sectors at 262456 on dm-11)
[ 390.294209] md/raid1:mdX: read error corrected (8 sectors at 262544 on dm-11)
[ 390.294287] md/raid1:mdX: read error corrected (8 sectors at 262624 on dm-11)
[ 390.294586] md/raid1:mdX: read error corrected (8 sectors at 263000 on dm-11)
[ 390.294712] md/raid1:mdX: redirecting sector 262168 to other mirror: dm-7
pretty much convicing.
So after testing btrfs and being not convinced, after doing some test on
a production zfs -- not convinced either -- I am going to ry
ext4+dm-integrity+dm-raid.
#! /bin/bash
set -e
function create_lo {
local f
f=$(losetup -f)
losetup $f $1
echo $f
}
# beware of the rm -r below!
tmp_dir=/tmp/$(basename $0)
mnt=/mnt
mkdir $tmp_dir
declare -a pvs
for p in pv1 pv2
do
truncate -s 250M $tmp_dir/$p
l=$(create_lo $tmp_dir/$p)
pvcreate $l
pvs+=($l)
done
vg=$(basename $0)-test
lv=test
vgcreate $vg ${pvs[*]}
vgdisplay $vg
lvcreate --type raid1 --raidintegrity y -m 1 -L 200M -n $lv $vg
lvdisplay $vg
# sync/integrity complete?
sleep 10
cat /proc/mdstat
echo
lvs -a -o name,copy_percent,devices $vg
echo
echo -n Type ENTER
read ignore
mkfs.ext4 -I 256 /dev/$vg/$lv
mount /dev/$vg/$lv $mnt
for f in $(seq 1 10)
do
# ignore errors
head -c 20M < /dev/random > $mnt/f_$f || true
done
(cd $mnt && find . -type f -print0 | xargs -0 md5sum > $tmp_dir/MD5SUMS)
# corrupting some data in one PV
count=5000
blocks=$(blockdev --getsz ${pvs[1]})
if [ $blocks -lt 32767 ]; then
factor=1
else
factor=$(( ($blocks - 1) / 32767))
fi
p=1
for i in $(seq 1 $count)
do
offset=$(($RANDOM * $factor))
echo ${pvs[$p]} $offset
dd if=/dev/random of=${pvs[$p]} bs=$(blockdev --getpbsz ${pvs[$p]}) seek=$offset count=1
# only doing on 1, not 0, since we have no way to avoid destroying the same sector!
#p=$((1 - p))
done
dd if=/dev/$vg/$lv of=/dev/null bs=32M
dmesg | tail
umount $mnt
lvremove -y $vg/$lv
vgremove -y $vg
for p in ${pvs[*]}
do
pvremove $p
losetup -d $p
done
rm -r $tmp_dir
Reply to: