Re: ZFS guidance

[Dan Ritter <dsr@randomstring.org>]

Gregory Seidman wrote: 
> I've been running 10+ LVM volumes on top of dmcrypt on top of md RAID1 on
> Debian for many, many years and it has served me well. I've been
> double-mirroring (i.e. three active drives in the RAID array) for the last
> several with the idea that I can manually fail a disk, pull it out (and
> replace with a fresh drive), and put it somewhere safe off-site as an easy
> approach to backup (and I'm only concerned with disaster recovery, not
> individual file recovery).
> 
> I have new server hardware I'm planning on moving things to, and I'm
> considering making a change to my approach. I've been hearing good things
> about ZFS for a long time, and I understand that encryption has been
> supported for several years. Assuming that I have three physical disks to
> dedicate (separate from the three I am currently using) I'm seeking
> guidance on the following:
> 
> 1. Can an entire ZFS array be encrypted, rather than individual volumes? I
> 	 don't want to have to enter the password for each volume, just once when
> 	 bringing up the whole array.

No, and that's a good thing.

Let's review the hierarchy:

a vdev (device) is one or more storage areas grouped together.
They can be:
    a file
    a partition
    a disk
    a mirror on 2 or more disks
    a RAIDZ (like RAID 5, 1 vdev worth of redundancy)
    a RAIDZ2 (like RAID6, 2 vdevs for redundancy)
    a RAIDZ3 (3 vdevs for redundancy)
    a spare
    a cache
    a log

a zpool is comprised of one or more zvols, and data will be shared (not
quite striped, but close) across all the zvols (unless they are spares,
caches, or logs)

a zvol is a described storage partition on a zpool. The special
case of a zvol is a zfs filesystem; the general case is a zvol
block device. A zvol is *not* pre-allocated space, and can grow
to fill the entire zpool unless you put a quota on it.

zvols are where encryption happens.

You can use a zvol as your root filesystem - and as your /home
and /var and /opt and whatever as well. It's perfectly
reasonable to have a single zfs zvol for everything on many
systems.

> 2. Is there a way to tune ZFS such that it can tolerate the loss of any two
>    out of three disks? Redundancy is more important to me than total
> 	 available storage.

You can make every vdev a 3-way mirror. (Or 4. Or 5.)

> 3. Is there any equivalent off-site backup mechanism like my current
>    fail-and-remove approach?

The much better way is to use zfs send/recv to copy an encrypted
zfs or zvol to a remote or local zpool. Let's say you have four
disks, all the same size, one of which is connected via esata or
usb3 or something. We'll call them sdb, sdc, sdd and the
external one sde.

put sdb,c,d in a 3-way mirror vdev,
put only that vdev in a zpool called pool,
and create one zfs filesystem, encrypted, on pool, called
storage.

Create another zpool called target, which only has one disk in
one vdev: sde.

You can take a snapshot of storage, then zfs send the storage
snapshot over to target. It will copy the data for you.

Let's suppose you do that on Monday, Tuesday, Wednesday, and
Thursday. Each time, the snapshots will hold a view of storage
from the appropriate day, so you can recover data you rm'd by
accident or overwrote. The first send takes a long time because
it needs to send over all the data, but the subsequent sends are
fast, because there's an existing snapshot over on target.

On Friday, you do the snapshot and send one last time, then
export target. It unmounts from your computer, and you take it
to the bank, where you put it in your safety deposit box. You
remove the other disk you have there (or a series of them,
whatever) and take it back home.

The first send will take a while, since it's going to update the
entire week's worth of data changes. Then it's smooth sailing
again.

If you don't feel like going to the bank, the zfs send can
operate over a network, over a VPN, over an SSH connection...

-dsr-