Re: rsync backup to ext3-formatted usb flash drive?
On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote:
> On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:
>
> > Hi. I'm using svn-fast-backup (in the subversion-tools package) to
> > make
> > rsync backups of my subversion repository. The place I'm backing
> > up to
> > is on an ext3-formatted usb flash drive, and I'm wondering if this
> > is a
> > wise thing to do.
> >
> > I know that flash drive blocks wear out after a certain number of
> > writes, and rsync uses hard links to avoid duplicate backup files. My
> > question is, does this mean that the inodes' reference counts will be
> > changed frequently enough that it might wear out that block of the
> > flash
> > drive any time soon? (I currently backup about 900 files one or two
> > times a day.) If so, is ext3 capable of using another block for the
> > inode, or will I lose the ability to read/write some files completely?
> >
> > Thanks,
> > Brian
>
>
> Here are some basic facts about USB flash drives:
>
>
> The underlying NAND flash RAM is guaranteed for 100K writes (older
> technology) or 1M (more recent). After that it's retention time
> starts to decrease, sometimes to as little as a few weeks, and with
> continued use eventually to complete failure. The retention time of
> a new, fully-functional NAND flash RAM is hundreds of years.
>
> USB flash drives have controller chips in the drive that, among other
> things, talk USB protocol to the host. One of the other things they
> do is try to distribute the writes over the available blocks,
> extending the lifetime of the device beyond the raw lifetime of the
> underlying NAND flash. But the firmware in the controller usually
> assumes you are using a FAT file-system. If, for example, you use
> EXT3 rather than FAT, you will probably confuse it. I don't know
> what the effect will be on device life, but I'm guessing it will
> probably be something like a return to the behavior of the raw NAND
> flash.
>
> Write blocks are large (4-8K or more). If you update one inode in a
> block, you have to read the entire block, update the inode and write
> the entire block back again. Each write to an inode causes an
> effective write to all the inodes in the same block. So there's
> roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode =
> 64 inodes/block) in the number of writes. Thus any given single
> inode is good for roughly 16K updates before it wears out.
>
> Let's suppose your rsync backup causes a small number of updates for
> each inode it touches. Call it 10, conservatively (it very well
> could be as low as 1 or 2 -- I don't know enough of the details of
> how rsync works at the filesystem level). That gives you about 1600
> rsync runs before you start to wear out the inodes. At twice a day,
> you can safely use your USB flash drive for 800 days, or between 2
> and 3 years.
>
> So here's what I'd do. Use your USB flash drive for a year, then
> retire it to the long-term archives and replace it with a new one.
> Let that be one of those things you do on or near your birthday, like
> schedule a checkup with your Doctor. Also: dismount the flash drive,
> force an fsck, and remount it, every time you do a backup. This will
> read and sanity-check every inode on the drive, without doing any
> writes. If the fsck shows random errors, replace the drive because
> it's probably starting to wear out.
>
> Sound reasonable?
>
>
> Rick
>
>
It does sound reasonable, with one sticking point: a lot of space in the
flash drive will never be used. I should have mentioned that; despite
having 900 files, the repository is currently only about 5 MB, and the
drive itself is 4 GB.
I'm looking at an alternative utility, svn-backup-dumps, in the same
package. I could do ~800 full backups, or many more incremental backups
with a full backup here and there, before running out of space. That
would only write most blocks once or twice, and once it filled up I
could erase old backups with only one write and reuse it, right? (If I
do this, should I reformat to FAT so the controllers won't be confused?)
Thanks,
Brian
Reply to: