[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: rsync backup to ext3-formatted usb flash drive?

On Mon, 2008-08-11 at 02:21 -0400, Rick Thomas wrote:
> On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:
> > Hi. I'm using svn-fast-backup (in the subversion-tools package) to  
> > make
> > rsync backups of my subversion repository.  The place I'm backing  
> > up to
> > is on an ext3-formatted usb flash drive, and I'm wondering if this  
> > is a
> > wise thing to do.
> >
> > I know that flash drive blocks wear out after a certain number of
> > writes, and rsync uses hard links to avoid duplicate backup files.  My
> > question is, does this mean that the inodes' reference counts will be
> > changed frequently enough that it might wear out that block of the  
> > flash
> > drive any time soon?  (I currently backup about 900 files one or two
> > times a day.)  If so, is ext3 capable of using another block for the
> > inode, or will I lose the ability to read/write some files completely?
> >
> > Thanks,
> > Brian
> Here are some basic facts about USB flash drives:
> The underlying NAND flash RAM is guaranteed for 100K writes (older  
> technology) or 1M (more recent).  After that it's retention time  
> starts to decrease, sometimes to as little as a few weeks, and with  
> continued use eventually to complete failure.  The retention time of  
> a new, fully-functional NAND flash RAM is hundreds of years.
> USB flash drives have controller chips in the drive that, among other  
> things, talk USB protocol to the host.  One of the other things they  
> do is try to distribute the writes over the available blocks,  
> extending the lifetime of the device beyond the raw lifetime of the  
> underlying NAND flash.  But the firmware in the controller usually  
> assumes you are using a FAT file-system.  If, for example, you use  
> EXT3 rather than FAT, you will probably confuse it.  I don't know  
> what the effect will be on device life, but I'm guessing it will  
> probably be something like a return to the behavior of the raw NAND  
> flash.
> Write blocks are large (4-8K or more).  If you update one inode in a  
> block, you have to read the entire block, update the inode and write  
> the entire block back again.  Each write to an inode causes an  
> effective write to all the inodes in the same block.  So there's  
> roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode =  
> 64 inodes/block) in the number of writes.  Thus any given single  
> inode is good for roughly 16K updates before it wears out.
> Let's suppose your rsync backup causes a small number of updates for  
> each inode it touches.  Call it 10, conservatively (it very well  
> could be as low as 1 or 2 -- I don't know enough of the details of  
> how rsync works at the filesystem level).  That gives you about 1600  
> rsync runs before you start to wear out the inodes.  At twice a day,  
> you can safely use your USB flash drive for 800 days, or between 2  
> and 3 years.
> So here's what I'd do.  Use your USB flash drive for a year, then  
> retire it to the long-term archives and replace it with a new one.   
> Let that be one of those things you do on or near your birthday, like  
> schedule a checkup with your Doctor.  Also: dismount the flash drive,  
> force an fsck, and remount it, every time you do a backup.  This will  
> read and sanity-check every inode on the drive, without doing any  
> writes.  If the fsck shows random errors, replace the drive because  
> it's probably starting to wear out.
> Sound reasonable?
> Rick

It does sound reasonable, with one sticking point: a lot of space in the
flash drive will never be used.  I should have mentioned that; despite
having 900 files, the repository is currently only about 5 MB, and the
drive itself is 4 GB.

I'm looking at an alternative utility, svn-backup-dumps, in the same
package.  I could do ~800 full backups, or many more incremental backups
with a full backup here and there, before running out of space.  That
would only write most blocks once or twice, and once it filled up I
could erase old backups with only one write and reuse it, right?  (If I
do this, should I reformat to FAT so the controllers won't be confused?)


Reply to: