On Aug 10, 2008, at 2:12 AM, Brian Wells wrote:
Hi. I'm using svn-fast-backup (in the subversion-tools package) to make rsync backups of my subversion repository. The place I'm backing up to is on an ext3-formatted usb flash drive, and I'm wondering if this is awise thing to do. I know that flash drive blocks wear out after a certain number of writes, and rsync uses hard links to avoid duplicate backup files. My question is, does this mean that the inodes' reference counts will bechanged frequently enough that it might wear out that block of the flashdrive any time soon? (I currently backup about 900 files one or two times a day.) If so, is ext3 capable of using another block for the inode, or will I lose the ability to read/write some files completely? Thanks, Brian
Here are some basic facts about USB flash drives:The underlying NAND flash RAM is guaranteed for 100K writes (older technology) or 1M (more recent). After that it's retention time starts to decrease, sometimes to as little as a few weeks, and with continued use eventually to complete failure. The retention time of a new, fully-functional NAND flash RAM is hundreds of years.
USB flash drives have controller chips in the drive that, among other things, talk USB protocol to the host. One of the other things they do is try to distribute the writes over the available blocks, extending the lifetime of the device beyond the raw lifetime of the underlying NAND flash. But the firmware in the controller usually assumes you are using a FAT file-system. If, for example, you use EXT3 rather than FAT, you will probably confuse it. I don't know what the effect will be on device life, but I'm guessing it will probably be something like a return to the behavior of the raw NAND flash.
Write blocks are large (4-8K or more). If you update one inode in a block, you have to read the entire block, update the inode and write the entire block back again. Each write to an inode causes an effective write to all the inodes in the same block. So there's roughly a 64x multiplier (8Kbytes/block divided by 128bytes/inode = 64 inodes/block) in the number of writes. Thus any given single inode is good for roughly 16K updates before it wears out.
Let's suppose your rsync backup causes a small number of updates for each inode it touches. Call it 10, conservatively (it very well could be as low as 1 or 2 -- I don't know enough of the details of how rsync works at the filesystem level). That gives you about 1600 rsync runs before you start to wear out the inodes. At twice a day, you can safely use your USB flash drive for 800 days, or between 2 and 3 years.
So here's what I'd do. Use your USB flash drive for a year, then retire it to the long-term archives and replace it with a new one. Let that be one of those things you do on or near your birthday, like schedule a checkup with your Doctor. Also: dismount the flash drive, force an fsck, and remount it, every time you do a backup. This will read and sanity-check every inode on the drive, without doing any writes. If the fsck shows random errors, replace the drive because it's probably starting to wear out.
Sound reasonable? Rick