[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

SUMMARY: rsync, fat32, firewire

Got my mixed-platform backup system running, and everything seems to be peachy. I learned a few lessons, so I thought I'd compile them here for posterity. None of these are anything new to an expert, but they may be to others.

What I'm doing is using rsync over the LAN to do backups of some windows 2000 machines and some unix/linux machines, onto a dedicated backup server that runs Debian Sarge. It has a couple of large IDE drives (250GB each), and a couple of external (firewire/USB) hard drives that I take offsite on a rotating basis. I'm using rsync running locally to bring the external drives up to date with the latest generations of the backups that are stored on the internal drives. Also on the internal drives are old generations of the rsync backups, which are created by use of "cp -al" so that files that don't change from one backup to the next are just done as multiple hard links to the same file. Result is that each backup job after the first one is just an incremental backup in terms of what gets transferred and how much additional disk space is consumed on the backup server. But any one of the backup generations appears as if it were a full backup because of the hard links. Nightly backup runs on some of the clients are measured in seconds, and I'll never have to run a full backup again unless something croaks.

Things I learned along the way.

FAT32 works better with linux than NTFS does. I wanted to be able to take the external hard drives and connect them to a windows machine for restores, so I went with FAT32, based on advice here and elsewhere that writing to an NTFS partition in linux is risky.

When you use rsync to sync an ext3 partition onto a FAT32 partition, a couple of things happen. One is that the FAT32 partition isn't going to like certain filename characters that are ok in the ext3 partition ... like filenames with ":" in them. The result is that files coming off of the linux/unix servers can't be stored correctly on the FAT32. My solution to this was to split the external drives into two partitions, one FAT32 for the windows backups and one ext3 for the linux/unix ones. Another option would have been to go ahead and use ext[23] and rely on a software solution that would allow a windows machine to access them.

In order to create the FAT32 partitions, I used mkfs on the linux machine. Windows 2000 is perfectly happy USING a large fat32 partition, but if you try to create one, it runs through what appears to be an entire format process, and then fails at the end with a message that the filesystem is too large. Even the windows help says that this is by design, and that it's got nothing to do with inherent limits to fat32, it's just to encourage (also known as force) you to use NTFS instead. Why they go ahead and let the program run through 99% of a time-consuming process before failing rather than just having it tell you right away is left as an exercise to the reader.

Another lesson is that the FAT32 partition should be mounted with the "shortname=mixed" option. If not, the rsync gets messed up. The default option is "lower," which means that files and directories with short names will have their names forced to lower case. But then rsync will think they're different from the ones it sees in the source that are upper case, and it will send the upper case ones again, and then delete the lower case ones. But since it's really the same files, the end result is it transfers a bunch of data and then deletes it all! Not good.

USB 2 works better than firewire. The external drives have both interfaces. When I used firewire, I was getting errors in the data transfer, and losing a lot of data. Using USB, this didn't happen.

Easiest way to set up the windows backup clients seemed to me to be to install cygwin, rsync, and ssh. A shell script takes care of the client side of the rsync transaction, and the backup server runs the rsync daemon via ssh. Then on the windows client, I wrote a batch file to launch the shell script, and installed that as a scheduled task. No messing with services on the windows end that way.

Now in hindsight, the backuppc package would probably have saved me some of this trouble, but I like the way this turned out in the end.

(This hotmail address may go away, once the spammers catch up to it. A more permanent address is rowan at crssa rutgers edu, with dots and a real at sign in the right places.)

Reply to: