Re: To mirror a huge tree...
On Tue, Jul 14, 2009 at 07:39:04PM -0500, Ron Johnson wrote:
> is it more efficient to use tar, rsync or "cp -r"?
cp -r
I assume that you need copy all the data. If you can avoid copying,
rsync can, of course be (much?) faster, depending on the saving.
However the mere work it ddoes for scanning the tree on both sides takes
extra time and memory.
>
> (Yes, I know that incremental backups would be faster with rsync. Also,
> most all of these files are already compressed, so tar or rsync -z or -j
> wouldn't help either.)
>
> With "cp -r" to an ext4 (with extents enabled) fs mounted on an external
> USB hard drive, I'm getting a consistent 30MBps, which is half of
> USB2.0's theoretical 480Mbps.
I suspect that this is the bottleneck here, anyway.
Data compression wouldn't have helped overcoming it, as you have to
decompress the data before it hits the USB bus.
>
> Thus, could I tweak a statistically significant more MBps using tar or
> rsync? Pre-fetching files or some such? As it stands now, 88% of my 8GB
> RAM is "cached".
Given that the write itself is the bottleneck, I'm not sure how much
this would help. It could slightly help to shorten the time, but the
latency of the proces will remain unchanged.
>
> Or is 30MBps about as good as I can get from the combination of the USB
> software and hardware?
--
Tzafrir Cohen | tzafrir@jabber.org | VIM is
http://tzafrir.org.il | | a Mutt's
tzafrir@cohens.org.il | | best
ICQ# 16849754 | | friend
Reply to: