on Mon, Feb 18, 2002 at 06:05:46PM -0800, Osamu Aoki (debian@aokiconsulting.com) wrote:
> On Mon, Feb 18, 2002 at 12:47:44PM -0500, Scott Henson wrote:
> > I need to move large amounts of data from one disk to another and then
> > tar it up for back up purposes. I have tried cp and mv, but both take
> > very large amounts of time with many ide resets and faults. The amount
> > of data I am trying to copy is somewhere in the neighborhood of 10 to 15
> > gigs. It is going from a fat32 fs to an ext2 fs. I also notice that
> ...
> Many utilities still have 2GB file size limitation hidden somewhere,
> even though kernel should be able to handle large files.
>
> So just do not listen to other posts suggesting "tar ..." or similar.
Tar has long had the ability to stream large quantities of data to and
from stdin and stdout -- tapes were a workaround for large dataset long
ago.
I've just completed an over-the-wire transfer of 90+ GB using tar
pipelined through ssh, essentially:
$ tar cvf - * | ssh -C user@host ( cd /target-path; tar xvf - )
This was run in a 'screen' session to allow attaching to and detaching
from the process.
A note -- this seems to be cpu bound, likely by the encryption, on the
slower (sending) of the two boxes, particularly as load seem symmetric
(typical of encryption but not compression) on both sides of the link.
Total transfer time was on the order of 50 hours.
This is on a set of RH systems, but the default cipher order is the same
as Debian: aes128-cbc, 3des-cbc, blowfish-cbc, cast128-cbc, arcfour. I
know blowfish is a lower-overhead cipher, and am told that arcfour is
better still. I've proposed we move to specifying blowfish as the
default cipher in ssh and sshd config files on all systems.
On a trusted network, encryption could be disabled entirely, further
improving throughput.
Peace.
--
Karsten M. Self <kmself@ix.netcom.com> http://kmself.home.netcom.com/
What part of "Gestalt" don't you understand? There is no K5 cabal
http://gestalt-system.sourceforge.net/ http://www.kuro5hin.org
Attachment:
pgpsgMiExumaW.pgp
Description: PGP signature