Re: Transfering large files (was: Unison hangs on copy)
On Tue, Jul 26, 2011 at 07:25:41AM -0400, Randy Kramer wrote:
> > So I tried rsyncinc another file, 23 M in size (with --progress
> > option), and surprise, it stops when 11% transfer is reached. Tried
> > with scp, and same magic number: 11% and it stops.
>
> I'm somewhat surprised, because when I was thinking of large files, I
> was thinking of CD image size files. (I used to transfer these over a
> 33 kbps modem, requiring about 65 hours per image, spread out over
> (typically) 5 nights. Oh, the good old days ;-)
Indeed, this is very frustrating, 20M is not a huge file, but it's the
size which is giving me trouble (ok, it could be less, haven't tried,
but I don't want to be too obsessive :-)
>
> I don't really have any good thoughts to offer. I'd be looking to make
> sure that no temporary storage areas have been filled (presumably /tmp
> on either machine), in fact, I'd probably do a df for all partitions on
> both machines and make sure all of them have plenty of space (a nice
> big multiple of 23 MB--I don't really know how much storage rsync, or
> scp need, but I'm just "grasping at straws" looking for possibilities.
>
> While rsync may do some thinking (i.e., pause) during a 23 MB file
> transfer, I don't think scp would, but there still might be something
> that is timing out.
I thought exactly the same thing, that's why I tried scp after rsync.
One of the machines reports this:
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 178G 146G 24G 87% /
tmpfs 1007M 0 1007M 0% /lib/init/rw
udev 1002M 172K 1002M 1% /dev
tmpfs 1007M 0 1007M 0% /dev/shm
This is the remote machine I'm trying to synchronize with every night.
I will check at home the other one when I can.
>
> All from me is just random speculation, however. Good luck--I hope
> someone else will have some better suggestions for you.
Fine, thanks for the speculation and the interest.
Victor
Reply to: