Re: FTP timeouts while mirorring with debmirror
Klaus Klein <k.klein@gmx.de> writes:
> Goswin von Brederlow wrote:
>> Klaus Klein <k.klein@gmx.de> writes:
>>> I occasionally mirror either from 'ftp2.de.debian.org' or
>>> 'ftp.de.debian.org' with the following script:
>>>
>>> #!/bin/bash
>>> set -o pipefail
>>> source="ftp2.de.debian.org"
>>> # source="ftp.de.debian.org"
>>> target="/srv/ftp"
>>> logdir=$target/log
>> ...
>> You do realize that debmirror supports a conffile, right?
>
> Another well kept secret. :-(
> It's not mentioned in 'man debmirror', 'info debmirror' nor 'debmirror
> --help'. Did I miss anything?
>
> 'man debmirror.conf' doesn't give me any info about the content of a /etc/debmirror.conf or /$home/debmirror.conf (just found those within the perl code) either. :-(
/usr/share/doc/debmirror/debmirror.conf
>>> I then realized that the timeout always occurs when debmirror starts
>>> to get a file ($ret = $ftp->get($file, $file);) but never while a file
>>> is retrieved.
>>> Before I start to dig my way further down into this (ftp?) problem, I would like to ask if someone else experiences the same problem or if this is my own personal 'bug'. :-)
>> If you do look into this you should look into the perl code for the
>> ftp module and not debmirror. I don't think I use it wrongly.
>
> Never said it would be your fault. ;-)
> I guess that since that it used to work fine and nothing changed on my
> side, that it's either an issue with my provider or with the servers.
As a stab in the dark I would guess the ftpserver or your providers
proxy closes the connection silently after every X files on its own
and does not correctly reopen it.
> Therefore I raised the question if someone else had experienced the same problem (which would indicate a server issue) or not (which would make me start to look into ftp sessions with other servers to rule out the provider)
>
> BTW, I in between modified the perl code a bit and, if a timeout
> occurs, initialize the fpt session and retry the download (up to five
> times). It shows that with one retry the according file and the next
> (approx.) 80 to 90 MB are downloaded until the next timeout.
>
>> That said how about using http? ftp is probably the worst protocol for
>> debmirror.
>
> http works fine, unfortunately it doesn't show the hash symbol to indicate the progress during the download of a file. :-(
>
> Why do you think that ftp is the worst protocol for debmirror?
Because it has to open a new data connect for every single file. For
small files the overhead of a tcp handshake is a killer.
> Cheers,
> Klaus
MfG
Goswin
Reply to: