[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: FTP timeouts while mirorring with debmirror

Klaus Klein <k.klein@gmx.de> writes:

> Goswin von Brederlow wrote:
>> Klaus Klein <k.klein@gmx.de> writes:
>>> I occasionally mirror either from 'ftp2.de.debian.org' or
>>> 'ftp.de.debian.org' with the following script:
>>>  #!/bin/bash
>>>  set -o pipefail
>>>  source="ftp2.de.debian.org"
>>>  # source="ftp.de.debian.org"
>>>  target="/srv/ftp"
>>>  logdir=$target/log
>> ...
>> You do realize that debmirror supports a conffile, right?
> Another well kept secret. :-(
> It's not mentioned in 'man debmirror', 'info debmirror' nor 'debmirror
> --help'. Did I miss anything?
> 'man debmirror.conf' doesn't give me any info about the content of a /etc/debmirror.conf or /$home/debmirror.conf (just found those within the perl code) either. :-(


>>> I then realized that the timeout always occurs when debmirror starts
>>> to get a file ($ret = $ftp->get($file, $file);) but never while a file
>>> is retrieved.
>>> Before I start to dig my way further down into this (ftp?) problem, I would like to ask if someone else experiences the same problem or if this is my own personal 'bug'. :-)
>> If you do look into this you should look into the perl code for the
>> ftp module and not debmirror. I don't think I use it wrongly.
> Never said it would be your fault. ;-)
> I guess that since that it used to work fine and nothing changed on my
> side, that it's either an issue with my provider or with the servers.

As a stab in the dark I would guess the ftpserver or your providers
proxy closes the connection silently after every X files on its own
and does not correctly reopen it.

> Therefore I raised the question if someone else had experienced the same problem (which would indicate a server issue) or not (which would make me start to look into ftp sessions with other servers to rule out the provider)
> BTW, I in between modified the perl code a bit and, if a timeout
> occurs, initialize the fpt session and retry the download (up to five
> times). It shows that with one retry the according file and the next
> (approx.) 80 to 90 MB are downloaded until the next timeout.
>> That said how about using http? ftp is probably the worst protocol for
>> debmirror.
> http works fine, unfortunately it doesn't show the hash symbol to indicate the progress during the download of a file. :-(
> Why do you think that ftp is the worst protocol for debmirror?

Because it has to open a new data connect for every single file. For
small files the overhead of a tcp handshake is a killer.

> Cheers,
> Klaus


Reply to: