Re: where to sync from?
On Wed, Mar 21, 2007 at 12:07:09PM -0400, Ricardo Yanez wrote:
> Many years ago, before 2002, we were syncing with ftp.br.debian.org,
> then one day, the entire debian/ tree was gone with the catastrophic
> result that our entire tree was deleted as well. It took us weeks to
> re-build the tree again (yeah, that slow! and debian in those days was a
> 10th of what is today). Based on this painful experience I've been very
> reluctant to use leaf mirrors to sync.
ftp.br.d.o had its share of problems in the past, yes, but that was probably
when it was hosted elsewhere. The alias changed hands a few times by now.
About that particular problem - please use the option --max-delete for
rsync. Set it to something like 50000, that should be safe enough.
> I mean, ftp.cl.debian.org is a leaf mirror, low in the hierarchy, and
> should stay there based on it's not-so-hot "stableness". I see
> ftp.br.debian.org is also a leaf mirror, which I interpret as not being so
> high in the hierarchy either. Maybe I'm wrong about my interpretation of
> Leaf, Secondary and Primary mirrors, but these categories must somehow
> reflect how committed the sponsor and admins are to the mirror, right?
In this particular case, we have a bit of an inconsistency. By assigning the
ftp.*.d.o alias, we at Debian want to recognize the quality of a mirror and
relay this information to all the users, too.
The mirrors in the western hemisphere are mostly dependent on the Northern
American mirrors, mainly because ftp-master.debian.org is in the US and the
layout of the Internet links in general works that way. It would be logical
for us Debianites to work with mirror admins in both North and South America
in order to provide the best service; for example, we should organize the
first tier of push-mirrors and provide assistance with the second tier;
allowing more mirrors to get pushed so that they can update more
efficiently.
Yet, we have this bizarre situation where the North American mirrors
have been continuously either neglected or struggling due to external
influences, despite the fact that they are actually the closest ones to
ftp-master.debian.org.
We have the ftp.debian.org which is constantly overloaded, but graciously
hosted at a university which won't kick us out (at least not yet). It is
maxing out its 100Mbit connection virtually all the time[1]. We have
ftp.us.debian.org as a round-robin of three, currently - one is ftp.d.o,
the other is another mirror at a university, and the third is
mirrors.kernel.org, hosted by ISC. All of those are generally under a high
load, because the US users aren't aided in order to spread the load. The
Canadians either; we used to have a separate ftp.ca.debian.org, but once
that site went offline, nobody worked to seek a replacement, and instead
all that load was shifted onto ftp.debian.org. It's been like that for years
now.
The selection of mirrors in the round-robin is controlled by a few Debian
admins who aren't particularly interested in improving it, and the selection
has only shrunk in the last few years (once upon a time, we had six machines
in it; now we only have three). I have tried to contribute to fixing this
problem, but was told to butt out, mostly because as a European I didn't
have enough information about the US matters; I then on a separate occasion
organized a syncproxy.wna.d.o on the west coast, but that's as far as that
went (that was needed to fix up the Australian mirrors). On a few occasions
I contributed a few bits and pieces (such as asking elmo to up
ftp-master.d.o tcp_wmem so that it does cross-continental syncs faster), but
the main problem remained unsolved. I later went AWOL and nobody seemed to
pick up the torch after me. Now I'm a bit more active again, and I see the
status quo remains; and also I lost my (unprivileged) account on
ftp-master.d.o so I can't even track the log files over there any more,
and my requests to have it reinstated are ignored for months.
The secondary mirrors in the US which aren't part of the round-robin alias
are mostly not pushed, and I guess most people just don't use them. These
mirrors end up servicing a local community, which was usually the reason
they were set up; as well as the few people who actually go through the huge
list and pick a mirror from it.
So, the logical thing to do here would be to get your two mirrors, which
are the official mirrors for two large countries, pushed from either
ftp-master.debian.org or from one of the ftp.us.debian.org sites.
But, I don't have any idea how to do that, because a) nobody cares to do
these things b) the sites are mostly overloaded anyway so they would
probably be slow to sync from, unless some special arrangements were made.
I guess that now that I spent time writing this mail, I could have another
go at trying to find out which debian-admin member now tends after these
things, and see if anything can be changed.
> In our case, the environment is somewhat hostile, perhaps unforgiven about
> the needs of a server. For example, over some weekends, people saw the
> mirror on and turned it off thinking someone must have forgotten it on,
> because that's what people do with their windows desktops. Today the
> server is locked up, which only reduces the frequency of these events. Or,
> someone unplugs a switch to plug an electric kettle. Even more common,
> power outages lasting days, particularly during weekends when some people
> think switching off the entire building at the main fuse box helps
> conserving energy and money. That's the kind of things we have to struggle
> against, not to mention a Computing Department that seems to be playing
> around with a filter.
I trust you plan to get it into a proper server room in the future :)
Many of us have experienced similar problems in the past; I sympathise :/
--
2. That which causes joy or happiness.
[1] http://saens.debian.org/mrtg/saens.eth0.html
Reply to: