[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: [Emdebian] Mirrors

On Fri, 18 Sep 2009 15:54:58 +0800
Andrew Lee <ajqlee@debian.org> wrote:

> Let's discuss this with debian-mirrors folks. They have good
> experience on setup mirrors for all kind of Debian related repository
> mirrors.

True - however we don't quite do things as Debian would do things,
principally due to the lack of an incoming.emdebian.org staging area
that results in the archive being continually updated. This is due to a
technical limitation (lack of sufficient space) and changes in the
scripts to cope with the extra space.

debian-mirrors people: feel free to ignore the thread until such time
as Grip has a more compatible setup. Sorry for any noise.

> Hector has setup a push trigger script on emdebian.org, based on the
> signal script example from http://www.debian.org/mirror/push_server
> And the script works to trigger ftp.tw.d.o to kick up ftpsync script
> on ftp.tw.d.o to sync from emdebian.org.
> Question:
> * What's the proper way to hook to send trigger to downstream mirrors
> on emdebian.org server?

Get more space onto www.emdebian.org so that we can use a more
Debian-like process in the first place at which point normal Debian
mirror techniques become appropriate. Right now, mirroring only makes
things worse.
> * What's the best path name to provide via rsyncd for mirrors?
> Current repositories name available on emdebian.org are:
> $ rsync -av emdebian.org::
> debian         	emdebian tools and native packages archive
> crush          	emdebian crush archive
> grip           	emdebian grip archive
> Folder name 'debian' is conflicted to main Debian's repo. So I placed
> emdebian.org as:
> http://ftp.tw.debian.org/emdebian.org/debian/
> http://ftp.tw.debian.org/emdebian.org/crush/
> http://ftp.tw.debian.org/emdebian.org/grip/

IIRC the toolchains will be migrating into Debian anyway so the whole
debian/ repository could become moot.

The native package support in debian/ is also largely unused now -
principally because the packages themselves do not need such rapid
changes that we need a new version before the old one has migrated
into testing.

That should leave only crush/ and grip/

However, I repeat my request that people *don't* mirror Grip but
instead run partial buildd's using the current emdebian-grip-server
package to add more packages to Grip. We don't need mirrors right now,
we need more machines preparing more packages resulting in bug reports
against emdebian-grip-server and hence better code in the package

Also, moving Grip processing to a backend server would free up space
for more packages to go onto the main emdebian.org server - as long as
that server isn't the one doing the work. This improvement has little
to do with mirroring.

Until we have an incoming staging area, I don't think it's practical to
mirror Grip or involve the debian-mirrors people.

> -Andrew

(please don't top post)
> >> Not practical. With Grip, the packages are added by the
> >> emdebian-grip-server package. The only real way is to do the call
> >> at the end. Right now, grip-cron.sh appears to need more than
> >> several hours to run - about 7-10hrs at the moment. I think it had
> >> something to do with the outage because it usually takes quite a
> >> bit less and the scripts are still trying to catch up with
> >> unstable. It could also be simply dealing with much, much larger
> >> packages which take a lot of time to Grip. I'm hoping to take a
> >> look at what is happening tomorrow but the time required means
> >> that I can't actually do anything with the repository in the
> >> evenings as reprepro is almost constantly locked (for
> >> additions/updates).
> > 
> > About your lags, i have been building toolchains on ant, that could
> > explain your timings.
> > According to Simon, the machine was down because KVM did not
> > attached to the right network interface, but else it is posible too.
> > 
> >>> Note that an scalable solution is better, because for each mirror
> >>> we add, we need to add a signal entry
> >> This kind of signal is not something that other machines running
> >> emdebian-grip-server could support. Plus, adding a signal every
> >> time is only going to turn grip-cron.sh into a 24hr script at
> >> which point it becomes unusable.




Neil Williams

Attachment: pgppTJ2QbCulg.pgp
Description: PGP signature

Reply to: