Re: Debian services and Debian infrastructure
On 07/02/14 at 18:16 +0100, Peter Palfrader wrote:
> On Sat, 08 Feb 2014, Thomas Goirand wrote:
>
> > It'd be super nice to have
> > the archive rebuild jobs running on the Debian infrastructure rather
> > than on AWS for example.
>
> I agree, and it has been proposed several times over the last few years.
> To say there was no interest whatsoever would overstate the amount of
> excitement those suggestions have received.
The archive rebuilds are currently running over a very short amount of
time (< 8 hours). That's an important feature, because it allows one to
rebuild a snapshot of the archive (or a quasi-snapshot, since that could
span two dinstalls) and make sure to find all build failures in that
snapshot. Unfortunately, to do that, one requires quite a lot of
computing power. As a wild guess, I would say 20 to 30 recent servers
(I've been using more than 100 EC2 VM for some rebuilds, especially when
doing two rebuilds at the same time to compare the rebuilds, e.g. gcc
4.n vs gcc 4.(n+1)).
We currently get this computing power for free from Amazon. We used to
get it from my employer through a research project (Grid'5000), and we
could return to that if needed: the move to Amazon was an attempt at
switching from 'only Lucas can work on that' to 'other people can work
on that', which was a success given at least David Suarez (general
purpose rebuilds + bug filing) and Sylvestre Ledru (clang rebuilds) have
been actively running rebuilds over the last year, and I haven't. Also,
we made sure of not using anything Amazon-specific in the rebuild
infrastructure.
Doing those rebuilds on Debian infrastructure would require either:
1) investing in a set of powerful machines to do the rebuilds ($50k as a
guess, $2k * 25 -- and that's probably the bare minimum we would need),
finding hosting for them, managing them. We get all of that for free
from Amazon, so I don't think that this would be good use of Debian's
funds.
2) degrading the service: loosen the requirement to rebuild
quasi-snapshots of the archive, or use snapshot.debian.org to build
a possibly older snapshot of the archive (which would result in
filing bugs that were already fixed).
None of those options look particularly exciting. That's why I never
explored further the idea of running those rebuilds on Debian
infrastructure. But maybe you have a nice solution, that you haven't
explained yet?
One thing we could do, though, is move all the "static" part of the
rebuild infrastructure to Debian infrastructure. That's the virtual
machine used to schedule builds on all temporary nodes, and the
virtual machine used to store logs.
Those logs used to be stored in http://people.d.o/~lucas/, which did not
work scale well to several people doing the rebuilds. I approached DSA
asking if they would be willing to provide similar archival space in a
place accessible by a team including non-DDs. (#debian-admin,
2013-06-03) All of the suggested solutions (mail all build logs as
attachments to the bug reports; ask someone else or a cron job to rsync
from external host to qa.d.o; push to buildd.d.o) had drawbacks or
required additional work that I didn't have time for at the time, so we
have just been using an EC2 VM (aws-logs.d.n) to store the logs since
then. Probably that should be revisited, but I'm no longer the good point
of contact for that (David Suarez is).
Lucas
Reply to: