Re: Status update
On Sat, Mar 17, 2001 at 06:56:31PM -0500, Roland McGrath wrote:
> > If you're interested in helping me get some of these to compile, let me
> > know - I'd like to avoid duplication of effort (I cheerfully will let
> > other people do any of the work they want to!)
>
> Since you are building so many packages, it would be helpful if I could
> look at an automated web page that tells me the status of each package,
> and ideally lets me get the full output of the last build attempt.
I hope Jeff will be able to push the status web pages through his fire walls.
> > If there is are Debian packages that *compiles cleanly* that you want,
> > please let me know. I don't have the time to chase source bugs right
> > now, but if something is buildable, I will keep it up to date.
>
> It would be great to have something automated to try new packages, if that
> is not hard for you to do.
This idea is so good, that I thought of it half a year ago already ;)
http://sourceforge.net/projects/turtle
"Perl classes and supporting scripts that allow to monitor a source
repository and process new or existing versions, for example by
downloading and compiling software automatically or on request and
uploading built packages to a binary repository."
(I should try to get a patent for that)
(If you d/l it, use the CVS version).
> What I have in mind is something that simply
> attempts to build a package and records what happened. Then these would be
> automatically classified into "built" and "failed to build"
And "out-of-date" (== to be build), "up-to-date", "other-arch" and soon
"claimed" (to be build by someone else, not us).
> (and maybe
> "built but with warnings" if you have some regex matching on the compile
> output); for things that support "make check" you could try that too and
> add another bit to the matrix in the output.
Interesting. I would prefer to stick with Debian interfaces solely for now,
but it should be easy to hack in. Maybe a per-package script that can be run
in the source tree post-compile and which return code is evaluated.
> For anything that builds
> successfully, then a human can take a quick look at the build log and see
> if it looks like it might really be usable, and then decide to actually try
> it out; when a human declares an autobuilt package is actually usable, it
> can be published.
Trying out is not a common step. For most packages, it is quite acceptable
if the version in the archive is broken for a few days. It just has to be
tried out by someone.
> For anything that fails to build, then a human can take
> a quick look at the build log and with very little effort decide from the
> kinds of errors whether or not they want to make the attempt to fix it.
Turtle sends an e-mail to a group of admins, including build logs and the
changes file. There is a mime type attached which handles signing the
changes file and sending it back. You can set up forwarding on some turtle
email address to have the changes file automatically installed and the
package uploaded immediately. Very convenient.
> It would be ideal to have this kind of system processing a queue of all
> source packages in the debian pool as new or updated ones arrive.
Done. In fact, it constantly compares the Packages file with the Source file
versions to determine what is up-to-date etc.
> It could
> prioritize the queue, doing first packages whose previous source versions
> were manually declared working, second packages that are entirely new,
> third packages whose previous source versions autobuilt ok, and lastly ones
> that never worked; or whatever the policy, but using those kinds of
> criteria to prioritize.
Currently it prioritizes using the package information: Packages in base, or
essential packages, library packages, and packages with a high Debian
priority are done first. Games and optional/extra stuff is done later.
I think that categorizing on past experience with building the package is
not feasible. At least it would be quite hackish to implement and perform
automatically. However, you can override the default priority at a
per-package level, so you can bump up the priority of packages you know
work/work not.
> With that particular prioritization, some packages
> might never get tried if there are always more important packages getting
> updated.
Speed will not be an issue. Currently we have a limit imposed by the
stability of the Hurd. Usually, an autobuilder can run for several hours
until it crashes. Most often it was proc dying. It seems a system() call
from perl with a long duration (hours) triggers some nasty bug.[1]
> If such a system is set up and does a little bit of extra
> coordination to farm out pieces of the work, then several of us can set up
> hurd machines that spend their spare time working on autobuilding while we
> sleep.
That's on the to-do list[2]. Coordinating several turtles is currently only
feasible on a per package base (split into distinct set of packages
monitored). But I am currently working out a scheme to delegate packages to
sub turtles on request.
Thanks,
Marcus
[1] This and the pfinet troubles are the reason why my initial efforts with
an autobuilder that runs 24x7 remotely failed.
[2] It seems I might get a permanent connection donated soon, and you bet that
I will get into autobuilding when this happens. Lots of cycles to spare.
--
`Rhubarb is no Egyptian god.' Debian http://www.debian.org brinkmd@debian.org
Marcus Brinkmann GNU http://www.gnu.org marcus@gnu.org
Marcus.Brinkmann@ruhr-uni-bochum.de
http://www.marcus-brinkmann.de
Reply to: