[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: wanna-build / how to sort packages on buildds?

On Sun, 1 May 2011 01:36:38 +0200, Andreas Barth wrote:

Sometimes we have a few packages we don't want to build on a certain
buildds. Sometimes this is because this package needs lots of ram. Or
it takes quite long and would waste the parallel building a machine
supports. Or whatever else. Of course a package could be in more than
one category.

Yes, you're facing basically the same problem I tried to address in 2000/2001 when doing my renderserver and later for what Multibuild was intended to do as well. ;-)

Now, what I would like to do is to write that down in a central file
with categories.

I would recommend to use a database, really.

That is, to mark packages as "builds only with more than one gigabyte
of ram". And to mark buildds as "has 6 cores", "only ... ram" - so
that I don't need to copy entries from buildd to buildd, but just say
"that new machine is the same class as ...", and that's it.

Another category would be "fast disk/raid". There are some packages with lots of disk accesses. When you can schedule those packages to a buildd that has faster disk access like in having multiple spindles for faster seeks, you can minimize build times as well. We faced that problem on m68k particularly on IDE vs SCSI disks on Amigas, as IDE was dog slow. Another example there was the faster disks on Amigas vs slower SCSI disks in Apple machines.

Now my question is just: How to do that efficient? I.e. how would such
a configuration file look like, and how the code to distribute the
package on the most fitting buildd(s)? (I.e. it's better to waste 5
out of 6 cores than to not build a package at all, but a package
needing at least 1g ram can't build on a buildd with only 512mb - but
no package should starve in the end.)
Ideas? Suggestions? Code?

Look at my update-buildd.net from Buildd.net, which I used to collect data from the buildds such as RAM, kernel, uptime, used swap and such (http://buildd.net/cgi/hostpackages.cgi?unstable_arch=m68k&searchtype=arrakis). I store this information into the database and also the build times of the packages. With this dataset it should be possible to have the wanna-buildd schedule packages in such a way to minimize the build times because you can decide which buildd is the most suitable buildd for the next package.

Ciao...            //      Fon: 0381-2744150
      Ingo       \X/       http://blog.windfluechter.net

gpg pubkey: http://www.juergensmann.de/ij_public_key.

Reply to: