[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: K3 images

Philip Charles wrote:


I will give as much notice as possible about when I freeze my archive
before the final build.
That is (please correct me if I'm wrong):
You make your special hardlink tree. Then you build the images based on this tree. Until final distribution of the images some packages won't be available on mirrors any more. These packages can only be found on the CD then.

The boot floppies are the same as K2, but baseGNU.tgz and its derived
floppy set will be different.

That was forseeable, as we have to expect a gnu (new) baseGNU.tgz for every CD set.

 baseGNU etc could give problems as the
jigdo build process will see them and not include them in the template for
CD1.  Any ideas?

Files available on mirrors will not get into the template.
The template is $((image - files_available_at_mirrors))
As we don't have a mirror for baseGNU.tgz (until now), this file will get into the template.

We have three issues, sorted by importance:
1) Producing _small_ Jigdo files for easy, fast and cheap/painless distribution. 2) Making the images capable of recycling old images to reduce download volume to changed packages only. 3) Finding a way for using Jigdo to get the images fast from New Zealand to the Rest Of The World.

ad 1)
When baseGNU.tgz is not treated specially, it will go into the template and increasing template size. As we aim to produce small images, and 50 MB is not exactly that small, getting baseGNU.tgz out of the template is desirable. Moreover we want to get all packages that are not available from mirros any more out of the template. As packages are taken from unstable and change over time, the number of unavailable but needed packages will increase in time.
Jigdo has a mechanism to solve this problem:
When packages are removed from the mirrors, they are hold back for some weeks (currently it's two weeks), to keep the weekly built "unstable" images rebuildable. The packages are checksumed, renamed to the checksum and moved to a special "fallback" directory. As the images are rebuilt every week, there is no need to keep them longer. If jigdo is unable to download them form the prefered mirror it looks in the fallback directory for the checksum, and downloads it. The downside of this approach is, that old packages are only available at one place, thus giving heavy load on that server. On the other side, image remain usefull for some period.

What can we do?
I wrote to Atilla Nagy, the maintainer of the fallback directory, but got no response. I made a script that is capaple of building jigdo images without local access to a debian mirror tree. It creates a directory with files that can't be found on servers and have to be added to the fallback directory. I can talk to our admins and ask for some space to host the files. I don't know how much traffic to expect. baseGNU.tgz: we can make the same with baseGNU.tgz, adding it to the fallback directory. Or we can leave it in the template, increasing template size. I prefere to add it to fallback.

ad 2)
This should work straigt forward: If a file can't be found on an old CD, the server is asked for the file, and if this failes, the file is searched in fallback. in worst case, the image has to be rsynced tofill the last gaps.

ad 3)
With my script you could produce an "transport set" for sending the bare information about your CD compilation. You have to make sure that all files can be found when rebuilding the images. Perhaps someone at the target machine can do your hardlink trick at the same time, and thus ensuring smooth rebuilding.

Attached is my script. I'm shure it's not best style... ;-)


Engineers motto: cheap, good, fast: choose any two
Patrick Strasser <past at sbox dot tugraz dot at>
Student of Telematik, Techn. University Graz, Austria

Reply to: