Re: K3 images

[Patrick Strasser <past@sbox.tugraz.at>]

Philip Charles wrote:

> Jigdo
>
> I will give as much notice as possible about when I freeze my archive
> before the final build.
>  
That is (please correct me if I'm wrong):
You make your special hardlink tree. Then you build the images based on 
this tree. Until final distribution of the images some packages won't be 
available on mirrors any more. These packages can only be found on the 
CD then.

> The boot floppies are the same as K2, but baseGNU.tgz and its derived
> floppy set will be different.
That was forseeable, as we have to expect a gnu (new) baseGNU.tgz for 
every CD set.

>  baseGNU etc could give problems as the
> jigdo build process will see them and not include them in the template for
> CD1.  Any ideas?
Files available on mirrors will not get into the template.
The template is $((image - files_available_at_mirrors))
As we don't have a mirror for baseGNU.tgz (until now), this file will 
get into the template.

We have three issues, sorted by importance:
1) Producing _small_ Jigdo files for easy, fast and cheap/painless 
distribution.
2) Making the images capable of recycling old images to reduce download 
volume to changed packages only.
3) Finding a way for using Jigdo to get the images fast from New Zealand 
to the Rest Of The World.

ad 1)
When baseGNU.tgz is not treated specially, it will go into the template 
and increasing template size. As we aim to produce small images, and 50 
MB is not exactly that small, getting baseGNU.tgz out of the template is 
desirable. Moreover we want to get all packages that are not available 
from mirros any more out of the template. As packages are taken from 
unstable and change over time, the number of unavailable but needed 
packages will increase in time.
Jigdo has a mechanism to solve this problem:
When packages are removed from the mirrors, they are hold back for some 
weeks (currently it's two weeks), to keep the weekly built "unstable" 
images rebuildable. The packages are checksumed, renamed to the checksum 
and moved to a special "fallback" directory. As the images are rebuilt 
every week, there is no need to keep them longer. If jigdo is unable to 
download them form the prefered mirror it looks in the fallback 
directory for the checksum, and downloads it.
The downside of this approach is, that old packages are only available 
at one place, thus giving heavy load on that server. On the other side, 
image remain usefull for some period.

What can we do?
I wrote to Atilla Nagy, the maintainer of the fallback directory, but 
got no response. I made a script that is capaple of building jigdo 
images without local access to a debian mirror tree. It creates a 
directory with files that can't be found on servers and have to be added 
to the fallback directory.
I can talk to our admins and ask for some space to host the files. I 
don't know how much traffic to expect.
baseGNU.tgz: we can make the same with baseGNU.tgz, adding it to the 
fallback directory. Or we can leave it in the template, increasing 
template size. I prefere to add it to fallback.

ad 2)
This should work straigt forward: If  a file can't be found on an old 
CD, the server is asked for the file, and if this failes,  the file is 
searched in fallback. in worst case, the image has to be rsynced tofill 
the last gaps.

ad 3)
With my script you could produce an "transport set" for sending the bare 
information about your CD compilation. You have to make sure that all 
files can be found when rebuilding the images. Perhaps someone at the 
target machine can do your hardlink trick at the same time, and thus 
ensuring smooth rebuilding.

Attached is my script. I'm shure it's not best style... ;-)

Patrick

--
Engineers motto: cheap, good, fast: choose any two
Patrick Strasser <past at sbox dot tugraz dot at>
Student of Telematik, Techn. University Graz, Austria