Re: K3 images
Philip Charles wrote:
I will give as much notice as possible about when I freeze my archive
before the final build.
That is (please correct me if I'm wrong):
You make your special hardlink tree. Then you build the images based on
this tree. Until final distribution of the images some packages won't be
available on mirrors any more. These packages can only be found on the
That was forseeable, as we have to expect a gnu (new) baseGNU.tgz for
every CD set.
The boot floppies are the same as K2, but baseGNU.tgz and its derived
floppy set will be different.
baseGNU etc could give problems as the
jigdo build process will see them and not include them in the template for
CD1. Any ideas?
Files available on mirrors will not get into the template.
The template is $((image - files_available_at_mirrors))
As we don't have a mirror for baseGNU.tgz (until now), this file will
get into the template.
We have three issues, sorted by importance:
1) Producing _small_ Jigdo files for easy, fast and cheap/painless
2) Making the images capable of recycling old images to reduce download
volume to changed packages only.
3) Finding a way for using Jigdo to get the images fast from New Zealand
to the Rest Of The World.
When baseGNU.tgz is not treated specially, it will go into the template
and increasing template size. As we aim to produce small images, and 50
MB is not exactly that small, getting baseGNU.tgz out of the template is
desirable. Moreover we want to get all packages that are not available
from mirros any more out of the template. As packages are taken from
unstable and change over time, the number of unavailable but needed
packages will increase in time.
Jigdo has a mechanism to solve this problem:
When packages are removed from the mirrors, they are hold back for some
weeks (currently it's two weeks), to keep the weekly built "unstable"
images rebuildable. The packages are checksumed, renamed to the checksum
and moved to a special "fallback" directory. As the images are rebuilt
every week, there is no need to keep them longer. If jigdo is unable to
download them form the prefered mirror it looks in the fallback
directory for the checksum, and downloads it.
The downside of this approach is, that old packages are only available
at one place, thus giving heavy load on that server. On the other side,
image remain usefull for some period.
What can we do?
I wrote to Atilla Nagy, the maintainer of the fallback directory, but
got no response. I made a script that is capaple of building jigdo
images without local access to a debian mirror tree. It creates a
directory with files that can't be found on servers and have to be added
to the fallback directory.
I can talk to our admins and ask for some space to host the files. I
don't know how much traffic to expect.
baseGNU.tgz: we can make the same with baseGNU.tgz, adding it to the
fallback directory. Or we can leave it in the template, increasing
template size. I prefere to add it to fallback.
This should work straigt forward: If a file can't be found on an old
CD, the server is asked for the file, and if this failes, the file is
searched in fallback. in worst case, the image has to be rsynced tofill
the last gaps.
With my script you could produce an "transport set" for sending the bare
information about your CD compilation. You have to make sure that all
files can be found when rebuilding the images. Perhaps someone at the
target machine can do your hardlink trick at the same time, and thus
ensuring smooth rebuilding.
Attached is my script. I'm shure it's not best style... ;-)
Engineers motto: cheap, good, fast: choose any two
Patrick Strasser <past at sbox dot tugraz dot at>
Student of Telematik, Techn. University Graz, Austria
- K3 images
- From: Philip Charles <email@example.com>