[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: CD release announcement, take 2

[I sent this before, but it doesn't seem to be getting through.]

Here's the next version of the CD release announcement to send to CD
mirror admins. If I don't get feedback, I'll assume it is OK like

I've just added include/exclude support to jigdo-mirror. PLEASE TEST
the attached version; if it works, I'll make a new jigdo release which
contains it.

Changes since the first version:
 - Complete set is 96 CD images (11*8 + 8 source), not 88
 - States that all images except 1_NONUS (84 images, 50 GB) will be
   available via rsync
 - Discusses excluding of certain files with
 - Mentions the DNS & server change from open to raff

From: Richard Atterer <atterer@debian.org>
Reply-To: debian-cd@lists.debian.org
Subject: Forthcoming release of CD images for Debian 3.0 rev0 ("woody")

This mail was sent to you because your address is stored in our
database of Debian CD mirror servers and your server is listed on
<http://www.debian.org/CD/http-ftp/>. Note that if you reply to this
mail, your answer will by default be sent to
<debian-cd@lists.debian.org>, a public mailing list.

Within the next weeks, Debian 3.0 "woody" will be released and new CD
images will be made available. The exact release date is not known yet
- subscribe to the debian-cd mailing list to be informed about it.

Here is some information about the release of the new CD images.

Required disc space grows to 50 or 57 GB
The images of the current stable distribution ("potato") need 17 GB of
disc space for 28 CD images. In contrast to this, the full set of
"woody" CDs needs about 57 GB for 96 CD images!

However, by default the 1_NONUS CDs will not be available for
rsync-based mirroring (and thus also debcdmirror), which reduces the
amount of space to 50 GB for 84 images.

The increase is due to the larger number of CDs (8, used to be 4) per
architecture, and the larger number of architectures (11, used to be

If you do not want to mirror the complete set of images, it is
possible to restrict which images to mirror and which to omit - see
below for instructions.

Setup changes on your mirror
You do not need to change anything about your current mirror setup if
you want to distribute the set of 84 CD images which needs about 50 GB
- the old "debcdmirror" scheme as well as rsync or FTP/HTTP mirroring
will continue to work.

However, consider changing the mirror setup as described below if one
of the following applies:

 - You want to update your mirror quickly. In our experience, the
   master site will be under heavy load around release time, possibly
   even to the point of not being reachable.
 - You already have a local "regular" Debian FTP mirror. In this case,
   the mirroring can be made much more efficient now.
 - You do not want to offer the full set of 96 CD images.

New way of mirroring: jigdo-mirror
jigdo is a new way of generating Debian CD images. A local (=same
machine) Debian FTP mirror is required for this. Additionally, if the
mirror does not run Linux on Intel, you'll have to compile jigdo
yourself - you need a recent C++ compiler (e.g. GCC 2.95) for this.

The jigdo-mirror script to automate mirroring of Debian's CD images is
new and needs more testing - if you can, please try it out now on the
3.0 pre-release images and report any success/failure to us!

jigdo-mirror takes packages from the mirror as well as special files
with ".jigdo" and ".template" extensions, and assembles the CD images
from all this information. This makes it similar to how debcdmirror
works, with the important difference that jigdo does not rely on rsync
to produce the final image.

A jigdo-based mirror requires
 - setting up a normal Debian FTP mirror <http://www.debian.org/mirror/>
 - setting up HTTP mirroring of the .jigdo/.template files at
 - setting up a cronjob which runs jigdo-mirror about once a day.
   Alternatively, you can also run jigdo-mirror manually whenever a
   new version of the CDs is released.
 - configuring jigdo-mirror. This should be easy, it hardly needs more
   information than the paths to the .jigdo/.template files and your
   Debian FTP mirror.

Excluding certain CD images from being stored on your mirror
The 96 images can be broken up as follows: For each of the 11
architectures (alpha, arm, hppa, i386, ia64, m68k, mips, mipsel,
powerpc, s390, sparc), there are 8 CDs, CD 1-7 and CD 1_NONUS. 
Furthermore, there are 8 "source" CDs, again 7 CDs and one NONUS
version of the first CD.

Mirror sites located in the United States should not mirror the
1_NONUS CD, because it contains software which is not distributable in
the US, e.g. because of patent issues. In contrast, mirrors outside
the US may *only* want to mirror CD 1_NONUS, not CD 1.

If 50 GB is too much for you, one possible way to restrict the needed
disc space to 17 GB is to mirror all CDs for i386 and CDs 1+2 of the
remaining architectures, and only to mirror one variant of the first
CD. Below are instructions on how to do this with the different
mirroring schemes.

rsync-based mirroring

  You can use rsync's include/exclude mechanism. Add the following
  switches to the rsync command:
  US sites:
    --exclude '*_NONUS*' --exclude 'source/*' --include 'i386/*' --exclude '*-[3-9].iso'
  Sites outside the US:
    --exclude '*-1.*' --exclude 'source/*' --include 'i386/*' --exclude '*-[3-9].iso'

jigdo-mirror based mirroring

  Use jigdo-mirror's include/exclude feature. The required setting is
  as follows:
  US sites:             include='i386/|-[12]'; exclude='source/|_NONUS'
  Sites outside the US: include='i386/|-[12]'; exclude='source/|-1\.'

mirroring based on debcdmirror

  Refer to the instructions on debcdmirror's include/exclude files in
  its documentation.

Changes in the setup of Debian's master CD mirror
In the past, the server where Debian CDs were generated was
cdimage.debian.org, a.k.a. open.hands.com, a machine in the UK. 
Starting with the release of woody, the cdimage DNS entry will point
to raff.debian.org (located in the US), while open will be available
as non-us.cdimage.debian.org.

Debian on CD:
Archives of the debian-cd mailing list:

The rsync utility:
rsync path for stable CD images:
  (Try not to mirror directly from the master site if possible.)
HTTP access is disabled for the images themselves, but the MD5SUM
files and .list files can be fetched here:

Retrieving Debian CDs with jigdo:
jigdo homepage:
How to set up jigdo-mirror:
Precompiled jigdo binary for i386 Linux, including documentation:
jigdo source code:

.jigdo/.template files for pre-release tests of Debian 3.0:
The .jigdo/.template files for 3.0 will be made available here:

The debcdmirror script:

In case of questions, feel free to reply to this mail. Thank you for
dedicating disc space to Debian CD images!

All the best,


  __   _
  |_) /|  Richard Atterer
  | \/¯|  http://atterer.net
  ¯ '` ¯
#! /bin/sh
#  __   _
#  |_) /|  Copyright (C) 2002 Richard Atterer
#  | \/¯|  <richard@atterer.net>
#  ¯ '` ¯
#  This program is free software; you can redistribute it and/or modify
#  it under the terms of the GNU General Public License, version 2. See
#  the file COPYING for details.

# Mirror script for Debian CD images, using Jigsaw Download. You first
# need to set up a conventional mirror (rsync/http/ftp-based) for the
# jigdo and template files, then this script can use those files and
# your local Debian mirror to create the full images automatically.

# This directory will be scanned for .jigdo files

# For any file $jigdoDir/somedir/file.jigdo, an attempt will be made
# to create all the images offered by file.jigdo in $imageDir/somedir/

# Temporary dir to use for creating images. Should be on the same
# partition as $imageDir, because mv is used to put finished images
# into $imageDir.

# Local Debian/Non-US mirrors. Can use http/ftp URLs, but beware that
# this may cause huge amounts of data to be downloaded repeatedly -
# the ftp/http server had better be on your LAN.

# Where to put the logfile. If undefined, log output goes to stdout.
# If defined, stderr is also redirected to the logfile
#logfile="$tmpDir/jigdo-mirror-`date +%y%m%d`.log"

# Include and exclude certain files. These are two regular extended
# expressions, matched case-sensitively. The input given to them is
# the filename of the .iso files relative to $imageDir, in the form
# "somedir/image.iso", e.g. "3.0rev0/i386/woody-i386-1.iso". The
# filtering is equivalent to
#  echo $name | egrep $include | egrep -v $exclude
# That is, first the list of files is restricted to those matching
# $include, then anything matching $exclude is removed.
include='.'  # include all files,
exclude='$^' # then exclude none
# Examples:
# US sites: Exclude non-US stuff
# US sites: All i386 images, images 1 and 2 of the rest
#include='i386/|-[12]'; exclude='source/|_NONUS'
# Sites outside the US: All i386 images, images 1 and 2 of the rest
#include='i386/|-[12]'; exclude='source/|-1\.'

# How to call jigdo-file or jigdo-port.
# CAREFUL: Make sure that jigdo-cache.db is not publically accessible
# from the internet since it contains local path info.
jigdoFile="jigdo-file --cache=$tmpDir/jigdo-cache.db --cache-expiry=1w --report=noprogress --no-check-files"
#jigdoFile="jigdo-port"; havePMA=false

# Any files older than $maxAge days are deleted from $imageDir, except
# when the variable is unset; in that case, nothing happens. WARNING:
# This really means *any* files, not just files generated by
# jigdo-mirror.

# In case only a few files are missing for the image to be complete,
# will download them from any fallback servers specified in the .jigdo
# file. Maximum number of missing files to download:

wgetOpts="--passive-ftp --no-directories --non-verbose"

# To find the template file, first the leafname of the template's URL
# is extracted from the .jigdo file. Next, for a .jigdo file named
# $jigdoDir/somedir/file.jigdo, the file
# $templateDir/somedir/leafnameFromURL is tried. If that isn't
# present, either the template is downloaded (if the URL is absolute)
# or looked for in $jigdoDir/somedir/templateURL (if the URL is
# relative). Default if unset is templateDir=$jigdoDir.

# If it is inconvenient for you to set the variables above, you can
# either specify a config file with settings on the command line...
if test "$1"; then
    . "$1"
elif test -r ~/.jigdo-mirror; then
    # ... or put the commands in "~/.jigdo-mirror"
    . ~/.jigdo-mirror
#  No user-serviceable parts below

# fetch <URL>...
# Download a file, storing it in the current dir
fetch() {
    if test "$#" -eq 0; then return 0; fi
    wget --user-agent="$userAgent" $wgetOpts "$@" || return 1
userAgent="jigdo-mirror/1.0 (`wget --version 2>/dev/null | (read ver; echo $ver)`)"

# isURI <string>
# Returns 0 (true) if the supplied string is a HTTP/FTP URL, otherwise 1
isURI() {
    case "$1" in
        http:*|ftp:*|HTTP:*|FTP:*|file:*|FILE:*) return 0;;
        *) return 1;

makeImage() {
    rm -f "image" "image.tmp"
    template=`basename "$templateURI"`
    if test -f "$templateDir/$dirName/$template"; then
        # Check for template in $templateDir
        log "    Found template \`\$templateDir/$dirName/$template'"
        ln -s "$templateDir/$dirName/$template" "template"
    elif isURI "$templateURI"; then
        # Absolute template URL - download
        log "    Template \`\$templateDir/$dirName/$template' not found, will download"
        if fetch "$templateURI" -O "template"; then true; else
            log "    Error getting template file"
            rm -f "image" "template"
            return 0
    elif test -f "$jigdoDir/$dirName/$templateURI"; then
        log "    Found template \`\$jigdoDir/$dirName/$templateURI'"
        ln -s "$jigdoDir/$dirName/$templateURI" "template"
        log "    Template file \`\$templateDir/$dirName/$template' not found"
        log "    Template file \`\$jigdoDir/$dirName/$templateURI' not found"
        rm -f "image" "template"
        return 0

    # If possible, check md5sum of template data
    if test "$templateMD5"; then
        set -- `$jigdoFile md5sum --report=quiet "template"`
        if test "$1" = "$templateMD5"; then
            log "    Template checksum is correct"
            log "    Error - template checksum mismatch"
            rm -f "image" "template"
            return 0
        log "    [WARNING - \`Template-MD5Sum' missing from image section]"

    # Try to merge any files into the image.
    if $havePMA; then
        $jigdoFile print-missing-all $ijtOpts $uriOpts \
        | egrep -v '^([a-zA-Z0-9.+_-]+:|$)' \
        | $jigdoFile make-image $ijtOpts --files-from=-
        $jigdoFile print-missing $ijtOpts $uriOpts \
        | egrep -v '^([a-zA-Z0-9.+_-]+:|$)' \
        | $jigdoFile make-image $ijtOpts --files-from=-
    if test "$jigdoErr" -ge 2; then
        log "    Error merging data from local filesystem"
        rm -f "image" "template"
        return 0

    # First try to download all files using the first URL in the
    # print-missing-all list. If any files remain missing, add another
    # pass, this time try to download the missing files using the 2nd
    # URL, and so on.
    for pass in x xx xxx xxxx xxxxx xxxxxx xxxxxxx xxxxxxxx; do
        if $havePMA; then
            $jigdoFile print-missing-all $ijtOpts $jigdoOpts $uriOpts \
            | egrep -i '^(http:|ftp:|$)' >"list"
            # Quick hack until jigdo-port supports print-missing-all
            $jigdoFile print-missing $ijtOpts $jigdoOpts $uriOpts \
            | egrep -i '^(http:|ftp:|$)' \
            | sed -n '/./{p;s/^.*$//;p;}' >"list"
        missingCount=`egrep '^$' <"list" | wc -l | sed -e 's/ *//g'`
        if test "$pass" = "x"; then
            if $localMirror; then true; else missingCount=0; fi
        if test "$missingCount" -gt "$maxMissing"; then
            log "    Too many files ($missingCount) missing in local mirror"
            rm -f "list" "image" "template"
            return 0
        # Accumulate URLs in $@, pass them to fetchAndMerge in batches
        set --
        while read url; do
            if test "$url" = ""; then count=""; continue; fi
            if test "$count" != "$pass"; then continue; fi
            if $noMorePasses; then
                log "    $missingCount parts still missing from image"
            set -- "$@" "$url"
            if test "$#" -ge "$filesPerFetch"; then
                if fetchAndMerge "$@"; then true; else
                    set --; noMorePasses=true
                set --
        done <"list"
        if test "$#" -ge 1; then
            if fetchAndMerge "$@"; then true; else break; fi
        if $noMorePasses; then break; fi
        if test -r "image"; then break; fi

    if test -r "image"; then
        # Finished - verify checksum
        if $jigdoFile verify $ijtOpts; then
            log "    Image checksum is correct, moving image into place"
            mkdir -p "$imageDir/$dirName"
            mv "image" "$imageDir/$dirName/$image"
            log "    Error - image checksum mismatch"
        log "    Image creation failed, list of missing files follows"
        $jigdoFile print-missing $ijtOpts $jigdoOpts $uriOpts
    rm -f "list" "image" "image.tmp" "template"
    return 0

# Given URLs, fetch them into $tmpDir, then merge them into image
fetchAndMerge() {
    (mkdir "$tmpDir/files"; cd "$tmpDir/files"; fetch "$@")
    # Merge into the image
    find "$tmpDir/files" -type f \
    | $jigdoFile make-image $ijtOpts --files-from=-
    if test "$jigdoErr" -ge 2; then
        exit 1
    # Delete tmpDir, to avoid taking up more space than necessary
    rm -rf "$tmpDir/files"

sectionEnd() {
  if test "$section" = "[Image]" -a "$image" \
          -a "$templateURI"; then
    log "  Image \`$image', template \`$templateURI'"
    set -- `echo "$dirName/$image" | egrep -- "$include" | egrep -v -- "$exclude"`
    if test "$#" -eq 0; then
      log "    \`\$imageDir/$dirName/$image' excluded by \$include/\$exclude"
    if test -f "$imageDir/$dirName/$image"; then
      if test "$jigdoDir/$jigdo" -nt "$imageDir/$dirName/$image";then
        log "    jigdo is newer - updating \`\$imageDir/$dirName/$image'"
        # Remove outdated image *immediately*, even in case
        # the subsequent attempt to regenerate it fails
        rm -f "$imageDir/$dirName/$image"
        log "    \`\$imageDir/$dirName/$image' is up to date"
        test "$maxAge" && touch "$imageDir/$dirName/$image"
      log "    Attempting to create \`\$imageDir/$dirName/$image'"

if test "$logfile"; then
    true >"$logfile"
    exec >>"$logfile"
    exec 2>>"$logfile"
log() { printf "%s: %s\n" "`date +'%Y-%m-%d %H:%M:%S'`" "$1"; }

log "imageDir:    $imageDir"
log "jigdoDir:    $jigdoDir"
log "templateDir: $templateDir"
# Remove slashes from dir names
uriOpts="--uri Debian='$debianMirror/' --uri Non-US='$nonusMirror/'"
ijtOpts="--image=image --jigdo=jigdo --template=template"
# Is the main mirror on the local disc?
case "$debianMirror $nonusMirror" in
    "file:"*" file:"*|/*" /"*) localMirror=true;;
    *) localMirror=false;;
if test -z "$havePMA"; then havePMA=true; fi
if test -z "$templateDir"; then templateDir="$jigdoDir"; fi
mkdir -p "$tmpDir" || true
cd "$tmpDir"

find "$jigdoDir" -name "*.jigdo" \
| sed -e "sÿ^$jigdoDir/ÿÿ" \
| while read jigdo; do

    log "Found \`\$jigdoDir/$jigdo'"
    dirName=`dirname "$jigdo"`
    if zcat "$jigdoDir/$jigdo" >"jigdo" 2>"/dev/null"; then
    elif test -f "$jigdoDir/$jigdo"; then
        rm -f "jigdo" "template"
        ln -s "$jigdoDir/$jigdo" "jigdo"
        log "  jigdo file not present/could not be unpacked - ignored"

    # Parse jigdo file, look for images
    while read REPLY; do
      set -- `echo "$REPLY" | sed -e 's/^ *\[ *\([^ ]*\) *\] *$/[\1]/; s/ *= */ /; s/['\''"$]//g'`
      case "$1" in
          unset image templateURI templateMD5 shortInfo info
        Filename) image="$2";;
        Template) templateURI="$2";;
        Template-MD5Sum) templateMD5="$2";;
        ShortInfo) shift; shortInfo="$*";;
        Info) shift; info="$*";;
    done <"jigdo"

    rm -f "jigdo"


if cd "$imageDir"; then
    if test "$maxAge"; then
        log "Expiring images older than $maxAge days"
        find . -type f -mtime +"$maxAge" \
        | while read file; do
            log "  Deleting \`\$imageDir/${file#./}'"
            rm -f "$file"
        # Remove empty directories
        find "$imageDir" -depth -mindepth 1 -type d -empty \
            -exec rmdir '{}' ';'

log "Exit $exitCode"
exit $exitCode

Attachment: pgpfGjaRG2NeC.pgp
Description: PGP signature

Reply to: