[I sent this before, but it doesn't seem to be getting through.]
Here's the next version of the CD release announcement to send to CD
mirror admins. If I don't get feedback, I'll assume it is OK like
that.
I've just added include/exclude support to jigdo-mirror. PLEASE TEST
the attached version; if it works, I'll make a new jigdo release which
contains it.
Changes since the first version:
- Complete set is 96 CD images (11*8 + 8 source), not 88
- States that all images except 1_NONUS (84 images, 50 GB) will be
available via rsync
- Discusses excluding of certain files with
jigdo-mirror/rsync/debcdmirror
- Mentions the DNS & server change from open to raff
----------------------------------------------------------------------
From: Richard Atterer <atterer@debian.org>
Reply-To: debian-cd@lists.debian.org
Subject: Forthcoming release of CD images for Debian 3.0 rev0 ("woody")
This mail was sent to you because your address is stored in our
database of Debian CD mirror servers and your server is listed on
<http://www.debian.org/CD/http-ftp/>. Note that if you reply to this
mail, your answer will by default be sent to
<debian-cd@lists.debian.org>, a public mailing list.
Within the next weeks, Debian 3.0 "woody" will be released and new CD
images will be made available. The exact release date is not known yet
- subscribe to the debian-cd mailing list to be informed about it.
Here is some information about the release of the new CD images.
Required disc space grows to 50 or 57 GB
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The images of the current stable distribution ("potato") need 17 GB of
disc space for 28 CD images. In contrast to this, the full set of
"woody" CDs needs about 57 GB for 96 CD images!
However, by default the 1_NONUS CDs will not be available for
rsync-based mirroring (and thus also debcdmirror), which reduces the
amount of space to 50 GB for 84 images.
The increase is due to the larger number of CDs (8, used to be 4) per
architecture, and the larger number of architectures (11, used to be
6).
If you do not want to mirror the complete set of images, it is
possible to restrict which images to mirror and which to omit - see
below for instructions.
Setup changes on your mirror
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You do not need to change anything about your current mirror setup if
you want to distribute the set of 84 CD images which needs about 50 GB
- the old "debcdmirror" scheme as well as rsync or FTP/HTTP mirroring
will continue to work.
However, consider changing the mirror setup as described below if one
of the following applies:
- You want to update your mirror quickly. In our experience, the
master site will be under heavy load around release time, possibly
even to the point of not being reachable.
- You already have a local "regular" Debian FTP mirror. In this case,
the mirroring can be made much more efficient now.
- You do not want to offer the full set of 96 CD images.
New way of mirroring: jigdo-mirror
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
jigdo is a new way of generating Debian CD images. A local (=same
machine) Debian FTP mirror is required for this. Additionally, if the
mirror does not run Linux on Intel, you'll have to compile jigdo
yourself - you need a recent C++ compiler (e.g. GCC 2.95) for this.
The jigdo-mirror script to automate mirroring of Debian's CD images is
new and needs more testing - if you can, please try it out now on the
3.0 pre-release images and report any success/failure to us!
jigdo-mirror takes packages from the mirror as well as special files
with ".jigdo" and ".template" extensions, and assembles the CD images
from all this information. This makes it similar to how debcdmirror
works, with the important difference that jigdo does not rely on rsync
to produce the final image.
A jigdo-based mirror requires
- setting up a normal Debian FTP mirror <http://www.debian.org/mirror/>
- setting up HTTP mirroring of the .jigdo/.template files at
<http://cdimage.debian.org/jigdo-area/current/jigdo/>
- setting up a cronjob which runs jigdo-mirror about once a day.
Alternatively, you can also run jigdo-mirror manually whenever a
new version of the CDs is released.
- configuring jigdo-mirror. This should be easy, it hardly needs more
information than the paths to the .jigdo/.template files and your
Debian FTP mirror.
Excluding certain CD images from being stored on your mirror
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The 96 images can be broken up as follows: For each of the 11
architectures (alpha, arm, hppa, i386, ia64, m68k, mips, mipsel,
powerpc, s390, sparc), there are 8 CDs, CD 1-7 and CD 1_NONUS.
Furthermore, there are 8 "source" CDs, again 7 CDs and one NONUS
version of the first CD.
Mirror sites located in the United States should not mirror the
1_NONUS CD, because it contains software which is not distributable in
the US, e.g. because of patent issues. In contrast, mirrors outside
the US may *only* want to mirror CD 1_NONUS, not CD 1.
If 50 GB is too much for you, one possible way to restrict the needed
disc space to 17 GB is to mirror all CDs for i386 and CDs 1+2 of the
remaining architectures, and only to mirror one variant of the first
CD. Below are instructions on how to do this with the different
mirroring schemes.
rsync-based mirroring
You can use rsync's include/exclude mechanism. Add the following
switches to the rsync command:
US sites:
--exclude '*_NONUS*' --exclude 'source/*' --include 'i386/*' --exclude '*-[3-9].iso'
Sites outside the US:
--exclude '*-1.*' --exclude 'source/*' --include 'i386/*' --exclude '*-[3-9].iso'
jigdo-mirror based mirroring
Use jigdo-mirror's include/exclude feature. The required setting is
as follows:
US sites: include='i386/|-[12]'; exclude='source/|_NONUS'
Sites outside the US: include='i386/|-[12]'; exclude='source/|-1\.'
mirroring based on debcdmirror
Refer to the instructions on debcdmirror's include/exclude files in
its documentation.
Changes in the setup of Debian's master CD mirror
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the past, the server where Debian CDs were generated was
cdimage.debian.org, a.k.a. open.hands.com, a machine in the UK.
Starting with the release of woody, the cdimage DNS entry will point
to raff.debian.org (located in the US), while open will be available
as non-us.cdimage.debian.org.
Links
~~~~~
Debian on CD:
<http://www.debian.org/CD/>
Archives of the debian-cd mailing list:
<http://lists.debian.org/debian-cd/>
The rsync utility:
<http://rsync.samba.org/>
rsync path for stable CD images:
<rsync://cdimage.debian.org/debian-cd/>
(Try not to mirror directly from the master site if possible.)
HTTP access is disabled for the images themselves, but the MD5SUM
files and .list files can be fetched here:
<http://cdimage.debian.org/cd-images/>
Retrieving Debian CDs with jigdo:
<http://www.debian.org/CD/jigdo-cd/>
jigdo homepage:
<http://atterer.net/jigdo/>
How to set up jigdo-mirror:
<http://atterer.net/jigdo/jigdo-mirror.html>
Precompiled jigdo binary for i386 Linux, including documentation:
<http://atterer.net/jigdo/jigdo-bin-0.6.7.tar.bz2>
jigdo source code:
<http://atterer.net/jigdo/jigdo-0.6.7.tar.bz2>
.jigdo/.template files for pre-release tests of Debian 3.0:
<http://cdimage.debian.org/jigdo-area/>
The .jigdo/.template files for 3.0 will be made available here:
<http://cdimage.debian.org/jigdo-area/current/jigdo/>
The debcdmirror script:
<http://cdimage.debian.org/~costar/debcdmirror/>
In case of questions, feel free to reply to this mail. Thank you for
dedicating disc space to Debian CD images!
All the best,
Richard
--
__ _
|_) /| Richard Atterer
| \/¯| http://atterer.net
¯ '` ¯
#! /bin/sh
# __ _
# |_) /| Copyright (C) 2002 Richard Atterer
# | \/¯| <richard@atterer.net>
# ¯ '` ¯
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License, version 2. See
# the file COPYING for details.
# Mirror script for Debian CD images, using Jigsaw Download. You first
# need to set up a conventional mirror (rsync/http/ftp-based) for the
# jigdo and template files, then this script can use those files and
# your local Debian mirror to create the full images automatically.
# This directory will be scanned for .jigdo files
jigdoDir="/home/ftp/debian/jigdo"
# For any file $jigdoDir/somedir/file.jigdo, an attempt will be made
# to create all the images offered by file.jigdo in $imageDir/somedir/
imageDir="/home/ftp/debian-cd"
# Temporary dir to use for creating images. Should be on the same
# partition as $imageDir, because mv is used to put finished images
# into $imageDir.
tmpDir="/home/jigdo-mirror-tmpdir"
# Local Debian/Non-US mirrors. Can use http/ftp URLs, but beware that
# this may cause huge amounts of data to be downloaded repeatedly -
# the ftp/http server had better be on your LAN.
debianMirror="file:/home/ftp/debian"
nonusMirror="file:/home/ftp/debian/non-US"
# Where to put the logfile. If undefined, log output goes to stdout.
# If defined, stderr is also redirected to the logfile
#logfile="$tmpDir/jigdo-mirror-`date +%y%m%d`.log"
# Include and exclude certain files. These are two regular extended
# expressions, matched case-sensitively. The input given to them is
# the filename of the .iso files relative to $imageDir, in the form
# "somedir/image.iso", e.g. "3.0rev0/i386/woody-i386-1.iso". The
# filtering is equivalent to
# echo $name | egrep $include | egrep -v $exclude
# That is, first the list of files is restricted to those matching
# $include, then anything matching $exclude is removed.
include='.' # include all files,
exclude='$^' # then exclude none
# Examples:
# US sites: Exclude non-US stuff
#exclude='_NONUS'
# US sites: All i386 images, images 1 and 2 of the rest
#include='i386/|-[12]'; exclude='source/|_NONUS'
# Sites outside the US: All i386 images, images 1 and 2 of the rest
#include='i386/|-[12]'; exclude='source/|-1\.'
# How to call jigdo-file or jigdo-port.
# CAREFUL: Make sure that jigdo-cache.db is not publically accessible
# from the internet since it contains local path info.
jigdoFile="jigdo-file --cache=$tmpDir/jigdo-cache.db --cache-expiry=1w --report=noprogress --no-check-files"
#jigdoFile="jigdo-port"; havePMA=false
# Any files older than $maxAge days are deleted from $imageDir, except
# when the variable is unset; in that case, nothing happens. WARNING:
# This really means *any* files, not just files generated by
# jigdo-mirror.
#maxAge=8
# In case only a few files are missing for the image to be complete,
# will download them from any fallback servers specified in the .jigdo
# file. Maximum number of missing files to download:
maxMissing=100
filesPerFetch=10
wgetOpts="--passive-ftp --no-directories --non-verbose"
# To find the template file, first the leafname of the template's URL
# is extracted from the .jigdo file. Next, for a .jigdo file named
# $jigdoDir/somedir/file.jigdo, the file
# $templateDir/somedir/leafnameFromURL is tried. If that isn't
# present, either the template is downloaded (if the URL is absolute)
# or looked for in $jigdoDir/somedir/templateURL (if the URL is
# relative). Default if unset is templateDir=$jigdoDir.
#templateDir="$jigdoDir"
# If it is inconvenient for you to set the variables above, you can
# either specify a config file with settings on the command line...
if test "$1"; then
. "$1"
elif test -r ~/.jigdo-mirror; then
# ... or put the commands in "~/.jigdo-mirror"
. ~/.jigdo-mirror
fi
#======================================================================
# No user-serviceable parts below
#======================================================================
# fetch <URL>...
# Download a file, storing it in the current dir
fetch() {
if test "$#" -eq 0; then return 0; fi
wget --user-agent="$userAgent" $wgetOpts "$@" || return 1
}
userAgent="jigdo-mirror/1.0 (`wget --version 2>/dev/null | (read ver; echo $ver)`)"
#______________________________________________________________________
# isURI <string>
# Returns 0 (true) if the supplied string is a HTTP/FTP URL, otherwise 1
isURI() {
case "$1" in
http:*|ftp:*|HTTP:*|FTP:*|file:*|FILE:*) return 0;;
*) return 1;
esac
}
#______________________________________________________________________
makeImage() {
rm -f "image" "image.tmp"
template=`basename "$templateURI"`
if test -f "$templateDir/$dirName/$template"; then
# Check for template in $templateDir
log " Found template \`\$templateDir/$dirName/$template'"
ln -s "$templateDir/$dirName/$template" "template"
elif isURI "$templateURI"; then
# Absolute template URL - download
log " Template \`\$templateDir/$dirName/$template' not found, will download"
if fetch "$templateURI" -O "template"; then true; else
log " Error getting template file"
exitCode=1
rm -f "image" "template"
return 0
fi
elif test -f "$jigdoDir/$dirName/$templateURI"; then
log " Found template \`\$jigdoDir/$dirName/$templateURI'"
ln -s "$jigdoDir/$dirName/$templateURI" "template"
else
log " Template file \`\$templateDir/$dirName/$template' not found"
log " Template file \`\$jigdoDir/$dirName/$templateURI' not found"
exitCode=1
rm -f "image" "template"
return 0
fi
# If possible, check md5sum of template data
if test "$templateMD5"; then
set -- `$jigdoFile md5sum --report=quiet "template"`
if test "$1" = "$templateMD5"; then
log " Template checksum is correct"
else
log " Error - template checksum mismatch"
exitCode=1
rm -f "image" "template"
return 0
fi
else
log " [WARNING - \`Template-MD5Sum' missing from image section]"
fi
# Try to merge any files into the image.
if $havePMA; then
$jigdoFile print-missing-all $ijtOpts $uriOpts \
| egrep -v '^([a-zA-Z0-9.+_-]+:|$)' \
| $jigdoFile make-image $ijtOpts --files-from=-
jigdoErr="$?"
else
$jigdoFile print-missing $ijtOpts $uriOpts \
| egrep -v '^([a-zA-Z0-9.+_-]+:|$)' \
| $jigdoFile make-image $ijtOpts --files-from=-
jigdoErr="$?"
fi
if test "$jigdoErr" -ge 2; then
log " Error merging data from local filesystem"
exitCode=1
rm -f "image" "template"
return 0
fi
# First try to download all files using the first URL in the
# print-missing-all list. If any files remain missing, add another
# pass, this time try to download the missing files using the 2nd
# URL, and so on.
noMorePasses=$localMirror
for pass in x xx xxx xxxx xxxxx xxxxxx xxxxxxx xxxxxxxx; do
if $havePMA; then
$jigdoFile print-missing-all $ijtOpts $jigdoOpts $uriOpts \
| egrep -i '^(http:|ftp:|$)' >"list"
else
# Quick hack until jigdo-port supports print-missing-all
$jigdoFile print-missing $ijtOpts $jigdoOpts $uriOpts \
| egrep -i '^(http:|ftp:|$)' \
| sed -n '/./{p;s/^.*$//;p;}' >"list"
fi
missingCount=`egrep '^$' <"list" | wc -l | sed -e 's/ *//g'`
if test "$pass" = "x"; then
if $localMirror; then true; else missingCount=0; fi
fi
if test "$missingCount" -gt "$maxMissing"; then
log " Too many files ($missingCount) missing in local mirror"
exitCode=1
rm -f "list" "image" "template"
return 0
fi
# Accumulate URLs in $@, pass them to fetchAndMerge in batches
set --
count=""
while read url; do
count="x$count"
if test "$url" = ""; then count=""; continue; fi
if test "$count" != "$pass"; then continue; fi
if $noMorePasses; then
log " $missingCount parts still missing from image"
fi
noMorePasses=false
set -- "$@" "$url"
if test "$#" -ge "$filesPerFetch"; then
if fetchAndMerge "$@"; then true; else
set --; noMorePasses=true
fi
set --
fi
done <"list"
if test "$#" -ge 1; then
if fetchAndMerge "$@"; then true; else break; fi
fi
if $noMorePasses; then break; fi
if test -r "image"; then break; fi
noMorePasses=true
done
if test -r "image"; then
# Finished - verify checksum
if $jigdoFile verify $ijtOpts; then
log " Image checksum is correct, moving image into place"
mkdir -p "$imageDir/$dirName"
mv "image" "$imageDir/$dirName/$image"
else
log " Error - image checksum mismatch"
exitCode=1
fi
else
log " Image creation failed, list of missing files follows"
$jigdoFile print-missing $ijtOpts $jigdoOpts $uriOpts
exitCode=1
fi
rm -f "list" "image" "image.tmp" "template"
return 0
}
#______________________________________________________________________
# Given URLs, fetch them into $tmpDir, then merge them into image
fetchAndMerge() {
(mkdir "$tmpDir/files"; cd "$tmpDir/files"; fetch "$@")
# Merge into the image
find "$tmpDir/files" -type f \
| $jigdoFile make-image $ijtOpts --files-from=-
jigdoErr="$?"
if test "$jigdoErr" -ge 2; then
exitCode=1
exit 1
fi
# Delete tmpDir, to avoid taking up more space than necessary
rm -rf "$tmpDir/files"
}
#______________________________________________________________________
sectionEnd() {
if test "$section" = "[Image]" -a "$image" \
-a "$templateURI"; then
log " Image \`$image', template \`$templateURI'"
set -- `echo "$dirName/$image" | egrep -- "$include" | egrep -v -- "$exclude"`
if test "$#" -eq 0; then
log " \`\$imageDir/$dirName/$image' excluded by \$include/\$exclude"
return
fi
if test -f "$imageDir/$dirName/$image"; then
if test "$jigdoDir/$jigdo" -nt "$imageDir/$dirName/$image";then
log " jigdo is newer - updating \`\$imageDir/$dirName/$image'"
# Remove outdated image *immediately*, even in case
# the subsequent attempt to regenerate it fails
rm -f "$imageDir/$dirName/$image"
makeImage
else
log " \`\$imageDir/$dirName/$image' is up to date"
test "$maxAge" && touch "$imageDir/$dirName/$image"
fi
else
log " Attempting to create \`\$imageDir/$dirName/$image'"
makeImage
fi
fi
}
#______________________________________________________________________
if test "$logfile"; then
true >"$logfile"
exec >>"$logfile"
exec 2>>"$logfile"
fi
log() { printf "%s: %s\n" "`date +'%Y-%m-%d %H:%M:%S'`" "$1"; }
#________________________________________
log "imageDir: $imageDir"
log "jigdoDir: $jigdoDir"
log "templateDir: $templateDir"
# Remove slashes from dir names
jigdoDir=${jigdoDir%/}
imageDir=${imageDir%/}
templateDir=${templateDir%/}
tmpDir=${tmpDir%/}
debianMirror=${debianMirror%/}
nonusMirror=${nonusMirror%/}
uriOpts="--uri Debian='$debianMirror/' --uri Non-US='$nonusMirror/'"
ijtOpts="--image=image --jigdo=jigdo --template=template"
# Is the main mirror on the local disc?
case "$debianMirror $nonusMirror" in
"file:"*" file:"*|/*" /"*) localMirror=true;;
*) localMirror=false;;
esac
if test -z "$havePMA"; then havePMA=true; fi
if test -z "$templateDir"; then templateDir="$jigdoDir"; fi
exitCode=0
mkdir -p "$tmpDir" || true
cd "$tmpDir"
#________________________________________
find "$jigdoDir" -name "*.jigdo" \
| sed -e "sÿ^$jigdoDir/ÿÿ" \
| while read jigdo; do
log "Found \`\$jigdoDir/$jigdo'"
dirName=`dirname "$jigdo"`
if zcat "$jigdoDir/$jigdo" >"jigdo" 2>"/dev/null"; then
true
elif test -f "$jigdoDir/$jigdo"; then
rm -f "jigdo" "template"
ln -s "$jigdoDir/$jigdo" "jigdo"
else
log " jigdo file not present/could not be unpacked - ignored"
exitCode=1
continue
fi
# Parse jigdo file, look for images
section=""
while read REPLY; do
set -- `echo "$REPLY" | sed -e 's/^ *\[ *\([^ ]*\) *\] *$/[\1]/; s/ *= */ /; s/['\''"$]//g'`
case "$1" in
"["*"]")
sectionEnd
unset image templateURI templateMD5 shortInfo info
section="$1";;
Filename) image="$2";;
Template) templateURI="$2";;
Template-MD5Sum) templateMD5="$2";;
ShortInfo) shift; shortInfo="$*";;
Info) shift; info="$*";;
esac
done <"jigdo"
sectionEnd
rm -f "jigdo"
done
#________________________________________
if cd "$imageDir"; then
if test "$maxAge"; then
log "Expiring images older than $maxAge days"
find . -type f -mtime +"$maxAge" \
| while read file; do
log " Deleting \`\$imageDir/${file#./}'"
rm -f "$file"
done
# Remove empty directories
find "$imageDir" -depth -mindepth 1 -type d -empty \
-exec rmdir '{}' ';'
fi
fi
#________________________________________
log "Exit $exitCode"
exit $exitCode
Attachment:
pgpfGjaRG2NeC.pgp
Description: PGP signature