We are currently getting build failures on one s390 buildd because it has an incomplete mirror in its /etc/apt/sources.list (only has unstable, not testing). When d-i is uploaded, that mirror is added in sources.list.udeb, but with s/unstable/lenny/, which means that apt-get update fails and thus the whole build fails. Attached a patch that checks the validity of each mirror before it is added. Assumption is that the buildd will have at least one source that is complete (which we're told is the case). As there are undoubtedly alternatives as to exactly how to check the validity of a mirror, I'd appreciate comments: - currently only checks network mirrors (deb ftp/http) - check uses HEAD command from libwww-perl (added to build-deps) to minimize network traffic - check tests that $mirror/dists/$suite/main/debian-installer/ exists, i.e. it tests that the suite/codename is supported and that the D-I section is present For now I've kept as close as possible to current code, but the script could possibly be extended: - maybe add a known reliable fallback mirror (such as ftp.nl.d.o) if no mirrors are valid - could maybe be used to automatically add *-proposed-updates for uploads to (old)stable or testing Cheers, FJP
commit cb747b775f1c5a9d964669dd9884b94eb5e81fc2
Author: Frans Pop <fjp@debian.org>
Date: Sun May 25 22:07:57 2008 +0200
Test sources before adding them to sources.list.udeb
We currently assume that all sources in the system's /etc/apt/sources.list
are valid for the target suite, which may not be true if it contains an
incomplete mirror. Example is the lxdebian s390 build which lists a
mirror that does not have testing and thus fails.
Solve this by testing that a mirror actually contains the main/d-i section
for the target suite.
diff --git a/installer/build/Makefile b/installer/build/Makefile
index 1718a81..4b29f8c 100644
--- a/installer/build/Makefile
+++ b/installer/build/Makefile
@@ -563,13 +563,7 @@ sources.list.udeb:
if [ "$(MIRROR)x" != "x" ]; then \
echo "deb $(MIRROR) $(USE_UDEBS_FROM) $(UDEB_COMPONENTS)"; \
else \
- grep '^deb[ \t]' $(SYSTEM_SOURCES_LIST) \
- |grep -v '^deb[ \t]cdrom:' \
- |grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' \
- |grep '[ \t]main' \
- |awk '{print $$1 " " $$2}' \
- |sed "s,/* *$$, $(USE_UDEBS_FROM) $(UDEB_COMPONENTS)," \
- |sed "s,^deb file,deb copy," \
+ gen-sources.list.udeb "$(SYSTEM_SOURCES_LIST)" $(USE_UDEBS_FROM) $(UDEB_COMPONENTS) \
|perl -ne 'print unless $$seen{$$_}; $$seen{$$_}=1' ; \
fi) > $@
diff --git a/installer/build/util/gen-sources.list.udeb b/installer/build/util/gen-sources.list.udeb
new file mode 100755
index 0000000..ba2dfa2
--- /dev/null
+++ b/installer/build/util/gen-sources.list.udeb
@@ -0,0 +1,32 @@
+#! /bin/sh
+set -e
+
+OLDIFS="$IFS"
+NL="
+"
+
+SOURCES_LIST=$1
+SUITE=$2
+UDEB_COMPONENTS=$3
+
+MIRRORS="$(grep '^deb[ \t]' $SOURCES_LIST | \
+ grep -v '^deb[ \t]cdrom:' | \
+ grep -v '\(security.debian.org\|volatile.debian.\(net\|org\)\)' | \
+ grep '[ \t]main' | \
+ awk '{print $1 " " $2}' | \
+ sed 's,^deb file,deb copy,' | \
+ sed 's,/* *$,,')"
+
+IFS=$NL
+for mirror in $MIRRORS; do
+ IFS="$OLDIFS"
+
+ tmirror="$(echo $mirror | sed "s/^deb //")"
+ if echo "$mirror" | grep -Eq "^deb (f|ht)tp" && \
+ ! HEAD $tmirror/dists/$SUITE/$UDEB_COMPONENTS/ >/dev/null 2>&1; then
+ echo "WARNING: mirror '$tmirror' appears to be invalid; skipping" >&2
+ continue
+ fi
+
+ echo "$mirror $SUITE $UDEB_COMPONENTS"
+done
diff --git a/installer/debian/control b/installer/debian/control
index 88dae28..d53f23e 100644
--- a/installer/debian/control
+++ b/installer/debian/control
@@ -7,7 +7,7 @@ Standards-Version: 3.6.2
Vcs-Svn: svn://svn.debian.org/d-i/trunk/installer
Build-Conflicts: libnewt-pic [mipsel]
# NOTE: Do not edit the next line by hand. See comment below.
-Build-Depends: debhelper (>= 4), apt, apt-utils, gnupg, debian-archive-keyring (>= 2006.11.22), dpkg (>= 1.13.9), dctrl-tools, bc, debiandoc-sgml, xsltproc, docbook-xsl, libbogl-dev [!kfreebsd-i386 !kfreebsd-amd64 !hurd-i386], glibc-pic, libslang2-pic (>= 2.0.6-4), libnewt-pic [!mipsel], libnewt-dev [mipsel], libgcc1 [i386 amd64], cramfsprogs [powerpc ppc64 ia64 mips mipsel arm armeb armel], genext2fs (>= 1.3-7.1), e2fsprogs, mklibs (>= 0.1.25), mklibs-copy [mips mipsel], genisoimage [!s390 !s390x], genromfs [sparc], hfsutils [powerpc ppc64], dosfstools [i386 ia64 m68k amd64], cpio, devio [arm armeb armel], slugimage (>= 0.10+r58-6) [arm armeb armel], nwutil [arm], syslinux (>= 2.11-0.1) [i386 amd64], palo [hppa], elilo [ia64], yaboot [powerpc ppc64], aboot (>= 0.9b-2) [alpha], silo [sparc], sparc-utils [sparc], genisovh [mips], delo [mipsel], tip22 [mips], colo (>= 1.21-1) [mipsel], sibyl [mips mipsel], atari-bootstrap [m68k], vmelilo [m68k], m68k-vme-tftplilo [m68k], amiboot [m68k], emile [m68k], emile-bootblocks [m68k], apex-nslu2 [arm armeb armel], tofrodos [i386 amd64 kfreebsd-i386 kfreebsd-amd64], mtools (>= 3.9.9-1) [i386 ia64 m68k amd64 kfreebsd-i386 kfreebsd-amd64], module-init-tools [i386 arm armeb armel amd64 alpha hppa ia64 m68k mips mipsel powerpc ppc64 s390 sparc], bf-utf-source [!s390 !s390x], upx-ucl (>= 3) [i386], mkvmlinuz [powerpc ppc64], openssl [arm armel], win32-loader [i386 amd64 kfreebsd-i386 kfreebsd-amd64]
+Build-Depends: debhelper (>= 4), apt, apt-utils, gnupg, debian-archive-keyring (>= 2006.11.22), dpkg (>= 1.13.9), dctrl-tools, libwww-perl, bc, debiandoc-sgml, xsltproc, docbook-xsl, libbogl-dev [!kfreebsd-i386 !kfreebsd-amd64 !hurd-i386], glibc-pic, libslang2-pic (>= 2.0.6-4), libnewt-pic [!mipsel], libnewt-dev [mipsel], libgcc1 [i386 amd64], cramfsprogs [powerpc ppc64 ia64 mips mipsel arm armeb armel], genext2fs (>= 1.3-7.1), e2fsprogs, mklibs (>= 0.1.25), mklibs-copy [mips mipsel], genisoimage [!s390 !s390x], genromfs [sparc], hfsutils [powerpc ppc64], dosfstools [i386 ia64 m68k amd64], cpio, devio [arm armeb armel], slugimage (>= 0.10+r58-6) [arm armeb armel], nwutil [arm], syslinux (>= 2.11-0.1) [i386 amd64], palo [hppa], elilo [ia64], yaboot [powerpc ppc64], aboot (>= 0.9b-2) [alpha], silo [sparc], sparc-utils [sparc], genisovh [mips], delo [mipsel], tip22 [mips], colo (>= 1.21-1) [mipsel], sibyl [mips mipsel], atari-bootstrap [m68k], vmelilo [m68k], m68k-vme-tftplilo [m68k], amiboot [m68k], emile [m68k], emile-bootblocks [m68k], apex-nslu2 [arm armeb armel], tofrodos [i386 amd64 kfreebsd-i386 kfreebsd-amd64], mtools (>= 3.9.9-1) [i386 ia64 m68k amd64 kfreebsd-i386 kfreebsd-amd64], module-init-tools [i386 arm armeb armel amd64 alpha hppa ia64 m68k mips mipsel powerpc ppc64 s390 sparc], bf-utf-source [!s390 !s390x], upx-ucl (>= 3) [i386], mkvmlinuz [powerpc ppc64], openssl [arm armel], win32-loader [i386 amd64 kfreebsd-i386 kfreebsd-amd64]
# This package has the worst Build-Depends in Debian, so it deserves some
# explanation. Note that this comment can also be used to generate a
# Build-Depends line, by running the debian/genbuilddeps program.
@@ -31,6 +31,8 @@ Build-Depends: debhelper (>= 4), apt, apt-utils, gnupg, debian-archive-keyring (
# version.
# - dctrl-tools
# Various manipulations of the udeb Packages files.
+# - libwww-perl
+# Used to test validity of mirrors.
# - bc
# Used for some image size calculations.
# - debiandoc-sgml
Attachment:
signature.asc
Description: This is a digitally signed message part.