Bug#829725: debootstrap: cannot bootstrap from repositories without InRelease files when --debian-installer set

[David McBride <dwm37@cam.ac.uk>]

Package: debootstrap
Version: 1.0.81
Severity: normal

I am using debootstrap in my own minimal system bootstrapper, and am making use
of the progress information reported to FH 3 as enabled by the
--debian-installer command-line flag.

Unfortunately, using this flag causes the behaviour of debootstrap to change,
specifically in its handling of InRelease / Release files.

During normal operation, debootstrap will first attempt to fetch an InRelease
file from the repository; if this is unavailable, i.e. the fetch fails with
404, then debootstrap will normally fall back to fetching a Release file
instead.  Indeed, this is the behaviour of debootstrap if --debian-installer is
not passed on the command-line.

debootstrap uses the wgetprogress() function to fetch these URLs.  It reads:

wgetprogress () {
        [ ! "$VERBOSE" ] && QSWITCH="-q"
        local ret=0
        if [ "$USE_DEBIANINSTALLER_INTERACTION" ] && [ "$PROGRESS_NEXT" ]; then
                wget "$@" 2>&1 >/dev/null | $PKGDETAILS "WGET%" $PROGRESS_NOW
$PROGRESS_NEXT $PROGRESS_END >&3
                ret=$?
        else
                wget $QSWITCH "$@"
                ret=$?
        fi
        return $ret
}

When the --debian-installer command-line flag is set, the first path of the if
branch will be taken - running the output of wget through a pipe, so that the
$PKGDETAILS command can be used to parse progress information provided by wget
and report it to FH 3 in a format usable by debian-installer.

However, if the URL passed to wget returns 404, then while the wget command
will fail, the $PKGDETAILS command, and thus the pipeline as a whole, does not,
and the wgetprocess () function thus erroneously returns success.

As a consequence, debootstrap does not fall back to fetching and using a
Release file as it should in this case, and the bootstrapping attempt as a
whole fails.

Because this is a POSIX shell-script, I don't believe there is a
straightforward mechanism to fetch the exit status of the wget command when it
is part of a pipeline.  (See: http://cfajohnson.com/shell/cus-faq-2.html#Q11).
It might be possible to enhance the code invoked by $PKGDETAILS to return a
fatal exit status if it does not definitely see a successful file retrieval?

A more direct work-around is to modify this function to unconditionally use the
second code path that does not attempt to invoke wget as part of a pipeline.
However, this does mean that you lose out on intra-file download progress
reporting.



-- System Information:
Debian Release: stretch/sid
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'unstable'), (500, 'stable'), (500, 'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.6.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages debootstrap depends on:
ii  wget  1.18-1

Versions of packages debootstrap recommends:
ii  debian-archive-keyring  2014.3
ii  gnupg                   1.4.20-6

debootstrap suggests no packages.

-- no debconf information