Patch to tidy up wget use, and add return=4 when 404's are seen
Hi,
I've tested this briefly, and it seems to work OK, but failed
with a grub error (that I notice has been mentioned by other, so
I'm currently assuming that's not related to this patch)
First I've made preseed_fetch callable as fetch_url, so that it can
be used outside preseeding without treading on the contents of
/var/run/preseed.last_location
There's also something to remove existing target files because I've not
yet tested if the use of wget -c will play well with the way preseed_fetch
gets called.
Then we have the tweaks to file and floppy fetch methods as suggested
in #422088
For the http method, I've gone for a slightly more complicated approach
than suggested in the bug, in order to avoid the need for a tmp file --
this can probably stand some explanation:
RET=$({ { wget "$@" 2>&1 && echo 0 >&3 ;} | if grep -q 'server returned error 404:'; then echo 4; else echo 1; fi;} 3>&1 | head -1)
This line does the following:
run wget, and redirect it's STDERR to STDOUT so that we can grep for 404's
if the wget succeeds, echo zero to file handle 3
which then gets redirected back to stdout
then, depending on the result of the grep, echo 1 or 4
that means that at this point the possible outputs are:
success:
0
1
404 error:
4
other errors:
1
which the head -1 then renders into 0, 4 & 1 respectively ready to be
used as the return from wget_plus()
Then we have a bug fix, that would otherwise leave ftp proxy unset, if the
http method script is being use for ftp (as may happen, since it's linked)
Then we come to the while loop in the http method, which I'm pretty sure
was a cut&paste from the code in net-retriever, so I've nicked the wget
-c stuff from there while I was about it. I've also replaced the while
with a for, as it makes the code cleaner.
This code is something of a puzzle though, since as it stands in
net-retriever, the while loop is apparently pointless, given that each
conditional branch ends in a return. Should we even be doing retries
here I wonder?
I take it net-retriever itself gets called multiple times (by anna?) if
there is a partial download, since otherwise the wget -c is also dead code.
If that's the case, perhaps the for loop should go, and I should put
the retries into preseed_fetch, and have that call fetch_url.
Finally, I' ve made net-retriever depend on preseed-common, and stripped
out all the duplicated code.
Cheers, Phil.
=-=-=-=-=-=-=-=-=-=-
Index: preseed/preseed_fetch
===================================================================
--- preseed/preseed_fetch (revision 51327)
+++ preseed/preseed_fetch (working copy)
@@ -3,12 +3,18 @@
. /usr/share/debconf/confmodule
. /lib/preseed/preseed.sh
-src="$1"
+url="$1"
dst="$2"
-last=/var/run/preseed.last_location
+# check if we're preseed_fetch -- deal with relative URLs if so
+if expr "$0" : '\(.*/\|^\)preseed_fetch$' >/dev/null ; then
+ last=/var/run/preseed.last_location
+ url=$(make_absolute_url "$url" "$(test -r $last && cat $last)")
+ # Am suspicious that there are assumptions that we can overwrite
+ # in preseeding, so lets discard old files for now
+ [ -e $dst ] && rm $dst
+fi
-url=$(make_absolute_url "$src" "$(test -r $last && cat $last)")
proto=${url%%://*}
. /lib/preseed/fetch-methods/$proto
Index: preseed/debian/changelog
===================================================================
--- preseed/debian/changelog (revision 51327)
+++ preseed/debian/changelog (working copy)
@@ -1,3 +1,15 @@
+preseed (1.35) UNRELEASED; urgency=low
+
+ * fix bug in fetch_method/http that would ignore ftp proxy
+ * take inspiration from net-retriever about wget -c
+ * add wrapper to wget that allows it to diferentiate 404s
+ * modify all fetch methods to return 4 for missing files
+ * add fetch_url (a symlink to preseed_fetch) to allow for
+ it's use for downloading without interfering with the
+ preseed relative path magic
+
+ -- Philip Hands <phil@hands.com> Fri, 15 Feb 2008 16:41:15 +0000
+
preseed (1.34) unstable; urgency=low
* debconf-set-selections: Fix parser to again allow use of tabs
Index: preseed/debian/rules
===================================================================
--- preseed/debian/rules (revision 51327)
+++ preseed/debian/rules (working copy)
@@ -20,6 +20,7 @@
done
dh_link -p network-preseed lib/preseed/fetch-methods/http \
lib/preseed/fetch-methods/ftp
+ dh_link -p preseed-common bin/preseed_fetch bin/fetch_url
dh_compress
dh_fixperms
dh_installdeb
Index: preseed/fetch-methods/file
===================================================================
--- preseed/fetch-methods/file (revision 51327)
+++ preseed/fetch-methods/file (working copy)
@@ -1,6 +1,8 @@
protocol_fetch() {
FILE="${1#file://*}"
- if [ ! -e "$FILE" ] || ! cp "$FILE" $2; then
+ if [ ! -e "$FILE" ] ; then
+ return 4
+ elif ! cp "$FILE" $2; then
return 1
else
return 0
Index: preseed/fetch-methods/http
===================================================================
--- preseed/fetch-methods/http (revision 51327)
+++ preseed/fetch-methods/http (working copy)
@@ -1,19 +1,31 @@
protocol_fetch() {
local url="$1"
local file="$2"
- iters=0
+ wget_plus() { # wrapper for wget that returns 4 for 404 errors
+ RET=$({ { wget "$@" 2>&1 && echo 0 >&3 ;} | if grep -q 'server returned error 404:'; then echo 4; else echo 1; fi;} 3>&1 | head -1)
+ return $RET
+ }
+
# use the proxy for wgets (should speed things up)
- if db_get mirror/http/proxy; then
- export http_proxy="$RET"
+ if db_get mirror/$proto/proxy; then
+ export ${proto}_proxy="$RET"
fi
- while [ $iters -lt 3 ]; do
- # TODO add progress bar
- if wget -q "$url" -O "$file"; then
- return 0
+ for i in 1 2 3; do
+ if [ -e "$file" ]; then
+ # busybox wget can sometimes become confused while
+ # resuming, so if it fails, restart
+ if wget_plus -c "$url" -O "$file"; then
+ return 0
+ fi
fi
- iters=$(($iters + 1))
+
+ wget_plus "$url" -O "$file"
+
+ if [ 1 != "$?" ]; then
+ return $RET
+ fi
done
return 1
}
Index: preseed/fetch-methods/floppy
===================================================================
--- preseed/fetch-methods/floppy (revision 51327)
+++ preseed/fetch-methods/floppy (working copy)
@@ -4,7 +4,9 @@
mountfloppy || true
touch /var/run/preseed-usedfloppy
- if [ ! -e "$FILE" ] || ! cp "$FILE" $2; then
+ if [ ! -e "$FILE" ] ; then
+ return 4
+ elif ! cp "$FILE" $2; then
return 1
else
return 0
Index: preseed/README.preseed_fetch
===================================================================
--- preseed/README.preseed_fetch (revision 51327)
+++ preseed/README.preseed_fetch (working copy)
@@ -1,6 +1,10 @@
The preseed_fetch script takes a source and destination address, and
copies the specified file from the source to the destination.
+The relative path functionality is only active if the script is invoked
+using the name "preseed_fetch". This is to allow it to be linked to
+fetch_url for more general use outside preseeding.
+
The source can be a fully qualified URL, an absolute path, or a relative path.
If the source is not a full URL, the contents of /var/run/preseed.last_location
Index: net-retriever/debian/control
===================================================================
--- net-retriever/debian/control (revision 51327)
+++ net-retriever/debian/control (working copy)
@@ -10,7 +10,7 @@
Package: net-retriever
XC-Package-Type: udeb
Architecture: all
-Depends: ${misc:Depends}, choose-mirror, configured-network, di-utils (>= 1.16), gpgv-udeb, debian-archive-keyring-udeb
+Depends: ${misc:Depends}, choose-mirror, configured-network, di-utils (>= 1.16), gpgv-udeb, debian-archive-keyring-udeb, preseed-common
Provides: retriever
Description: Fetch modules from the Internet
This is a retriever that uses wget to fetch files over http or ftp.
Index: net-retriever/debian/changelog
===================================================================
--- net-retriever/debian/changelog (revision 51327)
+++ net-retriever/debian/changelog (working copy)
@@ -1,3 +1,10 @@
+net-retriever (1.20) UNRELEASED; urgency=low
+
+ * take all wget references out, and instead use fetch_url from
+ preseed-common
+
+ -- Philip Hands <phil@hands.com> Fri, 15 Feb 2008 16:53:35 +0000
+
net-retriever (1.19) unstable; urgency=low
[ Updated translations ]
Index: net-retriever/net-retriever
===================================================================
--- net-retriever/net-retriever (revision 51327)
+++ net-retriever/net-retriever (working copy)
@@ -16,36 +16,11 @@
hostname="$RET"
db_get mirror/$protocol/directory
directory="$RET"
-db_get mirror/$protocol/proxy
-proxy="$RET"
-if [ -n "$proxy" ]; then
- if [ "$protocol" = http ]; then
- export http_proxy="$proxy"
- elif [ "$protocol" = ftp ]; then
- export ftp_proxy="$proxy"
- fi
-fi
+
keyring=/usr/share/keyrings/archive.gpg
fetch() {
- url="${protocol}://${hostname}${directory}/$1"
- iters=0
- while [ $iters -lt 3 ]; do
- if [ ! -e "$2" ]; then
- wget -q "$url" -O "$2"
- return $?
- else
- # busybox wget can sometimes become confused while
- # resuming, so if it fails, restart
- if wget -c -q "$url" -O "$2"; then
- return 0
- else
- wget -q "$url" -O "$2"
- return $?
- fi
- fi
- iters=$(($iters + 1))
- done
+ fetch_url "${protocol}://${hostname}${directory}/$1" "$2"
}
checkmatch() {
Reply to: