[PATCH] Support for alternative protocols
Hi guys,
I'm working at the EDF Center for High Performance Computing (France).
We use
debian-live to deploy the operating system of compute nodes.
The particularity of this task is that we need to deploy thousands of
nodes in
an effective and scalable manner in a few minutes.
For this reason we have decided to patch the live-boot package, to
extend the
functionality of the FETCH method.
The main issue is that the script interpreting the FETCH parameter is
not
modular.
So we had to patch it to add new download methods. Do you think it would
be
possible to change that in order to have one file per FETCH method? If
so, we
will not need to patch anything and will just drop files in the correct
places.
For the deployment, we tested two protocols:
1) P2P: in this case we pass to the kernel the FETCH parameter
containing the
location of a torrent file with the information to download the squashfs
image.
2) 9P: in this case we use the xget binary (not present in Debian) and
pass to
FETCH a URL that directly contains the name of the squashfs image.
For more information on the operation of xget you can consult the
following
paper:
http://permalink.lanl.gov/object/tr?what=info:lanl-repo/lareport/LA-UR-08-06574
The xget utility is not packaged for Debian yet. (though we did package
it
internally to use it). For now, the "xget" name conflicts with other
utilities
like the AIX command or with the command-line tool that downloads images
from an
XNAT server. So it might need a renaming before being able to upload it
in
Debian.
Best regards
Antonio J. Russo
################################################################################
--- /usr/share/initramfs-tools/hooks/live.orig 2014-10-19
19:28:01.782993679 +0200
+++ /usr/share/initramfs-tools/hooks/live 2014-10-19 19:31:34.942864040
+0200
@@ -185,6 +185,20 @@
copy_exec /usr/bin/wget /bin
fi
+# Program: ctorrent
+if [ -x /usr/bin/ctorrent ]
+then
+ [ "${QUIET}" ] || echo -n " ctorrent"
+ copy_exec /usr/bin/ctorrent /bin
+fi
+
+# Program: 9pget
+if [ -x /usr/sbin/9pget ]
+then
+ [ "${QUIET}" ] || echo -n " 9pget"
+ copy_exec /usr/sbin/9pget /bin
+fi
+
# Program: blockdev
if [ -x /sbin/blockdev ]
then
################################################################################
--- 9990-mount-http.sh.orig 2014-10-19 19:17:24.203345786 +0200
+++ 9990-mount-http.sh 2014-10-15 13:29:29.251197233 +0200
@@ -15,7 +15,7 @@
if [ -n "$url" ]
then
case "${extension}" in
- iso|squashfs|tgz|tar)
+ iso|squashfs|tgz|tar|torrent)
if [ "${extension}" = "iso" ]
then
mkdir -p "${alt_mountpoint}"
@@ -35,11 +35,51 @@
log_begin_msg "Trying tftp -g -b 10240 -r $rfile -l
${dest}/$lfile $ip"
tftp -g -b 10240 -r $rfile -l ${dest}/$lfile $ip
;;
-
+ 9pget*)
+
ip="$(dirname $url | sed -e 's|9pget://||g' -e 's|/.*$||g')"
+
server="$(echo $ip | awk -F ':' '{print $1}')"
+
isport="$(echo $ip | awk -F ':' '{print $2}')"
+
port=${isport:=20004}
+
rfile="$(echo $url | sed -e "s|9pget://$ip/||g")"
+ fpasswd="/etc/passwd"
+ fgroup="/etc/group"
+ # Dirty hack to prevent segmentation fault #
+ if [ ! -e $fpasswd ]
+ then
+ echo 'root:x:0:0:root:/root:/bin/bash' > $fpasswd
+ fi
+ if [ ! -e $fgroup ]
+ then
+ echo
'root:x:0:' > $fgroup
+ fi
+ # End #
+
log_begin_msg "Trying 9pget -n ${server} -p ${port} -s ${rfile} ${dest}"
+
/bin/9pget -n ${server} -p ${port} -s ${rfile} ${dest}
+ rm -f $fpasswd $fgroup
+ ;;
+
+ http*torrent)
+
torrentfile="/tmp/$(basename ${url})"
+
log_begin_msg "Trying wget ${url} -O ${torrentfile}"
+ wget
"${url}" -O "${torrentfile}"
+
imagefile="${dest}/`/bin/ctorrent -x ${torrentfile} | sed -n "s/<.*>
\+\([^ ]*\).*/\1/p"`"
+
/bin/ctorrent ${torrentfile} -s ${imagefile} &
+
pidofctorrent="$(pidof ctorrent)"
+
cmdexit=1
+ sleep 1
+ while [
${cmdexit} -ne 0 ]
+ do
+ sleep
$(tr -cd 1-9 </dev/urandom | head -c 1)
+
/bin/ctorrent -c ${torrentfile} -s ${imagefile} | grep -q "100%"
+
cmdexit=${?}
+ done
+ kill
-HUP ${pidofctorrent}
+ ;;
+
*)
log_begin_msg "Trying wget ${url} -O ${dest}/$(basename
${url})"
wget "${url}" -O "${dest}/$(basename ${url})"
- ;;
+ ;;
esac
else
log_begin_msg "Trying to mount ${url} on ${dest}/$(basename
${url})"
################################################################################
Reply to: