Hacking ftpsync to get mirror sync working again
As you probably are aware, our ftp.skolelinux.org mirror is no longer
updating properly because we are out of disk space. This also block the
ISO build. The cause is that the mirror size just exceed the amount of
disk space we have available, because a lot of new packages have been
uploaded to Debian after the Jessie release, and old versions of
packages are kept on the mirrors for several months after they are
replaced by newer versions.
In an effort to try to get the ISO build working again, I had the idea
to drop packages we do not use in our ISO from the mirror. I compared
the set of pool/ directories on the mirror with the ones on the latest
Jessie USB build, and then checked the disk footprint of the mirror
directories not used to build the USB ISO. This is the list of those
exceeding 1 GiB of disk space at the moment:
1006860 /skolelinux/administrator/ftparchive/debian/pool/main/p/pcl
1040756 /skolelinux/administrator/ftparchive/debian/pool/non-free/f/frogatto-data
1052588 /skolelinux/administrator/ftparchive/debian/pool/main/m/mrpt
1052716 /skolelinux/administrator/ftparchive/debian/pool/main/e/eclipse
1052988 /skolelinux/administrator/ftparchive/debian/pool/main/w/wesnoth-1.8
1084164 /skolelinux/administrator/ftparchive/debian/pool/main/l/llvm-toolchain-3.4
1088240 /skolelinux/administrator/ftparchive/debian/pool/main/g/gcc-mingw-w64
1107644 /skolelinux/administrator/ftparchive/debian/pool/main/t/texlive-base
1129768 /skolelinux/administrator/ftparchive/debian/pool/main/o/openclipart2
1158088 /skolelinux/administrator/ftparchive/debian/pool/main/h/hedgewars
1192140 /skolelinux/administrator/ftparchive/debian/pool/main/g/gazebo
1216136 /skolelinux/administrator/ftparchive/debian/pool/main/s/supertuxkart
1226288 /skolelinux/administrator/ftparchive/debian/pool/main/c/cmake
1230644 /skolelinux/administrator/ftparchive/debian/pool/main/t/texlive-lang
1249544 /skolelinux/administrator/ftparchive/debian/pool/main/0/0ad
1275240 /skolelinux/administrator/ftparchive/debian/pool/main/w/widelands
1282328 /skolelinux/administrator/ftparchive/debian/pool/main/b/blender
1292276 /skolelinux/administrator/ftparchive/debian/pool/main/n/ns3
1388596 /skolelinux/administrator/ftparchive/debian/pool/non-free/n/nvidia-graphics-drivers
1462336 /skolelinux/administrator/ftparchive/debian/pool/main/e/ember-media
1485924 /skolelinux/administrator/ftparchive/debian/pool/main/l/llvm-toolchain-3.6
1496444 /skolelinux/administrator/ftparchive/debian/pool/main/o/openoffice.org
1573520 /skolelinux/administrator/ftparchive/debian/pool/contrib/v/virtualbox
1663592 /skolelinux/administrator/ftparchive/debian/pool/main/m/megaglest-data
1691948 /skolelinux/administrator/ftparchive/debian/pool/main/c/calligra
1719520 /skolelinux/administrator/ftparchive/debian/pool/main/g/gcc-5-cross
1804900 /skolelinux/administrator/ftparchive/debian/pool/non-free/s/sauerbraten-data
1847268 /skolelinux/administrator/ftparchive/debian/pool/main/w/wesnoth-1.12
1865036 /skolelinux/administrator/ftparchive/debian/pool/main/g/ghc
2083892 /skolelinux/administrator/ftparchive/debian/pool/main/g/gcc-snapshot
2180632 /skolelinux/administrator/ftparchive/debian/pool/non-free/r/redeclipse-data
2224080 /skolelinux/administrator/ftparchive/debian/pool/main/w/wesnoth-1.10
2458604 /skolelinux/administrator/ftparchive/debian/pool/main/o/openjdk-6
2563272 /skolelinux/administrator/ftparchive/debian/pool/main/q/qtcreator
2661724 /skolelinux/administrator/ftparchive/debian/pool/non-free/a/alien-arena-data
2713444 /skolelinux/administrator/ftparchive/debian/pool/main/n/nexuiz-data
2806688 /skolelinux/administrator/ftparchive/debian/pool/main/r/redeclipse-data
3813724 /skolelinux/administrator/ftparchive/debian/pool/main/g/gcc-5
3955796 /skolelinux/administrator/ftparchive/debian/pool/main/0/0ad-data
4267848 /skolelinux/administrator/ftparchive/debian/pool/main/i/icedove
4291168 /skolelinux/administrator/ftparchive/debian/pool/main/t/texlive-extra
5240384 /skolelinux/administrator/ftparchive/debian/pool/main/f/flightgear-data
6222912 /skolelinux/administrator/ftparchive/debian/pool/main/c/ceph
6339100 /skolelinux/administrator/ftparchive/debian/pool/main/w/webkit2gtk
7046284 /skolelinux/administrator/ftparchive/debian/pool/non-free/n/nvidia-cuda-toolkit
7580316 /skolelinux/administrator/ftparchive/debian/pool/main/l/linux-2.6
9491168 /skolelinux/administrator/ftparchive/debian/pool/main/c/chromium-browser
9505960 /skolelinux/administrator/ftparchive/debian/pool/main/libr/libreoffice
I suspect there are some false positives here caused by some packages
being fetched from the security or updates repositories (like
libreoffice), but also see several packages we do not include in our
ISO. I updated ~builder/etc/ftpsync.conf and added this list of
packages to exclude from the mirroring, and am testing a full mirror
sync now:
# Exclude a few packages we do not include in the ISO build, to get
# the build going again
EXCLUDE="--exclude pool/main/c/chromium-browser/ \
--exclude pool/main/c/ceph/ \
--exclude pool/main/f/flightgear-data/ \
--exclude pool/main/f/flightgear/ \
--exclude pool/main/0/0ad-data/ \
--exclude pool/main/0/0ad/ \
--exclude pool/main/i/icedove/"
If this work to get the mirror update going again, I'll update the list
of packages to drop from our mirror a bit. For example, I believe we
are using chromium-browser, but decided to ignore it in the first run as
it was one of the bigger directories.
The proper fix is to move the ISO build to a machine with more disk
space. :)
--
Happy hacking
Petter Reinholdtsen
Reply to: