Bug#779228: Service may require more disk space
Control: retitle -1 lintian.d.o: disk space probably insufficient for scheduler
Hi,
> On lindsay.d.o I found it to be using 12GB before dying with "No more
> space left on device".
On archive-wide runs locally, the architectures amd64, all, and source
from unstable alone consume more than 50 GB of disk space. That is
probably because disk intensive runs tend to take a long time. They
bunch up.
I used a parallel scheduler that processes groups in alphabetic order
(please see below). Some large groups are also near each other
alphabetically (e.g. gcc-*) and may therefore run at the same time.
> Lintian will process all packages built from the same source/version
> together. This currently implies that all related packages must be
> unpacked fully at the same time. Unfortunately, for some package
> groups this can become extremely excessive.
Based on the size of processing groups in the archive (please have a
look at kicad-packages3d, for example) I could not detect excessive
disk consumption. While my scheduler is totally different from
lintian.d.o (which may or may not run in parallel), I believe there
was sufficient evidence to retitle the bug.
According to information obtained from DSA, the lintian.d.o service
recently had 32 GB of disk space, of which 12 GB appeared taken. (This
is from IRC; I do not have access to that service.) The available
space is probably insufficient.
My scheduler is included below. It uses an experimental branch for
JSON output, but any suitable output format can be used.
Kind regards
Felix Lechner
* * *
#!/usr/bin/bash
lintian="$1"
mirrorbase="$2"
resultsdir="$3"
find "$mirrorbase/pool/main/" -mindepth 2 -maxdepth 2 -type d | sort > sources
numsources=`wc -l sources`
echo "Found $numsources."
mkdir -p $resultsdir || true
command="$lintian --display-info --display-experimental --pedantic -L
+classification --show-overrides --exp-output format=json"
cat sources | xargs -L 1 -P 8 bash -c "directory=\$0; group=\`basename
\$directory\`; results=$resultsdir/\$group.json; if [ ! -e \$results ]
; then files=\`find \$directory -name '*.dsc' -o -name '*_amd64.deb'
-o -name '*_all.deb' | tr '\\\n' ' '\`; $command \$files > \$results
|| true ; fi"
times
Reply to: