Re: archive rebuilds, step 2

[Lucas Nussbaum <lucas@debian.org>]

Hi,

So, Step 2. I described it as:
    Scan the logs (that's done automatically), generate list of
    failures. Also upload the logs to some public place

At the end of Step 1, we have all the logs on EC2 in /tmp/logs.
In Step 2, we will:
- scan the logs to identify failed builds. For that, we will use
  cqa-scanlogs, which is installed on the master node, and available as
  part of the collab-qa-tools package, which source can be found on
  http://anonscm.debian.org/viewvc/collab-qa/collab-qa-tools/
  cqa-scanlogs parses build logs and extracts the interesting part
  automatically.
- move logs of failures to a public place. Currently, under
  http://people.debian.org/~lucas/logs/, but we need to change that.
- prepare a list of failures that will be used for Step 3, such as
  http://anonscm.debian.org/viewvc/collab-qa/archive-rebuilds/2013-05-09-unstable-amd64/failed.2013-05-09.txt?revision=2673&view=markup

That step is actually completely automatized in that script:
http://anonscm.debian.org/viewvc/collab-qa/archive-rebuilds/fetch-and-process-results-aws?revision=2276&view=markup
The problem is that everything is hardcoded.

Your tasks here are:
- build and install collab-qa-tools locally
- look at fetch-and-process-results-aws, understand what it does, do
  every step manually replacing steps you can't do as is (e.g. upload
  logs somewhere else)
- ideally, hack fetch-and-process-results-aws so that your own settings
  (different place to put logs) are available.

You will note that at the end of the script, we use a script called
"merge-results.rb". That script 'imports' a list of failures (+ bugs
filed) into the current list. That allows you to concentrate on the
unknown failures, and to auto-fill all known failures with the
corresponding bug numbers.

In Step 3, we will look at using the list of failures to file bugs.



Side task for Steps 1 and 2:
I am often asked to perform custom rebuilds. Typically, rebuild of
unstable with a custom GCC version. There's on my TODO list right now,
about rebuilding unstable with GCC 4.8. For that, Matthias Klose has
prepared at
   deb http://people.debian.org/~doko/tmp/tinstall ./
To run a custom rebuild, you need to specify a 'mode' to
generate-tasks-rebuild, e.g.:
./generate-tasks-rebuild --no-arch-all -m binary-only -m parallel -m gcc48-unstable -i gcc48 > tasks.gcc48.json
That mode is then added in the json description of tasks, so
process-task can use it to customize the way it calls sbuild, e.g.:
  if task['modes'].include?('gcc48-unstable')
    if not File::exists?('/tmp/change-gcc48-unstable')
      File::open('/tmp/change-gcc48-unstable', 'w') do |fd|
        fd.puts <<-EOF
#!/usr/bin/sudo bash
set -e -x
id
deb http://people.debian.org/~doko/tmp/gcc-4.8 ./ >> /etc/apt/sources.list
echo 'APT::Get::AllowUnauthenticated "1";' > /etc/apt/apt.conf.d/99allow-unauth
apt-get update
apt-get -y upgrade
EOF
      end
      system("chmod a+rx /tmp/change-gcc48-unstable")
    end
    sbuildopt += ' --chroot-setup-commands=/tmp/change-gcc48-unstable'
  end

Generally, when running such custom rebuilds, what's interesting is the
difference between a normal rebuild and the custom one. So usually, you do
something such as (example from README):
# gcc 4.8
./generate-tasks-rebuild --no-arch-all -m binary-only -m parallel -m gcc48-unstable -i gcc48 > tasks.gcc48.json
./generate-tasks-rebuild --no-arch-all -m binary-only -m parallel -i normal > tasks.normal.json
./merge-tasks tasks.gcc48.json tasks.normal.json > tasks.json

You task here is to hack process-task so that it can be used to run a custom
rebuild using doko packages, and to test that it works against a few packages.
(It might be that only the URL in process-task needs to be changed ; I'm not sure).

Then, move "normal" and "gcc48" logs to separate directories, use cqa-scanlogs
to list failures, and use 'compare-results.rb' to compare both list of
failures. Try to find one package that succeeds in the normal rebuild, but
fails in the gcc 4.8 one. (there's a list of bugs that should affect only gcc
4.8 filed by Matthias Klose at
http://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=ftbfs-gcc-4.8;users=debian-gcc@lists.debian.org)

Lucas