Proto-type britney test suite

[Niels Thykier <niels@thykier.net>]

Hi

I already mentioned this over IRC, but I have written a proto-type test
suite for britney.  While this is a "good thing", it sounds a lot better
than it actually is.

The primary issue with this suite is that you have to hand-craft *all*
the data needed, which is a pain.  I strongly suspect that the long term
implications of this is that tests will end up being very hard to
understand and that people will generally avoid writing tests.

The "second" problem is that the test-suite only checks the heidi
result, so it is cannot tell the difference between "pkg with rc bugs"
and "migrating pkgX would break pkgY".


To fix the primary issue, I am considering to create a small language to
describe various parts of the data (packages, their relations, their RC
bugs, etc) and then generate the "real" data from that.  I am not quite
ready to start on that yet, so I will follow up on this "soon".

For now, feel free to give the test suite a spin.  Either clone it from
[1] or run it from franck [2].  If you clone it, have a look at the
README (there are no "-h" or "--help").

In case you want to debug a test, you can read the results from the
"test-out/$test" dir.  This will be the state after the britney run.
All of britney's stdout output will be available in
"test-out/$test/log.txt" and the conf file will be
"test-out/$test/britney.conf".
  The expected testing state is "test-out/$test/expected" (it is a
"simplified" heidi file) and is compared to the HeidiResult file.  The
comparision is not affected by ordering of those files.

~Niels

[1]
ssh://release.debian.org/home/nthykier/britney-tests.git
http://release.debian.org/~nthykier/britney-tests.git/

Have a look at the README.

[2]
$ ~nthykier/bin/run-britney-tests \
   /srv/release.debian.org/britney/code/b2/britney.py \
   ~nthykier/britney-tests/t \
   test-out
Running basic-with-rc-reduction...done
Running basic-uninstallable...done
Running basic-with-rc...done
Running basic...done

Summery:
Ran 4 tests
Failed tests: 0

Note: The last argument ("test-out") must not exist and must be "creatable".