[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Proto-type britney test suite


I already mentioned this over IRC, but I have written a proto-type test
suite for britney.  While this is a "good thing", it sounds a lot better
than it actually is.

The primary issue with this suite is that you have to hand-craft *all*
the data needed, which is a pain.  I strongly suspect that the long term
implications of this is that tests will end up being very hard to
understand and that people will generally avoid writing tests.

The "second" problem is that the test-suite only checks the heidi
result, so it is cannot tell the difference between "pkg with rc bugs"
and "migrating pkgX would break pkgY".

To fix the primary issue, I am considering to create a small language to
describe various parts of the data (packages, their relations, their RC
bugs, etc) and then generate the "real" data from that.  I am not quite
ready to start on that yet, so I will follow up on this "soon".

For now, feel free to give the test suite a spin.  Either clone it from
[1] or run it from franck [2].  If you clone it, have a look at the
README (there are no "-h" or "--help").

In case you want to debug a test, you can read the results from the
"test-out/$test" dir.  This will be the state after the britney run.
All of britney's stdout output will be available in
"test-out/$test/log.txt" and the conf file will be
  The expected testing state is "test-out/$test/expected" (it is a
"simplified" heidi file) and is compared to the HeidiResult file.  The
comparision is not affected by ordering of those files.



Have a look at the README.

$ ~nthykier/bin/run-britney-tests \
   /srv/release.debian.org/britney/code/b2/britney.py \
   ~nthykier/britney-tests/t \
Running basic-with-rc-reduction...done
Running basic-uninstallable...done
Running basic-with-rc...done
Running basic...done

Ran 4 tests
Failed tests: 0

Note: The last argument ("test-out") must not exist and must be "creatable".

Reply to: