Re: dpkg testsuite (again)

[Esteban Manchado Velázquez <zoso@demiurgo.org>]

Hi,

On Thu, Jun 09, 2005 at 04:59:31AM +0100, Scott James Remnant wrote:
> [...]
> My main concerns so far with test suites is that the suite be:
> 
> 1) easy for users and casual developers to run, and be able to quickly
>    identify failed tests and use them to fix the code.
> 
> 2) sufficiently easy to use to encourage developers to write tests with
>    every code modification they make, and even with every bug fix.
> 
> 3) not require exotic or unusual support platforms or languages.
> 
> I've not yet come across an off-the-shelf suite that manages this for C;
> perhaps I'm a bit spoiled by pyunit and doctest, but I want something
> approaching that level of usability.

   I guess you're talking here mostly about unit tests, what about functional
tests?

> DejaGNU in particular doesn't seem to manage to satisfy any of these
> three concerns, the output of the suite is hard to use, the tests and
> especially C unit tests are hard to write and it requires TCL which is
> so widely incompatible with itself and no longer anywhere approaching a
> usual support platform.

   My idea is _not_ using DejaGNU "directly", but using it as a "base" to
build our testing framework. I really _don't know_ if DejaGNU gives us enough
to base the framework on it, or if it would be better to simply start fresh.

   Have you seen my current work? To add a dpkg-parsechangelog test, for
example, you simply drop two files in testsuite/dpkg-parsechangelog:
xxx.input and xxx.expected, where xxx is usually a number. The problem is, of
course, adding new "types" of tests, in which case you do have to program in
Tcl :-(

   About the C unit tests, currently (with my toy testsuite) you can simply
drop some *.c file with your new tests, and add it to the PROGRAMS variable in
the Makefile. Moreover, it's trivial, with a simple naming convention, to
remove the second step. The actual unit tests could be simpler to write (they
currently use the "check" library), but I guess we can do some simple
code-generation to remove the repetitive chunks. Do you have an idea of how
would you like to make them? pyunit-like, other?

   The output of the DejaGNU suite is too verbose for me, but perhaps not
particularly difficult to parse. I _thought_ that using DejaGNU for this would
let us use some testing front-end written by some other person, _but_ I really
don't know how many people use DejaGNU _and_ has written extensions, and I
don't know if it's better using DejaGNU and being able to use third-party
extensions, or not using it and being able to program in some non-Tcl language
:-)

   About the Tcl incompatibilities, I really don't know what you mean, my
experience with Tcl is almost limited to this experiment... care to explain?

   BTW, what platforms/languages do you consider non-exotic and non-unusual?
Ruby, Haskell, Python, Shell, Perl, OCaml?

> That being said, I did have some interesting discussions with tridge at
> LCA about how they test the samba suite.

   Any pointers or ideas?

-- 
Esteban Manchado Velázquez <zoso@demiurgo.org>
EuropeSwPatentFree - http://EuropeSwPatentFree.hispalinux.es
Help spread it through the Net in signatures, webpages, whatever!