[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Proto-type britney test suite

On 2011-10-18 14:56, Mehdi Dogguy wrote:
> On 15/10/2011 12:20, Niels Thykier wrote:
>> The primary issue with this suite is that you have to hand-craft *all*
>>  the data needed, which is a pain.  I strongly suspect that the long 
>> term implications of this is that tests will end up being very hard to
>>  understand and that people will generally avoid writing tests.
> I guess that's why you added "description" files for (almost) each test?
> I think it is enough to not forget why we added $some test.

Yes, at least for the more complicated ones.  Still I do not like the
amount of copy-waste(-errors) I have do(/fix) for each test, but it does
sort of work... :)

>> The "second" problem is that the test-suite only checks the heidi 
>> result, so it is cannot tell the difference between "pkg with rc bugs"
>>  and "migrating pkgX would break pkgY".
> But we do have (thanks to you) minimal tests for those two situations. So,
> somehow, they are distinguishable. But, I agree it could be a problem with
> more complicated tests.
>> For now, feel free to give the test suite a spin.  Either clone it from
>> [1] or run it from franck [2].  If you clone it, have a look at the
>> README (there are no "-h" or "--help").
> I didn't find time to play with it enough yet but I find this project very
> nice. It helps (some of) us to understand better what britney does in some
> specific situations, serves as a regression test base and tells new briney
> implementations what they should implement (at least).

Indeed, :)

> I have some minor patches:
> #1) fixes a typo.
> #2) makes run-single-test exit with a non zero value when test fails
> (the change is not pretty but kinda wfm™) Ideally, run-single-test should
> also tell us why the test failed (maybe by `cat diff`?).


> #) a bug in britney fixed by Adam recently (that requires disabling
> compatible mode). But the test passes using the buggy britney… so maybe I
> did something wrong.
> Kind regards,

And I applied your corrected version.  Thanks for the contributions :)


Reply to: