Re: Proto-type britney test suite
On 2011-10-18 14:56, Mehdi Dogguy wrote:
> On 15/10/2011 12:20, Niels Thykier wrote:
>>
>> The primary issue with this suite is that you have to hand-craft *all*
>> the data needed, which is a pain. I strongly suspect that the long
>> term implications of this is that tests will end up being very hard to
>> understand and that people will generally avoid writing tests.
>>
>
> I guess that's why you added "description" files for (almost) each test?
> I think it is enough to not forget why we added $some test.
>
Yes, at least for the more complicated ones. Still I do not like the
amount of copy-waste(-errors) I have do(/fix) for each test, but it does
sort of work... :)
>> The "second" problem is that the test-suite only checks the heidi
>> result, so it is cannot tell the difference between "pkg with rc bugs"
>> and "migrating pkgX would break pkgY".
>>
>
> But we do have (thanks to you) minimal tests for those two situations. So,
> somehow, they are distinguishable. But, I agree it could be a problem with
> more complicated tests.
>
>>
>> For now, feel free to give the test suite a spin. Either clone it from
>> [1] or run it from franck [2]. If you clone it, have a look at the
>> README (there are no "-h" or "--help").
>>
>
> I didn't find time to play with it enough yet but I find this project very
> nice. It helps (some of) us to understand better what britney does in some
> specific situations, serves as a regression test base and tells new briney
> implementations what they should implement (at least).
>
Indeed, :)
> I have some minor patches:
>
> #1) fixes a typo.
> #2) makes run-single-test exit with a non zero value when test fails
> (the change is not pretty but kinda wfm™) Ideally, run-single-test should
> also tell us why the test failed (maybe by `cat diff`?).
Applied
> #) a bug in britney fixed by Adam recently (that requires disabling
> compatible mode). But the test passes using the buggy britney… so maybe I
> did something wrong.
>
> Kind regards,
>
And I applied your corrected version. Thanks for the contributions :)
~Niels
Reply to: