Re: mercurial new test packages

[Antoine Beaupré <anarcat@orangeseeds.org>]

On 2018-06-29 03:41:15, Chris Lamb wrote:
> Antoine,
>
>> >> I am not sure why the test suite fails nor why the output varies from
>> >> one build to the next. Once a package is built, however, it passes the
>> >> test suite reliably.
> […]
>> Sure. I guess I see this from the perspective of "does the actual fix
>> work or not" as well. ;)
>
> Sorry to keep on about this but I still think we are talking past each
> other. You seem to be conflating and jumping between three separate
> concerns:
>
>  * A build that does not non-determistically fail in its testsuite (and
>    thus FTBFS randomly.).
>    
>  * Reliably detecting regressions ("introduce new…").
>  
>  * A bit-for-bit reproducible build - eg. your "test packages
>    unreproducible" note in data/dla-needed.txt when we both agreed that
>    is not a goal of LTS updates.
>    
> Can you please clarify exactly which are referring to in data/dla-
> needed.txt?

Sorry if I wasn't clear. By "unreproducible" I meant the first option
above, that is the build fails non-deterministically.

The reason I use the word "reproducible" here is because I specifically
think this is bound to the way the binary is build. And I think that
because when I *do* manage to build a binary (after a few attempts), it
reliably succeeds the test suite without problem.

So while reproducibility is not a goal of LTS updates per se, if they do
make a test suite fail, I think it's a worthwhile goal to fix
reproducibility of the package in some way. Or at least the test suite
should be fixed so it doesn't depend on constant output from the tested
commands, but I have not been successful in fixing it that way. There
*are* provisions to sort JSON arrays in the output, but those are not
currently used in the test suite. This is one of the places I'm thinking
of looking at next to fix this.

In the meantime, I postponed working on the package as I had to move on
to other things and there didn't seem to be a concensus on the packaged
suggested. I'll go back to it now to see if I can fix the test failures
and hopefully resolve this myself.

> These are all somewhat related to some degree but being casual about
> the terms really does not help anyone else who wants to pick up this
> package and ultimately fix it for our users.

Understood, apologies for the confusion. I hope the above makes it
clearer. In my defense, I have tried to delegate Mercurial to others in
the past and have not been succesful. Maybe it is because I wasn't clear
enough in my description but I do believe I have done deliberate and
good work to document my progress before giving back packages so far.

>> > Are you using btrfs?
>> 
>> Nope.
>
> (As in; that's often an easy to introduce some non-determinism
> filesystem ordering, rather than a diagnosis that needs knocking down..)

I am not sure I understand. Are you saying that btrfs makes it easier to
introduce non-deterministic filesystem ordering?

A.
-- 
Code is law.
                         - Lawrence Lessig