Re: Format of test specifications
Chris,
On Fri, Dec 7, 2018 at 1:24 PM Chris Lamb <lamby@debian.org> wrote:
>
> Could you elaborate more why? You do mention that changing 'Test-
> Architectures' (for example) should not trigger the rebuilding of
> packages but is that really a big concern? I mean, I throw away
> everything before running any test and it's not excessively slow to do
> so.
>
First of all, it is not urgent. My hope is that the test suite will
eventually permutate through several degrees of freedom. The goal is
to get as many *test cases* out of the test specifications as
possible.
That idea was formed in reaction to commit ccb387b4. There, a problem
arose when a test specification was filled with 'Package-Architecture:
any' while the harness provided 'amd64' (due to a particular
implementation of Test-Architectures). The check should have worked
either way.
Next, please consider that the current defaults for test cases are
package format '1.0 (native)' with 'Package-Architecture: all'. That
is not representative of the archive at all. As you know from my merge
requests, I went on to parametrize many templates. I never submitted
branches changing the default package architecture to 'any', because
they seemed like half-way solutions. The right way is to permutate for
each degree of freedom through all possible values.
When one test specification is turned into many test cases, some data
changes and some does not. The template fill data and the test options
can vary from one test case to another, while the constraints do not.
In fact, the constraints are currently stored up one directory.
The distinction between fill data and test options, on the other hand,
is helpful to avoid confusion. Among other things, it will safely
allow the field Package-Architecture to be renamed back to
Architecture (hoping to address your request here:
https://salsa.debian.org/lintian/lintian/merge_requests/46#note_46510).
As for rebuilding, perhaps I am more time sensitive. I run the full
suite dozens of times a day and would like to bring it down from six
minutes over here to about one (if my experiments hold up). When
adding permutations for package architecture and package format alone
(and there are additional degrees of freedom), the 670~ test
specifications turn into 6000 test cases, or more. Rebuilding is not a
problem with a single test specification or with hand edits, but
rather when such processes are automated.
That being said, I am not set one way or another. Please let's revisit
this proposal when my 'variations' branch is ready for your
consideration.
Reply to: