Hi Ian,
On 18-06-18 13:21, Ian Jackson wrote:
> I wrote:
>> 4. Can we have a way to trigger tests from updates of non-direct
>> rdepends ? At some point in the future maybe we will run tests of
>> whole batches of updates and then have some algorithm to chop out
>> what the failures are caused by, but for now it would be useful to
>> be able to declare a specific indirect dependency for test trigger.
>> Maybe an XS- header field ?
>
> In dgit, I wanted to do this because ISTM that some of the most likely
> sources of regressions are a handful of indirect dependencies.
Just curious, but if they are so important for your package, how come
they end up being only indirect dependencies?
> Mattia Rizzolo pointed out:
>> On Thu, May 03, 2018 at 10:38:45PM +0200, Paul Gevers wrote:
>>> Just add it as a test dependency in one of your tests?
>>
>> Just to share a bit that doesn't seem to be of public knowledge:
>> .dsc have a Testsuite-Triggers field that is autopoulated from the
>> d/tests/control file (IIRC). I believe you are looking exactly for
>> this field.
>
> I wanted to add some things to Testsuite-Triggers, rather than replace
> the field, so I couldn't just supply it in debian/control. Instead, I
> am adding, in debian/tests/control:
>
> Tests: hint-testsuite-triggers
> Tests-Directory: tests/tests
Tests-Directory is cruft if you don't expect the test to be run anyways.
> Depends: gnupg, patch, diffutils
> Restrictions: hint-testsuite-triggers
>
> This works, in the sense that it adds the information I wanted to the
> Testsuite-Triggers generated by dpkg-source. And it should cause all
> DEP-8 test runners to skip the test because of the unknown
> restriction, so it won't add any spurious test load.
You are forgetting to mention the rationale for not adding this to any
existing test. I think I know, but let's see if you have the same in mind.
I would prefer it if we could think of a better way. We are currently
discussing¹ if the exit codes of autopkgtest shouldn't be used further,
e.g. by britney, and one case I may want to consider a regression is if
an autopkgtest (without source change) went from PASS (exit code 0) to
PASS with SKIPped tests (exit code 2). If a test has SKIPped tests by
design that would be a shame.
> I chose to invent the restriction name `hint-dpkg-testsuite-triggers'
^^^^^ dropped below.
> for this purpose. Should this be added to README.package-tests.rst ?
>
> hint-testsuite-triggers
>
> This test exists purely as a hint to suggest when rerunning the
> tests is likely to be useful. Specifically, it influences the
> way dpkg-source generates the Testsuite-Triggeers .dsc header
^^ typo
> from test metadata: the Depends for this test are added to
> Testsuite-Triggers.
>
> The packages listed as Depends for this test are usually indirect
> dependencies, updates to which are considered likely to cause
I don't like the word "likely" here. Have some trust in your peers. At
most this is "elevated risk". Or "have in the past led to regressions"
> regressions in other tests defined in this package.
>
> There is currently no way to specify this hint on a per-test
> basis; but in any case the debian.org machinery is not able to
> think about triggering individual tests.
Are you considering the case here where we would trigger one test out of
the autopkgtest for pkg-a and another for pkg-b? Before that is
possible, we would need to change the Testsuite-Triggers mechanism again.
> The test with the hint-testsuite-triggers restriction should not
> actually be run. Future systems which understand per-test update
> triggering should treat the Depends of the test with this
> restriction, as triggering packages for all tests in this
> debian/tests/control.
>
> I haven't yet pushed the change with this restriction name in it,
> anywhere, so if people think a different name is better I'm open to
> suggestions.
I don't really care what name this carries though if we go for it.
Paul
¹ Weird place, but here:
https://salsa.debian.org/ci-team/autopkgtest/merge_requests/19
Attachment:
signature.asc
Description: OpenPGP digital signature