Hi Ian,
On 18-06-18 18:19, Ian Jackson wrote:
> Firstly thanks for your prompt and detailed review.
YW.
> Paul Gevers writes ("Re: DEP-8 pseudo-restriction "hint-dpkg-testsuite-triggers""):
>> On 18-06-18 13:21, Ian Jackson wrote:
>>> This works, in the sense that it adds the information I wanted to the
>>> Testsuite-Triggers generated by dpkg-source. And it should cause all
>>> DEP-8 test runners to skip the test because of the unknown
>>> restriction, so it won't add any spurious test load.
>>
>> You are forgetting to mention the rationale for not adding this to any
>> existing test. I think I know, but let's see if you have the same in mind.
>
> Adding the dependencies explicitly to an existing test will cause
> lossage (of one kind or another) if the target of the indirect
> dependency changes, defeating the point of indirect package
> dependencies and adding maintenance burden.
So you have indeed a different rationale, good. Mine was that those test
dependencies do get installed. In you gnupg case, they may even change
the set of packages being install. Will come back to that.
> Wrong test triggers are a very minor problem compared to wrong package
> dependencies, of course. And they are minor even compared to wrong
> test dependencies.
Ack.
>> I would prefer it if we could think of a better way. We are currently
>> discussing¹ if the exit codes of autopkgtest shouldn't be used further,
>> e.g. by britney, and one case I may want to consider a regression is if
>> an autopkgtest (without source change) went from PASS (exit code 0) to
>> PASS with SKIPped tests (exit code 2). If a test has SKIPped tests by
>> design that would be a shame.
>
> I think that proposal of yours is wrong. Any package might introduce
> new tests which are currently SKIP, and that is not a regression. We
> don't want to discourage people from adding new, more sophisticated,
> tests, even if the have requirements which can't yet be met in debci.
I was only considering the case where the source of the autopkgtest
didn't change (i.e. no new added test) but Simon rightfully brought up
in bug 901804 that the testbed may change over time and reduce support
of some required restriction. (Bug in CC, please drop that again if you
don't respond to this part or the three pieces below).
> If you want to do this you should analyse the actual set of test
> results, and check that there are no *individual tests* which went
> from PASS to SKIP; just looking at the exit status is not sufficient.>
> The summary output from autopkgtest ought to be suitable for this.
> And think this can be done without needing to teach the test
> triggering infrastructure about individual tests; although you do
> need to store the previous summary (whether in parsed or unparsed
> form).
And this wouldn't be enough either, at least not to distinguish between
SKIP due to regression or SKIP due to infra change.
> As a bonus, if you go in this direction, we'll end up with a more
> intelligent regression blocker too: even if some set of the tests are
> FAIL, regressions to the ones that are still PASS will still be
> detected.
Good point, interesting.
> And it would be possible to mark individual tests as flaky rather than
> whole packages. (I don't know how often that would be useful.)
That is already possible in the the master of autopkgtest. Or you mean
at the britney level? I don't think that would be containing the info in
the right place.
>>> The packages listed as Depends for this test are usually indirect
>>> dependencies, updates to which are considered likely to cause
>>> regressions in other tests defined in this package.
>>
>> I don't like the word "likely" here. Have some trust in your peers. At
>> most this is "elevated risk". Or "have in the past led to regressions"
>
> If you look below at my specific use cases, you can see that the
> changes to diff/patch which might cause regressions are only likely to
> do so due to infelicity in dpkg-dev, an intervening package. Quite
> reasonable improvements in diff/patch might break things, and require
> remedial work (ideally, then, in dpkg-dev).
>
> So I certainly don't want to imply any insults towards the named
> packages. They are often likely to be ones where the interactions are
> indirect but still intimate.
>
> How about "more likely" ? Or "are considered to pose a risk of
> regressions" ?
Fine. I like the second one better.
>>> There is currently no way to specify this hint on a per-test
>>> basis; but in any case the debian.org machinery is not able to
>>> think about triggering individual tests.
>>
>> Are you considering the case here where we would trigger one test out of
>> the autopkgtest for pkg-a and another for pkg-b? Before that is
>> possible, we would need to change the Testsuite-Triggers mechanism again.
>
> Yes. Or the system might discover that there is no point triggering
> any tests because all of the relevant ones have unmeetable
> restrictions, and all of the runnable ones are irrelevant to the
> update being performed.
>
> The Testsuite-Triggers mechanism is indeed rather crude. I don't like
> it very much but of course we didn't want the best to be the enemy of
> the good.
Fully ack.
> What I'm proposing here is indeed awkward. It is conceptually wrong -
> a fact which is highlighted in my description - but that conceptual
> wrongness is purely inherited from the Testsuite-Triggers
> arrangements, which I am trying to manipulate.
>
> If there are better alternatives that are likely to be deployable soon
> then I'm all ears...
>
>>> I haven't yet pushed the change with this restriction name in it,
>>> anywhere, so if people think a different name is better I'm open to
>>> suggestions.
>>
>> I don't really care what name this carries though if we go for it.
>
> OK.
>
>>> I wrote:
>>>> 4. Can we have a way to trigger tests from updates of non-direct
>>>> rdepends ? At some point in the future maybe we will run tests of
>>>> whole batches of updates and then have some algorithm to chop out
>>>> what the failures are caused by, but for now it would be useful to
>>>> be able to declare a specific indirect dependency for test trigger.
>>>> Maybe an XS- header field ?
>>>
>>> In dgit, I wanted to do this because ISTM that some of the most likely
>>> sources of regressions are a handful of indirect dependencies.
>>
>> Just curious, but if they are so important for your package, how come
>> they end up being only indirect dependencies?
>
> Happy to answer that:
>
> * gnupg:
>
> gnupg2 has a bunch of daemons and stuff which it tries to
> autostart, but the autostart and clean up is plagued with races and
> other bugs. On a number of previous occasions the dgit test suite
> has started failing because the gnupg races got worse, making my
> workarounds (which are gradually becoming more horrific)
> insufficient.
>
> I don't want to add a direct dependency because dgit is very
> agnostic about what `gpg' on PATH is actually like, and the package
> name `gnupg' means different things in different Debian releases.
> Because dgit Depends on devscripts and does much of its signing
> with debsign, there is guaranteed to be something suitable.
>
> Certainly I don't want to stop anyone installing gnupg1 if they can
> do so, nor do I want to stop them running the test suite with
> gnupg1.
>
> (Maybe this entry in Testsuite-Triggers is of no practical import
> with the current test runner, because dgit-infrastructure Depends
> on gpgv, so gnupg2 updates will cause dgit's tests to be rerun
> anyway.)
Considering your description above I give you the following alternative
suggestion: make a gnupg1 specific test, where you install gnupg instead
of gnupg2 and verify that it works also that way. For a while I was
testing my package dbconfig-common with both MariaDB and MySQL servers
(neither of which are my (indirect) dependencies as the server may run
on a different host).
> * patch/diffutils:
>
> dgit is quite fussy about how source packages with patch queues
> work. dpkg-source is of course responsible for the actual
> implementation, which is in terms of diff and patch. dpkg-source
> is *not* careful about exactly what the semantics of diff and patch
> are.
>
> We have already had one incident where a change to the behaviour of
> diff and patch led to an uncontrolled and unversioned semantic
> change to the feature set of the `3.0 (quilt)' source package
> format. (As a result, some newer packages cannot be unpacked on
> older Debian releases, and this was not detected until far too
> late.)
>
> Rerunning the dgit autopkgtests when diff or patch is updated seems
> like a sensible precaution to me. (Better would be for dpkg-source
> to have a comprehensive DEP-8 test suite would detect accidentally
> picking up new diff/patch features, and which avoids the meaning of
> existing source packages changing.)
I started writing in my previous mail a proposal about you helping your
dependency to create a good autopkgtest to cover your use case. I
deleted that. Seems like I should have left that in. :)
Paul
Attachment:
signature.asc
Description: OpenPGP digital signature