Re: ROCm CI: Scheduling for experimental active again
Hi Cory,
On 2024-01-10 21:35, Cordell Bloor wrote:
> There are other ways we might want to prune the triggers as well. Seem
> to be using the package build dependencies as the basis to determine if
> the tests are triggered, but the root of the dependency walking should
> really be the binary dependencies of the autopkgtest.
>
> For example, we're rerunning the rocsparse autopkgtest for a new
> experimental version of kmod. This is because kmod is a dependency of
> rocminfo and rocminfo is a dependency of hipcc. However, hipcc is only a
> build dependency of rocsparse-test, not a runtime dependency. So, when
> using a pre-built copy of rocsparse-test as we do in the autopkgtest,
> the updated kmod cannot affect the outcome of the test, and the test is
> redundant.
You mention build dependencies above, but the resolver actually only
looks at regular dependencies between binary packages, which goes too
far as you say.
For any given source package A, the resolver
(1) looks at all binary packages built by A
(2) collects all there reverse binary dependencies
(3) for all binary packages collected in (2), find source packages
(4) for all source packages found in (3), repeat from (1)
The assumption being that this covers all changes that could
theoretically affect a package.
For kmod, applying your findings to the steps above,
(1) src:rocminfo builds bin:rocminfo
(2) bin:rocminfo has bin:hipcc as reverse binary dependency
(3) bin:hipcc is from src:rocm-hipamd
(4) (1) src:rocm-hipamd builds bin:libamdhip64-5
... and thus, all the other tests are triggered.
> I could imagine that at some point we might add an autopkgtest that does
> depend on hipcc, but it would be nice if the triggering of each
> Test-Command could be filtered based on the Depends tree for that
> command. I think it would dramatically reduce the CI utilization without
> affecting the CI coverage, which will be necessary as we continue to
> scale up the number of packages we're testing.
That is an *excellent* suggestion!
Step (1) goes too far. For our purposes, it would suffice to look at
only the bin:foo-tests packages built by src:foo, rather than all its
packages.
I've created an issue [1] for this, and will fix it over the weekend.
Unfortunately it seems that APT does not expose debian/tests/control
dependency information, but since we have dedicated -tests packages, the
dependencies should generally be recorded in debian/control anyway, so
that users installing these packages can run the tests manually without
having to bother to "learn" autopkgtest [2].
By the way, the scheduler already has the ability to ignore triggers,
and I was about to add kmod to it. But your proposed solution is the
correct one.
Best,
Christian
[1] https://salsa.debian.org/rocm-team/rocm-dev-tools/-/issues/12
[2] This won't work when we have autopkgtests that recompile stuff, but
let's cross that bridge when we get there.
Reply to: