Re: RFC: rocm-autopkgtest-helper
Hi,
On 2024-10-13 16:10, Christian Kastner wrote:
> (B) One test per invocation, with arbitrary options
>
> Test-Command: rocm-autopkgtest-helper /path/to/test --output foo.xml --skip BadTest
>
> The downside is that one can no longer specify a directory, but a
> wrapper could solve this.
I've finished the first version of the test helper [1]. I've embedded
this version in new releases of hipsolver (1-autopkgtest, 1-executable)
and rocrand (2-autopkgtests, N-executables), and converted their
autopkgtests to make use of it.
I think this turned out nicely: d/tests/control of both packages are
minimal [2, 3], hipsolver needs no extra file, and rocrand just needs a
tiny wrapper to run N-executables [4].
Highlight
---------
First, I'd like to point out that the nice thing about the helper is
than what I describe below works for *any* test using it, not just our
own ROCm packages. This should meaningfully lower the barrier for other
maintainers to implement tests.
With increasing adoption, it also enables us to evolve test running
centrally: if we need a new feature, we just add it to the helper,
rather than ask maintainers to adapt all of their packages.
Basics
------
The test helper performs all the necessary checks to ensure that a GPU
is available for the test. If not, it skips the test.
Exporting system info
---------------------
A particularly useful feature is that rocm-testhelper can be instructed
to dump dmesg, firmware info, and rocminfo [5] output by means of
environment variables that we can set in a debci worker's config.
You can see the effect of this here, in the artifacts for hipsolver on
gfx1030: first, from a regular run [6], and then one with the following
added to the worker's 'debci_autopkgtest_args':
--env=ROCM_TESTHELPER_WITH_DMESG= --env=ROCM_TESTHELPER_WITH_DRI=/dri
With an image built by debci_3.10+rocm6 that I uploaded earlier today,
dmesg before and after the test were also saved as artifacts [7].
This way, export becomes worker-dependent, rather than requiring tests
to be defined with the needs-sudo or needs-root restrictions.
Exporting test-specific info
----------------------------
As you can see in the artifacts, I've also added an option to the
tests (not the runner) to export GTest output in XML form. This might
be a useful template for other maintainers to follow.
Within our CI, we can now grab this data, and post-process it to
generate test pass percentages, rather than just pass/fail.
Best,
Christian
[1]: which I named rocm-testhelper since technically, it can also be
used for non-autopkgtest execution.
[2]: https://salsa.debian.org/rocm-team/hipsolver/-/blob/master/debian/tests/control
[3]: https://salsa.debian.org/rocm-team/rocrand/-/blob/master/debian/tests/control
[4]: https://salsa.debian.org/rocm-team/rocrand/-/blob/master/debian/tests/run-testdir
[5]: Though this requires tests to depend on rocminfo, or rocminfo to
be installed in the base image.
[6]: https://ci.rocm.debian.net/data/autopkgtest/unstable/amd64+gfx1030/h/hipsolver/37173/artifacts.tar.gz
[7]: https://ci.rocm.debian.net/data/autopkgtest/unstable/amd64+gfx1030/h/hipsolver/37226/artifacts.tar.gz
Reply to: