Hi all,
TL;DR: if one wishes to expose gpu to an sbuild environment, to
run the build time test suite in almost the same configuration
as on buildd, one quick and easy but perhaps deemed not very
safe way to do it would be to bind mount the whole /dev into
schroots via /etc/schroot/default/fstab. For a more selective
way, see the next paragraphs.
In rocm-hipamd, I adjusted the test suite to be triggered only
if there is an amd gpu available[1]. For the moment, I only
check for the existence of the /dev/kfd character device, but
perhaps I should be more diligent in the verification? Anyway,
to trigger, this means the build environment must expose said
amdgpu, which is not the default behavior with the classical
sbuild based environment used on buildd servers I believe.
[1]: https://salsa.debian.org/rocm-team/rocm-hipamd/-/commit/6da7b85a08a142c5ac38cd856a3572a2e01168e5
To run the build, I'm using sbuild with an schroot slightly
adjusted to expose the gpu device, and only it. Here below is a
sample of my /etc/schroot/schroot.conf; note the profile set to
amdgpu:
# schroot.conf(5)
[sid-amd64-amdgpu]
aliases=amdgpu
description=Debian sid (unstable) with amdgpu binding
type=directory
directory=/mnt/archs/sid-amd64
union-type=overlay
users=emollier
groups=sbuild
root-groups=sbuild
profile=amdgpu
This profile is defined partly in /etc/schroot/amdgpu/. I make
the /etc/schroot/amdgpu/copyfiles from a simple copy of the same
file from the default profile, same thing for the nssdatabases.
Regarding /etc/schroot/amdgpu/fstab, it also includes a binding
for pulling the cards via /dev/dri/, but this could be exposed
by script if fine grained selection of gpu were needed I think:
# fstab: static file system information for chroots.
# Note that the mount point will be prefixed by the chroot path
# (CHROOT_PATH)
#
# <file system> <mount point> <type> <options> <dump> <pass>
/proc /proc none rw,bind 0 0
/sys /sys none rw,bind 0 0
/dev/pts /dev/pts none rw,bind 0 0
tmpfs /dev/shm tmpfs defaults 0 0
/dev/dri /dev/dri none rw,bind 0 0
Finally, to pull the /dev/kfd but not the rest of /dev, I added
the script /etc/schroot/setup.d/15mkkfd below; this basically
looks up how the real /dev/kfd character device looks like on
the host, then rebuilds it inside the schroot environment:
#!/bin/sh
set -e
. "$SETUP_DATA_DIR/common-data"
. "$SETUP_DATA_DIR/common-functions"
. "$SETUP_DATA_DIR/common-config"
mkamdgpunod () {
# We can bind mount /dev/dri/, so it is part of the profile's fstab.
# However, the node /dev/kfd must be reconstructed from the host's.
AMDGPU_DEV_DIR="$CHROOT_MOUNT_LOCATION/dev"
AMDGPU_KFD_MAJOR="$(stat --format 0x%t /dev/kfd)"
AMDGPU_KFD_MINOR="$(stat --format 0x%T /dev/kfd)"
AMDGPU_KFD_MODE="$(stat --format 0%a /dev/kfd)"
AMDGPU_KFD_USER="$(stat --format %U /dev/kfd)"
AMDGPU_KFD_GROUP="$(stat --format %G /dev/kfd)"
mknod "$AMDGPU_DEV_DIR/kfd" --mode "$AMDGPU_KFD_MODE" \
c "$AMDGPU_KFD_MAJOR" "$AMDGPU_KFD_MINOR"
chown "$AMDGPU_KFD_USER:$AMDGPU_KFD_GROUP" "$AMDGPU_DEV_DIR/kfd"
unset AMDGPU_KFD_GROUP AMDGPU_KFD_USER AMDGPU_KFD_MODE
unset AMDGPU_KFD_MINOR AMDGPU_KFD_MAJOR AMDGPU_DEV_DIR
}
if [ "$CHROOT_PROFILE" = "amdgpu" ] && [ "$STAGE" = "setup-start" ]
then test ! -c /dev/kfd || mkamdgpunod
fi
Once all this is setup, and assuming an otherwise working sbuild
configuration, I trigger builds of a given package by choosing
the amdgpu specific schroot with -c flag:
$ sbuild -c sid-amd64-amdgpu rocm-hipamd_5.0.0-1~exp1.dsc
I don't consider myself an schroot guru, so would welcome any
improvements one might see. As revealed by tests in the past
few weeks, the test suite may impede the display of the host, or
trigger kernel tainting, so I suppose exposing only an auxiliary
gpu might be more prudent, at least initially, thus the possible
need for selective device reconstruction below /dev/dri/.
Next step might be to wrap up some working autopkgtests for
these packages, and thus determine a similar configuration for
lxc, the debian ci container type for running autopkgtests. But
lazy me has been sticking to testing packages in schroot instead
of lxc so far, which is not a recommended way, so haven't a
procedure yet.
In hope this helps with setting up build-test environments,
Have a nice day, :)
--
Étienne Mollier <emollier@emlwks999.eu>
Fingerprint: 8f91 b227 c7d6 f2b1 948c 8236 793c f67e 8f0d 11da
Sent from /dev/pts/2, please excuse my verbosity.
On air: Karmakanic - The Spirit Remains The Same
Attachment:
signature.asc
Description: PGP signature