Re: ROCm installation
On 1/12/22 20:17, M. Zhou wrote:
Hi,
Thanks for the updates.
On Wed, 2022-01-12 at 18:14 +0100, Maxime Chambonnet wrote:
"Native" Debian packages are starting to cover a significant portion of
the
stack [2], and it would be great to figure out the installation topic
The word "native" is ambiguous to a portion of developers as it may
also refer a native (debian/source/format) package.
For other readers: it's "offician debian package" in contrast to
"third-party debian packages by upstream.
on how to install ROCm today.
After skimming through the mail I realize what you actually meant
is the "ROCm file installation layout" right?
Yes, totally, I forgot a bit to extract myself from my point of view!
The installation options and paths generally looked for by CMake
Lists/configs
are currently:
- various cmake project-specific flags for the install paths of the
components
HIP_CLANG_PATH, HIP_DEVICE_LIB_PATH, HIP_PATH, ROCM_PATH, ... see
[5]
Headers and libraries should installed under the standard path,
so that the compiler and linker should be able to find them without
additional flags. Just install all stuff to /usr should be enough.
Currently for example rocm-hipamd installs to /usr/hip, and
lintian yells a lot. All to /usr is quite not clear enough.
- /opt/rocm as a default backup
There is no way for `/opt` as official debian package. If any component
breaks without any specific file under /opt, then it is a bug to fix.
Right!
I see at least three choices, and sub-decisions to be made:
- Multi-arch or not
nvidia toolkit supports aarch64 and a few others.
Cross-compiling ROCm from Debian could be interesting in a near-
future.
The rocm libraries and binary executables are architecture dependent.
Most of them should have Architecture: any in d/control.
Cross-compiling ROCm is not something worth being looked at IMHO.
ROCm targets on high performance computing. A hardware architecture
really capable of "high performance computing" can't be too weak
to compile ROCm itself.
That said, making the installation layout Multi-Arch aware is a
good practice. Most of the packages may have Multi-Arch: same
as long as they contain architecture-dependent files.
- Nested or not
Other stacks and relatively important projects, such as postgresql
or
llvm go
nested (there is a central /usr/lib/{llvm-13, postgresql} directory,
often with a sub ./bin, ...)
I did not understand this question. Do you mean something like
/usr/lib/rocm-{4.5.2,5.0.0},
or
/usr/lib/rocm-4.5.2/llvm ?
Rather the first, not sure I see a difference, in all cases, it looks
nested under "rocm-something" to me. And we further down agree
that nested is probably not the way.
- Where to install machine-readable GPU code
There is at least 3 types of device-side (aka GPU) binary files -
.bc for bitcode,
.hsaco for HSA code object and
.co for code object.
How are these files read by ROCm? Is there anything like
"PYTHONPATH" for the gpu code files? We should choose a
supported path compatible to debian policy.
There is a cmake flag / environment variable for now,
HIP_DEVICE_LIB_PATH :<
The current preferred layout is /usr/amdgcn/*.bc
BTW, are these files architecture-independent? Namely,
can arm64 and amd64 produce the exactly the same (e.g.
md5sum-identical) output?
I don't know, we discussed it last jitsi meeting and
I believe that no one tried yet :)
Bitcode files are the machine readable form of the LLVM intermediate
representation. HSA (Heterogeneous System Architecture) and other
code object
files are AMD containers for GPU machine code. PostgreSQL does use
llvm
bitcode files: since the install path is nested, they are in
/usr/lib/postgresql/14/lib/bitcode.
Since it is arch-independent in the sense of the CPU architecture, I
have
been proposed that such code should reside in /usr/share.
Nested layout for llvm and postgresql intends to allow multiple
versions of the software co-exist on the same system. For example,
llvm-{11,12,13} may be installed simultaneously on Debian.
We debian rocm team does not have so many contributors to support
multiple versions. Just do it the simplest way as we can.
The official repacked nvidia-cuda-toolkit is not relevant
to such nested layout.
Agreed
What I tried to keep in mind is that:
- shared libraries should be easily discoverable in paths looked by
/etc/ld.so.conf
- there are only so much paths that cmake find_package in config mode
looks for [8].
Shared objects from Multi-arch aware library packages should be
put at /usr/lib/<multiarch-triplet>/ as long as they are indended
for public usage.
Don't be misled by complicated setups such as llvm, postgresql or
the upstream non-standard installation path. In the standard setup
everything is likely becoming simpler. When you started to think
about ld.so.conf for a regular official debian shlib package, I
doubt something had been going wrong.
Gentoo has basically finished their ROCm packaging. Feel free to
borrow them as their license permits.
Will look further at it!
I attached as an image a direct comparison between some arbitrary
combinations
of these decisions. The directories are bundled in the attached archive
too.
- install_layout_proposal_v1 goes
multi-arch, flattened, and with GPU code in /usr/share
- install_layout_proposal_v2 goes
"ante-multi-arch", nested, and with GPU code in /usr/lib
1. header.
installation path of architecture-dependent headers should contain
multi-arch triplet (e.g. x86_64-linux-gnu). In this case,
Architecture: any, Multi-Arch: same
if the headers are identical across all architectures, the multi-arch
triplet should be stripped.
Architecture: all. Multi-Arch: no (default)
I am not sure, maybe Cordell could help.
2. shared objects.
No need to nest as /usr/lib/rocm/lib. Just install every shared objects
to /usr/lib/<multi-arch-triplet>/ . Private shared objects (such as
plugins) may go to /usr/lib/<multi-arch-triplet/rocm/ .
Nested installation layout is really pointless unless you are
determined to support the co-existence of multiple ROCm versions
on Debian.
My vote on "maintaining co-existence of multiple versions of ROCm"
is disagree.
Understood and agreed!
Reply to: