Re: CMake musings for ROCm 4.5.2 (Was Re: ROCm packaging session memo) (Was Re: ROCm installation)
I took the liberty of adding Ye Luo from ANL, whom voiced various
legitimate concerns about the build system at ROCm githubs.
On 1/30/22 10:38, Cordell Bloor wrote:
> It doesn't work properly because it wasn't designed to do that in
> ROCm 4.5. It's only just now that support for any layout besides that
> of /opt/rocm and spack has become a goal. Upstream support probably
> won't be coming until ROCm 5.2 or later, but we should be able to get
> it fixed on Debian long before then by backporting patches (once
> they exist).
Yes :)
> I believe I mentioned in the past that you can correct this problem
> across all math libraries with minor changes to rocm-cmake.
Did not see?
> I've recently been made a maintainer for rocm-cmake, so I suppose it's
> now my job to help facilitate that. The primary question is what file
> layout Debian wants.
Cool!
> /usr/rocrand was the result of my quick-and-dirty sample build
> script. It's not a serious suggestion of anything. It's a prompt for
> us to discuss what layout Debian would like for the math libraries.
> For example, is there a preference between
> /usr/include/rocrand/rocrand.h and /usr/include/rocrand.h?
I asked on debian-devel and heard from Mo Zhou [1], veteran
Debian Developer whom initiated ROCm packaging here.
We mostly agreed in the thread that:
- headers should be in /usr/include/<package>, arch-independent. This
is ${CMAKE_INSTALL_INCLUDEIR}/<package>. (the second version of my
image and its install(DIRECTORY) is still wrong so). To answer your
question, the preference would be for
/usr/include/rocrand/rocrand.h .
- Libraries should be in multi-arch by default, at my place, it is
/usr/lib/x86_64-linux-gnu/ . This is cmake's ${CMAKE_INSTALL_LIBDIR},
by default, no need for anything else!
- The export should be with the library in multiarch, in
/usr/lib/<arch triple>(e.g. x86_64-linux-gnu)/cmake/<package>
which is ${CMAKE_INSTALL_LIBDIR}/cmake/<package> .
These three points above for the maths libraries, but generally for all
ROCm (especially HIPAMD)?
- Last but not least Etienne validated bit-for-bit reproducibility
between builds of the device libs on different CPU architectures,
and it was agreed between Jeremy and him that device libs would be
fine living in /usr/share/amdgcn/bitcode if it is ever validated.
> I have not dug deep enough into rocm-cmake to have a personal
> opinion. It was authored by a skilled developer, so I default to
> trusting his judgement.
>
> In general, though, the ROCm build code is a huge mess of unnecessary
> complexity. We can improve it, but there is no quick fix. To remove
> complex code, you need to know that the things you're deleting aren't
> fulfilling some requirement on some other platform. Unfortunately,
> there is no comprehensive listing of ROCm build requirements or
> supported platforms. Understanding why a line of code exists requires
> detective work.
>
> I spent the past year painstakingly studying the rocSOLVER build
> code. I tracked down why each line was added and checked if it was
> still relevant. Over the course of 2021, I removed hundreds of lines
> of unnecessary CMake. Simplifying the build code is not a fast
> process. At least, not in upstream.
I understand that removing is harder than adding, and you have been
doing valuable work, as in Ken Thompson's most productive days :)
I checked deeper rocm-cmake, and while I question the call to
ROCMInstallSymlinks.cmake, I can finally see use cases for
ROCMInstallTargets.cmake indeed, and it makes correct use of
GNUInstallDirs.
For rocSOLVER, these four lines seem to be what is crippling the
underlying install function:
rocsolver/library/src/CMakeLists.txt, line 249
rocm_install_targets(
TARGETS ${package_targets}
*********************************************************
INCLUDE
${CMAKE_SOURCE_DIR}/library/include
${CMAKE_BINARY_DIR}/include
PREFIX rocsolver
*********************************************************
)
I suspect that without them, we get very close to "Debian's preferred
install layout"!
And with PARSE_PREFIX and INSTALL_PREFIX, you still have latitude to
install wherever spack wants, or in /opt/rocm?
> Kitware is great. I get the impression it was not an easy process,
> but they managed to organize some of the low level details of the
> ROCm stack when building the HIP language support for CMake 3.21.
> We're not using CMake's HIP language support yet in the math
> libraries, but I think it will simplify a lot.
Today I learnt that such support even existed, thanks!
> Sincerely,
> Cory Bloor
Thank you,
Best regards, Maxime
[1] https://lists.debian.org/debian-ai/2022/01/msg00032.html
Reply to: