[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: CMake musings for ROCm 4.5.2 (Was Re: ROCm packaging session memo) (Was Re: ROCm installation)

I took the liberty of adding Ye Luo from ANL, whom voiced various
legitimate concerns about the build system at ROCm githubs.

On 1/30/22 10:38, Cordell Bloor wrote:
> It doesn't work properly because it wasn't designed to do that in
> ROCm 4.5. It's only just now that support for any layout besides that
> of /opt/rocm and spack has become a goal. Upstream support probably
> won't be coming until ROCm 5.2 or later, but we should be able to get
> it fixed on Debian long before then by backporting patches (once
> they exist).

Yes :)

> I believe I mentioned in the past that you can correct this problem
> across all math libraries with minor changes to rocm-cmake.

Did not see?

> I've recently been made a maintainer for rocm-cmake, so I suppose it's
> now my job to help facilitate that. The primary question is what file
> layout Debian wants.


> /usr/rocrand was the result of my quick-and-dirty sample build
> script. It's not a serious suggestion of anything. It's a prompt for
> us to discuss what layout Debian would like for the math libraries.
> For example, is there a preference between
> /usr/include/rocrand/rocrand.h and /usr/include/rocrand.h?

I asked on debian-devel and heard from Mo Zhou [1], veteran
Debian Developer whom initiated ROCm packaging here.
We mostly agreed in the thread that:

- headers should be in /usr/include/<package>, arch-independent. This
  is ${CMAKE_INSTALL_INCLUDEIR}/<package>. (the second version of my
  image and its install(DIRECTORY) is still wrong so). To answer your
  question, the preference would be for
  /usr/include/rocrand/rocrand.h .

- Libraries should be in multi-arch by default, at my place, it is
  /usr/lib/x86_64-linux-gnu/ . This is cmake's ${CMAKE_INSTALL_LIBDIR},
  by default, no need for anything else!

- The export should be with the library in multiarch, in
  /usr/lib/<arch triple>(e.g. x86_64-linux-gnu)/cmake/<package>
  which is ${CMAKE_INSTALL_LIBDIR}/cmake/<package> .

These three points above for the maths libraries, but generally for all
ROCm (especially HIPAMD)?

- Last but not least Etienne validated bit-for-bit reproducibility
  between builds of the device libs on different CPU architectures,
  and it was agreed between Jeremy and him that device libs would be
  fine living in /usr/share/amdgcn/bitcode if it is ever validated.

> I have not dug deep enough into rocm-cmake to have a personal
> opinion. It was authored by a skilled developer, so I default to
> trusting his judgement.
> In general, though, the ROCm build code is a huge mess of unnecessary
> complexity. We can improve it, but there is no quick fix. To remove
> complex code, you need to know that the things you're deleting aren't
> fulfilling some requirement on some other platform. Unfortunately,
> there is no comprehensive listing of ROCm build requirements or
> supported platforms. Understanding why a line of code exists requires
> detective work.
> I spent the past year painstakingly studying the rocSOLVER build
> code. I tracked down why each line was added and checked if it was
> still relevant. Over the course of 2021, I removed hundreds of lines
> of unnecessary CMake. Simplifying the build code is not a fast
> process. At least, not in upstream.

I understand that removing is harder than adding, and you have been
doing valuable work, as in Ken Thompson's most productive days :)

I checked deeper rocm-cmake, and while I question the call to
ROCMInstallSymlinks.cmake, I can finally see use cases for
ROCMInstallTargets.cmake indeed, and it makes correct use of

For rocSOLVER, these four lines seem to be what is crippling the
underlying install function:
rocsolver/library/src/CMakeLists.txt, line 249

  TARGETS ${package_targets}
  PREFIX rocsolver

I suspect that without them, we get very close to "Debian's preferred
install layout"!

And with PARSE_PREFIX and INSTALL_PREFIX, you still have latitude to
install wherever spack wants, or in /opt/rocm?

> Kitware is great. I get the impression it was not an easy process,
> but they managed to organize some of the low level details of the
> ROCm stack when building the HIP language support for CMake 3.21.
> We're not using CMake's HIP language support yet in the math
> libraries, but I think it will simplify a lot.

Today I learnt that such support even existed, thanks!

> Sincerely,
> Cory Bloor

Thank you,
Best regards, Maxime

[1] https://lists.debian.org/debian-ai/2022/01/msg00032.html

Reply to: