Hi Gavin,
Lastly, I have a working PR for sccache that can cache HIP compilations. Combining with a patch for HIPCC I was able to greatly speed up my workflow by exporting the environment variable HIP_CLANG_LAUNCHER=sccache so that I can rapidly edit, clean, and rebuild. I'm not too familiar with how Debian packaging works, but if you've also been bothered by the long compilation times hopefully this can help.
AMD uses sccache and ccache extensively within the internal CI systems for the ROCm project. The real trick is in the cache invalidation. I'll try to see if I can share more information about this. I also think you should also consider submitting that patch for hipcc upstream. It may or may not get accepted, but it's a good feature (and AMD used a similar patch internally for a while).
Looking further into the future, I believe that the greatest gains would come from caching amdgpu bitcode. You might need some support from clang to interpose the appropriate calls, but the device code optimization passes are where the compiler spends all its time during most library builds. Most source code and compiler flag changes don't affect the bitcode generated for the device code, so it has a much higher cache hit rate than caching based on the source code. I built a working prototype of this back when the rocm clang compiler was calling the external `opt` and `llc` utilities (circa ROCm 3.8).
I've adopted your HIPCC patch into the Debian package [1]. And I see that you had already submitted it upstream. Having now reviewed both your patches, I am very impressed by the quality of your work. Thank you very much for your contributions to the ROCm community.
Sincerely,
Cory Bloor
P.S. ccache has HIP language support, but it lacks the cache
invalidation logic that you've added for sccache.
[1]: https://lists.debian.org/debian-ai/2024/03/msg00030.html