Re: Building llama.cpp for AMD GPU using only Debian packages?

[Jamie Bainbridge <jamie.bainbridge@gmail.com>]

On Sat, 1 Feb 2025 at 00:47, Cordell Bloor <cgmb@slerp.xyz> wrote:
> [1]: I'd hoped that with so many friends on the team, I'd be able to
> have some influence on the technical direction of the library.
> Unfortunately, that proved not to be the case. It was a bit of a life
> lesson for me.
>
> Though whether AMD upstream
> accepts that tuning is an open question.

Open source software lives or dies by its community. With an
inattentive upstream as you describe above, it makes me wonder if
effort spent in ROCm is worth it.

ggeranov has also said llama.cpp is just maintaining ROCm support and
future effort will be put into Vulkan inference.

If Vulkan prompt processing could be sped up (maybe like jart's matmul
work? https://justine.lol/matmul/) then there would be little need to
use ROCm for llama.cpp at all.

I am only new here and I don't know how the rest of the ROCm ecosystem
operates, there is surely more than hobbyists running llama.cpp on
desktop GPUs, but if everyone else in this space is working on Vulkan
maybe Debian (and AMD) could leverage Vulkan more instead?

Jamie