Re: llama.cpp, whisper.cpp, ggml: Next steps

[Christian Kastner <ckk@debian.org>]

Hi Petter,

On 2025-02-18 13:04, Petter Reinholdtsen wrote:
> I would be happy to test the build and packages on my machine.  Is there
> a git repo yet?

I've pushed my work to [1].

This builds a package that apparently produces the right layout and
content. Basically, what I did for ggml-embedded-in-llama.cpp, with a
few improvements.

I say "apparently" because I haven't really tested this extensively yet.
The most significant change is how library and headers are made private;
the proper way to validate this is to build llama.cpp and whisper.cpp
against this. Specifically, disabling the embedded ggml build, and
pointing to the private directories. This should be doable entirely in
cmake.

I'll do the llama.cpp build tomorrow and will let you know how it went.
I think llama.cpp and whisper.cpp should be quite easy now. All the
complexity was in ggml.

There are still some warts of course, but I consider the result
shippable, and all further improvements should be transparent to other
users.

Best,
Christian

PS: I was concerned that packaging this, and the design decisions this
entails, might run against upstream's intentions. Upstream however was
quite receptive; we see the same challenges, and I think we can provide
valuable feedback and code back upstream.

[1]: https://salsa.debian.org/deeplearning-team/ggml