Re: Bug#1063673: ITP: llama.cpp -- Inference of Meta's LLaMA model (and others) in pure C/C++

[Christian Kastner <ckk@debian.org>]

Mo, thanks again for the pointer. This was great advice.

On 2024-12-22 20:42, Mo Zhou wrote:
> No need to do that manually. The Glibc already provided that kind of
> dispatching functionality when you build multiple solibs with different
> baselines. Please check
> 
>   https://lists.debian.org/debian-devel/2019/04/msg00057.html
> 
> Or concretely,
> 
>   section "Hardware capabilities" from ld.so(8)
> 
> I believe this is what you were expecting. But it seems that the
> avx2/ avx512 dispatch is missing from the man page. Don't know
> what's happening for their support.

This was updated with glibc 2.33, with a better solution that supports
the amd64 v1-v4 levels [2]. The 32-bit stuff was declared legacy, and
dropped in 2.37.

Using this new approach, I was elated to see that it doesn't just work
perfectly, it also works with non-standard library paths (lacking
stability, llama.cpp's libraries are kept private, for now). And the
baseline build should work fine for any other dynamic linker
implementation, so there is really no meaningful downside to using
hwcaps.

Below are the outputs for the CPU backend of llama.cpp. These are all
from the same QEMU VM on endeavour (EPYC 9354P) to which I assigned 8
cores and 64GiB RAM, and just booted with a different -cpu argument,
one matching each level. You can see the levels get picked up by the
linker, and the resulting effect on performance [3].

Best,
Christian

[2]: https://en.wikipedia.org/wiki/X86-64#Microarchitecture_levels

[3]: I could have "hacked" this to run on bare metal, too, but the VM
     approach was less intrusive to our production CI host.


# -cpu host, this one has AVX512
root@unstable-guest:~# ldd /usr/bin/llama-bench
        linux-vdso.so.1 (0x00007fc25698a000)
        libllama.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v4/libllama.so (0x00007fc256735000)
        libggml.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v4/libggml.so (0x00007fc256726000)
        libggml-base.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v4/libggml-base.so (0x00007fc256677000)
        [...]
root@unstable-guest:~# llama-bench -m ggml-model-q4_0.gguf 
| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         pp512 |        314.18 ± 0.29 |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         tg128 |        137.38 ± 0.08 |


# -cpu EPYC, up to AVX512
root@unstable-guest:~# ldd /usr/bin/llama-bench
        linux-vdso.so.1 (0x00007f275b5a3000)
        libllama.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v3/libllama.so (0x00007f275b34e000)
        libggml.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v3/libggml.so (0x00007f275b33f000)
        libggml-base.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v3/libggml-base.so (0x00007f275b290000)
        [...]
root@unstable-guest:~# llama-bench -m ggml-model-q4_0.gguf 
| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         pp512 |        281.55 ± 0.12 |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         tg128 |        139.31 ± 0.08 |


# -cpu Opteron_G5, up to SSE4.2
root@unstable-guest:~# ldd /usr/bin/llama-bench
        linux-vdso.so.1 (0x00007f2d23bc1000)
        libllama.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v2/libllama.so (0x00007f2d2392e000)
        libggml.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v2/libggml.so (0x00007f2d2391e000)
        libggml-base.so => /usr/lib/x86_64-linux-gnu/llama.cpp/glibc-hwcaps/x86-64-v2/libggml-base.so (0x00007f2d2386e000)
        [...]
root@unstable-guest:~# llama-bench -m ggml-model-q4_0.gguf 
| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         pp512 |         86.39 ± 0.53 |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         tg128 |         66.94 ± 0.07 |


# -cpu Opteron_G3, baseline SSE2
root@unstable-guest:~# ldd /usr/bin/llama-bench
        linux-vdso.so.1 (0x00007faf3fb6c000)
        libllama.so => /usr/lib/x86_64-linux-gnu/llama.cpp/libllama.so (0x00007faf3f917000)
        libggml.so => /usr/lib/x86_64-linux-gnu/llama.cpp/libggml.so (0x00007faf3f908000)
        libggml-base.so => /usr/lib/x86_64-linux-gnu/llama.cpp/libggml-base.so (0x00007faf3f859000)
        [...]
root@unstable-guest:~# llama-bench -m ggml-model-q4_0.gguf 
| model                          |       size |     params | backend    | threads |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         pp512 |         48.25 ± 0.03 |
| llama 1B Q4_0                  | 606.54 MiB |     1.10 B | CPU        |       8 |         tg128 |         41.14 ± 0.01 |