Re: Bug#1063673: ITP: llama.cpp -- Inference of Meta's LLaMA model (and others) in pure C/C++
On 12/22/24 13:03, Christian Kastner wrote:
To my point of view, llama.cpp is more suitable for source-based
distributions like Gentoo. In the past I proposed something similar for
Debian but the community was not interested in that.
Well, we could still keep this in mind and implement it experimentally
for some packages. After all, it's already happening in some sort with DKMS.
Given how data is processed today, our amd64 baseline really does impede
machine-learning software, so some solution will need to be found.
Apart from source-based alternative distribution for Debian, "bumping amd64
baseline for selected packages" is another project I proposed long time ago:
https://github.com/SIMDebian/SIMDebian
Software like Eigen3, TensorFlow can heavily benefit from the baseline bump.
At that time PyTorch did not have dispatch, but now it has already.
In terms of the BLAS/MKL-like approach for SIMD capability
dispatching ... I bet focusing on something else is more worthwhile.
I probably misunderstood something about the MKL packages. I saw
libmkl-avx
libmkl-avx2
libmkl-avx512
...
and I assumed that these are all the same library, just built with
different optimizations. But they don't seem to conflict with each
other, so I guess it's more subtle than that.
MKL might not be working the way you thought. The main library to link
against
is libmkl-rt, which will dynamically link to the right implementation based
on the run-time CPU capability. That's just their specific way of
implementing
the dynamic dispatch.
My idea was to just build libllama X times with various optimizations
enabled. So users can select the version best applicable to their CPU
and/or GPU, but we would also have a non-optimized version that would
satisfy our amd64 ISA requirements.
No need to do that manually. The Glibc already provided that kind of
dispatching functionality when you build multiple solibs with different
baselines. Please check
https://lists.debian.org/debian-devel/2019/04/msg00057.html
Or concretely,
section "Hardware capabilities" from ld.so(8)
I believe this is what you were expecting. But it seems that the avx2/avx512
dispatch is missing from the man page. Don't know what's happening for their
support.
BTW, my personal conclusion on the SIMDebian project is that, while bumping
baseline can indeed benefit a lot, it is not necessary to do so by
ourselves,
because if SIMD performance is really important to this software,
(1) the users will figure out how to compile using -march=native. Typically
consider this is also done in highly-customized environments like HPC. Those
power users will anyway recompile on their own to fit their need whatever we
package.
(2) the upstream will implement them soon after being requested, if the
software remains popular while not interested in implementing so, that
means SIMD was not necessary for them.
I'm surprised those discussions were already something happened 5 years ago.
Reply to: