The LLM I used to produce that exact news report was gpt-4o-mini, from openai. ChatGPT is the name of openai's LLM web interface and its underlying LLM model name could change. It took roughly 3 minutes to perform the bulk API calls. That said, I basically implemented support for all commonly seen LLM inference services: (4 commercial ones) openai, anthropic, google, xai, (4 self-hosted) llamafile, ollama, vllm, zmq (built-in but kind of outdated.) Other services missing from the list are also supported as long as it has compatibility mode to the openai api. For the particular use case like summarizing a mailing list, self-hosted one will be much slower to respond to the bulk API call unless it is hosted on a GPU cluster :-) Small LLMs are not necessarily smart enough. The open llm leaderboard[3] is a good reference for figuring out the best open-access llm for self-hosting. In terms of "Debian hosted computer with AMD GPU for LLM inference" -- that is exactly one of the long term goals of debian deep learning team (debian-ai@l.d.o). Team members are working to prepare the ROCm packages and the ROCm version of pytorch. I find ollama[1] and llamafile[2] quite handy to use locally if do not mind using software from outside of debian archive, with a spare GPU. [1] https://github.com/ollama/ollama [2] https://github.com/Mozilla-Ocho/llamafile [3] https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard On 11/9/24 05:19, PICCA Frederic-Emmanuel wrote:
is it via ChatGPT or an llm self hosted ? Can we imagine having a Debian hosted computer with and AMD GPU dedicated to this use case ? Se should provide these summaries letter for most of our mailing list :) cheers Fred ----- Le 9 Nov 24, à 14:09, Hector Oron zumbi@debian.org a écrit :Hello Lumin, El sáb, 9 nov 2024 a las 10:27, DebGPT (<lumin@debian.org>) escribió:This is an experiment, by letting LLM go through all 369 emails from debian-devel on Oct. The command for producing the news report is included below. Use debgpt's git HEAD if you want to try.First time I see this kind of email, I thought time ago that'd be a really cool use of AI, to produce a summary of mailing lists - since I struggle to read everything. I just want to thank you for putting this together and, at least from my side, this is very much appreciated. Regards -- Héctor Orón -.. . -... .. .- -. -.. . ...- . .-.. --- .--. . .-.