73969b44b1
feat: use bitsqueeze as communication compression library to replace the previous quantization/k_quantization/sparsity. Also add a new argument call comm_compression_threshold to allow user can set a threshold for compression (good for only compress prefilling tokens)
2025-12-06 21:08:10 +08:00
9e1a4f2942
feat: first draft of q2_k quantization support
2025-11-03 18:24:41 +08:00
45a78f2623
feat: first draft of f32 sparsity support
2025-10-06 10:00:50 +08:00
5f40a34e67
fix: modify the Makefile to support added code files
2025-08-23 11:40:14 +00:00
7480ffbd30
Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream
2025-08-07 20:19:18 +08:00
c949e531d5
feat: make llama-perplexity can support with multi node feature
2025-08-02 11:48:26 +00:00
2e8e42a5ad
Add speculative decoding support to the server and command-line interfaces
2025-06-30 09:35:35 +00:00
fbbc30c950
Merge branch 'speculative' into dev
2025-06-16 13:27:36 +04:00
dc875bbef9
fix speculative decoding
2025-06-13 08:18:12 +04:00
d6c8d322cd
fix try_connect
2025-06-12 12:26:10 +00:00
6439090920
reformat code
2025-06-03 23:53:24 +04:00
c54a6a0132
fix context shifting
2025-05-19 16:58:35 +04:00
2cc01483fd
support server mode
2025-05-14 18:28:46 +04:00
cf4fa04980
Add an independent profile tool
2025-04-18 03:27:20 +00:00
f5e874f75f
remove conda path
2025-02-23 01:38:13 +04:00
2d649c808d
add /usr/local/bin on darwin
2025-01-28 13:05:25 +04:00
dab6b2e1c2
fix option USE_HIGHS
2025-01-15 20:50:36 +04:00
18c96e8042
add option USE_HIGHS
2025-01-15 20:05:49 +04:00
9bead4291c
fix path not found
2025-01-15 16:55:07 +04:00
106ff79f08
include conda path only on Linux (if installing highs on Linux via conda)
2025-01-15 13:02:52 +04:00
73ed4dec68
add conda path to find highs
2025-01-15 12:16:38 +04:00
5d9aadf3d5
use highs to solve the allocation program
2025-01-15 10:04:04 +04:00
5fae6ac36f
add cpu flops test
2024-11-09 20:53:42 +04:00
ef7fdf70cc
add LLAMA_API llama_profile_device
2024-11-07 09:30:39 +04:00
407c71ae52
add cpu and gpu profile
2024-11-06 20:42:28 +04:00
9cd66f2145
add profiler
2024-11-05 20:29:09 +04:00
766ec7862b
test
2024-11-05 17:22:24 +04:00
2a01ff5fb1
init
2024-10-23 09:42:32 +04:00
c83ad6d01e
ggml-backend : add device and backend reg interfaces ( #9707 )
...
Co-authored-by: Johannes Gäßler <johannesg@5d6.de >
2024-10-03 01:49:47 +02:00
148844fe97
examples : remove benchmark ( #9704 )
...
ggml-ci
2024-10-02 10:14:44 +03:00
c35e586ea5
musa: enable building fat binaries, enable unified memory, and disable Flash Attention on QY1 (MTT S80) ( #9526 )
...
* mtgpu: add mp_21 support
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* mtgpu: disable flash attention on qy1 (MTT S80); disable q3_k and mul_mat_batched_cublas
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* mtgpu: enable unified memory
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* mtgpu: map cublasOperation_t to mublasOperation_t (sync code to latest)
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
---------
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2024-09-22 16:55:49 +02:00
19514d632e
cmake : do not hide GGML options + rename option ( #9465 )
...
* cmake : do not hide GGML options
ggml-ci
* build : rename flag GGML_CUDA_USE_GRAPHS -> GGML_CUDA_GRAPHS
for consistency
ggml-ci
2024-09-16 10:27:50 +03:00
6262d13e0b
common : reimplement logging ( #9418 )
...
https://github.com/ggerganov/llama.cpp/pull/9418
2024-09-15 20:46:12 +03:00
feff4aa846
server : add loading html page while model is loading ( #9468 )
...
* Adding loading page for '/' server requests
* set content when model is loading
* removed loading html file
* updated cmakelist
* updated makefile
* cleaned up whitespace
* cleanup for PR removed error
* updated server test to handle 503 HTML
* updated server test to handle 503 HTML
* ca†ch 503 before parsing json
* revert test
* account for both api and web browser requests
* precommit corrections
* eol fix
* revert changes to pre-commit
* removed print statement
* made loading message more descriptive
* also support .html files
---------
Co-authored-by: VJHack <flymyplane21@gmail.com >
Co-authored-by: Vinesh Janarthanan <36610342+VJHack@users.noreply.github.com >
2024-09-13 14:23:11 +02:00
2b00fa7997
riscv : modify Makefile and add a RISCV_VECT to print log info ( #9442 )
...
- Added ggml_cpu_has_riscv_v() in GGML to print system info in log
- Modified Makefile to only use flag when cross compiling for RISC-V
2024-09-12 14:24:31 +03:00
fb3f249815
make : do not run llama-gen-docs when building ( #9399 )
2024-09-10 09:23:33 +03:00
bfe76d4a17
common : move arg parser code to arg.cpp ( #9388 )
...
* common : move arg parser to arg.cpp
* better categorize args
* add cmake
* missing climits
* missing cstdarg
* common : more explicit includes
* fix build
* refactor gpt_params_parse
* update server readme
* fix test
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-09-09 23:36:09 +02:00
1b9ae5189c
common : refactor arg parser ( #9308 )
...
* (wip) argparser v3
* migrated
* add test
* handle env
* fix linux build
* add export-docs example
* fix build (2)
* skip build test-arg-parser on windows
* update server docs
* bring back missing --alias
* bring back --n-predict
* clarify test-arg-parser
* small correction
* add comments
* fix args with 2 values
* refine example-specific args
* no more lamba capture
Co-authored-by: slaren@users.noreply.github.com
* params.sparams
* optimize more
* export-docs --> gen-docs
2024-09-07 20:43:51 +02:00
df270ef745
llama : refactor sampling v2 ( #9294 )
...
- Add `struct llama_sampler` and `struct llama_sampler_i`
- Add `llama_sampler_` API
- Add `llama_sampler_chain_` API for chaining multiple samplers
- Remove `LLAMA_API_INTERNAL`
- Add `llama_perf_` API and remove old `llama_print_timings` and `llama_reset_timings`
2024-09-07 15:16:19 +03:00
5fd89a70ea
Vulkan Optimizations and Fixes ( #8959 )
...
* Optimize Vulkan REPEAT performance
* Use Vulkan GLSL fused multiply-add instruction where possible
* Add GGML_VULKAN_PERF option to output performance data per operator
* Rework and fix Vulkan descriptor set and descriptor pool handling
* Fix float32 concat f16 shader validation error
* Add Vulkan GROUP_NORM eps parameter
* Fix validation error with transfer queue memory barrier flags
* Remove trailing whitespaces
2024-08-14 18:32:53 +02:00
272e3bd95e
make : fix llava obj file race ( #8946 )
...
ggml-ci
2024-08-09 18:24:30 +03:00
3071c0a5f2
llava : support MiniCPM-V-2.5 ( #7599 )
...
* init
* rename
* add run android for termux in readme
* add android readme
* add instructions in readme
* change name in readme
* Update README.md
* fixed line
* add result in readme
* random pos_embed
* add positions index
* change for ollama
* change for ollama
* better pos_embed in clip
* support ollama
* updata cmakelist
* updata cmakelist
* rename wrapper
* clear code
* replace and organize code
* add link
* sync master
* fix warnings
* fix warnings
* fix bug in bicubic resize when need resize iamge smaller
* receive review comments and modify
* receive review comments and modify
* put all code into llava dir
* fix quality problem in pr code
* change n_layer
* add space in "-1"
* imitate reshape bug of python code
* fix bug in clip
* fix issues for merging
* fix llama-minicpmv-cli in cmake file
* change pr readme
* fix code review
* remove in line 33 directory in the /cmakelists.txt (not in example, in the main dir
* fix cmakefile
* add warn
* fix KEY_HAS_MINICPMV_PROJ
* remove load_image_size into clip_ctx
* remove the extern "C", MINICPMV_API
* fix uhd code for review comment
* delete minicpmv-wrapper in pr
* remove uhd_image_embed
* Modify 2 notes
* clip : style changes
* del common.h in clip
* fix Type-Check error
* fix Type-Check error
* fix Type-Check error
* fix Type-Check error
* fix makefile error
* fix ubuntu-make error
* try fix clip
* try fix 1
---------
Co-authored-by: Hongji Zhu <fireyoucan@gmail.com >
Co-authored-by: harvestingmoon <leewenyeong@gmail.com >
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com >
2024-08-09 13:33:53 +03:00
ebd541a570
make : clean llamafile objects ( #8923 )
...
`ggml/src/llamafile/sgemm.o` was not deleted on `make clean`
2024-08-08 11:44:51 +03:00
15fa07a5c5
make : use C compiler to build metal embed object ( #8899 )
...
* make : use C compiler to build metal embed object
* use rm + rmdir to avoid -r flag in rm
2024-08-07 18:24:05 +02:00
ed9d2854c9
Build: Fix potential race condition ( #8781 )
...
* Fix potential race condition as pointed out by @fairydreaming in #8776
* Reference the .o rather than rebuilding every time.
* Adding in CXXFLAGS and LDFLAGS
* Removing unnecessary linker flags.
2024-07-31 15:51:06 -04:00
e54c35e4fb
feat: Support Moore Threads GPU ( #8383 )
...
* Update doc for MUSA
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* Add GGML_MUSA in Makefile
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* Add GGML_MUSA in CMake
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* CUDA => MUSA
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* MUSA adds support for __vsubss4
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
* Fix CI build failure
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
---------
Signed-off-by: Xiaodong Ye <xiaodong.ye@mthreads.com >
2024-07-28 01:41:25 +02:00
2b1f616b20
ggml : reduce hash table reset cost ( #8698 )
...
* ggml : reduce hash table reset cost
* fix unreachable code warnings after GGML_ASSERT(false)
* GGML_ASSERT(false) -> GGML_ABORT("fatal error")
* GGML_ABORT use format string
2024-07-27 04:41:55 +02:00
be6d7c0791
examples : remove finetune and train-text-from-scratch ( #8669 )
...
* examples : remove finetune and train-text-from-scratch
* fix build
* update help message
* fix small typo for export-lora
2024-07-25 10:39:04 +02:00
de280085e7
examples : Fix llama-export-lora example ( #8607 )
...
* fix export-lora example
* add more logging
* reject merging subset
* better check
* typo
2024-07-23 23:48:37 +02:00
938943cdbf
llama : move vocab, grammar and sampling into separate files ( #8508 )
...
* llama : move sampling code into llama-sampling
ggml-ci
* llama : move grammar code into llama-grammar
ggml-ci
* cont
ggml-ci
* cont : pre-fetch rules
* cont
ggml-ci
* llama : deprecate llama_sample_grammar
* llama : move tokenizers into llama-vocab
ggml-ci
* make : update llama.cpp deps [no ci]
* llama : redirect external API to internal APIs
ggml-ci
* llama : suffix the internal APIs with "_impl"
ggml-ci
* llama : clean-up
2024-07-23 13:10:17 +03:00
DandinPower