c490e38ae7
docs: update CHANGES.md for BitSqueeze integration and new threshold arg
2025-12-06 21:22:52 +08:00
73969b44b1
feat: use bitsqueeze as communication compression library to replace the previous quantization/k_quantization/sparsity. Also add a new argument call comm_compression_threshold to allow user can set a threshold for compression (good for only compress prefilling tokens)
2025-12-06 21:08:10 +08:00
f90029e3c3
doc: update doc for q2_k
2025-11-03 19:44:38 +08:00
9e1a4f2942
feat: first draft of q2_k quantization support
2025-11-03 18:24:41 +08:00
25e5dd4645
doc: update docs for sparsity features
2025-10-06 14:11:03 +08:00
9e730dd29a
fix: update the bug of the sparsity
2025-10-06 12:22:42 +08:00
3207a4e2e8
fix: fix the syntax error
2025-10-06 10:31:43 +08:00
45a78f2623
feat: first draft of f32 sparsity support
2025-10-06 10:00:50 +08:00
d63cdf13b5
doc: add documentation for communication compression
2025-08-23 22:03:51 +08:00
3a97a82f52
feat: add args to allow user can control the communication datatype
2025-08-23 13:43:14 +00:00
b780a35577
feat: refactor and fix bug for q8_0 and can easily support q4_0 and fp32 in future
2025-08-23 12:46:26 +00:00
5f40a34e67
fix: modify the Makefile to support added code files
2025-08-23 11:40:14 +00:00
2a10bca030
feat: add support for logging quantize and dequantize time
2025-08-23 19:17:22 +08:00
952a67f9a8
feat: refactor to make it can better support for future requirements (can switch using different communication type)
2025-08-23 17:54:04 +08:00
01f772b652
fix: fix the problem that zmq need pointer as input
2025-08-08 21:30:21 +08:00
ff310e7882
feat: turn the communication datatype from float32 to int8
2025-08-08 21:18:33 +08:00
7480ffbd30
Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream
2025-08-07 20:19:18 +08:00
96191d3b6f
feat: add CLI flag to control communication and computation logging
2025-08-02 12:55:12 +00:00
e00cef66ac
docs: add comprehensive CHANGES.md documenting prima.cpp enhancements
2025-08-02 12:34:56 +00:00
c949e531d5
feat: make llama-perplexity can support with multi node feature
2025-08-02 11:48:26 +00:00
e48e955e81
Merge pull request #4 from DandinPower/feat/comm_tensors_dump
...
Add tensor dumping functionality for network communication analysis
2025-07-28 18:03:52 +08:00
9bc57a10f6
feat: add tensor dumping functionality with shape information
...
- Add --dump-folder CLI argument to enable tensor dumping during network communication
- Implement binary dump format with tensor shape metadata (n_embed, n_tokens)
- Dump both send and receive tensors with unique filenames and counters
- Include proper parameter passing from CLI to llama_send_tensors/llama_recv_tensors functions
The dump format includes: element_type(1B) + n_embed(8B) + n_tokens(8B) + tensor_size(8B) + data
2025-07-28 09:57:07 +00:00
bdf9d8e74b
llama-server: fix k-shift when output overlength
2025-07-17 21:03:41 +08:00
f032680cab
Add support for speculative decoding in llama-server
...
Add support for speculative decoding in llama-server
2025-07-13 21:54:41 +08:00
86ca21e49c
server: fix bugs when running speculative decoding
2025-07-13 21:52:59 +08:00
b019a707b8
server: fix bugs
2025-07-13 13:42:24 +08:00
0cf87c8837
fix: set cache_prompt default to true
2025-07-06 10:05:24 +08:00
eb0cac1da5
fix: update the sbatch_tokens from %u to %lu to match unsigned long
2025-07-06 02:38:26 +08:00
adad23df61
feat: add more detail batch information
2025-07-05 21:40:58 +08:00
a399e49194
feat: add comm & compute log for further gantt chart analysis
2025-07-05 18:09:11 +08:00
ca5996e7a6
fix: slot id
2025-06-30 09:35:35 +00:00
b4929d510a
fix: args in speculative
2025-06-30 09:35:35 +00:00
9bf6565df4
fix: load draft model first
2025-06-30 09:35:35 +00:00
d248f3c40e
fix: some fields in cparams_draft
2025-06-30 09:35:35 +00:00
2e8e42a5ad
Add speculative decoding support to the server and command-line interfaces
2025-06-30 09:35:35 +00:00
1ea2d61a97
speedup: add arg --keep-out-in-cuda to run the output layer on CUDA
2025-06-28 10:58:18 +04:00
e8d3e5a631
update README
2025-06-27 20:16:30 +04:00
11ce0d58f7
fix compute buffer estimate: don't reverse CUDA VRAM for output layer
2025-06-27 12:42:16 +00:00
3a03549fed
update README
2025-06-26 22:37:08 +04:00
ba59a1a07a
update README
2025-06-26 22:33:28 +04:00
aacfa8a231
fix compute buffer estimate: reserve 300 MiB VRAM to avoid potential OOM
2025-06-26 20:45:45 +04:00
a05022c05a
communication: use barrier instead of manually adding delay
2025-06-26 17:30:47 +04:00
3f27a25340
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:50:58 +04:00
729870fcd7
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:47:34 +04:00
50807fd4e1
halda: handle infeasible solution with weak device
2025-06-26 08:56:31 +04:00
72701ae872
fix compute buffer estimate: reserve 200 MiB VRAM to avoid potential OOM
2025-06-24 20:39:49 +04:00
4dde8458cf
fix compute buffer estimate: reserve 100 MiB VRAM to avoid potential OOM
2025-06-24 19:29:10 +04:00
90b1079d78
fix compute_buffer estimate: remove unused memory for CUDA device
2025-06-24 16:37:16 +04:00
16ba3564ce
fix compute_buffer estimate: add context GPU usage
2025-06-24 16:09:59 +04:00
f1f7e37cdd
Merge pull request #3 from DandinPower/feat/enable-rank-specific-gguf-file-loading
...
Feat/enable rank specific gguf file loading
2025-06-23 01:33:06 +08:00
DandinPower