This website requires JavaScript.
003fc6d511
docs: add CLAUDE.md
2026-04-08 09:41:13 +02:00
b35e13d4f2
feat: add q2_k fast and modify q2_k logic, add preliminary support for rms_norm_aware_importance but currently assumption is wrong and will lead to corrupted result
2025-12-24 21:16:11 +08:00
eb7675f923
feat: fix the profiling logic
2025-12-17 00:33:11 +08:00
c490e38ae7
docs: update CHANGES.md for BitSqueeze integration and new threshold arg
2025-12-06 21:22:52 +08:00
73969b44b1
feat: use bitsqueeze as communication compression library to replace the previous quantization/k_quantization/sparsity. Also add a new argument call comm_compression_threshold to allow user can set a threshold for compression (good for only compress prefilling tokens)
2025-12-06 21:08:10 +08:00
f90029e3c3
doc: update doc for q2_k
2025-11-03 19:44:38 +08:00
9e1a4f2942
feat: first draft of q2_k quantization support
2025-11-03 18:24:41 +08:00
25e5dd4645
doc: update docs for sparsity features
2025-10-06 14:11:03 +08:00
9e730dd29a
fix: update the bug of the sparsity
2025-10-06 12:22:42 +08:00
3207a4e2e8
fix: fix the syntax error
2025-10-06 10:31:43 +08:00
45a78f2623
feat: first draft of f32 sparsity support
2025-10-06 10:00:50 +08:00
d63cdf13b5
doc: add documentation for communication compression
2025-08-23 22:03:51 +08:00
3a97a82f52
feat: add args to allow user can control the communication datatype
2025-08-23 13:43:14 +00:00
b780a35577
feat: refactor and fix bug for q8_0 and can easily support q4_0 and fp32 in future
2025-08-23 12:46:26 +00:00
5f40a34e67
fix: modify the Makefile to support added code files
2025-08-23 11:40:14 +00:00
2a10bca030
feat: add support for logging quantize and dequantize time
2025-08-23 19:17:22 +08:00
952a67f9a8
feat: refactor to make it can better support for future requirements (can switch using different communication type)
2025-08-23 17:54:04 +08:00
01f772b652
fix: fix the problem that zmq need pointer as input
2025-08-08 21:30:21 +08:00
ff310e7882
feat: turn the communication datatype from float32 to int8
2025-08-08 21:18:33 +08:00
7480ffbd30
Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream
2025-08-07 20:19:18 +08:00
96191d3b6f
feat: add CLI flag to control communication and computation logging
2025-08-02 12:55:12 +00:00
e00cef66ac
docs: add comprehensive CHANGES.md documenting prima.cpp enhancements
2025-08-02 12:34:56 +00:00
c949e531d5
feat: make llama-perplexity can support with multi node feature
2025-08-02 11:48:26 +00:00
e48e955e81
Merge pull request #4 from DandinPower/feat/comm_tensors_dump
2025-07-28 18:03:52 +08:00
9bc57a10f6
feat: add tensor dumping functionality with shape information
2025-07-28 09:57:07 +00:00
bdf9d8e74b
llama-server: fix k-shift when output overlength
2025-07-17 21:03:41 +08:00
f032680cab
Add support for speculative decoding in llama-server
2025-07-13 21:54:41 +08:00
86ca21e49c
server: fix bugs when running speculative decoding
2025-07-13 21:52:59 +08:00
b019a707b8
server: fix bugs
2025-07-13 13:42:24 +08:00
0cf87c8837
fix: set cache_prompt default to true
2025-07-06 10:05:01 +08:00
eb0cac1da5
fix: update the sbatch_tokens from %u to %lu to match unsigned long
2025-07-06 02:38:26 +08:00
adad23df61
feat: add more detail batch information
2025-07-05 21:40:58 +08:00
a399e49194
feat: add comm & compute log for further gantt chart analysis
2025-07-05 18:09:11 +08:00
ca5996e7a6
fix: slot id
2025-06-30 07:31:05 +00:00
b4929d510a
fix: args in speculative
2025-06-30 04:35:59 +00:00
9bf6565df4
fix: load draft model first
2025-06-27 06:30:57 +00:00
d248f3c40e
fix: some fields in cparams_draft
2025-06-27 06:07:47 +00:00
2e8e42a5ad
Add speculative decoding support to the server and command-line interfaces
2025-06-23 20:36:32 +08:00
1ea2d61a97
speedup: add arg --keep-out-in-cuda to run the output layer on CUDA
2025-06-28 05:59:19 +00:00
e8d3e5a631
update README
2025-06-27 20:16:30 +04:00
11ce0d58f7
fix compute buffer estimate: don't reverse CUDA VRAM for output layer
2025-06-27 12:42:16 +00:00
3a03549fed
update README
2025-06-26 22:37:08 +04:00
ba59a1a07a
update README
2025-06-26 22:33:28 +04:00
aacfa8a231
fix compute buffer estimate: reserve 300 MiB VRAM to avoid potential OOM
2025-06-26 20:45:45 +04:00
a05022c05a
communication: use barrier instead of manually adding delay
2025-06-26 17:30:47 +04:00
3f27a25340
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:50:58 +04:00
729870fcd7
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:47:34 +04:00
50807fd4e1
halda: handle infeasible solution with weak device
2025-06-26 08:56:31 +04:00
72701ae872
fix compute buffer estimate: reserve 200 MiB VRAM to avoid potential OOM
2025-06-24 20:39:49 +04:00
4dde8458cf
fix compute buffer estimate: reserve 100 MiB VRAM to avoid potential OOM
2025-06-24 19:29:10 +04:00
90b1079d78
fix compute_buffer estimate: remove unused memory for CUDA device
2025-06-24 16:37:16 +04:00
16ba3564ce
fix compute_buffer estimate: add context GPU usage
2025-06-24 16:09:59 +04:00
f1f7e37cdd
Merge pull request #3 from DandinPower/feat/enable-rank-specific-gguf-file-loading
2025-06-23 01:33:06 +08:00
b51eb86676
doc: update documentation for multi splits feature
2025-06-22 22:58:07 +08:00
17edd6c96f
feat: remove the string_split function
2025-06-22 22:16:15 +08:00
c926088d6a
fix compute buffer estimate: test without highs
2025-06-22 16:27:55 +04:00
45e8b0420c
fix compute buffer estimate: tested on cuda
2025-06-22 08:10:57 +00:00
80e5b71b48
fix compute buffer estimate: tested on metal
2025-06-20 13:43:55 +04:00
6a898ef9d8
docs: clarify split loading usage
2025-06-19 19:55:56 +08:00
dd589561b4
improve the computing buffer estimate
2025-06-19 08:02:43 +00:00
0b4ffdfde5
Merge branch 'dev'
2025-06-17 09:40:27 +04:00
deeec668b8
fix: n_worker in draft model
2025-06-17 13:20:06 +08:00
2b902f89bd
fix: change default ip to 127.0.0.1 & improve args for setting ports
2025-06-17 08:23:25 +04:00
67c4f70357
fix: add log when serving as a proxy
2025-06-17 12:08:53 +08:00
6ff38b2a0c
add args: data-port and signal-port
2025-06-17 12:00:04 +08:00
104e3b2356
fix: replace localhost to 127.0.0.1
2025-06-17 11:27:58 +08:00
fbbc30c950
Merge branch 'speculative' into dev
2025-06-16 13:27:36 +04:00
dc797e94f5
Fix speculative decoding
2025-06-16 12:11:12 +04:00
dfb1feb54e
update README
2025-06-16 12:09:07 +04:00
45de284f3d
Merge branch 'fix' into speculative
2025-06-14 18:57:17 +04:00
f38cfc625c
Merge branch 'fix' into dev
2025-06-14 18:56:36 +04:00
b5ccd62135
fix n_gpu_layers allocation errors
2025-06-14 18:55:53 +04:00
0a535cbdc1
Merge branch 'speculative' of github.com:Lizonghang/prima.cpp into speculative
2025-06-13 13:31:12 +04:00
c9cae626cf
speculative: free sockets and send stop signal when inference ends
2025-06-13 13:30:29 +04:00
2687ef3126
speculative: free sockets and send stop signal when inference ends
2025-06-13 11:25:42 +04:00
dc875bbef9
fix speculative decoding
2025-06-13 08:18:12 +04:00
ba29717613
add feature: keep the forwarder if its previous device cannot directly connect to its next device.
2025-06-12 16:57:35 +04:00
d4618de991
fix: block when free socket
2025-06-11 21:47:58 +08:00
2039e3b0c1
fix: send and recv meta
2025-06-11 21:05:31 +08:00
d6c8d322cd
fix try_connect
2025-06-03 15:02:59 +08:00
d1b97f798e
support reconnection
2025-05-23 10:08:30 +00:00
e50b3aa473
Merge pull request #27 from Lizonghang/lizh_dev
2025-06-11 17:12:08 +04:00
3e6d831930
fix seq_id mismatch between head and worker devices
2025-06-11 17:10:21 +04:00
1e7ae71ce5
feat: Add configurable network port options
2025-06-10 23:16:57 +08:00
fb9b1f2b00
reformat llama.cpp
2025-06-09 13:04:22 +04:00
fbf853341b
add endpoint /v1/cancel
2025-06-07 11:34:38 +04:00
c8af1be27e
Merge pull request #24 from Lizonghang/lizh_dev
2025-06-07 01:02:10 +04:00
22a6ddef13
fix batch decoding and dynamic batching
2025-06-07 00:53:56 +04:00
e56be76bdf
assume only a single seq_id per token is needed
2025-06-07 00:42:44 +04:00
d8aea899d1
fix n_seq_id and seq_id
2025-06-06 23:58:03 +04:00
a1a2238831
add batch_all.n_seq_id and batch_all.seq_id to sync_meta
2025-06-06 23:36:53 +04:00
68ecc8509d
add batch_all.logits to sync_meta
2025-06-06 22:58:48 +04:00
500e066a2f
fix batch decoding and dynamic batching
2025-06-06 16:53:22 +04:00
ef1e10101e
add test for IQ1 and doc for device selection
2025-06-04 15:12:00 +04:00
27756ee182
fix: enable rolling back set assignment when all devices are assigned to M4 but no feasible solutions
2025-06-04 15:11:29 +04:00
6439090920
reformat code
2025-06-03 23:53:24 +04:00
b6fdbd541b
Merge branch 'dev' of github.com:Lizonghang/prima.cpp into dev
2025-06-03 18:20:17 +04:00
9f0ec78a4b
Merge branch 'dev' of github.com:Lizonghang/prima.cpp into dev
2025-06-03 18:18:53 +04:00
a01fafd126
Merge branch 'main' into dev
2025-06-03 17:56:47 +04:00
1b3b6a506f
fix: add warm-up in profiling to prevent init delay
2025-06-03 17:10:09 +04:00
kxkm