ff310e7882
feat: turn the communication datatype from float32 to int8
2025-08-08 21:18:33 +08:00
7480ffbd30
Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream
2025-08-07 20:19:18 +08:00
96191d3b6f
feat: add CLI flag to control communication and computation logging
2025-08-02 12:55:12 +00:00
e00cef66ac
docs: add comprehensive CHANGES.md documenting prima.cpp enhancements
2025-08-02 12:34:56 +00:00
c949e531d5
feat: make llama-perplexity can support with multi node feature
2025-08-02 11:48:26 +00:00
e48e955e81
Merge pull request #4 from DandinPower/feat/comm_tensors_dump
...
Add tensor dumping functionality for network communication analysis
2025-07-28 18:03:52 +08:00
9bc57a10f6
feat: add tensor dumping functionality with shape information
...
- Add --dump-folder CLI argument to enable tensor dumping during network communication
- Implement binary dump format with tensor shape metadata (n_embed, n_tokens)
- Dump both send and receive tensors with unique filenames and counters
- Include proper parameter passing from CLI to llama_send_tensors/llama_recv_tensors functions
The dump format includes: element_type(1B) + n_embed(8B) + n_tokens(8B) + tensor_size(8B) + data
2025-07-28 09:57:07 +00:00
bdf9d8e74b
llama-server: fix k-shift when output overlength
2025-07-17 21:03:41 +08:00
f032680cab
Add support for speculative decoding in llama-server
...
Add support for speculative decoding in llama-server
2025-07-13 21:54:41 +08:00
86ca21e49c
server: fix bugs when running speculative decoding
2025-07-13 21:52:59 +08:00
b019a707b8
server: fix bugs
2025-07-13 13:42:24 +08:00
0cf87c8837
fix: set cache_prompt default to true
2025-07-06 10:05:24 +08:00
eb0cac1da5
fix: update the sbatch_tokens from %u to %lu to match unsigned long
2025-07-06 02:38:26 +08:00
adad23df61
feat: add more detail batch information
2025-07-05 21:40:58 +08:00
a399e49194
feat: add comm & compute log for further gantt chart analysis
2025-07-05 18:09:11 +08:00
ca5996e7a6
fix: slot id
2025-06-30 09:35:35 +00:00
b4929d510a
fix: args in speculative
2025-06-30 09:35:35 +00:00
9bf6565df4
fix: load draft model first
2025-06-30 09:35:35 +00:00
d248f3c40e
fix: some fields in cparams_draft
2025-06-30 09:35:35 +00:00
2e8e42a5ad
Add speculative decoding support to the server and command-line interfaces
2025-06-30 09:35:35 +00:00
1ea2d61a97
speedup: add arg --keep-out-in-cuda to run the output layer on CUDA
2025-06-28 10:58:18 +04:00
e8d3e5a631
update README
2025-06-27 20:16:30 +04:00
11ce0d58f7
fix compute buffer estimate: don't reverse CUDA VRAM for output layer
2025-06-27 12:42:16 +00:00
3a03549fed
update README
2025-06-26 22:37:08 +04:00
ba59a1a07a
update README
2025-06-26 22:33:28 +04:00
aacfa8a231
fix compute buffer estimate: reserve 300 MiB VRAM to avoid potential OOM
2025-06-26 20:45:45 +04:00
a05022c05a
communication: use barrier instead of manually adding delay
2025-06-26 17:30:47 +04:00
3f27a25340
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:50:58 +04:00
729870fcd7
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:47:34 +04:00
50807fd4e1
halda: handle infeasible solution with weak device
2025-06-26 08:56:31 +04:00
72701ae872
fix compute buffer estimate: reserve 200 MiB VRAM to avoid potential OOM
2025-06-24 20:39:49 +04:00
4dde8458cf
fix compute buffer estimate: reserve 100 MiB VRAM to avoid potential OOM
2025-06-24 19:29:10 +04:00
90b1079d78
fix compute_buffer estimate: remove unused memory for CUDA device
2025-06-24 16:37:16 +04:00
16ba3564ce
fix compute_buffer estimate: add context GPU usage
2025-06-24 16:09:59 +04:00
f1f7e37cdd
Merge pull request #3 from DandinPower/feat/enable-rank-specific-gguf-file-loading
...
Feat/enable rank specific gguf file loading
2025-06-23 01:33:06 +08:00
b51eb86676
doc: update documentation for multi splits feature
2025-06-22 22:58:07 +08:00
17edd6c96f
feat: remove the string_split function
2025-06-22 22:16:15 +08:00
c926088d6a
fix compute buffer estimate: test without highs
2025-06-22 16:27:55 +04:00
45e8b0420c
fix compute buffer estimate: tested on cuda
2025-06-22 08:10:57 +00:00
80e5b71b48
fix compute buffer estimate: tested on metal
2025-06-20 13:43:55 +04:00
6a898ef9d8
docs: clarify split loading usage
2025-06-19 19:55:56 +08:00
dd589561b4
improve the computing buffer estimate
2025-06-19 08:02:43 +00:00
0b4ffdfde5
Merge branch 'dev'
2025-06-17 09:40:27 +04:00
deeec668b8
fix: n_worker in draft model
...
(cherry picked from commit 921ad2b453b24b715ad5db6a703fb3df65fdcb80)
2025-06-17 13:23:20 +08:00
2b902f89bd
fix: change default ip to 127.0.0.1 & improve args for setting ports
...
fix: change default ip to `127.0.0.1` & improve args for setting ports
2025-06-17 08:23:25 +04:00
67c4f70357
fix: add log when serving as a proxy
2025-06-17 12:08:53 +08:00
6ff38b2a0c
add args: data-port and signal-port
2025-06-17 12:00:04 +08:00
104e3b2356
fix: replace localhost to 127.0.0.1
2025-06-17 11:27:58 +08:00
fbbc30c950
Merge branch 'speculative' into dev
2025-06-16 13:27:36 +04:00
dc797e94f5
Fix speculative decoding
...
Power prima.cpp with speculative decoding: Further speeds up by up to 80%
2025-06-16 12:11:12 +04:00
DandinPower