b35e13d4f2
feat: add q2_k fast and modify q2_k logic, add preliminary support for rms_norm_aware_importance but currently assumption is wrong and will lead to corrupted result
2025-12-24 21:16:11 +08:00
c490e38ae7
docs: update CHANGES.md for BitSqueeze integration and new threshold arg
2025-12-06 21:22:52 +08:00
73969b44b1
feat: use bitsqueeze as communication compression library to replace the previous quantization/k_quantization/sparsity. Also add a new argument call comm_compression_threshold to allow user can set a threshold for compression (good for only compress prefilling tokens)
2025-12-06 21:08:10 +08:00
9e1a4f2942
feat: first draft of q2_k quantization support
2025-11-03 18:24:41 +08:00
45a78f2623
feat: first draft of f32 sparsity support
2025-10-06 10:00:50 +08:00
3a97a82f52
feat: add args to allow user can control the communication datatype
2025-08-23 13:43:14 +00:00
7480ffbd30
Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream
2025-08-07 20:19:18 +08:00
96191d3b6f
feat: add CLI flag to control communication and computation logging
2025-08-02 12:55:12 +00:00
9bc57a10f6
feat: add tensor dumping functionality with shape information
...
- Add --dump-folder CLI argument to enable tensor dumping during network communication
- Implement binary dump format with tensor shape metadata (n_embed, n_tokens)
- Dump both send and receive tensors with unique filenames and counters
- Include proper parameter passing from CLI to llama_send_tensors/llama_recv_tensors functions
The dump format includes: element_type(1B) + n_embed(8B) + n_tokens(8B) + tensor_size(8B) + data
2025-07-28 09:57:07 +00:00
bdf9d8e74b
llama-server: fix k-shift when output overlength
2025-07-17 21:03:41 +08:00
b019a707b8
server: fix bugs
2025-07-13 13:42:24 +08:00
b4929d510a
fix: args in speculative
2025-06-30 09:35:35 +00:00
d248f3c40e
fix: some fields in cparams_draft
2025-06-30 09:35:35 +00:00
2e8e42a5ad
Add speculative decoding support to the server and command-line interfaces
2025-06-30 09:35:35 +00:00
1ea2d61a97
speedup: add arg --keep-out-in-cuda to run the output layer on CUDA
2025-06-28 10:58:18 +04:00
a05022c05a
communication: use barrier instead of manually adding delay
2025-06-26 17:30:47 +04:00
729870fcd7
topo rebuild: add a delay to avoid packet interleaving
2025-06-26 14:47:34 +04:00
50807fd4e1
halda: handle infeasible solution with weak device
2025-06-26 08:56:31 +04:00
16ba3564ce
fix compute_buffer estimate: add context GPU usage
2025-06-24 16:09:59 +04:00
c926088d6a
fix compute buffer estimate: test without highs
2025-06-22 16:27:55 +04:00
45e8b0420c
fix compute buffer estimate: tested on cuda
2025-06-22 08:10:57 +00:00
80e5b71b48
fix compute buffer estimate: tested on metal
2025-06-20 13:43:55 +04:00
6a898ef9d8
docs: clarify split loading usage
2025-06-19 19:55:56 +08:00
dd589561b4
improve the computing buffer estimate
2025-06-19 08:02:43 +00:00
deeec668b8
fix: n_worker in draft model
...
(cherry picked from commit 921ad2b453b24b715ad5db6a703fb3df65fdcb80)
2025-06-17 13:23:20 +08:00
67c4f70357
fix: add log when serving as a proxy
2025-06-17 12:08:53 +08:00
6ff38b2a0c
add args: data-port and signal-port
2025-06-17 12:00:04 +08:00
fbbc30c950
Merge branch 'speculative' into dev
2025-06-16 13:27:36 +04:00
dfb1feb54e
update README
2025-06-16 12:09:07 +04:00
f38cfc625c
Merge branch 'fix' into dev
2025-06-14 18:56:36 +04:00
b5ccd62135
fix n_gpu_layers allocation errors
2025-06-14 18:55:53 +04:00
d4618de991
fix: block when free socket
2025-06-12 12:26:10 +00:00
2039e3b0c1
fix: send and recv meta
2025-06-12 12:26:10 +00:00
d6c8d322cd
fix try_connect
2025-06-12 12:26:10 +00:00
d1b97f798e
support reconnection
2025-06-12 12:26:09 +00:00
1e7ae71ce5
feat: Add configurable network port options
2025-06-10 23:16:57 +08:00
500e066a2f
fix batch decoding and dynamic batching
2025-06-06 16:53:22 +04:00
27756ee182
fix: enable rolling back set assignment when all devices are assigned to M4 but no feasible solutions
2025-06-04 15:11:29 +04:00
6439090920
reformat code
2025-06-03 23:53:24 +04:00
a01fafd126
Merge branch 'main' into dev
2025-06-03 17:56:47 +04:00
1b3b6a506f
fix: add warm-up in profiling to prevent init delay
2025-06-03 17:10:09 +04:00
b30f749e5e
fix n_embd cannot be divided by quantized block size
2025-06-03 14:06:31 +04:00
7b0ededd24
Merge branch 'dev' into feat/auto-exit
2025-05-20 02:04:14 +08:00
c54a6a0132
fix context shifting
2025-05-19 16:58:35 +04:00
8b61cb2fa4
fix: adapt the new topo
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:29 +00:00
0ad009a2f4
fix: update serialization and deserialization for next_ip in device_info
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:16 +00:00
4b36aef157
fix some bugs
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:08 +00:00
cc46aa9828
update rank and n_world
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:02 +00:00
fdd6694633
add topo rebuild
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:21:53 +00:00
26bb86c09b
Add tune_layer_allocation
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:21:22 +00:00
DandinPower