b35e13d4f2
feat: add q2_k fast and modify q2_k logic, add preliminary support for rms_norm_aware_importance but currently assumption is wrong and will lead to corrupted result
2025-12-24 21:16:11 +08:00
73969b44b1
feat: use bitsqueeze as communication compression library to replace the previous quantization/k_quantization/sparsity. Also add a new argument call comm_compression_threshold to allow user can set a threshold for compression (good for only compress prefilling tokens)
2025-12-06 21:08:10 +08:00
45a78f2623
feat: first draft of f32 sparsity support
2025-10-06 10:00:50 +08:00
3a97a82f52
feat: add args to allow user can control the communication datatype
2025-08-23 13:43:14 +00:00
7480ffbd30
Merge remote-tracking branch 'upstream/main' into merge/merge_86ca21e_from_upstream
2025-08-07 20:19:18 +08:00
96191d3b6f
feat: add CLI flag to control communication and computation logging
2025-08-02 12:55:12 +00:00
9bc57a10f6
feat: add tensor dumping functionality with shape information
...
- Add --dump-folder CLI argument to enable tensor dumping during network communication
- Implement binary dump format with tensor shape metadata (n_embed, n_tokens)
- Dump both send and receive tensors with unique filenames and counters
- Include proper parameter passing from CLI to llama_send_tensors/llama_recv_tensors functions
The dump format includes: element_type(1B) + n_embed(8B) + n_tokens(8B) + tensor_size(8B) + data
2025-07-28 09:57:07 +00:00
1ea2d61a97
speedup: add arg --keep-out-in-cuda to run the output layer on CUDA
2025-06-28 10:58:18 +04:00
45e8b0420c
fix compute buffer estimate: tested on cuda
2025-06-22 08:10:57 +00:00
80e5b71b48
fix compute buffer estimate: tested on metal
2025-06-20 13:43:55 +04:00
6a898ef9d8
docs: clarify split loading usage
2025-06-19 19:55:56 +08:00
dd589561b4
improve the computing buffer estimate
2025-06-19 08:02:43 +00:00
6ff38b2a0c
add args: data-port and signal-port
2025-06-17 12:00:04 +08:00
fbbc30c950
Merge branch 'speculative' into dev
2025-06-16 13:27:36 +04:00
dc875bbef9
fix speculative decoding
2025-06-13 08:18:12 +04:00
2039e3b0c1
fix: send and recv meta
2025-06-12 12:26:10 +00:00
d6c8d322cd
fix try_connect
2025-06-12 12:26:10 +00:00
d1b97f798e
support reconnection
2025-06-12 12:26:09 +00:00
3e6d831930
fix seq_id mismatch between head and worker devices
2025-06-11 17:10:21 +04:00
1e7ae71ce5
feat: Add configurable network port options
2025-06-10 23:16:57 +08:00
6439090920
reformat code
2025-06-03 23:53:24 +04:00
7b0ededd24
Merge branch 'dev' into feat/auto-exit
2025-05-20 02:04:14 +08:00
c54a6a0132
fix context shifting
2025-05-19 16:58:35 +04:00
cc46aa9828
update rank and n_world
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:02 +00:00
fdd6694633
add topo rebuild
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:21:53 +00:00
2fbc0c8da3
fix: reset -ngl to 0 when GPU is not used and reformat code
2025-05-14 13:27:20 +04:00
168c14f4e8
remove unnecessary profile when --lw is specified
2025-04-17 13:49:09 +00:00
fc1e2d3fc6
Added support for iq1s and iq1m quantization type
2025-04-17 10:27:53 +00:00
bcfdace59b
add args -k and --force
2025-03-11 20:44:36 +04:00
e2cda4cfa0
Removed support for GGML_TYPE_Q4_0_4_4, GGML_TYPE_0_4_8, and GGML_TYPE_0_8_8 (GGUF no longer supports these types)
2025-03-01 14:31:38 +00:00
7bf1b743fb
Merge branch 'dev' into lt_test
...
Merge dev branch updates into local branch lt_test.
2025-02-23 08:35:45 +00:00
f99e08b9fe
Added inference support for the Deepseek distilled model
2025-02-23 08:27:37 +00:00
c84f9d29fe
use arg prefetch and remove arg unload
2025-02-12 17:04:41 +04:00
1c0087e919
rename arg --keep-inp-out-in-metal to --keep-out-in-metal
2025-01-23 23:17:06 +04:00
78a544d716
add metal mem limit
2025-01-23 16:08:52 +04:00
facb4ea736
add option --keep-inp-out-in-metal and fix bugs in unmap
2025-01-22 11:15:19 +04:00
46e99218b4
add arg --cuda-mem
2025-01-16 09:15:34 +04:00
3d75b8576e
add api llama_model_set_n_gpu_layers
2025-01-15 10:48:19 +04:00
9279a2e3ff
fix error in llama_context_n_gpu_layers
2025-01-15 10:08:41 +04:00
5d9aadf3d5
use highs to solve the allocation program
2025-01-15 10:04:04 +04:00
b577c10d25
add noreturn flag for monitor
2025-01-07 20:38:11 +04:00
8e9ab45458
fix model bytes counter
2024-12-10 14:57:48 +04:00
d78fa427e7
add memory copy speed test
2024-12-09 10:07:42 +04:00
df813675d0
fix flops count and ram/vram speed test
2024-12-08 10:14:05 +04:00
cd823546dd
llama_profile_device: add arg n_predict
2024-12-06 16:37:25 +04:00
6f54a12c7d
add gpu support in llama_model_kvcache_size and llama_model_compute_buf_size
2024-11-29 21:06:32 +04:00
68ecabc8c3
add cpu_read_ram_bw, metal_read_vram_bw, cuda_read_vram_bw
2024-11-29 19:04:53 +04:00
0f73d12247
decrease compute buf from available memory
2024-11-29 11:15:54 +04:00
45a1e55eec
reduce kv cache from available memory
2024-11-28 20:21:21 +04:00
9a7bbce7ad
fix t_load_us
2024-11-28 15:55:21 +04:00
DandinPower