 Li, Zonghang
|
dc875bbef9
|
fix speculative decoding
|
2025-06-13 08:18:12 +04:00 |
|
|
Li, Zonghang
|
3e6d831930
|
fix seq_id mismatch between head and worker devices
|
2025-06-11 17:10:21 +04:00 |
|
|
Li, Zonghang
|
fb9b1f2b00
|
reformat llama.cpp
|
2025-06-09 13:04:22 +04:00 |
|
|
Li, Zonghang
|
22a6ddef13
|
fix batch decoding and dynamic batching
|
2025-06-07 00:53:56 +04:00 |
|
|
Lizonghang
|
e56be76bdf
|
assume only a single seq_id per token is needed
|
2025-06-07 00:42:44 +04:00 |
|
|
Lizonghang
|
d8aea899d1
|
fix n_seq_id and seq_id
|
2025-06-06 23:58:03 +04:00 |
|
|
Lizonghang
|
a1a2238831
|
add batch_all.n_seq_id and batch_all.seq_id to sync_meta
|
2025-06-06 23:36:53 +04:00 |
|
|
Lizonghang
|
68ecc8509d
|
add batch_all.logits to sync_meta
|
2025-06-06 22:58:48 +04:00 |
|
|
Lizonghang
|
500e066a2f
|
fix batch decoding and dynamic batching
|
2025-06-06 16:53:22 +04:00 |
|
|
Li, Zonghang
|
6439090920
|
reformat code
|
2025-06-03 23:53:24 +04:00 |
|
|
Li, Zonghang
|
a01fafd126
|
Merge branch 'main' into dev
|
2025-06-03 17:56:47 +04:00 |
|
|
Li, Zonghang
|
1b3b6a506f
|
fix: add warm-up in profiling to prevent init delay
|
2025-06-03 17:10:09 +04:00 |
|
|
Li, Zonghang
|
7b0ededd24
|
Merge branch 'dev' into feat/auto-exit
|
2025-05-20 02:04:14 +08:00 |
|
|
Lizonghang
|
421b3deca5
|
fix llama-cli pos sync
|
2025-05-19 18:08:27 +04:00 |
|
|
Lizonghang
|
c54a6a0132
|
fix context shifting
|
2025-05-19 16:58:35 +04:00 |
|
|
DeEMO
|
34eaa8224d
|
fix: handle socket closure and connection in llama_rebuild_topo
Signed-off-by: DeEMO <yzzxrx@gmail.com>
|
2025-05-19 09:22:35 +00:00 |
|
|
DeEMO
|
8b61cb2fa4
|
fix: adapt the new topo
Signed-off-by: DeEMO <yzzxrx@gmail.com>
|
2025-05-19 09:22:29 +00:00 |
|
|
DeEMO
|
df16b1876f
|
refactor: add zmq helper to generate message
Signed-off-by: DeEMO <yzzxrx@gmail.com>
|
2025-05-19 09:22:24 +00:00 |
|
|
DeEMO
|
4b36aef157
|
fix some bugs
Signed-off-by: DeEMO <yzzxrx@gmail.com>
|
2025-05-19 09:22:08 +00:00 |
|
|
DeEMO
|
cc46aa9828
|
update rank and n_world
Signed-off-by: DeEMO <yzzxrx@gmail.com>
|
2025-05-19 09:22:02 +00:00 |
|
|
DeEMO
|
fdd6694633
|
add topo rebuild
Signed-off-by: DeEMO <yzzxrx@gmail.com>
|
2025-05-19 09:21:53 +00:00 |
|
|
Lizonghang
|
2fbc0c8da3
|
fix: reset -ngl to 0 when GPU is not used and reformat code
|
2025-05-14 13:27:20 +04:00 |
|
|
DeEMO
|
168c14f4e8
|
remove unnecessary profile when --lw is specified
|
2025-04-17 13:49:09 +00:00 |
|
|
leeetao
|
fc1e2d3fc6
|
Added support for iq1s and iq1m quantization type
|
2025-04-17 10:27:53 +00:00 |
|
|
Zonghang Li
|
63b45a4c26
|
add args -k and --force
|
2025-03-11 22:09:39 +04:00 |
|
|
Zonghang Li
|
bcfdace59b
|
add args -k and --force
|
2025-03-11 20:44:36 +04:00 |
|
|
leeetao
|
45ec52c2cb
|
Added support for IQ1_M and IQ2_XXS quantization type
|
2025-03-07 16:56:16 +00:00 |
|
|
leeetao
|
6a416534c8
|
Fixed the alignment display of device performance
|
2025-03-07 07:46:30 +00:00 |
|
|
leeetao
|
2f049b8428
|
Added support for Q2K, IQ1s, IQ4NL quantization types
|
2025-03-04 15:22:55 +00:00 |
|
|
leeetao
|
e2cda4cfa0
|
Removed support for GGML_TYPE_Q4_0_4_4, GGML_TYPE_0_4_8, and GGML_TYPE_0_8_8 (GGUF no longer supports these types)
|
2025-03-01 14:31:38 +00:00 |
|
|
Lizonghang
|
9cbdf01645
|
fix support for Q5_0
|
2025-02-27 22:25:03 +04:00 |
|
|
Lizonghang
|
550fdcbc4f
|
add support for Q5_0
|
2025-02-27 21:47:14 +04:00 |
|
|
leeetao
|
42da179d66
|
Added parameter display for the distilled model of deepseek-qwen
|
2025-02-24 13:24:56 +00:00 |
|
|
leeetao
|
7bf1b743fb
|
Merge branch 'dev' into lt_test
Merge dev branch updates into local branch lt_test.
|
2025-02-23 08:35:45 +00:00 |
|
|
leeetao
|
f99e08b9fe
|
Added inference support for the Deepseek distilled model
|
2025-02-23 08:27:37 +00:00 |
|
|
Lizonghang
|
e219fada4e
|
disable timer
|
2025-02-19 16:24:12 +04:00 |
|
|
Lizonghang
|
c84f9d29fe
|
use arg prefetch and remove arg unload
|
2025-02-12 17:04:41 +04:00 |
|
|
Lizonghang
|
708b1d8c89
|
disable force fetching
|
2025-02-12 16:55:44 +04:00 |
|
|
Lizonghang
|
ea0e655a8b
|
disable force feteching
|
2025-02-12 16:55:21 +04:00 |
|
|
Lizonghang
|
b163918b46
|
disable prefetch in standalone mode
|
2025-02-12 00:17:33 +04:00 |
|
|
Lizonghang
|
6a50d494d2
|
increase prefetch dense
|
2025-02-11 17:25:06 +04:00 |
|
|
Lizonghang
|
65ad14140a
|
do not check loaded tensors due to increased latency
|
2025-02-11 17:10:11 +04:00 |
|
|
Lizonghang
|
3dd3138207
|
ignore tensors already in page cache when prefetching
|
2025-02-11 17:00:17 +04:00 |
|
|
Zonghang Li
|
261c88f058
|
skip tensors on CUDA in manage_graph_tensors
|
2025-02-11 09:49:17 +04:00 |
|
|
Zonghang Li
|
c4c6a642fc
|
manage_graph_tensors: fix segment prefetch
|
2025-02-08 22:44:38 +04:00 |
|
|
Lizonghang
|
8e41362af0
|
fix prefetch
|
2025-02-04 17:38:41 +04:00 |
|
|
Lizonghang
|
215151918f
|
fix gpu mem limit
|
2025-01-31 18:52:13 +04:00 |
|
|
Lizonghang
|
17cd8ba618
|
reverse 300MiB for Metal kernel
|
2025-01-31 16:24:44 +04:00 |
|
|
Lizonghang
|
dd632ee6df
|
ignore the first 5 evals due to preheat
|
2025-01-31 08:53:51 +04:00 |
|
|
Lizonghang
|
b680cb74fe
|
set POSIX_MADV_WILLNEED for the next subgraph
|
2025-01-30 13:29:34 +04:00 |
|