 Li, Zonghang
|
729870fcd7
|
topo rebuild: add a delay to avoid packet interleaving
|
2025-06-26 14:47:34 +04:00 |
|
|
Li, Zonghang
|
50807fd4e1
|
halda: handle infeasible solution with weak device
|
2025-06-26 08:56:31 +04:00 |
|
|
Li, Zonghang
|
72701ae872
|
fix compute buffer estimate: reserve 200 MiB VRAM to avoid potential OOM
|
2025-06-24 20:39:49 +04:00 |
|
|
Li, Zonghang
|
4dde8458cf
|
fix compute buffer estimate: reserve 100 MiB VRAM to avoid potential OOM
|
2025-06-24 19:29:10 +04:00 |
|
|
Li, Zonghang
|
90b1079d78
|
fix compute_buffer estimate: remove unused memory for CUDA device
|
2025-06-24 16:37:16 +04:00 |
|
|
Li, Zonghang
|
16ba3564ce
|
fix compute_buffer estimate: add context GPU usage
|
2025-06-24 16:09:59 +04:00 |
|
|
Li, Zonghang
|
c926088d6a
|
fix compute buffer estimate: test without highs
|
2025-06-22 16:27:55 +04:00 |
|
|
Zonghang Li
|
45e8b0420c
|
fix compute buffer estimate: tested on cuda
|
2025-06-22 08:10:57 +00:00 |
|
|
Li, Zonghang
|
80e5b71b48
|
fix compute buffer estimate: tested on metal
|
2025-06-20 13:43:55 +04:00 |
|
|
Zonghang Li
|
dd589561b4
|
improve the computing buffer estimate
|
2025-06-19 08:02:43 +00:00 |
|
|
Li, Zonghang
|
0b4ffdfde5
|
Merge branch 'dev'
|
2025-06-17 09:40:27 +04:00 |
|
|
DeEMO
|
deeec668b8
|
fix: n_worker in draft model
(cherry picked from commit 921ad2b453b24b715ad5db6a703fb3df65fdcb80)
|
2025-06-17 13:23:20 +08:00 |
|
|
Zonghang Li
|
2b902f89bd
|
fix: change default ip to 127.0.0.1 & improve args for setting ports
fix: change default ip to `127.0.0.1` & improve args for setting ports
|
2025-06-17 08:23:25 +04:00 |
|
|
DeEMO
|
67c4f70357
|
fix: add log when serving as a proxy
|
2025-06-17 12:08:53 +08:00 |
|
|
DeEMO
|
6ff38b2a0c
|
add args: data-port and signal-port
|
2025-06-17 12:00:04 +08:00 |
|
|
DeEMO
|
104e3b2356
|
fix: replace localhost to 127.0.0.1
|
2025-06-17 11:27:58 +08:00 |
|
|
Li, Zonghang
|
fbbc30c950
|
Merge branch 'speculative' into dev
|
2025-06-16 13:27:36 +04:00 |
|
|
Zonghang Li
|
dc797e94f5
|
Fix speculative decoding
Power prima.cpp with speculative decoding: Further speeds up by up to 80%
|
2025-06-16 12:11:12 +04:00 |
|
|
Li, Zonghang
|
dfb1feb54e
|
update README
|
2025-06-16 12:09:07 +04:00 |
|
|
Li, Zonghang
|
45de284f3d
|
Merge branch 'fix' into speculative
|
2025-06-14 18:57:17 +04:00 |
|
|
Li, Zonghang
|
f38cfc625c
|
Merge branch 'fix' into dev
|
2025-06-14 18:56:36 +04:00 |
|
|
Li, Zonghang
|
b5ccd62135
|
fix n_gpu_layers allocation errors
|
2025-06-14 18:55:53 +04:00 |
|
|
Li, Zonghang
|
0a535cbdc1
|
Merge branch 'speculative' of github.com:Lizonghang/prima.cpp into speculative
|
2025-06-13 13:31:12 +04:00 |
|
|
Li, Zonghang
|
c9cae626cf
|
speculative: free sockets and send stop signal when inference ends
|
2025-06-13 13:30:29 +04:00 |
|
|
Li, Zonghang
|
2687ef3126
|
speculative: free sockets and send stop signal when inference ends
|
2025-06-13 11:25:42 +04:00 |
|
|
Li, Zonghang
|
dc875bbef9
|
fix speculative decoding
|
2025-06-13 08:18:12 +04:00 |
|
|
Zonghang Li
|
ba29717613
|
add feature: keep the forwarder if its previous device cannot directly connect to its next device.
feat: nodes attempt connections during topology rebuild while preserving forwarders
|
2025-06-12 16:57:35 +04:00 |
|
|
DeEMO
|
d4618de991
|
fix: block when free socket
|
2025-06-12 12:26:10 +00:00 |
|
|
DeEMO
|
2039e3b0c1
|
fix: send and recv meta
|
2025-06-12 12:26:10 +00:00 |
|
|
DeEMO
|
d6c8d322cd
|
fix try_connect
|
2025-06-12 12:26:10 +00:00 |
|
|
DeEMO
|
d1b97f798e
|
support reconnection
|
2025-06-12 12:26:09 +00:00 |
|
|
Zonghang Li
|
e50b3aa473
|
Merge pull request #27 from Lizonghang/lizh_dev
Fix seq_id mismatch between the head and worker devices.
|
2025-06-11 17:12:08 +04:00 |
|
|
Li, Zonghang
|
3e6d831930
|
fix seq_id mismatch between head and worker devices
|
2025-06-11 17:10:21 +04:00 |
|
|
Li, Zonghang
|
fb9b1f2b00
|
reformat llama.cpp
|
2025-06-09 13:04:22 +04:00 |
|
|
Li, Zonghang
|
fbf853341b
|
add endpoint /v1/cancel
|
2025-06-07 11:34:38 +04:00 |
|
|
Zonghang Li
|
c8af1be27e
|
Merge pull request #24 from Lizonghang/lizh_dev
Fix batch decoding and dynamic batching.
|
2025-06-07 01:02:10 +04:00 |
|
|
Li, Zonghang
|
22a6ddef13
|
fix batch decoding and dynamic batching
|
2025-06-07 00:53:56 +04:00 |
|
|
Lizonghang
|
e56be76bdf
|
assume only a single seq_id per token is needed
|
2025-06-07 00:42:44 +04:00 |
|
|
Lizonghang
|
d8aea899d1
|
fix n_seq_id and seq_id
|
2025-06-06 23:58:03 +04:00 |
|
|
Lizonghang
|
a1a2238831
|
add batch_all.n_seq_id and batch_all.seq_id to sync_meta
|
2025-06-06 23:36:53 +04:00 |
|
|
Lizonghang
|
68ecc8509d
|
add batch_all.logits to sync_meta
|
2025-06-06 22:58:48 +04:00 |
|
|
Lizonghang
|
500e066a2f
|
fix batch decoding and dynamic batching
|
2025-06-06 16:53:22 +04:00 |
|
|
Lizonghang
|
ef1e10101e
|
add test for IQ1 and doc for device selection
|
2025-06-04 15:12:00 +04:00 |
|
|
Lizonghang
|
27756ee182
|
fix: enable rolling back set assignment when all devices are assigned to M4 but no feasible solutions
|
2025-06-04 15:11:29 +04:00 |
|
|
Li, Zonghang
|
6439090920
|
reformat code
|
2025-06-03 23:53:24 +04:00 |
|
|
Li, Zonghang
|
b6fdbd541b
|
Merge branch 'dev' of github.com:Lizonghang/prima.cpp into dev
|
2025-06-03 18:20:17 +04:00 |
|
|
Lizonghang
|
9f0ec78a4b
|
Merge branch 'dev' of github.com:Lizonghang/prima.cpp into dev
|
2025-06-03 18:18:53 +04:00 |
|
|
Li, Zonghang
|
a01fafd126
|
Merge branch 'main' into dev
|
2025-06-03 17:56:47 +04:00 |
|
|
Li, Zonghang
|
1b3b6a506f
|
fix: add warm-up in profiling to prevent init delay
|
2025-06-03 17:10:09 +04:00 |
|
|
Li, Zonghang
|
b30f749e5e
|
fix n_embd cannot be divided by quantized block size
|
2025-06-03 14:06:31 +04:00 |
|