This website requires JavaScript.
b30f749e5e
fix n_embd cannot be divided by quantized block size
2025-06-03 14:06:31 +04:00
e25c739ecf
Merge pull request #19 from yezhizi/feat/auto-exit
2025-05-20 02:04:50 +08:00
7b0ededd24
Merge branch 'dev' into feat/auto-exit
2025-05-20 02:04:14 +08:00
421b3deca5
fix llama-cli pos sync
2025-05-19 18:08:27 +04:00
c54a6a0132
fix context shifting
2025-05-19 16:58:35 +04:00
34eaa8224d
fix: handle socket closure and connection in llama_rebuild_topo
2025-05-16 20:48:51 +08:00
8b61cb2fa4
fix: adapt the new topo
2025-05-16 17:03:36 +08:00
df16b1876f
refactor: add zmq helper to generate message
2025-05-16 16:02:25 +08:00
0ad009a2f4
fix: update serialization and deserialization for next_ip in device_info
2025-05-16 15:26:16 +08:00
4b36aef157
fix some bugs
2025-05-15 06:25:12 +00:00
cc46aa9828
update rank and n_world
2025-05-15 13:57:16 +08:00
fdd6694633
add topo rebuild
2025-05-15 04:22:12 +00:00
26bb86c09b
Add tune_layer_allocation
2025-05-14 07:50:04 +00:00
07c4966a80
reduce fio data size to 1gb to speed up profiling
2025-05-14 21:26:01 +04:00
2cc01483fd
support server mode
2025-05-14 18:28:46 +04:00
ebd09fc83c
Merge branch 'dev'
2025-05-14 14:19:53 +04:00
258fb2d06b
add QA: How to manually profile a device
2025-05-14 14:19:20 +04:00
2fbc0c8da3
fix: reset -ngl to 0 when GPU is not used and reformat code
2025-05-14 13:27:20 +04:00
b44187e3af
Merge pull request #9 from yezhizi/rm-profile
2025-05-14 10:05:49 +08:00
e2de4511c5
Update README.md
2025-05-11 18:15:39 +08:00
ca95b1efad
Merge pull request #11 from eltociear/patch-1
2025-04-22 00:32:19 +08:00
fa1444ab3b
chore: update debug-test.sh
2025-04-21 23:40:40 +09:00
cf4fa04980
Add an independent profile tool
2025-04-18 03:27:20 +00:00
168c14f4e8
remove unnecessary profile when --lw is specified
2025-04-17 13:49:09 +00:00
9cb87f7923
add fio file to gitignore
2025-04-17 13:44:23 +00:00
fc1e2d3fc6
Added support for iq1s and iq1m quantization type
2025-04-17 10:27:53 +00:00
b212d74dc3
update Readme.md
2025-04-17 09:17:11 +00:00
f9702ec4c0
update README.md
2025-04-16 15:55:43 +04:00
5a5f103833
fix q6k and q80
2025-04-16 08:55:07 +04:00
b59d6d9cdf
Update README.md
2025-04-15 09:59:08 +08:00
6d13836c44
Update README.md
2025-04-11 01:41:43 +08:00
4845abf25e
Update README.md
2025-04-11 01:20:36 +08:00
4ab9087d69
Update LICENSE
2025-04-09 21:42:55 +08:00
e48b804730
update README.md
2025-04-09 13:55:30 +04:00
55f8dc588f
Update README.md
2025-04-09 10:56:25 +08:00
e421d788d3
update README
2025-04-08 23:15:43 +04:00
03ff9a7654
update README
2025-04-07 23:28:01 +04:00
a3a1f4499b
Update README.md
2025-04-07 22:14:44 +08:00
98d73778a6
Update README.md
2025-04-07 22:13:31 +08:00
5984b1b75f
Update README.md
2025-04-07 22:13:12 +08:00
ebd15b4112
Update README.md
2025-04-07 22:12:11 +08:00
35adc76337
Update README.md
2025-04-07 22:08:14 +08:00
87eb1aa7ec
update README
2025-04-07 18:06:57 +04:00
67b10034a7
update video to compare llama.cpp and prima.cpp
2025-04-07 18:01:04 +04:00
7631ddcdc7
ignore video
2025-04-07 17:59:23 +04:00
fffefb9259
update README
2025-04-07 17:57:57 +04:00
f97a97003b
fix type convert
2025-04-07 17:57:50 +04:00
3b264352e7
update README
2025-03-30 23:39:36 +04:00
3a6cb1768f
add logo
2025-03-30 17:21:42 +04:00
63b45a4c26
add args -k and --force
2025-03-11 22:09:39 +04:00
bcfdace59b
add args -k and --force
2025-03-11 20:44:36 +04:00
45ec52c2cb
Added support for IQ1_M and IQ2_XXS quantization type
2025-03-07 16:56:16 +00:00
230c68b80c
fixed the alignment display
2025-03-07 07:55:23 +00:00
6a416534c8
Fixed the alignment display of device performance
2025-03-07 07:46:30 +00:00
54c4c1c26e
Fixed the flops test for iq1s and q2k quantization types
2025-03-07 02:47:00 +00:00
2f049b8428
Added support for Q2K, IQ1s, IQ4NL quantization types
2025-03-04 15:22:55 +00:00
e2cda4cfa0
Removed support for GGML_TYPE_Q4_0_4_4, GGML_TYPE_0_4_8, and GGML_TYPE_0_8_8 (GGUF no longer supports these types)
2025-03-01 14:31:38 +00:00
9cbdf01645
fix support for Q5_0
2025-02-27 22:25:03 +04:00
c8e615d69c
fix n_m bound error
2025-02-27 21:59:04 +04:00
550fdcbc4f
add support for Q5_0
2025-02-27 21:47:14 +04:00
96e68679ce
fix upper bound and set calibration in halda
2025-02-27 17:00:27 +04:00
41f3708999
fix condition for gpu overload
2025-02-25 21:31:55 +04:00
224d14eb4c
Merge branch 'tao' into dev
2025-02-24 16:48:43 +00:00
42da179d66
Added parameter display for the distilled model of deepseek-qwen
2025-02-24 13:24:56 +00:00
e3a0d0007a
add gpu check in set calibration
2025-02-23 21:56:59 +04:00
7bf1b743fb
Merge branch 'dev' into lt_test
2025-02-23 08:35:45 +00:00
b4a9932d56
Added deepseek-r1-qwen vocabulary file
2025-02-23 08:33:57 +00:00
f99e08b9fe
Added inference support for the Deepseek distilled model
2025-02-23 08:27:37 +00:00
f5e874f75f
remove conda path
2025-02-23 01:38:13 +04:00
07a397360b
fix gpu underutilization
2025-02-19 16:30:18 +04:00
e219fada4e
disable timer
2025-02-19 16:24:12 +04:00
863393554a
add gpu underutilization calibration step
2025-02-17 18:54:18 +04:00
8532d030f3
fix bugs in bo
2025-02-15 18:10:11 +04:00
e64f237e04
fix bugs in available_mem calculation
2025-02-15 17:43:03 +04:00
64c4a47980
fix bugs and warnings
2025-02-15 17:33:03 +04:00
630556bc16
fix default allocation strategy to avoid OOM
2025-02-15 17:23:19 +04:00
fdfaaecd5e
disable device profiler in standalone mode
2025-02-12 17:12:30 +04:00
c84f9d29fe
use arg prefetch and remove arg unload
2025-02-12 17:04:41 +04:00
708b1d8c89
disable force fetching
2025-02-12 16:55:44 +04:00
ea0e655a8b
disable force feteching
2025-02-12 16:55:21 +04:00
b163918b46
disable prefetch in standalone mode
2025-02-12 00:17:33 +04:00
6a50d494d2
increase prefetch dense
2025-02-11 17:25:06 +04:00
65ad14140a
do not check loaded tensors due to increased latency
2025-02-11 17:10:11 +04:00
3dd3138207
ignore tensors already in page cache when prefetching
2025-02-11 17:00:17 +04:00
24974a488c
assume 10% of active pages can be compressed on macOS UMA
2025-02-11 11:06:33 +04:00
261c88f058
skip tensors on CUDA in manage_graph_tensors
2025-02-11 09:49:17 +04:00
c4c6a642fc
manage_graph_tensors: fix segment prefetch
2025-02-08 22:44:38 +04:00
d2bc5cd502
add pid as suffix to avoid conflicts with other processes
2025-02-07 10:29:22 +04:00
8e41362af0
fix prefetch
2025-02-04 17:38:41 +04:00
ec73e239c9
use 80% available mem as a conservative estimate
2025-02-03 18:10:05 +04:00
64089236eb
fix latency estimation in set m1
2025-02-03 07:56:02 +04:00
83b3d01844
fix delay estimation on macos
2025-02-01 10:37:56 +04:00
215151918f
fix gpu mem limit
2025-01-31 18:52:13 +04:00
17cd8ba618
reverse 300MiB for Metal kernel
2025-01-31 16:24:44 +04:00
dd632ee6df
ignore the first 5 evals due to preheat
2025-01-31 08:53:51 +04:00
fdecd4b54c
more active pages can be compressed
2025-01-30 23:17:07 +04:00
2bc7a56790
fix available mem estimation in termux
2025-01-30 21:23:05 +04:00
b680cb74fe
set POSIX_MADV_WILLNEED for the next subgraph
2025-01-30 13:29:34 +04:00
f9b4c46b74
ignore the first eval to make time test more accurate
2025-01-30 11:12:26 +04:00
cd758247e6
consider active pages compression in macos available memory estimation
2025-01-29 20:33:13 +04:00
Li, Zonghang