9bc57a10f6
feat: add tensor dumping functionality with shape information
...
- Add --dump-folder CLI argument to enable tensor dumping during network communication
- Implement binary dump format with tensor shape metadata (n_embed, n_tokens)
- Dump both send and receive tensors with unique filenames and counters
- Include proper parameter passing from CLI to llama_send_tensors/llama_recv_tensors functions
The dump format includes: element_type(1B) + n_embed(8B) + n_tokens(8B) + tensor_size(8B) + data
2025-07-28 09:57:07 +00:00
6a898ef9d8
docs: clarify split loading usage
2025-06-19 19:55:56 +08:00
1e7ae71ce5
feat: Add configurable network port options
2025-06-10 23:16:57 +08:00
27756ee182
fix: enable rolling back set assignment when all devices are assigned to M4 but no feasible solutions
2025-06-04 15:11:29 +04:00
6439090920
reformat code
2025-06-03 23:53:24 +04:00
a01fafd126
Merge branch 'main' into dev
2025-06-03 17:56:47 +04:00
b30f749e5e
fix n_embd cannot be divided by quantized block size
2025-06-03 14:06:31 +04:00
7b0ededd24
Merge branch 'dev' into feat/auto-exit
2025-05-20 02:04:14 +08:00
c54a6a0132
fix context shifting
2025-05-19 16:58:35 +04:00
8b61cb2fa4
fix: adapt the new topo
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:29 +00:00
4b36aef157
fix some bugs
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:08 +00:00
cc46aa9828
update rank and n_world
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:22:02 +00:00
fdd6694633
add topo rebuild
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:21:53 +00:00
26bb86c09b
Add tune_layer_allocation
...
Signed-off-by: DeEMO <yzzxrx@gmail.com >
2025-05-19 09:21:22 +00:00
2fbc0c8da3
fix: reset -ngl to 0 when GPU is not used and reformat code
2025-05-14 13:27:20 +04:00
168c14f4e8
remove unnecessary profile when --lw is specified
2025-04-17 13:49:09 +00:00
fc1e2d3fc6
Added support for iq1s and iq1m quantization type
2025-04-17 10:27:53 +00:00
f97a97003b
fix type convert
2025-04-07 17:57:50 +04:00
63b45a4c26
add args -k and --force
2025-03-11 22:09:39 +04:00
bcfdace59b
add args -k and --force
2025-03-11 20:44:36 +04:00
45ec52c2cb
Added support for IQ1_M and IQ2_XXS quantization type
2025-03-07 16:56:16 +00:00
2f049b8428
Added support for Q2K, IQ1s, IQ4NL quantization types
2025-03-04 15:22:55 +00:00
9cbdf01645
fix support for Q5_0
2025-02-27 22:25:03 +04:00
c8e615d69c
fix n_m bound error
2025-02-27 21:59:04 +04:00
550fdcbc4f
add support for Q5_0
2025-02-27 21:47:14 +04:00
96e68679ce
fix upper bound and set calibration in halda
2025-02-27 17:00:27 +04:00
41f3708999
fix condition for gpu overload
2025-02-25 21:31:55 +04:00
224d14eb4c
Merge branch 'tao' into dev
2025-02-24 16:48:43 +00:00
42da179d66
Added parameter display for the distilled model of deepseek-qwen
2025-02-24 13:24:56 +00:00
e3a0d0007a
add gpu check in set calibration
2025-02-23 21:56:59 +04:00
07a397360b
fix gpu underutilization
2025-02-19 16:30:18 +04:00
863393554a
add gpu underutilization calibration step
2025-02-17 18:54:18 +04:00
8532d030f3
fix bugs in bo
2025-02-15 18:10:11 +04:00
e64f237e04
fix bugs in available_mem calculation
2025-02-15 17:43:03 +04:00
64c4a47980
fix bugs and warnings
2025-02-15 17:33:03 +04:00
630556bc16
fix default allocation strategy to avoid OOM
2025-02-15 17:23:19 +04:00
fdfaaecd5e
disable device profiler in standalone mode
2025-02-12 17:12:30 +04:00
c84f9d29fe
use arg prefetch and remove arg unload
2025-02-12 17:04:41 +04:00
64089236eb
fix latency estimation in set m1
2025-02-03 07:56:02 +04:00
83b3d01844
fix delay estimation on macos
2025-02-01 10:37:56 +04:00
849b47ccd0
fix auto schedule logic
2025-01-29 13:13:37 +04:00
e7c6b830e6
fix auto schedule logic
2025-01-29 11:15:45 +04:00
631daadd92
test
2025-01-28 16:36:47 +04:00
36f353e374
check env path before calling fio to ensure we can find it
2025-01-28 13:06:08 +04:00
2934cf3e8e
reserve 200 mib for internal gpu usage
2025-01-27 22:14:12 +04:00
1e2b934d69
add bounds n[m]<=0 for devices without GPUs
2025-01-27 11:13:09 +04:00
1ca9a43bd1
keep the output layer weights in shared memory by default
2025-01-25 23:31:43 +04:00
f3dd5776eb
fix kappa and memory bounds, account for look-up table and input/output layer delay
2025-01-25 22:31:40 +04:00
9e4ba4f06a
fix w init error
2025-01-24 20:26:18 +04:00
1c0087e919
rename arg --keep-inp-out-in-metal to --keep-out-in-metal
2025-01-23 23:17:06 +04:00
DandinPower