vskiwi
fc1ae90111
## Summary DeepSeek V3.2 (`DeepseekV32ForCausalLM`) is already supported by exo's inference engine (architecture whitelisted in `model_cards.py`, DSML encoding added in #1548), but **doesn't work out of the box** due to two bugs: ### Bug 1: `warmup_inference` passes empty model ID `warmup_inference()` in `generate.py` accepts `model_id: ModelId` as a parameter but creates `TextGenerationTaskParams(model=ModelId(""), ...)` instead of using it. Since `_needs_dsml_encoding()` checks `"deepseek-v3.2" in task_params.model.lower()`, the empty string never matches → falls back to `tokenizer.apply_chat_template()` → **ValueError** because V3.2 has no Jinja chat template. **Fix:** `model=ModelId("")` → `model=model_id` (one line). ### Bug 2: `_needs_dsml_encoding` limited to tool calling `_needs_dsml_encoding()` returns `True` only when `task_params.tools` is present or tool messages exist in `chat_template_messages`. For warmup and regular chat requests without tools → `return False` → Jinja fallback → **ValueError**. Unlike V3.1 (which has a `.jinja` chat template file that transformers picks up automatically), V3.2 **has no Jinja template at all** — it uses Python-based DSML encoding for all message types. **Fix:** For V3.2, always return `True` — DSML encoding handles all message types. ### Catalog cards Added inference model cards for: - `mlx-community/DeepSeek-V3.2-8bit` - `mlx-community/DeepSeek-V3.2-4bit` Parameters taken from model `config.json` on HuggingFace, storage sizes from HF API. Capabilities include `thinking_toggle` (related: #1456). ## Notes - The model ID string matching approach (`"deepseek-v3.2" in model.lower()`) is acknowledged tech debt — see #1371 for the planned architecture-based approach. ## Test plan - [x] Start exo with DeepSeek V3.2 model → warmup should complete without crash - [x] Send a regular chat message (no tools) → should get a response - [x] Send a chat message with tools → should work as before - [x] V3.2 cards should appear in the dashboard model catalog --------- Co-authored-by: user <user@m1.note> Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net> Co-authored-by: Evan <evanev7@gmail.com>
54 lines
1.4 KiB
Bash
Executable File
54 lines
1.4 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -euo pipefail
|
|
|
|
[ $# -lt 1 ] && {
|
|
echo "Usage: $0 host1 [host2 ...]"
|
|
exit 1
|
|
}
|
|
|
|
[ -z "$(git status --porcelain)" ] || {
|
|
echo "Uncommitted changes"
|
|
exit 1
|
|
}
|
|
|
|
upstream=$(git rev-parse --abbrev-ref --symbolic-full-name "@{u}" 2>/dev/null) || {
|
|
echo "No upstream"
|
|
exit 1
|
|
}
|
|
commit=$(git rev-parse HEAD)
|
|
remote=${upstream%%/*}
|
|
remote_installable=$(git remote get-url "$remote" | sed -E "s#^(git@github.com:|https://github\.com/)([^/]+)/([^/]+)(\.git)?\$#github:\2/\3/$commit#")
|
|
|
|
git fetch -q "$remote"
|
|
git branch -r --contains "$commit" | grep -qE "^[[:space:]]*$remote/" || {
|
|
echo "Not pushed to $remote"
|
|
exit 1
|
|
}
|
|
|
|
echo "Deploying $commit to $# hosts..."
|
|
hosts=("$@")
|
|
cleanup() {
|
|
for host in "${hosts[@]}"; do
|
|
ssh -T -o BatchMode=yes "$host@$host" "pkill -f bin/exo" &
|
|
done
|
|
wait
|
|
jobs -pr | xargs -r kill 2>/dev/null || true
|
|
}
|
|
trap 'cleanup' EXIT INT TERM
|
|
|
|
colours=($'\e[31m' $'\e[32m' $'\e[33m' $'\e[34m')
|
|
reset=$'\e[0m'
|
|
i=0
|
|
for host; do
|
|
colour=${colours[i++ % 4]}
|
|
ssh -T -o BatchMode=yes -o ServerAliveInterval=30 "$host@$host" \
|
|
"EXO_LIBP2P_NAMESPACE=$commit /nix/var/nix/profiles/default/bin/nix run $remote_installable" 2>&1 |
|
|
awk -v p="${colour}[${host}]${reset}" '{ print p $0; fflush() }' &
|
|
done
|
|
|
|
for host; do
|
|
echo "Waiting for $host..."
|
|
until curl -sf "http://$host:52415/models" &>/dev/null; do sleep 1; done
|
|
done
|
|
wait
|