vskiwi
fc1ae90111
## Summary DeepSeek V3.2 (`DeepseekV32ForCausalLM`) is already supported by exo's inference engine (architecture whitelisted in `model_cards.py`, DSML encoding added in #1548), but **doesn't work out of the box** due to two bugs: ### Bug 1: `warmup_inference` passes empty model ID `warmup_inference()` in `generate.py` accepts `model_id: ModelId` as a parameter but creates `TextGenerationTaskParams(model=ModelId(""), ...)` instead of using it. Since `_needs_dsml_encoding()` checks `"deepseek-v3.2" in task_params.model.lower()`, the empty string never matches → falls back to `tokenizer.apply_chat_template()` → **ValueError** because V3.2 has no Jinja chat template. **Fix:** `model=ModelId("")` → `model=model_id` (one line). ### Bug 2: `_needs_dsml_encoding` limited to tool calling `_needs_dsml_encoding()` returns `True` only when `task_params.tools` is present or tool messages exist in `chat_template_messages`. For warmup and regular chat requests without tools → `return False` → Jinja fallback → **ValueError**. Unlike V3.1 (which has a `.jinja` chat template file that transformers picks up automatically), V3.2 **has no Jinja template at all** — it uses Python-based DSML encoding for all message types. **Fix:** For V3.2, always return `True` — DSML encoding handles all message types. ### Catalog cards Added inference model cards for: - `mlx-community/DeepSeek-V3.2-8bit` - `mlx-community/DeepSeek-V3.2-4bit` Parameters taken from model `config.json` on HuggingFace, storage sizes from HF API. Capabilities include `thinking_toggle` (related: #1456). ## Notes - The model ID string matching approach (`"deepseek-v3.2" in model.lower()`) is acknowledged tech debt — see #1371 for the planned architecture-based approach. ## Test plan - [x] Start exo with DeepSeek V3.2 model → warmup should complete without crash - [x] Send a regular chat message (no tools) → should get a response - [x] Send a chat message with tools → should work as before - [x] V3.2 cards should appear in the dashboard model catalog --------- Co-authored-by: user <user@m1.note> Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net> Co-authored-by: Evan <evanev7@gmail.com>
48 lines
1.0 KiB
Makefile
48 lines
1.0 KiB
Makefile
export NIX_CONFIG := "extra-experimental-features = nix-command flakes"
|
|
|
|
default: lint fmt
|
|
all: lint fmt check
|
|
|
|
fmt:
|
|
treefmt || nix fmt
|
|
|
|
lint:
|
|
uv run ruff check --fix
|
|
|
|
test:
|
|
uv run pytest src
|
|
|
|
check:
|
|
uv run basedpyright --project pyproject.toml
|
|
|
|
sync:
|
|
uv sync --all-packages
|
|
|
|
sync-clean:
|
|
uv sync --all-packages --force-reinstall --no-cache
|
|
|
|
rust-rebuild:
|
|
cargo run --bin stub_gen
|
|
uv sync --reinstall-package exo_pyo3_bindings
|
|
|
|
build-dashboard:
|
|
#!/usr/bin/env bash
|
|
cd dashboard
|
|
npm install
|
|
npm run build
|
|
|
|
package:
|
|
uv run pyinstaller packaging/pyinstaller/exo.spec
|
|
|
|
build-app: package
|
|
xcodebuild build -project app/EXO/EXO.xcodeproj -scheme EXO -configuration Debug -derivedDataPath app/EXO/build
|
|
@echo "\nBuild complete. Run with:\n open {{justfile_directory()}}/app/EXO/build/Build/Products/Debug/EXO.app"
|
|
|
|
clean:
|
|
rm -rf **/__pycache__
|
|
rm -rf target/
|
|
rm -rf .venv
|
|
rm -rf dashboard/node_modules
|
|
rm -rf dashboard/.svelte-kit
|
|
rm -rf dashboard/build
|