Speculative decoding:
- speculative/: MTPPredictor, MTPBatchGenerator, SpeculativeArraysCache,
speculative GDN kernel with per-step state output
- Correct GDN state + conv state rollback on draft rejection
- Proper rejection sampling with residual distribution correction
- Lazy drafting + single async_eval per cycle
- MTP prefill in submit() with correct token alignment
- Speculative warmup at startup
- EXO_SPECULATIVE=1, EXO_MTP_WEIGHTS, EXO_SPECULATIVE_GAMMA,
EXO_SPECULATIVE_TEMP, EXO_SPECULATIVE_ALPHA env vars
- EXO_DISABLE_LOGPROBS=1 to skip per-token logprobs extraction
Kernel patches:
- qwen3_5_moe/: batched fused kernels for 35B-A3B MoE decode
- qwen3_5/: LpB GEMV kernels for 27B/9B dense models
- Auto-detected by model_type, EXO_FUSED_KERNELS=0 to disable
- maybe_apply_patches() called after model load in utils_mlx.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Motivation
Nemotron Cascade and Nano failing at long decodes.
## Changes
Fixed upstream, just change pyproject and uv lock here.
## Test Plan
### Automated Testing
Tested with a reproduce script upstream
## Motivation
Addresses #1816
## Changes
Update on min prefix cache > min_prefix_hit_length **and** hit ratio >
_MIN_PREFIX_HIT_RATIO_TO_UPDATE
min_prefix_hit_length = max(1000, system prompt length) -> system
prompts must match exactly.
## Test Plan
### Manual Testing
Test on OpenCode and Claude Code
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
adds a path option to the /state endpoint, allowing you to query
subfields of state without grabbing the whole blob
## test plan
poking around in the api
This PR builds on https://github.com/exo-explore/exo/pull/1677 to enable
custom prompts sent from Firefox `browser.ml.chat` to EXO dashboard
using URL parameters in sidebar for summary and other browser
interactions. See "Summarize page" example below.
## Summary
- Parse `?q=<encoded prompt>` URL parameter on page load and auto-submit
it as a chat message
- Clean up the URL with `history.replaceState` to prevent re-submission
on refresh
- Defer auto-send until both cluster state and model list are loaded so
model auto-selection works correctly
## Context
Firefox's built-in AI sidebar (`about:config: browser.ml.chat.enabled`)
integrates with chat providers by appending the user's prompt as
`?q=<URL-encoded prompt>`. Previously the exo dashboard ignored this
parameter. Users can now configure `http://localhost:52415` as a Firefox
AI chatbot provider.
See: https://support.mozilla.org/en-US/kb/ai-chatbot
## Technical notes
- Frontend-only change in `dashboard/src/routes/+page.svelte`
- Uses a Svelte `$effect` that reacts to `pendingFirefoxQuery`, `data`
(cluster state), and `models.length` — fires exactly once when all three
are ready
- If no model is selected, `handleAutoSend` auto-picks the best
available model; if no model fits memory, a toast is shown
- If a model is selected but not running, the message is queued until
the model loads
## Testing
```
http://localhost:52415/?q=Hello+worldhttp://localhost:52415/?q=Summarize+this+page%3A+%5Bpage+title%5D+%5Bpage+url%5D
```
<img width="2056" height="1329" alt="image"
src="https://github.com/user-attachments/assets/74463eb4-ca1a-400d-806a-c19ba93147b9"
/>
## Motivation
Adding a custom model from the Hub tab shows "Added" toast but the model
doesn't appear in the All tab. You have to add it a second time for it
to work. Also, the "All" button in the model picker sidebar is too small
to read comfortably.
## Changes
**Race condition fix (`src/exo/api/main.py`):**
- Call `add_to_card_cache(card)` directly in `add_custom_model()` after
sending the `ForwarderCommand`, before the API response returns
**Sidebar sizing
(`dashboard/src/lib/components/FamilySidebar.svelte`):**
- Increased sidebar min-width from 72/64px to 80/72px
- Increased "All" icon from `w-5 h-5` to `w-6 h-6`
- Increased all sidebar labels from 9px to 11px
## Why It Works
`POST /models/add` sends a `ForwarderCommand(AddCustomModelCard)` and
returns immediately. The frontend then calls `GET /models` which reads
from `_card_cache`. But the cache was only updated by the worker event
handler after the event round-trips through the master — a race the
frontend almost always loses. By updating the cache directly in the API
handler, `GET /models` immediately reflects the new model. The worker's
later `add_to_card_cache` call is idempotent (dict key assignment).
## Test Plan
### Manual Testing
<!-- Hardware: any Mac -->
- Open model picker → Hub tab → add a custom model → verify it appears
in All tab on the first attempt
- Verify sidebar "All" button and other labels are visually larger and
readable
### Automated Testing
- `uv run basedpyright` passes with 0 errors
- `uv run ruff check` passes
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Motivation
Replace confusing EXO_MODELS_DIR/EXO_MODELS_PATH with clearer
multi-directory support, enabling automatic download spillover across
volumes.
## Changes
- EXO_MODELS_DIRS: colon-separated writable dirs (default always
prepended, first with enough space wins)
- EXO_MODELS_READ_ONLY_DIRS: colon-separated read-only dirs (protected
from deletion)
- select_download_dir(): picks writable dir by free space
- resolve_existing_model(): unified lookup across all dirs
- is_read_only_model_dir(): path-based read-only detection instead of
hardcoded flag
- Updated coordinator, worker, model cards, tests
## Why It Works
Default dir always included so zero-config behavior is unchanged. Disk
space checked at download time for automatic spillover. Read-only status
derived from path, not hardcoded.
## Test Plan
### Manual Testing
- No env vars set → identical behavior
- EXO_MODELS_DIRS=/Volumes/SSD/models → downloads to external storage
- EXO_MODELS_READ_ONLY_DIRS=/mnt/nfs → models found, deletion blocked
### Automated Testing
- 4 new tests in test_xdg_paths.py (prepend, default-only, overlap,
empty read-only)
- Existing tests updated to patch new constants
## Motivation
During master reelection, `_elect_loop` called `worker.shutdown()` (fire
& forget) then immediately created and started a new Worker.
This caused the old runner subprocess's Metal/GPU teardown to race with
the new worker's startup, resulting in `IOConnectUnmapMemory failed:
kr=0xe00002bc` errors and a full node hang requiring `^C`. Same issue
existed for `DownloadCoordinator`.
## Changes
- Added `anyio.Event`-based `_stopped` signal to `Worker` and
`DownloadCoordinator`, set at the end of their `run()` finally blocks
- Added `wait_stopped()` async method to both classes
- Updated `_elect_loop` to `await wait_stopped()` after calling
`shutdown()` on the old Worker and DownloadCoordinator before creating
replacements
## Why It Works
The old Worker's task group contains the RunnerSupervisor tasks, whose
finally blocks join the runner subprocess (with 5s timeout + SIGTERM +
SIGKILL escalation). By awaiting `wait_stopped()`, we guarantee the old
runner process has fully exited — including GPU memory cleanup — before
a new Worker can start and potentially access the GPU. This eliminates
the race without changing the shutdown mechanics themselves.
## Test Plan
### Manual Testing
Hardware: M4 Pro Mac Mini 24GB + M3 Ultra Mac Studio 96GB, connected via
Thunderbolt
**Repro steps:**
1. Start exo on two nodes with a model sharded across both (e.g.
`Josiefied-Qwen3-14B-abliterated-v3-4bit`)
2. Wait for "runner ready" on both
3. `kill -9` the master node
4. Observe the surviving node's re-election behavior
**Before fix (original crash):**
```
[ 11:02:39.0896AM ] Runner supervisor shutting down
[ 11:02:39.0905AM ] bye from the runner
[ 11:02:39.1052AM ] Stopping Worker
IOConnectUnmapMemory failed: kr=0xe00002bc
IOConnectUnmapMemory failed: kr=0xe00002bc
IOConnectUnmapMemory failed: kr=0xe00002bc
IOConnectUnmapMemory failed: kr=0xe00002bc
^C[ 11:03:45 ] ← hung for over a minute, required manual kill
```
**After fix (clean re-election):**
```
[ 12:15:22.4703PM ] runner loaded
[ 12:15:24.1672PM ] runner ready
[ 12:15:33.5393PM ] Waiting for other campaign to finish
[ 12:15:36.5409PM ] Node elected Master
[ 12:15:36.5413PM ] Unpausing API
```
No `IOConnectUnmapMemory` errors, no hang, no `^C` needed.
### Automated Testing
- No existing tests cover the `_elect_loop` re-election path; this is an
integration-level flow requiring a live router/election/worker stack
- All existing tests pass (307/308, 1 pre-existing Rust binding failure)
- basedpyright: 0 errors, ruff: all checks passed
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
missed in #1747, issue #1798.
### the issue
we didn't set the memory poll rate after failling the macmon preflight,
only after failing the followups - as we never ran macmon if preflight
failed, we never hit the followup errors etc.
### testing
requires testing on an m5 pro, but the core issue is solved.
## Motivation
PR #1747 fixed macmon support for M5 Pro/Max by pinning the
`swiftraccoon/macmon` fork in `flake.nix`. This works when running from
source (via Nix) but the distributed macOS `.app` build was still broken
on M5 Pro/Max because it was bundling the wrong macmon.
The error on M5 Pro/Max:
```
macmon preflight failed with return code -6: thread 'main' panicked at src/sources.rs:394:41
```
## Changes
- Removed `macmon` from `brew install` in `build-app.yml` — this was
installing the upstream `vladkens/macmon` which doesn't support M5
Pro/Max
- Added a new step that resolves the pinned macmon fork from the Nix dev
shell (same `swiftraccoon/macmon` at rev `9154d23` already defined in
`flake.nix`) and adds it to `$GITHUB_PATH`
- Added a safety `brew uninstall macmon` to ensure no Homebrew macmon
can shadow the pinned version
## Why It Works
PyInstaller bundles macmon via `shutil.which("macmon")`. Previously this
found the Homebrew (upstream) binary. Now it finds the Nix-overlayed
fork that has M5 Pro/Max support, because `$GITHUB_PATH` prepends the
Nix store path before the PyInstaller step runs.
## Test Plan
### Manual Testing
<!-- Hardware: M5 Pro -->
- Trigger a macOS build and verify the bundled macmon is the pinned fork
- Run the built `.app` on M5 Pro/Max and confirm macmon preflight
succeeds
### Automated Testing
- Existing CI build workflow will validate that the macmon binary is
found and bundled correctly
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## Motivation
Support cancelling image generation, similar to existing support for
cancelling text generation
## Changes
- Dashboard (app.svelte.ts): Wire up AbortController for both
generateImage and editImage API calls. On abort, show "Cancelled"
instead of an error. Clean up the controller in finally.
- Pipeline runner (pipeline/runner.py): Introduce a cancel_checker
callback and NaN-sentinel cancellation protocol for distributed
diffusion:
- _check_cancellation() - only rank 0 polls the cancel callback
- _send() - replaces data with NaN sentinels when cancelling, so
downstream ranks detect cancellation via _recv_and_check()
- _recv() / _recv_like() wrappers that eval and check for NaN sentinel
- After cancellation, drains any pending ring recv to prevent deadlock
- Skips partial image yields and final decode when cancelled
- Image runner (runner/image_models/runner.py): Deduplicate the
ImageGeneration and ImageEdits match arms into a shared
_run_image_task() method. Thread a cancel_checker closure (backed by the
existing cancel_receiver + cancelled_tasks set) into generate_image().
- Plumbing (distributed_model.py, generate.py): Pass cancel_checker
through the call chain.
## Why It Works
- Rank 0 is the only node that knows about task-level cancellation. When
it detects cancellation, it sends NaN tensors instead of real data.
Higher-order ranks detect the NaN sentinel on recv, set their own
_cancelling flag, and propagate NaN forward
- A drain step after the loop prevents the deadlock case where the last
rank already sent patches that the first would never consume.
- For single-node mode, the loop simply breaks immediately on
cancellation.
## Test Plan
### Automated Testing
New tests in src/exo/worker/tests/unittests/test_image
## Summary
DeepSeek V3.2 (`DeepseekV32ForCausalLM`) is already supported by exo's
inference engine (architecture whitelisted in `model_cards.py`, DSML
encoding added in #1548), but **doesn't work out of the box** due to two
bugs:
### Bug 1: `warmup_inference` passes empty model ID
`warmup_inference()` in `generate.py` accepts `model_id: ModelId` as a
parameter but creates `TextGenerationTaskParams(model=ModelId(""), ...)`
instead of using it. Since `_needs_dsml_encoding()` checks
`"deepseek-v3.2" in task_params.model.lower()`, the empty string never
matches → falls back to `tokenizer.apply_chat_template()` →
**ValueError** because V3.2 has no Jinja chat template.
**Fix:** `model=ModelId("")` → `model=model_id` (one line).
### Bug 2: `_needs_dsml_encoding` limited to tool calling
`_needs_dsml_encoding()` returns `True` only when `task_params.tools` is
present or tool messages exist in `chat_template_messages`. For warmup
and regular chat requests without tools → `return False` → Jinja
fallback → **ValueError**.
Unlike V3.1 (which has a `.jinja` chat template file that transformers
picks up automatically), V3.2 **has no Jinja template at all** — it uses
Python-based DSML encoding for all message types.
**Fix:** For V3.2, always return `True` — DSML encoding handles all
message types.
### Catalog cards
Added inference model cards for:
- `mlx-community/DeepSeek-V3.2-8bit`
- `mlx-community/DeepSeek-V3.2-4bit`
Parameters taken from model `config.json` on HuggingFace, storage sizes
from HF API. Capabilities include `thinking_toggle` (related: #1456).
## Notes
- The model ID string matching approach (`"deepseek-v3.2" in
model.lower()`) is acknowledged tech debt — see #1371 for the planned
architecture-based approach.
## Test plan
- [x] Start exo with DeepSeek V3.2 model → warmup should complete
without crash
- [x] Send a regular chat message (no tools) → should get a response
- [x] Send a chat message with tools → should work as before
- [x] V3.2 cards should appear in the dashboard model catalog
---------
Co-authored-by: user <user@m1.note>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Warmup occasionally had issues; e.g. #1748 and #1793 because we were
using a standard stream_generate, all of which are issues that are
resolved in the wrapper function mlx_generate.
**Enabling peers to be discovered in environments where mDNS is
unavailable (SSH sessions, headless servers, Docker).**
## Motivation
Exo discovers peers exclusively via mDNS, which works great on a local
network but breaks once you move beyond a single L2 broadcast domain:
- SSH sessions on macOS — TCC blocks mDNS multicast from non-GUI
sessions (#1488)
- Headless servers/rack machines — #1682 ("DGX Spark does not find other
nodes")
- Docker Compose — mDNS is often unavailable across container networks;
e.g. #1462 (E2E test framework) needs an alternative
Related works:
#1488 (working implementation made by @AlexCheema and closed because SSH
had a GUI workaround),
#1023 (Headscale WAN then closed due to merge conflicts),
#1656 (discovery cleanup, open).
This PR introduces an optional bootstrap mechanism for peer discovery
while leaving the existing mDNS behavior unchanged.
## Changes
Adds two new CLI flags:
- `--bootstrap-peers` (env: `EXO_BOOTSTRAP_PEERS`) — comma-separated
libp2p multiaddrs to dial on startup and retry periodically
- `--libp2p-port` — fixed TCP port for libp2p to listen on (default:
OS-assigned). Required when bootstrap peers, so other nodes know which
port to dial.
8 files:
- `rust/networking/src/discovery.rs`: Store bootstrap addrs, dial in
existing retry loop
- `rust/networking/src/swarm.rs`: Thread `bootstrap_peers` parameter to
`Behaviour`
- `rust/networking/examples/chatroom.rs`: Updated call site for new
create_swarm signature
- `rust/networking/tests/bootstrap_peers.rs`: Integration tests
- `rust/exo_pyo3_bindings/src/networking.rs`: Accept optional
`bootstrap_peers` in PyO3 constructor
- `rust/exo_pyo3_bindings/exo_pyo3_bindings.pyi` : Update type stub
- `src/exo/routing/router.py`: Pass peers to `NetworkingHandle`
- `src/exo/main.py` : `--bootstrap-peers` CLI arg +
`EXO_BOOTSTRAP_PEERS` env var
## Why It Works
Bootstrap peers are dialed in the existing retry loop — the same path
taken by peers when mDNS-discovered. The swarm handles connection, Noise
handshake, and gossipsub mesh joining from there.
PeerId is intentionally not required in the multiaddr, the Noise
handshake discovers it.
Docker Compose example:
```yaml
services:
exo-1:
environment:
EXO_BOOTSTRAP_PEERS: "/ip4/exo-2/tcp/30000"
exo-2:
environment:
EXO_BOOTSTRAP_PEERS: "/ip4/exo-1/tcp/30000"
```
## Test Plan
### Manual Testing
<details>
<summary>Docker Compose config</summary>
```
services:
exo-node1:
build:
context: .
dockerfile: Dockerfile.bootstrap-test
container_name: exo-bootstrap-node1
hostname: exo-node1
command: ["-q", "--libp2p-port", "30000", "--bootstrap-peers", "/ip4/172.30.20.3/tcp/30000"]
environment:
- EXO_LIBP2P_NAMESPACE=bootstrap-test
ports:
- "52415:52415"
networks:
bootstrap-net:
ipv4_address: 172.30.20.2
deploy:
resources:
limits:
memory: 4g
exo-node2:
build:
context: .
dockerfile: Dockerfile.bootstrap-test
container_name: exo-bootstrap-node2
hostname: exo-node2
command: ["-q", "--libp2p-port", "30000", "--bootstrap-peers", "/ip4/172.30.20.2/tcp/30000"]
environment:
- EXO_LIBP2P_NAMESPACE=bootstrap-test
ports:
- "52416:52415"
networks:
bootstrap-net:
ipv4_address: 172.30.20.3
deploy:
resources:
limits:
memory: 4g
networks:
bootstrap-net:
driver: bridge
ipam:
config:
- subnet: 172.30.20.0/24
```
</details>
Two containers on a bridge network (`172.30.20.0/24`), fixed IPs,
`--libp2p-port 30000`, cross-referencing `--bootstrap-peers`.
Both nodes found each other and established a connection then ran the
election protocol.
### Automated Testing
4 Rust integration tests in `rust/networking/tests/bootstrap_peers.rs`
(`cargo test -p networking`):
| Test | What it verifies | Result |
|------|-----------------|--------|
| `two_nodes_connect_via_bootstrap_peers` | Node B discovers Node A via
bootstrap addr (real TCP connection) | PASS |
| `create_swarm_with_empty_bootstrap_peers` | Backward compatibility —
no bootstrap peers works | PASS |
| `create_swarm_ignores_invalid_bootstrap_addrs` | Invalid multiaddrs
silently filtered | PASS |
| `create_swarm_with_fixed_port` | `listen_port` parameter works | PASS
|
All 4 pass. The connection test takes ~6s
---------
Signed-off-by: DeepZima <deepzima@outlook.com>
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
The Nemotron model family in the model picker sidebar was displaying as
"Nemotron" with a generic checkmark icon. Since these models are
NVIDIA's Nemotron models, the category should be branded as "NVIDIA"
with the official NVIDIA logo, consistent with how other families are
branded (e.g., "llama" → "Meta", "gpt-oss" → "OpenAI").
## Changes
- **FamilySidebar.svelte**: Added `nemotron: "NVIDIA"` to the
`familyNames` mapping so the sidebar displays "NVIDIA" instead of
"Nemotron"
- **FamilyLogos.svelte**: Added the NVIDIA "eye" logo as an inline SVG
for the `nemotron` family, matching the `viewBox="0 0 24 24"` /
`fill="currentColor"` pattern used by all other brand logos
- **ModelPickerModal.svelte**: Added `"nemotron"` to the `familyOrder`
array so NVIDIA appears in a consistent position in the sidebar
## Why It Works
The model picker derives categories from the `family` field in TOML
model cards. Nemotron models already have `family = "nemotron"`, but the
three UI components (display name, logo, sort order) lacked explicit
entries for it, causing fallback behavior (auto-capitalized name,
checkmark icon, alphabetical sorting). Adding explicit entries for all
three aligns NVIDIA with the existing brand pattern.
## Test Plan
### Manual Testing
<!-- Hardware: N/A - dashboard UI change only -->
- Built dashboard successfully (`npm run build`)
- Verified the NVIDIA logo renders in the sidebar alongside existing
brand logos
### Automated Testing
- No test changes needed — this is a purely cosmetic dashboard change
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
updates macmon to an upstream fork that fixes m5 max issues.
might see if the upstream version gets merged before we release.
---------
Co-authored-by: Alex Cheema <alexcheema123@gmail.com>
## Motivation
Move back to an MLX LM to improve the Qwen 3.5 experience.
## Test Plan
### Manual Testing
Seems to loop less from testing, no speed regressions.
## Motivation
Some users (primarily in China) are unable to access HuggingFace.co. HF
supports the HF_ENDPOINT env variable, and we also support it, but there
is no way to easily do that from the app currently.
## Test Plan
### Manual Testing
hf-mirror works
empty field works
google.com endpoint fails
a fix from pr #1643 which is now stale - should make prefill more
consistent on very slow machines
## testing
qwen-3.5-35b-a3b loads normally
gpt-oss-120b-mxfp4-q8 loads normally
## Motivation
Batch generation reports incorrect statistics, as mlx lm never clears
the original stats, meaning they get polluted over time.
The dashboard also seems considerably slower than bench statistics.
We also have a large discrepancy between B=1 batch generating and
mlx_generate.
Extracting logprobs is massively expensive, causing up to a 25% slowdown
compared to pure batching.
```
[ 12:02:01.1240AM | INFO ] step overhead: 3.49ms (next=12.49ms total=15.99ms)
[ 12:02:02.1600AM | INFO ] step overhead: 3.23ms (next=13.01ms total=16.24ms)
[ 12:02:03.2228AM | INFO ] step overhead: 3.28ms (next=13.38ms total=16.66ms)
[ 12:02:04.2798AM | INFO ] step overhead: 3.25ms (next=12.84ms total=16.10ms)
[ 12:02:05.3152AM | INFO ] step overhead: 3.18ms (next=12.61ms total=15.79ms)
[ 12:02:06.3522AM | INFO ] step overhead: 3.41ms (next=12.83ms total=16.25ms)
[ 12:02:07.3987AM | INFO ] step overhead: 3.38ms (next=13.14ms total=16.52ms)
[ 12:02:08.4537AM | INFO ] step overhead: 1.84ms (next=19.44ms total=21.28ms)
```
## Changes
1. Report stats ourselves instead of using mlx lm's stats for batch
generation (they use perf_counter anyway).
2. Adjust exo bench to match
3. Improve logprobs extraction speed by 10x, improving tps for dashboard
& any requests for logprobs
4. Use an SSE comment to align the speed to the real numbers at the end
of generation
5. Patch mlx for several optimizations given our assumptions and use
cases (e.g. use vllm style RoPE).
6. Switch MLX LM version to latest main, including support for Nemotron
Super and some Qwen3.5 fixes.
## Why It Works
1. Exo bench no longer reports polluted stats
2. Exo bench now handles the reported per-request stats rather than the
aggregate stats
3. The decode speed now jumps back to a real number at the end of the
generation
4. Large batch speedup for rotating KV cache models + 1:1 matching cache
with vllm
## Test Plan
### Manual Testing
Needs testing on OpenCode and CC
Needs eval testing
### Automated Testing
Only going to show the performance optimization difference after the
accurate reporting:
**GPT OSS 20B MXFP4 Q8 (large change)**
Before:
<img width="2466" height="1534" alt="image"
src="https://github.com/user-attachments/assets/88b50637-fca2-4db4-9413-b9eee6e2057e"
/>
<img width="2410" height="1240" alt="image"
src="https://github.com/user-attachments/assets/21e5c76a-2f5f-44d2-8953-121b3ebdbd68"
/>
After:
<img width="2476" height="1472" alt="image"
src="https://github.com/user-attachments/assets/fec5cfbd-fff8-430a-b12e-a329410107a2"
/>
<img width="2454" height="1236" alt="image"
src="https://github.com/user-attachments/assets/0400344b-a4a6-42c0-a9dd-4ee91ade714a"
/>
**Qwen 3.5 35B A3B 8bit (No change)**
Before:
<img width="2414" height="1396" alt="image"
src="https://github.com/user-attachments/assets/e75f0b38-df5d-49fd-ab90-bc1667d981b3"
/>
After:
<img width="2346" height="1234" alt="image"
src="https://github.com/user-attachments/assets/eabfb59c-851f-4d88-b927-e1e699a75cc6"
/>
**Llama 3.2 1B Instruct 4bit (small change)**
Before:
<img width="2516" height="1220" alt="image"
src="https://github.com/user-attachments/assets/c2873655-acff-4536-8263-fb8aea33db80"
/>
After:
<img width="2566" height="1370" alt="image"
src="https://github.com/user-attachments/assets/15f95c75-1c2f-4474-85a2-88c4d0a32543"
/>
## Motivation
Running exo in offline mode on a machine where a model has never been
downloaded causes a crash
## Changes
- `download_shard` now catches `FileNotFoundError` from
`fetch_file_list_with_cache` and returns a `not_started` progress
instead of propagating the exception
## Why It Works
A status query should never crash its caller. By returning
`not_started`, the coordinator's existing offline guard (`if
self.offline:` at line 198) is reached and emits a graceful
`DownloadFailed` event. This also eliminates the warning spam on startup
where the status iterator catches the same exception for every
predefined model.
## Test Plan
### Manual Testing
- Start exo with `--offline` on a machine with no cached models, request
a model via the API — should get a graceful failure instead of a crash
## Motivation
Custom model cards were only saved locally on the node that handled the
API request.
## Changes
- Added AddCustomModelCard and DeleteCustomModelCard commands
- Added CustomModelCardAdded and CustomModelCardDeleted events
- Added custom_model_cards field to cluster State
- Master handles new commands by emitting corresponding events
- Workers persist model cards to disk and update the in-memory cache on
event receipt
- Separated fetch_from_hf (pure fetch) from disk persistence (now
handled by event-sourcing layer)
- Exposed add_to_card_cache() helper for the worker to update the cache
## Why It Works
- Follows the existing event-sourcing pattern: API → Command → Master →
Event → all Workers
- Every node applies the same events, so custom model cards are
consistent across the cluster
## Test Plan
### Manual Testing
Add/delete a custom model via the API on one node, verify it
appears/disappears on all nodes
### Automated Testing
Existing tests cover apply() logic; new event types follow the same
discriminated-union pattern
## Motivation
When placing a model instance across the cluster, the master previously
only considered available RAM. This meant it could pick a node that
hasn't downloaded the model yet, even when another node already has it
(or is further along in downloading it).
## Changes
- Added download_status parameter to place_instance() in placement.py
- Added _get_node_download_fraction() to compute 0.0–1.0 download
progress per node/model
- Added _cycle_download_score() to sum download fractions across a
cycle's nodes
- Cycle selection now uses a (download_score, available_ram) tuple key —
download progress is the primary sort, RAM is the tiebreaker
- Passed self.state.downloads into place_instance() from master/main.py
## Why It Works
Python's tuple comparison gives download progress strict priority over
RAM, so a node with the model already downloaded will always be
preferred over one with more free RAM but no download.
## Test Plan
### Automated Testing
3 new tests cover: completed download preferred, higher partial progress
preferred, failed download not preferred over no-download node
## Summary
- **Add ISO date to verbose stderr format**: when running with `-v`, the
stderr timestamp changes from `HH:mm:ss.SSS` to `YYYY-MM-DD
HH:mm:ss.SSS`, matching the file log format. This makes it possible to
correlate entries across days when stderr is captured by launchd,
systemd, Docker, or file redirection.
- **File log respects verbosity**: `exo.log` now uses DEBUG level when
`-v` is passed, so the persistent log has the same detail as stderr.
Previously it was always INFO regardless of verbosity.
- **Startup banner with PID**: adds a visual separator and process ID to
the startup message, making it easy to identify session boundaries in
long-running logs (e.g. `grep "Starting EXO"`).
The non-verbose (default) stderr format is unchanged — end-user terminal
experience is not affected.
## Motivation
When exo runs as a service (launchd, systemd, Docker), stderr is
typically captured to a file. Without calendar dates in the timestamp,
it is impossible to tell which day a log entry belongs to. This caused
misidentification of log entries during a multi-day RDMA debugging
session on a 4-node cluster.
The file log (`exo.log`) already had dates and rotation, but only
captured INFO level — missing the DEBUG output needed for postmortem
analysis.
## Test plan
- [x] `basedpyright` — 0 errors, 0 warnings, 0 notes
- [x] `ruff check` — all checks passed
- [x] `pytest` — 249 passed, 1 skipped
Co-authored-by: user <user@m1.note>
## Motivation
Download progress in the dashboard was broken: mainly treating all
download statuses as ongoing
## Changes
- Backend (apply.py): Deduplicate download progress events by model_id
instead of full shard_metadata, preventing duplicate entries per node
- Dashboard (+page.svelte): Extract shared collectDownloadStatus()
helper that both getModelDownloadStatus and getInstanceDownloadStatus
use, eliminating ~100 lines of duplicated logic. Adds proper handling
for
DownloadCompleted/DownloadFailed events, uses a Map to deduplicate
per-node entries, and introduces a typed NodeDownloadStatus with
explicit status states (downloading/completed/partial/pending)
- ModelCard: Replace single aggregate progress bar with per-node
download bars, each color-coded by status. Instance preview now scopes
download status to participating nodes only
## Why It Works
- Deduplicating by model_id in apply.py ensures each node has exactly
one download entry per model
- The perNodeMap in the frontend keeps only the latest event per node,
preventing duplicate bars
- Handling DownloadCompleted allows the UI to show finished downloads
instead of dropping them
- Scoping instance previews to assigned nodes avoids showing irrelevant
download progress
## Motivation
Image generation/editing from the dashboard was broken. ChatForm
bypassed the parent's model-launch logic, so image requests fell through
to text chat.
## Changes
- Moved image routing logic from ChatForm.svelte to +page.svelte
(routeMessage())
- ChatForm now always delegates to parent via onAutoSend (made required)
- Fixed missing updateActiveConversation() call on message retry
## Why It Works
All sends now go through the parent's launch-then-route path, so image
models get launched before the request is dispatched to the correct
endpoint.
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
This was to fix a small issue which was that the StepFun logo was not
included in the sidebar, and I also noticed it from an open issue:
- #1662
## Changes
<!-- Describe what you changed in detail -->
I added a condition to the FamilyLogos where if the family is "step"
then the logo will be included in the sidebar.
## Test Plan
Open exo, go choose a model, and scroll down the sidebar until you see
the step logo, which should be there.
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
Check for the step logo in the sidebar.
## Summary
Fixes a bug where macmon metrics (GPU usage, temperature, power, RAM)
freeze permanently in the dashboard while non-macmon metrics (disk
usage) continue updating normally.
**Root cause: asyncio subprocess pipe transport flow control stall**
Observed on a 2-node Mac Studio M3 Ultra cluster (Python 3.13.12, anyio
4.11.0, macOS with kqueue) running EXO.app for ~36 hours. Diagnostics
confirmed:
1. **Only macmon monitoring is stuck** — disk metrics from the same
InfoGatherer continue updating, proving the Worker, EventRouter, and API
pipelines are healthy
2. **macmon IS producing data** — its stdout pipe is full at exactly
65536 bytes (64KB OS buffer), and macmon is blocked on write at 0% CPU
3. **The pipe read-end FD is still open** — the exo process holds it,
but asyncio isn't reading from it
4. **The stale GPU value (0.21) is wrong** — should be ~1.0 (matching
the other node under identical load)
This is consistent with asyncio's subprocess pipe transport getting
stuck in flow control: `pause_reading()` is called when the internal
buffer exceeds the high-water mark, but `resume_reading()` is never
called, permanently deregistering the FD from kqueue. The `receive()`
coroutine waits forever for data asyncio will never deliver.
```
$ lsof -p <macmon_pid>
macmon 74691 FD=1 PIPE SIZE=65536 ->0x5ae78ecf376ccd0e # full pipe
$ lsof -p <exo_pid> | grep <pipe_id>
exo 74689 FD=47 PIPE SIZE=65536 ->0xd129da474c1340f7 # read end held open, not consumed
```
Note: anyio 4.11's `Process.aclose()` already uses
`CancelScope(shield=True)` for cleanup during cancellation — this is NOT
an election cleanup issue (confirmed by @ciaranbor's testing).
**Fix:** Replace the `async for` iteration with an explicit `receive()`
inside `fail_after()`. If macmon produces no output for 10× its
configured interval (minimum 30s), `TimeoutError` fires, `open_process`
cleanup kills macmon and tears down the stale transport, and the loop
restarts with a fresh subprocess and fresh asyncio transport.
## Test plan
- [x] All 246 existing tests pass
- [ ] Verify macmon restarts after simulated pipe stall (e.g., `SIGSTOP`
macmon, wait for timeout, confirm restart and metrics resume)
- [ ] Long-running soak test on multi-node cluster to confirm the fix
prevents recurrence
## Motivation
The MLX version and git revision in nix/mlx.nix were hardcoded and had
to be manually kept in sync with uv.lock
## Changes
- flake.nix: Extract MLX git rev from uv.lock's source.git URL and pass
as uvLockMlxRev
- nix/mlx.nix: Use uvLockMlxVersion and uvLockMlxRev instead of
hardcoded values; remove version mismatch assertion
## Why It Works
uv.lock is already the source of truth — now Nix reads both version and
rev from it directly. The pinned fetchFromGitHub hash still guards
against unexpected changes.
## Problem
Running short completions (like `max_tokens=1` health check probes) can
finish so fast that `mlx_lm`'s internal `generation_time` rounds to
zero. When that happens, `BatchGenerator.stats()` in
`mlx_lm/generate.py` divides `generation_tokens / generation_time` and
throws a `ZeroDivisionError`, which kills the runner process.
EXO already handles this on its side — lines 289-293 in
`batch_generate.py` guard the TPS calculation with `if
generation_elapsed > 0`. But the call to `self._exo_gen.stats()` on line
294 goes into mlx_lm's *separate* timing code, which doesn't have the
same guard. Two different timers, only one is protected.
In my case, this was triggered by health check probes (content: `"a"`,
`max_tokens=1`). The generation completed in sub-microsecond time,
`generation_time` was exactly `0`, and the runner crashed. Since the
health check command stays in the queue and retries after recovery, it
created an infinite crash loop — every ~15 seconds the runner would load
the model, get the same health check, and die again.
## Fix
Wrap `self._exo_gen.stats()` in a `try/except ZeroDivisionError`. If it
throws, set `mlx_stats` to `None` and fall back to `0.0` for
`prompt_tps`. The only fields EXO reads from `mlx_stats` are
`prompt_tps` and `prompt_time` — losing them on a sub-microsecond
generation has no practical impact.
## Traceback
```
File "batch_generate.py", line 294, in step
mlx_stats = self._exo_gen.stats()
File "mlx_lm/generate.py", line 1224, in stats
self._stats.generation_tokens / self._stats.generation_time
ZeroDivisionError: division by zero
```
## Summary
- Dashboard now shows partial download progress for models that were
partially downloaded in a previous session, instead of showing 0%
- Both `getModelDownloadStatus()` and `getInstanceDownloadStatus()` now
handle `DownloadPending` entries that carry non-zero
`downloaded`/`total` bytes
Fixes#1042
## Root cause
When exo restarts with partially downloaded models, the
`DownloadCoordinator` emits `DownloadPending` events (because
`downloaded_this_session` is 0, even though real bytes exist on disk).
The main page dashboard only checked for `DownloadOngoing` entries, so
these partially downloaded models showed as 0%.
The dedicated `/downloads` page already handled this correctly — it
renders `DownloadPending` entries with a progress bar when `downloaded >
0`. This fix brings the same behavior to the main page.
## Changes
For both `getModelDownloadStatus()` and `getInstanceDownloadStatus()` in
`+page.svelte`:
- Accept `DownloadPending` in addition to `DownloadOngoing`
- For `DownloadPending` entries with `downloaded > 0` or `total > 0`,
synthesize a `DownloadProgress` object from the top-level fields (with
`speed: 0` and `etaMs: 0` since no active download is in progress)
- Skip `DownloadPending` entries where both `downloaded` and `total` are
0 (truly pending, not yet started)
## Test plan
- [ ] Partially download a model, quit exo, relaunch — dashboard should
show partial progress instead of 0%
- [ ] Fully downloaded models still show as complete
- [ ] Active downloads still show real-time progress with speed/ETA
- [ ] Models never downloaded show as not started (not falsely showing
progress)
- [ ] Dashboard builds without errors (`cd dashboard && npm run build`)
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
The responses API often does not provide tool args nested under a
"function" field. Since we follow the chat completions format of tools
in the backend (for MLX chat templates), we need to normalise to this
format.
## Test Plan
### Manual Testing
Works on n8n!
<img width="3442" height="2076" alt="image"
src="https://github.com/user-attachments/assets/9e11d679-0102-4d83-9a8e-b0a7a5898708"
/>
## Problem
Models with fewer KV heads than nodes crash during tensor parallelism.
For example, Qwen3.5 MoE models have only 2 KV heads — trying to shard
across 4 nodes produces empty tensors and a reshape error at runtime.
The placement system already validates `hidden_size % num_nodes == 0`
but doesn't check KV heads, so it creates configurations that look valid
but blow up when the worker tries to split the attention heads.
Affected models include Qwen3.5-35B-A3B, Qwen3.5-122B-A10B,
Qwen3.5-397B-A17B, Qwen3-Next-80B-A3B, and Qwen3-Coder-Next (all have 2
KV heads).
## Changes
**Placement validation** (`src/exo/master/placement.py`):
- Combined KV heads divisibility check with the existing hidden_size
filter in a single pass
- Cycles where `num_key_value_heads % len(cycle) != 0` are now excluded
for tensor sharding
- Error message includes both constraints when no valid cycle is found
**Model card schema** (`src/exo/shared/models/model_cards.py`):
- Added optional `num_key_value_heads` field to `ModelCard` and
`ConfigData`
- Extracted from HuggingFace `config.json` (handles both top-level and
`text_config` nesting)
- Passed through in `fetch_from_hf()` for dynamically fetched cards
**All 68 inference model cards**
(`resources/inference_model_cards/*.toml`):
- Populated `num_key_value_heads` from each model's HuggingFace config
**Utility script** (`scripts/fetch_kv_heads.py`):
- Fetches `num_key_value_heads` from HuggingFace and updates TOML cards
- `--missing`: only fills in cards that don't have the field yet
- `--all`: re-fetches and overwrites everything
- Uses tomlkit for safe TOML editing and ThreadPoolExecutor for parallel
fetches
## Behavior
- Instance previews no longer show tensor options for models that can't
split their KV heads across the cluster size
- `place_instance()` rejects with a clear error instead of crash-looping
- Pipeline parallelism is unaffected
- 2-node tensor still works for 2-KV-head models (2 ÷ 2 = 1)
- Field is optional — existing custom cards without it continue to work
(validation is skipped when `None`)
## Problem
Emojis and other multi-byte UTF-8 characters are rendered as `\ufffd`
(Unicode Replacement Character) in batch streaming responses.
Byte-level BPE tokenizers (like Qwen's) can split multi-byte UTF-8
characters (e.g. a 4-byte emoji) across multiple tokens. The batch
generator decodes each token independently with
`tokenizer.decode([token_id])`, which produces U+FFFD for partial byte
sequences.
The sequential generator doesn't have this problem — it uses
`stream_generate()` from mlx_lm, which internally uses
`StreamingDetokenizer` to buffer incomplete bytes.
## Fix
Use `StreamingDetokenizer` in the batch generator, matching the
sequential path:
- Each `_EngineTask` gets its own `StreamingDetokenizer` instance
- `add_token()` buffers tokens, holding back incomplete UTF-8 byte
sequences until they form valid characters
- `last_segment` returns only complete, valid text
- `finalize()` flushes any remaining buffered bytes when generation
completes
Empty text segments during buffering are harmless — they're already
handled correctly by the downstream streaming pipeline.
---------
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Motivation
Add power/energy usage tracking to /bench API responses.
## Changes
- New PowerSampler class that periodically samples sys_power from each
node during inference
- New PowerUsage / NodePowerStats API types
- Integrated into both bench chat completion and bench image generation
endpoints
## Why It Works
Runs concurrently via anyio.create_task_group, reads from existing
state.node_system heartbeat data — no new plumbing needed. Energy =
avg_power × elapsed_time.
## Test Plan
### Manual Testing
Run a /bench request and verify power_usage appears in the response.
### Automated Testing
7 async tests covering single/multi-node, energy math, dynamic power
changes, empty state, and lifecycle.
## Motivation
When a runner shuts down, there's no mechanism to remove it from
state.runners, causing stale state to accumulate.
## Changes
- apply.py: apply_runner_status_updated now handles RunnerShutdown
status by removing the runner from state instead of adding/updating it.
Removed the separate apply_runner_deleted function entirely.
- events.py: Removed the RunnerDeleted event type — runner cleanup is
now handled via RunnerStatusUpdated with a RunnerShutdown status.
- Tests: New test_apply_runner_deleted.py with 2 tests: verifying
RunnerShutdown removes the runner, and that a normal status update adds
it.
## Why It Works
Instead of a separate RunnerDeleted event and coordinating its emission
from master/placement, runner cleanup is handled naturally through the
existing RunnerStatusUpdated path — a RunnerShutdown status signals
removal. This is simpler and requires no changes to master or placement
logic.
## Test Plan
### Automated Testing
2 new unit tests covering RunnerShutdown removal and normal runner
status addition
## Summary
- Adds mobile-friendly drawer UI for the chat sidebar and model family
sidebar
- Introduces responsive header navigation with mobile-specific toggle
buttons
- Uses slide-in drawer overlay pattern on small screens instead of
collapsible sidebars
- Stores mobile drawer state in app store for consistent state
management
## Testing
Tested on desktop (responsive mode) and mobile viewport widths. Sidebars
now slide in as drawers on small screens with proper z-index layering
and close-on-outside-click behavior.
## Motivation
Following the changes made in #1632 !
Closes#1020
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
---------
Co-authored-by: Evan Quiney <evanev7@gmail.com>
## Motivation
When gpt-oss truncates a response mid-tool-call (e.g. hitting max
tokens), the finish_reason was swallowed because the parser was in
tool-accumulation mode and never yielded it back to the caller. This
caused the API to hang or fail to signal completion.
## Changes
In parse_gpt_oss, when accumulating tool call arguments and a
finish_reason arrives, yield the response (with empty text) so the
finish reason propagates
## Why It Works
The parser now checks for finish_reason on every chunk while inside a
tool call. When the model stops generating (truncation), the finish
signal is forwarded immediately rather than being silently consumed.
## Test Plan
### Automated Testing
Added tests covering truncated tool calls (finish_reason="length") and
plain text truncation
## Motivation
Two minor bugs in the download coordinator: cancelling a download didn't
reset its status to pending, and a single error in
_emit_existing_download_progress would kill the loop permanently.
## Changes
- Cancel emits pending status: When a download is cancelled, emit a
DownloadPending event so the UI/cluster state reflects that the model is
no longer actively downloading.
- Resilient progress emission loop: Move the try/except inside the while
loop so transient errors are logged but the periodic emission continues.
## Why It Works
- The cancel fix ensures the download status is consistent — without it,
a cancelled download would still appear as in-progress.
- The loop fix prevents a single exception from permanently stopping the
60-second periodic progress broadcast.
## Motivation
Updated documentation for v1.0.68 release
## Changes
**docs/api.md:**
- Added documentation for new API endpoints: Claude Messages API
(`/v1/messages`), OpenAI Responses API (`/v1/responses`), and Ollama API
compatibility endpoints
- Documented custom model management endpoints (`POST /models/add`,
`DELETE /models/custom/{model_id}`)
- Added `enable_thinking` parameter documentation for thinking-capable
models (DeepSeek V3.1, Qwen3, GLM-4.7)
- Documented usage statistics in responses (prompt_tokens,
completion_tokens, total_tokens)
- Added streaming event format documentation for all API types
- Updated image generation section with FLUX.1-Kontext-dev support and
new dimensions (1024x1365, 1365x1024)
- Added request cancellation documentation
- Updated complete endpoint summary with all new endpoints
- Added security notes about trust_remote_code being opt-in
**README.md:**
- Updated Features section to highlight multiple API compatibility
options
- Added Environment Variables section documenting all configuration
options (EXO_MODELS_PATH, EXO_OFFLINE, EXO_ENABLE_IMAGE_MODELS,
EXO_LIBP2P_NAMESPACE, EXO_FAST_SYNCH, EXO_TRACING_ENABLED)
- Expanded "Using the API" section with examples for Claude Messages
API, OpenAI Responses API, and Ollama API
- Added custom model loading documentation with security notes
- Updated file locations to include log files and custom model cards
paths
**CONTRIBUTING.md:**
- Added documentation for TOML model cards format and the API adapter
pattern
**docs/architecture.md:**
- Documented the adapter architecture introduced in PR #1167Closes#1653
---------
Co-authored-by: askmanu[bot] <192355599+askmanu[bot]@users.noreply.github.com>
Co-authored-by: Evan Quiney <evanev7@gmail.com>
## Problem
Models running through exo can get stuck in repetition loops —
generating the same text over and over until hitting the token limit. It
happens more with quantized models where probability distributions can
become degenerate.
`mlx_lm` has `make_logits_processors(repetition_penalty,
repetition_context_size)` that handles this, but it is never called
anywhere in the pipeline.
## Solution
Wire `repetition_penalty` and `repetition_context_size` through the
stack:
- `ChatCompletionRequest` → accept the params from the client
- `TextGenerationTaskParams` → carry them through the pipeline
- `chat_completions.py` adapter → map to internal params
- `generate.py` → call `make_logits_processors()` and merge into the
`logits_processors` list
When `repetition_penalty` is `None` (default), `make_logits_processors`
returns `[]` so existing behavior is unchanged.
## Usage
```json
{
"model": "mlx-community/...",
"messages": [...],
"repetition_penalty": 1.15,
"repetition_context_size": 64
}
```
## Motivation
Fixes#1648. Users adding custom models from HuggingFace got no clear
feedback that the model was successfully added — the model didn't appear
prominently in the list. Additionally, searches were limited to
`mlx-community` models with no fallback.
## Changes
- **Success feedback**: After adding a custom model, a toast
notification appears, the view auto-switches to "All Models", and the
newly added model is scrolled into view with a green highlight that
fades over 4 seconds.
- **HuggingFace search fallback**: The `/models/search` endpoint now
searches `mlx-community` first; if no results are found, it falls back
to searching all of HuggingFace.
- **Inline HF results**: When the main search bar finds no local
matches, HuggingFace search results appear inline with "+ Add" buttons
and a "See all results on Hub" link.
- **Full repo ID for non-mlx models**: Non-mlx-community models now
display the full repo ID (e.g., `meta-llama/Llama-3.1-8B`) instead of
just the short name.
## Why It Works
The toast + scroll + highlight gives immediate, unambiguous feedback
that the model was added and where it lives in the list. The HF search
fallback broadens discoverability while still prioritising mlx-community
models. Inline results in the main search bar mean users don't need to
navigate to the HF tab to discover new models.
## Test Plan
### Manual Testing
- Open model picker → HuggingFace Hub tab → search for a model → click
"+ Add"
- Verify: toast appears, view switches to All Models, model highlighted
with green glow
- Search for a term with no mlx-community results → verify fallback to
full HF search
- Non-mlx-community results show full repo ID (e.g., `org/model-name`)
- On All Models tab, search for a model not in the local list → verify
"From HuggingFace" section appears
### Automated Testing
- Existing tests cover the model catalog and API; no new automated tests
needed for UI behaviour changes
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
After deleting a model and re-downloading it, the CachedShardDownloader
returns the stale cached path, so ensure_shard short-circuits and no
download actually happens.
## Changes
- Added invalidate(model_id) method to the ShardDownloader ABC and all
implementations
- CachedShardDownloader.invalidate evicts cache entries matching the
model ID and delegates down
- DownloadCoordinator.delete_model calls invalidate after cancelling
active downloads, before deleting files
- Added end-to-end test that downloads, deletes, and re-downloads a
model through the coordinator
## Why It Works
The cache is cleared when a model is deleted, so the next ensure_shard
call performs a fresh download instead of returning the stale path.
## Test Plan
## Automated Testing
New test_re_download_after_delete_completes exercises the full download
→ delete → re-download flow through DownloadCoordinator with
CachedShardDownloader + SingletonShardDownloader wrappers matching
production.
Fixes#1657
## Motivation
The copy button on code blocks in the dashboard does nothing when
clicked. This affects all code blocks in assistant responses.
## Root Cause
In `MarkdownContent.svelte`, click event listeners are bound to
`.copy-code-btn` elements via `setupCopyButtons()` inside a Svelte 5
`$effect`. However, the effect fires before the DOM has been updated
with the new HTML, so `querySelectorAll(".copy-code-btn")` finds zero
buttons.
Additionally, during streaming, the `content` prop updates on every
token, causing the entire `{@html processedHtml}` to be re-rendered.
This destroys all previously bound event listeners, even if they were
successfully attached.
## Changes
Replaced the per-button `addEventListener` approach with **event
delegation** — a single click listener on the container element that
catches clicks bubbling up from any `.copy-code-btn` or
`.copy-math-btn`. This:
- Eliminates the timing issue (the listener exists before the buttons
are rendered)
- Survives HTML re-renders during streaming (no need to re-bind)
- Removes the need for `setupCopyButtons()` and the `data-listenerBound`
tracking
## Testing
1. Load any model
2. Prompt it to generate a code block (e.g. "write a hello world in
Python")
3. Click the copy button on the code block
4. Paste — the code is copied correctly
5. Verified the button also works during streaming (before generation
completes)
Co-authored-by: Wysie <wysie@users.noreply.github.com>
Closes#1586
## Summary
- track in-flight tasks in `RunnerSupervisor` (not only unacknowledged
pending tasks)
- when `_check_runner()` detects a crashed runner, emit
`ChunkGenerated(ErrorChunk)` for each in-flight command task
(`TextGeneration`, `ImageGeneration`, `ImageEdits`)
- keep existing `RunnerStatusUpdated(RunnerFailed)` emission so
planner/state still transition correctly
- add a unit test for supervisor crash path to ensure an error chunk is
emitted before failed runner status
## Why
`#1586` reports streams that can hang forever when runners crash during
warmup/loading. This keeps failure signaling at the runner-supervisor
layer, matching maintainer guidance in the issue thread.
## Validation
- attempted: `uv run pytest
src/exo/worker/tests/unittests/test_runner/test_runner_supervisor.py`
- blocked locally by environment disk exhaustion while uv tried to
materialize heavy CUDA wheels (`No space left on device` during
`nvidia-cudnn-cu13` extraction)
I kept the change scoped and added a targeted unit test for the failure
path.
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Adds reasoning_effort parameter support to both the Chat Completions and
Responses APIs, aligning with the OpenAI spec and enabling thinking
control for gpt-oss models
## Changes
- Added ReasoningEffort literal type ("none" | "minimal" | "low" |
"medium" | "high" | "xhigh") and a resolve_reasoning_params() helper
that cross-derives reasoning_effort ↔ enable_thinking when only one is
provided
- Added reasoning_effort field to ChatCompletionRequest and reasoning
(with Reasoning model) + enable_thinking to ResponsesRequest
- Both adapters now call resolve_reasoning_params() before building
TextGenerationTaskParams
- reasoning_effort is passed through to the MLX chat template as a
template variable
## Why It Works
resolve_reasoning_params is a pure function that normalises the two
overlapping knobs (reasoning_effort and enable_thinking) into a
consistent pair, so downstream code always has both values regardless of
which the caller supplied.
## Test Plan
### Automated Testing
Added test_resolve_reasoning_params.py with 10 test cases covering:
both-None, both-set passthrough, enable_thinking → effort derivation,
and effort → enable_thinking derivation for every ReasoningEffort
variant.
Apply schema-aware coercion to parsed tool-call arguments so
Hermes-style toolcalls can still return typed JSON (e.g. integer ids).
- pass request tools into parse_tool_calls
- coerce parsed argument values by function parameters schema
- add unit tests for coercion and unknown-tool passthrough
## Motivation
Models that use Hermes-based toolcall syntax (Qwen3.5) can't reliably
call tools with non-string parameters
Example tool:
```json
{
"type": "function",
"function": {
"name": "process",
"description": "Manage background processes",
"parameters": {
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["spawn", "output", "kill", "list"]
},
"id": {
"type": "integer",
"description": "Process id"
},
"command": {
"type": "string",
"description": "Command to run for spawn"
}
},
"required": ["action"],
"additionalProperties": false
}
}
}
```
Model transcript:
```
<tool_call>
<function=process>
<parameter=action>
output
</parameter>
<parameter=id>
0
</parameter>
</function>
</tool_call>
```
And the API returns:
`{"id":"a8f11689-d840-4ca5-ab1d-ead3678a11a9","name":"process","arguments":"{\"action\":
\"output\", \"id\": \"0\"}"}}`
Tool definition declared `id` as `integer`, the model output is
type-agnostic, and the translation layer treats everything as a string.
The same Qwen3.5-27B on OpenRouter and GPT-4.1-mini on openai obey the
function signature and emit correct call:
```
{"name":"process","arguments":"{\"action\": \"output\", \"id\": 0}"}}
```
Steps to reproduce:
```
❯ curl -sS -v http://localhost:52415/v1/chat/completions \
-H 'Content-Type: application/json' \ -d @- <<'JSON'
{
"model": "mlx-community/Qwen3.5-27B-4bit",
"stream": false,
"temperature": 0,
"messages": [
{
"role": "user",
"content": "Call the process tool with action=output and id=0. Do not explain anything. Just make the tool call."
}
],
"tools": [
{
"type": "function",
"function": {
"name": "process",
"description": "Manage background processes",
"parameters": {
"type": "object",
"properties": {
"action": {
"type": "string",
"enum": ["spawn", "output", "kill", "list"]
},
"id": {
"type": "integer",
"description": "Process id"
},
"command": {
"type": "string",
"description": "Command to run for spawn"
}
},
"required": ["action"],
"additionalProperties": false
}
}
}
],
"tool_choice": {
"type": "function",
"function": {
"name": "process"
}
}
}
JSON
```
Look for type of `id` function call
## Changes
Function call parameters are now converted to the types that the
function declaration has
## Why It Works
We now explicitly convert types where we know it (and skip if we don't)
## Test Plan
### Manual Testing
Create Qwen3.5-<ANY> instance, send the curl command above. Check that
'id' is now serialized as a number
### Automated Testing
Unit tests to cover basic type conversions
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Batching will require us to send tasks concurrently and queue them up.
Our current infrastructure cannot handle that all. This PR gets us
closer to this by allowing multiple tasks to be sent in parallel and
then queuing up tasks.
## Changes
Change Plan logic
Make runner main into a class
Add a "BatchGenerator" to which tasks can be submitted (although tasks
are handled sequentially) and sent back through an MpSender.
Refactor runner to accept tasks during generation
Keep the generator threading
Separate the runner into several files for better readability
## Test Plan
### Manual Testing
Tested manually, needs a lot more automated testing. Cancellation still
works on a single device. Needs checking on multiple devices.
### Automated Testing
---------
Co-authored-by: Evan Quiney <evanev7@gmail.com>
## Motivation
Qwen3.5 MoE models (e.g., `Qwen3.5-397B-A17B-6bit`) are now supported by
`mlx-lm` via `qwen3_5_moe` model type, but exo lacks tensor parallel
sharding support for this architecture. This prevents running large
Qwen3.5 models across multiple nodes.
Qwen3.5 uses a GatedDeltaNet hybrid attention mechanism similar to
Qwen3-Next, but with a different projection layout — separate
`in_proj_qkv`, `in_proj_z`, `in_proj_b`, `in_proj_a` instead of
Qwen3-Next's combined `in_proj_qkvz` and `in_proj_ba`. This requires
architecture-aware sharding logic.
## Changes (evan summary)
- enable qwen3_5 dense + moe tensor parallelism from config
- defensively skip evalling _cache.keys if it doesn't exist
- ignore kwargs in qwen35 pipeline masking and ensure pipeline segments match global model parameters for mask creation
- add sharding for qwen3_5 moe linear attention
- added another 6 million model cards
## Why It Works
Qwen3.5's GatedDeltaNet has an `in_proj_qkv` linear layer with three
concatenated sections: `[q(key_dim), k(key_dim), v(value_dim)]`. A naive
contiguous split (`segments=1`) would slice across section boundaries,
corrupting q/k/v values and producing garbled output.
By passing `segments=[key_dim, key_dim + key_dim]` to `shard_linear()`,
each section is split independently before distributing across devices.
This ensures every rank receives correctly aligned q, k, and v
components.
The remaining separate projections (`in_proj_z`, `in_proj_b`,
`in_proj_a`) and the MoE layers follow the same `all_to_sharded` /
`sharded_to_all` pattern already used for Qwen3-Next.
Some pipeline splits didn't include an ssm layer or a linear layer resulting in a subset of the model acting like it shouldn't create the appropriate masks for the next layer - we patch the model to manually create such masks.
## Test Plan
tensor sharded 2,3,4 models & pipeline sharded 2,3,4 with simple eval.
---------
Co-authored-by: hw <hw@hwStudio1.local>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Clicking on the mini topology view in the right sidebar during chat was
selecting/filtering nodes instead of navigating back to the home view.
The entire mini topology section is wrapped in a button that calls
`handleGoHome()`, but individual node clicks were intercepting the event
via `stopPropagation()`.
## Changes
- Removed `onNodeClick={togglePreviewNodeFilter}` from the mini
sidebar's `TopologyGraph` component
- Added `pointer-events-none` to the topology graph container so all
clicks pass through to the parent button
## Why It Works
`TopologyGraph` only registers click/hover handlers when `onNodeClick`
is provided. Removing it disables node interaction entirely in the mini
view. The `pointer-events-none` class ensures no SVG element can
intercept clicks — they all bubble up to the parent `<button
onclick={handleGoHome}>`, which resets the chat state and returns to the
main topology view.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Start exo and open the dashboard
- Start a chat to trigger the mini topology sidebar
- Click on the mini topology view → should navigate back to home
- Confirm nodes in the mini topology are not selectable or hoverable
### Automated Testing
- Dashboard builds successfully (`npm run build`)
## Summary
- Adds explicit `POST /v1/cancel/{command_id}` REST endpoint to cancel
in-flight text and image generation commands by ID
- Previously, cancellation only worked via HTTP disconnect (client
closes SSE connection, triggering `anyio.get_cancelled_exc_class()`).
Non-streaming clients and external consumers had no way to cleanly
cancel active generation
- The endpoint looks up the command in active generation queues, sends
`TaskCancelled` to notify workers, and closes the sender stream. Returns
404 (OpenAI error format) if the command is not found or already
completed
## Changes
| File | Change |
|------|--------|
| `src/exo/shared/types/api.py` | Add `CancelCommandResponse` model |
| `src/exo/master/api.py` | Import, route registration, `cancel_command`
handler |
| `src/exo/master/tests/test_cancel_command.py` | 3 test cases (404,
text cancel, image cancel) |
| `docs/api.md` | Document new endpoint + update summary table |
## Design Decisions
- Uses `self._send()` (not raw `command_sender.send()`) — respects API
pause state during elections
- Uses `raise HTTPException` — feeds into exo's centralized OpenAI-style
error handler
- Returns typed `CancelCommandResponse` — consistent with
`CreateInstanceResponse` / `DeleteInstanceResponse` patterns
- `sender.close()` is idempotent so concurrent cancel requests for the
same command are harmless
## Test Plan
- [x] `test_cancel_nonexistent_command_returns_404` — verifies 404 with
OpenAI error format
- [x] `test_cancel_active_text_generation` — verifies 200,
`sender.close()` called, `TaskCancelled` sent with correct
`cancelled_command_id`
- [x] `test_cancel_active_image_generation` — same verification for
image queue
- [x] `basedpyright` — 0 new errors
- [x] `ruff check` — all checks passed
- [x] `pytest` — 3/3 new tests pass, no regressions
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Fix for https://github.com/exo-explore/exo/issues/1630.
When loading a model that spans 2 nodes as the first action after
startup, Exo crashes with anyio.BrokenResourceError in
Master._command_processor. The receive end of the Master's event_sender
channel is already closed (open_receive_channels=0 ) when the Master
tries to send an InstanceCreated event.
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
In _elect_loop, when is_new_master is True and the EventRouter is
replaced, the Master's event_sender is now swapped in-place with a fresh
sender from the new EventRouter. This mirrors the existing pattern where
the Worker and DownloadCoordinator are already recreated with new
channels from the new router.
<!-- Describe what you changed in detail -->
## Why It Works
When is_new_master fires (which happens on every startup when a second
node joins), the old EventRouter is shut down and a new one is created.
Shutting down the old router cancels its _ingest coroutine, which closes
the receive end of every channel created by event_router.sender(). The
Worker and DownloadCoordinator are already recreated with senders from
the new router, but the Master was not — its event_sender still pointed
to the dead channel from the old router.
This patch replaces self.master.event_sender with a new sender from the
new EventRouter, so events flow through a live channel. We swap in-place
rather than recreating the Master to preserve its DiskEventLog, internal
state, and running tasks.
The reason loading a single-node model first appeared to work around the
issue is a timing race: if the model loads fast enough, the Master sends
its events before the election fires and kills the old channel. With a
2-node model, placement and RDMA setup take longer, so the election
completes first and the channel is already dead by the time the Master
tries to send.
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
2 x Mac Studio M4 Max 128GB, RDMA
<!-- What you did: -->
Loaded a model that requires 2 nodes and it works properly now.
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
Co-authored-by: Wysie <wysie@users.noreply.github.com>
## Motivation
Enable using the dashboard to delete traces
## Changes
- Backend (src/exo/master/api.py): Added POST /v1/traces/delete endpoint
that deletes trace files by task ID, with path traversal protection
- API types (src/exo/shared/types/api.py): Added DeleteTracesRequest and
DeleteTracesResponse models
- Dashboard store (app.svelte.ts): Added deleteTraces() method
- Traces page (+page.svelte): Added multi-select UI (click to select,
select all/deselect all) with a bulk delete button and confirmation
dialog
## Why It Works
- Uses existing _get_trace_path helper (now hardened against path
traversal) to resolve and delete trace files
- Dashboard selection state is reactive via Svelte 5 runes; deleted
traces refresh the list automatically
## Test Plan
### Manual Testing
- Select individual traces, select all, delete with confirmation dialog
- Verify traces are removed from disk and list refreshes
## Motivation
Since we need to do distributed communications between prefill step
sizes, the out-of-the-box stream_generate that we currently use prevents
pipeline parallel models from doing overlapped computation. While this
was technically a regression, this communication is necessary for
cancellation, and we will need various distributed communications in the
future (e.g. for coordinating batching).
500 lines are for one testing file, so the diffs aren't as bad as they
look!
## Changes
Added a special prefill function for pipeline parallel models
Edited the model to handle
Added a test to verify this new prefill and the original prefill produce
identical results
Improved type stubs to remove some type: ignores
## Why It Works
<img width="768" height="1246" alt="image"
src="https://github.com/user-attachments/assets/8986ff17-ac23-4a02-9bd7-e6253a0ca799"
/>
## Test Plan
### Manual Testing
Needs more testing, but seems good so far.
### Automated Testing
Passes CI, considerable speedup seen in benchmarks (up to 1.98x) on
prefill speed.
Before:
<img width="3280" height="1238" alt="image"
src="https://github.com/user-attachments/assets/9abc1cbc-ecdb-4e48-a675-2c4cb04a32a0"
/>
After:
<img width="3344" height="1236" alt="image"
src="https://github.com/user-attachments/assets/e03c7987-41b4-4950-9ac3-2840e774ce30"
/>
replace the nack & resend logic in the worker/download coordinator with
a dedicated subsystem in front of the topic router. this centralizes
that logic (and the concept of system ids) to reduce replication in the
codebase. each system reading or writing events now gets a clean stream
of events in and can trust written events will be retried until
acknowledged.
## Motivation
Coordination files for MLX distributed init were written to the current
working directory (./hosts_*.json)
## Changes
- Move coordination file creation to a tempfile.TemporaryDirectory(),
which auto-cleans on context manager exit
macmon appears to be going silent. time it out and restart it after
3*interval (Default 3s) and report warning.
Co-authored-by: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
sometimes we generate NaN logprobs for tokens, this causes pydantic
validation errors on the receiving end. in this case we should just not
send the logprob items
## Motivation
Users were reporting GPU timeout errors on Mac Minis, which we never saw
on testing with Mac Studios. It also seems to only happen with large
models.
## Changes
Eval specific distributed operations.
## Why It Works
As I wrote in a Slack message:
Basically, prefill is too slow for pipeline communications. If there are
both communications and GPU operations as part of an mlx graph, the
communications become subject to the GPU's 5 second command buffer
timeout.
For normal generation, I added evals to the communications (only during
prefill, as it slows down decode) to do this, fixing GPU timeouts.
But we don't do this during warmup, as the prompt is absolutely tiny.
This is still too slow on an M4 Pro on some models that it causes a GPU
timeout during warmup...
----------------------
This was one of the issues. However, there is another issue:
mx.all_gather sometimes reads stale data with FAST_SYNCH enabled. I'm
still investigating the root cause, but the code as it is now works on
Mac Minis.
## Test Plan
### Manual Testing
<img width="2762" height="1808" alt="image"
src="https://github.com/user-attachments/assets/27c88542-606c-4551-8f7c-bd2c0471f54e"
/>
<img width="2820" height="1898" alt="image"
src="https://github.com/user-attachments/assets/0ba3478c-ee39-438d-902c-92893db23d05"
/>
### Automated Testing
Needs a bunch on mac minis
## Motivation
Nodes silently shut down mid-inference with no exception logged. The
logs show a clean-looking shutdown cascade (unsubscribe all topics →
stop worker → runner communication closed) but no error explaining
*why*. This makes debugging cluster issues extremely difficult.
## Changes
**`src/exo/routing/router.py`** — Added `try/except Exception` around
`_networking_recv()` and `_networking_recv_connection_messages()` loops.
Logs the root cause at ERROR level via
`logger.opt(exception=...).error(...)` before re-raising. Uses
`Exception` (not `BaseException`) so clean SIGTERM cancellation is
unaffected.
**`src/exo/main.py`** — Wrapped `anyio.run(node.run)` in
`try/except/finally`:
- `except BaseException`: logs the fatal exception at CRITICAL level
through loguru
- `finally`: ensures `logger.info("EXO Shutdown complete")` and
`logger_cleanup()` always run, guaranteeing the async log queue is
flushed before process exit
## Why It Works
The Router's recv loops (`_networking_recv`,
`_networking_recv_connection_messages`) are infinite `while True` loops
calling Rust bindings with **no exception handling**. When the Rust
networking layer raises `ConnectionError` (e.g. channel closed), the
exception cascades silently through anyio task groups: Router → Node →
all components shut down. The exception was never logged because (1) no
try-except anywhere in the cascade, and (2) `logger_cleanup()` was
skipped when `anyio.run()` raised, so loguru's async queue was never
flushed.
## Test Plan
### Manual Testing
- Run `uv run exo`, send SIGTERM → should see clean shutdown with "EXO
Shutdown complete", no error/critical logs
- Next time the silent shutdown reproduces, logs should now show the
actual exception with full traceback at ERROR level from the recv loop,
plus CRITICAL level from main()
### Automated Testing
- All 222 existing tests pass (1 pre-existing failure in
`test_python.py::test_sleep_on_multiple_items` unrelated to this change)
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — all checks passed
- `nix fmt` — applied
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
The socket-based internet connectivity probes (connecting to
1.1.1.1/8.8.8.8/1.0.0.1 every 10 seconds) are flaky — they produce false
negatives on some networks/ISPs, causing downloads to silently fail or
get stuck. Instead of dynamically detecting connectivity, this replaces
it with an explicit offline mode that the user opts into.
## Changes
- **Removed internet check infrastructure**: Deleted
`_test_internet_connection()` (socket probes) and
`_check_internet_connection()` (10s polling loop) from
`DownloadCoordinator`
- **Renamed `internet_connection` → `offline`** on `ShardDownloader`
base class and all subclasses (`SingletonShardDownloader`,
`CachedShardDownloader`, `ResumableShardDownloader`)
- **Removed `on_connection_lost` callbacks** from download calls — no
longer needed without dynamic connectivity detection
- **Added `EXO_OFFLINE` env var support** in `constants.py` and `Args`
default in `main.py`
- **Added Offline Mode toggle** to macOS app Settings → General tab,
which sets `EXO_OFFLINE=true` in the process environment and restarts
## Why It Works
The download behavior (`skip_internet` conditionals in
`download_utils.py`) is unchanged — we just changed what drives it.
Instead of a flaky socket probe setting
`internet_connection=True/False`, the user explicitly sets offline mode
via:
1. `--offline` CLI flag
2. `EXO_OFFLINE=true` environment variable
3. macOS app Settings → General → Offline Mode toggle
This is more reliable and predictable. Users on flaky networks no longer
get intermittent download failures.
## Test Plan
### Manual Testing
- `uv run exo --offline` starts without internet checks, rejects
downloads for unavailable models
- `EXO_OFFLINE=true uv run exo` behaves identically
- macOS app Settings toggle persists across restarts and passes env var
to the exo process
### Automated Testing
- All existing tests pass (222 passed), including 8 offline-mode
specific tests
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — all checks passed
- `nix fmt` — 0 files changed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
should be release for 1.0.68 - let's synch our py version with our app
version - 0.3.68.
next minor release should be 1.4.0 and 0.4.0 respectively.
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
When navigating back to a previous conversation that used a different
model than the one currently loading, the UI incorrectly displayed the
other model's loading/download progress bar instead of the conversation
messages. Additionally, attempting to continue that old conversation by
sending a message would create an entirely new chat rather than
continuing the existing one.
## Changes
All changes in `dashboard/src/routes/+page.svelte`:
1. **Added fallthrough reset in the `chatLaunchState` restore
`$effect`**: When switching to a conversation whose model has no active
instance (not running, not downloading, not loading), the effect now
resets `chatLaunchState` to `"idle"` instead of leaving stale state from
a different model.
2. **Added `skipCreate` parameter to `launchModelForChat()`**: When
continuing an existing conversation (has messages),
`createConversation()` is skipped so the old conversation is preserved
rather than replaced with a new empty one.
3. **Reordered view conditional**: Progress views
(downloading/loading/launching) take priority, then conversation
messages (when idle with messages or model ready), then model selector
(idle with no messages). This ensures:
- Old conversations display normally when their model isn't running
- Download progress shows when relaunching a model for any conversation
- Model selector only appears when there are no messages
## Why It Works
- The `$effect` fallthrough prevents `chatLaunchState` from retaining a
stale value (e.g., `"downloading"`) from Model A when the user switches
to a conversation using Model B that has no active state.
- The `skipCreate` flag ensures `launchModelForChat` can be reused for
both new conversations (from model picker) and continuing existing ones
(from chat input) without always creating a new conversation.
- The reordered view logic ensures each state maps to the correct UI:
active launch → progress, existing messages → chat view, nothing → model
selector.
## Test Plan
### Manual Testing
- Load an old conversation while a different model is downloading →
should see conversation messages, not the other model's loading bar
- Send a message from that old conversation → model should
launch/download with progress shown → conversation continues with the
response appended
- Select a new model from the picker → should still create a new
conversation as before
- New conversation with model download → progress bar shows correctly
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
GPU locks because of prompt progress callbacks taking time. Current
solution: Don't fix it, make the symptom better
## Changes
Shortened timeout by 2x
Get event leak fixes from latest upstream
## Motivation
The "Load Model" dropdown in the right sidebar and the model selector
above the chat input were operating on independent state
(`selectedPreviewModelId` vs `selectedChatModel`). This caused several
UX issues:
- Selecting a model in one selector didn't update the other
- After going home from a chat, the chat selector showed "SELECT MODEL"
while the sidebar still showed the model
- On page refresh, only the sidebar restored the saved model
- Sending a chat message with a running model briefly showed the
recommended models view instead of starting the chat
## Changes
- **`handleModelPickerSelect`**: Also sets `selectedChatModel` when a
model is picked from the sidebar dropdown
- **`handleChatPickerSelect`**: Also sets `selectedPreviewModelId` when
a model is picked from the chat selector
- **`handleGoHome`**: Restores `selectedChatModel` from the sidebar's
`selectedModelId` instead of clearing it
- **`applyLaunchDefaults`**: Syncs `selectedChatModel` when restoring
saved defaults on page load
- **`handleChatSend`**: Sets `chatLaunchState = "ready"` before calling
`sendMessage` when the model is already running, ensuring the chat view
renders correctly on view transition
## Why It Works
The two selectors were backed by different store properties that were
never kept in sync. By updating both properties in every selection path
(sidebar pick, chat pick, go-home, page restore), they always reflect
the same model. The `chatLaunchState = "ready"` fix mirrors what
`launchModelForChat` already does (line 2649) and prevents the chat
state from being "idle" when transitioning from the welcome view to the
chat view.
## Test Plan
### Manual Testing
<!-- Hardware: MacBook Pro M4 Max 48GB -->
- Select a model from sidebar "Load Model" dropdown → chat input
selector updates to match
- Select a model from chat input selector → sidebar dropdown updates to
match
- Launch a model, chat with it, click "Go Home" → both selectors show
the same model
- Refresh the page → both selectors show the previously selected model
- With a running model, type and send a message from the welcome view →
chat starts directly without flashing the recommended models view
### Automated Testing
No new automated tests — this is a UI state synchronization fix in the
Svelte dashboard with no backend changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Follow-up to #1601 (downloads page contrast fix). The HOME and DOWNLOADS
navigation links in the top-right header use `text-exo-light-gray`
(`oklch(0.6 0 0)`) which is too dim against the dark header background.
## Changes
Changed both nav links in `HeaderNav.svelte` from `text-exo-light-gray`
to `text-white/70` for better visibility. Hover state
(`text-exo-yellow`) is unchanged.
## Why It Works
`text-white/70` provides noticeably better contrast against
`bg-exo-dark-gray` while still looking subdued relative to the yellow
accent color on hover. This is consistent with the approach used in
#1601.
## Test Plan
### Manual Testing
- Verified both links are clearly readable on the home page and
downloads page
- Hover state still transitions to yellow as expected
### Automated Testing
- Dashboard builds successfully
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Bumps opacity on dark-grey text/icons on the downloads page that were
nearly invisible against the dark background
- Informational text (GB sizes, speeds, disk free, model IDs) → full
`text-exo-light-gray`
- Interactive icons (delete, resume, retry) → `/70` at rest
- Hover-only elements (download button) → `/60`
## Test plan
- [ ] Open http://localhost:52415/downloads with models downloaded
- [ ] Verify GB downloaded amounts are clearly visible
- [ ] Verify delete trash icons are visible (not just on hover)
- [ ] Verify download speed text is readable
- [ ] Verify "paused" labels and resume buttons are visible
- [ ] Verify "X GB free" disk labels in column headers are readable
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Summary
- The cancellation logic for issue #1215 already exists in
`get_transition_events()` (`src/exo/master/placement.py:208-227`) — when
an instance is deleted, `TaskStatusUpdated(Cancelled)` events are
emitted for all Pending/Running tasks on that instance
- Combined with PR #1276's token-boundary cancellation in runners, the
full pipeline works end-to-end
- However, the existing test
`test_get_transition_events_delete_instance` passed `{}` for tasks, so
this path was never exercised
- This PR adds 4 tests covering the cancellation behavior:
- Running tasks are cancelled on instance deletion
- Pending tasks are cancelled on instance deletion
- Completed/Failed/TimedOut/Cancelled tasks are left alone
- Only tasks matching the deleted instance are cancelled
Closes#1215
## Test plan
- [x] `uv run pytest src/exo/master/tests/test_placement.py -v` — all 15
tests pass
- [x] `uv run basedpyright` — 0 errors
- [x] `uv run ruff check` — all checks passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Adds a **resume button** (download icon) to paused pending downloads
(those with partial progress)
- Adds a **download button** to not-started pending downloads
- Both buttons call the existing `startDownload()` function which
handles both new downloads and resuming partial ones
- Previously, paused downloads only showed a "paused" label with no
action, and not-started downloads showed "..." with no way to trigger
them
## Test plan
- [ ] Build dashboard (`cd dashboard && npm run build`)
- [ ] Start exo, navigate to Downloads tab
- [ ] Verify paused downloads show a clickable resume (download arrow)
icon below the progress bar
- [ ] Verify not-started pending downloads show a clickable download
icon
- [ ] Click both button types and confirm downloads start/resume
> Note: Screenshot could not be captured because the dashboard requires
the exo backend API to render, and exo has a pre-existing
`Keypair.generate()` startup bug on main.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Split from #1547 per review feedback.
When scanning existing download status, `get_downloaded_size()` returns
bytes from either the final file or its `.partial` counterpart — so a
`.partial` file with all bytes downloaded (e.g. process killed before
hash verification and rename) could be falsely reported as complete.
The previous approach (commit 3b54e7df) added a byte-comparison fallback
in the coordinator (`downloaded >= total > 0`), but this suffered from
the same `.partial` conflation issue.
**Fix:** Check whether the final (non-`.partial`) file actually exists
on disk before marking status as `"complete"` in `download_utils.py`.
This is the only reliable signal that hash verification passed and the
rename from `.partial` succeeded. The coordinator-level byte comparison
is removed since the source now reports correctly.
### Changes
- `download_utils.py`: Add `final_file_exists` check — only report
`"complete"` when the renamed, hash-verified file exists with matching
size
- `coordinator.py`: Revert to simple `progress.status == "complete"`
check, removing the byte-comparison fallback
**Note:** The corresponding byte-comparison workaround in #1547 should
also be removed.
## Test plan
- [x] basedpyright — 0 errors
- [x] ruff check — passes
- [x] pytest — 218 passed, 1 skipped (1 pre-existing Rust bindings
failure)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
When a model has multiple quantization variants (e.g., 4-bit, 8-bit),
expanding the group shows each variant's quantization and size — but
there's no way to see which specific nodes have each variant downloaded.
The top-level model group has an (i) info button, but the individual
variant rows don't. This makes it hard to tell which nodes have which
quantization.
## Changes
- Added an (i) info button to each expanded variant row in
`ModelPickerGroup.svelte`
- When clicked, opens the existing info modal scoped to that single
variant (showing its quantization, size, and which nodes have it
downloaded)
- Changed the variant row element from `<button>` to `<div
role="button">` to allow nesting the info `<button>` (matching the
pattern used by the top-level model row)
## Why It Works
Reuses the existing `onShowInfo` callback and info modal by constructing
a synthetic single-variant `ModelGroup` from the clicked variant. No
changes needed to `ModelPickerModal.svelte` — the info modal already
handles single-variant groups correctly.
## Test Plan
### Manual Testing
<!-- Hardware: MacBook Pro -->
- Open dashboard, open model picker
- Expand a model with multiple variants — each variant row now has an
(i) icon
- Click a variant's (i) — the info modal shows that single variant's
quantization/size and which nodes have it downloaded
### Automated Testing
- Dashboard builds successfully with `npm run build`
- `nix fmt` passes
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
The current chat experience requires users to manually select a model
before they can start chatting. This creates unnecessary friction —
especially for new users who may not know which model to pick.
Additionally, models that are already running or downloading appear
greyed out in the picker if available memory is low, making them
impossible to select even though they're already loaded.
## Changes
### New: ChatModelSelector component
- Category-based model recommendations (Best for Coding, Writing,
Agentic, Biggest) shown on the idle chat screen
- Tiered model ranking system (`AUTO_TIERS`) that picks the best model
fitting available memory
- Fixed-position tooltips that escape overflow-hidden ancestors
### Auto model selection from chat
- Typing in an empty chat auto-picks the best model using tier rankings
- Prefers already-running models over launching smaller new ones (avoids
the "launched a big model but recommends a smaller one" bug)
- If a model is already downloading/loading, attaches to existing
progress instead of launching a duplicate
### Launch-from-chat state machine
- Full `idle → launching → downloading → loading → ready` flow with
inline progress display
- `pendingAutoMessage` queue: messages typed during model launch are
remembered and sent once model is ready
- Fixed `$effect` race condition where restore effect competed with
auto-advance effect, causing lost messages
### Model picker improvements
- Instance status badges: green dot (ready), blue pulsing (downloading),
yellow pulsing (loading) on both group and variant rows
- "Ready" availability filter in ModelFilterPopover
- **Models with active instances (ready, downloading, loading) are never
greyed out** regardless of memory — they're already loaded/in-progress
and must be selectable
### Home page model display
- Model picker button shows the best running model name instead of "—
SELECT MODEL —" when a model is already active
- Uses the same `bestRunningModelId` tier-based derived as the chat
auto-selection
### Chat sidebar & form
- ChatSidebar conversation management with rename, delete, "New Chat"
- `userForcedIdle` guard prevents reactive effects from overriding
explicit user actions (e.g., clicking "New Chat")
- ChatForm simplified: uses ModelPickerModal instead of inline dropdown,
`modelDisplayOverride` prop for showing auto-selected models
## Why It Works
The tier-based auto-selection (`getAutoTierIndex` + `pickAutoModel`)
ensures users always get the best model their cluster can run. By
checking running instances first and comparing tiers, we avoid the bug
where loading a large model reduces available memory and causes a
smaller model to be recommended. The `$effect` ordering fix (moving the
`chatLaunchState === "ready"` guard after the `hasRunningInstance`
check) ensures pending messages are always delivered. The instance
status override in ModelPickerGroup (`anyVariantHasInstance`) ensures
models with active instances bypass memory-based greying regardless of
available RAM.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Launch exo, navigate to Chat view
- Verify idle chat shows category recommendations (Coding, Writing,
Agentic, Biggest) based on available memory
- Type a message without selecting a model → verify auto-selection picks
the best tier model
- Launch a model, go to New Chat → verify the input bar shows the
running model name (not "SELECT MODEL")
- Launch a model, go to Home → verify model picker button shows the
running model
- While a model is downloading, send a message → verify download
progress appears and message is queued
- Open model picker with a model running → verify it is NOT greyed out
and is selectable
- Click "New Chat" while in a conversation → verify it resets to idle
with model selector
### Automated Testing
- Dashboard builds successfully (`cd dashboard && npm run build`)
- No TypeScript errors in the build output
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
Large prompts (70K+ tokens / ~500KB+ JSON) cause exo to silently crash.
The root cause is an unhandled `PublishError::MessageTooLarge` from
gossipsub when serialized `TextGeneration` commands exceed the 1MB
`max_transmit_size` limit.
The error propagates as a generic Python exception through the PyO3
bindings. Since `_networking_publish` in `router.py` only catches
`NoPeersSubscribedToTopicError` and `AllQueuesFullError`, the unhandled
exception crashes the networking async task, causing exo to shut down
silently — no error message, no API response.
## Changes
- **Rust (PyO3 bindings):** Add `MessageTooLargeError` exception class
and handle `PublishError::MessageTooLarge` explicitly in the gossipsub
publish path, matching the existing pattern for
`NoPeersSubscribedToTopicError` and `AllQueuesFullError`
- **Python (router):** Catch `MessageTooLargeError` in
`_networking_publish` and log a warning with the message size,
preventing the networking task from crashing
## Reproduction
On a multi-node cluster with a large model (e.g., GLM-5 754B tensor
parallel over JACCL RDMA):
1. Send a chat completion request with ~70K+ tokens
2. exo silently shuts down — no error logged, curl gets no response
3. With shorter prompts (< ~50K tokens): works fine
## Test plan
- Verified `cargo check` passes for `networking` and `exo_pyo3_bindings`
crates
- Verified `ruff check` passes for modified Python files
- Manual testing on 4× Mac Studio M3 Ultra cluster: 50K token requests
pass, 70K+ previously caused silent shutdown, now logs a warning and
drops the oversized message gracefully
Co-authored-by: vsm <vsm@nomail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
in a few places we instantiate task groups early, in others we have an
optional task group.
this standardizes these patterns into a lazy task group, which queues
tasks until it is entered, at which point it enters its inner task group
and starts all its queued tasks. it should be noted that no queued tasks
will be started until the group itself is started.
we were checking if qwen3 had a transformers Qwen3DecoderLayer rather
than an mlx Qwen3MoeDecoderLayer causing an assertion error on loading
qwen models - this corrects it to the actual layer type
## Motivation
Large models + large prompts + pipeline RING = 0.2tps generation speed
Large models + large prompts + pipeline JACCL = 15 tps generation speed
Why? Well, MLX_METAL_FAST_SYNCH is on in pipeline JACCL.
## Changes
Just turn on fast synch everywhere, especially as GPU locks are old news
Also, changed to use mx.device_info as mx.metal.device_info is going to
be deprecated.
## Why It Works
Some magic thing that happens in the mlx backend. I really tried to find
a regression but couldn't. I will probably try again at some point.
## Test Plan
### Manual Testing
Did a bunch, no longer 0.2tps
### Automated Testing
We'll do that today.
The bench harness accessed the serialized DownloadCompleted field as
"totalBytes", but the Python field is `total: Memory` which serializes
to "total" (not snake_case, so the camelCase alias generator leaves it
unchanged). This caused a KeyError when --danger-delete-downloads
needed to free disk space by deleting existing models.
Changed the key from "totalBytes" to "total" to match the actual
serialized JSON structure.
Test plan:
- CI
- Reproduced KeyError on unfixed code by filling disk on a test node
to 14GB free and running exo-bench with a 16GB model
(Meta-Llama-3.1-8B-Instruct-bf16) with --danger-delete-downloads
- Verified fixed code successfully deletes smaller models to free
space and completes the benchmark run
## Summary
- **Complete onboarding wizard**: 7-step flow guiding new users from
Welcome → Your Devices (topology) → Add More Devices (animation) →
Choose Model → Download → Load → Chat
- **Native macOS integration**: NSPopover welcome callout anchored to
menu bar icon on first launch, polished DMG installer with
drag-to-Applications arrow
- **Dashboard UX polish**: auto-download on model select, toast
notifications, connection banner, skeleton loading, download progress in
header, recommended model tags, sidebar hidden in home state for cleaner
first impression
- **Settings & menu bar overhaul**: native Settings window with Advanced
tab, onboarding reset, chat sidebar toggle
## Test plan
- [ ] Fresh install: verify onboarding wizard appears and flows Welcome
→ Topology → Animation → Model → Download → Load → Chat
- [ ] Verify topology shows real device data in onboarding step 2
- [ ] Verify selecting a model in the main dashboard picker
auto-triggers download
- [ ] Verify chat sidebar is hidden on home view, appears when chat is
active
- [ ] Verify DMG installer has white background with curved arrow
- [ ] Verify NSPopover appears anchored to menu bar icon on first launch
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Summary
- When MlxJaccl (RDMA) placement is requested but no RDMA-connected
cycles exist, raise a clear ValueError instead of silently falling back
to non-RDMA cycles
- Split from #1519 for independent review
## Test plan
- [x] basedpyright — 0 errors
- [x] ruff check — passes
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- When clicking "Send Bug Report" in the macOS app, users are now
prompted with "What's the issue? (optional)" before the diagnostic
upload begins
- The user's description is included in the uploaded report JSON
(`user_description` field)
- After successful upload, a "Create GitHub Issue" button opens the
browser to `github.com/exo-explore/exo/issues/new` pre-filled with the
user's description, macOS version, and EXO version
## Changes
- **`ContentView.swift`**: Replaced simple button with multi-phase
inline UI (idle → prompting → sending → success/failure). Added
`openGitHubIssue()` helper using `URLComponents` for pre-filled GitHub
issue URLs.
- **`BugReportService.swift`**: Added `userDescription` parameter to
`sendReport()` and `makeReportJson()`, included in the JSON payload when
non-empty.
- Removed the previous `BugReportModal.svelte` approach (the feature
belongs in the macOS app, not the dashboard).
## Test plan
- [ ] Build the Xcode project and run the macOS app
- [ ] Click "Send Bug Report" → verify text prompt appears
- [ ] Type a description, click Send → verify upload succeeds and
"Create GitHub Issue" button appears
- [ ] Click "Create GitHub Issue" → verify browser opens with pre-filled
template
- [ ] Test Cancel returns to idle, test empty description still works
- [ ] Test failure case (e.g. with exo stopped) shows error and dismiss
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Remove legacy MlxIbvInstance references from ChatSidebar and ModelCard
components
- MlxIbv was replaced by MlxJaccl; these are leftover type checks
- Split from #1519 for independent review
## Test plan
- [x] Visual inspection of dashboard components
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- Changed blue info badge "RDMA AVAILABLE" to yellow warning badge "RDMA
NOT ENABLED" — more accurately describes the state
- Added hover tooltip with enable instructions to all views (was missing
in 2 of 4 instances)
- Warning icon instead of info icon, consistent with other cluster
warnings (TB cycle, macOS mismatch)
## Screenshots
**Badge (yellow warning):**
**Hover tooltip with instructions:**
## Test plan
- [x] Dashboard builds successfully
- [ ] Verify badge appears when 2+ TB5 nodes have RDMA disabled
- [ ] Verify hover tooltip shows in normal layout
- [ ] Verify hover tooltip shows in topology-only mode
- [ ] Verify dismiss button works
- [ ] Verify compact badge in status bar shows yellow warning
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
Models in EXO_MODELS_PATH are pre-downloaded into read-only directories
and must not be deleted. The DownloadCoordinator had no awareness of
these paths, so they never appeared as completed downloads in cluster
state, and the bench harness could attempt to delete them when freeing
disk space.
Added a `read_only: bool` field to `DownloadCompleted` (default False).
The DownloadCoordinator now checks `resolve_model_in_path` in
`_start_download`, proactively scans EXO_MODELS_PATH in
`_emit_existing_download_progress` to emit DownloadCompleted events for
all pre-downloaded models (overriding DownloadPending from the regular
scan), and refuses deletion of read-only models. The bench harness
filters out read-only models from deletion candidates.
Test plan:
- Ran with EXO_MODELS_PATH. Available models now show as downloaded in
the UI. There isn't good UI for the fact they can't be deleted, but it
should work with exo_bench.
The Qwen3-Coder-Next model card TOML files were missing family,
quantization, base_model, and capabilities fields. This caused them not
to appear under the Qwen family filter in the dashboard's model picker.
Added the missing metadata to all five variants (4bit, 5bit, 6bit, 8bit,
bf16), matching the format used by the existing Qwen3-Coder-480B model
cards.
Test plan:
- Eyeballs
Users with pre-existing model files (e.g. on shared NFS mounts or from
prior downloads) had no way to point exo at those directories without
going through the download coordinator. EXO_MODELS_DIR only moves the
download target directory, it doesn't support read-only search paths.
Added EXO_MODELS_PATH environment variable as a colon-separated list of
directories to search for models. When the worker's plan loop encounters
a DownloadModel task, it checks these directories first and emits a
synthetic DownloadCompleted event if found, bypassing the download
coordinator entirely. The runner's build_model_path also checks these
directories first so the correct path is used during model loading.
This keeps the existing event sourcing state machine unchanged — the
DownloadCompleted event propagates naturally through the system, so
_load_model and all downstream logic work without modification.
Test plan:
- `s1@s1s-Mac-Studio ~ % EXO_LIBP2P_NAMESPACE=jake EXO_MODELS_PATH="/Volumes/Definitely Leo's SSD" nix --extra-experimental-features 'nix-command flakes' run github:exo-explore/exo/f2babbc2f742357d97dc177619fec062ef545be4`
- Started mlx-community/Qwen3-Coder-Next-4bit - it's present on the disk
and it worked.
- Renamed one safetensor of mlx-community/Qwen3-Coder-Next-4bit on the
disk. It then started the download locally, as expected.
## Problem
When a request is cancelled during prefill, the cancellation is not
detected until `check_for_cancel_every` additional tokens have been
generated. This is because `tokens_since_last_cancel_check` is
initialized to `0`, meaning the first cancel check only happens after
generating `check_for_cancel_every` tokens post-prefill.
For long prefills (which are the most likely to be cancelled), this adds
unnecessary latency before the cancellation is actually honoured.
## Fix
Initialize `tokens_since_last_cancel_check` to `check_for_cancel_every`
instead of `0`, so the very first token generated after prefill triggers
an immediate cancel check.
```diff
- tokens_since_last_cancel_check = 0
+ tokens_since_last_cancel_check = check_for_cancel_every
```
## Impact
- Cancellations issued during prefill are detected immediately when
generation begins
- No change in behaviour for non-cancelled requests (the counter resets
to `0` after each check as before)
- 1 line changed
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Summary
- Add downloaded_bytes field to existing DownloadPending event for
accurate resume progress
- Minimal change per maintainer directive — no new download states
introduced
## Test plan
- [x] 42 tests passed, 1 skipped
- [x] Verified downloaded_bytes populates correctly for partial
downloads
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
we were seeing some crashes as lost download tasks were trying to push
data toward a deleted coordinator. this cancels download tasks with the
coordinator's shutdown on master election
## Motivation
We're going to be refactoring the llm inference code, so we should split
the runner up into parts while we can.
## Test Plan
### Manual Testing
Works on single node, at least.
### Automated Testing
Passes CI. Will be tested by our tests today.
addresses some election edge cases where a new worker with an old master
would get stuck on the old workers buffer index - we now use new system
ids each time we instantiate a node, and each event-producing system has
a unique system id for its lifespan (until the master moves).
The bench script downloads models during the planning phase but doesn't
record how long the download took, making it difficult to track download
performance for a given model over time.
Modified `run_planning_phase` to return download metadata: whether a
fresh download occurred, the wall-clock duration, and the model size in
bytes. These fields are included in every JSON output row alongside the
existing per-run metrics, and a summary line is logged to the console.
This allows filtering bench results by `download_occurred` and grouping
by `model_id` to compute average download times across runs.
Test plan:
```
# existing model
jake@maverick:/data/users/jake/repos/exo/ > nix run .#exo-bench -- --host s1 --model mlx-community/gpt-oss-120b-MXFP4-Q8 --pp 128 --tg 128
...
2026-02-20 15:23:49.081 | INFO | __main__:main:340 - Planning phase: checking downloads...
2026-02-20 15:23:49.152 | INFO | harness:run_planning_phase:402 - Started download on 12D3KooWKx41iikn188ozrxSdoG26g88jFCfie9wEA1eQR8csbPm
2026-02-20 15:23:49.184 | INFO | __main__:main:352 - Download: model already cached
...
Wrote results JSON: bench/results.json
jake@maverick:/data/users/jake/repos/exo/ > cat bench/results.json
[
{
"elapsed_s": 2.9446684420108795,
"output_text_preview": "The user just typed a long series of \"a\". Possibly they are testing. There's no explicit question. Could be they want a response? Might be a test of handling long input. We can respond politely, ask i",
"stats": {
"prompt_tps": 117.7872141515621,
"generation_tps": 85.49598231498028,
"prompt_tokens": 129,
"generation_tokens": 128,
"peak_memory_usage": {
"inBytes": 68215145744
}
},
"model_short_id": "gpt-oss-120b-MXFP4-Q8",
"model_id": "mlx-community/gpt-oss-120b-MXFP4-Q8",
"placement_sharding": "Pipeline",
"placement_instance_meta": "MlxRing",
"placement_nodes": 1,
"instance_id": "68babc2a-6e94-4c70-aa07-7ec681f7c856",
"pp_tokens": 128,
"tg": 128,
"repeat_index": 0
}
]%
# no change to output
```
```
# missing model
jake@maverick:/data/users/jake/repos/exo/ > nix run .#exo-bench -- --host s1 --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit --pp 128 --tg 128
...
2026-02-20 15:24:42.553 | INFO | __main__:main:340 - Planning phase: checking downloads...
2026-02-20 15:24:42.625 | INFO | harness:run_planning_phase:402 - Started download on 12D3KooWKx41iikn188ozrxSdoG26g88jFCfie9wEA1eQR8csbPm
2026-02-20 15:25:37.494 | INFO | __main__:main:350 - Download: 54.9s (freshly downloaded)
...
Wrote results JSON: bench/results.json
jake@maverick:/data/users/jake/repos/exo/ > cat bench/results.json
[
{
"elapsed_s": 1.500349276990164,
"output_text_preview": "It seems like you've entered a large number of 'a's. If you'd like to discuss something or ask a question, I'm here to help. If not, is there anything else I can assist you with? \n\nIf you're intereste",
"stats": {
"prompt_tps": 395.43264952543666,
"generation_tps": 128.03520443181478,
"prompt_tokens": 129,
"generation_tokens": 128,
"peak_memory_usage": {
"inBytes": 5116952079
}
},
"model_short_id": "Meta-Llama-3.1-8B-Instruct-4bit",
"model_id": "mlx-community/Meta-Llama-3.1-8B-Instruct-4bit",
"placement_sharding": "Pipeline",
"placement_instance_meta": "MlxRing",
"placement_nodes": 1,
"instance_id": "ccd9bd71-d4cc-4b75-a37f-98090544626a",
"pp_tokens": 128,
"tg": 128,
"repeat_index": 0,
"download_duration_s": 54.88322358299047
}
]%
# one new field
```
## Summary
- On macOS, memory monitoring relied exclusively on `macmon` — the
psutil fallback was explicitly disabled (`memory_poll_rate = None`)
- When `macmon` is not installed (e.g., mac-mini-2 through mac-mini-4 in
our cluster), **no memory data was reported**, causing nodes to show 0GB
memory in the cluster state
- This blocked the scheduler from placing shards on those nodes since it
had no memory data to work with
- Fix: when `macmon` is not found on Darwin, fall back to psutil-based
memory polling (`memory_poll_rate = 1`)
## Root cause
`InfoGatherer` has two memory monitoring paths:
1. `macmon` (Darwin-only): provides memory + GPU/CPU/power stats
2. `psutil` (non-Darwin fallback): provides memory via
`MemoryUsage.from_psutil()`
Line 378 disabled psutil on Darwin: `memory_poll_rate = None if
IS_DARWIN else 1`
Line 389 only starts macmon if the binary exists: `if
shutil.which("macmon") is not None`
If macmon is missing on Darwin, **neither path runs** — zero memory
reported.
## Test plan
- [ ] Verify `uv run basedpyright` passes (0 errors confirmed)
- [ ] Verify `uv run ruff check` passes (confirmed)
- [ ] Verify `uv run pytest src/exo/utils/info_gatherer/` passes (2/2
confirmed)
- [ ] Deploy to cluster nodes without macmon and verify memory appears
in `/state`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Summary
- Show estimated time remaining during prefill (prompt processing phase)
- Track prefill start time via performance.now() and extrapolate from
observed token throughput
- Display ~Xs remaining or ~Xm Ys remaining next to the percentage on
the progress bar
- Wait 200ms before showing ETA to ensure a stable sample window
## Changes
**PrefillProgressBar.svelte**: Add etaText derived computation that
calculates remaining time from (remainingTokens / tokensPerMs). Renders
in a new flex row below the progress bar alongside the percentage.
**app.svelte.ts**: Add startedAt: number field to PrefillProgress
interface. Set on first prefill_progress SSE event, preserved across
subsequent updates.
## Test plan
- [ ] Start inference with a long prompt (10k+ tokens) on a multi-node
cluster
- [ ] Verify the progress bar shows ~Xs remaining after ~200ms of
prefill
- [ ] Verify the ETA decreases as prefill progresses
- [ ] Verify short prefills (<200ms) dont flash a briefly-visible ETA
- [ ] Verify ETA disappears when prefill completes and token generation
begins
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Summary
- Fixed stale docstring in `seed_models()` that referenced
`.cache/huggingface/hub` when the function actually moves models to
`EXO_MODELS_DIR` (resolved via `ensure_models_dir()`)
- The old docstring was misleading for AI coding agents analyzing the
codebase, causing incorrect conclusions about model storage paths
## Changes
`src/exo/download/download_utils.py`: Updated docstring from `"Move
model in resources folder of app to .cache/huggingface/hub"` to `"Move
models from resources folder to EXO_MODELS_DIR."`
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
Coordinator can be a random rank. Let's just fix this to rank 0 as
that's what we typically assume.
## Test Plan
### Manual Testing
Works as normal on 2 nodes.
Let's wait for a little more testing to merge this.
---------
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
TB has better bandwidth and latency than ethernet. We should prioritise
TB5 where possible. This drastically improves distributed image
generation performance.
## Test Plan
### Manual Testing
Saw on the dashboard that TB (169.254) addresses were prioritised.
Tested that image models scale much better.
### Automated Testing
No regression on Kimi K2.5
## Motivation
Occasionally, memory does not get released when we shut down. There is
no reason to delay deleting the model.
Also handles can become None during shutdown, causing TypeErrors which
are not handled and bringing down exo.
Similarly, we were closing the event sender in the wrong place.
Also let's not verify the SSL certificate for http connections to local
peers, as this is failing sometimes and crashing.
## Test Plan
### Manual Testing
No more crashes as described.
## Motivation
OpenCode shows <think> tags and not thinking blocks as we aren't
following the API specs properly.
Claude was also getting horrible prefix cache hits because it sends
headers.
## Changes
Handle thinking tokens properly by placing them in think tags for each
of the API endpoints.
Also support DeepSeekV3.2 tool calling properly as a minor feature.
Strips Claude headers at the API level.
## Test Plan
### Manual Testing
Tested OpenCode manually
Needs testing with Claude.
### Automated Testing
All CI and tests passing - added a new e2e test for DeepSeekV32 tool
parsing.
Rebuilt from scratch (replaces PR #1543). Detects when Mac Studio uses
RDMA over en2 (TB5 port next to Ethernet) which does not support RDMA.
Shows dismissible warning banner with hover tooltip showing affected
devices, SVG rear panel illustration, and fix instructions. 205 lines in
+page.svelte.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
Our hits are really bad at the moment (0.2%). This PR makes it 98.5% on
average.
## Changes
Also adds an example for how to run Claude using Exo.
## Why It Works
Claude sends some billing and session headers that change with each
message.
## Test Plan
### Manual Testing
Works in manual testing.
this should never have been a separate event, but i didnt quite
communicate that well when this was merged. convert PrefillProgress to a
chunk like the rest of the runner responses.
tested with Llama-3.3-70B, prefill progress events still show up in the
dashboard as usual
c2f2111b extracted shared utilities from exo_bench.py into harness.py
but accidentally dropped the run_planning_phase function and
--danger-delete-downloads CLI argument in the process.
Restored run_planning_phase in harness.py (where its dependencies now
live) and re-added the --danger-delete-downloads argument to
add_common_instance_args. Re-wired the planning phase call in
exo_bench.py's main() before the benchmark loop.
## Summary
- `ResponsesStreamEvent` was defined in `openai_responses.py` as a union
of all 11 streaming event types but never imported or used anywhere in
the codebase
- Import it in the responses adapter and add a `_format_sse(event:
ResponsesStreamEvent) -> str` helper
- Replace 13 hardcoded `f"event: {type}\ndata:
{event.model_dump_json()}\n\n"` strings with `_format_sse()` calls
## Test plan
- [x] `uv run basedpyright` — 0 errors
- [x] `uv run ruff check` — all checks passed
- [x] `nix fmt` — 0 files changed
- [x] `uv run pytest` — 188 passed, 1 skipped
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
# Cancel task finalization (main.py)
After forwarding the cancel to the runner supervisor, emit TaskStatusUpdated(Complete) for the cancel task itself. This ensures the cancel task is properly removed from state.tasks.
## Motivation
There's no way to easily use the cancellation features we added! Also,
prefill can take ages so let's allow cancelling out of that.
## Changes
Wiring up our existing functionality to easily cancel during generation
(and adding stuff to do so during prefill)
## Test Plan
### Manual Testing
Tested it works during both prefill and decode.
### Automated testing
Needs testing to see if this causes a GPU timeout error on large prefill
on large models in pipeline parallel. However, from manually testing GLM
5 pipeline ring on 2 nodes, and from reading the code, it does not seem
like this will be the case.
## Motivation
Users processing long prompts have no visibility into when token
generation will start. This feature adds a progress bar showing prefill
progress, giving users real-time feedback during prompt processing.
## Changes
### Backend
- Added `PrefillProgress` event type with `command_id`,
`processed_tokens`, `total_tokens`
- Added `PrefillProgressResponse` type (though now using direct callback
approach)
- Wired `prompt_progress_callback` through MLX's `stream_generate()`
- Progress events sent directly from callback for real-time updates (not
batched)
- API generates SSE named events: `event: prefill_progress\ndata: {...}`
- Added `PrefillProgressData` dataclass and `StreamEvent` union type in
API
### Dashboard
- Added `PrefillProgress` interface to store
- Updated SSE parsing to handle `event:` lines (named events)
- Created `PrefillProgressBar.svelte` with animated progress bar
- Shows "Processing prompt: X/Y tokens" with percentage
- Progress bar disappears when first token arrives
## Why It Works
MLX's `stream_generate()` accepts a `prompt_progress_callback(processed,
total)` that's called after each prefill chunk. By sending events
directly from this callback (rather than yielding from the generator),
progress updates are sent in real-time during prefill.
Using SSE named events (`event: prefill_progress`) maintains full
OpenAI/Claude API compatibility - standard clients ignore named events
they don't recognize, while the exo dashboard explicitly listens for
them.
## Test Plan
### Manual Testing
- Hardware: MacBook Pro M3 Max
- Set `prefill_step_size=256` for more frequent updates
- Tested with long prompts (pasted large documents)
- Verified progress bar updates incrementally during prefill
- Confirmed progress bar disappears when generation starts
- Tested with curl - standard `data:` events still work normally
Here is it working:
https://github.com/user-attachments/assets/5cc6f075-c5b2-4a44-bb4d-9efb246bc5fe
### Automated Testing
- Type checker passes (0 errors)
- All 192 tests pass
- Dashboard builds successfully
### API Compatibility
- Named SSE events are ignored by OpenAI SDK clients
- Regular token data uses standard `data: {...}` format
- `[DONE]` sentinel works as expected
---
**Note:** `prefill_step_size` is temporarily set to 256 for testing.
Should be changed back to 2048 before merging for production
performance.
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Evan <evanev7@gmail.com>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Summary
- `_test_internet_connection()` previously only tried `1.1.1.1:443`,
which some ISPs/networks block, causing exo to incorrectly report no
internet and fail downloads on startup
- Now tries `1.1.1.1`, `8.8.8.8`, and `1.0.0.1` in sequence, succeeding
if any endpoint responds
- Returns early on first success for minimal latency in the common case
Fixes#1425
## Test plan
- [ ] Verify downloads work on networks that block `1.1.1.1`
- [ ] Verify existing behavior unchanged on networks where `1.1.1.1`
works
- [ ] Verify `internet_connection` is set to `False` only when all three
endpoints fail
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Summary
- `MpReceiver.close()` did not unblock threads stuck on `queue.get()` in
`receive_async()`, causing abandoned threads (via
`abandon_on_cancel=True`) to keep the Python process alive indefinitely
after tests pass
- This caused the `aarch64-darwin` CI jobs in PR #1462 to hang for ~6
hours until the GitHub Actions timeout killed them
- Sends an `_MpEndOfStream` sentinel before closing the buffer,
mirroring what `MpSender.close()` already does
## Test plan
- [x] `uv run basedpyright` — 0 errors
- [x] `uv run ruff check` — clean
- [x] `nix fmt` — 0 changed
- [x] `uv run pytest` — 188 passed, 1 skipped in 12s (no hang)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Summary
- Adds model cards for MiniMax M2.5 in three quantizations: 4bit (~129
GB), 6bit (~186 GB), 8bit (~243 GB)
- No code changes needed — `MiniMaxM2ForCausalLM` is already in the
tensor parallel whitelist and `MiniMaxShardingStrategy` is already
implemented in `auto_parallel.py`
- Credit to @vskiwi for confirming MiniMax M2.5 works out of the box
with existing code
Closes#1480
## Test plan
- [x] `basedpyright` passes with 0 errors
- [x] `ruff check` passes
- [x] `pytest` passes (260 passed, 1 skipped)
- [ ] Verify MiniMax M2.5 models appear in model selector on dashboard
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
GPT OSS tool calling issues.
## Changes
Fixes those and adds a bunch of evals for tool calling.
Fixes GLM5 prefix caching, where CacheList wasn't getting handled
properly.
Extracts a bunch of the setup functionality of exo bench to a harness
that can be reused elsewhere, such as in the tool calling eval.
## Test Plan
### Automated Testing
Let's run the evals for all models
## Summary
- Adds `thinking_toggle` capability to 26 model cards that support
toggling thinking mode on/off
- GPT-OSS models (20b, 120b) excluded — they always think and don't
support toggling
- Dashboard UI updated to check for `thinking_toggle` capability before
showing the toggle button
## Test plan
- [x] `uv run basedpyright` — 0 errors
- [x] `uv run ruff check` — all checks passed
- [x] `nix fmt` — 0 files changed
- [x] `uv run pytest` — 188 passed, 0 failed
- [x] Security review passed (no secrets, eval/exec, innerHTML, or dep
changes)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Closes#1510
There is currently no reliable way to run exo on an air-gapped or offline cluster where models are pre-staged on local disks. The two existing mechanisms — `--no-downloads` and `HF_HUB_OFFLINE=1` — each cover only a subset of the problem:
1. **`--no-downloads` blocks model loading**: When passed, `DownloadCoordinator` is not created. No `NodeDownloadProgress` events are ever emitted, so `_model_needs_download()` in `plan.py` perpetually returns `DownloadModel`, short-circuiting `_load_model()` and preventing the model from ever being loaded.
2. **`HF_HUB_OFFLINE=1` doesn't cover exo's aiohttp code**: exo's download pipeline primarily uses raw `aiohttp` for HTTP operations (file list fetching, file downloads, HEAD verification), not the `huggingface_hub` library. These calls will attempt connections and time out on air-gapped networks.
3. **`skip_internet` is not propagated to `download_file_with_retry()`**: Even when `internet_connection = False`, the `_download_file()` function still makes HTTP HEAD calls via `file_meta()` to verify local files and unconditionally attempts downloads for missing files.
## Changes
### `src/exo/main.py`
- Add `--offline` flag to `Args` with env var detection (`EXO_OFFLINE=1`, `HF_HUB_OFFLINE=1`)
- Pass `offline` to `DownloadCoordinator` at creation and re-creation (election loop)
### `src/exo/download/coordinator.py`
- Add `offline: bool = False` field
- In offline mode: set `internet_connection = False` immediately in `__post_init__`, skip `_test_internet_connection()` ping (avoids 3s timeout), skip `_check_internet_connection` periodic loop
- In `_start_download()`: if model is not fully available locally, emit `DownloadFailed` with clear message instead of starting a download task
### `src/exo/download/download_utils.py`
- Add `skip_internet: bool` parameter to `download_file_with_retry()` and `_download_file()`
- When `skip_internet=True` in `_download_file()`: return local file immediately without HTTP HEAD verification; raise `FileNotFoundError` for missing files
- Propagate `skip_internet` from `download_shard()` to `download_file_with_retry()`
### `src/exo/download/tests/test_offline_mode.py` (new)
- 8 tests covering `_download_file`, `download_file_with_retry`, and `fetch_file_list_with_cache` in offline mode
## Why It Works
Unlike `--no-downloads` which disables `DownloadCoordinator` entirely, `--offline` keeps the coordinator running in a restricted mode. The existing `_emit_existing_download_progress()` disk scanner still runs every 60 seconds, emitting `DownloadCompleted` events for pre-staged models. These events flow through the event-sourcing pipeline and populate `state.downloads`, which unblocks `_model_needs_download()` in `plan.py` — no changes to the planning logic required.
```
--offline flag
→ DownloadCoordinator (offline mode)
→ Skip 1.1.1.1 ping, internet_connection = False
→ _emit_existing_download_progress scans disk
→ Emits DownloadCompleted for pre-staged models
→ _model_needs_download sees DownloadCompleted
→ _load_model proceeds normally
```
## Test Plan
### Automated Testing
- `ruff check` — passes
- 8 new tests in `test_offline_mode.py` — all pass
- 11 existing download tests in `test_download_verification.py` — all pass (no regressions)
### Manual Testing
1. Pre-stage a model on disk (e.g., `~/.exo/models/mlx-community--Qwen3-0.6B-4bit/`)
2. Start exo with `--offline` (or `EXO_OFFLINE=1`)
3. Place an instance via API or dashboard
4. Verify: model loads into memory and inference works without any network calls
### Environment
- macOS (Apple Silicon), multi-node cluster with Thunderbolt interconnect
- Models pre-staged via rsync / NFS mount
## Motivation
- Image editing previously ignored input image dimensions, always
defaulting to 1024x1024
- Size dropdown was hidden in edit mode, giving users no control over
output dimensions
- Portrait/landscape presets used non-standard aspect ratios (1024x1365
/ 1365x1024)
## Changes
- Added "auto" size option that uses input image dimensions for edits,
defaults to 1024x1024 for generation
- Introduced ImageSize Literal type and normalize_image_size() validator
(replaces raw str size fields)
- Updated portrait/landscape presets to standard 1024x1536 / 1536x1024
- Made size selector visible in edit mode (previously hidden)
- Default size changed from "1024x1024" to "auto"
## Why It Works
- "auto" reads actual input image dimensions via PIL at generation time,
so edits preserve the original aspect ratio
- Pydantic field_validator on both ImageGenerationTaskParams and
ImageEditsTaskParams normalizes None → "auto", keeping the API
backward-compatible
## Test Plan
### Manual Testing
- Verify image edits output at the input image's native resolution when
size is "auto"
- Verify size dropdown appears and works in both generate and edit modes
## Motivation
Add GLM 5 support in favor of #1513
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
our old model card search path would override official model cards with
custom model cards - our packaged model cards should always be the
default here
## Summary
- Bumps `mlx-lm` from 0.30.6 to 0.30.7 in `pyproject.toml` and `uv.lock`
## Test plan
- [x] `uv lock` resolves successfully
- [x] `basedpyright` — no new errors (63 pre-existing in unrelated
`test_tool_call_tracker.py`)
- [x] `ruff check` — all checks passed
- [x] `nix fmt` — no formatting changes
- [x] `pytest` — 188 passed, 1 skipped
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Summary
- **Auto-open dashboard** in browser on first launch (uses
`~/.exo/.dashboard_opened` marker)
- **Welcome overlay** with "Choose a Model" CTA button when no model
instance is running
- **Tutorial progress messages** during model download → loading → ready
lifecycle stages
- **Fix conversation sidebar** text contrast — bumped to white text,
added active state background
- **Simplify technical jargon** — sharding/instance type/min nodes
hidden behind collapsible "Advanced Options" toggle; strategy display
hidden behind debug mode
- **Polished DMG installer** with drag-to-Applications layout, custom
branded background, and AppleScript-configured window positioning
## Test plan
- [ ] Launch exo for the first time (delete `~/.exo/.dashboard_opened`
to simulate) — browser should auto-open
- [ ] Verify welcome overlay appears on topology when no model is loaded
- [ ] Launch a model and verify download/loading/ready messages appear
in instance cards
- [ ] Check conversation sidebar text is readable (white on dark, yellow
when active)
- [ ] Verify "Advanced Options" toggle hides/shows sharding controls
- [ ] Build DMG with `packaging/dmg/create-dmg.sh` and verify
drag-to-Applications layout
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
There is an issue on Macs that means that an explicit synchronization is
necessary for memory to be updated from L1 cache. This means that GPU
locks can occur when a spin wait does not see the updated timestamp.
## Changes
Updated in my own personal fork.
## Why It Works
https://github.com/ARM-software/acle/releases
## Test Plan
### Manual Testing
Tested manually that no GPU locks occur (even with multiple simultaneous
instances running) and that the performance differential is negligible
(267 vs 269 tps on Llama 3.2 1B at an approx 10k context.)
------------------------------------------------------
I have seen a GPU lock, specifically when sending a particularly large
chat completion while the model was loading. However, I have since been
unable to reproduce and this may be something I did wrong. Please do
create an issue and tag me if any GPU locks do occur.
---------
Co-authored-by: Jake Hillion <jake@hillion.co.uk>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Users currently manage instances directly, which means if a node
disconnects or connections break, the instance dies and nothing
recreates it. MetaInstance is a declarative primitive: "ensure an
instance matching these parameters always exists." The reconciler
watches for unhealthy or missing backing instances and re-places them
automatically.
## Changes
- **MetaInstance type** (`meta_instance.py`): declarative constraint
with `model_id`, `min_nodes`, optional `node_ids`, and `sharding`
- **Reconciler** (`reconcile.py`): `find_unsatisfied_meta_instances`
checks which MetaInstances lack a healthy backing instance,
`try_place_for_meta_instance` creates one
- **Master loop** (`main.py`): periodically reconciles unsatisfied
MetaInstances; immediate placement on `CreateMetaInstance` command
- **API** (`api.py`): `create_meta_instance` / `delete_meta_instance` /
`GET /meta_instances` endpoints; delete cascades to backing instances
with task cancellation
- **Binding via `meta_instance_id` on Instance** (`instances.py`): no
separate binding event or backing map — the instance carries its parent
MetaInstance ID directly, eliminating race conditions in the reconciler
- **Dashboard**: sidebar shows MetaInstances with their backing instance
status; orphan instances (created directly) still shown separately
- **Tests**: constraint matching, connection health, unsatisfied
detection, exclusive binding, cascade delete with task cancellation
### Recent improvements
- **fix: cancel active tasks on cascade delete** — `DeleteMetaInstance`
now emits `TaskStatusUpdated(Cancelled)` for any Pending/Running tasks
on backing instances before emitting `InstanceDeleted`. Previously,
cascade-deleting backing instances left orphaned task references in
state.
- **Lifecycle logging** — added `logger.info`/`logger.warning` for:
`CreateMetaInstance` (model, min_nodes, sharding), `DeleteMetaInstance`
(with cascade count), reconciler placement success/failure, and retry
decisions with attempt counts in `InstanceHealthReconciler`.
- **GET `/meta_instances` endpoint** — lists all meta-instances without
needing to fetch full state.
- **2 regression tests** — `test_cascade_delete_cancels_active_tasks`
and `test_cascade_delete_skips_completed_tasks` verify the
cascade-delete event sequence.
## Why It Works
Putting `meta_instance_id` on `BaseInstance` makes binding inherent to
instance creation. When the reconciler creates an instance for a
MetaInstance, it tags it via `model_copy`. When the instance is deleted,
the binding disappears with it. This avoids the two bugs that a separate
binding mechanism would introduce:
1. Stale exclusion sets — the reconciler loop can't accidentally bind
two MetaInstances to the same instance
2. Delete ordering race — no window between deleting an instance and its
binding where the reconciler could re-place
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
- Created MetaInstance via dashboard, verified instance placed
- Verified delete cascades (deleting MetaInstance removes backing
instance)
- Verified orphan instances still work independently
### Automated Testing
- 30 tests in `test_meta_instance_edge_cases.py`: lifecycle, retry
logic, error handling, concurrent operations, cascade delete with task
cancellation
- 24 tests in `test_reconcile.py`: constraint matching, connection
health (single/multi-node, edge removal, IP changes), unsatisfied
detection, exclusive binding, idempotency
- All 261 tests pass
- basedpyright 0 errors, ruff clean, dashboard builds
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Fixes#1370
When the macOS app stops exo, GPU/system memory isn't released. This
happens because:
1. The macOS app calls `process.terminate()` (SIGTERM) but the Python
process only registers a graceful shutdown handler for SIGINT, not
SIGTERM. SIGTERM's default Python behavior raises `SystemExit` which
bypasses the cleanup cascade (runner subprocess MLX cleanup via
`mx.clear_cache()`, channel closing, etc.).
2. The app doesn't wait for the process to actually finish cleanup — it
immediately nils out the process reference.
## Changes
**`src/exo/main.py`**: Register SIGTERM handler alongside SIGINT so the
graceful shutdown cascade (`Node.shutdown()` → cancel task group →
worker/runner cleanup → `mx.clear_cache()` + `gc.collect()`) runs
regardless of which signal is received.
**`app/EXO/EXO/ExoProcessController.swift`**: Replace immediate
`process.terminate()` with escalating shutdown per @Evanev7's
suggestion:
1. Send SIGINT via `process.interrupt()` — triggers the registered
Python handler for graceful cleanup
2. Wait up to 5 seconds for the process to exit
3. If still running, escalate to SIGTERM via `process.terminate()`
4. Wait up to 3 seconds
5. If still running, force kill via SIGKILL
The escalation runs in a detached `Task` so the UI updates immediately
(status → stopped) without blocking.
## Why It Works
The root cause is that SIGTERM wasn't triggering the graceful shutdown
path. By registering a SIGTERM handler in Python and sending SIGINT
first from the macOS app, the process gets a chance to run the full
cleanup cascade: cancelling the task group, shutting down runners (which
call `del model; mx.clear_cache(); gc.collect()`), closing channels, and
flushing logs. The escalation to SIGTERM and SIGKILL ensures the process
always terminates even if graceful shutdown hangs.
## Test Plan
### Manual Testing
<!-- Hardware: Mac Studio M4 Max 128GB -->
- Start exo via macOS app, load a model, run inference
- Stop via the toggle switch, verify memory is released without
requiring a system restart
- Test rapid stop/start (restart) to ensure no race conditions
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `nix fmt` — no changes
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Evan Quiney <evanev7@gmail.com>
## Motivation
The current downloads page uses a node-centric card grid layout that is
messy and hard to read — the same model across different nodes appears
in separate cards, and deep nesting wastes space. This makes it
difficult to quickly see which models are on which nodes.
## Changes
Rewrote the downloads page
(`dashboard/src/routes/downloads/+page.svelte`) from a card grid to a
clean table layout:
- **Rows** = models (unique across all nodes)
- **Columns** = nodes (with disk free shown in header)
- **Cells** show status at a glance:
- ✅ Green checkmark + size for completed downloads
- 🟡 Yellow percentage + mini progress bar + speed for active downloads
- `...` for pending downloads
- ❌ Red X for failed downloads
- `--` for models not present on a node
- Delete/download action buttons appear on row hover
- Model name column is sticky on horizontal scroll (for many-node
clusters)
- Models sorted by number of nodes with completed downloads
- Imported shared utilities from `$lib/utils/downloads` instead of
inline re-implementations
### Backend: model directory in download events
- Added `model_directory` field to `BaseDownloadProgress` so all
download status events include the on-disk path
- Added `_model_dir()` helper to `DownloadCoordinator` to compute the
path from `EXO_MODELS_DIR`
- Dashboard uses this to show file location and enable "open in Finder"
for completed downloads
### Info modal
- Clicking a model name opens an info modal showing card details
(family, quantization, capabilities, storage size, layer count, tensor
parallelism support)
### Other fixes
- Fixed model name truncation in the table
- Excluded `tests/start_distributed_test.py` from pytest collection (CLI
script that calls `sys.exit()` at import time)
## Test Plan
- [x] `uv run basedpyright` — 0 errors
- [x] `uv run ruff check` — all passed
- [x] `nix fmt` — clean
- [x] `uv run pytest` — 188 passed, 1 skipped
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
currently nodes leaving and rejoining the cluster can lose their identity. We have no need to delete this data on node timing out, so let's just persist it.
Adds all the models that can fit onto a single M3 Ultra for single
machine benchmarks. Fixes the macOS version, GPU spec, and chip type for
maximum reproducibility. Specifies the minimum memory accordingly for
each type of model, using the smallest machine available (the smallest
M3 Ultra is 96GiB).
Test plan:
- Running this with some code that makes machines of this spec available
and stores the results. It works.
This will become part of a larger testing/stability strategy once we've
collected more of the data.
closing the http request to the api now
- sends a cancellation from the api
- writes that canellation in the master
- worker plans off the cancellation
- runner observes that cancellation after every generation step (+1
communication per token)
- cancellation happens synchronously to prevent gpu locks
closes#61
---------
Co-authored-by: Alex Cheema <alexcheema123@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
exo bench previously relied on the worker's plan loop to download
models, which could fail silently or run into disk space issues during
benchmarking. This made it difficult to diagnose download failures.
Added a planning phase that runs before benchmarking to explicitly
handle downloads. It checks available disk space on each node via the
/state endpoint and starts downloads via POST /download/start. When
the --danger-delete-downloads flag is set and there's insufficient
space, it deletes existing models from smallest to largest until
there's room for the benchmark model.
Test plan:
- CI
```
jake@maverick:/data/users/jake/repos/exo/ > nix run .#exo-bench -- --pp 128,2048,4096 --tg 128 --stdout --settle-timeout 10 --host s1 --model mlx-community/gpt-oss-120b-MXFP4-Q8
PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2026-02-16 12:12:11.807 | INFO | __main__:main:710 - pp/tg mode: combinations (product) - 3 pairs
Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
2026-02-16 12:12:13.455 | DEBUG | __main__:main:725 - [exo-bench] loaded tokenizer: mlx-community/gpt-oss-120b-MXFP4-Q8 for prompt sizer
2026-02-16 12:12:13.473 | DEBUG | __main__:main:761 - exo-bench model: short_id=gpt-oss-120b-MXFP4-Q8 full_id=mlx-community/gpt-oss-120b-MXFP4-Q8
2026-02-16 12:12:13.473 | INFO | __main__:main:762 - placements: 1
2026-02-16 12:12:13.474 | INFO | __main__:main:764 - - Pipeline / MlxRing / nodes=1
2026-02-16 12:12:13.474 | INFO | __main__:main:771 - Planning phase: checking downloads...
Traceback (most recent call last):
File "/nix/store/q31kmbcfr5bf97290bvbnhrvpc3fv824-source/bench/exo_bench.py", line 885, in <module>
raise SystemExit(main())
~~~~^^
File "/nix/store/q31kmbcfr5bf97290bvbnhrvpc3fv824-source/bench/exo_bench.py", line 772, in main
run_planning_phase(
~~~~~~~~~~~~~~~~~~^
client,
^^^^^^^
...<4 lines>...
settle_deadline,
^^^^^^^^^^^^^^^^
)
^
File "/nix/store/q31kmbcfr5bf97290bvbnhrvpc3fv824-source/bench/exo_bench.py", line 367, in run_planning_phase
raise RuntimeError(
...<2 lines>...
)
RuntimeError: Insufficient disk on 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj: need 65GB, have 55GB. Use --danger-delete-downloads to free space.
jake@maverick:/data/users/jake/repos/exo/ > nix run .#exo-bench -- --pp 128,2048,4096 --tg 128 --stdout --settle-timeout 10 --host s1 --model mlx-community/gpt-oss-120b-MXFP4-Q8 --danger-delete-downloads
PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2026-02-16 12:12:19.626 | INFO | __main__:main:710 - pp/tg mode: combinations (product) - 3 pairs
2026-02-16 12:12:21.262 | DEBUG | __main__:main:725 - [exo-bench] loaded tokenizer: mlx-community/gpt-oss-120b-MXFP4-Q8 for prompt sizer
2026-02-16 12:12:21.280 | DEBUG | __main__:main:761 - exo-bench model: short_id=gpt-oss-120b-MXFP4-Q8 full_id=mlx-community/gpt-oss-120b-MXFP4-Q8
2026-02-16 12:12:21.280 | INFO | __main__:main:762 - placements: 1
2026-02-16 12:12:21.280 | INFO | __main__:main:764 - - Pipeline / MlxRing / nodes=1
2026-02-16 12:12:21.280 | INFO | __main__:main:771 - Planning phase: checking downloads...
2026-02-16 12:12:21.336 | INFO | __main__:run_planning_phase:386 - Deleting mlx-community/Qwen3-0.6B-4bit from 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj (335MB)
2026-02-16 12:12:21.350 | INFO | __main__:run_planning_phase:386 - Deleting mlx-community/Llama-3.2-1B-Instruct-4bit from 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj (679MB)
2026-02-16 12:12:21.363 | INFO | __main__:run_planning_phase:386 - Deleting mlx-community/Llama-3.2-3B-Instruct-4bit from 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj (1740MB)
2026-02-16 12:12:21.373 | INFO | __main__:run_planning_phase:386 - Deleting mlx-community/Llama-3.2-3B-Instruct-8bit from 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj (3264MB)
2026-02-16 12:12:21.384 | INFO | __main__:run_planning_phase:386 - Deleting mlx-community/GLM-4.7-Flash-8bit from 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj (30366MB)
2026-02-16 12:12:21.413 | INFO | __main__:run_planning_phase:407 - Started download on 12D3KooWE2C7dzC9d9YJMEfWK3g8og7JdZj3HHXZ8VmGrXYAEnEj
```
It's not pretty but it works!
## Motivation
When a user deletes a message during an active streamed generation, it
can cause unexpected behavior. The delete confirmation text was also
misleading — it said "all responses after it" only for user messages,
which didn't accurately describe the behavior (all messages after the
deleted one are removed, regardless of role)
## Changes
- Prevent deletion during streaming: Disabled the delete button and
blocked handleDeleteClick when loading is true, with a visual indication
(dimmed button, cursor-not-allowed, tooltip change)
- Clarified delete confirmation text: Replaced role-specific wording
with a simpler, accurate message:
- Last message: "Delete this message?"
- Any other message: "Delete this message and all messages after it?"
## Why It Works
Guarding on the loading state at both the click handler and the button's
disabled attribute ensures no deletion can be triggered while a response
is being streamed
## Test Plan
### Manual Testing
- Verify the delete button is visually disabled and non-clickable while
a response is streaming
- Verify the tooltip shows "Cannot delete while generating" during
streaming
- Verify the last message shows "Delete this message?" confirmation
- Verify non-last messages show "Delete this message and all messages
after it?" confirmation
- Verify deletion works normally when not streaming
## Motivation
Models that only support image editing (ImageToImage but not
TextToImage) would silently attempt text-to-image generation when a user
submitted a text prompt without an attached image
## Changes
- Added an early return guard in handleSubmit() that prevents submission
when the selected model only supports image editing and no image is
attached (isEditOnlyWithoutImage)
- Fixed the text-to-image generation branch to use the more specific
modelSupportsTextToImage() check instead of the broad isImageModel(),
ensuring only models with TextToImage capability trigger generation from
text alone
- The existing isEditOnlyWithoutImage derived state (which was already
used for UI hints like placeholder text and button disabling) now also
blocks the actual submit path
## Why It Works
The text-to-image fallback now correctly checks
modelSupportsTextToImage() directly, so edit-only models no longer fall
through to the generation path
## Test Plan
### Manual Testing
- Select an edit-only image model (e.g., one with only ImageToImage
capability)
- Verify the send button is disabled and placeholder reads "Attach an
image to edit..." when no image is attached
- Attach an image and verify the form becomes submittable
- Select a text-to-image model and verify text-only prompts still
trigger generation normally
- Ensure pressing `enter` doesn't bypass check
The existing CI typecheck job used `uv run basedpyright` which depends
on a non-hermetic uv sync step. This replaces it with a fully hermetic
typecheck as a Nix flake check using the uv2nix virtual environment.
Added a `typecheckVenv` with dev dependencies, a `linuxOverlay` to
ignore native shared library deps (NVIDIA, torch, triton, mlx) that
aren't needed at type-check time, and `passthru` preservation plus
`.pyi` stub copying on the `exo-pyo3-bindings` overlay so basedpyright
can resolve the Rust bindings types. Also guarded the `mlx` Nix build
override to macOS only since it requires Metal. Removed the old
non-hermetic `typecheck` CI job since `nix flake check` now covers it.
The hermetic check ensures type checking uses exactly the locked
dependency versions and catches type errors without requiring a
working uv/pip environment.
Test plan:
- CI (`nix flake check` runs on x86_64-linux, aarch64-linux, aarch64-darwin)
- Verified `nix build ".#checks.x86_64-linux.typecheck"` passes with 0 errors
## Summary
- Clicking the **Recent** tab in the Model Picker crashed with
`TypeError: e.getModelFitStatus is not a function`
- The `ModelPickerGroup` component in the Recent tab section was missing
the `{getModelFitStatus}` prop, while all other tabs (e.g., the main
model list) passed it correctly
- Added the missing `{getModelFitStatus}` prop so the Recent tab renders
without errors, matching the behavior of the other tabs
## Test plan
- [ ] Open the dashboard and click **SELECT MODEL**
- [ ] Switch to the **Recent** tab — verify it renders without crashing
- [ ] Confirm model fit status indicators display correctly on recent
models
- [ ] Verify the other tabs (All, Favorites) still work as before
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Working version of #1366
## Changes
Add Step 3.5 Flash
## Test Plan
### Manual Testing
Works!
### Automated Testing
Running two processes tensor/pipeline sharded gives same logits as
single process.
## Motivation
When debugging cluster issues, it's useful to see which macOS version
each node is running — especially since version mismatches can cause
compatibility problems. The OS version data is already collected by the
identity gatherer but wasn't shown in the topology graph.
## Changes
- Added OS version label (e.g. "macOS 15.2") to each node in the
topology graph when debug mode is enabled
- Renders below the existing TB and RDMA debug labels using the same
styling conventions
- Sources data from the existing `nodeIdentities` store (no backend
changes needed)
## Why It Works
The `nodeIdentities` store already contains `osVersion` for each node.
We simply read it in the `TopologyGraph` component and append a text
label in the debug section, following the exact same pattern as the TB
and RDMA labels.
## Test Plan
### Manual Testing
<!-- Hardware: MacBook Pro -->
- Enable debug mode in the dashboard
- Verify OS version label appears below TB/RDMA labels on each node
- Verify label disappears when debug mode is disabled
### Automated Testing
- Dashboard build passes (`npm run build`)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Motivation
Fixes#1456. Models like DeepSeek V3.2, Qwen3, and GLM-4.7 always run in
thinking mode because their chat templates auto-inject `<think>`. Users
need a way to disable thinking for models that support both modes.
## Changes
**API**: Added `enable_thinking: bool | None` to `ChatCompletionRequest`
and `TextGenerationTaskParams`. Passed through the adapter to
`tokenizer.apply_chat_template()` as a kwarg (only when explicitly set,
so models without the template variable are unaffected).
**Dashboard**: Added a thinking toggle button in the chat input area.
Visible only when the selected model has both "text" and "thinking"
capabilities.
## Why It Works
Most thinking model chat templates (DeepSeek, Qwen3, GLM) accept
`enable_thinking` as a Jinja template variable. Passing
`enable_thinking=False` prevents the template from injecting `<think>`,
matching the vLLM convention.
## Test Plan
### Manual Testing
- `curl` with `"enable_thinking": false` against a thinking model — no
`<think>` in output
- Dashboard toggle visible for thinking models, hidden for text-only
models
### Automated Testing
- basedpyright: 0 errors
- ruff: clean
- pytest: 188 passed
- dashboard build: success
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
exo_bench.py fails if started too soon after a cluster starts because
the topology hasn't populated yet, resulting in no valid placements.
Extracted the preview-fetch-and-filter logic into a
`fetch_and_filter_placements` helper and added a retry loop with
exponential backoff (1s initial, 2x multiplier, 60s cap). The new
`--settle-timeout` flag controls how long to retry (default 0 = try
once, preserving existing behaviour). Each retry logs a warning
explaining the cluster may still be settling.
Test plan:
- Tested on several freshly started clusters. This used to fail a lot,
now it succeeds.
## Motivation
When a model download fails repeatedly (e.g. `ContentLengthError` on a
large model like `zai-org/GLM-5`), the download coordinator accumulates
duplicate progress callbacks — one per retry cycle. Each callback
independently throttles at 1 event/sec, so after N retries, every
download progress tick generates N events instead of 1. After an hour of
failures (~60 retry cycles), this produces ~60 `NodeDownloadProgress`
events/sec, overwhelming the master, delaying heartbeats, and causing
the node to time itself out.
### The callback accumulation cycle
1. `_start_download_task()` calls
`shard_downloader.on_progress(callback)` which **appends** to a list
2. Download fails → `DownloadFailed` status set, but old callback stays
in the list
3. 60s later: `_emit_existing_download_progress()` scans disk → resets
status to `DownloadPending`
4. Worker sends new `StartDownload` → coordinator accepts (guard didn't
check `DownloadFailed`)
5. `_start_download_task()` appends **another** callback
6. Each callback has its own throttle → N callbacks = N events per
progress tick
## Changes
### Commit 1: `src/exo/worker/main.py`
Move the `DownloadModel` backoff check **before** `TaskCreated` emission
in `plan_step()`. Previously `TaskCreated` was emitted unconditionally
every 0.1s even when backoff blocked the download command.
### Commit 2: `src/exo/download/coordinator.py`
1. **Register progress callback once** in `__post_init__` instead of
per-download in `_start_download_task()`. Uses a per-model throttle dict
instead of per-callback closure variables.
2. **Add `DownloadFailed` to the `_start_download()` guard** so
redundant `_start_download_task()` calls don't happen. Retries still
work because `_emit_existing_download_progress` resets `DownloadFailed`
→ `DownloadPending` by scanning disk every 60s.
## Why It Works
The root cause was callbacks accumulating in
`ResumableShardDownloader.on_progress_callbacks` (a list that only
appends, never clears). By registering one callback per coordinator
lifetime and guarding against re-entry on `DownloadFailed`, we ensure
exactly one progress event per model per progress tick regardless of how
many retry cycles have occurred.
## Test Plan
### Manual Testing
- Verified the download retry flow: failed download → 60s scan resets
status → new `StartDownload` accepted → download retries with single
callback
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `uv run pytest` — 188 passed
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Info gatherer monitors could silently stop posting events, causing stale
node state after rejoins. The macmon monitor was especially fragile — it
had no retry loop, so a crash or parse error would kill it permanently.
Worse, the unhandled exception would propagate to the TaskGroup and take
down *all* sibling monitors. Additionally, none of the monitors had
timeouts on their subprocess calls, so a hung `system_profiler` or
`networksetup` could stall a monitor indefinitely.
## Changes
- Wrap `_monitor_macmon` in a `while True` retry loop with `except
Exception`, matching the pattern used by all other monitors
- Add `fail_after` timeouts to all monitor loop bodies:
- 10s for lightweight commands (`_monitor_misc`, `_watch_system_info`,
`_gather_iface_map` init)
- 30s for heavier commands (`_monitor_system_profiler_thunderbolt_data`,
`_monitor_thunderbolt_bridge_status`)
- Remove unused `CalledProcessError` and `cast` imports
## Why It Works
All monitors now follow the same resilient pattern: `while True` → `try`
with `fail_after` → `except Exception` (logs warning) → `sleep`. If a
subprocess hangs, the timeout fires and `TimeoutError` is caught by the
existing `except Exception` handler. If macmon crashes, it restarts
after the interval instead of dying permanently. No single monitor
failure can cascade to kill the others.
## Test Plan
### Manual Testing
<!-- Hardware: macOS with macmon installed -->
<!-- What you did: -->
- Run exo, kill macmon process (`kill $(pgrep macmon)`), verify it
restarts and metrics resume
- Verify all monitors continue posting events after simulated hangs
### Automated Testing
- All 188 existing tests pass
- basedpyright: 0 errors
- ruff: all checks passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The downloads page previously only showed the approximate space used by
downloaded models (summed from completed download sizes), but did not
show how much disk space was actually available. This made it difficult
to know if a download would succeed before pressing the button.
Added disk space tracking to the InfoGatherer that polls the models
directory partition every 30 seconds. The DiskUsage type captures total
and available space, which flows through the event system to State and
is exposed via the /state API. The dashboard now displays "X on disk /
Y available" for each node in the downloads view.
Test plan:
- CI
## Motivation
When nodes in an exo cluster run different macOS versions, inference can
produce incompatible results or fail silently. Users currently have no
way to know this from the dashboard.
## Changes
- Added `get_os_version()` to `system_info.py` that returns the macOS
version (e.g. `"15.3"`) or platform name for non-Mac nodes
- Added `os_version` field to `NodeIdentity` and
`StaticNodeInformation`, gathered once at startup
- Propagated `os_version` through the event sourcing pipeline
(`apply.py`)
- Exposed `nodeIdentities` from the dashboard store with `osVersion`
- Added a derived `macosVersionMismatch` check in `+page.svelte` that
triggers when 2+ macOS nodes report different versions
- Rendered a yellow "INCOMPATIBLE macOS VERSIONS" warning badge
(matching the existing Thunderbolt Bridge cycle warning style) with a
hover tooltip listing each node's name and version, in all three
topology view sizes (large, medium, compact)
## Why It Works
The OS version is a static property gathered once at node startup via
`platform.mac_ver()`. It flows through the existing
`StaticNodeInformation` → `NodeGatheredInfo` event → `NodeIdentity`
state pipeline, so no new event types or state fields beyond
`os_version` on `NodeIdentity` are needed. The dashboard derives the
mismatch by comparing `osVersion` across all nodes whose version looks
like a macOS version string (starts with a digit).
## Test Plan
### Manual Testing
Hardware: 4x Mac Studio M2 Ultra 512GB (s18, s17 (2), james, mike),
connected via Thunderbolt
- s18 and s17 (2) on macOS 26.2, james and mike on macOS 26.3
- Verified the "INCOMPATIBLE macOS VERSIONS" warning badge appears in
the topology view
- Verified the hover tooltip lists all four nodes with their respective
versions
- Screenshots attached in comment below
### Automated Testing
- basedpyright: 0 errors
- ruff check: all checks passed
- nix fmt: no formatting changes needed
- Dashboard builds successfully
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
Several RDMA/Thunderbolt UX issues in the dashboard and macOS app:
1. **Debug mode showed "? ?" for RDMA connections** — the topology view
only extracted IPs from socket connections, not RDMA interface names
2. **No way to detect if RDMA is actually enabled** — the system only
knew about TB5 hardware and RDMA topology edges, not whether `rdma_ctl`
was enabled on each node
3. **False "RDMA AVAILABLE" info box** — showed on Mac Minis with idle
TB5 ports even when RDMA was already enabled, and on single nodes with
TB5
4. **macOS app only showed local RDMA status** — ran `rdma_ctl` locally
with no visibility into other nodes in the cluster
## Changes
### Dashboard: Fix RDMA debug labels (`0abc90c4`)
- Added `sourceRdmaIface` and `sinkRdmaIface` to `TopologyEdge`
interface
- Updated `TopologyGraph.svelte` and `ModelCard.svelte` to show `RDMA
en2 → en3` instead of `? ?`
### Dashboard: TB5 RDMA info box (`a3795552`, `8ce8e173`)
- Added dismissible info box when 2+ nodes have TB5 hardware but RDMA is
disabled
- Includes setup instructions (Recovery mode → `rdma_ctl enable` →
reboot, TB5 cables, macOS version match)
- Requires 2+ exo nodes with TB5 to avoid false positives from
single-node setups
### Backend: `rdma_ctl status` detection (`ae07239b`)
- Added `RdmaCtlStatus` event to `info_gatherer.py` — runs `rdma_ctl
status` with 5s timeout, `shutil.which` guard, and `OSError` handling
(polls every 10s on macOS)
- Added `NodeRdmaCtlStatus` model to `profiling.py` and `node_rdma_ctl`
field to `State`
- Handle in `apply.py` (event apply + node timeout cleanup)
- Exposed `nodeRdmaCtl` in dashboard store (`app.svelte.ts`)
- Info box detection now uses actual RDMA status instead of TB5 link
speeds
### Dashboard: Per-node RDMA debug labels (`ae07239b`)
- Debug mode shows `RDMA:ON` (green) or `RDMA:OFF` (dim) per node in
topology view, below the TB bridge label
### macOS app: Cluster-wide RDMA status from `/state` (`a1455b61`,
`d0d77b63`)
- Added `NodeRdmaCtlStatus` to `ClusterState.swift` — decoded from
`/state` endpoint
- Replaced local-only `rdma_ctl status` check with cluster-wide
`nodeRdmaCtl` from state
- Debug section shows per-node RDMA enabled/disabled for all nodes in
the cluster
- Still shows local `ibv_devices` and `ibv_devinfo` details (device
names, active ports) for richer local debugging
## Files changed
| Area | File | Change |
|------|------|--------|
| Backend | `src/exo/utils/info_gatherer/info_gatherer.py` |
`RdmaCtlStatus` event, monitor task |
| Backend | `src/exo/shared/types/profiling.py` | `NodeRdmaCtlStatus`
model |
| Backend | `src/exo/shared/types/state.py` | `node_rdma_ctl` field |
| Backend | `src/exo/shared/apply.py` | Event handler + timeout cleanup
|
| Dashboard | `dashboard/src/lib/stores/app.svelte.ts` | `nodeRdmaCtl` +
`nodeThunderbolt` in store |
| Dashboard | `dashboard/src/routes/+page.svelte` | Info box with RDMA
detection + instructions |
| Dashboard | `dashboard/src/lib/components/TopologyGraph.svelte` | RDMA
debug labels per node + fix "? ?" |
| Dashboard | `dashboard/src/lib/components/ModelCard.svelte` | RDMA
interface display fix |
| App | `app/EXO/EXO/Models/ClusterState.swift` | `NodeRdmaCtlStatus`
struct + decode |
| App | `app/EXO/EXO/ContentView.swift` | Cluster-wide RDMA view + local
device details |
| App | `app/EXO/EXO/Services/NetworkStatusService.swift` | Remove local
`rdma_ctl`, keep `ibv_*` |
## Test Plan
- [x] `uv run basedpyright` — 0 errors
- [x] `uv run ruff check` — pass
- [x] `nix fmt` — clean
- [x] `cd dashboard && npm run build` — success
- [x] `uv run pytest` — 188 passed
- [x] Xcode build — compiles (only pre-existing `dist/exo` resource
error)
- [x] Deployed to Mac Minis — `nodeRdmaCtl` shows `enabled: true`, no
false info box
- [x] Deployed to James cluster — RDMA debug labels show correctly
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
## Motivation
When frequently switching between models, it's tedious to search through
the full model list to find ones you've used before. A "Recent" tab
provides quick access to previously launched models.
## Changes
- **New store** (`dashboard/src/lib/stores/recents.svelte.ts`):
`RecentsStore` class persisting recently launched model IDs with
timestamps to localStorage (key: `exo-recent-models`). Caps at 20
entries, deduplicates on re-launch (moves to top).
- **FamilySidebar**: Added "Recent" tab between Favorites and Hub,
conditionally shown when there are recent models.
- **FamilyLogos**: Added clock/history icon for the recents tab.
- **ModelPickerModal**: Added `recentModelIds`/`hasRecents` props.
Derives single-variant `ModelGroup[]` from recent IDs and renders them
using the same `ModelPickerGroup` component as all other tabs —
consistent styling, memory grey-out, favorites, info button, download
indicators.
- **+page.svelte**: Calls `recordRecentLaunch(modelId)` after successful
instance launch. Passes reactive recent state to the modal.
## Why It Works
Follows the exact same pattern as the existing Favorites feature
(localStorage persistence, conditional tab display, reactive Svelte 5
`$state`/`$derived`). Recent models are wrapped as single-variant
`ModelGroup` objects so they reuse `ModelPickerGroup` for identical row
rendering across all tabs.
## Test Plan
### Manual Testing
<!-- Hardware: MacBook Pro -->
- Launch a model instance → reopen model picker → "Recent" tab appears
with the launched model
- Launch a second model → it appears at top of the Recent list
- Re-launch the first model → it moves back to top
- Search within the Recent tab filters the list
- Models that don't fit in memory are greyed out (same as All tab)
- Close/reopen browser → recents persist from localStorage
### Automated Testing
- Dashboard builds successfully (`npm run build`)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
In the model picker, non-runnable models were not clearly separated
between two different cases:
- model exceeds currently available RAM but can fit in total cluster
capacity
- model exceeds total cluster capacity
That made it harder to distinguish "not runnable right now" from "too
large for this cluster."
## Changes
- Added a tri-state fit status in dashboard model-picker flow:
- `fits_now`
- `fits_cluster_capacity`
- `too_large`
- Updated dashboard logic to compute both available cluster RAM and
total cluster RAM.
- Passed fit status through picker components.
- Updated model size color mapping:
- `fits_now` -> white/gray
- `fits_cluster_capacity` -> orange
- `too_large` -> red
- Updated group ordering for non-runnable models:
- orange groups (`fits_cluster_capacity`) are listed above red groups
(`too_large`).
## Why It Works
Launch safety is unchanged: selection is still gated by existing
placement feasibility (`canModelFit`), so models that cannot run now
remain disabled.
The new fit status is used for visual distinction and ordering only:
- runnable now
- fits cluster capacity but not free RAM now
- too large for cluster capacity
## Before
Non-runnable models were not clearly distinguished by temporary capacity
vs hard capacity limit.
## After
Model picker clearly separates states by both color and order:
- Runnable now (white/gray)
- Fits cluster capacity but not free RAM now (orange, disabled)
- Exceeds cluster capacity (red, disabled)
## Test Plan
### Manual testing
- Open model picker with live cluster memory telemetry.
- Verify white/gray models are selectable.
- Verify orange models are disabled and appear above red models.
- Verify red models are disabled and appear below orange models.
### Automated checks
- `npm --prefix dashboard run build` passes.
- `uv run basedpyright` passes.
- `uv run ruff check` passes.
- `npm --prefix dashboard run check` reports existing pre-change Svelte
diagnostics (same known SVG title/a11y items).
- `uv run pytest` in this local environment exits during collection due
existing `tests/start_distributed_test.py` `SystemExit` usage.
- No Python code was changed.
---------
Co-authored-by: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
## Summary
`exo` crashes on startup when the system's hard file descriptor limit is
below 65535, which occurs in macOS LaunchDaemon environments, Docker
containers, and other restricted setups.
**Root cause:** `resource.setrlimit(resource.RLIMIT_NOFILE, (max(soft,
65535), hard))` raises `ValueError` when `hard < 65535` because the soft
limit cannot exceed the hard limit.
**Fix:**
- **`main.py`**: Clamp the target soft limit to `min(max(soft, 65535),
hard)` so it never exceeds the hard limit
- **`utils_mlx.py`**: Query current limits instead of hardcoding `(2048,
4096)`, which both crashed on restricted systems and incorrectly lowered
the hard limit when it was set higher
## Test plan
- [x] `basedpyright` passes with 0 errors
- [x] `ruff check` passes
- [x] Verify startup works on system with hard limit < 65535 (tested in
macOS LaunchDaemon with hard limit 10240)
- [x] Verify startup still works on default macOS (hard limit typically
unlimited)
VecExt added a .map() convenience method on Vec<T> that simply called
.into_iter().map(f).collect(). This thin wrapper provided no
optimisation benefit and obscured a standard iterator pattern behind a
nightly feature gate and an extra dependency.
Replaced the single call site in exo_pyo3_bindings with the equivalent
iterator chain and removed the ext module, the extend dependency, and
the trait_alias feature gate from the util crate.
Test plan:
- CI
The util crate contained several unused items: NonemptyArray,
BoxedSliceExt, a blanket Sealed trait, an empty alias module, six unused
nightly feature gates, and five unused Cargo dependencies (thiserror,
once_cell, internment, derive_more, bon, recursion).
Removed all items that had no references outside their own definitions,
keeping only WakerDeque, VecExt, and the trait_alias feature gate which
are actively used by the networking and exo_pyo3_bindings crates.
Test plan:
- CI
The pinned nixpkgs provides apple-sdk 26.0, but building MLX requires
SDK 26.2. The upstream package reads versions.json via a relative path
at eval time, so it can't be overridden through callPackage args.
Added a thin overlay that copies the upstream apple-sdk source and
patches only metadata/versions.json to point at SDK 26.2. Also enabled
MLX_BUILD_CPU in the MLX nix build.
This avoids vendoring the entire apple-sdk package (~2200 lines) while
still getting the SDK version we need.
Test plan:
- CI
- Built and ran on two machines connected with Thunderbolt 5 - Kimi K2.5
starts in Tensor+RDMA and seems sensible.
## Motivation
Log rotation adds a bunch of .zst files. Let's send them all in the bug
reports.
This PR also standardises the logs so that they all include the
timestamp.
## Motivation
Standardises exo.log and event_log
## Changes
- exo.log and exo.log.zst are now in a exo_log directory.
- event_log is now timestamped and not numbered. The timestamps can be
sorted as they are YYYY-MM-DD-HH-MM
## Test Plan
### Manual Testing
Nothing crashes.
## Motivation
.exo.log currently contains all past history. This just makes it hard to
read and is unnecessarily expensive even on disk.
## Why It Works
Just uses loguru's rotation.
## Test Plan
### Manual Testing
exo.log is new.
<img width="1992" height="706" alt="image"
src="https://github.com/user-attachments/assets/9b293993-1141-43e7-b58e-0ddd2d4eda2e"
/>
The 15-second publish_queue_duration caused messages in peer queues to
be silently dropped. When events are dropped, workers detect gaps in the
event index sequence and request missing events via the NACK path
(RequestEventLog), but this recovery is inefficient.
Removed the timeout configuration - gossipsub now uses its default
behavior without time-based eviction. If queue buildup is a concern,
queue size should be limited explicitly rather than dropping by timeout.
Split error handling to log AllQueuesFullError as a warning (indicates
peers are unresponsive) while keeping NoPeersSubscribedToTopicError
silent (expected during startup and network partitions).
Test plan:
- CI
## Motivation
App keeps losing Local Network permissions.
## Changes
Don't save stuff to the app directory anymore. Instead, save to .exo.
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
Before:
<img width="1512" height="106" alt="image"
src="https://github.com/user-attachments/assets/544ef57e-b626-484d-941f-2472969aa208"
/>
After:
<img width="433" height="53" alt="Screenshot 2026-02-10 at 17 43 06"
src="https://github.com/user-attachments/assets/3de2856b-cdf6-4b35-aa8f-50440686344f"
/>
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
The master and API event logs (list[Event]) grew unbounded in RAM for
the lifetime of the process. Events are rarely read back (only for
RequestEventLog when a new node catches up, or the dashboard /events
endpoint).
Introduced a DiskEventLog class that writes length-prefixed msgpack
records to an append-only file, using a bounded LRU cache of byte
offsets for indexed access. On close, the active file is compressed
with ZSTD and rotated into a numbered archive slot, keeping the last 5
archives (events.1.bin.zst through events.5.bin.zst). On construction,
any stale active file from a crash is rotated before opening a fresh
log. The /events API endpoint now streams the JSON array one event at a
time rather than materializing the full list in memory. Deserialization
routes msgpack through json.dumps into Pydantic's validate_json() to
get correct JSON-mode coercion (e.g. string to enum) under strict mode.
This bounds memory usage to the LRU cache (128 entries) regardless of
event volume, while still supporting efficient sequential reads from
disk when needed.
Test plan:
- CI
- New unit tests for DiskEventLog: append/read, range queries, rotation
on close, stale file recovery, idempotent close, successive sessions,
archive retention limit (5 max)
- Tested on a cluster with 9000 events. /events continues working.
- On disk size is 3.9MiB with ~8000 events, and the compression is very
effective.
- Disconnected and rejoined a machine, it rejoined fine.
---------
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Motivation
Follows up on #1408. Image models (FLUX, Qwen-Image, etc.) don't have a
`config.json` on HuggingFace. Previously, image model TOML cards were
only loaded into `_card_cache` when `EXO_ENABLE_IMAGE_MODELS=true`. When
the flag was off but an image model was requested (e.g., via
`get_placement_previews`), `ModelCard.load()` fell through to
`fetch_from_hf()` which tried to download `config.json` — causing
`FileNotFoundError` spam. #1408 added defensive error handling; this PR
fixes the root cause.
## Changes
**`model_cards.py`**: Always include `image_model_cards/` in
`CARD_SEARCH_PATH` so image model TOML cards are always loaded into
`_card_cache`. `ModelCard.load()` then finds them directly and never
falls through to `fetch_from_hf()`. The `EXO_ENABLE_IMAGE_MODELS` flag
now controls whether image models appear in `get_model_cards()` (the
listing) rather than whether they're loaded at all.
## Why It Works
`fetch_from_hf()` is designed for text models only (it hardcodes
`tasks=[ModelTask.TextGeneration]` and requires `config.json`). Image
models should never reach that path. By always having them in the cache,
the lookup succeeds immediately and `fetch_from_hf()` is never called.
## Test Plan
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
The chat textarea was fully disabled during response generation,
preventing users from drafting their next message while waiting.
Removed the `disabled={loading}` attribute from the textarea element.
Submission is still blocked during generation by the early return in
`handleSubmit()` and the submit button's own disabled state.
Test plan:
- Ran on one machine. While a model was writing a really long poem, I
typed my next response. I couldn't submit it with Enter and the button
still said "Processing" greyed out. I could send the message after
generation finished.
Non-tagged builds (test-app branch, manual dispatch) only uploaded the
DMG as a GitHub artifact, which requires authentication to download.
Added an early exit path that uploads the DMG with a commit hash suffix
(EXO-<sha>.dmg) for non-tagged builds, making it publicly accessible
via S3.
Test plan:
- CI
-
https://github.com/exo-explore/exo/actions/runs/21837274032/job/63011907978
worked as intended
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
If any of the `InfoGatherer` monitor loops throw an unexpected
exception, the entire monitoring task crashes and never recovers. This
can silently stop memory, network, or Thunderbolt data collection for
the lifetime of the process.
## Changes
Wrap the body of each `while True` monitor loop in a try/except that
logs the exception as a warning and continues to the next iteration. The
sleep at the end of each loop runs regardless, providing natural backoff
before retry.
Affected methods: `_monitor_misc`,
`_monitor_system_profiler_thunderbolt_data`, `_monitor_memory_usage`,
`_watch_system_info`, `_monitor_thunderbolt_bridge_status`.
`_monitor_macmon` already had its own error handling so was left as-is.
## Why It Works
A transient error (e.g., a subprocess failing, a permission issue) in
one iteration no longer kills the loop. The warning log provides
visibility while the monitor continues collecting data on subsequent
iterations.
## Test Plan
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
check_reachable waits for all connection profile checks to be completed.
Since there are retries on failures, this can take around 20s to
resolve, preventing any instances from showing up. This feels very slow
for UX, and it slows down distributed testing.
## Changes
Made check_reachable an async generator.
## Test Plan
### Manual Testing
Works for me at least.
## Motivation
For large prompts and/or slow machines, users are running into GPU
timeout errors very often.
## Changes
Only during prefill, we eval distributed operations. We don't do this
during decode to maintain decode performance.
Raise the prefill step size to 8192 because now we can (we see a speedup
here).
We also now see a 2x speedup in pipeline parallel prefill by disabling
an unnecessary all_gather during prefill.
## Why It Works
GPU timeout errors happen in the Metal backend when GPU operations take
too long without making progress.
By isolating distributed operations, we can allow them to run without
any timeouts.
## Test Plan
### Manual Testing
Doesn't GPU timeout on 100k tokens on Minimax anymore. Also tested on
Kimi.
### Automated Testing
Needs more exo bench, but I think this is a good step in the right
direction.
## Motivation
When downloading image models, a missing config.json file triggers a
FileNotFoundError inside download_file_with_retry. This error was being
caught by the generic except Exception handler and retried 3 times
before failing. Then, the whole thing would be retried from the start
## Changes
- src/exo/download/download_utils.py: Added FileNotFoundError to the
list of immediately-raised exceptions in download_file_with_retry,
alongside HuggingFaceAuthenticationError. This prevents useless retries
when a file genuinely doesn't exist on the remote.
- src/exo/master/api.py: Wrapped ModelCard.load(model_id) in a
try/except that converts failures into an HTTPException(400) with a
descriptive error message, giving API consumers a clear error response.
## Why It Works
- FileNotFoundError is a deterministic error — the file won't appear on
retry, so re-raising immediately avoids 3 wasted download attempts with
exponential backoff.
- Catching ModelCard.load() failures and returning a 400 HTTP response
prevents unhandled exceptions from surfacing as opaque 500 errors in the
API.
## Test Plan
### Manual Testing
Verified an image model not in model cards does not cause an infinite
error loop
## Motivation
No way to view generated or attached images at full resolution in the
dashboard
## Changes
- New ImageLightbox.svelte — fullscreen overlay with download, close
(click-outside/Escape), and transitions
- ChatMessages.svelte — all images (input attachments + generated) are
now clickable to open in lightbox; added expand button to generated
image hover overlay
## Why It Works
Single expandedImageSrc state variable drives the lightbox — set it to
show, null to hide.
## Test Plan
### Manual Testing
- Click any image (attachment thumbnail or generated) → lightbox opens
- Close via Escape, click-outside, or close button
- Download button saves with correct extension
## Motivation
Maybe addresses #1303
## Changes
Add an mx barrier before warmup
## Why It Works
It might, it might not. Shouldn't break anything that's not already
broken though.
## Test Plan
### Manual Testing
The two machines I tested on were fine on GLM 4.7 Flash 8bit (the one in
exo.log in the issue). Obviously not definitive for anything, however.
<img width="594" height="878" alt="image"
src="https://github.com/user-attachments/assets/534d3ad6-16ef-4cb5-b823-43c8d4e1d3c6"
/>
## Motivation
Image models (FLUX, Qwen Image) had no family grouping or quantization
metadata in the dashboard
## Changes
- Added family, quantization, base_model, and capabilities fields to all
18 image model TOML cards (FLUX.1 variants + Qwen Image variants)
- Added FLUX and Qwen Image SVG logos to FamilyLogos.svelte
- Added "flux" and "qwen-image" families to the sidebar and family sort
order
- Added "Image Gen" and "Image Edit" capability filters in
ModelFilterPopover.svelte
- Added image edit icon/badge to ModelPickerGroup.svelte
- Made the model category sidebar scrollable to accommodate the new
entries
- Hidden scrollbars on model list panels
## Why It Works
Reuses the existing family/quantization grouping infrastructure that
LLMs already use, extending it to image models with appropriate metadata
and icons
## Test Plan
### Manual Testing
Verified image models behave like text models in the model list dialog
---------
Co-authored-by: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
exo-bench was gated behind isDarwin in python/parts.nix because it used
exoVenv, which pulls in MLX (Darwin-only). However, exo_bench.py is an
HTTP client that only needs loguru, transformers, huggingface-hub, and
tiktoken.
Made bench a uv workspace member with its own pyproject.toml declaring
only the minimal dependencies. Added a separate benchVenv in parts.nix
built from that workspace member, and moved exo-bench out of the
isDarwin block so it is available on all platforms.
Test plan:
- `nix run .#exo-bench -- --help` prints argparse help
---------
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
our event log request blasted the whole event log over libp2p, now it
just does the next 1000 messages - hopefully allowing nodes to catch up
a bit more consistently for long lived clusters
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
Kimi produces its own tool id. It gets confused when we generate our own
id.
## Changes
Add id to tool call item and parse Kimi id properly.
## Test Plan
### Manual Testing
<img width="3198" height="522" alt="image"
src="https://github.com/user-attachments/assets/d71ec2be-7f57-49dc-a569-d304cc430f4d"
/>
Long running Kimi K2.5 cluster querying itself through OpenCode running
on the same Kimi K2.5 instance.
## Motivation
Recent commit broke glm (non lite) sharding
## Why It Works
Assert is no longer hit, as isinstance check includes
GLM4MoeDecoderLayer.
Added type stubs to keep the type checker happy.
## Test Plan
### Manual Testing
Runs as expected without gibberish.
## Motivation
Add support for FLUX.1-Kontext-dev, an image editing variant of
FLUX.1-dev
## Changes
- New FluxKontextModelAdapter: Handles Kontext's image-to-image workflow
- encodes input image as conditioning latents with special position IDs,
generates from pure noise
- Model config: 57 transformer blocks (19 joint + 38 single), guidance
scale 4.0, ImageToImage task
- Pipeline updates: Added kontext_image_ids property to PromptData
interface, passed through diffusion runner
- Model cards: Added TOML configs for base, 4-bit, and 8-bit variants
- Dependency: mflux 0.15.4 → 0.15.5
- Utility: tmp/quantize_and_upload.py for quantizing and uploading
models to HuggingFace
## Test Plan
### Manual Testing
Works better than Qwen-Image-Edit
## Motivation
More dimensions for image generation
## Changes
- dashboard/src/lib/components/ImageParamsPanel.svelte: Added
"1024x1365" and "1365x1024" to the sizeOptions array
- dashboard/src/lib/stores/app.svelte.ts: Extended the size type in
ImageGenerationParams interface to include the two new dimension options
## Motivation
Allow users to directly configure num_sync_steps for distributed image
generation instead of deriving it from a factor of total steps.
## Changes
- Added num_sync_steps field to AdvancedImageParams API (range 1-50)
- Changed model configs from num_sync_steps_factor: float to
num_sync_steps: int
- Updated Flux/Qwen configs with direct values (1, 4, 7 respectively)
- Added slider control in dashboard advanced params panel
- Falls back to model default when not specified
## Why It Works
Decouples sync steps from inference steps, giving users direct control
over distributed inference synchronization while preserving sensible
defaults.
## Test Plan
### Manual Testing
- Generate images with various sync step values via dashboard slider
- Verify default behavior when parameter is unset
runs exo_bench remotely with some nice git QoL
## usage
run tests/auto_bench.sh host1 [host2]
exo bench will be run on those hosts and its output saved to
bench/commit_hash/*.json on all models currently downloaded
The uv.lock is churning constantly as different UV versions bounce it
between revisions. This is made worse by GitHub automatically hiding the
uv.lock changes, meaning it's hard to notice when this went wrong.
Set a minimum version for `uv` in pyproject.toml to fix this. I tried
quite a few versions (not all) and found 0.8.6 sets the revision to 3,
which I believe is the latest. This is from August 2025 so has been
around for a while.
Test plan:
```
jake@maverick:/data/users/jake/repos/exo/ > git checkout main uv.lock
jake@maverick:/data/users/jake/repos/exo/ > nix shell github:nixos/nixpkgs/3dce7f4a77812afd69efcbfe15e5223f98c5c69e#uv --command sh -c 'uv add pip --frozen && uv lock && uv remove pip --frozen && uv lock && uv --version'
Resolved 140 packages in 147ms
Added pip v26.0.1
Resolved 139 packages in 48ms
Removed pip v26.0.1
uv 0.8.6
```
## Motivation
MiniMax tensor sharding does not provide equivalent outputs to running
it as a single node because RMSNorm weights cannot be split without
affecting the output.
Qwen3Next sharding was broken, and something with Qwen3MoE was likely
changed upstream, as several variables no longer exist.
This also ballooned into fixing prefix caching for non-standard models
as Qwen3Next was behaving weirdly.
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
Worked for a 8 hour long eval at the same performance and a more similar
completion/reasoning token distribution.
---------
Co-authored-by: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
Co-authored-by: Alex Cheema <alexcheema123@gmail.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
`download_file_with_retry()` has a `break` in the generic exception
handler that exits the retry loop after the first transient failure.
This means network timeouts, connection resets, and server errors all
cause an immediate download failure — the two remaining retry attempts
never run.
## Changes
**download_utils.py**: Replaced `break` with logging and exponential
backoff in the generic exception handler, matching the existing
rate-limit handler behavior.
Before:
```python
except Exception as e:
on_connection_lost()
if attempt == n_attempts - 1:
raise e
break # exits loop immediately
```
After:
```python
except Exception as e:
on_connection_lost()
if attempt == n_attempts - 1:
raise e
logger.error(f"Download error on attempt {attempt + 1}/{n_attempts} ...")
logger.error(traceback.format_exc())
await asyncio.sleep(2.0**attempt)
```
## Why It Works
The `break` statement was bypassing the retry mechanism entirely.
Replacing it with the same log-and-backoff pattern used by the
`HuggingFaceRateLimitError` handler means all 3 attempts are actually
used before giving up. The exponential backoff (1s, 2s) gives transient
issues time to resolve between attempts.
## Test Plan
### Manual Testing
- Downloads that hit transient network errors now retry instead of
failing immediately
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `uv run pytest src/exo/download/tests/ -v` — 11 tests pass
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
a lot of our cleanup logic wasn't running leading to bad shutdown states
## changes
- added `try: except` blocks around most task groups
- made the runner shutdown code synchronous
- abandon the MpReceiver's recv_async thread on cancellation
- this only occurs during runner shutdown, the queue closing from the
other end should terminate the mp.Queue, cleaning up the thread in its
own time. i could try other methods if this is not sufficient.
## outcome
ctrl-c just works now! minus the tokio panic of course :) no more
hypercorn lifespan errors though!
## Motivation
Users browsing models in the picker need to know which models are
already downloaded and ready to run on their cluster, without having to
check the downloads page separately.
## Changes
- **ModelPickerModal.svelte**: Computes per-model download availability
by checking which nodes have `DownloadCompleted` entries and summing
their total RAM against the model's storage size. Passes availability
data to `ModelPickerGroup`. Enhances the info modal with a "Downloaded
on:" section showing node friendly names with green badges.
- **ModelPickerGroup.svelte**: Accepts new `downloadStatus` prop. Shows
a green checkmark-in-circle icon next to models that are downloaded on
sufficient nodes. Tooltip shows which nodes have the model.
- **+page.svelte**: Passes `downloadsData` and `topologyNodes` to
`ModelPickerModal`.
## Why It Works
The download state from `/state` already tracks per-node completed
downloads. The shared `getNodesWithModelDownloaded()` utility (from PR
#1375) finds nodes with `DownloadCompleted` entries for each model.
Total RAM is summed from the topology node data (using `ram_total`, not
`ram_available`) and compared to the model's `storage_size_megabytes` to
determine if there's enough aggregate memory. This is intentionally a
simple heuristic — not a full placement preview.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Open the model picker modal
- Verify downloaded models show a green checkmark icon
- Verify the checkmark appears dimmer for models downloaded on nodes
with insufficient total RAM
- Click the (i) info button on a downloaded model
- Verify "Downloaded on:" section appears with correct node names
- Verify models with no downloads show no indicator
### Automated Testing
- Dashboard builds successfully (`npm run build`)
- No new Python changes requiring type checking
> **Note:** This is a chained PR. Base branch is
`alexcheema/topology-download-indicators` (#1375).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Qwen3-Coder-Next just dropped on mlx-community in several quantizations.
It's an 80B MoE model (Qwen3NextForCausalLM) which we already have
tensor parallelism support for via QwenShardingStrategy — just needs
model cards.
## Changes
Added model cards for all 5 available quantizations:
- `mlx-community/Qwen3-Coder-Next-4bit` (~46GB)
- `mlx-community/Qwen3-Coder-Next-5bit` (~58GB)
- `mlx-community/Qwen3-Coder-Next-6bit` (~69GB)
- `mlx-community/Qwen3-Coder-Next-8bit` (~89GB)
- `mlx-community/Qwen3-Coder-Next-bf16` (~158GB)
All with `supports_tensor = true` since the architecture is already
supported.
## Why It Works
`Qwen3NextForCausalLM` is already handled by QwenShardingStrategy in
auto_parallel.py and is in the supports_tensor allowlist in
model_cards.py. No code changes needed — just the TOML card files.
## Test Plan
### Manual Testing
<!-- n/a - model card addition only -->
### Automated Testing
- `basedpyright` — 0 errors
- `ruff check` — passes
- `nix fmt` — no changes
- `pytest` — 173 passed, 1 skipped
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The macOS app required user interaction via AppleScript prompts to
install or uninstall network configuration components, making automated
deployments difficult.
Added --install and --uninstall command line flags that execute the
network setup scripts directly when running as root, bypassing GUI
prompts. Created a new main.swift entry point that parses CLI arguments
and delegates to NetworkSetupHelper's new direct execution methods.
This enables headless installation via `sudo EXO --install` for
automated deployment scenarios while preserving the existing GUI
behavior when launched normally.
Test plan:
- Deployed to a machine that didn't have the content installed. Got
blocked on the popup and EXO never launched.
- Relaunched EXO, confirmed it still never starts because of the popup.
- Ran `sudo /Applications/EXO.app/Contents/MacOS/EXO --install`
- Launched EXO - the API started as expected.
- Ran `sudo /Applications/EXO.app/Contents/MacOS/EXO --uninstall`
- Launched EXO - got the popup.
## Motivation
Adds uncertainty visualization to the chat interface, allowing users to
see token-level confidence scores and regenerate responses from any
point in the generation. This enables users to:
- Understand model confidence at each token
- Explore alternative completions by regenerating from uncertain tokens
- Debug and analyze model behavior
## Changes
### Uncertainty Visualization
- Add `TokenHeatmap` component showing token-level probability coloring
- Toggle uncertainty view per message with bar chart icon
- Display tooltip with probability, logprob, and top alternative tokens
on hover
### Regenerate from Token
- Add "Regenerate from here" button in token tooltip
- Use `continue_final_message` in chat template to continue within same
turn (no EOS tokens)
- Add `continue_from_prefix` flag to `ChatCompletionTaskParams`
### Request Cancellation
- Add `AbortController` to cancel in-flight requests when regenerating
mid-generation
- Handle `BrokenResourceError` server-side when client disconnects
gracefully
### Additional APIs
- Add Claude Messages API support (`/v1/messages`)
- Add OpenAI Responses API support (`/v1/responses`)
## Why It Works
- **Proper continuation**: Using `continue_final_message=True` instead
of `add_generation_prompt=True` keeps the assistant turn open, allowing
the model to continue naturally from the prefix without end-of-turn
markers
- **Clean cancellation**: AbortController aborts the HTTP request, and
server catches `BrokenResourceError` to avoid crashes
- **Stable hover during generation**: TokenHeatmap tracks hover by index
(stable across re-renders) with longer hide delay during generation
## Test Plan
### Manual Testing
<!-- Hardware: MacBook Pro M1 -->
- Send a message and verify logprobs are collected
- Enable uncertainty view and verify token coloring based on probability
- Hover over tokens to see tooltip with alternatives
- Click "Regenerate from here" on a token mid-response
- Verify the response continues naturally from that point
- Verify aborting mid-generation and regenerating works without server
crash
### Automated Testing
- Added tests for Claude Messages API adapter
- Added tests for OpenAI Responses API adapter
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Duplicate tasks are still observed.
## Changes
Moved task acknowledgement to after the runner has changed its status.
## Why It Works
Tasks now remain pending until the runner has updated its status.
## Test Plan
### Manual Testing
Seems to work fine from manual testing. Hard to test a race condition
though.
### Automated Testing
Updated the event ordering test.
## Motivation
Enable parallel classifier-free guidance (CFG) for Qwen image models.
CFG requires two forward passes (positive/negative prompts) - this
allows them to run on separate nodes simultaneously, reducing latency.
## Changes
- Added uses_cfg flag to ModelCard to identify CFG-based models
- Extended PipelineShardMetadata with CFG topology fields (cfg_rank,
cfg_world_size, peer device info)
- Updated placement to create two CFG groups with reversed ordering
(places CFG peers as ring neighbors)
- Refactored DiffusionRunner to process CFG branches separately with
exchange at last pipeline stage
- Added get_cfg_branch_data() to PromptData for single-branch embeddings
- Fixed seed handling in API for distributed consistency
- Fixed image yield to only emit from CFG rank 0 at last stage
- Increased num_sync_steps_factor from 0.125 to 0.25 for Qwen
## Why It Works
- 2 nodes + CFG: Both run all layers, process different CFG branches in
parallel
- 4+ even nodes + CFG: Hybrid - 2 CFG groups × N/2 pipeline stages
- Odd nodes or non-CFG: Falls back to pure pipeline parallelism
Ring topology places CFG peers as neighbors to enable direct exchange.
## Test Plan
### Manual Testing
Verified performance gain for Qwen-Image for 2 node and 4 node cluster.
Non-CFG models still work
### Automated Testing
Added tests in test_placement_utils.py covering 2-node CFG parallel,
4-node hybrid, odd-node fallback, and non-CFG pipeline modes.
## Motivation
Reimplements the model picker modal from #1191 on top of the custom
model support branch. Replaces the inline model dropdown with a
full-featured modal that groups models by base model, supports
filtering, favorites, and HuggingFace Hub search.
## Changes
**Backend:**
- Add `family`, `quantization`, `base_model`, `capabilities` metadata
fields to `ModelCard` and all 40 TOML model cards
- Pass new fields through `ModelListModel` and `get_models()` API
response
- Add `GET /models/search` endpoint using
`huggingface_hub.list_models()`
**Dashboard (7 new files):**
- `ModelPickerModal.svelte` — Main modal with search, family filtering,
HuggingFace Hub tab
- `ModelPickerGroup.svelte` — Expandable model group row with
quantization variants
- `FamilySidebar.svelte` — Vertical sidebar with family icons (All,
Favorites, Hub, model families)
- `FamilyLogos.svelte` — SVG icons for each model family
- `ModelFilterPopover.svelte` — Capability and size range filters
- `HuggingFaceResultItem.svelte` — HF search result item with
download/like counts
- `favorites.svelte.ts` — localStorage-backed favorites store
**Integration:**
- Replace inline dropdown in `+page.svelte` with button that opens
`ModelPickerModal`
- Custom models shown in Hub tab with delete support
**Polish:**
- Real brand logos (Meta, Qwen, DeepSeek, OpenAI, GLM, MiniMax, Kimi,
HuggingFace) from Simple Icons / LobeHub
- Clean SVG stroke icons for capabilities (thinking, code, vision, image
gen)
- Consistent `border-exo-yellow/10` borders, descriptive tooltips
throughout
- Cluster memory (used/total) shown in modal header
- Selected model highlight with checkmark for both single and
multi-variant groups
- Cursor pointer on all interactive elements, fix filter popover
click-outside bug
- Custom models now appear in All tab alongside built-in models
## Bug Fix: Gemma 3 EOS tokens
Also included in this branch: fix for Gemma 3 models generating infinite
`<end_of_turn>` tokens. The tokenizer's `eos_token_ids` was missing
token ID 106 (`<end_of_turn>`), so generation never stopped. The fix
appends this token to the EOS list after loading the tokenizer. Also
handles `eos_token_ids` being a `set` (not just a `list`).
## Why It Works
Model metadata (family, capabilities, etc.) is stored directly in TOML
cards rather than derived from heuristics, ensuring accuracy. The modal
groups models by `base_model` field so quantization variants appear
together. Custom models are separated into the Hub tab since they lack
grouping metadata.
## Test Plan
### Manual Testing
- Open dashboard, click model selector to open modal
- Browse models by family sidebar, search, and filters
- Expand model groups to see quantization variants
- Star favorites and verify persistence across page reloads
- Navigate to Hub tab, search and add models
- Verify error messages shown for invalid model IDs
- Run a Gemma 3 model and verify generation stops at `<end_of_turn>`
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `nix fmt` — clean
- `uv run pytest src/` — 173 passed
- `cd dashboard && npm run build` — builds successfully
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Users should be able to run any HuggingFace model, not just the ones we
ship TOML cards for. Continues the aim of #1191 with a minimal
implementation on top of the current TOML model card system.
Custom cards are saved to `~/.exo/custom_model_cards/` rather than the
bundled `resources/inference_model_cards/` because `RESOURCES_DIR` is
read-only in PyInstaller bundles (`sys._MEIPASS`). This also fixes
`fetch_from_hf` which was saving cards to the wrong path (`resources/`
root instead of `resources/inference_model_cards/`).
## Changes
- Add `EXO_CUSTOM_MODEL_CARDS_DIR` constant
(`~/.exo/custom_model_cards/`)
- Update `model_cards.py`: add custom dir to search path, fix
`save_to_custom_dir`, add `delete_custom_card`/`is_custom_card`
- Add `POST /models/add` and `DELETE /models/custom/{model_id}` API
endpoints
- Add `is_custom` field to `ModelListModel` API response
- Dashboard: add custom model input form in dropdown, delete button for
custom models, show actual API errors, auto-select newly added model
## Why It Works
Two separate directories for model cards: the bundled read-only
`resources/inference_model_cards/` for built-in cards, and user-writable
`~/.exo/custom_model_cards/` for custom cards. Both are scanned when
listing models. This works in all environments including PyInstaller
bundles where `RESOURCES_DIR` points to `sys._MEIPASS`.
## Test Plan
### Manual Testing
- Add a custom model via the dropdown (e.g.
`mlx-community/Llama-3.2-1B-Instruct-4bit`)
- Verify it appears in the model list with the delete (x) button
- Delete it and verify it disappears
- Try adding an invalid model ID and verify the actual error is shown
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `uv run pytest src/` — passes
- `cd dashboard && npm run build` — builds
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Offline users currently have to wait for every retry to fail before
being able to launch a model.
For users that restart clusters often or share API keys between devices,
we also spam HuggingFace with downloads every 5 minutes.
These issues are caused by _emit_existing_download_progress being
inefficient.
## Changes
- Only query HuggingFace once while EXO is running (assumption being
that a change should only be reflected on a new EXO session)
- Only query HuggingFace when there is an internet connection (polling
connectivity every 10 seconds)
- Request download progress if we switch from no connectivity ->
connected to reduce the wait.
- Reduce download progress sleep as it's no longer expensive (queries
cache most of the time).
- Reduce retries as 30 is way too many.
## Test Plan
### Manual Testing
Manually tested the behaviour.
### Automated Testing
None, should I add any? We do have some tests for this folder, but they
are probably not too helpful.
## Motivation
With the addition of the Responses API, we introduced `str |
list[InputMessage]` as the type for `TextGenerationTaskParams.input`
since the Responses API supports sending input as a plain string. But
there was no reason to leak that flexibility past the API adapter
boundary — it just meant every downstream consumer had to do `if
isinstance(messages, str):` checks, adding complexity for no benefit.
## Changes
- Changed `TextGenerationTaskParams.input` from `str |
list[InputMessage]` to `list[InputMessage]`
- Each API adapter (Chat Completions, Claude Messages, Responses) now
normalizes to `list[InputMessage]` at the boundary
- Removed `isinstance(task_params.input, str)` branches in
`utils_mlx.py` and `runner.py`
- Wrapped string inputs in `[InputMessage(role="user", content=...)]` in
the warmup path and all test files
## Why It Works
The API adapters are the only place where we deal with raw user input
formats. By normalizing there, all downstream code (worker, runner, MLX
engine) can just assume `list[InputMessage]` and skip the type-checking
branches. The type system (`basedpyright`) catches any missed call sites
at compile time.
## Test Plan
### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `nix fmt` — applied
- `uv run pytest` — 174 passed, 1 skipped
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Our distributed test now does a full query cycle for every model loaded
onto the relevant machine. This will help find bugs early, as it already
has found one with Qwen3 Next! I didn't write down what the error was
though. Gooooooood luck with that!
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
Tool-call requests can hang indefinitely when `max_tokens` truncates
generation mid-tool-call.
## Reproduction
1. Send a chat completion with `tools` and a low `max_tokens` (e.g. 65)
to Qwen3-0.6B
2. Model generates `<think>...</think>` then starts `<tool_call>` but
`max_tokens` cuts it off before `</tool_call>`
3. **Before this fix:** `parse_tool_calls` buffers tokens after
`<tool_call>`, generator exhausts, buffered tokens (including
`finish_reason`) are silently dropped → stream hangs forever
4. **After this fix:** buffered tokens are flushed as regular text with
`finish_reason` propagated → response returns normally with
`finish_reason: "length"`
Confirmed with fresh local testing: 4 unclosed tool call flushes
triggered in a single session. Also confirmed via production logs from
Jan 29 (2 occurrences).
## Changes
1. **`parse_tool_calls` unclosed tool call flush** — when the generator
exhausts inside an open `<tool_call>` block, flush buffered tokens as
regular text and propagate `finish_reason`
2. **GLM regex fix** — match literal `\n` (not escaped `\\n`) between
arg tags; handle missing `</arg_value>` via lookahead
3. **7 new unit tests** for `parse_tool_calls` covering unclosed,
closed, passthrough, and failed-parse scenarios
## Why It Works
- `parse_tool_calls` now has a post-loop check: if `in_tool_call` is
still true, it yields the buffered text with the tracked `finish_reason`
instead of silently dropping it
- The GLM regex now matches real-world output where newlines appear
between tags and `</arg_value>` may be absent
## Test Plan
### Manual Testing
- Qwen3-0.6B-4bit with `tools` + various `max_tokens` values (61-75)
- Confirmed responses return with `finish_reason: "length"` instead of
hanging
- Log output shows `"generator exhausted inside unclosed tool call,
flushing buffered text"`
### Automated Testing
- 7 new tests in `test_parse_tool_calls.py`
- Full test suite passes (`uv run pytest`)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
Co-authored-by: Evan <evanev7@gmail.com>
Co-authored-by: Jake Hillion <jake@hillion.co.uk>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
we didn't check before, raising a critical exception.
now we create ~/.config/exo on linux systems before touching config.toml.
this wasn't caught before since everything lives in ~/.exo on macos, and we no longer write the keypair to CONFIG_HOME, so config.toml has to do init work it avoided before.
## Motivation
Know what the hell is going on
## Changes
- Tracing library (src/exo/shared/tracing.py): trace() context manager,
Chrome Trace Format export, statistics computation
- Runner instrumentation
(src/exo/worker/engines/image/pipeline/runner.py): Wrapped sync/async
steps, compute blocks, and send/recv operations
- Trace collection: Workers send traces to master after task completion;
merged into ~/.exo/traces/trace_{task_id}.json
- API endpoints: List, fetch, stats, and raw download at /v1/traces/*
- Dashboard: Trace list and detail pages with Perfetto integration
## Why It Works
<img width="1236" height="767" alt="Screenshot 2026-01-30 at 19 00 09"
src="https://github.com/user-attachments/assets/73e6e46d-ba10-4e83-ba99-ff1c3f62ab05"
/>
<img width="1659" height="89" alt="Screenshot 2026-01-30 at 19 00 58"
src="https://github.com/user-attachments/assets/c0fd0e65-e4fc-4fd5-920d-b43b2887d591"
/>
## motivation
eagerly discovered peers through gossipsub were added to state. this left things looking broken from one-sided connections
## changes
the worker no longer writes topology edges from these gossipsub messages
we now strictly rely on http-discovered topology, which tends to be more reflective of the actual state of the systems connectivity
## Motivation
`hosts_*.json` files are local host configuration snapshots that
shouldn't be tracked in version control.
## Changes
Added `hosts_*.json` pattern to `.gitignore`.
## Why It Works
The glob pattern `hosts_*.json` matches any file starting with `hosts_`
and ending with `.json` in the repo root.
## Test Plan
### Manual Testing
- Verified that `hosts_*.json` files are ignored by git after this
change.
### Automated Testing
- No automated tests needed for a `.gitignore` change.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Add support for Claude Messages API and OpenAI Responses API to allow
users to interact with exo using these popular API formats. This enables
broader compatibility with existing tooling and SDKs that expect these
API formats.
## Architecture
Adapter logic lives exclusively in the API layer
(`src/exo/master/adapters/`). On the way in, each adapter converts its
API-specific request type (`ChatCompletionRequest`,
`ClaudeMessagesRequest`, `ResponsesRequest`) into
`TextGenerationTaskParams`. On the way out, each adapter converts the
`TokenChunk` stream back into its API-specific response format.
Everything inside the application — commands, worker, runner, event
sourcing — only sees `TextGenerationTaskParams` and `TokenChunk`. No
API-specific types cross the boundary.
```
API layer │ Application internals
│
Chat Completions → [adapter] → TextGenerationTaskParams ──→ │ ──→ TextGeneration command → Runner → TokenChunk ──→ │ ──→ [adapter] → ChatCompletionResponse
Claude Messages → [adapter] → TextGenerationTaskParams ──→ │ ──→ TextGeneration command → Runner → TokenChunk ──→ │ ──→ [adapter] → ClaudeMessagesResponse
Responses API → [adapter] → TextGenerationTaskParams ──→ │ ──→ TextGeneration command → Runner → TokenChunk ──→ │ ──→ [adapter] → ResponsesResponse
```
## Changes
### New Files
- `src/exo/shared/types/claude_api.py` - Pydantic types for Claude
Messages API
- `src/exo/shared/types/openai_responses.py` - Pydantic types for OpenAI
Responses API
- `src/exo/shared/types/text_generation.py` - Shared
`TextGenerationTaskParams` internal type
- `src/exo/master/adapters/chat_completions.py` - Chat Completions
adapter (streaming/non-streaming)
- `src/exo/master/adapters/claude.py` - Claude Messages adapter
(streaming/non-streaming)
- `src/exo/master/adapters/responses.py` - OpenAI Responses adapter
(streaming/non-streaming)
### Modified Files
- `src/exo/master/api.py` - Refactored to use adapters uniformly for all
endpoints; extracted `_resolve_and_validate_text_model` helper to
deduplicate model validation across all text endpoints; removed ad-hoc
`try/except ValueError` blocks from non-streaming paths
### New Endpoints
- `POST /v1/messages` - Claude Messages API (streaming and
non-streaming)
- `POST /v1/responses` - OpenAI Responses API (streaming and
non-streaming)
## Why It Works
All APIs are implemented as pure conversion adapters at the edge of the
application:
1. Adapter functions in `src/exo/master/adapters/` convert incoming
requests to `TextGenerationTaskParams`
2. `api.py` wraps the params in a `TextGeneration` command and sends it
through the existing command/event flow
3. The worker, runner, and event sourcing layers only handle
`TextGenerationTaskParams` and `TokenChunk` — they have no awareness of
Chat Completions, Claude, or Responses API formats
4. On response, adapter functions convert the `TokenChunk` stream back
to the caller's expected format
5. Model validation is handled by a single shared helper
(`_resolve_and_validate_text_model`), mirroring the existing
`_validate_image_model` pattern for image endpoints
No changes to core inference logic were needed.
### Streaming Formats
- **Chat Completions**: Uses `data: {...}\n\n` with `[DONE]` terminator
- **Claude**: Uses event types `message_start`, `content_block_start`,
`content_block_delta`, `content_block_stop`, `message_delta`,
`message_stop`
- **OpenAI Responses**: Uses event types `response.created`,
`response.in_progress`, `response.output_item.added`,
`response.content_part.added`, `response.output_text.delta`,
`response.output_text.done`, `response.content_part.done`,
`response.output_item.done`, `response.completed`
## Test Plan
### Manual Testing
Hardware: MacBook Pro M3 Max
**Non-streaming tests:**
```bash
# Chat Completions API
curl -X POST http://localhost:52415/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.2-1b", "messages": [{"role": "user", "content": "Hello"}], "max_tokens": 20}'
# Claude Messages API
curl -X POST http://localhost:52415/v1/messages \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.2-1b", "max_tokens": 50, "messages": [{"role": "user", "content": "Hello"}]}'
# OpenAI Responses API
curl -X POST http://localhost:52415/v1/responses \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.2-1b", "input": "Hello", "max_output_tokens": 20}'
```
**Streaming tests:**
```bash
# Chat Completions API (streaming)
curl -N -X POST http://localhost:52415/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.2-1b", "messages": [{"role": "user", "content": "Hello"}], "stream": true, "max_tokens": 20}'
# Claude Messages API (streaming)
curl -N -X POST http://localhost:52415/v1/messages \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.2-1b", "max_tokens": 50, "messages": [{"role": "user", "content": "Hello"}], "stream": true}'
# OpenAI Responses API (streaming)
curl -N -X POST http://localhost:52415/v1/responses \
-H "Content-Type: application/json" \
-d '{"model": "llama-3.2-1b", "input": "Hello", "stream": true, "max_output_tokens": 20}'
```
All endpoints tested successfully with proper response formats and
streaming events.
### Automated Testing
- Tests in `src/exo/master/tests/` all pass (85 tests)
- Type checker (basedpyright) passes with 0 errors
- Linter (ruff) passes
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Evan <evanev7@gmail.com>
## Motivation
Make exo bench faster for longer prompts, lengthen default timeouts and
use pairs for pp and tg.
## Changes
- Uses binary search to find correct prompt
- Flag to force all combinations if that is desired
`nix run .#exo` couldn't find `macmon` because the Nix wrapper scripts
didn't include it in PATH, causing `shutil.which("macmon")` to fail.
Added `--prefix PATH : ${pkgs.macmon}/bin` to the `makeWrapper` call,
conditional on Darwin via `lib.optionalString`, so macmon's binary is
available at runtime without modifying the user's system PATH.
Test plan:
- Verified `nix build .#exo` succeeds
- Checked wrapper script contains macmon store path in PATH prefix
## Motivation
If nodes have uneven memory, one node may evict cache that remains on
another node. This will break prefill on some setups.
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
we currently have no strict reuqirements that node ids persist across
sessions, so we can generate fresh nodeids each time
this avoids issues like #1332, but prevents further features such as
caching downloads or node-id dialling
Co-authored-by: rltakashige <rl.takashige@gmail.com>
some attempts to load model cards (i.e. build_base_shard) always went
through networking rather than using downloaded model cards. we should
always default to ModelCard.load in these scenarios
## Motivation
(Probably) the final missing piece of the Chat Completions API
## Changes
Add UsageStats
## Why It Works
OpenCode reviewed my PR and gave me stats:
<img width="1150" height="802" alt="image"
src="https://github.com/user-attachments/assets/ebc06bae-797f-4087-87d5-2f26cf60fc48"
/>
## Test Plan
### Automated Testing
No tests were broken.
## Motivation
A lot of changes happened without much attention to the state of exo
bench.
## Changes
Use TaggedModel for BenchChatCompletion so it serialises properly.
Don't break after gpt oss tool call to preserve parity with the rest of
the codebase.
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<img width="2856" height="678" alt="image"
src="https://github.com/user-attachments/assets/2e18cf0d-c0f8-467c-9763-1a6a59c8a327"
/>
Also tested GPT OSS tool calling in OpenCode
## Motivation
The dashboard UI attempted to parse all image generation responses as
SSE streams, even when streaming was disabled. This broke non-streaming
image generation.
## Changes
- Parse JSON responses directly when not streaming, use SSE parser only
when stream=true AND partialImages > 0
- explicitly disable partial images when not streaming
## Why It Works
Both API and dashboard now use the same condition (stream &&
partialImages > 0) to determine response format, ensuring correct
parsing.
## Test Plan
### Manual Testing
Non-streamed image generation results appear in the UI. Streamed image
generation still works
## Motivation
Warnings that go unchecked tend to accumulate and hide real issues.
Treating them as errors ensures they are addressed immediately, both
locally during development and in CI.
## Changes
Added `SWIFT_TREAT_WARNINGS_AS_ERRORS = YES` and
`GCC_TREAT_WARNINGS_AS_ERRORS = YES` to the **project-level** Debug and
Release build configurations in `project.pbxproj`. This applies to all
targets (EXO, EXOTests, EXOUITests).
## Why It Works
Xcode's `SWIFT_TREAT_WARNINGS_AS_ERRORS` and
`GCC_TREAT_WARNINGS_AS_ERRORS` build settings promote Swift and C/ObjC
warnings to errors at compile time. Setting them at the project level
means all targets inherit the policy without needing per-target or
CI-level overrides.
## Test Plan
### Manual Testing
- Built the EXO scheme in Release configuration with `xcodebuild` — no
warning-as-error failures from Swift or C/ObjC sources.
### Automated Testing
- CI already builds with `-configuration Release`, so it will
automatically enforce warnings-as-errors via the inherited project
settings — no CI changes needed.
Add uv2nix to build Python packages from uv.lock. This creates a fully
Nix-managed Python environment with the Rust bindings injected via overlay.
Changes:
- Add pyproject-nix, uv2nix, and pyproject-build-systems flake inputs
- Create python/parts.nix with overlays to inject Nix-built Rust wheel
- Export packages.exo on macOS (wraps exo/exo-master/exo-worker with dashboard)
- Add checks.lint (ruff, all platforms) and checks.pytest (macOS only)
- Simplify CI typecheck job using nicknovitski/nix-develop action
- Delete .github/actions/typecheck composite action (no longer needed)
- Add no-build-package for MLX packages in pyproject.toml (use wheels)
The Python build is currently macOS-only since MLX requires Metal. Linux
support will be added once the pyproject dependencies are simplified.
Test plan:
- Run `nix flake check` on macOS to verify pytest and lint pass
- Build exo package on macOS: `nix build .#exo`
- Verify CI pipeline passes with simplified typecheck job
In order to make testing and deployment simpler and more reproducible,
we want to provide a Nix derivation for our macOS .app build. We already
build the Rust and dashboard with Nix, but so far the Python has been
blocked because we haven't had an MLX build.
This change adds a Metal compiler derivation that uses `requireFile` to
be provided a NAR of the unfree macOS Metal compiler. It is documented
how to get this file, but effectively you have to trigger the download,
mount the DMG, and NAR the result. Once this is added to the store by
hash we can build MLX using it. The MLX build itself is quite self
explanatory.
Test plan:
- CI. We follow the instructions to grab the Metal compiler. Once this
is in Cachix we should really never do this again, and I can pin the
path too to ensure it doesn't leave.
- MLX tests run as part of the MLX derivation's build. They pass.
- `NIXPKGS_ALLOW_UNFREE=1 nix build .#mlx.passthru.tests.mlxTest
--impure --option sandbox false`
---------
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Summary
- Add 20-second startup delay to wait for macOS to finish network setup
after boot
- Update user-facing message to clarify the service configures local
networking, disables Thunderbolt Bridge (preventing packet storms), and
installs a Network Location
## Test plan
- [ ] Manual verification of Swift syntax
- [ ] Test network setup on macOS device after reboot
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
Fix "Network setup script failed" error when clicking uninstall button
and resolve Xcode compiler warnings.
## Changes
- NetworkSetupHelper.swift: Add || true guards and explicit return 0 in
find_and_enable_thunderbolt_bridge to prevent script failures with set
-euo pipefail
- ThunderboltBridgeService.swift: Use withCString and
withUnsafeMutablePointer for Authorization API calls to fix pointer
lifetime warnings
- EXOApp.swift: Mark showNotification as nonisolated to fix main actor
isolation warning
## Why It Works
- The uninstall script's Thunderbolt re-enable function could exit
non-zero in edge cases (no bridges, no matches). Since this is a cleanup
step, failures should not abort uninstall.
- Swift requires explicit pointer lifetime management when passing
strings/structs to C APIs.
- showNotification is called from a nonisolated delegate method and uses
thread-safe APIs.
## Test Plan
### Manual Testing
Hardware: MacBook Pro
- Clicked Uninstall button, verified it completes without error
- Built in Xcode, verified no warnings
### Automated Testing
N/A
## Motivation
Running RDMA from source is not well documented as is. Several
surprising things that took time to debug internally too.
App should be updated to detect MacOS versions in future.
## motivation
our shard downloader could not correctly fetch data for kimi-k2, as it
deferred some values to a text_config field.
## changes
config_data now prioritizes this field if it exists in information like
layer_count
## Summary
- Bump mlx from 0.30.3 to 0.30.4
## Test plan
- [x] `uv lock` succeeds
- [x] Type checking passes (`uv run basedpyright`)
- [x] Run inference tests
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
---------
Co-authored-by: Alex Cheema <41707476+AlexCheema@users.noreply.github.com>
## Motivation
Broken right now. We'll potentially add a better one later
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
Used for evals without any issue.
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
After the download verification fix, when files are re-downloaded due to
upstream changes (size mismatch), the download progress displays
correctly (completion %, bytes, file counts), but speed shows 0 B/s and
ETA shows "--" for both overall and per-file progress.
## Changes
- Modified `on_progress_wrapper` in `src/exo/download/download_utils.py`
to detect re-download scenarios
- Added re-download detection: when `curr_bytes < previous_downloaded`,
the file was deleted and download restarted
- On re-download: reset `start_time` to current time and set
`downloaded_this_session = curr_bytes`
- Added two tests to `test_download_verification.py` covering
re-download and continuing download scenarios
## Why It Works
The bug occurred because:
1. `file_progress` is initialized with the OLD local file size (e.g.,
1.5GB)
2. When `_download_file` detects size mismatch, it deletes the file and
starts fresh
3. Progress callback receives small `curr_bytes` (e.g., 8KB) but
compares against old size
4. `downloaded_this_session = 0 + (8KB - 1.5GB) = -1.5GB` (negative!)
5. Negative session bytes → 0 or negative speed → ETA shows "--"
The fix detects when `curr_bytes < previous_downloaded` (indicating
re-download started) and resets tracking to treat it as a fresh
download.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Download a model, modify a file to change its size, restart exo,
verify speed/ETA display correctly during re-download
### Automated Testing
- Added `TestProgressResetOnRedownload` class with two tests:
- `test_progress_resets_correctly_on_redownload`: Verifies progress
resets correctly when re-download starts
- `test_progress_accumulates_on_continuing_download`: Verifies
continuing downloads still accumulate correctly
- All 11 download tests pass
- Type checking (basedpyright): 0 errors
- Linting (ruff): All checks passed
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
OpenCode sends very large prompts, most of which are repeated on the
next call.
## Changes
Add prefix caching, reducing average time in prefill (in testing) from
40 seconds to 4. This massively improves user experience.
Also evicts KV caches from this prefix cache in a LRU-style manner.
## Why It Works
We no longer prefill repeatedly but rather use kv cache stored in
memory. A future update may want to use storage to make the prefix cache
larger.
## Test Plan
### Manual Testing
Tested speedup on OpenCode
### Automated Testing
Added a lot of tests
---------
Co-authored-by: David Hind <davehind@yahoo.co.uk>
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
Enable users to select and use quantized variants (8-bit, 4-bit) of
image models
## Changes
Use exolabs HF org for image models
## Why It Works
Quantized versions have been uploaded to exolabs HF org
## Test Plan
Loaded and ran different quantized variants. Confirmed lower memory
usage and different outputs for the same seed. Verified chat completion
still works.
## Motivation
Some models, on some configurations, would have several issues that
caused the model to be stuck on loading.
## Changes
Several loading issues were with upstream mlx lm shard loading for
tensor parallel.
GLM 4.7 Flash now uses GLM 4.7 Lite.
A final portion of the issues were from mlx memory not being properly
released before calling mx.eval(model), causing the system to run out of
memory.
## Test Plan
### Manual Testing
Done a bunch (thanks @AlexCheema), hopefully exhaustive.
### Automated Testing
A bunch of automated testing is imminent but not landed yet.
---------
Co-authored-by: Alex Cheema <alexcheema123@gmail.com>
## Motivation
When upstream files change on Hugging Face, exo's download system
doesn't detect the change and downloads get stuck. The only workaround
is deleting `~/.exo/models/` and the cache.
Root causes:
1. Existing files are never re-verified against remote metadata
2. File list cache is never invalidated, causing stale sizes to be used
## Changes
1. **Verify existing files against remote size** (`_download_file`):
Before returning early for existing files, verify the local file size
matches remote. If mismatched, delete and re-download. If network fails
(offline), fall back to trusting local file.
2. **Always try fresh file list first** (`fetch_file_list_with_cache`):
Always attempt to fetch fresh data from Hugging Face. On success, update
the cache. On failure, fall back to cached data if available.
3. **Clear cache on model delete** (`delete_model`): When a model is
deleted, also delete its cache entry to prevent stale metadata.
## Why It Works
- **Online**: Stale local files are detected via size mismatch and
re-downloaded. Fresh file list is always fetched and cache is updated.
- **Offline with cache**: Existing files are trusted. Cached file list
is used as fallback.
- **Offline without cache**: Fails gracefully (can't download without
knowing what files to get).
The size check is O(1) so there's no performance impact. Hash
verification still happens after download completes (existing behavior).
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Download a model, manually modify a local file's content, restart exo,
verify it re-downloads
### Automated Testing
Added 9 new tests in
`src/exo/download/tests/test_download_verification.py`:
- Re-download when file size changes upstream
- Skip download when file size matches
- Offline fallback uses local file
- Fetch fresh file list and update cache
- Fall back to cache when fetch fails
- Error propagates when no cache exists
- Model delete clears cache
- Delete when only cache exists
- Delete nonexistent model
All tests pass: `uv run pytest src/exo/download/tests/ -v`
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Summary
- When a node times out, its info gets cleared from state. The monitor
functions only sent data when something changed, leaving no mechanism to
re-populate this info after a timeout.
- Removes change-detection checks from `_monitor_misc`,
`_monitor_system_profiler_thunderbolt_data`, `_watch_system_info`, and
`_monitor_thunderbolt_bridge_status` so data is sent periodically
regardless of whether it changed.
## Test plan
- [ ] Verify type checker passes: `uv run basedpyright`
- [ ] Verify linter passes: `uv run ruff check`
- [ ] Verify tests pass: `uv run pytest`
- [ ] Manually test that node info is re-populated after a timeout by
observing cluster behavior
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The prettier-svelte package was rebuilding whenever any file in the
repository changed because dashboardStubSrc referenced inputs.self
directly. Since inputs.self's store path hash is computed from the
entire repository contents, any file modification invalidated the
derivation.
Added dashboardLockfileSrc using lib.cleanSourceWith to filter
inputs.self to only include package.json and package-lock.json from
the dashboard directory. Updated dashboardStubSrc to reference this
filtered source instead of inputs.self directly.
This ensures prettier-svelte only rebuilds when the lockfiles actually
change, significantly improving build caching for unrelated changes.
Test plan:
- Built prettier-svelte with nix build .#prettier-svelte
- Modified src/exo/main.py and rebuilt - same store path (no rebuild)
- Modified dashboard/package.json and rebuilt - different store path (rebuild triggered)
- Ran nix flake check successfully
## Summary
Reverts the state compaction feature (#1259) to investigate issues with
nodes staying as "unknown" after joining a cluster.
## Test plan
- [ ] Verify nodes properly show up after joining cluster
- [ ] Verify state catchup works correctly without compaction
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## motivation
a node joining a long-running cluster would bring down networking. this
attempts to mitigate that issue by compacting the state for catching up
new devices
## changes
introduces a new topic ("state_catchup") over which a full state can be
sent. currently the master sends the worker + api this new state, and
they update only if they have no other events applied - otherwise usual
NACK systems function
## testing
manually tested on two and eight nodes - its an improvement, not a fix
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
The LaunchDaemon approach for disabling Thunderbolt Bridge was removed
in commit 43f12f5d and replaced with dynamic cycle detection. However,
the LaunchDaemon runs automatically on reboot, ensuring the bridge is
always disabled before it can cause packet storms.
## Changes
- Restore `NetworkSetupHelper.promptAndInstallIfNeeded()` to install a
LaunchDaemon that disables Thunderbolt Bridge on startup
- Show user prompt explaining what will be installed before requesting
admin password
- Remove old cleanup-only logic from `EXOApp.swift`
- Installer removes any existing installation before installing fresh
(handles upgrades)
## Why It Works
The LaunchDaemon runs at boot with `RunAtLoad=true` and periodically
(every ~30 min), destroying bridge0 and disabling Thunderbolt Bridge
before it can cause packet storms. The daemon is only installed
once—`daemonAlreadyInstalled()` checks script content and plist config
match before prompting.
## Test Plan
### Manual Testing
- Run app first time → should see prompt → click Install → enter admin
password → daemon installed
- Run app again → no prompt (already installed)
- Reboot → bridge0 should be destroyed/disabled automatically
- Check daemon: `launchctl list | grep io.exo.networksetup`
- Check files: `/Library/LaunchDaemons/io.exo.networksetup.plist`,
`/Library/Application Support/EXO/disable_bridge.sh`
### Automated Testing
N/A - requires admin privileges and system-level changes
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
When a user switched conversations while a response was still streaming,
the streaming content would be written to the currently selected
conversation instead of the original one. For streamed image generation,
each partial image would be written to the open conversation
## Changes
Added helper methods to track and update the correct conversation during
streaming:
- updateConversationMessage() - Update a message in a specific
conversation by ID
- syncActiveMessagesIfNeeded() - Sync this.messages from target
conversation only if it's active
- conversationExists() - Check if a conversation still exists (handles
mid-stream deletion)
- persistConversation() - Persist a specific conversation to storage
- addMessageToConversation() - Add a message directly to a specific
conversation
## Why It Works
Capturing the conversation ID at the start of the request ensures we
know which conversation to update
## Test Plan
### Manual Testing
Tested switching conversation during generation across each model type
## Motivation
When using the dashboard's instance placement filter (clicking nodes in
the topology), it was filtering to placements that use exactly the
selected nodes. This isn't the expected behavior - users want to see
placements that include all selected nodes, but may also include
additional nodes.
For example, selecting nodes [A, B] should show placements using [A, B],
[A, B, C], [A, B, C, D], etc. - not just [A, B].
## Changes
- Added `required_nodes` parameter to `place_instance()` in
`placement.py`
- Filter cycles early in placement to only those containing all required
nodes (subset matching)
- Simplified `api.py` by removing the subgraph topology filtering and
passing `required_nodes` directly to placement
- Renamed internal `node_ids` variable to `placement_node_ids` to avoid
shadowing the parameter
## Why It Works
By filtering cycles at the placement level using
`required_nodes.issubset(cycle.node_ids)`, we ensure that only cycles
containing all the user-selected nodes are considered. This happens
early in the placement algorithm, so we don't waste time computing
placements that would be filtered out later.
## Test Plan
### Manual Testing
- Select nodes in the dashboard topology view
- Verify that placements shown include all selected nodes (but may
include additional nodes)
- Verify that placements not containing the selected nodes are filtered
out
### Automated Testing
- Existing placement tests pass
- `uv run pytest src/exo/master/tests/ -v` - 37 tests pass
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Packet storms occur with Thunderbolt bridge enabled on 2 machines
connected by Thunderbolt, not just 3+ node cycles as previously assumed.
The cycle detection was too conservative and missed this case.
## Changes
- Changed the minimum cycle length from >2 (3+ nodes) to >=2 (2+ nodes)
- Updated the early return threshold from `< 3` to `< 2` enabled nodes
- Updated docstring to reflect the new behavior
## Why It Works
A Thunderbolt bridge loop between just 2 machines can still create
broadcast storms when both have the bridge enabled. The previous
threshold of 3+ was based on an incorrect assumption that 2-node
connections wouldn't cause this problem.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
- Tested with 2 machines connected via Thunderbolt with bridge enabled
- Confirmed packet storms occur in this configuration
- Verified the fix correctly detects and handles 2-node cycles
### Automated Testing
- Existing topology tests cover cycle detection logic
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The downloads page showed model download progress but provided no way
for users to trigger downloads or remove completed models from disk.
Added API endpoints (POST /download/start, DELETE /download/{node_id}/{model_id})
that send StartDownload and DeleteDownload commands via the download_command_sender.
Updated the dashboard downloads page with per-model buttons: a download button
for incomplete downloads and a delete button for completed ones.
This allows users to manage downloads directly from the UI without needing
to trigger downloads through other means.
Test plan:
- Deployed on a 3 machine cluster. Did several downloads/deletions - all
work and the dashboard updates relatively fluently. It takes roughly 5
seconds to render a 131GB model deletion which isn't too bad.
The Worker previously owned the ShardDownloader directly via dependency
injection, which prevented --no-worker nodes from downloading and made
it impossible for multiple Workers to share a single downloader instance.
Moved download functionality to a new DownloadCoordinator component at
the Node level that communicates via the DOWNLOAD_COMMANDS pub/sub topic.
Workers now send StartDownload commands instead of calling the downloader
directly, and receive progress updates through the event-sourced state.
This decouples downloads from the Worker lifecycle and enables future
features like UI-triggered downloads to specific nodes and multi-worker
download sharing.
Test plan:
- Mostly tested in the next PR that adds explicit downloads/deletions to
the dashboard.
- Started a model that isn't downloaded - it works.
## Motivation
The 'regenerate' button was hardcoded to chat completion. Clicking
'regenerate' for image request would result in an error after the model
is loaded
## Changes
Store request type and dispatch to appropriate request upon regeneration
## Why It Works
We make sure to repeat the same request type as was performed originally
## Test Plan
### Manual Testing
Checked 'regenerate' works for chat completion, image generation, image
editing
The prettier-svelte formatter depended on the full dashboard build
(dashboardFull), causing the devshell to rebuild whenever any dashboard
source file changed.
Created a deps-only dream2nix derivation (deps.nix) that uses a stub
source containing only package.json, package-lock.json, and minimal
files for vite to succeed. Updated prettier-svelte to use this
derivation instead of dashboardFull.
The stub source is constant unless lockfiles change, so prettier-svelte
and the devshell no longer rebuild when dashboard source files are
modified.
Test plan:
- nix flake check passed
- nix fmt successfully formatted svelte files
## Motivation
Image editing was missing UI controls for quality, output format, and
advanced parameters that text-to-image generation already supported.
## Changes
- Added quality, output_format, and advanced_params to image edit API
endpoints
- Extended isImageModel check to include image editing models
## Why It Works
The API now accepts and forwards these settings for image edits, and the
UI displays the appropriate controls for image editing models.
## Test Plan
### Manual Testing
Verified parameters can be set in UI and that they progagate through to
model inference
## Motivation
Support OpenAI API `n` setting
## Changes
- Users can select `n` to generate more than one image with the same
prompt
- each image uses a different seed -> different results
- `stream` and `partial_images` settings can be overwritten in UI
## Motivation
Fixes tool calling crashes with GLM-4.7-Flash and Kimi-K2 models.
Related: #1254
Two distinct issues were causing crashes:
1. **Tool parser crashes** - The upstream GLM47 and Kimi tool parsers
call `.group()` on regex matches without checking for `None`, causing
`AttributeError` when the model outputs malformed tool calls
2. **Chat template crashes** - GLM's chat template expects
`tool_calls[].function.arguments` to be a dict, but OpenAI format
provides it as a JSON string, causing `'str object' has no attribute
'items'`
## Changes
**`src/exo/worker/runner/runner.py`:**
- Add `patch_glm_tokenizer()` - fixed version of mlx_lm's glm47 parser
with None checks
- Fix `patch_kimi_tokenizer()` - add None checks before calling
`.group()` on regex matches
- Add `ValueError` and `AttributeError` to exception handling in
`parse_tool_calls()`
**`src/exo/worker/engines/mlx/utils_mlx.py`:**
- Add `_normalize_tool_calls()` - parses
`tool_calls[].function.arguments` from JSON string to dict for templates
that expect dicts (like GLM-4.7-Flash)
## Why It Works
1. **Parser fixes**: By checking if regex matches are `None` before
calling `.group()`, we can raise a proper `ValueError` instead of
crashing with `AttributeError`
2. **Template fix**: The GLM-4.7-Flash chat template iterates over
arguments with `.items()`:
```jinja2
{% set _args = tc.arguments %}{% for k, v in _args.items() %}
```
OpenAI format has `arguments` as a JSON string.
`_normalize_tool_calls()` parses this to a dict before passing to the
template.
## Test Plan
### Manual Testing
- Hardware: Mac with GLM-4.7-Flash-4bit model
- Tested tool calling with GLM model - no longer crashes
### Automated Testing
- Existing tests pass (`uv run pytest`)
- Type checking passes (`uv run basedpyright`)
- Linting passes (`uv run ruff check`)
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Motivation
When selecting a model for placement, users often want to see placements
that utilize specific nodes in their cluster. Currently there's no way
to filter the placement previews to focus on configurations that include
particular machines.
## Changes
- **Backend**: Added `node_ids` query parameter to the
`/placement-previews` API endpoint. When provided, the endpoint filters
the topology to only include the specified nodes before generating
placements using the new `Topology.filter_to_nodes()` method.
- **Topology class**: Added `filter_to_nodes(node_ids)` method that
creates a new topology containing only the specified nodes and edges
between them.
- **App store**: Added `previewNodeFilter` state to track selected
nodes, with methods to toggle/clear the filter. Automatically cleans up
filter when nodes are removed from the cluster and re-fetches previews
when topology changes.
- **TopologyGraph component**: Added click handlers to toggle node
filter selection, hover effects to indicate clickable nodes, and visual
styling (yellow highlight for selected, dimmed for filtered-out nodes).
- **Main page**: Added filter indicator in top-right corner of topology
showing active filter count with a clear button.
## Why It Works
The filtering happens at the backend/placement generation level rather
than just filtering the results. This ensures we see all valid placement
combinations for the selected nodes, not just a subset that happened to
be generated for the full topology.
The visual feedback uses the same rendering approach as the existing
highlight system - state is tracked in Svelte and applied during render,
so it persists across data updates without flickering.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Click a node in topology → should show yellow highlight and filter
indicator
- Click another node → indicator shows "2 nodes", previews update to
show only placements using both
- Hover over nodes → subtle yellow highlight indicates they're clickable
- Click X on filter indicator → clears filter, shows all placements
again
- Disconnect a node while it's in filter → filter auto-removes that node
### Automated Testing
- Existing tests cover the Topology class; the new `filter_to_nodes`
method follows the same patterns
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
The previous approach installed a LaunchDaemon plist that ran
periodically to disable Thunderbolt Bridge. This required full admin
privileges upfront and ran regardless of whether a problematic loop
existed.
This change replaces that with dynamic detection - only prompting the
user when an actual TB bridge loop with 3+ machines is detected, and
using fine-grained SCPreferences authorization instead of full admin.
## Changes
**Backend (Python):**
- Added `ThunderboltBridgeStatus` model to track bridge enabled/exists
state per node
- Added `node_thunderbolt_bridge` and `thunderbolt_bridge_cycles` fields
to State
- Added `get_thunderbolt_bridge_cycles()` method to Topology class
- **Robust TB bridge detection:**
- Finds bridge network services from `-listnetworkserviceorder` (not
`-listallhardwareports` which can miss bridges)
- Checks each bridge's member interfaces via `ifconfig` to verify it
contains Thunderbolt interfaces
- Handles varying service names (e.g., "TB Bridge", "Thunderbolt
Bridge", "Bridge (bridge0)")
- Includes `service_name` in status for correct disable commands
- Added warning logs for all error cases in detection
- Updated `apply.py` to handle the new event type and recompute cycles
on node timeout
**Swift App:**
- New `ThunderboltBridgeService` that monitors for cycles from cluster
state
- Shows NSAlert when a cycle with >2 machines is detected
- Uses `SCPreferencesCreateWithAuthorization` with
`system.services.systemconfiguration.network` right for targeted
permissions
- **Auto-cleanup of legacy LaunchDaemon:** On app startup, checks for
and removes old plist/scripts (non-fatal if user cancels)
- **Periodic local network checking:** Re-checks every 10s so the
warning disappears when user grants permission
- **Fixed ClusterState model:** Updated to decode new granular state
fields (`nodeIdentities`, `nodeMemory`, `nodeSystem`,
`nodeThunderboltBridge`) with computed `nodeProfiles` property for
backwards compatibility
- **Fixed Topology model:** Updated to match actual JSON structure where
`nodes` is an array of strings (not objects) and `connections` is a
nested map (not flat array)
- Cleaned up `NetworkSetupHelper` by removing daemon installation code
(now only handles uninstall)
**Dashboard:**
- Added yellow warning badge on topology when TB bridge cycle detected
- On hover: highlights affected nodes in yellow on the topology graph
- Shows which machines are in the cycle with friendly names
- Provides copy-paste terminal command with the correct service name:
```
sudo networksetup -setnetworkserviceenabled "<service-name>" off
```
- Warning appears in all topology views (full, welcome, and minimized
chat sidebar)
- **Debug mode:** Shows "TB:ON" or "TB:OFF" status next to each node in
the topology
## Why It Works
- Cycle detection happens on the backend where we have full topology
information
- Only cycles with 3+ machines are flagged (2-node connections are fine)
- TB bridge detection is robust:
- Uses `-listnetworkserviceorder` to find bridges (works on all machines
tested)
- Verifies bridge membership via `ifconfig` to confirm Thunderbolt
interfaces
- Handles different service names across machines
- The Swift app reacts to detected cycles and prompts the user once per
cycle
- The dashboard provides visual feedback and actionable instructions
- `SCPreferencesCreateWithAuthorization` provides the minimal
permissions needed to modify network service state
- Legacy LaunchDaemon is automatically cleaned up on first launch with
this version
## Test Plan
### Manual Testing
Here EXO detected a TB bridge cycle:
#### Dashboard:
<img width="1363" height="884" alt="Screenshot 2026-01-21 at 10 07
30 PM"
src="https://github.com/user-attachments/assets/7da9c621-0c91-42c4-898e-4952188a1f61"
/>
#### Hovering the warning:
<img width="359" height="279" alt="Screenshot 2026-01-21 at 16 30 57"
src="https://github.com/user-attachments/assets/05501dcf-3d4a-4704-9f38-257748c05a53"
/>
#### macOS app warning popup:
<img width="270" height="410" alt="Screenshot 2026-01-21 at 16 29 08"
src="https://github.com/user-attachments/assets/45714427-08c3-4fb4-9e61-144925c51adf"
/>
### Which then asks for the user's password:
<img width="263" height="372" alt="Screenshot 2026-01-21 at 16 29 28"
src="https://github.com/user-attachments/assets/7502e591-596d-4128-8cf5-6a12674e27bc"
/>
Which when entered, successfully disables bridge and no longer shows the
warning on dashboard.
#### When it fails it shows the error message:
<img width="263" height="234" alt="Screenshot 2026-01-21 at 14 45 38"
src="https://github.com/user-attachments/assets/2d10b3d5-69d7-46ea-b631-d52d8651ab41"
/>
### Automated Testing
- Type checker: 0 errors (`uv run basedpyright`)
- Linter: All checks passed (`uv run ruff check`)
- Tests: 118 passed (`uv run pytest`)
- Dashboard: Builds successfully (`npm run build`)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
black-forest-labs models require hf auth and signup to download. We
don't handle this gracefully.
https://github.com/exo-explore/exo/issues/1242
## Changes
- Handle auth errors
- Surface error to UI and suggest resolution
- Support using HF_TOKEN env variable for auto
- Hide image functionality behind `EXO_ENABLE_IMAGE_MODELS=true` for now
## Why It Works
Users are presented with actionable feedback when issue occurs
## Test Plan
### Manual Testing
Confirmed loading black-forest-labs model in UI presents the issue in
the UI.
Confirmed both `hf auto login` and setting `HF_TOKEN` resolve the issue
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
Image generation feature is not stable and causing issues for users.
Fixes#1242
## Changes
- Commented out image model cards (flux1-schnell, flux1-dev, qwen-image,
qwen-image-edit-2509) in `src/exo/shared/models/model_cards.py`
- Added reference to issue #1242 in the comment explaining why they are
disabled
## Why It Works
By commenting out the model cards, these image models will no longer
appear in the model list, preventing users from attempting to use the
unstable feature until it is stabilized.
## Test Plan
### Manual Testing
- Run exo and verify image models no longer appear in the model list
### Automated Testing
- No changes to automated tests needed - this simply removes models from
the available list
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
The downloads page showed "0B / 0B" for models that haven't started
downloading yet because the download progress data only gets populated
after the file list is fetched from HuggingFace.
Added a fetch to the /models API endpoint on page mount and created a
helper function that falls back to storage_size_megabytes when the
download's totalBytes is 0.
This allows users to see the actual model size (e.g., "0 / 25GB")
before a download begins, which is helpful for a future feature that
lets you download models explicitly.
Test plan:
- Deployed to a cluster, the previous 0B now show sensible values.
## Motivation
When `get_shard_download_status()` runs, it iterates over all models in
`MODEL_CARDS` and calls `build_full_shard()` → `build_base_shard()` →
`ModelCard.from_hf()`. This unconditionally tried to download
`config.json` from HuggingFace, but image models (FLUX, Qwen-Image)
don't have a root-level config.json file, causing errors:
```
Error downloading shard: File not found: https://huggingface.co/black-forest-labs/FLUX.1-dev/resolve/main/config.json
Error downloading shard: File not found: https://huggingface.co/black-forest-labs/FLUX.1-schnell/resolve/main/config.json
Error downloading shard: File not found: https://huggingface.co/Qwen/Qwen-Image/resolve/main/config.json
Error downloading shard: File not found: https://huggingface.co/Qwen/Qwen-Image-Edit-2509/resolve/main/config.json
```
## Changes
### ModelCard.load() fix
- `build_base_shard()` now uses `ModelCard.load()` instead of
`ModelCard.from_hf()`
- `ModelCard.load()` iterates through `MODEL_CARDS.values()` to find a
match by `model_id`
### exo-bench fixes
- Use `name` field instead of `id` for model resolution
- Pass `full_model_id` to `/instance/previews` endpoint
- Make model name matching case-insensitive
- Update README example model name
## Why It Works
`MODEL_CARDS` uses short names as keys (e.g., `"flux1-schnell"`) but the
`model_id` values are HuggingFace paths (e.g.,
`"black-forest-labs/FLUX.1-schnell"`). When `ModelCard.load()` was
called with the HF path, it didn't match any key and fell back to
`from_hf()` which tried to download config.json.
The fix iterates through `MODEL_CARDS.values()` to find a match by
`model_id`, ensuring predefined models (including image models) use
their registry entries directly without network calls. A key lookup is
unnecessary since `load()` is always called with HF paths which don't
match the short-name keys.
## Test Plan
### Manual Testing
- Run exo and verify no more "Error downloading shard: File not found:
.../config.json" errors for image models
- Run exo-bench and verify model resolution works correctly
### Automated Testing
- `uv run basedpyright` - passes with 0 errors
- `uv run pytest` - all tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Way too much spam. Some logs were also obsolete, leading to users
thinking there was something wrong during expected behaviour.
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
Package prettier with Svelte support and add it to treefmt-nix to format
the dashboard. This change is brutal, I spent a long time trying to get
it nicer but it doesn't seem there's a good way to make this minimal.
Sorry for the noise!
This will make it easier for new contributors to get the formatting
right first time.
Also removes the `.prettierrc` because it turns out treefmt-nix was
ignoring it.
Test plan:
- CI
## Motivation
Enable distributed image generation across exo clusters
## Changes
- Added OpenAI-compatible /v1/images/generations and /v1/images/edits
API endpoints - Added /bench/images/generations and /bench/images/edits
endpoints that return generation statistics (timing, throughput metrics)
- Implemented PipeFusion distributed inference for diffusion models,
enabling patch-based parallelism across nodes
- Added model adapters for Flux (schnell, dev) and Qwen image models
## Why It Works
https://arxiv.org/abs/2405.14430
## Test Plan
### Manual Testing
- Generate images using /v1/images/generations endpoint with single and
multi-node clusters
- Test image editing via /v1/images/edits with source images
- Verify streaming partial images appear progressively in the dashboard
- Use /bench/images/generations to measure generation performance
- Test both Flux and Qwen model families
---------
Co-authored-by: Sami Khan <smsak99@gmail.com>
## Motivation
Test failed due to circular import
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
Tried importing and calling the functions, worked fine.
### Automated Testing
Tests pass again
## Motivation
MLX LM has given GPT OSS a shard method, but MLX does not have an update
to match.
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
Kimi K2 Thinking Pipeline RDMA was broken before.
## Why It Works
No clue tbh
## Test Plan
### Manual Testing
Kimi K2 Thinking and GPT OSS work at the same time on Pipeline RDMA.
Needs exo bench to check more thoroughly
### Automated Testing
Layer composition tests still pass.
This change uses the stronger typed ModelId, and introduces some
convenience methods. It also cleans up some code left over from #1204.
## Changes
`model_id: str -> model_id: ModelId`
`repo_id: str -> model_id: ModelId`
Introduces methods on ModelId, in particular ModelId.normalize() to
replace `/` with `--`.
This PR did introduce some circular imports, so has moved some code
around to try and limit them.
## Test Plan
Tests still pass, types still check. As this is about metadata, I
haven't tested inference.
## Motivation
The `download_progress_for_local_path` function and the "Handle local
paths" code block in `download_shard` are dead code that cannot be
reached in normal usage. The code checks if `model_id` (e.g.,
"mlx-community/Llama-3.2-3B-Instruct-4bit") exists as a filesystem path,
but model IDs are constrained to HuggingFace repo format and there's no
API pathway to pass local paths.
## Changes
- Removed `download_progress_for_local_path()` function (45 lines)
- Removed the "Handle local paths" block in `download_shard()` (7 lines)
## Why It Works
This code was added in PR #669 as part of a "feature-local-models"
branch, but the feature was never fully integrated. The check
`aios.path.exists(str(shard.model_card.model_id))` would only return
true if a directory literally named
"mlx-community/Llama-3.2-3B-Instruct-4bit" existed in the cwd, which
doesn't happen in practice. Offline caching is already handled by
`fetch_file_list_with_cache`.
## Test Plan
### Manual Testing
- Run exo normally and verify downloads still work
### Automated Testing
- Existing tests pass (this code had no test coverage)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
When `skip_download=True`, exo was logging a lot of unnecessary messages during periodic download status checks. This resulted in spammy logs that made it hard to see important messages.
## Changes
- Only log "Downloading ... with allow_patterns=..." when actually downloading (not when skip_download is true)
- Changed periodic download progress check logs from INFO to DEBUG level
## Why It Works
The `skip_download=True` parameter is used when checking download status without actually downloading. By guarding the log behind `if not skip_download:`, we avoid logging on every status check. Changing the periodic emitting logs to DEBUG level reduces noise while still keeping them available for debugging.
## Test Plan
### Manual Testing
- Run exo and observe that logs are less spammy during normal operation
- Use -v or -vv flags to see DEBUG logs when needed
### Automated Testing
- Existing tests cover this code path
## Motivation
<!-- Why is this change needed? What problem does it solve? -->
<!-- If it fixes an open issue, please link to the issue here -->
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
We have a lot of unneeded data in the model card - lets just keep the
necessary stuff and add back more data when we need it
## Test Plan
EXO still runs! (pipeline on 2)
Co-authored-by: rltakashige <rl.takashige@gmail.com>
## Motivation
Add support for GLM-4.7-Flash, a lighter variant of GLM-4.7 with the
`glm4_moe_lite` architecture. These models are smaller and faster while
maintaining good performance.
## Changes
1. **Added 4 new model cards** for GLM-4.7-Flash variants:
- `glm-4.7-flash-4bit` (~18 GB)
- `glm-4.7-flash-5bit` (~21 GB)
- `glm-4.7-flash-6bit` (~25 GB)
- `glm-4.7-flash-8bit` (~32 GB)
All variants have:
- `n_layers`: 47 (vs 91 in GLM-4.7)
- `hidden_size`: 2048 (vs 5120 in GLM-4.7)
- `supports_tensor`: True (native `shard()` method)
2. **Bumped mlx from 0.30.1 to 0.30.3** - required by mlx-lm 0.30.4
3. **Updated mlx-lm from 0.30.2 to 0.30.4** - adds `glm4_moe_lite`
architecture support
4. **Added type ignores** in `auto_parallel.py` for stricter type
annotations in new mlx-lm
5. **Fixed EOS token IDs** for GLM-4.7-Flash - uses different tokenizer
with IDs `[154820, 154827, 154829]` vs other GLM models' `[151336,
151329, 151338]`
6. **Renamed `MLX_IBV_DEVICES` to `MLX_JACCL_DEVICES`** - env var name
changed in new mlx
## Why It Works
The model cards follow the same pattern as existing GLM-4.7 models.
Tensor parallel support is enabled because GLM-4.7-Flash implements the
native `shard()` method in mlx-lm 0.30.4, which is automatically
detected in `auto_parallel.py`.
GLM-4.7-Flash uses a new tokenizer with different special token IDs.
Without the correct EOS tokens, generation wouldn't stop properly.
## Test Plan
### Manual Testing
Tested generation with GLM-4.7-Flash-4bit - now correctly stops at EOS
tokens.
### Automated Testing
- `basedpyright`: 0 errors
- `ruff check`: All checks passed
- `pytest`: 162/162 tests pass (excluding pre-existing
`test_distributed_fix.py` timeout failures)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Certain models hang at model loading in tensor parallel.
Hopefully closes#1205
## Changes
- Load layer by layer for tensor parallel sharding
- Move eval_with_timeout to auto_parallel.py to resolve circular import.
## Why It Works
The naive way to fix this is to use load model with lazy = False and
then shard in tensor parallel. However, this requires the entire model
to be loaded into memory.
Instead, we can load layer by layer and shard after loading. There is a
very small memory footprint to this, but it is negligible.
I tried loading layer by layer after the sharding, and this allowed
model loading but got stuck at warming up.
## Test Plan
### Manual Testing
GPT OSS loads with TP and FAST SYNCH. Kimi does too.
### Automated Testing
We need to run a suite of exo_bench before merging this!
## Motivation
For thinking models like GLM-4.7, the `<think>` tag is inserted by the
tokenizer's `apply_chat_template()` into the **prompt** (input). The
model generates tokens starting *after* this tag, so `<think>` never
appears in the streamed output. The frontend expects
`<think>...</think>` tags to extract and display thinking content.
**Log evidence:**
```
[gMASK]<sop><|system|>...<|user|>...<|assistant|><think>
```
The prompt ends with `<think>`, so the model generates content after it,
never returning the opening tag.
## Changes
- Added `detect_thinking_prompt_suffix()` helper function in
`utils_mlx.py` to detect if a prompt ends with `<think>` tag
- Added `parse_thinking_models()` generator wrapper in `runner.py` that
prepends the thinking tag to the output stream
- Modified the main generation loop to use the thinking wrapper for
non-GptOssModel models when a thinking prefix is detected
- Updated test mocks to handle the new `apply_chat_template` call
## Why It Works
The solution follows the same pattern as `parse_gpt_oss()` - a generator
wrapper that transforms the output stream. When the chat template ends
with `<think>`, we prepend this tag to the first generated token so the
frontend receives the complete `<think>...</think>` structure it
expects.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Run exo: `uv run exo`
- Send a chat request to GLM-4.7:
```bash
curl http://localhost:52415/v1/chat/completions -H "Content-Type:
application/json" -d '{
"model": "mlx-community/GLM-4.7-8bit-gs32",
"messages": [{"role": "user", "content": "What is 2+2?"}],
"stream": true
}'
```
- Verify the streamed response starts with `<think>` tag
- Verify the frontend dashboard correctly shows the thinking section
collapsed
### Automated Testing
- All 72 worker tests pass: `uv run pytest src/exo/worker/`
- Type checker passes: `uv run basedpyright`
- Linter passes: `uv run ruff check`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Ryuichi Leo Takashige <leo@exolabs.net>
## Motivation
The current `NodePerformanceProfile` is a monolithic object where every
update (even 1-second memory updates) replaces the entire profile,
touching unrelated data. Different fields update at vastly different
frequencies:
| Data | Update Frequency |
|------|------------------|
| Memory, System | 1 second |
| Thunderbolt | 5 seconds |
| Network interfaces | 10 seconds |
| Friendly name | 60 seconds |
| Model/Chip ID | Once at startup |
## Changes
Split into separate state mappings so each data type updates
independently:
- `node_identities`: Static and slow-changing data (model_id, chip_id,
friendly_name)
- `node_memory`: RAM and swap usage
- `node_system`: GPU usage, temperature, power, CPU metrics
- `node_network`: Network interface information
- `node_thunderbolt`: Thunderbolt interface identifiers
Added a backwards-compatible `node_profiles` property that reconstructs
`NodePerformanceProfile` from the granular mappings for dashboard
compatibility.
**Files modified:**
- `src/exo/shared/types/profiling.py` - Added `NodeIdentity`,
`NodeNetworkInfo`, `NodeThunderboltInfo` types
- `src/exo/shared/types/state.py` - Added 5 new mappings +
`node_profiles` property
- `src/exo/shared/apply.py` - Updated `apply_node_gathered_info` and
`apply_node_timed_out`
## Why It Works
Each info type now writes only to its specific mapping, avoiding
unnecessary updates to unrelated data. The `MacThunderboltConnections`
handler reads from `node_thunderbolt` instead of the old `node_profiles`
for RDMA connection mapping. The backwards-compatible property ensures
the dashboard continues to work unchanged.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
- Start exo and verify dashboard shows node info
- Verify memory/GPU updates stream correctly
- Check that node timeout properly cleans up all mappings
### Automated Testing
- All 162 existing tests pass
- basedpyright: 0 errors
- ruff check: All checks passed
- nix fmt: Applied
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
GPU timeouts often when prompt size > profile_step_size. It also happens
for seemingly random models.
## Changes
Add mx.depends for cache on the logits.
All gather at the model level rather than the layer level, reducing the
amount of data sent.
## Why It Works
mlx_lm's prefill loop only evaluates cache state, not logits.
When prompt > prefill_step_size, the all_gather is never evaluated,
causing GPU timeout.
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
Added failing test cases and then resolved them.
The daemonAlreadyInstalled() function checked that the startup script
file existed and validated plist properties, but did not compare the
actual script content. If the setupScript constant was updated in a new
app version, the stale on-disk script would not be detected or replaced.
Added a guard clause that reads the installed script from disk and
compares it against the expected setupScript content (with whitespace
normalization). When content differs, the function returns false,
triggering the reinstallation flow with an admin privileges prompt.
Test plan:
- Installed on a cluster that had the previous network config. Got the
popup asking for permissions. After accepting I could run Kimi K2
Thinking Tensor RDMA on all 4 nodes.
## Motivation
Information gathering is tightly coupled to MacMon - we should start
generalizing our information sources so we can add more in future.
## Changes
Added a new system to gather any information. Currently, it is attached
to the Worker - though this is mostly to keep the data processing logic
simple. It could be made independent quite easily.
I also refactored topology to include different kinds of connections as
we can gather RDMA connections without having a pre-existing socket
connection, and made the relevant placement updates. We should no longer
need the network locations script in the app.
Other sources of information now include:
- static node information like "model" and "chip" (macos, "Unknown"
fallback)
- device friendly name (macos, falls back to device hostname)
- network interfaces + ips (cross platform)
- thunderbolt interfaces (macos)
- thunderbolt connections (macos)
- RAM usage (cross platform)
- per-device configuration written to EXO_HOME/config.toml
## Limitations
Model and Chip are not cross platform concepts.
We do not differentiate between unified and non-unified memory systems.
A lot of this data collection is based on simple timers. Watching the SC
store on macos is the correct way to gather some of this information,
but requires a detour into rust for macos.
## Why It Works
The InfoGatherer is a generic subsystem which returns a union of metric
datatypes. It writes them to an event, which is applied to state. It is
currently re-spawned with the worker so each cluster receives the
correct information.
As for topology, macOS identifies TB ports with a uuid in
SPThunderboltDataType, and also stores remote uuids if it can find them.
These changes read that data with the system_profiler, hopefully not so
often as to cause notable performance impacts (though this should be
tuned) but frequently enough for moderate responsiveness.
As we can identify TB connections between devices without needing ips
attached to each interface, we can remove the network setup script
(almost) completely.
## Test Plan
### Manual Testing
Spawn RDMA instances without enabling DHCP on the RDMA interfaces.
### Automated Testing
Updated the current master and shared tests to cover the topology
refactor and new events.
---------
Co-authored-by: Sami Khan <smsak99@gmail.com>
Co-authored-by: Alex Cheema <alexcheema123@gmail.com>
Co-authored-by: Jake Hillion <jake@hillion.co.uk>
Updated README.md with documentation for four new features:
- added a "Benchmarking" section documenting the exo-bench tool for
measuring model performance across different placement configurations
- documented the custom namespace feature for cluster isolation in the
macOS app section
- added a "Configuration Options" subsection explaining the --no-worker
CLI flag for coordinator-only nodes
- added a "File Locations (Linux)" subsection documenting XDG Base
Directory Specification compliance on Linux systems
Issue #930
The on_progress callback was synchronous but always invoked from async
contexts, forcing the use of send_nowait() which could raise WouldBlock
if the channel buffer was full, potentially dropping progress updates.
Changed the callback type from `Callable[[ShardMetadata,
RepoDownloadProgress], None]` to return a coroutine, updated all
implementations to be async, and replaced send_nowait() with await
send() in the worker's download progress handler.
This allows proper backpressure handling when sending download progress
events through the channel, eliminating the "Footgun!" that was
previously documented in the code.
Test plan:
- Built a DMG and ran it on one node. All existing models showed as
downloaded.
- Downloaded a new model. The progress bar on the download page worked.
- Downloaded another new model. The progress bar on the home page
worked.
## Motivation
When models output LaTeX-formatted math proofs, the dashboard was not
rendering them correctly. Issues included:
- `\documentclass`, `\begin{document}`, `\usepackage` showing as raw
text
- `$...$` inline math with complex expressions (like `\frac`, `\ldots`)
not rendering due to markdown escaping backslashes
- `\begin{align*}...\end{align*}` and other math environments showing as
raw text
- `\emph{...}`, `\textbf{...}` LaTeX formatting commands not being
converted
- `$\require{...}$` (MathJax-specific) causing KaTeX errors
- `\begin{proof}...\end{proof}` showing as raw text
## Changes
Enhanced `MarkdownContent.svelte` with comprehensive LaTeX support:
**Math extraction before markdown processing:**
- Extract `$...$`, `$$...$$`, `\(...\)`, `\[...\]` into placeholders
before markdown processes the text
- Use alphanumeric placeholders (`MATHPLACEHOLDERINLINE0END`) that won't
be interpreted as HTML tags
- Restore and render with KaTeX after markdown processing
**LaTeX document command removal:**
- Strip `\documentclass{...}`, `\usepackage{...}`, `\begin{document}`,
`\end{document}`
- Strip `\maketitle`, `\title{...}`, `\author{...}`, `\date{...}`
- Strip `\require{...}` (MathJax-specific, not KaTeX)
- Replace `tikzpicture` environments with `[diagram]` placeholder
- Strip `\label{...}` cross-reference commands
**LaTeX math environments:**
- Convert `\begin{align*}`, `\begin{equation}`, `\begin{gather}`, etc.
to display math blocks
**LaTeX text formatting:**
- `\emph{...}` and `\textit{...}` → `<em>...</em>`
- `\textbf{...}` → `<strong>...</strong>`
- `\texttt{...}` → `<code>...</code>`
- `\underline{...}` → `<u>...</u>`
**LaTeX environments styling:**
- `\begin{proof}...\end{proof}` → styled proof block with QED symbol
- `\begin{theorem}`, `\begin{lemma}`, etc. → styled theorem blocks
**Display math enhancements:**
- Wrapped in styled container with subtle gold border
- "LaTeX" label and copy button appear on hover
- Dark theme KaTeX color overrides for better readability
- Custom scrollbar for overflow
## Why It Works
The key insight is that markdown processing was escaping backslashes in
LaTeX before KaTeX could see them. By extracting all math expressions
into alphanumeric placeholders *before* markdown runs, then restoring
them *after*, the LaTeX content passes through to KaTeX unmodified.
Using purely alphanumeric placeholders like `MATHPLACEHOLDERINLINE0END`
instead of `<<MATH_INLINE_0>>` prevents markdown from interpreting them
as HTML tags and stripping them.
## Test Plan
### Manual Testing
- Hardware: Any machine with the dashboard
- What you did:
- Ask model to "write a proof in latex"
- Verify inline math like `$x \in S$` renders correctly
- Verify display math like `\begin{align*}...\end{align*}` renders as
block
- Verify `\documentclass`, `\begin{document}` are stripped (not shown)
- Verify `\emph{...}` converts to italics
- Verify copy button works on display math blocks
- Test edge cases: `$5` (currency) stays as text, `\$50` (escaped)
becomes `$50`
Before:
<img width="799" height="637" alt="Screenshot 2026-01-19 at 11 51 22 AM"
src="https://github.com/user-attachments/assets/62a705b8-b3c2-47b8-afd0-5d0c1b240e44"
/>
After:
<img width="809" height="642" alt="Screenshot 2026-01-19 at 11 46 58 AM"
src="https://github.com/user-attachments/assets/4f35fa1d-333c-4285-bc68-58a50f8f148e"
/>
### Automated Testing
- Dashboard builds successfully with `npm run build`
- Existing functionality preserved
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
With a single pipeline layer, PipelineFirstLayer gets composed with
PipelineLastLayer.
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
### Automated Testing
Made failing tests. Fixed them!
## Motivation
Layer types does not contain either "sliding_attention" or
"full_attention" for pipeline parallel (single layer).
## Changes
<!-- Describe what you changed in detail -->
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
Manually tested single layer of GPT OSS. Doesn't crash
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
Uneven placements were causing issues for some users with lopsided
setups. While fixing, I ran into another issue with impossible
allocation of memory.
## Changes
- Allocate at least 1 layer per device.
- Catch overallocation of memory with an error.
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
Tested that GPT OSS is placed correctly.
### Automated Testing
Added breaking tests in the first commit. Resolved with new placement
algorithm in the second one.
## Motivation
The README showcases exo's features and benchmarks but doesn't show what
the dashboard actually looks like. Adding a screenshot helps users
understand what they'll get when they run exo.
## Changes
- Added dashboard screenshot to `docs/imgs/dashboard-cluster-view.png`:
Shows the cluster topology view with 4 × 512GB M3 Ultra Mac Studio
running DeepSeek v3.1 (8-bit) and Kimi-K2-Thinking (4-bit)
- Added a new "Dashboard" section to README.md below Features,
displaying the screenshot with caption
## Why It Works
Visual documentation helps users understand what exo offers before they
install it. The screenshot demonstrates the cluster management
capabilities.
## Test Plan
### Manual Testing
- Verified image renders correctly in GitHub markdown preview
### Automated Testing
- N/A - documentation only change
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Runners that crash can swallow errors - we should re-raise. Also the
exception handler annoyed me.
## Changes
The try: except in the runner's chat now re-raises.
## Motivation
mp sender occasionally does not have time to flush its events before
collect() is called, making the event ordering test fail.
## Changes
- Replace mp_channel with simple collector for event ordering test
- Also suppress warning for <frozen importlib._bootstrap>:488 <frozen
importlib._bootstrap>:488: DeprecationWarning: builtin type SwigPyObject
has no __module__ attribute
## Why It Works
<!-- Explain why your approach solves the problem -->
## Test Plan
### Manual Testing
<!-- Hardware: (e.g., MacBook Pro M1 Max 32GB, Mac Mini M2 16GB,
connected via Thunderbolt 4) -->
<!-- What you did: -->
<!-- - -->
### Automated Testing
Ran the test 100 times without it failing.
Support HTTPS_PROXY/HTTP_PROXY environment variables for proxy
configuration and SSL_CERT_FILE for custom CA certificates, enabling use
in corporate environments with SSL inspection.
## Motivation
Users in corporate environments often need to route traffic through HTTP
proxies and use custom CA certificates for SSL inspection. Without this
support, exo cannot download models in these network configurations.
## Changes
- Added `HTTPS_PROXY`/`HTTP_PROXY` environment variable support to
`create_http_session()` in `download_utils.py`
- Added `SSL_CERT_FILE` environment variable support for custom CA
certificate bundles, falling back to certifi's default bundle
## Why It Works
- `aiohttp.ClientSession` natively supports the `proxy` parameter for
routing requests through HTTP proxies
- `ssl.create_default_context(cafile=...)` accepts a custom CA bundle
path, allowing corporate CAs to be trusted
- Using environment variables is consistent with the codebase's existing
configuration patterns (e.g., `EXO_HOME`, `HF_ENDPOINT`)
## Test Plan
### Manual Testing
- Set `HTTPS_PROXY` environment variable and verified model downloads
route through proxy
- Set `SSL_CERT_FILE` to custom CA bundle and verified SSL verification
succeeds with corporate SSL inspection
### Automated Testing
- No automated tests added; this change is configuration-only and does
not alter existing behavior when environment variables are unset
## Motivation
CI failures can be avoided by running checks locally before committing.
This adds clear documentation to AGENTS.md so that AI agents (and
humans) know exactly which checks must pass before pushing code.
## Changes
Added a new "Pre-Commit Checks (REQUIRED)" section to AGENTS.md that:
- Lists all 4 required checks (basedpyright, ruff, nix fmt, pytest)
- Provides a one-liner to run all checks in sequence
- Notes that `nix fmt` changes must be staged before committing
- Explains that CI runs `nix flake check` which verifies everything
## Why It Works
Clear documentation prevents CI failures by ensuring contributors run
checks locally first. The one-liner command makes it easy to run all
checks before committing.
## Test Plan
### Manual Testing
- Verified the documented commands work correctly
### Automated Testing
- N/A - documentation only change
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
- Error chunks
- Use error handling in exo_bench.py
## Motivation
Return when an error occurs so that generation stops. Adding timeouts is
a separate TODO for model loading and chat completions.
## Changes
- Return HTTP exceptions as JSON responses in an OpenAI compatible
format.
- Context manager for generation to catch and return error messages.
- Use error handling in exo_bench.py.
## Test Plan
### Manual Testing
Manually tested that exo_bench returns on failures within and outside
generation
### Automated Testing
<!-- Describe changes to automated tests, or how existing tests cover
this change -->
<!-- - -->
## Motivation
Previous fix still failed in CI. Suspecting permissions issue with
GITHUB_TOKEN not being able to see draft releases via API.
## Changes
1. Add explicit `permissions: contents: write` to the job
2. Use `gh release list` first to check if draft exists (this uses a
different code path that might work better)
3. Add debug echo statements
## Test Plan
Delete v1.0.63 tag and re-push after merging.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Fixes the draft release detection that failed on the v1.0.63 release
attempt.
## Changes
The jq query was piped to `head -1` which truncated multi-line JSON
output to just `{`, causing the empty check to fail.
Changed to use `first // empty` in jq instead.
## Test Plan
Tested locally:
```bash
GITHUB_REF_NAME="v1.0.63"
gh api repos/exo-explore/exo/releases --jq "[.[] | select(.draft == true) | select(.name == \"$GITHUB_REF_NAME\")] | first // empty"
# Returns the full draft release JSON (2711 chars)
```
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
## Motivation
Closes#1140
Currently releases are uploaded to S3 for Sparkle updates but there's no
GitHub Release created, and Sparkle update dialogs don't show release
notes. Users have no visibility into what changed.
## Changes
- Added release workflow documentation comment at top of `build-app.yml`
- Added "Fetch release notes for Sparkle" step that converts markdown
from draft GitHub release to HTML
- Added "Inject release notes into appcast" step that embeds HTML in
appcast.xml with CDATA
- Added "Publish GitHub Release" step that attaches DMG and publishes
the draft
## Why It Works
- Sparkle's `<description>` tag supports HTML wrapped in CDATA for
rendering in update dialogs
- GitHub's markdown API (`/markdown`) converts the release notes to HTML
with proper formatting
- Draft releases allow writing polished notes before the build, then the
workflow publishes them automatically
- The workflow fails if no draft release exists, ensuring release notes
are always provided
## Test Plan
### Manual Testing
1. Create a draft GitHub release for a new tag with markdown release
notes
2. Push the tag to trigger the workflow
3. Verify the GitHub release is published with DMG attached
4. Download appcast.xml from S3 and verify
`<description><![CDATA[...]]></description>` contains HTML
5. Test Sparkle update dialog on macOS to confirm release notes appear
### Automated Testing
No automated tests added - this is CI workflow configuration.
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-16 16:47:33 +00:00
695 changed files with 80212 additions and 16300 deletions
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
dmcc73
