Add to nix flake

## Motivation

When a model download fails repeatedly (e.g. `ContentLengthError` on a
large model like `zai-org/GLM-5`), the download coordinator accumulates
duplicate progress callbacks — one per retry cycle. Each callback
independently throttles at 1 event/sec, so after N retries, every
download progress tick generates N events instead of 1. After an hour of
failures (~60 retry cycles), this produces ~60 `NodeDownloadProgress`
events/sec, overwhelming the master, delaying heartbeats, and causing
the node to time itself out.

### The callback accumulation cycle
1. `_start_download_task()` calls
`shard_downloader.on_progress(callback)` which **appends** to a list
2. Download fails → `DownloadFailed` status set, but old callback stays
in the list
3. 60s later: `_emit_existing_download_progress()` scans disk → resets
status to `DownloadPending`
4. Worker sends new `StartDownload` → coordinator accepts (guard didn't
check `DownloadFailed`)
5. `_start_download_task()` appends **another** callback
6. Each callback has its own throttle → N callbacks = N events per
progress tick

## Changes

### Commit 1: `src/exo/worker/main.py`
Move the `DownloadModel` backoff check **before** `TaskCreated` emission
in `plan_step()`. Previously `TaskCreated` was emitted unconditionally
every 0.1s even when backoff blocked the download command.

### Commit 2: `src/exo/download/coordinator.py`
1. **Register progress callback once** in `__post_init__` instead of
per-download in `_start_download_task()`. Uses a per-model throttle dict
instead of per-callback closure variables.
2. **Add `DownloadFailed` to the `_start_download()` guard** so
redundant `_start_download_task()` calls don't happen. Retries still
work because `_emit_existing_download_progress` resets `DownloadFailed`
→ `DownloadPending` by scanning disk every 60s.

## Why It Works

The root cause was callbacks accumulating in
`ResumableShardDownloader.on_progress_callbacks` (a list that only
appends, never clears). By registering one callback per coordinator
lifetime and guarding against re-entry on `DownloadFailed`, we ensure
exactly one progress event per model per progress tick regardless of how
many retry cycles have occurred.

## Test Plan

### Manual Testing
- Verified the download retry flow: failed download → 60s scan resets
status → new `StartDownload` accepted → download retries with single
callback

### Automated Testing
- `uv run basedpyright` — 0 errors
- `uv run ruff check` — passes
- `uv run pytest` — 188 passed

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

.github/workflows/build-app.yml

@@ -396,7 +396,7 @@ jobs:
           path: output/EXO-${{ env.RELEASE_VERSION }}.dmg
 
       - name: Upload to S3
-        if: env.SPARKLE_S3_BUCKET != '' && github.ref_type == 'tag'
+        if: env.SPARKLE_S3_BUCKET != ''
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -412,6 +412,12 @@ jobs:
             PREFIX="${PREFIX}/"
           fi
           DMG_NAME="EXO-${RELEASE_VERSION}.dmg"
+
+          if [[ "${{ github.ref_type }}" != "tag" ]]; then
+            aws s3 cp "$DMG_NAME" "s3://${SPARKLE_S3_BUCKET}/${PREFIX}EXO-${GITHUB_SHA}.dmg"
+            exit 0
+          fi
+
           aws s3 cp "$DMG_NAME" "s3://${SPARKLE_S3_BUCKET}/${PREFIX}${DMG_NAME}"
           if [[ "$IS_ALPHA" != "true" ]]; then
             aws s3 cp "$DMG_NAME" "s3://${SPARKLE_S3_BUCKET}/${PREFIX}EXO-latest.dmg"

.mlx_typings/mlx/core/__init__.pyi

@@ -2366,7 +2366,7 @@ class custom_function:
 def default_device() -> Device:
     """Get the default device."""
 
-def default_stream(device: Device) -> Stream:
+def default_stream(device: Device | DeviceType) -> Stream:
     """Get the device's default stream."""
 
 def degrees(a: array, /, *, stream: Stream | Device | None = ...) -> array:

AGENTS.md

@@ -119,3 +119,78 @@ From .cursorrules:
 ## Testing
 
 Tests use pytest-asyncio with `asyncio_mode = "auto"`. Tests are in `tests/` subdirectories alongside the code they test. The `EXO_TESTS=1` env var is set during tests.
+
+## Dashboard UI Testing & Screenshots
+
+### Building and Running the Dashboard
+```bash
+# Build the dashboard (must be done before running exo)
+cd dashboard && npm install && npm run build && cd ..
+
+# Start exo (serves the dashboard at http://localhost:52415)
+uv run exo &
+sleep 8  # Wait for server to start
+```
+
+### Taking Headless Screenshots with Playwright
+Use Playwright with headless Chromium for programmatic screenshots — no manual browser interaction needed.
+
+**Setup (one-time):**
+```bash
+npx --yes playwright install chromium
+cd /tmp && npm init -y && npm install playwright
+```
+
+**Taking screenshots:**
+```javascript
+// Run from /tmp where playwright is installed: cd /tmp && node -e "..."
+const { chromium } = require('playwright');
+(async () => {
+  const browser = await chromium.launch({ headless: true });
+  const page = await browser.newPage({ viewport: { width: 1280, height: 800 } });
+  await page.goto('http://localhost:52415', { waitUntil: 'networkidle' });
+  await page.waitForTimeout(2000);
+
+  // Inject test data into localStorage if needed (e.g., recent models)
+  await page.evaluate(() => {
+    localStorage.setItem('exo-recent-models', JSON.stringify([
+      { modelId: 'mlx-community/Qwen3-30B-A3B-4bit', launchedAt: Date.now() },
+    ]));
+  });
+  await page.reload({ waitUntil: 'networkidle' });
+  await page.waitForTimeout(2000);
+
+  // Interact with UI elements
+  await page.locator('text=SELECT MODEL').click();
+  await page.waitForTimeout(1000);
+
+  // Take screenshot
+  await page.screenshot({ path: '/tmp/screenshot.png', fullPage: false });
+  await browser.close();
+})();
+```
+
+### Uploading Images to GitHub PRs
+GitHub's API doesn't support direct image upload for PR comments. Workaround:
+
+1. **Commit images to the branch** (temporarily):
+   ```bash
+   cp /tmp/screenshot.png .
+   git add screenshot.png
+   git commit -m "temp: add screenshots for PR"
+   git push origin <branch>
+   COMMIT_SHA=$(git rev-parse HEAD)
+   ```
+
+2. **Post PR comment** referencing the raw image URL (uses permanent commit SHA so images survive deletion):
+   ```bash
+   gh pr comment <PR_NUMBER> --body "![Screenshot](https://raw.githubusercontent.com/exo-explore/exo/${COMMIT_SHA}/screenshot.png)"
+   ```
+
+3. **Remove the images** from the branch:
+   ```bash
+   git rm screenshot.png
+   git commit -m "chore: remove temporary screenshot files"
+   git push origin <branch>
+   ```
+   The images still render in the PR comment because they reference the permanent commit SHA.

Cargo.lock

@@ -353,31 +353,6 @@ dependencies = [
  "generic-array",
 ]
 
-[[package]]
-name = "bon"
-version = "3.8.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "ebeb9aaf9329dff6ceb65c689ca3db33dbf15f324909c60e4e5eef5701ce31b1"
-dependencies = [
- "bon-macros",
- "rustversion",
-]
-
-[[package]]
-name = "bon-macros"
-version = "3.8.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "77e9d642a7e3a318e37c2c9427b5a6a48aa1ad55dcd986f3034ab2239045a645"
-dependencies = [
- "darling",
- "ident_case",
- "prettyplease",
- "proc-macro2",
- "quote",
- "rustversion",
- "syn 2.0.111",
-]
-
 [[package]]
 name = "bs58"
 version = "0.5.1"
@@ -700,41 +675,6 @@ dependencies = [
  "syn 2.0.111",
 ]
 
-[[package]]
-name = "darling"
-version = "0.21.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9cdf337090841a411e2a7f3deb9187445851f91b309c0c0a29e05f74a00a48c0"
-dependencies = [
- "darling_core",
- "darling_macro",
-]
-
-[[package]]
-name = "darling_core"
-version = "0.21.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "1247195ecd7e3c85f83c8d2a366e4210d588e802133e1e355180a9870b517ea4"
-dependencies = [
- "fnv",
- "ident_case",
- "proc-macro2",
- "quote",
- "strsim",
- "syn 2.0.111",
-]
-
-[[package]]
-name = "darling_macro"
-version = "0.21.3"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "d38308df82d1080de0afee5d069fa14b0326a88c14f15c5ccda35b4a6c414c81"
-dependencies = [
- "darling_core",
- "quote",
- "syn 2.0.111",
-]
-
 [[package]]
 name = "data-encoding"
 version = "2.9.0"
@@ -1625,12 +1565,6 @@ dependencies = [
  "zerovec",
 ]
 
-[[package]]
-name = "ident_case"
-version = "1.0.1"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "b9e0384b61958566e926dc50660321d12159025e767c18e043daf26b70104c39"
-
 [[package]]
 name = "idna"
 version = "1.1.0"
@@ -1745,15 +1679,6 @@ dependencies = [
  "generic-array",
 ]
 
-[[package]]
-name = "internment"
-version = "0.8.6"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "636d4b0f6a39fd684effe2a73f5310df16a3fa7954c26d36833e98f44d1977a2"
-dependencies = [
- "hashbrown 0.15.5",
-]
-
 [[package]]
 name = "inventory"
 version = "0.3.21"
@@ -3307,16 +3232,6 @@ dependencies = [
  "zerocopy",
 ]
 
-[[package]]
-name = "prettyplease"
-version = "0.2.37"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b"
-dependencies = [
- "proc-macro2",
- "syn 2.0.111",
-]
-
 [[package]]
 name = "primeorder"
 version = "0.13.6"
@@ -3740,12 +3655,6 @@ dependencies = [
  "yasna",
 ]
 
-[[package]]
-name = "recursion"
-version = "0.5.4"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "9dba2197bf7b1d87b4dd460c195f4edeb45a94e82e8054f8d5f317c1f0e93ca1"
-
 [[package]]
 name = "redox_syscall"
 version = "0.5.18"
@@ -4807,15 +4716,6 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
 [[package]]
 name = "util"
 version = "0.0.1"
-dependencies = [
- "bon",
- "derive_more",
- "extend",
- "internment",
- "once_cell",
- "recursion",
- "thiserror 2.0.17",
-]
 
 [[package]]
 name = "uuid"

README.md

@@ -72,16 +72,23 @@ There are two ways to run exo:
 
 ### Run from Source (macOS)
 
+If you have [Nix](https://nixos.org/) installed, you can skip most of the steps below and run exo directly (after accepting the Cachix cache):
+
+```bash
+nix run .#exo
+```
+
 **Prerequisites:**
+- [Xcode](https://developer.apple.com/xcode/) (provides the Metal ToolChain required for MLX compilation)
 - [brew](https://github.com/Homebrew/brew) (for simple package management on macOS)
-  
+
   ```bash
   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
   ```
 - [uv](https://github.com/astral-sh/uv) (for Python dependency management)
 - [macmon](https://github.com/vladkens/macmon) (for hardware monitoring on Apple Silicon)
 - [node](https://github.com/nodejs/node) (for building the dashboard)
-  
+
   ```bash
   brew install uv macmon node
   ```

TODO.md

@@ -1,28 +1,14 @@
-2. Currently a lot of requests from the API are timing out, but we still process those requests internally. If an API request times out, we should cancel all corresponding tasks to that API request (why process a request with nobody listening).
 3. Task cancellation. When API http request gets cancelled, it should cancel corresponding task.
 4. I'd like to see profiled network latency / bandwidth.
 5. I'd like to see how much bandwidth each link is using.
-6. We should handle the case where one machine doesn't have the model downloaded and then other machines are waiting on it. In this case we get loads of timeout errors because the others are waiting for the one that needs to download the model.
 7. Solve the problem of in continuous batching when a new prompt comes in, it will block decode of the current batch until the prefill is complete.
 8. We want people to be able to copy models over to a new device without ever connecting EXO to the internet. Right now EXO require internet connection once to cache some files to check if a download is complete. Instead, we should simply check if there is a non-empty model folder locally with no .partial files. This indicates it's a fully downloaded model that can be loaded.
-10. More granular control over how to deploy instances.
-12. Nix is great but installing it is a pain and we have ended up in a lot of cases having PATH issues or installation issues. For example, after rebooting mike it seemed to no longer have a nix installation and needed reinstalling. It has a bunch of broken symlinks left over from nix that caused ssh to fail, making it even harder to debug. We need consistent environments (perhaps MDM) so we can guarantee nix is installed properly on each machine.
 13. Memory pressure instead of memory used.
 14. Show the type of each connection (TB5, Ethernet, etc.) in the UI. Refer to old exo: https://github.com/exo-explore/exo/blob/56f783b38dc6b08ce606b07a5386dc40dae00330/exo/helpers.py#L251
 15. Prioritise certain connection types (or by latency). TB5 > Ethernet > WiFi. Refer to old exo: https://github.com/exo-explore/exo/blob/56f783b38dc6b08ce606b07a5386dc40dae00330/exo/helpers.py#L251
 16. Dynamically switch to higher priority connection when it becomes available. Probably bring back InstanceReplacedAtomically.
 17. Faster model loads by streaming model from other devices in cluster.
 18. Add support for specifying the type of network connection to use in a test. Depends on 15/16.
-20. Add chat completion cancellations (e.g OpenWebUI has something for cancelling an ongoing request).
-23. Do we need cache_limit? We went back and forth on that a lot because we thought it might be causing issues. One problem is it sets it relative to model size. So if you have multiple models loaded in it will take the most recent model size for the cache_limit. This is problematic if you launch DeepSeek -> Llama for example.
-24. further openai/lmstudio api compatibility
 25. Rethink retry logic
-26. Task cancellation. When API http request gets cancelled, it should cancel corresponding task.
 27. Log cleanup - per-module log filters and default to DEBUG log levels
 28. Validate RDMA connections with ibv_devinfo in the info gatherer
-
-Potential refactors:
-
-2. Topology can be simplified
-
-Random errors we've run into:

app/EXO/EXO/ContentView.swift

@@ -563,21 +563,45 @@ struct ContentView: View {
     }
 
     private var rdmaStatusView: some View {
-        let rdma = networkStatusService.status.rdmaStatus
+        let rdmaStatuses = stateService.latestSnapshot?.nodeRdmaCtl ?? [:]
+        let localNodeId = stateService.localNodeId
+        let nodeProfiles = stateService.latestSnapshot?.nodeProfiles ?? [:]
+        let localDevices = networkStatusService.status.localRdmaDevices
+        let localPorts = networkStatusService.status.localRdmaActivePorts
+
         return VStack(alignment: .leading, spacing: 1) {
-            Text("RDMA: \(rdmaStatusText(rdma))")
-                .font(.caption2)
-                .foregroundColor(rdmaStatusColor(rdma))
-            if !rdma.devices.isEmpty {
-                Text("  Devices: \(rdma.devices.joined(separator: ", "))")
+            if rdmaStatuses.isEmpty {
+                Text("Cluster RDMA: No data")
+                    .font(.caption2)
+                    .foregroundColor(.secondary)
+            } else {
+                Text("Cluster RDMA Status:")
+                    .font(.caption2)
+                    .foregroundColor(.secondary)
+                ForEach(Array(rdmaStatuses.keys.sorted()), id: \.self) { nodeId in
+                    if let status = rdmaStatuses[nodeId] {
+                        let nodeName =
+                            nodeProfiles[nodeId]?.friendlyName ?? String(nodeId.prefix(8))
+                        let isLocal = nodeId == localNodeId
+                        let prefix = isLocal ? "  \(nodeName) (local):" : "  \(nodeName):"
+                        let statusText = status.enabled ? "Enabled" : "Disabled"
+                        let color: Color = status.enabled ? .green : .orange
+                        Text("\(prefix) \(statusText)")
+                            .font(.caption2)
+                            .foregroundColor(color)
+                    }
+                }
+            }
+            if !localDevices.isEmpty {
+                Text("  Local Devices: \(localDevices.joined(separator: ", "))")
                     .font(.caption2)
                     .foregroundColor(.secondary)
             }
-            if !rdma.activePorts.isEmpty {
-                Text("  Active Ports:")
+            if !localPorts.isEmpty {
+                Text("  Local Active Ports:")
                     .font(.caption2)
                     .foregroundColor(.secondary)
-                ForEach(rdma.activePorts, id: \.device) { port in
+                ForEach(localPorts, id: \.device) { port in
                     Text("    \(port.device) port \(port.port): \(port.state)")
                         .font(.caption2)
                         .foregroundColor(.green)
@@ -586,28 +610,6 @@ struct ContentView: View {
         }
     }
 
-    private func rdmaStatusText(_ rdma: RDMAStatus) -> String {
-        switch rdma.rdmaCtlEnabled {
-        case .some(true):
-            return "Enabled"
-        case .some(false):
-            return "Disabled"
-        case nil:
-            return rdma.devices.isEmpty ? "Not Available" : "Available"
-        }
-    }
-
-    private func rdmaStatusColor(_ rdma: RDMAStatus) -> Color {
-        switch rdma.rdmaCtlEnabled {
-        case .some(true):
-            return .green
-        case .some(false):
-            return .orange
-        case nil:
-            return rdma.devices.isEmpty ? .secondary : .green
-        }
-    }
-
     private var sendBugReportButton: some View {
         VStack(alignment: .leading, spacing: 4) {
             Button {

app/EXO/EXO/ExoProcessController.swift

@@ -28,6 +28,10 @@ final class ExoProcessController: ObservableObject {
         }
     }
 
+    static let exoDirectoryURL: URL = {
+        URL(fileURLWithPath: NSHomeDirectory()).appendingPathComponent(".exo")
+    }()
+
     @Published private(set) var status: Status = .stopped
     @Published private(set) var lastError: String?
     @Published private(set) var launchCountdownSeconds: Int?
@@ -78,7 +82,11 @@ final class ExoProcessController: ObservableObject {
 
             let child = Process()
             child.executableURL = executableURL
-            child.currentDirectoryURL = runtimeURL
+            let exoHomeURL = Self.exoDirectoryURL
+            try? FileManager.default.createDirectory(
+                at: exoHomeURL, withIntermediateDirectories: true
+            )
+            child.currentDirectoryURL = exoHomeURL
             child.environment = makeEnvironment(for: runtimeURL)
 
             child.standardOutput = FileHandle.nullDevice

app/EXO/EXO/Models/ClusterState.swift

@@ -15,6 +15,7 @@ struct ClusterState: Decodable {
     let nodeMemory: [String: MemoryInfo]
     let nodeSystem: [String: SystemInfo]
     let nodeThunderboltBridge: [String: ThunderboltBridgeStatus]
+    let nodeRdmaCtl: [String: NodeRdmaCtlStatus]
 
     /// Computed property for backwards compatibility - merges granular state into NodeProfile
     var nodeProfiles: [String: NodeProfile] {
@@ -65,6 +66,10 @@ struct ClusterState: Decodable {
             try container.decodeIfPresent(
                 [String: ThunderboltBridgeStatus].self, forKey: .nodeThunderboltBridge
             ) ?? [:]
+        self.nodeRdmaCtl =
+            try container.decodeIfPresent(
+                [String: NodeRdmaCtlStatus].self, forKey: .nodeRdmaCtl
+            ) ?? [:]
     }
 
     private enum CodingKeys: String, CodingKey {
@@ -78,6 +83,7 @@ struct ClusterState: Decodable {
         case nodeMemory
         case nodeSystem
         case nodeThunderboltBridge
+        case nodeRdmaCtl
     }
 }
 
@@ -159,6 +165,10 @@ struct ThunderboltBridgeStatus: Decodable {
     let serviceName: String?
 }
 
+struct NodeRdmaCtlStatus: Decodable {
+    let enabled: Bool
+}
+
 struct MemoryInfo: Decodable {
     let ramTotal: MemoryValue?
     let ramAvailable: MemoryValue?

app/EXO/EXO/Services/BugReportService.swift

@@ -44,16 +44,12 @@ struct BugReportService {
         let dayPrefix = Self.dayPrefixString(now)
         let prefix = "reports/\(dayPrefix)/\(timestamp)/"
 
-        let logData = readLog()
+        let logFiles = readAllLogs()
         let ifconfigText = try await captureIfconfig()
         let hostName = Host.current().localizedName ?? "unknown"
         let debugInfo = readDebugInfo()
 
-        async let stateResult = fetch(url: baseURL.appendingPathComponent("state"))
-        async let eventsResult = fetch(url: baseURL.appendingPathComponent("events"))
-
-        let stateData = try await stateResult
-        let eventsData = try await eventsResult
+        let stateData = try await fetch(url: baseURL.appendingPathComponent("state"))
 
         // Extract cluster TB bridge status from exo state
         let clusterTbBridgeStatus = extractClusterTbBridgeStatus(from: stateData)
@@ -67,12 +63,19 @@ struct BugReportService {
             clusterTbBridgeStatus: clusterTbBridgeStatus
         )
 
-        let uploads: [(path: String, data: Data?)] = [
-            ("\(prefix)exo.log", logData),
+        let eventLogFiles = readAllEventLogs()
+
+        var uploads: [(path: String, data: Data?)] = logFiles.map { (path, data) in
+            ("\(prefix)\(path)", data)
+        }
+        uploads.append(
+            contentsOf: eventLogFiles.map { (path, data) in
+                ("\(prefix)\(path)", data as Data?)
+            })
+        uploads.append(contentsOf: [
             ("\(prefix)state.json", stateData),
-            ("\(prefix)events.json", eventsData),
             ("\(prefix)report.json", reportJSON),
-        ]
+        ])
 
         let uploadItems: [(key: String, body: Data)] = uploads.compactMap { item in
             guard let body = item.data else { return nil }
@@ -149,11 +152,40 @@ struct BugReportService {
         return decoded.urls
     }
 
-    private func readLog() -> Data? {
-        let logURL = URL(fileURLWithPath: NSHomeDirectory())
+    private func readAllLogs() -> [(path: String, data: Data)] {
+        let dir = URL(fileURLWithPath: NSHomeDirectory())
             .appendingPathComponent(".exo")
-            .appendingPathComponent("exo.log")
-        return try? Data(contentsOf: logURL)
+            .appendingPathComponent("exo_log")
+        var results: [(path: String, data: Data)] = []
+
+        let contents = (try? FileManager.default.contentsOfDirectory(atPath: dir.path)) ?? []
+        for name in contents {
+            if let data = try? Data(contentsOf: dir.appendingPathComponent(name)) {
+                results.append(("exo_log/\(name)", data))
+            }
+        }
+
+        return results
+    }
+
+    private func readAllEventLogs() -> [(path: String, data: Data)] {
+        let eventLogDir = URL(fileURLWithPath: NSHomeDirectory())
+            .appendingPathComponent(".exo")
+            .appendingPathComponent("event_log")
+        var results: [(path: String, data: Data)] = []
+
+        for subdir in ["master", "api"] {
+            let dir = eventLogDir.appendingPathComponent(subdir)
+            let contents =
+                (try? FileManager.default.contentsOfDirectory(atPath: dir.path)) ?? []
+            for name in contents where name.hasPrefix("events.") {
+                if let data = try? Data(contentsOf: dir.appendingPathComponent(name)) {
+                    results.append(("event_log/\(subdir)/\(name)", data))
+                }
+            }
+        }
+
+        return results
     }
 
     private func captureIfconfig() async throws -> String {

app/EXO/EXO/Services/NetworkStatusService.swift

@@ -35,28 +35,18 @@ struct NetworkStatus: Equatable {
     let thunderboltBridgeState: ThunderboltState?
     let bridgeInactive: Bool?
     let interfaceStatuses: [InterfaceIpStatus]
-    let rdmaStatus: RDMAStatus
+    let localRdmaDevices: [String]
+    let localRdmaActivePorts: [RDMAPort]
 
     static let empty = NetworkStatus(
         thunderboltBridgeState: nil,
         bridgeInactive: nil,
         interfaceStatuses: [],
-        rdmaStatus: .empty
+        localRdmaDevices: [],
+        localRdmaActivePorts: []
     )
 }
 
-struct RDMAStatus: Equatable {
-    let rdmaCtlEnabled: Bool?
-    let devices: [String]
-    let activePorts: [RDMAPort]
-
-    var isAvailable: Bool {
-        rdmaCtlEnabled == true || !devices.isEmpty
-    }
-
-    static let empty = RDMAStatus(rdmaCtlEnabled: nil, devices: [], activePorts: [])
-}
-
 struct RDMAPort: Equatable {
     let device: String
     let port: String
@@ -80,31 +70,11 @@ private struct NetworkStatusFetcher {
             thunderboltBridgeState: readThunderboltBridgeState(),
             bridgeInactive: readBridgeInactive(),
             interfaceStatuses: readInterfaceStatuses(),
-            rdmaStatus: readRDMAStatus()
+            localRdmaDevices: readRDMADevices(),
+            localRdmaActivePorts: readRDMAActivePorts()
         )
     }
 
-    private func readRDMAStatus() -> RDMAStatus {
-        let rdmaCtlEnabled = readRDMACtlEnabled()
-        let devices = readRDMADevices()
-        let activePorts = readRDMAActivePorts()
-        return RDMAStatus(
-            rdmaCtlEnabled: rdmaCtlEnabled, devices: devices, activePorts: activePorts)
-    }
-
-    private func readRDMACtlEnabled() -> Bool? {
-        let result = runCommand(["rdma_ctl", "status"])
-        guard result.exitCode == 0 else { return nil }
-        let output = result.output.lowercased().trimmingCharacters(in: .whitespacesAndNewlines)
-        if output.contains("enabled") {
-            return true
-        }
-        if output.contains("disabled") {
-            return false
-        }
-        return nil
-    }
-
     private func readRDMADevices() -> [String] {
         let result = runCommand(["ibv_devices"])
         guard result.exitCode == 0 else { return [] }

bench/pyproject.toml

@@ -0,0 +1,16 @@
+[project]
+name = "exo-bench"
+version = "0.1.0"
+description = "Benchmarking tool for exo distributed inference"
+requires-python = ">=3.13"
+dependencies = [
+    "loguru>=0.7.3",
+    "transformers>=5.0.0",
+    "huggingface-hub>=0.33.4",
+    "tiktoken>=0.12.0",
+    "jinja2>=3.1.0",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"

dashboard/src/lib/components/ChatForm.svelte

@@ -599,9 +599,8 @@
             : isImageModel()
               ? "Describe the image you want to generate..."
               : placeholder}
-        disabled={loading}
         rows={1}
-        class="flex-1 resize-none bg-transparent text-foreground placeholder:text-exo-light-gray/60 placeholder:text-sm placeholder:tracking-[0.15em] placeholder:leading-7 focus:outline-none focus:ring-0 focus:border-none disabled:opacity-50 text-sm leading-7 font-mono"
+        class="flex-1 resize-none bg-transparent text-foreground placeholder:text-exo-light-gray/60 placeholder:text-sm placeholder:tracking-[0.15em] placeholder:leading-7 focus:outline-none focus:ring-0 focus:border-none text-sm leading-7 font-mono"
         style="min-height: 28px; max-height: 150px;"
       ></textarea>
 

dashboard/src/lib/components/ChatMessages.svelte

@@ -13,6 +13,7 @@
   import type { MessageAttachment } from "$lib/stores/app.svelte";
   import MarkdownContent from "./MarkdownContent.svelte";
   import TokenHeatmap from "./TokenHeatmap.svelte";
+  import ImageLightbox from "./ImageLightbox.svelte";
 
   interface Props {
     class?: string;
@@ -101,6 +102,9 @@
   let copiedMessageId = $state<string | null>(null);
   let expandedThinkingMessageIds = $state<Set<string>>(new Set());
 
+  // Lightbox state
+  let expandedImageSrc = $state<string | null>(null);
+
   // Uncertainty heatmap toggle
   let heatmapMessageIds = $state<Set<string>>(new Set());
 
@@ -389,10 +393,15 @@
                         class="flex items-center gap-2 bg-exo-dark-gray/60 border border-exo-yellow/20 rounded px-2 py-1 text-xs font-mono"
                       >
                         {#if attachment.type === "image" && attachment.preview}
+                          <!-- svelte-ignore a11y_no_noninteractive_element_interactions, a11y_click_events_have_key_events -->
                           <img
                             src={attachment.preview}
                             alt={attachment.name}
-                            class="w-12 h-12 object-cover rounded border border-exo-yellow/20"
+                            class="w-12 h-12 object-cover rounded border border-exo-yellow/20 cursor-pointer hover:border-exo-yellow/50 transition-colors"
+                            onclick={() => {
+                              if (attachment.preview)
+                                expandedImageSrc = attachment.preview;
+                            }}
                           />
                         {:else}
                           <span>{getAttachmentIcon(attachment)}</span>
@@ -466,15 +475,44 @@
                   <div class="mb-3">
                     {#each message.attachments.filter((a) => a.type === "generated-image") as attachment}
                       <div class="relative group/img inline-block">
+                        <!-- svelte-ignore a11y_no_noninteractive_element_interactions, a11y_click_events_have_key_events -->
                         <img
                           src={attachment.preview}
                           alt=""
-                          class="max-w-full max-h-[512px] rounded-lg border border-exo-yellow/20 shadow-lg shadow-black/20"
+                          class="max-w-full max-h-[512px] rounded-lg border border-exo-yellow/20 shadow-lg shadow-black/20 cursor-pointer"
+                          onclick={() => {
+                            if (attachment.preview)
+                              expandedImageSrc = attachment.preview;
+                          }}
                         />
                         <!-- Button overlay -->
                         <div
                           class="absolute top-2 right-2 flex gap-1 opacity-0 group-hover/img:opacity-100 transition-opacity"
                         >
+                          <!-- Expand button -->
+                          <button
+                            type="button"
+                            class="p-2 rounded-lg bg-exo-dark-gray/80 border border-exo-yellow/30 text-exo-yellow hover:bg-exo-dark-gray hover:border-exo-yellow/50 cursor-pointer"
+                            onclick={() => {
+                              if (attachment.preview)
+                                expandedImageSrc = attachment.preview;
+                            }}
+                            title="Expand image"
+                          >
+                            <svg
+                              class="w-4 h-4"
+                              fill="none"
+                              viewBox="0 0 24 24"
+                              stroke="currentColor"
+                              stroke-width="2"
+                            >
+                              <path
+                                stroke-linecap="round"
+                                stroke-linejoin="round"
+                                d="M4 8V4m0 0h4M4 4l5 5m11-1V4m0 0h-4m4 0l-5 5M4 16v4m0 0h4m-4 0l5-5m11 5l-5-5m5 5v-4m0 4h-4"
+                              />
+                            </svg>
+                          </button>
                           <!-- Edit button -->
                           <button
                             type="button"
@@ -789,3 +827,8 @@
     </button>
   {/if}
 </div>
+
+<ImageLightbox
+  src={expandedImageSrc}
+  onclose={() => (expandedImageSrc = null)}
+/>

dashboard/src/lib/components/FamilyLogos.svelte

@@ -13,6 +13,12 @@
       d="M12 2l3.09 6.26L22 9.27l-5 4.87 1.18 6.88L12 17.77l-6.18 3.25L7 14.14 2 9.27l6.91-1.01L12 2z"
     />
   </svg>
+{:else if family === "recents"}
+  <svg class="w-6 h-6 {className}" viewBox="0 0 24 24" fill="currentColor">
+    <path
+      d="M13 3a9 9 0 0 0-9 9H1l3.89 3.89.07.14L9 12H6c0-3.87 3.13-7 7-7s7 3.13 7 7-3.13 7-7 7c-1.93 0-3.68-.79-4.94-2.06l-1.42 1.42A8.954 8.954 0 0 0 13 21a9 9 0 0 0 0-18zm-1 5v5l4.28 2.54.72-1.21-3.5-2.08V8H12z"
+    />
+  </svg>
 {:else if family === "llama" || family === "meta"}
   <svg class="w-6 h-6 {className}" viewBox="0 0 24 24" fill="currentColor">
     <path
@@ -58,6 +64,18 @@
       d="M17.962 1.844h-4.326l-3.425 7.81H5.369V1.878H1.5V22h3.87v-8.477h6.824a3.025 3.025 0 002.743-1.75V22h3.87v-8.477a3.87 3.87 0 00-3.588-3.86v-.01h-2.125a3.94 3.94 0 002.323-2.12l2.545-5.689z"
     />
   </svg>
+{:else if family === "flux"}
+  <svg class="w-6 h-6 {className}" viewBox="0 0 24 24" fill="currentColor">
+    <path
+      d="M12 2L2 19h7.5l2.5-4.5L14.5 19H22L12 2zm0 4.5L16.5 17h-3l-1.5-2.7L10.5 17h-3L12 6.5z"
+    />
+  </svg>
+{:else if family === "qwen-image"}
+  <svg class="w-6 h-6 {className}" viewBox="0 0 24 24" fill="currentColor">
+    <path
+      d="M12.604 1.34c.393.69.784 1.382 1.174 2.075a.18.18 0 00.157.091h5.552c.174 0 .322.11.446.327l1.454 2.57c.19.337.24.478.024.837-.26.43-.513.864-.76 1.3l-.367.658c-.106.196-.223.28-.04.512l2.652 4.637c.172.301.111.494-.043.77-.437.785-.882 1.564-1.335 2.34-.159.272-.352.375-.68.37-.777-.016-1.552-.01-2.327.016a.099.099 0 00-.081.05 575.097 575.097 0 01-2.705 4.74c-.169.293-.38.363-.725.364-.997.003-2.002.004-3.017.002a.537.537 0 01-.465-.271l-1.335-2.323a.09.09 0 00-.083-.049H4.982c-.285.03-.553-.001-.805-.092l-1.603-2.77a.543.543 0 01-.002-.54l1.207-2.12a.198.198 0 000-.197 550.951 550.951 0 01-1.875-3.272l-.79-1.395c-.16-.31-.173-.496.095-.965.465-.813.927-1.625 1.387-2.436.132-.234.304-.334.584-.335a338.3 338.3 0 012.589-.001.124.124 0 00.107-.063l2.806-4.895a.488.488 0 01.422-.246c.524-.001 1.053 0 1.583-.006L11.704 1c.341-.003.724.032.9.34zm-3.432.403a.06.06 0 00-.052.03L6.254 6.788a.157.157 0 01-.135.078H3.253c-.056 0-.07.025-.041.074l5.81 10.156c.025.042.013.062-.034.063l-2.795.015a.218.218 0 00-.2.116l-1.32 2.31c-.044.078-.021.118.068.118l5.716.008c.046 0 .08.02.104.061l1.403 2.454c.046.081.092.082.139 0l5.006-8.76.783-1.382a.055.055 0 01.096 0l1.424 2.53a.122.122 0 00.107.062l2.763-.02a.04.04 0 00.035-.02.041.041 0 000-.04l-2.9-5.086a.108.108 0 010-.113l.293-.507 1.12-1.977c.024-.041.012-.062-.035-.062H9.2c-.059 0-.073-.026-.043-.077l1.434-2.505a.107.107 0 000-.114L9.225 1.774a.06.06 0 00-.053-.031zm6.29 8.02c.046 0 .058.02.034.06l-.832 1.465-2.613 4.585a.056.056 0 01-.05.029.058.058 0 01-.05-.029L8.498 9.841c-.02-.034-.01-.052.028-.054l.216-.012 6.722-.012z"
+    />
+  </svg>
 {:else if family === "huggingface"}
   <svg class="w-6 h-6 {className}" viewBox="0 0 24 24" fill="currentColor">
     <path

dashboard/src/lib/components/FamilySidebar.svelte

@@ -5,15 +5,22 @@
     families: string[];
     selectedFamily: string | null;
     hasFavorites: boolean;
+    hasRecents: boolean;
     onSelect: (family: string | null) => void;
   };
 
-  let { families, selectedFamily, hasFavorites, onSelect }: FamilySidebarProps =
-    $props();
+  let {
+    families,
+    selectedFamily,
+    hasFavorites,
+    hasRecents,
+    onSelect,
+  }: FamilySidebarProps = $props();
 
   // Family display names
   const familyNames: Record<string, string> = {
     favorites: "Favorites",
+    recents: "Recent",
     huggingface: "Hub",
     llama: "Meta",
     qwen: "Qwen",
@@ -22,6 +29,8 @@
     glm: "GLM",
     minimax: "MiniMax",
     kimi: "Kimi",
+    flux: "FLUX",
+    "qwen-image": "Qwen Img",
   };
 
   function getFamilyName(family: string): string {
@@ -32,7 +41,7 @@
 </script>
 
 <div
-  class="flex flex-col gap-1 py-2 px-1 border-r border-exo-yellow/10 bg-exo-medium-gray/30 min-w-[64px]"
+  class="flex flex-col gap-1 py-2 px-1 border-r border-exo-yellow/10 bg-exo-medium-gray/30 min-w-[64px] overflow-y-auto scrollbar-hide"
 >
   <!-- All models (no filter) -->
   <button
@@ -87,6 +96,31 @@
     </button>
   {/if}
 
+  <!-- Recent (only show if has recent models) -->
+  {#if hasRecents}
+    <button
+      type="button"
+      onclick={() => onSelect("recents")}
+      class="group flex flex-col items-center justify-center p-2 rounded transition-all duration-200 cursor-pointer {selectedFamily ===
+      'recents'
+        ? 'bg-exo-yellow/20 border-l-2 border-exo-yellow'
+        : 'hover:bg-white/5 border-l-2 border-transparent'}"
+      title="Recently launched models"
+    >
+      <FamilyLogos
+        family="recents"
+        class={selectedFamily === "recents"
+          ? "text-exo-yellow"
+          : "text-white/50 group-hover:text-white/70"}
+      />
+      <span
+        class="text-[9px] font-mono mt-0.5 {selectedFamily === 'recents'
+          ? 'text-exo-yellow'
+          : 'text-white/40 group-hover:text-white/60'}">Recent</span
+      >
+    </button>
+  {/if}
+
   <!-- HuggingFace Hub -->
   <button
     type="button"

dashboard/src/lib/components/ImageLightbox.svelte

@@ -0,0 +1,96 @@
+<script lang="ts">
+  import { fade, fly } from "svelte/transition";
+  import { cubicOut } from "svelte/easing";
+
+  interface Props {
+    src: string | null;
+    onclose: () => void;
+  }
+
+  let { src, onclose }: Props = $props();
+
+  function handleKeydown(e: KeyboardEvent) {
+    if (e.key === "Escape") {
+      onclose();
+    }
+  }
+
+  function extensionFromSrc(dataSrc: string): string {
+    const match = dataSrc.match(/^data:image\/(\w+)/);
+    if (match) return match[1] === "jpeg" ? "jpg" : match[1];
+    const urlMatch = dataSrc.match(/\.(\w+)(?:\?|$)/);
+    if (urlMatch) return urlMatch[1];
+    return "png";
+  }
+
+  function handleDownload(e: MouseEvent) {
+    e.stopPropagation();
+    if (!src) return;
+    const link = document.createElement("a");
+    link.href = src;
+    link.download = `image-${Date.now()}.${extensionFromSrc(src)}`;
+    link.click();
+  }
+
+  function handleClose(e: MouseEvent) {
+    e.stopPropagation();
+    onclose();
+  }
+</script>
+
+<svelte:window onkeydown={src ? handleKeydown : undefined} />
+
+{#if src}
+  <div
+    class="fixed inset-0 z-50 bg-black/90 backdrop-blur-sm flex items-center justify-center"
+    transition:fade={{ duration: 200 }}
+    onclick={onclose}
+    role="presentation"
+    onintrostart={() => (document.body.style.overflow = "hidden")}
+    onoutroend={() => (document.body.style.overflow = "")}
+  >
+    <div class="absolute top-4 right-4 flex gap-2 z-10">
+      <button
+        type="button"
+        class="p-2 rounded-lg bg-exo-dark-gray/80 border border-exo-yellow/30 text-exo-yellow hover:bg-exo-dark-gray hover:border-exo-yellow/50 cursor-pointer transition-colors"
+        onclick={handleDownload}
+        title="Download image"
+      >
+        <svg
+          class="w-5 h-5"
+          fill="none"
+          viewBox="0 0 24 24"
+          stroke="currentColor"
+          stroke-width="2"
+        >
+          <path
+            stroke-linecap="round"
+            stroke-linejoin="round"
+            d="M4 16v1a3 3 0 003 3h10a3 3 0 003-3v-1m-4-4l-4 4m0 0l-4-4m4 4V4"
+          />
+        </svg>
+      </button>
+      <button
+        type="button"
+        class="p-2 rounded-lg bg-exo-dark-gray/80 border border-exo-yellow/30 text-exo-yellow hover:bg-exo-dark-gray hover:border-exo-yellow/50 cursor-pointer transition-colors"
+        onclick={handleClose}
+        title="Close"
+      >
+        <svg class="w-5 h-5" viewBox="0 0 24 24" fill="currentColor">
+          <path
+            d="M19 6.41L17.59 5 12 10.59 6.41 5 5 6.41 10.59 12 5 17.59 6.41 19 12 13.41 17.59 19 19 17.59 13.41 12 19 6.41z"
+          />
+        </svg>
+      </button>
+    </div>
+
+    <!-- svelte-ignore a11y_no_noninteractive_element_interactions, a11y_click_events_have_key_events -->
+    <img
+      {src}
+      alt=""
+      class="max-w-[90vw] max-h-[90vh] object-contain rounded-lg shadow-2xl"
+      transition:fly={{ y: 20, duration: 300, easing: cubicOut }}
+      onclick={(e) => e.stopPropagation()}
+    />
+  </div>
+{/if}

dashboard/src/lib/components/ModelCard.svelte

@@ -422,9 +422,16 @@
     const bToACandidates: Array<{ ip: string; iface: string | null }> = [];
 
     for (const edge of topology.edges) {
-      const ip = edge.sendBackIp || "?";
-      const iface =
-        edge.sendBackInterface || getInterfaceForIp(edge.source, ip);
+      let ip: string;
+      let iface: string | null;
+
+      if (edge.sourceRdmaIface || edge.sinkRdmaIface) {
+        ip = "RDMA";
+        iface = `${edge.sourceRdmaIface || "?"} \u2192 ${edge.sinkRdmaIface || "?"}`;
+      } else {
+        ip = edge.sendBackIp || "?";
+        iface = edge.sendBackInterface || getInterfaceForIp(edge.source, ip);
+      }
 
       if (edge.source === nodeId1 && edge.target === nodeId2) {
         aToBCandidates.push({ ip, iface });

dashboard/src/lib/components/ModelFilterPopover.svelte

@@ -24,6 +24,8 @@
     { id: "thinking", label: "Thinking" },
     { id: "code", label: "Code" },
     { id: "vision", label: "Vision" },
+    { id: "image_gen", label: "Image Gen" },
+    { id: "image_edit", label: "Image Edit" },
   ];
 
   // Size ranges
@@ -142,6 +144,45 @@
                   stroke-linejoin="round"
                 /><circle cx="12" cy="12" r="3" /></svg
               >
+            {:else if cap.id === "image_gen"}
+              <svg
+                class="w-3.5 h-3.5 inline-block"
+                viewBox="0 0 24 24"
+                fill="none"
+                stroke="currentColor"
+                stroke-width="1.5"
+                ><rect
+                  x="3"
+                  y="3"
+                  width="18"
+                  height="18"
+                  rx="2"
+                  ry="2"
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                /><circle cx="8.5" cy="8.5" r="1.5" /><path
+                  d="M21 15l-5-5L5 21"
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                /></svg
+              >
+            {:else if cap.id === "image_edit"}
+              <svg
+                class="w-3.5 h-3.5 inline-block"
+                viewBox="0 0 24 24"
+                fill="none"
+                stroke="currentColor"
+                stroke-width="1.5"
+                ><path
+                  d="M11 4H4a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2v-7"
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                /><path
+                  d="M18.5 2.5a2.121 2.121 0 0 1 3 3L12 15l-4 1 1-4 9.5-9.5z"
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                /></svg
+              >
             {/if}
             <span class="ml-1">{cap.label}</span>
           </button>

dashboard/src/lib/components/ModelPickerGroup.svelte

@@ -26,6 +26,7 @@
     nodeNames: string[];
     nodeIds: string[];
   };
+  type ModelFitStatus = "fits_now" | "fits_cluster_capacity" | "too_large";
 
   type ModelPickerGroupProps = {
     group: ModelGroup;
@@ -33,11 +34,13 @@
     isFavorite: boolean;
     selectedModelId: string | null;
     canModelFit: (id: string) => boolean;
+    getModelFitStatus: (id: string) => ModelFitStatus;
     onToggleExpand: () => void;
     onSelectModel: (modelId: string) => void;
     onToggleFavorite: (baseModelId: string) => void;
     onShowInfo: (group: ModelGroup) => void;
     downloadStatusMap?: Map<string, DownloadAvailability>;
+    launchedAt?: number;
   };
 
   let {
@@ -46,11 +49,13 @@
     isFavorite,
     selectedModelId,
     canModelFit,
+    getModelFitStatus,
     onToggleExpand,
     onSelectModel,
     onToggleFavorite,
     onShowInfo,
     downloadStatusMap,
+    launchedAt,
   }: ModelPickerGroupProps = $props();
 
   // Group-level download status: show if any variant is downloaded
@@ -72,10 +77,45 @@
     return `${mb}MB`;
   }
 
+  function timeAgo(ts: number): string {
+    const seconds = Math.floor((Date.now() - ts) / 1000);
+    if (seconds < 60) return "just now";
+    const minutes = Math.floor(seconds / 60);
+    if (minutes < 60) return `${minutes}m ago`;
+    const hours = Math.floor(minutes / 60);
+    if (hours < 24) return `${hours}h ago`;
+    const days = Math.floor(hours / 24);
+    return `${days}d ago`;
+  }
+
   // Check if any variant can fit
   const anyVariantFits = $derived(
     group.variants.some((v) => canModelFit(v.id)),
   );
+  const groupFitStatus = $derived.by((): ModelFitStatus => {
+    let hasClusterCapacityOnly = false;
+    for (const variant of group.variants) {
+      const fitStatus = getModelFitStatus(variant.id);
+      if (fitStatus === "fits_now") {
+        return "fits_now";
+      }
+      if (fitStatus === "fits_cluster_capacity") {
+        hasClusterCapacityOnly = true;
+      }
+    }
+    return hasClusterCapacityOnly ? "fits_cluster_capacity" : "too_large";
+  });
+
+  function getSizeClassForFitStatus(fitStatus: ModelFitStatus): string {
+    switch (fitStatus) {
+      case "fits_now":
+        return "text-white/40";
+      case "fits_cluster_capacity":
+        return "text-orange-400/80";
+      case "too_large":
+        return "text-red-400/70";
+    }
+  }
 
   // Check if this group's model is currently selected (for single-variant groups)
   const isMainSelected = $derived(
@@ -200,9 +240,42 @@
               stroke-width="1.5"
               title="Supports image generation"
             >
-              <rect x="3" y="3" width="18" height="18" rx="2" ry="2" />
+              <rect
+                x="3"
+                y="3"
+                width="18"
+                height="18"
+                rx="2"
+                ry="2"
+                stroke-linecap="round"
+                stroke-linejoin="round"
+              />
               <circle cx="8.5" cy="8.5" r="1.5" />
-              <path d="M21 15l-5-5L5 21" />
+              <path
+                d="M21 15l-5-5L5 21"
+                stroke-linecap="round"
+                stroke-linejoin="round"
+              />
+            </svg>
+          {:else if cap === "image_edit"}
+            <svg
+              class="w-3.5 h-3.5 text-white/40 flex-shrink-0"
+              viewBox="0 0 24 24"
+              fill="none"
+              stroke="currentColor"
+              stroke-width="1.5"
+              title="Supports image editing"
+            >
+              <path
+                d="M11 4H4a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2v-7"
+                stroke-linecap="round"
+                stroke-linejoin="round"
+              />
+              <path
+                d="M18.5 2.5a2.121 2.121 0 0 1 3 3L12 15l-4 1 1-4 9.5-9.5z"
+                stroke-linecap="round"
+                stroke-linejoin="round"
+              />
             </svg>
           {/if}
         {/each}
@@ -211,7 +284,14 @@
 
     <!-- Size indicator (smallest variant) -->
     {#if !group.hasMultipleVariants && group.smallestVariant?.storage_size_megabytes}
-      <span class="text-xs font-mono text-white/30 flex-shrink-0">
+      {@const singleVariantFitStatus = getModelFitStatus(
+        group.smallestVariant.id,
+      )}
+      <span
+        class="text-xs font-mono flex-shrink-0 {getSizeClassForFitStatus(
+          singleVariantFitStatus,
+        )}"
+      >
         {formatSize(group.smallestVariant.storage_size_megabytes)}
       </span>
     {/if}
@@ -222,13 +302,24 @@
         .map((v) => v.storage_size_megabytes || 0)
         .filter((s) => s > 0)
         .sort((a, b) => a - b)}
-      <span class="text-xs font-mono text-white/30 flex-shrink-0">
+      <span
+        class="text-xs font-mono flex-shrink-0 {getSizeClassForFitStatus(
+          groupFitStatus,
+        )}"
+      >
         {group.variants.length} variants{#if sizes.length >= 2}{" "}({formatSize(
             sizes[0],
           )}-{formatSize(sizes[sizes.length - 1])}){/if}
       </span>
     {/if}
 
+    <!-- Time ago (for recent models) -->
+    {#if launchedAt}
+      <span class="text-xs font-mono text-white/20 flex-shrink-0">
+        {timeAgo(launchedAt)}
+      </span>
+    {/if}
+
     <!-- Download availability indicator -->
     {#if groupDownloadStatus && groupDownloadStatus.nodeIds.length > 0}
       <span
@@ -327,6 +418,7 @@
   {#if isExpanded && group.hasMultipleVariants}
     <div class="bg-black/20 border-t border-white/5">
       {#each group.variants as variant}
+        {@const fitStatus = getModelFitStatus(variant.id)}
         {@const modelCanFit = canModelFit(variant.id)}
         {@const isSelected = selectedModelId === variant.id}
         <button
@@ -351,7 +443,11 @@
           </span>
 
           <!-- Size -->
-          <span class="text-xs font-mono text-white/40 flex-1">
+          <span
+            class="text-xs font-mono flex-1 {getSizeClassForFitStatus(
+              fitStatus,
+            )}"
+          >
             {formatSize(variant.storage_size_megabytes)}
           </span>
 

dashboard/src/lib/components/ModelPickerModal.svelte

@@ -6,6 +6,7 @@
   import ModelFilterPopover from "./ModelFilterPopover.svelte";
   import HuggingFaceResultItem from "./HuggingFaceResultItem.svelte";
   import { getNodesWithModelDownloaded } from "$lib/utils/downloads";
+  import { getRecentEntries } from "$lib/stores/recents.svelte";
 
   interface ModelInfo {
     id: string;
@@ -46,13 +47,18 @@
     tags: string[];
   }
 
+  type ModelFitStatus = "fits_now" | "fits_cluster_capacity" | "too_large";
+
   type ModelPickerModalProps = {
     isOpen: boolean;
     models: ModelInfo[];
     selectedModelId: string | null;
     favorites: Set<string>;
+    recentModelIds?: string[];
+    hasRecents?: boolean;
     existingModelIds: Set<string>;
     canModelFit: (modelId: string) => boolean;
+    getModelFitStatus: (modelId: string) => ModelFitStatus;
     onSelect: (modelId: string) => void;
     onClose: () => void;
     onToggleFavorite: (baseModelId: string) => void;
@@ -76,8 +82,11 @@
     models,
     selectedModelId,
     favorites,
+    recentModelIds = [],
+    hasRecents: hasRecentsTab = false,
     existingModelIds,
     canModelFit,
+    getModelFitStatus,
     onSelect,
     onClose,
     onToggleFavorite,
@@ -363,6 +372,8 @@
       "deepseek",
       "gpt-oss",
       "llama",
+      "flux",
+      "qwen-image",
     ];
     return Array.from(families).sort((a, b) => {
       const aIdx = familyOrder.indexOf(a);
@@ -381,7 +392,11 @@
     // Filter by family
     if (selectedFamily === "favorites") {
       result = result.filter((g) => favorites.has(g.id));
-    } else if (selectedFamily && selectedFamily !== "huggingface") {
+    } else if (
+      selectedFamily &&
+      selectedFamily !== "huggingface" &&
+      selectedFamily !== "recents"
+    ) {
       result = result.filter((g) => g.family === selectedFamily);
     }
 
@@ -425,13 +440,23 @@
       );
     }
 
-    // Sort: models that fit first, then by size (largest first)
+    // Sort: fits-now first, then fits-cluster-capacity, then too-large
     result.sort((a, b) => {
-      const aFits = a.variants.some((v) => canModelFit(v.id));
-      const bFits = b.variants.some((v) => canModelFit(v.id));
+      const getGroupFitRank = (group: ModelGroup): number => {
+        let hasClusterCapacityOnly = false;
+        for (const variant of group.variants) {
+          const fitStatus = getModelFitStatus(variant.id);
+          if (fitStatus === "fits_now") return 0;
+          if (fitStatus === "fits_cluster_capacity") {
+            hasClusterCapacityOnly = true;
+          }
+        }
+        return hasClusterCapacityOnly ? 1 : 2;
+      };
 
-      if (aFits && !bFits) return -1;
-      if (!aFits && bFits) return 1;
+      const aRank = getGroupFitRank(a);
+      const bRank = getGroupFitRank(b);
+      if (aRank !== bRank) return aRank - bRank;
 
       return (
         (b.smallestVariant.storage_size_megabytes || 0) -
@@ -445,6 +470,48 @@
   // Check if any favorites exist
   const hasFavorites = $derived(favorites.size > 0);
 
+  // Timestamp lookup for recent models
+  const recentTimestamps = $derived(
+    new Map(getRecentEntries().map((e) => [e.modelId, e.launchedAt])),
+  );
+
+  // Recent models: single-variant ModelGroups in launch order
+  const recentGroups = $derived.by((): ModelGroup[] => {
+    if (!recentModelIds || recentModelIds.length === 0) return [];
+    const result: ModelGroup[] = [];
+    for (const id of recentModelIds) {
+      const model = models.find((m) => m.id === id);
+      if (model) {
+        result.push({
+          id: model.base_model || model.id,
+          name: model.name || model.id,
+          capabilities: model.capabilities || ["text"],
+          family: model.family || "",
+          variants: [model],
+          smallestVariant: model,
+          hasMultipleVariants: false,
+        });
+      }
+    }
+    return result;
+  });
+
+  // Filtered recent groups (apply search query)
+  const filteredRecentGroups = $derived.by((): ModelGroup[] => {
+    if (!searchQuery.trim()) return recentGroups;
+    const query = searchQuery.toLowerCase().trim();
+    return recentGroups.filter(
+      (g) =>
+        g.name.toLowerCase().includes(query) ||
+        g.variants.some(
+          (v) =>
+            v.id.toLowerCase().includes(query) ||
+            (v.name || "").toLowerCase().includes(query) ||
+            (v.quantization || "").toLowerCase().includes(query),
+        ),
+    );
+  });
+
   function toggleGroupExpanded(groupId: string) {
     const next = new Set(expandedGroups);
     if (next.has(groupId)) {
@@ -602,11 +669,12 @@
         families={uniqueFamilies}
         {selectedFamily}
         {hasFavorites}
+        hasRecents={hasRecentsTab}
         onSelect={(family) => (selectedFamily = family)}
       />
 
       <!-- Model list -->
-      <div class="flex-1 overflow-y-auto flex flex-col">
+      <div class="flex-1 overflow-y-auto scrollbar-hide flex flex-col">
         {#if selectedFamily === "huggingface"}
           <!-- HuggingFace Hub view -->
           <div class="flex-1 flex flex-col min-h-0">
@@ -624,7 +692,7 @@
             </div>
 
             <!-- Results list -->
-            <div class="flex-1 overflow-y-auto">
+            <div class="flex-1 overflow-y-auto scrollbar-hide">
               {#if hfIsLoadingTrending && hfTrendingModels.length === 0}
                 <div
                   class="flex items-center justify-center py-12 text-white/40"
@@ -709,6 +777,44 @@
               </div>
             </div>
           </div>
+        {:else if selectedFamily === "recents"}
+          <!-- Recent models view -->
+          {#if filteredRecentGroups.length === 0}
+            <div
+              class="flex flex-col items-center justify-center h-full text-white/40 p-8"
+            >
+              <svg
+                class="w-12 h-12 mb-3"
+                viewBox="0 0 24 24"
+                fill="currentColor"
+              >
+                <path
+                  d="M13 3a9 9 0 0 0-9 9H1l3.89 3.89.07.14L9 12H6c0-3.87 3.13-7 7-7s7 3.13 7 7-3.13 7-7 7c-1.93 0-3.68-.79-4.94-2.06l-1.42 1.42A8.954 8.954 0 0 0 13 21a9 9 0 0 0 0-18zm-1 5v5l4.28 2.54.72-1.21-3.5-2.08V8H12z"
+                />
+              </svg>
+              <p class="font-mono text-sm">
+                {searchQuery
+                  ? "No matching recent models"
+                  : "No recently launched models"}
+              </p>
+            </div>
+          {:else}
+            {#each filteredRecentGroups as group}
+              <ModelPickerGroup
+                {group}
+                isExpanded={expandedGroups.has(group.id)}
+                isFavorite={favorites.has(group.id)}
+                {selectedModelId}
+                {canModelFit}
+                onToggleExpand={() => toggleGroupExpanded(group.id)}
+                onSelectModel={handleSelect}
+                {onToggleFavorite}
+                onShowInfo={(g) => (infoGroup = g)}
+                downloadStatusMap={getVariantDownloadMap(group)}
+                launchedAt={recentTimestamps.get(group.variants[0]?.id ?? "")}
+              />
+            {/each}
+          {/if}
         {:else if filteredGroups.length === 0}
           <div
             class="flex flex-col items-center justify-center h-full text-white/40 p-8"
@@ -740,6 +846,7 @@
               isFavorite={favorites.has(group.id)}
               {selectedModelId}
               {canModelFit}
+              {getModelFitStatus}
               onToggleExpand={() => toggleGroupExpanded(group.id)}
               onSelectModel={handleSelect}
               {onToggleFavorite}

dashboard/src/lib/components/TopologyGraph.svelte

@@ -6,6 +6,7 @@
     isTopologyMinimized,
     debugMode,
     nodeThunderboltBridge,
+    nodeRdmaCtl,
     type NodeInfo,
   } from "$lib/stores/app.svelte";
 
@@ -31,6 +32,7 @@
   const data = $derived(topologyData());
   const debugEnabled = $derived(debugMode());
   const tbBridgeData = $derived(nodeThunderboltBridge());
+  const rdmaCtlData = $derived(nodeRdmaCtl());
 
   function getNodeLabel(nodeId: string): string {
     const node = data?.nodes?.[nodeId];
@@ -333,14 +335,27 @@
       if (edge.source === a) entry.aToB = true;
       else entry.bToA = true;
 
-      const ip = edge.sendBackIp || "?";
-      const ifaceInfo = getInterfaceLabel(edge.source, ip);
+      let ip: string;
+      let ifaceLabel: string;
+      let missingIface: boolean;
+
+      if (edge.sourceRdmaIface || edge.sinkRdmaIface) {
+        ip = "RDMA";
+        ifaceLabel = `${edge.sourceRdmaIface || "?"} \u2192 ${edge.sinkRdmaIface || "?"}`;
+        missingIface = false;
+      } else {
+        ip = edge.sendBackIp || "?";
+        const ifaceInfo = getInterfaceLabel(edge.source, ip);
+        ifaceLabel = ifaceInfo.label;
+        missingIface = ifaceInfo.missing;
+      }
+
       entry.connections.push({
         from: edge.source,
         to: edge.target,
         ip,
-        ifaceLabel: ifaceInfo.label,
-        missingIface: ifaceInfo.missing,
+        ifaceLabel,
+        missingIface,
       });
       pairMap.set(key, entry);
     });
@@ -1120,15 +1135,17 @@
           .text(` (${ramUsagePercent.toFixed(0)}%)`);
       }
 
-      // Debug mode: Show TB bridge status
+      // Debug mode: Show TB bridge and RDMA status
       if (debugEnabled) {
+        let debugLabelY =
+          nodeInfo.y +
+          iconBaseHeight / 2 +
+          (showFullLabels ? 32 : showCompactLabels ? 26 : 22);
+        const debugFontSize = showFullLabels ? 9 : 7;
+        const debugLineHeight = showFullLabels ? 11 : 9;
+
         const tbStatus = tbBridgeData[nodeInfo.id];
         if (tbStatus) {
-          const tbY =
-            nodeInfo.y +
-            iconBaseHeight / 2 +
-            (showFullLabels ? 32 : showCompactLabels ? 26 : 22);
-          const tbFontSize = showFullLabels ? 9 : 7;
           const tbColor = tbStatus.enabled
             ? "rgba(234,179,8,0.9)"
             : "rgba(100,100,100,0.7)";
@@ -1136,12 +1153,30 @@
           nodeG
             .append("text")
             .attr("x", nodeInfo.x)
-            .attr("y", tbY)
+            .attr("y", debugLabelY)
             .attr("text-anchor", "middle")
             .attr("fill", tbColor)
-            .attr("font-size", tbFontSize)
+            .attr("font-size", debugFontSize)
             .attr("font-family", "SF Mono, Monaco, monospace")
             .text(tbText);
+          debugLabelY += debugLineHeight;
+        }
+
+        const rdmaStatus = rdmaCtlData[nodeInfo.id];
+        if (rdmaStatus !== undefined) {
+          const rdmaColor = rdmaStatus.enabled
+            ? "rgba(74,222,128,0.9)"
+            : "rgba(100,100,100,0.7)";
+          const rdmaText = rdmaStatus.enabled ? "RDMA:ON" : "RDMA:OFF";
+          nodeG
+            .append("text")
+            .attr("x", nodeInfo.x)
+            .attr("y", debugLabelY)
+            .attr("text-anchor", "middle")
+            .attr("fill", rdmaColor)
+            .attr("font-size", debugFontSize)
+            .attr("font-family", "SF Mono, Monaco, monospace")
+            .text(rdmaText);
         }
       }
     });

dashboard/src/lib/stores/app.svelte.ts

@@ -49,6 +49,7 @@ export interface NodeInfo {
   };
   last_macmon_update: number;
   friendly_name?: string;
+  os_version?: string;
 }
 
 export interface TopologyEdge {
@@ -56,6 +57,8 @@ export interface TopologyEdge {
   target: string;
   sendBackIp?: string;
   sendBackInterface?: string;
+  sourceRdmaIface?: string;
+  sinkRdmaIface?: string;
 }
 
 export interface TopologyData {
@@ -76,6 +79,8 @@ interface RawNodeIdentity {
   modelId?: string;
   chipId?: string;
   friendlyName?: string;
+  osVersion?: string;
+  osBuildVersion?: string;
 }
 
 interface RawMemoryUsage {
@@ -225,6 +230,19 @@ interface RawStateResponse {
   nodeMemory?: Record<string, RawMemoryUsage>;
   nodeSystem?: Record<string, RawSystemPerformanceProfile>;
   nodeNetwork?: Record<string, RawNodeNetworkInfo>;
+  // Thunderbolt identifiers per node
+  nodeThunderbolt?: Record<
+    string,
+    {
+      interfaces: Array<{
+        rdmaInterface: string;
+        domainUuid: string;
+        linkSpeed: string;
+      }>;
+    }
+  >;
+  // RDMA ctl status per node
+  nodeRdmaCtl?: Record<string, { enabled: boolean }>;
   // Thunderbolt bridge status per node
   nodeThunderboltBridge?: Record<
     string,
@@ -425,6 +443,7 @@ function transformTopology(
       },
       last_macmon_update: Date.now() / 1000,
       friendly_name: identity?.friendlyName,
+      os_version: identity?.osVersion,
     };
   }
 
@@ -437,6 +456,8 @@ function transformTopology(
         if (!Array.isArray(edgeList)) continue;
         for (const edge of edgeList) {
           let sendBackIp: string | undefined;
+          let sourceRdmaIface: string | undefined;
+          let sinkRdmaIface: string | undefined;
           if (edge && typeof edge === "object" && "sinkMultiaddr" in edge) {
             const multiaddr = edge.sinkMultiaddr;
             if (multiaddr) {
@@ -444,10 +465,23 @@ function transformTopology(
                 multiaddr.ip_address ||
                 extractIpFromMultiaddr(multiaddr.address);
             }
+          } else if (
+            edge &&
+            typeof edge === "object" &&
+            "sourceRdmaIface" in edge
+          ) {
+            sourceRdmaIface = edge.sourceRdmaIface;
+            sinkRdmaIface = edge.sinkRdmaIface;
           }
 
           if (nodes[source] && nodes[sink] && source !== sink) {
-            edges.push({ source, target: sink, sendBackIp });
+            edges.push({
+              source,
+              target: sink,
+              sendBackIp,
+              sourceRdmaIface,
+              sinkRdmaIface,
+            });
           }
         }
       }
@@ -490,12 +524,32 @@ class AppStore {
   instances = $state<Record<string, unknown>>({});
   runners = $state<Record<string, unknown>>({});
   downloads = $state<Record<string, unknown[]>>({});
+  nodeDisk = $state<
+    Record<
+      string,
+      { total: { inBytes: number }; available: { inBytes: number } }
+    >
+  >({});
   placementPreviews = $state<PlacementPreview[]>([]);
   selectedPreviewModelId = $state<string | null>(null);
   isLoadingPreviews = $state(false);
   previewNodeFilter = $state<Set<string>>(new Set());
   lastUpdate = $state<number | null>(null);
+  nodeIdentities = $state<Record<string, RawNodeIdentity>>({});
   thunderboltBridgeCycles = $state<string[][]>([]);
+  nodeThunderbolt = $state<
+    Record<
+      string,
+      {
+        interfaces: Array<{
+          rdmaInterface: string;
+          domainUuid: string;
+          linkSpeed: string;
+        }>;
+      }
+    >
+  >({});
+  nodeRdmaCtl = $state<Record<string, { enabled: boolean }>>({});
   nodeThunderboltBridge = $state<
     Record<
       string,
@@ -1206,6 +1260,15 @@ class AppStore {
       if (data.downloads) {
         this.downloads = data.downloads;
       }
+      if (data.nodeDisk) {
+        this.nodeDisk = data.nodeDisk;
+      }
+      // Node identities (for OS version mismatch detection)
+      this.nodeIdentities = data.nodeIdentities ?? {};
+      // Thunderbolt identifiers per node
+      this.nodeThunderbolt = data.nodeThunderbolt ?? {};
+      // RDMA ctl status per node
+      this.nodeRdmaCtl = data.nodeRdmaCtl ?? {};
       // Thunderbolt bridge cycles
       this.thunderboltBridgeCycles = data.thunderboltBridgeCycles ?? [];
       // Thunderbolt bridge status per node
@@ -2958,6 +3021,7 @@ export const topologyData = () => appStore.topologyData;
 export const instances = () => appStore.instances;
 export const runners = () => appStore.runners;
 export const downloads = () => appStore.downloads;
+export const nodeDisk = () => appStore.nodeDisk;
 export const placementPreviews = () => appStore.placementPreviews;
 export const selectedPreviewModelId = () => appStore.selectedPreviewModelId;
 export const isLoadingPreviews = () => appStore.isLoadingPreviews;
@@ -3038,7 +3102,12 @@ export const setChatSidebarVisible = (visible: boolean) =>
   appStore.setChatSidebarVisible(visible);
 export const refreshState = () => appStore.fetchState();
 
-// Thunderbolt bridge status
+// Node identities (for OS version mismatch detection)
+export const nodeIdentities = () => appStore.nodeIdentities;
+
+// Thunderbolt & RDMA status
+export const nodeThunderbolt = () => appStore.nodeThunderbolt;
+export const nodeRdmaCtl = () => appStore.nodeRdmaCtl;
 export const thunderboltBridgeCycles = () => appStore.thunderboltBridgeCycles;
 export const nodeThunderboltBridge = () => appStore.nodeThunderboltBridge;
 

dashboard/src/lib/stores/recents.svelte.ts

@@ -0,0 +1,75 @@
+/**
+ * RecentsStore - Manages recently launched models with localStorage persistence
+ */
+
+import { browser } from "$app/environment";
+
+const RECENTS_KEY = "exo-recent-models";
+const MAX_RECENT_MODELS = 20;
+
+interface RecentEntry {
+  modelId: string;
+  launchedAt: number;
+}
+
+class RecentsStore {
+  recents = $state<RecentEntry[]>([]);
+
+  constructor() {
+    if (browser) {
+      this.loadFromStorage();
+    }
+  }
+
+  private loadFromStorage() {
+    try {
+      const stored = localStorage.getItem(RECENTS_KEY);
+      if (stored) {
+        const parsed = JSON.parse(stored) as RecentEntry[];
+        this.recents = parsed;
+      }
+    } catch (error) {
+      console.error("Failed to load recent models:", error);
+    }
+  }
+
+  private saveToStorage() {
+    try {
+      localStorage.setItem(RECENTS_KEY, JSON.stringify(this.recents));
+    } catch (error) {
+      console.error("Failed to save recent models:", error);
+    }
+  }
+
+  recordLaunch(modelId: string) {
+    // Remove existing entry for this model (if any) to move it to top
+    const filtered = this.recents.filter((r) => r.modelId !== modelId);
+    // Prepend new entry
+    const next = [{ modelId, launchedAt: Date.now() }, ...filtered];
+    // Cap at max
+    this.recents = next.slice(0, MAX_RECENT_MODELS);
+    this.saveToStorage();
+  }
+
+  getRecentModelIds(): string[] {
+    return this.recents.map((r) => r.modelId);
+  }
+
+  hasAny(): boolean {
+    return this.recents.length > 0;
+  }
+
+  clearAll() {
+    this.recents = [];
+    this.saveToStorage();
+  }
+}
+
+export const recentsStore = new RecentsStore();
+
+export const hasRecents = () => recentsStore.hasAny();
+export const getRecentModelIds = () => recentsStore.getRecentModelIds();
+export const getRecentEntries = () => recentsStore.recents;
+export const recordRecentLaunch = (modelId: string) =>
+  recentsStore.recordLaunch(modelId);
+export const clearRecents = () => recentsStore.clearAll();

dashboard/src/routes/+page.svelte

@@ -12,6 +12,11 @@
     toggleFavorite,
     getFavoritesSet,
   } from "$lib/stores/favorites.svelte";
+  import {
+    hasRecents,
+    getRecentModelIds,
+    recordRecentLaunch,
+  } from "$lib/stores/recents.svelte";
   import {
     hasStartedChat,
     isTopologyMinimized,
@@ -37,8 +42,11 @@
     toggleTopologyOnlyMode,
     chatSidebarVisible,
     toggleChatSidebarVisible,
+    nodeThunderbolt,
+    nodeRdmaCtl,
     thunderboltBridgeCycles,
     nodeThunderboltBridge,
+    nodeIdentities,
     type DownloadProgress,
     type PlacementPreview,
   } from "$lib/stores/app.svelte";
@@ -62,8 +70,50 @@
   const sidebarVisible = $derived(chatSidebarVisible());
   const tbBridgeCycles = $derived(thunderboltBridgeCycles());
   const tbBridgeData = $derived(nodeThunderboltBridge());
+  const identitiesData = $derived(nodeIdentities());
+  const tbIdentifiers = $derived(nodeThunderbolt());
+  const rdmaCtlData = $derived(nodeRdmaCtl());
   const nodeFilter = $derived(previewNodeFilter());
 
+  // Detect macOS version mismatches across cluster nodes
+  const macosVersionMismatch = $derived.by(() => {
+    if (!identitiesData) return null;
+    const entries = Object.entries(identitiesData);
+    // Filter to macOS nodes (version starts with a digit, e.g. "15.3")
+    const macosNodes = entries.filter(([_, id]) => {
+      const v = id.osVersion;
+      return v && v !== "Unknown" && /^\d/.test(v);
+    });
+    if (macosNodes.length < 2) return null;
+    // Compare on buildVersion for precise mismatch detection
+    const buildVersions = new Set(
+      macosNodes.map(([_, id]) => id.osBuildVersion ?? id.osVersion),
+    );
+    if (buildVersions.size <= 1) return null;
+    return macosNodes.map(([nodeId, id]) => ({
+      nodeId,
+      friendlyName: getNodeName(nodeId),
+      version: id.osVersion!,
+      buildVersion: id.osBuildVersion ?? "Unknown",
+    }));
+  });
+
+  // Detect TB5 nodes where RDMA is not enabled
+  const tb5WithoutRdma = $derived.by(() => {
+    const rdmaCtl = rdmaCtlData;
+    if (!rdmaCtl) return false;
+    const ids = tbIdentifiers;
+    if (!ids) return false;
+    // Find nodes with TB5 hardware (any TB interface)
+    const tb5NodeIds = Object.entries(ids)
+      .filter(([_, node]) => node.interfaces.length > 0)
+      .map(([id]) => id);
+    if (tb5NodeIds.length < 2) return false;
+    // At least one TB5 node has RDMA disabled
+    return tb5NodeIds.some((id) => rdmaCtl[id]?.enabled !== true);
+  });
+  let tb5InfoDismissed = $state(false);
+
   // Helper to get friendly node name from node ID
   function getNodeName(nodeId: string): string {
     const node = data?.nodes?.[nodeId];
@@ -96,6 +146,12 @@
     }
   }
 
+  // Warning icon SVG path (reused across warning snippets)
+  const warningIconPath =
+    "M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z";
+  const infoIconPath =
+    "M13 16h-1v-4h-1m1-4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z";
+
   let mounted = $state(false);
 
   // Instance launch state
@@ -113,6 +169,10 @@
       capabilities?: string[];
     }>
   >([]);
+  type ModelMemoryFitStatus =
+    | "fits_now"
+    | "fits_cluster_capacity"
+    | "too_large";
 
   // Model tasks lookup for ChatForm - maps both short IDs and full HuggingFace IDs
   const modelTasks = $derived(() => {
@@ -228,6 +288,10 @@
   // Favorites state (reactive)
   const favoritesSet = $derived(getFavoritesSet());
 
+  // Recent models state (reactive)
+  const recentModelIds = $derived(getRecentModelIds());
+  const showRecentsTab = $derived(hasRecents());
+
   // Slider dragging state
   let isDraggingSlider = $state(false);
   let sliderTrackElement: HTMLDivElement | null = $state(null);
@@ -450,14 +514,41 @@
     );
   });
 
+  // Calculate total memory in the cluster (in GB)
+  const clusterTotalMemoryGB = $derived(() => {
+    if (!data) return 0;
+    return (
+      Object.values(data.nodes).reduce((acc, n) => {
+        const total =
+          n.macmon_info?.memory?.ram_total ?? n.system_info?.memory ?? 0;
+        return acc + total;
+      }, 0) /
+      (1024 * 1024 * 1024)
+    );
+  });
+
+  function getModelMemoryFitStatus(model: {
+    id: string;
+    name?: string;
+    storage_size_megabytes?: number;
+  }): ModelMemoryFitStatus {
+    const modelSizeGB = getModelSizeGB(model);
+    if (modelSizeGB <= availableMemoryGB()) {
+      return "fits_now";
+    }
+    if (modelSizeGB <= clusterTotalMemoryGB()) {
+      return "fits_cluster_capacity";
+    }
+    return "too_large";
+  }
+
   // Check if a model has enough memory to run
   function hasEnoughMemory(model: {
     id: string;
     name?: string;
     storage_size_megabytes?: number;
   }): boolean {
-    const modelSizeGB = getModelSizeGB(model);
-    return modelSizeGB <= availableMemoryGB();
+    return getModelMemoryFitStatus(model) === "fits_now";
   }
 
   // Sorted models for dropdown - biggest first, unrunnable at the end
@@ -630,6 +721,9 @@
         // Always auto-select the newly launched model so the user chats to what they just launched
         setSelectedChatModel(modelId);
 
+        // Record the launch in recent models history
+        recordRecentLaunch(modelId);
+
         // Scroll to the bottom of instances container to show the new instance
         // Use multiple attempts to ensure DOM has updated with the new instance
         const scrollToBottom = () => {
@@ -1657,6 +1751,249 @@
   });
 </script>
 
+{#snippet clusterWarnings()}
+  {#if tbBridgeCycles.length > 0 || macosVersionMismatch || (tb5WithoutRdma && !tb5InfoDismissed)}
+    <div class="absolute top-4 left-4 flex flex-col gap-2 z-40">
+      {#if tbBridgeCycles.length > 0}
+        {@const cycle = tbBridgeCycles[0]}
+        {@const serviceName = getTbBridgeServiceName(cycle)}
+        {@const disableCmd = `sudo networksetup -setnetworkserviceenabled "${serviceName}" off`}
+        <div class="group relative" role="alert">
+          <div
+            class="flex items-center gap-2 px-3 py-2 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm cursor-help"
+          >
+            <svg
+              class="w-5 h-5 text-yellow-400 flex-shrink-0"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d={warningIconPath}
+              />
+            </svg>
+            <span class="text-sm font-mono text-yellow-200">
+              THUNDERBOLT BRIDGE CYCLE DETECTED
+            </span>
+          </div>
+
+          <!-- Tooltip on hover -->
+          <div
+            class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-yellow-500/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
+          >
+            <p class="text-xs text-white/80 mb-2">
+              A network routing cycle was detected between nodes connected via
+              Thunderbolt Bridge. This can cause connectivity issues.
+            </p>
+            <p class="text-xs text-white/60 mb-2">
+              <span class="text-yellow-300">Affected nodes:</span>
+              {cycle.map(getNodeName).join(" → ")}
+            </p>
+            <p class="text-xs text-white/60 mb-1">
+              <span class="text-yellow-300">To fix:</span> Disable the Thunderbolt
+              Bridge on one of the affected nodes:
+            </p>
+            <button
+              type="button"
+              onclick={() => copyToClipboard(disableCmd)}
+              class="w-full flex items-center gap-2 text-[10px] font-mono bg-exo-black/60 px-2 py-1.5 rounded text-exo-yellow break-all text-left hover:bg-exo-black/80 transition-colors cursor-pointer group/copy"
+              title="Click to copy"
+            >
+              <span class="flex-1">{disableCmd}</span>
+              <svg
+                class="w-3.5 h-3.5 flex-shrink-0 text-white/40 group-hover/copy:text-exo-yellow transition-colors"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                {#if copiedCommand}
+                  <path
+                    stroke-linecap="round"
+                    stroke-linejoin="round"
+                    d="M5 13l4 4L19 7"
+                  />
+                {:else}
+                  <path
+                    stroke-linecap="round"
+                    stroke-linejoin="round"
+                    d="M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z"
+                  />
+                {/if}
+              </svg>
+            </button>
+          </div>
+        </div>
+      {/if}
+
+      {#if macosVersionMismatch}
+        <div class="group relative" role="alert">
+          <div
+            class="flex items-center gap-2 px-3 py-2 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm cursor-help"
+          >
+            <svg
+              class="w-5 h-5 text-yellow-400 flex-shrink-0"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d={warningIconPath}
+              />
+            </svg>
+            <span class="text-sm font-mono text-yellow-200">
+              INCOMPATIBLE macOS VERSIONS
+            </span>
+          </div>
+
+          <!-- Tooltip on hover -->
+          <div
+            class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-yellow-500/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
+          >
+            <p class="text-xs text-white/80 mb-2">
+              Nodes in this cluster are running different macOS versions. This
+              may cause inference compatibility issues.
+            </p>
+            <div class="text-xs text-white/60 mb-2">
+              <span class="text-yellow-300">Node versions:</span>
+              {#each macosVersionMismatch as node}
+                <div class="ml-2">
+                  {node.friendlyName} — macOS {node.version} ({node.buildVersion})
+                </div>
+              {/each}
+            </div>
+            <p class="text-xs text-white/60">
+              <span class="text-yellow-300">Suggested action:</span> Update all nodes
+              to the same macOS version for best compatibility.
+            </p>
+          </div>
+        </div>
+      {/if}
+
+      {#if tb5WithoutRdma && !tb5InfoDismissed}
+        <div
+          class="flex items-center gap-2 px-3 py-2 rounded border border-blue-400/50 bg-blue-400/10 backdrop-blur-sm"
+          role="status"
+        >
+          <svg
+            class="w-5 h-5 text-blue-400 flex-shrink-0"
+            fill="none"
+            viewBox="0 0 24 24"
+            stroke="currentColor"
+            stroke-width="2"
+          >
+            <path
+              stroke-linecap="round"
+              stroke-linejoin="round"
+              d={infoIconPath}
+            />
+          </svg>
+          <span class="text-sm font-mono text-blue-200"> RDMA AVAILABLE </span>
+          <button
+            type="button"
+            onclick={() => (tb5InfoDismissed = true)}
+            class="ml-1 text-blue-300/60 hover:text-blue-200 transition-colors cursor-pointer"
+            title="Dismiss"
+          >
+            <svg
+              class="w-4 h-4"
+              fill="none"
+              viewBox="0 0 24 24"
+              stroke="currentColor"
+              stroke-width="2"
+            >
+              <path
+                stroke-linecap="round"
+                stroke-linejoin="round"
+                d="M6 18L18 6M6 6l12 12"
+              />
+            </svg>
+          </button>
+        </div>
+      {/if}
+    </div>
+  {/if}
+{/snippet}
+
+{#snippet clusterWarningsCompact()}
+  {#if tbBridgeCycles.length > 0 || macosVersionMismatch || (tb5WithoutRdma && !tb5InfoDismissed)}
+    <div class="absolute top-2 left-2 flex flex-col gap-1">
+      {#if tbBridgeCycles.length > 0}
+        <div
+          class="flex items-center gap-1.5 px-2 py-1 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm"
+          title="Thunderbolt Bridge cycle detected"
+        >
+          <svg
+            class="w-3.5 h-3.5 text-yellow-400"
+            fill="none"
+            viewBox="0 0 24 24"
+            stroke="currentColor"
+            stroke-width="2"
+          >
+            <path
+              stroke-linecap="round"
+              stroke-linejoin="round"
+              d={warningIconPath}
+            />
+          </svg>
+          <span class="text-[10px] font-mono text-yellow-200">TB CYCLE</span>
+        </div>
+      {/if}
+      {#if macosVersionMismatch}
+        <div
+          class="flex items-center gap-1.5 px-2 py-1 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm"
+          title="Incompatible macOS versions detected"
+        >
+          <svg
+            class="w-3.5 h-3.5 text-yellow-400"
+            fill="none"
+            viewBox="0 0 24 24"
+            stroke="currentColor"
+            stroke-width="2"
+          >
+            <path
+              stroke-linecap="round"
+              stroke-linejoin="round"
+              d={warningIconPath}
+            />
+          </svg>
+          <span class="text-[10px] font-mono text-yellow-200"
+            >macOS MISMATCH</span
+          >
+        </div>
+      {/if}
+      {#if tb5WithoutRdma && !tb5InfoDismissed}
+        <div
+          class="flex items-center gap-1.5 px-2 py-1 rounded border border-blue-400/50 bg-blue-400/10 backdrop-blur-sm"
+          title="Thunderbolt 5 detected — RDMA can be enabled for better performance"
+        >
+          <svg
+            class="w-3.5 h-3.5 text-blue-400"
+            fill="none"
+            viewBox="0 0 24 24"
+            stroke="currentColor"
+            stroke-width="2"
+          >
+            <path
+              stroke-linecap="round"
+              stroke-linejoin="round"
+              d={infoIconPath}
+            />
+          </svg>
+          <span class="text-[10px] font-mono text-blue-200">RDMA AVAILABLE</span
+          >
+        </div>
+      {/if}
+    </div>
+  {/if}
+{/snippet}
+
 <!-- Global event listeners for slider dragging -->
 <svelte:window
   onmousemove={handleSliderMouseMove}
@@ -1724,17 +2061,40 @@
             onNodeClick={togglePreviewNodeFilter}
           />
 
-          <!-- Thunderbolt Bridge Cycle Warning -->
-          {#if tbBridgeCycles.length > 0}
-            {@const cycle = tbBridgeCycles[0]}
-            {@const serviceName = getTbBridgeServiceName(cycle)}
-            {@const disableCmd = `sudo networksetup -setnetworkserviceenabled "${serviceName}" off`}
-            <div class="absolute top-4 left-4 group" role="alert">
-              <div
-                class="flex items-center gap-2 px-3 py-2 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm cursor-help"
+          {@render clusterWarnings()}
+
+          <!-- TB5 RDMA Available Info -->
+          {#if tb5WithoutRdma && !tb5InfoDismissed}
+            <div
+              class="absolute left-4 flex items-center gap-2 px-3 py-2 rounded border border-blue-400/50 bg-blue-400/10 backdrop-blur-sm"
+              class:top-16={tbBridgeCycles.length > 0}
+              class:top-4={tbBridgeCycles.length === 0}
+              role="status"
+            >
+              <svg
+                class="w-5 h-5 text-blue-400 flex-shrink-0"
+                fill="none"
+                viewBox="0 0 24 24"
+                stroke="currentColor"
+                stroke-width="2"
+              >
+                <path
+                  stroke-linecap="round"
+                  stroke-linejoin="round"
+                  d="M13 16h-1v-4h-1m1-4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"
+                />
+              </svg>
+              <span class="text-sm font-mono text-blue-200">
+                RDMA AVAILABLE
+              </span>
+              <button
+                type="button"
+                onclick={() => (tb5InfoDismissed = true)}
+                class="ml-1 text-blue-300/60 hover:text-blue-200 transition-colors cursor-pointer"
+                title="Dismiss"
               >
                 <svg
-                  class="w-5 h-5 text-yellow-400 flex-shrink-0"
+                  class="w-4 h-4"
                   fill="none"
                   viewBox="0 0 24 24"
                   stroke="currentColor"
@@ -1743,60 +2103,10 @@
                   <path
                     stroke-linecap="round"
                     stroke-linejoin="round"
-                    d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
+                    d="M6 18L18 6M6 6l12 12"
                   />
                 </svg>
-                <span class="text-sm font-mono text-yellow-200">
-                  THUNDERBOLT BRIDGE CYCLE DETECTED
-                </span>
-              </div>
-
-              <!-- Tooltip on hover -->
-              <div
-                class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-yellow-500/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
-              >
-                <p class="text-xs text-white/80 mb-2">
-                  A network routing cycle was detected between nodes connected
-                  via Thunderbolt Bridge. This can cause connectivity issues.
-                </p>
-                <p class="text-xs text-white/60 mb-2">
-                  <span class="text-yellow-300">Affected nodes:</span>
-                  {cycle.map(getNodeName).join(" → ")}
-                </p>
-                <p class="text-xs text-white/60 mb-1">
-                  <span class="text-yellow-300">To fix:</span> Disable the Thunderbolt
-                  Bridge on one of the affected nodes:
-                </p>
-                <button
-                  type="button"
-                  onclick={() => copyToClipboard(disableCmd)}
-                  class="w-full flex items-center gap-2 text-[10px] font-mono bg-exo-black/60 px-2 py-1.5 rounded text-exo-yellow break-all text-left hover:bg-exo-black/80 transition-colors cursor-pointer group/copy"
-                  title="Click to copy"
-                >
-                  <span class="flex-1">{disableCmd}</span>
-                  <svg
-                    class="w-3.5 h-3.5 flex-shrink-0 text-white/40 group-hover/copy:text-exo-yellow transition-colors"
-                    fill="none"
-                    viewBox="0 0 24 24"
-                    stroke="currentColor"
-                    stroke-width="2"
-                  >
-                    {#if copiedCommand}
-                      <path
-                        stroke-linecap="round"
-                        stroke-linejoin="round"
-                        d="M5 13l4 4L19 7"
-                      />
-                    {:else}
-                      <path
-                        stroke-linecap="round"
-                        stroke-linejoin="round"
-                        d="M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z"
-                      />
-                    {/if}
-                  </svg>
-                </button>
-              </div>
+              </button>
             </div>
           {/if}
 
@@ -1843,17 +2153,21 @@
               onNodeClick={togglePreviewNodeFilter}
             />
 
-            <!-- Thunderbolt Bridge Cycle Warning -->
-            {#if tbBridgeCycles.length > 0}
-              {@const cycle = tbBridgeCycles[0]}
-              {@const serviceName = getTbBridgeServiceName(cycle)}
-              {@const disableCmd = `sudo networksetup -setnetworkserviceenabled "${serviceName}" off`}
-              <div class="absolute top-4 left-4 group" role="alert">
+            {@render clusterWarnings()}
+
+            <!-- TB5 RDMA Available Info -->
+            {#if tb5WithoutRdma && !tb5InfoDismissed}
+              <div
+                class="absolute left-4 group"
+                class:top-16={tbBridgeCycles.length > 0}
+                class:top-4={tbBridgeCycles.length === 0}
+                role="status"
+              >
                 <div
-                  class="flex items-center gap-2 px-3 py-2 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm cursor-help"
+                  class="flex items-center gap-2 px-3 py-2 rounded border border-blue-400/50 bg-blue-400/10 backdrop-blur-sm"
                 >
                   <svg
-                    class="w-5 h-5 text-yellow-400 flex-shrink-0"
+                    class="w-5 h-5 text-blue-400 flex-shrink-0"
                     fill="none"
                     viewBox="0 0 24 24"
                     stroke="currentColor"
@@ -1862,60 +2176,62 @@
                     <path
                       stroke-linecap="round"
                       stroke-linejoin="round"
-                      d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
+                      d="M13 16h-1v-4h-1m1-4h.01M21 12a9 9 0 11-18 0 9 9 0 0118 0z"
                     />
                   </svg>
-                  <span class="text-sm font-mono text-yellow-200">
-                    THUNDERBOLT BRIDGE CYCLE DETECTED
+                  <span class="text-sm font-mono text-blue-200">
+                    RDMA AVAILABLE
                   </span>
-                </div>
-
-                <!-- Tooltip on hover -->
-                <div
-                  class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-yellow-500/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
-                >
-                  <p class="text-xs text-white/80 mb-2">
-                    A network routing cycle was detected between nodes connected
-                    via Thunderbolt Bridge. This can cause connectivity issues.
-                  </p>
-                  <p class="text-xs text-white/60 mb-2">
-                    <span class="text-yellow-300">Affected nodes:</span>
-                    {cycle.map(getNodeName).join(" → ")}
-                  </p>
-                  <p class="text-xs text-white/60 mb-1">
-                    <span class="text-yellow-300">To fix:</span> Disable the Thunderbolt
-                    Bridge on one of the affected nodes:
-                  </p>
                   <button
                     type="button"
-                    onclick={() => copyToClipboard(disableCmd)}
-                    class="w-full flex items-center gap-2 text-[10px] font-mono bg-exo-black/60 px-2 py-1.5 rounded text-exo-yellow break-all text-left hover:bg-exo-black/80 transition-colors cursor-pointer group/copy"
-                    title="Click to copy"
+                    onclick={() => (tb5InfoDismissed = true)}
+                    class="ml-1 text-blue-300/60 hover:text-blue-200 transition-colors cursor-pointer"
+                    title="Dismiss"
                   >
-                    <span class="flex-1">{disableCmd}</span>
                     <svg
-                      class="w-3.5 h-3.5 flex-shrink-0 text-white/40 group-hover/copy:text-exo-yellow transition-colors"
+                      class="w-4 h-4"
                       fill="none"
                       viewBox="0 0 24 24"
                       stroke="currentColor"
                       stroke-width="2"
                     >
-                      {#if copiedCommand}
-                        <path
-                          stroke-linecap="round"
-                          stroke-linejoin="round"
-                          d="M5 13l4 4L19 7"
-                        />
-                      {:else}
-                        <path
-                          stroke-linecap="round"
-                          stroke-linejoin="round"
-                          d="M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z"
-                        />
-                      {/if}
+                      <path
+                        stroke-linecap="round"
+                        stroke-linejoin="round"
+                        d="M6 18L18 6M6 6l12 12"
+                      />
                     </svg>
                   </button>
                 </div>
+
+                <!-- Tooltip on hover -->
+                <div
+                  class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-blue-400/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
+                >
+                  <p class="text-xs text-white/80 mb-2">
+                    Thunderbolt 5 hardware detected on multiple nodes. Enable
+                    RDMA for significantly faster inter-node communication.
+                  </p>
+                  <p class="text-xs text-white/60 mb-1.5">
+                    <span class="text-blue-300">To enable:</span>
+                  </p>
+                  <ol
+                    class="text-xs text-white/60 list-decimal list-inside space-y-0.5 mb-1.5"
+                  >
+                    <li>Connect nodes with TB5 cables</li>
+                    <li>Boot to Recovery (hold power 10s → Options)</li>
+                    <li>
+                      Run
+                      <code class="text-blue-300 bg-blue-400/10 px-1 rounded"
+                        >rdma_ctl enable</code
+                      >
+                    </li>
+                    <li>Reboot</li>
+                  </ol>
+                  <p class="text-xs text-white/40">
+                    Requires macOS 26.2+, TB5 cables, and matching OS versions.
+                  </p>
+                </div>
               </div>
             {/if}
 
@@ -2773,30 +3089,7 @@
                   onNodeClick={togglePreviewNodeFilter}
                 />
 
-                <!-- Thunderbolt Bridge Cycle Warning (compact) -->
-                {#if tbBridgeCycles.length > 0}
-                  <div
-                    class="absolute top-2 left-2 flex items-center gap-1.5 px-2 py-1 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm"
-                    title="Thunderbolt Bridge cycle detected - click to view details"
-                  >
-                    <svg
-                      class="w-3.5 h-3.5 text-yellow-400"
-                      fill="none"
-                      viewBox="0 0 24 24"
-                      stroke="currentColor"
-                      stroke-width="2"
-                    >
-                      <path
-                        stroke-linecap="round"
-                        stroke-linejoin="round"
-                        d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
-                      />
-                    </svg>
-                    <span class="text-[10px] font-mono text-yellow-200"
-                      >TB CYCLE</span
-                    >
-                  </div>
-                {/if}
+                {@render clusterWarningsCompact()}
               </div>
             </button>
 
@@ -3252,11 +3545,17 @@
   {models}
   {selectedModelId}
   favorites={favoritesSet}
+  {recentModelIds}
+  hasRecents={showRecentsTab}
   existingModelIds={new Set(models.map((m) => m.id))}
   canModelFit={(modelId) => {
     const model = models.find((m) => m.id === modelId);
     return model ? hasEnoughMemory(model) : false;
   }}
+  getModelFitStatus={(modelId): ModelMemoryFitStatus => {
+    const model = models.find((m) => m.id === modelId);
+    return model ? getModelMemoryFitStatus(model) : "too_large";
+  }}
   onSelect={handleModelPickerSelect}
   onClose={() => (isModelPickerOpen = false)}
   onToggleFavorite={toggleFavorite}

dashboard/src/routes/downloads/+page.svelte

@@ -3,6 +3,7 @@
   import {
     topologyData,
     downloads,
+    nodeDisk,
     type DownloadProgress,
     refreshState,
     lastUpdate as lastUpdateStore,
@@ -37,10 +38,13 @@
     nodeId: string;
     nodeName: string;
     models: ModelEntry[];
+    diskAvailable?: number;
+    diskTotal?: number;
   };
 
   const data = $derived(topologyData());
   const downloadsData = $derived(downloads());
+  const nodeDiskData = $derived(nodeDisk());
 
   function getNodeLabel(nodeId: string): string {
     const node = data?.nodes?.[nodeId];
@@ -327,10 +331,17 @@
           ];
         }
 
+        // Get disk info for this node
+        const diskInfo = nodeDiskData?.[nodeId];
+        const diskAvailable = diskInfo?.available?.inBytes;
+        const diskTotal = diskInfo?.total?.inBytes;
+
         built.push({
           nodeId,
           nodeName: getNodeLabel(nodeId),
           models,
+          diskAvailable,
+          diskTotal,
         });
       }
 
@@ -417,6 +428,14 @@
                 <div class="text-xs text-exo-light-gray font-mono truncate">
                   {node.nodeId}
                 </div>
+                <div class="text-xs text-exo-light-gray font-mono mt-1">
+                  {formatBytes(
+                    node.models
+                      .filter((m) => m.status === "completed")
+                      .reduce((sum, m) => sum + m.totalBytes, 0),
+                  )} models{#if node.diskAvailable != null}
+                    - {formatBytes(node.diskAvailable)} free{/if}
+                </div>
               </div>
               <div
                 class="text-xs font-mono uppercase tracking-wider whitespace-nowrap shrink-0 text-right"
@@ -429,13 +448,6 @@
                     / {node.models.length} models</span
                   >
                 </div>
-                <div class="text-exo-light-gray normal-case tracking-normal">
-                  {formatBytes(
-                    node.models
-                      .filter((m) => m.status === "completed")
-                      .reduce((sum, m) => sum + m.totalBytes, 0),
-                  )} on disk
-                </div>
               </div>
             </div>
 

flake.nix

@@ -83,6 +83,9 @@
           _module.args.pkgs = import inputs.nixpkgs {
             inherit system;
             config.allowUnfreePredicate = pkg: (pkg.pname or "") == "metal-toolchain";
+            overlays = [
+              (import ./nix/apple-sdk-overlay.nix)
+            ];
           };
           treefmt = {
             projectRootFile = "flake.nix";
@@ -112,7 +115,7 @@
           packages = lib.optionalAttrs pkgs.stdenv.hostPlatform.isDarwin (
             let
               uvLock = builtins.fromTOML (builtins.readFile ./uv.lock);
-              mlxPackage = builtins.head (builtins.filter (p: p.name == "mlx") uvLock.package);
+              mlxPackage = builtins.head (builtins.filter (p: p.name == "mlx" && p.source ? git) uvLock.package);
               uvLockMlxVersion = mlxPackage.version;
             in
             {

nix/apple-sdk-overlay.nix

@@ -0,0 +1,18 @@
+# Overlay that builds apple-sdk with a custom versions.json (for SDK 26.2).
+# The upstream nixpkgs package reads versions.json at eval time via a relative
+# path, so we can't override it through callPackage args. Instead, we copy
+# the upstream source and patch the one file.
+final: _prev:
+let
+  upstreamSrc = final.path + "/pkgs/by-name/ap/apple-sdk";
+  patchedSrc = final.runCommandLocal "apple-sdk-src-patched" { } ''
+    cp -r ${upstreamSrc} $out
+    chmod -R u+w $out
+    cp ${./apple-sdk/metadata/versions.json} $out/metadata/versions.json
+  '';
+in
+{
+  apple-sdk_26 = final.callPackage (patchedSrc + "/package.nix") {
+    darwinSdkMajorVersion = "26";
+  };
+}

nix/apple-sdk/metadata/versions.json

@@ -0,0 +1,26 @@
+{
+  "14": {
+    "urls": [
+      "https://swcdn.apple.com/content/downloads/14/48/052-59890-A_I0F5YGAY0Y/p9n40hio7892gou31o1v031ng6fnm9sb3c/CLTools_macOSNMOS_SDK.pkg",
+      "https://web.archive.org/web/20250211001355/https://swcdn.apple.com/content/downloads/14/48/052-59890-A_I0F5YGAY0Y/p9n40hio7892gou31o1v031ng6fnm9sb3c/CLTools_macOSNMOS_SDK.pkg"
+    ],
+    "version": "14.4",
+    "hash": "sha256-QozDiwY0Czc0g45vPD7G4v4Ra+3DujCJbSads3fJjjM="
+  },
+  "15": {
+    "urls": [
+      "https://swcdn.apple.com/content/downloads/52/01/082-41241-A_0747ZN8FHV/dectd075r63pppkkzsb75qk61s0lfee22j/CLTools_macOSNMOS_SDK.pkg",
+      "https://web.archive.org/web/20250530132510/https://swcdn.apple.com/content/downloads/52/01/082-41241-A_0747ZN8FHV/dectd075r63pppkkzsb75qk61s0lfee22j/CLTools_macOSNMOS_SDK.pkg"
+    ],
+    "version": "15.5",
+    "hash": "sha256-HBiSJuw1XBUK5R/8Sj65c3rftSEvQl/O9ZZVp/g1Amo="
+  },
+  "26": {
+    "urls": [
+      "https://swcdn.apple.com/content/downloads/60/22/089-71960-A_W8BL1RUJJ6/5zkyplomhk1cm7z6xja2ktgapnhhti6wwd/CLTools_macOSNMOS_SDK.pkg",
+      "https://web.archive.org/web/20250915230423/https://swcdn.apple.com/content/downloads/60/22/089-71960-A_W8BL1RUJJ6/5zkyplomhk1cm7z6xja2ktgapnhhti6wwd/CLTools_macOSNMOS_SDK.pkg"
+    ],
+    "version": "26.2",
+    "hash": "sha256-hXRlMieVv0smna5uiWRwq87IWOaPWtAjAldbi+wQXcw="
+  }
+}

nix/mlx.nix

@@ -41,16 +41,16 @@ let
 
   mlx = stdenv.mkDerivation rec {
     pname = "mlx";
-    version = let v = "0.30.5"; in
+    version = let v = "0.30.7.dev20260216+d94f81a2"; in
       assert v == uvLockMlxVersion || throw "MLX version mismatch: nix/mlx.nix has ${v} but uv.lock has ${uvLockMlxVersion}. Update both the version and hash in nix/mlx.nix.";
       v;
     pyproject = true;
 
     src = fetchFromGitHub {
-      owner = "ml-explore";
-      repo = "mlx";
-      tag = "v${version}";
-      hash = "sha256-OJk6jPlbaSlsUdk3ADz3tWcRzTWXRof3/q8Soe1AO6w=";
+      owner = "rltakashige";
+      repo = "mlx-jaccl-fix-small-recv";
+      rev = "d94f81a2ce7346f56567a17eb08c4df5a602ed40";
+      hash = "sha256-7Qp3uwMAW+izJ70Jz5ZP1P+4aDiiB+/6nG8QofshzLg=";
     };
 
     patches = [
@@ -85,6 +85,7 @@ let
         (lib.cmakeOptionType "filepath" "FETCHCONTENT_SOURCE_DIR_JSON" "${nlohmann_json.src}")
         (lib.cmakeOptionType "filepath" "FETCHCONTENT_SOURCE_DIR_NANOBIND" "${nanobind}")
         (lib.cmakeBool "FETCHCONTENT_FULLY_DISCONNECTED" true)
+        (lib.cmakeBool "MLX_BUILD_CPU" true)
         (lib.cmakeBool "MLX_BUILD_METAL" true)
         (lib.cmakeOptionType "filepath" "FETCHCONTENT_SOURCE_DIR_METAL_CPP" "${metal_cpp}")
         (lib.cmakeOptionType "string" "CMAKE_OSX_DEPLOYMENT_TARGET" "${apple-sdk_26.version}")

pyproject.toml

@@ -17,8 +17,8 @@ dependencies = [
     "loguru>=0.7.3",
     "exo_pyo3_bindings", # rust bindings
     "anyio==4.11.0",
-    "mlx==0.30.5; sys_platform == 'darwin'",
-    "mlx[cpu]==0.30.5; sys_platform == 'linux'",
+    "mlx; sys_platform == 'darwin'",
+    "mlx[cpu]==0.30.6; sys_platform == 'linux'",
     "mlx-lm==0.30.6",
     "tiktoken>=0.12.0", # required for kimi k2 tokenizer
     "hypercorn>=0.18.0",
@@ -28,6 +28,8 @@ dependencies = [
     "pillow>=11.0,<12.0", # compatibility with mflux
     "mflux==0.15.5",
     "python-multipart>=0.0.21",
+    "msgspec>=0.19.0",
+    "zstandard>=0.23.0",
 ]
 
 [project.scripts]
@@ -57,10 +59,12 @@ dev = [
 [tool.uv.workspace]
 members = [
     "rust/exo_pyo3_bindings",
+    "bench",
 ]
 
 [tool.uv.sources]
 exo_pyo3_bindings = { workspace = true }
+mlx = { git = "https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git", branch = "address-rdma-gpu-locks", marker = "sys_platform == 'darwin'" }
 #mlx-lm = { git = "https://github.com/davidmcc73/mlx-lm", branch = "stable" }
 # Uncomment to use local mlx/mlx-lm development versions:
 # mlx = { path = "/Users/Shared/mlx", editable=true }

python/parts.nix

@@ -69,6 +69,16 @@
         text = ''exec python ${path} "$@"'';
       };
 
+      benchVenv = pythonSet.mkVirtualEnv "exo-bench-env" {
+        exo-bench = [ ];
+      };
+
+      mkBenchScript = name: path: pkgs.writeShellApplication {
+        inherit name;
+        runtimeInputs = [ benchVenv ];
+        text = ''exec python ${path} "$@"'';
+      };
+
       mkSimplePythonScript = name: path: pkgs.writeShellApplication {
         inherit name;
         runtimeInputs = [ pkgs.python313 ];
@@ -96,8 +106,8 @@
           exo = exoPackage;
           # Test environment for running pytest outside of Nix sandbox (needs GPU access)
           exo-test-env = testVenv;
-          exo-bench = mkPythonScript "exo-bench" (inputs.self + /bench/exo_bench.py);
         } // {
+        exo-bench = mkBenchScript "exo-bench" (inputs.self + /bench/exo_bench.py);
         exo-get-all-models-on-cluster = mkSimplePythonScript "exo-get-all-models-on-cluster" (inputs.self + /tests/get_all_models_on_cluster.py);
       };
 

resources/image_model_cards/exolabs--FLUX.1-Kontext-dev-4bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["ImageToImage"]
+family = "flux"
+quantization = "4bit"
+base_model = "FLUX.1 Kontext"
+capabilities = ["image_edit"]
 
 [storage_size]
 in_bytes = 15475325472

resources/image_model_cards/exolabs--FLUX.1-Kontext-dev-8bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["ImageToImage"]
+family = "flux"
+quantization = "8bit"
+base_model = "FLUX.1 Kontext"
+capabilities = ["image_edit"]
 
 [storage_size]
 in_bytes = 21426029632

resources/image_model_cards/exolabs--FLUX.1-Kontext-dev.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["ImageToImage"]
+family = "flux"
+quantization = ""
+base_model = "FLUX.1 Kontext"
+capabilities = ["image_edit"]
 
 [storage_size]
 in_bytes = 33327437952

resources/image_model_cards/exolabs--FLUX.1-Krea-dev-4bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = "4bit"
+base_model = "FLUX.1 Krea"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 15475325472

resources/image_model_cards/exolabs--FLUX.1-Krea-dev-8bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = "8bit"
+base_model = "FLUX.1 Krea"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 21426029632

resources/image_model_cards/exolabs--FLUX.1-Krea-dev.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = ""
+base_model = "FLUX.1 Krea"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 33327437952

resources/image_model_cards/exolabs--FLUX.1-dev-4bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = "4bit"
+base_model = "FLUX.1 Dev"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 15475325472

resources/image_model_cards/exolabs--FLUX.1-dev-8bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = "8bit"
+base_model = "FLUX.1 Dev"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 21426029632

resources/image_model_cards/exolabs--FLUX.1-dev.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = ""
+base_model = "FLUX.1 Dev"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 33327437952

resources/image_model_cards/exolabs--FLUX.1-schnell-4bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = "4bit"
+base_model = "FLUX.1 Schnell"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 15470210592

resources/image_model_cards/exolabs--FLUX.1-schnell-8bit.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = "8bit"
+base_model = "FLUX.1 Schnell"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 21415799872

resources/image_model_cards/exolabs--FLUX.1-schnell.toml

@@ -3,6 +3,10 @@ n_layers = 57
 hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
+family = "flux"
+quantization = ""
+base_model = "FLUX.1 Schnell"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 33306978432

resources/image_model_cards/exolabs--Qwen-Image-4bit.toml

@@ -4,6 +4,10 @@ hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
 uses_cfg = true
+family = "qwen-image"
+quantization = "4bit"
+base_model = "Qwen Image"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 26799533856

resources/image_model_cards/exolabs--Qwen-Image-8bit.toml

@@ -4,6 +4,10 @@ hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
 uses_cfg = true
+family = "qwen-image"
+quantization = "8bit"
+base_model = "Qwen Image"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 37014734400

resources/image_model_cards/exolabs--Qwen-Image-Edit-2509-4bit.toml

@@ -4,6 +4,10 @@ hidden_size = 1
 supports_tensor = false
 tasks = ["ImageToImage"]
 uses_cfg = true
+family = "qwen-image"
+quantization = "4bit"
+base_model = "Qwen Image Edit"
+capabilities = ["image_edit"]
 
 [storage_size]
 in_bytes = 26799533856

resources/image_model_cards/exolabs--Qwen-Image-Edit-2509-8bit.toml

@@ -4,6 +4,10 @@ hidden_size = 1
 supports_tensor = false
 tasks = ["ImageToImage"]
 uses_cfg = true
+family = "qwen-image"
+quantization = "8bit"
+base_model = "Qwen Image Edit"
+capabilities = ["image_edit"]
 
 [storage_size]
 in_bytes = 37014734400

resources/image_model_cards/exolabs--Qwen-Image-Edit-2509.toml

@@ -4,6 +4,10 @@ hidden_size = 1
 supports_tensor = false
 tasks = ["ImageToImage"]
 uses_cfg = true
+family = "qwen-image"
+quantization = ""
+base_model = "Qwen Image Edit"
+capabilities = ["image_edit"]
 
 [storage_size]
 in_bytes = 57445135488

resources/image_model_cards/exolabs--Qwen-Image.toml

@@ -4,6 +4,10 @@ hidden_size = 1
 supports_tensor = false
 tasks = ["TextToImage"]
 uses_cfg = true
+family = "qwen-image"
+quantization = ""
+base_model = "Qwen Image"
+capabilities = ["image_gen"]
 
 [storage_size]
 in_bytes = 57445135488

rust/exo_pyo3_bindings/src/networking.rs

@@ -22,7 +22,6 @@ use pyo3::{Bound, Py, PyErr, PyResult, PyTraverseError, PyVisit, Python, pymetho
 use pyo3_stub_gen::derive::{gen_stub_pyclass, gen_stub_pyclass_enum, gen_stub_pymethods};
 use std::net::IpAddr;
 use tokio::sync::{Mutex, mpsc, oneshot};
-use util::ext::VecExt as _;
 
 mod exception {
     use pyo3::types::PyTuple;
@@ -532,7 +531,9 @@ impl PyNetworkingHandle {
             .recv_many_py(limit)
             .allow_threads_py() // allow-threads-aware async call
             .await?
-            .map(|(t, d)| (t, d.pybytes())))
+            .into_iter()
+            .map(|(t, d)| (t, d.pybytes()))
+            .collect())
     }
 
     // TODO: rn this blocks main thread if anything else is awaiting the channel (bc its a mutex)

rust/networking/src/swarm.rs

@@ -106,7 +106,6 @@ mod behaviour {
     use crate::{alias, discovery};
     use libp2p::swarm::NetworkBehaviour;
     use libp2p::{gossipsub, identity};
-    use std::time::Duration;
 
     /// Behavior of the Swarm which composes all desired behaviors:
     /// Right now its just [`discovery::Behaviour`] and [`gossipsub::Behaviour`].
@@ -134,7 +133,6 @@ mod behaviour {
         gossipsub::Behaviour::new(
             MessageAuthenticity::Signed(keypair.clone()),
             ConfigBuilder::default()
-                .publish_queue_duration(Duration::from_secs(15))
                 .max_transmit_size(1024 * 1024)
                 .validation_mode(ValidationMode::Strict)
                 .build()

rust/util/Cargo.toml

@@ -13,13 +13,3 @@ path = "src/lib.rs"
 workspace = true
 
 [dependencies]
-# macro dependencies
-extend = { workspace = true }
-
-# utility dependencies
-thiserror = { workspace = true }
-once_cell = { workspace = true }
-internment = { workspace = true }
-derive_more = { workspace = true }
-bon = { workspace = true }
-recursion = { workspace = true }

rust/util/src/lib.rs

@@ -1,53 +1 @@
-//! TODO: crate documentation
-//!
-//! this is here as a placeholder documentation
-//!
-//!
-
-// enable Rust-unstable features for convenience
-#![feature(trait_alias)]
-#![feature(stmt_expr_attributes)]
-#![feature(type_alias_impl_trait)]
-#![feature(specialization)]
-#![feature(unboxed_closures)]
-#![feature(const_trait_impl)]
-#![feature(fn_traits)]
-
-pub mod nonempty;
 pub mod wakerdeque;
-
-pub(crate) mod private {
-    // sealed traits support
-    pub trait Sealed {}
-    impl<T: ?Sized> Sealed for T {}
-}
-
-/// Namespace for all the type/trait aliases used by this crate.
-pub(crate) mod alias {}
-
-/// Namespace for crate-wide extension traits/methods
-pub mod ext {
-    use extend::ext;
-
-    #[ext(pub, name = BoxedSliceExt)]
-    impl<T> Box<[T]> {
-        #[inline]
-        fn map<B, F>(self, f: F) -> Box<[B]>
-        where
-            F: FnMut(T) -> B,
-        {
-            self.into_iter().map(f).collect()
-        }
-    }
-
-    #[ext(pub, name = VecExt)]
-    impl<T> Vec<T> {
-        #[inline]
-        fn map<B, F>(self, f: F) -> Vec<B>
-        where
-            F: FnMut(T) -> B,
-        {
-            self.into_iter().map(f).collect()
-        }
-    }
-}

rust/util/src/nonempty.rs

@@ -1,138 +0,0 @@
-use std::slice::SliceIndex;
-use std::{ops, slice};
-use thiserror::Error;
-
-#[derive(Error, Debug)]
-#[error("Cannot create to `NonemptyArray` because the supplied slice is empty")]
-pub struct EmptySliceError;
-
-/// A pointer to a non-empty fixed-size slice allocated on the heap.
-#[derive(Clone, Debug, PartialEq, Eq, Hash, PartialOrd, Ord)]
-#[repr(transparent)]
-pub struct NonemptyArray<T>(Box<[T]>);
-
-#[allow(clippy::arbitrary_source_item_ordering)]
-impl<T> NonemptyArray<T> {
-    #[inline]
-    pub fn singleton(value: T) -> Self {
-        Self(Box::new([value]))
-    }
-
-    #[allow(clippy::missing_errors_doc)]
-    #[inline]
-    pub fn try_from_boxed_slice<S: Into<Box<[T]>>>(
-        boxed_slice: S,
-    ) -> Result<Self, EmptySliceError> {
-        let boxed_slice = boxed_slice.into();
-        if boxed_slice.is_empty() {
-            Err(EmptySliceError)
-        } else {
-            Ok(Self(boxed_slice))
-        }
-    }
-
-    #[must_use]
-    #[inline]
-    pub fn into_boxed_slice(self) -> Box<[T]> {
-        self.0
-    }
-
-    #[must_use]
-    #[inline]
-    pub fn to_vec(&self) -> Vec<T>
-    where
-        T: Clone,
-    {
-        self.0.to_vec()
-    }
-
-    #[must_use]
-    #[inline]
-    pub const fn as_slice(&self) -> &[T] {
-        &self.0
-    }
-
-    #[allow(clippy::indexing_slicing)]
-    #[must_use]
-    #[inline]
-    pub fn first(&self) -> &T {
-        &self.0[0]
-    }
-
-    #[allow(clippy::indexing_slicing, clippy::arithmetic_side_effects)]
-    #[must_use]
-    #[inline]
-    pub fn last(&self) -> &T {
-        &self.0[self.0.len() - 1]
-    }
-
-    #[must_use]
-    #[inline]
-    pub fn get<I>(&self, index: I) -> Option<&I::Output>
-    where
-        I: SliceIndex<[T]>,
-    {
-        self.0.get(index)
-    }
-
-    #[allow(clippy::len_without_is_empty)]
-    #[must_use]
-    #[inline]
-    pub const fn len(&self) -> usize {
-        self.0.len()
-    }
-
-    #[allow(clippy::iter_without_into_iter)]
-    #[inline]
-    pub fn iter(&self) -> slice::Iter<'_, T> {
-        self.0.iter()
-    }
-
-    #[allow(clippy::iter_without_into_iter)]
-    #[inline]
-    pub fn iter_mut(&mut self) -> slice::IterMut<'_, T> {
-        self.0.iter_mut()
-    }
-
-    #[inline]
-    #[must_use]
-    pub fn map<U, F: FnMut(T) -> U>(self, f: F) -> NonemptyArray<U> {
-        NonemptyArray(self.0.into_iter().map(f).collect())
-    }
-}
-
-impl<T> From<NonemptyArray<T>> for Box<[T]> {
-    #[inline]
-    fn from(value: NonemptyArray<T>) -> Self {
-        value.into_boxed_slice()
-    }
-}
-
-impl<T> ops::Index<usize> for NonemptyArray<T> {
-    type Output = T;
-
-    #[inline]
-    fn index(&self, index: usize) -> &Self::Output {
-        self.0.index(index)
-    }
-}
-
-impl<T> IntoIterator for NonemptyArray<T> {
-    type Item = T;
-    type IntoIter = std::vec::IntoIter<T>;
-
-    #[inline]
-    fn into_iter(self) -> Self::IntoIter {
-        self.into_boxed_slice().into_vec().into_iter()
-    }
-}
-
-impl<'a, T> IntoIterator for &'a NonemptyArray<T> {
-    type Item = &'a T;
-    type IntoIter = slice::Iter<'a, T>;
-
-    #[inline]
-    fn into_iter(self) -> Self::IntoIter {
-        self.iter()
-    }
-}

src/exo/download/coordinator.py

@@ -56,8 +56,49 @@ class DownloadCoordinator:
     event_receiver: Receiver[Event] = field(init=False)
     _tg: TaskGroup = field(init=False, default_factory=anyio.create_task_group)
 
+    # Per-model throttle for download progress events
+    _last_progress_time: dict[ModelId, float] = field(default_factory=dict)
+
     def __post_init__(self) -> None:
         self.event_sender, self.event_receiver = channel[Event]()
+        self.shard_downloader.on_progress(self._download_progress_callback)
+
+    async def _download_progress_callback(
+        self, callback_shard: ShardMetadata, progress: RepoDownloadProgress
+    ) -> None:
+        model_id = callback_shard.model_card.model_id
+        throttle_interval_secs = 1.0
+
+        if progress.status == "complete":
+            completed = DownloadCompleted(
+                shard_metadata=callback_shard,
+                node_id=self.node_id,
+                total_bytes=progress.total_bytes,
+            )
+            self.download_status[model_id] = completed
+            await self.event_sender.send(
+                NodeDownloadProgress(download_progress=completed)
+            )
+            if model_id in self.active_downloads:
+                del self.active_downloads[model_id]
+            self._last_progress_time.pop(model_id, None)
+        elif (
+            progress.status == "in_progress"
+            and current_time() - self._last_progress_time.get(model_id, 0.0)
+            > throttle_interval_secs
+        ):
+            ongoing = DownloadOngoing(
+                node_id=self.node_id,
+                shard_metadata=callback_shard,
+                download_progress=map_repo_download_progress_to_download_progress_data(
+                    progress
+                ),
+            )
+            self.download_status[model_id] = ongoing
+            await self.event_sender.send(
+                NodeDownloadProgress(download_progress=ongoing)
+            )
+            self._last_progress_time[model_id] = current_time()
 
     async def run(self) -> None:
         logger.info("Starting DownloadCoordinator")
@@ -119,12 +160,12 @@ class DownloadCoordinator:
     async def _start_download(self, shard: ShardMetadata) -> None:
         model_id = shard.model_card.model_id
 
-        # Check if already downloading or complete
+        # Check if already downloading, complete, or recently failed
         if model_id in self.download_status:
             status = self.download_status[model_id]
-            if isinstance(status, (DownloadOngoing, DownloadCompleted)):
+            if isinstance(status, (DownloadOngoing, DownloadCompleted, DownloadFailed)):
                 logger.debug(
-                    f"Download for {model_id} already in progress or complete, skipping"
+                    f"Download for {model_id} already in progress, complete, or failed, skipping"
                 )
                 return
 
@@ -169,46 +210,6 @@ class DownloadCoordinator:
         self.download_status[model_id] = status
         self.event_sender.send_nowait(NodeDownloadProgress(download_progress=status))
 
-        last_progress_time = 0.0
-        throttle_interval_secs = 1.0
-
-        async def download_progress_callback(
-            callback_shard: ShardMetadata, progress: RepoDownloadProgress
-        ) -> None:
-            nonlocal last_progress_time
-
-            if progress.status == "complete":
-                completed = DownloadCompleted(
-                    shard_metadata=callback_shard,
-                    node_id=self.node_id,
-                    total_bytes=progress.total_bytes,
-                )
-                self.download_status[callback_shard.model_card.model_id] = completed
-                await self.event_sender.send(
-                    NodeDownloadProgress(download_progress=completed)
-                )
-                # Clean up active download tracking
-                if callback_shard.model_card.model_id in self.active_downloads:
-                    del self.active_downloads[callback_shard.model_card.model_id]
-            elif (
-                progress.status == "in_progress"
-                and current_time() - last_progress_time > throttle_interval_secs
-            ):
-                ongoing = DownloadOngoing(
-                    node_id=self.node_id,
-                    shard_metadata=callback_shard,
-                    download_progress=map_repo_download_progress_to_download_progress_data(
-                        progress
-                    ),
-                )
-                self.download_status[callback_shard.model_card.model_id] = ongoing
-                await self.event_sender.send(
-                    NodeDownloadProgress(download_progress=ongoing)
-                )
-                last_progress_time = current_time()
-
-        self.shard_downloader.on_progress(download_progress_callback)
-
         async def download_wrapper() -> None:
             try:
                 await self.shard_downloader.ensure_shard(shard)
@@ -283,6 +284,12 @@ class DownloadCoordinator:
                     _,
                     progress,
                 ) in self.shard_downloader.get_shard_download_status():
+                    model_id = progress.shard.model_card.model_id
+
+                    # Active downloads emit progress via the callback — don't overwrite
+                    if model_id in self.active_downloads:
+                        continue
+
                     if progress.status == "complete":
                         status: DownloadProgress = DownloadCompleted(
                             node_id=self.node_id,

src/exo/download/download_utils.py

@@ -457,6 +457,8 @@ async def download_file_with_retry(
             )
         except HuggingFaceAuthenticationError:
             raise
+        except FileNotFoundError:
+            raise
         except HuggingFaceRateLimitError as e:
             if attempt == n_attempts - 1:
                 raise e

src/exo/main.py

@@ -251,7 +251,8 @@ class Node:
 def main():
     args = Args.parse()
     soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
-    resource.setrlimit(resource.RLIMIT_NOFILE, (max(soft, 65535), hard))
+    target = min(max(soft, 65535), hard)
+    resource.setrlimit(resource.RLIMIT_NOFILE, (target, hard))
 
     mp.set_start_method("spawn")
     # TODO: Refactor the current verbosity system

src/exo/master/adapters/chat_completions.py

@@ -3,7 +3,6 @@
 import time
 from collections.abc import AsyncGenerator
 from typing import Any
-from uuid import uuid4
 
 from exo.shared.types.api import (
     ChatCompletionChoice,
@@ -141,7 +140,7 @@ async def generate_chat_stream(
         if isinstance(chunk, ToolCallChunk):
             tool_call_deltas = [
                 ToolCall(
-                    id=str(uuid4()),
+                    id=tool.id,
                     index=i,
                     function=tool,
                 )
@@ -207,7 +206,7 @@ async def collect_chat_response(
         if isinstance(chunk, ToolCallChunk):
             tool_calls.extend(
                 ToolCall(
-                    id=str(uuid4()),
+                    id=tool.id,
                     index=i,
                     function=tool,
                 )

src/exo/master/adapters/claude.py

@@ -3,7 +3,6 @@
 import json
 from collections.abc import AsyncGenerator
 from typing import Any
-from uuid import uuid4
 
 from exo.shared.types.api import FinishReason
 from exo.shared.types.chunks import ErrorChunk, TokenChunk, ToolCallChunk
@@ -179,7 +178,7 @@ async def collect_claude_response(
             for tool in chunk.tool_calls:
                 tool_use_blocks.append(
                     ClaudeToolUseBlock(
-                        id=f"toolu_{uuid4().hex[:24]}",
+                        id=f"toolu_{tool.id}",
                         name=tool.name,
                         input=json.loads(tool.arguments),  # pyright: ignore[reportAny]
                     )
@@ -264,7 +263,7 @@ async def generate_claude_stream(
 
             # Emit tool_use content blocks
             for tool in chunk.tool_calls:
-                tool_id = f"toolu_{uuid4().hex[:24]}"
+                tool_id = f"toolu_{tool.id}"
                 tool_input_json = tool.arguments
 
                 # content_block_start for tool_use

src/exo/master/adapters/responses.py

@@ -3,7 +3,6 @@
 from collections.abc import AsyncGenerator
 from itertools import count
 from typing import Any
-from uuid import uuid4
 
 from exo.shared.types.chunks import ErrorChunk, TokenChunk, ToolCallChunk
 from exo.shared.types.common import CommandId
@@ -140,8 +139,8 @@ async def collect_responses_response(
             for tool in chunk.tool_calls:
                 function_call_items.append(
                     ResponseFunctionCallItem(
-                        id=f"fc_{uuid4().hex[:24]}",
-                        call_id=f"call_{uuid4().hex[:24]}",
+                        id=f"fc_{tool.id}",
+                        call_id=f"call_{tool.id}",
                         name=tool.name,
                         arguments=tool.arguments,
                     )
@@ -246,8 +245,8 @@ async def generate_responses_stream(
         if isinstance(chunk, ToolCallChunk):
             last_stats = chunk.stats or last_stats
             for tool in chunk.tool_calls:
-                fc_id = f"fc_{uuid4().hex[:24]}"
-                call_id = f"call_{uuid4().hex[:24]}"
+                fc_id = f"fc_{tool.id}"
+                call_id = f"call_{tool.id}"
 
                 # response.output_item.added for function_call
                 fc_item = ResponseFunctionCallItem(

src/exo/master/api.py

@@ -3,7 +3,7 @@ import contextlib
 import json
 import random
 import time
-from collections.abc import AsyncGenerator, Awaitable, Callable
+from collections.abc import AsyncGenerator, Awaitable, Callable, Iterator
 from datetime import datetime, timezone
 from http import HTTPStatus
 from pathlib import Path
@@ -37,11 +37,13 @@ from exo.master.adapters.responses import (
     generate_responses_stream,
     responses_request_to_text_generation,
 )
+from exo.master.event_log import DiskEventLog
 from exo.master.image_store import ImageStore
 from exo.master.placement import place_instance as get_instance_placements
 from exo.shared.apply import apply
 from exo.shared.constants import (
     DASHBOARD_DIR,
+    EXO_EVENT_LOG_DIR,
     EXO_IMAGE_CACHE_DIR,
     EXO_MAX_CHUNK_SIZE,
     EXO_TRACING_CACHE_DIR,
@@ -146,6 +148,8 @@ from exo.utils.banner import print_startup_banner
 from exo.utils.channels import Receiver, Sender, channel
 from exo.utils.event_buffer import OrderedBuffer
 
+_API_EVENT_LOG_DIR = EXO_EVENT_LOG_DIR / "api"
+
 
 def _format_to_content_type(image_format: Literal["png", "jpeg", "webp"] | None) -> str:
     return f"image/{image_format or 'png'}"
@@ -175,7 +179,7 @@ class API:
         election_receiver: Receiver[ElectionMessage],
     ) -> None:
         self.state = State()
-        self._event_log: list[Event] = []
+        self._event_log = DiskEventLog(_API_EVENT_LOG_DIR)
         self.command_sender = command_sender
         self.download_command_sender = download_command_sender
         self.global_event_receiver = global_event_receiver
@@ -223,6 +227,8 @@ class API:
 
     def reset(self, new_session_id: SessionId, result_clock: int):
         logger.info("Resetting API State")
+        self._event_log.close()
+        self._event_log = DiskEventLog(_API_EVENT_LOG_DIR)
         self.state = State()
         self.session_id = new_session_id
         self.event_buffer = OrderedBuffer[Event]()
@@ -289,7 +295,7 @@ class API:
         self.app.post("/v1/messages", response_model=None)(self.claude_messages)
         self.app.post("/v1/responses", response_model=None)(self.openai_responses)
         self.app.get("/state")(lambda: self.state)
-        self.app.get("/events")(lambda: self._event_log)
+        self.app.get("/events")(self.stream_events)
         self.app.post("/download/start")(self.start_download)
         self.app.delete("/download/{node_id}/{model_id:path}")(self.delete_download)
         self.app.get("/v1/traces")(self.list_traces)
@@ -386,7 +392,12 @@ class API:
         if len(list(self.state.topology.list_nodes())) == 0:
             return PlacementPreviewResponse(previews=[])
 
-        model_card = await ModelCard.load(model_id)
+        try:
+            model_card = await ModelCard.load(model_id)
+        except Exception as exc:
+            raise HTTPException(
+                status_code=400, detail=f"Failed to load model card: {exc}"
+            ) from exc
         instance_combinations: list[tuple[Sharding, InstanceMeta, int]] = []
         for sharding in (Sharding.Pipeline, Sharding.Tensor):
             for instance_meta in (InstanceMeta.MlxRing, InstanceMeta.MlxJaccl):
@@ -689,6 +700,22 @@ class API:
             )
         return resolved_model
 
+    def stream_events(self) -> StreamingResponse:
+        def _generate_json_array(events: Iterator[Event]) -> Iterator[str]:
+            yield "["
+            first = True
+            for event in events:
+                if not first:
+                    yield ","
+                first = False
+                yield event.model_dump_json()
+            yield "]"
+
+        return StreamingResponse(
+            _generate_json_array(self._event_log.read_all()),
+            media_type="application/json",
+        )
+
     async def get_image(self, image_id: str) -> FileResponse:
         stored = self._image_store.get(Id(image_id))
         if stored is None:
@@ -1337,6 +1364,7 @@ class API:
                     with anyio.CancelScope(shield=True):
                         shutdown_ev.set()
         finally:
+            self._event_log.close()
             self.command_sender.close()
             self.global_event_receiver.close()
 

src/exo/master/event_log.py

@@ -0,0 +1,187 @@
+import contextlib
+import json
+from collections import OrderedDict
+from collections.abc import Iterator
+from datetime import datetime, timezone
+from io import BufferedRandom, BufferedReader
+from pathlib import Path
+
+import msgspec
+import zstandard
+from loguru import logger
+from pydantic import TypeAdapter
+
+from exo.shared.types.events import Event
+
+_EVENT_ADAPTER: TypeAdapter[Event] = TypeAdapter(Event)
+
+_HEADER_SIZE = 4  # uint32 big-endian
+_OFFSET_CACHE_SIZE = 128
+_MAX_ARCHIVES = 5
+
+
+def _serialize_event(event: Event) -> bytes:
+    return msgspec.msgpack.encode(event.model_dump(mode="json"))
+
+
+def _deserialize_event(raw: bytes) -> Event:
+    # Decode msgpack into a Python dict, then re-encode as JSON for Pydantic.
+    # Pydantic's validate_json() uses JSON-mode coercion (e.g. string -> enum)
+    # even under strict=True, whereas validate_python() does not. Going through
+    # JSON is the only way to get correct round-trip deserialization without
+    # disabling strict mode or adding casts everywhere.
+    as_json = json.dumps(msgspec.msgpack.decode(raw, type=dict))
+    return _EVENT_ADAPTER.validate_json(as_json)
+
+
+def _unpack_header(header: bytes) -> int:
+    return int.from_bytes(header, byteorder="big")
+
+
+def _skip_record(f: BufferedReader) -> bool:
+    """Skip one length-prefixed record. Returns False on EOF."""
+    header = f.read(_HEADER_SIZE)
+    if len(header) < _HEADER_SIZE:
+        return False
+    f.seek(_unpack_header(header), 1)
+    return True
+
+
+def _read_record(f: BufferedReader) -> Event | None:
+    """Read one length-prefixed record. Returns None on EOF."""
+    header = f.read(_HEADER_SIZE)
+    if len(header) < _HEADER_SIZE:
+        return None
+    length = _unpack_header(header)
+    payload = f.read(length)
+    if len(payload) < length:
+        return None
+    return _deserialize_event(payload)
+
+
+class DiskEventLog:
+    """Append-only event log backed by a file on disk.
+
+    On-disk format: sequence of length-prefixed msgpack records.
+    Each record is [4-byte big-endian uint32 length][msgpack payload].
+
+    Uses a bounded LRU cache of event index → byte offset for efficient
+    random access without storing an offset per event.
+    """
+
+    def __init__(self, directory: Path) -> None:
+        self._directory = directory
+        self._directory.mkdir(parents=True, exist_ok=True)
+        self._active_path = directory / "events.bin"
+        self._offset_cache: OrderedDict[int, int] = OrderedDict()
+        self._count: int = 0
+
+        # Rotate stale active file from a previous session/crash
+        if self._active_path.exists():
+            self._rotate(self._active_path, self._directory)
+
+        self._file: BufferedRandom = open(self._active_path, "w+b")  # noqa: SIM115
+
+    def _cache_offset(self, idx: int, offset: int) -> None:
+        self._offset_cache[idx] = offset
+        self._offset_cache.move_to_end(idx)
+        if len(self._offset_cache) > _OFFSET_CACHE_SIZE:
+            self._offset_cache.popitem(last=False)
+
+    def _seek_to(self, f: BufferedReader, target_idx: int) -> None:
+        """Seek f to the byte offset of event target_idx, using cache or scanning forward."""
+        if target_idx in self._offset_cache:
+            self._offset_cache.move_to_end(target_idx)
+            f.seek(self._offset_cache[target_idx])
+            return
+
+        # Find the highest cached index before target_idx
+        scan_from_idx = 0
+        scan_from_offset = 0
+        for cached_idx in self._offset_cache:
+            if cached_idx < target_idx:
+                scan_from_idx = cached_idx
+                scan_from_offset = self._offset_cache[cached_idx]
+
+        # Scan forward, skipping records
+        f.seek(scan_from_offset)
+        for _ in range(scan_from_idx, target_idx):
+            _skip_record(f)
+
+        self._cache_offset(target_idx, f.tell())
+
+    def append(self, event: Event) -> None:
+        packed = _serialize_event(event)
+        self._file.write(len(packed).to_bytes(_HEADER_SIZE, byteorder="big"))
+        self._file.write(packed)
+        self._count += 1
+
+    def read_range(self, start: int, end: int) -> Iterator[Event]:
+        """Yield events from index start (inclusive) to end (exclusive)."""
+        end = min(end, self._count)
+        if start < 0 or end < 0 or start >= end:
+            return
+
+        self._file.flush()
+        with open(self._active_path, "rb") as f:
+            self._seek_to(f, start)
+            for _ in range(end - start):
+                event = _read_record(f)
+                if event is None:
+                    break
+                yield event
+
+            # Cache where we ended up so the next sequential read is a hit
+            if end < self._count:
+                self._cache_offset(end, f.tell())
+
+    def read_all(self) -> Iterator[Event]:
+        """Yield all events from the log one at a time."""
+        if self._count == 0:
+            return
+        self._file.flush()
+        with open(self._active_path, "rb") as f:
+            for _ in range(self._count):
+                event = _read_record(f)
+                if event is None:
+                    break
+                yield event
+
+    def __len__(self) -> int:
+        return self._count
+
+    def close(self) -> None:
+        """Close the file and rotate active file to compressed archive."""
+        if self._file.closed:
+            return
+        self._file.close()
+        if self._active_path.exists() and self._count > 0:
+            self._rotate(self._active_path, self._directory)
+        elif self._active_path.exists():
+            self._active_path.unlink()
+
+    @staticmethod
+    def _rotate(source: Path, directory: Path) -> None:
+        """Compress source into a timestamped archive.
+
+        Keeps at most ``_MAX_ARCHIVES`` compressed copies.  Oldest beyond
+        the limit are deleted.
+        """
+        try:
+            stamp = datetime.now(timezone.utc).strftime("%Y-%m-%d_%H-%M-%S_%f")
+            dest = directory / f"events.{stamp}.bin.zst"
+            compressor = zstandard.ZstdCompressor()
+            with open(source, "rb") as f_in, open(dest, "wb") as f_out:
+                compressor.copy_stream(f_in, f_out)
+            source.unlink()
+            logger.info(f"Rotated event log: {source} -> {dest}")
+
+            # Prune oldest archives beyond the limit
+            archives = sorted(directory.glob("events.*.bin.zst"))
+            for old in archives[:-_MAX_ARCHIVES]:
+                old.unlink()
+        except Exception as e:
+            logger.opt(exception=e).warning(f"Failed to rotate event log {source}")
+            # Clean up the source even if compression fails
+            with contextlib.suppress(OSError):
+                source.unlink()

src/exo/master/main.py

@@ -4,6 +4,7 @@ import anyio
 from anyio.abc import TaskGroup
 from loguru import logger
 
+from exo.master.event_log import DiskEventLog
 from exo.master.placement import (
     add_instance_to_placements,
     cancel_unnecessary_downloads,
@@ -12,7 +13,7 @@ from exo.master.placement import (
     place_instance,
 )
 from exo.shared.apply import apply
-from exo.shared.constants import EXO_TRACING_ENABLED
+from exo.shared.constants import EXO_EVENT_LOG_DIR, EXO_TRACING_ENABLED
 from exo.shared.types.commands import (
     CreateInstance,
     DeleteInstance,
@@ -88,8 +89,7 @@ class Master:
             local_event_receiver.clone_sender()
         )
         self._multi_buffer = MultiSourceBuffer[NodeId, Event]()
-        # TODO: not have this
-        self._event_log: list[Event] = []
+        self._event_log = DiskEventLog(EXO_EVENT_LOG_DIR / "master")
         self._pending_traces: dict[TaskId, dict[int, list[TraceEventData]]] = {}
         self._expected_ranks: dict[TaskId, set[int]] = {}
 
@@ -103,6 +103,7 @@ class Master:
                 tg.start_soon(self._loopback_processor)
                 tg.start_soon(self._plan)
         finally:
+            self._event_log.close()
             self.global_event_sender.close()
             self.local_event_receiver.close()
             self.command_receiver.close()
@@ -332,10 +333,13 @@ class Master:
                                 ]
                         case RequestEventLog():
                             # We should just be able to send everything, since other buffers will ignore old messages
-                            for i in range(command.since_idx, len(self._event_log)):
-                                await self._send_event(
-                                    IndexedEvent(idx=i, event=self._event_log[i])
-                                )
+                            # rate limit to 1000 at a time
+                            end = min(command.since_idx + 1000, len(self._event_log))
+                            for i, event in enumerate(
+                                self._event_log.read_range(command.since_idx, end),
+                                start=command.since_idx,
+                            ):
+                                await self._send_event(IndexedEvent(idx=i, event=event))
                     for event in generated_events:
                         await self.event_sender.send(event)
                 except ValueError as e:

src/exo/master/tests/test_event_log.py

@@ -0,0 +1,155 @@
+from pathlib import Path
+
+import pytest
+
+from exo.master.event_log import DiskEventLog
+from exo.shared.types.events import TestEvent
+
+
+@pytest.fixture
+def log_dir(tmp_path: Path) -> Path:
+    return tmp_path / "event_log"
+
+
+def test_append_and_read_back(log_dir: Path):
+    log = DiskEventLog(log_dir)
+    events = [TestEvent() for _ in range(5)]
+    for e in events:
+        log.append(e)
+
+    assert len(log) == 5
+
+    result = list(log.read_all())
+    assert len(result) == 5
+    for original, restored in zip(events, result, strict=True):
+        assert original.event_id == restored.event_id
+
+    log.close()
+
+
+def test_read_range(log_dir: Path):
+    log = DiskEventLog(log_dir)
+    events = [TestEvent() for _ in range(10)]
+    for e in events:
+        log.append(e)
+
+    result = list(log.read_range(3, 7))
+    assert len(result) == 4
+    for i, restored in enumerate(result):
+        assert events[3 + i].event_id == restored.event_id
+
+    log.close()
+
+
+def test_read_range_bounds(log_dir: Path):
+    log = DiskEventLog(log_dir)
+    events = [TestEvent() for _ in range(3)]
+    for e in events:
+        log.append(e)
+
+    # Start beyond count
+    assert list(log.read_range(5, 10)) == []
+    # Negative start
+    assert list(log.read_range(-1, 2)) == []
+    # End beyond count is clamped
+    result = list(log.read_range(1, 100))
+    assert len(result) == 2
+
+    log.close()
+
+
+def test_empty_log(log_dir: Path):
+    log = DiskEventLog(log_dir)
+    assert len(log) == 0
+    assert list(log.read_all()) == []
+    assert list(log.read_range(0, 10)) == []
+    log.close()
+
+
+def _archives(log_dir: Path) -> list[Path]:
+    return sorted(log_dir.glob("events.*.bin.zst"))
+
+
+def test_rotation_on_close(log_dir: Path):
+    log = DiskEventLog(log_dir)
+    log.append(TestEvent())
+    log.close()
+
+    active = log_dir / "events.bin"
+    assert not active.exists()
+
+    archives = _archives(log_dir)
+    assert len(archives) == 1
+    assert archives[0].stat().st_size > 0
+
+
+def test_rotation_on_construction_with_stale_file(log_dir: Path):
+    log_dir.mkdir(parents=True, exist_ok=True)
+    (log_dir / "events.bin").write_bytes(b"stale data")
+
+    log = DiskEventLog(log_dir)
+    archives = _archives(log_dir)
+    assert len(archives) == 1
+    assert archives[0].exists()
+    assert len(log) == 0
+
+    log.close()
+
+
+def test_empty_log_no_archive(log_dir: Path):
+    """Closing an empty log should not leave an archive."""
+    log = DiskEventLog(log_dir)
+    log.close()
+
+    active = log_dir / "events.bin"
+
+    assert not active.exists()
+    assert _archives(log_dir) == []
+
+
+def test_close_is_idempotent(log_dir: Path):
+    log = DiskEventLog(log_dir)
+    log.append(TestEvent())
+    log.close()
+    archive = _archives(log_dir)
+    log.close()  # should not raise
+
+    assert _archives(log_dir) == archive
+
+
+def test_successive_sessions(log_dir: Path):
+    """Simulate two master sessions: both archives should be kept."""
+    log1 = DiskEventLog(log_dir)
+    log1.append(TestEvent())
+    log1.close()
+
+    first_archive = _archives(log_dir)[-1]
+
+    log2 = DiskEventLog(log_dir)
+    log2.append(TestEvent())
+    log2.append(TestEvent())
+    log2.close()
+
+    # Session 1 archive shifted to slot 2, session 2 in slot 1
+    second_archive = _archives(log_dir)[-1]
+    should_be_first_archive = _archives(log_dir)[-2]
+
+    assert first_archive.exists()
+    assert second_archive.exists()
+    assert first_archive != second_archive
+    assert should_be_first_archive == first_archive
+
+
+def test_rotation_keeps_at_most_5_archives(log_dir: Path):
+    """After 7 sessions, only the 5 most recent archives should remain."""
+    all_archives: list[Path] = []
+    for _ in range(7):
+        log = DiskEventLog(log_dir)
+        log.append(TestEvent())
+        log.close()
+        all_archives.append(_archives(log_dir)[-1])
+
+    for old in all_archives[:2]:
+        assert not old.exists()
+    for recent in all_archives[2:]:
+        assert recent.exists()

src/exo/routing/router.py

@@ -207,10 +207,10 @@ class Router:
                 try:
                     logger.trace(f"Sending message on {topic} with payload {data}")
                     await self._net.gossipsub_publish(topic, data)
-                # As a hack, this also catches AllQueuesFull
-                # Need to fix that ASAP.
-                except (NoPeersSubscribedToTopicError, AllQueuesFullError):
+                except NoPeersSubscribedToTopicError:
                     pass
+                except AllQueuesFullError:
+                    logger.warning(f"All peer queues full, dropping message on {topic}")
 
 
 def get_node_id_keypair(

src/exo/shared/apply.py

@@ -31,6 +31,7 @@ from exo.shared.types.events import (
 from exo.shared.types.profiling import (
     NodeIdentity,
     NodeNetworkInfo,
+    NodeRdmaCtlStatus,
     NodeThunderboltInfo,
     ThunderboltBridgeStatus,
 )
@@ -47,7 +48,9 @@ from exo.utils.info_gatherer.info_gatherer import (
     MemoryUsage,
     MiscData,
     NodeConfig,
+    NodeDiskUsage,
     NodeNetworkInterfaces,
+    RdmaCtlStatus,
     StaticNodeInformation,
     ThunderboltBridgeInfo,
 )
@@ -223,6 +226,9 @@ def apply_node_timed_out(event: NodeTimedOut, state: State) -> State:
     node_memory = {
         key: value for key, value in state.node_memory.items() if key != event.node_id
     }
+    node_disk = {
+        key: value for key, value in state.node_disk.items() if key != event.node_id
+    }
     node_system = {
         key: value for key, value in state.node_system.items() if key != event.node_id
     }
@@ -239,6 +245,9 @@ def apply_node_timed_out(event: NodeTimedOut, state: State) -> State:
         for key, value in state.node_thunderbolt_bridge.items()
         if key != event.node_id
     }
+    node_rdma_ctl = {
+        key: value for key, value in state.node_rdma_ctl.items() if key != event.node_id
+    }
     # Only recompute cycles if the leaving node had TB bridge enabled
     leaving_node_status = state.node_thunderbolt_bridge.get(event.node_id)
     leaving_node_had_tb_enabled = (
@@ -256,10 +265,12 @@ def apply_node_timed_out(event: NodeTimedOut, state: State) -> State:
             "last_seen": last_seen,
             "node_identities": node_identities,
             "node_memory": node_memory,
+            "node_disk": node_disk,
             "node_system": node_system,
             "node_network": node_network,
             "node_thunderbolt": node_thunderbolt,
             "node_thunderbolt_bridge": node_thunderbolt_bridge,
+            "node_rdma_ctl": node_rdma_ctl,
             "thunderbolt_bridge_cycles": thunderbolt_bridge_cycles,
         }
     )
@@ -288,6 +299,8 @@ def apply_node_gathered_info(event: NodeGatheredInfo, state: State) -> State:
             update["node_memory"] = {**state.node_memory, event.node_id: info.memory}
         case MemoryUsage():
             update["node_memory"] = {**state.node_memory, event.node_id: info}
+        case NodeDiskUsage():
+            update["node_disk"] = {**state.node_disk, event.node_id: info.disk_usage}
         case NodeConfig():
             pass
         case MiscData():
@@ -302,7 +315,12 @@ def apply_node_gathered_info(event: NodeGatheredInfo, state: State) -> State:
         case StaticNodeInformation():
             current_identity = state.node_identities.get(event.node_id, NodeIdentity())
             new_identity = current_identity.model_copy(
-                update={"model_id": info.model, "chip_id": info.chip}
+                update={
+                    "model_id": info.model,
+                    "chip_id": info.chip,
+                    "os_version": info.os_version,
+                    "os_build_version": info.os_build_version,
+                }
             )
             update["node_identities"] = {
                 **state.node_identities,
@@ -354,6 +372,11 @@ def apply_node_gathered_info(event: NodeGatheredInfo, state: State) -> State:
                         new_tb_bridge, state.node_network
                     )
                 )
+        case RdmaCtlStatus():
+            update["node_rdma_ctl"] = {
+                **state.node_rdma_ctl,
+                event.node_id: NodeRdmaCtlStatus(enabled=info.enabled),
+            }
 
     return state.model_copy(update=update)
 

src/exo/shared/constants.py

@@ -43,7 +43,8 @@ DASHBOARD_DIR = (
 )
 
 # Log files (data/logs or cache)
-EXO_LOG = EXO_CACHE_HOME / "exo.log"
+EXO_LOG_DIR = EXO_CACHE_HOME / "exo_log"
+EXO_LOG = EXO_LOG_DIR / "exo.log"
 EXO_TEST_LOG = EXO_CACHE_HOME / "exo_test.log"
 
 # Identity (config)
@@ -60,6 +61,7 @@ EXO_MAX_CHUNK_SIZE = 512 * 1024
 
 EXO_CUSTOM_MODEL_CARDS_DIR = EXO_DATA_HOME / "custom_model_cards"
 
+EXO_EVENT_LOG_DIR = EXO_DATA_HOME / "event_log"
 EXO_IMAGE_CACHE_DIR = EXO_CACHE_HOME / "images"
 EXO_TRACING_CACHE_DIR = EXO_CACHE_HOME / "traces"
 

src/exo/shared/logging.py

@@ -1,11 +1,30 @@
 import logging
 import sys
+from collections.abc import Iterator
 from pathlib import Path
 
+import zstandard
 from hypercorn import Config
 from hypercorn.logging import Logger as HypercornLogger
 from loguru import logger
 
+_MAX_LOG_ARCHIVES = 5
+
+
+def _zstd_compress(filepath: str) -> None:
+    source = Path(filepath)
+    dest = source.with_suffix(source.suffix + ".zst")
+    cctx = zstandard.ZstdCompressor()
+    with open(source, "rb") as f_in, open(dest, "wb") as f_out:
+        cctx.copy_stream(f_in, f_out)
+    source.unlink()
+
+
+def _once_then_never() -> Iterator[bool]:
+    yield True
+    while True:
+        yield False
+
 
 class InterceptLogger(HypercornLogger):
     def __init__(self, config: Config):
@@ -53,13 +72,16 @@ def logger_setup(log_file: Path | None, verbosity: int = 0):
             enqueue=True,
         )
     if log_file:
+        rotate_once = _once_then_never()
         logger.add(
             log_file,
             format="[ {time:YYYY-MM-DD HH:mm:ss.SSS} | {level: <8} | {name}:{function}:{line} ] {message}",
             level="INFO",
             colorize=False,
             enqueue=True,
-            rotation="1 week",
+            rotation=lambda _, __: next(rotate_once),
+            retention=_MAX_LOG_ARCHIVES,
+            compression=_zstd_compress,
         )
 
 

src/exo/shared/models/model_cards.py

@@ -30,11 +30,11 @@ from exo.utils.pydantic_ext import CamelCaseModel
 # kinda ugly...
 # TODO: load search path from config.toml
 _custom_cards_dir = Path(str(EXO_CUSTOM_MODEL_CARDS_DIR))
-_csp = [Path(RESOURCES_DIR) / "inference_model_cards", _custom_cards_dir]
-if EXO_ENABLE_IMAGE_MODELS:
-    _csp.append(Path(RESOURCES_DIR) / "image_model_cards")
-
-CARD_SEARCH_PATH = _csp
+CARD_SEARCH_PATH = [
+    Path(RESOURCES_DIR) / "inference_model_cards",
+    Path(RESOURCES_DIR) / "image_model_cards",
+    _custom_cards_dir,
+]
 
 _card_cache: dict[ModelId, "ModelCard"] = {}
 
@@ -49,10 +49,16 @@ async def _refresh_card_cache():
                 pass
 
 
+def _is_image_card(card: "ModelCard") -> bool:
+    return any(t in (ModelTask.TextToImage, ModelTask.ImageToImage) for t in card.tasks)
+
+
 async def get_model_cards() -> list["ModelCard"]:
     if len(_card_cache) == 0:
         await _refresh_card_cache()
-    return list(_card_cache.values())
+    if EXO_ENABLE_IMAGE_MODELS:
+        return list(_card_cache.values())
+    return [c for c in _card_cache.values() if not _is_image_card(c)]
 
 
 class ModelTask(str, Enum):

src/exo/shared/types/api.py

@@ -1,6 +1,7 @@
 import time
 from collections.abc import Generator
 from typing import Annotated, Any, Literal
+from uuid import uuid4
 
 from pydantic import BaseModel, Field, field_validator
 from pydantic_core import PydanticUseDefault
@@ -60,6 +61,7 @@ class ChatCompletionMessageText(BaseModel):
 
 
 class ToolCallItem(BaseModel):
+    id: str = Field(default_factory=lambda: str(uuid4()))
     name: str
     arguments: str
 

src/exo/shared/types/profiling.py

@@ -1,4 +1,6 @@
+import shutil
 from collections.abc import Sequence
+from pathlib import Path
 from typing import Literal, Self
 
 import psutil
@@ -38,6 +40,22 @@ class MemoryUsage(CamelCaseModel):
         )
 
 
+class DiskUsage(CamelCaseModel):
+    """Disk space usage for the models directory."""
+
+    total: Memory
+    available: Memory
+
+    @classmethod
+    def from_path(cls, path: Path) -> Self:
+        """Get disk usage stats for the partition containing path."""
+        total, _used, free = shutil.disk_usage(path)
+        return cls(
+            total=Memory.from_bytes(total),
+            available=Memory.from_bytes(free),
+        )
+
+
 class SystemPerformanceProfile(CamelCaseModel):
     # TODO: flops_fp16: float
 
@@ -63,6 +81,8 @@ class NodeIdentity(CamelCaseModel):
     model_id: str = "Unknown"
     chip_id: str = "Unknown"
     friendly_name: str = "Unknown"
+    os_version: str = "Unknown"
+    os_build_version: str = "Unknown"
 
 
 class NodeNetworkInfo(CamelCaseModel):
@@ -77,6 +97,12 @@ class NodeThunderboltInfo(CamelCaseModel):
     interfaces: Sequence[ThunderboltIdentifier] = []
 
 
+class NodeRdmaCtlStatus(CamelCaseModel):
+    """Whether RDMA is enabled on this node (via rdma_ctl)."""
+
+    enabled: bool
+
+
 class ThunderboltBridgeStatus(CamelCaseModel):
     """Whether the Thunderbolt Bridge network service is enabled on this node."""
 

src/exo/shared/types/state.py

@@ -8,9 +8,11 @@ from pydantic.alias_generators import to_camel
 from exo.shared.topology import Topology, TopologySnapshot
 from exo.shared.types.common import NodeId
 from exo.shared.types.profiling import (
+    DiskUsage,
     MemoryUsage,
     NodeIdentity,
     NodeNetworkInfo,
+    NodeRdmaCtlStatus,
     NodeThunderboltInfo,
     SystemPerformanceProfile,
     ThunderboltBridgeStatus,
@@ -49,10 +51,12 @@ class State(CamelCaseModel):
     # Granular node state mappings (update independently at different frequencies)
     node_identities: Mapping[NodeId, NodeIdentity] = {}
     node_memory: Mapping[NodeId, MemoryUsage] = {}
+    node_disk: Mapping[NodeId, DiskUsage] = {}
     node_system: Mapping[NodeId, SystemPerformanceProfile] = {}
     node_network: Mapping[NodeId, NodeNetworkInfo] = {}
     node_thunderbolt: Mapping[NodeId, NodeThunderboltInfo] = {}
     node_thunderbolt_bridge: Mapping[NodeId, ThunderboltBridgeStatus] = {}
+    node_rdma_ctl: Mapping[NodeId, NodeRdmaCtlStatus] = {}
 
     # Detected cycles where all nodes have Thunderbolt bridge enabled (>2 nodes)
     thunderbolt_bridge_cycles: Sequence[Sequence[NodeId]] = []

src/exo/shared/types/thunderbolt.py

@@ -12,6 +12,7 @@ class ThunderboltConnection(CamelCaseModel):
 class ThunderboltIdentifier(CamelCaseModel):
     rdma_interface: str
     domain_uuid: str
+    link_speed: str = ""
 
 
 ## Intentionally minimal, only collecting data we care about - there's a lot more
@@ -19,6 +20,7 @@ class ThunderboltIdentifier(CamelCaseModel):
 
 class _ReceptacleTag(BaseModel, extra="ignore"):
     receptacle_id_key: str | None = None
+    current_speed_key: str | None = None
 
 
 class _ConnectivityItem(BaseModel, extra="ignore"):
@@ -42,7 +44,9 @@ class ThunderboltConnectivityData(BaseModel, extra="ignore"):
         # if tag not in ifaces: return None
         iface = f"rdma_{ifaces[tag]}"
         return ThunderboltIdentifier(
-            rdma_interface=iface, domain_uuid=self.domain_uuid_key
+            rdma_interface=iface,
+            domain_uuid=self.domain_uuid_key,
+            link_speed=self.receptacle_1_tag.current_speed_key or "",
         )
 
     def conn(self) -> ThunderboltConnection | None:

src/exo/utils/channels.py

@@ -141,6 +141,9 @@ class MpSender[T]:
         )
         self._state.buffer.join_thread()
 
+    def cancel_join(self) -> None:
+        self._state.buffer.cancel_join_thread()
+
     # == context manager support ==
     def __enter__(self) -> Self:
         return self
@@ -212,6 +215,9 @@ class MpReceiver[T]:
         )
         self._state.buffer.join_thread()
 
+    def cancel_join(self) -> None:
+        self._state.buffer.cancel_join_thread()
+
     # == iterator support ==
     def __iter__(self) -> Self:
         return self

src/exo/utils/info_gatherer/info_gatherer.py

@@ -8,16 +8,17 @@ from subprocess import CalledProcessError
 from typing import Self, cast
 
 import anyio
-from anyio import create_task_group, open_process
+from anyio import create_task_group, fail_after, open_process, to_thread
 from anyio.abc import TaskGroup
 from anyio.streams.buffered import BufferedByteReceiveStream
 from anyio.streams.text import TextReceiveStream
 from loguru import logger
 from pydantic import ValidationError
 
-from exo.shared.constants import EXO_CONFIG_FILE
+from exo.shared.constants import EXO_CONFIG_FILE, EXO_MODELS_DIR
 from exo.shared.types.memory import Memory
 from exo.shared.types.profiling import (
+    DiskUsage,
     MemoryUsage,
     NetworkInterfaceInfo,
     ThunderboltBridgeStatus,
@@ -31,7 +32,13 @@ from exo.utils.channels import Sender
 from exo.utils.pydantic_ext import TaggedModel
 
 from .macmon import MacmonMetrics
-from .system_info import get_friendly_name, get_model_and_chip, get_network_interfaces
+from .system_info import (
+    get_friendly_name,
+    get_model_and_chip,
+    get_network_interfaces,
+    get_os_build_version,
+    get_os_version,
+)
 
 IS_DARWIN = sys.platform == "darwin"
 
@@ -177,11 +184,18 @@ class StaticNodeInformation(TaggedModel):
 
     model: str
     chip: str
+    os_version: str
+    os_build_version: str
 
     @classmethod
     async def gather(cls) -> Self:
         model, chip = await get_model_and_chip()
-        return cls(model=model, chip=chip)
+        return cls(
+            model=model,
+            chip=chip,
+            os_version=get_os_version(),
+            os_build_version=await get_os_build_version(),
+        )
 
 
 class NodeNetworkInterfaces(TaggedModel):
@@ -196,6 +210,28 @@ class MacThunderboltConnections(TaggedModel):
     conns: Sequence[ThunderboltConnection]
 
 
+class RdmaCtlStatus(TaggedModel):
+    enabled: bool
+
+    @classmethod
+    async def gather(cls) -> Self | None:
+        if not IS_DARWIN or shutil.which("rdma_ctl") is None:
+            return None
+        try:
+            with anyio.fail_after(5):
+                proc = await anyio.run_process(["rdma_ctl", "status"], check=False)
+        except (TimeoutError, OSError):
+            return None
+        if proc.returncode != 0:
+            return None
+        output = proc.stdout.decode("utf-8").lower().strip()
+        if "enabled" in output:
+            return cls(enabled=True)
+        if "disabled" in output:
+            return cls(enabled=False)
+        return None
+
+
 class ThunderboltBridgeInfo(TaggedModel):
     status: ThunderboltBridgeStatus
 
@@ -284,6 +320,20 @@ class MiscData(TaggedModel):
         return cls(friendly_name=await get_friendly_name())
 
 
+class NodeDiskUsage(TaggedModel):
+    """Disk space information for the models directory."""
+
+    disk_usage: DiskUsage
+
+    @classmethod
+    async def gather(cls) -> Self:
+        return cls(
+            disk_usage=await to_thread.run_sync(
+                lambda: DiskUsage.from_path(EXO_MODELS_DIR)
+            )
+        )
+
+
 async def _gather_iface_map() -> dict[str, str] | None:
     proc = await anyio.run_process(
         ["networksetup", "-listallhardwareports"], check=False
@@ -310,10 +360,12 @@ GatheredInfo = (
     | NodeNetworkInterfaces
     | MacThunderboltIdentifiers
     | MacThunderboltConnections
+    | RdmaCtlStatus
     | ThunderboltBridgeInfo
     | NodeConfig
     | MiscData
     | StaticNodeInformation
+    | NodeDiskUsage
 )
 
 
@@ -326,6 +378,9 @@ class InfoGatherer:
     memory_poll_rate: float | None = None if IS_DARWIN else 1
     macmon_interval: float | None = 1 if IS_DARWIN else None
     thunderbolt_bridge_poll_interval: float | None = 10 if IS_DARWIN else None
+    static_info_poll_interval: float | None = 60
+    rdma_ctl_poll_interval: float | None = 10 if IS_DARWIN else None
+    disk_poll_interval: float | None = 30
     _tg: TaskGroup = field(init=False, default_factory=create_task_group)
 
     async def run(self):
@@ -335,43 +390,67 @@ class InfoGatherer:
                     tg.start_soon(self._monitor_macmon, macmon_path)
                 tg.start_soon(self._monitor_system_profiler_thunderbolt_data)
                 tg.start_soon(self._monitor_thunderbolt_bridge_status)
+                tg.start_soon(self._monitor_rdma_ctl_status)
             tg.start_soon(self._watch_system_info)
             tg.start_soon(self._monitor_memory_usage)
             tg.start_soon(self._monitor_misc)
+            tg.start_soon(self._monitor_static_info)
+            tg.start_soon(self._monitor_disk_usage)
 
             nc = await NodeConfig.gather()
             if nc is not None:
                 await self.info_sender.send(nc)
-            sni = await StaticNodeInformation.gather()
-            await self.info_sender.send(sni)
 
     def shutdown(self):
         self._tg.cancel_scope.cancel()
 
+    async def _monitor_static_info(self):
+        if self.static_info_poll_interval is None:
+            return
+        while True:
+            try:
+                with fail_after(30):
+                    await self.info_sender.send(await StaticNodeInformation.gather())
+            except Exception as e:
+                logger.warning(f"Error gathering static node info: {e}")
+            await anyio.sleep(self.static_info_poll_interval)
+
     async def _monitor_misc(self):
         if self.misc_poll_interval is None:
             return
         while True:
-            await self.info_sender.send(await MiscData.gather())
+            try:
+                with fail_after(10):
+                    await self.info_sender.send(await MiscData.gather())
+            except Exception as e:
+                logger.warning(f"Error gathering misc data: {e}")
             await anyio.sleep(self.misc_poll_interval)
 
     async def _monitor_system_profiler_thunderbolt_data(self):
         if self.system_profiler_interval is None:
             return
-        iface_map = await _gather_iface_map()
-        if iface_map is None:
-            return
 
         while True:
-            data = await ThunderboltConnectivity.gather()
-            assert data is not None
+            try:
+                with fail_after(30):
+                    iface_map = await _gather_iface_map()
+                    if iface_map is None:
+                        raise ValueError("Failed to gather interface map")
 
-            idents = [it for i in data if (it := i.ident(iface_map)) is not None]
-            await self.info_sender.send(MacThunderboltIdentifiers(idents=idents))
+                    data = await ThunderboltConnectivity.gather()
+                    assert data is not None
 
-            conns = [it for i in data if (it := i.conn()) is not None]
-            await self.info_sender.send(MacThunderboltConnections(conns=conns))
+                    idents = [
+                        it for i in data if (it := i.ident(iface_map)) is not None
+                    ]
+                    await self.info_sender.send(
+                        MacThunderboltIdentifiers(idents=idents)
+                    )
 
+                    conns = [it for i in data if (it := i.conn()) is not None]
+                    await self.info_sender.send(MacThunderboltConnections(conns=conns))
+            except Exception as e:
+                logger.warning(f"Error gathering Thunderbolt data: {e}")
             await anyio.sleep(self.system_profiler_interval)
 
     async def _monitor_memory_usage(self):
@@ -384,52 +463,95 @@ class InfoGatherer:
         if self.memory_poll_rate is None:
             return
         while True:
-            await self.info_sender.send(
-                MemoryUsage.from_psutil(override_memory=override_memory)
-            )
+            try:
+                await self.info_sender.send(
+                    MemoryUsage.from_psutil(override_memory=override_memory)
+                )
+            except Exception as e:
+                logger.warning(f"Error gathering memory usage: {e}")
             await anyio.sleep(self.memory_poll_rate)
 
     async def _watch_system_info(self):
         if self.interface_watcher_interval is None:
             return
         while True:
-            nics = await get_network_interfaces()
-            await self.info_sender.send(NodeNetworkInterfaces(ifaces=nics))
+            try:
+                with fail_after(10):
+                    nics = await get_network_interfaces()
+                    await self.info_sender.send(NodeNetworkInterfaces(ifaces=nics))
+            except Exception as e:
+                logger.warning(f"Error gathering network interfaces: {e}")
             await anyio.sleep(self.interface_watcher_interval)
 
     async def _monitor_thunderbolt_bridge_status(self):
         if self.thunderbolt_bridge_poll_interval is None:
             return
         while True:
-            curr = await ThunderboltBridgeInfo.gather()
-            if curr is not None:
-                await self.info_sender.send(curr)
+            try:
+                with fail_after(30):
+                    curr = await ThunderboltBridgeInfo.gather()
+                    if curr is not None:
+                        await self.info_sender.send(curr)
+            except Exception as e:
+                logger.warning(f"Error gathering Thunderbolt Bridge status: {e}")
             await anyio.sleep(self.thunderbolt_bridge_poll_interval)
 
+    async def _monitor_rdma_ctl_status(self):
+        if self.rdma_ctl_poll_interval is None:
+            return
+        while True:
+            try:
+                curr = await RdmaCtlStatus.gather()
+                if curr is not None:
+                    await self.info_sender.send(curr)
+            except Exception as e:
+                logger.warning(f"Error gathering RDMA ctl status: {e}")
+            await anyio.sleep(self.rdma_ctl_poll_interval)
+
+    async def _monitor_disk_usage(self):
+        if self.disk_poll_interval is None:
+            return
+        while True:
+            try:
+                with fail_after(5):
+                    await self.info_sender.send(await NodeDiskUsage.gather())
+            except Exception as e:
+                logger.warning(f"Error gathering disk usage: {e}")
+            await anyio.sleep(self.disk_poll_interval)
+
     async def _monitor_macmon(self, macmon_path: str):
         if self.macmon_interval is None:
             return
         # macmon pipe --interval [interval in ms]
-        try:
-            async with await open_process(
-                [macmon_path, "pipe", "--interval", str(self.macmon_interval * 1000)]
-            ) as p:
-                if not p.stdout:
-                    logger.critical("MacMon closed stdout")
-                    return
-                async for text in TextReceiveStream(
-                    BufferedByteReceiveStream(p.stdout)
-                ):
-                    await self.info_sender.send(MacmonMetrics.from_raw_json(text))
-        except CalledProcessError as e:
-            stderr_msg = "no stderr"
-            stderr_output = cast(bytes | str | None, e.stderr)
-            if stderr_output is not None:
-                stderr_msg = (
-                    stderr_output.decode()
-                    if isinstance(stderr_output, bytes)
-                    else str(stderr_output)
+        while True:
+            try:
+                async with await open_process(
+                    [
+                        macmon_path,
+                        "pipe",
+                        "--interval",
+                        str(self.macmon_interval * 1000),
+                    ]
+                ) as p:
+                    if not p.stdout:
+                        logger.critical("MacMon closed stdout")
+                        return
+                    async for text in TextReceiveStream(
+                        BufferedByteReceiveStream(p.stdout)
+                    ):
+                        await self.info_sender.send(MacmonMetrics.from_raw_json(text))
+            except CalledProcessError as e:
+                stderr_msg = "no stderr"
+                stderr_output = cast(bytes | str | None, e.stderr)
+                if stderr_output is not None:
+                    stderr_msg = (
+                        stderr_output.decode()
+                        if isinstance(stderr_output, bytes)
+                        else str(stderr_output)
+                    )
+                logger.warning(
+                    f"MacMon failed with return code {e.returncode}: {stderr_msg}"
                 )
-            logger.warning(
-                f"MacMon failed with return code {e.returncode}: {stderr_msg}"
-            )
+            except Exception as e:
+                logger.warning(f"Error in macmon monitor: {e}")
+            await anyio.sleep(self.macmon_interval)

src/exo/utils/info_gatherer/net_profile.py

@@ -1,4 +1,5 @@
-from collections.abc import Mapping
+from collections import defaultdict
+from collections.abc import AsyncGenerator, Mapping
 
 import anyio
 import httpx
@@ -8,6 +9,7 @@ from loguru import logger
 from exo.shared.topology import Topology
 from exo.shared.types.common import NodeId
 from exo.shared.types.profiling import NodeNetworkInfo
+from exo.utils.channels import Sender, channel
 
 REACHABILITY_ATTEMPTS = 3
 
@@ -80,10 +82,10 @@ async def check_reachable(
     topology: Topology,
     self_node_id: NodeId,
     node_network: Mapping[NodeId, NodeNetworkInfo],
-) -> dict[NodeId, set[str]]:
-    """Check which nodes are reachable and return their IPs."""
+) -> AsyncGenerator[tuple[str, NodeId], None]:
+    """Yield (ip, node_id) pairs as reachability probes complete."""
 
-    reachable: dict[NodeId, set[str]] = {}
+    send, recv = channel[tuple[str, NodeId]]()
 
     # these are intentionally httpx's defaults so we can tune them later
     timeout = httpx.Timeout(timeout=5.0)
@@ -93,6 +95,18 @@ async def check_reachable(
         keepalive_expiry=5,
     )
 
+    async def _probe(
+        target_ip: str,
+        expected_node_id: NodeId,
+        client: httpx.AsyncClient,
+        send: Sender[tuple[str, NodeId]],
+    ) -> None:
+        async with send:
+            out: defaultdict[NodeId, set[str]] = defaultdict(set)
+            await check_reachability(target_ip, expected_node_id, out, client)
+            if expected_node_id in out:
+                await send.send((target_ip, expected_node_id))
+
     async with (
         httpx.AsyncClient(timeout=timeout, limits=limits) as client,
         create_task_group() as tg,
@@ -103,12 +117,9 @@ async def check_reachable(
             if node_id == self_node_id:
                 continue
             for iface in node_network[node_id].interfaces:
-                tg.start_soon(
-                    check_reachability,
-                    iface.ip_address,
-                    node_id,
-                    reachable,
-                    client,
-                )
+                tg.start_soon(_probe, iface.ip_address, node_id, client, send.clone())
+        send.close()
 
-    return reachable
+        with recv:
+            async for item in recv:
+                yield item

src/exo/utils/info_gatherer/system_info.py

@@ -1,3 +1,4 @@
+import platform
 import socket
 import sys
 from subprocess import CalledProcessError
@@ -8,6 +9,34 @@ from anyio import run_process
 from exo.shared.types.profiling import InterfaceType, NetworkInterfaceInfo
 
 
+def get_os_version() -> str:
+    """Return the OS version string for this node.
+
+    On macOS this is the macOS version (e.g. ``"15.3"``).
+    On other platforms it falls back to the platform name (e.g. ``"Linux"``).
+    """
+    if sys.platform == "darwin":
+        version = platform.mac_ver()[0]
+        return version if version else "Unknown"
+    return platform.system() or "Unknown"
+
+
+async def get_os_build_version() -> str:
+    """Return the macOS build version string (e.g. ``"24D5055b"``).
+
+    On non-macOS platforms, returns ``"Unknown"``.
+    """
+    if sys.platform != "darwin":
+        return "Unknown"
+
+    try:
+        process = await run_process(["sw_vers", "-buildVersion"])
+    except CalledProcessError:
+        return "Unknown"
+
+    return process.stdout.decode("utf-8", errors="replace").strip() or "Unknown"
+
+
 async def get_friendly_name() -> str:
     """
     Asynchronously gets the 'Computer Name' (friendly name) of a Mac.

src/exo/worker/engines/mlx/auto_parallel.py

@@ -121,10 +121,15 @@ class PipelineFirstLayer(CustomMlxLayer):
         super().__init__(original_layer)
         self.r: int = r
         self.group = group
+        self.is_prefill: bool = False
 
     def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
         if self.r != 0:
             x = mx.distributed.recv_like(x, (self.r - 1), group=self.group)
+            if self.is_prefill:
+                # We want to avoid GPU timeout errors by evalling the distributed operation
+                # so that it stays on CPU, which does not have a timeout.
+                mx.eval(x)
         return self.original_layer(x, *args, **kwargs)
 
 
@@ -141,6 +146,7 @@ class PipelineLastLayer(CustomMlxLayer):
         self.s: int = s
         self.group = group
         self.original_layer_signature = signature(self.original_layer.__call__)
+        self.is_prefill: bool = False
 
     def __call__(self, x: mx.array, *args: object, **kwargs: object) -> mx.array:
         cache = self.original_layer_signature.bind_partial(
@@ -155,14 +161,25 @@ class PipelineLastLayer(CustomMlxLayer):
             )
             if cache is not None:
                 cache.keys = mx.depends(cache.keys, output)  # type: ignore[reportUnknownMemberType]
+            if self.is_prefill:
+                mx.eval(output)
+                if cache is not None:
+                    mx.eval(cache.keys)  # type: ignore
 
-        output = mx.distributed.all_gather(output, group=self.group)[
-            -output.shape[0] :
-        ]  # type :ignore
+        if not self.is_prefill:
+            output = mx.distributed.all_gather(output, group=self.group)[
+                -output.shape[0] :
+            ]
 
         return output
 
 
+def set_pipeline_prefill(model: nn.Module, is_prefill: bool) -> None:
+    for layer in model.layers:  # type: ignore
+        if isinstance(layer, (PipelineFirstLayer, PipelineLastLayer)):
+            layer.is_prefill = is_prefill
+
+
 def _inner_model(model: nn.Module) -> nn.Module:
     inner = getattr(model, "model", None)
     if isinstance(inner, nn.Module):

src/exo/worker/engines/mlx/generator/generate.py

@@ -24,6 +24,7 @@ from exo.shared.types.worker.runner_response import (
     GenerationResponse,
 )
 from exo.worker.engines.mlx import Model
+from exo.worker.engines.mlx.auto_parallel import set_pipeline_prefill
 from exo.worker.engines.mlx.cache import (
     CacheSnapshot,
     KVPrefixCache,
@@ -56,6 +57,7 @@ def prefill(
     sampler: Callable[[mx.array], mx.array],
     prompt_tokens: mx.array,
     cache: KVCacheType,
+    group: mx.distributed.Group | None = None,
 ) -> tuple[float, int, list[CacheSnapshot]]:
     """Prefill the KV cache with prompt tokens.
 
@@ -83,6 +85,11 @@ def prefill(
         if has_ssm:
             snapshots.append(snapshot_ssm_states(cache))
 
+    set_pipeline_prefill(model, is_prefill=True)
+
+    mx_barrier(group)
+    logger.info("Ready to prefill")
+
     # Use max_tokens=1 because max_tokens=0 does not work.
     # We just throw away the generated token - we only care about filling the cache
     for _ in stream_generate(
@@ -92,13 +99,15 @@ def prefill(
         max_tokens=1,
         sampler=sampler,
         prompt_cache=cache,
-        prefill_step_size=2048,
+        prefill_step_size=8192,
         kv_group_size=KV_GROUP_SIZE,
         kv_bits=KV_BITS,
         prompt_progress_callback=progress_callback,
     ):
         break  # Stop after first iteration - cache is now filled
 
+    set_pipeline_prefill(model, is_prefill=False)
+
     # stream_generate added 1 extra generated token to the cache, so we should trim it.
     # Because of needing to roll back arrays cache, we will generate on 2 tokens so trim 1 more.
     pre_gen = deepcopy(snapshots[-2]) if has_ssm else None
@@ -145,6 +154,8 @@ def warmup_inference(
     # Use a default sampler for warmup
     sampler = make_sampler(temp=0.0)
 
+    mx_barrier(group)
+
     logger.info("Generating warmup tokens")
     for _r in stream_generate(
         model=model,
@@ -300,11 +311,7 @@ def mlx_generate(
 
     # Prefill cache with all tokens except the last one
     prefill_tps, prefill_tokens, ssm_snapshots_list = prefill(
-        model,
-        tokenizer,
-        sampler,
-        prompt_tokens[:-1],
-        caches,
+        model, tokenizer, sampler, prompt_tokens[:-1], caches, group
     )
     cache_snapshots: list[CacheSnapshot] | None = ssm_snapshots_list or None
 
@@ -320,6 +327,9 @@ def mlx_generate(
     reasoning_tokens = 0
     think_start = tokenizer.think_start
     think_end = tokenizer.think_end
+
+    mx_barrier(group)
+
     for completion_tokens, out in enumerate(
         stream_generate(
             model=model,
@@ -329,8 +339,7 @@ def mlx_generate(
             sampler=sampler,
             logits_processors=logits_processors,
             prompt_cache=caches,
-            # TODO: Dynamically change prefill step size to be the maximum possible without timing out.
-            prefill_step_size=2048,
+            prefill_step_size=1,
             kv_group_size=KV_GROUP_SIZE,
             kv_bits=KV_BITS,
         ),

src/exo/worker/engines/mlx/utils_mlx.py

@@ -1,6 +1,5 @@
 import json
 import os
-import resource
 import sys
 import time
 from pathlib import Path
@@ -63,8 +62,6 @@ from exo.worker.engines.mlx.auto_parallel import (
 from exo.worker.runner.bootstrap import logger
 
 Group = mx.distributed.Group
-# Needed for 8 bit model
-resource.setrlimit(resource.RLIMIT_NOFILE, (2048, 4096))
 
 
 # TODO: Test this

src/exo/worker/main.py

@@ -1,3 +1,4 @@
+from collections import defaultdict
 from datetime import datetime, timezone
 from random import random
 from typing import Iterator
@@ -183,6 +184,14 @@ class Worker:
             )
             if task is None:
                 continue
+
+            # Gate DownloadModel on backoff BEFORE emitting TaskCreated
+            # to prevent flooding the event log with useless events
+            if isinstance(task, DownloadModel):
+                model_id = task.shard_metadata.model_card.model_id
+                if not self._download_backoff.should_proceed(model_id):
+                    continue
+
             logger.info(f"Worker plan: {task.__class__.__name__}")
             assert task.task_status
             await self.event_sender.send(TaskCreated(task_id=task.task_id, task=task))
@@ -198,9 +207,6 @@ class Worker:
                     )
                 case DownloadModel(shard_metadata=shard):
                     model_id = shard.model_card.model_id
-                    if not self._download_backoff.should_proceed(model_id):
-                        continue
-
                     self._download_backoff.record_attempt(model_id)
 
                     await self.download_command_sender.send(
@@ -345,29 +351,29 @@ class Worker:
             edges = set(
                 conn.edge for conn in self.state.topology.out_edges(self.node_id)
             )
-            conns = await check_reachable(
+            conns: defaultdict[NodeId, set[str]] = defaultdict(set)
+            async for ip, nid in check_reachable(
                 self.state.topology,
                 self.node_id,
                 self.state.node_network,
-            )
-            for nid in conns:
-                for ip in conns[nid]:
-                    edge = SocketConnection(
-                        # nonsense multiaddr
-                        sink_multiaddr=Multiaddr(address=f"/ip4/{ip}/tcp/52415")
-                        if "." in ip
-                        # nonsense multiaddr
-                        else Multiaddr(address=f"/ip6/{ip}/tcp/52415"),
-                    )
-                    if edge not in edges:
-                        logger.debug(f"ping discovered {edge=}")
-                        await self.event_sender.send(
-                            TopologyEdgeCreated(
-                                conn=Connection(
-                                    source=self.node_id, sink=nid, edge=edge
-                                )
-                            )
+            ):
+                if ip in conns[nid]:
+                    continue
+                conns[nid].add(ip)
+                edge = SocketConnection(
+                    # nonsense multiaddr
+                    sink_multiaddr=Multiaddr(address=f"/ip4/{ip}/tcp/52415")
+                    if "." in ip
+                    # nonsense multiaddr
+                    else Multiaddr(address=f"/ip6/{ip}/tcp/52415"),
+                )
+                if edge not in edges:
+                    logger.debug(f"ping discovered {edge=}")
+                    await self.event_sender.send(
+                        TopologyEdgeCreated(
+                            conn=Connection(source=self.node_id, sink=nid, edge=edge)
                         )
+                    )
 
             for conn in self.state.topology.out_edges(self.node_id):
                 if not isinstance(conn.edge, SocketConnection):
@@ -377,8 +383,7 @@ class Worker:
                     continue
                 if (
                     conn.sink not in conns
-                    or conn.edge.sink_multiaddr.ip_address
-                    not in conns.get(conn.sink, set())
+                    or conn.edge.sink_multiaddr.ip_address not in conns[conn.sink]
                 ):
                     logger.debug(f"ping failed to discover {conn=}")
                     await self.event_sender.send(TopologyEdgeDeleted(conn=conn))

src/exo/worker/runner/bootstrap.py

@@ -41,6 +41,8 @@ def entrypoint(
         main(bound_instance, event_sender, task_receiver)
     except ClosedResourceError:
         logger.warning("Runner communication closed unexpectedly")
+    except KeyboardInterrupt:
+        logger.info("Runner received interrupt, shutting down")
     except Exception as e:
         logger.opt(exception=e).warning(
             f"Runner {bound_instance.bound_runner_id} crashed with critical exception {e}"
@@ -55,7 +57,9 @@ def entrypoint(
         try:
             event_sender.close()
             task_receiver.close()
+        except Exception:
+            pass
         finally:
-            event_sender.join()
-            task_receiver.join()
+            event_sender.cancel_join()
+            task_receiver.cancel_join()
             logger.info("bye from the runner")

src/exo/worker/runner/runner.py

@@ -1,5 +1,6 @@
 import base64
 import json
+import resource
 import time
 from collections.abc import Generator
 from functools import cache
@@ -112,6 +113,9 @@ def main(
     event_sender: MpSender[Event],
     task_receiver: MpReceiver[Task],
 ):
+    soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
+    resource.setrlimit(resource.RLIMIT_NOFILE, (min(max(soft, 2048), hard), hard))
+
     instance, runner_id, shard_metadata = (
         bound_instance.instance,
         bound_instance.bound_runner_id,
@@ -810,8 +814,9 @@ def patch_kimi_tokenizer(tokenizer: TokenizerWrapper):
 
     # kimi has a fixed function naming scheme, with a json formatted arg
     #   functions.multiply:0 <|tool_call_argument_begin|> {"a": 2, "b": 3}
+    #   Also needs to handle tools like call_0<|tool_call_argument_begin|>{"filePath": "..."}
     _func_name_regex = re.compile(
-        r"^\s*(.+):\d+\s*<\|tool_call_argument_begin\|>", re.DOTALL
+        r"^\s*(.+)[:](\d+)\s*<\|tool_call_argument_begin\|>", re.DOTALL
     )
     _func_arg_regex = re.compile(r"<\|tool_call_argument_begin\|>\s*(.*)\s*", re.DOTALL)
 
@@ -835,9 +840,10 @@ def patch_kimi_tokenizer(tokenizer: TokenizerWrapper):
         func_name_match = _func_name_regex.search(text)
         if func_name_match is None:
             raise ValueError(f"Could not parse function name from tool call: {text!r}")
-        func_name = func_name_match.group(1)
+        original_func_name = func_name_match.group(1)
+        tool_id = func_name_match.group(2)
         # strip off the `functions.` prefix, if it exists.
-        func_name = func_name[func_name.find(".") + 1 :]
+        func_name = original_func_name[original_func_name.find(".") + 1 :]
 
         func_args_match = _func_arg_regex.search(text)
         if func_args_match is None:
@@ -846,7 +852,11 @@ def patch_kimi_tokenizer(tokenizer: TokenizerWrapper):
         # the args should be valid json - no need to check against our tools to deserialize
         arg_dct = _deserialize(func_args)  # pyright: ignore[reportAny]
 
-        return dict(name=func_name, arguments=arg_dct)  # pyright: ignore[reportAny]
+        return dict(
+            id=f"{original_func_name}:{tool_id}",
+            name=func_name,
+            arguments=arg_dct,  # pyright: ignore[reportAny]
+        )
 
     tokenizer._tool_call_start = tool_call_start
     tokenizer._tool_call_end = tool_call_end
@@ -929,7 +939,13 @@ def _validate_single_tool(obj: dict[str, Any]) -> ToolCallItem:
         and ((args := obj.get("arguments")) is not None)
         and isinstance(name, str)
     ):
-        return ToolCallItem(name=name, arguments=json.dumps(args))
+        raw_id: object = obj.get("id")
+        extra = {"id": str(raw_id)} if raw_id is not None else {}
+        return ToolCallItem(
+            **extra,
+            name=name,
+            arguments=json.dumps(args),
+        )
     else:
         raise ValidationError
 

uv.lock

@@ -16,6 +16,7 @@ prerelease-mode = "allow"
 [manifest]
 members = [
     "exo",
+    "exo-bench",
     "exo-pyo3-bindings",
 ]
 
@@ -192,14 +193,20 @@ sdist = { url = "https://files.pythonhosted.org/packages/eb/56/b1ba7935a17738ae8
 wheels = [
     { url = "https://files.pythonhosted.org/packages/b0/1e/d22cc63332bd59b06481ceaac49d6c507598642e2230f201649058a7e704/cffi-2.0.0-cp313-cp313-manylinux1_i686.manylinux2014_i686.manylinux_2_17_i686.manylinux_2_5_i686.whl", hash = "sha256:07b271772c100085dd28b74fa0cd81c8fb1a3ba18b21e03d7c27f3436a10606b", size = 212446, upload-time = "2025-09-08T23:23:03.472Z" },
     { url = "https://files.pythonhosted.org/packages/a9/f5/a2c23eb03b61a0b8747f211eb716446c826ad66818ddc7810cc2cc19b3f2/cffi-2.0.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d48a880098c96020b02d5a1f7d9251308510ce8858940e6fa99ece33f610838b", size = 220101, upload-time = "2025-09-08T23:23:04.792Z" },
+    { url = "https://files.pythonhosted.org/packages/f2/7f/e6647792fc5850d634695bc0e6ab4111ae88e89981d35ac269956605feba/cffi-2.0.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:f93fd8e5c8c0a4aa1f424d6173f14a892044054871c771f8566e4008eaa359d2", size = 207948, upload-time = "2025-09-08T23:23:06.127Z" },
+    { url = "https://files.pythonhosted.org/packages/cb/1e/a5a1bd6f1fb30f22573f76533de12a00bf274abcdc55c8edab639078abb6/cffi-2.0.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:dd4f05f54a52fb558f1ba9f528228066954fee3ebe629fc1660d874d040ae5a3", size = 206422, upload-time = "2025-09-08T23:23:07.753Z" },
     { url = "https://files.pythonhosted.org/packages/98/df/0a1755e750013a2081e863e7cd37e0cdd02664372c754e5560099eb7aa44/cffi-2.0.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c8d3b5532fc71b7a77c09192b4a5a200ea992702734a2e9279a37f2478236f26", size = 219499, upload-time = "2025-09-08T23:23:09.648Z" },
     { url = "https://files.pythonhosted.org/packages/50/e1/a969e687fcf9ea58e6e2a928ad5e2dd88cc12f6f0ab477e9971f2309b57c/cffi-2.0.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:d9b29c1f0ae438d5ee9acb31cadee00a58c46cc9c0b2f9038c6b0b3470877a8c", size = 222928, upload-time = "2025-09-08T23:23:10.928Z" },
     { url = "https://files.pythonhosted.org/packages/36/54/0362578dd2c9e557a28ac77698ed67323ed5b9775ca9d3fe73fe191bb5d8/cffi-2.0.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6d50360be4546678fc1b79ffe7a66265e28667840010348dd69a314145807a1b", size = 221302, upload-time = "2025-09-08T23:23:12.42Z" },
     { url = "https://files.pythonhosted.org/packages/d6/43/0e822876f87ea8a4ef95442c3d766a06a51fc5298823f884ef87aaad168c/cffi-2.0.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:24b6f81f1983e6df8db3adc38562c83f7d4a0c36162885ec7f7b77c7dcbec97b", size = 220049, upload-time = "2025-09-08T23:23:20.853Z" },
+    { url = "https://files.pythonhosted.org/packages/b4/89/76799151d9c2d2d1ead63c2429da9ea9d7aac304603de0c6e8764e6e8e70/cffi-2.0.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:12873ca6cb9b0f0d3a0da705d6086fe911591737a59f28b7936bdfed27c0d47c", size = 207793, upload-time = "2025-09-08T23:23:22.08Z" },
+    { url = "https://files.pythonhosted.org/packages/bb/dd/3465b14bb9e24ee24cb88c9e3730f6de63111fffe513492bf8c808a3547e/cffi-2.0.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:d9b97165e8aed9272a6bb17c01e3cc5871a594a446ebedc996e2397a1c1ea8ef", size = 206300, upload-time = "2025-09-08T23:23:23.314Z" },
     { url = "https://files.pythonhosted.org/packages/47/d9/d83e293854571c877a92da46fdec39158f8d7e68da75bf73581225d28e90/cffi-2.0.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:afb8db5439b81cf9c9d0c80404b60c3cc9c3add93e114dcae767f1477cb53775", size = 219244, upload-time = "2025-09-08T23:23:24.541Z" },
     { url = "https://files.pythonhosted.org/packages/2b/0f/1f177e3683aead2bb00f7679a16451d302c436b5cbf2505f0ea8146ef59e/cffi-2.0.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:737fe7d37e1a1bffe70bd5754ea763a62a066dc5913ca57e957824b72a85e205", size = 222828, upload-time = "2025-09-08T23:23:26.143Z" },
     { url = "https://files.pythonhosted.org/packages/c6/0f/cafacebd4b040e3119dcb32fed8bdef8dfe94da653155f9d0b9dc660166e/cffi-2.0.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:38100abb9d1b1435bc4cc340bb4489635dc2f0da7456590877030c9b3d40b0c1", size = 220926, upload-time = "2025-09-08T23:23:27.873Z" },
     { url = "https://files.pythonhosted.org/packages/be/b4/c56878d0d1755cf9caa54ba71e5d049479c52f9e4afc230f06822162ab2f/cffi-2.0.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:7cc09976e8b56f8cebd752f7113ad07752461f48a58cbba644139015ac24954c", size = 221593, upload-time = "2025-09-08T23:23:31.91Z" },
+    { url = "https://files.pythonhosted.org/packages/e0/0d/eb704606dfe8033e7128df5e90fee946bbcb64a04fcdaa97321309004000/cffi-2.0.0-cp314-cp314t-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:92b68146a71df78564e4ef48af17551a5ddd142e5190cdf2c5624d0c3ff5b2e8", size = 209354, upload-time = "2025-09-08T23:23:33.214Z" },
+    { url = "https://files.pythonhosted.org/packages/d8/19/3c435d727b368ca475fb8742ab97c9cb13a0de600ce86f62eab7fa3eea60/cffi-2.0.0-cp314-cp314t-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:b1e74d11748e7e98e2f426ab176d4ed720a64412b6a15054378afdb71e0f37dc", size = 208480, upload-time = "2025-09-08T23:23:34.495Z" },
     { url = "https://files.pythonhosted.org/packages/d0/44/681604464ed9541673e486521497406fadcc15b5217c3e326b061696899a/cffi-2.0.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:28a3a209b96630bca57cce802da70c266eb08c6e97e5afd61a75611ee6c64592", size = 221584, upload-time = "2025-09-08T23:23:36.096Z" },
     { url = "https://files.pythonhosted.org/packages/25/8e/342a504ff018a2825d395d44d63a767dd8ebc927ebda557fecdaca3ac33a/cffi-2.0.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7553fb2090d71822f02c629afe6042c299edf91ba1bf94951165613553984512", size = 224443, upload-time = "2025-09-08T23:23:37.328Z" },
     { url = "https://files.pythonhosted.org/packages/e1/5e/b666bacbbc60fbf415ba9988324a132c9a7a0448a9a8f125074671c0f2c3/cffi-2.0.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6c6c373cfc5c83a975506110d17457138c8c63016b563cc9ed6e056a82f13ce4", size = 223437, upload-time = "2025-09-08T23:23:38.945Z" },
@@ -305,8 +312,10 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/5c/49/498c86566a1d80e978b42f0d702795f69887005548c041636df6ae1ca64c/cryptography-46.0.3-cp311-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:01ca9ff2885f3acc98c29f1860552e37f6d7c7d013d7334ff2a9de43a449315d", size = 4450807, upload-time = "2025-10-15T23:16:56.414Z" },
     { url = "https://files.pythonhosted.org/packages/4b/0a/863a3604112174c8624a2ac3c038662d9e59970c7f926acdcfaed8d61142/cryptography-46.0.3-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:6eae65d4c3d33da080cff9c4ab1f711b15c1d9760809dad6ea763f3812d254cb", size = 4299615, upload-time = "2025-10-15T23:16:58.442Z" },
     { url = "https://files.pythonhosted.org/packages/64/02/b73a533f6b64a69f3cd3872acb6ebc12aef924d8d103133bb3ea750dc703/cryptography-46.0.3-cp311-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:e5bf0ed4490068a2e72ac03d786693adeb909981cc596425d09032d372bcc849", size = 4016800, upload-time = "2025-10-15T23:17:00.378Z" },
+    { url = "https://files.pythonhosted.org/packages/25/d5/16e41afbfa450cde85a3b7ec599bebefaef16b5c6ba4ec49a3532336ed72/cryptography-46.0.3-cp311-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:5ecfccd2329e37e9b7112a888e76d9feca2347f12f37918facbb893d7bb88ee8", size = 4984707, upload-time = "2025-10-15T23:17:01.98Z" },
     { url = "https://files.pythonhosted.org/packages/c9/56/e7e69b427c3878352c2fb9b450bd0e19ed552753491d39d7d0a2f5226d41/cryptography-46.0.3-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:a2c0cd47381a3229c403062f764160d57d4d175e022c1df84e168c6251a22eec", size = 4482541, upload-time = "2025-10-15T23:17:04.078Z" },
     { url = "https://files.pythonhosted.org/packages/78/f6/50736d40d97e8483172f1bb6e698895b92a223dba513b0ca6f06b2365339/cryptography-46.0.3-cp311-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:549e234ff32571b1f4076ac269fcce7a808d3bf98b76c8dd560e42dbc66d7d91", size = 4299464, upload-time = "2025-10-15T23:17:05.483Z" },
+    { url = "https://files.pythonhosted.org/packages/00/de/d8e26b1a855f19d9994a19c702fa2e93b0456beccbcfe437eda00e0701f2/cryptography-46.0.3-cp311-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:c0a7bb1a68a5d3471880e264621346c48665b3bf1c3759d682fc0864c540bd9e", size = 4950838, upload-time = "2025-10-15T23:17:07.425Z" },
     { url = "https://files.pythonhosted.org/packages/8f/29/798fc4ec461a1c9e9f735f2fc58741b0daae30688f41b2497dcbc9ed1355/cryptography-46.0.3-cp311-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:10b01676fc208c3e6feeb25a8b83d81767e8059e1fe86e1dc62d10a3018fa926", size = 4481596, upload-time = "2025-10-15T23:17:09.343Z" },
     { url = "https://files.pythonhosted.org/packages/15/8d/03cd48b20a573adfff7652b76271078e3045b9f49387920e7f1f631d125e/cryptography-46.0.3-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:0abf1ffd6e57c67e92af68330d05760b7b7efb243aab8377e583284dbab72c71", size = 4426782, upload-time = "2025-10-15T23:17:11.22Z" },
     { url = "https://files.pythonhosted.org/packages/fa/b1/ebacbfe53317d55cf33165bda24c86523497a6881f339f9aae5c2e13e57b/cryptography-46.0.3-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:a04bee9ab6a4da801eb9b51f1b708a1b5b5c9eb48c03f74198464c66f0d344ac", size = 4698381, upload-time = "2025-10-15T23:17:12.829Z" },
@@ -314,8 +323,10 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/c5/fd/bc1daf8230eaa075184cbbf5f8cd00ba9db4fd32d63fb83da4671b72ed8a/cryptography-46.0.3-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:39b6755623145ad5eff1dab323f4eae2a32a77a7abef2c5089a04a3d04366715", size = 4435078, upload-time = "2025-10-15T23:17:23.042Z" },
     { url = "https://files.pythonhosted.org/packages/82/98/d3bd5407ce4c60017f8ff9e63ffee4200ab3e23fe05b765cab805a7db008/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:db391fa7c66df6762ee3f00c95a89e6d428f4d60e7abc8328f4fe155b5ac6e54", size = 4293460, upload-time = "2025-10-15T23:17:24.885Z" },
     { url = "https://files.pythonhosted.org/packages/26/e9/e23e7900983c2b8af7a08098db406cf989d7f09caea7897e347598d4cd5b/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:78a97cf6a8839a48c49271cdcbd5cf37ca2c1d6b7fdd86cc864f302b5e9bf459", size = 3995237, upload-time = "2025-10-15T23:17:26.449Z" },
+    { url = "https://files.pythonhosted.org/packages/91/15/af68c509d4a138cfe299d0d7ddb14afba15233223ebd933b4bbdbc7155d3/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_ppc64le.whl", hash = "sha256:dfb781ff7eaa91a6f7fd41776ec37c5853c795d3b358d4896fdbb5df168af422", size = 4967344, upload-time = "2025-10-15T23:17:28.06Z" },
     { url = "https://files.pythonhosted.org/packages/ca/e3/8643d077c53868b681af077edf6b3cb58288b5423610f21c62aadcbe99f4/cryptography-46.0.3-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:6f61efb26e76c45c4a227835ddeae96d83624fb0d29eb5df5b96e14ed1a0afb7", size = 4466564, upload-time = "2025-10-15T23:17:29.665Z" },
     { url = "https://files.pythonhosted.org/packages/0e/43/c1e8726fa59c236ff477ff2b5dc071e54b21e5a1e51aa2cee1676f1c986f/cryptography-46.0.3-cp314-cp314t-manylinux_2_34_aarch64.whl", hash = "sha256:23b1a8f26e43f47ceb6d6a43115f33a5a37d57df4ea0ca295b780ae8546e8044", size = 4292415, upload-time = "2025-10-15T23:17:31.686Z" },
+    { url = "https://files.pythonhosted.org/packages/42/f9/2f8fefdb1aee8a8e3256a0568cffc4e6d517b256a2fe97a029b3f1b9fe7e/cryptography-46.0.3-cp314-cp314t-manylinux_2_34_ppc64le.whl", hash = "sha256:b419ae593c86b87014b9be7396b385491ad7f320bde96826d0dd174459e54665", size = 4931457, upload-time = "2025-10-15T23:17:33.478Z" },
     { url = "https://files.pythonhosted.org/packages/79/30/9b54127a9a778ccd6d27c3da7563e9f2d341826075ceab89ae3b41bf5be2/cryptography-46.0.3-cp314-cp314t-manylinux_2_34_x86_64.whl", hash = "sha256:50fc3343ac490c6b08c0cf0d704e881d0d660be923fd3076db3e932007e726e3", size = 4466074, upload-time = "2025-10-15T23:17:35.158Z" },
     { url = "https://files.pythonhosted.org/packages/ac/68/b4f4a10928e26c941b1b6a179143af9f4d27d88fe84a6a3c53592d2e76bf/cryptography-46.0.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:22d7e97932f511d6b0b04f2bfd818d73dcd5928db509460aaf48384778eb6d20", size = 4420569, upload-time = "2025-10-15T23:17:37.188Z" },
     { url = "https://files.pythonhosted.org/packages/a3/49/3746dab4c0d1979888f125226357d3262a6dd40e114ac29e3d2abdf1ec55/cryptography-46.0.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:d55f3dffadd674514ad19451161118fd010988540cee43d8bc20675e775925de", size = 4681941, upload-time = "2025-10-15T23:17:39.236Z" },
@@ -323,8 +334,10 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/26/42/fa8389d4478368743e24e61eea78846a0006caffaf72ea24a15159215a14/cryptography-46.0.3-cp38-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:15ab9b093e8f09daab0f2159bb7e47532596075139dd74365da52ecc9cb46c5d", size = 4440029, upload-time = "2025-10-15T23:17:49.837Z" },
     { url = "https://files.pythonhosted.org/packages/5f/eb/f483db0ec5ac040824f269e93dd2bd8a21ecd1027e77ad7bdf6914f2fd80/cryptography-46.0.3-cp38-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:46acf53b40ea38f9c6c229599a4a13f0d46a6c3fa9ef19fc1a124d62e338dfa0", size = 4297222, upload-time = "2025-10-15T23:17:51.357Z" },
     { url = "https://files.pythonhosted.org/packages/fd/cf/da9502c4e1912cb1da3807ea3618a6829bee8207456fbbeebc361ec38ba3/cryptography-46.0.3-cp38-abi3-manylinux_2_28_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:10ca84c4668d066a9878890047f03546f3ae0a6b8b39b697457b7757aaf18dbc", size = 4012280, upload-time = "2025-10-15T23:17:52.964Z" },
+    { url = "https://files.pythonhosted.org/packages/6b/8f/9adb86b93330e0df8b3dcf03eae67c33ba89958fc2e03862ef1ac2b42465/cryptography-46.0.3-cp38-abi3-manylinux_2_28_ppc64le.whl", hash = "sha256:36e627112085bb3b81b19fed209c05ce2a52ee8b15d161b7c643a7d5a88491f3", size = 4978958, upload-time = "2025-10-15T23:17:54.965Z" },
     { url = "https://files.pythonhosted.org/packages/d1/a0/5fa77988289c34bdb9f913f5606ecc9ada1adb5ae870bd0d1054a7021cc4/cryptography-46.0.3-cp38-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:1000713389b75c449a6e979ffc7dcc8ac90b437048766cef052d4d30b8220971", size = 4473714, upload-time = "2025-10-15T23:17:56.754Z" },
     { url = "https://files.pythonhosted.org/packages/14/e5/fc82d72a58d41c393697aa18c9abe5ae1214ff6f2a5c18ac470f92777895/cryptography-46.0.3-cp38-abi3-manylinux_2_34_aarch64.whl", hash = "sha256:b02cf04496f6576afffef5ddd04a0cb7d49cf6be16a9059d793a30b035f6b6ac", size = 4296970, upload-time = "2025-10-15T23:17:58.588Z" },
+    { url = "https://files.pythonhosted.org/packages/78/06/5663ed35438d0b09056973994f1aec467492b33bd31da36e468b01ec1097/cryptography-46.0.3-cp38-abi3-manylinux_2_34_ppc64le.whl", hash = "sha256:71e842ec9bc7abf543b47cf86b9a743baa95f4677d22baa4c7d5c69e49e9bc04", size = 4940236, upload-time = "2025-10-15T23:18:00.897Z" },
     { url = "https://files.pythonhosted.org/packages/fc/59/873633f3f2dcd8a053b8dd1d38f783043b5fce589c0f6988bf55ef57e43e/cryptography-46.0.3-cp38-abi3-manylinux_2_34_x86_64.whl", hash = "sha256:402b58fc32614f00980b66d6e56a5b4118e6cb362ae8f3fda141ba4689bd4506", size = 4472642, upload-time = "2025-10-15T23:18:02.749Z" },
     { url = "https://files.pythonhosted.org/packages/3d/39/8e71f3930e40f6877737d6f69248cf74d4e34b886a3967d32f919cc50d3b/cryptography-46.0.3-cp38-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:ef639cb3372f69ec44915fafcd6698b6cc78fbe0c2ea41be867f6ed612811963", size = 4423126, upload-time = "2025-10-15T23:18:04.85Z" },
     { url = "https://files.pythonhosted.org/packages/cd/c7/f65027c2810e14c3e7268353b1681932b87e5a48e65505d8cc17c99e36ae/cryptography-46.0.3-cp38-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:3b51b8ca4f1c6453d8829e1eb7299499ca7f313900dd4d89a24b8b87c0a780d4", size = 4686573, upload-time = "2025-10-15T23:18:06.908Z" },
@@ -364,9 +377,10 @@ dependencies = [
     { name = "hypercorn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "loguru", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "mflux", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
-    { name = "mlx", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
-    { name = "mlx", extra = ["cpu"], marker = "sys_platform == 'linux'" },
+    { name = "mlx", version = "0.30.6", source = { registry = "https://pypi.org/simple" }, extra = ["cpu"], marker = "sys_platform == 'linux'" },
+    { name = "mlx", version = "0.30.7.dev20260216+d94f81a2", source = { git = "https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git?branch=address-rdma-gpu-locks#d94f81a2ce7346f56567a17eb08c4df5a602ed40" }, marker = "sys_platform == 'darwin'" },
     { name = "mlx-lm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+    { name = "msgspec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "openai-harmony", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "psutil", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -376,6 +390,7 @@ dependencies = [
     { name = "tiktoken", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "tomlkit", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "types-aiofiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+    { name = "zstandard", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
 ]
 
 [package.dev-dependencies]
@@ -401,9 +416,10 @@ requires-dist = [
     { name = "hypercorn", specifier = ">=0.18.0" },
     { name = "loguru", specifier = ">=0.7.3" },
     { name = "mflux", specifier = "==0.15.5" },
-    { name = "mlx", marker = "sys_platform == 'darwin'", specifier = "==0.30.5" },
-    { name = "mlx", extras = ["cpu"], marker = "sys_platform == 'linux'", specifier = "==0.30.5" },
+    { name = "mlx", marker = "sys_platform == 'darwin'", git = "https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git?branch=address-rdma-gpu-locks" },
+    { name = "mlx", extras = ["cpu"], marker = "sys_platform == 'linux'", specifier = "==0.30.6" },
     { name = "mlx-lm", specifier = "==0.30.6" },
+    { name = "msgspec", specifier = ">=0.19.0" },
     { name = "openai-harmony", specifier = ">=0.0.8" },
     { name = "pillow", specifier = ">=11.0,<12.0" },
     { name = "psutil", specifier = ">=7.0.0" },
@@ -413,6 +429,7 @@ requires-dist = [
     { name = "tiktoken", specifier = ">=0.12.0" },
     { name = "tomlkit", specifier = ">=0.14.0" },
     { name = "types-aiofiles", specifier = ">=24.1.0.20250708" },
+    { name = "zstandard", specifier = ">=0.23.0" },
 ]
 
 [package.metadata.requires-dev]
@@ -425,6 +442,27 @@ dev = [
     { name = "ruff", specifier = ">=0.11.13" },
 ]
 
+[[package]]
+name = "exo-bench"
+version = "0.1.0"
+source = { editable = "bench" }
+dependencies = [
+    { name = "huggingface-hub", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+    { name = "jinja2", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+    { name = "loguru", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+    { name = "tiktoken", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+    { name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
+]
+
+[package.metadata]
+requires-dist = [
+    { name = "huggingface-hub", specifier = ">=0.33.4" },
+    { name = "jinja2", specifier = ">=3.1.0" },
+    { name = "loguru", specifier = ">=0.7.3" },
+    { name = "tiktoken", specifier = ">=0.12.0" },
+    { name = "transformers", specifier = ">=5.0.0" },
+]
+
 [[package]]
 name = "exo-pyo3-bindings"
 version = "0.1.0"
@@ -982,8 +1020,8 @@ dependencies = [
     { name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "huggingface-hub", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "matplotlib", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
-    { name = "mlx", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
-    { name = "mlx", extra = ["cuda13"], marker = "sys_platform == 'linux'" },
+    { name = "mlx", version = "0.30.6", source = { registry = "https://pypi.org/simple" }, extra = ["cuda13"], marker = "sys_platform == 'linux'" },
+    { name = "mlx", version = "0.30.7.dev20260216+d94f81a2", source = { git = "https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git?branch=address-rdma-gpu-locks#d94f81a2ce7346f56567a17eb08c4df5a602ed40" }, marker = "sys_platform == 'darwin'" },
     { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "opencv-python", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "piexif", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -1008,22 +1046,16 @@ wheels = [
 
 [[package]]
 name = "mlx"
-version = "0.30.5"
+version = "0.30.6"
 source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "mlx-metal", marker = "sys_platform == 'darwin'" },
+resolution-markers = [
+    "sys_platform == 'linux'",
 ]
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/99/bf/e399eac4371a9d80a4bff41e1500a914a852d026ef84c30c8e427964752f/mlx-0.30.5-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:1fd93ce3de56b01699dc18a9673c89510f1772125fc48f6385dca66be34500fc", size = 572736, upload-time = "2026-02-03T04:27:58.063Z" },
-    { url = "https://files.pythonhosted.org/packages/56/e3/ad176bc95791eb09dd4414811c8fc27741c3e4832824bf1f7881f8103928/mlx-0.30.5-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:8e24d40e8c5827a242d7b8ba2379c8ece5776b7163e05ceb7a110ec161582f91", size = 572738, upload-time = "2026-02-03T04:27:59.4Z" },
-    { url = "https://files.pythonhosted.org/packages/34/16/a1b4da06eec1efe8bb84ab450c790e7291a8610d619517965d71d31aa094/mlx-0.30.5-cp313-cp313-macosx_26_0_arm64.whl", hash = "sha256:481555f8632a54a36405a80348d88631564cecf1c16c3a9a9bf971b11c353887", size = 572813, upload-time = "2026-02-03T04:28:03.41Z" },
-    { url = "https://files.pythonhosted.org/packages/d6/7c/97c1539a593393c65cf19a0ae89005fafa13321cfb185dbb409fe9b0a955/mlx-0.30.5-cp313-cp313-manylinux_2_35_aarch64.whl", hash = "sha256:e89df90121910595529d37711a8a14f345e468b6e6caa453f40578acc90b7297", size = 621230, upload-time = "2026-02-03T04:28:05.632Z" },
-    { url = "https://files.pythonhosted.org/packages/47/ef/a266e0584ab5e301568eecf1b1e83ba92e87a395de6a7539404386ffd555/mlx-0.30.5-cp313-cp313-manylinux_2_35_x86_64.whl", hash = "sha256:027412cf65319c6d09d726a4910795811c995adeecd4d19b859a5c4eab0b0cf2", size = 662868, upload-time = "2026-02-03T04:28:07.034Z" },
-    { url = "https://files.pythonhosted.org/packages/a7/7f/2089917d98a12528a43f50e0cae675432322a36683252451d06819dbf8b1/mlx-0.30.5-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:ce54786846716f43b18733af1ccbd7f4801d659e6191da9d8297e88950533533", size = 574246, upload-time = "2026-02-03T04:28:08.368Z" },
-    { url = "https://files.pythonhosted.org/packages/d7/09/7950b1587fab3fb4922371628d57b643297ec7e336dcfb44acf2738b087f/mlx-0.30.5-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:da405026dadbf7e5c838bdbf941484c307ad4629b8f137e52782c3093ef00fc9", size = 574245, upload-time = "2026-02-03T04:28:10.23Z" },
-    { url = "https://files.pythonhosted.org/packages/37/86/8592f35b2cb25077bb5e8bf0566b9bb2061b36baa774db3e5990e59848a4/mlx-0.30.5-cp314-cp314-macosx_26_0_arm64.whl", hash = "sha256:d2cc2d131f65f83141ffeb858eda3ae4b694030b2e1d7ed393b3a8f5fa6944f8", size = 574262, upload-time = "2026-02-03T04:28:11.548Z" },
-    { url = "https://files.pythonhosted.org/packages/3d/c7/9393ae41bff83a69321aea8d8f481faf530f3b3f5912e364094ef2245e2b/mlx-0.30.5-cp314-cp314-manylinux_2_35_aarch64.whl", hash = "sha256:3a58a108ed4cea1f644fcf0cf2a7fbaabcc43f3022da46b61cf50a7d80fe77f8", size = 623221, upload-time = "2026-02-03T04:28:13.406Z" },
-    { url = "https://files.pythonhosted.org/packages/74/e9/ee0fa04cb294382f778c7800367e8bcceac1832d0e17554ab8cabd2fc48a/mlx-0.30.5-cp314-cp314-manylinux_2_35_x86_64.whl", hash = "sha256:e623031b72e49b3c2ffbca203cc6a853f0c6c11afe271a9b410598d3fd9479f8", size = 662788, upload-time = "2026-02-03T04:28:14.764Z" },
+    { url = "https://files.pythonhosted.org/packages/93/06/280f6f2ba80520a7109730425eda0d966658793aa0d02d8be8d351f75253/mlx-0.30.6-cp313-cp313-manylinux_2_35_aarch64.whl", hash = "sha256:67e6c9e30a9faeacc209917ef5523177cf9b086914b6b5d83ff886e4294b727d", size = 622011, upload-time = "2026-02-06T03:45:28.165Z" },
+    { url = "https://files.pythonhosted.org/packages/fe/35/f872afbee9c079cc69924d9e9c46f5663adb7da58cba3511db082dd307c1/mlx-0.30.6-cp313-cp313-manylinux_2_35_x86_64.whl", hash = "sha256:47db8b16fcb6f6c5a47c0bdb24ed377b41237017ac93aa6cb6aa206c9bdf82e4", size = 663650, upload-time = "2026-02-06T03:45:30.315Z" },
+    { url = "https://files.pythonhosted.org/packages/3f/54/9d9e06804fb2088202a2cdf60458e00b221f71420bea285720b60f9e82b5/mlx-0.30.6-cp314-cp314-manylinux_2_35_aarch64.whl", hash = "sha256:9ceddede4af0de31d1f6b3099f70e5469d60cd7c546975dedbdbeab3519cab3f", size = 624002, upload-time = "2026-02-06T03:45:36Z" },
+    { url = "https://files.pythonhosted.org/packages/42/92/3140a15a50cb1f9267a6552171e1dfa577861de53e093124bc43707f2a0e/mlx-0.30.6-cp314-cp314-manylinux_2_35_x86_64.whl", hash = "sha256:4a6ffd2d16728cf95f63a1b555d7c2eaeea686a0e6b73228bd265411cb5d77a4", size = 663569, upload-time = "2026-02-06T03:45:37.242Z" },
 ]
 
 [package.optional-dependencies]
@@ -1034,18 +1066,26 @@ cuda13 = [
     { name = "mlx-cuda-13", marker = "sys_platform == 'linux'" },
 ]
 
+[[package]]
+name = "mlx"
+version = "0.30.7.dev20260216+d94f81a2"
+source = { git = "https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git?branch=address-rdma-gpu-locks#d94f81a2ce7346f56567a17eb08c4df5a602ed40" }
+resolution-markers = [
+    "sys_platform == 'darwin'",
+]
+
 [[package]]
 name = "mlx-cpu"
-version = "0.30.5"
+version = "0.30.6"
 source = { registry = "https://pypi.org/simple" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/d7/23/5837effd6be5ed322a456623a4ad51d7c569ee0234284d96c081f73f8bda/mlx_cpu-0.30.5-py3-none-manylinux_2_35_aarch64.whl", hash = "sha256:5e848db0830f70be3e706e465badf0570ebee3379470de9fb84c8d5b45b9e2e5", size = 8685370, upload-time = "2026-02-03T03:08:32.512Z" },
-    { url = "https://files.pythonhosted.org/packages/9f/0e/b5c98e9a2d767a967b5f0e464375bb4bbc4da2190a51586ef3bdca4e7875/mlx_cpu-0.30.5-py3-none-manylinux_2_35_x86_64.whl", hash = "sha256:3d32fef8958c3e1abff39e77309d715f131609ad63c8c3be0f54e2778fbd36ba", size = 10256264, upload-time = "2026-02-03T03:08:34.872Z" },
+    { url = "https://files.pythonhosted.org/packages/c1/fd/b52c37d8514c321675bd9c8dacf1a26f493bb1004adbb3986feda76f2776/mlx_cpu-0.30.6-py3-none-manylinux_2_35_aarch64.whl", hash = "sha256:d4ca39400e06a762ab8c43c1da7e4ada87d9880b864d10eff8023dc910c76985", size = 8689310, upload-time = "2026-02-06T01:48:34.598Z" },
+    { url = "https://files.pythonhosted.org/packages/67/a1/f097b2042c3373abeb0365b52a5f5fc9176fe5c14682483b53012f714697/mlx_cpu-0.30.6-py3-none-manylinux_2_35_x86_64.whl", hash = "sha256:573baa804539b64fc3e1740d7055e80e7b47eaa4139bbace69d6f9896d2081e4", size = 10262929, upload-time = "2026-02-06T01:48:37.157Z" },
 ]
 
 [[package]]
 name = "mlx-cuda-13"
-version = "0.30.5"
+version = "0.30.6"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "nvidia-cublas", marker = "sys_platform == 'linux'" },
@@ -1054,8 +1094,8 @@ dependencies = [
     { name = "nvidia-nccl-cu13", marker = "sys_platform == 'linux'" },
 ]
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/bf/cb/10622469930c6f4024cc9853be387158d9e39e0e19d97ee07120265e9ddc/mlx_cuda_13-0.30.5-py3-none-manylinux_2_35_aarch64.whl", hash = "sha256:6cd61592b4624ced451b312f1ce0cb6c951c438d9639c1b56dc43abfebb0fbaf", size = 66899934, upload-time = "2026-02-03T04:10:55.162Z" },
-    { url = "https://files.pythonhosted.org/packages/f8/9e/144f35c6b9ceaec2ddda184868e58e2a09188e2f82547dd1eabc74adddd4/mlx_cuda_13-0.30.5-py3-none-manylinux_2_35_x86_64.whl", hash = "sha256:b88db12c6eec02667902edc5a7ef3febac0822412a1477259f06abb564d09724", size = 69738714, upload-time = "2026-02-03T04:10:58.941Z" },
+    { url = "https://files.pythonhosted.org/packages/08/6b/27cd4abed63ca9e4086536acde4486ea7a762dda5b36bb867ed954efe93a/mlx_cuda_13-0.30.6-py3-none-manylinux_2_35_aarch64.whl", hash = "sha256:1a5e58eaf4b1a4eb6cb0111ecb4099f6f1380403599f8992ec93dfbad8d9c6f1", size = 66904875, upload-time = "2026-02-06T02:49:19.183Z" },
+    { url = "https://files.pythonhosted.org/packages/b1/ae/8864834f1c4637bd2f5da7fd8a6d2fb554a0bad16fbe53bb5ecdfcf1611f/mlx_cuda_13-0.30.6-py3-none-manylinux_2_35_x86_64.whl", hash = "sha256:35792e799bbf1498f3e0792c5c34c64bd13075de6d34d88e223de4b1000bab82", size = 69745930, upload-time = "2026-02-06T02:49:22.844Z" },
 ]
 
 [[package]]
@@ -1064,7 +1104,7 @@ version = "0.30.6"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "jinja2", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
-    { name = "mlx", marker = "sys_platform == 'darwin'" },
+    { name = "mlx", version = "0.30.7.dev20260216+d94f81a2", source = { git = "https://github.com/rltakashige/mlx-jaccl-fix-small-recv.git?branch=address-rdma-gpu-locks#d94f81a2ce7346f56567a17eb08c4df5a602ed40" }, marker = "sys_platform == 'darwin'" },
     { name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "protobuf", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -1076,16 +1116,6 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/20/5f/01d281f1fa8a1521d5936659beb4f5ab1f32b463d059263cf9d4cef969d9/mlx_lm-0.30.6-py3-none-any.whl", hash = "sha256:a7405bd581eacc4bf8209d7a6b7f23629585a0d7c6740c2a97e51fee35b3b0e1", size = 379451, upload-time = "2026-02-04T21:27:43.222Z" },
 ]
 
-[[package]]
-name = "mlx-metal"
-version = "0.30.5"
-source = { registry = "https://pypi.org/simple" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/5f/60/1bb7019c10645a3a7f3c5decea9241f5ff18133da96d396dabf27dd6f6c4/mlx_metal-0.30.5-py3-none-macosx_14_0_arm64.whl", hash = "sha256:f8287d0204170231aefda0b03acbb3b54336848be83e7916d7ef18d44e23593a", size = 38259369, upload-time = "2026-02-03T04:27:19.468Z" },
-    { url = "https://files.pythonhosted.org/packages/31/4a/0b6a59629f9ee4de76c1b018a0c4d5a38844048f6ebccc2c72d2dba89bf0/mlx_metal-0.30.5-py3-none-macosx_15_0_arm64.whl", hash = "sha256:9ce0b169ec7043d46d63a7b04e048fdd51246a134c8ee7eccb9dfb8db9f3f468", size = 38254849, upload-time = "2026-02-03T04:27:23.08Z" },
-    { url = "https://files.pythonhosted.org/packages/8e/d4/038a7281a44366d963b30b3867423f59f65e3dae479cffe0e63855c600f3/mlx_metal-0.30.5-py3-none-macosx_26_0_arm64.whl", hash = "sha256:04cfe0bf2b95a294c5072e7084a708ca3e1ba9454775af17b1be2877a4606a9f", size = 47536555, upload-time = "2026-02-03T04:27:26.385Z" },
-]
-
 [[package]]
 name = "more-itertools"
 version = "10.8.0"
@@ -1104,6 +1134,32 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" },
 ]
 
+[[package]]
+name = "msgspec"
+version = "0.20.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/ea/9c/bfbd12955a49180cbd234c5d29ec6f74fe641698f0cd9df154a854fc8a15/msgspec-0.20.0.tar.gz", hash = "sha256:692349e588fde322875f8d3025ac01689fead5901e7fb18d6870a44519d62a29", size = 317862, upload-time = "2025-11-24T03:56:28.934Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/8a/d1/b902d38b6e5ba3bdddbec469bba388d647f960aeed7b5b3623a8debe8a76/msgspec-0.20.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:9c1ff8db03be7598b50dd4b4a478d6fe93faae3bd54f4f17aa004d0e46c14c46", size = 196463, upload-time = "2025-11-24T03:55:43.405Z" },
+    { url = "https://files.pythonhosted.org/packages/57/b6/eff0305961a1d9447ec2b02f8c73c8946f22564d302a504185b730c9a761/msgspec-0.20.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:f6532369ece217fd37c5ebcfd7e981f2615628c21121b7b2df9d3adcf2fd69b8", size = 188650, upload-time = "2025-11-24T03:55:44.761Z" },
+    { url = "https://files.pythonhosted.org/packages/99/93/f2ec1ae1de51d3fdee998a1ede6b2c089453a2ee82b5c1b361ed9095064a/msgspec-0.20.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f9a1697da2f85a751ac3cc6a97fceb8e937fc670947183fb2268edaf4016d1ee", size = 218834, upload-time = "2025-11-24T03:55:46.441Z" },
+    { url = "https://files.pythonhosted.org/packages/28/83/36557b04cfdc317ed8a525c4993b23e43a8fbcddaddd78619112ca07138c/msgspec-0.20.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7fac7e9c92eddcd24c19d9e5f6249760941485dff97802461ae7c995a2450111", size = 224917, upload-time = "2025-11-24T03:55:48.06Z" },
+    { url = "https://files.pythonhosted.org/packages/8f/56/362037a1ed5be0b88aced59272442c4b40065c659700f4b195a7f4d0ac88/msgspec-0.20.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:f953a66f2a3eb8d5ea64768445e2bb301d97609db052628c3e1bcb7d87192a9f", size = 222821, upload-time = "2025-11-24T03:55:49.388Z" },
+    { url = "https://files.pythonhosted.org/packages/92/75/fa2370ec341cedf663731ab7042e177b3742645c5dd4f64dc96bd9f18a6b/msgspec-0.20.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:247af0313ae64a066d3aea7ba98840f6681ccbf5c90ba9c7d17f3e39dbba679c", size = 227227, upload-time = "2025-11-24T03:55:51.125Z" },
+    { url = "https://files.pythonhosted.org/packages/bb/18/62dc13ab0260c7d741dda8dc7f481495b93ac9168cd887dda5929880eef8/msgspec-0.20.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:eead16538db1b3f7ec6e3ed1f6f7c5dec67e90f76e76b610e1ffb5671815633a", size = 196407, upload-time = "2025-11-24T03:55:55.001Z" },
+    { url = "https://files.pythonhosted.org/packages/dd/1d/b9949e4ad6953e9f9a142c7997b2f7390c81e03e93570c7c33caf65d27e1/msgspec-0.20.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:703c3bb47bf47801627fb1438f106adbfa2998fe586696d1324586a375fca238", size = 188889, upload-time = "2025-11-24T03:55:56.311Z" },
+    { url = "https://files.pythonhosted.org/packages/1e/19/f8bb2dc0f1bfe46cc7d2b6b61c5e9b5a46c62298e8f4d03bbe499c926180/msgspec-0.20.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6cdb227dc585fb109305cee0fd304c2896f02af93ecf50a9c84ee54ee67dbb42", size = 219691, upload-time = "2025-11-24T03:55:57.908Z" },
+    { url = "https://files.pythonhosted.org/packages/b8/8e/6b17e43f6eb9369d9858ee32c97959fcd515628a1df376af96c11606cf70/msgspec-0.20.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:27d35044dd8818ac1bd0fedb2feb4fbdff4e3508dd7c5d14316a12a2d96a0de0", size = 224918, upload-time = "2025-11-24T03:55:59.322Z" },
+    { url = "https://files.pythonhosted.org/packages/1c/db/0e833a177db1a4484797adba7f429d4242585980b90882cc38709e1b62df/msgspec-0.20.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b4296393a29ee42dd25947981c65506fd4ad39beaf816f614146fa0c5a6c91ae", size = 223436, upload-time = "2025-11-24T03:56:00.716Z" },
+    { url = "https://files.pythonhosted.org/packages/c3/30/d2ee787f4c918fd2b123441d49a7707ae9015e0e8e1ab51aa7967a97b90e/msgspec-0.20.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:205fbdadd0d8d861d71c8f3399fe1a82a2caf4467bc8ff9a626df34c12176980", size = 227190, upload-time = "2025-11-24T03:56:02.371Z" },
+    { url = "https://files.pythonhosted.org/packages/4c/06/3da3fc9aaa55618a8f43eb9052453cfe01f82930bca3af8cea63a89f3a11/msgspec-0.20.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:f84703e0e6ef025663dd1de828ca028774797b8155e070e795c548f76dde65d5", size = 200389, upload-time = "2025-11-24T03:56:06.375Z" },
+    { url = "https://files.pythonhosted.org/packages/83/3b/cc4270a5ceab40dfe1d1745856951b0a24fd16ac8539a66ed3004a60c91e/msgspec-0.20.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:7c83fc24dd09cf1275934ff300e3951b3adc5573f0657a643515cc16c7dee131", size = 193198, upload-time = "2025-11-24T03:56:07.742Z" },
+    { url = "https://files.pythonhosted.org/packages/cd/ae/4c7905ac53830c8e3c06fdd60e3cdcfedc0bbc993872d1549b84ea21a1bd/msgspec-0.20.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5f13ccb1c335a124e80c4562573b9b90f01ea9521a1a87f7576c2e281d547f56", size = 225973, upload-time = "2025-11-24T03:56:09.18Z" },
+    { url = "https://files.pythonhosted.org/packages/d9/da/032abac1de4d0678d99eaeadb1323bd9d247f4711c012404ba77ed6f15ca/msgspec-0.20.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:17c2b5ca19f19306fc83c96d85e606d2cc107e0caeea85066b5389f664e04846", size = 229509, upload-time = "2025-11-24T03:56:10.898Z" },
+    { url = "https://files.pythonhosted.org/packages/69/52/fdc7bdb7057a166f309e0b44929e584319e625aaba4771b60912a9321ccd/msgspec-0.20.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d931709355edabf66c2dd1a756b2d658593e79882bc81aae5964969d5a291b63", size = 230434, upload-time = "2025-11-24T03:56:12.48Z" },
+    { url = "https://files.pythonhosted.org/packages/cb/fe/1dfd5f512b26b53043884e4f34710c73e294e7cc54278c3fe28380e42c37/msgspec-0.20.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:565f915d2e540e8a0c93a01ff67f50aebe1f7e22798c6a25873f9fda8d1325f8", size = 231758, upload-time = "2025-11-24T03:56:13.765Z" },
+]
+
 [[package]]
 name = "multidict"
 version = "6.7.0"
@@ -2452,3 +2508,37 @@ wheels = [
     { url = "https://files.pythonhosted.org/packages/cf/72/6a85bba382f22cf78add705d8c3731748397d986e197e53ecc7835e76de7/yarl-1.22.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:c0232bce2170103ec23c454e54a57008a9a72b5d1c3105dc2496750da8cfa47c", size = 341473, upload-time = "2025-10-06T14:12:10.994Z" },
     { url = "https://files.pythonhosted.org/packages/73/ae/b48f95715333080afb75a4504487cbe142cae1268afc482d06692d605ae6/yarl-1.22.0-py3-none-any.whl", hash = "sha256:1380560bdba02b6b6c90de54133c81c9f2a453dee9912fe58c1dcced1edb7cff", size = 46814, upload-time = "2025-10-06T14:12:53.872Z" },
 ]
+
+[[package]]
+name = "zstandard"
+version = "0.25.0"
+source = { registry = "https://pypi.org/simple" }
+sdist = { url = "https://files.pythonhosted.org/packages/fd/aa/3e0508d5a5dd96529cdc5a97011299056e14c6505b678fd58938792794b1/zstandard-0.25.0.tar.gz", hash = "sha256:7713e1179d162cf5c7906da876ec2ccb9c3a9dcbdffef0cc7f70c3667a205f0b", size = 711513, upload-time = "2025-09-14T22:15:54.002Z" }
+wheels = [
+    { url = "https://files.pythonhosted.org/packages/35/0b/8df9c4ad06af91d39e94fa96cc010a24ac4ef1378d3efab9223cc8593d40/zstandard-0.25.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ec996f12524f88e151c339688c3897194821d7f03081ab35d31d1e12ec975e94", size = 795735, upload-time = "2025-09-14T22:17:26.042Z" },
+    { url = "https://files.pythonhosted.org/packages/3f/06/9ae96a3e5dcfd119377ba33d4c42a7d89da1efabd5cb3e366b156c45ff4d/zstandard-0.25.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a1a4ae2dec3993a32247995bdfe367fc3266da832d82f8438c8570f989753de1", size = 640440, upload-time = "2025-09-14T22:17:27.366Z" },
+    { url = "https://files.pythonhosted.org/packages/d9/14/933d27204c2bd404229c69f445862454dcc101cd69ef8c6068f15aaec12c/zstandard-0.25.0-cp313-cp313-manylinux2010_i686.manylinux2014_i686.manylinux_2_12_i686.manylinux_2_17_i686.whl", hash = "sha256:e96594a5537722fdfb79951672a2a63aec5ebfb823e7560586f7484819f2a08f", size = 5343070, upload-time = "2025-09-14T22:17:28.896Z" },
+    { url = "https://files.pythonhosted.org/packages/6d/db/ddb11011826ed7db9d0e485d13df79b58586bfdec56e5c84a928a9a78c1c/zstandard-0.25.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:bfc4e20784722098822e3eee42b8e576b379ed72cca4a7cb856ae733e62192ea", size = 5063001, upload-time = "2025-09-14T22:17:31.044Z" },
+    { url = "https://files.pythonhosted.org/packages/db/00/87466ea3f99599d02a5238498b87bf84a6348290c19571051839ca943777/zstandard-0.25.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:457ed498fc58cdc12fc48f7950e02740d4f7ae9493dd4ab2168a47c93c31298e", size = 5394120, upload-time = "2025-09-14T22:17:32.711Z" },
+    { url = "https://files.pythonhosted.org/packages/2b/95/fc5531d9c618a679a20ff6c29e2b3ef1d1f4ad66c5e161ae6ff847d102a9/zstandard-0.25.0-cp313-cp313-manylinux2014_s390x.manylinux_2_17_s390x.whl", hash = "sha256:fd7a5004eb1980d3cefe26b2685bcb0b17989901a70a1040d1ac86f1d898c551", size = 5451230, upload-time = "2025-09-14T22:17:34.41Z" },
+    { url = "https://files.pythonhosted.org/packages/63/4b/e3678b4e776db00f9f7b2fe58e547e8928ef32727d7a1ff01dea010f3f13/zstandard-0.25.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:8e735494da3db08694d26480f1493ad2cf86e99bdd53e8e9771b2752a5c0246a", size = 5547173, upload-time = "2025-09-14T22:17:36.084Z" },
+    { url = "https://files.pythonhosted.org/packages/4e/d5/ba05ed95c6b8ec30bd468dfeab20589f2cf709b5c940483e31d991f2ca58/zstandard-0.25.0-cp313-cp313-musllinux_1_1_aarch64.whl", hash = "sha256:3a39c94ad7866160a4a46d772e43311a743c316942037671beb264e395bdd611", size = 5046736, upload-time = "2025-09-14T22:17:37.891Z" },
+    { url = "https://files.pythonhosted.org/packages/50/d5/870aa06b3a76c73eced65c044b92286a3c4e00554005ff51962deef28e28/zstandard-0.25.0-cp313-cp313-musllinux_1_1_x86_64.whl", hash = "sha256:172de1f06947577d3a3005416977cce6168f2261284c02080e7ad0185faeced3", size = 5576368, upload-time = "2025-09-14T22:17:40.206Z" },
+    { url = "https://files.pythonhosted.org/packages/5d/35/398dc2ffc89d304d59bc12f0fdd931b4ce455bddf7038a0a67733a25f550/zstandard-0.25.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3c83b0188c852a47cd13ef3bf9209fb0a77fa5374958b8c53aaa699398c6bd7b", size = 4954022, upload-time = "2025-09-14T22:17:41.879Z" },
+    { url = "https://files.pythonhosted.org/packages/9a/5c/36ba1e5507d56d2213202ec2b05e8541734af5f2ce378c5d1ceaf4d88dc4/zstandard-0.25.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:1673b7199bbe763365b81a4f3252b8e80f44c9e323fc42940dc8843bfeaf9851", size = 5267889, upload-time = "2025-09-14T22:17:43.577Z" },
+    { url = "https://files.pythonhosted.org/packages/70/e8/2ec6b6fb7358b2ec0113ae202647ca7c0e9d15b61c005ae5225ad0995df5/zstandard-0.25.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:0be7622c37c183406f3dbf0cba104118eb16a4ea7359eeb5752f0794882fc250", size = 5433952, upload-time = "2025-09-14T22:17:45.271Z" },
+    { url = "https://files.pythonhosted.org/packages/7b/01/b5f4d4dbc59ef193e870495c6f1275f5b2928e01ff5a81fecb22a06e22fb/zstandard-0.25.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:5f5e4c2a23ca271c218ac025bd7d635597048b366d6f31f420aaeb715239fc98", size = 5814054, upload-time = "2025-09-14T22:17:47.08Z" },
+    { url = "https://files.pythonhosted.org/packages/b2/e5/fbd822d5c6f427cf158316d012c5a12f233473c2f9c5fe5ab1ae5d21f3d8/zstandard-0.25.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4f187a0bb61b35119d1926aee039524d1f93aaf38a9916b8c4b78ac8514a0aaf", size = 5360113, upload-time = "2025-09-14T22:17:48.893Z" },
+    { url = "https://files.pythonhosted.org/packages/3d/5c/f8923b595b55fe49e30612987ad8bf053aef555c14f05bb659dd5dbe3e8a/zstandard-0.25.0-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:e29f0cf06974c899b2c188ef7f783607dbef36da4c242eb6c82dcd8b512855e3", size = 795887, upload-time = "2025-09-14T22:17:54.198Z" },
+    { url = "https://files.pythonhosted.org/packages/8d/09/d0a2a14fc3439c5f874042dca72a79c70a532090b7ba0003be73fee37ae2/zstandard-0.25.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:05df5136bc5a011f33cd25bc9f506e7426c0c9b3f9954f056831ce68f3b6689f", size = 640658, upload-time = "2025-09-14T22:17:55.423Z" },
+    { url = "https://files.pythonhosted.org/packages/5d/7c/8b6b71b1ddd517f68ffb55e10834388d4f793c49c6b83effaaa05785b0b4/zstandard-0.25.0-cp314-cp314-manylinux2010_i686.manylinux_2_12_i686.manylinux_2_28_i686.whl", hash = "sha256:f604efd28f239cc21b3adb53eb061e2a205dc164be408e553b41ba2ffe0ca15c", size = 5379849, upload-time = "2025-09-14T22:17:57.372Z" },
+    { url = "https://files.pythonhosted.org/packages/a4/86/a48e56320d0a17189ab7a42645387334fba2200e904ee47fc5a26c1fd8ca/zstandard-0.25.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:223415140608d0f0da010499eaa8ccdb9af210a543fac54bce15babbcfc78439", size = 5058095, upload-time = "2025-09-14T22:17:59.498Z" },
+    { url = "https://files.pythonhosted.org/packages/f8/ad/eb659984ee2c0a779f9d06dbfe45e2dc39d99ff40a319895df2d3d9a48e5/zstandard-0.25.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e54296a283f3ab5a26fc9b8b5d4978ea0532f37b231644f367aa588930aa043", size = 5551751, upload-time = "2025-09-14T22:18:01.618Z" },
+    { url = "https://files.pythonhosted.org/packages/61/b3/b637faea43677eb7bd42ab204dfb7053bd5c4582bfe6b1baefa80ac0c47b/zstandard-0.25.0-cp314-cp314-manylinux2014_s390x.manylinux_2_17_s390x.manylinux_2_28_s390x.whl", hash = "sha256:ca54090275939dc8ec5dea2d2afb400e0f83444b2fc24e07df7fdef677110859", size = 6364818, upload-time = "2025-09-14T22:18:03.769Z" },
+    { url = "https://files.pythonhosted.org/packages/31/dc/cc50210e11e465c975462439a492516a73300ab8caa8f5e0902544fd748b/zstandard-0.25.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e09bb6252b6476d8d56100e8147b803befa9a12cea144bbe629dd508800d1ad0", size = 5560402, upload-time = "2025-09-14T22:18:05.954Z" },
+    { url = "https://files.pythonhosted.org/packages/c9/ae/56523ae9c142f0c08efd5e868a6da613ae76614eca1305259c3bf6a0ed43/zstandard-0.25.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9ec8c642d1ec73287ae3e726792dd86c96f5681eb8df274a757bf62b750eae7", size = 4955108, upload-time = "2025-09-14T22:18:07.68Z" },
+    { url = "https://files.pythonhosted.org/packages/98/cf/c899f2d6df0840d5e384cf4c4121458c72802e8bda19691f3b16619f51e9/zstandard-0.25.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:a4089a10e598eae6393756b036e0f419e8c1d60f44a831520f9af41c14216cf2", size = 5269248, upload-time = "2025-09-14T22:18:09.753Z" },
+    { url = "https://files.pythonhosted.org/packages/1b/c0/59e912a531d91e1c192d3085fc0f6fb2852753c301a812d856d857ea03c6/zstandard-0.25.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:f67e8f1a324a900e75b5e28ffb152bcac9fbed1cc7b43f99cd90f395c4375344", size = 5430330, upload-time = "2025-09-14T22:18:11.966Z" },
+    { url = "https://files.pythonhosted.org/packages/a0/1d/7e31db1240de2df22a58e2ea9a93fc6e38cc29353e660c0272b6735d6669/zstandard-0.25.0-cp314-cp314-musllinux_1_2_s390x.whl", hash = "sha256:9654dbc012d8b06fc3d19cc825af3f7bf8ae242226df5f83936cb39f5fdc846c", size = 5811123, upload-time = "2025-09-14T22:18:13.907Z" },
+    { url = "https://files.pythonhosted.org/packages/f6/49/fac46df5ad353d50535e118d6983069df68ca5908d4d65b8c466150a4ff1/zstandard-0.25.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:4203ce3b31aec23012d3a4cf4a2ed64d12fea5269c49aed5e4c3611b938e4088", size = 5359591, upload-time = "2025-09-14T22:18:16.465Z" },
+]