Replace LaunchDaemon with dynamic Thunderbolt Bridge loop detection (#1222)
## Motivation The previous approach installed a LaunchDaemon plist that ran periodically to disable Thunderbolt Bridge. This required full admin privileges upfront and ran regardless of whether a problematic loop existed. This change replaces that with dynamic detection - only prompting the user when an actual TB bridge loop with 3+ machines is detected, and using fine-grained SCPreferences authorization instead of full admin. ## Changes **Backend (Python):** - Added `ThunderboltBridgeStatus` model to track bridge enabled/exists state per node - Added `node_thunderbolt_bridge` and `thunderbolt_bridge_cycles` fields to State - Added `get_thunderbolt_bridge_cycles()` method to Topology class - **Robust TB bridge detection:** - Finds bridge network services from `-listnetworkserviceorder` (not `-listallhardwareports` which can miss bridges) - Checks each bridge's member interfaces via `ifconfig` to verify it contains Thunderbolt interfaces - Handles varying service names (e.g., "TB Bridge", "Thunderbolt Bridge", "Bridge (bridge0)") - Includes `service_name` in status for correct disable commands - Added warning logs for all error cases in detection - Updated `apply.py` to handle the new event type and recompute cycles on node timeout **Swift App:** - New `ThunderboltBridgeService` that monitors for cycles from cluster state - Shows NSAlert when a cycle with >2 machines is detected - Uses `SCPreferencesCreateWithAuthorization` with `system.services.systemconfiguration.network` right for targeted permissions - **Auto-cleanup of legacy LaunchDaemon:** On app startup, checks for and removes old plist/scripts (non-fatal if user cancels) - **Periodic local network checking:** Re-checks every 10s so the warning disappears when user grants permission - **Fixed ClusterState model:** Updated to decode new granular state fields (`nodeIdentities`, `nodeMemory`, `nodeSystem`, `nodeThunderboltBridge`) with computed `nodeProfiles` property for backwards compatibility - **Fixed Topology model:** Updated to match actual JSON structure where `nodes` is an array of strings (not objects) and `connections` is a nested map (not flat array) - Cleaned up `NetworkSetupHelper` by removing daemon installation code (now only handles uninstall) **Dashboard:** - Added yellow warning badge on topology when TB bridge cycle detected - On hover: highlights affected nodes in yellow on the topology graph - Shows which machines are in the cycle with friendly names - Provides copy-paste terminal command with the correct service name: ``` sudo networksetup -setnetworkserviceenabled "<service-name>" off ``` - Warning appears in all topology views (full, welcome, and minimized chat sidebar) - **Debug mode:** Shows "TB:ON" or "TB:OFF" status next to each node in the topology ## Why It Works - Cycle detection happens on the backend where we have full topology information - Only cycles with 3+ machines are flagged (2-node connections are fine) - TB bridge detection is robust: - Uses `-listnetworkserviceorder` to find bridges (works on all machines tested) - Verifies bridge membership via `ifconfig` to confirm Thunderbolt interfaces - Handles different service names across machines - The Swift app reacts to detected cycles and prompts the user once per cycle - The dashboard provides visual feedback and actionable instructions - `SCPreferencesCreateWithAuthorization` provides the minimal permissions needed to modify network service state - Legacy LaunchDaemon is automatically cleaned up on first launch with this version ## Test Plan ### Manual Testing Here EXO detected a TB bridge cycle: #### Dashboard: <img width="1363" height="884" alt="Screenshot 2026-01-21 at 10 07 30 PM" src="https://github.com/user-attachments/assets/7da9c621-0c91-42c4-898e-4952188a1f61" /> #### Hovering the warning: <img width="359" height="279" alt="Screenshot 2026-01-21 at 16 30 57" src="https://github.com/user-attachments/assets/05501dcf-3d4a-4704-9f38-257748c05a53" /> #### macOS app warning popup: <img width="270" height="410" alt="Screenshot 2026-01-21 at 16 29 08" src="https://github.com/user-attachments/assets/45714427-08c3-4fb4-9e61-144925c51adf" /> ### Which then asks for the user's password: <img width="263" height="372" alt="Screenshot 2026-01-21 at 16 29 28" src="https://github.com/user-attachments/assets/7502e591-596d-4128-8cf5-6a12674e27bc" /> Which when entered, successfully disables bridge and no longer shows the warning on dashboard. #### When it fails it shows the error message: <img width="263" height="234" alt="Screenshot 2026-01-21 at 14 45 38" src="https://github.com/user-attachments/assets/2d10b3d5-69d7-46ea-b631-d52d8651ab41" /> ### Automated Testing - Type checker: 0 errors (`uv run basedpyright`) - Linter: All checks passed (`uv run ruff check`) - Tests: 118 passed (`uv run pytest`) - Dashboard: Builds successfully (`npm run build`) --------- Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -14,6 +14,7 @@ struct ContentView: View {
|
||||
@EnvironmentObject private var networkStatusService: NetworkStatusService
|
||||
@EnvironmentObject private var localNetworkChecker: LocalNetworkChecker
|
||||
@EnvironmentObject private var updater: SparkleUpdater
|
||||
@EnvironmentObject private var thunderboltBridgeService: ThunderboltBridgeService
|
||||
@State private var focusedNode: NodeViewModel?
|
||||
@State private var deletingInstanceIDs: Set<String> = []
|
||||
@State private var showAllNodes = false
|
||||
@@ -468,6 +469,44 @@ struct ContentView: View {
|
||||
}
|
||||
}
|
||||
|
||||
/// Shows TB bridge status for all nodes from exo cluster state
|
||||
private var clusterThunderboltBridgeView: some View {
|
||||
let bridgeStatuses = stateService.latestSnapshot?.nodeThunderboltBridge ?? [:]
|
||||
let localNodeId = stateService.localNodeId
|
||||
let nodeProfiles = stateService.latestSnapshot?.nodeProfiles ?? [:]
|
||||
|
||||
return VStack(alignment: .leading, spacing: 1) {
|
||||
if bridgeStatuses.isEmpty {
|
||||
Text("Cluster TB Bridge: No data")
|
||||
.font(.caption2)
|
||||
.foregroundColor(.secondary)
|
||||
} else {
|
||||
Text("Cluster TB Bridge Status:")
|
||||
.font(.caption2)
|
||||
.foregroundColor(.secondary)
|
||||
ForEach(Array(bridgeStatuses.keys.sorted()), id: \.self) { nodeId in
|
||||
if let status = bridgeStatuses[nodeId] {
|
||||
let nodeName =
|
||||
nodeProfiles[nodeId]?.friendlyName ?? String(nodeId.prefix(8))
|
||||
let isLocal = nodeId == localNodeId
|
||||
let prefix = isLocal ? " \(nodeName) (local):" : " \(nodeName):"
|
||||
let statusText =
|
||||
!status.exists
|
||||
? "N/A"
|
||||
: (status.enabled ? "Enabled" : "Disabled")
|
||||
let color: Color =
|
||||
!status.exists
|
||||
? .secondary
|
||||
: (status.enabled ? .red : .green)
|
||||
Text("\(prefix) \(statusText)")
|
||||
.font(.caption2)
|
||||
.foregroundColor(color)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private var interfaceIpList: some View {
|
||||
let statuses = networkStatusService.status.interfaceStatuses
|
||||
return VStack(alignment: .leading, spacing: 1) {
|
||||
@@ -510,6 +549,7 @@ struct ContentView: View {
|
||||
Text(thunderboltStatusText)
|
||||
.font(.caption2)
|
||||
.foregroundColor(thunderboltStatusColor)
|
||||
clusterThunderboltBridgeView
|
||||
interfaceIpList
|
||||
rdmaStatusView
|
||||
sendBugReportButton
|
||||
|
||||
@@ -21,6 +21,7 @@ struct EXOApp: App {
|
||||
@StateObject private var networkStatusService: NetworkStatusService
|
||||
@StateObject private var localNetworkChecker: LocalNetworkChecker
|
||||
@StateObject private var updater: SparkleUpdater
|
||||
@StateObject private var thunderboltBridgeService: ThunderboltBridgeService
|
||||
private let terminationObserver: TerminationObserver
|
||||
private let ciContext = CIContext(options: nil)
|
||||
|
||||
@@ -41,10 +42,13 @@ struct EXOApp: App {
|
||||
let localNetwork = LocalNetworkChecker()
|
||||
_localNetworkChecker = StateObject(wrappedValue: localNetwork)
|
||||
_updater = StateObject(wrappedValue: updater)
|
||||
let thunderboltBridge = ThunderboltBridgeService(clusterStateService: service)
|
||||
_thunderboltBridgeService = StateObject(wrappedValue: thunderboltBridge)
|
||||
enableLaunchAtLoginIfNeeded()
|
||||
NetworkSetupHelper.ensureLaunchDaemonInstalled()
|
||||
// Check local network access BEFORE launching exo
|
||||
localNetwork.check()
|
||||
// Remove old LaunchDaemon components if they exist (from previous versions)
|
||||
cleanupLegacyNetworkSetup()
|
||||
// Check local network access periodically (warning disappears when user grants permission)
|
||||
localNetwork.startPeriodicChecking(interval: 10)
|
||||
controller.scheduleLaunch(after: 15)
|
||||
service.startPolling()
|
||||
networkStatus.startPolling()
|
||||
@@ -58,6 +62,7 @@ struct EXOApp: App {
|
||||
.environmentObject(networkStatusService)
|
||||
.environmentObject(localNetworkChecker)
|
||||
.environmentObject(updater)
|
||||
.environmentObject(thunderboltBridgeService)
|
||||
} label: {
|
||||
menuBarIcon
|
||||
}
|
||||
@@ -130,6 +135,37 @@ struct EXOApp: App {
|
||||
"Failed to register EXO for launch at login: \(error.localizedDescription)")
|
||||
}
|
||||
}
|
||||
|
||||
private func cleanupLegacyNetworkSetup() {
|
||||
guard NetworkSetupHelper.hasInstalledComponents() else { return }
|
||||
// Dispatch async to ensure app is ready before showing alert
|
||||
DispatchQueue.main.async {
|
||||
let alert = NSAlert()
|
||||
alert.messageText = "EXO Network Configuration"
|
||||
alert.informativeText =
|
||||
"EXO needs to configure local network discovery on your device. This requires granting permission once."
|
||||
alert.alertStyle = .informational
|
||||
alert.addButton(withTitle: "Continue")
|
||||
alert.addButton(withTitle: "Later")
|
||||
|
||||
let response = alert.runModal()
|
||||
guard response == .alertFirstButtonReturn else {
|
||||
Logger().info("User deferred legacy network setup cleanup")
|
||||
return
|
||||
}
|
||||
|
||||
do {
|
||||
try NetworkSetupHelper.uninstall()
|
||||
Logger().info("Cleaned up legacy network setup components")
|
||||
} catch {
|
||||
// Non-fatal: user may have cancelled admin prompt or cleanup may have
|
||||
// partially succeeded. The app will continue normally.
|
||||
Logger().warning(
|
||||
"Could not clean up legacy network setup (non-fatal): \(error.localizedDescription)"
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Helper for managing EXO's launch-at-login registration
|
||||
|
||||
@@ -5,17 +5,43 @@ import Foundation
|
||||
struct ClusterState: Decodable {
|
||||
let instances: [String: ClusterInstance]
|
||||
let runners: [String: RunnerStatusSummary]
|
||||
let nodeProfiles: [String: NodeProfile]
|
||||
let tasks: [String: ClusterTask]
|
||||
let topology: Topology?
|
||||
let downloads: [String: [NodeDownloadStatus]]
|
||||
let thunderboltBridgeCycles: [[String]]
|
||||
|
||||
// Granular node state (split from the old nodeProfiles)
|
||||
let nodeIdentities: [String: NodeIdentity]
|
||||
let nodeMemory: [String: MemoryInfo]
|
||||
let nodeSystem: [String: SystemInfo]
|
||||
let nodeThunderboltBridge: [String: ThunderboltBridgeStatus]
|
||||
|
||||
/// Computed property for backwards compatibility - merges granular state into NodeProfile
|
||||
var nodeProfiles: [String: NodeProfile] {
|
||||
var profiles: [String: NodeProfile] = [:]
|
||||
let allNodeIds = Set(nodeIdentities.keys)
|
||||
.union(nodeMemory.keys)
|
||||
.union(nodeSystem.keys)
|
||||
for nodeId in allNodeIds {
|
||||
let identity = nodeIdentities[nodeId]
|
||||
let memory = nodeMemory[nodeId]
|
||||
let system = nodeSystem[nodeId]
|
||||
profiles[nodeId] = NodeProfile(
|
||||
modelId: identity?.modelId,
|
||||
chipId: identity?.chipId,
|
||||
friendlyName: identity?.friendlyName,
|
||||
memory: memory,
|
||||
system: system
|
||||
)
|
||||
}
|
||||
return profiles
|
||||
}
|
||||
|
||||
init(from decoder: Decoder) throws {
|
||||
let container = try decoder.container(keyedBy: CodingKeys.self)
|
||||
let rawInstances = try container.decode([String: TaggedInstance].self, forKey: .instances)
|
||||
self.instances = rawInstances.mapValues(\.instance)
|
||||
self.runners = try container.decode([String: RunnerStatusSummary].self, forKey: .runners)
|
||||
self.nodeProfiles = try container.decode([String: NodeProfile].self, forKey: .nodeProfiles)
|
||||
let rawTasks =
|
||||
try container.decodeIfPresent([String: TaggedTask].self, forKey: .tasks) ?? [:]
|
||||
self.tasks = rawTasks.compactMapValues(\.task)
|
||||
@@ -24,15 +50,34 @@ struct ClusterState: Decodable {
|
||||
try container.decodeIfPresent([String: [TaggedNodeDownload]].self, forKey: .downloads)
|
||||
?? [:]
|
||||
self.downloads = rawDownloads.mapValues { $0.compactMap(\.status) }
|
||||
self.thunderboltBridgeCycles =
|
||||
try container.decodeIfPresent([[String]].self, forKey: .thunderboltBridgeCycles) ?? []
|
||||
|
||||
// Granular node state
|
||||
self.nodeIdentities =
|
||||
try container.decodeIfPresent([String: NodeIdentity].self, forKey: .nodeIdentities)
|
||||
?? [:]
|
||||
self.nodeMemory =
|
||||
try container.decodeIfPresent([String: MemoryInfo].self, forKey: .nodeMemory) ?? [:]
|
||||
self.nodeSystem =
|
||||
try container.decodeIfPresent([String: SystemInfo].self, forKey: .nodeSystem) ?? [:]
|
||||
self.nodeThunderboltBridge =
|
||||
try container.decodeIfPresent(
|
||||
[String: ThunderboltBridgeStatus].self, forKey: .nodeThunderboltBridge
|
||||
) ?? [:]
|
||||
}
|
||||
|
||||
private enum CodingKeys: String, CodingKey {
|
||||
case instances
|
||||
case runners
|
||||
case nodeProfiles
|
||||
case topology
|
||||
case tasks
|
||||
case downloads
|
||||
case thunderboltBridgeCycles
|
||||
case nodeIdentities
|
||||
case nodeMemory
|
||||
case nodeSystem
|
||||
case nodeThunderboltBridge
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,6 +147,18 @@ struct NodeProfile: Decodable {
|
||||
let system: SystemInfo?
|
||||
}
|
||||
|
||||
struct NodeIdentity: Decodable {
|
||||
let modelId: String?
|
||||
let chipId: String?
|
||||
let friendlyName: String?
|
||||
}
|
||||
|
||||
struct ThunderboltBridgeStatus: Decodable {
|
||||
let enabled: Bool
|
||||
let exists: Bool
|
||||
let serviceName: String?
|
||||
}
|
||||
|
||||
struct MemoryInfo: Decodable {
|
||||
let ramTotal: MemoryValue?
|
||||
let ramAvailable: MemoryValue?
|
||||
@@ -120,16 +177,51 @@ struct SystemInfo: Decodable {
|
||||
}
|
||||
|
||||
struct Topology: Decodable {
|
||||
let nodes: [TopologyNode]
|
||||
let connections: [TopologyConnection]?
|
||||
/// Node IDs in the topology
|
||||
let nodes: [String]
|
||||
/// Flattened list of connections (source -> sink pairs)
|
||||
let connections: [TopologyConnection]
|
||||
|
||||
init(from decoder: Decoder) throws {
|
||||
let container = try decoder.container(keyedBy: CodingKeys.self)
|
||||
self.nodes = try container.decodeIfPresent([String].self, forKey: .nodes) ?? []
|
||||
|
||||
// Connections come as nested map: { source: { sink: [edges] } }
|
||||
// We flatten to array of (source, sink) pairs
|
||||
var flatConnections: [TopologyConnection] = []
|
||||
if let nested = try container.decodeIfPresent(
|
||||
[String: [String: [AnyCodable]]].self, forKey: .connections
|
||||
) {
|
||||
for (source, sinks) in nested {
|
||||
for sink in sinks.keys {
|
||||
flatConnections.append(
|
||||
TopologyConnection(localNodeId: source, sendBackNodeId: sink))
|
||||
}
|
||||
}
|
||||
}
|
||||
self.connections = flatConnections
|
||||
}
|
||||
|
||||
private enum CodingKeys: String, CodingKey {
|
||||
case nodes
|
||||
case connections
|
||||
}
|
||||
}
|
||||
|
||||
struct TopologyNode: Decodable {
|
||||
let nodeId: String
|
||||
let nodeProfile: NodeProfile
|
||||
/// Placeholder for decoding arbitrary JSON values we don't need to inspect
|
||||
private struct AnyCodable: Decodable {
|
||||
init(from decoder: Decoder) throws {
|
||||
// Just consume the value without storing it
|
||||
_ = try? decoder.singleValueContainer().decode(Bool.self)
|
||||
_ = try? decoder.singleValueContainer().decode(Int.self)
|
||||
_ = try? decoder.singleValueContainer().decode(Double.self)
|
||||
_ = try? decoder.singleValueContainer().decode(String.self)
|
||||
_ = try? decoder.singleValueContainer().decode([AnyCodable].self)
|
||||
_ = try? decoder.singleValueContainer().decode([String: AnyCodable].self)
|
||||
}
|
||||
}
|
||||
|
||||
struct TopologyConnection: Decodable {
|
||||
struct TopologyConnection {
|
||||
let localNodeId: String
|
||||
let sendBackNodeId: String
|
||||
}
|
||||
|
||||
@@ -55,12 +55,16 @@ struct BugReportService {
|
||||
let stateData = try await stateResult
|
||||
let eventsData = try await eventsResult
|
||||
|
||||
// Extract cluster TB bridge status from exo state
|
||||
let clusterTbBridgeStatus = extractClusterTbBridgeStatus(from: stateData)
|
||||
|
||||
let reportJSON = makeReportJson(
|
||||
timestamp: timestamp,
|
||||
hostName: hostName,
|
||||
ifconfig: ifconfigText,
|
||||
debugInfo: debugInfo,
|
||||
isManual: isManual
|
||||
isManual: isManual,
|
||||
clusterTbBridgeStatus: clusterTbBridgeStatus
|
||||
)
|
||||
|
||||
let uploads: [(path: String, data: Data?)] = [
|
||||
@@ -178,18 +182,19 @@ struct BugReportService {
|
||||
}
|
||||
|
||||
private func readThunderboltBridgeDisabled() -> Bool? {
|
||||
let result = runCommand([
|
||||
"/usr/sbin/networksetup", "-getnetworkserviceenabled", "Thunderbolt Bridge",
|
||||
])
|
||||
guard result.exitCode == 0 else { return nil }
|
||||
let output = result.output.lowercased()
|
||||
if output.contains("enabled") {
|
||||
return false
|
||||
// Dynamically find the Thunderbolt Bridge service (don't assume the name)
|
||||
guard let serviceName = ThunderboltBridgeDetector.findThunderboltBridgeServiceName() else {
|
||||
// No bridge containing Thunderbolt interfaces exists
|
||||
return nil
|
||||
}
|
||||
if output.contains("disabled") {
|
||||
return true
|
||||
|
||||
guard let isEnabled = ThunderboltBridgeDetector.isServiceEnabled(serviceName: serviceName)
|
||||
else {
|
||||
return nil
|
||||
}
|
||||
return nil
|
||||
|
||||
// Return true if disabled, false if enabled
|
||||
return !isEnabled
|
||||
}
|
||||
|
||||
private func readInterfaces() -> [DebugInfo.InterfaceStatus] {
|
||||
@@ -268,11 +273,12 @@ struct BugReportService {
|
||||
hostName: String,
|
||||
ifconfig: String,
|
||||
debugInfo: DebugInfo,
|
||||
isManual: Bool
|
||||
isManual: Bool,
|
||||
clusterTbBridgeStatus: [[String: Any]]?
|
||||
) -> Data? {
|
||||
let system = readSystemMetadata()
|
||||
let exo = readExoMetadata()
|
||||
let payload: [String: Any] = [
|
||||
var payload: [String: Any] = [
|
||||
"timestamp": timestamp,
|
||||
"host": hostName,
|
||||
"ifconfig": ifconfig,
|
||||
@@ -282,9 +288,38 @@ struct BugReportService {
|
||||
"exo_commit": exo.commit as Any,
|
||||
"report_type": isManual ? "manual" : "automated",
|
||||
]
|
||||
if let tbStatus = clusterTbBridgeStatus {
|
||||
payload["cluster_thunderbolt_bridge"] = tbStatus
|
||||
}
|
||||
return try? JSONSerialization.data(withJSONObject: payload, options: [.prettyPrinted])
|
||||
}
|
||||
|
||||
/// Extracts cluster-wide Thunderbolt Bridge status from exo state JSON
|
||||
private func extractClusterTbBridgeStatus(from stateData: Data?) -> [[String: Any]]? {
|
||||
guard let data = stateData,
|
||||
let json = try? JSONSerialization.jsonObject(with: data) as? [String: Any],
|
||||
let nodeThunderboltBridge = json["node_thunderbolt_bridge"] as? [String: [String: Any]]
|
||||
else {
|
||||
return nil
|
||||
}
|
||||
|
||||
var result: [[String: Any]] = []
|
||||
for (nodeId, status) in nodeThunderboltBridge {
|
||||
var entry: [String: Any] = ["node_id": nodeId]
|
||||
if let enabled = status["enabled"] as? Bool {
|
||||
entry["enabled"] = enabled
|
||||
}
|
||||
if let exists = status["exists"] as? Bool {
|
||||
entry["exists"] = exists
|
||||
}
|
||||
if let serviceName = status["service_name"] as? String {
|
||||
entry["service_name"] = serviceName
|
||||
}
|
||||
result.append(entry)
|
||||
}
|
||||
return result.isEmpty ? nil : result
|
||||
}
|
||||
|
||||
private func readSystemMetadata() -> [String: Any] {
|
||||
let hostname = safeRunCommand(["/bin/hostname"])
|
||||
let computerName = safeRunCommand(["/usr/sbin/scutil", "--get", "ComputerName"])
|
||||
|
||||
@@ -41,6 +41,7 @@ final class LocalNetworkChecker: ObservableObject {
|
||||
|
||||
private var connection: NWConnection?
|
||||
private var checkTask: Task<Void, Never>?
|
||||
private var periodicTask: Task<Void, Never>?
|
||||
|
||||
/// Whether we've completed at least one check (stored in UserDefaults)
|
||||
private var hasCompletedInitialCheck: Bool {
|
||||
@@ -48,10 +49,39 @@ final class LocalNetworkChecker: ObservableObject {
|
||||
set { UserDefaults.standard.set(newValue, forKey: Self.hasCompletedInitialCheckKey) }
|
||||
}
|
||||
|
||||
/// Checks if local network access is working.
|
||||
/// Checks if local network access is working (one-time check).
|
||||
func check() {
|
||||
performCheck()
|
||||
}
|
||||
|
||||
/// Starts periodic checking of local network access.
|
||||
/// Re-checks every `interval` seconds so the warning disappears when user grants permission.
|
||||
func startPeriodicChecking(interval: TimeInterval = 10) {
|
||||
stopPeriodicChecking()
|
||||
// Do an immediate check first
|
||||
performCheck()
|
||||
// Then schedule periodic checks
|
||||
periodicTask = Task { [weak self] in
|
||||
while !Task.isCancelled {
|
||||
try? await Task.sleep(nanoseconds: UInt64(interval * 1_000_000_000))
|
||||
guard !Task.isCancelled else { break }
|
||||
self?.performCheck()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Stops periodic checking.
|
||||
func stopPeriodicChecking() {
|
||||
periodicTask?.cancel()
|
||||
periodicTask = nil
|
||||
}
|
||||
|
||||
private func performCheck() {
|
||||
checkTask?.cancel()
|
||||
status = .checking
|
||||
// Only show "checking" status on first check to avoid UI flicker
|
||||
if status == .unknown {
|
||||
status = .checking
|
||||
}
|
||||
|
||||
// Use longer timeout on first launch to allow time for permission prompt
|
||||
let isFirstCheck = !hasCompletedInitialCheck
|
||||
@@ -60,12 +90,15 @@ final class LocalNetworkChecker: ObservableObject {
|
||||
checkTask = Task { [weak self] in
|
||||
guard let self else { return }
|
||||
|
||||
Self.logger.info("Checking local network connectivity (first check: \(isFirstCheck))")
|
||||
Self.logger.debug("Checking local network connectivity (first check: \(isFirstCheck))")
|
||||
let result = await self.checkConnectivity(timeout: timeout)
|
||||
self.status = result
|
||||
self.hasCompletedInitialCheck = true
|
||||
|
||||
Self.logger.info("Local network check complete: \(result.displayText)")
|
||||
// Only log on state changes or first check to reduce noise
|
||||
if isFirstCheck || result != self.status {
|
||||
Self.logger.info("Local network check: \(result.displayText)")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -141,6 +174,7 @@ final class LocalNetworkChecker: ObservableObject {
|
||||
}
|
||||
|
||||
func stop() {
|
||||
stopPeriodicChecking()
|
||||
checkTask?.cancel()
|
||||
checkTask = nil
|
||||
connection?.cancel()
|
||||
|
||||
@@ -7,48 +7,10 @@ enum NetworkSetupHelper {
|
||||
private static let daemonLabel = "io.exo.networksetup"
|
||||
private static let scriptDestination =
|
||||
"/Library/Application Support/EXO/disable_bridge.sh"
|
||||
// Legacy script path from older versions
|
||||
private static let legacyScriptDestination =
|
||||
"/Library/Application Support/EXO/disable_bridge_enable_dhcp.sh"
|
||||
private static let plistDestination = "/Library/LaunchDaemons/io.exo.networksetup.plist"
|
||||
private static let requiredStartInterval: Int = 1791
|
||||
|
||||
private static let setupScript = """
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
PREFS="/Library/Preferences/SystemConfiguration/preferences.plist"
|
||||
|
||||
# Remove bridge0 interface
|
||||
ifconfig bridge0 &>/dev/null && {
|
||||
ifconfig bridge0 | grep -q 'member' && {
|
||||
ifconfig bridge0 | awk '/member/ {print $2}' | xargs -n1 ifconfig bridge0 deletem 2>/dev/null || true
|
||||
}
|
||||
ifconfig bridge0 destroy 2>/dev/null || true
|
||||
}
|
||||
|
||||
# Remove Thunderbolt Bridge from VirtualNetworkInterfaces in preferences.plist
|
||||
/usr/libexec/PlistBuddy -c "Delete :VirtualNetworkInterfaces:Bridge:bridge0" "$PREFS" 2>/dev/null || true
|
||||
|
||||
networksetup -listnetworkservices | grep -q "Thunderbolt Bridge" && {
|
||||
networksetup -setnetworkserviceenabled "Thunderbolt Bridge" off
|
||||
} || true
|
||||
"""
|
||||
|
||||
static func ensureLaunchDaemonInstalled() {
|
||||
// Use .utility priority to match NSAppleScript's internal QoS and avoid priority inversion
|
||||
Task.detached(priority: .utility) {
|
||||
do {
|
||||
if daemonAlreadyInstalled() {
|
||||
return
|
||||
}
|
||||
try await installLaunchDaemon()
|
||||
logger.info("Network setup launch daemon installed and started")
|
||||
} catch {
|
||||
logger.error(
|
||||
"Network setup launch daemon failed: \(error.localizedDescription, privacy: .public)"
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Removes all EXO network setup components from the system.
|
||||
/// This includes the LaunchDaemon, scripts, logs, and network location.
|
||||
@@ -63,8 +25,9 @@ enum NetworkSetupHelper {
|
||||
static func hasInstalledComponents() -> Bool {
|
||||
let manager = FileManager.default
|
||||
let scriptExists = manager.fileExists(atPath: scriptDestination)
|
||||
let legacyScriptExists = manager.fileExists(atPath: legacyScriptDestination)
|
||||
let plistExists = manager.fileExists(atPath: plistDestination)
|
||||
return scriptExists || plistExists
|
||||
return scriptExists || legacyScriptExists || plistExists
|
||||
}
|
||||
|
||||
private static func makeUninstallScript() -> String {
|
||||
@@ -73,6 +36,7 @@ enum NetworkSetupHelper {
|
||||
|
||||
LABEL="\(daemonLabel)"
|
||||
SCRIPT_DEST="\(scriptDestination)"
|
||||
LEGACY_SCRIPT_DEST="\(legacyScriptDestination)"
|
||||
PLIST_DEST="\(plistDestination)"
|
||||
LOG_OUT="/var/log/\(daemonLabel).log"
|
||||
LOG_ERR="/var/log/\(daemonLabel).err.log"
|
||||
@@ -83,8 +47,9 @@ enum NetworkSetupHelper {
|
||||
# Remove LaunchDaemon plist
|
||||
rm -f "$PLIST_DEST"
|
||||
|
||||
# Remove the script and parent directory if empty
|
||||
# Remove the script (current and legacy paths) and parent directory if empty
|
||||
rm -f "$SCRIPT_DEST"
|
||||
rm -f "$LEGACY_SCRIPT_DEST"
|
||||
rmdir "$(dirname "$SCRIPT_DEST")" 2>/dev/null || true
|
||||
|
||||
# Remove log files
|
||||
@@ -98,99 +63,42 @@ enum NetworkSetupHelper {
|
||||
networksetup -deletelocation exo 2>/dev/null || true
|
||||
} || true
|
||||
|
||||
# Re-enable Thunderbolt Bridge if it exists
|
||||
networksetup -listnetworkservices | grep -q "Thunderbolt Bridge" && {
|
||||
networksetup -setnetworkserviceenabled "Thunderbolt Bridge" on 2>/dev/null || true
|
||||
} || true
|
||||
# Re-enable any Thunderbolt Bridge service if it exists
|
||||
# We find it dynamically by looking for bridges containing Thunderbolt interfaces
|
||||
find_and_enable_thunderbolt_bridge() {
|
||||
# Get Thunderbolt interface devices from hardware ports
|
||||
tb_devices=$(networksetup -listallhardwareports 2>/dev/null | awk '
|
||||
/^Hardware Port:/ { port = tolower(substr($0, 16)) }
|
||||
/^Device:/ { if (port ~ /thunderbolt/) print substr($0, 9) }
|
||||
')
|
||||
[ -z "$tb_devices" ] && return 0
|
||||
|
||||
# For each bridge device, check if it contains Thunderbolt interfaces
|
||||
for bridge in bridge0 bridge1 bridge2; do
|
||||
members=$(ifconfig "$bridge" 2>/dev/null | awk '/member:/ {print $2}')
|
||||
[ -z "$members" ] && continue
|
||||
|
||||
for tb_dev in $tb_devices; do
|
||||
if echo "$members" | grep -qx "$tb_dev"; then
|
||||
# Find the service name for this bridge device
|
||||
service_name=$(networksetup -listnetworkserviceorder 2>/dev/null | awk -v dev="$bridge" '
|
||||
/^\\([0-9*]/ { gsub(/^\\([0-9*]+\\) /, ""); svc = $0 }
|
||||
/Device:/ && $0 ~ dev { print svc; exit }
|
||||
')
|
||||
if [ -n "$service_name" ]; then
|
||||
networksetup -setnetworkserviceenabled "$service_name" on 2>/dev/null || true
|
||||
return 0
|
||||
fi
|
||||
fi
|
||||
done
|
||||
done
|
||||
}
|
||||
find_and_enable_thunderbolt_bridge
|
||||
|
||||
echo "EXO network components removed successfully"
|
||||
"""
|
||||
}
|
||||
|
||||
private static func daemonAlreadyInstalled() -> Bool {
|
||||
let manager = FileManager.default
|
||||
let scriptExists = manager.fileExists(atPath: scriptDestination)
|
||||
let plistExists = manager.fileExists(atPath: plistDestination)
|
||||
guard scriptExists, plistExists else { return false }
|
||||
guard
|
||||
let installedScript = try? String(contentsOfFile: scriptDestination, encoding: .utf8),
|
||||
installedScript.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
== setupScript.trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
else {
|
||||
return false
|
||||
}
|
||||
guard
|
||||
let data = try? Data(contentsOf: URL(fileURLWithPath: plistDestination)),
|
||||
let plist = try? PropertyListSerialization.propertyList(
|
||||
from: data, options: [], format: nil) as? [String: Any]
|
||||
else {
|
||||
return false
|
||||
}
|
||||
guard
|
||||
let interval = plist["StartInterval"] as? Int,
|
||||
interval == requiredStartInterval
|
||||
else {
|
||||
return false
|
||||
}
|
||||
if let programArgs = plist["ProgramArguments"] as? [String],
|
||||
programArgs.contains(scriptDestination) == false
|
||||
{
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
private static func installLaunchDaemon() async throws {
|
||||
let installerScript = makeInstallerScript()
|
||||
try runShellAsAdmin(installerScript)
|
||||
}
|
||||
|
||||
private static func makeInstallerScript() -> String {
|
||||
"""
|
||||
set -euo pipefail
|
||||
|
||||
LABEL="\(daemonLabel)"
|
||||
SCRIPT_DEST="\(scriptDestination)"
|
||||
PLIST_DEST="\(plistDestination)"
|
||||
|
||||
mkdir -p "$(dirname "$SCRIPT_DEST")"
|
||||
|
||||
cat > "$SCRIPT_DEST" <<'EOF_SCRIPT'
|
||||
\(setupScript)
|
||||
EOF_SCRIPT
|
||||
chmod 755 "$SCRIPT_DEST"
|
||||
|
||||
cat > "$PLIST_DEST" <<'EOF_PLIST'
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>\(daemonLabel)</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/bin/bash</string>
|
||||
<string>\(scriptDestination)</string>
|
||||
</array>
|
||||
<key>StartInterval</key>
|
||||
<integer>\(requiredStartInterval)</integer>
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>/var/log/\(daemonLabel).log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/var/log/\(daemonLabel).err.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF_PLIST
|
||||
|
||||
launchctl bootout system/"$LABEL" >/dev/null 2>&1 || true
|
||||
launchctl bootstrap system "$PLIST_DEST"
|
||||
launchctl enable system/"$LABEL"
|
||||
launchctl kickstart -k system/"$LABEL"
|
||||
"""
|
||||
}
|
||||
|
||||
private static func runShellAsAdmin(_ script: String) throws {
|
||||
let escapedScript =
|
||||
script
|
||||
|
||||
@@ -153,22 +153,18 @@ private struct NetworkStatusFetcher {
|
||||
}
|
||||
|
||||
private func readThunderboltBridgeState() -> ThunderboltState? {
|
||||
let result = runCommand(["networksetup", "-getnetworkserviceenabled", "Thunderbolt Bridge"])
|
||||
guard result.exitCode == 0 else {
|
||||
let lower = result.output.lowercased() + result.error.lowercased()
|
||||
if lower.contains("not a recognized network service") {
|
||||
return .deleted
|
||||
}
|
||||
// Dynamically find the Thunderbolt Bridge service (don't assume the name)
|
||||
guard let serviceName = ThunderboltBridgeDetector.findThunderboltBridgeServiceName() else {
|
||||
// No bridge containing Thunderbolt interfaces exists
|
||||
return .deleted
|
||||
}
|
||||
|
||||
guard let isEnabled = ThunderboltBridgeDetector.isServiceEnabled(serviceName: serviceName)
|
||||
else {
|
||||
return nil
|
||||
}
|
||||
let output = result.output.lowercased()
|
||||
if output.contains("enabled") {
|
||||
return .enabled
|
||||
}
|
||||
if output.contains("disabled") {
|
||||
return .disabled
|
||||
}
|
||||
return nil
|
||||
|
||||
return isEnabled ? .enabled : .disabled
|
||||
}
|
||||
|
||||
private func readBridgeInactive() -> Bool? {
|
||||
|
||||
@@ -0,0 +1,194 @@
|
||||
import Foundation
|
||||
import os.log
|
||||
|
||||
/// Utility for dynamically detecting Thunderbolt Bridge network services.
|
||||
/// This mirrors the Python logic in info_gatherer.py - we never assume the service
|
||||
/// is named "Thunderbolt Bridge", instead we find bridges containing Thunderbolt interfaces.
|
||||
enum ThunderboltBridgeDetector {
|
||||
private static let logger = Logger(
|
||||
subsystem: "io.exo.EXO", category: "ThunderboltBridgeDetector")
|
||||
|
||||
struct CommandResult {
|
||||
let exitCode: Int32
|
||||
let output: String
|
||||
let error: String
|
||||
}
|
||||
|
||||
/// Find the network service name of a bridge containing Thunderbolt interfaces.
|
||||
/// Returns nil if no such bridge exists.
|
||||
static func findThunderboltBridgeServiceName() -> String? {
|
||||
// 1. Get all Thunderbolt interface devices (e.g., en2, en3)
|
||||
guard let thunderboltDevices = getThunderboltDevices(), !thunderboltDevices.isEmpty else {
|
||||
logger.debug("No Thunderbolt devices found")
|
||||
return nil
|
||||
}
|
||||
logger.debug("Found Thunderbolt devices: \(thunderboltDevices.joined(separator: ", "))")
|
||||
|
||||
// 2. Get bridge services from network service order
|
||||
guard let bridgeServices = getBridgeServices(), !bridgeServices.isEmpty else {
|
||||
logger.debug("No bridge services found")
|
||||
return nil
|
||||
}
|
||||
logger.debug("Found bridge services: \(bridgeServices.keys.joined(separator: ", "))")
|
||||
|
||||
// 3. Find a bridge that contains Thunderbolt interfaces
|
||||
for (bridgeDevice, serviceName) in bridgeServices {
|
||||
let members = getBridgeMembers(bridgeDevice: bridgeDevice)
|
||||
logger.debug(
|
||||
"Bridge \(bridgeDevice) (\(serviceName)) has members: \(members.joined(separator: ", "))"
|
||||
)
|
||||
|
||||
// Check if any Thunderbolt device is a member of this bridge
|
||||
if !members.isDisjoint(with: thunderboltDevices) {
|
||||
logger.info(
|
||||
"Found Thunderbolt Bridge service: '\(serviceName)' (device: \(bridgeDevice))")
|
||||
return serviceName
|
||||
}
|
||||
}
|
||||
|
||||
logger.debug("No bridge found containing Thunderbolt interfaces")
|
||||
return nil
|
||||
}
|
||||
|
||||
/// Get Thunderbolt interface device names (e.g., en2, en3) from hardware ports.
|
||||
private static func getThunderboltDevices() -> Set<String>? {
|
||||
let result = runCommand(["networksetup", "-listallhardwareports"])
|
||||
guard result.exitCode == 0 else {
|
||||
logger.warning("networksetup -listallhardwareports failed: \(result.error)")
|
||||
return nil
|
||||
}
|
||||
|
||||
var thunderboltDevices: Set<String> = []
|
||||
var currentPort: String?
|
||||
|
||||
for line in result.output.components(separatedBy: .newlines) {
|
||||
let trimmed = line.trimmingCharacters(in: .whitespaces)
|
||||
if trimmed.hasPrefix("Hardware Port:") {
|
||||
currentPort = String(trimmed.dropFirst("Hardware Port:".count)).trimmingCharacters(
|
||||
in: .whitespaces)
|
||||
} else if trimmed.hasPrefix("Device:"), let port = currentPort {
|
||||
let device = String(trimmed.dropFirst("Device:".count)).trimmingCharacters(
|
||||
in: .whitespaces)
|
||||
if port.lowercased().contains("thunderbolt") {
|
||||
thunderboltDevices.insert(device)
|
||||
}
|
||||
currentPort = nil
|
||||
}
|
||||
}
|
||||
|
||||
return thunderboltDevices
|
||||
}
|
||||
|
||||
/// Get mapping of bridge device -> service name from network service order.
|
||||
private static func getBridgeServices() -> [String: String]? {
|
||||
let result = runCommand(["networksetup", "-listnetworkserviceorder"])
|
||||
guard result.exitCode == 0 else {
|
||||
logger.warning("networksetup -listnetworkserviceorder failed: \(result.error)")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Parse service order to find bridge devices and their service names
|
||||
// Format: "(1) Service Name\n(Hardware Port: ..., Device: bridge0)\n"
|
||||
var bridgeServices: [String: String] = [:]
|
||||
var currentService: String?
|
||||
|
||||
for line in result.output.components(separatedBy: .newlines) {
|
||||
let trimmed = line.trimmingCharacters(in: .whitespaces)
|
||||
|
||||
// Match "(N) Service Name" or "(*) Service Name" (disabled)
|
||||
// but NOT "(Hardware Port: ...)" lines
|
||||
if trimmed.hasPrefix("("), trimmed.contains(")"),
|
||||
!trimmed.hasPrefix("(Hardware Port:")
|
||||
{
|
||||
if let parenEnd = trimmed.firstIndex(of: ")") {
|
||||
let afterParen = trimmed.index(after: parenEnd)
|
||||
if afterParen < trimmed.endIndex {
|
||||
currentService =
|
||||
String(trimmed[afterParen...])
|
||||
.trimmingCharacters(in: .whitespaces)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Match "(Hardware Port: ..., Device: bridgeX)"
|
||||
else if let service = currentService, trimmed.contains("Device: bridge") {
|
||||
// Extract device name from "..., Device: bridge0)"
|
||||
if let deviceRange = trimmed.range(of: "Device: ") {
|
||||
let afterDevice = trimmed[deviceRange.upperBound...]
|
||||
if let parenIndex = afterDevice.firstIndex(of: ")") {
|
||||
let device = String(afterDevice[..<parenIndex])
|
||||
bridgeServices[device] = service
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return bridgeServices
|
||||
}
|
||||
|
||||
/// Get member interfaces of a bridge device via ifconfig.
|
||||
private static func getBridgeMembers(bridgeDevice: String) -> Set<String> {
|
||||
let result = runCommand(["ifconfig", bridgeDevice])
|
||||
guard result.exitCode == 0 else {
|
||||
logger.debug("ifconfig \(bridgeDevice) failed")
|
||||
return []
|
||||
}
|
||||
|
||||
var members: Set<String> = []
|
||||
for line in result.output.components(separatedBy: .newlines) {
|
||||
let trimmed = line.trimmingCharacters(in: .whitespaces)
|
||||
if trimmed.hasPrefix("member:") {
|
||||
let parts = trimmed.split(separator: " ")
|
||||
if parts.count > 1 {
|
||||
members.insert(String(parts[1]))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return members
|
||||
}
|
||||
|
||||
/// Check if a network service is enabled.
|
||||
static func isServiceEnabled(serviceName: String) -> Bool? {
|
||||
let result = runCommand(["networksetup", "-getnetworkserviceenabled", serviceName])
|
||||
guard result.exitCode == 0 else {
|
||||
logger.warning("Failed to check if '\(serviceName)' is enabled: \(result.error)")
|
||||
return nil
|
||||
}
|
||||
|
||||
let output = result.output.lowercased().trimmingCharacters(in: .whitespacesAndNewlines)
|
||||
if output.contains("enabled") {
|
||||
return true
|
||||
}
|
||||
if output.contains("disabled") {
|
||||
return false
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
private static func runCommand(_ arguments: [String]) -> CommandResult {
|
||||
let process = Process()
|
||||
process.launchPath = "/usr/bin/env"
|
||||
process.arguments = arguments
|
||||
|
||||
let stdout = Pipe()
|
||||
let stderr = Pipe()
|
||||
process.standardOutput = stdout
|
||||
process.standardError = stderr
|
||||
|
||||
do {
|
||||
try process.run()
|
||||
} catch {
|
||||
return CommandResult(exitCode: -1, output: "", error: error.localizedDescription)
|
||||
}
|
||||
process.waitUntilExit()
|
||||
|
||||
let outputData = stdout.fileHandleForReading.readDataToEndOfFile()
|
||||
let errorData = stderr.fileHandleForReading.readDataToEndOfFile()
|
||||
|
||||
return CommandResult(
|
||||
exitCode: process.terminationStatus,
|
||||
output: String(decoding: outputData, as: UTF8.self),
|
||||
error: String(decoding: errorData, as: UTF8.self)
|
||||
)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,258 @@
|
||||
import AppKit
|
||||
import Combine
|
||||
import Foundation
|
||||
import Security
|
||||
import SystemConfiguration
|
||||
import os.log
|
||||
|
||||
@MainActor
|
||||
final class ThunderboltBridgeService: ObservableObject {
|
||||
private static let logger = Logger(subsystem: "io.exo.EXO", category: "ThunderboltBridge")
|
||||
|
||||
@Published private(set) var detectedCycle: [String]?
|
||||
@Published private(set) var hasPromptedForCurrentCycle = false
|
||||
@Published private(set) var lastError: String?
|
||||
|
||||
private weak var clusterStateService: ClusterStateService?
|
||||
private var cancellables = Set<AnyCancellable>()
|
||||
private var previousCycleSignature: String?
|
||||
|
||||
init(clusterStateService: ClusterStateService) {
|
||||
self.clusterStateService = clusterStateService
|
||||
setupObserver()
|
||||
}
|
||||
|
||||
private func setupObserver() {
|
||||
guard let service = clusterStateService else { return }
|
||||
|
||||
service.$latestSnapshot
|
||||
.compactMap { $0 }
|
||||
.sink { [weak self] snapshot in
|
||||
self?.checkForCycles(snapshot: snapshot)
|
||||
}
|
||||
.store(in: &cancellables)
|
||||
}
|
||||
|
||||
private func checkForCycles(snapshot: ClusterState) {
|
||||
let cycles = snapshot.thunderboltBridgeCycles
|
||||
|
||||
// Only consider cycles with more than 2 nodes
|
||||
guard let firstCycle = cycles.first, firstCycle.count > 2 else {
|
||||
// No problematic cycles detected, reset state
|
||||
if detectedCycle != nil {
|
||||
detectedCycle = nil
|
||||
previousCycleSignature = nil
|
||||
hasPromptedForCurrentCycle = false
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// Create a signature for this cycle to detect if it changed
|
||||
let cycleSignature = firstCycle.sorted().joined(separator: ",")
|
||||
|
||||
// If this is a new/different cycle, reset the prompt state
|
||||
if cycleSignature != previousCycleSignature {
|
||||
previousCycleSignature = cycleSignature
|
||||
hasPromptedForCurrentCycle = false
|
||||
}
|
||||
|
||||
detectedCycle = firstCycle
|
||||
|
||||
// Only prompt once per cycle
|
||||
if !hasPromptedForCurrentCycle {
|
||||
showDisableBridgePrompt(nodeIds: firstCycle)
|
||||
}
|
||||
}
|
||||
|
||||
private func showDisableBridgePrompt(nodeIds: [String]) {
|
||||
hasPromptedForCurrentCycle = true
|
||||
|
||||
// Get friendly names for the nodes if available
|
||||
let nodeNames = nodeIds.map { nodeId -> String in
|
||||
if let snapshot = clusterStateService?.latestSnapshot,
|
||||
let profile = snapshot.nodeProfiles[nodeId],
|
||||
let friendlyName = profile.friendlyName, !friendlyName.isEmpty
|
||||
{
|
||||
return friendlyName
|
||||
}
|
||||
return String(nodeId.prefix(8)) // Use first 8 chars of node ID as fallback
|
||||
}
|
||||
let machineNames = nodeNames.joined(separator: ", ")
|
||||
|
||||
let alert = NSAlert()
|
||||
alert.messageText = "Thunderbolt Bridge Loop Detected"
|
||||
alert.informativeText = """
|
||||
A Thunderbolt Bridge loop has been detected between \(nodeNames.count) machines: \(machineNames).
|
||||
|
||||
This can cause network packet storms and connectivity issues. Would you like to disable Thunderbolt Bridge on this machine to break the loop?
|
||||
"""
|
||||
alert.alertStyle = .warning
|
||||
alert.addButton(withTitle: "Disable Bridge")
|
||||
alert.addButton(withTitle: "Not Now")
|
||||
|
||||
let response = alert.runModal()
|
||||
|
||||
if response == .alertFirstButtonReturn {
|
||||
Task {
|
||||
await disableThunderboltBridge()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func disableThunderboltBridge() async {
|
||||
Self.logger.info("Attempting to disable Thunderbolt Bridge via SCPreferences")
|
||||
lastError = nil
|
||||
|
||||
do {
|
||||
try await disableThunderboltBridgeWithSCPreferences()
|
||||
Self.logger.info("Successfully disabled Thunderbolt Bridge")
|
||||
} catch {
|
||||
Self.logger.error(
|
||||
"Failed to disable Thunderbolt Bridge: \(error.localizedDescription, privacy: .public)"
|
||||
)
|
||||
lastError = error.localizedDescription
|
||||
showErrorAlert(message: error.localizedDescription)
|
||||
}
|
||||
}
|
||||
|
||||
private func disableThunderboltBridgeWithSCPreferences() async throws {
|
||||
// 1. Create authorization reference
|
||||
var authRef: AuthorizationRef?
|
||||
var status = AuthorizationCreate(nil, nil, [], &authRef)
|
||||
guard status == errAuthorizationSuccess, let authRef = authRef else {
|
||||
throw ThunderboltBridgeError.authorizationFailed
|
||||
}
|
||||
|
||||
defer { AuthorizationFree(authRef, [.destroyRights]) }
|
||||
|
||||
// 2. Request specific network configuration rights
|
||||
let rightName = "system.services.systemconfiguration.network"
|
||||
var item = AuthorizationItem(
|
||||
name: rightName,
|
||||
valueLength: 0,
|
||||
value: nil,
|
||||
flags: 0
|
||||
)
|
||||
var rights = AuthorizationRights(count: 1, items: &item)
|
||||
|
||||
status = AuthorizationCopyRights(
|
||||
authRef,
|
||||
&rights,
|
||||
nil,
|
||||
[.extendRights, .interactionAllowed],
|
||||
nil
|
||||
)
|
||||
guard status == errAuthorizationSuccess else {
|
||||
if status == errAuthorizationCanceled {
|
||||
throw ThunderboltBridgeError.authorizationCanceled
|
||||
}
|
||||
throw ThunderboltBridgeError.authorizationDenied
|
||||
}
|
||||
|
||||
// 3. Create SCPreferences with authorization
|
||||
guard
|
||||
let prefs = SCPreferencesCreateWithAuthorization(
|
||||
kCFAllocatorDefault,
|
||||
"EXO" as CFString,
|
||||
nil,
|
||||
authRef
|
||||
)
|
||||
else {
|
||||
throw ThunderboltBridgeError.preferencesCreationFailed
|
||||
}
|
||||
|
||||
// 4. Lock, modify, commit
|
||||
guard SCPreferencesLock(prefs, true) else {
|
||||
throw ThunderboltBridgeError.lockFailed
|
||||
}
|
||||
|
||||
defer {
|
||||
SCPreferencesUnlock(prefs)
|
||||
}
|
||||
|
||||
// 5. Find the Thunderbolt Bridge service dynamically (don't assume the name)
|
||||
guard let targetServiceName = ThunderboltBridgeDetector.findThunderboltBridgeServiceName()
|
||||
else {
|
||||
throw ThunderboltBridgeError.serviceNotFound
|
||||
}
|
||||
|
||||
guard let allServices = SCNetworkServiceCopyAll(prefs) as? [SCNetworkService] else {
|
||||
throw ThunderboltBridgeError.servicesNotFound
|
||||
}
|
||||
|
||||
var found = false
|
||||
for service in allServices {
|
||||
if let name = SCNetworkServiceGetName(service) as String?,
|
||||
name == targetServiceName
|
||||
{
|
||||
guard SCNetworkServiceSetEnabled(service, false) else {
|
||||
throw ThunderboltBridgeError.disableFailed
|
||||
}
|
||||
found = true
|
||||
Self.logger.info(
|
||||
"Found and disabled Thunderbolt Bridge service: '\(targetServiceName)'")
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if !found {
|
||||
throw ThunderboltBridgeError.serviceNotFound
|
||||
}
|
||||
|
||||
// 6. Commit and apply
|
||||
guard SCPreferencesCommitChanges(prefs) else {
|
||||
throw ThunderboltBridgeError.commitFailed
|
||||
}
|
||||
|
||||
guard SCPreferencesApplyChanges(prefs) else {
|
||||
throw ThunderboltBridgeError.applyFailed
|
||||
}
|
||||
}
|
||||
|
||||
private func showErrorAlert(message: String) {
|
||||
let alert = NSAlert()
|
||||
alert.messageText = "Failed to Disable Thunderbolt Bridge"
|
||||
alert.informativeText = message
|
||||
alert.alertStyle = .critical
|
||||
alert.addButton(withTitle: "OK")
|
||||
alert.runModal()
|
||||
}
|
||||
}
|
||||
|
||||
enum ThunderboltBridgeError: LocalizedError {
|
||||
case authorizationFailed
|
||||
case authorizationCanceled
|
||||
case authorizationDenied
|
||||
case preferencesCreationFailed
|
||||
case lockFailed
|
||||
case servicesNotFound
|
||||
case serviceNotFound
|
||||
case disableFailed
|
||||
case commitFailed
|
||||
case applyFailed
|
||||
|
||||
var errorDescription: String? {
|
||||
switch self {
|
||||
case .authorizationFailed:
|
||||
return "Failed to create authorization"
|
||||
case .authorizationCanceled:
|
||||
return "Authorization was canceled by user"
|
||||
case .authorizationDenied:
|
||||
return "Authorization was denied"
|
||||
case .preferencesCreationFailed:
|
||||
return "Failed to access network preferences"
|
||||
case .lockFailed:
|
||||
return "Failed to lock network preferences for modification"
|
||||
case .servicesNotFound:
|
||||
return "Could not retrieve network services"
|
||||
case .serviceNotFound:
|
||||
return "Thunderbolt Bridge service not found"
|
||||
case .disableFailed:
|
||||
return "Failed to disable Thunderbolt Bridge service"
|
||||
case .commitFailed:
|
||||
return "Failed to save network configuration changes"
|
||||
case .applyFailed:
|
||||
return "Failed to apply network configuration changes"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -86,7 +86,7 @@ struct TopologyViewModel {
|
||||
|
||||
extension ClusterState {
|
||||
func topologyViewModel(localNodeId: String?) -> TopologyViewModel? {
|
||||
let topologyNodeIds = Set(topology?.nodes.map(\.nodeId) ?? [])
|
||||
let topologyNodeIds = Set(topology?.nodes ?? [])
|
||||
let allNodes = nodeViewModels().filter {
|
||||
topologyNodeIds.isEmpty || topologyNodeIds.contains($0.id)
|
||||
}
|
||||
@@ -95,8 +95,8 @@ extension ClusterState {
|
||||
let nodesById = Dictionary(uniqueKeysWithValues: allNodes.map { ($0.id, $0) })
|
||||
var orderedNodes: [NodeViewModel] = []
|
||||
if let topologyNodes = topology?.nodes {
|
||||
for topoNode in topologyNodes {
|
||||
if let viewModel = nodesById[topoNode.nodeId] {
|
||||
for nodeId in topologyNodes {
|
||||
if let viewModel = nodesById[nodeId] {
|
||||
orderedNodes.append(viewModel)
|
||||
}
|
||||
}
|
||||
@@ -116,7 +116,7 @@ extension ClusterState {
|
||||
|
||||
let nodeIds = Set(orderedNodes.map(\.id))
|
||||
let edgesArray: [TopologyEdgeViewModel] =
|
||||
topology?.connections?.compactMap { connection in
|
||||
topology?.connections.compactMap { connection in
|
||||
guard nodeIds.contains(connection.localNodeId),
|
||||
nodeIds.contains(connection.sendBackNodeId)
|
||||
else { return nil }
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
topologyData,
|
||||
isTopologyMinimized,
|
||||
debugMode,
|
||||
nodeThunderboltBridge,
|
||||
type NodeInfo,
|
||||
} from "$lib/stores/app.svelte";
|
||||
|
||||
@@ -21,6 +22,7 @@
|
||||
const isMinimized = $derived(isTopologyMinimized());
|
||||
const data = $derived(topologyData());
|
||||
const debugEnabled = $derived(debugMode());
|
||||
const tbBridgeData = $derived(nodeThunderboltBridge());
|
||||
|
||||
function getNodeLabel(nodeId: string): string {
|
||||
const node = data?.nodes?.[nodeId];
|
||||
@@ -1064,6 +1066,31 @@
|
||||
.attr("fill", "rgba(179,179,179,0.7)")
|
||||
.text(` (${ramUsagePercent.toFixed(0)}%)`);
|
||||
}
|
||||
|
||||
// Debug mode: Show TB bridge status
|
||||
if (debugEnabled) {
|
||||
const tbStatus = tbBridgeData[nodeInfo.id];
|
||||
if (tbStatus) {
|
||||
const tbY =
|
||||
nodeInfo.y +
|
||||
iconBaseHeight / 2 +
|
||||
(showFullLabels ? 32 : showCompactLabels ? 26 : 22);
|
||||
const tbFontSize = showFullLabels ? 9 : 7;
|
||||
const tbColor = tbStatus.enabled
|
||||
? "rgba(234,179,8,0.9)"
|
||||
: "rgba(100,100,100,0.7)";
|
||||
const tbText = tbStatus.enabled ? "TB:ON" : "TB:OFF";
|
||||
nodeG
|
||||
.append("text")
|
||||
.attr("x", nodeInfo.x)
|
||||
.attr("y", tbY)
|
||||
.attr("text-anchor", "middle")
|
||||
.attr("fill", tbColor)
|
||||
.attr("font-size", tbFontSize)
|
||||
.attr("font-family", "SF Mono, Monaco, monospace")
|
||||
.text(tbText);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -190,6 +190,13 @@ interface RawStateResponse {
|
||||
nodeMemory?: Record<string, RawMemoryUsage>;
|
||||
nodeSystem?: Record<string, RawSystemPerformanceProfile>;
|
||||
nodeNetwork?: Record<string, RawNodeNetworkInfo>;
|
||||
// Thunderbolt bridge status per node
|
||||
nodeThunderboltBridge?: Record<
|
||||
string,
|
||||
{ enabled: boolean; exists: boolean; serviceName?: string | null }
|
||||
>;
|
||||
// Thunderbolt bridge cycles (nodes with bridge enabled forming loops)
|
||||
thunderboltBridgeCycles?: string[][];
|
||||
}
|
||||
|
||||
export interface MessageAttachment {
|
||||
@@ -420,6 +427,13 @@ class AppStore {
|
||||
selectedPreviewModelId = $state<string | null>(null);
|
||||
isLoadingPreviews = $state(false);
|
||||
lastUpdate = $state<number | null>(null);
|
||||
thunderboltBridgeCycles = $state<string[][]>([]);
|
||||
nodeThunderboltBridge = $state<
|
||||
Record<
|
||||
string,
|
||||
{ enabled: boolean; exists: boolean; serviceName?: string | null }
|
||||
>
|
||||
>({});
|
||||
|
||||
// UI state
|
||||
isTopologyMinimized = $state(false);
|
||||
@@ -1008,6 +1022,10 @@ class AppStore {
|
||||
if (data.downloads) {
|
||||
this.downloads = data.downloads;
|
||||
}
|
||||
// Thunderbolt bridge cycles
|
||||
this.thunderboltBridgeCycles = data.thunderboltBridgeCycles ?? [];
|
||||
// Thunderbolt bridge status per node
|
||||
this.nodeThunderboltBridge = data.nodeThunderboltBridge ?? {};
|
||||
this.lastUpdate = Date.now();
|
||||
} catch (error) {
|
||||
console.error("Error fetching state:", error);
|
||||
@@ -2134,6 +2152,10 @@ export const setChatSidebarVisible = (visible: boolean) =>
|
||||
appStore.setChatSidebarVisible(visible);
|
||||
export const refreshState = () => appStore.fetchState();
|
||||
|
||||
// Thunderbolt bridge status
|
||||
export const thunderboltBridgeCycles = () => appStore.thunderboltBridgeCycles;
|
||||
export const nodeThunderboltBridge = () => appStore.nodeThunderboltBridge;
|
||||
|
||||
// Image generation params
|
||||
export const imageGenerationParams = () => appStore.getImageGenerationParams();
|
||||
export const setImageGenerationParams = (
|
||||
|
||||
@@ -28,6 +28,8 @@
|
||||
toggleTopologyOnlyMode,
|
||||
chatSidebarVisible,
|
||||
toggleChatSidebarVisible,
|
||||
thunderboltBridgeCycles,
|
||||
nodeThunderboltBridge,
|
||||
type DownloadProgress,
|
||||
type PlacementPreview,
|
||||
} from "$lib/stores/app.svelte";
|
||||
@@ -49,6 +51,40 @@
|
||||
const debugEnabled = $derived(debugMode());
|
||||
const topologyOnlyEnabled = $derived(topologyOnlyMode());
|
||||
const sidebarVisible = $derived(chatSidebarVisible());
|
||||
const tbBridgeCycles = $derived(thunderboltBridgeCycles());
|
||||
const tbBridgeData = $derived(nodeThunderboltBridge());
|
||||
|
||||
// Helper to get friendly node name from node ID
|
||||
function getNodeName(nodeId: string): string {
|
||||
const node = data?.nodes?.[nodeId];
|
||||
return node?.friendly_name || nodeId.slice(0, 8) + "...";
|
||||
}
|
||||
|
||||
// Helper to get the thunderbolt bridge service name from a cycle
|
||||
function getTbBridgeServiceName(cycle: string[]): string {
|
||||
// Try to find service name from any node in the cycle
|
||||
for (const nodeId of cycle) {
|
||||
const nodeData = tbBridgeData?.[nodeId];
|
||||
if (nodeData?.serviceName) {
|
||||
return nodeData.serviceName;
|
||||
}
|
||||
}
|
||||
return "Thunderbolt Bridge"; // Fallback if no service name found
|
||||
}
|
||||
|
||||
// Copy to clipboard state and function
|
||||
let copiedCommand = $state(false);
|
||||
async function copyToClipboard(text: string) {
|
||||
try {
|
||||
await navigator.clipboard.writeText(text);
|
||||
copiedCommand = true;
|
||||
setTimeout(() => {
|
||||
copiedCommand = false;
|
||||
}, 2000);
|
||||
} catch (err) {
|
||||
console.error("Failed to copy:", err);
|
||||
}
|
||||
}
|
||||
|
||||
let mounted = $state(false);
|
||||
|
||||
@@ -1615,6 +1651,83 @@
|
||||
class="w-full h-full"
|
||||
highlightedNodes={highlightedNodes()}
|
||||
/>
|
||||
|
||||
<!-- Thunderbolt Bridge Cycle Warning -->
|
||||
{#if tbBridgeCycles.length > 0}
|
||||
{@const cycle = tbBridgeCycles[0]}
|
||||
{@const serviceName = getTbBridgeServiceName(cycle)}
|
||||
{@const disableCmd = `sudo networksetup -setnetworkserviceenabled "${serviceName}" off`}
|
||||
<div class="absolute top-4 left-4 group" role="alert">
|
||||
<div
|
||||
class="flex items-center gap-2 px-3 py-2 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm cursor-help"
|
||||
>
|
||||
<svg
|
||||
class="w-5 h-5 text-yellow-400 flex-shrink-0"
|
||||
fill="none"
|
||||
viewBox="0 0 24 24"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
|
||||
/>
|
||||
</svg>
|
||||
<span class="text-sm font-mono text-yellow-200">
|
||||
THUNDERBOLT BRIDGE CYCLE DETECTED
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<!-- Tooltip on hover -->
|
||||
<div
|
||||
class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-yellow-500/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
|
||||
>
|
||||
<p class="text-xs text-white/80 mb-2">
|
||||
A network routing cycle was detected between nodes connected
|
||||
via Thunderbolt Bridge. This can cause connectivity issues.
|
||||
</p>
|
||||
<p class="text-xs text-white/60 mb-2">
|
||||
<span class="text-yellow-300">Affected nodes:</span>
|
||||
{cycle.map(getNodeName).join(" → ")}
|
||||
</p>
|
||||
<p class="text-xs text-white/60 mb-1">
|
||||
<span class="text-yellow-300">To fix:</span> Disable the Thunderbolt
|
||||
Bridge on one of the affected nodes:
|
||||
</p>
|
||||
<button
|
||||
type="button"
|
||||
onclick={() => copyToClipboard(disableCmd)}
|
||||
class="w-full flex items-center gap-2 text-[10px] font-mono bg-exo-black/60 px-2 py-1.5 rounded text-exo-yellow break-all text-left hover:bg-exo-black/80 transition-colors cursor-pointer group/copy"
|
||||
title="Click to copy"
|
||||
>
|
||||
<span class="flex-1">{disableCmd}</span>
|
||||
<svg
|
||||
class="w-3.5 h-3.5 flex-shrink-0 text-white/40 group-hover/copy:text-exo-yellow transition-colors"
|
||||
fill="none"
|
||||
viewBox="0 0 24 24"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
{#if copiedCommand}
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M5 13l4 4L19 7"
|
||||
/>
|
||||
{:else}
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z"
|
||||
/>
|
||||
{/if}
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
<!-- Exit topology-only mode button -->
|
||||
<button
|
||||
type="button"
|
||||
@@ -1655,6 +1768,82 @@
|
||||
class="w-full h-full"
|
||||
highlightedNodes={highlightedNodes()}
|
||||
/>
|
||||
|
||||
<!-- Thunderbolt Bridge Cycle Warning -->
|
||||
{#if tbBridgeCycles.length > 0}
|
||||
{@const cycle = tbBridgeCycles[0]}
|
||||
{@const serviceName = getTbBridgeServiceName(cycle)}
|
||||
{@const disableCmd = `sudo networksetup -setnetworkserviceenabled "${serviceName}" off`}
|
||||
<div class="absolute top-4 left-4 group" role="alert">
|
||||
<div
|
||||
class="flex items-center gap-2 px-3 py-2 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm cursor-help"
|
||||
>
|
||||
<svg
|
||||
class="w-5 h-5 text-yellow-400 flex-shrink-0"
|
||||
fill="none"
|
||||
viewBox="0 0 24 24"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
|
||||
/>
|
||||
</svg>
|
||||
<span class="text-sm font-mono text-yellow-200">
|
||||
THUNDERBOLT BRIDGE CYCLE DETECTED
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<!-- Tooltip on hover -->
|
||||
<div
|
||||
class="absolute top-full left-0 mt-2 w-80 p-3 rounded border border-yellow-500/30 bg-exo-dark-gray/95 backdrop-blur-sm opacity-0 invisible group-hover:opacity-100 group-hover:visible transition-all duration-200 z-50 shadow-lg"
|
||||
>
|
||||
<p class="text-xs text-white/80 mb-2">
|
||||
A network routing cycle was detected between nodes connected
|
||||
via Thunderbolt Bridge. This can cause connectivity issues.
|
||||
</p>
|
||||
<p class="text-xs text-white/60 mb-2">
|
||||
<span class="text-yellow-300">Affected nodes:</span>
|
||||
{cycle.map(getNodeName).join(" → ")}
|
||||
</p>
|
||||
<p class="text-xs text-white/60 mb-1">
|
||||
<span class="text-yellow-300">To fix:</span> Disable the Thunderbolt
|
||||
Bridge on one of the affected nodes:
|
||||
</p>
|
||||
<button
|
||||
type="button"
|
||||
onclick={() => copyToClipboard(disableCmd)}
|
||||
class="w-full flex items-center gap-2 text-[10px] font-mono bg-exo-black/60 px-2 py-1.5 rounded text-exo-yellow break-all text-left hover:bg-exo-black/80 transition-colors cursor-pointer group/copy"
|
||||
title="Click to copy"
|
||||
>
|
||||
<span class="flex-1">{disableCmd}</span>
|
||||
<svg
|
||||
class="w-3.5 h-3.5 flex-shrink-0 text-white/40 group-hover/copy:text-exo-yellow transition-colors"
|
||||
fill="none"
|
||||
viewBox="0 0 24 24"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
{#if copiedCommand}
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M5 13l4 4L19 7"
|
||||
/>
|
||||
{:else}
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M8 16H6a2 2 0 01-2-2V6a2 2 0 012-2h8a2 2 0 012 2v2m-6 12h8a2 2 0 002-2v-8a2 2 0 00-2-2h-8a2 2 0 00-2 2v8a2 2 0 002 2z"
|
||||
/>
|
||||
{/if}
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<!-- Chat Input - Below topology -->
|
||||
@@ -2602,6 +2791,31 @@
|
||||
class="relative aspect-square bg-exo-dark-gray rounded-lg overflow-hidden"
|
||||
>
|
||||
<TopologyGraph highlightedNodes={highlightedNodes()} />
|
||||
|
||||
<!-- Thunderbolt Bridge Cycle Warning (compact) -->
|
||||
{#if tbBridgeCycles.length > 0}
|
||||
<div
|
||||
class="absolute top-2 left-2 flex items-center gap-1.5 px-2 py-1 rounded border border-yellow-500/50 bg-yellow-500/10 backdrop-blur-sm"
|
||||
title="Thunderbolt Bridge cycle detected - click to view details"
|
||||
>
|
||||
<svg
|
||||
class="w-3.5 h-3.5 text-yellow-400"
|
||||
fill="none"
|
||||
viewBox="0 0 24 24"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z"
|
||||
/>
|
||||
</svg>
|
||||
<span class="text-[10px] font-mono text-yellow-200"
|
||||
>TB CYCLE</span
|
||||
>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
</button>
|
||||
|
||||
|
||||
@@ -172,33 +172,6 @@
|
||||
}
|
||||
|
||||
let downloadOverview = $state<NodeEntry[]>([]);
|
||||
let models = $state<Array<{ id: string; storage_size_megabytes?: number }>>(
|
||||
[],
|
||||
);
|
||||
|
||||
async function fetchModels() {
|
||||
try {
|
||||
const response = await fetch("/models");
|
||||
if (response.ok) {
|
||||
const data = await response.json();
|
||||
models = data.data || [];
|
||||
}
|
||||
} catch (error) {
|
||||
console.error("Failed to fetch models:", error);
|
||||
}
|
||||
}
|
||||
|
||||
function getModelTotalBytes(
|
||||
modelId: string,
|
||||
downloadTotalBytes: number,
|
||||
): number {
|
||||
if (downloadTotalBytes > 0) return downloadTotalBytes;
|
||||
const model = models.find((m) => m.id === modelId);
|
||||
if (model?.storage_size_megabytes) {
|
||||
return model.storage_size_megabytes * 1024 * 1024;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
$effect(() => {
|
||||
try {
|
||||
@@ -373,7 +346,6 @@
|
||||
onMount(() => {
|
||||
// Ensure we fetch at least once when visiting downloads directly
|
||||
refreshState();
|
||||
fetchModels();
|
||||
});
|
||||
</script>
|
||||
|
||||
@@ -482,7 +454,7 @@
|
||||
{#if model.status !== "completed"}
|
||||
<div class="text-[11px] text-exo-light-gray font-mono">
|
||||
{formatBytes(model.downloadedBytes)} / {formatBytes(
|
||||
getModelTotalBytes(model.modelId, model.totalBytes),
|
||||
model.totalBytes,
|
||||
)}
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
@@ -87,12 +87,12 @@ def place_instance(
|
||||
|
||||
smallest_cycles = get_smallest_cycles(cycles_with_sufficient_memory)
|
||||
|
||||
smallest_tb_cycles = [
|
||||
cycle for cycle in smallest_cycles if topology.is_thunderbolt_cycle(cycle)
|
||||
smallest_rdma_cycles = [
|
||||
cycle for cycle in smallest_cycles if topology.is_rdma_cycle(cycle)
|
||||
]
|
||||
|
||||
if smallest_tb_cycles != []:
|
||||
smallest_cycles = smallest_tb_cycles
|
||||
if command.instance_meta == InstanceMeta.MlxJaccl and smallest_rdma_cycles != []:
|
||||
smallest_cycles = smallest_rdma_cycles
|
||||
|
||||
cycles_with_leaf_nodes: list[Cycle] = [
|
||||
cycle
|
||||
|
||||
@@ -197,49 +197,6 @@ def get_shard_assignments(
|
||||
)
|
||||
|
||||
|
||||
def get_hosts_from_subgraph(cycle_digraph: Topology) -> list[Host]:
|
||||
cycles = cycle_digraph.get_cycles()
|
||||
expected_length = len(list(cycle_digraph.list_nodes()))
|
||||
cycles = [cycle for cycle in cycles if len(cycle) == expected_length]
|
||||
if not cycles:
|
||||
if expected_length > 1:
|
||||
logger.warning(
|
||||
f"No cycles of length {expected_length} found even though chosen subgraph contained {expected_length} nodes"
|
||||
)
|
||||
return []
|
||||
|
||||
cycle = cycles[0]
|
||||
|
||||
get_thunderbolt = False
|
||||
if cycle_digraph.is_thunderbolt_cycle(cycle):
|
||||
get_thunderbolt = True
|
||||
|
||||
logger.debug(f"Using thunderbolt cycle: {get_thunderbolt}")
|
||||
|
||||
hosts: list[Host] = []
|
||||
for i in range(len(cycle)):
|
||||
current_node = cycle.node_ids[i]
|
||||
next_node = cycle.node_ids[(i + 1) % len(cycle)]
|
||||
|
||||
for connection in cycle_digraph.get_all_connections_between(
|
||||
source=current_node, sink=next_node
|
||||
):
|
||||
if not isinstance(connection, SocketConnection):
|
||||
continue
|
||||
|
||||
if get_thunderbolt and not connection.is_thunderbolt():
|
||||
continue
|
||||
|
||||
host = Host(
|
||||
ip=connection.sink_multiaddr.ip_address,
|
||||
port=connection.sink_multiaddr.port,
|
||||
)
|
||||
hosts.append(host)
|
||||
break
|
||||
|
||||
return hosts
|
||||
|
||||
|
||||
def get_mlx_jaccl_devices_matrix(
|
||||
selected_cycle: list[NodeId],
|
||||
cycle_digraph: Topology,
|
||||
@@ -265,9 +222,6 @@ def get_mlx_jaccl_devices_matrix(
|
||||
matrix[i][j] = conn.source_rdma_iface
|
||||
break
|
||||
else:
|
||||
logger.warning(
|
||||
f"Failed to find interface name between {node_i} and {node_j}"
|
||||
)
|
||||
raise ValueError(
|
||||
"Current jaccl backend requires all-to-all RDMA connections"
|
||||
)
|
||||
@@ -279,22 +233,11 @@ def _find_connection_ip(
|
||||
node_i: NodeId,
|
||||
node_j: NodeId,
|
||||
cycle_digraph: Topology,
|
||||
) -> Generator[tuple[str, bool]]:
|
||||
) -> Generator[str, None, None]:
|
||||
"""Find all IP addresses that connect node i to node j."""
|
||||
for connection in cycle_digraph.get_all_connections_between(node_i, node_j):
|
||||
if isinstance(connection, SocketConnection):
|
||||
yield connection.sink_multiaddr.ip_address, connection.is_thunderbolt()
|
||||
|
||||
|
||||
def _find_interface_name_for_ip(
|
||||
ip_address: str, node_network: NodeNetworkInfo
|
||||
) -> str | None:
|
||||
"""Find the interface name for an IP address on a node (any interface)."""
|
||||
for interface in node_network.interfaces:
|
||||
if interface.ip_address == ip_address:
|
||||
return interface.name
|
||||
|
||||
return None
|
||||
yield connection.sink_multiaddr.ip_address
|
||||
|
||||
|
||||
def _find_ip_prioritised(
|
||||
@@ -303,43 +246,19 @@ def _find_ip_prioritised(
|
||||
cycle_digraph: Topology,
|
||||
node_network: Mapping[NodeId, NodeNetworkInfo],
|
||||
) -> str | None:
|
||||
# TODO: Actually prioritize in the correct Ethernet > Wifi > Non-TB > TB order.
|
||||
"""Find an IP address between nodes with prioritization.
|
||||
|
||||
Priority order:
|
||||
1. en0 (Ethernet on Mac Studio, WiFi on MacBook)
|
||||
2. en1 (WiFi on Mac Studio, Ethernet on MacBook)
|
||||
3. Non-Thunderbolt connections
|
||||
4. Any other IP address
|
||||
Priority: ethernet > wifi > unknown > thunderbolt
|
||||
"""
|
||||
ips = list(_find_connection_ip(node_id, other_node_id, cycle_digraph))
|
||||
# We expect a unique iface -> ip mapping
|
||||
iface_map = {
|
||||
_find_interface_name_for_ip(
|
||||
ip, node_network.get(other_node_id, NodeNetworkInfo())
|
||||
): ip
|
||||
for ip, _ in ips
|
||||
if not ips:
|
||||
return None
|
||||
other_network = node_network.get(other_node_id, NodeNetworkInfo())
|
||||
ip_to_type = {
|
||||
iface.ip_address: iface.interface_type for iface in other_network.interfaces
|
||||
}
|
||||
|
||||
en0_ip = iface_map.get("en0")
|
||||
if en0_ip:
|
||||
return en0_ip
|
||||
|
||||
en1_ip = iface_map.get("en1")
|
||||
if en1_ip:
|
||||
return en1_ip
|
||||
|
||||
non_thunderbolt_ip = next(
|
||||
(ip for (ip, is_thunderbolt) in ips if not is_thunderbolt), None
|
||||
)
|
||||
|
||||
if non_thunderbolt_ip:
|
||||
return non_thunderbolt_ip
|
||||
|
||||
if ips:
|
||||
return ips[0][0]
|
||||
|
||||
return None
|
||||
priority = {"ethernet": 0, "wifi": 1, "unknown": 2, "thunderbolt": 3}
|
||||
return min(ips, key=lambda ip: priority.get(ip_to_type.get(ip, "unknown"), 2))
|
||||
|
||||
|
||||
def get_mlx_ring_hosts_by_node(
|
||||
@@ -381,9 +300,6 @@ def get_mlx_ring_hosts_by_node(
|
||||
node_id, other_node_id, cycle_digraph, node_network
|
||||
)
|
||||
if connection_ip is None:
|
||||
logger.warning(
|
||||
f"Failed to find prioritised connection IP between {node_id} and {other_node_id}"
|
||||
)
|
||||
raise ValueError(
|
||||
"MLX ring backend requires connectivity between neighbouring nodes"
|
||||
)
|
||||
@@ -416,9 +332,6 @@ def get_mlx_jaccl_coordinators(
|
||||
if ip is not None:
|
||||
return ip
|
||||
|
||||
logger.warning(
|
||||
f"Failed to find directly connected ip between {n} and {coordinator}"
|
||||
)
|
||||
raise ValueError(
|
||||
"Current jaccl backend requires all participating devices to be able to communicate"
|
||||
)
|
||||
|
||||
@@ -3,7 +3,6 @@ import pytest
|
||||
from exo.master.placement_utils import (
|
||||
allocate_layers_proportionally,
|
||||
filter_cycles_by_memory,
|
||||
get_hosts_from_subgraph,
|
||||
get_mlx_jaccl_coordinators,
|
||||
get_shard_assignments,
|
||||
get_smallest_cycles,
|
||||
@@ -14,7 +13,7 @@ from exo.master.tests.conftest import (
|
||||
)
|
||||
from exo.shared.models.model_cards import ModelCard, ModelId, ModelTask
|
||||
from exo.shared.topology import Topology
|
||||
from exo.shared.types.common import Host, NodeId
|
||||
from exo.shared.types.common import NodeId
|
||||
from exo.shared.types.memory import Memory
|
||||
from exo.shared.types.profiling import (
|
||||
NetworkInterfaceInfo,
|
||||
@@ -273,45 +272,6 @@ def test_get_shard_assignments(
|
||||
)
|
||||
|
||||
|
||||
def test_get_hosts_from_subgraph():
|
||||
# arrange
|
||||
node_a_id = NodeId()
|
||||
node_b_id = NodeId()
|
||||
node_c_id = NodeId()
|
||||
topology = Topology()
|
||||
|
||||
topology.add_node(node_a_id)
|
||||
topology.add_node(node_b_id)
|
||||
topology.add_node(node_c_id)
|
||||
|
||||
connection1 = Connection(
|
||||
source=node_a_id, sink=node_b_id, edge=create_socket_connection(1)
|
||||
)
|
||||
connection2 = Connection(
|
||||
source=node_b_id, sink=node_c_id, edge=create_socket_connection(2)
|
||||
)
|
||||
connection3 = Connection(
|
||||
source=node_c_id, sink=node_a_id, edge=create_socket_connection(3)
|
||||
)
|
||||
|
||||
topology.add_connection(connection1)
|
||||
topology.add_connection(connection2)
|
||||
topology.add_connection(connection3)
|
||||
|
||||
# act
|
||||
hosts = get_hosts_from_subgraph(topology)
|
||||
|
||||
# assert
|
||||
assert len(hosts) == 3
|
||||
expected_hosts = [
|
||||
Host(ip="169.254.0.1", port=1234),
|
||||
Host(ip="169.254.0.2", port=1234),
|
||||
Host(ip="169.254.0.3", port=1234),
|
||||
]
|
||||
for expected_host in expected_hosts:
|
||||
assert expected_host in hosts
|
||||
|
||||
|
||||
def test_get_mlx_jaccl_coordinators():
|
||||
# arrange
|
||||
node_a_id = NodeId()
|
||||
|
||||
@@ -30,6 +30,7 @@ from exo.shared.types.profiling import (
|
||||
NodeIdentity,
|
||||
NodeNetworkInfo,
|
||||
NodeThunderboltInfo,
|
||||
ThunderboltBridgeStatus,
|
||||
)
|
||||
from exo.shared.types.state import State
|
||||
from exo.shared.types.tasks import Task, TaskId, TaskStatus
|
||||
@@ -46,6 +47,7 @@ from exo.utils.info_gatherer.info_gatherer import (
|
||||
NodeConfig,
|
||||
NodeNetworkInterfaces,
|
||||
StaticNodeInformation,
|
||||
ThunderboltBridgeInfo,
|
||||
)
|
||||
|
||||
|
||||
@@ -225,6 +227,21 @@ def apply_node_timed_out(event: NodeTimedOut, state: State) -> State:
|
||||
for key, value in state.node_thunderbolt.items()
|
||||
if key != event.node_id
|
||||
}
|
||||
node_thunderbolt_bridge = {
|
||||
key: value
|
||||
for key, value in state.node_thunderbolt_bridge.items()
|
||||
if key != event.node_id
|
||||
}
|
||||
# Only recompute cycles if the leaving node had TB bridge enabled
|
||||
leaving_node_status = state.node_thunderbolt_bridge.get(event.node_id)
|
||||
leaving_node_had_tb_enabled = (
|
||||
leaving_node_status is not None and leaving_node_status.enabled
|
||||
)
|
||||
thunderbolt_bridge_cycles = (
|
||||
topology.get_thunderbolt_bridge_cycles(node_thunderbolt_bridge, node_network)
|
||||
if leaving_node_had_tb_enabled
|
||||
else [list(cycle) for cycle in state.thunderbolt_bridge_cycles]
|
||||
)
|
||||
return state.model_copy(
|
||||
update={
|
||||
"downloads": downloads,
|
||||
@@ -235,6 +252,8 @@ def apply_node_timed_out(event: NodeTimedOut, state: State) -> State:
|
||||
"node_system": node_system,
|
||||
"node_network": node_network,
|
||||
"node_thunderbolt": node_thunderbolt,
|
||||
"node_thunderbolt_bridge": node_thunderbolt_bridge,
|
||||
"thunderbolt_bridge_cycles": thunderbolt_bridge_cycles,
|
||||
}
|
||||
)
|
||||
|
||||
@@ -312,6 +331,22 @@ def apply_node_gathered_info(event: NodeGatheredInfo, state: State) -> State:
|
||||
if tb_conn.sink_uuid in conn_map
|
||||
]
|
||||
topology.replace_all_out_rdma_connections(event.node_id, as_rdma_conns)
|
||||
case ThunderboltBridgeInfo():
|
||||
new_tb_bridge: dict[NodeId, ThunderboltBridgeStatus] = {
|
||||
**state.node_thunderbolt_bridge,
|
||||
event.node_id: info.status,
|
||||
}
|
||||
update["node_thunderbolt_bridge"] = new_tb_bridge
|
||||
# Only recompute cycles if the enabled status changed
|
||||
old_status = state.node_thunderbolt_bridge.get(event.node_id)
|
||||
old_enabled = old_status.enabled if old_status else False
|
||||
new_enabled = info.status.enabled
|
||||
if old_enabled != new_enabled:
|
||||
update["thunderbolt_bridge_cycles"] = (
|
||||
topology.get_thunderbolt_bridge_cycles(
|
||||
new_tb_bridge, state.node_network
|
||||
)
|
||||
)
|
||||
|
||||
return state.model_copy(update=update)
|
||||
|
||||
|
||||
+86
-15
@@ -7,6 +7,11 @@ import rustworkx as rx
|
||||
from pydantic import BaseModel, ConfigDict
|
||||
|
||||
from exo.shared.types.common import NodeId
|
||||
from exo.shared.types.profiling import (
|
||||
InterfaceType,
|
||||
NodeNetworkInfo,
|
||||
ThunderboltBridgeStatus,
|
||||
)
|
||||
from exo.shared.types.topology import (
|
||||
Connection,
|
||||
Cycle,
|
||||
@@ -188,24 +193,25 @@ class Topology:
|
||||
cycles.append(Cycle(node_ids=[node_id]))
|
||||
return cycles
|
||||
|
||||
def get_cycles_tb(self) -> list[Cycle]:
|
||||
tb_edges = [
|
||||
def get_rdma_cycles(self) -> list[Cycle]:
|
||||
rdma_edges = [
|
||||
(u, v, conn)
|
||||
for u, v, conn in self._graph.weighted_edge_list()
|
||||
if conn.is_thunderbolt()
|
||||
if isinstance(conn, RDMAConnection)
|
||||
]
|
||||
|
||||
tb_graph: rx.PyDiGraph[NodeId, SocketConnection] = rx.PyDiGraph()
|
||||
tb_graph.add_nodes_from(self._graph.nodes())
|
||||
rdma_graph: rx.PyDiGraph[NodeId, SocketConnection | RDMAConnection] = (
|
||||
rx.PyDiGraph()
|
||||
)
|
||||
rdma_graph.add_nodes_from(self._graph.nodes())
|
||||
|
||||
for u, v, conn in tb_edges:
|
||||
if isinstance(conn, SocketConnection):
|
||||
tb_graph.add_edge(u, v, conn)
|
||||
for u, v, conn in rdma_edges:
|
||||
rdma_graph.add_edge(u, v, conn)
|
||||
|
||||
cycle_idxs = rx.simple_cycles(tb_graph)
|
||||
cycle_idxs = rx.simple_cycles(rdma_graph)
|
||||
cycles: list[Cycle] = []
|
||||
for cycle_idx in cycle_idxs:
|
||||
cycle = Cycle(node_ids=[tb_graph[idx] for idx in cycle_idx])
|
||||
cycle = Cycle(node_ids=[rdma_graph[idx] for idx in cycle_idx])
|
||||
cycles.append(cycle)
|
||||
|
||||
return cycles
|
||||
@@ -219,18 +225,83 @@ class Topology:
|
||||
topology.add_connection(connection)
|
||||
return topology
|
||||
|
||||
def is_thunderbolt_cycle(self, cycle: Cycle) -> bool:
|
||||
def is_rdma_cycle(self, cycle: Cycle) -> bool:
|
||||
node_idxs = [node for node in cycle]
|
||||
rx_idxs = [self._vertex_indices[idx] for idx in node_idxs]
|
||||
for rid in rx_idxs:
|
||||
for neighbor_rid in self._graph.neighbors(rid):
|
||||
if neighbor_rid not in rx_idxs:
|
||||
continue
|
||||
has_tb = False
|
||||
has_rdma = False
|
||||
for edge in self._graph.get_all_edge_data(rid, neighbor_rid):
|
||||
if edge.is_thunderbolt():
|
||||
has_tb = True
|
||||
if isinstance(edge, RDMAConnection):
|
||||
has_rdma = True
|
||||
break
|
||||
if not has_tb:
|
||||
if not has_rdma:
|
||||
return False
|
||||
return True
|
||||
|
||||
def get_thunderbolt_bridge_cycles(
|
||||
self,
|
||||
node_tb_bridge_status: Mapping[NodeId, ThunderboltBridgeStatus],
|
||||
node_network: Mapping[NodeId, NodeNetworkInfo],
|
||||
) -> list[list[NodeId]]:
|
||||
"""
|
||||
Find cycles in the Thunderbolt topology where all nodes have TB bridge enabled.
|
||||
Only returns cycles with >2 nodes (3+ machines in a loop), as cycles with
|
||||
2 or fewer nodes don't cause the broadcast storm problem.
|
||||
"""
|
||||
enabled_nodes = {
|
||||
node_id
|
||||
for node_id, status in node_tb_bridge_status.items()
|
||||
if status.enabled
|
||||
}
|
||||
|
||||
if len(enabled_nodes) < 3:
|
||||
return []
|
||||
|
||||
thunderbolt_ips = _get_ips_with_interface_type(
|
||||
enabled_nodes, node_network, "thunderbolt"
|
||||
)
|
||||
|
||||
# Build subgraph with only TB bridge enabled nodes and thunderbolt connections
|
||||
graph: rx.PyDiGraph[NodeId, SocketConnection | RDMAConnection] = rx.PyDiGraph()
|
||||
node_to_idx: dict[NodeId, int] = {}
|
||||
|
||||
for node_id in enabled_nodes:
|
||||
if node_id in self._vertex_indices:
|
||||
node_to_idx[node_id] = graph.add_node(node_id)
|
||||
|
||||
for u, v, conn in self._graph.weighted_edge_list():
|
||||
source_id, sink_id = self._graph[u], self._graph[v]
|
||||
if source_id not in node_to_idx or sink_id not in node_to_idx:
|
||||
continue
|
||||
# Include connection if it's over a thunderbolt interface
|
||||
if (
|
||||
isinstance(conn, SocketConnection)
|
||||
and conn.sink_multiaddr.ip_address in thunderbolt_ips
|
||||
):
|
||||
graph.add_edge(node_to_idx[source_id], node_to_idx[sink_id], conn)
|
||||
if isinstance(conn, RDMAConnection):
|
||||
graph.add_edge(node_to_idx[source_id], node_to_idx[sink_id], conn)
|
||||
|
||||
return [
|
||||
[graph[idx] for idx in cycle]
|
||||
for cycle in rx.simple_cycles(graph)
|
||||
if len(cycle) > 2
|
||||
]
|
||||
|
||||
|
||||
def _get_ips_with_interface_type(
|
||||
node_ids: set[NodeId],
|
||||
node_network: Mapping[NodeId, NodeNetworkInfo],
|
||||
interface_type: InterfaceType,
|
||||
) -> set[str]:
|
||||
"""Get all IP addresses on interfaces of the specified type for the given nodes."""
|
||||
ips: set[str] = set()
|
||||
for node_id in node_ids:
|
||||
network_info = node_network.get(node_id, NodeNetworkInfo())
|
||||
for iface in network_info.interfaces:
|
||||
if iface.interface_type == interface_type:
|
||||
ips.add(iface.ip_address)
|
||||
return ips
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
from collections.abc import Sequence
|
||||
from typing import Self
|
||||
from typing import Literal, Self
|
||||
|
||||
import psutil
|
||||
|
||||
@@ -48,9 +48,13 @@ class SystemPerformanceProfile(CamelCaseModel):
|
||||
ecpu_usage: float = 0.0
|
||||
|
||||
|
||||
InterfaceType = Literal["wifi", "ethernet", "thunderbolt", "unknown"]
|
||||
|
||||
|
||||
class NetworkInterfaceInfo(CamelCaseModel):
|
||||
name: str
|
||||
ip_address: str
|
||||
interface_type: InterfaceType = "unknown"
|
||||
|
||||
|
||||
class NodeIdentity(CamelCaseModel):
|
||||
@@ -71,3 +75,11 @@ class NodeThunderboltInfo(CamelCaseModel):
|
||||
"""Thunderbolt interface identifiers for a node."""
|
||||
|
||||
interfaces: Sequence[ThunderboltIdentifier] = []
|
||||
|
||||
|
||||
class ThunderboltBridgeStatus(CamelCaseModel):
|
||||
"""Whether the Thunderbolt Bridge network service is enabled on this node."""
|
||||
|
||||
enabled: bool
|
||||
exists: bool
|
||||
service_name: str | None = None
|
||||
|
||||
@@ -13,6 +13,7 @@ from exo.shared.types.profiling import (
|
||||
NodeNetworkInfo,
|
||||
NodeThunderboltInfo,
|
||||
SystemPerformanceProfile,
|
||||
ThunderboltBridgeStatus,
|
||||
)
|
||||
from exo.shared.types.tasks import Task, TaskId
|
||||
from exo.shared.types.worker.downloads import DownloadProgress
|
||||
@@ -51,6 +52,10 @@ class State(CamelCaseModel):
|
||||
node_system: Mapping[NodeId, SystemPerformanceProfile] = {}
|
||||
node_network: Mapping[NodeId, NodeNetworkInfo] = {}
|
||||
node_thunderbolt: Mapping[NodeId, NodeThunderboltInfo] = {}
|
||||
node_thunderbolt_bridge: Mapping[NodeId, ThunderboltBridgeStatus] = {}
|
||||
|
||||
# Detected cycles where all nodes have Thunderbolt bridge enabled (>2 nodes)
|
||||
thunderbolt_bridge_cycles: Sequence[Sequence[NodeId]] = []
|
||||
|
||||
@field_serializer("topology", mode="plain")
|
||||
def _encode_topology(self, value: Topology) -> TopologySnapshot:
|
||||
|
||||
@@ -21,9 +21,6 @@ class RDMAConnection(FrozenModel):
|
||||
source_rdma_iface: str
|
||||
sink_rdma_iface: str
|
||||
|
||||
def is_thunderbolt(self) -> bool:
|
||||
return True
|
||||
|
||||
|
||||
class SocketConnection(FrozenModel):
|
||||
sink_multiaddr: Multiaddr
|
||||
@@ -31,9 +28,6 @@ class SocketConnection(FrozenModel):
|
||||
def __hash__(self):
|
||||
return hash(self.sink_multiaddr.ip_address)
|
||||
|
||||
def is_thunderbolt(self) -> bool:
|
||||
return str(self.sink_multiaddr.ipv4_address).startswith("169.254")
|
||||
|
||||
|
||||
class Connection(FrozenModel):
|
||||
source: NodeId
|
||||
|
||||
@@ -19,6 +19,7 @@ from exo.shared.types.memory import Memory
|
||||
from exo.shared.types.profiling import (
|
||||
MemoryUsage,
|
||||
NetworkInterfaceInfo,
|
||||
ThunderboltBridgeStatus,
|
||||
)
|
||||
from exo.shared.types.thunderbolt import (
|
||||
ThunderboltConnection,
|
||||
@@ -34,6 +35,142 @@ from .system_info import get_friendly_name, get_model_and_chip, get_network_inte
|
||||
IS_DARWIN = sys.platform == "darwin"
|
||||
|
||||
|
||||
async def _get_thunderbolt_devices() -> set[str] | None:
|
||||
"""Get Thunderbolt interface device names (e.g., en2, en3) from hardware ports.
|
||||
|
||||
Returns None if the networksetup command fails.
|
||||
"""
|
||||
result = await anyio.run_process(
|
||||
["networksetup", "-listallhardwareports"],
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.warning(
|
||||
f"networksetup -listallhardwareports failed with code "
|
||||
f"{result.returncode}: {result.stderr.decode()}"
|
||||
)
|
||||
return None
|
||||
|
||||
output = result.stdout.decode()
|
||||
thunderbolt_devices: set[str] = set()
|
||||
current_port: str | None = None
|
||||
|
||||
for line in output.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("Hardware Port:"):
|
||||
current_port = line.split(":", 1)[1].strip()
|
||||
elif line.startswith("Device:") and current_port:
|
||||
device = line.split(":", 1)[1].strip()
|
||||
if "thunderbolt" in current_port.lower():
|
||||
thunderbolt_devices.add(device)
|
||||
current_port = None
|
||||
|
||||
return thunderbolt_devices
|
||||
|
||||
|
||||
async def _get_bridge_services() -> dict[str, str] | None:
|
||||
"""Get mapping of bridge device -> service name from network service order.
|
||||
|
||||
Returns None if the networksetup command fails.
|
||||
"""
|
||||
result = await anyio.run_process(
|
||||
["networksetup", "-listnetworkserviceorder"],
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.warning(
|
||||
f"networksetup -listnetworkserviceorder failed with code "
|
||||
f"{result.returncode}: {result.stderr.decode()}"
|
||||
)
|
||||
return None
|
||||
|
||||
# Parse service order to find bridge devices and their service names
|
||||
# Format: "(1) Service Name\n(Hardware Port: ..., Device: bridge0)\n"
|
||||
service_order_output = result.stdout.decode()
|
||||
bridge_services: dict[str, str] = {} # device -> service name
|
||||
current_service: str | None = None
|
||||
|
||||
for line in service_order_output.splitlines():
|
||||
line = line.strip()
|
||||
# Match "(N) Service Name" or "(*) Service Name" (disabled)
|
||||
# but NOT "(Hardware Port: ...)" lines
|
||||
if (
|
||||
line
|
||||
and line.startswith("(")
|
||||
and ")" in line
|
||||
and not line.startswith("(Hardware Port:")
|
||||
):
|
||||
paren_end = line.index(")")
|
||||
if paren_end + 2 <= len(line):
|
||||
current_service = line[paren_end + 2 :]
|
||||
# Match "(Hardware Port: ..., Device: bridgeX)"
|
||||
elif current_service and "Device: bridge" in line:
|
||||
# Extract device name from "..., Device: bridge0)"
|
||||
device_start = line.find("Device: ") + len("Device: ")
|
||||
device_end = line.find(")", device_start)
|
||||
if device_end > device_start:
|
||||
device = line[device_start:device_end]
|
||||
bridge_services[device] = current_service
|
||||
|
||||
return bridge_services
|
||||
|
||||
|
||||
async def _get_bridge_members(bridge_device: str) -> set[str]:
|
||||
"""Get member interfaces of a bridge device via ifconfig."""
|
||||
result = await anyio.run_process(
|
||||
["ifconfig", bridge_device],
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.debug(f"ifconfig {bridge_device} failed with code {result.returncode}")
|
||||
return set()
|
||||
|
||||
members: set[str] = set()
|
||||
ifconfig_output = result.stdout.decode()
|
||||
for line in ifconfig_output.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith("member:"):
|
||||
parts = line.split()
|
||||
if len(parts) > 1:
|
||||
members.add(parts[1])
|
||||
|
||||
return members
|
||||
|
||||
|
||||
async def _find_thunderbolt_bridge(
|
||||
bridge_services: dict[str, str], thunderbolt_devices: set[str]
|
||||
) -> str | None:
|
||||
"""Find the service name of a bridge containing Thunderbolt interfaces.
|
||||
|
||||
Returns the service name if found, None otherwise.
|
||||
"""
|
||||
for bridge_device, service_name in bridge_services.items():
|
||||
members = await _get_bridge_members(bridge_device)
|
||||
if members & thunderbolt_devices: # intersection is non-empty
|
||||
return service_name
|
||||
return None
|
||||
|
||||
|
||||
async def _is_service_enabled(service_name: str) -> bool | None:
|
||||
"""Check if a network service is enabled.
|
||||
|
||||
Returns True if enabled, False if disabled, None on error.
|
||||
"""
|
||||
result = await anyio.run_process(
|
||||
["networksetup", "-getnetworkserviceenabled", service_name],
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
logger.warning(
|
||||
f"networksetup -getnetworkserviceenabled '{service_name}' "
|
||||
f"failed with code {result.returncode}: {result.stderr.decode()}"
|
||||
)
|
||||
return None
|
||||
|
||||
stdout = result.stdout.decode().strip().lower()
|
||||
return stdout == "enabled"
|
||||
|
||||
|
||||
class StaticNodeInformation(TaggedModel):
|
||||
"""Node information that should NEVER change, to be gathered once at startup"""
|
||||
|
||||
@@ -58,6 +195,66 @@ class MacThunderboltConnections(TaggedModel):
|
||||
conns: Sequence[ThunderboltConnection]
|
||||
|
||||
|
||||
class ThunderboltBridgeInfo(TaggedModel):
|
||||
status: ThunderboltBridgeStatus
|
||||
|
||||
@classmethod
|
||||
async def gather(cls) -> Self | None:
|
||||
"""Check if a Thunderbolt Bridge network service is enabled on this node.
|
||||
|
||||
Detection approach:
|
||||
1. Find all Thunderbolt interface devices (en2, en3, etc.) from hardware ports
|
||||
2. Find bridge devices from network service order (not hardware ports, as
|
||||
bridges may not appear there)
|
||||
3. Check each bridge's members via ifconfig
|
||||
4. If a bridge contains Thunderbolt interfaces, it's a Thunderbolt Bridge
|
||||
5. Check if that network service is enabled
|
||||
"""
|
||||
if not IS_DARWIN:
|
||||
return None
|
||||
|
||||
def _no_bridge_status() -> Self:
|
||||
return cls(
|
||||
status=ThunderboltBridgeStatus(
|
||||
enabled=False, exists=False, service_name=None
|
||||
)
|
||||
)
|
||||
|
||||
try:
|
||||
tb_devices = await _get_thunderbolt_devices()
|
||||
if tb_devices is None:
|
||||
return _no_bridge_status()
|
||||
|
||||
bridge_services = await _get_bridge_services()
|
||||
if not bridge_services:
|
||||
return _no_bridge_status()
|
||||
|
||||
tb_service_name = await _find_thunderbolt_bridge(
|
||||
bridge_services, tb_devices
|
||||
)
|
||||
if not tb_service_name:
|
||||
return _no_bridge_status()
|
||||
|
||||
enabled = await _is_service_enabled(tb_service_name)
|
||||
if enabled is None:
|
||||
return cls(
|
||||
status=ThunderboltBridgeStatus(
|
||||
enabled=False, exists=True, service_name=tb_service_name
|
||||
)
|
||||
)
|
||||
|
||||
return cls(
|
||||
status=ThunderboltBridgeStatus(
|
||||
enabled=enabled,
|
||||
exists=True,
|
||||
service_name=tb_service_name,
|
||||
)
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Failed to gather Thunderbolt Bridge info: {e}")
|
||||
return None
|
||||
|
||||
|
||||
class NodeConfig(TaggedModel):
|
||||
"""Node configuration from EXO_CONFIG_FILE, reloaded from the file only at startup. Other changes should come in through the API and propagate from there"""
|
||||
|
||||
@@ -111,6 +308,7 @@ GatheredInfo = (
|
||||
| NodeNetworkInterfaces
|
||||
| MacThunderboltIdentifiers
|
||||
| MacThunderboltConnections
|
||||
| ThunderboltBridgeInfo
|
||||
| NodeConfig
|
||||
| MiscData
|
||||
| StaticNodeInformation
|
||||
@@ -125,6 +323,7 @@ class InfoGatherer:
|
||||
system_profiler_interval: float | None = 5 if IS_DARWIN else None
|
||||
memory_poll_rate: float | None = None if IS_DARWIN else 1
|
||||
macmon_interval: float | None = 1 if IS_DARWIN else None
|
||||
thunderbolt_bridge_poll_interval: float | None = 10 if IS_DARWIN else None
|
||||
_tg: TaskGroup = field(init=False, default_factory=create_task_group)
|
||||
|
||||
async def run(self):
|
||||
@@ -133,6 +332,7 @@ class InfoGatherer:
|
||||
if (macmon_path := shutil.which("macmon")) is not None:
|
||||
tg.start_soon(self._monitor_macmon, macmon_path)
|
||||
tg.start_soon(self._monitor_system_profiler_thunderbolt_data)
|
||||
tg.start_soon(self._monitor_thunderbolt_bridge_status)
|
||||
tg.start_soon(self._watch_system_info)
|
||||
tg.start_soon(self._monitor_memory_usage)
|
||||
tg.start_soon(self._monitor_misc)
|
||||
@@ -206,6 +406,17 @@ class InfoGatherer:
|
||||
await self.info_sender.send(NodeNetworkInterfaces(ifaces=nics))
|
||||
await anyio.sleep(self.interface_watcher_interval)
|
||||
|
||||
async def _monitor_thunderbolt_bridge_status(self):
|
||||
if self.thunderbolt_bridge_poll_interval is None:
|
||||
return
|
||||
prev: ThunderboltBridgeInfo | None = None
|
||||
while True:
|
||||
curr = await ThunderboltBridgeInfo.gather()
|
||||
if curr is not None and prev != curr:
|
||||
prev = curr
|
||||
await self.info_sender.send(curr)
|
||||
await anyio.sleep(self.thunderbolt_bridge_poll_interval)
|
||||
|
||||
async def _monitor_macmon(self, macmon_path: str):
|
||||
if self.macmon_interval is None:
|
||||
return
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
import socket
|
||||
import sys
|
||||
from subprocess import CalledProcessError
|
||||
from subprocess import CalledProcessError, run
|
||||
|
||||
import psutil
|
||||
from anyio import run_process
|
||||
|
||||
from exo.shared.types.profiling import NetworkInterfaceInfo
|
||||
from exo.shared.types.profiling import InterfaceType, NetworkInterfaceInfo
|
||||
|
||||
|
||||
async def get_friendly_name() -> str:
|
||||
@@ -28,6 +28,38 @@ async def get_friendly_name() -> str:
|
||||
return process.stdout.decode("utf-8", errors="replace").strip() or hostname
|
||||
|
||||
|
||||
def _get_interface_types_from_networksetup() -> dict[str, InterfaceType]:
|
||||
"""Parse networksetup -listallhardwareports to get interface types."""
|
||||
if sys.platform != "darwin":
|
||||
return {}
|
||||
try:
|
||||
result = run(
|
||||
["networksetup", "-listallhardwareports"], capture_output=True, text=True
|
||||
)
|
||||
except Exception:
|
||||
return {}
|
||||
|
||||
types: dict[str, InterfaceType] = {}
|
||||
current_type: InterfaceType = "unknown"
|
||||
|
||||
for line in result.stdout.splitlines():
|
||||
if line.startswith("Hardware Port:"):
|
||||
port_name = line.split(":", 1)[1].strip()
|
||||
if "Wi-Fi" in port_name:
|
||||
current_type = "wifi"
|
||||
elif "Ethernet" in port_name or "LAN" in port_name:
|
||||
current_type = "ethernet"
|
||||
elif port_name.startswith("Thunderbolt"):
|
||||
current_type = "thunderbolt"
|
||||
else:
|
||||
current_type = "unknown"
|
||||
elif line.startswith("Device:"):
|
||||
device = line.split(":", 1)[1].strip()
|
||||
types[device] = current_type
|
||||
|
||||
return types
|
||||
|
||||
|
||||
def get_network_interfaces() -> list[NetworkInterfaceInfo]:
|
||||
"""
|
||||
Retrieves detailed network interface information on macOS.
|
||||
@@ -36,13 +68,18 @@ def get_network_interfaces() -> list[NetworkInterfaceInfo]:
|
||||
Returns a list of NetworkInterfaceInfo objects.
|
||||
"""
|
||||
interfaces_info: list[NetworkInterfaceInfo] = []
|
||||
interface_types = _get_interface_types_from_networksetup()
|
||||
|
||||
for iface, services in psutil.net_if_addrs().items():
|
||||
for service in services:
|
||||
match service.family:
|
||||
case socket.AF_INET | socket.AF_INET6:
|
||||
interfaces_info.append(
|
||||
NetworkInterfaceInfo(name=iface, ip_address=service.address)
|
||||
NetworkInterfaceInfo(
|
||||
name=iface,
|
||||
ip_address=service.address,
|
||||
interface_type=interface_types.get(iface, "unknown"),
|
||||
)
|
||||
)
|
||||
case _:
|
||||
pass
|
||||
|
||||
@@ -21,7 +21,7 @@ def exo_shard_downloader(max_parallel_downloads: int = 8) -> ShardDownloader:
|
||||
|
||||
|
||||
async def build_base_shard(model_id: ModelId) -> ShardMetadata:
|
||||
model_card = await ModelCard.load(model_id)
|
||||
model_card = await ModelCard.from_hf(model_id)
|
||||
return PipelineShardMetadata(
|
||||
model_card=model_card,
|
||||
device_rank=0,
|
||||
|
||||
Reference in New Issue
Block a user