fix: shield macmon cleanup from cancellation to prevent orphaned process (#1714)

## Summary

Fixes a bug where macmon metrics (GPU usage, temperature, power, RAM)
freeze permanently in the dashboard while non-macmon metrics (disk
usage) continue updating normally.

**Root cause: asyncio subprocess pipe transport flow control stall**

Observed on a 2-node Mac Studio M3 Ultra cluster (Python 3.13.12, anyio
4.11.0, macOS with kqueue) running EXO.app for ~36 hours. Diagnostics
confirmed:

1. **Only macmon monitoring is stuck** — disk metrics from the same
InfoGatherer continue updating, proving the Worker, EventRouter, and API
pipelines are healthy
2. **macmon IS producing data** — its stdout pipe is full at exactly
65536 bytes (64KB OS buffer), and macmon is blocked on write at 0% CPU
3. **The pipe read-end FD is still open** — the exo process holds it,
but asyncio isn't reading from it
4. **The stale GPU value (0.21) is wrong** — should be ~1.0 (matching
the other node under identical load)

This is consistent with asyncio's subprocess pipe transport getting
stuck in flow control: `pause_reading()` is called when the internal
buffer exceeds the high-water mark, but `resume_reading()` is never
called, permanently deregistering the FD from kqueue. The `receive()`
coroutine waits forever for data asyncio will never deliver.

```
$ lsof -p <macmon_pid>
macmon  74691  FD=1  PIPE  SIZE=65536  ->0x5ae78ecf376ccd0e  # full pipe

$ lsof -p <exo_pid> | grep <pipe_id>
exo     74689  FD=47  PIPE  SIZE=65536  ->0xd129da474c1340f7  # read end held open, not consumed
```

Note: anyio 4.11's `Process.aclose()` already uses
`CancelScope(shield=True)` for cleanup during cancellation — this is NOT
an election cleanup issue (confirmed by @ciaranbor's testing).

**Fix:** Replace the `async for` iteration with an explicit `receive()`
inside `fail_after()`. If macmon produces no output for 10× its
configured interval (minimum 30s), `TimeoutError` fires, `open_process`
cleanup kills macmon and tears down the stale transport, and the loop
restarts with a fresh subprocess and fresh asyncio transport.

## Test plan

- [x] All 246 existing tests pass
- [ ] Verify macmon restarts after simulated pipe stall (e.g., `SIGSTOP`
macmon, wait for timeout, confirm restart and metrics resume)
- [ ] Long-running soak test on multi-node cluster to confirm the fix
prevents recurrence

src/exo/utils/info_gatherer/info_gatherer.py

@@ -529,6 +529,9 @@ class InfoGatherer:
         if self.macmon_interval is None:
             return
         # macmon pipe --interval [interval in ms]
+        # Timeout: if macmon produces no output for this many seconds, restart it.
+        # macmon writes every macmon_interval seconds, so 10x that is generous.
+        read_timeout = max(self.macmon_interval * 10, 30)
         while True:
             try:
                 async with await open_process(
@@ -542,10 +545,15 @@ class InfoGatherer:
                     if not p.stdout:
                         logger.critical("MacMon closed stdout")
                         return
-                    async for text in TextReceiveStream(
-                        BufferedByteReceiveStream(p.stdout)
-                    ):
+                    stream = TextReceiveStream(BufferedByteReceiveStream(p.stdout))
+                    while True:
+                        with fail_after(read_timeout):
+                            text = await stream.receive()
                         await self.info_sender.send(MacmonMetrics.from_raw_json(text))
+            except TimeoutError:
+                logger.warning(
+                    f"MacMon produced no output for {read_timeout}s, restarting"
+                )
             except CalledProcessError as e:
                 stderr_msg = "no stderr"
                 stderr_output = cast(bytes | str | None, e.stderr)