Files
Aperant/apps/backend/services
Andy 4091d1d4b5 fix: prevent OOM, orphaned agents, and unbounded growth during overnight builds (#1813)
* fix(stability): prevent OOM, orphaned agents, and unbounded growth during overnight builds

Address multiple crash/stability issues observed during long-running autonomous builds:

Backend:
- Skip stuck subtasks in get_next_subtask() using attempt_history.json
- Add retry with exponential backoff + jitter for LadybugDB lock contention
- Time-window filter attempt counts (2h window) to prevent unbounded accumulation
- Trim attempt history per subtask (cap at 50) to bound file size
- Use timezone-aware UTC datetimes throughout recovery manager

Frontend:
- Kill all agents on window close to prevent orphaned processes
- Circuit breaker: kill agents after 10 consecutive renderer disposal errors
- Cap batch queue logs at 100 entries (OOM prevention in IPC batching)
- Cap task log entries at 5000 per task (OOM prevention in store)

Tests:
- Add lock retry logic tests (lock detection, backoff, retry exhaustion)
- Add stuck subtask skipping tests (skip, corrupt JSON, all-stuck)
- Add time-window filtering and attempt trimming tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(stability): address PR review findings for crash stability

- Add .catch() to async killAll() calls to prevent unhandled promise rejections
  (index.ts on window close, utils.ts circuit breaker)
- Reset circuitBreakerTriggered on successful send so it can re-trigger after
  renderer recovery followed by a second crash
- Fix agentManagerRef type to reflect async killAll() signature
- Switch %-format logging to f-strings for consistency with codebase convention

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 11:23:17 +01:00
..
2026-01-02 11:56:36 +01:00