diff --git a/CLAUDE.md b/CLAUDE.md index 61c2fbe..b23f4db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -33,18 +33,23 @@ Frontend, desktop, and tooling commands live in their nested CLAUDE.md. ``` YAML scenario → compile_runtime3.py → Runtime 3 IR → ESP32 / Web player ↓ - Voice bridge → Piper TTS (Tower:8001) + Voice bridge :8200 (MacStudio) + ↳ /tts → F5-TTS-MLX (cache → F5 → safe wav) + ↳ /voice/intent → LiteLLM npc-fast (MLX 7B Q4) + ↳ /voice/transcribe → whisper.cpp :8300 ↓ - Hints engine → /hints/ask + Hints engine → /hints/ask (LLM rewrite via hints-deep MLX 32B Q4) ``` Key surfaces: - **Scenario IR**: `game/scenarios/zacus_v2.yaml` → `tools/scenario/compile_runtime3.py` → portable Runtime 3 IR. Contract: `specs/ZACUS_RUNTIME_3_SPEC.md`. - **Authoring**: `frontend-v3/` (pnpm monorepo: `apps/atelier/` Scratch-like studio + `apps/dashboard/` game-master live view). - **Firmware**: - - `ESP32_ZACUS/` submodule — Freenove ESP32-S3 master (NPC engine, voice pipeline, vision/QR, media manager). + - `ESP32_ZACUS/` submodule — Freenove ESP32-S3 master (NPC engine, voice pipeline, vision/QR, media manager). Branche `feat/idf-migration` en cours (port Arduino → ESP-IDF 5.4). - `PLIP_FIRMWARE/` — retro telephone annex (ES8388 dev kit bringup, Si3210 PCB target). REST endpoint consumed by the master ESP32. -- **Voice / NPC**: Piper TTS on Tower:8001 (zacus voice = tom-medium). NPC phrases in `game/scenarios/npc_phrases.yaml`. MP3 pool generator: `tools/tts/generate_npc_pool.py`. +- **Voice / NPC**: F5-TTS-MLX in-process sur MacStudio (`studio:8200`), voix Zacus clonée via `~/zacus_reference.wav`. Pré-rendu via `tools/tts/generate_npc_pool.py --backend f5`. Service down WAV hardcoded en graceful degradation. NPC phrases : `game/scenarios/npc_phrases.yaml`. +- **Hints engine**: `tools/hints/server.py` FastAPI (LLM rewrite via LiteLLM `hints-deep`, anti-cheat, adaptive level, safety filter `game/scenarios/hints_safety.yaml`). +- **Inference stack**: voir `tools/macstudio/README.md` — MLX-LM (npc + hints), whisper.cpp, F5-TTS, LiteLLM proxy `:4000`. Configs versionnées dans `tools/macstudio/`. - **MCP hardware**: `tools/dev/mcp_hardware_server.py` (stdio, 6 tools). - **Desktop hub**: `desktop/` (Electron, macOS, bundles V3 frontends, talks USB serial). @@ -76,8 +81,9 @@ Key surfaces: ## Infrastructure -- **Tower** (`clems@192.168.0.120`): Piper TTS FR on `:8001` (voices: tom-medium / siwis / upmc). -- **KXKM-AI** (`kxkm@kxkm-ai` via Tailscale): RTX 4090, GPU inference. +- **MacStudio** (`ssh studio`, M3 Ultra 512 GB) : **stack inférence Zacus complète** — MLX-LM (`npc-fast` 7B Q4 :8501, `hints-deep` 32B Q4 :8500), whisper.cpp `large-v3-turbo-q5_0` :8300, F5-TTS-MLX in-process via voice-bridge :8200 (cache disque + watchdog `*/2` + `service_down.wav` graceful), LiteLLM proxy :4000. Voir `tools/macstudio/README.md`. +- **Tower** (`clems@192.168.0.120`) : Suite Numérique, n8n, LiteLLM Tower (legacy). **Plus dans la chaîne live Zacus** ; container Docker `zacus-tts-piper` survit pour batch historique uniquement (voix EN-only, FR à déployer si besoin). +- **KXKM-AI** (`kxkm@kxkm-ai` via Tailscale) : RTX 4090, GPU inference (fine-tuning, plus utilisé en runtime). - SSH is key-based only — never `sshpass`. ## Language diff --git a/tools/macstudio/README.md b/tools/macstudio/README.md index 72bc47b..eb294c5 100644 --- a/tools/macstudio/README.md +++ b/tools/macstudio/README.md @@ -1,30 +1,81 @@ # MacStudio inference stack — config sources of truth -Deployment configs for the MacStudio M3 Ultra (`ssh studio`) hosting Ollama, +Deployment configs for the MacStudio M3 Ultra (`ssh studio`, +`clems@MacStudio-de-MonsieurB.local`, 512 GB) hosting MLX-LM, Ollama, whisper.cpp, F5-TTS, and the LiteLLM proxy. Spec: `docs/superpowers/specs/2026-05-03-tts-stt-llm-macstudio-design.md`. +## Stack live (post-P1 part1-9) + +Inference and voice stack runs entirely on studio. Tower no longer participates +in the live Zacus runtime path. + +| Service | Port | Backend | LiteLLM alias(es) | Monitoring | +|---------|------|---------|-------------------|------------| +| MLX-LM 7B Q4 | `:8501` | `mlx_lm.server` | `npc-fast` (~12 tok/s warm) | `curl -s http://studio:8501/v1/models` | +| MLX-LM 32B Q4 | `:8500` | `mlx_lm.server` | `hints-deep` (~4.4 tok/s warm) | `curl -s http://studio:8500/v1/models` | +| Ollama | `:11434` | `ollama serve` | `npc-ollama-7b`, `hints-ollama-72b` (legacy A/B) | `curl -s http://studio:11434/api/tags` | +| whisper.cpp | `:8300` | `whisper-server` (`large-v3-turbo-q5_0`) | `whisper-fr`, `stt-fr` | `pgrep -fl whisper-server` | +| F5-TTS-MLX | `:8200` (in-process) | voice-bridge `/tts` + `/voice/{transcribe,intent,ws}` | `tts-zacus-primary` | `curl -s http://studio:8200/healthz` | +| LiteLLM proxy | `:4000` | `litellm` | (master_key `sk-zacus-local-dev-do-not-share`) | `curl -s http://studio:4000/health/readiness` | + ## Files | Local path | Remote path on studio | Purpose | |------------|-----------------------|---------| -| `litellm-config.yaml` | `~/litellm-config/config.yaml` | LiteLLM proxy (`:4000`) — 6 model aliases (`npc-fast`, `hints-deep`, `whisper-fr`, `stt-fr`, `tts-zacus-primary`, `tts-zacus-fallback`) | -| `voice-bridge/main.py` | `~/voice-bridge/main.py` | FastAPI voice-bridge scaffold (`:8200`) — routes `/voice/transcribe` (whisper :8300), `/voice/intent` (LiteLLM `npc-fast`), `/tts` (F5 primary, Piper Tower:8001 fallback) | -| `voice-bridge/requirements.txt` | `~/voice-bridge/requirements.txt` | FastAPI deps (fastapi, httpx, uvicorn) | +| `litellm-config.yaml` | `~/litellm-config/config.yaml` | LiteLLM proxy (`:4000`) — 6+ model aliases (`npc-fast`, `hints-deep`, `whisper-fr`, `stt-fr`, `tts-zacus-primary`, `npc-ollama-7b`, `hints-ollama-72b`) | +| `voice-bridge/main.py` | `~/voice-bridge/main.py` | FastAPI voice-bridge (`:8200`) — routes `/voice/transcribe` (whisper :8300), `/voice/intent` (LiteLLM `npc-fast`), `/tts` (F5-TTS-MLX in-process), `/voice/ws` | +| `voice-bridge/requirements.txt` | `~/voice-bridge/requirements.txt` | FastAPI deps (fastapi, httpx, uvicorn, f5-tts-mlx) | +| `voice-bridge/watchdog.sh` | `~/voice-bridge/watchdog.sh` | Watchdog: relance voice-bridge si `pgrep` absent | -## Persistence (headless macOS — no launchd) +## Aliases LiteLLM + +Model routing exposed via the proxy on `:4000`. Clients hit aliases (never raw +backends) so we can swap MLX ↔ Ollama without code changes. + +| Alias | Backend | Use case | +|-------|---------|----------| +| `npc-fast` | MLX-LM 7B Q4 (`:8501`) | NPC dialogue, low-latency intent | +| `hints-deep` | MLX-LM 32B Q4 (`:8500`) | Hint generation, longer reasoning | +| `npc-ollama-7b` | Ollama 7B (`:11434`) | Legacy A/B comparator for `npc-fast` | +| `hints-ollama-72b` | Ollama 72B (`:11434`) | Legacy A/B comparator for `hints-deep` | +| `whisper-fr` / `stt-fr` | whisper.cpp (`:8300`) | French ASR | +| `tts-zacus-primary` | F5-TTS-MLX via voice-bridge (`:8200`) | Zacus voice synthesis | + +## Persistence (headless macOS 26 — no launchd) Studio runs headless; `launchctl` over SSH is broken. All daemons run via -`nohup` for current uptime + `@reboot` crontab for reboot persistence. +`nohup` for current uptime + `@reboot` crontab for reboot persistence. Plus a +`*/2` watchdog entry that relaunches voice-bridge on crash. ```cron +@reboot /opt/homebrew/bin/node_exporter >> /Users/clems/node_exporter.log 2>&1 @reboot /Users/clems/.local/bin/ollama serve >> /Users/clems/ollama.log 2>&1 @reboot /Users/clems/whisper-install/whisper.cpp/build/bin/whisper-server -m /Users/clems/whisper-install/whisper.cpp/models/ggml-large-v3-turbo-q5_0.bin --port 8300 -t 8 --host 127.0.0.1 -l fr >> /Users/clems/whisper-server.log 2>&1 @reboot /Users/clems/.local/bin/litellm --config /Users/clems/litellm-config/config.yaml --port 4000 --host 0.0.0.0 >> /Users/clems/litellm.log 2>&1 +@reboot /Users/clems/.local/bin/mlx_lm.server --model --port 8500 >> /Users/clems/mlx-32b.log 2>&1 +@reboot /Users/clems/.local/bin/mlx_lm.server --model --port 8501 >> /Users/clems/mlx-7b.log 2>&1 +@reboot /Users/clems/voice-bridge/.venv/bin/uvicorn main:app --host 0.0.0.0 --port 8200 >> /Users/clems/voice-bridge.log 2>&1 +*/2 * * * * /Users/clems/voice-bridge/watchdog.sh >> /Users/clems/voice-bridge.watchdog.log 2>&1 ``` Edit on studio with `crontab -e`. +## Resilience + +- **Watchdog** — `*/2 * * * * /Users/clems/voice-bridge/watchdog.sh` relance + voice-bridge si `pgrep -f "uvicorn main:app"` est vide (auto-recovery sous 2 + minutes après un crash). +- **Cache TTS disque** — `~/voice-bridge/cache/` indexé par + `sha256(text + voice_ref + steps)`. Hit cache = retour WAV immédiat (pas de + ré-synthèse). Purge : + ```bash + curl -X DELETE http://studio:8200/tts/cache + ``` +- **Graceful degradation** — `service_down.wav` (hardcoded fallback) servi par + voice-bridge si F5-TTS lève une exception irrécupérable. Le runtime ESP32 + reçoit toujours un WAV valide, jamais un 5xx silencieux. + ## Sync local → remote After editing here, push to studio: @@ -41,21 +92,40 @@ ssh studio 'pkill -HUP -f "litellm --config" || pgrep -fl litellm' # reload Li ```bash ssh studio 'curl -s http://localhost:11434/api/tags | jq .models[].name' # ollama models +ssh studio 'curl -s http://localhost:8500/v1/models' # mlx 32B (hints-deep) +ssh studio 'curl -s http://localhost:8501/v1/models' # mlx 7B (npc-fast) ssh studio 'curl -s http://localhost:4000/health/readiness' # litellm +ssh studio 'curl -s http://localhost:8200/healthz' # voice-bridge ssh studio 'curl -s -F "file=@/Users/clems/whisper-install/whisper.cpp/samples/jfk.wav" http://localhost:8300/inference' # whisper -ssh studio 'pgrep -fl ollama; pgrep -fl whisper-server; pgrep -fl litellm' # daemons +ssh studio 'pgrep -fl ollama; pgrep -fl whisper-server; pgrep -fl litellm; pgrep -fl mlx_lm.server; pgrep -fl "uvicorn main:app"' # daemons ``` -## Known limitations (2026-05-03) +## Pré-render NPC pool (F5-TTS) -- `qwen2.5:72b-instruct-q8_0` via Ollama gives ~2 tok/s on M3 Ultra. Switch - to MLX-LM backend or Q4_K_M quant if hints latency unacceptable (cible - spec: <8 s for 50 tokens, currently ~25 s). -- F5-TTS uses base multilingual checkpoint (no FR-specific). Quality - acceptable but no Zacus voice clone yet — needs `zacus_reference.wav` to - be recorded (P2). -- Voice-bridge scaffold is **not running** as daemon yet. Activate when - F5-TTS in-process integration is wired. +Generate the full NPC phrase pool against the live voice-bridge (idempotent — +skips already-cached entries). Output: `hotline_tts/f5/.wav` plus +`hotline_tts/f5/manifest.json`. + +```bash +python3 tools/tts/generate_npc_pool.py \ + --backend f5 \ + --voice-bridge-url http://studio:8200 +``` + +Cache key derivation must mirror `voice-bridge/main.py::_voice_ref_token`. See +`tools/CLAUDE.md` for backend semantics. + +## Known issues / TODO + +- **`steps` non borné** côté payload `/tts` — un client malicieux peut envoyer + `steps=999` et saturer la GPU queue. Ajouter `steps: int = Field(ge=1, le=64)` + dans le schéma Pydantic de voice-bridge. +- **Default `--voice-bridge-url 192.168.0.150`** dans certains scripts — IP LAN + studio à confirmer selon le réseau du client. En itinérance préférer le + Tailscale `100.116.92.12` (ou alias DNS `studio`). +- **`zacus_reference.wav`** à enregistrer (action humaine, prereq P2). Sans ce + fichier, F5-TTS-MLX tombe sur le checkpoint multilingue de base — qualité + acceptable mais pas de Zacus voice clone. ## Anti-patterns