le-mystere-professeur-zacus/docs/voice-pipeline-roadmap.md

Voice pipeline roadmap — 2026-05-24

Snapshot of the voice stack after the macM1 hints + Kyutai STT + WS fallback session. Lives here so the next session (this repo or others) can pick up the right context without re-discovering every gotcha.

Current state

ESP32 firmware (ESP-IDF feat/idf-migration)
   │
   │ WebSocket /voice/ws  (PCM16 16 kHz, json control frames)
   ▼
voice-bridge :8200  (FastAPI, F5-TTS-MLX in-process, MacStudio)
   │   STT_BACKEND = kyutai (default since commit 43b5ddc)
   │
   ├── /voice/ws STT     → kyutai_stt.py → ws://localhost:8304
   │                       (msgpack, /api/asr-streaming, partials forwarded)
   │                       ↳ fallback whisper.cpp on KyutaiSttError
   │
   ├── /voice/ws intent  → LiteLLM :4000 → npc-fast (MLX Qwen2.5-7B Q4 :8501)
   │
   ├── /voice/ws TTS     → cache → F5 → Piper Tower :8001 (commit a5b00aa)
   │                       (Piper EN-only today, plugs the silence)
   │
   └── /usage/stats      → polled by dashboard useVoiceUsage (5 s)

hints engine :8311  (FastAPI, macM1)
   └── /hints/ask       → LiteLLM Studio :4000 → hints-deep (MLX 32B Q4 :8500)

Studio crontab @reboot: whisper.cpp :8300, MLX :8500/:8501, moshi-server :8304
macM1 LaunchAgents:    cc.zacus.hints (:8311), cc.ailiance.whisperx (:9500)

Runbooks :

• tools/macstudio/MACM1_HINTS_DEPLOY.md
• tools/macstudio/MOSHI_STT_DEPLOY.md

Open chantiers

Ordered by ROI / effort, not by priority — pick what matters next.

⭐ Glossary biasing for Kyutai STT

Goal : fix U-SON → eu son nez, Cherchez → chercher, all the metier vocabulary holes the generic 1B model can't know about.

Constraint discovered this session : moshi-server 0.6.4 Rust does not expose any TextPrompt / InitialPrompt message via the WS (rust/moshi-server/src/asr.rs:17 enum InMsg = Init|Marker|Audio|OggOpus only). The PyTorch prompt mechanism uses on_text_logits_hook which is in-process only.

Three viable alternatives :

Approach	Latency	Maintenance	Notes
Post-correction LLM (npc-fast "rewrite this transcript knowing ...")	+500 ms	nil	use the same model already in the chain
Audio prompt pre-pended (cf. stt_from_file_with_prompt_pytorch.py)	+2-3 s	regen TTS cache	dose pre-roll = glossary spoken aloud, skip N first Word events
Fuzzy alias dictionary (firmware voice_dispatcher already has one)	0	manual	extend per-puzzle; smallest scope, highest ROI

Recommended : start with the fuzzy alias dictionary extension. The firmware already runs this on the intent path (see core mem 20793 + voice_dispatcher fuzzy alias matching for Whisper transcription errors). Adding per-puzzle metier terms there is ~30 lines + a small YAML. Reserve post-correction LLM for the day fuzzy aliases plateau.

#3 Server-side end-of-utterance

Status : skipped 2026-05-24. The firmware already sends {"type":"end"} via its own VAD AFE (voice_pipeline_ws.c:332). Adding Silero on the server = double VAD + ~500 MB torch dep, almost no gain.

If reopened : the cleanest path is to consume the Step events already emitted by moshi-server ({type:"Step", prs:[f32], buffered_pcm}) — Kyutai has a built-in semantic VAD. prs semantics are undocumented (it's a projection p[0] of the Mimi LM logits), needs empirical reverse-engineering. Budget: 1-2 h of test sessions with a printed-Step client, then a tiny auto-marker on N consecutive low-pr frames rule in kyutai_stt.py.

#4 Kokoro / TTS fallback

Status : Piper fallback wired into /voice/ws (commit a5b00aa). Kokoro evaluated and rejected for Zacus :

• 1 FR voice only (af + a handful), maintainer flags G2P FR as weak
• Mediocre quality on metier terms
• Piper already in place as fallback infrastructure

Real follow-up : deploy a Piper FR model on Tower (or on Studio next to whisper.cpp). The Tower instance is EN-only today, so the fallback currently produces an English pronunciation of French input — better than silence, but worth fixing. ~30 min :

# On Tower (or wherever you want the FR fallback):
mkdir -p /opt/piper-fr/models
cd /opt/piper-fr/models
curl -L -o fr_FR-siwis-medium.onnx \
  https://huggingface.co/rhasspy/piper-voices/resolve/main/fr/fr_FR/siwis/medium/fr_FR-siwis-medium.onnx
curl -L -o fr_FR-siwis-medium.onnx.json \
  https://huggingface.co/rhasspy/piper-voices/resolve/main/fr/fr_FR/siwis/medium/fr_FR-siwis-medium.onnx.json
# Wire a separate piper-server (OpenAI-compatible) on :8002 then
# set PIPER_URL=http://192.168.0.120:8002 in voice-bridge env.

Plus loin : ailiance OpenAI-Realtime wrap

Goal : expose Kyutai (and later Moshi end-to-end) behind an OpenAI-Realtime-compatible API endpoint inside the ailiance gateway. This is a product differentiator — ailiance becomes "sovereign EU Realtime API" vs OpenAI's hosted offering.

Scope :

• ailiance gateway already runs on electron-server:9300 (FastAPI, source in ailiance/ailiance private repo, see /home/electron/ailiance/)
• Add a /v1/realtime WebSocket endpoint that speaks OpenAI's Realtime protocol (session events, input_audio_buffer, response.create, etc.)
• Internally adapt:
  • incoming input_audio_buffer.append → moshi-server :8304 Audio frames
  • moshi-server Word/EndWord → emit OpenAI conversation.item.input_audio_transcription.completed
  • response generation: call LiteLLM npc-fast (or Helium when Moshi full-duplex lands), stream tokens, generate TTS via F5 or Kokoro/Piper
• ~400-600 lines Python, ~3 days

Pre-requisites :

• moshi-server STT live on Studio (done)
• A spec mapping OpenAI Realtime events ↔ moshi-server msgpack events (write this first, it's the hard part)

Next session entry point : cd ailiance/ailiance && code . ; the adapter goes in ailiance/realtime/ as a sub-router of the gateway.

Plus loin : baby-brain migration whisperx → Kyutai

Goal : replace whisperx-server :9500 on macM1 with Kyutai STT for the conversational path. Keep whisperx for diarisation/speaker ID since Kyutai doesn't do that.

Scope :

• Two STT endpoints in baby-brain: one streaming (Kyutai) for the live avatar/voice agent, one batched-with-diarisation (whisperx) for the WML clustering path that needs speaker labels
• Touch points : baby_brain/identity/stt_bridge.py (the WS client), baby_brain/identity/visual_bridge.py (avatar mood from voice)
• Decision needed : run a 2nd moshi-server on macM1 (local low latency, ~600 MB extra RAM in an already-tight env), or point at the Studio :8304 instance (1 RTT extra)
• ~1 day

Pre-requisites :

• baby-brain repo cleanup (last session was at PR/refactor stage)
• benchmark whisperx vs Kyutai on baby-brain's actual conversational corpus (which is closer to human voice than the Zacus F5 WAVs)

Next session entry point : cd ~/code/baby-brain && code . ; talk to the stt_bridge.py author about whether they prefer the WSc proxy approach or a direct in-process Python client (the Python moshi-mlx package would work on macM1 since baby-brain already runs Python).

Cross-cutting things learned this session

• MLX workers on Studio bind to 127.0.0.1 (see lsof proof in MACM1_HINTS_DEPLOY.md). Anything Tailnet-reachable on Studio has to be explicitly 0.0.0.0 in its launch args. Today the only exceptions are the gateway-routed services (:9301, :9303, :9327 etc.) and the new :8304 Kyutai.
• Non-interactive SSH on Studio doesn't see Homebrew (cmake, etc.). Always prepend PATH=/opt/homebrew/bin:$PATH in cargo/make invocations shipped over ssh ....
• pyo3 0.23.5 + Python 3.14+ needs PYO3_USE_ABI3_FORWARD_COMPATIBILITY=1 at build time. Bit anyone building Rust packages with Python bindings.
• Voice-bridge venv has no pip (uv-style minimal venv). Always use uv pip install --python /Users/clems/voice-bridge/.venv/bin/python … to add deps.
• WS /voice/ws is already implemented end-to-end (server + ESP-IDF client) — don't reinvent the protocol, just add backends to the existing branches.