Custom ESP32-WROOM-32E + Si3210 SLIC board replacing the ESP32-A1S
dev kit (final PLIP target, RJ9 handset). Generated via tools/
gen_si3210_lib.py (symbol + QFN-38 footprint), gen_schematic.py
(parses standard KiCad libs, 45 instances, 157 global labels on pin
ends) and gen_pcb.py (58x38mm outline, 41 footprints placed per the
plan, nets assigned, unrouted).
ERC: 0 violations. Schematic follows the spec GPIO map and the plan's
passive network. PCB placed not routed (Freerouting + DRC + gerbers
remain — see README). BOM in JLCPCB format; CP2102N/transformer/RJ9
LCSC parts marked TBD.
WARNINGS (README): the Si3210 -72V DC-DC stage is simplified and must
be checked against the Skyworks datasheet/AN35 before fab; line
protection is minimal; human review required before ordering.
- ConsolePane shows the ScenePlayer narration (simLog, green) under
the validation entries, plus a current-scene badge in the header.
- voice-bridge accepts rate + emotion on /tts: rate (clamped 0.5..2.0)
drives the F5 sampler speed; emotion is accepted and cache-keyed but
not rendered yet (no F5 emotion control). Legacy cache keys are
preserved when both fields are absent/default — the NPC pool
generator contract is untouched.
- Bump ESP32_ZACUS: GET/POST /game/peers (PR ESP32_ZACUS#4) — PLIP
provisioning by curl, live registration without reboot.
Verified: py_compile OK, tsc clean, vite build green, master firmware
build green.
The Run button fed blocks_studio v2 YAML to the legacy gameplay engine,
which only parses old zacus_v3 scenarios. loadScenario now tries the v2
document first and plays it through a new ScenePlayer: « Zacus dit »
drives the NPC bubble + Web Speech (rate applied, emotion mapped to the
stage mood), timers wait, goto/branch/intent jump between scenes,
hardware actions are narrated into a capped simLog. Legacy YAML still
falls back to ZacusScenarioEngine. Visualization fidelity only — the
authoritative compiler remains blocks_to_runtime3.py.
Verified: tsc clean, vite build green.
The Atelier exposed a legacy 12-block gameplay-config language whose
export the gateway could not compile (no blocks_studio_version) — and
no start/end blocks. It now speaks the same language as the zacus-hub
native Studio: the 38-block scene catalog (sceneStart/sceneEnd, npcSay
voice lines, HW/ESP-NOW/M5/PLIP, logic) lives once in
@zacus/shared/blockly; the hub copy is GENERATED via
tools/dev/sync_blockly_catalog.sh.
- Export/import rewritten to blocks_studio_version: 2 (flat nodes,
next pointers, logicIf slots) via shared blocks-doc.ts — what
POST /v1/flash compiles to Runtime 3 IR, and what the hub Studio
reads/writes (full interop).
- « Zacus dit » gains voice (voice-bridge voice_ref), emotion and rate
fields; the compiler forwards them as optional tts_say fields
(omitted at defaults, backward compatible).
- Editor validation now mirrors compiler rules: missing sceneStart,
duplicate scene ids, unknown goto targets, empty voice lines.
Verified: tsc clean, vite build green, v2->IR round-trip OK
(entry STEP_intro, tts_say{voice_ref,emotion,rate}).
DeployPanel + useDeploy hook: saves the current Blockly workspace YAML
to the gateway (PUT /v1/studio/scenario/atelier_live.yaml) then flashes
the selected board (POST /v1/flash/{board}). Boards without an IP that
the master relays over ESP-NOW (plip, box3, p7_coffre) are flagged in
the selector and routed through the master automatically; the panel
shows the FlashResult steps and relay fan-out. Gateway gains a
CORSMiddleware (ZACUS_HUB_CORS_ORIGINS, default *) so the browser
client can reach it — auth stays on the bearer token
(VITE_GATEWAY_TOKEN / VITE_GATEWAY_BASE_URL).
Verified: tsc --noEmit clean, vite build green, py_compile OK.
PLIP was out of scope by design (Wi-Fi/HTTP-only); brought back in by
request. New src/scenario_now.{h,cpp}: owns the single esp_now recv
callback (no MSG_* demux needed — PLIP has no legacy ESP-NOW traffic),
reassembles scenario_mesh frames ({seq:u16 LE, total:u16 LE} + <=236 B
payload, <=64 KiB, 5 s sender-silence timeout) in a worker task, then
persists to LittleFS /scenario.json (temp-then-rename). Optional
consumer hook for a future Runtime 3 engine; stored-and-logged without
one. network_task calls scenario_now_init() once the station is up
(idempotent, repeated on reconnect).
Build green: pio run (devkit_es8388), RAM 14.8%, Flash 63.1%.
Gateway (tools/zacus-gateway):
- /v1/state now aggregates live per-group progress from the hints engine
/hints/sessions (the only backend exposing game state), with graceful
fallback when the engine is offline or admin-protected. Flows through both
the REST endpoint and the WS push (shared build_state). Documents
HINTS_ADMIN_KEY.
App (apps/zacus-hub):
- GameMaster "Groupes en cours" section: active groups, hint/penalty counts,
current puzzle, and a state_detail note. SessionSnapshot DTO + extended
GameState.
Submodule:
- ESP32_ZACUS -> feat/idf-migration b2267f2 (ESP-NOW scenario hot-load
receivers; three firmwares build green under ESP-IDF 5.4.4).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two new specs:
- 2026-05-24-zacus-hub-app.md — architecture, sprints, auth,
open decisions for the SwiftUI hub app + gateway.
- 2026-05-24-firmware-scenario-hotload.md — firmware Phase 2
plan for /game/scenario + ESP-NOW mesh relay (the master
side has shipped on the submodule; the relay + peer
receivers + scenario_engine_reload remain to do).
Submodule ESP32_ZACUS bumped to d220d94 on feat/idf-migration
(POST /game/scenario hot-load on master, scenario_server.c on
BOX-3, ota_server max_uri_handlers 8->16, sdkconfig.qemu).
New multiplatform SwiftUI app under apps/zacus-hub/ generated via
XcodeGen (project.yml). Three modes share a HubSession actor that
talks to tools/zacus-gateway over Tailscale-only HTTP/WS.
Game-master: live state stream (WebSocket /v1/state/ws), backend
health badges, hint trigger, journal.
Companion: push-to-talk via AVAudioRecorder (M4A AAC 16 kHz),
fallback file picker, hint request, transcript via voice-bridge
proxy.
Studio: YAML editor with validate/save (existing /v1/studio
routes), plus a Scratch tab hosting a Blockly + zelos-renderer
workspace in WKWebView. 38 custom Zacus block types across 10
categories (Scènes, NPC, Audio, LCD, Hardware ESP32, ESP-NOW,
BOX-3, M5, PLIP, Logique), .sb3 export/import via JSZip (procedure
calls + embedded YAML for lossless round-trip), Flasher sheet that
calls /v1/flash/{board} with auto/hot/cold strategies and shows
per-board step results including the cold-flash idf.py command.
ATS exception domains cover the Tailscale CGNAT range plus the
electron-server / studio / macm1 hostnames so the WebView and
URLSession can hit plain HTTP on the tailnet without TLS.
Token stays in the Keychain (cc.saillant.zacus.hub) — never
committed to source.
New thin FastAPI aggregator under tools/zacus-gateway/ that sits
between the SwiftUI Hub app and the existing voice/hints/firmware
surfaces. Exposes:
- GET /v1/auth/ping, /v1/backends/health, /v1/state (+ WS)
- POST /v1/gm/hint, /v1/companion/{hint,voice/intent,voice/
transcribe,voice/tts}
- GET/PUT/validate/compile /v1/studio/scenario/*
- GET /v1/flash/boards, POST /v1/flash/{board}
blocks_to_runtime3.py walks a blocks_studio_version: 2 YAML and
emits a Runtime 3 IR with the new condition action for logicIf
slots (body/else), keeping branching inside a single step.
boards.yaml registry declares master / box3 / plip / p7_coffre with
hot_endpoint, cold_data_dir, and espnow_relay_peers; the flash
endpoint compiles the IR then either POSTs to the board's HTTP
surface or stages it for cold-flash via idf.py.
flash_bridge.sh works around the docker bridge subnet collision
on electron-server by chaining gateway compile + scp + POST from a
host already on the WiFi LAN.
systemd-friendly: bearer auth via ZACUS_HUB_TOKEN, request
fan-out via a single httpx AsyncClient. .env / .venv git-ignored.
Extend the Runtime 3 IR spec with a standalone 'condition' action
kind that takes {expr, then[], else[]} sublists, keeping branching
local to the current step (no extra transitions or step duplication).
The validator now accepts both string-form (legacy firmware import)
and dict-form actions, recursively validating condition.then /
condition.else as nested action lists. STANDARD_ACTION_KINDS gains
the full set of blocks_studio kinds (audio, LCD, hardware, ESP-NOW,
BOX-3, M5, PLIP) so the blocks-to-IR codec output passes validation
end-to-end.
Both chantiers live in other repos but the validating infrastructure
(Kyutai :8304, voice-bridge :8200, Kokoro :8002, LiteLLM :4000) is
documented here. Specs are implementable as-is — file layout, API
shapes, integration order, bench targets, risks, concrete entry
commands.
ailiance/realtime/ : OpenAI Realtime API wrap. ~9 events,
5 modules, single process, one upstream Kyutai WS per client.
baby-brain stt migration : router keeps whisperx for diarisation
(WML needs speakers), routes the conversational path to Kyutai.
Includes the ssh one-liner to add a babybrain-tailnet auth key
to Studio Kyutai config when work starts.
Adds Kokoro-82M FR (ff_siwis voice, mlx-audio runtime) as the
second-tier TTS fallback after F5, before Piper, on the /voice/ws
path. Reason: Piper Tower is English-only as of 2026-05 and butchers
French utterances; Kokoro-FR is small (1.5 GB venv), fast on warm
(~1.4 s for 5 words), and FR-native.
New tools/macstudio/kokoro-fr/ houses the self-contained FastAPI
server (POST /synthesize, drop-in field shape with Piper). Persisted
on Studio :8002 via crontab @reboot. Verified roundtrip: Kokoro FR
synth → Kyutai STT → 'USON est sensible, vous savez, ...' (clean).
voice-bridge gains:
- KOKORO_URL env (default http://localhost:8002, set to 0 to disable)
- _kokoro_fallback() helper symmetric to _piper_fallback()
- /voice/ws chain: cache -> F5 -> Kokoro -> Piper -> empty speak_end
RMS-based silence detector on the receive loop closes the
utterance automatically when WS_VAD_SILENCE_TIMEOUT_MS of
contiguous silence follows at least WS_VAD_MIN_SPEECH_MS of
detected speech. Firmware's local VAD AFE still sends
{type:end} normally — this is a backstop for the cases where
the firmware loses the end frame or crashes mid-stream.
Two trigger paths:
- frame-level: silence accumulates across incoming PCM frames
- idle-timeout: no frame arrives at all for the silence
window (firmware froze mid-stream)
Zero deps added — pure numpy RMS. Disable via WS_VAD_ENABLED=0;
tune thresholds via env. Smoke-tested with a client that never
sends {type:end}: server closes via auto-EOU, full pipeline runs.
Server-side fuzzy alias table applied to the final STT transcript
before forwarding to /voice/intent. Closes the metier-vocabulary
gap of generic STT models (U-SON, LA 440 Hz, Zacus, ...) without
needing per-request prompt biasing — which moshi-server's Rust
runtime does not expose anyway.
Loader is soft on pyyaml (degrades to no-op if missing) and on
the yaml file itself (logs + skips on parse error). Patterns are
case-insensitive with auto word-boundaries, with a (?...) escape
hatch for advanced regex.
Per-puzzle entries layer on top of globals — firmware passes
puzzle_id in the hello frame; legacy clients omit it and get
just the global polish. Loaded once at module import; restart
voice-bridge to reload.
Unit tested on 5 representative transcripts (eu son nez -> U-SON,
diapazon -> diapason, le rasus -> Zacus, le la 440 hertz -> LA 440 Hz).
Snapshot of state after this session + actionable plan for the
five remaining chantiers (glossary biasing, server-side EOU,
TTS fallback, ailiance Realtime wrap, baby-brain migration).
Each chantier documents what was discovered (e.g. Kyutai Rust
server has no TextPrompt message, Step VAD prs are undocumented),
why we did/didn't act on it this session, and a precise entry
point for the next session — so the next person/agent doesn't
have to re-grep the moshi-server source to know what's possible.
The /tts HTTP handler already chained cache -> F5 -> Piper Tower.
The /voice/ws path was missing the Piper branch — on F5 timeout
or load failure it sent an empty speak_end and the firmware got
silence.
Wire the same _piper_fallback() helper after the F5 path, mirror
the wav-to-pcm conversion done on the cache hit branch. Best
effort: any error in the fallback leaves pcm=None and the WS
behaves as before. backend tag distinguishes 'f5' / 'cache' /
'piper_fallback' in the ws_tts_* log events.
Note: Piper on Tower is EN-only today (per root CLAUDE.md), so
a FR utterance falls back to EN pronunciation. Deploying a Piper
FR model is the real follow-up; this just plugs the silence.
/voice/ws no longer waits for the full audio buffer + a single
whisper.cpp batch call. STT_BACKEND=kyutai (default) routes
through the new kyutai_stt module: streams PCM16 16 kHz -> 24 kHz
to moshi-server on :8304, emits {type:stt,final:false,text:...}
partials to the firmware as Word events arrive, then a final:true
after the marker echo.
Fallback to the legacy whisper.cpp batch path on KyutaiSttError so
a downed moshi-server keeps the firmware loop alive. Flip to
STT_BACKEND=whispercpp to disable Kyutai during upgrade / A-B.
Verified end-to-end on say-Thomas WAV: 18 partials + 1 final, then
intent + F5 TTS as before. Server logs backend=kyutai per call.
moshi-server 0.6.4 Rust (Candle/Metal) on Studio :8304 serves
kyutai/stt-1b-en_fr-candle through /api/asr-streaming WebSocket
(msgpack). Persisted via crontab @reboot matching the other
Studio service start pattern.
Validated against Zacus F5 WAV (hallucinations, all STT engines
share them — corpus issue) and a clean 'say -v Thomas' FR WAV
(14/17 words verbatim). Wiring into voice-bridge is left for a
dedicated chantier — see file.
Phase A deploy on 2026-05-23: hints engine on macM1:8311 calling
back to Studio LiteLLM:4000 (which routes to local MLX hints-deep).
LiteLLM macM1 dropped from plan: Studio MLX workers bind to
127.0.0.1 only (lsof confirms), so a macM1 proxy can't reach them.
Rebinding the workers to 0.0.0.0 is Phase B.
Module logger is LOG (line 130). The bare 'log' references in the
default-key / default-admin warnings crashed the server at boot
whenever LITELLM_MASTER_KEY was unset (default placeholder path).
NameError surfaced when bringing up the hints engine on macM1.
Problem: pio build failed because HTTPClient::POST expects a non-const uint8_t* and the slice 14 client passed const uint8_t* via reinterpret_cast. Approach: drop the const qualifier — body is a local stack buffer, so the cast is safe (no shared mutable aliasing). Tradeoffs: kept the comment in place to document why the cast is benign so a future reader does not re-introduce const for ergonomics and re-break the build.
Problem: voice-bridge GET /usage/stats (commit 1ac5741) exposes
LLM token + F5 TTS + STT cost counters but the dashboard had no
window into them — GMs were blind to runaway token spend during
playtests.
Approach: dedicated useVoiceUsage hook (separate 5 s polling cadence
from the 2 s /health/ready probe) keeps a bounded ring of the last
60 snapshots in memory for delta math, exposes resetUsage() that
sends X-Admin-Key when configured. VoiceUsagePanel renders LLM
total + tokens/min with a tiny inline SVG sparkline, F5 TTS calls/
seconds/cache hit rate, STT calls/seconds, and a confirm-gated
Reset button. Status pill flips to burst above 1000 tok/min.
Tradeoffs: rolling-window tok/min uses receivedAtMs (local clock,
not server uptime_s deltas) so we tolerate clock skew but lose
precision when the page sleeps. Sparkline is plain SVG to avoid
pulling in a charting lib. Counters reset on bridge restart, same
caveat as the prom textfile dumper (commit ba56395).
Problem: voice-bridge /usage/stats (commit 1ac5741) exposes JSON
counters but had no path into the existing studio:9100 node_exporter
for Grafana dashboarding.
Approach: small bash dumper (usage_to_prom.sh) renders the JSON into
the Prometheus textfile format, atomic mv into
/Users/clems/textfile_collector/. Cron once a minute. Doc covers
metric semantics, PromQL queries (token rate, cache ratio, inverse
RTF), and a 4-row Grafana panel layout.
Tradeoffs: counters reset on bridge restart (in-memory state in
main.py) — panels must use rate()/increase() with reset-tolerant
windows. node_exporter on studio still lacks the
--collector.textfile.directory flag (sudo/user edit needed); README
documents the one-line crontab change required to expose the metrics.
Problem: PlatformIO build failed because lib_deps listed
schreibfaul1/ESP32-audioI2S, which is not published on the registry,
and network_task was an empty stub so the slice 14 hook client could
not actually reach zacus-master.local.
Approach: drop the unresolvable schreibfaul1 dep (esphome's fork
already covers ES8388 I2S). Flesh out network_task to read WiFi
credentials from NVS (namespace "wifi"), fall back to an open
ZACUS-SETUP SSID for bringup, wait up to 30 s for association,
advertise plip.local via ESPmDNS, and probe zacus-master.local once
so the hook client's hostname-based POSTs resolve. A 5 s health loop
reconnects on drop and re-reads NVS in case the desktop pushes new
credentials at runtime.
Tradeoffs: the open SSID fallback is dev-only and must be replaced
with SoftAP provisioning before shipping. mDNS probe is logged but
non-fatal so the hook client can still retry resolution per-request.
Build verification with `pio run` was blocked by sandbox in this
session; the dep change is mechanical and the source compiles
against stock ESP32 Arduino headers (WiFi.h, ESPmDNS.h, Preferences.h).
Problem: the voice-bridge synthesizes audio, runs whisper, and forwards
to LiteLLM npc-fast and (in the future) hints-deep without any
visibility on cumulative token consumption or audio compute time. Soak
runs and cost extrapolation are blind.
Approach: aggregate per-bucket counters in module-level state under an
asyncio Lock. Buckets: npc_fast, hints_deep, tts (f5_calls,
f5_seconds, cache_hits), stt (calls, audio_seconds). Hooks live in
/voice/intent (LiteLLM usage dict), _intent_complete in /voice/ws,
/tts (cache hit vs F5 with audio duration computed from PCM), WS TTS
path, and /voice/transcribe with a best-effort RIFF sniffer for the
multipart body. New endpoints GET /usage/stats (Cache-Control no-cache,
no auth — LAN only) and POST /usage/reset (gated by VOICE_BRIDGE_ADMIN_KEY
when set). Background asyncio task logs a usage_tick JSON snapshot every
USAGE_TICK_INTERVAL_S (default 300 s) for grep-friendly history.
Tradeoffs: chose dict-under-lock over a real metrics library
(prometheus_client) to keep the daemon dependency-light and because
the readout is meant to feed a Prometheus textfile collector cron from
node_exporter (recommended in the README), not a direct /metrics scrape.
audio_seconds may stay 0 for transcribe calls when the WAV header is
unrecognisable — best-effort, calls counter still increments. Reset
endpoint follows the same admin-key-optional pattern as DELETE /tts/cache.
MacStudio spec: P1 part13 marked delivered (commit d59282a, voice-bridge 0.5 with CORSMiddleware + NPC_SYSTEM_PROMPT injection for npc-fast persona). Risks section gains a new entry pointing to the LAN-only security checklist in tools/macstudio/README.md (commit 695b7f7). Acceptance gates: 4 new boxes checked.
Voice-pipeline ESP-SR spec: C_slice 12 marked delivered (commit 71bc9a5, mDNS zacus-master.local + game_endpoint REST NVS-backed) and C_slice 13 (commit a743cfc, voice_dispatcher fuzzy keyword aliases). Risks rewritten: PLIP /voice/hook decroche is now the primary trigger, custom Espressif wake-word is optional, microWakeWord remains an open-source alternative path. Gates: 3 new boxes checked.
Tradeoffs: kept the wholesale 'Total core' line and the §9 Open questions entry on mDNS untouched even though both are now obsolete — minimum-diff to satisfy the verrou and keep review easy. Both can be revised in a follow-up pass.
Problem: the LITELLM master key was committed in plain text across five
files. The .gitignore did not block .env files, leaving the door open
for future accidental commits of real credentials. Admin DELETE on the
voice-bridge cache and the hints engine sessions endpoints fall back to
no-auth when their ADMIN_KEY env is unset, with no warning to operators.
Approach: keep the placeholder default so the LAN-only dev workflow
keeps working without any breaking change, but make every operator
aware of the risk. Added gitignore patterns (.env*, *.secret,
secrets.{yaml,yml,json}, credentials.json, *-secret.*) with a
!.env.example whitelist for the template. Created
tools/macstudio/.env.example documenting the four secrets (master key,
two admin keys, CORS) and recommending ~/.zacus.env as the local store.
Voice-bridge and hints engine now log a boot WARN when the default key
or no admin key is in use. README gained a Security section listing the
LAN-only assumption, the no-auth surfaces, and a four-step
pre-deployment checklist (rotate key, source env, audit Cloudflare
tunnel routes, verify boot logs).
Tradeoffs: chose loud-warning over fail-fast because smoke runs and
tests would otherwise need the real key wired before they can boot at
all; loud warnings surface the issue without breaking dev. Did not
rewrite git history to scrub the placeholder — its value is now
treated as known-public, and any deployment outside the closed LAN
must rotate it before going live.
voice-bridge 0.5: CORSMiddleware allowing dashboard/atelier dev origins (env CORS_ORIGINS), and inject NPC_SYSTEM_PROMPT (Professeur Zacus persona, accepts Zaku/Zacusse variants, FR 2-sentence dramatic, never reveal solution) before npc-fast calls in /voice/intent and /voice/ws. Caller-supplied messages or system override the default. Bump ESP32_ZACUS to slice-13 voice_dispatcher fuzzy aliases.
Add tools/macstudio/smoke_e2e.py exercising the full voice-bridge loop
(/health/ready, /tts, /voice/ws hello+stream+intent+TTS, /tts/cache/stats)
without requiring the ESP32 firmware. Three modes (tts/ws/full); generates
a French PCM16 16 kHz fixture via `say -v Thomas` + afconvert when needed;
saves received WAVs and a structured JSON report under /tmp.
Wires `make smoke-voice` and documents the smoke test under
tools/macstudio/README.md (prereqs, exit codes 0/1/2, typical latencies).
Local run hit /tts cache miss + WS round-trip in ~3 s with cache hit on
the second pass; cache delta logging confirmed.
Claude Worker claude2
electron