feat: AI integration — voice pipeline, hints engine, MCP server, analytics, security
- Voice pipeline: ESP32 WebSocket client → voice bridge → LLM → Piper TTS (Tower :8001) - Hints engine: 3 puzzles (LA_440, LEFOU_PIANO, QR_FINALE), anti-cheat, 3 hint levels - MCP hardware server: 6 tools (puzzle, audio, LED, camera, scenario, status), stdio transport - Analytics: ESP32 module + 6 web endpoints + Dashboard UI with chat interface - Security: auth middleware (Bearer NVS), rate limiting, input validation on 30 endpoints - Frontend: code-split (1.1MB → 210KB initial), ErrorBoundary, API timeout, WS reconnect - Tests: 24 Python + 38 TypeScript + 18 MCP = 80 project tests (+ 19 mascarade) - Specs: AI_INTEGRATION_SPEC, MCP_HARDWARE_SERVER_SPEC, QA_TEST_MATRIX_SPEC - Docs: SECURITY, DEPLOYMENT_RUNBOOK, voice pipeline guide, AI architecture map - 6 AI agent definitions (.github/agents/ai_*.md) - TUI orchestration script (tools/dev/zacus_tui.py) - Docker compose TTS for Tower + KXKM-AI - CHANGELOG, README, mkdocs.yml updated - Cycle detection (DFS) in runtime3 validator - Sprint plan: plans/SPRINT_AI_INTEGRATION.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
L'électron rare
+3
-3
@@ -4,7 +4,7 @@
|
||||
Docs agents govern `docs/`, `README.md`, `esp32_audio/README.md`, and any onboarding / briefing files (`.github/agents/`, `docs/_generated/`).
|
||||
|
||||
## Doit
|
||||
- Avant toute mise à jour, lire `docs/AGENT_TODO.md` pour ne pas dupliquer les efforts de coordination.
|
||||
- Avant toute mise à jour, lire `hardware/firmware/docs/AGENT_TODO.md` pour ne pas dupliquer les efforts de coordination.
|
||||
- Refléter tout changement d’organisation ou de procédure dans `docs/TEST_SCRIPT_COORDINATOR.md`, `docs/TEST_COHERENCE_AUDIT_RUNBOOK.md`, et les AGENTS concernés.
|
||||
- Maintenir `docs/_generated/COCKPIT_COMMANDS.md` en cohérence avec `tools/dev/cockpit_commands.yaml` via `python3 tools/dev/gen_cockpit_docs.py`.
|
||||
- Documenter les nouveaux artefacts (gates/tests) dans la section adéquate du runbook et mentionner les chemins dans le reporting template.
|
||||
@@ -12,7 +12,7 @@ Docs agents govern `docs/`, `README.md`, `esp32_audio/README.md`, and any onboar
|
||||
|
||||
## Artefacts
|
||||
- Les docs n’intègrent pas d’artefacts binaires ; citez plutôt les dossiers `artifacts/` lorsque vous décrivez un protocole ou un gate réussi.
|
||||
- Pour les instructions “TODO”, mettez à jour `docs/AGENT_TODO.md` (bloc 4/5) en parallèle des docs.
|
||||
- Pour les instructions “TODO”, mettez à jour `hardware/firmware/docs/AGENT_TODO.md` (bloc 4/5) en parallèle des docs.
|
||||
|
||||
## Bonnes pratiques
|
||||
- Chaque mise à jour de runbook ou guide doit inclure un rappel sur la nécessité de respecter `docs/AGENT_TODO.md` comme tracker unique.
|
||||
- Chaque mise à jour de runbook ou guide doit inclure un rappel sur la nécessité de respecter `hardware/firmware/docs/AGENT_TODO.md` comme tracker unique.
|
||||
|
||||
@@ -4,15 +4,15 @@
|
||||
Firmware agents cover `esp32_audio/`, `ui/esp8266_oled/`, `ui/rp2040_tft/`, and shared `protocol/` code that drives the MCU binaries.
|
||||
|
||||
## Doit
|
||||
- Lire puis mettre à jour `docs/AGENT_TODO.md` avant d’agir (le tracker unique garde la trace de l’état de chaque gate).
|
||||
- Lire puis mettre à jour `hardware/firmware/docs/AGENT_TODO.md` avant d’agir (le tracker unique garde la trace de l’état de chaque gate).
|
||||
- Respecter le contrat global (`AGENTS.md`) et les règles d’outillage de `tools/dev/AGENTS.md` (gates/scripts centralisés, artefacts hors git).
|
||||
- S’assurer que tout build se passe via `platformio.ini`/`build_all.sh` ou `Makefile fast-*`, et documenter les artefacts dans `docs/AGENT_TODO.md`.
|
||||
- Pour les modifications de structure, ajouter un commentaire dans `docs/AGENT_TODO.md` + mentionner le commit dans `docs/TEST_SCRIPT_COORDINATOR.md`.
|
||||
- Toujours signaler l’état des UI Link / LittleFS / I2S dans `docs/AGENT_TODO.md:6-14` lorsqu’on manipule ces composants.
|
||||
- S’assurer que tout build se passe via `platformio.ini`/`build_all.sh` ou `Makefile fast-*`, et documenter les artefacts dans `hardware/firmware/docs/AGENT_TODO.md`.
|
||||
- Pour les modifications de structure, ajouter un commentaire dans `hardware/firmware/docs/AGENT_TODO.md` + mentionner le commit dans `docs/TEST_SCRIPT_COORDINATOR.md`.
|
||||
- Toujours signaler l’état des UI Link / LittleFS / I2S dans `hardware/firmware/docs/AGENT_TODO.md:6-14` lorsqu’on manipule ces composants.
|
||||
|
||||
## Reporting
|
||||
- Chaque run doit produire des artefacts `artifacts/<phase>/<timestamp>` ; les chemins doivent figurer dans la case correspondante du TODO.
|
||||
- Données hardware (logs, ports) restent hors git : référencer les fichiers dans `docs/AGENT_TODO.md` ou le rapport final.
|
||||
- Données hardware (logs, ports) restent hors git : référencer les fichiers dans `hardware/firmware/docs/AGENT_TODO.md` ou le rapport final.
|
||||
|
||||
## References
|
||||
- Branche principale : `docs/SPRINT_RECOMMENDATIONS.md`, `README.md`, `protocol/ui_link_v2.md`
|
||||
|
||||
@@ -4,15 +4,15 @@
|
||||
Tests agents focus on files under `tools/dev/`, `tools/test/`, `esp32_audio/tests/`, and the artifacts/log directories (`artifacts/`, `logs/`).
|
||||
|
||||
## Doit
|
||||
- Toujours lire/mettre à jour `docs/AGENT_TODO.md` pour annoncer les gates en cours ou bloqués (ex. smoke, stress, audit).
|
||||
- Toujours lire/mettre à jour `hardware/firmware/docs/AGENT_TODO.md` pour annoncer les gates en cours ou bloqués (ex. smoke, stress, audit).
|
||||
- Exécuter les calibrations/gates via les scripts officiels (`tools/dev/run_matrix_and_smoke.sh`, `tools/dev/run_smoke_tests.sh`, `tools/dev/run_stress_tests.py`, `tools/test/audit_coherence.py`), puis consigner les artefacts sous `artifacts/`.
|
||||
- Tenir `docs/TEST_SCRIPT_COORDINATOR.md` à jour avec les résultats et les chemins d’évidence (section “Exécutions récentes” + reporting template).
|
||||
- Valider les commandes cockpit via `tools/dev/cockpit_commands.yaml` et `docs/_generated/COCKPIT_COMMANDS.md` (regénération avec `python3 tools/dev/gen_cockpit_docs.py` si besoin).
|
||||
- Mentionner les échecs réseau (HTTP/WebSocket/WiFi) dans `docs/AGENT_TODO.md` et `docs/TEST_SCRIPT_COORDINATOR.md` pour la coordination.
|
||||
- Mentionner les échecs réseau (HTTP/WebSocket/WiFi) dans `hardware/firmware/docs/AGENT_TODO.md` et `docs/TEST_SCRIPT_COORDINATOR.md` pour la coordination.
|
||||
|
||||
## Artefacts
|
||||
- Les logs exigent `meta.json`, `git.txt`, `commands.txt`, `summary.md` (au minimum) ; vérifiez les répertoires demandés dans `docs/TEST_SCRIPT_COORDINATOR.md`.
|
||||
- Pas de commit d’artefacts ; les chemins doivent être mentionnés dans le TODO.
|
||||
|
||||
## Bonnes pratiques
|
||||
- En cas de panne persistante (UI link, stress panic, scénario manquant), capturez les traces dans `artifacts/rc_live/<timestamp>` et notez la prochaine action dans `docs/AGENT_TODO.md`.
|
||||
- En cas de panne persistante (UI link, stress panic, scénario manquant), capturez les traces dans `artifacts/rc_live/<timestamp>` et notez la prochaine action dans `hardware/firmware/docs/AGENT_TODO.md`.
|
||||
|
||||
@@ -0,0 +1,235 @@
|
||||
# Analyse IA & Intégration — Le Mystere du Professeur Zacus
|
||||
|
||||
> Generee le 2026-03-21 par analyse exhaustive (firmware, frontend, tooling, docs, web research)
|
||||
|
||||
---
|
||||
|
||||
## 1. SWOT — Firmware ESP32-S3
|
||||
|
||||
### Forces
|
||||
- Architecture modulaire (audio/UI/network/scenario managers)
|
||||
- Gestion memoire PSRAM mature (caps_allocator, fallback chains)
|
||||
- Audio I2S avec protection underrun, DMA async
|
||||
- LVGL avec DMA flush async, SIMD optionnel
|
||||
- Runtime 3 step-based avec transitions event-driven
|
||||
|
||||
### Faiblesses (CRITIQUES)
|
||||
| ID | Severite | Issue | Fichier |
|
||||
|----|----------|-------|---------|
|
||||
| FW-01 | CRITICAL | Credentials WiFi en dur | storage_manager.cpp:73 |
|
||||
| FW-02 | CRITICAL | API web sans authentification | main.cpp:5932-5960 |
|
||||
| FW-03 | HIGH | Watchdog timeout (calculator eval) | main.cpp + platformio.ini |
|
||||
| FW-04 | HIGH | Pas de validation input API | main.cpp:5945-5950 |
|
||||
| FW-05 | HIGH | Pas de rate limiting | main.cpp:5200-5960 |
|
||||
| FW-06 | HIGH | Pas de timeout JSON parsing | main.cpp |
|
||||
| FW-07 | MEDIUM | LVGL fragmentation (54KB pool) | platformio.ini:80 |
|
||||
| FW-08 | MEDIUM | Audio underrun sans recovery | audio_manager.cpp:407-418 |
|
||||
| FW-09 | MEDIUM | Buffer overflow string ops | ui_manager.cpp:145 |
|
||||
| FW-10 | MEDIUM | Pas de HTTPS/TLS | main.cpp:5966 |
|
||||
|
||||
### Opportunites
|
||||
- OTA firmware updates (partition scheme compatible)
|
||||
- Secure Boot + Flash encryption (ESP32-S3 natif)
|
||||
- Auth middleware centralise pour webOnApi()
|
||||
- Watchdog supervisor software
|
||||
|
||||
---
|
||||
|
||||
## 2. SWOT — Frontend React+Blockly
|
||||
|
||||
### Forces
|
||||
- Architecture composants clean (4 onglets)
|
||||
- API client complet (30+ endpoints, dual protocol)
|
||||
- Blockly bidirectionnel (workspace <-> YAML)
|
||||
- TypeScript strict + Zod validation
|
||||
- Accessibilite (aria-label, aria-live)
|
||||
|
||||
### Faiblesses
|
||||
| ID | Severite | Issue | Fichier |
|
||||
|----|----------|-------|---------|
|
||||
| FE-01 | HIGH | Zero tests (0% coverage) | — |
|
||||
| FE-02 | HIGH | Pas de React ErrorBoundary | App.tsx |
|
||||
| FE-03 | HIGH | Pas de timeout API requests | api.ts:21-34 |
|
||||
| FE-04 | MEDIUM | Blockly registration globale mutable | BlocklyDesigner.tsx:35-86 |
|
||||
| FE-05 | MEDIUM | Pas de reconnexion WebSocket | api.ts:274-289 |
|
||||
| FE-06 | MEDIUM | Bundle bloat (Blockly+Monaco ~2.5MB) | package.json |
|
||||
| FE-07 | LOW | Tab state non persiste | App.tsx:22 |
|
||||
| FE-08 | LOW | Pas de dark mode | App.css |
|
||||
|
||||
---
|
||||
|
||||
## 3. SWOT — Python Tooling
|
||||
|
||||
### Forces
|
||||
- Pipeline clair (compile -> simulate -> validate -> export)
|
||||
- Validation semantique comprehensive
|
||||
- Simulation deterministe avec detection cycles (max_steps)
|
||||
- Shell scripts robustes (set -euo pipefail)
|
||||
|
||||
### Faiblesses
|
||||
| ID | Severite | Issue | Fichier |
|
||||
|----|----------|-------|---------|
|
||||
| PY-01 | HIGH | Seulement 5 tests (pas de negatifs) | test_runtime3_routes.py |
|
||||
| PY-02 | HIGH | Pas de detection cycles transitions | runtime3_common.py:227-233 |
|
||||
| PY-03 | MEDIUM | Schema version hard-codee (v1 only) | runtime3_common.py:196 |
|
||||
| PY-04 | MEDIUM | normalize_token() fallback silencieux | runtime3_common.py:31-33 |
|
||||
| PY-05 | LOW | Pas de TypedDict/dataclass partout | runtime3_common.py |
|
||||
|
||||
---
|
||||
|
||||
## 4. Documentation — Etat
|
||||
|
||||
| Zone | Completude | Action |
|
||||
|------|-----------|--------|
|
||||
| Architecture (8 maps) | 100% | A jour |
|
||||
| Specifications (13 specs) | 90% | 3 specs critiques manquantes |
|
||||
| Getting Started | 95% | OK |
|
||||
| Operations | 30% | Runbook manquant |
|
||||
| Securite | 10% | Stub seulement |
|
||||
| Tests/QA | 40% | Pas de matrice unifiee |
|
||||
|
||||
### Specs MANQUANTES
|
||||
1. `DEPLOYMENT_RUNBOOK.md` — procedures terrain
|
||||
2. `SECURITY.md` — modele auth, menaces, remediations
|
||||
3. `MCP_HARDWARE_SERVER_SPEC.md` — integration mascarade MCP
|
||||
4. `ANALYTICS_OBSERVABILITY_SPEC.md` — telemetrie temps reel
|
||||
5. `QA_TEST_MATRIX_SPEC.md` — matrice de tests formelle
|
||||
6. `NETWORK_TOPOLOGY_SPEC.md` — ESP-NOW format messages
|
||||
|
||||
### Fichiers OBSOLETES a supprimer
|
||||
- `docs/AGENTS 2.md`, `docs/AGENT_TODO 2.md` (duplicates)
|
||||
- `docs/AGENTS_DOCS.md`, `docs/AGENTS_FIRMWARE.md` (remplace par .github/agents/)
|
||||
- `docs/GENERER_UN_SCENARIO_STORY_V2.md` (references obsoletes)
|
||||
|
||||
---
|
||||
|
||||
## 5. Etat de l'Art IA 2026 — Opportunites d'Integration
|
||||
|
||||
### TOP 5 Technologies Prioritaires
|
||||
|
||||
| # | Technologie | Usage Zacus | Maturite | Licence |
|
||||
|---|------------|-------------|----------|---------|
|
||||
| 1 | **ESP-SR v2.0** (Espressif) | Wake word "Hey Zacus" + commandes vocales offline (300 mots) | Production | Espressif |
|
||||
| 2 | **Coqui XTTS-v2** | Cloner la voix du Prof Zacus (6s sample) pour narration dynamique | Production | MPL-2.0 |
|
||||
| 3 | **ESP-DL v3.2** | Detection objets on-device (YOLOv11n, 7 FPS) pour puzzles physiques | Production | MIT |
|
||||
| 4 | **ESP RainMaker MCP** | Controle materiel via LLM ("allume la lampe UV salle 3") | Production | Apache 2.0 |
|
||||
| 5 | **AudioCraft MusicGen** | Musique ambiante generative par salle/puzzle sur KXKM-AI | Production | MIT/CC-BY-NC |
|
||||
|
||||
### Projets de Reference
|
||||
|
||||
| Projet | Stars | Pertinence | URL |
|
||||
|--------|-------|-----------|-----|
|
||||
| **XiaoZhi ESP32** | 25k+ | Architecture quasi-identique (ESP32-S3 + wake + LLM + TTS via MCP) | github.com/78/xiaozhi-esp32 |
|
||||
| **Willow** | — | Pipeline voix ESP32-S3 <500ms latence | github.com/HeyWillow/willow |
|
||||
| **ClueControl** | — | Puzzles Arduino escape room (RFID, maglocks) | github.com/ClueControl |
|
||||
| **EscapeRoom (devlinb)** | — | Backend Node.js anti-prompt-injection pour hints IA | github.com/devlinb/escaperoom |
|
||||
| **IoT-MCP (Duke)** | — | Framework MCP pour IoT, 205ms latence, 74KB RAM | github.com/Duke-CEI-Center/IoT-MCP-Servers |
|
||||
|
||||
### Architecture IA Cible
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph ESP32-S3["ESP32-S3 (On-Device)"]
|
||||
SR[ESP-SR v2.0<br/>Wake Word + Commands]
|
||||
DL[ESP-DL v3.2<br/>Object Detection]
|
||||
CAM[Camera OV2640]
|
||||
MIC[Microphone I2S]
|
||||
SPK[Speaker I2S]
|
||||
end
|
||||
|
||||
subgraph Server["Serveur mascarade"]
|
||||
LLM[LLM via mascarade API<br/>Hints adaptatifs]
|
||||
TTS[Coqui XTTS-v2<br/>Voix Prof Zacus]
|
||||
MCP[MCP Server<br/>Hardware Control]
|
||||
ANALYTICS[Analytics Engine<br/>Difficulte adaptative]
|
||||
end
|
||||
|
||||
subgraph KXKM["KXKM-AI (RTX 4090)"]
|
||||
MUSIC[AudioCraft MusicGen<br/>Ambient + SFX]
|
||||
TRAIN[Fine-tune modeles<br/>voix/detection custom]
|
||||
end
|
||||
|
||||
MIC --> SR
|
||||
CAM --> DL
|
||||
SR -->|"commande vocale"| MCP
|
||||
DL -->|"objet detecte"| MCP
|
||||
MCP -->|"action puzzle"| ESP32-S3
|
||||
MCP <-->|"API mascarade"| LLM
|
||||
LLM -->|"hint text"| TTS
|
||||
TTS -->|"audio stream"| SPK
|
||||
MUSIC -->|"ambient MP3"| SPK
|
||||
ESP32-S3 -->|"telemetrie"| ANALYTICS
|
||||
ANALYTICS -->|"ajuster difficulte"| LLM
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Plan d'Integration IA — Phases
|
||||
|
||||
### Phase A: Fondations Securite (P0 — 1-2 semaines)
|
||||
1. Supprimer credentials WiFi en dur → NVS + provisioning QR
|
||||
2. Ajouter auth Bearer token sur tous les endpoints API
|
||||
3. Input validation + rate limiting
|
||||
4. Augmenter LVGL pool 54→96KB
|
||||
5. Augmenter stack Arduino 16→24KB
|
||||
|
||||
### Phase B: Voice Pipeline (P1 — 2-4 semaines)
|
||||
1. Integrer ESP-SR v2.0 pour wake word "Hey Zacus"
|
||||
2. Deployer Coqui XTTS-v2 en Docker sur VM mascarade
|
||||
3. Pipeline: ESP32 mic → WiFi stream → mascarade → LLM → TTS → ESP32 speaker
|
||||
4. Commandes vocales offline (MultiNet, 50 mots FR)
|
||||
5. Ref: XiaoZhi ESP32 architecture
|
||||
|
||||
### Phase C: Vision & Detection (P1 — 2-4 semaines)
|
||||
1. Integrer ESP-DL v3.2 pour detection objets puzzle
|
||||
2. Entrainer modele custom (props specifiques Zacus)
|
||||
3. Face detection pour comptage joueurs (ESP-WHO)
|
||||
4. Au-dela du QR basique: detection indices physiques
|
||||
|
||||
### Phase D: LLM Hints Adaptatifs (P2 — 4-6 semaines)
|
||||
1. API mascarade comme backend LLM pour hints contextuels
|
||||
2. Prompt engineering anti-triche (ref: devlinb/escaperoom)
|
||||
3. Analytics temps reel → ajustement difficulte
|
||||
4. Prof Zacus comme NPC LLM avec memoire conversation
|
||||
|
||||
### Phase E: Audio Generatif (P2 — 2-3 semaines)
|
||||
1. AudioCraft MusicGen sur KXKM-AI (RTX 4090)
|
||||
2. Generation ambiante par salle/puzzle
|
||||
3. SFX dynamiques via Stable Audio Open
|
||||
4. Streaming vers ESP32 speakers
|
||||
|
||||
### Phase F: MCP & Orchestration (P3 — 4-6 semaines)
|
||||
1. MCP server hardware (ESP RainMaker MCP pattern)
|
||||
2. Integration mascarade MCP existant
|
||||
3. Controle naturel-language de tous les peripheriques
|
||||
4. Dashboard game master temps reel
|
||||
|
||||
---
|
||||
|
||||
## 7. Corrections Prioritaires Code
|
||||
|
||||
### Immediate (cette semaine)
|
||||
```
|
||||
FW-01: storage_manager.cpp — NVS credentials
|
||||
FW-02: main.cpp — Bearer auth middleware
|
||||
FW-03: platformio.ini — stack 16→24KB
|
||||
FE-02: App.tsx — ErrorBoundary wrapper
|
||||
FE-03: api.ts — timeout 5s defaut
|
||||
```
|
||||
|
||||
### Court terme (2 semaines)
|
||||
```
|
||||
FW-04-06: main.cpp — input validation, rate limit, JSON timeout
|
||||
FW-07: platformio.ini — LVGL pool 54→96KB
|
||||
PY-01: tests — 5→25+ tests avec negatifs
|
||||
PY-02: runtime3_common.py — detection cycles
|
||||
FE-01: frontend — premiers tests Vitest
|
||||
```
|
||||
|
||||
### Moyen terme (1 mois)
|
||||
```
|
||||
FW-08-10: audio recovery, string safety, TLS
|
||||
FE-04-06: Blockly cleanup, WS reconnect, bundle split
|
||||
PY-03-05: schema migration, TypedDict, normalize warnings
|
||||
DOCS: specs manquantes + cleanup obsoletes
|
||||
```
|
||||
@@ -0,0 +1,446 @@
|
||||
# Deployment Runbook
|
||||
|
||||
## Status
|
||||
- State: draft
|
||||
- Date: 2026-03-21
|
||||
- Audience: Game master, field technician
|
||||
- Related: `docs/QUICKSTART.md`, `docs/SECURITY.md`
|
||||
|
||||
---
|
||||
|
||||
## 1) Pre-Deployment Checklist
|
||||
|
||||
### Hardware
|
||||
- [ ] ESP32-S3 Freenove board powered and accessible via USB
|
||||
- [ ] Speaker connected to I2S output (verified with test tone)
|
||||
- [ ] LED strips connected and addressed (WS2812B data pin)
|
||||
- [ ] Camera OV2640 connected (if vision features enabled)
|
||||
- [ ] Microphone INMP441 connected (if voice features enabled)
|
||||
- [ ] All puzzle actuators (servos, relays) wired and tested
|
||||
- [ ] Backup ESP32 board available on-site
|
||||
|
||||
### Network
|
||||
- [ ] WiFi access point configured and powered
|
||||
- [ ] SSID and password documented (not the default)
|
||||
- [ ] ESP32 IP address reserved (DHCP reservation or static)
|
||||
- [ ] mascarade server reachable from game WiFi (if AI features)
|
||||
- [ ] Game master device (laptop/tablet) on same network
|
||||
|
||||
### Content
|
||||
- [ ] Scenario YAML validated: `bash tools/test/run_content_checks.sh`
|
||||
- [ ] Runtime 3 IR compiled: `python3 tools/scenario/compile_runtime3.py game/scenarios/zacus_v2.yaml`
|
||||
- [ ] Runtime 3 simulation passed: `python3 tools/scenario/simulate_runtime3.py game/scenarios/zacus_v2.yaml`
|
||||
- [ ] Audio files present in `hardware/firmware/data/audio/`
|
||||
- [ ] All audio files referenced in scenario exist on filesystem
|
||||
- [ ] Printable materials generated and printed
|
||||
|
||||
### Security
|
||||
- [ ] API bearer token generated and stored in NVS
|
||||
- [ ] WiFi credentials stored in NVS (not hardcoded)
|
||||
- [ ] `.env` file configured on server with matching token
|
||||
- [ ] Serial debug disabled in production firmware
|
||||
- [ ] Review `docs/SECURITY.md` checklist
|
||||
|
||||
---
|
||||
|
||||
## 2) Firmware Flash Procedure
|
||||
|
||||
### 2.1 Prerequisites
|
||||
|
||||
```bash
|
||||
# Install PlatformIO CLI (if not installed)
|
||||
pip install platformio
|
||||
|
||||
# Verify toolchain
|
||||
pio platform update espressif32
|
||||
```
|
||||
|
||||
### 2.2 Build Firmware
|
||||
|
||||
```bash
|
||||
cd hardware/firmware
|
||||
|
||||
# Build for Freenove ESP32-S3
|
||||
pio run -e freenove_esp32s3
|
||||
|
||||
# Build for ESP8266 OLED (secondary display)
|
||||
pio run -e esp8266_oled
|
||||
```
|
||||
|
||||
### 2.3 Flash Firmware
|
||||
|
||||
```bash
|
||||
# Connect ESP32-S3 via USB, identify port
|
||||
ls /dev/tty.usb* # macOS
|
||||
ls /dev/ttyUSB* # Linux
|
||||
|
||||
# Flash firmware + bootloader + partition table
|
||||
pio run -e freenove_esp32s3 -t upload --upload-port /dev/tty.usbmodem*
|
||||
|
||||
# Monitor serial output to verify boot
|
||||
pio device monitor -e freenove_esp32s3 --port /dev/tty.usbmodem* -b 115200
|
||||
```
|
||||
|
||||
### 2.4 Expected Boot Output
|
||||
|
||||
```
|
||||
[INFO] Zacus Runtime 3 v3.1
|
||||
[INFO] Free heap: 245000 bytes
|
||||
[INFO] PSRAM: 8388608 bytes
|
||||
[INFO] WiFi connecting to: EscapeRoom-Net
|
||||
[INFO] WiFi connected, IP: 192.168.0.50
|
||||
[INFO] HTTP server started on port 8080
|
||||
[INFO] Scenario loaded: ZACUS_V2 (11 steps)
|
||||
[INFO] Initial step: RTC_ESP_ETAPE1
|
||||
[INFO] Audio manager ready
|
||||
[INFO] LED manager ready (60 LEDs)
|
||||
[INFO] Ready.
|
||||
```
|
||||
|
||||
### 2.5 Provisioning Credentials
|
||||
|
||||
On first boot (or after NVS erase):
|
||||
|
||||
```bash
|
||||
# Erase NVS to force setup mode
|
||||
pio run -e freenove_esp32s3 -t erase
|
||||
|
||||
# Flash and boot — device enters AP mode "Zacus-Setup"
|
||||
pio run -e freenove_esp32s3 -t upload
|
||||
|
||||
# Connect to "Zacus-Setup" WiFi from your phone/laptop
|
||||
# Open http://192.168.4.1 in browser
|
||||
# Enter WiFi SSID, password, and API token
|
||||
# Device reboots in STA mode
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3) Content Deployment (LittleFS)
|
||||
|
||||
### 3.1 Prepare Filesystem Image
|
||||
|
||||
Content files are stored in `hardware/firmware/data/`:
|
||||
|
||||
```
|
||||
data/
|
||||
story/
|
||||
runtime3/
|
||||
DEFAULT.json # Runtime 3 IR (compiled)
|
||||
scenarios/
|
||||
DEFAULT.json # Legacy scenario
|
||||
audio/
|
||||
ambient_01.mp3
|
||||
hint_01.mp3
|
||||
sfx_unlock.mp3
|
||||
...
|
||||
config/
|
||||
device.json # Device-specific config
|
||||
```
|
||||
|
||||
### 3.2 Upload LittleFS Image
|
||||
|
||||
```bash
|
||||
cd hardware/firmware
|
||||
|
||||
# Build and upload filesystem image
|
||||
pio run -e freenove_esp32s3 -t uploadfs --upload-port /dev/tty.usbmodem*
|
||||
```
|
||||
|
||||
### 3.3 Verify Content Upload
|
||||
|
||||
```bash
|
||||
# Via serial
|
||||
> ls /littlefs/story/runtime3/
|
||||
DEFAULT.json (4523 bytes)
|
||||
|
||||
# Via API (requires auth)
|
||||
curl -H "Authorization: Bearer <token>" http://192.168.0.50:8080/api/status
|
||||
# Should show: "scenario": "ZACUS_V2", "steps": 11
|
||||
```
|
||||
|
||||
### 3.4 Update Content Without Reflashing Firmware
|
||||
|
||||
For content-only updates (new scenario, audio files):
|
||||
|
||||
```bash
|
||||
# Recompile scenario
|
||||
python3 tools/scenario/compile_runtime3.py game/scenarios/zacus_v2.yaml
|
||||
|
||||
# Copy to firmware data directory
|
||||
cp artifacts/runtime3/latest/DEFAULT.json hardware/firmware/data/story/runtime3/
|
||||
|
||||
# Upload only filesystem (firmware unchanged)
|
||||
pio run -e freenove_esp32s3 -t uploadfs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 4) Frontend Deployment
|
||||
|
||||
### 4.1 Build Frontend
|
||||
|
||||
```bash
|
||||
cd frontend-scratch-v2
|
||||
|
||||
# Install dependencies
|
||||
npm install
|
||||
|
||||
# Run tests
|
||||
npm test
|
||||
|
||||
# Run lint
|
||||
npm run lint
|
||||
|
||||
# Build for production
|
||||
VITE_STORY_API_BASE=http://192.168.0.50:8080 npm run build
|
||||
```
|
||||
|
||||
Build output is in `frontend-scratch-v2/dist/`.
|
||||
|
||||
### 4.2 Serve Frontend
|
||||
|
||||
**Option A: From ESP32 (embedded)**
|
||||
```bash
|
||||
# Copy built files to firmware data directory
|
||||
cp -r frontend-scratch-v2/dist/* hardware/firmware/data/www/
|
||||
|
||||
# Upload filesystem
|
||||
cd hardware/firmware
|
||||
pio run -e freenove_esp32s3 -t uploadfs
|
||||
```
|
||||
|
||||
The ESP32 serves the frontend at `http://<esp32-ip>:8080/`.
|
||||
|
||||
**Option B: From separate server**
|
||||
```bash
|
||||
# Serve with any static file server
|
||||
npx serve frontend-scratch-v2/dist -l 3000
|
||||
|
||||
# Or via zacus.sh
|
||||
./tools/dev/zacus.sh frontend-serve
|
||||
```
|
||||
|
||||
### 4.3 Using the Shell Shortcut
|
||||
|
||||
```bash
|
||||
# Build + lint + test in one command
|
||||
./tools/dev/zacus.sh frontend-build
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5) Network Setup
|
||||
|
||||
### 5.1 WiFi Access Point Configuration
|
||||
|
||||
| Parameter | Recommended Value |
|
||||
|-----------|------------------|
|
||||
| SSID | `EscapeRoom-Net` (hidden optional) |
|
||||
| Security | WPA2-PSK |
|
||||
| Channel | 1, 6, or 11 (least congested) |
|
||||
| Band | 2.4 GHz (ESP32 does not support 5 GHz) |
|
||||
| DHCP range | 192.168.0.50 - 192.168.0.99 |
|
||||
| DNS | Not required (local network only) |
|
||||
|
||||
### 5.2 IP Address Assignments
|
||||
|
||||
| Device | IP | Port | Purpose |
|
||||
|--------|-----|------|---------|
|
||||
| WiFi AP / Router | 192.168.0.1 | — | Gateway |
|
||||
| ESP32 main | 192.168.0.50 | 8080 | Game device |
|
||||
| ESP32 salle 2 | 192.168.0.51 | 8080 | Secondary device |
|
||||
| Game master laptop | 192.168.0.10 | 3000 | Dashboard |
|
||||
| mascarade server | 192.168.0.119 | 4001 | AI backend |
|
||||
|
||||
### 5.3 ESP-NOW Pairing
|
||||
|
||||
For multi-device setups using ESP-NOW mesh:
|
||||
|
||||
```bash
|
||||
# On primary device (serial console)
|
||||
> espnow pair
|
||||
Pairing mode active. Press button on secondary device...
|
||||
|
||||
# On secondary device (serial console)
|
||||
> espnow join
|
||||
Scanning for primary...
|
||||
Paired with zacus-main (MAC: AA:BB:CC:DD:EE:FF)
|
||||
```
|
||||
|
||||
### 5.4 Verify Network Connectivity
|
||||
|
||||
```bash
|
||||
# Ping ESP32
|
||||
ping 192.168.0.50
|
||||
|
||||
# Check API health
|
||||
curl http://192.168.0.50:8080/health
|
||||
# Expected: {"status":"ok"}
|
||||
|
||||
# Check authenticated endpoint
|
||||
curl -H "Authorization: Bearer <token>" http://192.168.0.50:8080/api/status
|
||||
# Expected: full status JSON
|
||||
|
||||
# Check mascarade (if AI features)
|
||||
curl http://192.168.0.119:4001/health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6) Post-Deployment Validation
|
||||
|
||||
### 6.1 Functional Checks
|
||||
|
||||
| # | Check | Command / Action | Expected |
|
||||
|---|-------|-----------------|----------|
|
||||
| 1 | Device boots | Power on, watch serial | "Ready." message |
|
||||
| 2 | WiFi connects | Serial log | IP address assigned |
|
||||
| 3 | API responds | `curl /health` | `{"status":"ok"}` |
|
||||
| 4 | Auth works | `curl /api/status` with token | 200 OK |
|
||||
| 5 | Auth rejects | `curl /api/status` without token | 401 |
|
||||
| 6 | Scenario loaded | Check `/api/status` response | Correct scenario ID |
|
||||
| 7 | Audio plays | Trigger audio via API or button | Sound from speaker |
|
||||
| 8 | LEDs respond | Trigger LED via API or scenario | Correct colors |
|
||||
| 9 | Frontend loads | Open browser to device IP | UI renders |
|
||||
| 10 | Full walkthrough | Play through scenario start to finish | All transitions work |
|
||||
|
||||
### 6.2 Automated Validation Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# tools/deploy/validate.sh
|
||||
set -euo pipefail
|
||||
|
||||
ESP_IP="${1:-192.168.0.50}"
|
||||
TOKEN="${2:-$(cat .env | grep API_TOKEN | cut -d= -f2)}"
|
||||
|
||||
echo "Validating deployment at $ESP_IP..."
|
||||
|
||||
# Health check
|
||||
curl -sf "http://$ESP_IP:8080/health" | jq .status
|
||||
echo "Health: OK"
|
||||
|
||||
# Auth check
|
||||
HTTP_CODE=$(curl -sf -o /dev/null -w "%{http_code}" \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
"http://$ESP_IP:8080/api/status")
|
||||
[ "$HTTP_CODE" = "200" ] && echo "Auth: OK" || echo "Auth: FAIL ($HTTP_CODE)"
|
||||
|
||||
# Scenario check
|
||||
SCENARIO=$(curl -sf -H "Authorization: Bearer $TOKEN" \
|
||||
"http://$ESP_IP:8080/api/status" | jq -r .scenario)
|
||||
echo "Scenario: $SCENARIO"
|
||||
|
||||
# Step count
|
||||
STEPS=$(curl -sf -H "Authorization: Bearer $TOKEN" \
|
||||
"http://$ESP_IP:8080/api/status" | jq .steps)
|
||||
echo "Steps: $STEPS"
|
||||
|
||||
echo "Validation complete."
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 7) Rollback Procedure
|
||||
|
||||
### 7.1 Firmware Rollback
|
||||
|
||||
```bash
|
||||
# Keep previous firmware binary
|
||||
cp .pio/build/freenove_esp32s3/firmware.bin backups/firmware_$(date +%Y%m%d).bin
|
||||
|
||||
# To rollback:
|
||||
pio run -e freenove_esp32s3 -t upload --upload-port /dev/tty.usbmodem* \
|
||||
--firmware backups/firmware_20260320.bin
|
||||
```
|
||||
|
||||
### 7.2 Content Rollback
|
||||
|
||||
```bash
|
||||
# Content is versioned in git
|
||||
git log --oneline hardware/firmware/data/story/
|
||||
|
||||
# Restore previous version
|
||||
git checkout HEAD~1 -- hardware/firmware/data/story/runtime3/DEFAULT.json
|
||||
|
||||
# Re-upload filesystem
|
||||
cd hardware/firmware
|
||||
pio run -e freenove_esp32s3 -t uploadfs
|
||||
```
|
||||
|
||||
### 7.3 Frontend Rollback
|
||||
|
||||
```bash
|
||||
# Previous builds in git
|
||||
git log --oneline frontend-scratch-v2/
|
||||
|
||||
# Restore and rebuild
|
||||
git checkout HEAD~1 -- frontend-scratch-v2/
|
||||
cd frontend-scratch-v2 && npm run build
|
||||
```
|
||||
|
||||
### 7.4 Emergency: Factory Reset
|
||||
|
||||
```bash
|
||||
# Full erase (firmware + NVS + filesystem)
|
||||
pio run -e freenove_esp32s3 -t erase
|
||||
|
||||
# Reflash everything from scratch
|
||||
pio run -e freenove_esp32s3 -t upload
|
||||
pio run -e freenove_esp32s3 -t uploadfs
|
||||
|
||||
# Re-provision credentials via AP mode
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 8) Troubleshooting Quick Reference
|
||||
|
||||
| Symptom | Likely Cause | Fix |
|
||||
|---------|-------------|-----|
|
||||
| No serial output | Wrong USB port or baud rate | Try `115200` baud, check cable (data-capable) |
|
||||
| Boot loop | Stack overflow or panic | Increase stack in `platformio.ini` (16 -> 24 KB) |
|
||||
| WiFi won't connect | Wrong credentials or channel | Erase NVS, re-provision. Check 2.4 GHz only |
|
||||
| WiFi keeps disconnecting | Signal too weak or interference | Move AP closer, change channel |
|
||||
| API returns 401 | Token mismatch | Re-provision token via AP setup portal |
|
||||
| API returns 429 | Rate limited | Wait 1 s, reduce request frequency |
|
||||
| No audio | I2S pin mismatch or volume 0 | Check `platformio.ini` pin defines, set volume > 0 |
|
||||
| Audio distorted | DMA buffer underrun | Reduce concurrent tasks, increase DMA buffer count |
|
||||
| LEDs wrong color | GRB/RGB order mismatch | Check LED type in code (WS2812B = GRB) |
|
||||
| LEDs flicker | Insufficient power supply | Use dedicated 5V supply, add 100uF cap |
|
||||
| LVGL crash | Pool too small | Increase to 96 KB in `platformio.ini` |
|
||||
| Scenario stuck | Missing transition for current state | Check `DEFAULT.json` transitions, use serial `> status` |
|
||||
| ESP-NOW not pairing | Different WiFi channels | Both devices must be on same channel |
|
||||
| Camera timeout | I2C conflict or power issue | Check camera ribbon cable, power cycle |
|
||||
| OOM / heap exhaustion | Memory leak or too many features | Monitor with `> heap`, disable unused features |
|
||||
| Frontend won't connect | Wrong `VITE_STORY_API_BASE` | Rebuild with correct ESP32 IP |
|
||||
| mascarade unreachable | Network or Docker issue | Check `ping`, `docker ps`, firewall rules |
|
||||
|
||||
### Serial Debug Commands
|
||||
|
||||
```
|
||||
> status # Device state, heap, uptime
|
||||
> heap # Detailed memory breakdown
|
||||
> wifi # WiFi RSSI, IP, channel
|
||||
> scenario # Current step, available transitions
|
||||
> transition <event> # Manually trigger a transition
|
||||
> audio list # List available audio files
|
||||
> audio play <file> # Play an audio file
|
||||
> led test # Cycle through LED test patterns
|
||||
> reboot # Software restart
|
||||
> factory-reset # Erase NVS + reboot into AP mode
|
||||
```
|
||||
|
||||
### Useful Monitoring Commands
|
||||
|
||||
```bash
|
||||
# Watch serial output continuously
|
||||
pio device monitor -e freenove_esp32s3 -b 115200
|
||||
|
||||
# Stream API status every 5 seconds
|
||||
watch -n 5 'curl -s -H "Authorization: Bearer <token>" http://192.168.0.50:8080/api/status | jq .'
|
||||
|
||||
# Check WiFi signal from ESP32 perspective
|
||||
curl -s -H "Authorization: Bearer <token>" http://192.168.0.50:8080/api/status | jq .wifi_rssi
|
||||
```
|
||||
@@ -1,87 +0,0 @@
|
||||
# Structure & Configuration Fixes (2026-03-01)
|
||||
|
||||
## Summary
|
||||
Fixed the double `hardware/firmware` directory duplication and locked `cockpit.sh` to use ONLY PlatformIO CLI.
|
||||
|
||||
## Changes Made
|
||||
|
||||
### 1. Directory Structure Correction
|
||||
**Problem:** Double nesting `hardware/firmware/hardware/firmware/` contained actual sources
|
||||
**Solution:**
|
||||
- Moved `esp32_audio/`, `ui_freenove_allinone/`, `ui/` to correct location
|
||||
- Removed empty `hardware/firmware/hardware/` directory
|
||||
- New clean structure: `hardware/firmware/{esp32_audio/, ui_freenove_allinone/, ui/, lib/, ...}`
|
||||
|
||||
### 2. platformio.ini Fixes
|
||||
**Problem:** Paths referenced `hardware/firmware` double, wrong `libs` path
|
||||
**Changes:**
|
||||
```ini
|
||||
# Before:
|
||||
src_dir = hardware/firmware
|
||||
build_flags = -Ihardware/firmware/esp32_audio/src -I$PROJECT_DIR/hardware/libs/story
|
||||
|
||||
# After:
|
||||
src_dir = .
|
||||
build_flags = -I./esp32_audio/src -I$PROJECT_DIR/lib/story
|
||||
```
|
||||
|
||||
**All corrections:**
|
||||
- `src_dir`: `hardware/firmware` → `.` (current directory only)
|
||||
- Include paths: Removed all `hardware/firmware` double-references
|
||||
- Library paths: `libs/` → `lib/` (correct singular form)
|
||||
- Relative includes: `./lib/story` instead of complex relative paths
|
||||
|
||||
### 3. cockpit.sh Locked to PIO-Only
|
||||
**Problem:** Complex script with hardware detection, artifact management, build orchestration
|
||||
**Solution:** Rewritten as pure PlatformIO CLI wrapper
|
||||
|
||||
**Removed:**
|
||||
- ❌ Hardware auto-detection
|
||||
- ❌ build_all.sh integration
|
||||
- ❌ Artifact collection/logging
|
||||
- ❌ Complex FX/graphics verification
|
||||
- ❌ 600+ lines of legacy code
|
||||
|
||||
**Added:**
|
||||
- ✅ Direct pio run / pio run -t upload / pio device monitor
|
||||
- ✅ Environment listing (envs) and port detection (ports)
|
||||
- ✅ Locked contract: "ONLY uses PIO (PlatformIO CLI)"
|
||||
- ✅ Clean 113-line script with help, examples, clear contract
|
||||
|
||||
**New usage:**
|
||||
```bash
|
||||
./tools/dev/cockpit.sh build [env] # pio run -e $env
|
||||
./tools/dev/cockpit.sh flash [env] # pio run -e $env -t upload
|
||||
./tools/dev/cockpit.sh monitor [env] # pio device monitor -e $env
|
||||
./tools/dev/cockpit.sh go [env] # Build + Flash + Monitor combo
|
||||
./tools/dev/cockpit.sh envs # List available environments
|
||||
./tools/dev/cockpit.sh ports # List connected serial ports
|
||||
./tools/dev/cockpit.sh help # Show help with contract
|
||||
```
|
||||
|
||||
## Files Modified
|
||||
1. `platformio.ini` - Fixed all path references
|
||||
2. `tools/dev/cockpit.sh` - Rewrote to locked PIO-only contract
|
||||
|
||||
## Verification
|
||||
```bash
|
||||
# Verify directory structure
|
||||
ls -1d hardware/firmware/{esp32_audio,ui_freenove_allinone,ui,lib}
|
||||
# Output: All 4 dirs present ✓
|
||||
|
||||
# Verify cockpit.sh works
|
||||
./tools/dev/cockpit.sh envs
|
||||
# Output: Lists 9+ environments ✓
|
||||
|
||||
# Verify no more hardware/firmware double-refs
|
||||
grep -c 'hardware/firmware' platformio.ini
|
||||
# Output: 0 ✓
|
||||
```
|
||||
|
||||
## Notes
|
||||
- Build may show missing headers (e.g., `boot_protocol_runtime.h`) - this is a separate issue in source files, not related to the path corrections
|
||||
- cockpit.sh now has a strict locked contract: zero custom logic, zero shell complexity
|
||||
- All future cockpit.sh changes must use ONLY `pio` commands, no custom detection/scripts
|
||||
|
||||
## Rollback (if needed)
|
||||
Original files backed up in git; use `git checkout hardware/firmware/platformio.ini tools/dev/cockpit.sh` to revert.
|
||||
@@ -1,91 +1,96 @@
|
||||
# Générer un scénario STORY V2 (firmware ESP32)
|
||||
# Générer un scénario STORY V2
|
||||
|
||||
Ce guide explique **comment créer/modifier un scénario** pour le firmware ESP32 sans toucher au moteur C++.
|
||||
Ce guide explique **comment créer, compiler et tester un scénario** avec le Runtime 3 sans toucher au moteur C++.
|
||||
|
||||
## Prérequis
|
||||
|
||||
Depuis ce dossier:
|
||||
- Python 3.x avec PyYAML installé (`pip install pyyaml`)
|
||||
- Le dépôt cloné à la racine du projet
|
||||
|
||||
```bash
|
||||
cd hardware/firmware/esp32_audio
|
||||
## 1) Source canonique du scénario
|
||||
|
||||
Le fichier de référence est :
|
||||
|
||||
```
|
||||
game/scenarios/zacus_v2.yaml
|
||||
```
|
||||
|
||||
## 1) Créer (ou dupliquer) un scénario YAML
|
||||
C'est la **single source of truth**. Toute modification de scénario doit passer par ce fichier YAML.
|
||||
|
||||
Point de départ recommandé:
|
||||
## 2) Édition visuelle (studio Blockly)
|
||||
|
||||
- template: `docs/protocols/story_specs/templates/scenario.template.yaml`
|
||||
- exemple existant: `docs/protocols/story_specs/scenarios/default_unlock_win_etape2.yaml`
|
||||
- schéma de référence: `docs/protocols/story_specs/schema/story_spec_v1.yaml`
|
||||
Pour éditer le scénario graphiquement, utiliser le studio React + Blockly :
|
||||
|
||||
Crée un nouveau fichier dans `docs/protocols/story_specs/scenarios/`, par exemple:
|
||||
```bash
|
||||
cd frontend-scratch-v2
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
`docs/protocols/story_specs/scenarios/mon_scenario.yaml`
|
||||
|
||||
Exemple prêt à l'emploi dans le repo:
|
||||
|
||||
- `docs/protocols/story_specs/scenarios/example_unlock_express.yaml`
|
||||
- `docs/protocols/story_specs/scenarios/example_unlock_express_done.yaml`
|
||||
|
||||
## 2) Définir la structure minimale
|
||||
|
||||
Dans le YAML, renseigne au minimum:
|
||||
|
||||
- `id` (identifiant unique)
|
||||
- `version` (utiliser la version supportée par le générateur; dans ce repo les exemples sont en `2`)
|
||||
- `initial_step`
|
||||
- `app_bindings`
|
||||
- `steps`
|
||||
- `transitions`
|
||||
|
||||
Règle pratique:
|
||||
- chaque `target_step_id` doit pointer vers une étape existante,
|
||||
- les IDs doivent être uniques,
|
||||
- les transitions doivent permettre d’atteindre une fin logique (`DONE` ou équivalent).
|
||||
Le studio permet de manipuler les étapes, transitions et événements visuellement, puis d'exporter vers le format YAML V2.
|
||||
|
||||
## 3) Valider le scénario
|
||||
|
||||
```bash
|
||||
make story-validate
|
||||
make scenario-validate
|
||||
```
|
||||
|
||||
Cette commande vérifie:
|
||||
Cette commande vérifie :
|
||||
|
||||
- la conformité de structure,
|
||||
- les IDs / références,
|
||||
- la conformité de structure au schéma V2,
|
||||
- les IDs / références croisées,
|
||||
- la cohérence des transitions,
|
||||
- les bindings des mini-apps.
|
||||
|
||||
Corrige les erreurs jusqu’à obtenir une validation propre.
|
||||
Corriger les erreurs jusqu'à obtenir une validation propre.
|
||||
|
||||
## 4) Générer le code C++
|
||||
## 4) Compiler avec le Runtime 3
|
||||
|
||||
```bash
|
||||
make story-gen
|
||||
make runtime3-compile
|
||||
```
|
||||
|
||||
Le générateur produit les fichiers utilisés par le runtime V2:
|
||||
|
||||
- `src/story/generated/scenarios_gen.h`
|
||||
- `src/story/generated/scenarios_gen.cpp`
|
||||
- `src/story/generated/apps_gen.h`
|
||||
- `src/story/generated/apps_gen.cpp`
|
||||
|
||||
## 5) Compiler et flasher
|
||||
Cela exécute `tools/scenario/compile_runtime3.py` sur le scénario par défaut. Pour spécifier un fichier :
|
||||
|
||||
```bash
|
||||
make runtime3-compile SCENARIO=game/scenarios/zacus_v2.yaml
|
||||
```
|
||||
|
||||
## 5) Simuler le scénario
|
||||
|
||||
```bash
|
||||
make runtime3-simulate
|
||||
```
|
||||
|
||||
Cela exécute `tools/scenario/simulate_runtime3.py` et déroule le scénario hors firmware pour vérifier les transitions et les états finaux.
|
||||
|
||||
## 6) Vérifier les pivots et générer le bundle firmware
|
||||
|
||||
```bash
|
||||
make runtime3-verify
|
||||
make runtime3-firmware-bundle
|
||||
```
|
||||
|
||||
- `runtime3-verify` valide les points pivots du graphe de scénario.
|
||||
- `runtime3-firmware-bundle` produit le bundle prêt à flasher sur l'ESP32.
|
||||
|
||||
## 7) Lancer les tests unitaires Runtime 3
|
||||
|
||||
```bash
|
||||
make runtime3-test
|
||||
```
|
||||
|
||||
## 8) Compiler et flasher le firmware
|
||||
|
||||
```bash
|
||||
cd ESP32_ZACUS
|
||||
pio run -e esp32dev
|
||||
```
|
||||
|
||||
Puis upload selon ton port série:
|
||||
|
||||
```bash
|
||||
pio run -e esp32dev -t upload --upload-port /dev/ttyUSB0
|
||||
```
|
||||
|
||||
## 6) Tester côté série
|
||||
## 9) Tester côté série
|
||||
|
||||
Dans le moniteur série, commandes utiles:
|
||||
Dans le moniteur série, commandes utiles :
|
||||
|
||||
- `STORY_V2_ENABLE ON`
|
||||
- `STORY_V2_LIST`
|
||||
@@ -97,15 +102,27 @@ Dans le moniteur série, commandes utiles:
|
||||
## Workflow court (copier-coller)
|
||||
|
||||
```bash
|
||||
cd hardware/firmware/esp32_audio
|
||||
make story-validate
|
||||
make story-gen
|
||||
pio run -e esp32dev
|
||||
make scenario-validate
|
||||
make runtime3-compile
|
||||
make runtime3-simulate
|
||||
make runtime3-verify
|
||||
```
|
||||
|
||||
## Toutes les cibles Makefile disponibles
|
||||
|
||||
| Cible | Description |
|
||||
|---|---|
|
||||
| `scenario-validate` | Validation YAML du scénario |
|
||||
| `runtime3-compile` | Compilation Runtime 3 |
|
||||
| `runtime3-simulate` | Simulation hors firmware |
|
||||
| `runtime3-verify` | Vérification des pivots |
|
||||
| `runtime3-test` | Tests unitaires Runtime 3 |
|
||||
| `runtime3-firmware-bundle` | Export bundle firmware ESP32 |
|
||||
|
||||
## Conseils pour éviter les blocages
|
||||
|
||||
- Commence simple: 3 à 4 étapes max au début.
|
||||
- N’ajoute qu’un seul nouveau type d’événement à la fois.
|
||||
- Teste chaque transition clé au moniteur série (`STORY_V2_EVENT ...`).
|
||||
- Garde un scénario « de secours » valide dans `docs/protocols/story_specs/scenarios/`.
|
||||
- Commencer simple : 3 à 4 étapes max au début.
|
||||
- N'ajouter qu'un seul nouveau type d'événement à la fois.
|
||||
- Toujours valider (`make scenario-validate`) avant de compiler.
|
||||
- Tester chaque transition clé au moniteur série (`STORY_V2_EVENT ...`).
|
||||
- Utiliser `make runtime3-simulate` pour déboguer sans flasher.
|
||||
|
||||
+77
-61
@@ -1,82 +1,98 @@
|
||||
# Quickstart
|
||||
|
||||
## En un coup d’œil
|
||||
- Source de vérité : `game/scenarios/zacus_v2.yaml` (dérive tout le reste : kit MJ, audio, printables).
|
||||
- Public : 6 à 14 enfants, 105 minutes (45 + 60).
|
||||
- Matériel : `kit-maitre-du-jeu/`, `printables/`, `game/scenarios/zacus_v2.yaml`, `audio/manifests/zacus_v2_audio.yaml`, `hardware/firmware/` (ESP32 + écran tactile).
|
||||
- Licences : contenus créatifs CC BY-NC 4.0 (`LICENSES/CC-BY-NC-4.0.txt`), code/script MIT (`LICENSES/MIT.txt`).
|
||||
|
||||
## Installer
|
||||
1. Copie les imprimables depuis `printables/export/pdf/` ou génère-les via `printables/manifests/zacus_v2_printables.yaml` + `printables/src/prompts/`.
|
||||
2. Prépare un lecteur audio avec `audio/manifests/zacus_v2_audio.yaml` et les fichiers de `game/prompts/audio/`.
|
||||
3. Positionne les stations selon `kit-maitre-du-jeu/plan-stations-et-mise-en-place.md`, prépare les rôles (`distribution-des-roles.md`) et l’accueil rapide (`script-minute-par-minute.md`).
|
||||
4. Flash et configure l’ESP32 en suivant `hardware/firmware/README.md` pour que l’interface réagisse aux stations imprimées.
|
||||
|
||||
## Préparer l’électronique
|
||||
|
||||
1. Installe PlatformIO (`pip install -U platformio`) et branche l’ESP32 décrit dans `hardware/firmware/README.md`, écran + alim.
|
||||
2. Compile/flash avec un profil supporté (`pio run -e esp32dev -t upload`), puis charge les assets `pio run -e esp32dev -t uploadfs` si nécessaire.
|
||||
3. Vérifie que l’écran affiche le scénario (`python3 tools/scenario/validate_scenario.py ...`), la connectique audio/led est prête, et que l’ESP32 reste branché pendant toute la partie.
|
||||
|
||||
## Déroulé express
|
||||
1. Accueil + immersion (0-10 min) : boucle audio d’`intro.md`, attribution des rôles, mise au courant sur les règles anti-chaos.
|
||||
2. Support d’enquête (10-80 min) : station par station, fiche d’enquête, audio `incident.md`, indices printables.
|
||||
3. Synthèse & final (80-105 min) : audio `accusation.md`, accusation finale, audio `solution.md`, lecture `solution-complete.md`.
|
||||
|
||||
## Outils de validation
|
||||
- Scénario : `python3 tools/scenario/validate_scenario.py game/scenarios/zacus_v2.yaml`
|
||||
- Audio : `python3 tools/audio/validate_manifest.py audio/manifests/zacus_v2_audio.yaml`
|
||||
- Printables : `python3 tools/printables/validate_manifest.py printables/manifests/zacus_v2_printables.yaml`
|
||||
- Export Markdown : `python3 tools/scenario/export_md.py game/scenarios/zacus_v2.yaml`
|
||||
|
||||
## Frontend V2
|
||||
- Frontend canon : `frontend-scratch-v2/`.
|
||||
- Démarrage :
|
||||
- `cd frontend-scratch-v2`
|
||||
- `npm install`
|
||||
- `VITE_STORY_API_BASE=http://<esp_ip>:8080 npm run dev`
|
||||
|
||||
## Outils test firmware
|
||||
|
||||
Entrée rapide:
|
||||
## Vue canonique
|
||||
- Source de vérité: `game/scenarios/zacus_v2.yaml`
|
||||
- Studio auteur: `frontend-scratch-v2/`
|
||||
- Runtime portable: Zacus Runtime 3
|
||||
- Cible terrain principale: `hardware/firmware` sur `freenove_esp32s3`
|
||||
- Shell d'orchestration: `tools/dev/zacus.sh`
|
||||
|
||||
## Valider le contenu
|
||||
```bash
|
||||
python3 tools/test/zacus_menu.py
|
||||
bash tools/setup/install_validators.sh
|
||||
bash tools/test/run_content_checks.sh --install-missing
|
||||
```
|
||||
|
||||
Codex CLI intégré : `./tools/dev/zacus.sh codex --prompt tools/dev/codex_prompts/zacus_overhaul_one_shot.md`
|
||||
Préflight USB ZeroClaw : `./tools/dev/zacus.sh zeroclaw-preflight --require-port`
|
||||
|
||||
Checks contenu (sans hardware):
|
||||
Checks inclus:
|
||||
- validation scénario
|
||||
- compilation Runtime 3
|
||||
- simulation Runtime 3
|
||||
- validation runtime bundle
|
||||
- validation audio
|
||||
- validation printables
|
||||
- export Markdown
|
||||
|
||||
## Compiler le runtime portable
|
||||
```bash
|
||||
bash tools/test/run_content_checks.sh
|
||||
python3 tools/scenario/compile_runtime3.py game/scenarios/zacus_v2.yaml
|
||||
python3 tools/scenario/simulate_runtime3.py game/scenarios/zacus_v2.yaml
|
||||
python3 tools/scenario/verify_runtime3_pivots.py game/scenarios/zacus_v2.yaml
|
||||
python3 -m unittest discover -s tests/runtime3 -p 'test_*.py'
|
||||
python3 tools/scenario/export_runtime3_firmware_bundle.py game/scenarios/zacus_v2.yaml
|
||||
```
|
||||
|
||||
Suites série disponibles:
|
||||
|
||||
Ou via le shell canonique:
|
||||
```bash
|
||||
python3 tools/test/run_serial_suite.py --list-suites
|
||||
./tools/dev/zacus.sh runtime3-compile
|
||||
./tools/dev/zacus.sh runtime3-simulate
|
||||
./tools/dev/zacus.sh runtime3-verify
|
||||
./tools/dev/zacus.sh runtime3-test
|
||||
./tools/dev/zacus.sh frontend-test
|
||||
```
|
||||
|
||||
Mode laptop/CI (sans carte branchée):
|
||||
Les artefacts sont écrits sous `artifacts/runtime3/<timestamp>/`.
|
||||
Le bundle firmware canonique est écrit dans `hardware/firmware/data/story/runtime3/DEFAULT.json`.
|
||||
|
||||
## Démarrer le studio React + Blockly
|
||||
```bash
|
||||
python3 tools/test/run_serial_suite.py --suite smoke_plus --allow-no-hardware
|
||||
python3 tools/test/zacus_menu.py --action smoke --allow-no-hardware
|
||||
cd frontend-scratch-v2
|
||||
npm install
|
||||
npm test
|
||||
npm run lint
|
||||
VITE_STORY_API_BASE=http://<esp_ip>:8080 npm run dev
|
||||
```
|
||||
|
||||
Pour résoudre les ports sans matériel, la commande `./tools/dev/zacus.sh ports` écrit le JSON contractuel dans `artifacts/ports/<timestamp>/ports_resolve.json` et met à jour `artifacts/ports/latest_ports_resolve.json`. Pour mocker les CP2102 utilisez `ZACUS_MOCK_PORTS=1 ZACUS_PORTS_FIXTURE=tools/test/fixtures/ports_list_macos.txt ./tools/dev/zacus.sh ports`.
|
||||
Pour un build local:
|
||||
```bash
|
||||
./tools/dev/zacus.sh frontend-build
|
||||
```
|
||||
|
||||
Codex CLI intégré : `./tools/dev/zacus.sh codex --prompt tools/dev/codex_prompts/zacus_overhaul_one_shot.md`. Le script note aussi l’état des ports résolus et place les logs dans `artifacts/codex/<timestamp>/`.
|
||||
## Générer la documentation
|
||||
```bash
|
||||
python3 -m pip install --user --break-system-packages -r tools/requirements/docs.txt
|
||||
python3 -m mkdocs build --strict
|
||||
```
|
||||
|
||||
Guide orchestration ZeroClaw: `docs/zeroclaw_orchestration.md`
|
||||
Ou:
|
||||
```bash
|
||||
./tools/dev/zacus.sh docs-build
|
||||
```
|
||||
|
||||
Dépendances optionnelles:
|
||||
- `pip install pyyaml` pour `run_content_checks.sh`
|
||||
- `pip install pyserial` pour suites USB, UI Link sim et console série
|
||||
## Valider le firmware
|
||||
```bash
|
||||
cd hardware/firmware
|
||||
pio run -e freenove_esp32s3
|
||||
pio run -e esp8266_oled
|
||||
```
|
||||
|
||||
## Pour aller plus loin
|
||||
- Crée une variante : duplique le YAML, modifie `canon` et `solution` et revalide avec le script.
|
||||
- Ajoute des cartes imprimables en t’inspirant des prompts dans `printables/src/prompts/`.
|
||||
- Mets à jour ce quickstart via un PR si le montage change (p. ex. nouvelle station ou nouvelle plage horaire).
|
||||
Chemin mono-carte recommandé:
|
||||
```bash
|
||||
cd hardware/firmware
|
||||
ZACUS_ENV="freenove_esp32s3" ./tools/dev/run_matrix_and_smoke.sh
|
||||
```
|
||||
|
||||
## Utiliser le shell Zacus
|
||||
```bash
|
||||
./tools/dev/zacus.sh content-checks
|
||||
./tools/dev/zacus.sh ports
|
||||
./tools/dev/zacus.sh artifacts-summary
|
||||
./tools/dev/zacus.sh menu
|
||||
```
|
||||
|
||||
Le menu utilise `gum` si disponible et repasse en mode texte sinon.
|
||||
|
||||
## Où regarder ensuite
|
||||
- `docs/architecture/index.md`
|
||||
- `plans/master-plan.md`
|
||||
- `todos/master.md`
|
||||
- `hardware/firmware/docs/AGENT_TODO.md`
|
||||
|
||||
@@ -1,106 +0,0 @@
|
||||
# RC AutoFix (Scheduled) Workflow
|
||||
|
||||
This document describes the GitHub Actions workflow at .github/workflows/rc-autofix-cicd.yml.
|
||||
|
||||
## Trigger Manually (GitHub UI)
|
||||
|
||||
1. Go to Actions tab
|
||||
2. Select "RC AutoFix (Scheduled)"
|
||||
3. Click "Run workflow"
|
||||
|
||||
## Trigger via CLI
|
||||
|
||||
```bash
|
||||
gh workflow run rc-autofix-cicd.yml
|
||||
```
|
||||
|
||||
## Automatic Scheduled Run
|
||||
|
||||
- Runs daily at 2 AM UTC (adjust cron expression as needed)
|
||||
- Can be modified in workflow file under schedule.cron
|
||||
|
||||
## Environment Variables Explained
|
||||
|
||||
| Variable | Value | Purpose |
|
||||
|----------|-------|---------|
|
||||
| ZACUS_GIT_AUTOCOMMIT | 1 | Enable automatic git commits after Codex fixes |
|
||||
| ZACUS_GIT_ALLOW_WRITE | 1 | Allow git write operations (required for commits) |
|
||||
| ZACUS_GIT_NO_CONFIRM | 1 | Skip confirmation prompts (required for automation) |
|
||||
| ZACUS_REQUIRE_HW | 0 | Do not require physical hardware (simulation mode) |
|
||||
| ZACUS_SKIP_UPLOAD | 1 | Skip ESP32/ESP8266 upload (stability + speed) |
|
||||
|
||||
## Workflow Steps
|
||||
|
||||
1. Checkout code from current branch
|
||||
2. Setup Python and PlatformIO
|
||||
3. Bootstrap Zacus environment
|
||||
4. Configure Git user for commits
|
||||
5. Run RC AutoFix with auto-commit enabled
|
||||
6. Push changes if any commits were made
|
||||
7. Upload artifacts for review (30-day retention)
|
||||
8. Notify on failure (optional: configure GitHub issue comment)
|
||||
|
||||
## Tracking Results
|
||||
|
||||
### Via GitHub Actions UI
|
||||
|
||||
- Check "Run workflow" and the latest run
|
||||
- View step logs for detailed output
|
||||
- Download artifacts (rc_autofix.log, codex_last_message.md, etc.)
|
||||
|
||||
### Via Git History
|
||||
|
||||
```bash
|
||||
git log --oneline | grep "Auto-fix:" # Show all auto-fix commits
|
||||
git show HEAD # Review latest commit
|
||||
```
|
||||
|
||||
## Failure Modes
|
||||
|
||||
| Failure | Action |
|
||||
|---------|--------|
|
||||
| RC pre-condition fails | Workflow continues (will fail unless fix applied) |
|
||||
| Codex execution fails | Check logs in artifact/codex_last_message.md |
|
||||
| Git commit fails | Check git config and repository permissions |
|
||||
| Push fails | Check branch protection rules and credentials |
|
||||
|
||||
## Customization
|
||||
|
||||
### Change Schedule
|
||||
|
||||
Edit the cron expression:
|
||||
|
||||
```yaml
|
||||
schedule:
|
||||
- cron: '0 2 * * 1-5' # Mon-Fri at 2 AM UTC
|
||||
- cron: '0 6 * * *' # Daily at 6 AM UTC
|
||||
```
|
||||
|
||||
### Disable Auto-Commit
|
||||
|
||||
Set ZACUS_GIT_AUTOCOMMIT: '0' (requires manual commit review).
|
||||
|
||||
### Add Hardware Testing
|
||||
|
||||
Set ZACUS_REQUIRE_HW: '1' (requires physical ESP32/ESP8266 and a CI runner with USB).
|
||||
|
||||
### Add Slack Notification
|
||||
|
||||
```yaml
|
||||
- name: Notify Slack on failure
|
||||
if: failure()
|
||||
uses: slackapi/slack-github-action@v1.24.0
|
||||
with:
|
||||
webhook-url: ${{ secrets.SLACK_WEBHOOK }}
|
||||
payload: |
|
||||
{
|
||||
"text": "RC AutoFix failed - check artifacts"
|
||||
}
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- Zacus CLI: hardware/firmware/tools/dev/zacus.sh
|
||||
- RC Live Codex Status: hardware/firmware/docs/RC_LIVE_CODEX_STATUS.md
|
||||
- Codex Prompt Maintenance: hardware/firmware/docs/CODEX_PROMPT_MAINTENANCE.md
|
||||
- Agent Briefing: .github/agents/AGENT_BRIEFINGS.md
|
||||
@@ -1,11 +0,0 @@
|
||||
<!-- REPO_STATE:v1 -->
|
||||
Repo: le-mystere-professeur-zacus
|
||||
Branch: codex/repo-state-zacus
|
||||
HEAD: 3fec631c43c66a6ae12c12d2f652c8800d22b28f
|
||||
HeadDate: 2026-02-21T02:59:01+01:00
|
||||
HeadSubject: feat(freenove): stabilize fallback AP outside local wifi (#103)
|
||||
RepoURL: https://github.com/electron-rare/le-mystere-professeur-zacus.git
|
||||
ProjectKind: hardware_firmware_hybrid
|
||||
PivotChanges: [{"path": "hardware/firmware/data/story/apps/APP_WIFI.json", "tags": ["firmware_build_test", "hardware_validation"]}, {"path": "hardware/firmware/docs/AGENT_TODO.md", "tags": ["firmware_build_test", "hardware_validation"]}, {"path": "hardware/firmware/hardware/firmware/ui_freenove_allinone/include/network_manager.h", "tags": ["firmware_build_test", "hardware_validation"]}, {"path": "hardware/firmware/hardware/firmware/ui_freenove_allinone/src/main.cpp", "tags": ["firmware_build_test", "hardware_validation"]}, {"path": "hardware/firmware/hardware/firmware/ui_freenove_allinone/src/network_manager.cpp", "tags": ["firmware_build_test", "hardware_validation"]}]
|
||||
ImpactGates: firmware_build_test, hardware_validation
|
||||
GeneratedAtUTC: 2026-02-21T02:23:01Z
|
||||
@@ -0,0 +1,319 @@
|
||||
# Security Specification
|
||||
|
||||
## Status
|
||||
- State: draft
|
||||
- Date: 2026-03-21
|
||||
- Supersedes: previous 4-line stub
|
||||
- Related: `specs/AI_INTEGRATION_SPEC.md`, `specs/MCP_HARDWARE_SERVER_SPEC.md`
|
||||
|
||||
## 1) Threat Model
|
||||
|
||||
### 1.1 Deployment Context
|
||||
|
||||
The Zacus escape room system operates on a **local WiFi network** in a controlled physical space. It is not exposed to the public internet. The primary attack surface is:
|
||||
|
||||
| Vector | Risk | Likelihood |
|
||||
|--------|------|-----------|
|
||||
| Local network access (same WiFi) | Players or visitors on the game WiFi | HIGH |
|
||||
| Physical access to ESP32 device | Players can touch/see the hardware | HIGH |
|
||||
| Firmware extraction (USB/JTAG) | Sophisticated attacker with tools | LOW |
|
||||
| Server-side (mascarade VM) | Only reachable via LAN/Tailscale | LOW |
|
||||
| Supply chain (dependencies) | Compromised PlatformIO/npm package | VERY LOW |
|
||||
|
||||
### 1.2 Assets to Protect
|
||||
|
||||
| Asset | Sensitivity | Impact if Compromised |
|
||||
|-------|------------|----------------------|
|
||||
| WiFi credentials | HIGH | Network access, lateral movement |
|
||||
| API bearer tokens | HIGH | Full device control |
|
||||
| Game scenario content | MEDIUM | Spoilers, cheating |
|
||||
| Player interaction data | LOW | Privacy (no PII collected) |
|
||||
| Firmware binary | LOW | IP, reverse engineering |
|
||||
| mascarade API key | HIGH | Full orchestration control |
|
||||
|
||||
### 1.3 Threat Actors
|
||||
|
||||
1. **Curious player**: Tries to cheat by accessing the API from their phone
|
||||
2. **Tech-savvy visitor**: Scans the network, finds ESP32 endpoints
|
||||
3. **Malicious local user**: Attempts injection, DoS, or firmware dump
|
||||
4. **Remote attacker**: Not applicable (no internet exposure)
|
||||
|
||||
## 2) API Authentication
|
||||
|
||||
### 2.1 Bearer Token Scheme
|
||||
|
||||
All ESP32 HTTP API endpoints require a Bearer token:
|
||||
|
||||
```
|
||||
GET /api/status
|
||||
Authorization: Bearer <token>
|
||||
|
||||
Response 200: { "step": "STEP_U_SON_PROTO", ... }
|
||||
Response 401: { "error": "unauthorized" }
|
||||
```
|
||||
|
||||
### 2.2 Token Lifecycle
|
||||
|
||||
| Phase | Mechanism |
|
||||
|-------|-----------|
|
||||
| Generation | Random 48-char hex string (cryptographically secure) |
|
||||
| Storage (ESP32) | NVS encrypted partition |
|
||||
| Storage (server) | `.env` file, not committed to git |
|
||||
| Provisioning | QR code scanned at setup (see 3.2) |
|
||||
| Rotation | Manual, at each deployment |
|
||||
| Revocation | Reflash NVS or change `.env` |
|
||||
|
||||
### 2.3 Endpoint Protection Matrix
|
||||
|
||||
| Endpoint | Auth Required | Rate Limit | Notes |
|
||||
|----------|:------------:|:----------:|-------|
|
||||
| `GET /api/status` | Yes | 10/s | Device health |
|
||||
| `GET /api/scenario` | Yes | 5/s | Current state |
|
||||
| `POST /api/scenario/transition` | Yes | 2/s | State change |
|
||||
| `POST /api/audio` | Yes | 5/s | Audio control |
|
||||
| `POST /api/led` | Yes | 10/s | LED control |
|
||||
| `GET /api/camera` | Yes | 2/s | Snapshot |
|
||||
| `POST /api/puzzle` | Yes | 2/s | Puzzle state |
|
||||
| `GET /` | No | 20/s | Web UI (static) |
|
||||
| `GET /health` | No | 20/s | Health check |
|
||||
| `WS /ws` | Yes (on connect) | — | Event stream |
|
||||
|
||||
### 2.4 Implementation (Firmware)
|
||||
|
||||
```cpp
|
||||
// Auth middleware for AsyncWebServer
|
||||
bool checkAuth(AsyncWebServerRequest *request) {
|
||||
if (!request->hasHeader("Authorization")) {
|
||||
request->send(401, "application/json", "{\"error\":\"unauthorized\"}");
|
||||
return false;
|
||||
}
|
||||
String authHeader = request->header("Authorization");
|
||||
if (!authHeader.startsWith("Bearer ")) {
|
||||
request->send(401, "application/json", "{\"error\":\"unauthorized\"}");
|
||||
return false;
|
||||
}
|
||||
String token = authHeader.substring(7);
|
||||
String storedToken = nvs_get_string("auth", "api_token");
|
||||
if (token != storedToken) {
|
||||
request->send(401, "application/json", "{\"error\":\"unauthorized\"}");
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
```
|
||||
|
||||
## 3) WiFi Credential Management
|
||||
|
||||
### 3.1 Current State (CRITICAL — FW-01)
|
||||
|
||||
WiFi credentials are currently **hardcoded** in `storage_manager.cpp:73`. This must be remediated immediately.
|
||||
|
||||
### 3.2 Target: NVS + QR Provisioning
|
||||
|
||||
**Storage**: ESP32 NVS (Non-Volatile Storage) encrypted partition.
|
||||
|
||||
**Provisioning flow**:
|
||||
1. First boot: ESP32 starts in AP mode (`Zacus-Setup`)
|
||||
2. Game master connects and opens captive portal
|
||||
3. Game master scans QR code containing WiFi config:
|
||||
```
|
||||
WIFI:T:WPA;S:EscapeRoom-Net;P:s3cur3p4ss;;
|
||||
```
|
||||
4. ESP32 stores credentials in NVS, restarts in STA mode
|
||||
5. On connection failure: fallback to AP mode after 30 s
|
||||
|
||||
**NVS keys**:
|
||||
| Key | Namespace | Encrypted |
|
||||
|-----|-----------|:---------:|
|
||||
| `wifi_ssid` | `network` | Yes |
|
||||
| `wifi_pass` | `network` | Yes |
|
||||
| `api_token` | `auth` | Yes |
|
||||
| `device_id` | `device` | No |
|
||||
| `mcp_token` | `auth` | Yes |
|
||||
|
||||
### 3.3 NVS Encryption
|
||||
|
||||
Enable NVS encryption with a flash encryption key:
|
||||
```ini
|
||||
# platformio.ini
|
||||
board_build.partitions = partitions_encrypted.csv
|
||||
board_build.flash_mode = dio
|
||||
board_build.esp-idf.sdkconfig =
|
||||
CONFIG_NVS_ENCRYPTION=y
|
||||
CONFIG_SECURE_FLASH_ENC_ENABLED=y
|
||||
```
|
||||
|
||||
## 4) Input Validation
|
||||
|
||||
### 4.1 Requirements
|
||||
|
||||
All API inputs must be validated before processing:
|
||||
|
||||
| Input | Validation |
|
||||
|-------|-----------|
|
||||
| JSON body | Max size 4 KB, max depth 5 levels |
|
||||
| `step_id` | Alphanumeric + underscore, max 64 chars, must exist in scenario |
|
||||
| `event_name` | Alphanumeric + underscore, max 64 chars |
|
||||
| `audio_path` | Must start with `/audio/`, no `..`, max 128 chars |
|
||||
| `color` | Regex `^#[0-9a-fA-F]{6}$` or named color enum |
|
||||
| `volume` | Integer 0-100 |
|
||||
| `brightness` | Integer 0-255 |
|
||||
| `device_id` | Alphanumeric + hyphen, max 32 chars |
|
||||
|
||||
### 4.2 JSON Parsing Safety
|
||||
|
||||
```cpp
|
||||
// Safe JSON parsing with limits
|
||||
StaticJsonDocument<4096> doc; // 4 KB max
|
||||
DeserializationError err = deserializeJson(doc, body, DeserializationOption::NestingLimit(5));
|
||||
if (err) {
|
||||
request->send(400, "application/json", "{\"error\":\"invalid_json\"}");
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
### 4.3 String Safety
|
||||
|
||||
- All string operations use length-bounded functions (`strncpy`, `snprintf`)
|
||||
- No `sprintf` or unbounded `strcpy` in firmware code
|
||||
- LVGL text buffers use fixed-size arrays with null termination
|
||||
|
||||
## 5) Rate Limiting
|
||||
|
||||
### 5.1 Token Bucket Algorithm
|
||||
|
||||
Each endpoint has a token bucket rate limiter:
|
||||
|
||||
```cpp
|
||||
struct RateLimiter {
|
||||
uint32_t tokens;
|
||||
uint32_t max_tokens;
|
||||
uint32_t refill_rate_ms; // ms per token refill
|
||||
uint32_t last_refill;
|
||||
};
|
||||
```
|
||||
|
||||
### 5.2 Limits
|
||||
|
||||
| Category | Rate | Burst |
|
||||
|----------|------|-------|
|
||||
| Read endpoints (GET) | 10/s | 20 |
|
||||
| Write endpoints (POST) | 5/s | 10 |
|
||||
| Camera capture | 2/s | 3 |
|
||||
| State transitions | 2/s | 5 |
|
||||
| WebSocket messages | 20/s | 50 |
|
||||
| Static files | 20/s | 50 |
|
||||
|
||||
### 5.3 Response on Limit
|
||||
|
||||
```
|
||||
HTTP 429 Too Many Requests
|
||||
Retry-After: 1
|
||||
Content-Type: application/json
|
||||
|
||||
{"error": "rate_limited", "retry_after_ms": 1000}
|
||||
```
|
||||
|
||||
## 6) CORS Policy
|
||||
|
||||
### 6.1 Configuration
|
||||
|
||||
```cpp
|
||||
// CORS headers for AsyncWebServer
|
||||
server.on("/*", HTTP_OPTIONS, [](AsyncWebServerRequest *request) {
|
||||
AsyncWebServerResponse *response = request->beginResponse(204);
|
||||
response->addHeader("Access-Control-Allow-Origin", allowedOrigin);
|
||||
response->addHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS");
|
||||
response->addHeader("Access-Control-Allow-Headers", "Authorization, Content-Type");
|
||||
response->addHeader("Access-Control-Max-Age", "3600");
|
||||
request->send(response);
|
||||
});
|
||||
```
|
||||
|
||||
### 6.2 Allowed Origins
|
||||
|
||||
| Environment | Allowed Origin |
|
||||
|-------------|---------------|
|
||||
| Development | `http://localhost:5173` (Vite dev server) |
|
||||
| Production | `http://<esp32-ip>:8080` (self-hosted UI) |
|
||||
| Game master | `http://<gm-dashboard-ip>:3000` |
|
||||
|
||||
Wildcard `*` is **never** used.
|
||||
|
||||
## 7) Future Security Enhancements
|
||||
|
||||
### 7.1 Secure Boot (Phase 2)
|
||||
|
||||
Enable ESP32-S3 Secure Boot v2:
|
||||
- RSA-3072 signature verification on boot
|
||||
- Prevents unsigned firmware from running
|
||||
- Key stored in eFuse (one-time programmable)
|
||||
|
||||
```ini
|
||||
# platformio.ini (future)
|
||||
board_build.esp-idf.sdkconfig =
|
||||
CONFIG_SECURE_BOOT=y
|
||||
CONFIG_SECURE_BOOT_V2_ENABLED=y
|
||||
```
|
||||
|
||||
### 7.2 Flash Encryption (Phase 2)
|
||||
|
||||
Enable flash encryption to protect firmware and NVS:
|
||||
- AES-256-XTS encryption
|
||||
- Prevents firmware extraction via SPI flash dump
|
||||
- Development mode allows re-flashing; release mode is permanent
|
||||
|
||||
### 7.3 TLS / HTTPS (Phase 3)
|
||||
|
||||
Upgrade HTTP API to HTTPS:
|
||||
- Self-signed certificate stored in NVS
|
||||
- Certificate pinning in frontend and MCP client
|
||||
- Adds ~50 KB RAM overhead on ESP32
|
||||
|
||||
### 7.4 OTA Security (Phase 3)
|
||||
|
||||
Signed OTA updates:
|
||||
- Firmware images signed with project key
|
||||
- ESP32 verifies signature before applying update
|
||||
- Rollback on failed verification
|
||||
|
||||
## 8) Incident Response
|
||||
|
||||
### 8.1 Severity Levels
|
||||
|
||||
| Level | Description | Example |
|
||||
|-------|------------|---------|
|
||||
| SEV-1 | Game completely compromised | Token leaked, full device control by attacker |
|
||||
| SEV-2 | Security bypass discovered | Auth bypass, injection working |
|
||||
| SEV-3 | Vulnerability found, not exploited | Missing validation on one endpoint |
|
||||
| SEV-4 | Hardening improvement | Missing rate limit on low-risk endpoint |
|
||||
|
||||
### 8.2 Response Procedure
|
||||
|
||||
1. **Detect**: Monitor serial logs, API access patterns, unexpected state changes
|
||||
2. **Contain**: Power off affected devices if SEV-1/SEV-2
|
||||
3. **Assess**: Identify scope (which devices, which data)
|
||||
4. **Remediate**:
|
||||
- Rotate all tokens immediately
|
||||
- Reflash firmware with fix
|
||||
- Update NVS credentials
|
||||
5. **Review**: Post-incident analysis, update threat model
|
||||
|
||||
### 8.3 Contact
|
||||
|
||||
Security issues: open a private issue on the GitHub repository or contact the project maintainer directly. Do not disclose vulnerabilities publicly before a fix is available.
|
||||
|
||||
## 9) Security Checklist (Pre-Deployment)
|
||||
|
||||
- [ ] WiFi credentials stored in NVS, not in source code
|
||||
- [ ] API bearer token generated and provisioned
|
||||
- [ ] All API endpoints require authentication
|
||||
- [ ] Input validation on all POST endpoints
|
||||
- [ ] Rate limiting enabled
|
||||
- [ ] CORS configured with specific origins
|
||||
- [ ] No credentials in git history
|
||||
- [ ] Serial debug output disabled in production build
|
||||
- [ ] Firmware compiled with optimization (-Os)
|
||||
- [ ] NVS encryption enabled
|
||||
- [ ] Default passwords changed
|
||||
- [ ] `.env` file excluded from git (in `.gitignore`)
|
||||
+34
-189
@@ -1,196 +1,41 @@
|
||||
# Repository Structure
|
||||
|
||||
## Overview
|
||||
## Canon de refonte
|
||||
|
||||
This repository contains all materials for "Le Mystère du Professeur Zacus" - a complete investigation kit for birthday parties (6-14 children, 105 minutes), with a laboratory/scientific campus theme.
|
||||
|
||||
## Root Structure
|
||||
|
||||
```
|
||||
le-mystere-professeur-zacus/
|
||||
├── .github/ # GitHub configuration (workflows, templates)
|
||||
├── audio/ # Audio manifests and prompts
|
||||
├── docs/ # Documentation
|
||||
├── examples/ # Example usages and demonstrations
|
||||
├── game/ # Core game scenarios and rules
|
||||
├── hardware/ # Firmware for microcontrollers
|
||||
├── kit-maitre-du-jeu/ # Game master kit (scripts, solutions, setup)
|
||||
├── ops/ # Operations scripts and utilities
|
||||
├── printables/ # Printable materials (badges, cards, clues)
|
||||
├── scenario-ai-coherence/# AI-generated scenario content (human-validated)
|
||||
├── tools/ # Validation and export scripts
|
||||
└── [root config files] # LICENSE, README, CONTRIBUTING, etc.
|
||||
```mermaid
|
||||
flowchart TD
|
||||
YAML["game/scenarios/*.yaml"] --> Runtime3["tools/scenario Runtime 3"]
|
||||
YAML --> Exports["audio / printables / kit MJ"]
|
||||
Runtime3 --> Studio["frontend-scratch-v2"]
|
||||
Runtime3 --> Firmware["hardware/firmware"]
|
||||
Plans["memory + plans + todos"] --> Studio
|
||||
Plans --> Firmware
|
||||
Docs["docs/architecture + specs"] --> Plans
|
||||
```
|
||||
|
||||
## Detailed Directory Structure
|
||||
## Répertoires principaux
|
||||
- `.github/`: CI et politiques de validation.
|
||||
- `audio/`: manifestes et ressources audio.
|
||||
- `docs/`: quickstart, architecture, benchmark OSS, runbooks.
|
||||
- `frontend-scratch-v2/`: studio auteur React + Blockly et prévisualisation Runtime 3.
|
||||
- `game/`: scénarios YAML canoniques.
|
||||
- `hardware/`: firmware, scripts de test terrain et documentation matériel.
|
||||
- `kit-maitre-du-jeu/`: déroulé MJ, organisation des stations, scripts de partie.
|
||||
- `memory/`: mémoire projet transversale.
|
||||
- `plans/`: plans directeurs et plans d'agents.
|
||||
- `printables/`: manifestes et sources d'indices imprimables.
|
||||
- `specs/`: spécifications canoniques de la refonte.
|
||||
- `todos/`: backlog opérationnel de la refonte.
|
||||
- `tools/`: validateurs, compilateur/simulateur Runtime 3, shells TUI/CLI.
|
||||
|
||||
### `.github/`
|
||||
GitHub-specific configuration files:
|
||||
- **`ISSUE_TEMPLATE/`**: Issue templates for bugs and features
|
||||
- **`PULL_REQUEST_TEMPLATE.md`**: Pull request template
|
||||
- **`workflows/`**: CI/CD workflows for validation
|
||||
## Règles de structure
|
||||
- `game/scenarios/*.yaml` reste la source de vérité narrative.
|
||||
- `hardware/firmware/esp32/` est en lecture seule.
|
||||
- `frontend-scratch-v2/` est le front canonique; les chemins legacy doivent sortir du flux principal.
|
||||
- Les artefacts d'exécution restent hors git et sont référencés depuis les TODO ou rapports.
|
||||
|
||||
### `audio/`
|
||||
Audio content organization:
|
||||
- **`manifests/`**: YAML manifests describing audio requirements
|
||||
- **`game/prompts/audio/`**: Audio generation prompts (intro, incidents, solutions)
|
||||
|
||||
Validation: `python3 tools/audio/validate_manifest.py`
|
||||
|
||||
### `docs/`
|
||||
Comprehensive documentation:
|
||||
- **`BRANCHES.md`**: Git branch strategy and workflows
|
||||
- **`STRUCTURE.md`**: This file - repository architecture
|
||||
- **`QUICKSTART.md`**: Express setup guide
|
||||
- **`STYLEGUIDE.md`**: Writing style and tone guidelines
|
||||
- **`WORKFLOWS.md`**: Development workflows
|
||||
- **`GLOSSARY.md`**: Terms and definitions
|
||||
- **`index.md`**: Documentation index
|
||||
- **`repo-status.md`**: Current repository status
|
||||
- **`maintenance-repo.md`**: Maintenance procedures
|
||||
- **`repo-audit.md`**: Audit reports
|
||||
- **`_generated/`**: Auto-generated documentation
|
||||
- **`assets/`**: Documentation images and diagrams
|
||||
|
||||
### `examples/`
|
||||
Cross-functional examples and demonstrations. See `examples/README.md` for index.
|
||||
|
||||
### `game/`
|
||||
Core game content (canonical):
|
||||
- **`scenarios/`**: YAML scenario definitions (single source of truth)
|
||||
- `zacus_v2.yaml`: Canon scenario with acts, runtime steps and terrain clues
|
||||
- **`prompts/`**: Generation prompts for various content
|
||||
- **`exports/`**: Generated exports (not versioned)
|
||||
|
||||
Validation: `python3 tools/scenario/validate_scenario.py`
|
||||
|
||||
### `hardware/`
|
||||
Microcontroller firmware and hardware integration:
|
||||
- **`firmware/`**:
|
||||
- `esp32/`: ESP32 firmware (main controller, WiFi, audio, story engine)
|
||||
- `arduino/`: Arduino-compatible variants
|
||||
- `README.md`: Branch strategy and setup
|
||||
|
||||
**Note**: Firmware uses Git branches for development variants instead of separate directories:
|
||||
- `main:hardware/firmware/` → stable releases
|
||||
- `hardware/esp32-dev` → active development
|
||||
- `hardware/esp32-experimental` → R&D
|
||||
|
||||
See `hardware/firmware/README.md` for complete details.
|
||||
|
||||
### `kit-maitre-du-jeu/`
|
||||
Game Master materials:
|
||||
- Minute-by-minute script (ready-to-play)
|
||||
- Complete solution (culprit, motive, method, timeline)
|
||||
- Material checklist and 4-station setup
|
||||
- Role distribution (modular for 6-14 children)
|
||||
- Anti-chaos guide and station map
|
||||
- PDF exports
|
||||
|
||||
### `ops/`
|
||||
Operational scripts:
|
||||
- **`check_scope.sh`**: Validate branch scope compliance
|
||||
- **`codex_story_ia.sh`**: Story-IA workflow helper
|
||||
|
||||
### `printables/`
|
||||
Printable materials for the game:
|
||||
- **`src/`**: Source files and generation prompts
|
||||
- **`export/pdf/`**: Exported PDFs (locally generated, not versioned)
|
||||
- **`export/png/`**: Exported PNGs (locally generated, not versioned)
|
||||
- **`manifests/`**: YAML manifests for printables validation
|
||||
- **`WORKFLOW.md`**: Generation and export workflow
|
||||
|
||||
Validation: `python3 tools/printables/validate_manifest.py`
|
||||
|
||||
### `scenario-ai-coherence/`
|
||||
AI-generated scenario coherence artifacts (human-validated):
|
||||
- **`version-finale/`**: Validated final content
|
||||
- Future: intermediate versions, generation logs
|
||||
|
||||
This folder contains AI-generated scenario content that has been reviewed and approved by human maintainers.
|
||||
|
||||
### `tools/`
|
||||
Python validation and export scripts:
|
||||
- **`scenario/`**: Scenario validation and export
|
||||
- `validate_scenario.py`: Validate YAML scenarios
|
||||
- `export_md.py`: Generate markdown from scenarios
|
||||
- **`audio/`**: Audio manifest validation
|
||||
- `validate_manifest.py`: Validate audio manifests
|
||||
- **`printables/`**: Printables manifest validation
|
||||
- `validate_manifest.py`: Validate printables manifests
|
||||
- **`qa/`**: Quality assurance utilities (optional for firmware)
|
||||
|
||||
### Root Configuration Files
|
||||
|
||||
- **`README.md`**: Main project overview
|
||||
- **`LICENSE.md`**: Dual licensing (MIT for code, CC BY-NC 4.0 for content)
|
||||
- **`CONTRIBUTING.md`**: Contribution guidelines
|
||||
- **`CODEX_RULES.md`**: Branch scope rules for AI assistants
|
||||
- **`CHANGELOG.md`**: Version history
|
||||
- **`AGENTS.md`**: AI agent descriptions
|
||||
- **`SECURITY.md`**: Security policy
|
||||
- **`CODE_OF_CONDUCT.md`**: Community standards
|
||||
- **`DISCLAIMER.md`**: Legal disclaimer
|
||||
- **`.gitignore`**: Git ignore patterns
|
||||
- **`Makefile`**: Build and validation commands
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
game/scenarios/*.yaml (CANON)
|
||||
↓
|
||||
tools/scenario/validate_scenario.py → validation
|
||||
↓
|
||||
tools/scenario/export_md.py → docs/_generated/
|
||||
↓
|
||||
printables/src/ + audio/prompts/ → printable PDFs + audio files
|
||||
↓
|
||||
kit-maitre-du-jeu/ → complete game master package
|
||||
```
|
||||
|
||||
## Branch Strategy
|
||||
|
||||
See `BRANCHES.md` for complete branch workflow. Key branches:
|
||||
- **`main`**: Stable integration branch
|
||||
- **`hardware/firmware`**: Firmware development
|
||||
- **`generation/story-esp`**: Story engine for ESP32
|
||||
- **`generation/story-ia`**: Game content and scenarios
|
||||
- **`scripts/generation`**: Validation and export tools
|
||||
- **`documentation`**: Documentation updates
|
||||
|
||||
## Validation Workflow
|
||||
|
||||
```bash
|
||||
# Validate scenario
|
||||
python3 tools/scenario/validate_scenario.py game/scenarios/zacus_v2.yaml
|
||||
|
||||
# Export markdown
|
||||
python3 tools/scenario/export_md.py game/scenarios/zacus_v2.yaml
|
||||
|
||||
# Validate audio manifest
|
||||
python3 tools/audio/validate_manifest.py audio/manifests/zacus_v2_audio.yaml
|
||||
|
||||
# Validate printables manifest
|
||||
python3 tools/printables/validate_manifest.py printables/manifests/zacus_v2_printables.yaml
|
||||
|
||||
# Or use Makefile
|
||||
make scenario-validate
|
||||
make export
|
||||
make audio-validate
|
||||
make printables-validate
|
||||
make all-validate
|
||||
```
|
||||
|
||||
## Licenses
|
||||
|
||||
- **Code** (firmware, scripts, tools): MIT License
|
||||
- **Creative content** (documents, PDFs, assets, prompts): CC BY-NC 4.0
|
||||
- See `LICENSE.md` and `LICENSES/` for full text
|
||||
|
||||
## Contributing
|
||||
|
||||
See `CONTRIBUTING.md` for contribution guidelines and review process.
|
||||
|
||||
## Status
|
||||
|
||||
Current project status tracked in `docs/repo-status.md`.
|
||||
## Chemins de référence
|
||||
- Architecture: `docs/architecture/index.md`
|
||||
- Runtime: `specs/ZACUS_RUNTIME_3_SPEC.md`
|
||||
- Studio: `specs/STORY_DESIGNER_SCRATCH_LIKE_SPEC.md`
|
||||
- Pilotage: `plans/master-plan.md`, `todos/master.md`
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Agent Matrix
|
||||
|
||||
| Agent | Scope | Inputs | Outputs |
|
||||
| --- | --- | --- | --- |
|
||||
| PM / Architecture | Global audit, sequencing, dependency map | repo state, plans, docs | `plans/master-plan.md`, architecture maps |
|
||||
| Runtime / Compiler | Runtime 3 schema, compiler, simulator | YAML scenarios, runtime spec | IR JSON, simulator traces, migration rules |
|
||||
| Frontend Studio | React + Blockly authoring studio | Runtime 3 contract, API contract | Studio UI, previews, lint/build gates |
|
||||
| Firmware Adapter | Runtime execution on device | IR JSON, firmware APIs | buildable firmware, runtime adapter |
|
||||
| Content / Narrative | Scenario migration and feature cards | YAML canon, audio/printables manifests | consistent scenario graph, content backlog |
|
||||
| Tooling / TUI / Logs | CLI/TUI wrappers and evidence | scripts, CI, artifact dirs | `tools/dev/zacus.sh`, validation entrypoints |
|
||||
| Docs / Knowledge | Specs, Mermaid maps, READMEs | all workstreams | updated docs, MkDocs site |
|
||||
| QA / Release | Gates, proofs, release readiness | builds, logs, simulations | validation matrix, release checklist |
|
||||
|
||||
## Coordination Rule
|
||||
- `memory/project-memory.md` stores cross-wave context.
|
||||
- `plans/agents/*.md` store agent-specific execution plans.
|
||||
- `todos/*.md` store the next concrete chain of work.
|
||||
@@ -0,0 +1,235 @@
|
||||
# AI Integration Map
|
||||
|
||||
## Mindmap — AI Capabilities
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((Zacus AI))
|
||||
On-Device
|
||||
ESP-SR v2.0
|
||||
WakeNet "Hey Zacus"
|
||||
MultiNet 50 commands FR
|
||||
16-bit PCM 16kHz
|
||||
280 KB PSRAM
|
||||
ESP-DL v3.2
|
||||
YOLOv11n INT8
|
||||
8 prop classes
|
||||
5-7 FPS 320x240
|
||||
600 KB PSRAM
|
||||
Fallback
|
||||
Tap-to-talk UI
|
||||
QR code scan
|
||||
Pre-recorded hints
|
||||
Server
|
||||
mascarade LLM
|
||||
Adaptive hints
|
||||
Anti-spoiler prompts
|
||||
Conversation memory
|
||||
Ollama backend
|
||||
Coqui XTTS-v2
|
||||
Voice cloning 6s ref
|
||||
French TTS
|
||||
Chunked streaming
|
||||
Docker container
|
||||
MCP Server
|
||||
puzzle_set_state
|
||||
audio_play
|
||||
led_set
|
||||
camera_capture
|
||||
scenario_advance
|
||||
GPU KXKM-AI
|
||||
AudioCraft MusicGen
|
||||
Room ambients 30-60s
|
||||
SFX generation
|
||||
Batch pre-generation
|
||||
MP3 128kbps output
|
||||
Fine-tune Pipeline
|
||||
Qwen2.5-Coder-1.5B
|
||||
Unsloth + SimPO
|
||||
GGUF Q4_K_M export
|
||||
P2P distribute_task
|
||||
Analytics
|
||||
Difficulty adaptation
|
||||
Step timing telemetry
|
||||
Hint usage tracking
|
||||
Player count detection
|
||||
```
|
||||
|
||||
## Flowchart — Data Flows
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Input["Player Input"]
|
||||
MIC[Microphone I2S]
|
||||
CAM[Camera OV2640]
|
||||
BTN[Buttons / Touch]
|
||||
QR[QR Scanner]
|
||||
end
|
||||
|
||||
subgraph OnDevice["ESP32-S3 On-Device AI"]
|
||||
SR[ESP-SR v2.0<br/>Wake + Commands]
|
||||
DL[ESP-DL v3.2<br/>Object Detection]
|
||||
RT3[Runtime 3 Engine<br/>Step Machine]
|
||||
AUD[Audio Manager<br/>I2S DMA]
|
||||
LED[LED Manager<br/>WS2812B]
|
||||
UI[LVGL Display]
|
||||
end
|
||||
|
||||
subgraph Network["Network Layer"]
|
||||
WIFI[WiFi HTTP Client]
|
||||
ESPNOW[ESP-NOW Mesh]
|
||||
WS[WebSocket Events]
|
||||
end
|
||||
|
||||
subgraph Server["mascarade Server"]
|
||||
API[mascarade API<br/>/api/v1/send]
|
||||
MCP[MCP Hardware Server<br/>JSON-RPC 2.0]
|
||||
TTS[Coqui XTTS-v2<br/>Docker :5002]
|
||||
OLLAMA[Ollama<br/>:11434]
|
||||
ADAPT[Adaptive Engine<br/>Difficulty Params]
|
||||
end
|
||||
|
||||
subgraph GPU["KXKM-AI RTX 4090"]
|
||||
MUSICGEN[AudioCraft MusicGen<br/>Ambient Generation]
|
||||
STABAUD[Stable Audio Open<br/>SFX Generation]
|
||||
FINETUNE[Fine-tune Pipeline<br/>Unsloth]
|
||||
end
|
||||
|
||||
subgraph Output["Player Output"]
|
||||
SPK[Speaker I2S]
|
||||
LEDS[LED Strips]
|
||||
SCREEN[LCD Display]
|
||||
SERVO[Puzzle Actuators]
|
||||
end
|
||||
|
||||
%% Input -> On-Device
|
||||
MIC -->|PCM 16kHz| SR
|
||||
CAM -->|320x240 RGB| DL
|
||||
BTN -->|GPIO events| RT3
|
||||
QR -->|decoded text| RT3
|
||||
|
||||
%% On-Device processing
|
||||
SR -->|command token| RT3
|
||||
DL -->|detection event| RT3
|
||||
RT3 -->|play command| AUD
|
||||
RT3 -->|color command| LED
|
||||
RT3 -->|screen update| UI
|
||||
|
||||
%% On-Device -> Network
|
||||
SR -->|complex query| WIFI
|
||||
RT3 -->|hint request| WIFI
|
||||
RT3 -->|telemetry| WIFI
|
||||
RT3 <-->|peer sync| ESPNOW
|
||||
RT3 -->|state events| WS
|
||||
|
||||
%% Network -> Server
|
||||
WIFI -->|POST /api/v1/send| API
|
||||
WIFI -->|JSON-RPC| MCP
|
||||
WS -->|status stream| MCP
|
||||
|
||||
%% Server processing
|
||||
API -->|inference| OLLAMA
|
||||
OLLAMA -->|hint text| API
|
||||
API -->|text| TTS
|
||||
MCP -->|tool dispatch| API
|
||||
API -->|telemetry| ADAPT
|
||||
ADAPT -->|params| API
|
||||
|
||||
%% Server -> Device
|
||||
TTS -->|PCM stream| AUD
|
||||
MCP -->|puzzle cmd| SERVO
|
||||
MCP -->|led cmd| LED
|
||||
MCP -->|audio cmd| AUD
|
||||
|
||||
%% GPU -> Server (offline)
|
||||
MUSICGEN -.->|ambient MP3| AUD
|
||||
STABAUD -.->|SFX WAV| AUD
|
||||
FINETUNE -.->|GGUF model| OLLAMA
|
||||
|
||||
%% Output
|
||||
AUD --> SPK
|
||||
LED --> LEDS
|
||||
UI --> SCREEN
|
||||
RT3 --> SERVO
|
||||
|
||||
%% Styling
|
||||
style OnDevice fill:#1a3a1a,color:#fff
|
||||
style Server fill:#1a1a3a,color:#fff
|
||||
style GPU fill:#3a1a1a,color:#fff
|
||||
```
|
||||
|
||||
## Latency Map
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A["Player speaks"] -->|0ms| B["ESP-SR wake"]
|
||||
B -->|200ms| C["Command parse"]
|
||||
C -->|500ms| D{"On-device<br/>or server?"}
|
||||
D -->|"on-device<br/>0ms"| E["Runtime 3 action"]
|
||||
D -->|"server<br/>50ms network"| F["mascarade LLM"]
|
||||
F -->|"2000ms inference"| G["TTS synthesis"]
|
||||
G -->|"2000ms render"| H["Audio stream"]
|
||||
H -->|"100ms buffer"| I["Speaker plays"]
|
||||
|
||||
E -->|"50ms"| J["LED/Audio/Servo"]
|
||||
|
||||
style D fill:#dd6,stroke:#333
|
||||
```
|
||||
|
||||
**Critical path (voice hint)**: 200 + 500 + 50 + 2000 + 2000 + 100 = ~4850 ms worst case, target < 3000 ms.
|
||||
|
||||
## Component Integration Matrix
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Legend
|
||||
direction LR
|
||||
L1[Implemented] --- L2[Planned Phase B]
|
||||
L2 --- L3[Planned Phase C+]
|
||||
end
|
||||
|
||||
subgraph Current["Currently Implemented"]
|
||||
R3[Runtime 3 Engine]
|
||||
AUDIO[Audio Manager I2S]
|
||||
LEDM[LED Manager]
|
||||
LVGL[LVGL UI]
|
||||
WIFIC[WiFi + HTTP API]
|
||||
ESPNM[ESP-NOW Manager]
|
||||
STOR[Storage Manager]
|
||||
end
|
||||
|
||||
subgraph PhaseB["Phase B: Voice"]
|
||||
WAKE[WakeNet Custom]
|
||||
MULTI[MultiNet FR]
|
||||
XTTS[XTTS-v2 Docker]
|
||||
STREAM[Audio Streaming]
|
||||
end
|
||||
|
||||
subgraph PhaseC["Phase C+: Vision, LLM, Music"]
|
||||
YOLO[YOLOv11n ESP-DL]
|
||||
HINTS[LLM Hints]
|
||||
MCPS[MCP Server]
|
||||
MGEN[MusicGen Batch]
|
||||
end
|
||||
|
||||
R3 --> WAKE
|
||||
R3 --> YOLO
|
||||
R3 --> HINTS
|
||||
AUDIO --> STREAM
|
||||
STREAM --> XTTS
|
||||
WIFIC --> MCPS
|
||||
MCPS --> HINTS
|
||||
HINTS --> XTTS
|
||||
AUDIO --> MGEN
|
||||
WAKE --> MULTI
|
||||
|
||||
style Current fill:#2d6,stroke:#333
|
||||
style PhaseB fill:#69d,stroke:#333
|
||||
style PhaseC fill:#d69,stroke:#333
|
||||
```
|
||||
|
||||
## Notes
|
||||
- On-device AI (ESP-SR, ESP-DL) runs without network — zero-latency fallback.
|
||||
- Server AI (LLM, TTS, MCP) requires WiFi — graceful degradation to cached hints.
|
||||
- GPU AI (AudioCraft) is offline batch only — no runtime dependency.
|
||||
- All AI features are additive: the base Runtime 3 game works without any AI.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Component Map
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Studio
|
||||
App["App.tsx"]
|
||||
Designer["BlocklyDesigner.tsx"]
|
||||
ScenarioLib["lib/scenario.ts"]
|
||||
Runtime3Lib["lib/runtime3.ts"]
|
||||
ApiLib["lib/api.ts"]
|
||||
end
|
||||
|
||||
subgraph Canon
|
||||
ScenarioYaml["game/scenarios/zacus_v2.yaml"]
|
||||
end
|
||||
|
||||
subgraph Runtime3
|
||||
CompilerPy["tools/scenario/compile_runtime3.py"]
|
||||
RuntimeCommon["tools/scenario/runtime3_common.py"]
|
||||
SimPy["tools/scenario/simulate_runtime3.py"]
|
||||
end
|
||||
|
||||
subgraph Device
|
||||
Firmware["hardware/firmware"]
|
||||
StoryCore["lib/story runtime adapter"]
|
||||
UI["Freenove UI + API"]
|
||||
end
|
||||
|
||||
ScenarioYaml --> ScenarioLib
|
||||
Designer --> ScenarioLib
|
||||
ScenarioLib --> Runtime3Lib
|
||||
Runtime3Lib --> App
|
||||
ScenarioYaml --> CompilerPy
|
||||
CompilerPy --> RuntimeCommon
|
||||
RuntimeCommon --> SimPy
|
||||
RuntimeCommon --> Firmware
|
||||
Firmware --> StoryCore
|
||||
StoryCore --> UI
|
||||
ApiLib --> UI
|
||||
```
|
||||
|
||||
## Notes
|
||||
- The React studio owns authoring ergonomics and previews.
|
||||
- The Python compiler/simulator owns deterministic offline validation.
|
||||
- Firmware consumes the Runtime 3 contract and exposes runtime APIs.
|
||||
@@ -0,0 +1,21 @@
|
||||
# Data Flow Map
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
Author["Author edits Blockly graph"] --> Draft["Scenario draft in browser"]
|
||||
Draft --> Yaml["Canonical YAML"]
|
||||
Yaml --> Validate["Scenario validators"]
|
||||
Yaml --> Compile["Runtime 3 compiler"]
|
||||
Compile --> IR["Runtime 3 JSON"]
|
||||
IR --> Sim["Local simulator"]
|
||||
IR --> Firmware["Firmware adapter"]
|
||||
Yaml --> Exports["Audio / printables / MJ kit exports"]
|
||||
Firmware --> Api["Runtime API + status"]
|
||||
Api --> Studio["Studio dashboard + diagnostics"]
|
||||
Api --> Evidence["Logs / artifacts / AGENT_TODO evidence"]
|
||||
```
|
||||
|
||||
## Guarantees
|
||||
- Narrative truth remains in YAML during the migration window.
|
||||
- Runtime truth is emitted as versioned IR JSON.
|
||||
- Hardware evidence stays outside git and is referenced from planning/TODO docs.
|
||||
@@ -0,0 +1,34 @@
|
||||
# Feature Map
|
||||
|
||||
## Core Gameplay Features
|
||||
|
||||
| Feature | Canon input | Runtime pivot | Delivery surfaces |
|
||||
| --- | --- | --- | --- |
|
||||
| LA 440 stabilization | `puzzles.PUZZLE_LA_440` | `STEP_LA_DETECTOR` | studio, firmware, audio, GM kit |
|
||||
| LEFOU piano unlock | `puzzles.PUZZLE_PIANO_ALPHABET_5` | `STEP_LEFOU_DETECTOR` | studio, firmware, printables, GM kit |
|
||||
| QR WIN finale | `puzzles.PUZZLE_QR_WIN` | `STEP_QR_DETECTOR` | studio, firmware camera, archives setup |
|
||||
| Media hub / finale | `firmware.media_hub` | `SCENE_MEDIA_MANAGER` | firmware, studio dashboard |
|
||||
|
||||
## Authoring Features
|
||||
|
||||
```mermaid
|
||||
mindmap
|
||||
root((Studio Zacus))
|
||||
Blockly editor
|
||||
Steps
|
||||
Transitions
|
||||
Apps
|
||||
YAML preview
|
||||
Runtime 3 preview
|
||||
Validate
|
||||
Deploy
|
||||
Dashboard
|
||||
Media
|
||||
Network
|
||||
```
|
||||
|
||||
## Refactor Priorities
|
||||
1. Authoring parity with the current canonical scenario.
|
||||
2. Runtime 3 compilation and simulation.
|
||||
3. Combined-board execution on Freenove.
|
||||
4. Documentation and cleanup of legacy references.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Zacus Architecture
|
||||
|
||||
Cette section est le point d’entrée canonique de la refonte Zacus V3.
|
||||
|
||||
## Vue d’ensemble
|
||||
- Le contenu narratif reste défini dans `game/scenarios/*.yaml`.
|
||||
- Le studio auteur canonique est `frontend-scratch-v2/`.
|
||||
- Le nouveau noyau cible est **Zacus Runtime 3**, compilé en IR JSON.
|
||||
- Le firmware devient un adaptateur d’exécution et d’API.
|
||||
|
||||
## Cartes publiées
|
||||
- [System Map](system-map.md)
|
||||
- [Component Map](component-map.md)
|
||||
- [Data Flow Map](data-flow-map.md)
|
||||
- [Feature Map](feature-map.md)
|
||||
- [Migration Map](migration-map.md)
|
||||
- [Agent Matrix](agent-matrix.md)
|
||||
- [Release Map](release-map.md)
|
||||
|
||||
## Principe directeur
|
||||
Une seule architecture visible:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Author["Auteur Zacus"] --> Studio["Studio React + Blockly"]
|
||||
Studio --> Canon["YAML canonique"]
|
||||
Canon --> Compiler["Compilateur Runtime 3"]
|
||||
Compiler --> IR["IR JSON versionné"]
|
||||
IR --> Firmware["Adaptateur firmware"]
|
||||
Firmware --> Device["Carte Freenove / périphériques"]
|
||||
Firmware --> API["API runtime + observabilité"]
|
||||
API --> Studio
|
||||
```
|
||||
@@ -0,0 +1,23 @@
|
||||
# Migration Map
|
||||
|
||||
## Migration Route
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
LegacyDocs["Conflicting docs/specs"] --> CanonDocs["Unified architecture docs"]
|
||||
LegacyFront["fronted dev web UI"] --> ReactStudio["frontend-scratch-v2"]
|
||||
StoryV2["Story V2 runtime"] --> Runtime3["Zacus Runtime 3"]
|
||||
DirectFirmwareStory["Firmware-owned story semantics"] --> FirmwareAdapter["Firmware adapter over IR"]
|
||||
AdHocTracking["Scattered TODO/plan refs"] --> RefactorCoord["memory/ + plans/ + todos/"]
|
||||
```
|
||||
|
||||
## Rules
|
||||
- No destructive cleanup before replacement proof.
|
||||
- Legacy routes are quarantined first, then removed.
|
||||
- Hardware evidence stays in `hardware/firmware/docs/AGENT_TODO.md` until the adapter route is stable.
|
||||
|
||||
## Current Quarantine Targets
|
||||
- `fronted dev web UI/`
|
||||
- Svelte/Cytoscape specs and references
|
||||
- `zacus_v1` references outside archive or historical context
|
||||
- duplicate docs such as `docs/AGENTS 2.md` and `docs/AGENT_TODO 2.md`
|
||||
@@ -0,0 +1,20 @@
|
||||
# Release Map
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Wave0["Wave 0 Baseline"] --> Wave1["Wave 1 Canon"]
|
||||
Wave1 --> Wave2["Wave 2 Runtime 3"]
|
||||
Wave2 --> Wave3["Wave 3 Studio"]
|
||||
Wave3 --> Wave4["Wave 4 Firmware"]
|
||||
Wave4 --> Wave5["Wave 5 Cleanup"]
|
||||
Wave5 --> Wave6["Wave 6 Cutover"]
|
||||
```
|
||||
|
||||
## Exit Criteria
|
||||
- Wave 0: baseline evidence captured and dirty-tree risks understood.
|
||||
- Wave 1: architecture/spec/docs/plans canonized.
|
||||
- Wave 2: Runtime 3 compiler and simulator usable on the canonical scenario.
|
||||
- Wave 3: React/Blockly studio builds and previews Runtime 3.
|
||||
- Wave 4: Freenove build path consumes the runtime contract without regression.
|
||||
- Wave 5: legacy paths archived or deleted with proof of replacement.
|
||||
- Wave 6: README, docs site, CI, and operator entrypoints all point to the new canon.
|
||||
@@ -0,0 +1,48 @@
|
||||
# System Map
|
||||
|
||||
## Canonical System
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph Authoring
|
||||
YAML["game/scenarios/zacus_v2.yaml"]
|
||||
Blockly["frontend-scratch-v2 Blockly graph"]
|
||||
end
|
||||
|
||||
subgraph Runtime3
|
||||
Compiler["compile_runtime3.py / runtime3.ts"]
|
||||
IR["Runtime 3 IR JSON"]
|
||||
Sim["simulate_runtime3.py"]
|
||||
end
|
||||
|
||||
subgraph Delivery
|
||||
Audio["audio/manifests"]
|
||||
Printables["printables/manifests"]
|
||||
Kit["kit-maitre-du-jeu"]
|
||||
Docs["docs + MkDocs"]
|
||||
end
|
||||
|
||||
subgraph Device
|
||||
Firmware["hardware/firmware adapter"]
|
||||
Board["freenove_esp32s3"]
|
||||
Media["media / network / UI"]
|
||||
end
|
||||
|
||||
YAML --> Compiler
|
||||
Blockly --> Compiler
|
||||
Compiler --> IR
|
||||
IR --> Sim
|
||||
IR --> Firmware
|
||||
YAML --> Audio
|
||||
YAML --> Printables
|
||||
YAML --> Kit
|
||||
YAML --> Docs
|
||||
Firmware --> Board
|
||||
Firmware --> Media
|
||||
```
|
||||
|
||||
## Design Notes
|
||||
- YAML stays canonical for narrative truth during migration.
|
||||
- Blockly is the preferred authoring UX, not the runtime artifact itself.
|
||||
- Runtime 3 IR is the portable contract between authoring and execution.
|
||||
- Firmware owns board integration, not story semantics.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 3.1 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.6 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.5 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.4 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.4 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.9 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.1 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.6 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.5 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.4 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.4 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.9 MiB |
@@ -0,0 +1,31 @@
|
||||
# OSS Benchmark
|
||||
|
||||
## Adopt / Align
|
||||
|
||||
| Tool | Why it matters | Reference |
|
||||
| --- | --- | --- |
|
||||
| Blockly | Canonical block-based authoring model for the Zacus studio. | https://developers.google.com/blockly |
|
||||
| Mermaid | Lightweight diagrams embedded in docs and specs. | https://mermaid.js.org/ |
|
||||
| Material for MkDocs | Fast documentation shell for the rebuilt knowledge base. | https://squidfunk.github.io/mkdocs-material/ |
|
||||
| Charm Gum | Optional shell TUI layer with graceful fallback. | https://github.com/charmbracelet/gum |
|
||||
|
||||
## Benchmark / Inspiration
|
||||
|
||||
| Project | Why it is relevant | Reference |
|
||||
| --- | --- | --- |
|
||||
| Node-RED | Flow-based authoring patterns and graph ergonomics. | https://nodered.org/ |
|
||||
| Ink | Narrative authoring and branching conventions. | https://www.inklestudios.com/ink/ |
|
||||
| Yarn Spinner | Dialogue/runtime separation patterns. | https://www.yarnspinner.dev/ |
|
||||
| LVGL | Embedded UI runtime constraints and display patterns. | https://lvgl.io/ |
|
||||
| SquareLine Studio | Embedded UI workflow inspiration for screen authoring. | https://squareline.io/ |
|
||||
| EEZ Studio | Embedded UI/editor workflow benchmark for device tooling. | https://www.envox.eu/studio/studio-introduction/ |
|
||||
|
||||
## Zacus-specific Decision
|
||||
- Zacus Runtime 3 remains custom and portable.
|
||||
- External OSS is used as a reference or shell, not as a drop-in replacement for the story runtime.
|
||||
|
||||
## Fresh Notes (2026-03-16)
|
||||
- Blockly remains the strongest fit for Zacus authoring because it is still positioned as a customizable client-side block editor for application-specific languages.
|
||||
- Node-RED remains the best ergonomics benchmark for event-driven flows that can run both on low-cost edge devices and in the cloud.
|
||||
- LVGL stays the right embedded UI reference because it explicitly targets small memory footprints and broad MCU/OS/display portability.
|
||||
- EEZ Studio is the strongest workflow benchmark for embedded authoring because it combines templates, visual debugging, execution logs, and timeline-driven UI tooling.
|
||||
+24
-112
@@ -1,119 +1,31 @@
|
||||
---
|
||||
layout: default
|
||||
title: Le Mystère du Professeur Zacus
|
||||
description: Une enquête scientifique immersive pour anniversaire (printables, audio, modules électroniques).
|
||||
---
|
||||
# Zacus V3
|
||||
|
||||
# 🎩 Le Mystère du Professeur Zacus
|
||||
Point d'entrée documentaire de la refonte Zacus.
|
||||
|
||||
|
||||
## Canon
|
||||
- YAML source de vérité: `game/scenarios/zacus_v2.yaml`
|
||||
- Studio auteur: `frontend-scratch-v2/`
|
||||
- Runtime portable: Zacus Runtime 3
|
||||
- Cible terrain: Freenove ESP32-S3 via `hardware/firmware/`
|
||||
|
||||
Le Mystère du Professeur Zacus : une enquête scientifique immersive pour anniversaire, jouable en famille ou entre ami·e·s.
|
||||
## Démarrage
|
||||
- [Quickstart](QUICKSTART.md)
|
||||
- [Architecture](architecture/index.md)
|
||||
- [Benchmark OSS](benchmark-oss.md)
|
||||
|
||||
Scénarios modulaires (YAML), supports imprimables, audio, et accessoires électroniques (ESP32/Arduino).
|
||||
## Carte rapide
|
||||
|
||||
<div class="badges">
|
||||
<div class="badge"><img src="./assets/icons/icon_hat.png" alt="hat" />Scénarios modulaires</div>
|
||||
<div class="badge"><img src="./assets/icons/icon_audio.png" alt="audio" />Audio & checkpoints</div>
|
||||
<div class="badge"><img src="./assets/icons/icon_chip.png" alt="chip" />Modules électroniques</div>
|
||||
<div class="badge"><img src="./assets/icons/icon_flask.png" alt="flask" />Thème labo/science</div>
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Structure du projet
|
||||
|
||||
```text
|
||||
├── game/scenarios/*.yaml Scénarios (source de vérité)
|
||||
├── audio/manifests/*.yaml Manifeste audio + fichiers associés
|
||||
├── printables/manifests/*.yaml Manifeste printables + PDF/PNG
|
||||
├── hardware/firmware/esp32/ Firmware (lecture seule sans approbation)
|
||||
├── tools/ Outils Python (validation, export, génération)
|
||||
├── docs/ Guides + mini-site GitHub Pages
|
||||
├── assets/ Images (README / marketing)
|
||||
```mermaid
|
||||
flowchart LR
|
||||
YAML["Scenario YAML"] --> Runtime3["Runtime 3"]
|
||||
Runtime3 --> Studio["React + Blockly studio"]
|
||||
Runtime3 --> Firmware["Firmware adapter"]
|
||||
YAML --> Exports["Audio / printables / MJ kit"]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Démarrage rapide
|
||||
|
||||
1. Imprime les printables (cartes, indices).
|
||||
2. (Option) Flash les modules électroniques.
|
||||
3. Lis le scénario YAML et place les indices.
|
||||
4. Lance la partie et suis le guide MJ.
|
||||
|
||||
---
|
||||
|
||||
## Guides & documentation
|
||||
|
||||
- [STRUCTURE.md](STRUCTURE.md) : Architecture détaillée
|
||||
- [QUICKSTART.md](QUICKSTART.md) : Démarrage express
|
||||
- [WORKFLOWS.md](WORKFLOWS.md) : Workflows validation/export
|
||||
- [faq.md](faq.md) : FAQ dépannage
|
||||
|
||||
---
|
||||
|
||||
## Licences
|
||||
|
||||
- Code : MIT — voir [../LICENSE](../LICENSE)
|
||||
- Contenu créatif : CC BY‑NC 4.0 — voir [../LICENSE-CONTENT.md](../LICENSE-CONTENT.md)
|
||||
|
||||
---
|
||||
|
||||
## Assets
|
||||
|
||||
- Couverture : assets/cover.png
|
||||
- OpenGraph : assets/og.png
|
||||
- Poster : assets/poster.png
|
||||
- Logo : assets/logo.png
|
||||
- Diagramme : assets/diagram.png
|
||||
- Aperçu printables : assets/printables.png
|
||||
|
||||
## Démo
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Jouer (MJ)
|
||||
|
||||
### Checklist rapide
|
||||
1. Imprime les **printables** (cartes, indices).
|
||||
2. (Option) Flash les modules électroniques.
|
||||
3. Lis le scénario et place les indices.
|
||||
4. Lance la partie et suis le guide MJ.
|
||||
|
||||
👉 Voir aussi : [FAQ (flash / SD / audio)](./faq)
|
||||
|
||||
---
|
||||
|
||||
## Développer / contribuer
|
||||
|
||||
### Structure du dépôt
|
||||
- `game/scenarios/` — scénarios (source de vérité)
|
||||
- `audio/` — manifestes et ressources audio
|
||||
- `printables/` — manifestes + PDF/PNG
|
||||
- `tools/` — outils Python (validation, export)
|
||||
- `hardware/firmware/esp32/` — firmware (**lecture seule** sans approbation)
|
||||
|
||||
### Démarrage rapide
|
||||
```bash
|
||||
python3 -m pip install -r tools/requirements.txt
|
||||
python3 tools/scenario/validate_scenario.py game/scenarios/mon_scenario.yaml
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Visuels & matériel
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
## Licences
|
||||
- **Code** : MIT (`LICENSE`)
|
||||
- **Contenu** : CC BY‑NC 4.0 (`LICENSE-CONTENT.md`)
|
||||
## Liens utiles
|
||||
- [Repository Structure](STRUCTURE.md)
|
||||
- Spécification Runtime 3: `specs/ZACUS_RUNTIME_3_SPEC.md`
|
||||
- Spécification studio: `specs/STORY_DESIGNER_SCRATCH_LIKE_SPEC.md`
|
||||
- Plans: `plans/master-plan.md`
|
||||
- Todos: `todos/master.md`
|
||||
|
||||
@@ -1,67 +0,0 @@
|
||||
# Audit de cohérence du dépôt
|
||||
|
||||
Date: 2026-02-12 (mise à jour cohérence globale)
|
||||
Périmètre: **contenu documentaire + structure fichiers**, sans modification firmware.
|
||||
|
||||
## Résumé exécutif
|
||||
|
||||
- Le dépôt est globalement bien structuré en 3 piliers: `kit-maitre-du-jeu/`, `printables/`, `hardware/`.
|
||||
- La zone la plus mature côté exécution est le firmware ESP32/ESP8266 (documenté et outillé).
|
||||
- Le principal point de cohérence identifié concernait les chemins printables documentés mais absents.
|
||||
- Les images PNG printables sont désormais centralisées sous `printables/export/png/`.
|
||||
|
||||
## Couverture de revue
|
||||
|
||||
- Revue des fichiers Markdown/README du dépôt (hors licences externes inchangées).
|
||||
- Vérification de cohérence des chemins documentés vs arborescence versionnée.
|
||||
- Vérification des assets image versionnés.
|
||||
|
||||
## Analyse du contenu des fichiers
|
||||
|
||||
### 1) Documentation racine
|
||||
- `README.md` décrit clairement le périmètre et les licences.
|
||||
- `CONTRIBUTING.md` couvre les règles éditoriales et le workflow.
|
||||
- `CHANGELOG.md` contient désormais une release `0.2.0` et un `Unreleased` exploitable.
|
||||
|
||||
### 2) Kit MJ
|
||||
- Le dossier `kit-maitre-du-jeu/` est cohérent et orienté animation terrain.
|
||||
- Les sous-docs couvrent script, solution, rôles, anti-chaos et stations.
|
||||
|
||||
### 3) Printables
|
||||
- La convention de nommage est explicite.
|
||||
- Le dépôt déclarait une structure `src/export` non matérialisée: ce point est désormais aligné.
|
||||
|
||||
### 4) Hardware
|
||||
- Arborescence hardware lisible (`bom`, `wiring`, `firmware`, `enclosure`).
|
||||
- Le firmware reste hors périmètre de modification dans ce cycle.
|
||||
|
||||
## Analyse des images
|
||||
|
||||
Commande utilisée:
|
||||
- `rg --files | rg -i '\\.(png|jpe?g|gif|webp|svg|bmp)$'`
|
||||
|
||||
Résultat:
|
||||
- Des previews PNG sont bien versionnées, avec un classement thématique dans `printables/export/png/`.
|
||||
|
||||
Conséquence:
|
||||
- La revue visuelle des imprimables est possible directement depuis le dépôt.
|
||||
- Recommandé: maintenir ce classement à chaque nouvel export.
|
||||
|
||||
## Corrections de cohérence appliquées
|
||||
|
||||
1. Création de la structure printables attendue:
|
||||
- `printables/src/`
|
||||
- `printables/export/pdf/`
|
||||
- `printables/export/png/`
|
||||
2. Création des sous-dossiers invitations cohérents avec son README:
|
||||
- `printables/invitations/src/`
|
||||
- `printables/invitations/export/pdf/`
|
||||
- `printables/invitations/export/png/`
|
||||
3. Ajout de README ciblés dans les nouveaux dossiers printables.
|
||||
4. Regroupement des PNG du dossier `printables/` vers `printables/export/png/` avec sous-dossiers (`general`, `fiche-enquete`, `personnages`, `zones`).
|
||||
|
||||
## Recommandations immédiates (hors firmware)
|
||||
|
||||
- Ajouter 1 à 3 documents pilotes dans `printables/export/pdf/` et leurs previews dans `printables/export/png/`.
|
||||
- Maintenir `CHANGELOG.md` à chaque PR doc-only.
|
||||
- Utiliser la checklist PR de `CONTRIBUTING.md` comme gate de merge.
|
||||
@@ -1,54 +0,0 @@
|
||||
# Analyse approfondie du dépôt — 2026-02-25
|
||||
|
||||
## Objectif
|
||||
Aller plus loin que le lot conversation bundle en évaluant l'état global du repo (docs, workflows, contenu gameplay, outillage validation, CI).
|
||||
|
||||
## Méthode
|
||||
Commandes de revue utilisées (échantillon):
|
||||
- `rg --files`
|
||||
- `rg -n "validate_|workflow|scenario" .github/workflows tools docs game`
|
||||
- `python3 tools/scenario/validate_runtime_bundle.py`
|
||||
- `python3 tools/scenario/validate_scenario.py game/scenarios/zacus_v1.yaml`
|
||||
- `python3 tools/scenario/export_md.py game/scenarios/zacus_v1.yaml`
|
||||
- `python3 tools/audio/validate_manifest.py audio/manifests/zacus_v1_audio.yaml`
|
||||
- `python3 tools/printables/validate_manifest.py printables/manifests/zacus_v1_printables.yaml`
|
||||
|
||||
## Constat global
|
||||
|
||||
### 1) Source de vérité gameplay
|
||||
- ✅ `game/scenarios/zacus_v1.yaml` est bien utilisé comme source canon.
|
||||
- ✅ Les exports MJ/docs générés sont cohérents avec le canon courant.
|
||||
- ⚠️ Le bundle `scenario-ai-coherence/` reste parallèle: utile pour itérer, mais il faut éviter toute dérive silencieuse.
|
||||
|
||||
### 2) Gates G3 / validation
|
||||
- ✅ Validateurs scénario/audio/printables présents et opérationnels.
|
||||
- ✅ `validate_runtime_bundle.py` existe et est branché dans CI (`.github/workflows/validate.yml`).
|
||||
- ⚠️ Le diff fonctionnel gameplay est scripté localement, mais pas encore exécuté automatiquement en CI.
|
||||
|
||||
### 3) Documentation et guidage équipe
|
||||
- ✅ `docs/WORKFLOWS.md` décrit l'enchaînement validation.
|
||||
- ✅ Spec/arch conversation bundle documentent le périmètre.
|
||||
- ⚠️ Il manque un guide "release gate" unique qui enchaîne G0→G5 en une seule checklist opérationnelle.
|
||||
|
||||
### 4) Risques principaux
|
||||
- Divergence entre bundle conversationnel et canon gameplay si la promotion n'est pas rituelle.
|
||||
- G4/G5 encore partiels (pas de lot release/compliance structuré dans ce cycle).
|
||||
- Dépendance implicite à PyYAML: bootstrap présent, mais doit rester obligatoire en CI locale/README.
|
||||
|
||||
## Plan de progression recommandé (prochaines PRs)
|
||||
|
||||
### PR-A (QA/Test)
|
||||
- Ajouter un job CI dédié qui exécute aussi `tools/scenario/diff_gameplay_scenarios.py`.
|
||||
- Publier l'artifact markdown de diff en sortie CI (upload artifact GitHub Actions).
|
||||
|
||||
### PR-B (Doc/Release)
|
||||
- Créer `docs/RELEASE_GATES.md` avec checklist G0→G5 exécutable.
|
||||
- Standardiser le template de PR pour exiger la section "Gates satisfaites" + "limitations".
|
||||
|
||||
### PR-C (Compliance)
|
||||
- Ajouter un inventaire SBOM/licences minimal pour G5 (au moins dépendances Python validateurs + manifest repo).
|
||||
- Centraliser les preuves dans `evidence/` avec index daté.
|
||||
|
||||
## Décision proposée maintenant
|
||||
- Garder le canon gameplay tel quel.
|
||||
- Passer sur PR-A en priorité pour fermer l'écart CI sur le diff fonctionnel.
|
||||
@@ -1,32 +0,0 @@
|
||||
# Repo Status Report
|
||||
|
||||
## Tree summary
|
||||
- `kit-maitre-du-jeu/`: full GM script, solutions, checklists, and export folders for PDF/PNG deliverables.
|
||||
- `printables/`: `src/` plus `export/{pdf,png}/` for invitations, cards, badges, and the one-page rule set.
|
||||
- `hardware/`: BOMs, wiring docs, and the ESP32/Arduino firmware flow with `.pio` dependencies (libraries such as Adafruit and ESP8266Audio naturally expand here).
|
||||
- `docs/`: maintenance plan, repo audit, and asset folder (`docs/assets/` contains the repo map SVG already referenced in `README.md`).
|
||||
- `examples/` and `scenario-ai-coherence/` (renamed for portability): auxiliary content and tools, now safe for Windows/CI.
|
||||
|
||||
## Empty / TODO files
|
||||
- `printables/invitations/export/pdf/.gitkeep`
|
||||
- `printables/invitations/export/png/.gitkeep`
|
||||
- `printables/invitations/src/.gitkeep`
|
||||
- assorted `.nojekyll`, `.uno.test.skip`, and upstream library placeholders under `hardware/firmware/esp32/.pio/` (e.g., Adafruit BusIO examples); none appear to contain TODO hints, but they are empty files that may surprise contributors or scripts.
|
||||
|
||||
## Broken links
|
||||
- `hardware/firmware/esp32/.pio/libdeps/esp32dev/ESP8266Audio/README.md` line 250 → `examples/StreamMP3FromHTTP_SPIRAM/Schema_Spiram.png` (missing in that vendor snapshot).
|
||||
- `hardware/firmware/esp32/.pio/libdeps/esp32_release/ESP8266Audio/README.md` line 250 → same missing `Schema_Spiram.png`.
|
||||
- `hardware/firmware/esp32/.pio/libdeps/esp32dev/Mozzi/README.md` line 116 → `extras/NEWS.txt` (not included in the copied tree).
|
||||
- `hardware/firmware/esp32/.pio/libdeps/esp32_release/Mozzi/README.md` line 116 → same missing `extras/NEWS.txt`.
|
||||
|
||||
## Naming / portability issues
|
||||
- `scenario-ai-coherence/` and `scenario-ai-coherence/version-finale/` now avoid spaces/colon, keeping the tree stable sur Windows/CI tooling. Continue to prefer hyphenated, lowercase identifiers pour tout nouveau sous-dossier.
|
||||
|
||||
## Licences
|
||||
- `README.md`, `CONTRIBUTING.md`, and `LICENSE.md` sont alignés autour du modèle CC BY-NC 4.0 pour les contenus créatifs et MIT pour le code (`LICENSES/CC-BY-NC-4.0.txt`, `LICENSES/MIT.txt`). Garder la même nomenclature accélère la revue des contributions.
|
||||
|
||||
## Patch plan
|
||||
- Continuer à surveiller les fichiers vides (`.gitkeep`, `.nojekyll`, `.uno.test.skip`) pour décider s’ils restent ou disparaissent lors d’un nettoyage futur.
|
||||
- Corriger ou supprimer les liens cassés des bibliothèques `ESP8266Audio`/`Mozzi` si les assets nécessaires ne sont plus plastifiés.
|
||||
- Valider chaque nouvelle version de scénario et de manifeste audio avec `tools/scenario/validate_scenario.py` et `tools/audio/validate_manifest.py`.
|
||||
- Mettre à jour Quickstart, Styleguide et le workflow printables si la structure du kit ou les fichiers audio changent.
|
||||
@@ -1,52 +0,0 @@
|
||||
{
|
||||
"schema_version": "repo_state.v1",
|
||||
"generated_at_utc": "2026-02-21T02:23:01Z",
|
||||
"repo": "le-mystere-professeur-zacus",
|
||||
"repo_url": "https://github.com/electron-rare/le-mystere-professeur-zacus.git",
|
||||
"branch": "codex/repo-state-zacus",
|
||||
"head": "3fec631c43c66a6ae12c12d2f652c8800d22b28f",
|
||||
"head_date": "2026-02-21T02:59:01+01:00",
|
||||
"head_subject": "feat(freenove): stabilize fallback AP outside local wifi (#103)",
|
||||
"project_kind": "hardware_firmware_hybrid",
|
||||
"pivot_changes": [
|
||||
{
|
||||
"path": "hardware/firmware/data/story/apps/APP_WIFI.json",
|
||||
"tags": [
|
||||
"firmware_build_test",
|
||||
"hardware_validation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "hardware/firmware/docs/AGENT_TODO.md",
|
||||
"tags": [
|
||||
"firmware_build_test",
|
||||
"hardware_validation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "hardware/firmware/hardware/firmware/ui_freenove_allinone/include/network_manager.h",
|
||||
"tags": [
|
||||
"firmware_build_test",
|
||||
"hardware_validation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "hardware/firmware/hardware/firmware/ui_freenove_allinone/src/main.cpp",
|
||||
"tags": [
|
||||
"firmware_build_test",
|
||||
"hardware_validation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "hardware/firmware/hardware/firmware/ui_freenove_allinone/src/network_manager.cpp",
|
||||
"tags": [
|
||||
"firmware_build_test",
|
||||
"hardware_validation"
|
||||
]
|
||||
}
|
||||
],
|
||||
"impact_gates": [
|
||||
"firmware_build_test",
|
||||
"hardware_validation"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,299 @@
|
||||
# Voice Pipeline Guide — Professeur Zacus
|
||||
|
||||
Technical guide for integrating voice interaction into the Zacus ESP32-S3 installation.
|
||||
|
||||
## Overview
|
||||
|
||||
The voice pipeline enables visitors to interact with Professeur Zacus using natural speech. The system uses a split architecture: wake word detection runs locally on the ESP32-S3 (via ESP-SR), while speech-to-text, LLM inference, and text-to-speech run server-side via the mascarade bridge.
|
||||
|
||||
## Hardware Requirements
|
||||
|
||||
### INMP441 MEMS Microphone
|
||||
|
||||
| INMP441 Pin | ESP32-S3 GPIO | Notes |
|
||||
|-------------|---------------|-------|
|
||||
| SCK (BCLK) | GPIO 41 | I2S0 bit clock |
|
||||
| WS (LRCLK) | GPIO 42 | I2S0 word select |
|
||||
| SD (DATA) | GPIO 2 | I2S0 data in |
|
||||
| L/R | GND | Left channel select |
|
||||
| VDD | 3.3V | Power supply |
|
||||
| GND | GND | Ground |
|
||||
|
||||
**Important:** The microphone uses I2S0, which is a separate bus from the speaker output on I2S1 (MAX98357A). Both can operate simultaneously for full-duplex audio.
|
||||
|
||||
### Speaker (already present)
|
||||
|
||||
The Freenove All-in-One board includes a MAX98357A I2S amplifier on I2S1. No additional wiring needed for audio output.
|
||||
|
||||
### Bill of Materials (voice-specific)
|
||||
|
||||
| Part | Qty | Approx. Cost | Source |
|
||||
|------|-----|-------------|--------|
|
||||
| INMP441 breakout | 1 | 3-5 EUR | AliExpress, Amazon |
|
||||
| Dupont wires (F-F) | 5 | included | — |
|
||||
|
||||
## Software Requirements
|
||||
|
||||
### ESP-IDF + esp-sr Component
|
||||
|
||||
The ESP-SR library (WakeNet, MultiNet, AFE) requires the ESP-IDF framework. The current project uses Arduino framework via PlatformIO.
|
||||
|
||||
**Option A: Full migration to ESP-IDF (recommended for production)**
|
||||
|
||||
```ini
|
||||
; platformio.ini changes
|
||||
[env:freenove_allinone]
|
||||
framework = espidf
|
||||
; All Arduino-style code must be rewritten to ESP-IDF APIs
|
||||
```
|
||||
|
||||
**Option B: Arduino as ESP-IDF component (gradual migration)**
|
||||
|
||||
```ini
|
||||
; platformio.ini changes
|
||||
[env:freenove_allinone]
|
||||
framework = arduino, espidf
|
||||
; Allows mixing Arduino and ESP-IDF code
|
||||
; esp-sr can be added as an IDF component
|
||||
```
|
||||
|
||||
### esp-sr Component Registration
|
||||
|
||||
Create or update `idf_component.yml`:
|
||||
```yaml
|
||||
dependencies:
|
||||
espressif/esp-sr:
|
||||
version: "^1.0.0"
|
||||
# Includes: WakeNet, MultiNet, AFE (AEC + NS + VAD)
|
||||
```
|
||||
|
||||
### Partition Table
|
||||
|
||||
ESP-SR models need flash space. Update `partitions.csv`:
|
||||
```
|
||||
# Name, Type, SubType, Offset, Size, Flags
|
||||
model, data, spiffs, , 1500K, ; WakeNet + optional MultiNet models
|
||||
```
|
||||
|
||||
## Voice Flow — Sequence Diagram
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Mic as INMP441<br/>(I2S0)
|
||||
participant AFE as ESP-SR AFE<br/>(AEC+NS+VAD)
|
||||
participant WN as WakeNet9<br/>("Prof Zacus")
|
||||
participant WS as WebSocket<br/>Client
|
||||
participant Bridge as mascarade<br/>bridge:4001
|
||||
participant LLM as LLM<br/>(Qwen/GPT)
|
||||
participant TTS as Piper TTS<br/>(French)
|
||||
participant Spk as Speaker<br/>(I2S1)
|
||||
|
||||
Note over Mic,AFE: LISTENING state
|
||||
Mic->>AFE: 16kHz 16-bit PCM (continuous)
|
||||
AFE->>WN: Cleaned audio frames
|
||||
WN-->>WN: Scanning for wake word...
|
||||
|
||||
Note over WN,WS: WAKE_DETECTED state
|
||||
WN->>WS: Wake word detected!
|
||||
Spk->>Spk: Play chime / LED feedback
|
||||
|
||||
Note over WS,Bridge: STREAMING state
|
||||
WS->>Bridge: {"type":"hello","wakeword":"professeur zacus"}
|
||||
loop Until VAD silence
|
||||
AFE->>WS: Processed audio frames
|
||||
WS->>Bridge: OPUS-encoded binary frames
|
||||
Bridge-->>WS: {"type":"stt","text":"..."} (interim)
|
||||
end
|
||||
WS->>Bridge: {"type":"end_of_speech"}
|
||||
|
||||
Note over Bridge,LLM: PROCESSING state
|
||||
Bridge->>LLM: Final transcription -> LLM prompt
|
||||
LLM->>Bridge: Response text (streamed)
|
||||
|
||||
Note over Bridge,Spk: SPEAKING state
|
||||
Bridge->>TTS: Response text
|
||||
TTS->>Bridge: OPUS audio stream
|
||||
Bridge->>WS: {"type":"tts","state":"start"}
|
||||
loop TTS audio chunks
|
||||
Bridge->>WS: OPUS-encoded binary frames
|
||||
WS->>Spk: Decode OPUS -> PCM -> I2S1
|
||||
end
|
||||
Bridge->>WS: {"type":"tts","state":"end"}
|
||||
|
||||
Note over Mic,Spk: Back to LISTENING
|
||||
```
|
||||
|
||||
## XiaoZhi WebSocket Protocol
|
||||
|
||||
The voice pipeline follows the [XiaoZhi-ESP32](https://github.com/78/xiaozhi-esp32) WebSocket protocol, which is well-documented and battle-tested.
|
||||
|
||||
### Client -> Server Messages
|
||||
|
||||
| Type | Format | Description |
|
||||
|------|--------|-------------|
|
||||
| Hello | `{"type":"hello","version":1,"wakeword":"..."}` | Session start after wake detection |
|
||||
| Audio | Binary (OPUS frames) | 20ms OPUS-encoded audio at 16kHz mono |
|
||||
| Abort | `{"type":"abort"}` | Cancel current interaction |
|
||||
| End of speech | `{"type":"end_of_speech"}` | VAD detected end of utterance |
|
||||
|
||||
### Server -> Client Messages
|
||||
|
||||
| Type | Format | Description |
|
||||
|------|--------|-------------|
|
||||
| STT result | `{"type":"stt","text":"..."}` | Interim/final transcription |
|
||||
| LLM text | `{"type":"llm","text":"...","emotion":"..."}` | Streamed LLM response |
|
||||
| TTS control | `{"type":"tts","state":"start\|end"}` | TTS playback boundaries |
|
||||
| TTS audio | Binary (OPUS frames) | OPUS-encoded speech audio |
|
||||
| Emotion | `{"type":"emotion","value":"happy\|thinking\|..."}` | Facial expression hint |
|
||||
|
||||
### Audio Encoding
|
||||
|
||||
- **Codec:** OPUS (RFC 6716)
|
||||
- **Frame size:** 20ms
|
||||
- **Sample rate:** 16kHz
|
||||
- **Channels:** Mono
|
||||
- **Bitrate:** ~16kbps (voice-optimized)
|
||||
- **Library:** `libopus` via ESP-IDF component or bundled in esp-sr
|
||||
|
||||
## Server-Side Requirements
|
||||
|
||||
### mascarade Bridge Endpoint
|
||||
|
||||
The mascarade bridge (running on GrosMac or VM) needs a `/voice` WebSocket endpoint that:
|
||||
|
||||
1. Receives OPUS audio frames from ESP32
|
||||
2. Decodes OPUS -> PCM
|
||||
3. Runs STT (Whisper via faster-whisper or whisper.cpp)
|
||||
4. Sends transcription to LLM pipeline (existing mascarade flow)
|
||||
5. Sends LLM response text to TTS
|
||||
6. Encodes TTS output as OPUS
|
||||
7. Streams OPUS frames back to ESP32
|
||||
|
||||
This endpoint does not exist yet in mascarade. It will be added as a new route in the bridge service.
|
||||
|
||||
### TTS Services
|
||||
|
||||
Two TTS options are provided via Docker Compose (`tools/dev/docker-compose.tts.yml`):
|
||||
|
||||
**Piper TTS (recommended for low latency)**
|
||||
- CPU-only, runs on VM (192.168.0.119)
|
||||
- French voice: `fr_FR-siwis-medium` or `fr_FR-upmc-medium`
|
||||
- Latency: ~200ms for short sentences
|
||||
- OpenAI-compatible API on port 8000
|
||||
|
||||
**Coqui XTTS-v2 (for voice cloning)**
|
||||
- GPU required, deploy on KXKM-AI (RTX 4090)
|
||||
- Can clone Professeur Zacus's voice from ~30s of sample audio
|
||||
- Higher quality but higher latency (~1-2s)
|
||||
- API on port 5002
|
||||
|
||||
### STT Service
|
||||
|
||||
- **Recommended:** faster-whisper with `large-v3` model
|
||||
- **Language:** French (`--language fr`)
|
||||
- **Deploy on:** KXKM-AI for GPU inference, or VM for CPU (slower)
|
||||
- **Alternative:** whisper.cpp for CPU-optimized inference
|
||||
|
||||
## Custom Wake Word — Espressif Process
|
||||
|
||||
To get a custom "Professeur Zacus" wake word model for WakeNet9:
|
||||
|
||||
1. **Contact Espressif** via https://www.espressif.com/en/contact-us
|
||||
2. **Provide:**
|
||||
- Target phrase: "Professeur Zacus"
|
||||
- Language: French
|
||||
- 200+ audio recordings of the phrase (diverse speakers, environments)
|
||||
- Target platform: ESP32-S3 with WakeNet9
|
||||
3. **Timeline:** 2-4 weeks
|
||||
4. **Cost:** Varies (may be free for open-source/educational projects)
|
||||
5. **Delivery:** `.wn9` model file to flash to partition
|
||||
|
||||
### Recording Tips for Training Data
|
||||
|
||||
- Record in the target environment (exhibition space)
|
||||
- Include male/female voices, different ages
|
||||
- Include background noise variations
|
||||
- Record at 16kHz, 16-bit, mono WAV
|
||||
- Minimum 200 recordings, ideally 500+
|
||||
- Use the INMP441 mic itself for best acoustic match
|
||||
|
||||
## French Voice Command Strategy
|
||||
|
||||
**Local MultiNet is NOT recommended for French** because:
|
||||
- MultiNet only supports Chinese, English, and a few other languages
|
||||
- French phonetics are complex (liaisons, nasals, silent letters)
|
||||
- Limited command vocabulary even in supported languages
|
||||
|
||||
**Strategy: Server-side ASR for all speech recognition**
|
||||
|
||||
1. Wake word detection: Local (WakeNet9, language-agnostic acoustic model)
|
||||
2. All speech-to-text: Server-side via Whisper (excellent French support)
|
||||
3. Intent parsing: LLM-based via mascarade (natural language understanding)
|
||||
4. No local MultiNet needed — everything after wake word goes to server
|
||||
|
||||
This simplifies the ESP32 firmware (no MultiNet model in flash) and gives unlimited French vocabulary through the LLM.
|
||||
|
||||
## Migration Path — Arduino to ESP-IDF
|
||||
|
||||
### Phase 1: Scaffold (current)
|
||||
- Voice pipeline header + stubs in Arduino framework
|
||||
- No ESP-SR dependency
|
||||
- Compiles and runs (functions return safe defaults)
|
||||
|
||||
### Phase 2: Arduino-as-Component
|
||||
- Switch PlatformIO to `framework = arduino, espidf`
|
||||
- Add esp-sr as IDF component
|
||||
- Implement I2S mic input and AFE initialization
|
||||
- Test wake word detection with "Hi Lexin" placeholder
|
||||
- Keep all existing Arduino code unchanged
|
||||
|
||||
### Phase 3: WebSocket Streaming
|
||||
- Add OPUS encoder (IDF component)
|
||||
- Implement WebSocket client for mascarade bridge
|
||||
- Stream audio on wake detection
|
||||
- Receive and play TTS responses
|
||||
|
||||
### Phase 4: Production
|
||||
- Order custom "Professeur Zacus" wake word
|
||||
- Full ESP-IDF migration (optional, for memory optimization)
|
||||
- AEC tuning (echo cancellation between speaker and mic)
|
||||
- LED/display feedback during voice states
|
||||
|
||||
## Memory Budget
|
||||
|
||||
| Component | RAM Type | Size | Notes |
|
||||
|-----------|----------|------|-------|
|
||||
| WakeNet9 | PSRAM | ~340 KB | Single wake word model |
|
||||
| AFE (1ch) | PSRAM | ~1.1 MB | AEC + noise suppression + VAD |
|
||||
| OPUS codec | Internal | ~50 KB | Encoder + decoder |
|
||||
| Audio buffers | PSRAM | ~64 KB | Ring buffers for I2S + WebSocket |
|
||||
| **Total voice** | **Mixed** | **~1.5 MB** | |
|
||||
| LVGL + UI | PSRAM | ~2 MB | Existing allocation |
|
||||
| Camera | PSRAM | ~500 KB | JPEG buffer |
|
||||
| **System total** | **PSRAM** | **~4 MB** | Of 8 MB available |
|
||||
|
||||
## File Structure
|
||||
|
||||
```
|
||||
ESP32_ZACUS/ui_freenove_allinone/
|
||||
├── include/voice/
|
||||
│ └── voice_pipeline.h # Public API (this scaffold)
|
||||
├── src/voice/
|
||||
│ └── voice_pipeline.cpp # Stub implementation
|
||||
└── ...
|
||||
|
||||
docs/voice/
|
||||
└── VOICE_PIPELINE_GUIDE.md # This document
|
||||
|
||||
tools/dev/
|
||||
└── docker-compose.tts.yml # TTS services (Piper + XTTS-v2)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [XiaoZhi-ESP32](https://github.com/78/xiaozhi-esp32) — Open-source voice assistant for ESP32-S3
|
||||
- [ESP-SR Programming Guide](https://docs.espressif.com/projects/esp-sr/en/latest/) — WakeNet, MultiNet, AFE
|
||||
- [Piper TTS](https://github.com/rhasspy/piper) — Fast local TTS with French models
|
||||
- [Coqui XTTS-v2](https://github.com/coqui-ai/TTS) — Voice cloning TTS
|
||||
- [faster-whisper](https://github.com/guillaumekln/faster-whisper) — CTranslate2-based Whisper
|
||||
- [OPUS codec](https://opus-codec.org/) — Low-latency audio codec
|
||||
Reference in New Issue
Block a user