feat: AI integration — voice pipeline, hints engine, MCP server, analytics, security

- Voice pipeline: ESP32 WebSocket client → voice bridge → LLM → Piper TTS (Tower :8001)
- Hints engine: 3 puzzles (LA_440, LEFOU_PIANO, QR_FINALE), anti-cheat, 3 hint levels
- MCP hardware server: 6 tools (puzzle, audio, LED, camera, scenario, status), stdio transport
- Analytics: ESP32 module + 6 web endpoints + Dashboard UI with chat interface
- Security: auth middleware (Bearer NVS), rate limiting, input validation on 30 endpoints
- Frontend: code-split (1.1MB → 210KB initial), ErrorBoundary, API timeout, WS reconnect
- Tests: 24 Python + 38 TypeScript + 18 MCP = 80 project tests (+ 19 mascarade)
- Specs: AI_INTEGRATION_SPEC, MCP_HARDWARE_SERVER_SPEC, QA_TEST_MATRIX_SPEC
- Docs: SECURITY, DEPLOYMENT_RUNBOOK, voice pipeline guide, AI architecture map
- 6 AI agent definitions (.github/agents/ai_*.md)
- TUI orchestration script (tools/dev/zacus_tui.py)
- Docker compose TTS for Tower + KXKM-AI
- CHANGELOG, README, mkdocs.yml updated
- Cycle detection (DFS) in runtime3 validator
- Sprint plan: plans/SPRINT_AI_INTEGRATION.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
L'électron rare
2026-03-22 13:52:45 +01:00
parent c2fc583bae
commit 20aed903ba
135 changed files with 11028 additions and 5989 deletions
+3 -3
View File
@@ -123,9 +123,9 @@ Coordonner les agents et developpeurs pour:
- `python3 tools/scenario/export_md.py game/scenarios/zacus_v2.yaml`
- `python3 tools/audio/validate_manifest.py audio/manifests/zacus_v2_audio.yaml`
- `python3 tools/printables/validate_manifest.py printables/manifests/zacus_v2_printables.yaml`
- `npm --prefix 'fronted dev web UI' run lint`
- `npm --prefix 'fronted dev web UI' run build`
- `npm --prefix 'fronted dev web UI' run test:unit -- --run`
- `npm --prefix 'frontend-scratch-v2' run lint`
- `npm --prefix 'frontend-scratch-v2' run build`
- importer/exporter YAML + IR Runtime 3 sans perte fonctionnelle
## Regle de handoff entre agents
Chaque ticket passe seulement avec ces evidences:
+365
View File
@@ -0,0 +1,365 @@
# AI Integration Specification
## Status
- State: draft
- Date: 2026-03-21
- Depends on: `ZACUS_RUNTIME_3_SPEC.md`, `MCP_HARDWARE_SERVER_SPEC.md`, `FIRMWARE_WEB_DATA_CONTRACT.md`
## 1) Objective
Integrate AI capabilities into the Zacus escape room platform across three tiers:
- **On-device** (ESP32-S3): low-latency voice commands and object detection
- **Server** (mascarade + Docker): LLM reasoning, TTS voice cloning, MCP orchestration
- **GPU** (KXKM-AI RTX 4090): generative audio, model fine-tuning
The AI layer enriches the escape room without replacing the deterministic Runtime 3 scenario engine. AI features degrade gracefully: if the server is unreachable, the game continues with pre-recorded audio and static hints.
## 2) Architecture Overview
```mermaid
flowchart TD
subgraph ESP32["ESP32-S3 On-Device AI"]
MIC[Microphone I2S]
SPK[Speaker I2S]
CAM[Camera OV2640]
SR[ESP-SR v2.0<br/>Wake Word + Commands]
DL[ESP-DL v3.2<br/>Object Detection]
RT[Runtime 3 Engine]
end
subgraph Server["mascarade Server (VM / GrosMac)"]
API[mascarade API<br/>LLM orchestration]
TTS[Coqui XTTS-v2<br/>Voice Cloning Docker]
MCP[MCP Hardware Server<br/>Tool dispatch]
ADAPT[Adaptive Difficulty<br/>Analytics Engine]
end
subgraph GPU["KXKM-AI (RTX 4090 24GB)"]
MUSIC[AudioCraft MusicGen<br/>Ambient + SFX]
TRAIN[Fine-tune Pipeline<br/>Unsloth + SimPO]
end
MIC --> SR
CAM --> DL
SR -->|"voice command"| RT
SR -->|"complex query WiFi"| MCP
DL -->|"object detected"| RT
DL -->|"detection event"| MCP
RT -->|"hint request"| API
MCP <-->|"JSON-RPC 2.0"| API
API -->|"hint text"| TTS
TTS -->|"PCM audio stream"| SPK
MUSIC -->|"ambient MP3 pre-gen"| SPK
RT -->|"telemetry"| ADAPT
ADAPT -->|"difficulty params"| API
TRAIN -.->|"updated models"| SR
TRAIN -.->|"updated models"| DL
```
## 3) On-Device AI
### 3.1 ESP-SR v2.0 — Wake Word + Voice Commands
**Purpose**: Hands-free interaction during gameplay. Players say "Hey Zacus" then a command.
| Parameter | Value |
|-----------|-------|
| Framework | ESP-SR v2.0 (WakeNet + MultiNet) |
| Wake word | "Hey Zacus" (custom trained, WakeNet Q8) |
| Command vocabulary | 50 French commands (MultiNet, expandable to 300) |
| Latency | < 200 ms wake detection, < 500 ms command recognition |
| Memory | ~280 KB PSRAM (WakeNet 120 KB + MultiNet 160 KB) |
| Audio format | 16-bit PCM, 16 kHz, mono |
| Microphone | INMP441 I2S MEMS |
**Command categories**:
- Navigation: "indice", "aide", "repeter", "suivant"
- Puzzle control: "valider", "annuler", "recommencer"
- Meta: "temps restant", "score", "pause"
**Fallback**: If wake word detection fails 3 times, the UI displays a tap-to-talk button.
### 3.2 ESP-DL v3.2 — Object Detection
**Purpose**: Detect physical puzzle props placed in front of the camera.
| Parameter | Value |
|-----------|-------|
| Framework | ESP-DL v3.2 |
| Model | YOLOv11n quantized (INT8) |
| Input | 320x240 RGB from OV2640 |
| FPS | 5-7 FPS inference |
| Memory | ~450 KB PSRAM (model) + 150 KB (input buffer) |
| Classes | 8 custom (fiole, clef, parchemin, cristal, engrenage, miroir, boussole, amulette) |
| Confidence threshold | 0.65 |
**Detection events**:
```json
{
"event_type": "object_detected",
"event_name": "DETECT_FIOLE",
"class": "fiole",
"confidence": 0.82,
"bbox": [45, 60, 180, 220],
"timestamp_ms": 1234567890
}
```
These events feed into Runtime 3 transitions like any other `event_type`.
### 3.3 Memory Budget (ESP32-S3, 8MB PSRAM)
| Component | PSRAM | Internal SRAM |
|-----------|-------|---------------|
| ESP-SR (WakeNet + MultiNet) | 280 KB | 12 KB |
| ESP-DL (YOLOv11n INT8) | 600 KB | 20 KB |
| LVGL UI | 96 KB | 8 KB |
| Audio DMA buffers | 64 KB | 4 KB |
| Runtime 3 engine | 48 KB | 16 KB |
| Network stack (WiFi + HTTP) | 80 KB | 32 KB |
| LittleFS cache | 32 KB | — |
| **Total used** | **1,200 KB** | **92 KB** |
| **Available** | 8,192 KB | 512 KB |
| **Headroom** | 85% | 82% |
### 3.4 Task Priority (FreeRTOS)
| Task | Priority | Core | Stack |
|------|----------|------|-------|
| Audio I2S (DMA) | 24 | 1 | 4 KB |
| ESP-SR inference | 20 | 1 | 8 KB |
| ESP-DL inference | 18 | 0 | 8 KB |
| Runtime 3 loop | 15 | 0 | 8 KB |
| LVGL tick | 12 | 0 | 4 KB |
| WiFi/HTTP | 10 | 0 | 6 KB |
| Idle | 0 | * | 2 KB |
## 4) Server-Side AI
### 4.1 Coqui XTTS-v2 — Voice Cloning
**Purpose**: Generate dynamic narration in Professor Zacus's voice.
| Parameter | Value |
|-----------|-------|
| Model | XTTS-v2 (Coqui) |
| Deployment | Docker container on mascarade VM |
| Reference sample | 6-second WAV of Zacus voice |
| Output format | PCM 22050 Hz 16-bit mono |
| Latency target | < 2 s for 20-word sentence |
| Language | French (fr) |
| Streaming | Chunked HTTP response (256-sample chunks) |
| GPU required | No (CPU inference acceptable for short utterances) |
| Memory | ~2 GB container |
**API endpoint** (Docker internal):
```
POST /api/tts
Content-Type: application/json
{
"text": "Bravo, vous avez trouve la fiole sacree!",
"speaker_wav": "/data/voices/zacus_ref.wav",
"language": "fr"
}
Response: audio/wav stream
```
**Integration with mascarade**: The MCP server calls TTS after receiving hint text from the LLM. Audio is streamed to the ESP32 via chunked HTTP.
### 4.2 LLM Adaptive Hints via mascarade API
**Purpose**: Context-aware, anti-cheat hints personalized to player progress.
**Flow**:
1. ESP32 sends hint request with context (current step, elapsed time, failed attempts)
2. mascarade API routes to configured LLM provider
3. System prompt enforces anti-spoiler rules
4. Response text is sent to TTS for voice synthesis
5. Difficulty parameters adjust based on analytics
**Request format** (ESP32 -> mascarade):
```json
{
"endpoint": "/api/v1/send",
"payload": {
"provider": "ollama",
"model": "mascarade-coder",
"messages": [
{
"role": "system",
"content": "Tu es le Professeur Zacus. Donne un indice sans reveler la solution. Adapte le niveau: {{difficulty}}."
},
{
"role": "user",
"content": "Nous sommes bloques a l'etape {{step_id}} depuis {{elapsed_min}} minutes. Tentatives: {{attempts}}."
}
]
}
}
```
**Anti-cheat prompt engineering** (ref: devlinb/escaperoom):
- Never reveal full solutions
- Escalate hints progressively (vague -> specific -> near-answer)
- Maximum 3 hints per puzzle per session
- Log all hint requests for game master review
**Latency target**: < 3 s end-to-end (LLM + TTS + network).
### 4.3 MCP Hardware Server
See `MCP_HARDWARE_SERVER_SPEC.md` for full specification.
The MCP server exposes ESP32 hardware as LLM-callable tools:
- `puzzle_set_state` — lock/unlock puzzle elements
- `audio_play` — trigger audio on device speakers
- `led_set` — control LED strips (color, pattern, brightness)
- `camera_capture` — take a snapshot from OV2640
- `scenario_advance` — trigger a Runtime 3 transition
## 5) GPU AI (KXKM-AI)
### 5.1 AudioCraft MusicGen — Generative Audio
**Purpose**: Generate ambient music and sound effects per room/puzzle.
| Parameter | Value |
|-----------|-------|
| Model | MusicGen-small (300M) or MusicGen-medium (1.5B) |
| Hardware | KXKM-AI, RTX 4090 24 GB, 62 GB RAM |
| Generation mode | Pre-generation (not real-time) |
| Output format | WAV 32 kHz stereo, converted to MP3 128 kbps for ESP32 |
| Duration | 30-60 s loops per room |
| Prompt template | "atmospheric mysterious escape room music, {{room_theme}}, ambient, looping" |
| Latency | ~10 s per 30 s clip (offline batch) |
**Workflow**:
1. Game designer specifies room themes in scenario YAML
2. Batch generation script produces ambient tracks on KXKM-AI
3. Tracks are transcoded to MP3 128 kbps mono (ESP32 compatible)
4. Uploaded to LittleFS or served via HTTP
5. Runtime 3 `audio_pack_id` references generated tracks
**SFX generation** (Stable Audio Open):
- Short effect sounds (unlock, alarm, discovery)
- 2-5 s duration
- Triggered by Runtime 3 events
### 5.2 Fine-Tune Pipeline
| Parameter | Value |
|-----------|-------|
| Base model | Qwen2.5-Coder-1.5B |
| Method | Unsloth + SimPO |
| Dataset | Custom Zacus hint pairs + Magicoder-OSS-Instruct-75K |
| Training time | ~6 min on RTX 4090 |
| Output | GGUF Q4_K_M (~941 MB) deployed to Ollama on VM |
| Trigger | P2P `distribute_task` via mascarade mesh |
## 6) Data Flow Summary
```mermaid
sequenceDiagram
participant P as Player
participant E as ESP32-S3
participant M as mascarade API
participant T as Coqui XTTS-v2
participant K as KXKM-AI
Note over E: On-device AI (ESP-SR, ESP-DL)
P->>E: "Hey Zacus, un indice"
E->>E: ESP-SR: wake + command parse
E->>M: POST /api/v1/send (hint request + context)
M->>M: LLM generates hint text
M->>T: POST /api/tts (hint text, zacus voice)
T-->>E: Chunked audio stream (PCM)
E->>P: Speaker plays Zacus voice hint
Note over K: Pre-generation (offline)
K->>K: MusicGen batch: room ambients
K-->>E: MP3 files via HTTP/LittleFS upload
```
## 7) Latency Targets
| Path | Target | Acceptable | Notes |
|------|--------|-----------|-------|
| Wake word detection | < 200 ms | < 500 ms | On-device, no network |
| Voice command recognition | < 500 ms | < 1 s | On-device, MultiNet |
| Object detection (single frame) | < 200 ms | < 400 ms | On-device, ESP-DL |
| LLM hint (text only) | < 2 s | < 4 s | Network + LLM inference |
| TTS synthesis (20 words) | < 2 s | < 4 s | Server CPU |
| End-to-end voice hint | < 3 s | < 6 s | Wake -> LLM -> TTS -> speaker |
| Ambient music start | < 500 ms | < 1 s | Pre-loaded MP3 |
## 8) Graceful Degradation
| Failure | Fallback |
|---------|----------|
| WiFi disconnected | Pre-recorded hints from LittleFS, no LLM |
| mascarade API down | Cached hint bank (3 hints per puzzle in JSON) |
| TTS service down | LLM text displayed on LVGL screen |
| ESP-SR model corrupt | Tap-to-talk UI button, no voice |
| ESP-DL model corrupt | QR code scanning fallback for object validation |
| KXKM-AI offline | Pre-generated ambient tracks already on device |
## 9) Phase Rollout Plan
### Phase A: Security Foundations (P0 — 1-2 weeks)
- NVS credential storage (replace hardcoded WiFi)
- Bearer token auth on all API endpoints
- Input validation + rate limiting
- LVGL pool increase 54 -> 96 KB
- Arduino stack increase 16 -> 24 KB
### Phase B: Voice Pipeline (P1 — 2-4 weeks)
1. Integrate ESP-SR v2.0 WakeNet custom wake word
2. Train "Hey Zacus" model with ESP-SR training toolkit
3. Deploy Coqui XTTS-v2 Docker on mascarade VM
4. Implement chunked audio streaming ESP32 <- Server
5. Add MultiNet command vocabulary (50 FR commands)
6. Reference architecture: XiaoZhi ESP32
### Phase C: Vision & Detection (P1 — 2-4 weeks)
1. Integrate ESP-DL v3.2 with quantized YOLOv11n
2. Collect and annotate prop dataset (8 classes, 500+ images)
3. Train custom model, export INT8 for ESP32
4. Wire detection events into Runtime 3 transitions
5. Face detection for player counting (ESP-WHO)
### Phase D: LLM Adaptive Hints (P2 — 4-6 weeks)
1. Design hint prompt templates with anti-spoiler rules
2. Implement hint request API in firmware HTTP client
3. Add analytics telemetry (step timing, attempts, hint count)
4. Build adaptive difficulty engine in mascarade
5. Professor Zacus as NPC LLM with conversation memory
### Phase E: Generative Audio (P2 — 2-3 weeks)
1. Deploy AudioCraft MusicGen on KXKM-AI
2. Create room theme prompts from scenario YAML
3. Batch generate ambient tracks (30-60 s loops)
4. Transcode to MP3 128 kbps mono for ESP32
5. SFX generation with Stable Audio Open
### Phase F: MCP & Orchestration (P3 — 4-6 weeks)
1. Implement MCP hardware server (see `MCP_HARDWARE_SERVER_SPEC.md`)
2. Register in mascarade MCP registry
3. Natural language hardware control for game masters
4. Real-time game master dashboard
## 10) Dependencies
| Dependency | Version | License | Source |
|------------|---------|---------|--------|
| ESP-SR | v2.0 | Espressif | espressif/esp-sr |
| ESP-DL | v3.2 | MIT | espressif/esp-dl |
| Coqui XTTS-v2 | latest | MPL-2.0 | coqui-ai/TTS |
| AudioCraft MusicGen | latest | MIT / CC-BY-NC-4.0 | facebookresearch/audiocraft |
| Stable Audio Open | latest | Stability AI CLA | stabilityai/stable-audio-open |
| mascarade API | main | Private | electron-rare/mascarade |
| Ollama | latest | MIT | ollama/ollama |
| Qwen2.5-Coder-1.5B | latest | Apache 2.0 | Qwen |
| Unsloth | latest | Apache 2.0 | unslothai/unsloth |
+1 -1
View File
@@ -17,7 +17,7 @@ Le bundle conversationnel est maintenu dans :
- Aucune refonte des manifests audio/printables.
## Risques
- Le bundle peut diverger du scénario canonique `game/scenarios/zacus_v1.yaml`.
- Le bundle peut diverger du scénario canonique `game/scenarios/zacus_v2.yaml`.
- Les formats `scenario_runtime.json`/templates peuvent nécessiter une validation dédiée ultérieure.
## Prochaine étape proposée
+5 -4
View File
@@ -27,13 +27,14 @@ Le firmware reste **intouché** par cette spécification.
- contraintes éditoriales
- configuration médias/printables (formats demandés)
- Sorties attendues :
- `YAML` conforme **Story V2** pour `game/scenarios/*.yaml`
- `YAML` canonique pour `game/scenarios/*.yaml`
- `IR JSON` conforme **Zacus Runtime 3** pour les previews et adaptateurs firmware
- `manifest_yaml` + `markdown` pour le bundle imprimables
- objet `diagnostic` (`rationale`, `source`, `warnings`)
- Services inclus :
- Docker LLM local (Ollama)
- Gateway HTTP locale (script Python)
- Frontend web Studio (React/Svelte selon stack du dépôt)
- Frontend web Studio React + Blockly (`frontend-scratch-v2`)
## 4) Architecture cible
@@ -220,9 +221,9 @@ curl http://127.0.0.1:8787/health
```
```bash
cp "fronted dev web UI/.env.local.example" "fronted dev web UI/.env.local"
cp "frontend-scratch-v2/.env.local.example" "frontend-scratch-v2/.env.local"
# VITE_ZACUS_STUDIO_AI_URL=http://127.0.0.1:8787/story_generate
npm --prefix "fronted dev web UI" run dev
npm --prefix "frontend-scratch-v2" run dev
```
### One-liner
+503
View File
@@ -0,0 +1,503 @@
# MCP Hardware Server Specification
## Status
- State: draft
- Date: 2026-03-21
- Depends on: `AI_INTEGRATION_SPEC.md`, `FIRMWARE_WEB_DATA_CONTRACT.md`
- Reference: MCP specification (modelcontextprotocol.io), ESP RainMaker MCP pattern, IoT-MCP (Duke CEI)
## 1) Objective
Define an MCP (Model Context Protocol) server that exposes ESP32-S3 hardware capabilities as LLM-callable tools. This enables natural-language hardware control from mascarade, game master dashboards, and automated scenario orchestration.
## 2) Architecture
```mermaid
flowchart LR
subgraph LLM["mascarade (LLM Host)"]
Client[MCP Client<br/>call_tool_http]
Registry[MCP Registry]
end
subgraph MCPServer["MCP Hardware Server (Python)"]
Transport[HTTP Transport<br/>JSON-RPC 2.0]
Auth[Bearer Token Auth]
Router[Tool Router]
end
subgraph ESP32["ESP32-S3 Devices"]
API1[Device 1 HTTP API<br/>:8080]
API2[Device 2 HTTP API<br/>:8080]
ESPNOW[ESP-NOW Mesh]
end
Client -->|"JSON-RPC 2.0"| Transport
Registry -.->|"discovery"| Transport
Transport --> Auth --> Router
Router -->|"HTTP REST"| API1
Router -->|"HTTP REST"| API2
API1 <-->|"ESP-NOW"| API2
API1 <-->|"ESP-NOW"| ESPNOW
```
## 3) Transport
### 3.1 HTTP Transport (Primary)
The MCP server runs as a Python HTTP service registered in mascarade's MCP registry.
| Parameter | Value |
|-----------|-------|
| Protocol | JSON-RPC 2.0 over HTTP POST |
| Endpoint | `POST /mcp` |
| Port | 8790 (configurable via `MCP_HARDWARE_PORT`) |
| Content-Type | `application/json` |
| Auth | Bearer token (same as `MASCARADE_API_KEY`) |
### 3.2 stdio Transport (Local Development)
For local testing, the server also supports stdio transport per MCP spec:
```bash
python -m zacus_mcp_server --transport stdio
```
### 3.3 Registration in mascarade
```python
# mascarade MCP registry entry
{
"name": "zacus-hardware",
"description": "ESP32-S3 escape room hardware control",
"transport": "http",
"url": "http://localhost:8790/mcp",
"auth": {"type": "bearer", "token_env": "MASCARADE_API_KEY"},
"enabled_env": "ZACUS_MCP_ENABLED"
}
```
## 4) Authentication Model
### 4.1 MCP Server Auth (mascarade -> MCP Server)
- Bearer token in `Authorization` header
- Token matches `MASCARADE_API_KEY` environment variable
- Requests without valid token receive `401 Unauthorized`
### 4.2 MCP Server -> ESP32 Auth
- Bearer token in `Authorization` header on ESP32 HTTP API
- Token stored in ESP32 NVS (provisioned at setup)
- Per-device token support for multi-device deployments
### 4.3 Token Hierarchy
```
mascarade API key
└── MCP server validates incoming requests
└── Per-device ESP32 tokens
└── ESP32 validates hardware commands
```
## 5) Tool Definitions
### 5.1 `puzzle_set_state`
Control puzzle lock/unlock state and trigger associated effects.
```json
{
"name": "puzzle_set_state",
"description": "Set the state of a puzzle element (lock, unlock, reset). Triggers associated LED and audio effects.",
"inputSchema": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"description": "Target ESP32 device identifier"
},
"puzzle_id": {
"type": "string",
"description": "Puzzle identifier from scenario runtime",
"enum": ["PUZZLE_FIOLE", "PUZZLE_COFFRE", "PUZZLE_MIROIR", "PUZZLE_ENGRENAGE", "PUZZLE_CRYSTAL", "PUZZLE_BOUSSOLE"]
},
"state": {
"type": "string",
"enum": ["locked", "unlocked", "reset"],
"description": "Target state"
},
"effects": {
"type": "boolean",
"default": true,
"description": "Play associated LED/audio effects on state change"
}
},
"required": ["device_id", "puzzle_id", "state"]
}
}
```
**ESP32 API mapping**: `POST /api/puzzle` with body `{"id": "...", "state": "...", "effects": true}`
### 5.2 `audio_play`
Play audio files or streams on device speakers.
```json
{
"name": "audio_play",
"description": "Play an audio file or stream on the ESP32 speaker. Supports local files (LittleFS) and HTTP URLs.",
"inputSchema": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"description": "Target ESP32 device identifier"
},
"source": {
"type": "string",
"description": "Audio source: LittleFS path (/audio/hint_01.mp3) or HTTP URL"
},
"volume": {
"type": "integer",
"minimum": 0,
"maximum": 100,
"default": 70,
"description": "Playback volume (0-100)"
},
"loop": {
"type": "boolean",
"default": false,
"description": "Loop playback continuously"
},
"action": {
"type": "string",
"enum": ["play", "stop", "pause", "resume"],
"default": "play"
}
},
"required": ["device_id", "source"]
}
}
```
**ESP32 API mapping**: `POST /api/audio` with body `{"src": "...", "vol": 70, "loop": false, "action": "play"}`
### 5.3 `led_set`
Control LED strips and individual LEDs.
```json
{
"name": "led_set",
"description": "Control LED strips: set color, pattern, brightness. Supports WS2812B addressable LEDs.",
"inputSchema": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"description": "Target ESP32 device identifier"
},
"zone": {
"type": "string",
"description": "LED zone identifier",
"enum": ["ambient", "puzzle", "alert", "all"]
},
"color": {
"type": "string",
"description": "Hex color (#RRGGBB) or named color",
"pattern": "^(#[0-9a-fA-F]{6}|red|green|blue|white|off|warm|cold|purple|orange)$"
},
"pattern": {
"type": "string",
"enum": ["solid", "breathe", "chase", "rainbow", "pulse", "off"],
"default": "solid"
},
"brightness": {
"type": "integer",
"minimum": 0,
"maximum": 255,
"default": 128
},
"duration_ms": {
"type": "integer",
"description": "Auto-off after duration (0 = indefinite)",
"default": 0
}
},
"required": ["device_id", "zone", "color"]
}
}
```
**ESP32 API mapping**: `POST /api/led` with body `{"zone": "...", "color": "...", "pattern": "solid", "bright": 128}`
### 5.4 `camera_capture`
Capture a snapshot from the OV2640 camera.
```json
{
"name": "camera_capture",
"description": "Capture a JPEG snapshot from the ESP32 camera. Returns base64-encoded image.",
"inputSchema": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"description": "Target ESP32 device identifier"
},
"resolution": {
"type": "string",
"enum": ["QQVGA", "QVGA", "VGA"],
"default": "QVGA",
"description": "Capture resolution (160x120, 320x240, 640x480)"
},
"quality": {
"type": "integer",
"minimum": 10,
"maximum": 63,
"default": 20,
"description": "JPEG quality (lower = better, 10-63)"
}
},
"required": ["device_id"]
}
}
```
**ESP32 API mapping**: `GET /api/camera?res=QVGA&q=20` returns `image/jpeg`
### 5.5 `scenario_advance`
Trigger a Runtime 3 transition on the device.
```json
{
"name": "scenario_advance",
"description": "Trigger a Runtime 3 scenario transition. Used by game masters to manually advance or reset the game.",
"inputSchema": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"description": "Target ESP32 device identifier"
},
"event_type": {
"type": "string",
"enum": ["button", "serial", "timer", "audio_done", "unlock", "espnow", "action", "manual"],
"description": "Event type per Runtime 3 transition model"
},
"event_name": {
"type": "string",
"description": "Event name token (e.g., UNLOCK_COFFRE, MANUAL_ADVANCE)"
},
"target_step_id": {
"type": "string",
"description": "Optional: force transition to specific step (game master override)"
}
},
"required": ["device_id", "event_type", "event_name"]
}
}
```
**ESP32 API mapping**: `POST /api/scenario/transition` with body `{"event_type": "...", "event_name": "...", "target": "..."}`
### 5.6 `device_status`
Query device health and current state.
```json
{
"name": "device_status",
"description": "Get current device status: free memory, current scenario step, uptime, WiFi RSSI, sensor readings.",
"inputSchema": {
"type": "object",
"properties": {
"device_id": {
"type": "string",
"description": "Target ESP32 device identifier"
}
},
"required": ["device_id"]
}
}
```
**ESP32 API mapping**: `GET /api/status` returns JSON status object
## 6) Message Format (JSON-RPC 2.0)
### 6.1 Request
```json
{
"jsonrpc": "2.0",
"id": "req-001",
"method": "tools/call",
"params": {
"name": "led_set",
"arguments": {
"device_id": "zacus-main",
"zone": "puzzle",
"color": "#00FF00",
"pattern": "pulse",
"brightness": 200
}
}
}
```
### 6.2 Success Response
```json
{
"jsonrpc": "2.0",
"id": "req-001",
"result": {
"content": [
{
"type": "text",
"text": "LED zone 'puzzle' set to #00FF00 pulse at brightness 200 on device zacus-main"
}
]
}
}
```
### 6.3 Error Response
```json
{
"jsonrpc": "2.0",
"id": "req-001",
"error": {
"code": -32000,
"message": "Device unreachable",
"data": {
"device_id": "zacus-main",
"detail": "HTTP timeout after 5000ms"
}
}
}
```
### 6.4 Error Codes
| Code | Meaning |
|------|---------|
| -32700 | Parse error (malformed JSON) |
| -32600 | Invalid request |
| -32601 | Method not found |
| -32602 | Invalid params |
| -32603 | Internal error |
| -32000 | Device unreachable |
| -32001 | Device busy (command in progress) |
| -32002 | Auth failed (ESP32 token) |
| -32003 | Puzzle state conflict |
## 7) Sequence Diagrams
### 7.1 LLM-Driven Puzzle Unlock
```mermaid
sequenceDiagram
participant GM as Game Master
participant M as mascarade LLM
participant MCP as MCP Hardware Server
participant E as ESP32-S3
GM->>M: "Deverrouille le coffre dans la salle 2"
M->>M: Parse intent -> puzzle_set_state
M->>MCP: JSON-RPC tools/call puzzle_set_state
MCP->>MCP: Validate params + auth
MCP->>E: POST /api/puzzle {"id":"PUZZLE_COFFRE","state":"unlocked"}
E->>E: Servo unlock + LED green + SFX
E-->>MCP: 200 OK {"state":"unlocked","effects_played":true}
MCP-->>M: JSON-RPC result
M-->>GM: "Le coffre de la salle 2 est maintenant deverrouille."
```
### 7.2 Voice Command -> Hardware Action
```mermaid
sequenceDiagram
participant P as Player
participant E as ESP32-S3
participant MCP as MCP Hardware Server
participant M as mascarade LLM
participant T as TTS
P->>E: "Hey Zacus, allume la lumiere UV"
E->>E: ESP-SR: wake + command parse
E->>MCP: JSON-RPC tools/call led_set (zone:puzzle, color:purple)
MCP->>E: POST /api/led {"zone":"puzzle","color":"#7B00FF"}
E->>E: UV LED on
E-->>MCP: 200 OK
MCP-->>E: JSON-RPC result
Note over E: Optional: confirm via TTS
E->>M: Hint request: "confirme action UV"
M->>T: TTS "La lumiere UV est activee"
T-->>E: Audio stream
E->>P: Speaker: "La lumiere UV est activee"
```
## 8) Device Discovery
### 8.1 Static Configuration (Phase 1)
Devices are configured in `.env`:
```env
ZACUS_DEVICES='[{"id":"zacus-main","host":"192.168.0.50","port":8080,"token":"..."}]'
```
### 8.2 mDNS Discovery (Phase 2)
ESP32 devices advertise `_zacus._tcp` via mDNS. The MCP server discovers devices automatically:
```
_zacus._tcp.local.
zacus-main._zacus._tcp.local. 8080 TXT "room=main" "version=3.1"
zacus-salle2._zacus._tcp.local. 8080 TXT "room=salle2" "version=3.1"
```
### 8.3 ESP-NOW Mesh (Phase 3)
The primary ESP32 acts as a gateway for ESP-NOW mesh devices. The MCP server sends commands to the gateway, which relays via ESP-NOW to secondary devices.
## 9) Rate Limiting & Safety
| Constraint | Value |
|------------|-------|
| Max requests per device | 10/s |
| Max concurrent tool calls | 3 |
| Command timeout | 5 s |
| Retry on timeout | 1 retry with 2 s backoff |
| Servo actuation cooldown | 500 ms between movements |
| LED transition min interval | 100 ms |
**Safety guards**:
- No two conflicting puzzle state changes within 1 s
- Audio volume hard-capped at device level (not bypassable via MCP)
- Camera capture rate limited to 2/s to prevent overheating
- Game master override always available via `scenario_advance` with `event_type: "manual"`
## 10) Implementation Plan
### Phase 1: Core Server (2 weeks)
- Python MCP server with HTTP transport
- Tool definitions: `puzzle_set_state`, `audio_play`, `led_set`, `device_status`
- Bearer auth, static device config
- Register in mascarade MCP registry
- Unit tests (pytest)
### Phase 2: Camera + Scenario (2 weeks)
- `camera_capture` tool with base64 response
- `scenario_advance` tool with Runtime 3 integration
- mDNS device discovery
- Integration tests with ESP32 hardware
### Phase 3: Mesh + Dashboard (3 weeks)
- ESP-NOW mesh relay through gateway device
- Real-time game master dashboard (WebSocket feed)
- Multi-device orchestration (synchronized effects)
- Load testing and latency benchmarks
+47
View File
@@ -0,0 +1,47 @@
# Handoff - Équipe Front-end (Media Manager)
## Contexte
- FSM de travail: `scenario-ai-coherence/zacus_conversation_bundle_v3/fsm_mermaid.md`
- Contrat media manager: `specs/MEDIA_MANAGER_RUNTIME_SPEC.md`
- Scope: aucune modification firmware.
## Objectif métier
- Exposer le Media Manager de manière fiable à larrivée en fin de scénario.
- Accepter les identifiants step/scene mixtes sans casser lUI.
- Sappuyer sur les APIs media existantes pour listage, lecture et enregistrement.
## Cibles de sortie (artefacts)
- `artifacts/runtime-sync/<date>/media-manager-web-checks.md`
- `artifacts/runtime-sync/<date>/media-manager-front-mapping.md`
- notes de migration/ajustement de parsing si nécessaire.
- Spec détaillée d'intégration: `specs/MEDIA_MANAGER_FRONTEND_SPEC.md`
## Actions obligatoires
1. Parsing statut:
- lire `story.screen` en priorité.
- fallback tolérant sur `story.step == STEP_MEDIA_MANAGER`.
- ne pas supposer `step_id` prefixé `STEP_` ou `SCENE_`.
2. Mapping Media Hub:
- activer vue hub quand `story.screen == SCENE_MEDIA_MANAGER`.
- si non dispo, fallback `story.step == STEP_MEDIA_MANAGER`.
3. Endpoints media:
- `/api/media/files` (`kind` music/picture/recorder)
- `/api/media/play`, `/api/media/stop`
- `/api/media/record/start`, `/api/media/record/stop`
- `/api/media/record/status`
4. Gestion erreurs:
- gérer `ok=false` et `error` depuis `/api/control` et endpoints media.
- afficher `media.last_error` en debug écran media.
5. Affichage media:
- afficher clairement `media.record_simulated` pour éviter les ambiguïtés denregistrement réel.
## Critères dacceptation
- Le Media Hub saffiche à la fin de scénario sans crash.
- `SCENE_MEDIA_MANAGER` déclenche bien la vue média même avec `STEP_MEDIA_MANAGER` possible en fallback.
- `media.playing` bascule correctement via play/stop.
- Les erreurs API sont présentées de façon exploitable par lopérateur.
## Remarques équipe
- Ne pas changer les contrats runtime, seulement adapter la consommation UI.
- Toute divergence métier (ex: cible `SCENE_MEDIA_MANAGER` encore présente) doit être reportée via ticket "scenario".
- Implémentation front alignée au studio React + Blockly (`frontend-scratch-v2`) avec consommation API côté UI.
+120
View File
@@ -0,0 +1,120 @@
# Spécification Frontend — Media Manager (Mode final)
## Contexte
- Source de vérité runtime: `hardware/firmware/data/story/scenarios/DEFAULT.json`
- FSM de travail: `scenario-ai-coherence/zacus_conversation_bundle_v3/fsm_mermaid.md`
- Handoff frontend: `specs/MEDIA_MANAGER_FRONTEND_HANDOFF.md`
- Contrat media runtime: `specs/MEDIA_MANAGER_RUNTIME_SPEC.md`
Ce document formalise **ce que le frontend doit consommer** pour être robuste quand le scénario arrive sur la phase finale media manager.
## 1) Objectifs
1. Détecter de façon fiable lentrée Media Manager malgré la mixité `STEP_*`/`SCENE_*`.
2. Consommer les endpoints médias sans hypothèses sur lextension de fichier.
3. Exposer des erreurs exploitables (`ok=false`, `error`) sans crash UI.
4. Gérer le cas de lock NVS via laffichage de statut de boot mode.
## 2) Détection d’écran / état actif
### 2.1 Règle de détection (priorité)
Le frontend doit considérer que le Media Manager est actif si lune des conditions est vraie:
1. `story.screen === "SCENE_MEDIA_MANAGER"`
2. `story.step === "STEP_MEDIA_MANAGER"`
3. fallback legacy: payload status expose un champ équivalent `step`/`scene`/`currentStep` contenant `MEDIA_MANAGER`.
### 2.2 Tolérance IDs mixtes
- Ne jamais supposer que `step` commence par `STEP_`.
- Ne jamais supposer que `screen` commence par `SCENE_`.
- Conserver les deux champs en mémoire pour les logs daudit.
## 3) Modèle de données utilisé par le frontend
### 3.1 Polling / stream
- Le frontend peut continuer avec la mécanique actuelle (stream en live si dispo, sinon polling).
- Le payload de référence doit contenir au minimum:
- `story.scenario`
- `story.step`
- `story.screen`
- `media.ready`
- `media.playing`
- `media.recording`
- `media.record_simulated`
- `media.last_error`
- `media.record_limit_seconds`
- `media.record_elapsed_seconds`
### 3.2 Priorité daffichage
1. `story.screen` prend le pas sur `story.step` pour lUI.
2. `media.last_error` doit être prioritaire sur le message générique du formulaire daction.
## 4) Contrat API (front) à implémenter côté UI
### 4.1 Endpoints requis
- `GET /api/media/files?kind=music|picture|recorder`
- réponse attendue 200: `{ ok: true, kind: "...", files: ["/music/a.mp3"] }`
- réponse 400 sur kind invalide: `{ ok: false, kind: "...", error: "invalid_kind" }`
- `POST /api/media/play` body:
- `{ "path": "/music/file.mp3" }` ou `{ "file": "file.mp3" }`
- `POST /api/media/stop`
- `POST /api/media/record/start` body:
- `{ "seconds": 20, "filename": "take_1.wav" }`
- `POST /api/media/record/stop`
- `GET /api/media/record/status` (ou récupération de `media` dans `/api/status`)
- `POST /api/control` fallback pour compatibilité opérationnelle.
### 4.2 Format de succès/erreur UI
- Traiter tout appel média comme opération tri-state:
- succès: `{ ok: true, action: "...", ... }`
- erreur: `{ ok: false, error: "...", action: "..." }`
- En cas derreur, afficher immédiatement `media.last_error` + statut de commande.
## 5) Comportements de l’écran Media Hub
### 5.1 Listing
- Charger **les 3 catégories** `music`, `picture`, `recorder` en parallèle ou séquentiel.
- Afficher une section par catégorie quand disponible.
- Aucun filtrage dextension dur codé côté UI.
### 5.2 Lecture
- Le clic sur un item déclenche `MEDIA_PLAY`.
- Interdire un replay simultané si `media.playing === true` sans `stop` explicite préalable.
- Le changement d’état doit refléter `media.playing` de lAPI.
### 5.3 Enregistrement
- Bouton denregistrement -> `MEDIA_RECORD_START`.
- Respecter le `media.record_limit_seconds` du runtime.
- Afficher `record_simulated` pour expliciter la nature simulée/placeholder du capture path.
### 5.4 Erreurs
- Si `media.last_error` est non vide: montrer un encadré derreur persistante avec timestamp et action recommandée.
- Aucun toast silencieux.
## 6) Cas spéciaux
### 6.1 Lock NVS / boot mode
- Le backend peut avoir `kLockNvsMediaManagerMode` actif.
- Si lock actif:
- afficher l’état de lock (ou trace de refus) sans bloquer la navigation media actuelle.
- proposer des boutons de fallback (`BOOT_MODE_STATUS`, `BOOT_MODE_SET MEDIA_MANAGER`, `BOOT_MODE_CLEAR`) selon disponibilité.
### 6.2 Legacy compatibility
- Si lAPI legacy remonte uniquement `current_step`, dériver `story.step`.
- Si `media` absent, désactiver actions médias et afficher `status incomplet`.
## 7) Acceptance frontend (conforme)
1. Arrivée en fin de scénario -> `SCENE_MEDIA_MANAGER` affichée et hub actif.
2. `STEP_MEDIA_MANAGER` reconnu en fallback.
3. `/api/media/files` retourne des listes sans parsing dextension.
4. Play/stop alterne `media.playing` sans erreur persistante.
5. Record start/stop met à jour `media.recording` et respecte la limite temporelle.
6. Une erreur `kind` invalide renvoie bien `ok=false` et erreur visible.
## 8) Tests à préparer (non-code)
- **Contract mock**: payload `story.screen: SCENE_MEDIA_MANAGER` avec step mixte.
- **Contract mock**: payload sans `screen` mais `step: STEP_MEDIA_MANAGER`.
- **API mock**: `kind=video` -> 400.
- **UI mock**: replay play en cascade (double clic) sans erreur de crash.
- **UI mock**: `record_simulated=true` et `media.last_error` rempli.
## 9) Artefacts attendus par équipe
- `artifacts/runtime-sync/<date>/media-manager-front-checklist.md`
- `artifacts/runtime-sync/<date>/media-manager-front-mapping.md`
## 10) Sortie et limites
- Implémentation réalisée dans `frontend-scratch-v2` (consommation UI seulement).
- Toute divergence détectée (`SCENE_MEDIA_MANAGER` restant sur une transition runtime attendue) est remontée via `SCN-601-MEDIA-BRIDGE`.
+2 -2
View File
@@ -10,8 +10,8 @@
- Runbook sync (section FW): `specs/MEDIA_MANAGER_SYNC_RUNBOOK.md`
## Pack Equipe Front-end
- Handoff opérationnel frontend: `fronted dev web UI/specs/MEDIA_MANAGER_FRONTEND_HANDOFF.md`
- Spécification détaillée frontend: `fronted dev web UI/specs/MEDIA_MANAGER_FRONTEND_SPEC.md`
- Handoff opérationnel frontend: `specs/MEDIA_MANAGER_FRONTEND_HANDOFF.md`
- Spécification détaillée frontend: `specs/MEDIA_MANAGER_FRONTEND_SPEC.md`
- Runbook sync (section WEB): `specs/MEDIA_MANAGER_SYNC_RUNBOOK.md`
- Contrat runtime API global: `specs/FIRMWARE_WEB_DATA_CONTRACT.md`
+309
View File
@@ -0,0 +1,309 @@
# QA Test Matrix Specification
## Status
- State: draft
- Date: 2026-03-21
- Depends on: `ZACUS_RUNTIME_3_SPEC.md`, `FIRMWARE_WEB_DATA_CONTRACT.md`, `AI_INTEGRATION_SPEC.md`
## 1) Objective
Define a unified test matrix covering all components of the Zacus platform: firmware (C++), frontend (TypeScript/React), tooling (Python), content (YAML/JSON), and AI integrations. Establish coverage targets, tooling, and CI gates.
## 2) Test Pyramid
```mermaid
graph TD
E2E["E2E Tests<br/>Hardware-in-Loop<br/>~10 tests"]
INT["Integration Tests<br/>API Contract + Scenario Pipeline<br/>~30 tests"]
UNIT["Unit Tests<br/>Python + TypeScript + C++<br/>~100+ tests"]
SMOKE["Smoke / Content Checks<br/>Validators + Schema<br/>~20 checks"]
SEC["Security Tests<br/>Auth + Injection + Fuzzing<br/>~15 tests"]
PERF["Performance Tests<br/>Memory + Latency + Endurance<br/>~10 tests"]
E2E --> INT --> UNIT
SMOKE --> UNIT
SEC --> INT
PERF --> INT
style UNIT fill:#2d6,stroke:#333
style INT fill:#69d,stroke:#333
style E2E fill:#d69,stroke:#333
style SMOKE fill:#9d6,stroke:#333
style SEC fill:#d66,stroke:#333
style PERF fill:#dd6,stroke:#333
```
## 3) Unit Tests
### 3.1 Python Tooling (Runtime 3)
| ID | Test | File | Priority |
|----|------|------|----------|
| PY-U-01 | Compile valid YAML to IR JSON | `test_compile_runtime3.py` | P0 |
| PY-U-02 | Compile invalid YAML (missing fields) | `test_compile_runtime3.py` | P0 |
| PY-U-03 | Simulate linear scenario (no cycles) | `test_simulate_runtime3.py` | P0 |
| PY-U-04 | Simulate scenario with branch/merge | `test_simulate_runtime3.py` | P1 |
| PY-U-05 | Detect transition cycles (max_steps) | `test_simulate_runtime3.py` | P0 |
| PY-U-06 | Validate step_id uniqueness | `test_validate_runtime3.py` | P0 |
| PY-U-07 | Validate transition target exists | `test_validate_runtime3.py` | P0 |
| PY-U-08 | normalize_token edge cases | `test_runtime3_common.py` | P1 |
| PY-U-09 | Schema version migration (v1 -> v2) | `test_runtime3_common.py` | P2 |
| PY-U-10 | Export firmware bundle structure | `test_export_runtime3.py` | P1 |
| PY-U-11 | Pivot verification pass/fail | `test_verify_pivots.py` | P1 |
| PY-U-12 | Audio manifest validation | `test_audio_validation.py` | P1 |
| PY-U-13 | Printables manifest validation | `test_printables_validation.py` | P1 |
**Runner**: `uv run python -m pytest tests/runtime3/ -v`
**Coverage target**: 80%
**Tool**: pytest + coverage.py
### 3.2 TypeScript Frontend (React + Blockly)
| ID | Test | File | Priority |
|----|------|------|----------|
| FE-U-01 | ScenarioLib: load valid YAML | `scenario.test.ts` | P0 |
| FE-U-02 | ScenarioLib: reject malformed YAML | `scenario.test.ts` | P0 |
| FE-U-03 | Runtime3Lib: compile to IR | `runtime3.test.ts` | P0 |
| FE-U-04 | Runtime3Lib: validate schema version | `runtime3.test.ts` | P1 |
| FE-U-05 | API client: request formatting | `api.test.ts` | P0 |
| FE-U-06 | API client: timeout handling | `api.test.ts` | P0 |
| FE-U-07 | API client: error response parsing | `api.test.ts` | P1 |
| FE-U-08 | Blockly: workspace to YAML round-trip | `blockly.test.ts` | P1 |
| FE-U-09 | Blockly: custom block registration | `blockly.test.ts` | P2 |
| FE-U-10 | Zod schema validation | `schemas.test.ts` | P1 |
| FE-U-11 | App component renders tabs | `App.test.tsx` | P1 |
| FE-U-12 | ErrorBoundary catches throw | `ErrorBoundary.test.tsx` | P1 |
**Runner**: `npm test` (Vitest)
**Coverage target**: 70%
**Tool**: Vitest + @testing-library/react
### 3.3 C++ Firmware
| ID | Test | File | Priority |
|----|------|------|----------|
| FW-U-01 | JSON parser: valid scenario | `test_json_parser.cpp` | P0 |
| FW-U-02 | JSON parser: malformed input | `test_json_parser.cpp` | P0 |
| FW-U-03 | Transition engine: event dispatch | `test_transitions.cpp` | P0 |
| FW-U-04 | Transition engine: priority ordering | `test_transitions.cpp` | P1 |
| FW-U-05 | Audio manager: buffer lifecycle | `test_audio.cpp` | P1 |
| FW-U-06 | Storage manager: NVS read/write | `test_storage.cpp` | P1 |
| FW-U-07 | LED manager: color conversion | `test_led.cpp` | P2 |
| FW-U-08 | Input validation: API params | `test_input_validation.cpp` | P0 |
| FW-U-09 | Rate limiter: token bucket | `test_rate_limiter.cpp` | P1 |
| FW-U-10 | PSRAM allocator: fallback chain | `test_allocator.cpp` | P1 |
**Runner**: `pio test -e native` (PlatformIO native test)
**Coverage target**: 60% (limited by hardware abstraction)
**Tool**: PlatformIO Unity test framework
## 4) Integration Tests
### 4.1 API Contract Tests
| ID | Test | Priority |
|----|------|----------|
| INT-01 | ESP32 API: GET /api/status returns valid JSON | P0 |
| INT-02 | ESP32 API: POST /api/scenario/transition changes step | P0 |
| INT-03 | ESP32 API: POST /api/audio plays file | P1 |
| INT-04 | ESP32 API: POST /api/led sets color | P1 |
| INT-05 | ESP32 API: unauthorized request returns 401 | P0 |
| INT-06 | ESP32 API: invalid JSON returns 400 | P0 |
| INT-07 | ESP32 API: rate limit triggers 429 | P1 |
| INT-08 | mascarade API: POST /api/v1/send returns hint | P1 |
| INT-09 | MCP server: tools/list returns all tools | P0 |
| INT-10 | MCP server: tools/call puzzle_set_state | P1 |
**Runner**: `uv run python -m pytest tests/integration/ -v -m integration`
**Tool**: pytest + httpx (async HTTP client)
### 4.2 Scenario Pipeline Tests
| ID | Test | Priority |
|----|------|----------|
| PIPE-01 | YAML -> compile -> simulate -> export: full pipeline | P0 |
| PIPE-02 | Modified YAML -> recompile preserves step IDs | P0 |
| PIPE-03 | Firmware bundle matches expected schema | P0 |
| PIPE-04 | Blockly export -> YAML -> compile round-trip | P1 |
| PIPE-05 | Multi-scenario compilation (batch) | P2 |
**Runner**: `uv run python -m pytest tests/pipeline/ -v`
## 5) End-to-End Tests (Hardware-in-Loop)
| ID | Test | Priority |
|----|------|----------|
| E2E-01 | Flash firmware -> boot -> API responds | P0 |
| E2E-02 | Upload scenario -> play through all steps | P0 |
| E2E-03 | Audio playback: file plays, completion event fires | P1 |
| E2E-04 | LED sequence: scenario-driven color changes | P1 |
| E2E-05 | ESP-NOW: pair two devices, relay message | P1 |
| E2E-06 | OTA update: push new firmware, device reboots | P2 |
| E2E-07 | Serial command suite: all commands respond | P0 |
| E2E-08 | WiFi reconnection after dropout | P1 |
| E2E-09 | Watchdog recovery after hang | P2 |
| E2E-10 | Full game session (90 min endurance) | P1 |
**Runner**: Manual or CI with hardware runner
**Tool**: pytest + pyserial (serial commands) + httpx (API verification)
### 5.1 Serial Test Suite
The firmware exposes a serial command interface for testing:
```
> status
OK step=STEP_U_SON_PROTO uptime=12345 heap=245000
> transition UNLOCK_COFFRE
OK step=STEP_COFFRE_OPEN
> audio play /audio/hint_01.mp3
OK playing hint_01.mp3
> led puzzle #FF0000 solid
OK led_set zone=puzzle color=#FF0000
```
Serial tests validate all commands and expected responses.
## 6) Smoke Tests (Content Checks)
| ID | Check | Gate | Tool |
|----|-------|------|------|
| SMOKE-01 | Scenario YAML schema valid | CI | yamllint + custom validator |
| SMOKE-02 | All step_ids referenced in transitions exist | CI | compile_runtime3.py |
| SMOKE-03 | All audio files referenced in YAML exist | CI | validate_audio.sh |
| SMOKE-04 | All printable assets referenced exist | CI | validate_printables.sh |
| SMOKE-05 | Runtime 3 IR compiles without errors | CI | compile_runtime3.py |
| SMOKE-06 | Runtime 3 simulation completes (no deadlocks) | CI | simulate_runtime3.py |
| SMOKE-07 | Firmware compiles for freenove_esp32s3 | CI | pio run |
| SMOKE-08 | Firmware compiles for esp8266_oled | CI | pio run |
| SMOKE-09 | Frontend builds without errors | CI | npm run build |
| SMOKE-10 | Frontend lint passes | CI | npm run lint |
| SMOKE-11 | MkDocs builds without warnings | CI | mkdocs build --strict |
| SMOKE-12 | No hardcoded credentials in source | CI | grep + custom script |
**Runner**: `bash tools/test/run_content_checks.sh`
**CI trigger**: Every push to main, every PR
## 7) Security Tests
| ID | Test | Priority | Tool |
|----|------|----------|------|
| SEC-01 | API endpoints require Bearer token | P0 | httpx |
| SEC-02 | Invalid token returns 401 | P0 | httpx |
| SEC-03 | SQL/NoSQL injection in API params | P1 | custom fuzzer |
| SEC-04 | XSS in scenario text fields | P1 | custom fuzzer |
| SEC-05 | Path traversal in audio file paths | P0 | httpx |
| SEC-06 | JSON bomb (deeply nested) rejected | P1 | httpx |
| SEC-07 | Oversized request body rejected (>1 MB) | P1 | httpx |
| SEC-08 | Rate limiting enforced (>10 req/s) | P1 | httpx |
| SEC-09 | CORS: only allowed origins accepted | P1 | httpx |
| SEC-10 | No credentials in firmware binary | P0 | strings + grep |
| SEC-11 | NVS credentials not in plaintext flash dump | P1 | esptool |
| SEC-12 | WebSocket auth on connect | P1 | websockets |
| SEC-13 | MCP server: tool call auth validated | P0 | pytest |
| SEC-14 | Prompt injection in LLM hint requests | P1 | custom prompts |
| SEC-15 | WiFi deauth resilience (reconnect) | P2 | aireplay-ng |
**Runner**: `uv run python -m pytest tests/security/ -v -m security`
## 8) Performance Tests
| ID | Test | Target | Tool |
|----|------|--------|------|
| PERF-01 | ESP32 free heap after boot | > 200 KB | serial monitor |
| PERF-02 | ESP32 free heap after 1h runtime | > 150 KB (no leak) | serial monitor |
| PERF-03 | API response time (GET /status) | < 50 ms | httpx + timing |
| PERF-04 | API response time (POST /transition) | < 100 ms | httpx + timing |
| PERF-05 | Audio playback start latency | < 200 ms | oscilloscope/logic analyzer |
| PERF-06 | LED update latency | < 50 ms | logic analyzer |
| PERF-07 | Frontend build size | < 3 MB gzip | npm run build |
| PERF-08 | Frontend initial load time | < 2 s (LAN) | Lighthouse |
| PERF-09 | Runtime 3 compile time (50 steps) | < 1 s | pytest benchmark |
| PERF-10 | 90-min endurance: no crash, no memory leak | Pass | serial + API monitor |
**Runner**: `uv run python -m pytest tests/performance/ -v -m perf --benchmark`
## 9) Test Environments
| Environment | Purpose | Hardware | Network |
|-------------|---------|----------|---------|
| **Local dev** | Unit + smoke tests | MacBook (GrosMac) | None required |
| **Native test** | C++ unit tests (no hardware) | Any x86/ARM | None |
| **Hardware bench** | Integration + E2E | ESP32-S3 Freenove + USB | WiFi AP |
| **CI runner** | Smoke + unit + lint | GitHub Actions | Cloud |
| **Staging mesh** | Multi-device E2E | 2-3 ESP32-S3 + AP | Dedicated WiFi |
| **Field test** | Full game session | Complete room setup | Production WiFi |
### 9.1 CI Pipeline (GitHub Actions)
```mermaid
flowchart LR
Push["Push / PR"] --> Lint["Lint<br/>ESLint + Ruff"]
Lint --> Smoke["Smoke Tests<br/>Content Checks"]
Smoke --> Unit["Unit Tests<br/>Python + TS"]
Unit --> Build["Build<br/>Firmware + Frontend"]
Build --> Gate{"All Pass?"}
Gate -->|Yes| Merge["Allow Merge"]
Gate -->|No| Block["Block PR"]
```
**CI jobs**:
```yaml
jobs:
lint:
- npm run lint (frontend)
- ruff check tools/ tests/ (python)
smoke:
- bash tools/test/run_content_checks.sh
unit-python:
- uv run python -m pytest tests/runtime3/ -v --cov
unit-frontend:
- cd frontend-scratch-v2 && npm test -- --coverage
build-firmware:
- cd hardware/firmware && pio run -e freenove_esp32s3
build-frontend:
- cd frontend-scratch-v2 && npm run build
docs:
- python -m mkdocs build --strict
```
## 10) Coverage Targets
| Component | Current | Target (Phase 1) | Target (Phase 2) |
|-----------|---------|-------------------|-------------------|
| Python tooling | ~20% (5 tests) | 60% (25 tests) | 80% (40 tests) |
| Frontend (TS) | 0% | 40% (12 tests) | 70% (25 tests) |
| Firmware (C++) | 0% | 30% (10 tests) | 60% (20 tests) |
| Content checks | 80% | 90% (12 checks) | 95% (15 checks) |
| Integration | 0% | 30% (5 tests) | 60% (10 tests) |
| Security | 0% | 40% (6 tests) | 70% (12 tests) |
## 11) Test Data Management
### 11.1 Fixtures
- `tests/fixtures/valid_scenario.yaml` — minimal valid scenario (3 steps)
- `tests/fixtures/complex_scenario.yaml` — full scenario with branches (15 steps)
- `tests/fixtures/invalid_*.yaml` — various malformed scenarios
- `tests/fixtures/runtime3_ir.json` — expected compiled output
- `tests/fixtures/firmware_bundle/` — expected export structure
### 11.2 Mocks
- `tests/mocks/esp32_api.py` — Mock ESP32 HTTP API (httpx responder)
- `tests/mocks/mascarade_api.py` — Mock mascarade API
- `tests/mocks/serial_device.py` — Mock serial port (pyserial loopback)
## 12) Defect Tracking
| Severity | Response Time | Resolution Time |
|----------|--------------|-----------------|
| CRITICAL (game-blocking) | 1 hour | 24 hours |
| HIGH (feature broken) | 4 hours | 72 hours |
| MEDIUM (degraded UX) | 24 hours | 1 week |
| LOW (cosmetic) | 1 week | Next release |
All defects tracked as GitHub Issues with labels: `bug`, `severity/{critical,high,medium,low}`, `component/{firmware,frontend,tooling,content,ai}`.
+87
View File
@@ -0,0 +1,87 @@
# Spécification frontend - Zacus Studio React + Blockly
## Objectif
Définir le frontend canonique de la refonte:
- édition Blockly-first,
- génération YAML canonique,
- prévisualisation IR Zacus Runtime 3,
- pilotage runtime et diagnostics via API.
## Stack retenue
- React 19
- Vite
- Blockly
- Monaco Editor
- Zod pour la validation locale
Implémentation active:
- `frontend-scratch-v2/src/App.tsx`
- `frontend-scratch-v2/src/components/BlocklyDesigner.tsx`
- `frontend-scratch-v2/src/lib/scenario.ts`
- `frontend-scratch-v2/src/lib/runtime3.ts`
- `frontend-scratch-v2/src/lib/api.ts`
## Décision UX
- Le moteur auteur canonique est Blockly, pas Cytoscape.
- Le YAML reste visible comme vérité éditoriale.
- L'IR Runtime 3 est visible en lecture seule comme contrat d'exécution.
- Le studio doit rester utilisable sans carte branchée pour l'édition et la simulation locale.
## Modèle de données auteur
`StoryGraphDocument`
- `scenarioId`
- `version`
- `initialStep`
- `steps[]`
- `steps[].transitions[]`
`StepTransition`
- `eventType`
- `eventName`
- `targetStepId`
- `priority`
- `afterMs`
## Flux principal
```mermaid
flowchart LR
Blockly["Blockly graph"] --> Scenario["Scenario document"]
Scenario --> YAML["Canonical YAML"]
Scenario --> Runtime3["Runtime 3 preview"]
YAML --> Validate["Scenario validation"]
Runtime3 --> Deploy["Firmware adapter / simulator"]
```
## Exigences fonctionnelles
- Le designer doit produire un scénario valide même si aucune transition explicite n'est dessinée.
- Le parser doit réimporter:
- `runtime3.steps`
- `firmware.steps`
- `steps_narrative`
- `firmware.steps_reference_order` (bridge deprecated; prefer `firmware.steps`)
- Les transitions invalides doivent être signalées localement.
- Le studio doit exposer:
- vue YAML canonique,
- vue IR Runtime 3,
- build/lint sans erreur,
- branchement vers `VITE_STORY_API_BASE`.
## Contrat runtime
- Le frontend ne définit pas la logique d'exécution finale.
- Le frontend compile une représentation auteur vers l'IR Runtime 3.
- Les routes runtime restent web-first et compatibles avec l'adaptateur firmware.
## Critères d'acceptation
- `npm run lint` vert.
- `npm run build` vert.
- Roundtrip Blockly -> YAML -> Runtime 3 cohérent sur `zacus_v2.yaml`.
- Les transitions affichées dans le studio correspondent aux cibles runtime.
- Le YAML affiché dans le studio reflète exactement l'état Blockly courant.
## Hors-scope
- Réactivation du frontend legacy Svelte/Cytoscape.
- Réécriture du firmware depuis le frontend.
- Authoring multi-scenarios complexe tant que le flux canonique V3 n'est pas stabilisé.
+41 -1
View File
@@ -9,6 +9,7 @@
Ce contrat couvre les payloads utilises par l'integration Web:
1. Story V2 (`/api/story/*`, WebSocket `/api/story/stream`)
2. Legacy Freenove (`/api/status`, `/api/stream`, `/api/scenario/*`, `/api/control`, `/api/network/*`)
3. Runtime 3 adapter (`/api/runtime3/*`)
Le contrat est strict sur les champs consommes par le frontend.
@@ -246,7 +247,19 @@ Reponse 200 (core minimal consomme):
"scenario": "DEFAULT",
"step": "SCENE_LEFOU_DETECTOR",
"screen": "SCENE_LEFOU_DETECTOR",
"audio_pack": "PACK_CONFIRM_WIN_ETAPE2"
"audio_pack": "PACK_CONFIRM_WIN_ETAPE2",
"runtime_contract": "runtime3+story_v2_adapter"
},
"runtime3": {
"discovered": true,
"loaded": true,
"path": "/story/runtime3/DEFAULT.json",
"schema_version": "zacus.runtime3.v1",
"scenario_id": "ZACUS_V2",
"entry_step_id": "STEP_U_SON_PROTO",
"step_count": 9,
"transition_count": 8,
"error": ""
},
"network": {
"state": "STA",
@@ -255,6 +268,33 @@ Reponse 200 (core minimal consomme):
}
```
### 4) Runtime 3 adapter
### 4.1 GET `/api/runtime3/status`
Réponse 200:
```json
{
"discovered": true,
"loaded": true,
"path": "/story/runtime3/DEFAULT.json",
"schema_version": "zacus.runtime3.v1",
"scenario_id": "ZACUS_V2",
"scenario_version": 3,
"entry_step_id": "STEP_U_SON_PROTO",
"source_kind": "yaml",
"generated_by": "tools/scenario/compile_runtime3.py",
"migration_mode": "firmware_import",
"step_count": 9,
"transition_count": 8,
"size_bytes": 4096,
"error": ""
}
```
### 4.2 GET `/api/runtime3/document`
- Retourne le document Runtime 3 actuellement chargé depuis LittleFS.
- Réponse `404` si aucun artefact Runtime 3 nest disponible.
Schema core (strict):
```json
{
+56
View File
@@ -0,0 +1,56 @@
# Zacus Runtime 3 Specification
## Goal
Define a portable runtime artifact that can be compiled from YAML or Blockly-authored content and executed by firmware or simulated locally.
## Design Principles
- Narrative canon remains in YAML during migration.
- Runtime IR is deterministic, versioned, and JSON serializable.
- Firmware consumes Runtime 3 but does not own narrative semantics.
- Import from legacy canonical YAML is allowed in a `linear_import` migration mode.
## IR Shape
```json
{
"schema_version": "zacus.runtime3.v1",
"scenario": {
"id": "ZACUS_V2",
"version": 3,
"title": "Le mystère du Professeur Zacus",
"entry_step_id": "STEP_U_SON_PROTO",
"source_kind": "yaml"
},
"steps": [
{
"id": "STEP_U_SON_PROTO",
"scene_id": "SCENE_U_SON_PROTO",
"audio_pack_id": "",
"actions": [],
"apps": [],
"transitions": []
}
],
"metadata": {
"migration_mode": "native",
"generated_by": "zacus_runtime3"
}
}
```
## Transition Model
- `event_type`: `button`, `serial`, `timer`, `audio_done`, `unlock`, `espnow`, `action`
- `event_name`: opaque token preserved as runtime contract
- `target_step_id`: required
- `priority`: integer, lower first
- `after_ms`: only meaningful for timer-style transitions
## Migration Modes
- `native`: authored directly for Runtime 3 with explicit transitions
- `linear_import`: derived from the legacy canonical YAML using ordered steps when no explicit graph exists
## Execution Model
- Runtime loads one IR document.
- Entry step is resolved from `scenario.entry_step_id`.
- A simulator can replay deterministic transition events without hardware.
- Firmware adapters may enrich steps with board-specific actions, but not mutate story flow semantics.