feat: AI integration — voice pipeline, hints engine, MCP server, analytics, security
- Voice pipeline: ESP32 WebSocket client → voice bridge → LLM → Piper TTS (Tower :8001) - Hints engine: 3 puzzles (LA_440, LEFOU_PIANO, QR_FINALE), anti-cheat, 3 hint levels - MCP hardware server: 6 tools (puzzle, audio, LED, camera, scenario, status), stdio transport - Analytics: ESP32 module + 6 web endpoints + Dashboard UI with chat interface - Security: auth middleware (Bearer NVS), rate limiting, input validation on 30 endpoints - Frontend: code-split (1.1MB → 210KB initial), ErrorBoundary, API timeout, WS reconnect - Tests: 24 Python + 38 TypeScript + 18 MCP = 80 project tests (+ 19 mascarade) - Specs: AI_INTEGRATION_SPEC, MCP_HARDWARE_SERVER_SPEC, QA_TEST_MATRIX_SPEC - Docs: SECURITY, DEPLOYMENT_RUNBOOK, voice pipeline guide, AI architecture map - 6 AI agent definitions (.github/agents/ai_*.md) - TUI orchestration script (tools/dev/zacus_tui.py) - Docker compose TTS for Tower + KXKM-AI - CHANGELOG, README, mkdocs.yml updated - Cycle detection (DFS) in runtime3 validator - Sprint plan: plans/SPRINT_AI_INTEGRATION.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -123,9 +123,9 @@ Coordonner les agents et developpeurs pour:
|
||||
- `python3 tools/scenario/export_md.py game/scenarios/zacus_v2.yaml`
|
||||
- `python3 tools/audio/validate_manifest.py audio/manifests/zacus_v2_audio.yaml`
|
||||
- `python3 tools/printables/validate_manifest.py printables/manifests/zacus_v2_printables.yaml`
|
||||
- `npm --prefix 'fronted dev web UI' run lint`
|
||||
- `npm --prefix 'fronted dev web UI' run build`
|
||||
- `npm --prefix 'fronted dev web UI' run test:unit -- --run`
|
||||
- `npm --prefix 'frontend-scratch-v2' run lint`
|
||||
- `npm --prefix 'frontend-scratch-v2' run build`
|
||||
- importer/exporter YAML + IR Runtime 3 sans perte fonctionnelle
|
||||
|
||||
## Regle de handoff entre agents
|
||||
Chaque ticket passe seulement avec ces evidences:
|
||||
|
||||
@@ -0,0 +1,365 @@
|
||||
# AI Integration Specification
|
||||
|
||||
## Status
|
||||
- State: draft
|
||||
- Date: 2026-03-21
|
||||
- Depends on: `ZACUS_RUNTIME_3_SPEC.md`, `MCP_HARDWARE_SERVER_SPEC.md`, `FIRMWARE_WEB_DATA_CONTRACT.md`
|
||||
|
||||
## 1) Objective
|
||||
|
||||
Integrate AI capabilities into the Zacus escape room platform across three tiers:
|
||||
- **On-device** (ESP32-S3): low-latency voice commands and object detection
|
||||
- **Server** (mascarade + Docker): LLM reasoning, TTS voice cloning, MCP orchestration
|
||||
- **GPU** (KXKM-AI RTX 4090): generative audio, model fine-tuning
|
||||
|
||||
The AI layer enriches the escape room without replacing the deterministic Runtime 3 scenario engine. AI features degrade gracefully: if the server is unreachable, the game continues with pre-recorded audio and static hints.
|
||||
|
||||
## 2) Architecture Overview
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
subgraph ESP32["ESP32-S3 On-Device AI"]
|
||||
MIC[Microphone I2S]
|
||||
SPK[Speaker I2S]
|
||||
CAM[Camera OV2640]
|
||||
SR[ESP-SR v2.0<br/>Wake Word + Commands]
|
||||
DL[ESP-DL v3.2<br/>Object Detection]
|
||||
RT[Runtime 3 Engine]
|
||||
end
|
||||
|
||||
subgraph Server["mascarade Server (VM / GrosMac)"]
|
||||
API[mascarade API<br/>LLM orchestration]
|
||||
TTS[Coqui XTTS-v2<br/>Voice Cloning Docker]
|
||||
MCP[MCP Hardware Server<br/>Tool dispatch]
|
||||
ADAPT[Adaptive Difficulty<br/>Analytics Engine]
|
||||
end
|
||||
|
||||
subgraph GPU["KXKM-AI (RTX 4090 24GB)"]
|
||||
MUSIC[AudioCraft MusicGen<br/>Ambient + SFX]
|
||||
TRAIN[Fine-tune Pipeline<br/>Unsloth + SimPO]
|
||||
end
|
||||
|
||||
MIC --> SR
|
||||
CAM --> DL
|
||||
SR -->|"voice command"| RT
|
||||
SR -->|"complex query WiFi"| MCP
|
||||
DL -->|"object detected"| RT
|
||||
DL -->|"detection event"| MCP
|
||||
RT -->|"hint request"| API
|
||||
MCP <-->|"JSON-RPC 2.0"| API
|
||||
API -->|"hint text"| TTS
|
||||
TTS -->|"PCM audio stream"| SPK
|
||||
MUSIC -->|"ambient MP3 pre-gen"| SPK
|
||||
RT -->|"telemetry"| ADAPT
|
||||
ADAPT -->|"difficulty params"| API
|
||||
TRAIN -.->|"updated models"| SR
|
||||
TRAIN -.->|"updated models"| DL
|
||||
```
|
||||
|
||||
## 3) On-Device AI
|
||||
|
||||
### 3.1 ESP-SR v2.0 — Wake Word + Voice Commands
|
||||
|
||||
**Purpose**: Hands-free interaction during gameplay. Players say "Hey Zacus" then a command.
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Framework | ESP-SR v2.0 (WakeNet + MultiNet) |
|
||||
| Wake word | "Hey Zacus" (custom trained, WakeNet Q8) |
|
||||
| Command vocabulary | 50 French commands (MultiNet, expandable to 300) |
|
||||
| Latency | < 200 ms wake detection, < 500 ms command recognition |
|
||||
| Memory | ~280 KB PSRAM (WakeNet 120 KB + MultiNet 160 KB) |
|
||||
| Audio format | 16-bit PCM, 16 kHz, mono |
|
||||
| Microphone | INMP441 I2S MEMS |
|
||||
|
||||
**Command categories**:
|
||||
- Navigation: "indice", "aide", "repeter", "suivant"
|
||||
- Puzzle control: "valider", "annuler", "recommencer"
|
||||
- Meta: "temps restant", "score", "pause"
|
||||
|
||||
**Fallback**: If wake word detection fails 3 times, the UI displays a tap-to-talk button.
|
||||
|
||||
### 3.2 ESP-DL v3.2 — Object Detection
|
||||
|
||||
**Purpose**: Detect physical puzzle props placed in front of the camera.
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Framework | ESP-DL v3.2 |
|
||||
| Model | YOLOv11n quantized (INT8) |
|
||||
| Input | 320x240 RGB from OV2640 |
|
||||
| FPS | 5-7 FPS inference |
|
||||
| Memory | ~450 KB PSRAM (model) + 150 KB (input buffer) |
|
||||
| Classes | 8 custom (fiole, clef, parchemin, cristal, engrenage, miroir, boussole, amulette) |
|
||||
| Confidence threshold | 0.65 |
|
||||
|
||||
**Detection events**:
|
||||
```json
|
||||
{
|
||||
"event_type": "object_detected",
|
||||
"event_name": "DETECT_FIOLE",
|
||||
"class": "fiole",
|
||||
"confidence": 0.82,
|
||||
"bbox": [45, 60, 180, 220],
|
||||
"timestamp_ms": 1234567890
|
||||
}
|
||||
```
|
||||
|
||||
These events feed into Runtime 3 transitions like any other `event_type`.
|
||||
|
||||
### 3.3 Memory Budget (ESP32-S3, 8MB PSRAM)
|
||||
|
||||
| Component | PSRAM | Internal SRAM |
|
||||
|-----------|-------|---------------|
|
||||
| ESP-SR (WakeNet + MultiNet) | 280 KB | 12 KB |
|
||||
| ESP-DL (YOLOv11n INT8) | 600 KB | 20 KB |
|
||||
| LVGL UI | 96 KB | 8 KB |
|
||||
| Audio DMA buffers | 64 KB | 4 KB |
|
||||
| Runtime 3 engine | 48 KB | 16 KB |
|
||||
| Network stack (WiFi + HTTP) | 80 KB | 32 KB |
|
||||
| LittleFS cache | 32 KB | — |
|
||||
| **Total used** | **1,200 KB** | **92 KB** |
|
||||
| **Available** | 8,192 KB | 512 KB |
|
||||
| **Headroom** | 85% | 82% |
|
||||
|
||||
### 3.4 Task Priority (FreeRTOS)
|
||||
|
||||
| Task | Priority | Core | Stack |
|
||||
|------|----------|------|-------|
|
||||
| Audio I2S (DMA) | 24 | 1 | 4 KB |
|
||||
| ESP-SR inference | 20 | 1 | 8 KB |
|
||||
| ESP-DL inference | 18 | 0 | 8 KB |
|
||||
| Runtime 3 loop | 15 | 0 | 8 KB |
|
||||
| LVGL tick | 12 | 0 | 4 KB |
|
||||
| WiFi/HTTP | 10 | 0 | 6 KB |
|
||||
| Idle | 0 | * | 2 KB |
|
||||
|
||||
## 4) Server-Side AI
|
||||
|
||||
### 4.1 Coqui XTTS-v2 — Voice Cloning
|
||||
|
||||
**Purpose**: Generate dynamic narration in Professor Zacus's voice.
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Model | XTTS-v2 (Coqui) |
|
||||
| Deployment | Docker container on mascarade VM |
|
||||
| Reference sample | 6-second WAV of Zacus voice |
|
||||
| Output format | PCM 22050 Hz 16-bit mono |
|
||||
| Latency target | < 2 s for 20-word sentence |
|
||||
| Language | French (fr) |
|
||||
| Streaming | Chunked HTTP response (256-sample chunks) |
|
||||
| GPU required | No (CPU inference acceptable for short utterances) |
|
||||
| Memory | ~2 GB container |
|
||||
|
||||
**API endpoint** (Docker internal):
|
||||
```
|
||||
POST /api/tts
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"text": "Bravo, vous avez trouve la fiole sacree!",
|
||||
"speaker_wav": "/data/voices/zacus_ref.wav",
|
||||
"language": "fr"
|
||||
}
|
||||
|
||||
Response: audio/wav stream
|
||||
```
|
||||
|
||||
**Integration with mascarade**: The MCP server calls TTS after receiving hint text from the LLM. Audio is streamed to the ESP32 via chunked HTTP.
|
||||
|
||||
### 4.2 LLM Adaptive Hints via mascarade API
|
||||
|
||||
**Purpose**: Context-aware, anti-cheat hints personalized to player progress.
|
||||
|
||||
**Flow**:
|
||||
1. ESP32 sends hint request with context (current step, elapsed time, failed attempts)
|
||||
2. mascarade API routes to configured LLM provider
|
||||
3. System prompt enforces anti-spoiler rules
|
||||
4. Response text is sent to TTS for voice synthesis
|
||||
5. Difficulty parameters adjust based on analytics
|
||||
|
||||
**Request format** (ESP32 -> mascarade):
|
||||
```json
|
||||
{
|
||||
"endpoint": "/api/v1/send",
|
||||
"payload": {
|
||||
"provider": "ollama",
|
||||
"model": "mascarade-coder",
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "Tu es le Professeur Zacus. Donne un indice sans reveler la solution. Adapte le niveau: {{difficulty}}."
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Nous sommes bloques a l'etape {{step_id}} depuis {{elapsed_min}} minutes. Tentatives: {{attempts}}."
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Anti-cheat prompt engineering** (ref: devlinb/escaperoom):
|
||||
- Never reveal full solutions
|
||||
- Escalate hints progressively (vague -> specific -> near-answer)
|
||||
- Maximum 3 hints per puzzle per session
|
||||
- Log all hint requests for game master review
|
||||
|
||||
**Latency target**: < 3 s end-to-end (LLM + TTS + network).
|
||||
|
||||
### 4.3 MCP Hardware Server
|
||||
|
||||
See `MCP_HARDWARE_SERVER_SPEC.md` for full specification.
|
||||
|
||||
The MCP server exposes ESP32 hardware as LLM-callable tools:
|
||||
- `puzzle_set_state` — lock/unlock puzzle elements
|
||||
- `audio_play` — trigger audio on device speakers
|
||||
- `led_set` — control LED strips (color, pattern, brightness)
|
||||
- `camera_capture` — take a snapshot from OV2640
|
||||
- `scenario_advance` — trigger a Runtime 3 transition
|
||||
|
||||
## 5) GPU AI (KXKM-AI)
|
||||
|
||||
### 5.1 AudioCraft MusicGen — Generative Audio
|
||||
|
||||
**Purpose**: Generate ambient music and sound effects per room/puzzle.
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Model | MusicGen-small (300M) or MusicGen-medium (1.5B) |
|
||||
| Hardware | KXKM-AI, RTX 4090 24 GB, 62 GB RAM |
|
||||
| Generation mode | Pre-generation (not real-time) |
|
||||
| Output format | WAV 32 kHz stereo, converted to MP3 128 kbps for ESP32 |
|
||||
| Duration | 30-60 s loops per room |
|
||||
| Prompt template | "atmospheric mysterious escape room music, {{room_theme}}, ambient, looping" |
|
||||
| Latency | ~10 s per 30 s clip (offline batch) |
|
||||
|
||||
**Workflow**:
|
||||
1. Game designer specifies room themes in scenario YAML
|
||||
2. Batch generation script produces ambient tracks on KXKM-AI
|
||||
3. Tracks are transcoded to MP3 128 kbps mono (ESP32 compatible)
|
||||
4. Uploaded to LittleFS or served via HTTP
|
||||
5. Runtime 3 `audio_pack_id` references generated tracks
|
||||
|
||||
**SFX generation** (Stable Audio Open):
|
||||
- Short effect sounds (unlock, alarm, discovery)
|
||||
- 2-5 s duration
|
||||
- Triggered by Runtime 3 events
|
||||
|
||||
### 5.2 Fine-Tune Pipeline
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Base model | Qwen2.5-Coder-1.5B |
|
||||
| Method | Unsloth + SimPO |
|
||||
| Dataset | Custom Zacus hint pairs + Magicoder-OSS-Instruct-75K |
|
||||
| Training time | ~6 min on RTX 4090 |
|
||||
| Output | GGUF Q4_K_M (~941 MB) deployed to Ollama on VM |
|
||||
| Trigger | P2P `distribute_task` via mascarade mesh |
|
||||
|
||||
## 6) Data Flow Summary
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant P as Player
|
||||
participant E as ESP32-S3
|
||||
participant M as mascarade API
|
||||
participant T as Coqui XTTS-v2
|
||||
participant K as KXKM-AI
|
||||
|
||||
Note over E: On-device AI (ESP-SR, ESP-DL)
|
||||
|
||||
P->>E: "Hey Zacus, un indice"
|
||||
E->>E: ESP-SR: wake + command parse
|
||||
E->>M: POST /api/v1/send (hint request + context)
|
||||
M->>M: LLM generates hint text
|
||||
M->>T: POST /api/tts (hint text, zacus voice)
|
||||
T-->>E: Chunked audio stream (PCM)
|
||||
E->>P: Speaker plays Zacus voice hint
|
||||
|
||||
Note over K: Pre-generation (offline)
|
||||
K->>K: MusicGen batch: room ambients
|
||||
K-->>E: MP3 files via HTTP/LittleFS upload
|
||||
```
|
||||
|
||||
## 7) Latency Targets
|
||||
|
||||
| Path | Target | Acceptable | Notes |
|
||||
|------|--------|-----------|-------|
|
||||
| Wake word detection | < 200 ms | < 500 ms | On-device, no network |
|
||||
| Voice command recognition | < 500 ms | < 1 s | On-device, MultiNet |
|
||||
| Object detection (single frame) | < 200 ms | < 400 ms | On-device, ESP-DL |
|
||||
| LLM hint (text only) | < 2 s | < 4 s | Network + LLM inference |
|
||||
| TTS synthesis (20 words) | < 2 s | < 4 s | Server CPU |
|
||||
| End-to-end voice hint | < 3 s | < 6 s | Wake -> LLM -> TTS -> speaker |
|
||||
| Ambient music start | < 500 ms | < 1 s | Pre-loaded MP3 |
|
||||
|
||||
## 8) Graceful Degradation
|
||||
|
||||
| Failure | Fallback |
|
||||
|---------|----------|
|
||||
| WiFi disconnected | Pre-recorded hints from LittleFS, no LLM |
|
||||
| mascarade API down | Cached hint bank (3 hints per puzzle in JSON) |
|
||||
| TTS service down | LLM text displayed on LVGL screen |
|
||||
| ESP-SR model corrupt | Tap-to-talk UI button, no voice |
|
||||
| ESP-DL model corrupt | QR code scanning fallback for object validation |
|
||||
| KXKM-AI offline | Pre-generated ambient tracks already on device |
|
||||
|
||||
## 9) Phase Rollout Plan
|
||||
|
||||
### Phase A: Security Foundations (P0 — 1-2 weeks)
|
||||
- NVS credential storage (replace hardcoded WiFi)
|
||||
- Bearer token auth on all API endpoints
|
||||
- Input validation + rate limiting
|
||||
- LVGL pool increase 54 -> 96 KB
|
||||
- Arduino stack increase 16 -> 24 KB
|
||||
|
||||
### Phase B: Voice Pipeline (P1 — 2-4 weeks)
|
||||
1. Integrate ESP-SR v2.0 WakeNet custom wake word
|
||||
2. Train "Hey Zacus" model with ESP-SR training toolkit
|
||||
3. Deploy Coqui XTTS-v2 Docker on mascarade VM
|
||||
4. Implement chunked audio streaming ESP32 <- Server
|
||||
5. Add MultiNet command vocabulary (50 FR commands)
|
||||
6. Reference architecture: XiaoZhi ESP32
|
||||
|
||||
### Phase C: Vision & Detection (P1 — 2-4 weeks)
|
||||
1. Integrate ESP-DL v3.2 with quantized YOLOv11n
|
||||
2. Collect and annotate prop dataset (8 classes, 500+ images)
|
||||
3. Train custom model, export INT8 for ESP32
|
||||
4. Wire detection events into Runtime 3 transitions
|
||||
5. Face detection for player counting (ESP-WHO)
|
||||
|
||||
### Phase D: LLM Adaptive Hints (P2 — 4-6 weeks)
|
||||
1. Design hint prompt templates with anti-spoiler rules
|
||||
2. Implement hint request API in firmware HTTP client
|
||||
3. Add analytics telemetry (step timing, attempts, hint count)
|
||||
4. Build adaptive difficulty engine in mascarade
|
||||
5. Professor Zacus as NPC LLM with conversation memory
|
||||
|
||||
### Phase E: Generative Audio (P2 — 2-3 weeks)
|
||||
1. Deploy AudioCraft MusicGen on KXKM-AI
|
||||
2. Create room theme prompts from scenario YAML
|
||||
3. Batch generate ambient tracks (30-60 s loops)
|
||||
4. Transcode to MP3 128 kbps mono for ESP32
|
||||
5. SFX generation with Stable Audio Open
|
||||
|
||||
### Phase F: MCP & Orchestration (P3 — 4-6 weeks)
|
||||
1. Implement MCP hardware server (see `MCP_HARDWARE_SERVER_SPEC.md`)
|
||||
2. Register in mascarade MCP registry
|
||||
3. Natural language hardware control for game masters
|
||||
4. Real-time game master dashboard
|
||||
|
||||
## 10) Dependencies
|
||||
|
||||
| Dependency | Version | License | Source |
|
||||
|------------|---------|---------|--------|
|
||||
| ESP-SR | v2.0 | Espressif | espressif/esp-sr |
|
||||
| ESP-DL | v3.2 | MIT | espressif/esp-dl |
|
||||
| Coqui XTTS-v2 | latest | MPL-2.0 | coqui-ai/TTS |
|
||||
| AudioCraft MusicGen | latest | MIT / CC-BY-NC-4.0 | facebookresearch/audiocraft |
|
||||
| Stable Audio Open | latest | Stability AI CLA | stabilityai/stable-audio-open |
|
||||
| mascarade API | main | Private | electron-rare/mascarade |
|
||||
| Ollama | latest | MIT | ollama/ollama |
|
||||
| Qwen2.5-Coder-1.5B | latest | Apache 2.0 | Qwen |
|
||||
| Unsloth | latest | Apache 2.0 | unslothai/unsloth |
|
||||
@@ -17,7 +17,7 @@ Le bundle conversationnel est maintenu dans :
|
||||
- Aucune refonte des manifests audio/printables.
|
||||
|
||||
## Risques
|
||||
- Le bundle peut diverger du scénario canonique `game/scenarios/zacus_v1.yaml`.
|
||||
- Le bundle peut diverger du scénario canonique `game/scenarios/zacus_v2.yaml`.
|
||||
- Les formats `scenario_runtime.json`/templates peuvent nécessiter une validation dédiée ultérieure.
|
||||
|
||||
## Prochaine étape proposée
|
||||
|
||||
@@ -27,13 +27,14 @@ Le firmware reste **intouché** par cette spécification.
|
||||
- contraintes éditoriales
|
||||
- configuration médias/printables (formats demandés)
|
||||
- Sorties attendues :
|
||||
- `YAML` conforme **Story V2** pour `game/scenarios/*.yaml`
|
||||
- `YAML` canonique pour `game/scenarios/*.yaml`
|
||||
- `IR JSON` conforme **Zacus Runtime 3** pour les previews et adaptateurs firmware
|
||||
- `manifest_yaml` + `markdown` pour le bundle imprimables
|
||||
- objet `diagnostic` (`rationale`, `source`, `warnings`)
|
||||
- Services inclus :
|
||||
- Docker LLM local (Ollama)
|
||||
- Gateway HTTP locale (script Python)
|
||||
- Frontend web Studio (React/Svelte selon stack du dépôt)
|
||||
- Frontend web Studio React + Blockly (`frontend-scratch-v2`)
|
||||
|
||||
## 4) Architecture cible
|
||||
|
||||
@@ -220,9 +221,9 @@ curl http://127.0.0.1:8787/health
|
||||
```
|
||||
|
||||
```bash
|
||||
cp "fronted dev web UI/.env.local.example" "fronted dev web UI/.env.local"
|
||||
cp "frontend-scratch-v2/.env.local.example" "frontend-scratch-v2/.env.local"
|
||||
# VITE_ZACUS_STUDIO_AI_URL=http://127.0.0.1:8787/story_generate
|
||||
npm --prefix "fronted dev web UI" run dev
|
||||
npm --prefix "frontend-scratch-v2" run dev
|
||||
```
|
||||
|
||||
### One-liner
|
||||
|
||||
@@ -0,0 +1,503 @@
|
||||
# MCP Hardware Server Specification
|
||||
|
||||
## Status
|
||||
- State: draft
|
||||
- Date: 2026-03-21
|
||||
- Depends on: `AI_INTEGRATION_SPEC.md`, `FIRMWARE_WEB_DATA_CONTRACT.md`
|
||||
- Reference: MCP specification (modelcontextprotocol.io), ESP RainMaker MCP pattern, IoT-MCP (Duke CEI)
|
||||
|
||||
## 1) Objective
|
||||
|
||||
Define an MCP (Model Context Protocol) server that exposes ESP32-S3 hardware capabilities as LLM-callable tools. This enables natural-language hardware control from mascarade, game master dashboards, and automated scenario orchestration.
|
||||
|
||||
## 2) Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph LLM["mascarade (LLM Host)"]
|
||||
Client[MCP Client<br/>call_tool_http]
|
||||
Registry[MCP Registry]
|
||||
end
|
||||
|
||||
subgraph MCPServer["MCP Hardware Server (Python)"]
|
||||
Transport[HTTP Transport<br/>JSON-RPC 2.0]
|
||||
Auth[Bearer Token Auth]
|
||||
Router[Tool Router]
|
||||
end
|
||||
|
||||
subgraph ESP32["ESP32-S3 Devices"]
|
||||
API1[Device 1 HTTP API<br/>:8080]
|
||||
API2[Device 2 HTTP API<br/>:8080]
|
||||
ESPNOW[ESP-NOW Mesh]
|
||||
end
|
||||
|
||||
Client -->|"JSON-RPC 2.0"| Transport
|
||||
Registry -.->|"discovery"| Transport
|
||||
Transport --> Auth --> Router
|
||||
Router -->|"HTTP REST"| API1
|
||||
Router -->|"HTTP REST"| API2
|
||||
API1 <-->|"ESP-NOW"| API2
|
||||
API1 <-->|"ESP-NOW"| ESPNOW
|
||||
```
|
||||
|
||||
## 3) Transport
|
||||
|
||||
### 3.1 HTTP Transport (Primary)
|
||||
|
||||
The MCP server runs as a Python HTTP service registered in mascarade's MCP registry.
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Protocol | JSON-RPC 2.0 over HTTP POST |
|
||||
| Endpoint | `POST /mcp` |
|
||||
| Port | 8790 (configurable via `MCP_HARDWARE_PORT`) |
|
||||
| Content-Type | `application/json` |
|
||||
| Auth | Bearer token (same as `MASCARADE_API_KEY`) |
|
||||
|
||||
### 3.2 stdio Transport (Local Development)
|
||||
|
||||
For local testing, the server also supports stdio transport per MCP spec:
|
||||
```bash
|
||||
python -m zacus_mcp_server --transport stdio
|
||||
```
|
||||
|
||||
### 3.3 Registration in mascarade
|
||||
|
||||
```python
|
||||
# mascarade MCP registry entry
|
||||
{
|
||||
"name": "zacus-hardware",
|
||||
"description": "ESP32-S3 escape room hardware control",
|
||||
"transport": "http",
|
||||
"url": "http://localhost:8790/mcp",
|
||||
"auth": {"type": "bearer", "token_env": "MASCARADE_API_KEY"},
|
||||
"enabled_env": "ZACUS_MCP_ENABLED"
|
||||
}
|
||||
```
|
||||
|
||||
## 4) Authentication Model
|
||||
|
||||
### 4.1 MCP Server Auth (mascarade -> MCP Server)
|
||||
|
||||
- Bearer token in `Authorization` header
|
||||
- Token matches `MASCARADE_API_KEY` environment variable
|
||||
- Requests without valid token receive `401 Unauthorized`
|
||||
|
||||
### 4.2 MCP Server -> ESP32 Auth
|
||||
|
||||
- Bearer token in `Authorization` header on ESP32 HTTP API
|
||||
- Token stored in ESP32 NVS (provisioned at setup)
|
||||
- Per-device token support for multi-device deployments
|
||||
|
||||
### 4.3 Token Hierarchy
|
||||
|
||||
```
|
||||
mascarade API key
|
||||
└── MCP server validates incoming requests
|
||||
└── Per-device ESP32 tokens
|
||||
└── ESP32 validates hardware commands
|
||||
```
|
||||
|
||||
## 5) Tool Definitions
|
||||
|
||||
### 5.1 `puzzle_set_state`
|
||||
|
||||
Control puzzle lock/unlock state and trigger associated effects.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "puzzle_set_state",
|
||||
"description": "Set the state of a puzzle element (lock, unlock, reset). Triggers associated LED and audio effects.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"device_id": {
|
||||
"type": "string",
|
||||
"description": "Target ESP32 device identifier"
|
||||
},
|
||||
"puzzle_id": {
|
||||
"type": "string",
|
||||
"description": "Puzzle identifier from scenario runtime",
|
||||
"enum": ["PUZZLE_FIOLE", "PUZZLE_COFFRE", "PUZZLE_MIROIR", "PUZZLE_ENGRENAGE", "PUZZLE_CRYSTAL", "PUZZLE_BOUSSOLE"]
|
||||
},
|
||||
"state": {
|
||||
"type": "string",
|
||||
"enum": ["locked", "unlocked", "reset"],
|
||||
"description": "Target state"
|
||||
},
|
||||
"effects": {
|
||||
"type": "boolean",
|
||||
"default": true,
|
||||
"description": "Play associated LED/audio effects on state change"
|
||||
}
|
||||
},
|
||||
"required": ["device_id", "puzzle_id", "state"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**ESP32 API mapping**: `POST /api/puzzle` with body `{"id": "...", "state": "...", "effects": true}`
|
||||
|
||||
### 5.2 `audio_play`
|
||||
|
||||
Play audio files or streams on device speakers.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "audio_play",
|
||||
"description": "Play an audio file or stream on the ESP32 speaker. Supports local files (LittleFS) and HTTP URLs.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"device_id": {
|
||||
"type": "string",
|
||||
"description": "Target ESP32 device identifier"
|
||||
},
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Audio source: LittleFS path (/audio/hint_01.mp3) or HTTP URL"
|
||||
},
|
||||
"volume": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 100,
|
||||
"default": 70,
|
||||
"description": "Playback volume (0-100)"
|
||||
},
|
||||
"loop": {
|
||||
"type": "boolean",
|
||||
"default": false,
|
||||
"description": "Loop playback continuously"
|
||||
},
|
||||
"action": {
|
||||
"type": "string",
|
||||
"enum": ["play", "stop", "pause", "resume"],
|
||||
"default": "play"
|
||||
}
|
||||
},
|
||||
"required": ["device_id", "source"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**ESP32 API mapping**: `POST /api/audio` with body `{"src": "...", "vol": 70, "loop": false, "action": "play"}`
|
||||
|
||||
### 5.3 `led_set`
|
||||
|
||||
Control LED strips and individual LEDs.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "led_set",
|
||||
"description": "Control LED strips: set color, pattern, brightness. Supports WS2812B addressable LEDs.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"device_id": {
|
||||
"type": "string",
|
||||
"description": "Target ESP32 device identifier"
|
||||
},
|
||||
"zone": {
|
||||
"type": "string",
|
||||
"description": "LED zone identifier",
|
||||
"enum": ["ambient", "puzzle", "alert", "all"]
|
||||
},
|
||||
"color": {
|
||||
"type": "string",
|
||||
"description": "Hex color (#RRGGBB) or named color",
|
||||
"pattern": "^(#[0-9a-fA-F]{6}|red|green|blue|white|off|warm|cold|purple|orange)$"
|
||||
},
|
||||
"pattern": {
|
||||
"type": "string",
|
||||
"enum": ["solid", "breathe", "chase", "rainbow", "pulse", "off"],
|
||||
"default": "solid"
|
||||
},
|
||||
"brightness": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 255,
|
||||
"default": 128
|
||||
},
|
||||
"duration_ms": {
|
||||
"type": "integer",
|
||||
"description": "Auto-off after duration (0 = indefinite)",
|
||||
"default": 0
|
||||
}
|
||||
},
|
||||
"required": ["device_id", "zone", "color"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**ESP32 API mapping**: `POST /api/led` with body `{"zone": "...", "color": "...", "pattern": "solid", "bright": 128}`
|
||||
|
||||
### 5.4 `camera_capture`
|
||||
|
||||
Capture a snapshot from the OV2640 camera.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "camera_capture",
|
||||
"description": "Capture a JPEG snapshot from the ESP32 camera. Returns base64-encoded image.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"device_id": {
|
||||
"type": "string",
|
||||
"description": "Target ESP32 device identifier"
|
||||
},
|
||||
"resolution": {
|
||||
"type": "string",
|
||||
"enum": ["QQVGA", "QVGA", "VGA"],
|
||||
"default": "QVGA",
|
||||
"description": "Capture resolution (160x120, 320x240, 640x480)"
|
||||
},
|
||||
"quality": {
|
||||
"type": "integer",
|
||||
"minimum": 10,
|
||||
"maximum": 63,
|
||||
"default": 20,
|
||||
"description": "JPEG quality (lower = better, 10-63)"
|
||||
}
|
||||
},
|
||||
"required": ["device_id"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**ESP32 API mapping**: `GET /api/camera?res=QVGA&q=20` returns `image/jpeg`
|
||||
|
||||
### 5.5 `scenario_advance`
|
||||
|
||||
Trigger a Runtime 3 transition on the device.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "scenario_advance",
|
||||
"description": "Trigger a Runtime 3 scenario transition. Used by game masters to manually advance or reset the game.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"device_id": {
|
||||
"type": "string",
|
||||
"description": "Target ESP32 device identifier"
|
||||
},
|
||||
"event_type": {
|
||||
"type": "string",
|
||||
"enum": ["button", "serial", "timer", "audio_done", "unlock", "espnow", "action", "manual"],
|
||||
"description": "Event type per Runtime 3 transition model"
|
||||
},
|
||||
"event_name": {
|
||||
"type": "string",
|
||||
"description": "Event name token (e.g., UNLOCK_COFFRE, MANUAL_ADVANCE)"
|
||||
},
|
||||
"target_step_id": {
|
||||
"type": "string",
|
||||
"description": "Optional: force transition to specific step (game master override)"
|
||||
}
|
||||
},
|
||||
"required": ["device_id", "event_type", "event_name"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**ESP32 API mapping**: `POST /api/scenario/transition` with body `{"event_type": "...", "event_name": "...", "target": "..."}`
|
||||
|
||||
### 5.6 `device_status`
|
||||
|
||||
Query device health and current state.
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "device_status",
|
||||
"description": "Get current device status: free memory, current scenario step, uptime, WiFi RSSI, sensor readings.",
|
||||
"inputSchema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"device_id": {
|
||||
"type": "string",
|
||||
"description": "Target ESP32 device identifier"
|
||||
}
|
||||
},
|
||||
"required": ["device_id"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**ESP32 API mapping**: `GET /api/status` returns JSON status object
|
||||
|
||||
## 6) Message Format (JSON-RPC 2.0)
|
||||
|
||||
### 6.1 Request
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "req-001",
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "led_set",
|
||||
"arguments": {
|
||||
"device_id": "zacus-main",
|
||||
"zone": "puzzle",
|
||||
"color": "#00FF00",
|
||||
"pattern": "pulse",
|
||||
"brightness": 200
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.2 Success Response
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "req-001",
|
||||
"result": {
|
||||
"content": [
|
||||
{
|
||||
"type": "text",
|
||||
"text": "LED zone 'puzzle' set to #00FF00 pulse at brightness 200 on device zacus-main"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.3 Error Response
|
||||
|
||||
```json
|
||||
{
|
||||
"jsonrpc": "2.0",
|
||||
"id": "req-001",
|
||||
"error": {
|
||||
"code": -32000,
|
||||
"message": "Device unreachable",
|
||||
"data": {
|
||||
"device_id": "zacus-main",
|
||||
"detail": "HTTP timeout after 5000ms"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 6.4 Error Codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| -32700 | Parse error (malformed JSON) |
|
||||
| -32600 | Invalid request |
|
||||
| -32601 | Method not found |
|
||||
| -32602 | Invalid params |
|
||||
| -32603 | Internal error |
|
||||
| -32000 | Device unreachable |
|
||||
| -32001 | Device busy (command in progress) |
|
||||
| -32002 | Auth failed (ESP32 token) |
|
||||
| -32003 | Puzzle state conflict |
|
||||
|
||||
## 7) Sequence Diagrams
|
||||
|
||||
### 7.1 LLM-Driven Puzzle Unlock
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant GM as Game Master
|
||||
participant M as mascarade LLM
|
||||
participant MCP as MCP Hardware Server
|
||||
participant E as ESP32-S3
|
||||
|
||||
GM->>M: "Deverrouille le coffre dans la salle 2"
|
||||
M->>M: Parse intent -> puzzle_set_state
|
||||
M->>MCP: JSON-RPC tools/call puzzle_set_state
|
||||
MCP->>MCP: Validate params + auth
|
||||
MCP->>E: POST /api/puzzle {"id":"PUZZLE_COFFRE","state":"unlocked"}
|
||||
E->>E: Servo unlock + LED green + SFX
|
||||
E-->>MCP: 200 OK {"state":"unlocked","effects_played":true}
|
||||
MCP-->>M: JSON-RPC result
|
||||
M-->>GM: "Le coffre de la salle 2 est maintenant deverrouille."
|
||||
```
|
||||
|
||||
### 7.2 Voice Command -> Hardware Action
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant P as Player
|
||||
participant E as ESP32-S3
|
||||
participant MCP as MCP Hardware Server
|
||||
participant M as mascarade LLM
|
||||
participant T as TTS
|
||||
|
||||
P->>E: "Hey Zacus, allume la lumiere UV"
|
||||
E->>E: ESP-SR: wake + command parse
|
||||
E->>MCP: JSON-RPC tools/call led_set (zone:puzzle, color:purple)
|
||||
MCP->>E: POST /api/led {"zone":"puzzle","color":"#7B00FF"}
|
||||
E->>E: UV LED on
|
||||
E-->>MCP: 200 OK
|
||||
MCP-->>E: JSON-RPC result
|
||||
Note over E: Optional: confirm via TTS
|
||||
E->>M: Hint request: "confirme action UV"
|
||||
M->>T: TTS "La lumiere UV est activee"
|
||||
T-->>E: Audio stream
|
||||
E->>P: Speaker: "La lumiere UV est activee"
|
||||
```
|
||||
|
||||
## 8) Device Discovery
|
||||
|
||||
### 8.1 Static Configuration (Phase 1)
|
||||
|
||||
Devices are configured in `.env`:
|
||||
```env
|
||||
ZACUS_DEVICES='[{"id":"zacus-main","host":"192.168.0.50","port":8080,"token":"..."}]'
|
||||
```
|
||||
|
||||
### 8.2 mDNS Discovery (Phase 2)
|
||||
|
||||
ESP32 devices advertise `_zacus._tcp` via mDNS. The MCP server discovers devices automatically:
|
||||
```
|
||||
_zacus._tcp.local.
|
||||
zacus-main._zacus._tcp.local. 8080 TXT "room=main" "version=3.1"
|
||||
zacus-salle2._zacus._tcp.local. 8080 TXT "room=salle2" "version=3.1"
|
||||
```
|
||||
|
||||
### 8.3 ESP-NOW Mesh (Phase 3)
|
||||
|
||||
The primary ESP32 acts as a gateway for ESP-NOW mesh devices. The MCP server sends commands to the gateway, which relays via ESP-NOW to secondary devices.
|
||||
|
||||
## 9) Rate Limiting & Safety
|
||||
|
||||
| Constraint | Value |
|
||||
|------------|-------|
|
||||
| Max requests per device | 10/s |
|
||||
| Max concurrent tool calls | 3 |
|
||||
| Command timeout | 5 s |
|
||||
| Retry on timeout | 1 retry with 2 s backoff |
|
||||
| Servo actuation cooldown | 500 ms between movements |
|
||||
| LED transition min interval | 100 ms |
|
||||
|
||||
**Safety guards**:
|
||||
- No two conflicting puzzle state changes within 1 s
|
||||
- Audio volume hard-capped at device level (not bypassable via MCP)
|
||||
- Camera capture rate limited to 2/s to prevent overheating
|
||||
- Game master override always available via `scenario_advance` with `event_type: "manual"`
|
||||
|
||||
## 10) Implementation Plan
|
||||
|
||||
### Phase 1: Core Server (2 weeks)
|
||||
- Python MCP server with HTTP transport
|
||||
- Tool definitions: `puzzle_set_state`, `audio_play`, `led_set`, `device_status`
|
||||
- Bearer auth, static device config
|
||||
- Register in mascarade MCP registry
|
||||
- Unit tests (pytest)
|
||||
|
||||
### Phase 2: Camera + Scenario (2 weeks)
|
||||
- `camera_capture` tool with base64 response
|
||||
- `scenario_advance` tool with Runtime 3 integration
|
||||
- mDNS device discovery
|
||||
- Integration tests with ESP32 hardware
|
||||
|
||||
### Phase 3: Mesh + Dashboard (3 weeks)
|
||||
- ESP-NOW mesh relay through gateway device
|
||||
- Real-time game master dashboard (WebSocket feed)
|
||||
- Multi-device orchestration (synchronized effects)
|
||||
- Load testing and latency benchmarks
|
||||
@@ -0,0 +1,47 @@
|
||||
# Handoff - Équipe Front-end (Media Manager)
|
||||
|
||||
## Contexte
|
||||
- FSM de travail: `scenario-ai-coherence/zacus_conversation_bundle_v3/fsm_mermaid.md`
|
||||
- Contrat media manager: `specs/MEDIA_MANAGER_RUNTIME_SPEC.md`
|
||||
- Scope: aucune modification firmware.
|
||||
|
||||
## Objectif métier
|
||||
- Exposer le Media Manager de manière fiable à l’arrivée en fin de scénario.
|
||||
- Accepter les identifiants step/scene mixtes sans casser l’UI.
|
||||
- S’appuyer sur les APIs media existantes pour listage, lecture et enregistrement.
|
||||
|
||||
## Cibles de sortie (artefacts)
|
||||
- `artifacts/runtime-sync/<date>/media-manager-web-checks.md`
|
||||
- `artifacts/runtime-sync/<date>/media-manager-front-mapping.md`
|
||||
- notes de migration/ajustement de parsing si nécessaire.
|
||||
- Spec détaillée d'intégration: `specs/MEDIA_MANAGER_FRONTEND_SPEC.md`
|
||||
|
||||
## Actions obligatoires
|
||||
1. Parsing statut:
|
||||
- lire `story.screen` en priorité.
|
||||
- fallback tolérant sur `story.step == STEP_MEDIA_MANAGER`.
|
||||
- ne pas supposer `step_id` prefixé `STEP_` ou `SCENE_`.
|
||||
2. Mapping Media Hub:
|
||||
- activer vue hub quand `story.screen == SCENE_MEDIA_MANAGER`.
|
||||
- si non dispo, fallback `story.step == STEP_MEDIA_MANAGER`.
|
||||
3. Endpoints media:
|
||||
- `/api/media/files` (`kind` music/picture/recorder)
|
||||
- `/api/media/play`, `/api/media/stop`
|
||||
- `/api/media/record/start`, `/api/media/record/stop`
|
||||
- `/api/media/record/status`
|
||||
4. Gestion erreurs:
|
||||
- gérer `ok=false` et `error` depuis `/api/control` et endpoints media.
|
||||
- afficher `media.last_error` en debug écran media.
|
||||
5. Affichage media:
|
||||
- afficher clairement `media.record_simulated` pour éviter les ambiguïtés d’enregistrement réel.
|
||||
|
||||
## Critères d’acceptation
|
||||
- Le Media Hub s’affiche à la fin de scénario sans crash.
|
||||
- `SCENE_MEDIA_MANAGER` déclenche bien la vue média même avec `STEP_MEDIA_MANAGER` possible en fallback.
|
||||
- `media.playing` bascule correctement via play/stop.
|
||||
- Les erreurs API sont présentées de façon exploitable par l’opérateur.
|
||||
|
||||
## Remarques équipe
|
||||
- Ne pas changer les contrats runtime, seulement adapter la consommation UI.
|
||||
- Toute divergence métier (ex: cible `SCENE_MEDIA_MANAGER` encore présente) doit être reportée via ticket "scenario".
|
||||
- Implémentation front alignée au studio React + Blockly (`frontend-scratch-v2`) avec consommation API côté UI.
|
||||
@@ -0,0 +1,120 @@
|
||||
# Spécification Frontend — Media Manager (Mode final)
|
||||
|
||||
## Contexte
|
||||
- Source de vérité runtime: `hardware/firmware/data/story/scenarios/DEFAULT.json`
|
||||
- FSM de travail: `scenario-ai-coherence/zacus_conversation_bundle_v3/fsm_mermaid.md`
|
||||
- Handoff frontend: `specs/MEDIA_MANAGER_FRONTEND_HANDOFF.md`
|
||||
- Contrat media runtime: `specs/MEDIA_MANAGER_RUNTIME_SPEC.md`
|
||||
|
||||
Ce document formalise **ce que le frontend doit consommer** pour être robuste quand le scénario arrive sur la phase finale media manager.
|
||||
|
||||
## 1) Objectifs
|
||||
1. Détecter de façon fiable l’entrée Media Manager malgré la mixité `STEP_*`/`SCENE_*`.
|
||||
2. Consommer les endpoints médias sans hypothèses sur l’extension de fichier.
|
||||
3. Exposer des erreurs exploitables (`ok=false`, `error`) sans crash UI.
|
||||
4. Gérer le cas de lock NVS via l’affichage de statut de boot mode.
|
||||
|
||||
## 2) Détection d’écran / état actif
|
||||
### 2.1 Règle de détection (priorité)
|
||||
Le frontend doit considérer que le Media Manager est actif si l’une des conditions est vraie:
|
||||
1. `story.screen === "SCENE_MEDIA_MANAGER"`
|
||||
2. `story.step === "STEP_MEDIA_MANAGER"`
|
||||
3. fallback legacy: payload status expose un champ équivalent `step`/`scene`/`currentStep` contenant `MEDIA_MANAGER`.
|
||||
|
||||
### 2.2 Tolérance IDs mixtes
|
||||
- Ne jamais supposer que `step` commence par `STEP_`.
|
||||
- Ne jamais supposer que `screen` commence par `SCENE_`.
|
||||
- Conserver les deux champs en mémoire pour les logs d’audit.
|
||||
|
||||
## 3) Modèle de données utilisé par le frontend
|
||||
### 3.1 Polling / stream
|
||||
- Le frontend peut continuer avec la mécanique actuelle (stream en live si dispo, sinon polling).
|
||||
- Le payload de référence doit contenir au minimum:
|
||||
- `story.scenario`
|
||||
- `story.step`
|
||||
- `story.screen`
|
||||
- `media.ready`
|
||||
- `media.playing`
|
||||
- `media.recording`
|
||||
- `media.record_simulated`
|
||||
- `media.last_error`
|
||||
- `media.record_limit_seconds`
|
||||
- `media.record_elapsed_seconds`
|
||||
|
||||
### 3.2 Priorité d’affichage
|
||||
1. `story.screen` prend le pas sur `story.step` pour l’UI.
|
||||
2. `media.last_error` doit être prioritaire sur le message générique du formulaire d’action.
|
||||
|
||||
## 4) Contrat API (front) à implémenter côté UI
|
||||
### 4.1 Endpoints requis
|
||||
- `GET /api/media/files?kind=music|picture|recorder`
|
||||
- réponse attendue 200: `{ ok: true, kind: "...", files: ["/music/a.mp3"] }`
|
||||
- réponse 400 sur kind invalide: `{ ok: false, kind: "...", error: "invalid_kind" }`
|
||||
- `POST /api/media/play` body:
|
||||
- `{ "path": "/music/file.mp3" }` ou `{ "file": "file.mp3" }`
|
||||
- `POST /api/media/stop`
|
||||
- `POST /api/media/record/start` body:
|
||||
- `{ "seconds": 20, "filename": "take_1.wav" }`
|
||||
- `POST /api/media/record/stop`
|
||||
- `GET /api/media/record/status` (ou récupération de `media` dans `/api/status`)
|
||||
- `POST /api/control` fallback pour compatibilité opérationnelle.
|
||||
|
||||
### 4.2 Format de succès/erreur UI
|
||||
- Traiter tout appel média comme opération tri-state:
|
||||
- succès: `{ ok: true, action: "...", ... }`
|
||||
- erreur: `{ ok: false, error: "...", action: "..." }`
|
||||
- En cas d’erreur, afficher immédiatement `media.last_error` + statut de commande.
|
||||
|
||||
## 5) Comportements de l’écran Media Hub
|
||||
### 5.1 Listing
|
||||
- Charger **les 3 catégories** `music`, `picture`, `recorder` en parallèle ou séquentiel.
|
||||
- Afficher une section par catégorie quand disponible.
|
||||
- Aucun filtrage d’extension dur codé côté UI.
|
||||
|
||||
### 5.2 Lecture
|
||||
- Le clic sur un item déclenche `MEDIA_PLAY`.
|
||||
- Interdire un replay simultané si `media.playing === true` sans `stop` explicite préalable.
|
||||
- Le changement d’état doit refléter `media.playing` de l’API.
|
||||
|
||||
### 5.3 Enregistrement
|
||||
- Bouton d’enregistrement -> `MEDIA_RECORD_START`.
|
||||
- Respecter le `media.record_limit_seconds` du runtime.
|
||||
- Afficher `record_simulated` pour expliciter la nature simulée/placeholder du capture path.
|
||||
|
||||
### 5.4 Erreurs
|
||||
- Si `media.last_error` est non vide: montrer un encadré d’erreur persistante avec timestamp et action recommandée.
|
||||
- Aucun toast silencieux.
|
||||
|
||||
## 6) Cas spéciaux
|
||||
### 6.1 Lock NVS / boot mode
|
||||
- Le backend peut avoir `kLockNvsMediaManagerMode` actif.
|
||||
- Si lock actif:
|
||||
- afficher l’état de lock (ou trace de refus) sans bloquer la navigation media actuelle.
|
||||
- proposer des boutons de fallback (`BOOT_MODE_STATUS`, `BOOT_MODE_SET MEDIA_MANAGER`, `BOOT_MODE_CLEAR`) selon disponibilité.
|
||||
|
||||
### 6.2 Legacy compatibility
|
||||
- Si l’API legacy remonte uniquement `current_step`, dériver `story.step`.
|
||||
- Si `media` absent, désactiver actions médias et afficher `status incomplet`.
|
||||
|
||||
## 7) Acceptance frontend (conforme)
|
||||
1. Arrivée en fin de scénario -> `SCENE_MEDIA_MANAGER` affichée et hub actif.
|
||||
2. `STEP_MEDIA_MANAGER` reconnu en fallback.
|
||||
3. `/api/media/files` retourne des listes sans parsing d’extension.
|
||||
4. Play/stop alterne `media.playing` sans erreur persistante.
|
||||
5. Record start/stop met à jour `media.recording` et respecte la limite temporelle.
|
||||
6. Une erreur `kind` invalide renvoie bien `ok=false` et erreur visible.
|
||||
|
||||
## 8) Tests à préparer (non-code)
|
||||
- **Contract mock**: payload `story.screen: SCENE_MEDIA_MANAGER` avec step mixte.
|
||||
- **Contract mock**: payload sans `screen` mais `step: STEP_MEDIA_MANAGER`.
|
||||
- **API mock**: `kind=video` -> 400.
|
||||
- **UI mock**: replay play en cascade (double clic) sans erreur de crash.
|
||||
- **UI mock**: `record_simulated=true` et `media.last_error` rempli.
|
||||
|
||||
## 9) Artefacts attendus par équipe
|
||||
- `artifacts/runtime-sync/<date>/media-manager-front-checklist.md`
|
||||
- `artifacts/runtime-sync/<date>/media-manager-front-mapping.md`
|
||||
|
||||
## 10) Sortie et limites
|
||||
- Implémentation réalisée dans `frontend-scratch-v2` (consommation UI seulement).
|
||||
- Toute divergence détectée (`SCENE_MEDIA_MANAGER` restant sur une transition runtime attendue) est remontée via `SCN-601-MEDIA-BRIDGE`.
|
||||
@@ -10,8 +10,8 @@
|
||||
- Runbook sync (section FW): `specs/MEDIA_MANAGER_SYNC_RUNBOOK.md`
|
||||
|
||||
## Pack Equipe Front-end
|
||||
- Handoff opérationnel frontend: `fronted dev web UI/specs/MEDIA_MANAGER_FRONTEND_HANDOFF.md`
|
||||
- Spécification détaillée frontend: `fronted dev web UI/specs/MEDIA_MANAGER_FRONTEND_SPEC.md`
|
||||
- Handoff opérationnel frontend: `specs/MEDIA_MANAGER_FRONTEND_HANDOFF.md`
|
||||
- Spécification détaillée frontend: `specs/MEDIA_MANAGER_FRONTEND_SPEC.md`
|
||||
- Runbook sync (section WEB): `specs/MEDIA_MANAGER_SYNC_RUNBOOK.md`
|
||||
- Contrat runtime API global: `specs/FIRMWARE_WEB_DATA_CONTRACT.md`
|
||||
|
||||
|
||||
@@ -0,0 +1,309 @@
|
||||
# QA Test Matrix Specification
|
||||
|
||||
## Status
|
||||
- State: draft
|
||||
- Date: 2026-03-21
|
||||
- Depends on: `ZACUS_RUNTIME_3_SPEC.md`, `FIRMWARE_WEB_DATA_CONTRACT.md`, `AI_INTEGRATION_SPEC.md`
|
||||
|
||||
## 1) Objective
|
||||
|
||||
Define a unified test matrix covering all components of the Zacus platform: firmware (C++), frontend (TypeScript/React), tooling (Python), content (YAML/JSON), and AI integrations. Establish coverage targets, tooling, and CI gates.
|
||||
|
||||
## 2) Test Pyramid
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
E2E["E2E Tests<br/>Hardware-in-Loop<br/>~10 tests"]
|
||||
INT["Integration Tests<br/>API Contract + Scenario Pipeline<br/>~30 tests"]
|
||||
UNIT["Unit Tests<br/>Python + TypeScript + C++<br/>~100+ tests"]
|
||||
SMOKE["Smoke / Content Checks<br/>Validators + Schema<br/>~20 checks"]
|
||||
SEC["Security Tests<br/>Auth + Injection + Fuzzing<br/>~15 tests"]
|
||||
PERF["Performance Tests<br/>Memory + Latency + Endurance<br/>~10 tests"]
|
||||
|
||||
E2E --> INT --> UNIT
|
||||
SMOKE --> UNIT
|
||||
SEC --> INT
|
||||
PERF --> INT
|
||||
|
||||
style UNIT fill:#2d6,stroke:#333
|
||||
style INT fill:#69d,stroke:#333
|
||||
style E2E fill:#d69,stroke:#333
|
||||
style SMOKE fill:#9d6,stroke:#333
|
||||
style SEC fill:#d66,stroke:#333
|
||||
style PERF fill:#dd6,stroke:#333
|
||||
```
|
||||
|
||||
## 3) Unit Tests
|
||||
|
||||
### 3.1 Python Tooling (Runtime 3)
|
||||
|
||||
| ID | Test | File | Priority |
|
||||
|----|------|------|----------|
|
||||
| PY-U-01 | Compile valid YAML to IR JSON | `test_compile_runtime3.py` | P0 |
|
||||
| PY-U-02 | Compile invalid YAML (missing fields) | `test_compile_runtime3.py` | P0 |
|
||||
| PY-U-03 | Simulate linear scenario (no cycles) | `test_simulate_runtime3.py` | P0 |
|
||||
| PY-U-04 | Simulate scenario with branch/merge | `test_simulate_runtime3.py` | P1 |
|
||||
| PY-U-05 | Detect transition cycles (max_steps) | `test_simulate_runtime3.py` | P0 |
|
||||
| PY-U-06 | Validate step_id uniqueness | `test_validate_runtime3.py` | P0 |
|
||||
| PY-U-07 | Validate transition target exists | `test_validate_runtime3.py` | P0 |
|
||||
| PY-U-08 | normalize_token edge cases | `test_runtime3_common.py` | P1 |
|
||||
| PY-U-09 | Schema version migration (v1 -> v2) | `test_runtime3_common.py` | P2 |
|
||||
| PY-U-10 | Export firmware bundle structure | `test_export_runtime3.py` | P1 |
|
||||
| PY-U-11 | Pivot verification pass/fail | `test_verify_pivots.py` | P1 |
|
||||
| PY-U-12 | Audio manifest validation | `test_audio_validation.py` | P1 |
|
||||
| PY-U-13 | Printables manifest validation | `test_printables_validation.py` | P1 |
|
||||
|
||||
**Runner**: `uv run python -m pytest tests/runtime3/ -v`
|
||||
**Coverage target**: 80%
|
||||
**Tool**: pytest + coverage.py
|
||||
|
||||
### 3.2 TypeScript Frontend (React + Blockly)
|
||||
|
||||
| ID | Test | File | Priority |
|
||||
|----|------|------|----------|
|
||||
| FE-U-01 | ScenarioLib: load valid YAML | `scenario.test.ts` | P0 |
|
||||
| FE-U-02 | ScenarioLib: reject malformed YAML | `scenario.test.ts` | P0 |
|
||||
| FE-U-03 | Runtime3Lib: compile to IR | `runtime3.test.ts` | P0 |
|
||||
| FE-U-04 | Runtime3Lib: validate schema version | `runtime3.test.ts` | P1 |
|
||||
| FE-U-05 | API client: request formatting | `api.test.ts` | P0 |
|
||||
| FE-U-06 | API client: timeout handling | `api.test.ts` | P0 |
|
||||
| FE-U-07 | API client: error response parsing | `api.test.ts` | P1 |
|
||||
| FE-U-08 | Blockly: workspace to YAML round-trip | `blockly.test.ts` | P1 |
|
||||
| FE-U-09 | Blockly: custom block registration | `blockly.test.ts` | P2 |
|
||||
| FE-U-10 | Zod schema validation | `schemas.test.ts` | P1 |
|
||||
| FE-U-11 | App component renders tabs | `App.test.tsx` | P1 |
|
||||
| FE-U-12 | ErrorBoundary catches throw | `ErrorBoundary.test.tsx` | P1 |
|
||||
|
||||
**Runner**: `npm test` (Vitest)
|
||||
**Coverage target**: 70%
|
||||
**Tool**: Vitest + @testing-library/react
|
||||
|
||||
### 3.3 C++ Firmware
|
||||
|
||||
| ID | Test | File | Priority |
|
||||
|----|------|------|----------|
|
||||
| FW-U-01 | JSON parser: valid scenario | `test_json_parser.cpp` | P0 |
|
||||
| FW-U-02 | JSON parser: malformed input | `test_json_parser.cpp` | P0 |
|
||||
| FW-U-03 | Transition engine: event dispatch | `test_transitions.cpp` | P0 |
|
||||
| FW-U-04 | Transition engine: priority ordering | `test_transitions.cpp` | P1 |
|
||||
| FW-U-05 | Audio manager: buffer lifecycle | `test_audio.cpp` | P1 |
|
||||
| FW-U-06 | Storage manager: NVS read/write | `test_storage.cpp` | P1 |
|
||||
| FW-U-07 | LED manager: color conversion | `test_led.cpp` | P2 |
|
||||
| FW-U-08 | Input validation: API params | `test_input_validation.cpp` | P0 |
|
||||
| FW-U-09 | Rate limiter: token bucket | `test_rate_limiter.cpp` | P1 |
|
||||
| FW-U-10 | PSRAM allocator: fallback chain | `test_allocator.cpp` | P1 |
|
||||
|
||||
**Runner**: `pio test -e native` (PlatformIO native test)
|
||||
**Coverage target**: 60% (limited by hardware abstraction)
|
||||
**Tool**: PlatformIO Unity test framework
|
||||
|
||||
## 4) Integration Tests
|
||||
|
||||
### 4.1 API Contract Tests
|
||||
|
||||
| ID | Test | Priority |
|
||||
|----|------|----------|
|
||||
| INT-01 | ESP32 API: GET /api/status returns valid JSON | P0 |
|
||||
| INT-02 | ESP32 API: POST /api/scenario/transition changes step | P0 |
|
||||
| INT-03 | ESP32 API: POST /api/audio plays file | P1 |
|
||||
| INT-04 | ESP32 API: POST /api/led sets color | P1 |
|
||||
| INT-05 | ESP32 API: unauthorized request returns 401 | P0 |
|
||||
| INT-06 | ESP32 API: invalid JSON returns 400 | P0 |
|
||||
| INT-07 | ESP32 API: rate limit triggers 429 | P1 |
|
||||
| INT-08 | mascarade API: POST /api/v1/send returns hint | P1 |
|
||||
| INT-09 | MCP server: tools/list returns all tools | P0 |
|
||||
| INT-10 | MCP server: tools/call puzzle_set_state | P1 |
|
||||
|
||||
**Runner**: `uv run python -m pytest tests/integration/ -v -m integration`
|
||||
**Tool**: pytest + httpx (async HTTP client)
|
||||
|
||||
### 4.2 Scenario Pipeline Tests
|
||||
|
||||
| ID | Test | Priority |
|
||||
|----|------|----------|
|
||||
| PIPE-01 | YAML -> compile -> simulate -> export: full pipeline | P0 |
|
||||
| PIPE-02 | Modified YAML -> recompile preserves step IDs | P0 |
|
||||
| PIPE-03 | Firmware bundle matches expected schema | P0 |
|
||||
| PIPE-04 | Blockly export -> YAML -> compile round-trip | P1 |
|
||||
| PIPE-05 | Multi-scenario compilation (batch) | P2 |
|
||||
|
||||
**Runner**: `uv run python -m pytest tests/pipeline/ -v`
|
||||
|
||||
## 5) End-to-End Tests (Hardware-in-Loop)
|
||||
|
||||
| ID | Test | Priority |
|
||||
|----|------|----------|
|
||||
| E2E-01 | Flash firmware -> boot -> API responds | P0 |
|
||||
| E2E-02 | Upload scenario -> play through all steps | P0 |
|
||||
| E2E-03 | Audio playback: file plays, completion event fires | P1 |
|
||||
| E2E-04 | LED sequence: scenario-driven color changes | P1 |
|
||||
| E2E-05 | ESP-NOW: pair two devices, relay message | P1 |
|
||||
| E2E-06 | OTA update: push new firmware, device reboots | P2 |
|
||||
| E2E-07 | Serial command suite: all commands respond | P0 |
|
||||
| E2E-08 | WiFi reconnection after dropout | P1 |
|
||||
| E2E-09 | Watchdog recovery after hang | P2 |
|
||||
| E2E-10 | Full game session (90 min endurance) | P1 |
|
||||
|
||||
**Runner**: Manual or CI with hardware runner
|
||||
**Tool**: pytest + pyserial (serial commands) + httpx (API verification)
|
||||
|
||||
### 5.1 Serial Test Suite
|
||||
|
||||
The firmware exposes a serial command interface for testing:
|
||||
|
||||
```
|
||||
> status
|
||||
OK step=STEP_U_SON_PROTO uptime=12345 heap=245000
|
||||
|
||||
> transition UNLOCK_COFFRE
|
||||
OK step=STEP_COFFRE_OPEN
|
||||
|
||||
> audio play /audio/hint_01.mp3
|
||||
OK playing hint_01.mp3
|
||||
|
||||
> led puzzle #FF0000 solid
|
||||
OK led_set zone=puzzle color=#FF0000
|
||||
```
|
||||
|
||||
Serial tests validate all commands and expected responses.
|
||||
|
||||
## 6) Smoke Tests (Content Checks)
|
||||
|
||||
| ID | Check | Gate | Tool |
|
||||
|----|-------|------|------|
|
||||
| SMOKE-01 | Scenario YAML schema valid | CI | yamllint + custom validator |
|
||||
| SMOKE-02 | All step_ids referenced in transitions exist | CI | compile_runtime3.py |
|
||||
| SMOKE-03 | All audio files referenced in YAML exist | CI | validate_audio.sh |
|
||||
| SMOKE-04 | All printable assets referenced exist | CI | validate_printables.sh |
|
||||
| SMOKE-05 | Runtime 3 IR compiles without errors | CI | compile_runtime3.py |
|
||||
| SMOKE-06 | Runtime 3 simulation completes (no deadlocks) | CI | simulate_runtime3.py |
|
||||
| SMOKE-07 | Firmware compiles for freenove_esp32s3 | CI | pio run |
|
||||
| SMOKE-08 | Firmware compiles for esp8266_oled | CI | pio run |
|
||||
| SMOKE-09 | Frontend builds without errors | CI | npm run build |
|
||||
| SMOKE-10 | Frontend lint passes | CI | npm run lint |
|
||||
| SMOKE-11 | MkDocs builds without warnings | CI | mkdocs build --strict |
|
||||
| SMOKE-12 | No hardcoded credentials in source | CI | grep + custom script |
|
||||
|
||||
**Runner**: `bash tools/test/run_content_checks.sh`
|
||||
**CI trigger**: Every push to main, every PR
|
||||
|
||||
## 7) Security Tests
|
||||
|
||||
| ID | Test | Priority | Tool |
|
||||
|----|------|----------|------|
|
||||
| SEC-01 | API endpoints require Bearer token | P0 | httpx |
|
||||
| SEC-02 | Invalid token returns 401 | P0 | httpx |
|
||||
| SEC-03 | SQL/NoSQL injection in API params | P1 | custom fuzzer |
|
||||
| SEC-04 | XSS in scenario text fields | P1 | custom fuzzer |
|
||||
| SEC-05 | Path traversal in audio file paths | P0 | httpx |
|
||||
| SEC-06 | JSON bomb (deeply nested) rejected | P1 | httpx |
|
||||
| SEC-07 | Oversized request body rejected (>1 MB) | P1 | httpx |
|
||||
| SEC-08 | Rate limiting enforced (>10 req/s) | P1 | httpx |
|
||||
| SEC-09 | CORS: only allowed origins accepted | P1 | httpx |
|
||||
| SEC-10 | No credentials in firmware binary | P0 | strings + grep |
|
||||
| SEC-11 | NVS credentials not in plaintext flash dump | P1 | esptool |
|
||||
| SEC-12 | WebSocket auth on connect | P1 | websockets |
|
||||
| SEC-13 | MCP server: tool call auth validated | P0 | pytest |
|
||||
| SEC-14 | Prompt injection in LLM hint requests | P1 | custom prompts |
|
||||
| SEC-15 | WiFi deauth resilience (reconnect) | P2 | aireplay-ng |
|
||||
|
||||
**Runner**: `uv run python -m pytest tests/security/ -v -m security`
|
||||
|
||||
## 8) Performance Tests
|
||||
|
||||
| ID | Test | Target | Tool |
|
||||
|----|------|--------|------|
|
||||
| PERF-01 | ESP32 free heap after boot | > 200 KB | serial monitor |
|
||||
| PERF-02 | ESP32 free heap after 1h runtime | > 150 KB (no leak) | serial monitor |
|
||||
| PERF-03 | API response time (GET /status) | < 50 ms | httpx + timing |
|
||||
| PERF-04 | API response time (POST /transition) | < 100 ms | httpx + timing |
|
||||
| PERF-05 | Audio playback start latency | < 200 ms | oscilloscope/logic analyzer |
|
||||
| PERF-06 | LED update latency | < 50 ms | logic analyzer |
|
||||
| PERF-07 | Frontend build size | < 3 MB gzip | npm run build |
|
||||
| PERF-08 | Frontend initial load time | < 2 s (LAN) | Lighthouse |
|
||||
| PERF-09 | Runtime 3 compile time (50 steps) | < 1 s | pytest benchmark |
|
||||
| PERF-10 | 90-min endurance: no crash, no memory leak | Pass | serial + API monitor |
|
||||
|
||||
**Runner**: `uv run python -m pytest tests/performance/ -v -m perf --benchmark`
|
||||
|
||||
## 9) Test Environments
|
||||
|
||||
| Environment | Purpose | Hardware | Network |
|
||||
|-------------|---------|----------|---------|
|
||||
| **Local dev** | Unit + smoke tests | MacBook (GrosMac) | None required |
|
||||
| **Native test** | C++ unit tests (no hardware) | Any x86/ARM | None |
|
||||
| **Hardware bench** | Integration + E2E | ESP32-S3 Freenove + USB | WiFi AP |
|
||||
| **CI runner** | Smoke + unit + lint | GitHub Actions | Cloud |
|
||||
| **Staging mesh** | Multi-device E2E | 2-3 ESP32-S3 + AP | Dedicated WiFi |
|
||||
| **Field test** | Full game session | Complete room setup | Production WiFi |
|
||||
|
||||
### 9.1 CI Pipeline (GitHub Actions)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Push["Push / PR"] --> Lint["Lint<br/>ESLint + Ruff"]
|
||||
Lint --> Smoke["Smoke Tests<br/>Content Checks"]
|
||||
Smoke --> Unit["Unit Tests<br/>Python + TS"]
|
||||
Unit --> Build["Build<br/>Firmware + Frontend"]
|
||||
Build --> Gate{"All Pass?"}
|
||||
Gate -->|Yes| Merge["Allow Merge"]
|
||||
Gate -->|No| Block["Block PR"]
|
||||
```
|
||||
|
||||
**CI jobs**:
|
||||
```yaml
|
||||
jobs:
|
||||
lint:
|
||||
- npm run lint (frontend)
|
||||
- ruff check tools/ tests/ (python)
|
||||
smoke:
|
||||
- bash tools/test/run_content_checks.sh
|
||||
unit-python:
|
||||
- uv run python -m pytest tests/runtime3/ -v --cov
|
||||
unit-frontend:
|
||||
- cd frontend-scratch-v2 && npm test -- --coverage
|
||||
build-firmware:
|
||||
- cd hardware/firmware && pio run -e freenove_esp32s3
|
||||
build-frontend:
|
||||
- cd frontend-scratch-v2 && npm run build
|
||||
docs:
|
||||
- python -m mkdocs build --strict
|
||||
```
|
||||
|
||||
## 10) Coverage Targets
|
||||
|
||||
| Component | Current | Target (Phase 1) | Target (Phase 2) |
|
||||
|-----------|---------|-------------------|-------------------|
|
||||
| Python tooling | ~20% (5 tests) | 60% (25 tests) | 80% (40 tests) |
|
||||
| Frontend (TS) | 0% | 40% (12 tests) | 70% (25 tests) |
|
||||
| Firmware (C++) | 0% | 30% (10 tests) | 60% (20 tests) |
|
||||
| Content checks | 80% | 90% (12 checks) | 95% (15 checks) |
|
||||
| Integration | 0% | 30% (5 tests) | 60% (10 tests) |
|
||||
| Security | 0% | 40% (6 tests) | 70% (12 tests) |
|
||||
|
||||
## 11) Test Data Management
|
||||
|
||||
### 11.1 Fixtures
|
||||
|
||||
- `tests/fixtures/valid_scenario.yaml` — minimal valid scenario (3 steps)
|
||||
- `tests/fixtures/complex_scenario.yaml` — full scenario with branches (15 steps)
|
||||
- `tests/fixtures/invalid_*.yaml` — various malformed scenarios
|
||||
- `tests/fixtures/runtime3_ir.json` — expected compiled output
|
||||
- `tests/fixtures/firmware_bundle/` — expected export structure
|
||||
|
||||
### 11.2 Mocks
|
||||
|
||||
- `tests/mocks/esp32_api.py` — Mock ESP32 HTTP API (httpx responder)
|
||||
- `tests/mocks/mascarade_api.py` — Mock mascarade API
|
||||
- `tests/mocks/serial_device.py` — Mock serial port (pyserial loopback)
|
||||
|
||||
## 12) Defect Tracking
|
||||
|
||||
| Severity | Response Time | Resolution Time |
|
||||
|----------|--------------|-----------------|
|
||||
| CRITICAL (game-blocking) | 1 hour | 24 hours |
|
||||
| HIGH (feature broken) | 4 hours | 72 hours |
|
||||
| MEDIUM (degraded UX) | 24 hours | 1 week |
|
||||
| LOW (cosmetic) | 1 week | Next release |
|
||||
|
||||
All defects tracked as GitHub Issues with labels: `bug`, `severity/{critical,high,medium,low}`, `component/{firmware,frontend,tooling,content,ai}`.
|
||||
@@ -0,0 +1,87 @@
|
||||
# Spécification frontend - Zacus Studio React + Blockly
|
||||
|
||||
## Objectif
|
||||
|
||||
Définir le frontend canonique de la refonte:
|
||||
- édition Blockly-first,
|
||||
- génération YAML canonique,
|
||||
- prévisualisation IR Zacus Runtime 3,
|
||||
- pilotage runtime et diagnostics via API.
|
||||
|
||||
## Stack retenue
|
||||
- React 19
|
||||
- Vite
|
||||
- Blockly
|
||||
- Monaco Editor
|
||||
- Zod pour la validation locale
|
||||
|
||||
Implémentation active:
|
||||
- `frontend-scratch-v2/src/App.tsx`
|
||||
- `frontend-scratch-v2/src/components/BlocklyDesigner.tsx`
|
||||
- `frontend-scratch-v2/src/lib/scenario.ts`
|
||||
- `frontend-scratch-v2/src/lib/runtime3.ts`
|
||||
- `frontend-scratch-v2/src/lib/api.ts`
|
||||
|
||||
## Décision UX
|
||||
- Le moteur auteur canonique est Blockly, pas Cytoscape.
|
||||
- Le YAML reste visible comme vérité éditoriale.
|
||||
- L'IR Runtime 3 est visible en lecture seule comme contrat d'exécution.
|
||||
- Le studio doit rester utilisable sans carte branchée pour l'édition et la simulation locale.
|
||||
|
||||
## Modèle de données auteur
|
||||
|
||||
`StoryGraphDocument`
|
||||
- `scenarioId`
|
||||
- `version`
|
||||
- `initialStep`
|
||||
- `steps[]`
|
||||
- `steps[].transitions[]`
|
||||
|
||||
`StepTransition`
|
||||
- `eventType`
|
||||
- `eventName`
|
||||
- `targetStepId`
|
||||
- `priority`
|
||||
- `afterMs`
|
||||
|
||||
## Flux principal
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
Blockly["Blockly graph"] --> Scenario["Scenario document"]
|
||||
Scenario --> YAML["Canonical YAML"]
|
||||
Scenario --> Runtime3["Runtime 3 preview"]
|
||||
YAML --> Validate["Scenario validation"]
|
||||
Runtime3 --> Deploy["Firmware adapter / simulator"]
|
||||
```
|
||||
|
||||
## Exigences fonctionnelles
|
||||
- Le designer doit produire un scénario valide même si aucune transition explicite n'est dessinée.
|
||||
- Le parser doit réimporter:
|
||||
- `runtime3.steps`
|
||||
- `firmware.steps`
|
||||
- `steps_narrative`
|
||||
- `firmware.steps_reference_order` (bridge deprecated; prefer `firmware.steps`)
|
||||
- Les transitions invalides doivent être signalées localement.
|
||||
- Le studio doit exposer:
|
||||
- vue YAML canonique,
|
||||
- vue IR Runtime 3,
|
||||
- build/lint sans erreur,
|
||||
- branchement vers `VITE_STORY_API_BASE`.
|
||||
|
||||
## Contrat runtime
|
||||
- Le frontend ne définit pas la logique d'exécution finale.
|
||||
- Le frontend compile une représentation auteur vers l'IR Runtime 3.
|
||||
- Les routes runtime restent web-first et compatibles avec l'adaptateur firmware.
|
||||
|
||||
## Critères d'acceptation
|
||||
- `npm run lint` vert.
|
||||
- `npm run build` vert.
|
||||
- Roundtrip Blockly -> YAML -> Runtime 3 cohérent sur `zacus_v2.yaml`.
|
||||
- Les transitions affichées dans le studio correspondent aux cibles runtime.
|
||||
- Le YAML affiché dans le studio reflète exactement l'état Blockly courant.
|
||||
|
||||
## Hors-scope
|
||||
- Réactivation du frontend legacy Svelte/Cytoscape.
|
||||
- Réécriture du firmware depuis le frontend.
|
||||
- Authoring multi-scenarios complexe tant que le flux canonique V3 n'est pas stabilisé.
|
||||
@@ -9,6 +9,7 @@
|
||||
Ce contrat couvre les payloads utilises par l'integration Web:
|
||||
1. Story V2 (`/api/story/*`, WebSocket `/api/story/stream`)
|
||||
2. Legacy Freenove (`/api/status`, `/api/stream`, `/api/scenario/*`, `/api/control`, `/api/network/*`)
|
||||
3. Runtime 3 adapter (`/api/runtime3/*`)
|
||||
|
||||
Le contrat est strict sur les champs consommes par le frontend.
|
||||
|
||||
@@ -246,7 +247,19 @@ Reponse 200 (core minimal consomme):
|
||||
"scenario": "DEFAULT",
|
||||
"step": "SCENE_LEFOU_DETECTOR",
|
||||
"screen": "SCENE_LEFOU_DETECTOR",
|
||||
"audio_pack": "PACK_CONFIRM_WIN_ETAPE2"
|
||||
"audio_pack": "PACK_CONFIRM_WIN_ETAPE2",
|
||||
"runtime_contract": "runtime3+story_v2_adapter"
|
||||
},
|
||||
"runtime3": {
|
||||
"discovered": true,
|
||||
"loaded": true,
|
||||
"path": "/story/runtime3/DEFAULT.json",
|
||||
"schema_version": "zacus.runtime3.v1",
|
||||
"scenario_id": "ZACUS_V2",
|
||||
"entry_step_id": "STEP_U_SON_PROTO",
|
||||
"step_count": 9,
|
||||
"transition_count": 8,
|
||||
"error": ""
|
||||
},
|
||||
"network": {
|
||||
"state": "STA",
|
||||
@@ -255,6 +268,33 @@ Reponse 200 (core minimal consomme):
|
||||
}
|
||||
```
|
||||
|
||||
### 4) Runtime 3 adapter
|
||||
|
||||
### 4.1 GET `/api/runtime3/status`
|
||||
Réponse 200:
|
||||
```json
|
||||
{
|
||||
"discovered": true,
|
||||
"loaded": true,
|
||||
"path": "/story/runtime3/DEFAULT.json",
|
||||
"schema_version": "zacus.runtime3.v1",
|
||||
"scenario_id": "ZACUS_V2",
|
||||
"scenario_version": 3,
|
||||
"entry_step_id": "STEP_U_SON_PROTO",
|
||||
"source_kind": "yaml",
|
||||
"generated_by": "tools/scenario/compile_runtime3.py",
|
||||
"migration_mode": "firmware_import",
|
||||
"step_count": 9,
|
||||
"transition_count": 8,
|
||||
"size_bytes": 4096,
|
||||
"error": ""
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 GET `/api/runtime3/document`
|
||||
- Retourne le document Runtime 3 actuellement chargé depuis LittleFS.
|
||||
- Réponse `404` si aucun artefact Runtime 3 n’est disponible.
|
||||
|
||||
Schema core (strict):
|
||||
```json
|
||||
{
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# Zacus Runtime 3 Specification
|
||||
|
||||
## Goal
|
||||
Define a portable runtime artifact that can be compiled from YAML or Blockly-authored content and executed by firmware or simulated locally.
|
||||
|
||||
## Design Principles
|
||||
- Narrative canon remains in YAML during migration.
|
||||
- Runtime IR is deterministic, versioned, and JSON serializable.
|
||||
- Firmware consumes Runtime 3 but does not own narrative semantics.
|
||||
- Import from legacy canonical YAML is allowed in a `linear_import` migration mode.
|
||||
|
||||
## IR Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "zacus.runtime3.v1",
|
||||
"scenario": {
|
||||
"id": "ZACUS_V2",
|
||||
"version": 3,
|
||||
"title": "Le mystère du Professeur Zacus",
|
||||
"entry_step_id": "STEP_U_SON_PROTO",
|
||||
"source_kind": "yaml"
|
||||
},
|
||||
"steps": [
|
||||
{
|
||||
"id": "STEP_U_SON_PROTO",
|
||||
"scene_id": "SCENE_U_SON_PROTO",
|
||||
"audio_pack_id": "",
|
||||
"actions": [],
|
||||
"apps": [],
|
||||
"transitions": []
|
||||
}
|
||||
],
|
||||
"metadata": {
|
||||
"migration_mode": "native",
|
||||
"generated_by": "zacus_runtime3"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Transition Model
|
||||
- `event_type`: `button`, `serial`, `timer`, `audio_done`, `unlock`, `espnow`, `action`
|
||||
- `event_name`: opaque token preserved as runtime contract
|
||||
- `target_step_id`: required
|
||||
- `priority`: integer, lower first
|
||||
- `after_ms`: only meaningful for timer-style transitions
|
||||
|
||||
## Migration Modes
|
||||
- `native`: authored directly for Runtime 3 with explicit transitions
|
||||
- `linear_import`: derived from the legacy canonical YAML using ordered steps when no explicit graph exists
|
||||
|
||||
## Execution Model
|
||||
- Runtime loads one IR document.
|
||||
- Entry step is resolved from `scenario.entry_step_id`.
|
||||
- A simulator can replay deterministic transition events without hardware.
|
||||
- Firmware adapters may enrich steps with board-specific actions, but not mutate story flow semantics.
|
||||
Reference in New Issue
Block a user