le-mystere-professeur-zacus/docs/voice/VOICE_PIPELINE_GUIDE.md

Voice Pipeline Guide — Professeur Zacus

Technical guide for integrating voice interaction into the Zacus ESP32-S3 installation.

Overview

The voice pipeline enables visitors to interact with Professeur Zacus using natural speech. The system uses a split architecture: wake word detection runs locally on the ESP32-S3 (via ESP-SR), while speech-to-text, LLM inference, and text-to-speech run server-side via the mascarade bridge.

Hardware Requirements

INMP441 MEMS Microphone

INMP441 Pin	ESP32-S3 GPIO	Notes
SCK (BCLK)	GPIO 41	I2S0 bit clock
WS (LRCLK)	GPIO 42	I2S0 word select
SD (DATA)	GPIO 2	I2S0 data in
L/R	GND	Left channel select
VDD	3.3V	Power supply
GND	GND	Ground

Important: The microphone uses I2S0, which is a separate bus from the speaker output on I2S1 (MAX98357A). Both can operate simultaneously for full-duplex audio.

Speaker (already present)

The Freenove All-in-One board includes a MAX98357A I2S amplifier on I2S1. No additional wiring needed for audio output.

Bill of Materials (voice-specific)

Part	Qty	Approx. Cost	Source
INMP441 breakout	1	3-5 EUR	AliExpress, Amazon
Dupont wires (F-F)	5	included	—

Software Requirements

ESP-IDF + esp-sr Component

The ESP-SR library (WakeNet, MultiNet, AFE) requires the ESP-IDF framework. The current project uses Arduino framework via PlatformIO.

Option A: Full migration to ESP-IDF (recommended for production)

; platformio.ini changes
[env:freenove_allinone]
framework = espidf
; All Arduino-style code must be rewritten to ESP-IDF APIs

Option B: Arduino as ESP-IDF component (gradual migration)

; platformio.ini changes
[env:freenove_allinone]
framework = arduino, espidf
; Allows mixing Arduino and ESP-IDF code
; esp-sr can be added as an IDF component

esp-sr Component Registration

Create or update idf_component.yml:

dependencies:
  espressif/esp-sr:
    version: "^1.0.0"
    # Includes: WakeNet, MultiNet, AFE (AEC + NS + VAD)

Partition Table

ESP-SR models need flash space. Update partitions.csv:

# Name,   Type, SubType, Offset,  Size,    Flags
model,    data, spiffs,  ,        1500K,   ; WakeNet + optional MultiNet models

Voice Flow — Sequence Diagram

sequenceDiagram
    participant Mic as INMP441<br/>(I2S0)
    participant AFE as ESP-SR AFE<br/>(AEC+NS+VAD)
    participant WN as WakeNet9<br/>("Prof Zacus")
    participant WS as WebSocket<br/>Client
    participant Bridge as mascarade<br/>bridge:4001
    participant LLM as LLM<br/>(Qwen/GPT)
    participant TTS as Piper TTS<br/>(French)
    participant Spk as Speaker<br/>(I2S1)

    Note over Mic,AFE: LISTENING state
    Mic->>AFE: 16kHz 16-bit PCM (continuous)
    AFE->>WN: Cleaned audio frames
    WN-->>WN: Scanning for wake word...

    Note over WN,WS: WAKE_DETECTED state
    WN->>WS: Wake word detected!
    Spk->>Spk: Play chime / LED feedback

    Note over WS,Bridge: STREAMING state
    WS->>Bridge: {"type":"hello","wakeword":"professeur zacus"}
    loop Until VAD silence
        AFE->>WS: Processed audio frames
        WS->>Bridge: OPUS-encoded binary frames
        Bridge-->>WS: {"type":"stt","text":"..."} (interim)
    end
    WS->>Bridge: {"type":"end_of_speech"}

    Note over Bridge,LLM: PROCESSING state
    Bridge->>LLM: Final transcription -> LLM prompt
    LLM->>Bridge: Response text (streamed)

    Note over Bridge,Spk: SPEAKING state
    Bridge->>TTS: Response text
    TTS->>Bridge: OPUS audio stream
    Bridge->>WS: {"type":"tts","state":"start"}
    loop TTS audio chunks
        Bridge->>WS: OPUS-encoded binary frames
        WS->>Spk: Decode OPUS -> PCM -> I2S1
    end
    Bridge->>WS: {"type":"tts","state":"end"}

    Note over Mic,Spk: Back to LISTENING

XiaoZhi WebSocket Protocol

The voice pipeline follows the XiaoZhi-ESP32 WebSocket protocol, which is well-documented and battle-tested.

Client -> Server Messages

Type	Format	Description
Hello	{"type":"hello","version":1,"wakeword":"..."}	Session start after wake detection
Audio	Binary (OPUS frames)	20ms OPUS-encoded audio at 16kHz mono
Abort	{"type":"abort"}	Cancel current interaction
End of speech	{"type":"end_of_speech"}	VAD detected end of utterance

Server -> Client Messages

Type	Format	Description
STT result	{"type":"stt","text":"..."}	Interim/final transcription
LLM text	{"type":"llm","text":"...","emotion":"..."}	Streamed LLM response
TTS control	{"type":"tts","state":"start|end"}	TTS playback boundaries
TTS audio	Binary (OPUS frames)	OPUS-encoded speech audio
Emotion	{"type":"emotion","value":"happy|thinking|..."}	Facial expression hint

Audio Encoding

• Codec: OPUS (RFC 6716)
• Frame size: 20ms
• Sample rate: 16kHz
• Channels: Mono
• Bitrate: ~16kbps (voice-optimized)
• Library: libopus via ESP-IDF component or bundled in esp-sr

Server-Side Requirements

mascarade Bridge Endpoint

The mascarade bridge (running on GrosMac or VM) needs a /voice WebSocket endpoint that:

1. Receives OPUS audio frames from ESP32
2. Decodes OPUS -> PCM
3. Runs STT (Whisper via faster-whisper or whisper.cpp)
4. Sends transcription to LLM pipeline (existing mascarade flow)
5. Sends LLM response text to TTS
6. Encodes TTS output as OPUS
7. Streams OPUS frames back to ESP32

This endpoint does not exist yet in mascarade. It will be added as a new route in the bridge service.

TTS Services

Two TTS options are provided via Docker Compose (tools/dev/docker-compose.tts.yml):

Piper TTS (recommended for low latency)

• CPU-only, runs on VM (192.168.0.119)
• French voice: fr_FR-siwis-medium or fr_FR-upmc-medium
• Latency: ~200ms for short sentences
• OpenAI-compatible API on port 8000

Coqui XTTS-v2 (for voice cloning)

• GPU required, deploy on KXKM-AI (RTX 4090)
• Can clone Professeur Zacus's voice from ~30s of sample audio
• Higher quality but higher latency (~1-2s)
• API on port 5002

STT Service

• Recommended: faster-whisper with large-v3 model
• Language: French (--language fr)
• Deploy on: KXKM-AI for GPU inference, or VM for CPU (slower)
• Alternative: whisper.cpp for CPU-optimized inference

Custom Wake Word — Espressif Process

To get a custom "Professeur Zacus" wake word model for WakeNet9:

1. Contact Espressif via https://www.espressif.com/en/contact-us
2. Provide:
   • Target phrase: "Professeur Zacus"
   • Language: French
   • 200+ audio recordings of the phrase (diverse speakers, environments)
   • Target platform: ESP32-S3 with WakeNet9
3. Timeline: 2-4 weeks
4. Cost: Varies (may be free for open-source/educational projects)
5. Delivery: .wn9 model file to flash to partition

Recording Tips for Training Data

• Record in the target environment (exhibition space)
• Include male/female voices, different ages
• Include background noise variations
• Record at 16kHz, 16-bit, mono WAV
• Minimum 200 recordings, ideally 500+
• Use the INMP441 mic itself for best acoustic match

French Voice Command Strategy

Local MultiNet is NOT recommended for French because:

• MultiNet only supports Chinese, English, and a few other languages
• French phonetics are complex (liaisons, nasals, silent letters)
• Limited command vocabulary even in supported languages

Strategy: Server-side ASR for all speech recognition

1. Wake word detection: Local (WakeNet9, language-agnostic acoustic model)
2. All speech-to-text: Server-side via Whisper (excellent French support)
3. Intent parsing: LLM-based via mascarade (natural language understanding)
4. No local MultiNet needed — everything after wake word goes to server

This simplifies the ESP32 firmware (no MultiNet model in flash) and gives unlimited French vocabulary through the LLM.

Migration Path — Arduino to ESP-IDF

Phase 1: Scaffold (current)

• Voice pipeline header + stubs in Arduino framework
• No ESP-SR dependency
• Compiles and runs (functions return safe defaults)

Phase 2: Arduino-as-Component

• Switch PlatformIO to framework = arduino, espidf
• Add esp-sr as IDF component
• Implement I2S mic input and AFE initialization
• Test wake word detection with "Hi Lexin" placeholder
• Keep all existing Arduino code unchanged

Phase 3: WebSocket Streaming

• Add OPUS encoder (IDF component)
• Implement WebSocket client for mascarade bridge
• Stream audio on wake detection
• Receive and play TTS responses

Phase 4: Production

• Order custom "Professeur Zacus" wake word
• Full ESP-IDF migration (optional, for memory optimization)
• AEC tuning (echo cancellation between speaker and mic)
• LED/display feedback during voice states

Memory Budget

Component	RAM Type	Size	Notes
WakeNet9	PSRAM	~340 KB	Single wake word model
AFE (1ch)	PSRAM	~1.1 MB	AEC + noise suppression + VAD
OPUS codec	Internal	~50 KB	Encoder + decoder
Audio buffers	PSRAM	~64 KB	Ring buffers for I2S + WebSocket
Total voice	Mixed	~1.5 MB
LVGL + UI	PSRAM	~2 MB	Existing allocation
Camera	PSRAM	~500 KB	JPEG buffer
System total	PSRAM	~4 MB	Of 8 MB available

File Structure

ESP32_ZACUS/ui_freenove_allinone/
├── include/voice/
│   └── voice_pipeline.h          # Public API (this scaffold)
├── src/voice/
│   └── voice_pipeline.cpp        # Stub implementation
└── ...

docs/voice/
└── VOICE_PIPELINE_GUIDE.md       # This document

tools/dev/
└── docker-compose.tts.yml        # TTS services (Piper + XTTS-v2)

References

• XiaoZhi-ESP32 — Open-source voice assistant for ESP32-S3
• ESP-SR Programming Guide — WakeNet, MultiNet, AFE
• Piper TTS — Fast local TTS with French models
• Coqui XTTS-v2 — Voice cloning TTS
• faster-whisper — CTranslate2-based Whisper
• OPUS codec — Low-latency audio codec