fix(voice): RMS-amplify STT input for weak voice
Repo State / repo-state (push) Failing after 51s
Validate Zacus refactor / validate (pull_request) Failing after 6m24s
Validate Zacus refactor / validate (push) Failing after 5m57s
Repo State / repo-state (pull_request) Failing after 10m40s

Condition the handset capture by RMS (not peak) before whisper: the
voice body is ~0.5-1.5% FS and a transient peak defeated peak-normalise,
leaving it inaudible. RMS-normalise to ~20% FS + clip. Recovers weak
captures whisper returned '' for (verified: -> 'C'est parti !').
This commit was merged in pull request #168.
This commit is contained in:
clement
2026-06-18 00:31:37 +02:00
parent 73fa3650a1
commit 44cd100a80
+15 -14
View File
@@ -1941,17 +1941,18 @@ def _build_wav(pcm: bytes, sr: int, ch: int, sw: int) -> bytes:
return out.getvalue()
def _normalize_wav_for_stt(wav_bytes: bytes, target_peak: float = 0.7,
max_gain: float = 40.0, hp_cutoff_hz: float = 110.0) -> bytes:
"""Condition a PLIP handset capture for Kyutai STT: high-pass then normalise.
def _normalize_wav_for_stt(wav_bytes: bytes, target_rms: float = 0.20,
max_gain: float = 50.0, hp_cutoff_hz: float = 110.0) -> bytes:
"""Condition a PLIP handset capture for the STT: high-pass then RMS-amplify.
Two problems with the SLIC handset path: (1) it injects low-frequency rumble
/ DC drift that swamps the speech and makes Kyutai output nothing — a
high-pass (~110 Hz, box moving-average subtraction) removes it; (2) the mic
comes in very quiet (~2-5 % FS even at +24 dB PGA), so peak-normalise after
filtering (gain capped so silence isn't blown up). WITHOUT the high-pass the
exact same capture transcribes empty; WITH it, clean French — verified on the
bench."""
Two problems with the SLIC handset path: (1) low-frequency rumble/DC drift
that swamps the speech — a high-pass (~110 Hz, box moving-average subtraction)
removes it; (2) the voice body comes in VERY quiet (~0.5-1.5 % FS RMS). We
amplify by RMS (NOT peak — a transient peak defeats peak-normalisation and
leaves the body inaudible) to ~20 % FS, then hard-clip the rare transient.
Verified: weak captures whisper returned '' for transcribe correctly once
RMS-boosted (e.g. → "C'est parti !"). Silence gets blown up to noise but the
whisper hallucination filter drops the result."""
if _np is None:
return wav_bytes
parsed = _wav_pcm(wav_bytes)
@@ -1967,11 +1968,11 @@ def _normalize_wav_for_stt(wav_bytes: bytes, target_peak: float = 0.7,
if arr.size > k:
lp = _np.convolve(arr, _np.ones(k, dtype=_np.float32) / k, mode="same")
arr = arr - lp
peak = float(_np.max(_np.abs(arr))) or 1.0
gain = min((target_peak * 32767.0) / peak, max_gain)
rms = float(_np.sqrt(_np.mean(arr * arr))) or 1.0
gain = min((target_rms * 32767.0) / rms, max_gain)
arr = _np.clip(arr * gain, -32768, 32767).astype(_np.int16)
logging.info("STT conditioning: high-pass %.0f Hz, peak %.1f%% FS → gain x%.1f",
hp_cutoff_hz, 100 * peak / 32768, gain)
logging.info("STT conditioning: high-pass %.0f Hz, RMS %.2f%% FS → gain x%.1f",
hp_cutoff_hz, 100 * rms / 32768, gain)
return _build_wav(arr.tobytes(), sr, ch, sw)