From 8ad309de33bc04ecfebe5806ff6e26098f937981 Mon Sep 17 00:00:00 2001 From: clement Date: Sun, 28 Jun 2026 12:44:13 +0200 Subject: [PATCH] docs: air-piano implementation plan 6 tasks (TDD): Python strike detector + bridge + publisher wiring; SC note engine + modulation; loading + env docs. Default instrument follows live ~melodyInst via \live. --- .../2026-06-28-air-piano-iphone-fingers.md | 880 ++++++++++++++++++ ...6-06-28-air-piano-iphone-fingers-design.md | 5 +- 2 files changed, 883 insertions(+), 2 deletions(-) create mode 100644 docs/superpowers/plans/2026-06-28-air-piano-iphone-fingers.md diff --git a/docs/superpowers/plans/2026-06-28-air-piano-iphone-fingers.md b/docs/superpowers/plans/2026-06-28-air-piano-iphone-fingers.md new file mode 100644 index 0000000..4e27161 --- /dev/null +++ b/docs/superpowers/plans/2026-06-28-air-piano-iphone-fingers.md @@ -0,0 +1,880 @@ +# Air-Piano (iPhone fingers into SuperCollider) Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let the performer play notes "like a piano" in the air — each finger +strikes a scale degree, hand position shifts octave, hand spacing modulates +global params — by bringing per-finger iPhone hand data into SuperCollider. + +**Architecture:** Hybrid. Python (`data_only_viz`) detects finger strikes from +the raw 21-joint hand data it already holds and emits a discrete OSC +`/pose/finger` event per strike on the existing `:57121` channel. SuperCollider +(`sound_algo`) owns all musicality (scale, finger→degree, octave, instrument, +velocity) and continuous modulation read from the `/pose/hands` data already in +`~handFeat`. + +**Tech Stack:** Python 3.11+ via `uv`, `python-osc` (`SimpleUDPClient`), numpy, +pytest. SuperCollider (sclang) `OSCdef`, `Synth`, `Bus.control`. + +## Global Constraints + +- Python: use `uv` for everything (never pip/poetry/conda directly). +- Python tests live in `data_only_viz/tests/`, run with + `cd data_only_viz && uv run pytest tests/ -v`. +- Mutate `state.*` only under `with state.lock():`. +- Guard every optional ML/import with `try/except ImportError` (n/a here, no new + heavy deps). +- SC env vars are lowercase `~xxx` (uppercase parses as CLASSNAME). +- `.load`-compatible `.scd` files MUST have exactly ONE top-level `(...)` block + and pass the parens/brackets balance check (P:0 B:0). Use the + `validating-scd-files` skill after each `.scd` edit. +- Commits: subject ≤ 50 chars, body ≤ 72 chars/line, no AI attribution, no + `--no-verify`, no underscore in the commit scope. +- No emojis in code/docs/commits. +- Comments: French in `sound_algo/*.scd` (project convention), English + elsewhere. + +--- + +### Task 1: Python — `FingerStrikeDetector` + +Pure signal logic: detect per-finger downward strikes from raw hand joints, +isolating finger flexion from whole-hand motion by measuring the fingertip +relative to its base knuckle. No I/O, fully unit-testable. + +**Files:** +- Create: `data_only_viz/finger_strike.py` +- Test: `data_only_viz/tests/test_finger_strike.py` + +**Interfaces:** +- Consumes: a `hands` list shaped like `state.persons_hands` — a list of hands, + each hand a sequence of ≥21 landmarks where each landmark is indexable + (`lm[0]=x, lm[1]=y, lm[2]=z`) or has `.x/.y/.z`. Reuses `_coord`, `_finite`, + `_clamp` from `data_only_viz/hand_features.py`. +- Produces: + - `class StrikeEvent` (dataclass) with fields + `hand:int, finger:int, strike_speed:float, z:float, tipx:float, tipy:float`. + - `class FingerStrikeDetector` with + `__init__(self, vel_thresh=0.02, refractory_ms=120.0, speed_scale=0.10, history_slots=2)` + and `step(self, hands:list, t_now:float) -> list[StrikeEvent]`. + - Module constants `FINGERTIPS=(4,8,12,16,20)` and `FINGER_BASES=(2,5,9,13,17)`. + +- [ ] **Step 1: Write the failing test** + +Create `data_only_viz/tests/test_finger_strike.py`: + +```python +"""Tests for FingerStrikeDetector (air-piano strike detection).""" +from __future__ import annotations + +from data_only_viz.finger_strike import ( + FingerStrikeDetector, + StrikeEvent, + FINGERTIPS, + FINGER_BASES, +) + + +def _hand(tip_y_by_finger: dict[int, float], base_y: float = 0.4, + cx: float = 0.3) -> list[list[float]]: + """Build a 21-landmark hand. Every base knuckle sits at base_y; each + fingertip sits at base_y unless overridden in tip_y_by_finger (keyed by + finger 0..4). x is set near cx so L/R slotting is deterministic.""" + lm = [[cx, base_y, 0.0] for _ in range(21)] + for f, base_idx in enumerate(FINGER_BASES): + lm[base_idx] = [cx, base_y, 0.0] + for f, tip_idx in enumerate(FINGERTIPS): + ty = tip_y_by_finger.get(f, base_y) + lm[tip_idx] = [cx, ty, 0.0] + return lm + + +def test_downward_spike_fires_exactly_one_strike(): + det = FingerStrikeDetector(vel_thresh=0.02, refractory_ms=120.0) + # frame 0: neutral (primes prev), index tip level with base + det.step([_hand({1: 0.40})], t_now=0.00) + # frame 1: index tip drops 0.06 below -> downward velocity 0.06 > thresh + e1 = det.step([_hand({1: 0.46})], t_now=0.04) + # frame 2: tip stays down -> velocity ~0, must NOT refire + e2 = det.step([_hand({1: 0.46})], t_now=0.08) + strikes = e1 + e2 + assert len(strikes) == 1 + assert strikes[0].finger == 1 + assert strikes[0].hand == 0 + assert strikes[0].strike_speed > 0.0 + + +def test_whole_hand_translation_does_not_fire(): + det = FingerStrikeDetector(vel_thresh=0.02) + out = [] + # tip and base move down together each frame -> relative y constant + for i, by in enumerate((0.40, 0.50, 0.60, 0.70)): + out += det.step([_hand({1: by}, base_y=by)], t_now=i * 0.04) + assert out == [] + + +def test_refractory_blocks_second_strike(): + det = FingerStrikeDetector(vel_thresh=0.02, refractory_ms=120.0) + det.step([_hand({1: 0.40})], t_now=0.00) # prime + a = det.step([_hand({1: 0.46})], t_now=0.02) # strike 1 + det.step([_hand({1: 0.40})], t_now=0.04) # lift -> rearm + b = det.step([_hand({1: 0.46})], t_now=0.06) # within 120 ms -> blocked + assert len(a) == 1 + assert b == [] + + +def test_strike_speed_scales_with_velocity(): + soft = FingerStrikeDetector(vel_thresh=0.02, speed_scale=0.10) + hard = FingerStrikeDetector(vel_thresh=0.02, speed_scale=0.10) + soft.step([_hand({1: 0.40})], t_now=0.0) + hard.step([_hand({1: 0.40})], t_now=0.0) + s = soft.step([_hand({1: 0.44})], t_now=0.04) # delta 0.04 + h = hard.step([_hand({1: 0.50})], t_now=0.04) # delta 0.10 + assert h[0].strike_speed > s[0].strike_speed +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd data_only_viz && uv run pytest tests/test_finger_strike.py -v` +Expected: FAIL with `ModuleNotFoundError: No module named 'data_only_viz.finger_strike'` + +- [ ] **Step 3: Write minimal implementation** + +Create `data_only_viz/finger_strike.py`: + +```python +"""Air-piano finger strike detection from raw MediaPipe hand joints. + +Mirrors hand_features.py conventions (L/R slotting by cx, finite-guarded +coordinate extraction). A "strike" is a fast downward motion of a fingertip +RELATIVE to its base knuckle, so translating the whole hand does not fire all +fingers. Output feeds the OSC /pose/finger route consumed by SuperCollider. +""" +from __future__ import annotations + +from dataclasses import dataclass + +from data_only_viz.hand_features import _clamp, _coord, _finite + +# MediaPipe 21-kp hand: fingertip and base-knuckle indices per finger +# (thumb, index, middle, ring, pinky). Thumb base = ThumbMP (2). +FINGERTIPS: tuple[int, ...] = (4, 8, 12, 16, 20) +FINGER_BASES: tuple[int, ...] = (2, 5, 9, 13, 17) + + +@dataclass +class StrikeEvent: + hand: int # 0 = left slot (leftmost cx), 1 = right slot + finger: int # 0..4 = thumb, index, middle, ring, pinky + strike_speed: float + z: float + tipx: float + tipy: float + + +class _FingerState: + __slots__ = ("prev_rel", "armed", "last_t") + + def __init__(self) -> None: + self.prev_rel: float | None = None + self.armed: bool = True + self.last_t: float = -1e9 + + +class FingerStrikeDetector: + def __init__(self, vel_thresh: float = 0.02, refractory_ms: float = 120.0, + speed_scale: float = 0.10, history_slots: int = 2) -> None: + self.vel_thresh = vel_thresh + self.refractory_s = refractory_ms / 1000.0 + self.speed_scale = max(1e-6, speed_scale) + # state[slot][finger] + self._state = [[_FingerState() for _ in range(5)] + for _ in range(history_slots)] + + def _slot_hands(self, hands: list) -> list: + """Validate and order hands leftmost-first (slot 0 = L), like + HandFeatureExtractor. Returns up to 2 (cx, landmarks) ordered.""" + cand = [] + for lm in hands: + try: + if lm is None or len(lm) < 21: + continue + except TypeError: + continue + xs = [_finite(_coord(p, "x", 0), 0.5) for p in lm[:21]] + cx = _clamp(sum(xs) / len(xs), 0.0, 1.0) + cand.append((cx, lm)) + cand.sort(key=lambda c: c[0]) + return cand[:len(self._state)] + + def step(self, hands: list, t_now: float) -> list[StrikeEvent]: + ordered = self._slot_hands(hands) + present = set(range(len(ordered))) + events: list[StrikeEvent] = [] + for slot, (_cx, lm) in enumerate(ordered): + for f in range(5): + tip = lm[FINGERTIPS[f]] + base = lm[FINGER_BASES[f]] + tip_y = _finite(_coord(tip, "y", 1), 0.5) + base_y = _finite(_coord(base, "y", 1), 0.5) + rel = tip_y - base_y # +down (image y grows downward) + st = self._state[slot][f] + if st.prev_rel is None: + st.prev_rel = rel + continue + vel = rel - st.prev_rel # +down velocity per frame + st.prev_rel = rel + if vel < 0.0: # lifting -> rearm + st.armed = True + if (vel > self.vel_thresh and st.armed + and (t_now - st.last_t) >= self.refractory_s): + st.armed = False + st.last_t = t_now + events.append(StrikeEvent( + hand=slot, finger=f, + strike_speed=_clamp(vel / self.speed_scale, 0.0, 1.0), + z=_finite(_coord(tip, "z", 2, 0.0), 0.0), + tipx=_finite(_coord(tip, "x", 0), 0.5), + tipy=tip_y, + )) + # reset slots not present this tick so re-entry does not spike + for slot in range(len(self._state)): + if slot not in present: + for f in range(5): + self._state[slot][f].prev_rel = None + self._state[slot][f].armed = True + return events +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd data_only_viz && uv run pytest tests/test_finger_strike.py -v` +Expected: PASS (4 passed) + +- [ ] **Step 5: Commit** + +```bash +git add data_only_viz/finger_strike.py data_only_viz/tests/test_finger_strike.py +git commit -m "feat: add finger strike detector" +``` + +--- + +### Task 2: Python — `PoseSoundBridge.send_finger()` + +Add the OSC emitter for the new `/pose/finger` event, mirroring `send_hands`. + +**Files:** +- Modify: `data_only_viz/pose_bridge.py` (class `PoseSoundBridge`, after + `send_hands`) +- Test: `data_only_viz/tests/test_pose_bridge_finger.py` + +**Interfaces:** +- Consumes: nothing new (uses the existing `self._client` SimpleUDPClient bound + to `127.0.0.1:57121` from `PoseSoundBridge.__init__`). +- Produces: `PoseSoundBridge.send_finger(self, ev) -> None` where `ev` is a + `StrikeEvent` (duck-typed: needs `.hand .finger .strike_speed .z .tipx + .tipy`). Emits OSC `/pose/finger` with args + `[pid=0, hand, finger, strike_speed, z, tipx, tipy]` (ints for pid/hand/finger, + floats for the rest). Strike events are NOT throttled (they are already + rate-limited by the detector's refractory window). + +- [ ] **Step 1: Write the failing test** + +Create `data_only_viz/tests/test_pose_bridge_finger.py`: + +```python +"""Tests for PoseSoundBridge.send_finger.""" +from __future__ import annotations + +from unittest.mock import MagicMock + +from data_only_viz.finger_strike import StrikeEvent +from data_only_viz.pose_bridge import PoseSoundBridge + + +def _bridge() -> PoseSoundBridge: + b = PoseSoundBridge() + b._client = MagicMock() # stub the UDP client + return b + + +def test_send_finger_emits_expected_osc(): + b = _bridge() + ev = StrikeEvent(hand=1, finger=2, strike_speed=0.5, z=0.1, + tipx=0.7, tipy=0.6) + b.send_finger(ev) + b._client.send_message.assert_called_once() + addr, args = b._client.send_message.call_args[0] + assert addr == "/pose/finger" + assert args[0] == 0 # pid + assert args[1] == 1 # hand + assert args[2] == 2 # finger + assert abs(args[3] - 0.5) < 1e-6 + assert len(args) == 7 +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd data_only_viz && uv run pytest tests/test_pose_bridge_finger.py -v` +Expected: FAIL with `AttributeError: 'PoseSoundBridge' object has no attribute 'send_finger'` + +- [ ] **Step 3: Write minimal implementation** + +In `data_only_viz/pose_bridge.py`, add this method to `PoseSoundBridge` +immediately after the `send_hands` method (search for `def send_hands`): + +```python + def send_finger(self, ev) -> None: + """Emit one air-piano strike event. Not throttled: the detector's + refractory window already rate-limits these.""" + args = [ + 0, # pid (single performer) + int(ev.hand), + int(ev.finger), + float(ev.strike_speed), + float(ev.z), + float(ev.tipx), + float(ev.tipy), + ] + try: + self._client.send_message("/pose/finger", args) + except OSError: + pass + self._vj("/pose/finger", args) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `cd data_only_viz && uv run pytest tests/test_pose_bridge_finger.py -v` +Expected: PASS (1 passed) + +- [ ] **Step 5: Commit** + +```bash +git add data_only_viz/pose_bridge.py data_only_viz/tests/test_pose_bridge_finger.py +git commit -m "feat: emit /pose/finger over osc bridge" +``` + +--- + +### Task 3: Python — wire strike emission into `ActionHeadPublisher` + +Run the detector each tick on `state.persons_hands` (under lock), gated by +`FINGER_PIANO`, and emit each strike. Thresholds come from env vars. + +**Files:** +- Modify: `data_only_viz/action_head_pub.py` + (`ActionHeadPublisher.__init__` and `_tick`; add `_emit_fingers`) +- Test: `data_only_viz/tests/test_action_head_finger_emit.py` + +**Interfaces:** +- Consumes: `FingerStrikeDetector.step` (Task 1), `PoseSoundBridge.send_finger` + (Task 2), `state.persons_hands` (existing), `state.lock()` (existing). +- Produces: `ActionHeadPublisher._emit_fingers(self, t_now: float) -> None`, + called from `_tick` after `_emit_hands`. No-op when `FINGER_PIANO` env is unset + or `"0"`. Reads env once in `__init__`: + `FINGER_PIANO` (enable), `FINGER_STRIKE_VEL` (float, default 0.02), + `FINGER_STRIKE_REFRACTORY_MS` (float, default 120). + +- [ ] **Step 1: Write the failing test** + +Create `data_only_viz/tests/test_action_head_finger_emit.py`: + +```python +"""ActionHeadPublisher emits /pose/finger when FINGER_PIANO is enabled.""" +from __future__ import annotations + +import threading +from unittest.mock import MagicMock + +from data_only_viz.finger_strike import FINGERTIPS, FINGER_BASES + + +class _FakeState: + def __init__(self) -> None: + self.persons_hands = [] + self._lock = threading.RLock() + + def lock(self): + return self._lock + + +def _hand(index_tip_y: float, base_y: float = 0.4): + lm = [[0.3, base_y, 0.0] for _ in range(21)] + lm[FINGERTIPS[1]] = [0.3, index_tip_y, 0.0] + lm[FINGER_BASES[1]] = [0.3, base_y, 0.0] + return lm + + +def _pub(monkeypatch, enabled: bool): + monkeypatch.setenv("FINGER_PIANO", "1" if enabled else "0") + monkeypatch.setenv("FINGER_STRIKE_VEL", "0.02") + monkeypatch.setenv("FINGER_STRIKE_REFRACTORY_MS", "120") + from data_only_viz.action_head_pub import ActionHeadPublisher + # bypass __init__ heavy ckpt load: construct then set only what we touch + pub = ActionHeadPublisher.__new__(ActionHeadPublisher) + pub.state = _FakeState() + pub.bridge = MagicMock() + pub._init_finger_piano() # helper that reads env + builds detector + return pub + + +def test_emit_fingers_fires_when_enabled(monkeypatch): + pub = _pub(monkeypatch, enabled=True) + pub.state.persons_hands = [_hand(0.40)] + pub._emit_fingers(0.00) # prime + pub.state.persons_hands = [_hand(0.46)] # downward strike + pub._emit_fingers(0.04) + assert pub.bridge.send_finger.called + + +def test_emit_fingers_noop_when_disabled(monkeypatch): + pub = _pub(monkeypatch, enabled=False) + pub.state.persons_hands = [_hand(0.40)] + pub._emit_fingers(0.00) + pub.state.persons_hands = [_hand(0.46)] + pub._emit_fingers(0.04) + assert not pub.bridge.send_finger.called +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `cd data_only_viz && uv run pytest tests/test_action_head_finger_emit.py -v` +Expected: FAIL with `AttributeError: ... has no attribute '_init_finger_piano'` + +- [ ] **Step 3: Write minimal implementation** + +In `data_only_viz/action_head_pub.py`: + +(3a) Add the import near the top of `__init__`'s body sibling imports — at the +end of `__init__` (after `self._hand_ext = HandFeatureExtractor()`, line ~102), +add: + +```python + self._init_finger_piano() +``` + +(3b) Add these two methods to `ActionHeadPublisher` (place after `_emit_hands`): + +```python + def _init_finger_piano(self) -> None: + """Read FINGER_PIANO env config and build the strike detector.""" + import os + from data_only_viz.finger_strike import FingerStrikeDetector + self._finger_enabled = os.environ.get("FINGER_PIANO", "0") not in ( + "0", "", "false", "False", + ) + vel = float(os.environ.get("FINGER_STRIKE_VEL", "0.02")) + refr = float(os.environ.get("FINGER_STRIKE_REFRACTORY_MS", "120")) + self._finger_det = FingerStrikeDetector( + vel_thresh=vel, refractory_ms=refr, + ) + + def _emit_fingers(self, t_now: float) -> None: + """Detect finger strikes on persons_hands and emit /pose/finger.""" + if not getattr(self, "_finger_enabled", False): + return + with self.state.lock(): + hands = list(getattr(self.state, "persons_hands", None) or []) + for ev in self._finger_det.step(hands, t_now): + self.bridge.send_finger(ev) +``` + +(3c) In `_tick`, after the existing `self._emit_hands(t_now)` line (line ~147), +add: + +```python + self._emit_fingers(t_now) +``` + +- [ ] **Step 4: Run tests to verify they pass** + +Run: `cd data_only_viz && uv run pytest tests/test_action_head_finger_emit.py tests/test_action_head_pub.py -v` +Expected: PASS (existing action-head pub tests stay green; 2 new tests pass) + +- [ ] **Step 5: Commit** + +```bash +git add data_only_viz/action_head_pub.py data_only_viz/tests/test_action_head_finger_emit.py +git commit -m "feat: wire finger strikes into pose publisher" +``` + +--- + +### Task 4: SC — `finger_piano.scd` (resolve + OSCdef + `~fp` API + modulation) + +The air-piano brain in SuperCollider. A single top-level `(...)` block, +`.load`-compatible. Note resolution is factored into the pure function +`~fpResolve` so it can be unit-tested without a running server. + +**Files:** +- Create: `sound_algo/control/finger_piano.scd` +- Test: `sound_algo/tests/test_finger_piano.scd` + +**Interfaces:** +- Consumes: `~handFeat` dict (from `data_feeds.scd`: `\lx \ly \rx \ry \dist`), + `~melodyInst` (current live instrument symbol), `palette/scales.scd` scale + arrays (`~scaleMinorPent`, `~scaleDorian`, ...). MIDI→Hz via `.midicps`. +- Produces (env vars / functions): + - `~fpEnabled` (Boolean), `~fpInst` (Symbol or `\live`), `~fpScaleArr` + (Array of MIDI notes), `~fpMap` (`[[5 L degrees],[5 R degrees]]`), + `~fpHandOct` (`[Lsemi, Rsemi]`), `~fpBaseOct` (Integer octaves), + `~fpOctSpan` (Float octaves of hand-x transpose), `~fpSens` (Float), + `~fpVelSource` (`\strike`|`\z`), `~fpHold` (Float seconds), `~fpAmp` (Float). + - `~fpResolve.(hand, finger, strikeSpeed, z, tipx, tipy)` → + `Event` with `\note` (Integer MIDI), `\freq` (Float Hz), `\amp` (Float), + `\inst` (Symbol). Pure; reads `~handFeat`/state but triggers nothing. + - `~fpResolveInst.()` → Symbol (the `\live` sentinel resolves to + `~melodyInst ? \pluck`). + - `~fpModTick.()` → updates `~fpCut` and `~fpFx` (Floats, one-pole smoothed) + from `~handFeat[\dist]` and mean hand Y; writes to control buses + `~fpCutBus`/`~fpFxBus` when the server is running. + - Live wrappers: `~fpOn`, `~fpOff`, `~fpInst_`, `~fpScale_`, `~fpSens_`, + `~fpOctave_`, `~fpVelSource_`. + - `OSCdef(\poseFinger)` on `/pose/finger` calling `~fpResolve` then triggering + `Synth`. + +- [ ] **Step 1: Write the failing test** + +Create `sound_algo/tests/test_finger_piano.scd`: + +```supercollider +// Headless unit test for finger_piano note resolution. +// Run: sclang sound_algo/tests/test_finger_piano.scd +( +var base, ok = true, ev; +base = thisProcess.nowExecutingPath.dirname.dirname ++ "/"; +// minimal deps: scales + a fake handFeat + load finger_piano +(base ++ "palette/scales.scd").load; +~handFeat = Dictionary.new; +~handFeat[\lx] = 0.5; ~handFeat[\rx] = 0.5; +~handFeat[\ly] = 0.5; ~handFeat[\ry] = 0.5; +~melodyInst = \pluck; +(base ++ "control/finger_piano.scd").load; + +// default scale is minor pentatonic; left thumb (hand 0, finger 0) -> degree 0 +ev = ~fpResolve.(0, 0, 1.0, 0.0, 0.5, 0.5); +("note L thumb = " ++ ev[\note]).postln; +if(ev[\note] != ~fpScaleArr.wrapAt(0)) { + ok = false; "FAIL: L thumb note wrong".postln; +}; + +// right hand defaults one octave up (~fpHandOct[1] = 12) +ev = ~fpResolve.(1, 0, 1.0, 0.0, 0.5, 0.5); +if(ev[\note] != (~fpScaleArr.wrapAt(0) + 12)) { + ok = false; "FAIL: R thumb octave wrong".postln; +}; + +// instrument follows live via \live sentinel +~fpInst = \live; +if(~fpResolveInst.() != \pluck) { ok = false; "FAIL: live inst".postln; }; +~fpInst = \fm_bell; +if(~fpResolveInst.() != \fm_bell) { ok = false; "FAIL: pinned inst".postln; }; + +// modulation tick updates ~fpCut from dist +~handFeat[\dist] = 1.0; +~fpModTick.(); +if(~fpCut.isNil) { ok = false; "FAIL: fpCut nil".postln; }; + +if(ok) { "PASS test_finger_piano".postln } { "TESTS FAILED".postln }; +0.exit; +) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `sclang sound_algo/tests/test_finger_piano.scd` +Expected: FAIL — sclang error that `control/finger_piano.scd` does not exist +(open failed / `~fpResolve` is nil). + +- [ ] **Step 3: Write minimal implementation** + +Create `sound_algo/control/finger_piano.scd`: + +```supercollider +// ===================================================================== +// control/finger_piano.scd -- Air-piano : doigts iPhone -> notes SC +// +// Recoit /pose/finger (emis par data_only_viz quand FINGER_PIANO=1) : +// /pose/finger +// Chaque doigt = un degre de gamme. La position de main decale l'octave, +// la vitesse de frappe donne la velocite. Module aussi des params globaux +// depuis l'ecart des mains (~handFeat[\dist]). +// +// Charge par data_only/boot.scd et boot.data-only.scd apres data_feeds.scd. +// Un seul bloc top-level (compatible .load). +// ===================================================================== +( +// ---------- ETAT / CONFIG (idempotent : ne reset pas si deja defini) ---- +~fpEnabled = ~fpEnabled ? false; +~fpInst = ~fpInst ? \live; // \live = suit ~melodyInst +~fpScaleArr = ~fpScaleArr ? ~scaleMinorPent; +~fpMap = ~fpMap ? [[0,1,2,3,4], [0,1,2,3,4]]; // [L, R] degres +~fpHandOct = ~fpHandOct ? [0, 12]; // L octave basse, R +1 octave +~fpBaseOct = ~fpBaseOct ? 0; // octaves globales en plus +~fpOctSpan = ~fpOctSpan ? 1; // +/- octaves selon x de la main +~fpSens = ~fpSens ? 1.0; +~fpVelSource = ~fpVelSource ? \strike; // \strike ou \z +~fpHold = ~fpHold ? 0.35; // duree avant gate=0 (sec) +~fpAmp = ~fpAmp ? 0.3; +~fpCut = ~fpCut ? 0.0; +~fpFx = ~fpFx ? 0.0; + +// Bus de modulation continue (sorties a mapper sur un synth/param). +if(~fpCutBus.isNil and: { s.serverRunning }) { + ~fpCutBus = Bus.control(s, 1); + ~fpFxBus = Bus.control(s, 1); +}; + +// Gammes nommees accessibles via ~fpScale_ +~fpScales = ( + minorPent: ~scaleMinorPent, minor: ~scaleMinor, dorian: ~scaleDorian, + phrygian: ~scalePhrygian, blues: ~scaleBlues, japanese: ~scaleJapanese, + hijaz: ~scaleHijaz, hirajoshi: ~scaleHirajoshi +); + +// ---------- RESOLUTION (pure, testable) --------------------------------- +~fpResolveInst = { + if(~fpInst == \live or: { ~fpInst.isNil }) { ~melodyInst ? \pluck } + { ~fpInst }; +}; + +~fpResolve = { |hand, finger, strikeSpeed = 1.0, z = 0.0, tipx = 0.5, tipy = 0.5| + var deg, handX, octFromX, transpose, note, vel, amp; + deg = ~fpMap[hand][finger] ? 0; + handX = if(hand == 0) { ~handFeat[\lx] ? 0.5 } { ~handFeat[\rx] ? 0.5 }; + // x autour de 0.5 -> +/- ~fpOctSpan octaves + octFromX = ((handX - 0.5) * 2 * ~fpOctSpan).round(1.0).asInteger; + transpose = (~fpHandOct[hand] ? 0) + (~fpBaseOct * 12) + (octFromX * 12); + note = (~fpScaleArr.wrapAt(deg) + transpose).asInteger; + vel = if(~fpVelSource == \z) { z } { strikeSpeed }; + amp = (~fpAmp * (0.25 + (0.75 * vel)) * ~fpSens).clip(0.0, 1.0); + (note: note, freq: note.midicps, amp: amp, inst: ~fpResolveInst.()); +}; + +// ---------- MODULATION CONTINUE ----------------------------------------- +// dist (0..1) -> cutoff exp 200..6000 Hz ; mean Y (0..1) -> fx 0..1 +~fpModTick = { + var dist, meanY, cutHz, fx; + dist = (~handFeat[\dist] ? 0.0).clip(0.0, 1.0); + meanY = (((~handFeat[\ly] ? 0.5) + (~handFeat[\ry] ? 0.5)) * 0.5) + .clip(0.0, 1.0); + // lissage one-pole + ~fpCut = ~fpCut + (0.2 * (dist - ~fpCut)); + ~fpFx = ~fpFx + (0.2 * (meanY - ~fpFx)); + cutHz = (200 * (30 ** ~fpCut)); // 200 -> 6000 Hz + fx = ~fpFx; + if(~fpCutBus.notNil) { ~fpCutBus.set(cutHz) }; + if(~fpFxBus.notNil) { ~fpFxBus.set(fx) }; +}; + +// ---------- OSC : declenchement note ------------------------------------ +OSCdef(\poseFinger, { |msg| + var ev, syn; + if(~fpEnabled == true) { + // msg = [/pose/finger, pid, hand, finger, strikeSpeed, z, tipx, tipy] + ev = ~fpResolve.(msg[2].asInteger, msg[3].asInteger, + msg[4], msg[5], msg[6], msg[7]); + if(s.serverRunning) { + syn = Synth(ev[\inst], [\freq, ev[\freq], \amp, ev[\amp], \gate, 1]); + SystemClock.sched(~fpHold, { syn.set(\gate, 0); nil }); + }; + }; +}, '/pose/finger'); + +// ---------- API LIVE (env-direct wrappers) ------------------------------ +~fpOn = { ~fpEnabled = true; "[fp] air-piano ON".postln; }; +~fpOff = { ~fpEnabled = false; "[fp] air-piano OFF".postln; }; +~fpInst_ = { |sym| ~fpInst = sym; ("[fp] inst = " ++ sym).postln; }; +~fpScale_ = { |name| + ~fpScaleArr = ~fpScales[name] ? ~fpScaleArr; + ("[fp] scale = " ++ name).postln; +}; +~fpSens_ = { |v| ~fpSens = v; }; +~fpOctave_ = { |n| ~fpBaseOct = n; }; +~fpVelSource_ = { |sym| ~fpVelSource = sym; }; + +"[OK] finger_piano : ~fpOn / ~fpScale_(\\dorian) / ~fpInst_(\\fm_bell)".postln; +) +``` + +- [ ] **Step 4: Run test + balance check to verify they pass** + +Run: `sclang sound_algo/tests/test_finger_piano.scd` +Expected: `PASS test_finger_piano` + +Then run the parens/brackets balance check (must be P:0 B:0): + +Run: +```bash +awk 'BEGIN{p=0;b=0} {for(i=1;i<=length($0);i++){c=substr($0,i,1); if(c=="(")p++; if(c==")")p--; if(c=="[")b++; if(c=="]")b--}} END{print "P:"p" B:"b}' sound_algo/control/finger_piano.scd +``` +Expected: `P:0 B:0` + +- [ ] **Step 5: Commit** + +```bash +git add sound_algo/control/finger_piano.scd sound_algo/tests/test_finger_piano.scd +git commit -m "feat: air-piano finger note engine in sc" +``` + +--- + +### Task 5: SC — hook modulation into `data_feeds.scd`, wire loading + docs + +Make `~fpModTick` run on every `/pose/hands` frame, load `finger_piano.scd` in +both data-only boot paths, and document the env vars. + +**Files:** +- Modify: `sound_algo/control/data_feeds.scd:502-512` (`\poseHands` OSCdef) +- Modify: `sound_algo/data_only/boot.scd` (after the `data_feeds.scd` load, + line ~151) +- Modify: `sound_algo/boot.data-only.scd` (after the `data_feeds.scd` load, + line ~40) +- Modify: `CLAUDE.md` (Environment variables table) + +**Interfaces:** +- Consumes: `~fpModTick` (Task 4), loaded `finger_piano.scd`. +- Produces: continuous calls to `~fpModTick` at the `/pose/hands` rate (~30 Hz); + `finger_piano.scd` present in the data-only runtime so `/pose/finger` is + handled and `~fp*` API is available. + +- [ ] **Step 1: Add the modulation hook in `data_feeds.scd`** + +In `sound_algo/control/data_feeds.scd`, change the `\poseHands` OSCdef +(line 511 `~handFeat[\dist] = msg[10];`) so the closing of the function calls +the modulation tick. Replace: + +```supercollider + ~handFeat[\dist] = msg[10]; +}, '/pose/hands'); +``` + +with: + +```supercollider + ~handFeat[\dist] = msg[10]; + ~fpModTick !? { ~fpModTick.value }; // air-piano modulation continue +}, '/pose/hands'); +``` + +- [ ] **Step 2: Load `finger_piano.scd` in `data_only/boot.scd`** + +In `sound_algo/data_only/boot.scd`, find the data_feeds load (line ~151, +`(~repoRoot ++ "control/data_feeds.scd").load;`) and add immediately after it: + +```supercollider + (~repoRoot ++ "control/finger_piano.scd").load; +``` + +- [ ] **Step 3: Load `finger_piano.scd` in `boot.data-only.scd`** + +In `sound_algo/boot.data-only.scd`, find the data_feeds load (line ~40, +`(~base ++ "control/data_feeds.scd").load;`) and add immediately after it: + +```supercollider + (~base ++ "control/finger_piano.scd").load; +``` + +- [ ] **Step 4: Document env vars in root `CLAUDE.md`** + +In `/Users/electron/Documents/Projets/AV-Live/CLAUDE.md`, in the second +`## Environment variables` table (the one starting with `POSE_FILTER`), add +these rows at the end of the table body: + +```markdown +| `FINGER_PIANO` | `0` | `1` enables air-piano finger strike emission from data_only_viz | +| `FINGER_STRIKE_VEL` | `0.02` | downward relative-velocity threshold for a strike (normalized image units/frame) | +| `FINGER_STRIKE_REFRACTORY_MS` | `120` | min ms between strikes per finger | +``` + +- [ ] **Step 5: Verify balance of the edited `.scd` and re-run the SC test** + +Run (both must report `P:0 B:0`): +```bash +awk 'BEGIN{p=0;b=0} {for(i=1;i<=length($0);i++){c=substr($0,i,1); if(c=="(")p++; if(c==")")p--; if(c=="[")b++; if(c=="]")b--}} END{print "P:"p" B:"b}' sound_algo/control/data_feeds.scd +sclang sound_algo/tests/test_finger_piano.scd +``` +Expected: `P:0 B:0` and `PASS test_finger_piano` + +- [ ] **Step 6: Commit** + +```bash +git add sound_algo/control/data_feeds.scd sound_algo/data_only/boot.scd sound_algo/boot.data-only.scd CLAUDE.md +git commit -m "feat: load air-piano and hook hand modulation" +``` + +--- + +### Task 6: End-to-end manual smoke (no code) + Python suite green + +Final verification gate. No new code; confirm the whole chain and the full +Python suite. + +**Files:** none (verification only). + +- [ ] **Step 1: Full Python suite stays green** + +Run: `cd data_only_viz && uv run pytest tests/ -v` +Expected: all pass (new finger tests + existing action-head/pose-bridge tests). + +- [ ] **Step 2: Live smoke with iPhone (manual)** + +With the iPhone ARBodyTracker streaming and data_only_viz running: + +```bash +FINGER_PIANO=1 uv run python -m data_only_viz.main +``` + +In the SC data-only session, enable and play: + +```supercollider +~fpOn.(); +~fpScale_.(\dorian); +~fpInst_.(\live); // suit l'instrument melodique courant +``` + +Expected: striking a finger downward triggers a note; right hand sounds one +octave above the left; moving a hand left/right shifts the octave; spreading the +hands raises `~fpCut` (check `~fpCut.postln` rises toward 1.0). Tune +`FINGER_STRIKE_VEL` if strikes are too eager / too stiff. + +- [ ] **Step 3: Commit (only if smoke required a threshold tweak in docs)** + +If you changed a default threshold based on the smoke test, update the +`CLAUDE.md` default and the `FingerStrikeDetector` default to match, then: + +```bash +git add -A +git commit -m "tune: finger strike default threshold" +``` + +--- + +## Self-Review + +**Spec coverage:** +- `/pose/finger` message → Task 2 (emit), Task 3 (wire), Task 4 (receive). OK. +- Python strike detector with relative-to-MCP flexion → Task 1. OK. +- Env vars `FINGER_PIANO` / `FINGER_STRIKE_VEL` / `FINGER_STRIKE_REFRACTORY_MS` + → Task 3 (read), Task 5 step 4 (doc). OK. +- SC air-piano: finger→degree, octave from hand x, velocity from strike/z, + scale resolution, configurable instrument following `~melodyInst` → Task 4 + (`~fpResolve`, `~fpResolveInst`). OK. +- Continuous modulation from `~handFeat[\dist]` + mean position → Task 4 + (`~fpModTick`), Task 5 step 1 (hook). OK. +- `~fp` live API → Task 4. OK. +- Runtime loading in data-only paths → Task 5 steps 2-3. OK. +- Tests: Python pytest (Tasks 1-3) + SC headless test + balance check + (Task 4-5) + manual smoke (Task 6). OK. +- Single-performer assumption (pid=0) → Task 2 emits pid=0. OK. + +**Placeholder scan:** every code step contains full code; no TBD/TODO. OK. + +**Type consistency:** `StrikeEvent` fields (`hand, finger, strike_speed, z, +tipx, tipy`) identical across Tasks 1-3. `send_finger(ev)` arg order matches the +`/pose/finger` arg order consumed by `~fpResolve` (hand=msg[2], finger=msg[3], +strikeSpeed=msg[4], z=msg[5], tipx=msg[6], tipy=msg[7]). `~fpResolve` / +`~fpResolveInst` / `~fpModTick` / `~fpCut` names match between Task 4 impl and +the Task 4 test and the Task 5 hook. OK. diff --git a/docs/superpowers/specs/2026-06-28-air-piano-iphone-fingers-design.md b/docs/superpowers/specs/2026-06-28-air-piano-iphone-fingers-design.md index 77e6abb..a701048 100644 --- a/docs/superpowers/specs/2026-06-28-air-piano-iphone-fingers-design.md +++ b/docs/superpowers/specs/2026-06-28-air-piano-iphone-fingers-design.md @@ -148,8 +148,9 @@ Contents: 5. Trigger the instrument `~fpInst` as a one-shot (`Synth(~fpInst, [\freq, f, \amp, vel, \gate, 1, ...])` with its own envelope `doneAction: 2`, or play through the existing melody instrument - path). Default instrument: an existing pleasant lead/keys SynthDef; no new - SynthDef unless the user later wants a dedicated piano timbre. + path). Default instrument follows the current live melody instrument + (`~melodyInst`) via the `\live` sentinel, reassignable to any SynthDef + (`~fpInst_.(\fm_bell)`); no new SynthDef. - **Continuous modulation hook:** extend `data_feeds.scd`'s `\poseHands` OSCdef with a single call to `~fpModTick.value` (defined in `finger_piano.scd`, a no-op stub if the file is not loaded). `~fpModTick`: