L'électron rare
1c07211dd8
CI build oscope-of / build-check (push) Has been cancelled
Live session outcomes: (1) pinch thresholds validated for inner fingers frozen as defaults (ratio_on 0.50, margin 0.05, ext_min 1); (2) panel frames + X/Y gauges now permanent (no early-return when nothing is detected); (3) wxcvbn source-bundle shortcuts gated off by default (VIZ_SOURCE_KEYS=1 re-enables) — a stray key mid-performance switched the audio scene.
7.9 KiB
7.9 KiB
AV-Live
Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout. RC0.1+ (tag v0.1.0-rc1) ajoute le pipeline Multi-HMR distribué M5 ↔ macm1 sur LAN gigabit + 10 scènes Metal pose-réactives + unified 3D armature wireframe (body+face+hands).
Communication
Toujours répondre en français à l'utilisateur. Code, commentaires de code, commits et docs en anglais.
Stack par sous-projet
| Sous-projet | Stack |
|---|---|
sound_algo/ |
SuperCollider (sclang + scsynth), 1099 SynthDefs, 345 tracks |
oscope-of/ |
openFrameworks C++, libusb (Hantek bulk), GLSL 150 GL 3.2 core |
launcher/ |
SwiftUI menubar app, Swift Package Manager |
data_only_viz/ |
Python 3.11+ via uv, Metal natif (pyobjc), multi-backends pose |
web_realart/ |
Node.js, Express, OSC bridge |
Where to Look
| Tâche | Emplacement |
|---|---|
| Ajouter / modifier un SynthDef, track, palette | sound_algo/ (déjà documenté en nested) |
| Toucher au visualiseur, shaders, FFT, Hantek | oscope-of/ |
| Modifier l'app menubar macOS | launcher/ |
| Détection pose / mesh / body tracking | data_only_viz/ |
| Bridge web / UI de live coding | web_realart/ |
| Plans / specs en cours | docs/superpowers/plans/ |
| Multi-HMR remote server pyobjc | data_only_viz/scripts/multihmr_server.py (tourne sur macm1) |
| Mesh dense rigger 27 fps perçu | data_only_viz/mesh_rigger.py |
| 3D wireframe armature (body+face+hands) | launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift |
| Pose → Metal scenes uniforms | launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift + Resources/scene.metal |
| Filter chain (median + Kalman + lookahead + IK) | data_only_viz/pose_filter.py |
| DINO re-id pid matching | data_only_viz/dino_reid.py |
RC0.1+ environment variables
| Env | Default | Effect |
|---|---|---|
MULTIHMR_BACKEND |
pytorch |
pytorch, coreml, remote |
MULTIHMR_REMOTE_HOST |
127.0.0.1 |
macm1 IP for remote inference |
MULTIHMR_REMOTE_JPEG |
1 |
JPEG q=80 on the wire |
MULTIHMR_REMOTE_ASYNC |
1 |
client double-buffer queue |
MULTIHMR_SERVER_BACKEND |
pyobjc |
server: pyobjc or coremltools |
MULTIHMR_LOOP_FPS |
30 |
Python worker loop target_fps |
AVBODY_HOST |
127.0.0.1 |
route TCP mesh + OSC to remote AVLiveBody |
MEDIAPIPE_DELEGATE |
gpu |
Metal SRGBA (faster); cpu fallback for stability |
POSE_FILTER |
median+kalman+lookahead+ik |
filter chain stages |
MULTIHMR_REID |
dino |
DINO cosine matching, iou fallback |
Network topology
| Host | mDNS | IP (DHCP) | Role |
|---|---|---|---|
| GrosMac M5 | grosmac.local |
LAN | Source + visualisation (AVLiveBody + data_only_viz + data_feeds) |
| macm1 M1 Max | supra-m1.local |
192.168.0.175 |
Sink (Multi-HMR CoreML + Apple Vision ANE + SMPL-X TCP) |
| iPhone 16 Pro | (Personal Hotspot) | DHCP | ARKit/LiDAR pub via OSC /body3d/kp |
AVBODY_HOST / MULTIHMR_REMOTE_HOST accept mDNS hostnames — résiste aux changements DHCP (notamment iPhone hotspot 172.20.10.x).
Environment variables
| Env | Default | Effect |
|---|---|---|
POSE_FILTER |
median+kalman+lookahead+ik |
filter chain stages — extra: one_euro_joints (joint-space CHI 2012 One Euro, inserted before kalman), one_euro_bones (bone-vector One Euro applied after SMPL-X fusion in multi.py), arkit_fuse (overrides 14 body slots with ARKit ARSkeleton3D from the iOS app, expects /body3d/kp on :57128) |
IPHONE_OSC_PORT |
57128 |
UDP port the iPhone ARBodyTracker app pushes /body3d/kp to (always-on listener in data_only_viz) |
VIDEO_ROTATE |
none |
rotate the source video frame before MediaPipe + display: none/ccw/cw/180. Applied at multi.py read; pose detection + skeleton overlay stay aligned. ARKit 3D joints (gravity-aligned world) are unaffected |
ICP_FUSION |
0 |
1 to enable LiDAR↔SMPL-X ICP fusion (cf. docs/ICP_FUSION.md) |
ICP_LIDAR_HOST |
(unset) | iPhone ARBodyTracker IP when ICP_FUSION=1 |
ICP_LIDAR_PORT |
5500 |
iPhone ARMesh TCP port |
ICP_LIDAR_EXTRINSIC |
~/.config/av-live/lidar_extrinsic.json |
extrinsic JSON path |
FINGER_PIANO |
0 |
1 enables air-piano finger strike emission from data_only_viz |
FINGER_STRIKE_VEL |
0.02 |
downward relative-velocity threshold for a strike (normalized image units/frame) |
FINGER_STRIKE_REFRACTORY_MS |
120 |
min ms between strikes per finger |
FINGER_SOURCE |
auto |
hand source for finger gestures: auto (iPhone Vision hands if fresh, else MediaPipe), iphone, mediapipe |
FINGER_DEBUG |
0 |
1 logs each detected strike/pinch (hand, finger) — for threshold tuning |
PINCH_RATIO_ON |
0.50 |
thumb-to-finger distance / hand size below which a pinch engages (0.50 validated live 2026-07-02 for ring/pinky reach) |
PINCH_RATIO_OFF |
0.65 |
distance/size above which the pinch re-arms (hysteresis) |
PINCH_REFRACTORY_MS |
250 |
min ms between pinches per finger |
PINCH_MARGIN |
0.05 |
closest-wins margin between nearest and runner-up fingertip (0.05 validated live: 0.20 rejected middle/ring/pinky pinches) |
PINCH_EXT_RATIO |
1.35 |
tip-to-wrist distance / hand size above which a finger counts as extended (open-hand pinch gate) |
PINCH_EXT_MIN |
1 |
min extended non-pinching fingers required for a pinch to engage; 0 disables the open-hand gate (1 validated live: inner-finger pinches curl neighbours) |
PINCH_DEBOUNCE_FRAMES |
3 |
consecutive qualifying frames before a pinch engage fires (release stays immediate); 1 disables |
HAND_CONF_MIN |
0.45 |
min Vision hand confidence to draw (raised from 0.3; applies to both overlay and side panels) |
HAND_PERSIST_FRAMES |
3 |
consecutive frames a hand track must match before being drawn (1 = off, all hands drawn immediately) |
HAND_SWAP_LR |
0 |
Safety knob: invert the Vision chirality interpretation when routing hands (gestures + panels). Chirality validated correct live 2026-07-02; default 0. Set 1 if a future iPhone app build delivers inverted chirality |
HAND_NEAR_MIN |
0.10 |
Minimum wrist-to-middle-MCP distance (normalised) for a hand to trigger pinch/strike. Hands smaller than this are gated out. Display panels are unaffected |
HAND_PERSIST_GRACE |
2 |
grace frames before dropping a hand track (0 = strict per-frame) |
HAND_NEAR_OFF |
0.08 |
hand size hysteresis off-threshold for gesture slot stabilizer |
HAND_HOLD_FRAMES |
2 |
gesture slot: frames to hold last hand on Vision miss |
HAND_FACE_MIN |
0.5 |
palm-spread ratio threshold — gesture slot arms only when hand_facing >= this (palm-facing ~0.7-1.0, side-on < 0.4; 0.8× hysteresis for off) |
ARKIT_BONE_MAX |
0.5 |
max ARKit body bone length in normalized units; longer bones are skipped (tracking-loss garbage filter) |
PINCH_ENABLE |
0 |
enable pinch gesture detection |
VIZ_HUD |
0 |
show debug HUD overlay |
CONCERT_MIRROR |
1 |
mirror video horizontally (!=0 enables) |
Conventions globales
- Python : uv systématiquement (jamais pip/poetry/conda directs).
- Pas d'emojis dans code/docs/commits sauf demande explicite.
- Commits : sujet ≤ 50 char, body ≤ 72 char/ligne, pas d'attribution AI, pas de
--no-verify, pas d'underscore dans le scope (hooks enforcent). *.pt,*.ckpt,*.safetensors,*.mlpackageexclus par.gitignoreracine.
Agent Workflow
Explore localise → librarian lit (>500 lignes) → tu raisonnes → general-purpose implémente → validator vérifie. Lance les tâches indépendantes en parallèle.
Guidance imbriquée
Chaque sous-projet majeur a son propre CLAUDE.md. Claude charge automatiquement le plus proche du fichier édité (« closest wins »). N'ajouter ici que ce qui s'applique à TOUS les sous-projets.