AV-Live/CLAUDE.md

AV-Live

Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout. RC0.1+ (tag v0.1.0-rc1) ajoute le pipeline Multi-HMR distribué M5 ↔ macm1 sur LAN gigabit + 10 scènes Metal pose-réactives + unified 3D armature wireframe (body+face+hands).

Communication

Toujours répondre en français à l'utilisateur. Code, commentaires de code, commits et docs en anglais.

Stack par sous-projet

Sous-projet	Stack
sound_algo/	SuperCollider (sclang + scsynth), 1099 SynthDefs, 345 tracks
oscope-of/	openFrameworks C++, libusb (Hantek bulk), GLSL 150 GL 3.2 core
launcher/	SwiftUI menubar app, Swift Package Manager
data_only_viz/	Python 3.11+ via uv, Metal natif (pyobjc), multi-backends pose
web_realart/	Node.js, Express, OSC bridge

Where to Look

Tâche	Emplacement
Ajouter / modifier un SynthDef, track, palette	sound_algo/ (déjà documenté en nested)
Toucher au visualiseur, shaders, FFT, Hantek	oscope-of/
Modifier l'app menubar macOS	launcher/
Détection pose / mesh / body tracking	data_only_viz/
Bridge web / UI de live coding	web_realart/
Plans / specs en cours	docs/superpowers/plans/
Multi-HMR remote server pyobjc	data_only_viz/scripts/multihmr_server.py (tourne sur macm1)
Mesh dense rigger 27 fps perçu	data_only_viz/mesh_rigger.py
3D wireframe armature (body+face+hands)	launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift
Pose → Metal scenes uniforms	launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift + Resources/scene.metal
Filter chain (median + Kalman + lookahead + IK)	data_only_viz/pose_filter.py
DINO re-id pid matching	data_only_viz/dino_reid.py

RC0.1+ environment variables

Env	Default	Effect
MULTIHMR_BACKEND	pytorch	pytorch, coreml, remote
MULTIHMR_REMOTE_HOST	127.0.0.1	macm1 IP for remote inference
MULTIHMR_REMOTE_JPEG	1	JPEG q=80 on the wire
MULTIHMR_REMOTE_ASYNC	1	client double-buffer queue
MULTIHMR_SERVER_BACKEND	pyobjc	server: pyobjc or coremltools
MULTIHMR_LOOP_FPS	30	Python worker loop target_fps
AVBODY_HOST	127.0.0.1	route TCP mesh + OSC to remote AVLiveBody
MEDIAPIPE_DELEGATE	cpu	gpu Metal SRGBA (faster, flake on M5)
POSE_FILTER	median+kalman+lookahead+ik	filter chain stages
MULTIHMR_REID	dino	DINO cosine matching, iou fallback

Network topology

Host	mDNS	IP (DHCP)	Role
GrosMac M5	grosmac.local	LAN	Source + visualisation (AVLiveBody + data_only_viz + data_feeds)
macm1 M1 Max	supra-m1.local	192.168.0.175	Sink (Multi-HMR CoreML + Apple Vision ANE + SMPL-X TCP)
iPhone 16 Pro	(Personal Hotspot)	DHCP	ARKit/LiDAR pub via OSC /body3d/kp

AVBODY_HOST / MULTIHMR_REMOTE_HOST accept mDNS hostnames — résiste aux changements DHCP (notamment iPhone hotspot 172.20.10.x).

Environment variables

Env	Default	Effect
POSE_FILTER	median+kalman+lookahead+ik	filter chain stages — extra: one_euro_joints (joint-space CHI 2012 One Euro, inserted before kalman), one_euro_bones (bone-vector One Euro applied after SMPL-X fusion in multi.py), arkit_fuse (overrides 14 body slots with ARKit ARSkeleton3D from the iOS app, expects /body3d/kp on :57128)
IPHONE_OSC_PORT	57128	UDP port the iPhone ARBodyTracker app pushes /body3d/kp to (always-on listener in data_only_viz)
VIDEO_ROTATE	none	rotate the source video frame before MediaPipe + display: none/ccw/cw/180. Applied at multi.py read; pose detection + skeleton overlay stay aligned. ARKit 3D joints (gravity-aligned world) are unaffected
ICP_FUSION	0	1 to enable LiDAR↔SMPL-X ICP fusion (cf. docs/ICP_FUSION.md)
ICP_LIDAR_HOST	(unset)	iPhone ARBodyTracker IP when ICP_FUSION=1
ICP_LIDAR_PORT	5500	iPhone ARMesh TCP port
ICP_LIDAR_EXTRINSIC	~/.config/av-live/lidar_extrinsic.json	extrinsic JSON path
FINGER_PIANO	0	1 enables air-piano finger strike emission from data_only_viz
FINGER_STRIKE_VEL	0.02	downward relative-velocity threshold for a strike (normalized image units/frame)
FINGER_STRIKE_REFRACTORY_MS	120	min ms between strikes per finger
FINGER_SOURCE	auto	hand source for finger gestures: auto (iPhone Vision hands if fresh, else MediaPipe), iphone, mediapipe
FINGER_DEBUG	0	1 logs each detected strike/pinch (hand, finger) — for threshold tuning
PINCH_RATIO_ON	0.45	thumb-to-finger distance / hand size below which a pinch engages (clip toggle)
PINCH_RATIO_OFF	0.65	distance/size above which the pinch re-arms (hysteresis)
PINCH_REFRACTORY_MS	250	min ms between pinches per finger
PINCH_EXT_RATIO	1.35	tip-to-wrist distance / hand size above which a finger counts as extended (open-hand pinch gate)
PINCH_EXT_MIN	2	min extended non-pinching fingers required for a pinch to engage; 0 disables the open-hand gate
PINCH_DEBOUNCE_FRAMES	3	consecutive qualifying frames before a pinch engage fires (release stays immediate); 1 disables

Conventions globales

• Python : uv systématiquement (jamais pip/poetry/conda directs).
• Pas d'emojis dans code/docs/commits sauf demande explicite.
• Commits : sujet ≤ 50 char, body ≤ 72 char/ligne, pas d'attribution AI, pas de --no-verify, pas d'underscore dans le scope (hooks enforcent).
• *.pt, *.ckpt, *.safetensors, *.mlpackage exclus par .gitignore racine.

Agent Workflow

Explore localise → librarian lit (>500 lignes) → tu raisonnes → general-purpose implémente → validator vérifie. Lance les tâches indépendantes en parallèle.

Guidance imbriquée

Chaque sous-projet majeur a son propre CLAUDE.md. Claude charge automatiquement le plus proche du fichier édité (« closest wins »). N'ajouter ici que ce qui s'applique à TOUS les sous-projets.