From b6cd8cf4c74d05a68423e70f35799aeab301d9aa Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?L=27=C3=A9lectron=20rare?= <108685187+electron-rare@users.noreply.github.com> Date: Thu, 14 May 2026 02:46:32 +0200 Subject: [PATCH] docs: RC0.1+ architecture + env vars README header pinned to v0.1.0-rc1. New section "RC0.1+ distributed Multi-HMR architecture" with ASCII diagram of the M5 capture host <-> macm1 compute host pipeline (TCP :57140 pyobjc CoreML, JPEG q80, async double-buffer, MeshRigger 27 fps perceived) and the unified 3D armature wireframe on macm1 AVLiveBody. Env var reference table : MULTIHMR_BACKEND, MULTIHMR_REMOTE_*, MULTIHMR_SERVER_BACKEND, MULTIHMR_LOOP_FPS, AVBODY_HOST, MEDIAPIPE_DELEGATE, POSE_FILTER, MULTIHMR_REID*. Root CLAUDE.md ; extended with the same env vars + file locations for mesh_rigger, Skeleton3DRenderer, pose_filter, dino_reid, multihmr_server, scene.metal. Observed ceilings under MLX contention on macm1 : 25 fps loop, 9 fps fresh predict, 27 fps perceived via the rigger. --- CLAUDE.md | 23 ++++++++++++++- README.md | 86 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 107 insertions(+), 2 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 11cdd7c..01c0536 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,6 @@ # AV-Live -Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout. +Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout. **RC0.1+ (tag `v0.1.0-rc1`)** ajoute le pipeline Multi-HMR distribué M5 ↔ macm1 sur LAN gigabit + 10 scènes Metal pose-réactives + unified 3D armature wireframe (body+face+hands). ## Communication @@ -26,6 +26,27 @@ Toujours répondre en français à l'utilisateur. Code, commentaires de code, co | Détection pose / mesh / body tracking | `data_only_viz/` | | Bridge web / UI de live coding | `web_realart/` | | Plans / specs en cours | `docs/superpowers/plans/` | +| Multi-HMR remote server pyobjc | `data_only_viz/scripts/multihmr_server.py` (tourne sur macm1) | +| Mesh dense rigger 27 fps perçu | `data_only_viz/mesh_rigger.py` | +| 3D wireframe armature (body+face+hands) | `launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift` | +| Pose → Metal scenes uniforms | `launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift` + `Resources/scene.metal` | +| Filter chain (median + Kalman + lookahead + IK) | `data_only_viz/pose_filter.py` | +| DINO re-id pid matching | `data_only_viz/dino_reid.py` | + +## RC0.1+ environment variables + +| Env | Default | Effect | +|-----|---------|--------| +| `MULTIHMR_BACKEND` | `pytorch` | `pytorch`, `coreml`, `remote` | +| `MULTIHMR_REMOTE_HOST` | `127.0.0.1` | macm1 IP for remote inference | +| `MULTIHMR_REMOTE_JPEG` | `1` | JPEG q=80 on the wire | +| `MULTIHMR_REMOTE_ASYNC` | `1` | client double-buffer queue | +| `MULTIHMR_SERVER_BACKEND` | `pyobjc` | server: `pyobjc` or `coremltools` | +| `MULTIHMR_LOOP_FPS` | `30` | Python worker loop target_fps | +| `AVBODY_HOST` | `127.0.0.1` | route TCP mesh + OSC to remote AVLiveBody | +| `MEDIAPIPE_DELEGATE` | `cpu` | `gpu` Metal SRGBA (faster, flake on M5) | +| `POSE_FILTER` | `median+kalman+lookahead+ik` | filter chain stages | +| `MULTIHMR_REID` | `dino` | DINO cosine matching, `iou` fallback | ## Conventions globales diff --git a/README.md b/README.md index b0c24c8..fb7bee3 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,7 @@ # AV-Live +**Release : `v0.1.0-rc1` (2026-05-14)** — distributed Multi-HMR (M5 ↔ macm1 LAN), unified 3D armature openpos, hybrid mesh rigger 27 fps perceived, 10 pose-reactive Metal scenes. See [RC0.1+ architecture](#rc01-distributed-multi-hmr-architecture) below. + > **Live coding audio-visual performance system** built around a SuperCollider sound engine, an openFrameworks oscilloscope visualizer driven by a real Hantek 6022BL USB scope, a macOS menubar launcher, and a Metal-native pose / body-mesh visualizer that listens to the same audio bus. > > 15 scripted demoparties · ~47 fullscreen visuals (26 3D parametric meshes + 18 procedural shaders + 3 dedicated C++ scenes) · 5 OS pixel-art shaders · 14 retro OS logos · 33 GLSL shader pairs (~65 files) · 1099 SynthDefs across 368 tracks (23 albums × 16) · **20 real-world data feeds** · 7 pose-estimation backends · **SMPL-X body mesh** (10 475 vertices via Multi-HMR + RealityKit) · **3 launch modes** (Full / Data-only / Body Mesh) — all reactive to the audio physically passing through the scope probes. @@ -101,7 +103,89 @@ Metal-native fullscreen visualizer (pyobjc, MTKView, ~60 fps) driven by webcam p - **Renderer** : Metal pipelines compiled at runtime (`shaders/*.metal`), `bg_pipeline` (full-screen FBM) + `skel_pipeline` (skeleton lines). SMPL face topology shipped as binary (`mesh_topology.py`) for RealityKit-compatible mesh rendering. - **Tracker** : One Euro Filter on keypoints + IoU multi-person association (`scipy.linear_sum_assignment`, ByteTrack-like). - **OSC out → sclang** `:57121` : `/pose/count`, `/pose/center`, `/pose/wrist`, `/pose/head`, `/pose/sho_span`, `/pose/limb_span`. -- **Thread-safe state** : `state.py` exposes `State.lock()` ; dataclasses `PoseKp`, `NLFPerson` (vertices_3d, joints_3d), and a multi-person container. +- **OSC out → AVLiveBody** `:57126` UDP (mode openpos, mode 9 / touche `p`) : `/pose/skel`, `/face/kp` (68 dlib landmarks), `/hand/kp` (21 × 2 hands), `/pose3d/kp` (33 MediaPipe pose_world_landmarks 3D meters). +- **TCP out → AVLiveBody** `:57130` : SMPL-X dense mesh (10475 verts) frame-packed binary, 30 Hz rigged interpolation between Multi-HMR keyframes. +- **Thread-safe state** : `state.py` exposes `State.lock()` ; dataclasses `PoseKp`, `Kp3D`, `SMPLXPerson`, multi-person container. + +## RC0.1+ distributed Multi-HMR architecture + +`feat/action-head` + tag `v0.1.0-rc1` (2026-05-14). The body-mesh pipeline is now **distributed across two Apple-silicon Macs over LAN gigabit** : + +``` + M5 (capture host) macm1 (compute host) + ┌─────────────────────────────────┐ ┌──────────────────────────────────┐ + │ Caméra MacBook Pro │ │ Multi-HMR server :57140 (pyobjc) │ + │ data_only_viz/ │ JPEG q80 │ multihmr_full_672_s.mlpackage │ + │ ├─ multi_hmr_worker ├──TCP────▶│ ├─ Pyobjc direct CoreML.fwk │ + │ │ backend=remote (async) │ │ ├─ ~87 ms predict (M1 Max GPU) │ + │ ├─ MultiHMRRemoteBackend │◀──RSP───┤ └─ 6 outputs (v3d 10475, │ + │ │ queue maxsize 2/3 │ │ transl, scores, betas, │ + │ │ JPEG encode q80 │ │ expression, joints 127) │ + │ │ │ │ │ + │ ├─ MediaPipe Holistic │ │ AVLiveBody display (RealityKit) │ + │ │ Metal GPU delegate │ │ ├─ MeshRenderer (TCP :57130) │ + │ │ pose 33 + face 478 + │ /pose/* │ │ SMPL-X dense, low-level │ + │ │ hand 21×2 ├──UDP────▶│ │ mesh, 30 fps rigged interp │ + │ ├─ pose_bridge (UDP :57126) │ /face/* │ │ │ + │ ├─ smplx_tcp (TCP :57130, 30 Hz) │ /hand/* │ ├─ Skeleton3DRenderer (OSC) │ + │ ├─ MeshRigger (Hungarian + DINO │ /pose3d/│ │ 1 fused LowLevelMesh / │ + │ │ re-id, sticky pid 0.30/0.15) │ │ │ person : body 33 + face │ + │ └─ PoseFilterChain │ │ │ 68 + hand 21×2 = 143 vts │ + │ (median + Kalman CV + │ │ │ 288 line indices │ + │ lookahead + IK clamps) │ │ └─ 10 Metal scenes (storm, │ + │ │ │ tunnel, plasma, kaleido, │ + │ TCP loop fps : 25 │ │ voronoi, metaballs, │ + │ Multi-HMR fresh fps : 9 │ │ starfield, bars, hands3d, │ + │ MeshRigger perceived : 27 │ │ openpos), all consume │ + │ │ │ pose uniforms (mouth, │ + └─────────────────────────────────┘ │ velocity, head_tilt, │ + │ arm_spread, eye_open, │ + │ finger_pinch, body_xyz) │ + └──────────────────────────────────┘ +``` + +### Key technical wins (commits in `feat/action-head`) + +| Commit | Win | +|--------|-----| +| `5800156` | Multi-HMR ViT-S/672 → CoreML mlpackage (FP32, 6 outputs) | +| `52588b9` | roma.rotmat_to_rotvec → branchless atan2 (fixed all-NaN v3d/transl bug) | +| `4e7101c` | NaN/Inf guard on v3d before TCP ship | +| `2c8094c` | MeshRigger : 27 fps perceived via Hungarian pid match + Vision pelvis delta | +| `1f623fe` | AVLiveBody mirror webcam preview (CATransform3D scale -1) | +| `9838da3` | Remote inference protocol (TCP :57140) + multi-buffer async client | +| `67302e7` | Pyobjc server (drops coremltools overhead) + MediaPipe Metal GPU SRGBA | +| `1828d7c` | 12 new SceneUniforms (mouth, eye, head, finger, body) drive 5 scenes | +| `bd46f6e` | Lift Python self-throttle 10 → 30 fps loop, fresh fps metric | +| latest | 10 Metal scenes pose-reactive + SMPL-X 127 joints output (finger props) | + +### Environment toggles + +| Env var | Default | Effect | +|---------|---------|--------| +| `MULTIHMR_BACKEND` | `pytorch` | `pytorch`, `coreml`, `remote` | +| `MULTIHMR_REMOTE_HOST` | `127.0.0.1` | macm1 IP (gigabit LAN) | +| `MULTIHMR_REMOTE_JPEG` | `1` | JPEG q=80 compression on the wire | +| `MULTIHMR_REMOTE_ASYNC` | `1` | client double-buffer queue (maxsize 2/3) | +| `MULTIHMR_SERVER_BACKEND` | `pyobjc` | server : `pyobjc` or `coremltools` | +| `MULTIHMR_LOOP_FPS` | `30` | Python loop target_fps (formerly capped at 10) | +| `AVBODY_HOST` | `127.0.0.1` | route TCP mesh + OSC pose to a remote AVLiveBody | +| `MEDIAPIPE_DELEGATE` | `cpu` | `gpu` Metal SRGBA (faster but IOSurface flake on M5) | +| `POSE_FILTER` | `median+kalman+lookahead+ik` | toggle filter chain stages | +| `MULTIHMR_REID` | `dino` if mlpackage present | `dino` (cosine match) or `iou` | +| `MULTIHMR_REID_ALPHA` | `0.5` | IoU vs cosine weight (0=DINO only, 1=IoU only) | + +### Hardware ceilings observed (M5 + M1 Max 32c GPU, LAN gigabit) + +| Path | Predict | Live loop | Mesh rigged perceived | +|------|---------|-----------|------------------------| +| M5 local (PyTorch MPS) | 270 ms | 3.5 fps | 27 fps | +| M5 local (CoreML FP32) | 139 ms | 6.8 fps | 27 fps | +| Remote macm1 (idle GPU) | 53 ms | ~18 fps | 27 fps | +| Remote macm1 (under MLX contention) | 87 ms | 25 fps loop / 9 fps fresh | 27 fps | +| Studio M3 Ultra (80c GPU, *training only*) | est. 35 ms (~30 fps) | — | — | + +Hard ceiling on macm1 ≈ 18 fps fresh predict (unified memory bandwidth + CoreML sync overhead) ; further gains require moving MLX servers off macm1 or quantising the model. ### 🎬 Demoparties — 15 narrative demos Each demoparty is a multi-act scripted show with unique scroller text, background sequence, scroller style, and SuperCollider album track :