VIZ_DEFAULT_MODE=<name> boots the viz on that mode and disables the
auto-openpos switch entirely, so the mode is truly always active (user
request: the t-key voronoi effect permanent). Concert launcher defaults
to voronoi; "auto" keeps the historical behavior.
PinchDetector now emits state=1 on engage and state=0 on release
(and on hand-disappear). send_pinch appends the state as the 4th
OSC arg so SC can distinguish hold from lift. Launcher switches
from FINGER_PIANO to PINCH_ENABLE to gate only the pinch detector.
The Metal GPU MediaPipe delegate (set as default earlier to fix pose
choppiness) crashes under sustained load: fatal CVPixelBuffer alloc
failure (-6662) in gpu_buffer_storage_cv_pixel_buffer.cc /
Image::ConvertToGpu(), on M1 Max too. The choppiness it addressed was CPU
starvation (baby-brain + model servers), now stopped -- so the CPU
delegate has headroom and is both fluid and stable. Default to cpu,
overridable via MEDIAPIPE_DELEGATE=gpu for short sessions.
Wire the hand-authored families into the 84 presets and make the launcher
boot the live section.
Changes:
- patterns_for: strip the _intro/_live/_outro suffix before the
PRESET_FAMILY lookup, so all 84 section presets resolve to their
morceau's family.
- PRESET_FAMILY: retarget mel families to the new banks where they fit -
house (detroit/rave/trance/melodic/electro), psy (hypnotic/psytrance/
minimal), dub (dub_techno/melodic_deep), ambient (ambient_intro); drum
families now style ride/shaker/clap/perc per dfam.
- launch_concert.sh: MATRIX_PRESET default -> techno_drive_live.
Impact: genre presets now use the authored families. test_families PASS,
test_sections PASS, regen deterministic, VALIDATE PASS 84/84.
Repurpose AV-Live Concert.app to launch the matrix arranger:
boot SC with MATRIX_ONLY, keep the pose pipeline (feeds the
/pose/* capture-modulation sources), drop the morceau button,
and load a preset (MATRIX_PRESET, default techno_drive) + start
the playhead instead of switching to the concert scene.
Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).
Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.
ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.
Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.
VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.
Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.
Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.
Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.
Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.
Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.
Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.
Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.
Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.
Multi-HMR inference offloaded to a remote macm1 (M1 Max, 32-core
GPU) via a custom TCP protocol on :57140. JPEG q=80 compression
(~80 KB vs 1.35 MB raw) + double-buffer async pipeline (3-thread
server reader/worker/writer, single-buffer client) overlaps net
I/O with predict. Live: 9.8 fps Multi-HMR keyframe (vs 6.8 fps
M5 local CoreML), 27 fps perceived via MeshRigger.
Distributed display option : AVBODY_HOST env routes the TCP mesh
and UDP OSC streams to a remote AVLiveBody. The Swift launcher
runs on macm1 in the user GUI session (via osascript do shell
script over SSH), receiving the full mesh+skeleton+face+hand
stream from M5 capture.
Skeleton3DRenderer now bundles body 33 joints + dlib 68 face
landmarks + 21x2 hand landmarks into a single procedural
LowLevelMesh (line topology) per person : 143 vertices, 288 line
indices, 4 draw calls. Anatomical connectors body nose -> face
nose bridge and body wrists -> hand wrists.
MediaPipe delegate is selectable via MEDIAPIPE_DELEGATE=gpu|cpu
(default cpu after observed Metal GpuBuffer crashes on macOS).
Multi-HMR ViT-L checkpoint also tested : 350 ms on macm1 GPU
(2.9 fps) -- too slow to ship, ViT-S remains default. Conversion
script generalized via MULTIHMR_CKPT_NAME env.
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
CoreML mlpackage reverted to FLOAT32 compute_precision: the FP16
build, even with the roma branchless rotmat patch, visibly
degraded the mesh on extreme poses (atan2 near theta=pi + SMPL-X
decoder drift). FP32 stays 2.3x faster than PyTorch MPS (139 ms
vs 270 ms; 8 fps live with 3 workers).
MeshRigger pid matching now uses Hungarian assignment on bbox IoU
with sticky cache (new_iou >= 0.30 AND old_iou < 0.15 to switch).
Replaces fragile pelvis-distance heuristic. 5 new tests pass.
CATransform3D scale x=-1 on the AVCaptureVideoPreviewLayer so the
user sees themselves like in a mirror (left in reality = left on
screen). Overlays (skeleton/face/hand/mesh) keep their raw camera
coordinates for now; align via X-flip if user requests it.
Stream MediaPipe Holistic face landmarks (68 dlib subset of 478),
hand landmarks (21 left + 21 right), and pose world landmarks (33
3D xyz meters) over OSC :57126 to AVLiveBody. Launcher renders
face/hand as SwiftUI Canvas overlay and the 3D skeleton as a
RealityKit armature (sphere joints + cylinder bones, color per
chain) toggled via p / mode openpos.
Multi-HMR worker now also starts MediaPipe Multi in parallel so
both the dense SMPL-X mesh (TCP 57130, PyTorch backend) and the
skeleton/face/hand streams (OSC 57126) feed the launcher from
one Python process.
Launcher AppDelegate forces .regular activation so SwiftPM
binaries actually show their WindowGroup without a bundle.
Tests: 9 new pytest cases (4 body3d + 5 face/hand), all green.
CoreML conversion still produces NaN on v3d/transl; PyTorch
backend is the working path for now.
Phase 1 fusion : AV-Live-Body absorbe la couche Metal viz et
ecoute les data pose via OSC.
Metal scene (10 viz modes : storm, tunnel, plasma, kaleido,
voronoi, metaballs, starfield, bars, hands3d, openpos) :
- Resources/scene.metal copie depuis data_only_viz, compile au
runtime via MTLLibrary.makeLibrary(source).
- SceneRenderer.swift : MTKViewDelegate qui rebuild
SceneUniforms (20 floats, miroir du struct Metal) et drive
bg_pipeline (full-screen triangle).
- BodyView : nouveau MTKView entre la cam preview et l'ARView,
zPosition intermediaire, alpha pour laisser passer la cam.
- RenderSettings : showScene + vizMode (0..9), picker 10
boutons numerotes dans SettingsPanel + libelle du mode actif
affiche dans la row 'Scene Metal (<name>)'.
Pose OSC :
- PoseOSCListener.swift : UDP listener :57126, parser OSC
minimal (i, f, s args), @MainActor dispatch des Published.
Stocke un PoseFrame par pid (center, head, wrists, sho_span,
yaw, pitch) avec GC 2 s.
- data_only_viz/pose_bridge.py : 2e SimpleUDPClient broadcast
vers 127.0.0.1:57126 (try/except OSError pour silencer si
AVLiveBody pas la). Throttle 30 Hz partage.
Phase 2 (futur) : rendu skeleton entities RealityKit (spheres +
cylindres) consommant PoseOSCListener.persons.
Package.swift : ajout Resources/scene.metal en .copy.
Web data-only fully wired contre les 20 feeds :
dashboard.js : 8 nouveaux handlers (gdelt, wikimedia, tides,
moon, atc, pose, mempool, github, rte_eco2mix). Sparklines +
seuils alert/warn/green coherents.
map.js : marker GDELT (purple) en plus des seismes/foudre/
avions/ISS/volcans, TTL 60s comme les autres ephemeres.
control.html : nouvelle section 'Modes visuels (oF / Metal)'
avec 9 boutons (storm/tunnel/plasma/kaleido/voronoi/metaballs/
stars/bars/hands3d). Le handler scene-btn est splitte par
container (#scenes vs #vizmodes) pour ne pas melanger
l'etat actif.
server.js : whitelist /control/viz* + routing par path. Les
/control/viz* partent vers oF (:57123), tout le reste vers
SC (:57121). Nouveau env OF_PORT_OUT.
Audit fixes :
- LaunchMode displayName Data-only : 'SC + oF + feeds, no web'
-> 'SC + Metal viz + feeds + web' (l'ancien libelle datait
d'avant le toggle Web UI dans dataOnly).
- SettingsPanel : toggle Squelette etiquete '(a venir)' tant
que le rendu n'est pas implemente cote MeshRenderer.
alpha 0.5 -> 0.25 : lerp s'etale sur la frame Python
au lieu de converger en 100 ms puis figer. Persistance
entite 500 ms : une frame Multi-HMR ratee n'efface plus
le mesh. Skip recompute normales pendant interp ticks
(stale ~150 ms invisible vs 24 ms/tick economises).
Lerp SMPL-X vertices between Multi-HMR frames (Python at ~3 fps)
via SceneEvents.Update subscription. Throttle to 30 fps (1 tick
of 2) with alpha=0.5 to keep MainActor cost bounded. Skip normal
recompute during interp ticks — stale ~150 ms is invisible vs
the 24 ms/tick CPU cost it saves.
Pipeline web data-only complet :
Cote feeds, OSC target :57124 ajoute au profil data-only.toml
pour que le bridge.py diffuse vers : SC (57121) + oF (57123)
+ Web (57124).
Cote web (data_only_viz/web/) :
- server.js : Express :3211 + WS broadcast + osc UDP :57124
parse les paths /data/<feed>/<sub> en JSON et diffuse
- public/dashboard.html : grille de cards live avec sparklines
SVG vanilla (USGS, SWPC Kp/wind/Bz/X-ray, Blitz, OpenSky,
Bluesky, meteo, air, ISS, volcans, social_buzz, grid)
- public/map.html : Leaflet dark fullscreen avec markers
ephemeres (60s TTL) pour seismes/foudre/avions/volcans +
marker ISS persistant qui suit la position
Cote launcher :
- ProcessManager : startDataWeb / stopDataWeb + openDataDashboard
/ openDataMap, port :3211 hardcode, script attendu a
metalVizDir/web/server.js
- MenuBarContent : nouvelle row 'Dashboard data-only' en mode
.dataOnly et .bodyMesh + boutons Dashboard/Carte monde quand
le serveur tourne
- stopAll inclut dataWebProc
Le toggle 'Fil de fer' du panel etait un placeholder UI sans
rendu. Implementation maintenant fonctionnelle :
- loadFaces precompute les indices d'aretes : pour chaque
triangle (a,b,c) on emet 3 lignes (a,b)(b,c)(c,a). 62724
aretes total pour 20908 triangles SMPL-X.
- makeEntity cree un ModelEntity sibling avec un LowLevelMesh
en topologie .line, UnlitMaterial blanc, attache comme child.
- updateMeshVertices ecrit les memes positions dans le buffer
du wireframe (seulement si actif).
- applyWireframeSetting toggle entity.isEnabled sur tous les
wireframes vivants.
- BodyView.updateNSView appelle applyWireframeSetting avec
settings.showWireframe a chaque tick.
Ajoute une ProcessRow AV-Live-Body toujours visible dans le
menubar (sauf en mode bodyMesh ou la ligne est deja affichee).
L'utilisateur peut maintenant lancer la fenetre RealityKit a la
demande depuis n'importe quel mode pour acceder au panel de
reglages visuels (touche S).
Le guard 'useMultiHMR' dans startBodyApp est supprime : sans
worker Multi-HMR la fenetre affichera juste la cam et un fond
vide pour le mesh, mais le panel reste utilisable pour tester
les sliders / toggles.
Le projet est dirige en francais (CLAUDE.md root), donc l'UI
utilisateur suit. Traduction des libelles visibles :
- Reglages AV-Live-Body / Couches / Webcam / Maillage / Lumieres / Vue
- Maillage SMPL-X / Fil de fer / Squelette (articulations)
- Opacite / Metallique / Rugosite
- Principale / Remplissage / Contre-jour
- Champ de vision / Luminosite du fond
Les commentaires et identifiants Swift restent en anglais
conformement aux conventions du repo.
Restructures SettingsPanel into 5 sections : Layers / Webcam /
Mesh / Lights / View. Adds explicit toggles for each visual
layer with a pink icon when active : webcam, mesh fill,
wireframe, skeleton joints. The two new layers (wireframe and
skeleton) are wired into RenderSettings but not yet rendered ;
the toggles are the contract for the next renderer pass.
Panel is now scrollable + larger (320x620) to fit the extra
controls cleanly.
L'électron rare