AV-Live

electron-rare/AV-Live

Fork 0

Commit Graph

Author	SHA1	Message	Date
L'électron rare	1828d7ca3e	feat(av-live): scenes react + finger joints Two new capabilities on top of RC0.1 : 1. Pose drives Metal scenes via 12 new SceneUniforms (144 bytes total, 36 floats): mouth_open, eye_open_l/r, head_tilt, head_yaw, finger_pinch_l/r, body_x/y/z, body_height, arm_spread, pose_velocity. BodyView.updateNSView derives them from PoseOSCListener.faces / hands / body3d with EMA smoothing (alpha=0.7). Five scenes wired : storm (velocity+tilt), plasma (mouth+yaw), kaleido (arm_spread segments), starfield (pinch -> star density), bars (height+eye). 2. Multi-HMR mlpackage extended with 6th output : 127 SMPL-X joints (4, 127, 3) incl. 30 finger joints (indices 25-54). TracedMHMR stacks the joints already computed in smpl_layer.py. Output names shifted (var_2420 etc.); constants updated in multihmr_coreml.py and multihmr_server.py. Propagation to client backends + Skeleton3DRenderer finger override remains TODO.	2026-05-14 02:31:10 +02:00
L'électron rare	67302e7ca2	perf(av-live): pyobjc + GPU MediaPipe + queues Three perf optimizations stacked on top of the distributed pipeline: 1. multihmr_server.py ported to pyobjc CoreML.framework direct (drops the ~30 ms coremltools.MLModel.predict overhead). Compiles the mlpackage to .mlmodelc on load, then predicts via MLDictionaryFeatureProvider + MLMultiArray ctypes memcpy. Toggle MULTIHMR_SERVER_BACKEND=coremltools for fallback. setup script installs pyobjc-framework-CoreML on the macm1 venv. 2. MediaPipe Holistic now uses GPU Metal delegate on M5. Required wrapping camera frames as SRGBA (4-channel) instead of SRGB -- the Metal CVPixelBuffer path rejects 3-channel formats. Bench M5 standalone : pose 6.7 -> 2.9 ms, face 4.0 -> 1.0 ms, hand 6.1 -> 3.2 ms. Frees ~10 ms CPU per frame for OSC + rigger. 3. Remote client queues bumped maxsize 1 -> 2/3 (in/out) to absorb jitter without stalling capture.	2026-05-14 02:17:29 +02:00
L'électron rare	9838da3137	feat(av-live): distributed Multi-HMR + fused mesh Multi-HMR inference offloaded to a remote macm1 (M1 Max, 32-core GPU) via a custom TCP protocol on :57140. JPEG q=80 compression (~80 KB vs 1.35 MB raw) + double-buffer async pipeline (3-thread server reader/worker/writer, single-buffer client) overlaps net I/O with predict. Live: 9.8 fps Multi-HMR keyframe (vs 6.8 fps M5 local CoreML), 27 fps perceived via MeshRigger. Distributed display option : AVBODY_HOST env routes the TCP mesh and UDP OSC streams to a remote AVLiveBody. The Swift launcher runs on macm1 in the user GUI session (via osascript do shell script over SSH), receiving the full mesh+skeleton+face+hand stream from M5 capture. Skeleton3DRenderer now bundles body 33 joints + dlib 68 face landmarks + 21x2 hand landmarks into a single procedural LowLevelMesh (line topology) per person : 143 vertices, 288 line indices, 4 draw calls. Anatomical connectors body nose -> face nose bridge and body wrists -> hand wrists. MediaPipe delegate is selectable via MEDIAPIPE_DELEGATE=gpu\|cpu (default cpu after observed Metal GpuBuffer crashes on macOS). Multi-HMR ViT-L checkpoint also tested : 350 ms on macm1 GPU (2.9 fps) -- too slow to ship, ViT-S remains default. Conversion script generalized via MULTIHMR_CKPT_NAME env.	2026-05-14 02:02:33 +02:00

Author

SHA1

Message

Date

L'électron rare

1828d7ca3e

feat(av-live): scenes react + finger joints

Two new capabilities on top of RC0.1 :

1. Pose drives Metal scenes via 12 new SceneUniforms (144 bytes total,
   36 floats): mouth_open, eye_open_l/r, head_tilt, head_yaw,
   finger_pinch_l/r, body_x/y/z, body_height, arm_spread, pose_velocity.
   BodyView.updateNSView derives them from PoseOSCListener.faces /
   hands / body3d with EMA smoothing (alpha=0.7). Five scenes wired :
   storm (velocity+tilt), plasma (mouth+yaw), kaleido (arm_spread
   segments), starfield (pinch -> star density), bars (height+eye).

2. Multi-HMR mlpackage extended with 6th output : 127 SMPL-X joints
   (4, 127, 3) incl. 30 finger joints (indices 25-54). TracedMHMR
   stacks the joints already computed in smpl_layer.py. Output
   names shifted (var_2420 etc.); constants updated in
   multihmr_coreml.py and multihmr_server.py. Propagation to
   client backends + Skeleton3DRenderer finger override remains TODO.

2026-05-14 02:31:10 +02:00

L'électron rare

67302e7ca2

perf(av-live): pyobjc + GPU MediaPipe + queues

Three perf optimizations stacked on top of the distributed pipeline:

1. multihmr_server.py ported to pyobjc CoreML.framework direct
   (drops the ~30 ms coremltools.MLModel.predict overhead). Compiles
   the mlpackage to .mlmodelc on load, then predicts via
   MLDictionaryFeatureProvider + MLMultiArray ctypes memcpy. Toggle
   MULTIHMR_SERVER_BACKEND=coremltools for fallback. setup script
   installs pyobjc-framework-CoreML on the macm1 venv.

2. MediaPipe Holistic now uses GPU Metal delegate on M5. Required
   wrapping camera frames as SRGBA (4-channel) instead of SRGB --
   the Metal CVPixelBuffer path rejects 3-channel formats. Bench
   M5 standalone : pose 6.7 -> 2.9 ms, face 4.0 -> 1.0 ms, hand
   6.1 -> 3.2 ms. Frees ~10 ms CPU per frame for OSC + rigger.

3. Remote client queues bumped maxsize 1 -> 2/3 (in/out) to absorb
   jitter without stalling capture.

2026-05-14 02:17:29 +02:00

L'électron rare

9838da3137

feat(av-live): distributed Multi-HMR + fused mesh

Multi-HMR inference offloaded to a remote macm1 (M1 Max, 32-core
GPU) via a custom TCP protocol on :57140. JPEG q=80 compression
(~80 KB vs 1.35 MB raw) + double-buffer async pipeline (3-thread
server reader/worker/writer, single-buffer client) overlaps net
I/O with predict. Live: 9.8 fps Multi-HMR keyframe (vs 6.8 fps
M5 local CoreML), 27 fps perceived via MeshRigger.

Distributed display option : AVBODY_HOST env routes the TCP mesh
and UDP OSC streams to a remote AVLiveBody. The Swift launcher
runs on macm1 in the user GUI session (via osascript do shell
script over SSH), receiving the full mesh+skeleton+face+hand
stream from M5 capture.

Skeleton3DRenderer now bundles body 33 joints + dlib 68 face
landmarks + 21x2 hand landmarks into a single procedural
LowLevelMesh (line topology) per person : 143 vertices, 288 line
indices, 4 draw calls. Anatomical connectors body nose -> face
nose bridge and body wrists -> hand wrists.

MediaPipe delegate is selectable via MEDIAPIPE_DELEGATE=gpu|cpu
(default cpu after observed Metal GpuBuffer crashes on macOS).

Multi-HMR ViT-L checkpoint also tested : 350 ms on macm1 GPU
(2.9 fps) -- too slow to ship, ViT-S remains default. Conversion
script generalized via MULTIHMR_CKPT_NAME env.

2026-05-14 02:02:33 +02:00

3 Commits