Two changes that take loop fps from 9.8 to 20+ without touching
the predict cost :
1. MultiHMRWorker default target_fps 10 -> 30. The Python loop was
self-capping at 10 fps via sleep(period - dt) regardless of the
backend. With the remote async pipeline (queue maxsize=2) the
actual fresh-predict rate stays at the macm1 server ceiling
(~9 fps under MLX contention), but the loop now submits frames
at 30 Hz and re-emits MeshRigger-interpolated state at the same
cadence -- the mesh becomes visually fluid at the loop rate even
when fresh predicts are slower. Tunable via MULTIHMR_LOOP_FPS.
2. Short-circuit reused-humans : when infer() returns None (queue
empty), skip dedup/tracker/One-Euro reprocessing and just sleep
to the next slot. Saves ~3-5 ms CPU per idle iteration and
avoids advancing the smoother on identical state.
Heartbeat log extended with (fresh=Y) so we can distinguish loop
rate (visualiser feed) from real predict rate. Bench remote macm1
under MLX contention : hb[coreml]: 25.0 fps (fresh=9.0).
Root cause for the predict gap 53 vs 87 ms : MLX servers
mlx_lm.server :8502/8503 + LoRA on macm1 burn ~30 ms of unified
memory bandwidth per Multi-HMR predict. Not a software fix.
Three perf optimizations stacked on top of the distributed pipeline:
1. multihmr_server.py ported to pyobjc CoreML.framework direct
(drops the ~30 ms coremltools.MLModel.predict overhead). Compiles
the mlpackage to .mlmodelc on load, then predicts via
MLDictionaryFeatureProvider + MLMultiArray ctypes memcpy. Toggle
MULTIHMR_SERVER_BACKEND=coremltools for fallback. setup script
installs pyobjc-framework-CoreML on the macm1 venv.
2. MediaPipe Holistic now uses GPU Metal delegate on M5. Required
wrapping camera frames as SRGBA (4-channel) instead of SRGB --
the Metal CVPixelBuffer path rejects 3-channel formats. Bench
M5 standalone : pose 6.7 -> 2.9 ms, face 4.0 -> 1.0 ms, hand
6.1 -> 3.2 ms. Frees ~10 ms CPU per frame for OSC + rigger.
3. Remote client queues bumped maxsize 1 -> 2/3 (in/out) to absorb
jitter without stalling capture.
Multi-HMR inference offloaded to a remote macm1 (M1 Max, 32-core
GPU) via a custom TCP protocol on :57140. JPEG q=80 compression
(~80 KB vs 1.35 MB raw) + double-buffer async pipeline (3-thread
server reader/worker/writer, single-buffer client) overlaps net
I/O with predict. Live: 9.8 fps Multi-HMR keyframe (vs 6.8 fps
M5 local CoreML), 27 fps perceived via MeshRigger.
Distributed display option : AVBODY_HOST env routes the TCP mesh
and UDP OSC streams to a remote AVLiveBody. The Swift launcher
runs on macm1 in the user GUI session (via osascript do shell
script over SSH), receiving the full mesh+skeleton+face+hand
stream from M5 capture.
Skeleton3DRenderer now bundles body 33 joints + dlib 68 face
landmarks + 21x2 hand landmarks into a single procedural
LowLevelMesh (line topology) per person : 143 vertices, 288 line
indices, 4 draw calls. Anatomical connectors body nose -> face
nose bridge and body wrists -> hand wrists.
MediaPipe delegate is selectable via MEDIAPIPE_DELEGATE=gpu|cpu
(default cpu after observed Metal GpuBuffer crashes on macOS).
Multi-HMR ViT-L checkpoint also tested : 350 ms on macm1 GPU
(2.9 fps) -- too slow to ship, ViT-S remains default. Conversion
script generalized via MULTIHMR_CKPT_NAME env.
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
CoreML mlpackage reverted to FLOAT32 compute_precision: the FP16
build, even with the roma branchless rotmat patch, visibly
degraded the mesh on extreme poses (atan2 near theta=pi + SMPL-X
decoder drift). FP32 stays 2.3x faster than PyTorch MPS (139 ms
vs 270 ms; 8 fps live with 3 workers).
MeshRigger pid matching now uses Hungarian assignment on bbox IoU
with sticky cache (new_iou >= 0.30 AND old_iou < 0.15 to switch).
Replaces fragile pelvis-distance heuristic. 5 new tests pass.
Multi-HMR PyTorch MPS sort occasionnellement des v3d avec NaN/Inf
ou vertices a magnitudes extremes (>5 m). Ces frames glissaient
sans guard jusqu'a AVLiveBody et apparaissaient comme pics/trous
sur le mesh.
Approche : np.isfinite check sur v3d + transl_np, skip person si
invalide. Plus sanity clamp sur |v3d|.max() > 5.0 m : humains ~2 m,
au-dela = garbage du modele, skip aussi. Le receiver Swift garde
le dernier good mesh pendant les frames skip via retain_window
500 ms (dc7de90).
Tradeoff : on perd la frame entiere du pid au lieu de juste les
verts NaN. Acceptable car corruption est globale au tensor v3d
quand elle arrive (jamais juste 2-3 verts isoles).
Permet de desactiver le rigger 30 fps (mesh_rigger.py, ajoute en
2c8094c) via env var MESH_RIG=0 pour debugger les deformations
mesh sans changer le code.
Root cause of v3d/transl NaN identified: roma.rotmat_to_rotvec
uses torch.empty + 8 index_put_ on a buffer that CoreML mlprogram
translates as scatter_nd over a garbage-initialised tensor. Cells
that the scatter chain misses keep NaN; the subsequent quat /
norm propagates NaN to every vertex.
Patch: branchless atan2 formulation (stack/clamp/norm/atan2 only),
no torch.empty, no index_put_. Precision drift vs roma original:
2.26e-6 L_inf on random batches. Mlpackage now validates all
outputs finite (1.27e-4 L_inf vs PyTorch eager on v3d).
Bench standalone: 65 ms median FP16 (15.3 fps, target met).
Live with 3 parallel workers: 8 fps Multi-HMR keyframe rate
(2.3x speedup vs PyTorch MPS baseline 3.5 fps); rigging still
ships at 15-20 fps perceived.
Output names shifted post-patch (var_2541 -> var_2412 etc) so
multihmr_coreml.py constants updated.
Add standalone MP4 extractor for action-head v3 training data via
MediaPipe HolisticLandmarker (VIDEO mode). Unlike extract_j3d_offline.py
(SMPL-X path), writes real hands_kp (42, 3) and mouth_open from face
landmarks instead of zeros.
- scripts/extract_mediapipe_offline.py: full pipeline (open video,
iterate frames, map body33/face/hands -> j3d32 + hands_kp42 +
mouth_open, write jsonl rows matching dataset.py schema)
- tests/test_extract_mediapipe_offline.py: 8 pure-numpy unit tests;
no mediapipe runtime required
Enables hand-aware training from recorded footage without SMPL-X
or GPU at extraction time.
CATransform3D scale x=-1 on the AVCaptureVideoPreviewLayer so the
user sees themselves like in a mirror (left in reality = left on
screen). Overlays (skeleton/face/hand/mesh) keep their raw camera
coordinates for now; align via X-flip if user requests it.
MULTIHMR_AUTOCAST=1 enables MPS mixed precision for the ViT-S
backbone. Tested 2026-05-13: slower than fp32 baseline (400ms vs
270ms) -- overhead cast within forward exceeds matmul savings on
M5. Off by default; FP16 .half() crashes MPS matmul accumulator,
left out entirely.
apple_vision_pose face parser short-circuits to return immediately
since pyobjc 11 cannot dereference VNFaceLandmarkRegion2D
pointsInImageOfSize_ result. Removes ObjCPointerWarning spam at
30 fps (9 regions per face).
MeshRigger module : entre deux keyframes Multi-HMR (~3.5 fps mesh
dense PyTorch MPS), on translate rigidement le mesh keyframe via le
delta pelvis 2D Apple Vision (30 fps body ANE) projete a profondeur
constante. SMPLXTCPSender bumpe a 30 fps target et applique le rig
sur chaque tick. Verifie live : 27 fps TCP soutenu, 100% rigged,
keyframe Multi-HMR a 3.2 fps -> ~8x speedup perceptuel dans la
fenetre RealityKit AVLiveBody.
Limitations connues :
- Translation seule (pas de rotation ni de LBS articule)
- Pelvis 2D delta projete a Z constant du keyframe
- Pas de matching d'identite robuste Vision <-> Multi-HMR : on prend
la personne Vision la plus proche du pelvis keyframe projete
Add a parallel-pose worker selector (env AV_LIVE_PARALLEL_POSE)
defaulting to "both": runs Apple Vision body 2D on ANE alongside
MediaPipe Holistic on CPU XNNPACK, while Multi-HMR PyTorch streams
the dense mesh on MPS. Modes "apple_vision" or "mediapipe" to pick
one. Body keypoints land in state.persons_body either way.
Face landmark parser via pyobjc remains blocked: pointAtIndex_
selector arity confuses pyobjc 11 and pointsInImageOfSize_ returns
an opaque PyObjCPointer with no address handle. Keep the
ANE-detected count for logging, fall back to MediaPipe face/hand
fin landmarks until a Swift bridge is in place.
Adresse final review du feature action-head :
- action_head_pub.py + extract_j3d_offline.py : CoreMLArray
wraps numpy mais n'a pas __array__ ; unwrap via .numpy()
avant np.asarray pour eviter object-array silencieux
quand persons_smplx vient du backend CoreML. extract_j3d
ramene depuis main (manquait sur feat suite au merge c52271e).
- train_on_studio.sh : TRAIN_ARGS quote defensivement via
printf %q + reject single quotes pour eviter injection
via le payload single-quoted sur bastion.
Standalone publisher polls state at 30 Hz, extracts 22-joint
positions from either persons_smplx (vertex anchors) or
persons_body3d (MediaPipe 33→22 map), runs ActionHead.step()
per pid, and emits /pose/action + /pose/kin + lifecycle OSC.
- action_head_pub.py: ActionHeadPublisher thread with dedup
via smplx_last_t / pose_last_t; purges lost pids
- tests/test_action_head_pub.py: 4 unit tests (39 total pass)
- multi.py: import + instantiate + start publisher in __init__
Add comment in TracedMHMR.forward documenting that CoreML conversion
produces all-NaN on v3d/transl while PyTorch eager works. Tested FP32,
K_inv closed-form, simplified subtract+divide projection, nan_to_num
masking. Root cause is an op-level mistranslation in the v3d/transl
path; needs sub-wrapper bisection. Workaround: PyTorch backend.
Implement Task 10 of action-head plan. Records webcam frames +
timestamps for action-head training using OpenCV. Outputs MP4 video
+ timestamp text file to ~/.cache/av-live-action/raw/ with 672x672
square crop, configurable fps/session/camera, interactive q-quit.
Stream MediaPipe Holistic face landmarks (68 dlib subset of 478),
hand landmarks (21 left + 21 right), and pose world landmarks (33
3D xyz meters) over OSC :57126 to AVLiveBody. Launcher renders
face/hand as SwiftUI Canvas overlay and the 3D skeleton as a
RealityKit armature (sphere joints + cylinder bones, color per
chain) toggled via p / mode openpos.
Multi-HMR worker now also starts MediaPipe Multi in parallel so
both the dense SMPL-X mesh (TCP 57130, PyTorch backend) and the
skeleton/face/hand streams (OSC 57126) feed the launcher from
one Python process.
Launcher AppDelegate forces .regular activation so SwiftPM
binaries actually show their WindowGroup without a bundle.
Tests: 9 new pytest cases (4 body3d + 5 face/hand), all green.
CoreML conversion still produces NaN on v3d/transl; PyTorch
backend is the working path for now.
Add interactive console TUI for manual label review of
auto-labeled action datasets. Displays ASCII skeleton,
kinetics, and proposed label with confidence. User can
accept proposed label, choose manual override (1/2/3),
skip, or quit. Reads auto-labeled JSONL and writes
validated rows to reviewed dataset.
rsync excludes .venv/__pycache__/.pytest_cache + uv sync ajoute
extra multihmr (torch). End-to-end valide: smoke 160 windows
3 epochs MPS studio en ~4s, ckpt rsync back OK.
Le precedent printf %q sur-quotait $HOME -> mkdir recevait 0 arg.
On utilise des chemins absolus /Users/clems/av-live-action/* cote
studio et single-quotes pour proteger des bastion expansions.
Add OSCdef handlers for /pose/action, /pose/kin, /pose/enter,
/pose/leave routes emitted by data_only_viz pose_bridge. Store
person state and kinematics in ~poseState and ~poseKin dicts.
Ajoute 4 methodes OSC pour action_head et localisation cinetique:
send_action(pid, label_idx, probs, t_now, force) envoie /pose/action
avec [pid, label_idx, prob_0, prob_1, prob_2] pour les 3 classes.
send_kin(pid, kin, t_now, force) envoie /pose/kin avec [pid, kin[0],
kin[1], kin[2]] pour angles de poignets/coudes.
send_enter/send_leave envoient /pose/enter et /pose/leave pour cycle
vie des personnes.
Throttle reuse _period/_last_t existants; force=True bypass throttle.
Wrapper bash : rsync dataset+code grosmac->studio via bastion
electron-server, exec uv run train_action_head --device mps sur
M3 Ultra, rsync checkpoint back. SSH direct cassee depuis reboot
studio 2026-05-12 ; route via bastion documentee dans CLAUDE.md.
Add evaluation script to compute test accuracy, confusion matrix, and
inference latency on a trained action-head checkpoint. Reuses existing
WindowDataset and model infrastructure from training pipeline.
Falls back to evaluating on full dataset if test split is empty
(edge case with <4 sessions).
Add train_action_head.py with WindowDataset, class-weighted
CrossEntropy, AdamW optimizer, per-epoch train/val loop, and
best-val-acc checkpoint saving. Add smoke tests verifying
2-epoch run and checkpoint loadability via ActionHead.
- WindowDataset computes position/velocity/accel features inline
- _class_weights balances imbalanced label distribution
- train() returns history dict (train/val loss and acc)
- CLI entry point for --device mps/cuda/cpu production runs
Convert Multi-HMR ViT-S 672 to CoreML mlpackage and wire as optional
worker backend via MULTIHMR_BACKEND=coreml env var.
Inference path uses pyobjc + native CoreML framework (Python 3.14 has
no libcoremlpython binding). Conversion done in a separate Py 3.12
venv; einsum cascade patched (camera intrinsics broadcast + smplx
landmarks) via setup_multihmr.sh, idempotent on re-clone.
Bench: 28 ms mock, 100-170 ms live (~13 fps, 4x PyTorch MPS). ANE
compile fails on this model; CPU+GPU is the sweet spot.
Brings 9ab82e4 (MediaPipe Multi default + rich pose OSC) +
96a326d (av-live-body scene Metal) + 214b154 (HUD + menus).
Conflicts on duplicate launcher Swift commits (667f63c vs
96a326d) resolved with -X theirs (prefer main version, since
214b154 builds on it).
Phase 1 fusion : AV-Live-Body absorbe la couche Metal viz et
ecoute les data pose via OSC.
Metal scene (10 viz modes : storm, tunnel, plasma, kaleido,
voronoi, metaballs, starfield, bars, hands3d, openpos) :
- Resources/scene.metal copie depuis data_only_viz, compile au
runtime via MTLLibrary.makeLibrary(source).
- SceneRenderer.swift : MTKViewDelegate qui rebuild
SceneUniforms (20 floats, miroir du struct Metal) et drive
bg_pipeline (full-screen triangle).
- BodyView : nouveau MTKView entre la cam preview et l'ARView,
zPosition intermediaire, alpha pour laisser passer la cam.
- RenderSettings : showScene + vizMode (0..9), picker 10
boutons numerotes dans SettingsPanel + libelle du mode actif
affiche dans la row 'Scene Metal (<name>)'.
Pose OSC :
- PoseOSCListener.swift : UDP listener :57126, parser OSC
minimal (i, f, s args), @MainActor dispatch des Published.
Stocke un PoseFrame par pid (center, head, wrists, sho_span,
yaw, pitch) avec GC 2 s.
- data_only_viz/pose_bridge.py : 2e SimpleUDPClient broadcast
vers 127.0.0.1:57126 (try/except OSError pour silencer si
AVLiveBody pas la). Throttle 30 Hz partage.
Phase 2 (futur) : rendu skeleton entities RealityKit (spheres +
cylindres) consommant PoseOSCListener.persons.
Package.swift : ajout Resources/scene.metal en .copy.
Phase 1 fusion : AV-Live-Body absorbe la couche Metal viz et
ecoute les data pose via OSC.
Metal scene (10 viz modes : storm, tunnel, plasma, kaleido,
voronoi, metaballs, starfield, bars, hands3d, openpos) :
- Resources/scene.metal copie depuis data_only_viz, compile au
runtime via MTLLibrary.makeLibrary(source).
- SceneRenderer.swift : MTKViewDelegate qui rebuild
SceneUniforms (20 floats, miroir du struct Metal) et drive
bg_pipeline (full-screen triangle).
- BodyView : nouveau MTKView entre la cam preview et l'ARView,
zPosition intermediaire, alpha pour laisser passer la cam.
- RenderSettings : showScene + vizMode (0..9), picker 10
boutons numerotes dans SettingsPanel + libelle du mode actif
affiche dans la row 'Scene Metal (<name>)'.
Pose OSC :
- PoseOSCListener.swift : UDP listener :57126, parser OSC
minimal (i, f, s args), @MainActor dispatch des Published.
Stocke un PoseFrame par pid (center, head, wrists, sho_span,
yaw, pitch) avec GC 2 s.
- data_only_viz/pose_bridge.py : 2e SimpleUDPClient broadcast
vers 127.0.0.1:57126 (try/except OSError pour silencer si
AVLiveBody pas la). Throttle 30 Hz partage.
Phase 2 (futur) : rendu skeleton entities RealityKit (spheres +
cylindres) consommant PoseOSCListener.persons.
Package.swift : ajout Resources/scene.metal en .copy.
3 fixes pour reduire le bruit de logs des sources publiques :
wikimedia.py : ajoute User-Agent descriptif. WMF EventStreams
refuse l'UA httpx par defaut avec 403 Forbidden. Le 1.0+URL
descriptif est obligatoire pour le rate-limiting et les abuse
reports.
netzfrequenz.py : backoff exponentiel cap a 5 min sur les ws
disconnect. La source (mainsfrequenz.de) est NXDOMAIN depuis
mai 2026 ; on logguait toutes les 3 s soit ~28 800/jour. Avec
backoff x1.6 et log uniquement attempt==1 ou attempt%10, on
descend a ~30 log/jour.
config.data-only.toml : opensky poll_seconds 20 -> 60. L'API
anonyme est credit-based (10 req/min budget). 60s reste
raisonnable pour le viz tout en restaurant le credit.
Comparatif backbone DINOv2 ViT-S 672 sur deux Apple Silicon :
Hardware | CoreML GPU | CoreML ALL | PyTorch MPS
M5 (16 GB) | 25.1 ms | 157.3 ms | 274.7 ms
M1 Max(32G) | 63.3 ms | 154.1 ms | 110.6 ms
M5 GPU 2.5x plus rapide en CoreML (per-core efficiency vs
32 cores M1 Max). M1 Max 2.5x plus rapide en PyTorch MPS
(M5 souffrait probablement de throttle accumule).
ANE handicape les deux (~155 ms vs 25-63 ms GPU) — DINOv2
ViT-S non ANE-friendly.
macM1 workers MLX pauses via SIGSTOP/SIGCONT pour bench
isole.
L'électron rare