Add TAG_SKELETON2D=6, SKEL2D_BYTES=819, decode_skeleton2D() to
iphone_usb_bridge.py; handler in IphoneUSBSource writing
persons_arkit_2d / persons_arkit_2d_t to state; new state fields
in State dataclass. TDD: test_arkit_body.py (round-trip + wrong
length → None). Import check verified.
The ARBodyTracker iOS app serves its ARKit 91-joint skeleton over the
device TCP :7000 in AVLiveWire framing, not OSC, so data_only_viz never
received it. This bridge tunnels to :7000 through usbmuxd (ListDevices +
Connect, port byte-swapped), demuxes the AVLiveWire stream (magic AVL1,
19-byte BE header), decodes SkeletonPayload (91 interleaved BE xyz floats
+ 91 validity bytes) and republishes each valid joint as OSC /body3d/kp
pid joint_idx x y z to :57128 for the arkit_fuse POSE_FILTER stage.
Verified on macm1: usbmux connect to device :7000, frames forwarded.
Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.
A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.
Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.
MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.
Three perf optimizations stacked on top of the distributed pipeline:
1. multihmr_server.py ported to pyobjc CoreML.framework direct
(drops the ~30 ms coremltools.MLModel.predict overhead). Compiles
the mlpackage to .mlmodelc on load, then predicts via
MLDictionaryFeatureProvider + MLMultiArray ctypes memcpy. Toggle
MULTIHMR_SERVER_BACKEND=coremltools for fallback. setup script
installs pyobjc-framework-CoreML on the macm1 venv.
2. MediaPipe Holistic now uses GPU Metal delegate on M5. Required
wrapping camera frames as SRGBA (4-channel) instead of SRGB --
the Metal CVPixelBuffer path rejects 3-channel formats. Bench
M5 standalone : pose 6.7 -> 2.9 ms, face 4.0 -> 1.0 ms, hand
6.1 -> 3.2 ms. Frees ~10 ms CPU per frame for OSC + rigger.
3. Remote client queues bumped maxsize 1 -> 2/3 (in/out) to absorb
jitter without stalling capture.
Multi-HMR inference offloaded to a remote macm1 (M1 Max, 32-core
GPU) via a custom TCP protocol on :57140. JPEG q=80 compression
(~80 KB vs 1.35 MB raw) + double-buffer async pipeline (3-thread
server reader/worker/writer, single-buffer client) overlaps net
I/O with predict. Live: 9.8 fps Multi-HMR keyframe (vs 6.8 fps
M5 local CoreML), 27 fps perceived via MeshRigger.
Distributed display option : AVBODY_HOST env routes the TCP mesh
and UDP OSC streams to a remote AVLiveBody. The Swift launcher
runs on macm1 in the user GUI session (via osascript do shell
script over SSH), receiving the full mesh+skeleton+face+hand
stream from M5 capture.
Skeleton3DRenderer now bundles body 33 joints + dlib 68 face
landmarks + 21x2 hand landmarks into a single procedural
LowLevelMesh (line topology) per person : 143 vertices, 288 line
indices, 4 draw calls. Anatomical connectors body nose -> face
nose bridge and body wrists -> hand wrists.
MediaPipe delegate is selectable via MEDIAPIPE_DELEGATE=gpu|cpu
(default cpu after observed Metal GpuBuffer crashes on macOS).
Multi-HMR ViT-L checkpoint also tested : 350 ms on macm1 GPU
(2.9 fps) -- too slow to ship, ViT-S remains default. Conversion
script generalized via MULTIHMR_CKPT_NAME env.
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
Root cause of v3d/transl NaN identified: roma.rotmat_to_rotvec
uses torch.empty + 8 index_put_ on a buffer that CoreML mlprogram
translates as scatter_nd over a garbage-initialised tensor. Cells
that the scatter chain misses keep NaN; the subsequent quat /
norm propagates NaN to every vertex.
Patch: branchless atan2 formulation (stack/clamp/norm/atan2 only),
no torch.empty, no index_put_. Precision drift vs roma original:
2.26e-6 L_inf on random batches. Mlpackage now validates all
outputs finite (1.27e-4 L_inf vs PyTorch eager on v3d).
Bench standalone: 65 ms median FP16 (15.3 fps, target met).
Live with 3 parallel workers: 8 fps Multi-HMR keyframe rate
(2.3x speedup vs PyTorch MPS baseline 3.5 fps); rigging still
ships at 15-20 fps perceived.
Output names shifted post-patch (var_2541 -> var_2412 etc) so
multihmr_coreml.py constants updated.
Add standalone MP4 extractor for action-head v3 training data via
MediaPipe HolisticLandmarker (VIDEO mode). Unlike extract_j3d_offline.py
(SMPL-X path), writes real hands_kp (42, 3) and mouth_open from face
landmarks instead of zeros.
- scripts/extract_mediapipe_offline.py: full pipeline (open video,
iterate frames, map body33/face/hands -> j3d32 + hands_kp42 +
mouth_open, write jsonl rows matching dataset.py schema)
- tests/test_extract_mediapipe_offline.py: 8 pure-numpy unit tests;
no mediapipe runtime required
Enables hand-aware training from recorded footage without SMPL-X
or GPU at extraction time.
Adresse final review du feature action-head :
- action_head_pub.py + extract_j3d_offline.py : CoreMLArray
wraps numpy mais n'a pas __array__ ; unwrap via .numpy()
avant np.asarray pour eviter object-array silencieux
quand persons_smplx vient du backend CoreML. extract_j3d
ramene depuis main (manquait sur feat suite au merge c52271e).
- train_on_studio.sh : TRAIN_ARGS quote defensivement via
printf %q + reject single quotes pour eviter injection
via le payload single-quoted sur bastion.
Add comment in TracedMHMR.forward documenting that CoreML conversion
produces all-NaN on v3d/transl while PyTorch eager works. Tested FP32,
K_inv closed-form, simplified subtract+divide projection, nan_to_num
masking. Root cause is an op-level mistranslation in the v3d/transl
path; needs sub-wrapper bisection. Workaround: PyTorch backend.
Implement Task 10 of action-head plan. Records webcam frames +
timestamps for action-head training using OpenCV. Outputs MP4 video
+ timestamp text file to ~/.cache/av-live-action/raw/ with 672x672
square crop, configurable fps/session/camera, interactive q-quit.
Stream MediaPipe Holistic face landmarks (68 dlib subset of 478),
hand landmarks (21 left + 21 right), and pose world landmarks (33
3D xyz meters) over OSC :57126 to AVLiveBody. Launcher renders
face/hand as SwiftUI Canvas overlay and the 3D skeleton as a
RealityKit armature (sphere joints + cylinder bones, color per
chain) toggled via p / mode openpos.
Multi-HMR worker now also starts MediaPipe Multi in parallel so
both the dense SMPL-X mesh (TCP 57130, PyTorch backend) and the
skeleton/face/hand streams (OSC 57126) feed the launcher from
one Python process.
Launcher AppDelegate forces .regular activation so SwiftPM
binaries actually show their WindowGroup without a bundle.
Tests: 9 new pytest cases (4 body3d + 5 face/hand), all green.
CoreML conversion still produces NaN on v3d/transl; PyTorch
backend is the working path for now.
rsync excludes .venv/__pycache__/.pytest_cache + uv sync ajoute
extra multihmr (torch). End-to-end valide: smoke 160 windows
3 epochs MPS studio en ~4s, ckpt rsync back OK.
Le precedent printf %q sur-quotait $HOME -> mkdir recevait 0 arg.
On utilise des chemins absolus /Users/clems/av-live-action/* cote
studio et single-quotes pour proteger des bastion expansions.
Wrapper bash : rsync dataset+code grosmac->studio via bastion
electron-server, exec uv run train_action_head --device mps sur
M3 Ultra, rsync checkpoint back. SSH direct cassee depuis reboot
studio 2026-05-12 ; route via bastion documentee dans CLAUDE.md.
Convert Multi-HMR ViT-S 672 to CoreML mlpackage and wire as optional
worker backend via MULTIHMR_BACKEND=coreml env var.
Inference path uses pyobjc + native CoreML framework (Python 3.14 has
no libcoremlpython binding). Conversion done in a separate Py 3.12
venv; einsum cascade patched (camera intrinsics broadcast + smplx
landmarks) via setup_multihmr.sh, idempotent on re-clone.
Bench: 28 ms mock, 100-170 ms live (~13 fps, 4x PyTorch MPS). ANE
compile fails on this model; CPU+GPU is the sweet spot.
Patch 10 cumule : aten::diagonal override pour supporter offset=0
dim1=1 dim2=2 sur tensor rank-3 (B, N, N). Utilise par
roma.rotmat_to_rotvec invoque dans Multi-HMR head.
Implementation : reshape (B, N, N) -> (B, N*N), gather indices
diagonale aplaties [0, N+1, 2N+2, ...] axis=1.
Cascade T3 status : 10 patches actifs, conversion progresse.
Blocker 11 = reshape size mismatch ([1, 51, 3] -> [204, 3, 1])
= incoherence model-side post topk (facteur 4 = K=num_persons).
Plus une op manquante, du model surgery requis.
T3 roadmap mis a jour avec investigation Step B1-B3 pour reprise.
Cascade T3 progression (9 patches cumules) :
6. tile no-op pour reps=[]
7. new_ones converter (aten::new_ones manquait)
8. concat 0d->1d auto-promote
9. clamp_min/max avec dtype promotion
Bloque maintenant a aten::diagonal avec offset/dim non-default.
Cascade encore en cours, on continue.
Task 2 — apply_topk(K=4) validee comme drop-in remplacant
apply_threshold dans la tete Multi-HMR. Cosine sim = 1.000000
sur 4/4 detections image reelle, MAE 2-4 mm.
Script : scripts/probe_head_topk.py.
Task 3 — full Multi-HMR convert tente. Patches appliques :
1. apply_threshold -> apply_topk monkey-patch
2. backbone.encoder.interpolate_pos_encoding -> buffer fige
3. utils.camera.inverse_perspective_projection -> closed-form
jit.trace OK, mais conversion coremltools echoue sur op
suivantes en cascade. Estimation restante : 1-2 j.
Script : scripts/coreml_full_probe.py.
Le breakthrough probe v4 backbone seul 11.8x reste la victoire.
Task 4 worker integration bloquee tant que T3 pas complete.
Fork electron-rare/SMPLer-X (S-Lab License 1.0, non-comm
seulement) ajoute en git submodule a third_party/SMPLer-X.
Setup script utilise le submodule au lieu de cloner upstream.
pyproject : nouvel extra 'smplerx' (torch + mmcv-lite +
ultralytics + smplx + timm). NOTICE.md documente la posture
non-commerciale heritee de Multi-HMR et SMPLer-X.
Le portage inference ARM Mac est l'etape suivante : SMPLer-X
vendorise sa propre mmpose (transformer_utils/mmpose), donc
seul mmcv.Config a besoin d'un substitut (mmcv-lite). Le
detecteur mmdet sera remplace par YOLO Ultralytics.
NOT YET DONE: smpler_x_worker.py (model load + inference,
~1 j) ; bench camera ; integration main.py.
Multi-HMR ViT-S is the only S variant published (672x672 only).
Camera bench on M5 MPS: 3.8 fps median (228 ms/frame), no speedup
vs ViT-L — bottleneck is SMPL-X head, not DINOv2 backbone.
Heartbeat moved before no-detect early-return so FPS metric stays
meaningful with empty camera view.
Path to a working model load on Mac :
- demo.py imports multi_hmr_anny (needs private 'anny' pkg) and
utils.render (needs pyrender + OpenGL offscreen). We skip both
by stubbing the modules in sys.modules and loading Model
directly from model.py.
- utils/__init__.py pulls in roma/trimesh/pillow/tqdm ; added
to the multihmr extra.
- SMPLX_DIR and MEAN_PARAMS are relative paths in
utils/constants.py ; worker chdir's into the repo for the
load, restores cwd in finally. Setup script symlinks
multi-hmr/models -> ../models and downloads
smpl_mean_params.npz from openmmlab-share.
Smoke test : ViT-L checkpoint loads in 4 s, 317 M params,
48 missing / 16 unexpected state-dict keys (normal cross-arch
drift — SMPL_Layer-side, not the backbone).
User provided SMPLX_NEUTRAL.npz locally so we can generate the real
faces file. Adds the extraction script and the produced binary
(20908 triangles x 3 indices x int32 LE = 250896 bytes) consumed
by the upcoming AV-Live-Body RealityKit target.
Adds the `multihmr` extra (smplx, einops, torchgeometry, ...) and
a setup script that clones the Naver Multi-HMR repo and downloads
multiHMR_896_L (ViT-L, 1.28 GB) with a HuggingFace fallback.
NLF was abandoned because its bundled YOLO detector is TorchScript
traced with CUDA hardcoded (aten::empty_strided fails on CPU/MPS).
Multi-HMR is pure PyTorch and stays MPS-friendly.
SMPL-X NEUTRAL.npz still requires manual registration on
smpl-x.is.tue.mpg.de (MPII academic license).
New standalone visualizer: OSC listener, pose bridge,
Apple Vision / CoreML / YOLO pose backends, Euro filter,
fine analysis, mesh topology, holistic renderer.
Metal shader (scene.metal) for GPU-accelerated drawing.
L'électron rare