Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.
A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.
Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.
MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.
Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.
Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.
Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
(scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
(PointCloud roundtrip + ICP convergence on translated copy),
gated by `pytest.importorskip("open3d")`
Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.
Restore 166 lines of feed implementations (VELIB Paris, SYTADIN
trafic Ile-de-France, HUB'EAU water, GCN seismic, GDELT events,
SWPC space weather, RTE eco2mix) that existed in the local canon
but were missing from origin/main after the 2026-05-13 cherry-pick
recovery cycle. Verified unique via diff against fresh clone.
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
Apres cherry-pick des 14 commits action-head, le working tree main
divergeait encore de feat/action-head sur 21 fichiers a cause des
resolutions modify/delete qui avaient pris la mauvaise version
(c52271e botched merge avait drop ces fichiers, le cherry-pick
voyait alors "modify in feat / deleted in main" et conservait la
mauvaise version dans certains cas).
Cette commit synchronise le working tree main vers feat/action-head
en checkout-ant tous les fichiers depuis feat. Equivalent fonctionnel
a un fast-forward merge sans toucher l'historique.
Files restored : Skeleton3DRenderer.swift, FaceHandOverlay.swift,
test_multihmr_coreml.py, test_pose_bridge_*.py (4 test files).
Files updated : action_head.py, action_head_pub.py, multi.py,
pose_bridge.py, state.py, extract_j3d_offline.py, training/*.py,
SettingsPanel.swift, data_feeds.scd, et plus.
Result : main == feat/action-head sur le working tree, historique
granulaire preserve via les cherry-picks precedents.
CATransform3D scale x=-1 on the AVCaptureVideoPreviewLayer so the
user sees themselves like in a mirror (left in reality = left on
screen). Overlays (skeleton/face/hand/mesh) keep their raw camera
coordinates for now; align via X-flip if user requests it.
MULTIHMR_AUTOCAST=1 enables MPS mixed precision for the ViT-S
backbone. Tested 2026-05-13: slower than fp32 baseline (400ms vs
270ms) -- overhead cast within forward exceeds matmul savings on
M5. Off by default; FP16 .half() crashes MPS matmul accumulator,
left out entirely.
apple_vision_pose face parser short-circuits to return immediately
since pyobjc 11 cannot dereference VNFaceLandmarkRegion2D
pointsInImageOfSize_ result. Removes ObjCPointerWarning spam at
30 fps (9 regions per face).
Add a parallel-pose worker selector (env AV_LIVE_PARALLEL_POSE)
defaulting to "both": runs Apple Vision body 2D on ANE alongside
MediaPipe Holistic on CPU XNNPACK, while Multi-HMR PyTorch streams
the dense mesh on MPS. Modes "apple_vision" or "mediapipe" to pick
one. Body keypoints land in state.persons_body either way.
Face landmark parser via pyobjc remains blocked: pointAtIndex_
selector arity confuses pyobjc 11 and pointsInImageOfSize_ returns
an opaque PyObjCPointer with no address handle. Keep the
ANE-detected count for logging, fall back to MediaPipe face/hand
fin landmarks until a Swift bridge is in place.
Adresse final review du feature action-head :
- action_head_pub.py + extract_j3d_offline.py : CoreMLArray
wraps numpy mais n'a pas __array__ ; unwrap via .numpy()
avant np.asarray pour eviter object-array silencieux
quand persons_smplx vient du backend CoreML. extract_j3d
ramene depuis main (manquait sur feat suite au merge c52271e).
- train_on_studio.sh : TRAIN_ARGS quote defensivement via
printf %q + reject single quotes pour eviter injection
via le payload single-quoted sur bastion.
CoreML mlpackage reverted to FLOAT32 compute_precision: the FP16
build, even with the roma branchless rotmat patch, visibly
degraded the mesh on extreme poses (atan2 near theta=pi + SMPL-X
decoder drift). FP32 stays 2.3x faster than PyTorch MPS (139 ms
vs 270 ms; 8 fps live with 3 workers).
MeshRigger pid matching now uses Hungarian assignment on bbox IoU
with sticky cache (new_iou >= 0.30 AND old_iou < 0.15 to switch).
Replaces fragile pelvis-distance heuristic. 5 new tests pass.
Multi-HMR PyTorch MPS sort occasionnellement des v3d avec NaN/Inf
ou vertices a magnitudes extremes (>5 m). Ces frames glissaient
sans guard jusqu'a AVLiveBody et apparaissaient comme pics/trous
sur le mesh.
Approche : np.isfinite check sur v3d + transl_np, skip person si
invalide. Plus sanity clamp sur |v3d|.max() > 5.0 m : humains ~2 m,
au-dela = garbage du modele, skip aussi. Le receiver Swift garde
le dernier good mesh pendant les frames skip via retain_window
500 ms (dc7de90).
Tradeoff : on perd la frame entiere du pid au lieu de juste les
verts NaN. Acceptable car corruption est globale au tensor v3d
quand elle arrive (jamais juste 2-3 verts isoles).
Multi-HMR PyTorch MPS sort occasionnellement des v3d avec NaN/Inf
ou vertices a magnitudes extremes (>5 m). Ces frames glissaient
sans guard jusqu'a AVLiveBody et apparaissaient comme pics/trous
sur le mesh.
Approche : np.isfinite check sur v3d + transl_np, skip person si
invalide. Plus sanity clamp sur |v3d|.max() > 5.0 m : humains ~2 m,
au-dela = garbage du modele, skip aussi. Le receiver Swift garde
le dernier good mesh pendant les frames skip via retain_window
500 ms (dc7de90).
Tradeoff : on perd la frame entiere du pid au lieu de juste les
verts NaN. Acceptable car corruption est globale au tensor v3d
quand elle arrive (jamais juste 2-3 verts isoles).
Permet de desactiver le rigger 30 fps (mesh_rigger.py, ajoute en
2c8094c) via env var MESH_RIG=0 pour debugger les deformations
mesh sans changer le code.
Root cause of v3d/transl NaN identified: roma.rotmat_to_rotvec
uses torch.empty + 8 index_put_ on a buffer that CoreML mlprogram
translates as scatter_nd over a garbage-initialised tensor. Cells
that the scatter chain misses keep NaN; the subsequent quat /
norm propagates NaN to every vertex.
Patch: branchless atan2 formulation (stack/clamp/norm/atan2 only),
no torch.empty, no index_put_. Precision drift vs roma original:
2.26e-6 L_inf on random batches. Mlpackage now validates all
outputs finite (1.27e-4 L_inf vs PyTorch eager on v3d).
Bench standalone: 65 ms median FP16 (15.3 fps, target met).
Live with 3 parallel workers: 8 fps Multi-HMR keyframe rate
(2.3x speedup vs PyTorch MPS baseline 3.5 fps); rigging still
ships at 15-20 fps perceived.
Output names shifted post-patch (var_2541 -> var_2412 etc) so
multihmr_coreml.py constants updated.
Add standalone MP4 extractor for action-head v3 training data via
MediaPipe HolisticLandmarker (VIDEO mode). Unlike extract_j3d_offline.py
(SMPL-X path), writes real hands_kp (42, 3) and mouth_open from face
landmarks instead of zeros.
- scripts/extract_mediapipe_offline.py: full pipeline (open video,
iterate frames, map body33/face/hands -> j3d32 + hands_kp42 +
mouth_open, write jsonl rows matching dataset.py schema)
- tests/test_extract_mediapipe_offline.py: 8 pure-numpy unit tests;
no mediapipe runtime required
Enables hand-aware training from recorded footage without SMPL-X
or GPU at extraction time.
CATransform3D scale x=-1 on the AVCaptureVideoPreviewLayer so the
user sees themselves like in a mirror (left in reality = left on
screen). Overlays (skeleton/face/hand/mesh) keep their raw camera
coordinates for now; align via X-flip if user requests it.
MULTIHMR_AUTOCAST=1 enables MPS mixed precision for the ViT-S
backbone. Tested 2026-05-13: slower than fp32 baseline (400ms vs
270ms) -- overhead cast within forward exceeds matmul savings on
M5. Off by default; FP16 .half() crashes MPS matmul accumulator,
left out entirely.
apple_vision_pose face parser short-circuits to return immediately
since pyobjc 11 cannot dereference VNFaceLandmarkRegion2D
pointsInImageOfSize_ result. Removes ObjCPointerWarning spam at
30 fps (9 regions per face).
MeshRigger module : entre deux keyframes Multi-HMR (~3.5 fps mesh
dense PyTorch MPS), on translate rigidement le mesh keyframe via le
delta pelvis 2D Apple Vision (30 fps body ANE) projete a profondeur
constante. SMPLXTCPSender bumpe a 30 fps target et applique le rig
sur chaque tick. Verifie live : 27 fps TCP soutenu, 100% rigged,
keyframe Multi-HMR a 3.2 fps -> ~8x speedup perceptuel dans la
fenetre RealityKit AVLiveBody.
Limitations connues :
- Translation seule (pas de rotation ni de LBS articule)
- Pelvis 2D delta projete a Z constant du keyframe
- Pas de matching d'identite robuste Vision <-> Multi-HMR : on prend
la personne Vision la plus proche du pelvis keyframe projete
Add a parallel-pose worker selector (env AV_LIVE_PARALLEL_POSE)
defaulting to "both": runs Apple Vision body 2D on ANE alongside
MediaPipe Holistic on CPU XNNPACK, while Multi-HMR PyTorch streams
the dense mesh on MPS. Modes "apple_vision" or "mediapipe" to pick
one. Body keypoints land in state.persons_body either way.
Face landmark parser via pyobjc remains blocked: pointAtIndex_
selector arity confuses pyobjc 11 and pointsInImageOfSize_ returns
an opaque PyObjCPointer with no address handle. Keep the
ANE-detected count for logging, fall back to MediaPipe face/hand
fin landmarks until a Swift bridge is in place.
L'électron rare