HandPersistenceGate (min_frames=3, radius=0.15) suppresses ghost
hands that flash for fewer than min_frames consecutive frames.
Gate is called once per frame in the renderer on persons_hands;
result filters both the video overlay loop and the side panels.
hand_plausible now rejects wrists outside [-0.1, 1.1] (out-of-frame
ghost anchor check). panel_segments gains a conf_min param.
Env knobs read at renderer init (single parse, no per-frame cost):
HAND_CONF_MIN (default 0.45, raised from 0.3)
HAND_PERSIST_FRAMES (default 3)
ARKIT_BONE_MAX (default 0.5, passed to arkit_segments)
New tests: 12 gate tests + 4 wrist-bounds tests.
arkit_segments now accepts max_len and bounds params.
Bones whose endpoints fall outside [-bounds, 1+bounds] or whose
length exceeds max_len are skipped (tracking-loss garbage).
Defaults: max_len=0.5, bounds=0.1. Renderer will pass
max_len=ARKIT_BONE_MAX explicitly (Part 3).
Tests updated to use realistic bone lengths; new tests added
for the out-of-bounds and overlong-bone cases.
NSEvent keyboard monitors were placed at the END of
_start_pose_worker, unreachable: every backend path returns
early (lines 369/384/399/413/440) so monitors were never
installed. ESC, space (concertNext) and keymaps were silently
dead in all real concert paths.
Move the local + global monitor installation into
applicationDidFinishLaunching_, after window focus setup,
non-headless path only. The --multi-hmr headless early-return
(line 95) is unaffected.
_read_sources used .get("pid")/.get("v3d") (dict API) but
multi_hmr_worker emits SMPLXPerson dataclasses (field: vertices_3d).
AttributeError was swallowed by LOG.exception at 30 Hz.
- Dual-read via isinstance check: dataclass path uses p.pid /
p.vertices_3d / p.expression; dict path keeps p.get() unchanged
- 3 new tests: real SMPLXPerson emits OSC, expression field accepted,
legacy dict format still works
persons_arkit_2d_t was written by IphoneUSBSource but never
read. A USB drop left the frozen skeleton on screen indefinitely.
- Add arkit_2d_fresh(ts_by_pid, now, max_age=1.0) pure helper
in hand_display.py (8 TDD tests, all green)
- renderer._update_skeleton: use_arkit now requires at least one
pid younger than 1 s (replaces bool check)
- Per-pid freshness gate in draw loop skips stale pids
Add panel_rect/panel_segments/panel_frame helpers to hand_display.py
(PANEL_SIDE=0.30, PANEL_MARGIN=0.02, PANEL_INNER=0.10). Segments use
uniform pixel-space scale so the hand aspect ratio is preserved
regardless of view aspect. mirror=True flips X within the panel.
In renderer._update_skeleton add push_panel (no mirror flip) and a
panels block before the segs==0 epilogue: routes persons_hands to
left/right panel via chirality list when aligned, screen-cx fallback
otherwise. Draws frame (pid 7) + wireframe (pid 5 left / 6 right).
Runs on every path including use_arkit. Guards: _mp_bones None, empty
hands, misaligned chirality, height==0.
26 tests pass (10 new for panel helpers).
The AV-Live data-only text HUD hid the lower-left corner of the
performance display and would collide with the new left-hand panel.
VIZ_HUD=1 re-enables it; the H key toggle guards the absent view.
_decode_hands now returns (hands, chirality) tuple instead of a bare
list. Landmark .z keeps the raw wire value; .c = clamp(z, 0, 1).
Chirality byte (1=right, else 0=left) is decoded per hand and stored
in State.persons_hands_chirality under the same lock as
persons_hands_iphone.
Under --iphone-usb, skip all three MediaPipe detectors
(pose, face, hand) and rebuild the MP-33 body from
state.persons_arkit_{2d,joints} via new arkit_body_2d /
arkit_body_3d builders. Add head anchor (ARKIT_HEAD_IDX=51,
needs live verify) as MP33 slot 0. Webcam path unchanged.
Add TAG_SKELETON2D=6, SKEL2D_BYTES=819, decode_skeleton2D() to
iphone_usb_bridge.py; handler in IphoneUSBSource writing
persons_arkit_2d / persons_arkit_2d_t to state; new state fields
in State dataclass. TDD: test_arkit_body.py (round-trip + wrong
length → None). Import check verified.
PinchDetector now emits state=1 on engage and state=0 on release
(and on hand-disappear). send_pinch appends the state as the 4th
OSC arg so SC can distinguish hold from lift. Launcher switches
from FINGER_PIANO to PINCH_ENABLE to gate only the pinch detector.
The iPhone USB source gave up permanently if the first connect_device()
failed (app not streaming on :7000 at startup -- phone locked, app not
launched/crashed, USB hiccup): start() returned False and the read thread
(which holds the reconnect loop) never ran. Auto-reconnect only covered a
mid-session drop, never a startup race.
Now start() launches the thread regardless and _run reconnects from the
top of its loop, so the initial connect retries with backoff exactly like
a mid-session drop -- the source connects whenever the iPhone app appears,
no data_only_viz restart needed.
Test: tests/test_iphone_usb_reconnect.py (start() keeps retrying when no
device). The 2 unrelated pre-existing collection errors are unchanged.
Add PoseSoundBridge.send_mouth(pid, mouth_open) mirroring send_kin
pattern; call it in ActionHeadPublisher._tick after send_kin using
the already-in-scope `mouth` scalar; add route to module docstring.
3 unit tests (address, VJ mirror, boundary values) all green.
iPhone Vision hands (on-device, .right upright, y already
top-left, rotation-invariant) are stored in a dedicated state
field and used by the strike detector when fresh (FINGER_SOURCE
auto/iphone/mediapipe). More stable than MediaPipe on the
rotated downscaled video. Adds FINGER_DEBUG strike logging.
Rotate the iPhone/webcam frame before MediaPipe and display
so detection, skeleton overlay and the shown video stay
aligned after the phone is physically rotated. ARKit 3D
world joints are orientation-invariant and untouched.
Pre-existing failure: test_pose_bridge_hands builds the
bridge via __new__, so _vj raised AttributeError on _vdmx.
Defensive getattr makes the full suite green.
Strengthen test_send_finger_emits_expected_osc to verify all seven OSC
arguments by value (z, tipx, tipy) and their types. Ensures OSC tags and
SC ~fpResolve reader get correct int/float types.
Add /pose/skel OSC route emitting nose, shoulders, hips,
knees and ankles (18 floats) from _emit_person. Add
\poseSkel OSCdef in data_feeds.scd storing the 9 joint
positions per pid; cleanup on \poseLeave.
Env-gated VDMX_OSC_HOST (+ VDMX_OSC_PORT, default 1234) forwards the
VJ-useful pose signals (/pose/hands, /pose/kin, /pose/center,
/pose/action, /pose/count) to VDMX OSC input so the body drives any
VDMX parameter. Off by default (no env = no mirror). Verified on macm1:
hands/kin/action received on the VDMX port; None without the env.
When the iPhone USB stream drops (app backgrounds, USB hiccup, decode
end), the reader thread now reconnects with capped backoff instead of
spinning dead on read()=(False,None) forever. The stale frame is cleared
on drop so read() reports no-frame during the gap (no frozen pose).
Verified on macm1: app kill -> "reconnecting" -> relaunch -> detection
resumes (body=1). Unit tests still 10/10.
The body3d / MediaPipe fallback feeding action-head silently failed:
getattr(kp, "x", kp[0]) eagerly evaluated the kp[0] fallback, which
raises TypeError on the non-subscriptable Kp3D dataclass, so
_kp_list_to_array returned None and every body was skipped. This
blocked /pose/kin and /pose/action for any MediaPipe-only mode (webcam
and iphone-usb), so SC body-play never saw a body (hasBody=false ->
no notes). Use attribute access for Kp3D, index fallback only for
indexable rows. Verified: /pose/kin + /pose/action now flow (97 vs 0).
Add --iphone-usb flag so the pose worker reads frames from
IphoneUSBSource instead of the Mac webcam. Skip Mac-camera TCC
when iphone_usb is set; IphoneUSBSource handles its own usbmuxd
connect. MediaPipe still runs on iPhone frames; arkit_fuse refines
body joints as before.
Handle TAG_SKELETON and TAG_HANDS in IphoneUSBSource._run.
Skeleton: decode_skeleton() -> (91,3) float32 array into
persons_arkit_joints[pid] + persons_arkit_last_t[pid].
Hands: _decode_hands() parses count:u8 + per-hand chirality:u8
+ 21x(x,y,z) BE f32 into list[list[PoseKp]] for persons_hands,
compatible with HandFeatureExtractor.step().
cv2-VideoCapture-shaped IphoneUSBSource backed by the iPhone
ARBodyTracker USB stream (usbmuxd). Decodes AVLiveWire HEVC
video to 640x480 BGR frames for MediaPipe. Thread-safe read().
Skeleton and hands (Task 2) dispatched via state but not yet
wired.
The ARBodyTracker iOS app serves its ARKit 91-joint skeleton over the
device TCP :7000 in AVLiveWire framing, not OSC, so data_only_viz never
received it. This bridge tunnels to :7000 through usbmuxd (ListDevices +
Connect, port byte-swapped), demuxes the AVLiveWire stream (magic AVL1,
19-byte BE header), decodes SkeletonPayload (91 interleaved BE xyz floats
+ 91 validity bytes) and republishes each valid joint as OSC /body3d/kp
pid joint_idx x y z to :57128 for the arkit_fuse POSE_FILTER stage.
Verified on macm1: usbmux connect to device :7000, frames forwarded.
DETRPoseWorker._load_py_config imported LazyConfig from
src.misc.lazy_config, which the DETRPose repo does not expose (it lives
in src.core, as tools/inference/torch_inf.py uses it). The import raised
ImportError with no fallback, dropping into a hand-rolled exec() that
cannot resolve the lazy config's relative imports ("from .include..."),
so model loading died with KeyError "'__name__' not in globals". Try
src.core first, then the old path, before the exec fallback. DETRPose
now loads on macm1 (HGNetV2 backbone auto-downloads, no exception).
The whole data_only_viz pytest run was aborted at collection because
test_multihmr_remote imported REQ_PAYLOAD_LEN, a constant that had been
dropped from multihmr_remote while RSP_PAYLOAD_LEN stayed — restore it
as the RAW-format request payload size. The test had also drifted from
the protocol: decode_request now returns a third timing field, so the
two unpack sites take (img, K, _); and the backend defaults to async
double-buffering (MULTIHMR_REMOTE_ASYNC), under which infer() returns
None when the out-queue is empty — the two loopback stub tests now pin
MULTIHMR_REMOTE_ASYNC=0 for a deterministic sync round-trip. Suite goes
from aborted-at-collection to 151 passed, 14 skipped.
L'électron rare