iOS auto-lock tears down the USBServer TCP listener within
seconds, breaking AVLiveBody Mac-side connect. Disable the
idle timer for the lifetime of ContentView, restore on exit.
Apple Development cert + Automatic signing makes Xcode demand a
Mac Development cert that no one has. Switch to manual ad-hoc
(CODE_SIGN_IDENTITY = -) so any contributor can build. Drop
hardened runtime; re-enable for distribution builds.
- Extract arkitToRealityKit helper, dedupe 3 call sites.
- Add onDisappear consumer.stop() to terminate USB read loop.
- Replace @State with let for SceneController (stable class id).
- Add NSLog diagnostics in VideoQuad+MeshEntity silent guards.
Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).
Capture consumer weakly in the onVideoFrame closure so the
USBSkeletonConsumer can be deallocated and its background thread
exits cleanly. Guard the mesh-fusion path when consumer is gone.
Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.
Render SMPL-X dense meshes (10475 verts) from Multi-HMR with
pooled ModelEntity per person. Triangle indices loaded from the
bundled smplx_faces.bin (flat UInt32 triplets, copied from the
legacy launcher target). xcodegen folder-scanning bundles the
.bin under Contents/Resources/ — no project.yml change needed.
NSPanGestureRecognizer lives in AppKit on macOS; without the
import the AVLiveBody module failed to emit. T5 leftover
surfaced once T6/T7/T8 made the target compilable.
Flat 1.6x0.9m plane at z=-2m, textured per-frame from
CVPixelBuffer via CIImage -> CGImage -> TextureResource.
Per-frame TextureResource creation is the known perf hot
spot, isolated here for later LowLevelTexture upgrade.
Yellow marker spheres pooled per pid; ARKit (x,y,z) ->
RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for
macOS RealityKit Material.Color. Build deferred to T8.
Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to
dispatch only on .changed, replacing the Task wrapper with
MainActor.assumeIsolated. Guard SceneController.setUp() with a
didSetUp flag so duplicate makeNSView calls do not re-install
gestures or re-add anchors.
RealityKit scene plumbing: SceneController owns ARView, orbital
camera, and holders for VideoQuad/SkeletonEntity/MeshEntity.
SceneView is the SwiftUI NSViewRepresentable bridge.
Build intentionally deferred to T8 (holder types land in T6-T8).
Xcode compiles .mlpackage resources to .mlmodelc at build time;
look up the compiled artifact directly and drop the redundant
MLModel.compileModel step. Also rewrite BodyFusion docstring to
match actual single-person pelvis-z override behaviour.
Context: Task 4 of the macOS rewrite needs the dense-mesh half of
the pipeline alongside the USB skeleton consumer landed in task 3.
Approach: Add a CoreML wrapper that mirrors the validated Python
reference (data_only_viz/multihmr_coreml.py) and a pure-logic
fusion stage that corrects the mesh pelvis depth using the
LiDAR-precise USB skeleton.
Changes:
- MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image
input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at
10475 vertices, det threshold 0.3.
- BodyFusion.swift: stateless fuse(persons, skeletons) overrides
the highest-score mesh translation.z with the skeleton pelvis Z
when available, passes through otherwise.
- BodyFusionTests.swift: pelvis override and pass-through cases.
Impact: Unlocks the mesh renderer wiring in later tasks and gives
the macOS app metrically-correct depth in front of the camera.
Move the `thread` property write inside the stateLock-held
region in start(); t.start() stays outside since the thread
cannot run before start() is called. Removes a latent race.
Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload
keyed by pid and owns video decoding directly, dropping the legacy
ArkitOSCListener conversion layer.
Context: the new native AVLiveBody app needs the proven iPhone-Mac
usbmux transport layer. These files are self-contained, depending
only on AVLiveWire plus Apple system frameworks, so they cross the
rewrite boundary unchanged.
Approach: copy the three transport files and their unit tests
byte-for-byte from launcher/AV-Live-Body, then make the test target
buildable.
Changes:
- Add usb/USBMuxProtocol.swift, usb/USBClient.swift and
usb/VideoDecoder.swift under Sources/AVLiveBody.
- Add USBMuxProtocolTests.swift and USBClientTests.swift under
Tests/AVLiveBodyTests.
- Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so
xcodebuild can code sign the now-populated test bundle.
Impact: the usbmux pipeline is available in the rewrite and its
six unit tests run green under xcodebuild test.
Add an empty buildable native macOS app generated via xcodegen,
sibling of iphone-arbody. Depends on the shared AVLiveWire package.
Later tasks add the USB pipeline and RealityKit scene.
10-task plan: scaffold the xcodegen app, migrate the USB
pipeline, build the RealityKit scene (video quad, skeleton,
mesh), wire it, archive the legacy app.
Clean-rewrite spec: fresh native macOS Xcode app for the
iPhone-USB body pipeline. Reuses the tested USB components,
single RealityKit scene (video quad + skeleton + mesh),
drops all legacy MediaPipe/viz/data-feed code.
Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML
Swift wrapper, BodyFusion (ARKit depth correction), mesh
pipeline wiring. Completes the spec.
ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.
Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.
VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.
The CoreML Multi-HMR model was fine; the "0 detections" bug
was caller-side. Add ImageNet normalization in infer() (the
DINOv2 backbone needs it; raw [0,1] input collapsed all
scores) and update stale hardcoded output var names to match
the re-converted mlpackage. Bump the latency test threshold
to the realistic ~140 ms full-model figure.
Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.
ARBodySession header comment and Info.plist usage strings
still described the removed OSC/UDP path; update them to the
USB transport and drop the dead _osc._udp Bonjour service.
Add VideoEncoder (VideoToolbox HEVC) and stream encoded
frames over USB as .video AVLiveWire frames alongside the
skeleton. Remove the legacy OSC/UDP fanout and its host/port
config UI — the iPhone link is now USB-only.
Plan for HEVC video capture (VideoToolbox) over the USB
transport and removal of the legacy OSC sender. Skeleton
USB path already exists; this adds the video half.
Report .idle (not .listening) when NWListener creation fails,
and drop payloads larger than the demuxer's 8 MB cap so the
receiver never silently discards an oversized frame.
Feeds 20 framed skeleton payloads through StreamDemuxer in
7-byte chunks (worst-case TCP fragmentation). Fixed a split
range operator from the plan that did not parse.
The xcodegen project did not declare the shared AVLiveWire
package, so USBServer.swift would fail to import it in the
generated Xcode project. Add it as a local package dep.
Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.
Add USBServer: an NWListener on a fixed local TCP port that
usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames
and exposes a connection-state callback.
Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.
Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.
Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.
Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.
Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.
Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.
A corrupt header with a huge UInt32 length made feed buffer
forever waiting for bytes that never arrive. Add an 8 MB max
payload cap; a header exceeding it is treated as corrupt, its
magic is skipped, and the demuxer resyncs on the next frame.
Add StreamDemuxer that accepts arbitrary byte chunks from a
non-frame-aligned stream and emits complete (FrameHeader, Data)
frames, resyncing on the magic prefix after corruption.
Add FrameHeader, a fixed-size binary record so the demuxer can
frame and resync the iPhone USB stream. Layout is big-endian:
4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32.
The magic prefix lets a reader detect and skip corrupt bytes.
Decoding rejects short buffers and bad magic by returning nil.
Big-endian append/parse helpers are added as Data/UInt extensions
to keep the codec self-contained.
Bite-sized TDD plan for the network-free USB byte-pipe:
shared AVLiveWire frame format, native usbmux client,
iOS TCP frame server, incremental stream demuxer.
Brainstormed design for replacing the OSC/network iPhone-Mac
link with a wired USB transport via usbmuxd. iPhone streams
ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML
and renders the mesh. Network-free, single native macOS app.
ARBodySession: publish 2D-projected skeleton snapshot for live
overlay rendering on the iPhone screen alongside the camera feed.
ContentView: SkeletonOverlay drawing on top of the AR view, with
mock T-pose for Xcode previews (useMockBackground, useMockSkeleton).
iOS 14+ silently blocks UDP to LAN addresses without this key.
The first time the app tries to send to 192.168.0.159, iOS will
prompt the user to allow Local Network Access; the prompt must
be accepted or the OSC stream never reaches the Mac.
Also adds NSBonjourServices declaring _osc._udp so the system
treats the connection as a recognised service.
Add a "Network topology" section to top-level CLAUDE.md doc
the 3-host layout (GrosMac source, Supra-M1 sink via mDNS,
iPhone via Personal Hotspot DHCP).
mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST
accept hostname.local instead of IPs, so the cluster survives
DHCP rotations on iPhone hotspot (172.20.10.x).
iOS 17+ Swift Package app (.swiftpm) streaming ARKit body
joints via OSC UDP to two destinations:
:57128 -> data_only_viz/iphone_osc_listener.py
:57129 -> launcher/AV-Live-Body ArkitOSCListener.swift
Features:
- ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported
- 91 joints per body, /body3d/kp pid joint_idx x y z
- 30 fps throttle
- SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second
- Inline OSC encoder (no external dep)
Env mesh (TCP :5500) NOT yet implemented; requires a separate
ARWorldTrackingConfiguration session. ICP fusion path runs on
bench data only until phase 2.
Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.
Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.
A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.
Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.
MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.
Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.
Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.
Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
(scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
(PointCloud roundtrip + ICP convergence on translated copy),
gated by `pytest.importorskip("open3d")`
Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
L'électron rare