Files
AV-Live/docs/superpowers/plans/2026-05-18-macos-usb-consumer.md
T
Clément SAILLANT 5c61112826 docs: add AGENTS.md skeleton (#1)
* docs(plans): action-head v3 + branch sync notes

Update plan header :
- v2 (Task 18) + v3 (Task 19) extensions chronology
- Studio train validated, ckpt action_head_v3.pt landed
- Mesh NaN-guard debug trail (commit 4e7101c)
- Branch convergence main == feat/action-head
- Pointers to memories project_action_head_v3, etc.

* feat(av-live): openpos 3D + DINO reid + filter

Three improvements wired end-to-end:

1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
   RealityKit AnchorEntity in BodyView, toggled by showSkeleton
   or vizMode==9. PoseOSCListener now parses /pose3d/count and
   /pose3d/kp (plus restored /face/* and /hand/* paths).

2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
   MeshRigger combines Hungarian IoU with cosine similarity over
   a per-pid embedding history (deque maxlen=10), weighted by
   MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
   DINO mlpackage absent or scipy missing. state.last_frame_rgb
   buffer added so the rigger can crop bbox regions for embedding.

3. PoseFilterChain on pose_world_landmarks:
   median (anti-spike) -> Kalman constant-velocity ->
   50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
   via POSE_FILTER env (default median+kalman+lookahead+ik).
   <2 ms per frame for typical 1-2 persons.

Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.

* fix(av-live-body): restore face+hand+3D (f540158)

Three regressions after recent merges, all restored to match the
original f540158 design:

1. FaceHandOverlay was no longer instantiated in ContentView.
   Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
   with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).

2. Skeleton3DRenderer was not attached. BodyView now creates an
   AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
   and ties its visibility to vizMode==9 or showSkeleton toggle.

3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
   skeleton actually reads as 3D instead of looking flat.

MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).

* feat(av-live): wireframe skel + face/hand filter

Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.

FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).

pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.

CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.

* deps(icp): add open3d optional extra + smoke test

Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.

Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.

Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
  (scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
  (PointCloud roundtrip + ICP convergence on translated copy),
  gated by `pytest.importorskip("open3d")`

Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.

* feat(icp): LiDAR TCP frame decoder + tests

* feat(icp): LiDAR TCP socket reader with reconnect

* feat(icp): extrinsic dataclass + JSON persistence

* feat(icp): Kabsch + calibration CLI scaffold

* feat(state): persons_arkit_joints fields

* feat(viz): ARKit 91 -> MP 33 joint map

* feat(viz): iphone OSC listener :57128

* feat(viz): arkit_fuse stage overrides 14 slots

* feat(viz): arkit pelvis z locks cam translation

* feat(viz): iphone OSC listener auto-start

* docs: arkit fusion env vars

* feat(icp): point-to-plane register + reject gate

* feat(icp): partition LiDAR per pid by max-dist

* feat(icp): FusionWorker + State.lidar_points

* feat(icp): wire fusion thread behind ICP_FUSION

Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.

A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.

Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.

MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.

* test(icp): synthetic latency + convergence bench

* docs(icp): env vars + calibration procedure

* docs(plans): icp lidar mesh + arkit joints

Two complementary fusion plans landed in parallel on 2026-05-14:
- iphone-lidar-multihmr-fusion : ARKit 91 joints -> MP33 fuse stage +
  pelvis z override (already implemented in 7 commits)
- icp-lidar-smplx-fusion : LiDAR mesh point-to-plane ICP onto SMPL-X
  10475 verts (12 tasks executed via subagent-driven-development)

Both paths coexist; joints are sparse+fast (60 Hz), mesh is dense+slow
(5-10 Hz). See docs/ICP_FUSION.md for the integration topology.

* feat(icp): predict_once via CoreML backend

* feat(av-live-body): wire ArkitOSCListener :57129

Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.

* feat(ios): iphone ARBodyTracker swiftpm app

iOS 17+ Swift Package app (.swiftpm) streaming ARKit body
joints via OSC UDP to two destinations:

  :57128 -> data_only_viz/iphone_osc_listener.py
  :57129 -> launcher/AV-Live-Body ArkitOSCListener.swift

Features:
- ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported
- 91 joints per body, /body3d/kp pid joint_idx x y z
- 30 fps throttle
- SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second
- Inline OSC encoder (no external dep)

Env mesh (TCP :5500) NOT yet implemented; requires a separate
ARWorldTrackingConfiguration session. ICP fusion path runs on
bench data only until phase 2.

* feat(data-feeds): 10 open-data OSC publisher

* feat(viz): DataFeeds OSC listener + HUD

* chore: gitignore tweaks

* docs: network topology + mDNS hostnames

Add a "Network topology" section to top-level CLAUDE.md doc
the 3-host layout (GrosMac source, Supra-M1 sink via mDNS,
iPhone via Personal Hotspot DHCP).

mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST
accept hostname.local instead of IPs, so the cluster survives
DHCP rotations on iPhone hotspot (172.20.10.x).

* fix(ios): add NSLocalNetworkUsageDescription

iOS 14+ silently blocks UDP to LAN addresses without this key.
The first time the app tries to send to 192.168.0.159, iOS will
prompt the user to allow Local Network Access; the prompt must
be accepted or the OSC stream never reaches the Mac.

Also adds NSBonjourServices declaring _osc._udp so the system
treats the connection as a recognised service.

* docs: network topology + gitignore hygiene

- CLAUDE.md: add mDNS hostname table (grosmac.local, supra-m1.local,
  iPhone hotspot 172.20.10.x). AVBODY_HOST / MULTIHMR_REMOTE_HOST
  accept hostnames — resilient to DHCP rotation.
- .gitignore: exclude .remember/ tool state and iCloud '* 2'
  collision artifacts.

* feat(ios): ARBody skeleton2D + overlay preview

ARBodySession: publish 2D-projected skeleton snapshot for live
overlay rendering on the iPhone screen alongside the camera feed.
ContentView: SkeletonOverlay drawing on top of the AR view, with
mock T-pose for Xcode previews (useMockBackground, useMockSkeleton).

* docs: iPhone USB body-tracking link design

Brainstormed design for replacing the OSC/network iPhone-Mac
link with a wired USB transport via usbmuxd. iPhone streams
ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML
and renders the mesh. Network-free, single native macOS app.

* docs: iPhone USB transport plan (1 of 3)

Bite-sized TDD plan for the network-free USB byte-pipe:
shared AVLiveWire frame format, native usbmux client,
iOS TCP frame server, incremental stream demuxer.

* feat(avlivewire): shared wire package skeleton

* feat(avlivewire): fixed 19-byte frame header codec

Add FrameHeader, a fixed-size binary record so the demuxer can
frame and resync the iPhone USB stream. Layout is big-endian:
4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32.

The magic prefix lets a reader detect and skip corrupt bytes.
Decoding rejects short buffers and bad magic by returning nil.
Big-endian append/parse helpers are added as Data/UInt extensions
to keep the codec self-contained.

* chore: ignore SwiftPM .build artifacts

Both AVLiveWire and AV-Live-Body produce .build/ on swift
test; ignore them so they never get accidentally staged.

* feat(avlivewire): skeleton and video codecs

Add SkeletonPayload (91 ARKit joints + per-joint validity) and
VideoPayload (one HEVC access unit + keyframe flag) with
big-endian encode/decode. Reuses Task 2 Data/UInt32 helpers.

* feat(avlivewire): incremental stream demuxer

Add StreamDemuxer that accepts arbitrary byte chunks from a
non-frame-aligned stream and emits complete (FrameHeader, Data)
frames, resyncing on the magic prefix after corruption.

* fix(avlivewire): cap demuxer payload length

A corrupt header with a huge UInt32 length made feed buffer
forever waiting for bytes that never arrive. Add an 8 MB max
payload cap; a header exceeding it is treated as corrupt, its
magic is skipped, and the demuxer resyncs on the next frame.

* feat(av-live-body): usbmux message codec

Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.

Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.

* feat(av-live-body): usbmux device client

Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.

Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.

* feat(av-live-body): usbmuxd unix socket transport

Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.

* fix(av-live-body): harden unix socket transport

Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
  instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
  idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.

* feat(ios): USB TCP frame server

Add USBServer: an NWListener on a fixed local TCP port that
usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames
and exposes a connection-state callback.

* build: depend on shared AVLiveWire package

Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.

* build(ios): add AVLiveWire package to xcodegen

The xcodegen project did not declare the shared AVLiveWire
package, so USBServer.swift would fail to import it in the
generated Xcode project. Add it as a local package dep.

* test(avlivewire): end-to-end chunked loopback

Feeds 20 framed skeleton payloads through StreamDemuxer in
7-byte chunks (worst-case TCP fragmentation). Fixed a split
range operator from the plan that did not parse.

* fix(ios): guard USBServer listener and payload

Report .idle (not .listening) when NWListener creation fails,
and drop payloads larger than the demuxer's 8 MB cap so the
receiver never silently discards an oversized frame.

* chore: ignore .swiftpm editor state dirs

swift test / Xcode create hidden .swiftpm dirs inside
packages; ignore them so they never get staged.

* docs: iPhone capture plan (2 of 3)

Plan for HEVC video capture (VideoToolbox) over the USB
transport and removal of the legacy OSC sender. Skeleton
USB path already exists; this adds the video half.

* feat(ios): HEVC video capture, drop OSC

Add VideoEncoder (VideoToolbox HEVC) and stream encoded
frames over USB as .video AVLiveWire frames alongside the
skeleton. Remove the legacy OSC/UDP fanout and its host/port
config UI — the iPhone link is now USB-only.

* docs(ios): refresh stale OSC references

ARBodySession header comment and Info.plist usage strings
still described the removed OSC/UDP path; update them to the
USB transport and drop the dead _osc._udp Bonjour service.

* docs: macOS USB consumer plan (3a of 3)

Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.

* feat(av-live-body): USB skeleton consumer

Background usbmux read loop feeding StreamDemuxer; republishes
.skeleton frames as 91-joint ArkitBodyFrames and forwards
.video payloads. Removed stale iCloud collision duplicate
source files that broke the build.

* fix(data-only): CoreML Multi-HMR usage bugs

The CoreML Multi-HMR model was fine; the "0 detections" bug
was caller-side. Add ImageNet normalization in infer() (the
DINOv2 backbone needs it; raw [0,1] input collapsed all
scores) and update stale hardcoded output var names to match
the re-converted mlpackage. Bump the latency test threshold
to the realistic ~140 ms full-model figure.

* feat(av-live-body): HEVC video decoder

VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.

* feat(av-live-body): render 91-joint USB skeleton

Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.

* feat(av-live-body): wire USB consumer to renderer

ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.

* docs: macOS Multi-HMR mesh plan (3b of 3)

Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML
Swift wrapper, BodyFusion (ARKit depth correction), mesh
pipeline wiring. Completes the spec.

* docs: AVLiveBody macOS rewrite design

Clean-rewrite spec: fresh native macOS Xcode app for the
iPhone-USB body pipeline. Reuses the tested USB components,
single RealityKit scene (video quad + skeleton + mesh),
drops all legacy MediaPipe/viz/data-feed code.

* docs: AVLiveBody macOS rewrite plan

10-task plan: scaffold the xcodegen app, migrate the USB
pipeline, build the RealityKit scene (video quad, skeleton,
mesh), wire it, archive the legacy app.

* feat(avlivebody-mac): scaffold xcode app

Add an empty buildable native macOS app generated via xcodegen,
sibling of iphone-arbody. Depends on the shared AVLiveWire package.
Later tasks add the USB pipeline and RealityKit scene.

* feat(avlivebody-mac): migrate usb transport

Context: the new native AVLiveBody app needs the proven iPhone-Mac
usbmux transport layer. These files are self-contained, depending
only on AVLiveWire plus Apple system frameworks, so they cross the
rewrite boundary unchanged.

Approach: copy the three transport files and their unit tests
byte-for-byte from launcher/AV-Live-Body, then make the test target
buildable.

Changes:
- Add usb/USBMuxProtocol.swift, usb/USBClient.swift and
  usb/VideoDecoder.swift under Sources/AVLiveBody.
- Add USBMuxProtocolTests.swift and USBClientTests.swift under
  Tests/AVLiveBodyTests.
- Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so
  xcodebuild can code sign the now-populated test bundle.

Impact: the usbmux pipeline is available in the rewrite and its
six unit tests run green under xcodebuild test.

* feat(avlivebody-mac): usb skeleton consumer

Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload
keyed by pid and owns video decoding directly, dropping the legacy
ArkitOSCListener conversion layer.

* fix(avlivebody-mac): guard thread store with lock

Move the `thread` property write inside the stateLock-held
region in start(); t.start() stays outside since the thread
cannot run before start() is called. Removes a latent race.

* feat(avlivebody-mac): multi-hmr and body fusion

Context: Task 4 of the macOS rewrite needs the dense-mesh half of
the pipeline alongside the USB skeleton consumer landed in task 3.

Approach: Add a CoreML wrapper that mirrors the validated Python
reference (data_only_viz/multihmr_coreml.py) and a pure-logic
fusion stage that corrects the mesh pelvis depth using the
LiDAR-precise USB skeleton.

Changes:
- MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image
  input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at
  10475 vertices, det threshold 0.3.
- BodyFusion.swift: stateless fuse(persons, skeletons) overrides
  the highest-score mesh translation.z with the skeleton pelvis Z
  when available, passes through otherwise.
- BodyFusionTests.swift: pelvis override and pass-through cases.

Impact: Unlocks the mesh renderer wiring in later tasks and gives
the macOS app metrically-correct depth in front of the camera.

* fix(avlivebody-mac): load mlmodelc, clarify fusion

Xcode compiles .mlpackage resources to .mlmodelc at build time;
look up the compiled artifact directly and drop the redundant
MLModel.compileModel step. Also rewrite BodyFusion docstring to
match actual single-person pelvis-z override behaviour.

* feat(avlivebody-mac): scene controller + view

RealityKit scene plumbing: SceneController owns ARView, orbital
camera, and holders for VideoQuad/SkeletonEntity/MeshEntity.
SceneView is the SwiftUI NSViewRepresentable bridge.

Build intentionally deferred to T8 (holder types land in T6-T8).

* fix(avlivebody): orbit gesture + setUp guard

Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to
dispatch only on .changed, replacing the Task wrapper with
MainActor.assumeIsolated. Guard SceneController.setUp() with a
didSetUp flag so duplicate makeNSView calls do not re-install
gestures or re-add anchors.

* feat(avlivebody-mac): 91-joint skeleton entity

Yellow marker spheres pooled per pid; ARKit (x,y,z) ->
RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for
macOS RealityKit Material.Color. Build deferred to T8.

* feat(avlivebody-mac): video quad

Flat 1.6x0.9m plane at z=-2m, textured per-frame from
CVPixelBuffer via CIImage -> CGImage -> TextureResource.
Per-frame TextureResource creation is the known perf hot
spot, isolated here for later LowLevelTexture upgrade.

* fix(avlivebody-mac): appkit import for orbit

NSPanGestureRecognizer lives in AppKit on macOS; without the
import the AVLiveBody module failed to emit. T5 leftover
surfaced once T6/T7/T8 made the target compilable.

* feat(avlivebody-mac): smpl-x mesh entity

Render SMPL-X dense meshes (10475 verts) from Multi-HMR with
pooled ModelEntity per person. Triangle indices loaded from the
bundled smplx_faces.bin (flat UInt32 triplets, copied from the
legacy launcher target). xcodegen folder-scanning bundles the
.bin under Contents/Resources/ — no project.yml change needed.

* feat(avlivebody-mac): wire scene + status bar

Replace placeholder window with ContentView wiring
USBSkeletonConsumer, SceneController, MultiHMRCoreML and
BodyFusion per the T9 dataflow plan.

* chore: archive legacy AV-Live-Body

Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.

* fix(avlivebody): break onVideoFrame retain cycle

Capture consumer weakly in the onVideoFrame closure so the
USBSkeletonConsumer can be deallocated and its background thread
exits cleanly. Guard the mesh-fusion path when consumer is gone.

* fix(launcher): disable body spawn post-archive

Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).

* docs(avlivebody-mac): contributor setup README

Document prerequisites, mlpackage copy, signing xcconfig, and
xcodegen/xcodebuild commands. Points at design spec and plan.

* refactor(avlivebody): axis helper + cleanups

- Extract arkitToRealityKit helper, dedupe 3 call sites.
- Add onDisappear consumer.stop() to terminate USB read loop.
- Replace @State with let for SceneController (stable class id).
- Add NSLog diagnostics in VideoQuad+MeshEntity silent guards.

* fix(avlivebody): ad-hoc signing for local dev

Apple Development cert + Automatic signing makes Xcode demand a
Mac Development cert that no one has. Switch to manual ad-hoc
(CODE_SIGN_IDENTITY = -) so any contributor can build. Drop
hardened runtime; re-enable for distribution builds.

* feat(arbody): keep iphone awake while streaming

iOS auto-lock tears down the USBServer TCP listener within
seconds, breaking AVLiveBody Mac-side connect. Disable the
idle timer for the lifetime of ContentView, restore on exit.

* docs: add AGENTS.md skeleton
2026-05-21 12:41:33 +02:00

25 KiB

macOS USB Consumer Implementation Plan (Plan 3a of 3)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Make the macOS AVLiveBody app consume the iPhone's USB stream — connect via usbmuxd, demux AVLiveWire frames, render the 91-joint skeleton on screen, and HEVC-decode the video — without the Multi-HMR dense-mesh step (deferred to Plan 3b).

Architecture: A new USBSkeletonConsumer runs the blocking UnixMuxTransport/USBClient read loop on a dedicated background thread, feeds bytes through StreamDemuxer, and republishes .skeleton frames as @Published ARKit-shaped body frames plus a .video callback. Skeleton3DRenderer's long-standing // TODO: render yellow ARKit markers (line 138) is completed so the 91-joint USB skeleton actually draws. A new VideoDecoder turns .video VideoPayloads into CVPixelBuffers via VTDecompressionSession.

Tech Stack: Swift 5 (language mode v5), macOS 15, RealityKit, VideoToolbox, AVLiveWire (already a dependency of AV-Live-Body), XCTest.

Companion spec: docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md Prerequisites: Plan 1 (transport, merged), Plan 2 (iOS capture, merged). Out of scope: MultiHMRCoreML, BodyFusion, dense-mesh rendering — Plan 3b, gated on a confirmed CoreML Multi-HMR .mlpackage.


Verification

AV-Live-Body is a macOS target — it builds on the host:

cd launcher/AV-Live-Body && swift build
cd launcher/AV-Live-Body && swift test

Each task ends with swift build (and swift test where a test was added) succeeding.


File Structure

File Responsibility
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift NEW. Background USB read loop → StreamDemuxer@Published body frames + video callback
launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift NEW. VTDecompressionSession HEVC decode: VideoPayloadCVPixelBuffer
launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift NEW. Unit test for the SkeletonPayloadArkitBodyFrame mapping
launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift MODIFY. Complete the line-138 TODO: draw 91 USB-skeleton joint markers
launcher/AV-Live-Body/Sources/AVLiveBody/ArkitOSCListener.swift REFERENCE only — reuse its nested ArkitBodyFrame type
launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift MODIFY. Own a USBSkeletonConsumer, start it in .onAppear
launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift MODIFY. Thread the consumer into Skeleton3DRenderer.attach

Task 1: USBSkeletonConsumer

USBSkeletonConsumer owns the blocking USB read loop on a background Thread. It reconnects on drop. It republishes .skeleton frames as ArkitOSCListener.ArkitBodyFrame (the existing 91-joint body type, so Skeleton3DRenderer can consume them with no new type) and forwards .video payloads via a callback. It is not @MainActor: the loop runs off-main and hops to main only for @Published writes — the same pattern as ArkitOSCListener.

Files:

  • Create: launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift

  • Test: launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift

  • Step 1: Write the failing test

launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift:

import XCTest
import AVLiveWire
@testable import AVLiveBody

final class USBSkeletonConsumerTests: XCTestCase {
    func testSkeletonPayloadMapsToBodyFrame() {
        var p = SkeletonPayload()
        p.joints[0] = SIMD3(1, 2, 3)
        p.valid[0] = true
        p.joints[90] = SIMD3(-4, 5, -6)
        p.valid[90] = true
        let frame = USBSkeletonConsumer.bodyFrame(pid: 7, from: p)
        XCTAssertEqual(frame.pid, 7)
        XCTAssertEqual(frame.joints.count, 91)
        XCTAssertEqual(frame.hasJoint.count, 91)
        XCTAssertEqual(frame.joints[0], SIMD3(1, 2, 3))
        XCTAssertTrue(frame.hasJoint[0])
        XCTAssertEqual(frame.joints[90], SIMD3(-4, 5, -6))
        XCTAssertFalse(frame.hasJoint[1])
    }
}
  • Step 2: Run the test to verify it fails

Run: cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests Expected: FAIL — USBSkeletonConsumer undefined.

  • Step 3: Write the implementation

launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift:

import AVLiveWire
import Combine
import Foundation

/// Connects to the tethered iPhone over USB (usbmuxd), demuxes the
/// AVLiveWire stream, and republishes skeleton frames (as the existing
/// 91-joint `ArkitOSCListener.ArkitBodyFrame`) plus video payloads.
/// The blocking transport runs on a dedicated background thread; only
/// `@Published` writes hop to the main queue.
final class USBSkeletonConsumer: ObservableObject {
    /// 91-joint body frames keyed by pid — same shape `Skeleton3DRenderer`
    /// already consumes from `ArkitOSCListener`.
    @Published var bodies: [Int: ArkitOSCListener.ArkitBodyFrame] = [:]
    @Published var connected = false

    /// Called (on the main queue) for every decoded `.video` frame.
    var onVideo: ((VideoPayload) -> Void)?

    /// TCP port the iPhone `USBServer` listens on (must match the iOS
    /// app's `USBServer.port`).
    static let devicePort: UInt16 = 7000

    private let stateLock = NSLock()
    private var running = false
    private var thread: Thread?

    private var isRunning: Bool {
        stateLock.lock(); defer { stateLock.unlock() }
        return running
    }

    func start() {
        stateLock.lock()
        if running { stateLock.unlock(); return }
        running = true
        stateLock.unlock()
        let t = Thread { [weak self] in self?.loop() }
        t.name = "cc.avlive.usbconsumer"
        t.start()
        thread = t
    }

    func stop() {
        stateLock.lock(); running = false; stateLock.unlock()
    }

    /// Pure mapping `SkeletonPayload` -> `ArkitBodyFrame`. Static so it
    /// is unit-testable without a transport.
    static func bodyFrame(pid: Int, from p: SkeletonPayload)
        -> ArkitOSCListener.ArkitBodyFrame {
        var f = ArkitOSCListener.ArkitBodyFrame()
        f.pid = pid
        f.joints = p.joints
        f.hasJoint = p.valid
        f.seenAt = CFAbsoluteTimeGetCurrent()
        return f
    }

    // MARK: - Background read loop

    private func loop() {
        while isRunning {
            guard let transport = UnixMuxTransport() else {
                Thread.sleep(forTimeInterval: 1.0); continue
            }
            let client = USBClient(transport: transport)
            guard let dev = client.listDevices().first,
                  client.connect(deviceID: dev,
                                 port: Self.devicePort) else {
                transport.close()
                Thread.sleep(forTimeInterval: 1.0); continue
            }
            publishConnected(true)
            var demux = StreamDemuxer()
            while isRunning {
                guard let chunk = transport.readStream(),
                      !chunk.isEmpty else { break }
                for frame in demux.feed(chunk) { route(frame) }
            }
            transport.close()
            publishConnected(false)
            if isRunning { Thread.sleep(forTimeInterval: 1.0) }
        }
    }

    private func route(_ frame: StreamDemuxer.Frame) {
        switch frame.header.tag {
        case .skeleton:
            guard let payload =
                SkeletonPayload(decoding: frame.payload) else { return }
            let pid = Int(frame.header.pid)
            let body = Self.bodyFrame(pid: pid, from: payload)
            DispatchQueue.main.async { [weak self] in
                self?.bodies[pid] = body
            }
        case .video:
            guard let payload =
                VideoPayload(decoding: frame.payload) else { return }
            DispatchQueue.main.async { [weak self] in
                self?.onVideo?(payload)
            }
        case .meta:
            break
        }
    }

    private func publishConnected(_ value: Bool) {
        DispatchQueue.main.async { [weak self] in
            self?.connected = value
        }
    }
}
  • Step 4: Run the test to verify it passes

Run: cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests Expected: PASS, 1 test.

If ArkitOSCListener.ArkitBodyFrame has no memberwise mutability or a different field set than pid/joints/hasJoint/seenAt, read ArkitOSCListener.swift and adjust bodyFrame to match the actual struct (it is a struct ArkitBodyFrame: Equatable with var pid, var joints: [SIMD3<Float>], var hasJoint: [Bool], var seenAt).

  • Step 5: Run the full suite + commit

Run: cd launcher/AV-Live-Body && swift test Expected: PASS, all tests (7: prior 6 + this 1).

git add launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift
git commit -m "feat(av-live-body): USB skeleton consumer"

(subject ≤50 chars; add a short body — the hook rejects subject-only.)


Task 2: VideoDecoder

VideoDecoder turns .video VideoPayloads into CVPixelBuffers. A keyframe payload carries the HEVC parameter sets prepended (each as a 4-byte big-endian length prefix + NAL bytes — the format Plan 2's iOS VideoEncoder produces); the decoder builds its CMVideoFormatDescription from those, then decodes subsequent access units.

Files:

  • Create: launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift

  • Step 1: Write the implementation

launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift:

import AVLiveWire
import CoreMedia
import CoreVideo
import Foundation
import VideoToolbox

/// HEVC decoder. Feed `VideoPayload`s in; receive `CVPixelBuffer`s via
/// `onFrame`. Keyframe payloads must carry the VPS/SPS/PPS parameter
/// sets prepended as 4-byte-length-prefixed NAL units (the layout the
/// iOS `VideoEncoder` emits); the decoder (re)builds its format
/// description from those.
final class VideoDecoder {
    var onFrame: ((CVPixelBuffer) -> Void)?

    private var session: VTDecompressionSession?
    private var formatDesc: CMVideoFormatDescription?

    /// Decode one access unit.
    func decode(_ payload: VideoPayload) {
        var au = payload.data
        if payload.isKeyframe {
            // Split the prepended parameter sets from the frame data.
            let (params, rest) = Self.splitParameterSets(au)
            if !params.isEmpty {
                rebuildFormat(params)
            }
            au = rest
        }
        guard let fmt = formatDesc, !au.isEmpty else { return }
        if session == nil { makeSession(fmt) }
        guard let session else { return }
        guard let block = Self.blockBuffer(au) else { return }
        var sample: CMSampleBuffer?
        var sampleSize = au.count
        guard CMSampleBufferCreateReady(
            allocator: kCFAllocatorDefault, dataBuffer: block,
            formatDescription: fmt, sampleCount: 1, sampleTimingEntryCount: 0,
            sampleTimingArray: nil, sampleSizeEntryCount: 1,
            sampleSizeArray: &sampleSize,
            sampleBufferOut: &sample) == noErr, let sample else { return }
        VTDecompressionSessionDecodeFrame(
            session, sampleBuffer: sample, flags: [],
            infoFlagsOut: nil) { [weak self] status, _, image, _, _ in
                guard status == noErr, let image else { return }
                self?.onFrame?(image)
            }
    }

    func stop() {
        if let session { VTDecompressionSessionInvalidate(session) }
        session = nil
        formatDesc = nil
    }

    deinit { stop() }

    // MARK: - Helpers

    /// Parameter sets are 4-byte-length-prefixed NAL units at the head
    /// of a keyframe payload. The first NAL whose type is a VCL slice
    /// marks the start of frame data — but to stay simple and robust,
    /// we treat every leading NAL as a parameter set until the running
    /// concatenation can build a valid HEVC format description; the
    /// remainder is the frame. Returns (parameterSetData, frameData).
    private static func splitParameterSets(_ data: Data)
        -> (Data, Data) {
        // Parameter set NALs for HEVC: VPS=32, SPS=33, PPS=34
        // (nal_unit_type = (firstByte >> 1) & 0x3F).
        var offset = 0
        let bytes = [UInt8](data)
        var paramEnd = 0
        while offset + 4 <= bytes.count {
            let len = (Int(bytes[offset]) << 24)
                | (Int(bytes[offset + 1]) << 16)
                | (Int(bytes[offset + 2]) << 8)
                | Int(bytes[offset + 3])
            let nalStart = offset + 4
            guard len > 0, nalStart + len <= bytes.count else { break }
            let nalType = (Int(bytes[nalStart]) >> 1) & 0x3F
            if nalType == 32 || nalType == 33 || nalType == 34 {
                offset = nalStart + len
                paramEnd = offset
            } else {
                break
            }
        }
        return (data.prefix(paramEnd),
                data.suffix(from: data.startIndex
                    .advanced(by: paramEnd)))
    }

    private func rebuildFormat(_ paramData: Data) {
        var sets: [[UInt8]] = []
        let bytes = [UInt8](paramData)
        var offset = 0
        while offset + 4 <= bytes.count {
            let len = (Int(bytes[offset]) << 24)
                | (Int(bytes[offset + 1]) << 16)
                | (Int(bytes[offset + 2]) << 8)
                | Int(bytes[offset + 3])
            let start = offset + 4
            guard len > 0, start + len <= bytes.count else { break }
            sets.append(Array(bytes[start..<start + len]))
            offset = start + len
        }
        guard sets.count >= 3 else { return }
        let pointers = sets.map { UnsafePointer<UInt8>($0) }
        let sizes = sets.map { $0.count }
        var fmt: CMFormatDescription?
        let status = pointers.withUnsafeBufferPointer { pBuf in
            sizes.withUnsafeBufferPointer { sBuf in
                CMVideoFormatDescriptionCreateFromHEVCParameterSets(
                    allocator: kCFAllocatorDefault,
                    parameterSetCount: sets.count,
                    parameterSetPointers: pBuf.baseAddress!,
                    parameterSetSizes: sBuf.baseAddress!,
                    nalUnitHeaderLength: 4, extensions: nil,
                    formatDescriptionOut: &fmt)
            }
        }
        if status == noErr, let fmt {
            formatDesc = fmt
            if let session { VTDecompressionSessionInvalidate(session) }
            session = nil
        }
    }

    private func makeSession(_ fmt: CMVideoFormatDescription) {
        let attrs: [CFString: Any] = [
            kCVPixelBufferPixelFormatTypeKey:
                kCVPixelFormatType_32BGRA,
        ]
        VTDecompressionSessionCreate(
            allocator: kCFAllocatorDefault, formatDescription: fmt,
            decoderSpecification: nil,
            imageBufferAttributes: attrs as CFDictionary,
            outputCallback: nil, decompressionSessionOut: &session)
    }

    private static func blockBuffer(_ data: Data) -> CMBlockBuffer? {
        var block: CMBlockBuffer?
        guard CMBlockBufferCreateWithMemoryBlock(
            allocator: kCFAllocatorDefault, memoryBlock: nil,
            blockLength: data.count, blockAllocator: kCFAllocatorDefault,
            customBlockSource: nil, offsetToData: 0,
            dataLength: data.count, flags: 0,
            blockBufferOut: &block) == noErr, let block else {
            return nil
        }
        var ok = false
        data.withUnsafeBytes { raw in
            if CMBlockBufferReplaceDataBytes(
                with: raw.baseAddress!, blockBuffer: block,
                offsetIntoDestination: 0,
                dataLength: data.count) == noErr { ok = true }
        }
        return ok ? block : nil
    }
}
  • Step 2: Verify it compiles

Run: cd launcher/AV-Live-Body && swift build Expected: build succeeds. If a VideoToolbox/CoreMedia signature differs on this SDK, fix minimally — the behavior (build a format description from the prepended parameter sets, decode the rest) must be preserved.

  • Step 3: Commit
git add launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift
git commit -m "feat(av-live-body): HEVC video decoder"

Task 3: Render the 91-joint USB skeleton

Skeleton3DRenderer already subscribes to a 91-joint ARKit body publisher into lastArkit but never draws it — Skeleton3DRenderer.swift:138 is // TODO: render yellow ARKit markers from lastArkit in update(). Complete it: draw the 91 joints as small yellow spheres.

Files:

  • Modify: launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift

  • Step 1: Read the renderer

Read Skeleton3DRenderer.swift fully. Note: PersonEntities (the per-pid entity struct), lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame], makePerson(pid:parent:), the update(frames:) 30 fps tick, and the RealityKit space conversion used for MediaPipe joints (SIMD3(k.x, -k.y, -k.z)).

  • Step 2: Add 91 ARKit marker entities to PersonEntities

In the PersonEntities struct, add a field:

        var arkitMarkers: [ModelEntity]   // 91 yellow ARKit joint spheres

In makePerson(pid:parent:), after the hand spheres are built, create 91 yellow marker spheres (reuse the jointRadius-sized sphere mesh, a yellow SimpleMaterial), parent them to root, start them disabled, and include arkitMarkers: in the returned PersonEntities(...):

        let arkitMat = SimpleMaterial(
            color: .systemYellow, roughness: 0.6, isMetallic: false)
        var arkitMarkers: [ModelEntity] = []
        arkitMarkers.reserveCapacity(91)
        for _ in 0..<91 {
            let e = ModelEntity(mesh: sphereMesh, materials: [arkitMat])
            e.isEnabled = false
            root.addChild(e)
            arkitMarkers.append(e)
        }
  • Step 3: Draw the ARKit markers each tick

Replace the line // TODO: render yellow ARKit markers from lastArkit in update() (Skeleton3DRenderer.swift:138) — leave the comment removed — and add, at the end of update(frames:) (after the existing per-pid loop), a call to a new private method applyArkit(). Then add the method:

    /// Draw the 91-joint ARKit/USB skeletons as yellow joint markers.
    /// ARKit joints are world-space metric; convert to RealityKit
    /// space (x, y, z) -> (x, -y, -z) like the MediaPipe path.
    private func applyArkit() {
        for (pid, entities) in persons {
            guard let frame = lastArkit[pid] else {
                for m in entities.arkitMarkers { m.isEnabled = false }
                continue
            }
            let n = min(91, entities.arkitMarkers.count,
                        frame.joints.count)
            for i in 0..<n {
                let marker = entities.arkitMarkers[i]
                if frame.hasJoint[i] {
                    let j = frame.joints[i]
                    marker.transform.translation =
                        SIMD3<Float>(j.x, -j.y, -j.z)
                    marker.isEnabled = true
                } else {
                    marker.isEnabled = false
                }
            }
            for i in n..<entities.arkitMarkers.count {
                entities.arkitMarkers[i].isEnabled = false
            }
        }
    }

Note: applyArkit() iterates persons, which is only populated for pids seen in the MediaPipe frames. If the USB skeleton must show when there is no MediaPipe pose, also create a PersonEntities for each pid present in lastArkit. To keep Task 3 minimal, in update(frames:) before applyArkit(), ensure entities exist for ARKit-only pids:

        for pid in lastArkit.keys where persons[pid] == nil {
            persons[pid] = makePerson(pid: pid, parent: anchor)
            lastSeenAt[pid] = now
        }
  • Step 4: Verify build + tests

Run: cd launcher/AV-Live-Body && swift build — Expected: succeeds. Run: cd launcher/AV-Live-Body && swift test — Expected: all tests still pass (no regression).

  • Step 5: Commit
git add launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift
git commit -m "feat(av-live-body): render 91-joint USB skeleton"

Task 4: Wire the consumer into the app

Construct USBSkeletonConsumer in the app, start/stop it with the scene, and feed it into Skeleton3DRenderer in place of (or alongside) ArkitOSCListener.

Files:

  • Modify: launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift

  • Modify: launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift

  • Step 1: Read the two files

Read AVLiveBodyApp.swift and BodyView.swift. Identify: where the @StateObject listeners are declared in ContentView, where .onAppear starts them, how ArkitOSCListener is passed into BodyView, and where BodyView.makeNSView calls skel3d.attach(to:listener:arkitListener:).

  • Step 2: Own and start the consumer

In AVLiveBodyApp.swift's ContentView, add a @StateObject:

    @StateObject private var usbConsumer = USBSkeletonConsumer()

In .onAppear, alongside the existing listener .start() calls, add usbConsumer.start(). If there is an .onDisappear, add usbConsumer.stop().

  • Step 3: Thread the consumer to the renderer

Skeleton3DRenderer.attach currently takes arkitListener: ArkitOSCListener?. The simplest correct change: give USBSkeletonConsumer the same role. Add an overload / extra parameter so attach can subscribe to usbConsumer.$bodies exactly as it subscribes to arkitListener.$bodies (both publish [Int: ArkitOSCListener.ArkitBodyFrame]). Concretely, in Skeleton3DRenderer.attach, accept usbConsumer: USBSkeletonConsumer? and, if non-nil, subscribe its $bodies into lastArkit with the same sink already used for arkitListener (the arkitSub Combine subscription). Pass usbConsumer from ContentViewBodyViewmakeNSViewskel3d.attach(...), mirroring how arkitListener is already threaded.

If arkitListener (the OSC one) is now redundant, it may be passed as nil; do not delete ArkitOSCListener in this plan (other code or Plan 3b cleanup may still reference it).

  • Step 4: Verify build

Run: cd launcher/AV-Live-Body && swift build — Expected: succeeds. Run: cd launcher/AV-Live-Body && swift test — Expected: no regression.

  • Step 5: Commit
git add launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift
git commit -m "feat(av-live-body): wire USB consumer to renderer"

Task 5: Final verification

  • Step 1: Clean build + full test suite
cd launcher/AV-Live-Body && swift build && swift test

Expected: build succeeds; all tests pass (7: prior 6 + Task 1's).

  • Step 2: Confirm the integration seam

USBSkeletonConsumer.devicePort (7000) must equal the iOS app's USBServer.port. Verify:

grep -rn "port.*7000\|devicePort" \
  launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift \
  iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/USBServer.swift

Expected: both sides use 7000.

  • Step 3: Commit any fix (only if Step 2 found a mismatch).

Self-Review

  • Spec coverage: This plan implements the spec's USBClient consumption inside AVLiveBody, the VideoDecoder unit, and the skeleton render path. MultiHMRCoreML, BodyFusion, and dense-mesh rendering are explicitly Plan 3b (gated on a confirmed CoreML Multi-HMR .mlpackage).
  • Placeholders: none — new files have complete code; modify tasks cite exact files and the line-138 TODO, and instruct the implementer to read exact context for AVLiveBodyApp.swift/BodyView.swift (whose current line numbers are not reproduced here).
  • Type consistency: USBSkeletonConsumer.bodyFrame returns ArkitOSCListener.ArkitBodyFrame; Skeleton3DRenderer already stores lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame], so the consumer is type-compatible with the existing arkitSub path. VideoDecoder consumes VideoPayload exactly as Plan 2's VideoEncoder produces it (parameter sets prepended, 4-byte big-endian length prefixes).
  • Known risks: (1) BodyView owns Skeleton3DRenderer, so Task 4 threads a new object through ContentViewBodyViewattach — multi-file, follow the existing arkitListener threading exactly. (2) StreamDemuxer.findMagic copies the whole buffer per feed(); for HEVC video this is a perf risk — acceptable for Plan 3a, revisit if frame rate suffers. (3) The HEVC parameter-set split in VideoDecoder assumes the iOS encoder's exact prepend layout — this is the Plan 2 ↔ Plan 3a integration seam; validate on real device data.