* docs(plans): action-head v3 + branch sync notes Update plan header : - v2 (Task 18) + v3 (Task 19) extensions chronology - Studio train validated, ckpt action_head_v3.pt landed - Mesh NaN-guard debug trail (commit4e7101c) - Branch convergence main == feat/action-head - Pointers to memories project_action_head_v3, etc. * feat(av-live): openpos 3D + DINO reid + filter Three improvements wired end-to-end: 1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a RealityKit AnchorEntity in BodyView, toggled by showSkeleton or vizMode==9. PoseOSCListener now parses /pose3d/count and /pose3d/kp (plus restored /face/* and /hand/* paths). 2. DINO re-id (dinov2_vits14, ~9 ms ANE forward): MeshRigger combines Hungarian IoU with cosine similarity over a per-pid embedding history (deque maxlen=10), weighted by MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if DINO mlpackage absent or scipy missing. state.last_frame_rgb buffer added so the rigger can crop bbox regions for embedding. 3. PoseFilterChain on pose_world_landmarks: median (anti-spike) -> Kalman constant-velocity -> 50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable via POSE_FILTER env (default median+kalman+lookahead+ik). <2 ms per frame for typical 1-2 persons. Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py, all green. Live validated by user: skeleton spawns, mesh stays stable. * fix(av-live-body): restore face+hand+3D (f540158) Three regressions after recent merges, all restored to match the originalf540158design: 1. FaceHandOverlay was no longer instantiated in ContentView. Added back as a SwiftUI Canvas overlay (68 dlib face landmarks with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta). 2. Skeleton3DRenderer was not attached. BodyView now creates an AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer and ties its visibility to vizMode==9 or showSkeleton toggle. 3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D skeleton actually reads as 3D instead of looking flat. MeshRenderer exposes pelvisWorld map per pid for future interconnect uses (not auto-applied -- design keeps mesh and skeleton each in their own coord space perf540158). * feat(av-live): wireframe skel + face/hand filter Skeleton3DRenderer now renders a wireframe: joint radius 1 mm (quasi-invisible), bone radius 3 mm (line-like). Replaces the chunky bead armature with a clean filaire silhouette covering body 33 joints + face 68 dlib + hands 21x2, all 3D. FaceHandOverlay 2D Canvas removed from ContentView -- face and hand landmarks now live in the same 3D RealityKit armature as the body skeleton (Skeleton3DRenderer.applyFace / applyHands, anchored on nose joint 0 + wrist joints 15/16). pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms lookahead) and HandFilterChain. multi.py wires them after the 2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE), bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all green. CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms), INT8 weight quant = no live gain (GPU compute-bound). 6.4-6.8 fps live is the hardware ceiling on this model. quantize_multihmr_int8.py left in scripts/ for future trials. * deps(icp): add open3d optional extra + smoke test Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a point-cloud library to align iPhone LiDAR scans with Multi-HMR SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz LiDAR cadence. Approach: Add a dedicated `lidar` optional-dep group so the heavy dependency stays opt-in. Pin Python to 3.12 implicitly via the regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312 wheels (cp314 absent). Smoke test guards future regressions. Changes: - pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20` - uv.lock: regenerated with open3d 0.19 + transitive deps (scikit-learn, scipy, dash stack, etc.) - tests/test_open3d_smoke.py: two-test smoke suite (PointCloud roundtrip + ICP convergence on translated copy), gated by `pytest.importorskip("open3d")` Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh alignment, transform publication) without forcing open3d on contributors who only run the base pose pipeline. * feat(icp): LiDAR TCP frame decoder + tests * feat(icp): LiDAR TCP socket reader with reconnect * feat(icp): extrinsic dataclass + JSON persistence * feat(icp): Kabsch + calibration CLI scaffold * feat(state): persons_arkit_joints fields * feat(viz): ARKit 91 -> MP 33 joint map * feat(viz): iphone OSC listener :57128 * feat(viz): arkit_fuse stage overrides 14 slots * feat(viz): arkit pelvis z locks cam translation * feat(viz): iphone OSC listener auto-start * docs: arkit fusion env vars * feat(icp): point-to-plane register + reject gate * feat(icp): partition LiDAR per pid by max-dist * feat(icp): FusionWorker + State.lidar_points * feat(icp): wire fusion thread behind ICP_FUSION Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in earlier tasks into the live data_only_viz pipeline without disturbing the existing ARKit pelvis fuse path or the Multi-HMR worker thread. A new IcpFusionThread pulls LiDAR frames from LidarTCPReader, stages them into State, and applies in-place ICP registration on state.persons_smplx[*].vertices_3d. It runs as a separate daemon thread parallel to MultiHMRWorker rather than inline per frame — the autonomous-worker architecture didn't fit the plan's per-frame call site, so we adapted to a polling thread at 8 Hz. Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the default code path is untouched. Shutdown wired through applicationWillTerminate_. MultiHMRWorker.predict_once is added as a documented stub (NotImplementedError) because the existing PyTorch run loop is too coupled to the camera and MPS lifecycle for a clean single-shot extraction. calibrate_lidar.py keeps its placeholder until a follow-up refactor extracts a pure _infer(rgb) helper. * test(icp): synthetic latency + convergence bench * docs(icp): env vars + calibration procedure * docs(plans): icp lidar mesh + arkit joints Two complementary fusion plans landed in parallel on 2026-05-14: - iphone-lidar-multihmr-fusion : ARKit 91 joints -> MP33 fuse stage + pelvis z override (already implemented in 7 commits) - icp-lidar-smplx-fusion : LiDAR mesh point-to-plane ICP onto SMPL-X 10475 verts (12 tasks executed via subagent-driven-development) Both paths coexist; joints are sparse+fast (60 Hz), mesh is dense+slow (5-10 Hz). See docs/ICP_FUSION.md for the integration topology. * feat(icp): predict_once via CoreML backend * feat(av-live-body): wire ArkitOSCListener :57129 Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic port (57129, distinct from Python's 57128 fuse input). Plumbed through ContentView -> BodyView -> Skeleton3DRenderer so the ARKit joints can be overlayed alongside Multi-HMR mesh. * feat(ios): iphone ARBodyTracker swiftpm app iOS 17+ Swift Package app (.swiftpm) streaming ARKit body joints via OSC UDP to two destinations: :57128 -> data_only_viz/iphone_osc_listener.py :57129 -> launcher/AV-Live-Body ArkitOSCListener.swift Features: - ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported - 91 joints per body, /body3d/kp pid joint_idx x y z - 30 fps throttle - SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second - Inline OSC encoder (no external dep) Env mesh (TCP :5500) NOT yet implemented; requires a separate ARWorldTrackingConfiguration session. ICP fusion path runs on bench data only until phase 2. * feat(data-feeds): 10 open-data OSC publisher * feat(viz): DataFeeds OSC listener + HUD * chore: gitignore tweaks * docs: network topology + mDNS hostnames Add a "Network topology" section to top-level CLAUDE.md doc the 3-host layout (GrosMac source, Supra-M1 sink via mDNS, iPhone via Personal Hotspot DHCP). mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST accept hostname.local instead of IPs, so the cluster survives DHCP rotations on iPhone hotspot (172.20.10.x). * fix(ios): add NSLocalNetworkUsageDescription iOS 14+ silently blocks UDP to LAN addresses without this key. The first time the app tries to send to 192.168.0.159, iOS will prompt the user to allow Local Network Access; the prompt must be accepted or the OSC stream never reaches the Mac. Also adds NSBonjourServices declaring _osc._udp so the system treats the connection as a recognised service. * docs: network topology + gitignore hygiene - CLAUDE.md: add mDNS hostname table (grosmac.local, supra-m1.local, iPhone hotspot 172.20.10.x). AVBODY_HOST / MULTIHMR_REMOTE_HOST accept hostnames — resilient to DHCP rotation. - .gitignore: exclude .remember/ tool state and iCloud '* 2' collision artifacts. * feat(ios): ARBody skeleton2D + overlay preview ARBodySession: publish 2D-projected skeleton snapshot for live overlay rendering on the iPhone screen alongside the camera feed. ContentView: SkeletonOverlay drawing on top of the AR view, with mock T-pose for Xcode previews (useMockBackground, useMockSkeleton). * docs: iPhone USB body-tracking link design Brainstormed design for replacing the OSC/network iPhone-Mac link with a wired USB transport via usbmuxd. iPhone streams ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML and renders the mesh. Network-free, single native macOS app. * docs: iPhone USB transport plan (1 of 3) Bite-sized TDD plan for the network-free USB byte-pipe: shared AVLiveWire frame format, native usbmux client, iOS TCP frame server, incremental stream demuxer. * feat(avlivewire): shared wire package skeleton * feat(avlivewire): fixed 19-byte frame header codec Add FrameHeader, a fixed-size binary record so the demuxer can frame and resync the iPhone USB stream. Layout is big-endian: 4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32. The magic prefix lets a reader detect and skip corrupt bytes. Decoding rejects short buffers and bad magic by returning nil. Big-endian append/parse helpers are added as Data/UInt extensions to keep the codec self-contained. * chore: ignore SwiftPM .build artifacts Both AVLiveWire and AV-Live-Body produce .build/ on swift test; ignore them so they never get accidentally staged. * feat(avlivewire): skeleton and video codecs Add SkeletonPayload (91 ARKit joints + per-joint validity) and VideoPayload (one HEVC access unit + keyframe flag) with big-endian encode/decode. Reuses Task 2 Data/UInt32 helpers. * feat(avlivewire): incremental stream demuxer Add StreamDemuxer that accepts arbitrary byte chunks from a non-frame-aligned stream and emits complete (FrameHeader, Data) frames, resyncing on the magic prefix after corruption. * fix(avlivewire): cap demuxer payload length A corrupt header with a huge UInt32 length made feed buffer forever waiting for bytes that never arrive. Add an 8 MB max payload cap; a header exceeding it is treated as corrupt, its magic is skipped, and the demuxer resyncs on the next frame. * feat(av-live-body): usbmux message codec Add USBMuxProtocol, a codec for Apple's usbmuxd request/response protocol: a 16-byte little-endian header (length, version=1, message=8 plist, tag) followed by an XML property list. Wire an AVLiveBodyTests test target into Package.swift (none existed) so swift test runs the round-trip and header coverage. * feat(av-live-body): usbmux device client Add USBClient for usbmux device discovery and connect-to-port, with an injectable MuxTransport so tests need no real device. Harden USBMuxProtocol.readLE32 to return an optional with a bounds check, avoiding an out-of-range crash on truncated data. * feat(av-live-body): usbmuxd unix socket transport Add UnixMuxTransport, the production MuxTransport that opens a blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed packet reads (4-byte LE length prefix) and raw stream reads for the tunneled byte stream after a successful Connect. * fix(av-live-body): harden unix socket transport Apply four code-review fixes to UnixMuxTransport: - send() now loops on partial writes and retries on EINTR instead of discarding write(2)'s return value. - Add deinit and an fd = -1 sentinel so close() is idempotent and the descriptor cannot leak. - precondition guards strcpy against sun_path overflow. - readN() distinguishes EOF from error and retries EINTR. * feat(ios): USB TCP frame server Add USBServer: an NWListener on a fixed local TCP port that usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames and exposes a connection-state callback. * build: depend on shared AVLiveWire package Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on the local shared/AVLiveWire package so the wire format is defined once. iOS USBServer imports it; macOS use lands in Plan 3. * build(ios): add AVLiveWire package to xcodegen The xcodegen project did not declare the shared AVLiveWire package, so USBServer.swift would fail to import it in the generated Xcode project. Add it as a local package dep. * test(avlivewire): end-to-end chunked loopback Feeds 20 framed skeleton payloads through StreamDemuxer in 7-byte chunks (worst-case TCP fragmentation). Fixed a split range operator from the plan that did not parse. * fix(ios): guard USBServer listener and payload Report .idle (not .listening) when NWListener creation fails, and drop payloads larger than the demuxer's 8 MB cap so the receiver never silently discards an oversized frame. * chore: ignore .swiftpm editor state dirs swift test / Xcode create hidden .swiftpm dirs inside packages; ignore them so they never get staged. * docs: iPhone capture plan (2 of 3) Plan for HEVC video capture (VideoToolbox) over the USB transport and removal of the legacy OSC sender. Skeleton USB path already exists; this adds the video half. * feat(ios): HEVC video capture, drop OSC Add VideoEncoder (VideoToolbox HEVC) and stream encoded frames over USB as .video AVLiveWire frames alongside the skeleton. Remove the legacy OSC/UDP fanout and its host/port config UI — the iPhone link is now USB-only. * docs(ios): refresh stale OSC references ARBodySession header comment and Info.plist usage strings still described the removed OSC/UDP path; update them to the USB transport and drop the dead _osc._udp Bonjour service. * docs: macOS USB consumer plan (3a of 3) Plan for consuming the iPhone USB stream in AVLiveBody: USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render. Multi-HMR dense mesh deferred to Plan 3b. * feat(av-live-body): USB skeleton consumer Background usbmux read loop feeding StreamDemuxer; republishes .skeleton frames as 91-joint ArkitBodyFrames and forwards .video payloads. Removed stale iCloud collision duplicate source files that broke the build. * fix(data-only): CoreML Multi-HMR usage bugs The CoreML Multi-HMR model was fine; the "0 detections" bug was caller-side. Add ImageNet normalization in infer() (the DINOv2 backbone needs it; raw [0,1] input collapsed all scores) and update stale hardcoded output var names to match the re-converted mlpackage. Bump the latency test threshold to the realistic ~140 ms full-model figure. * feat(av-live-body): HEVC video decoder VTDecompressionSession decoder for .video VideoPayloads. Rebuilds the format description from the parameter sets prepended to keyframe payloads by the iOS VideoEncoder. * feat(av-live-body): render 91-joint USB skeleton Complete the long-standing TODO: draw the 91 ARKit/USB skeleton joints as yellow markers, fed from lastArkit. Spawn entity trees for ARKit-only pids so the USB skeleton shows without a MediaPipe pose. * feat(av-live-body): wire USB consumer to renderer ContentView owns and starts a USBSkeletonConsumer, threaded through BodyView into Skeleton3DRenderer.attach. The renderer subscribes its $bodies into lastArkit, so the iPhone's USB skeleton drives the on-screen 91-joint markers. * docs: macOS Multi-HMR mesh plan (3b of 3) Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML Swift wrapper, BodyFusion (ARKit depth correction), mesh pipeline wiring. Completes the spec. * docs: AVLiveBody macOS rewrite design Clean-rewrite spec: fresh native macOS Xcode app for the iPhone-USB body pipeline. Reuses the tested USB components, single RealityKit scene (video quad + skeleton + mesh), drops all legacy MediaPipe/viz/data-feed code. * docs: AVLiveBody macOS rewrite plan 10-task plan: scaffold the xcodegen app, migrate the USB pipeline, build the RealityKit scene (video quad, skeleton, mesh), wire it, archive the legacy app. * feat(avlivebody-mac): scaffold xcode app Add an empty buildable native macOS app generated via xcodegen, sibling of iphone-arbody. Depends on the shared AVLiveWire package. Later tasks add the USB pipeline and RealityKit scene. * feat(avlivebody-mac): migrate usb transport Context: the new native AVLiveBody app needs the proven iPhone-Mac usbmux transport layer. These files are self-contained, depending only on AVLiveWire plus Apple system frameworks, so they cross the rewrite boundary unchanged. Approach: copy the three transport files and their unit tests byte-for-byte from launcher/AV-Live-Body, then make the test target buildable. Changes: - Add usb/USBMuxProtocol.swift, usb/USBClient.swift and usb/VideoDecoder.swift under Sources/AVLiveBody. - Add USBMuxProtocolTests.swift and USBClientTests.swift under Tests/AVLiveBodyTests. - Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so xcodebuild can code sign the now-populated test bundle. Impact: the usbmux pipeline is available in the rewrite and its six unit tests run green under xcodebuild test. * feat(avlivebody-mac): usb skeleton consumer Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload keyed by pid and owns video decoding directly, dropping the legacy ArkitOSCListener conversion layer. * fix(avlivebody-mac): guard thread store with lock Move the `thread` property write inside the stateLock-held region in start(); t.start() stays outside since the thread cannot run before start() is called. Removes a latent race. * feat(avlivebody-mac): multi-hmr and body fusion Context: Task 4 of the macOS rewrite needs the dense-mesh half of the pipeline alongside the USB skeleton consumer landed in task 3. Approach: Add a CoreML wrapper that mirrors the validated Python reference (data_only_viz/multihmr_coreml.py) and a pure-logic fusion stage that corrects the mesh pelvis depth using the LiDAR-precise USB skeleton. Changes: - MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at 10475 vertices, det threshold 0.3. - BodyFusion.swift: stateless fuse(persons, skeletons) overrides the highest-score mesh translation.z with the skeleton pelvis Z when available, passes through otherwise. - BodyFusionTests.swift: pelvis override and pass-through cases. Impact: Unlocks the mesh renderer wiring in later tasks and gives the macOS app metrically-correct depth in front of the camera. * fix(avlivebody-mac): load mlmodelc, clarify fusion Xcode compiles .mlpackage resources to .mlmodelc at build time; look up the compiled artifact directly and drop the redundant MLModel.compileModel step. Also rewrite BodyFusion docstring to match actual single-person pelvis-z override behaviour. * feat(avlivebody-mac): scene controller + view RealityKit scene plumbing: SceneController owns ARView, orbital camera, and holders for VideoQuad/SkeletonEntity/MeshEntity. SceneView is the SwiftUI NSViewRepresentable bridge. Build intentionally deferred to T8 (holder types land in T6-T8). * fix(avlivebody): orbit gesture + setUp guard Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to dispatch only on .changed, replacing the Task wrapper with MainActor.assumeIsolated. Guard SceneController.setUp() with a didSetUp flag so duplicate makeNSView calls do not re-install gestures or re-add anchors. * feat(avlivebody-mac): 91-joint skeleton entity Yellow marker spheres pooled per pid; ARKit (x,y,z) -> RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for macOS RealityKit Material.Color. Build deferred to T8. * feat(avlivebody-mac): video quad Flat 1.6x0.9m plane at z=-2m, textured per-frame from CVPixelBuffer via CIImage -> CGImage -> TextureResource. Per-frame TextureResource creation is the known perf hot spot, isolated here for later LowLevelTexture upgrade. * fix(avlivebody-mac): appkit import for orbit NSPanGestureRecognizer lives in AppKit on macOS; without the import the AVLiveBody module failed to emit. T5 leftover surfaced once T6/T7/T8 made the target compilable. * feat(avlivebody-mac): smpl-x mesh entity Render SMPL-X dense meshes (10475 verts) from Multi-HMR with pooled ModelEntity per person. Triangle indices loaded from the bundled smplx_faces.bin (flat UInt32 triplets, copied from the legacy launcher target). xcodegen folder-scanning bundles the .bin under Contents/Resources/ — no project.yml change needed. * feat(avlivebody-mac): wire scene + status bar Replace placeholder window with ContentView wiring USBSkeletonConsumer, SceneController, MultiHMRCoreML and BodyFusion per the T9 dataflow plan. * chore: archive legacy AV-Live-Body Superseded by avlivebody-mac/ on 2026-05-18. See docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md for the rewrite design and rationale. * fix(avlivebody): break onVideoFrame retain cycle Capture consumer weakly in the onVideoFrame closure so the USBSkeletonConsumer can be deallocated and its background thread exits cleanly. Guard the mesh-fusion path when consumer is gone. * fix(launcher): disable body spawn post-archive Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/. New native Xcode app lives at avlivebody-mac/; no swift run path. startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18). * docs(avlivebody-mac): contributor setup README Document prerequisites, mlpackage copy, signing xcconfig, and xcodegen/xcodebuild commands. Points at design spec and plan. * refactor(avlivebody): axis helper + cleanups - Extract arkitToRealityKit helper, dedupe 3 call sites. - Add onDisappear consumer.stop() to terminate USB read loop. - Replace @State with let for SceneController (stable class id). - Add NSLog diagnostics in VideoQuad+MeshEntity silent guards. * fix(avlivebody): ad-hoc signing for local dev Apple Development cert + Automatic signing makes Xcode demand a Mac Development cert that no one has. Switch to manual ad-hoc (CODE_SIGN_IDENTITY = -) so any contributor can build. Drop hardened runtime; re-enable for distribution builds. * feat(arbody): keep iphone awake while streaming iOS auto-lock tears down the USBServer TCP listener within seconds, breaking AVLiveBody Mac-side connect. Disable the idle timer for the lifetime of ContentView, restore on exit. * docs: add AGENTS.md skeleton
25 KiB
macOS USB Consumer Implementation Plan (Plan 3a of 3)
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Make the macOS AVLiveBody app consume the iPhone's USB stream — connect via usbmuxd, demux AVLiveWire frames, render the 91-joint skeleton on screen, and HEVC-decode the video — without the Multi-HMR dense-mesh step (deferred to Plan 3b).
Architecture: A new USBSkeletonConsumer runs the blocking UnixMuxTransport/USBClient read loop on a dedicated background thread, feeds bytes through StreamDemuxer, and republishes .skeleton frames as @Published ARKit-shaped body frames plus a .video callback. Skeleton3DRenderer's long-standing // TODO: render yellow ARKit markers (line 138) is completed so the 91-joint USB skeleton actually draws. A new VideoDecoder turns .video VideoPayloads into CVPixelBuffers via VTDecompressionSession.
Tech Stack: Swift 5 (language mode v5), macOS 15, RealityKit, VideoToolbox, AVLiveWire (already a dependency of AV-Live-Body), XCTest.
Companion spec: docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md
Prerequisites: Plan 1 (transport, merged), Plan 2 (iOS capture, merged).
Out of scope: MultiHMRCoreML, BodyFusion, dense-mesh rendering — Plan 3b, gated on a confirmed CoreML Multi-HMR .mlpackage.
Verification
AV-Live-Body is a macOS target — it builds on the host:
cd launcher/AV-Live-Body && swift build
cd launcher/AV-Live-Body && swift test
Each task ends with swift build (and swift test where a test was
added) succeeding.
File Structure
| File | Responsibility |
|---|---|
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift |
NEW. Background USB read loop → StreamDemuxer → @Published body frames + video callback |
launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift |
NEW. VTDecompressionSession HEVC decode: VideoPayload → CVPixelBuffer |
launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift |
NEW. Unit test for the SkeletonPayload → ArkitBodyFrame mapping |
launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift |
MODIFY. Complete the line-138 TODO: draw 91 USB-skeleton joint markers |
launcher/AV-Live-Body/Sources/AVLiveBody/ArkitOSCListener.swift |
REFERENCE only — reuse its nested ArkitBodyFrame type |
launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift |
MODIFY. Own a USBSkeletonConsumer, start it in .onAppear |
launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift |
MODIFY. Thread the consumer into Skeleton3DRenderer.attach |
Task 1: USBSkeletonConsumer
USBSkeletonConsumer owns the blocking USB read loop on a background
Thread. It reconnects on drop. It republishes .skeleton frames as
ArkitOSCListener.ArkitBodyFrame (the existing 91-joint body type, so
Skeleton3DRenderer can consume them with no new type) and forwards
.video payloads via a callback. It is not @MainActor: the loop
runs off-main and hops to main only for @Published writes — the same
pattern as ArkitOSCListener.
Files:
-
Create:
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift -
Test:
launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift -
Step 1: Write the failing test
launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift:
import XCTest
import AVLiveWire
@testable import AVLiveBody
final class USBSkeletonConsumerTests: XCTestCase {
func testSkeletonPayloadMapsToBodyFrame() {
var p = SkeletonPayload()
p.joints[0] = SIMD3(1, 2, 3)
p.valid[0] = true
p.joints[90] = SIMD3(-4, 5, -6)
p.valid[90] = true
let frame = USBSkeletonConsumer.bodyFrame(pid: 7, from: p)
XCTAssertEqual(frame.pid, 7)
XCTAssertEqual(frame.joints.count, 91)
XCTAssertEqual(frame.hasJoint.count, 91)
XCTAssertEqual(frame.joints[0], SIMD3(1, 2, 3))
XCTAssertTrue(frame.hasJoint[0])
XCTAssertEqual(frame.joints[90], SIMD3(-4, 5, -6))
XCTAssertFalse(frame.hasJoint[1])
}
}
- Step 2: Run the test to verify it fails
Run: cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests
Expected: FAIL — USBSkeletonConsumer undefined.
- Step 3: Write the implementation
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift:
import AVLiveWire
import Combine
import Foundation
/// Connects to the tethered iPhone over USB (usbmuxd), demuxes the
/// AVLiveWire stream, and republishes skeleton frames (as the existing
/// 91-joint `ArkitOSCListener.ArkitBodyFrame`) plus video payloads.
/// The blocking transport runs on a dedicated background thread; only
/// `@Published` writes hop to the main queue.
final class USBSkeletonConsumer: ObservableObject {
/// 91-joint body frames keyed by pid — same shape `Skeleton3DRenderer`
/// already consumes from `ArkitOSCListener`.
@Published var bodies: [Int: ArkitOSCListener.ArkitBodyFrame] = [:]
@Published var connected = false
/// Called (on the main queue) for every decoded `.video` frame.
var onVideo: ((VideoPayload) -> Void)?
/// TCP port the iPhone `USBServer` listens on (must match the iOS
/// app's `USBServer.port`).
static let devicePort: UInt16 = 7000
private let stateLock = NSLock()
private var running = false
private var thread: Thread?
private var isRunning: Bool {
stateLock.lock(); defer { stateLock.unlock() }
return running
}
func start() {
stateLock.lock()
if running { stateLock.unlock(); return }
running = true
stateLock.unlock()
let t = Thread { [weak self] in self?.loop() }
t.name = "cc.avlive.usbconsumer"
t.start()
thread = t
}
func stop() {
stateLock.lock(); running = false; stateLock.unlock()
}
/// Pure mapping `SkeletonPayload` -> `ArkitBodyFrame`. Static so it
/// is unit-testable without a transport.
static func bodyFrame(pid: Int, from p: SkeletonPayload)
-> ArkitOSCListener.ArkitBodyFrame {
var f = ArkitOSCListener.ArkitBodyFrame()
f.pid = pid
f.joints = p.joints
f.hasJoint = p.valid
f.seenAt = CFAbsoluteTimeGetCurrent()
return f
}
// MARK: - Background read loop
private func loop() {
while isRunning {
guard let transport = UnixMuxTransport() else {
Thread.sleep(forTimeInterval: 1.0); continue
}
let client = USBClient(transport: transport)
guard let dev = client.listDevices().first,
client.connect(deviceID: dev,
port: Self.devicePort) else {
transport.close()
Thread.sleep(forTimeInterval: 1.0); continue
}
publishConnected(true)
var demux = StreamDemuxer()
while isRunning {
guard let chunk = transport.readStream(),
!chunk.isEmpty else { break }
for frame in demux.feed(chunk) { route(frame) }
}
transport.close()
publishConnected(false)
if isRunning { Thread.sleep(forTimeInterval: 1.0) }
}
}
private func route(_ frame: StreamDemuxer.Frame) {
switch frame.header.tag {
case .skeleton:
guard let payload =
SkeletonPayload(decoding: frame.payload) else { return }
let pid = Int(frame.header.pid)
let body = Self.bodyFrame(pid: pid, from: payload)
DispatchQueue.main.async { [weak self] in
self?.bodies[pid] = body
}
case .video:
guard let payload =
VideoPayload(decoding: frame.payload) else { return }
DispatchQueue.main.async { [weak self] in
self?.onVideo?(payload)
}
case .meta:
break
}
}
private func publishConnected(_ value: Bool) {
DispatchQueue.main.async { [weak self] in
self?.connected = value
}
}
}
- Step 4: Run the test to verify it passes
Run: cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests
Expected: PASS, 1 test.
If ArkitOSCListener.ArkitBodyFrame has no memberwise mutability or a
different field set than pid/joints/hasJoint/seenAt, read
ArkitOSCListener.swift and adjust bodyFrame to match the actual
struct (it is a struct ArkitBodyFrame: Equatable with var pid,
var joints: [SIMD3<Float>], var hasJoint: [Bool], var seenAt).
- Step 5: Run the full suite + commit
Run: cd launcher/AV-Live-Body && swift test
Expected: PASS, all tests (7: prior 6 + this 1).
git add launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift
git commit -m "feat(av-live-body): USB skeleton consumer"
(subject ≤50 chars; add a short body — the hook rejects subject-only.)
Task 2: VideoDecoder
VideoDecoder turns .video VideoPayloads into CVPixelBuffers. A
keyframe payload carries the HEVC parameter sets prepended (each as a
4-byte big-endian length prefix + NAL bytes — the format Plan 2's iOS
VideoEncoder produces); the decoder builds its
CMVideoFormatDescription from those, then decodes subsequent access
units.
Files:
-
Create:
launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift -
Step 1: Write the implementation
launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift:
import AVLiveWire
import CoreMedia
import CoreVideo
import Foundation
import VideoToolbox
/// HEVC decoder. Feed `VideoPayload`s in; receive `CVPixelBuffer`s via
/// `onFrame`. Keyframe payloads must carry the VPS/SPS/PPS parameter
/// sets prepended as 4-byte-length-prefixed NAL units (the layout the
/// iOS `VideoEncoder` emits); the decoder (re)builds its format
/// description from those.
final class VideoDecoder {
var onFrame: ((CVPixelBuffer) -> Void)?
private var session: VTDecompressionSession?
private var formatDesc: CMVideoFormatDescription?
/// Decode one access unit.
func decode(_ payload: VideoPayload) {
var au = payload.data
if payload.isKeyframe {
// Split the prepended parameter sets from the frame data.
let (params, rest) = Self.splitParameterSets(au)
if !params.isEmpty {
rebuildFormat(params)
}
au = rest
}
guard let fmt = formatDesc, !au.isEmpty else { return }
if session == nil { makeSession(fmt) }
guard let session else { return }
guard let block = Self.blockBuffer(au) else { return }
var sample: CMSampleBuffer?
var sampleSize = au.count
guard CMSampleBufferCreateReady(
allocator: kCFAllocatorDefault, dataBuffer: block,
formatDescription: fmt, sampleCount: 1, sampleTimingEntryCount: 0,
sampleTimingArray: nil, sampleSizeEntryCount: 1,
sampleSizeArray: &sampleSize,
sampleBufferOut: &sample) == noErr, let sample else { return }
VTDecompressionSessionDecodeFrame(
session, sampleBuffer: sample, flags: [],
infoFlagsOut: nil) { [weak self] status, _, image, _, _ in
guard status == noErr, let image else { return }
self?.onFrame?(image)
}
}
func stop() {
if let session { VTDecompressionSessionInvalidate(session) }
session = nil
formatDesc = nil
}
deinit { stop() }
// MARK: - Helpers
/// Parameter sets are 4-byte-length-prefixed NAL units at the head
/// of a keyframe payload. The first NAL whose type is a VCL slice
/// marks the start of frame data — but to stay simple and robust,
/// we treat every leading NAL as a parameter set until the running
/// concatenation can build a valid HEVC format description; the
/// remainder is the frame. Returns (parameterSetData, frameData).
private static func splitParameterSets(_ data: Data)
-> (Data, Data) {
// Parameter set NALs for HEVC: VPS=32, SPS=33, PPS=34
// (nal_unit_type = (firstByte >> 1) & 0x3F).
var offset = 0
let bytes = [UInt8](data)
var paramEnd = 0
while offset + 4 <= bytes.count {
let len = (Int(bytes[offset]) << 24)
| (Int(bytes[offset + 1]) << 16)
| (Int(bytes[offset + 2]) << 8)
| Int(bytes[offset + 3])
let nalStart = offset + 4
guard len > 0, nalStart + len <= bytes.count else { break }
let nalType = (Int(bytes[nalStart]) >> 1) & 0x3F
if nalType == 32 || nalType == 33 || nalType == 34 {
offset = nalStart + len
paramEnd = offset
} else {
break
}
}
return (data.prefix(paramEnd),
data.suffix(from: data.startIndex
.advanced(by: paramEnd)))
}
private func rebuildFormat(_ paramData: Data) {
var sets: [[UInt8]] = []
let bytes = [UInt8](paramData)
var offset = 0
while offset + 4 <= bytes.count {
let len = (Int(bytes[offset]) << 24)
| (Int(bytes[offset + 1]) << 16)
| (Int(bytes[offset + 2]) << 8)
| Int(bytes[offset + 3])
let start = offset + 4
guard len > 0, start + len <= bytes.count else { break }
sets.append(Array(bytes[start..<start + len]))
offset = start + len
}
guard sets.count >= 3 else { return }
let pointers = sets.map { UnsafePointer<UInt8>($0) }
let sizes = sets.map { $0.count }
var fmt: CMFormatDescription?
let status = pointers.withUnsafeBufferPointer { pBuf in
sizes.withUnsafeBufferPointer { sBuf in
CMVideoFormatDescriptionCreateFromHEVCParameterSets(
allocator: kCFAllocatorDefault,
parameterSetCount: sets.count,
parameterSetPointers: pBuf.baseAddress!,
parameterSetSizes: sBuf.baseAddress!,
nalUnitHeaderLength: 4, extensions: nil,
formatDescriptionOut: &fmt)
}
}
if status == noErr, let fmt {
formatDesc = fmt
if let session { VTDecompressionSessionInvalidate(session) }
session = nil
}
}
private func makeSession(_ fmt: CMVideoFormatDescription) {
let attrs: [CFString: Any] = [
kCVPixelBufferPixelFormatTypeKey:
kCVPixelFormatType_32BGRA,
]
VTDecompressionSessionCreate(
allocator: kCFAllocatorDefault, formatDescription: fmt,
decoderSpecification: nil,
imageBufferAttributes: attrs as CFDictionary,
outputCallback: nil, decompressionSessionOut: &session)
}
private static func blockBuffer(_ data: Data) -> CMBlockBuffer? {
var block: CMBlockBuffer?
guard CMBlockBufferCreateWithMemoryBlock(
allocator: kCFAllocatorDefault, memoryBlock: nil,
blockLength: data.count, blockAllocator: kCFAllocatorDefault,
customBlockSource: nil, offsetToData: 0,
dataLength: data.count, flags: 0,
blockBufferOut: &block) == noErr, let block else {
return nil
}
var ok = false
data.withUnsafeBytes { raw in
if CMBlockBufferReplaceDataBytes(
with: raw.baseAddress!, blockBuffer: block,
offsetIntoDestination: 0,
dataLength: data.count) == noErr { ok = true }
}
return ok ? block : nil
}
}
- Step 2: Verify it compiles
Run: cd launcher/AV-Live-Body && swift build
Expected: build succeeds. If a VideoToolbox/CoreMedia signature differs
on this SDK, fix minimally — the behavior (build a format description
from the prepended parameter sets, decode the rest) must be preserved.
- Step 3: Commit
git add launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift
git commit -m "feat(av-live-body): HEVC video decoder"
Task 3: Render the 91-joint USB skeleton
Skeleton3DRenderer already subscribes to a 91-joint ARKit body
publisher into lastArkit but never draws it — Skeleton3DRenderer.swift:138
is // TODO: render yellow ARKit markers from lastArkit in update().
Complete it: draw the 91 joints as small yellow spheres.
Files:
-
Modify:
launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift -
Step 1: Read the renderer
Read Skeleton3DRenderer.swift fully. Note: PersonEntities (the
per-pid entity struct), lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame],
makePerson(pid:parent:), the update(frames:) 30 fps tick, and the
RealityKit space conversion used for MediaPipe joints
(SIMD3(k.x, -k.y, -k.z)).
- Step 2: Add 91 ARKit marker entities to
PersonEntities
In the PersonEntities struct, add a field:
var arkitMarkers: [ModelEntity] // 91 yellow ARKit joint spheres
In makePerson(pid:parent:), after the hand spheres are built, create
91 yellow marker spheres (reuse the jointRadius-sized sphere mesh, a
yellow SimpleMaterial), parent them to root, start them disabled,
and include arkitMarkers: in the returned PersonEntities(...):
let arkitMat = SimpleMaterial(
color: .systemYellow, roughness: 0.6, isMetallic: false)
var arkitMarkers: [ModelEntity] = []
arkitMarkers.reserveCapacity(91)
for _ in 0..<91 {
let e = ModelEntity(mesh: sphereMesh, materials: [arkitMat])
e.isEnabled = false
root.addChild(e)
arkitMarkers.append(e)
}
- Step 3: Draw the ARKit markers each tick
Replace the line // TODO: render yellow ARKit markers from lastArkit in update()
(Skeleton3DRenderer.swift:138) — leave the comment removed — and add,
at the end of update(frames:) (after the existing per-pid loop), a
call to a new private method applyArkit(). Then add the method:
/// Draw the 91-joint ARKit/USB skeletons as yellow joint markers.
/// ARKit joints are world-space metric; convert to RealityKit
/// space (x, y, z) -> (x, -y, -z) like the MediaPipe path.
private func applyArkit() {
for (pid, entities) in persons {
guard let frame = lastArkit[pid] else {
for m in entities.arkitMarkers { m.isEnabled = false }
continue
}
let n = min(91, entities.arkitMarkers.count,
frame.joints.count)
for i in 0..<n {
let marker = entities.arkitMarkers[i]
if frame.hasJoint[i] {
let j = frame.joints[i]
marker.transform.translation =
SIMD3<Float>(j.x, -j.y, -j.z)
marker.isEnabled = true
} else {
marker.isEnabled = false
}
}
for i in n..<entities.arkitMarkers.count {
entities.arkitMarkers[i].isEnabled = false
}
}
}
Note: applyArkit() iterates persons, which is only populated for
pids seen in the MediaPipe frames. If the USB skeleton must show
when there is no MediaPipe pose, also create a PersonEntities for
each pid present in lastArkit. To keep Task 3 minimal, in
update(frames:) before applyArkit(), ensure entities exist for
ARKit-only pids:
for pid in lastArkit.keys where persons[pid] == nil {
persons[pid] = makePerson(pid: pid, parent: anchor)
lastSeenAt[pid] = now
}
- Step 4: Verify build + tests
Run: cd launcher/AV-Live-Body && swift build — Expected: succeeds.
Run: cd launcher/AV-Live-Body && swift test — Expected: all tests
still pass (no regression).
- Step 5: Commit
git add launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift
git commit -m "feat(av-live-body): render 91-joint USB skeleton"
Task 4: Wire the consumer into the app
Construct USBSkeletonConsumer in the app, start/stop it with the
scene, and feed it into Skeleton3DRenderer in place of (or alongside)
ArkitOSCListener.
Files:
-
Modify:
launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift -
Modify:
launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift -
Step 1: Read the two files
Read AVLiveBodyApp.swift and BodyView.swift. Identify: where the
@StateObject listeners are declared in ContentView, where .onAppear
starts them, how ArkitOSCListener is passed into BodyView, and where
BodyView.makeNSView calls skel3d.attach(to:listener:arkitListener:).
- Step 2: Own and start the consumer
In AVLiveBodyApp.swift's ContentView, add a @StateObject:
@StateObject private var usbConsumer = USBSkeletonConsumer()
In .onAppear, alongside the existing listener .start() calls, add
usbConsumer.start(). If there is an .onDisappear, add
usbConsumer.stop().
- Step 3: Thread the consumer to the renderer
Skeleton3DRenderer.attach currently takes
arkitListener: ArkitOSCListener?. The simplest correct change: give
USBSkeletonConsumer the same role. Add an overload / extra parameter
so attach can subscribe to usbConsumer.$bodies exactly as it
subscribes to arkitListener.$bodies (both publish
[Int: ArkitOSCListener.ArkitBodyFrame]). Concretely, in
Skeleton3DRenderer.attach, accept usbConsumer: USBSkeletonConsumer?
and, if non-nil, subscribe its $bodies into lastArkit with the same
sink already used for arkitListener (the arkitSub Combine
subscription). Pass usbConsumer from ContentView → BodyView →
makeNSView → skel3d.attach(...), mirroring how arkitListener is
already threaded.
If arkitListener (the OSC one) is now redundant, it may be passed as
nil; do not delete ArkitOSCListener in this plan (other code or
Plan 3b cleanup may still reference it).
- Step 4: Verify build
Run: cd launcher/AV-Live-Body && swift build — Expected: succeeds.
Run: cd launcher/AV-Live-Body && swift test — Expected: no regression.
- Step 5: Commit
git add launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift
git commit -m "feat(av-live-body): wire USB consumer to renderer"
Task 5: Final verification
- Step 1: Clean build + full test suite
cd launcher/AV-Live-Body && swift build && swift test
Expected: build succeeds; all tests pass (7: prior 6 + Task 1's).
- Step 2: Confirm the integration seam
USBSkeletonConsumer.devicePort (7000) must equal the iOS app's
USBServer.port. Verify:
grep -rn "port.*7000\|devicePort" \
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift \
iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/USBServer.swift
Expected: both sides use 7000.
- Step 3: Commit any fix (only if Step 2 found a mismatch).
Self-Review
- Spec coverage: This plan implements the spec's
USBClientconsumption insideAVLiveBody, theVideoDecoderunit, and the skeleton render path.MultiHMRCoreML,BodyFusion, and dense-mesh rendering are explicitly Plan 3b (gated on a confirmed CoreML Multi-HMR.mlpackage). - Placeholders: none — new files have complete code; modify tasks
cite exact files and the line-138 TODO, and instruct the implementer
to read exact context for
AVLiveBodyApp.swift/BodyView.swift(whose current line numbers are not reproduced here). - Type consistency:
USBSkeletonConsumer.bodyFramereturnsArkitOSCListener.ArkitBodyFrame;Skeleton3DRendereralready storeslastArkit: [Int: ArkitOSCListener.ArkitBodyFrame], so the consumer is type-compatible with the existingarkitSubpath.VideoDecoderconsumesVideoPayloadexactly as Plan 2'sVideoEncoderproduces it (parameter sets prepended, 4-byte big-endian length prefixes). - Known risks: (1)
BodyViewownsSkeleton3DRenderer, so Task 4 threads a new object throughContentView→BodyView→attach— multi-file, follow the existingarkitListenerthreading exactly. (2)StreamDemuxer.findMagiccopies the whole buffer perfeed(); for HEVC video this is a perf risk — acceptable for Plan 3a, revisit if frame rate suffers. (3) The HEVC parameter-set split inVideoDecoderassumes the iOS encoder's exact prepend layout — this is the Plan 2 ↔ Plan 3a integration seam; validate on real device data.