docs: macOS USB consumer plan (3a of 3)

Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.
This commit is contained in:
L'électron rare
2026-05-18 17:43:11 +02:00
parent 35b18f6ca5
commit 131c93f885
@@ -0,0 +1,655 @@
# macOS USB Consumer Implementation Plan (Plan 3a of 3)
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make the macOS `AVLiveBody` app consume the iPhone's USB stream — connect via `usbmuxd`, demux `AVLiveWire` frames, render the 91-joint skeleton on screen, and HEVC-decode the video — without the Multi-HMR dense-mesh step (deferred to Plan 3b).
**Architecture:** A new `USBSkeletonConsumer` runs the blocking `UnixMuxTransport`/`USBClient` read loop on a dedicated background thread, feeds bytes through `StreamDemuxer`, and republishes `.skeleton` frames as `@Published` ARKit-shaped body frames plus a `.video` callback. `Skeleton3DRenderer`'s long-standing `// TODO: render yellow ARKit markers` (line 138) is completed so the 91-joint USB skeleton actually draws. A new `VideoDecoder` turns `.video` `VideoPayload`s into `CVPixelBuffer`s via `VTDecompressionSession`.
**Tech Stack:** Swift 5 (language mode v5), macOS 15, RealityKit, VideoToolbox, `AVLiveWire` (already a dependency of `AV-Live-Body`), `XCTest`.
**Companion spec:** `docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md`
**Prerequisites:** Plan 1 (transport, merged), Plan 2 (iOS capture, merged).
**Out of scope:** `MultiHMRCoreML`, `BodyFusion`, dense-mesh rendering — Plan 3b, gated on a confirmed CoreML Multi-HMR `.mlpackage`.
---
## Verification
`AV-Live-Body` is a macOS target — it builds on the host:
```bash
cd launcher/AV-Live-Body && swift build
cd launcher/AV-Live-Body && swift test
```
Each task ends with `swift build` (and `swift test` where a test was
added) succeeding.
---
## File Structure
| File | Responsibility |
|------|----------------|
| `launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift` | NEW. Background USB read loop → `StreamDemuxer``@Published` body frames + video callback |
| `launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift` | NEW. `VTDecompressionSession` HEVC decode: `VideoPayload``CVPixelBuffer` |
| `launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift` | NEW. Unit test for the `SkeletonPayload``ArkitBodyFrame` mapping |
| `launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift` | MODIFY. Complete the line-138 TODO: draw 91 USB-skeleton joint markers |
| `launcher/AV-Live-Body/Sources/AVLiveBody/ArkitOSCListener.swift` | REFERENCE only — reuse its nested `ArkitBodyFrame` type |
| `launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift` | MODIFY. Own a `USBSkeletonConsumer`, start it in `.onAppear` |
| `launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift` | MODIFY. Thread the consumer into `Skeleton3DRenderer.attach` |
---
## Task 1: USBSkeletonConsumer
`USBSkeletonConsumer` owns the blocking USB read loop on a background
`Thread`. It reconnects on drop. It republishes `.skeleton` frames as
`ArkitOSCListener.ArkitBodyFrame` (the existing 91-joint body type, so
`Skeleton3DRenderer` can consume them with no new type) and forwards
`.video` payloads via a callback. It is **not** `@MainActor`: the loop
runs off-main and hops to main only for `@Published` writes — the same
pattern as `ArkitOSCListener`.
**Files:**
- Create: `launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift`
- Test: `launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift`
- [ ] **Step 1: Write the failing test**
`launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift`:
```swift
import XCTest
import AVLiveWire
@testable import AVLiveBody
final class USBSkeletonConsumerTests: XCTestCase {
func testSkeletonPayloadMapsToBodyFrame() {
var p = SkeletonPayload()
p.joints[0] = SIMD3(1, 2, 3)
p.valid[0] = true
p.joints[90] = SIMD3(-4, 5, -6)
p.valid[90] = true
let frame = USBSkeletonConsumer.bodyFrame(pid: 7, from: p)
XCTAssertEqual(frame.pid, 7)
XCTAssertEqual(frame.joints.count, 91)
XCTAssertEqual(frame.hasJoint.count, 91)
XCTAssertEqual(frame.joints[0], SIMD3(1, 2, 3))
XCTAssertTrue(frame.hasJoint[0])
XCTAssertEqual(frame.joints[90], SIMD3(-4, 5, -6))
XCTAssertFalse(frame.hasJoint[1])
}
}
```
- [ ] **Step 2: Run the test to verify it fails**
Run: `cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests`
Expected: FAIL — `USBSkeletonConsumer` undefined.
- [ ] **Step 3: Write the implementation**
`launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift`:
```swift
import AVLiveWire
import Combine
import Foundation
/// Connects to the tethered iPhone over USB (usbmuxd), demuxes the
/// AVLiveWire stream, and republishes skeleton frames (as the existing
/// 91-joint `ArkitOSCListener.ArkitBodyFrame`) plus video payloads.
/// The blocking transport runs on a dedicated background thread; only
/// `@Published` writes hop to the main queue.
final class USBSkeletonConsumer: ObservableObject {
/// 91-joint body frames keyed by pid same shape `Skeleton3DRenderer`
/// already consumes from `ArkitOSCListener`.
@Published var bodies: [Int: ArkitOSCListener.ArkitBodyFrame] = [:]
@Published var connected = false
/// Called (on the main queue) for every decoded `.video` frame.
var onVideo: ((VideoPayload) -> Void)?
/// TCP port the iPhone `USBServer` listens on (must match the iOS
/// app's `USBServer.port`).
static let devicePort: UInt16 = 7000
private let stateLock = NSLock()
private var running = false
private var thread: Thread?
private var isRunning: Bool {
stateLock.lock(); defer { stateLock.unlock() }
return running
}
func start() {
stateLock.lock()
if running { stateLock.unlock(); return }
running = true
stateLock.unlock()
let t = Thread { [weak self] in self?.loop() }
t.name = "cc.avlive.usbconsumer"
t.start()
thread = t
}
func stop() {
stateLock.lock(); running = false; stateLock.unlock()
}
/// Pure mapping `SkeletonPayload` -> `ArkitBodyFrame`. Static so it
/// is unit-testable without a transport.
static func bodyFrame(pid: Int, from p: SkeletonPayload)
-> ArkitOSCListener.ArkitBodyFrame {
var f = ArkitOSCListener.ArkitBodyFrame()
f.pid = pid
f.joints = p.joints
f.hasJoint = p.valid
f.seenAt = CFAbsoluteTimeGetCurrent()
return f
}
// MARK: - Background read loop
private func loop() {
while isRunning {
guard let transport = UnixMuxTransport() else {
Thread.sleep(forTimeInterval: 1.0); continue
}
let client = USBClient(transport: transport)
guard let dev = client.listDevices().first,
client.connect(deviceID: dev,
port: Self.devicePort) else {
transport.close()
Thread.sleep(forTimeInterval: 1.0); continue
}
publishConnected(true)
var demux = StreamDemuxer()
while isRunning {
guard let chunk = transport.readStream(),
!chunk.isEmpty else { break }
for frame in demux.feed(chunk) { route(frame) }
}
transport.close()
publishConnected(false)
if isRunning { Thread.sleep(forTimeInterval: 1.0) }
}
}
private func route(_ frame: StreamDemuxer.Frame) {
switch frame.header.tag {
case .skeleton:
guard let payload =
SkeletonPayload(decoding: frame.payload) else { return }
let pid = Int(frame.header.pid)
let body = Self.bodyFrame(pid: pid, from: payload)
DispatchQueue.main.async { [weak self] in
self?.bodies[pid] = body
}
case .video:
guard let payload =
VideoPayload(decoding: frame.payload) else { return }
DispatchQueue.main.async { [weak self] in
self?.onVideo?(payload)
}
case .meta:
break
}
}
private func publishConnected(_ value: Bool) {
DispatchQueue.main.async { [weak self] in
self?.connected = value
}
}
}
```
- [ ] **Step 4: Run the test to verify it passes**
Run: `cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests`
Expected: PASS, 1 test.
If `ArkitOSCListener.ArkitBodyFrame` has no memberwise mutability or a
different field set than `pid`/`joints`/`hasJoint`/`seenAt`, read
`ArkitOSCListener.swift` and adjust `bodyFrame` to match the actual
struct (it is a `struct ArkitBodyFrame: Equatable` with `var pid`,
`var joints: [SIMD3<Float>]`, `var hasJoint: [Bool]`, `var seenAt`).
- [ ] **Step 5: Run the full suite + commit**
Run: `cd launcher/AV-Live-Body && swift test`
Expected: PASS, all tests (7: prior 6 + this 1).
```bash
git add launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift
git commit -m "feat(av-live-body): USB skeleton consumer"
```
(subject ≤50 chars; add a short body — the hook rejects subject-only.)
---
## Task 2: VideoDecoder
`VideoDecoder` turns `.video` `VideoPayload`s into `CVPixelBuffer`s. A
keyframe payload carries the HEVC parameter sets prepended (each as a
4-byte big-endian length prefix + NAL bytes — the format Plan 2's iOS
`VideoEncoder` produces); the decoder builds its
`CMVideoFormatDescription` from those, then decodes subsequent access
units.
**Files:**
- Create: `launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift`
- [ ] **Step 1: Write the implementation**
`launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift`:
```swift
import AVLiveWire
import CoreMedia
import CoreVideo
import Foundation
import VideoToolbox
/// HEVC decoder. Feed `VideoPayload`s in; receive `CVPixelBuffer`s via
/// `onFrame`. Keyframe payloads must carry the VPS/SPS/PPS parameter
/// sets prepended as 4-byte-length-prefixed NAL units (the layout the
/// iOS `VideoEncoder` emits); the decoder (re)builds its format
/// description from those.
final class VideoDecoder {
var onFrame: ((CVPixelBuffer) -> Void)?
private var session: VTDecompressionSession?
private var formatDesc: CMVideoFormatDescription?
/// Decode one access unit.
func decode(_ payload: VideoPayload) {
var au = payload.data
if payload.isKeyframe {
// Split the prepended parameter sets from the frame data.
let (params, rest) = Self.splitParameterSets(au)
if !params.isEmpty {
rebuildFormat(params)
}
au = rest
}
guard let fmt = formatDesc, !au.isEmpty else { return }
if session == nil { makeSession(fmt) }
guard let session else { return }
guard let block = Self.blockBuffer(au) else { return }
var sample: CMSampleBuffer?
var sampleSize = au.count
guard CMSampleBufferCreateReady(
allocator: kCFAllocatorDefault, dataBuffer: block,
formatDescription: fmt, sampleCount: 1, sampleTimingEntryCount: 0,
sampleTimingArray: nil, sampleSizeEntryCount: 1,
sampleSizeArray: &sampleSize,
sampleBufferOut: &sample) == noErr, let sample else { return }
VTDecompressionSessionDecodeFrame(
session, sampleBuffer: sample, flags: [],
infoFlagsOut: nil) { [weak self] status, _, image, _, _ in
guard status == noErr, let image else { return }
self?.onFrame?(image)
}
}
func stop() {
if let session { VTDecompressionSessionInvalidate(session) }
session = nil
formatDesc = nil
}
deinit { stop() }
// MARK: - Helpers
/// Parameter sets are 4-byte-length-prefixed NAL units at the head
/// of a keyframe payload. The first NAL whose type is a VCL slice
/// marks the start of frame data but to stay simple and robust,
/// we treat every leading NAL as a parameter set until the running
/// concatenation can build a valid HEVC format description; the
/// remainder is the frame. Returns (parameterSetData, frameData).
private static func splitParameterSets(_ data: Data)
-> (Data, Data) {
// Parameter set NALs for HEVC: VPS=32, SPS=33, PPS=34
// (nal_unit_type = (firstByte >> 1) & 0x3F).
var offset = 0
let bytes = [UInt8](data)
var paramEnd = 0
while offset + 4 <= bytes.count {
let len = (Int(bytes[offset]) << 24)
| (Int(bytes[offset + 1]) << 16)
| (Int(bytes[offset + 2]) << 8)
| Int(bytes[offset + 3])
let nalStart = offset + 4
guard len > 0, nalStart + len <= bytes.count else { break }
let nalType = (Int(bytes[nalStart]) >> 1) & 0x3F
if nalType == 32 || nalType == 33 || nalType == 34 {
offset = nalStart + len
paramEnd = offset
} else {
break
}
}
return (data.prefix(paramEnd),
data.suffix(from: data.startIndex
.advanced(by: paramEnd)))
}
private func rebuildFormat(_ paramData: Data) {
var sets: [[UInt8]] = []
let bytes = [UInt8](paramData)
var offset = 0
while offset + 4 <= bytes.count {
let len = (Int(bytes[offset]) << 24)
| (Int(bytes[offset + 1]) << 16)
| (Int(bytes[offset + 2]) << 8)
| Int(bytes[offset + 3])
let start = offset + 4
guard len > 0, start + len <= bytes.count else { break }
sets.append(Array(bytes[start..<start + len]))
offset = start + len
}
guard sets.count >= 3 else { return }
let pointers = sets.map { UnsafePointer<UInt8>($0) }
let sizes = sets.map { $0.count }
var fmt: CMFormatDescription?
let status = pointers.withUnsafeBufferPointer { pBuf in
sizes.withUnsafeBufferPointer { sBuf in
CMVideoFormatDescriptionCreateFromHEVCParameterSets(
allocator: kCFAllocatorDefault,
parameterSetCount: sets.count,
parameterSetPointers: pBuf.baseAddress!,
parameterSetSizes: sBuf.baseAddress!,
nalUnitHeaderLength: 4, extensions: nil,
formatDescriptionOut: &fmt)
}
}
if status == noErr, let fmt {
formatDesc = fmt
if let session { VTDecompressionSessionInvalidate(session) }
session = nil
}
}
private func makeSession(_ fmt: CMVideoFormatDescription) {
let attrs: [CFString: Any] = [
kCVPixelBufferPixelFormatTypeKey:
kCVPixelFormatType_32BGRA,
]
VTDecompressionSessionCreate(
allocator: kCFAllocatorDefault, formatDescription: fmt,
decoderSpecification: nil,
imageBufferAttributes: attrs as CFDictionary,
outputCallback: nil, decompressionSessionOut: &session)
}
private static func blockBuffer(_ data: Data) -> CMBlockBuffer? {
var block: CMBlockBuffer?
guard CMBlockBufferCreateWithMemoryBlock(
allocator: kCFAllocatorDefault, memoryBlock: nil,
blockLength: data.count, blockAllocator: kCFAllocatorDefault,
customBlockSource: nil, offsetToData: 0,
dataLength: data.count, flags: 0,
blockBufferOut: &block) == noErr, let block else {
return nil
}
var ok = false
data.withUnsafeBytes { raw in
if CMBlockBufferReplaceDataBytes(
with: raw.baseAddress!, blockBuffer: block,
offsetIntoDestination: 0,
dataLength: data.count) == noErr { ok = true }
}
return ok ? block : nil
}
}
```
- [ ] **Step 2: Verify it compiles**
Run: `cd launcher/AV-Live-Body && swift build`
Expected: build succeeds. If a VideoToolbox/CoreMedia signature differs
on this SDK, fix minimally — the behavior (build a format description
from the prepended parameter sets, decode the rest) must be preserved.
- [ ] **Step 3: Commit**
```bash
git add launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift
git commit -m "feat(av-live-body): HEVC video decoder"
```
---
## Task 3: Render the 91-joint USB skeleton
`Skeleton3DRenderer` already subscribes to a 91-joint ARKit body
publisher into `lastArkit` but never draws it — `Skeleton3DRenderer.swift:138`
is `// TODO: render yellow ARKit markers from lastArkit in update()`.
Complete it: draw the 91 joints as small yellow spheres.
**Files:**
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift`
- [ ] **Step 1: Read the renderer**
Read `Skeleton3DRenderer.swift` fully. Note: `PersonEntities` (the
per-pid entity struct), `lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame]`,
`makePerson(pid:parent:)`, the `update(frames:)` 30 fps tick, and the
RealityKit space conversion used for MediaPipe joints
(`SIMD3(k.x, -k.y, -k.z)`).
- [ ] **Step 2: Add 91 ARKit marker entities to `PersonEntities`**
In the `PersonEntities` struct, add a field:
```swift
var arkitMarkers: [ModelEntity] // 91 yellow ARKit joint spheres
```
In `makePerson(pid:parent:)`, after the hand spheres are built, create
91 yellow marker spheres (reuse the `jointRadius`-sized sphere mesh, a
yellow `SimpleMaterial`), parent them to `root`, start them disabled,
and include `arkitMarkers:` in the returned `PersonEntities(...)`:
```swift
let arkitMat = SimpleMaterial(
color: .systemYellow, roughness: 0.6, isMetallic: false)
var arkitMarkers: [ModelEntity] = []
arkitMarkers.reserveCapacity(91)
for _ in 0..<91 {
let e = ModelEntity(mesh: sphereMesh, materials: [arkitMat])
e.isEnabled = false
root.addChild(e)
arkitMarkers.append(e)
}
```
- [ ] **Step 3: Draw the ARKit markers each tick**
Replace the line `// TODO: render yellow ARKit markers from lastArkit in update()`
(`Skeleton3DRenderer.swift:138`) — leave the comment removed — and add,
at the end of `update(frames:)` (after the existing per-pid loop), a
call to a new private method `applyArkit()`. Then add the method:
```swift
/// Draw the 91-joint ARKit/USB skeletons as yellow joint markers.
/// ARKit joints are world-space metric; convert to RealityKit
/// space (x, y, z) -> (x, -y, -z) like the MediaPipe path.
private func applyArkit() {
for (pid, entities) in persons {
guard let frame = lastArkit[pid] else {
for m in entities.arkitMarkers { m.isEnabled = false }
continue
}
let n = min(91, entities.arkitMarkers.count,
frame.joints.count)
for i in 0..<n {
let marker = entities.arkitMarkers[i]
if frame.hasJoint[i] {
let j = frame.joints[i]
marker.transform.translation =
SIMD3<Float>(j.x, -j.y, -j.z)
marker.isEnabled = true
} else {
marker.isEnabled = false
}
}
for i in n..<entities.arkitMarkers.count {
entities.arkitMarkers[i].isEnabled = false
}
}
}
```
Note: `applyArkit()` iterates `persons`, which is only populated for
pids seen in the MediaPipe `frames`. If the USB skeleton must show
when there is no MediaPipe pose, also create a `PersonEntities` for
each pid present in `lastArkit`. To keep Task 3 minimal, in
`update(frames:)` before `applyArkit()`, ensure entities exist for
ARKit-only pids:
```swift
for pid in lastArkit.keys where persons[pid] == nil {
persons[pid] = makePerson(pid: pid, parent: anchor)
lastSeenAt[pid] = now
}
```
- [ ] **Step 4: Verify build + tests**
Run: `cd launcher/AV-Live-Body && swift build` — Expected: succeeds.
Run: `cd launcher/AV-Live-Body && swift test` — Expected: all tests
still pass (no regression).
- [ ] **Step 5: Commit**
```bash
git add launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift
git commit -m "feat(av-live-body): render 91-joint USB skeleton"
```
---
## Task 4: Wire the consumer into the app
Construct `USBSkeletonConsumer` in the app, start/stop it with the
scene, and feed it into `Skeleton3DRenderer` in place of (or alongside)
`ArkitOSCListener`.
**Files:**
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift`
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift`
- [ ] **Step 1: Read the two files**
Read `AVLiveBodyApp.swift` and `BodyView.swift`. Identify: where the
`@StateObject` listeners are declared in `ContentView`, where `.onAppear`
starts them, how `ArkitOSCListener` is passed into `BodyView`, and where
`BodyView.makeNSView` calls `skel3d.attach(to:listener:arkitListener:)`.
- [ ] **Step 2: Own and start the consumer**
In `AVLiveBodyApp.swift`'s `ContentView`, add a `@StateObject`:
```swift
@StateObject private var usbConsumer = USBSkeletonConsumer()
```
In `.onAppear`, alongside the existing listener `.start()` calls, add
`usbConsumer.start()`. If there is an `.onDisappear`, add
`usbConsumer.stop()`.
- [ ] **Step 3: Thread the consumer to the renderer**
`Skeleton3DRenderer.attach` currently takes
`arkitListener: ArkitOSCListener?`. The simplest correct change: give
`USBSkeletonConsumer` the same role. Add an overload / extra parameter
so `attach` can subscribe to `usbConsumer.$bodies` exactly as it
subscribes to `arkitListener.$bodies` (both publish
`[Int: ArkitOSCListener.ArkitBodyFrame]`). Concretely, in
`Skeleton3DRenderer.attach`, accept `usbConsumer: USBSkeletonConsumer?`
and, if non-nil, subscribe its `$bodies` into `lastArkit` with the same
sink already used for `arkitListener` (the `arkitSub` Combine
subscription). Pass `usbConsumer` from `ContentView``BodyView`
`makeNSView``skel3d.attach(...)`, mirroring how `arkitListener` is
already threaded.
If `arkitListener` (the OSC one) is now redundant, it may be passed as
`nil`; do not delete `ArkitOSCListener` in this plan (other code or
Plan 3b cleanup may still reference it).
- [ ] **Step 4: Verify build**
Run: `cd launcher/AV-Live-Body && swift build` — Expected: succeeds.
Run: `cd launcher/AV-Live-Body && swift test` — Expected: no regression.
- [ ] **Step 5: Commit**
```bash
git add launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift
git commit -m "feat(av-live-body): wire USB consumer to renderer"
```
---
## Task 5: Final verification
- [ ] **Step 1: Clean build + full test suite**
```bash
cd launcher/AV-Live-Body && swift build && swift test
```
Expected: build succeeds; all tests pass (7: prior 6 + Task 1's).
- [ ] **Step 2: Confirm the integration seam**
`USBSkeletonConsumer.devicePort` (7000) must equal the iOS app's
`USBServer.port`. Verify:
```bash
grep -rn "port.*7000\|devicePort" \
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift \
iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/USBServer.swift
```
Expected: both sides use `7000`.
- [ ] **Step 3: Commit any fix** (only if Step 2 found a mismatch).
---
## Self-Review
- **Spec coverage:** This plan implements the spec's `USBClient`
consumption inside `AVLiveBody`, the `VideoDecoder` unit, and the
skeleton render path. `MultiHMRCoreML`, `BodyFusion`, and dense-mesh
rendering are explicitly Plan 3b (gated on a confirmed CoreML
Multi-HMR `.mlpackage`).
- **Placeholders:** none — new files have complete code; modify tasks
cite exact files and the line-138 TODO, and instruct the implementer
to read exact context for `AVLiveBodyApp.swift`/`BodyView.swift`
(whose current line numbers are not reproduced here).
- **Type consistency:** `USBSkeletonConsumer.bodyFrame` returns
`ArkitOSCListener.ArkitBodyFrame`; `Skeleton3DRenderer` already
stores `lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame]`, so the
consumer is type-compatible with the existing `arkitSub` path.
`VideoDecoder` consumes `VideoPayload` exactly as Plan 2's
`VideoEncoder` produces it (parameter sets prepended, 4-byte
big-endian length prefixes).
- **Known risks:** (1) `BodyView` owns `Skeleton3DRenderer`, so Task 4
threads a new object through `ContentView``BodyView``attach`
multi-file, follow the existing `arkitListener` threading exactly.
(2) `StreamDemuxer.findMagic` copies the whole buffer per `feed()`;
for HEVC video this is a perf risk — acceptable for Plan 3a, revisit
if frame rate suffers. (3) The HEVC parameter-set split in
`VideoDecoder` assumes the iOS encoder's exact prepend layout —
this is the Plan 2 ↔ Plan 3a integration seam; validate on real
device data.