docs: iPhone capture plan (2 of 3)

Plan for HEVC video capture (VideoToolbox) over the USB transport and removal of the legacy OSC sender. Skeleton USB path already exists; this adds the video half.
2026-05-18 17:23:38 +02:00
parent 5e58c0c4fe
commit 39457dde41
2 changed files with 524 additions and 0 deletions
@@ -0,0 +1,430 @@
+# iPhone Capture Implementation Plan (Plan 2 of 3)
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Make the iOS `ARBodyTracker` app stream the camera RGB video (HEVC) over the USB transport alongside the ARKit skeleton, and retire the legacy OSC/UDP sender — so the iPhone is a self-contained, network-free capture source.
+
+**Architecture:** `ARBodySession` already captures the ARKit 91-joint skeleton and sends it as `AVLiveWire` `.skeleton` frames through `USBServer` (built in Plan 1). This plan adds a `VideoEncoder` (VideoToolbox hardware HEVC) that encodes each `ARFrame.capturedImage` and sends it as `.video` frames through the same `USBServer`. The OSC/UDP fanout (`/body3d/kp` to `host:57128/57129`) and its `ContentView` config fields are removed.
+
+**Tech Stack:** Swift 5.10, ARKit, VideoToolbox, CoreMedia, `AVLiveWire` (local package), iOS 17. Build verification via `xcodebuild`.
+
+**Companion spec:** `docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md`
+**Prerequisite:** Plan 1 (`docs/superpowers/plans/2026-05-18-iphone-usb-transport.md`) — merged.
+
+---
+
+## Verification note
+
+The iOS app is an iOS-only target; it cannot be built with `swift build`
+on a macOS host. The verification command for every task is:
+
+```bash
+cd iphone-arbody && xcodegen generate && \
+xcodebuild -project ARBodyTracker.xcodeproj -scheme ARBodyTracker \
+  -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' \
+  -configuration Debug build
+```
+
+Expected: `** BUILD SUCCEEDED **`. VideoToolbox HEVC encoding and ARKit
+body tracking only run fully on a physical device — runtime behavior is
+an owner on-device check, out of this plan's automated scope.
+
+---
+
+## File Structure
+
+| File | Responsibility |
+|------|----------------|
+| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/VideoEncoder.swift` | NEW. VideoToolbox HEVC hardware encoder: `CVPixelBuffer` → `VideoPayload` via callback |
+| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift` | MODIFY. Add video encoding in `session(_:didUpdate:)`; remove OSC fanout |
+| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift` | MODIFY. Remove OSC host/port config UI |
+
+---
+
+## Task 1: VideoEncoder
+
+`VideoEncoder` wraps a `VTCompressionSession` configured for HEVC. It
+accepts `CVPixelBuffer`s and invokes `onPayload` with a `VideoPayload`
+(keyframe flag + the access-unit bytes; for keyframes the HEVC
+parameter sets are prepended so the Mac decoder is self-sufficient).
+
+**Files:**
+- Create: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/VideoEncoder.swift`
+
+- [ ] **Step 1: Create the file**
+
+```swift
+import AVLiveWire
+import CoreMedia
+import CoreVideo
+import Foundation
+import VideoToolbox
+
+/// Hardware HEVC encoder. Feed `CVPixelBuffer`s from ARKit frames in;
+/// receive one `VideoPayload` per encoded access unit via `onPayload`.
+/// Keyframe payloads carry the VPS/SPS/PPS parameter sets prepended,
+/// each as a 4-byte-length-prefixed NAL unit, so the Mac decoder can
+/// build its format description without a side channel.
+final class VideoEncoder {
+    var onPayload: ((VideoPayload) -> Void)?
+
+    private var session: VTCompressionSession?
+    private let lock = NSLock()
+
+    /// Create the compression session for a given frame size.
+    func start(width: Int32, height: Int32) {
+        stop()
+        var s: VTCompressionSession?
+        let status = VTCompressionSessionCreate(
+            allocator: kCFAllocatorDefault,
+            width: width, height: height,
+            codecType: kCMVideoCodecType_HEVC,
+            encoderSpecification: nil,
+            imageBufferAttributes: nil,
+            compressedDataAllocator: nil,
+            outputCallback: nil,
+            refcon: nil,
+            compressionSessionOut: &s)
+        guard status == noErr, let s else {
+            NSLog("VideoEncoder: VTCompressionSessionCreate failed %d",
+                  status)
+            return
+        }
+        VTSessionSetProperty(s, key: kVTCompressionPropertyKey_RealTime,
+                             value: kCFBooleanTrue)
+        VTSessionSetProperty(s,
+            key: kVTCompressionPropertyKey_AllowFrameReordering,
+            value: kCFBooleanFalse)
+        VTSessionSetProperty(s,
+            key: kVTCompressionPropertyKey_MaxKeyFrameInterval,
+            value: 30 as CFNumber)
+        VTCompressionSessionPrepareToEncodeFrames(s)
+        session = s
+    }
+
+    /// Encode one frame. `pts` is the capture timestamp in seconds.
+    func encode(_ pixelBuffer: CVPixelBuffer, pts: Double) {
+        lock.lock(); let s = session; lock.unlock()
+        guard let s else { return }
+        let time = CMTime(seconds: pts, preferredTimescale: 1_000_000)
+        VTCompressionSessionEncodeFrame(
+            s, imageBuffer: pixelBuffer, presentationTimeStamp: time,
+            duration: .invalid, frameProperties: nil,
+            infoFlagsOut: nil) { [weak self] status, _, sample in
+                guard status == noErr, let sample else { return }
+                self?.handle(sample)
+            }
+    }
+
+    func stop() {
+        lock.lock(); let s = session; session = nil; lock.unlock()
+        if let s {
+            VTCompressionSessionInvalidate(s)
+        }
+    }
+
+    deinit { stop() }
+
+    // MARK: - Sample → VideoPayload
+
+    private func handle(_ sample: CMSampleBuffer) {
+        let isKeyframe = !Self.notSync(sample)
+        var out = Data()
+        if isKeyframe, let fmt = CMSampleBufferGetFormatDescription(sample) {
+            out.append(Self.parameterSets(fmt))
+        }
+        if let block = CMSampleBufferGetDataBuffer(sample) {
+            var lengthOut = 0
+            var ptr: UnsafeMutablePointer<Int8>?
+            if CMBlockBufferGetDataPointer(
+                block, atOffset: 0, lengthAtOffsetOut: nil,
+                totalLengthOut: &lengthOut,
+                dataPointerOut: &ptr) == noErr, let ptr {
+                out.append(UnsafeBufferPointer(
+                    start: UnsafeRawPointer(ptr)
+                        .assumingMemoryBound(to: UInt8.self),
+                    count: lengthOut))
+            }
+        }
+        guard !out.isEmpty else { return }
+        onPayload?(VideoPayload(isKeyframe: isKeyframe, data: out))
+    }
+
+    /// True if the sample is NOT a sync (key) frame.
+    private static func notSync(_ sample: CMSampleBuffer) -> Bool {
+        guard let arr = CMSampleBufferGetSampleAttachmentsArray(
+            sample, createIfNecessary: false),
+            CFArrayGetCount(arr) > 0 else { return false }
+        let dict = unsafeBitCast(CFArrayGetValueAtIndex(arr, 0),
+                                 to: CFDictionary.self)
+        let key = Unmanaged.passUnretained(
+            kCMSampleAttachmentKey_NotSync).toOpaque()
+        return CFDictionaryContainsKey(dict, key)
+    }
+
+    /// Concatenate the HEVC VPS/SPS/PPS parameter sets, each as a
+    /// 4-byte big-endian length prefix followed by the NAL bytes.
+    private static func parameterSets(
+        _ fmt: CMFormatDescription) -> Data {
+        var count = 0
+        CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(
+            fmt, parameterSetIndex: 0, parameterSetPointerOut: nil,
+            parameterSetSizeOut: nil, parameterSetCountOut: &count,
+            nalUnitHeaderLengthOut: nil)
+        var data = Data()
+        for i in 0..<count {
+            var ptr: UnsafePointer<UInt8>?
+            var size = 0
+            guard CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(
+                fmt, parameterSetIndex: i,
+                parameterSetPointerOut: &ptr,
+                parameterSetSizeOut: &size,
+                parameterSetCountOut: nil,
+                nalUnitHeaderLengthOut: nil) == noErr,
+                let ptr else { continue }
+            var be = UInt32(size).bigEndian
+            withUnsafeBytes(of: &be) { data.append(contentsOf: $0) }
+            data.append(UnsafeBufferPointer(start: ptr, count: size))
+        }
+        return data
+    }
+}
+```
+
+- [ ] **Step 2: Verify it compiles**
+
+Run the verification command from the "Verification note" section above.
+Expected: `** BUILD SUCCEEDED **` (the new file compiles within the
+target).
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/VideoEncoder.swift
+git commit -m "feat(ios): VideoToolbox HEVC encoder"
+```
+
+(subject ≤50 chars; add a short body — the commit hook rejects
+subject-only messages; no AI attribution.)
+
+---
+
+## Task 2: Stream video from ARBodySession
+
+Wire `VideoEncoder` into the ARKit frame loop. On each `didUpdate`
+frame already processed for skeletons, also encode `capturedImage` and
+send the resulting `VideoPayload` over the existing `USBServer`.
+
+**Files:**
+- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift`
+
+- [ ] **Step 1: Add the encoder property and its payload wiring**
+
+In `ARBodySession`, next to `private let usb = USBServer()` (currently
+line 45), add:
+
+```swift
+    private let videoEncoder = VideoEncoder()
+    private var videoStarted = false
+```
+
+In `init()`, after the `usb.onState = { ... }` block, add the encoder
+output wiring:
+
+```swift
+        videoEncoder.onPayload = { [weak self] payload in
+            Task { @MainActor in
+                guard let self, self.usbState == .connected else {
+                    return
+                }
+                self.usb.send(tag: .video, pid: -1,
+                              timestamp: self.lastFrameTime,
+                              payload: payload.encoded())
+            }
+        }
+```
+
+- [ ] **Step 2: Encode the captured image in the frame loop**
+
+In `session(_:didUpdate:)`, inside the `Task { @MainActor in ... }`
+block, after `self.lastFrameTime = t` and before the anchor loop,
+add video encoding:
+
+```swift
+            // Start the encoder lazily once the first frame size is
+            // known, then encode every (throttled) frame.
+            let img = frame.capturedImage
+            let w = Int32(CVPixelBufferGetWidth(img))
+            let h = Int32(CVPixelBufferGetHeight(img))
+            if !self.videoStarted, w > 0, h > 0 {
+                self.videoEncoder.start(width: w, height: h)
+                self.videoStarted = true
+            }
+            if self.videoStarted {
+                self.videoEncoder.encode(img, pts: t)
+            }
+```
+
+- [ ] **Step 3: Stop the encoder on stop()**
+
+In `stop()`, after `usb.stop()`, add:
+
+```swift
+        videoEncoder.stop()
+        videoStarted = false
+```
+
+- [ ] **Step 4: Verify it compiles**
+
+Run the verification command. Expected: `** BUILD SUCCEEDED **`.
+
+- [ ] **Step 5: Commit**
+
+```bash
+git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift
+git commit -m "feat(ios): stream HEVC video over USB"
+```
+
+---
+
+## Task 3: Remove the legacy OSC sender
+
+The OSC/UDP fanout is the network dependency the autonomous USB design
+removes. Delete it from `ARBodySession`.
+
+**Files:**
+- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift`
+
+- [ ] **Step 1: Delete OSC members and methods**
+
+In `ARBodySession.swift`, delete:
+- the stored properties `host`, `pythonPort`, `swiftPort` (currently
+  lines 39-41) and `conns` (line 44);
+- the `configure(host:pythonPort:swiftPort:sendEnvMesh:)` method —
+  replace it with a parameterless `configure(sendEnvMesh:)`:
+  ```swift
+  func configure(sendEnvMesh: Bool) {
+      self.sendEnvMesh = sendEnvMesh
+  }
+  ```
+- the call `openUDP()` in `start()`;
+- in `stop()`, the lines `for c in conns { c.cancel() }` and
+  `conns.removeAll()`;
+- the entire `// MARK: - UDP fanout` section: `openUDP()` and
+  `sendDatagram(_:)`;
+- the `publishJoints(pid:body:)` method and its call site in
+  `session(_:didUpdate:)` (`self.publishJoints(pid: count, body: body)`);
+- the `sendOSC(addr:args:)` call for `/body3d/count` in
+  `session(_:didUpdate:)`;
+- the `// MARK: - OSC minimal encoder` section: the `OSCArg` enum,
+  `sendOSC(addr:args:)`, and `appendOSCString(_:into:)`.
+
+After deletion, `Network` is still needed (`USBServer` uses it
+indirectly — actually `USBServer` imports its own `Network`). Remove
+`import Network` from `ARBodySession.swift` only if no symbol from it
+remains; if `NWConnection`/`NWEndpoint` no longer appear in the file,
+remove the import.
+
+- [ ] **Step 2: Verify it compiles**
+
+Run the verification command. Expected: `** BUILD SUCCEEDED **`. If the
+build reports an unused `import` or an unresolved symbol, fix it
+minimally (remove the dead import, or keep it if still referenced).
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift
+git commit -m "refactor(ios): drop legacy OSC sender"
+```
+
+---
+
+## Task 4: Simplify ContentView
+
+`ContentView` exposes OSC host/port text fields that no longer have a
+backing. Remove them; keep the USB status indicator and Start/Stop.
+
+**Files:**
+- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift`
+
+- [ ] **Step 1: Remove OSC state and UI**
+
+In `ContentView`:
+- delete the `@State` properties `host`, `pythonPort`, `swiftPort`
+  (currently lines 7-9);
+- in `controlPanel`, delete the `HStack { Text("Host") ... }` block and
+  the `HStack { Text("Py") ... Text("Swift") ... }` block (the two
+  rows of OSC text fields, currently lines 85-102);
+- in the Start/Stop button action, replace the `session.configure(
+  host:pythonPort:swiftPort:sendEnvMesh:)` call with
+  `session.configure(sendEnvMesh: sendEnvMesh)`.
+
+Keep: `sendEnvMesh` toggle, Start/Stop button, status text, the USB
+status dot/label, and the bodies/frames/jointsPerSec line.
+
+- [ ] **Step 2: Verify it compiles**
+
+Run the verification command. Expected: `** BUILD SUCCEEDED **`. The
+three `#Preview` blocks at the end of the file construct
+`ContentView(useMockBackground:useMockSkeleton:)` — those parameters
+are unaffected; the previews must still compile.
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift
+git commit -m "refactor(ios): drop OSC config from ContentView"
+```
+
+---
+
+## Task 5: Final build verification
+
+- [ ] **Step 1: Full clean build**
+
+```bash
+cd iphone-arbody && xcodegen generate && \
+xcodebuild -project ARBodyTracker.xcodeproj -scheme ARBodyTracker \
+  -sdk iphonesimulator -destination 'generic/platform=iOS Simulator' \
+  -configuration Debug clean build
+```
+
+Expected: `** BUILD SUCCEEDED **`, zero errors.
+
+- [ ] **Step 2: Confirm no OSC references remain**
+
+```bash
+grep -rn -E "OSC|57128|57129|openUDP|sendDatagram" \
+  iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/
+```
+
+Expected: no matches in `ARBodySession.swift` or `ContentView.swift`.
+(`USBServer.swift` and `VideoEncoder.swift` never had OSC.) Comments
+mentioning history are acceptable; live OSC code is not.
+
+- [ ] **Step 3: Commit any cleanup** (only if Step 2 found stragglers)
+
+---
+
+## Self-Review
+
+- **Spec coverage:** This plan implements the spec's `VideoEncoder`
+  unit and the `ARBodySession` "exposes video frames / OSC sender
+  removed" requirement. `ARBodySession` already builds and sends
+  `SkeletonPayload` over `USBServer` (delivered via the recovery
+  branch + Plan 1), so no skeleton-path task is needed. `ContentView`
+  simplification follows from OSC removal.
+- **Placeholders:** none — every step has concrete code or an exact
+  command and expected output.
+- **Type consistency:** `VideoPayload`, `FrameTag.video`,
+  `USBServer.send(tag:pid:timestamp:payload:)` are used consistently
+  with their Plan 1 / `AVLiveWire` definitions. `VideoEncoder.start`
+  takes `Int32` width/height matching `CVPixelBufferGetWidth`'s `Int`
+  cast to `Int32`.
+- **Known risk:** the `VideoEncoder` VideoToolbox code compiles on the
+  simulator but HEVC hardware encoding and the exact access-unit /
+  parameter-set byte layout can only be validated on a physical
+  device. Plan 3's `VideoDecoder` must agree with the framing chosen
+  here (length-prefixed parameter sets prepended to keyframe payloads);
+  this is the integration seam to verify when Plan 3 is built.
@@ -0,0 +1,94 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+iPhone-only app that streams ARKit `ARBodyAnchor` joints (91 per body) to
+the AV-Live stack on a tethered Mac. Part of the parent `AV-Live/`
+monorepo — read `../CLAUDE.md` for global conventions (commit style,
+French/English split, no emoji, etc.).
+
+## Two build forms, one source tree
+
+Both flavours compile the same files under
+`ARBodyTracker.swiftpm/Sources/ARBodyTracker/`:
+
+| Form | When to use | How |
+|------|-------------|-----|
+| `ARBodyTracker.swiftpm/` (Swift Package) | Quick local iteration; previews | open `Package.swift` in Xcode |
+| `ARBodyTracker.xcodeproj` (generated) | Device deploy with reproducible signing | `xcodegen generate` from `project.yml` |
+
+The `.xcodeproj` is **gitignored** — regenerate after every change to
+`project.yml` or after adding sources. `Config/Local.xcconfig` (also
+gitignored) carries `DEVELOPMENT_TEAM`; copy from
+`Config/Local.xcconfig.example` on a fresh clone.
+
+## Common commands
+
+```bash
+brew install xcodegen                       # one-time
+cp Config/Local.xcconfig.example Config/Local.xcconfig  # set DEVELOPMENT_TEAM
+xcodegen generate                           # writes ARBodyTracker.xcodeproj
+open ARBodyTracker.xcodeproj                # then ⌘R on a connected iPhone
+
+# AVLiveWire — shared wire-format package (consumed by both the iOS
+# app and launcher/AV-Live-Body on macOS). Tests live with it.
+cd ../shared/AVLiveWire && swift test
+cd ../shared/AVLiveWire && swift test --filter LoopbackTests
+```
+
+`xcodegen generate` is the one command to remember: any time `project.yml`
+or the dep on `../shared/AVLiveWire` changes, the generated project must
+be rebuilt before opening Xcode.
+
+## Architecture
+
+```
+ARBodyTrackerApp ── ContentView ── ARViewContainer (ARView)
+                       │            │
+                       │            └─ ARBodySession (ARSessionDelegate)
+                       │                  ├─ OSC fanout (UDP) ─► host:57128 (Python data_only_viz)
+                       │                  │                      host:57129 (Swift AV-Live-Body)
+                       │                  ├─ USBServer (TCP :7000, AVLiveWire frames)
+                       │                  └─ @Published skeleton2D ──┐
+                       └─ SkeletonOverlay (SwiftUI Canvas) ◄─────────┘
+```
+
+- **`ARBodySession`** owns the `ARSession` configured with
+  `ARBodyTrackingConfiguration` (RGB-only; LiDAR `sceneDepth` is not
+  available on this config, do not re-add it — it crashes the session).
+  Throttles to 30 fps. Projects joints to 2D via
+  `ARCamera.projectPoint(_:orientation:viewportSize:)`; the viewport
+  size is fed back from the SwiftUI `GeometryReader`.
+- **Transport** is dual: legacy OSC/UDP (still default), and a new
+  USB/TCP path using `AVLiveWire` framed messages. The Mac side bridges
+  via `usbmuxd`; the iOS side just listens on a fixed local port. No
+  WiFi involved on the USB path.
+- **`AVLiveWire`** (in `../shared/AVLiveWire`) defines a fixed 19-byte
+  big-endian header (`AVL1` magic + tag + pid + timestamp + length) and
+  an incremental `StreamDemuxer`. Both the iOS `USBServer` and the
+  macOS consumer link against it so the wire format lives in exactly
+  one place.
+- **`SkeletonOverlay`** draws joints and bones from a published
+  `SkeletonSnapshot`; bone parent indices come from
+  `ARSkeletonDefinition.defaultBody3D.parentIndices` at runtime. For
+  Xcode previews the snapshot/parents are replaced by a 16-joint
+  stick-figure mock (ARKit's `neutralBodySkeleton3D` returns nil off
+  device).
+
+## Gotchas
+
+- **Frame semantics:** `ARBodyTrackingConfiguration` rejects
+  `.sceneDepth` (world-tracking only) and spams `ABPKPersonIDTracker:
+  Portrait image is not supported` per-frame if
+  `.personSegmentationWithDepth` is added. Keep `feats` empty unless a
+  newly-needed semantic is verified against
+  `ARBodyTrackingConfiguration.supportsFrameSemantics(_:)`.
+- **Signing without an Apple ID logged into Xcode:** with
+  `CODE_SIGN_STYLE = Automatic`, Xcode needs the account in
+  Settings → Accounts to download a profile. If only the keychain
+  cert is present, the build fails with "No Account for Team …".
+  Reconnect the account, or fall back to a manually installed profile
+  via `CODE_SIGN_STYLE = Manual` + `PROVISIONING_PROFILE_SPECIFIER`.
+- **`UIDeviceFamily` warning:** `Info.plist` and `TARGETED_DEVICE_FAMILY`
+  duplicate the device family. The build setting wins; the Info.plist
+  key only generates a warning and can be removed if desired.