Hybrid architecture: Python detects finger strikes from
iPhone hand joints, SC owns musicality (scale, octave,
instrument) and continuous modulation from hand spacing.
Extensible ~ccGesture registry (declarative gesture->event, engine
evaluates with hold/cooldown) so triggers are unlimited. Phase 1:
skeleton data layer (/pose/skel -> ~poseSkel), the framework, a master
event-FX layer (master filter + stutter + event synths), and a first
batch of FX/dramatic gestures (jump/stomp/squat/spin/tpose). Harmonic/
melodic triggers + ~ccHarmony are Phase 2 on the same framework.
Engine first (Task 1: ~concert + ~poseCenter + Hypno Drift + integration),
then one morceau per task (8 self-contained .load files registering via
~concertAdd with their own \cc_* SynthDefs), then a live smoke. Complete
working SuperCollider per task; headless sclang load/advance tests on
GrosMac, audio tuning at the macm1 smoke.
Body-driven techno concert: a sequential setlist of 8 custom-mapped
morceaux (hard/dub/acid/hypnotic) advanced by an arms-crossed gesture.
Concert engine (~concert) + uniform morceau module interface
(start/update/tick/stop + ctx) + crossfade + /pose/center wiring. New
\concert scene alongside body-play. Plan decomposes: engine first, then
morceaux in batches.
Three-task plan: IphoneUSBSource with HEVC video decode (cv2 .read
contract, reusing the de-risked decode), ARKit skeleton + Vision hands
into State, and --iphone-usb wiring in multi.py/main.py. Tests run on
macm1 against the live iPhone; plus a manual live smoke.
Drive data_only_viz entirely from the iPhone over USB: HEVC video ->
PyAV decode -> MediaPipe, ARKit skeleton -> arkit_fuse, Vision hands ->
hand features. IphoneUSBSource implements the cv2 .read() contract so it
drops into multi.py in place of cv2.VideoCapture; --iphone-usb flag.
HEVC decode de-risked (256 frames at ~28fps from real iPhone AUs).
Five-task TDD plan (headless sclang tests) for the body-play module:
bp_* SynthDef palette, modes table + scale quantizer, pure pose->music
mapping functions, the engine (clock/arp/lead/rhythm/crossfade), and
boot wiring with a live smoke step.
Design for body-driven musical generation in the data-only patch:
action-head label selects one of three complete modes (scale+tempo+
instruments), the right hand plays a gesture-triggered lead while the
left hand holds a chord an arpeggio plays in rhythm, and body speed
drives the tempo. New scene_pose_play.scd + synthdefs_play.scd
(bp_* palette), separate from the existing pose->FX module.
Phase 0: capture hands (Vision hand pose) + face (Vision face landmarks) on the iPhone from AR rear-camera frames, stream over new AVLiveWire tags 3/4. Phase 1: port settings panel + 3 lights + mesh material into avlivebody-mac. Phase 2: 10 Metal shader modes + 3D hand/face skeleton, uniforms fed from the iPhone stream.
Vertical slice: MediaPipe hand features drive both the data_only FX rack (SuperCollider) and the Metal scene via one shared signal. Modulation only, no note generation. Includes reliability fixes (boot loads scene_pose_action, FX stubs wired, finite guards).
* docs(plans): action-head v3 + branch sync notes
Update plan header :
- v2 (Task 18) + v3 (Task 19) extensions chronology
- Studio train validated, ckpt action_head_v3.pt landed
- Mesh NaN-guard debug trail (commit 4e7101c)
- Branch convergence main == feat/action-head
- Pointers to memories project_action_head_v3, etc.
* feat(av-live): openpos 3D + DINO reid + filter
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
* fix(av-live-body): restore face+hand+3D (f540158)
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
* feat(av-live): wireframe skel + face/hand filter
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
* deps(icp): add open3d optional extra + smoke test
Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.
Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.
Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
(scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
(PointCloud roundtrip + ICP convergence on translated copy),
gated by `pytest.importorskip("open3d")`
Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.
* feat(icp): LiDAR TCP frame decoder + tests
* feat(icp): LiDAR TCP socket reader with reconnect
* feat(icp): extrinsic dataclass + JSON persistence
* feat(icp): Kabsch + calibration CLI scaffold
* feat(state): persons_arkit_joints fields
* feat(viz): ARKit 91 -> MP 33 joint map
* feat(viz): iphone OSC listener :57128
* feat(viz): arkit_fuse stage overrides 14 slots
* feat(viz): arkit pelvis z locks cam translation
* feat(viz): iphone OSC listener auto-start
* docs: arkit fusion env vars
* feat(icp): point-to-plane register + reject gate
* feat(icp): partition LiDAR per pid by max-dist
* feat(icp): FusionWorker + State.lidar_points
* feat(icp): wire fusion thread behind ICP_FUSION
Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.
A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.
Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.
MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.
* test(icp): synthetic latency + convergence bench
* docs(icp): env vars + calibration procedure
* docs(plans): icp lidar mesh + arkit joints
Two complementary fusion plans landed in parallel on 2026-05-14:
- iphone-lidar-multihmr-fusion : ARKit 91 joints -> MP33 fuse stage +
pelvis z override (already implemented in 7 commits)
- icp-lidar-smplx-fusion : LiDAR mesh point-to-plane ICP onto SMPL-X
10475 verts (12 tasks executed via subagent-driven-development)
Both paths coexist; joints are sparse+fast (60 Hz), mesh is dense+slow
(5-10 Hz). See docs/ICP_FUSION.md for the integration topology.
* feat(icp): predict_once via CoreML backend
* feat(av-live-body): wire ArkitOSCListener :57129
Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.
* feat(ios): iphone ARBodyTracker swiftpm app
iOS 17+ Swift Package app (.swiftpm) streaming ARKit body
joints via OSC UDP to two destinations:
:57128 -> data_only_viz/iphone_osc_listener.py
:57129 -> launcher/AV-Live-Body ArkitOSCListener.swift
Features:
- ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported
- 91 joints per body, /body3d/kp pid joint_idx x y z
- 30 fps throttle
- SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second
- Inline OSC encoder (no external dep)
Env mesh (TCP :5500) NOT yet implemented; requires a separate
ARWorldTrackingConfiguration session. ICP fusion path runs on
bench data only until phase 2.
* feat(data-feeds): 10 open-data OSC publisher
* feat(viz): DataFeeds OSC listener + HUD
* chore: gitignore tweaks
* docs: network topology + mDNS hostnames
Add a "Network topology" section to top-level CLAUDE.md doc
the 3-host layout (GrosMac source, Supra-M1 sink via mDNS,
iPhone via Personal Hotspot DHCP).
mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST
accept hostname.local instead of IPs, so the cluster survives
DHCP rotations on iPhone hotspot (172.20.10.x).
* fix(ios): add NSLocalNetworkUsageDescription
iOS 14+ silently blocks UDP to LAN addresses without this key.
The first time the app tries to send to 192.168.0.159, iOS will
prompt the user to allow Local Network Access; the prompt must
be accepted or the OSC stream never reaches the Mac.
Also adds NSBonjourServices declaring _osc._udp so the system
treats the connection as a recognised service.
* docs: network topology + gitignore hygiene
- CLAUDE.md: add mDNS hostname table (grosmac.local, supra-m1.local,
iPhone hotspot 172.20.10.x). AVBODY_HOST / MULTIHMR_REMOTE_HOST
accept hostnames — resilient to DHCP rotation.
- .gitignore: exclude .remember/ tool state and iCloud '* 2'
collision artifacts.
* feat(ios): ARBody skeleton2D + overlay preview
ARBodySession: publish 2D-projected skeleton snapshot for live
overlay rendering on the iPhone screen alongside the camera feed.
ContentView: SkeletonOverlay drawing on top of the AR view, with
mock T-pose for Xcode previews (useMockBackground, useMockSkeleton).
* docs: iPhone USB body-tracking link design
Brainstormed design for replacing the OSC/network iPhone-Mac
link with a wired USB transport via usbmuxd. iPhone streams
ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML
and renders the mesh. Network-free, single native macOS app.
* docs: iPhone USB transport plan (1 of 3)
Bite-sized TDD plan for the network-free USB byte-pipe:
shared AVLiveWire frame format, native usbmux client,
iOS TCP frame server, incremental stream demuxer.
* feat(avlivewire): shared wire package skeleton
* feat(avlivewire): fixed 19-byte frame header codec
Add FrameHeader, a fixed-size binary record so the demuxer can
frame and resync the iPhone USB stream. Layout is big-endian:
4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32.
The magic prefix lets a reader detect and skip corrupt bytes.
Decoding rejects short buffers and bad magic by returning nil.
Big-endian append/parse helpers are added as Data/UInt extensions
to keep the codec self-contained.
* chore: ignore SwiftPM .build artifacts
Both AVLiveWire and AV-Live-Body produce .build/ on swift
test; ignore them so they never get accidentally staged.
* feat(avlivewire): skeleton and video codecs
Add SkeletonPayload (91 ARKit joints + per-joint validity) and
VideoPayload (one HEVC access unit + keyframe flag) with
big-endian encode/decode. Reuses Task 2 Data/UInt32 helpers.
* feat(avlivewire): incremental stream demuxer
Add StreamDemuxer that accepts arbitrary byte chunks from a
non-frame-aligned stream and emits complete (FrameHeader, Data)
frames, resyncing on the magic prefix after corruption.
* fix(avlivewire): cap demuxer payload length
A corrupt header with a huge UInt32 length made feed buffer
forever waiting for bytes that never arrive. Add an 8 MB max
payload cap; a header exceeding it is treated as corrupt, its
magic is skipped, and the demuxer resyncs on the next frame.
* feat(av-live-body): usbmux message codec
Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.
Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.
* feat(av-live-body): usbmux device client
Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.
Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.
* feat(av-live-body): usbmuxd unix socket transport
Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.
* fix(av-live-body): harden unix socket transport
Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.
* feat(ios): USB TCP frame server
Add USBServer: an NWListener on a fixed local TCP port that
usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames
and exposes a connection-state callback.
* build: depend on shared AVLiveWire package
Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.
* build(ios): add AVLiveWire package to xcodegen
The xcodegen project did not declare the shared AVLiveWire
package, so USBServer.swift would fail to import it in the
generated Xcode project. Add it as a local package dep.
* test(avlivewire): end-to-end chunked loopback
Feeds 20 framed skeleton payloads through StreamDemuxer in
7-byte chunks (worst-case TCP fragmentation). Fixed a split
range operator from the plan that did not parse.
* fix(ios): guard USBServer listener and payload
Report .idle (not .listening) when NWListener creation fails,
and drop payloads larger than the demuxer's 8 MB cap so the
receiver never silently discards an oversized frame.
* chore: ignore .swiftpm editor state dirs
swift test / Xcode create hidden .swiftpm dirs inside
packages; ignore them so they never get staged.
* docs: iPhone capture plan (2 of 3)
Plan for HEVC video capture (VideoToolbox) over the USB
transport and removal of the legacy OSC sender. Skeleton
USB path already exists; this adds the video half.
* feat(ios): HEVC video capture, drop OSC
Add VideoEncoder (VideoToolbox HEVC) and stream encoded
frames over USB as .video AVLiveWire frames alongside the
skeleton. Remove the legacy OSC/UDP fanout and its host/port
config UI — the iPhone link is now USB-only.
* docs(ios): refresh stale OSC references
ARBodySession header comment and Info.plist usage strings
still described the removed OSC/UDP path; update them to the
USB transport and drop the dead _osc._udp Bonjour service.
* docs: macOS USB consumer plan (3a of 3)
Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.
* feat(av-live-body): USB skeleton consumer
Background usbmux read loop feeding StreamDemuxer; republishes
.skeleton frames as 91-joint ArkitBodyFrames and forwards
.video payloads. Removed stale iCloud collision duplicate
source files that broke the build.
* fix(data-only): CoreML Multi-HMR usage bugs
The CoreML Multi-HMR model was fine; the "0 detections" bug
was caller-side. Add ImageNet normalization in infer() (the
DINOv2 backbone needs it; raw [0,1] input collapsed all
scores) and update stale hardcoded output var names to match
the re-converted mlpackage. Bump the latency test threshold
to the realistic ~140 ms full-model figure.
* feat(av-live-body): HEVC video decoder
VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.
* feat(av-live-body): render 91-joint USB skeleton
Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.
* feat(av-live-body): wire USB consumer to renderer
ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.
* docs: macOS Multi-HMR mesh plan (3b of 3)
Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML
Swift wrapper, BodyFusion (ARKit depth correction), mesh
pipeline wiring. Completes the spec.
* docs: AVLiveBody macOS rewrite design
Clean-rewrite spec: fresh native macOS Xcode app for the
iPhone-USB body pipeline. Reuses the tested USB components,
single RealityKit scene (video quad + skeleton + mesh),
drops all legacy MediaPipe/viz/data-feed code.
* docs: AVLiveBody macOS rewrite plan
10-task plan: scaffold the xcodegen app, migrate the USB
pipeline, build the RealityKit scene (video quad, skeleton,
mesh), wire it, archive the legacy app.
* feat(avlivebody-mac): scaffold xcode app
Add an empty buildable native macOS app generated via xcodegen,
sibling of iphone-arbody. Depends on the shared AVLiveWire package.
Later tasks add the USB pipeline and RealityKit scene.
* feat(avlivebody-mac): migrate usb transport
Context: the new native AVLiveBody app needs the proven iPhone-Mac
usbmux transport layer. These files are self-contained, depending
only on AVLiveWire plus Apple system frameworks, so they cross the
rewrite boundary unchanged.
Approach: copy the three transport files and their unit tests
byte-for-byte from launcher/AV-Live-Body, then make the test target
buildable.
Changes:
- Add usb/USBMuxProtocol.swift, usb/USBClient.swift and
usb/VideoDecoder.swift under Sources/AVLiveBody.
- Add USBMuxProtocolTests.swift and USBClientTests.swift under
Tests/AVLiveBodyTests.
- Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so
xcodebuild can code sign the now-populated test bundle.
Impact: the usbmux pipeline is available in the rewrite and its
six unit tests run green under xcodebuild test.
* feat(avlivebody-mac): usb skeleton consumer
Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload
keyed by pid and owns video decoding directly, dropping the legacy
ArkitOSCListener conversion layer.
* fix(avlivebody-mac): guard thread store with lock
Move the `thread` property write inside the stateLock-held
region in start(); t.start() stays outside since the thread
cannot run before start() is called. Removes a latent race.
* feat(avlivebody-mac): multi-hmr and body fusion
Context: Task 4 of the macOS rewrite needs the dense-mesh half of
the pipeline alongside the USB skeleton consumer landed in task 3.
Approach: Add a CoreML wrapper that mirrors the validated Python
reference (data_only_viz/multihmr_coreml.py) and a pure-logic
fusion stage that corrects the mesh pelvis depth using the
LiDAR-precise USB skeleton.
Changes:
- MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image
input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at
10475 vertices, det threshold 0.3.
- BodyFusion.swift: stateless fuse(persons, skeletons) overrides
the highest-score mesh translation.z with the skeleton pelvis Z
when available, passes through otherwise.
- BodyFusionTests.swift: pelvis override and pass-through cases.
Impact: Unlocks the mesh renderer wiring in later tasks and gives
the macOS app metrically-correct depth in front of the camera.
* fix(avlivebody-mac): load mlmodelc, clarify fusion
Xcode compiles .mlpackage resources to .mlmodelc at build time;
look up the compiled artifact directly and drop the redundant
MLModel.compileModel step. Also rewrite BodyFusion docstring to
match actual single-person pelvis-z override behaviour.
* feat(avlivebody-mac): scene controller + view
RealityKit scene plumbing: SceneController owns ARView, orbital
camera, and holders for VideoQuad/SkeletonEntity/MeshEntity.
SceneView is the SwiftUI NSViewRepresentable bridge.
Build intentionally deferred to T8 (holder types land in T6-T8).
* fix(avlivebody): orbit gesture + setUp guard
Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to
dispatch only on .changed, replacing the Task wrapper with
MainActor.assumeIsolated. Guard SceneController.setUp() with a
didSetUp flag so duplicate makeNSView calls do not re-install
gestures or re-add anchors.
* feat(avlivebody-mac): 91-joint skeleton entity
Yellow marker spheres pooled per pid; ARKit (x,y,z) ->
RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for
macOS RealityKit Material.Color. Build deferred to T8.
* feat(avlivebody-mac): video quad
Flat 1.6x0.9m plane at z=-2m, textured per-frame from
CVPixelBuffer via CIImage -> CGImage -> TextureResource.
Per-frame TextureResource creation is the known perf hot
spot, isolated here for later LowLevelTexture upgrade.
* fix(avlivebody-mac): appkit import for orbit
NSPanGestureRecognizer lives in AppKit on macOS; without the
import the AVLiveBody module failed to emit. T5 leftover
surfaced once T6/T7/T8 made the target compilable.
* feat(avlivebody-mac): smpl-x mesh entity
Render SMPL-X dense meshes (10475 verts) from Multi-HMR with
pooled ModelEntity per person. Triangle indices loaded from the
bundled smplx_faces.bin (flat UInt32 triplets, copied from the
legacy launcher target). xcodegen folder-scanning bundles the
.bin under Contents/Resources/ — no project.yml change needed.
* feat(avlivebody-mac): wire scene + status bar
Replace placeholder window with ContentView wiring
USBSkeletonConsumer, SceneController, MultiHMRCoreML and
BodyFusion per the T9 dataflow plan.
* chore: archive legacy AV-Live-Body
Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.
* fix(avlivebody): break onVideoFrame retain cycle
Capture consumer weakly in the onVideoFrame closure so the
USBSkeletonConsumer can be deallocated and its background thread
exits cleanly. Guard the mesh-fusion path when consumer is gone.
* fix(launcher): disable body spawn post-archive
Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).
* docs(avlivebody-mac): contributor setup README
Document prerequisites, mlpackage copy, signing xcconfig, and
xcodegen/xcodebuild commands. Points at design spec and plan.
* refactor(avlivebody): axis helper + cleanups
- Extract arkitToRealityKit helper, dedupe 3 call sites.
- Add onDisappear consumer.stop() to terminate USB read loop.
- Replace @State with let for SceneController (stable class id).
- Add NSLog diagnostics in VideoQuad+MeshEntity silent guards.
* fix(avlivebody): ad-hoc signing for local dev
Apple Development cert + Automatic signing makes Xcode demand a
Mac Development cert that no one has. Switch to manual ad-hoc
(CODE_SIGN_IDENTITY = -) so any contributor can build. Drop
hardened runtime; re-enable for distribution builds.
* feat(arbody): keep iphone awake while streaming
iOS auto-lock tears down the USBServer TCP listener within
seconds, breaking AVLiveBody Mac-side connect. Disable the
idle timer for the lifetime of ContentView, restore on exit.
* docs: add AGENTS.md skeleton
Nine-task TDD plan: three pure-C++ units unit-tested with clang++
(spectrogram buffer, demo signal, reused FFT/analyzer checks) then
five openFrameworks GL units verified visually, building the stereo
sphere layer by layer.
Brainstorming output for a new AV-Live sibling project: a minimal
openFrameworks app rendering a dual-channel Hantek 6022 capture as a
single 3D sphere with three composable layers (spectrogram skin +
waveform displacement, orbit rings, point cloud). Reuses oscope-of's
capture and analysis layer.
Comparatif backbone DINOv2 ViT-S 672 sur deux Apple Silicon :
Hardware | CoreML GPU | CoreML ALL | PyTorch MPS
M5 (16 GB) | 25.1 ms | 157.3 ms | 274.7 ms
M1 Max(32G) | 63.3 ms | 154.1 ms | 110.6 ms
M5 GPU 2.5x plus rapide en CoreML (per-core efficiency vs
32 cores M1 Max). M1 Max 2.5x plus rapide en PyTorch MPS
(M5 souffrait probablement de throttle accumule).
ANE handicape les deux (~155 ms vs 25-63 ms GPU) — DINOv2
ViT-S non ANE-friendly.
macM1 workers MLX pauses via SIGSTOP/SIGCONT pour bench
isole.
clement