Files
AV-Live/data_only_viz/main.py
T
Clément SAILLANT 5c61112826 docs: add AGENTS.md skeleton (#1)
* docs(plans): action-head v3 + branch sync notes

Update plan header :
- v2 (Task 18) + v3 (Task 19) extensions chronology
- Studio train validated, ckpt action_head_v3.pt landed
- Mesh NaN-guard debug trail (commit 4e7101c)
- Branch convergence main == feat/action-head
- Pointers to memories project_action_head_v3, etc.

* feat(av-live): openpos 3D + DINO reid + filter

Three improvements wired end-to-end:

1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
   RealityKit AnchorEntity in BodyView, toggled by showSkeleton
   or vizMode==9. PoseOSCListener now parses /pose3d/count and
   /pose3d/kp (plus restored /face/* and /hand/* paths).

2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
   MeshRigger combines Hungarian IoU with cosine similarity over
   a per-pid embedding history (deque maxlen=10), weighted by
   MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
   DINO mlpackage absent or scipy missing. state.last_frame_rgb
   buffer added so the rigger can crop bbox regions for embedding.

3. PoseFilterChain on pose_world_landmarks:
   median (anti-spike) -> Kalman constant-velocity ->
   50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
   via POSE_FILTER env (default median+kalman+lookahead+ik).
   <2 ms per frame for typical 1-2 persons.

Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.

* fix(av-live-body): restore face+hand+3D (f540158)

Three regressions after recent merges, all restored to match the
original f540158 design:

1. FaceHandOverlay was no longer instantiated in ContentView.
   Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
   with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).

2. Skeleton3DRenderer was not attached. BodyView now creates an
   AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
   and ties its visibility to vizMode==9 or showSkeleton toggle.

3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
   skeleton actually reads as 3D instead of looking flat.

MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).

* feat(av-live): wireframe skel + face/hand filter

Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.

FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).

pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.

CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.

* deps(icp): add open3d optional extra + smoke test

Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.

Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.

Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
  (scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
  (PointCloud roundtrip + ICP convergence on translated copy),
  gated by `pytest.importorskip("open3d")`

Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.

* feat(icp): LiDAR TCP frame decoder + tests

* feat(icp): LiDAR TCP socket reader with reconnect

* feat(icp): extrinsic dataclass + JSON persistence

* feat(icp): Kabsch + calibration CLI scaffold

* feat(state): persons_arkit_joints fields

* feat(viz): ARKit 91 -> MP 33 joint map

* feat(viz): iphone OSC listener :57128

* feat(viz): arkit_fuse stage overrides 14 slots

* feat(viz): arkit pelvis z locks cam translation

* feat(viz): iphone OSC listener auto-start

* docs: arkit fusion env vars

* feat(icp): point-to-plane register + reject gate

* feat(icp): partition LiDAR per pid by max-dist

* feat(icp): FusionWorker + State.lidar_points

* feat(icp): wire fusion thread behind ICP_FUSION

Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.

A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.

Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.

MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.

* test(icp): synthetic latency + convergence bench

* docs(icp): env vars + calibration procedure

* docs(plans): icp lidar mesh + arkit joints

Two complementary fusion plans landed in parallel on 2026-05-14:
- iphone-lidar-multihmr-fusion : ARKit 91 joints -> MP33 fuse stage +
  pelvis z override (already implemented in 7 commits)
- icp-lidar-smplx-fusion : LiDAR mesh point-to-plane ICP onto SMPL-X
  10475 verts (12 tasks executed via subagent-driven-development)

Both paths coexist; joints are sparse+fast (60 Hz), mesh is dense+slow
(5-10 Hz). See docs/ICP_FUSION.md for the integration topology.

* feat(icp): predict_once via CoreML backend

* feat(av-live-body): wire ArkitOSCListener :57129

Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.

* feat(ios): iphone ARBodyTracker swiftpm app

iOS 17+ Swift Package app (.swiftpm) streaming ARKit body
joints via OSC UDP to two destinations:

  :57128 -> data_only_viz/iphone_osc_listener.py
  :57129 -> launcher/AV-Live-Body ArkitOSCListener.swift

Features:
- ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported
- 91 joints per body, /body3d/kp pid joint_idx x y z
- 30 fps throttle
- SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second
- Inline OSC encoder (no external dep)

Env mesh (TCP :5500) NOT yet implemented; requires a separate
ARWorldTrackingConfiguration session. ICP fusion path runs on
bench data only until phase 2.

* feat(data-feeds): 10 open-data OSC publisher

* feat(viz): DataFeeds OSC listener + HUD

* chore: gitignore tweaks

* docs: network topology + mDNS hostnames

Add a "Network topology" section to top-level CLAUDE.md doc
the 3-host layout (GrosMac source, Supra-M1 sink via mDNS,
iPhone via Personal Hotspot DHCP).

mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST
accept hostname.local instead of IPs, so the cluster survives
DHCP rotations on iPhone hotspot (172.20.10.x).

* fix(ios): add NSLocalNetworkUsageDescription

iOS 14+ silently blocks UDP to LAN addresses without this key.
The first time the app tries to send to 192.168.0.159, iOS will
prompt the user to allow Local Network Access; the prompt must
be accepted or the OSC stream never reaches the Mac.

Also adds NSBonjourServices declaring _osc._udp so the system
treats the connection as a recognised service.

* docs: network topology + gitignore hygiene

- CLAUDE.md: add mDNS hostname table (grosmac.local, supra-m1.local,
  iPhone hotspot 172.20.10.x). AVBODY_HOST / MULTIHMR_REMOTE_HOST
  accept hostnames — resilient to DHCP rotation.
- .gitignore: exclude .remember/ tool state and iCloud '* 2'
  collision artifacts.

* feat(ios): ARBody skeleton2D + overlay preview

ARBodySession: publish 2D-projected skeleton snapshot for live
overlay rendering on the iPhone screen alongside the camera feed.
ContentView: SkeletonOverlay drawing on top of the AR view, with
mock T-pose for Xcode previews (useMockBackground, useMockSkeleton).

* docs: iPhone USB body-tracking link design

Brainstormed design for replacing the OSC/network iPhone-Mac
link with a wired USB transport via usbmuxd. iPhone streams
ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML
and renders the mesh. Network-free, single native macOS app.

* docs: iPhone USB transport plan (1 of 3)

Bite-sized TDD plan for the network-free USB byte-pipe:
shared AVLiveWire frame format, native usbmux client,
iOS TCP frame server, incremental stream demuxer.

* feat(avlivewire): shared wire package skeleton

* feat(avlivewire): fixed 19-byte frame header codec

Add FrameHeader, a fixed-size binary record so the demuxer can
frame and resync the iPhone USB stream. Layout is big-endian:
4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32.

The magic prefix lets a reader detect and skip corrupt bytes.
Decoding rejects short buffers and bad magic by returning nil.
Big-endian append/parse helpers are added as Data/UInt extensions
to keep the codec self-contained.

* chore: ignore SwiftPM .build artifacts

Both AVLiveWire and AV-Live-Body produce .build/ on swift
test; ignore them so they never get accidentally staged.

* feat(avlivewire): skeleton and video codecs

Add SkeletonPayload (91 ARKit joints + per-joint validity) and
VideoPayload (one HEVC access unit + keyframe flag) with
big-endian encode/decode. Reuses Task 2 Data/UInt32 helpers.

* feat(avlivewire): incremental stream demuxer

Add StreamDemuxer that accepts arbitrary byte chunks from a
non-frame-aligned stream and emits complete (FrameHeader, Data)
frames, resyncing on the magic prefix after corruption.

* fix(avlivewire): cap demuxer payload length

A corrupt header with a huge UInt32 length made feed buffer
forever waiting for bytes that never arrive. Add an 8 MB max
payload cap; a header exceeding it is treated as corrupt, its
magic is skipped, and the demuxer resyncs on the next frame.

* feat(av-live-body): usbmux message codec

Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.

Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.

* feat(av-live-body): usbmux device client

Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.

Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.

* feat(av-live-body): usbmuxd unix socket transport

Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.

* fix(av-live-body): harden unix socket transport

Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
  instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
  idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.

* feat(ios): USB TCP frame server

Add USBServer: an NWListener on a fixed local TCP port that
usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames
and exposes a connection-state callback.

* build: depend on shared AVLiveWire package

Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.

* build(ios): add AVLiveWire package to xcodegen

The xcodegen project did not declare the shared AVLiveWire
package, so USBServer.swift would fail to import it in the
generated Xcode project. Add it as a local package dep.

* test(avlivewire): end-to-end chunked loopback

Feeds 20 framed skeleton payloads through StreamDemuxer in
7-byte chunks (worst-case TCP fragmentation). Fixed a split
range operator from the plan that did not parse.

* fix(ios): guard USBServer listener and payload

Report .idle (not .listening) when NWListener creation fails,
and drop payloads larger than the demuxer's 8 MB cap so the
receiver never silently discards an oversized frame.

* chore: ignore .swiftpm editor state dirs

swift test / Xcode create hidden .swiftpm dirs inside
packages; ignore them so they never get staged.

* docs: iPhone capture plan (2 of 3)

Plan for HEVC video capture (VideoToolbox) over the USB
transport and removal of the legacy OSC sender. Skeleton
USB path already exists; this adds the video half.

* feat(ios): HEVC video capture, drop OSC

Add VideoEncoder (VideoToolbox HEVC) and stream encoded
frames over USB as .video AVLiveWire frames alongside the
skeleton. Remove the legacy OSC/UDP fanout and its host/port
config UI — the iPhone link is now USB-only.

* docs(ios): refresh stale OSC references

ARBodySession header comment and Info.plist usage strings
still described the removed OSC/UDP path; update them to the
USB transport and drop the dead _osc._udp Bonjour service.

* docs: macOS USB consumer plan (3a of 3)

Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.

* feat(av-live-body): USB skeleton consumer

Background usbmux read loop feeding StreamDemuxer; republishes
.skeleton frames as 91-joint ArkitBodyFrames and forwards
.video payloads. Removed stale iCloud collision duplicate
source files that broke the build.

* fix(data-only): CoreML Multi-HMR usage bugs

The CoreML Multi-HMR model was fine; the "0 detections" bug
was caller-side. Add ImageNet normalization in infer() (the
DINOv2 backbone needs it; raw [0,1] input collapsed all
scores) and update stale hardcoded output var names to match
the re-converted mlpackage. Bump the latency test threshold
to the realistic ~140 ms full-model figure.

* feat(av-live-body): HEVC video decoder

VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.

* feat(av-live-body): render 91-joint USB skeleton

Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.

* feat(av-live-body): wire USB consumer to renderer

ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.

* docs: macOS Multi-HMR mesh plan (3b of 3)

Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML
Swift wrapper, BodyFusion (ARKit depth correction), mesh
pipeline wiring. Completes the spec.

* docs: AVLiveBody macOS rewrite design

Clean-rewrite spec: fresh native macOS Xcode app for the
iPhone-USB body pipeline. Reuses the tested USB components,
single RealityKit scene (video quad + skeleton + mesh),
drops all legacy MediaPipe/viz/data-feed code.

* docs: AVLiveBody macOS rewrite plan

10-task plan: scaffold the xcodegen app, migrate the USB
pipeline, build the RealityKit scene (video quad, skeleton,
mesh), wire it, archive the legacy app.

* feat(avlivebody-mac): scaffold xcode app

Add an empty buildable native macOS app generated via xcodegen,
sibling of iphone-arbody. Depends on the shared AVLiveWire package.
Later tasks add the USB pipeline and RealityKit scene.

* feat(avlivebody-mac): migrate usb transport

Context: the new native AVLiveBody app needs the proven iPhone-Mac
usbmux transport layer. These files are self-contained, depending
only on AVLiveWire plus Apple system frameworks, so they cross the
rewrite boundary unchanged.

Approach: copy the three transport files and their unit tests
byte-for-byte from launcher/AV-Live-Body, then make the test target
buildable.

Changes:
- Add usb/USBMuxProtocol.swift, usb/USBClient.swift and
  usb/VideoDecoder.swift under Sources/AVLiveBody.
- Add USBMuxProtocolTests.swift and USBClientTests.swift under
  Tests/AVLiveBodyTests.
- Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so
  xcodebuild can code sign the now-populated test bundle.

Impact: the usbmux pipeline is available in the rewrite and its
six unit tests run green under xcodebuild test.

* feat(avlivebody-mac): usb skeleton consumer

Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload
keyed by pid and owns video decoding directly, dropping the legacy
ArkitOSCListener conversion layer.

* fix(avlivebody-mac): guard thread store with lock

Move the `thread` property write inside the stateLock-held
region in start(); t.start() stays outside since the thread
cannot run before start() is called. Removes a latent race.

* feat(avlivebody-mac): multi-hmr and body fusion

Context: Task 4 of the macOS rewrite needs the dense-mesh half of
the pipeline alongside the USB skeleton consumer landed in task 3.

Approach: Add a CoreML wrapper that mirrors the validated Python
reference (data_only_viz/multihmr_coreml.py) and a pure-logic
fusion stage that corrects the mesh pelvis depth using the
LiDAR-precise USB skeleton.

Changes:
- MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image
  input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at
  10475 vertices, det threshold 0.3.
- BodyFusion.swift: stateless fuse(persons, skeletons) overrides
  the highest-score mesh translation.z with the skeleton pelvis Z
  when available, passes through otherwise.
- BodyFusionTests.swift: pelvis override and pass-through cases.

Impact: Unlocks the mesh renderer wiring in later tasks and gives
the macOS app metrically-correct depth in front of the camera.

* fix(avlivebody-mac): load mlmodelc, clarify fusion

Xcode compiles .mlpackage resources to .mlmodelc at build time;
look up the compiled artifact directly and drop the redundant
MLModel.compileModel step. Also rewrite BodyFusion docstring to
match actual single-person pelvis-z override behaviour.

* feat(avlivebody-mac): scene controller + view

RealityKit scene plumbing: SceneController owns ARView, orbital
camera, and holders for VideoQuad/SkeletonEntity/MeshEntity.
SceneView is the SwiftUI NSViewRepresentable bridge.

Build intentionally deferred to T8 (holder types land in T6-T8).

* fix(avlivebody): orbit gesture + setUp guard

Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to
dispatch only on .changed, replacing the Task wrapper with
MainActor.assumeIsolated. Guard SceneController.setUp() with a
didSetUp flag so duplicate makeNSView calls do not re-install
gestures or re-add anchors.

* feat(avlivebody-mac): 91-joint skeleton entity

Yellow marker spheres pooled per pid; ARKit (x,y,z) ->
RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for
macOS RealityKit Material.Color. Build deferred to T8.

* feat(avlivebody-mac): video quad

Flat 1.6x0.9m plane at z=-2m, textured per-frame from
CVPixelBuffer via CIImage -> CGImage -> TextureResource.
Per-frame TextureResource creation is the known perf hot
spot, isolated here for later LowLevelTexture upgrade.

* fix(avlivebody-mac): appkit import for orbit

NSPanGestureRecognizer lives in AppKit on macOS; without the
import the AVLiveBody module failed to emit. T5 leftover
surfaced once T6/T7/T8 made the target compilable.

* feat(avlivebody-mac): smpl-x mesh entity

Render SMPL-X dense meshes (10475 verts) from Multi-HMR with
pooled ModelEntity per person. Triangle indices loaded from the
bundled smplx_faces.bin (flat UInt32 triplets, copied from the
legacy launcher target). xcodegen folder-scanning bundles the
.bin under Contents/Resources/ — no project.yml change needed.

* feat(avlivebody-mac): wire scene + status bar

Replace placeholder window with ContentView wiring
USBSkeletonConsumer, SceneController, MultiHMRCoreML and
BodyFusion per the T9 dataflow plan.

* chore: archive legacy AV-Live-Body

Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.

* fix(avlivebody): break onVideoFrame retain cycle

Capture consumer weakly in the onVideoFrame closure so the
USBSkeletonConsumer can be deallocated and its background thread
exits cleanly. Guard the mesh-fusion path when consumer is gone.

* fix(launcher): disable body spawn post-archive

Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).

* docs(avlivebody-mac): contributor setup README

Document prerequisites, mlpackage copy, signing xcconfig, and
xcodegen/xcodebuild commands. Points at design spec and plan.

* refactor(avlivebody): axis helper + cleanups

- Extract arkitToRealityKit helper, dedupe 3 call sites.
- Add onDisappear consumer.stop() to terminate USB read loop.
- Replace @State with let for SceneController (stable class id).
- Add NSLog diagnostics in VideoQuad+MeshEntity silent guards.

* fix(avlivebody): ad-hoc signing for local dev

Apple Development cert + Automatic signing makes Xcode demand a
Mac Development cert that no one has. Switch to manual ad-hoc
(CODE_SIGN_IDENTITY = -) so any contributor can build. Drop
hardened runtime; re-enable for distribution builds.

* feat(arbody): keep iphone awake while streaming

iOS auto-lock tears down the USBServer TCP listener within
seconds, breaking AVLiveBody Mac-side connect. Disable the
idle timer for the lifetime of ContentView, restore on exit.

* docs: add AGENTS.md skeleton
2026-05-21 12:41:33 +02:00

688 lines
33 KiB
Python
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Entry point du visualizer Metal pour le mode data-only.
Lance :
- Une fenetre AppKit avec une MTKView plein-cadre
- Un thread OSC en arriere-plan qui ecoute :57123
- La run loop NSApplication (jamais retournee tant que la window est ouverte)
Usage :
uv run python -m data_only_viz.main
uv run python -m data_only_viz.main --port 57123 --fullscreen
Le main thread DOIT etre la run loop AppKit (regle macOS). Le listener
OSC tourne dans un thread daemon.
"""
from __future__ import annotations
import argparse
import logging
import os
import signal
import sys
import objc
from AppKit import (
NSApp,
NSApplication,
NSApplicationActivationPolicyRegular,
NSBackingStoreBuffered,
NSColor,
NSEvent,
NSEventMaskKeyDown,
NSData,
NSFont,
NSImage,
NSImageScaleProportionallyUpOrDown,
NSImageView,
NSMakeRect,
NSObject,
NSScreen,
NSTextView,
NSTimer,
NSWindow,
NSWindowStyleMaskClosable,
NSWindowStyleMaskMiniaturizable,
NSWindowStyleMaskResizable,
NSWindowStyleMaskTitled,
NSViewWidthSizable,
NSViewHeightSizable,
)
from MetalKit import MTKView
from pythonosc.udp_client import SimpleUDPClient
from .osc_listener import OscListener
from .renderer import MetalRenderer
from .state import (
KEYMAP_AUDIO, KEYMAP_SOURCE, KEYMAP_SOURCE_NUM, KEYMAP_VIDEO, State,
)
LOG = logging.getLogger("main")
class AppDelegate(NSObject):
def initWithOpts_(self, opts): # noqa: N802
self = objc.super(AppDelegate, self).init()
if self is None:
return None
self._opts = opts
self._state = State()
self._listener = OscListener(self._state, host="127.0.0.1", port=opts.port)
# Sender vers sclang pour les changements de scene depuis le clavier
self._scClient = SimpleUDPClient("127.0.0.1", opts.sclang_port)
self._pose_worker = None
return self
def applicationDidFinishLaunching_(self, notification): # noqa: N802
LOG.info("applicationDidFinishLaunching: start")
# Mode headless : --multi-hmr cede l'affichage a AV-Live-Body
# (Swift RealityKit). On garde seulement le worker pose + le
# sender TCP, donc pas de NSWindow / Metal renderer / HUD.
if getattr(self._opts, "multi_hmr", False):
LOG.info("multi-hmr mode: headless (no NSWindow)")
self._headless = True
from .multi_hmr_worker import MultiHMRWorker
if not MultiHMRWorker.is_available():
LOG.error(
"Multi-HMR requested via --multi-hmr but checkpoint is missing. "
"Run scripts/setup_multihmr.sh first, or omit --multi-hmr to use MediaPipe."
)
sys.exit(2)
self._listener.start()
self._start_pose_worker()
LOG.info("headless ready — OSC :%d, TCP sender :57130",
self._opts.port)
return
self._headless = False
# 1) Fenetre
screen = NSScreen.mainScreen().frame()
w, h = self._opts.width, self._opts.height
x = (screen.size.width - w) / 2
y = (screen.size.height - h) / 2
style = (NSWindowStyleMaskTitled
| NSWindowStyleMaskClosable
| NSWindowStyleMaskResizable
| NSWindowStyleMaskMiniaturizable)
self._window = NSWindow.alloc().initWithContentRect_styleMask_backing_defer_(
NSMakeRect(x, y, w, h), style, NSBackingStoreBuffered, False)
self._window.setTitle_("AV-Live · data-only viz")
self._window.setBackgroundColor_(NSColor.blackColor())
LOG.info("window created")
# 2) Container fullscreen avec 2 couches :
# z=0 NSImageView : flux webcam live PLEIN-CADRE (sous le Metal)
# z=1 MTKView : skeleton + viz effects (transparent par-dessus)
from AppKit import NSView
self._container = NSView.alloc().initWithFrame_(NSMakeRect(0, 0, w, h))
self._container.setWantsLayer_(True)
self._container.layer().setBackgroundColor_(
NSColor.blackColor().CGColor())
# 2a) Webcam fond plein-cadre
self._cam = NSImageView.alloc().initWithFrame_(NSMakeRect(0, 0, w, h))
self._cam.setImageScaling_(NSImageScaleProportionallyUpOrDown)
self._cam.setAutoresizingMask_(NSViewWidthSizable | NSViewHeightSizable)
# Background visible meme sans frame (debug)
self._cam.setWantsLayer_(True)
self._cam.layer().setBackgroundColor_(
NSColor.colorWithCalibratedRed_green_blue_alpha_(0.05, 0.05, 0.1, 1)
.CGColor())
self._container.addSubview_(self._cam)
# 2b) MTKView par-dessus, transparent : on voit la webcam dessous
self._renderer = MetalRenderer.alloc().initWithState_(self._state)
LOG.info("renderer created")
device = self._renderer.device()
self._mtkview = MTKView.alloc().initWithFrame_device_(
NSMakeRect(0, 0, w, h), device)
self._mtkview.setDelegate_(self._renderer)
self._mtkview.setPreferredFramesPerSecond_(60)
self._mtkview.setColorPixelFormat_(80) # BGRA8Unorm
# Transparence : on rend le MTKView en RGBA et on laisse la
# CAMetalLayer ne pas etre opaque pour voir le NSImageView dessous.
layer = self._mtkview.layer()
if layer is not None:
layer.setOpaque_(False)
# MTKView expose framebufferOnly directement
try: self._mtkview.setFramebufferOnly_(False)
except Exception: pass
self._mtkview.setClearColor_((0.0, 0.0, 0.0, 0.0))
self._mtkview.setAutoresizingMask_(NSViewWidthSizable | NSViewHeightSizable)
self._container.addSubview_(self._mtkview)
LOG.info("mtkview configured (transparent overlay)")
self._window.setContentView_(self._container)
# Window au premier plan : level = floating, always on top jusqu'a
# ce qu'on perde le focus. Sinon la fenetre peut etre cachee
# derriere l'IDE / le terminal.
self._window.setLevel_(3) # NSFloatingWindowLevel
self._window.makeKeyAndOrderFront_(None)
NSApp().activateIgnoringOtherApps_(True)
self._window.makeFirstResponder_(self._container)
LOG.info("window shown + key focus forced + floating level")
# 2b) HUD : overlay NSTextView semi-transparent au-dessus du MTKView.
# NSTextView prend en charge le rendu CoreText sans avoir a passer
# par un MTLBuffer texte. On le pose comme subview du MTKView.
self._hud = NSTextView.alloc().initWithFrame_(
NSMakeRect(12, 12, 340, 240))
self._hud.setEditable_(False)
self._hud.setSelectable_(False)
self._hud.setDrawsBackground_(True)
self._hud.setBackgroundColor_(
NSColor.colorWithCalibratedRed_green_blue_alpha_(0, 0, 0, 0.45))
self._hud.setFont_(NSFont.fontWithName_size_("Menlo", 11))
self._hud.setTextColor_(NSColor.whiteColor())
self._hud.setAutoresizingMask_(NSViewHeightSizable)
self._mtkview.addSubview_(self._hud)
# Timer 10 Hz, rafraichit le texte avec les valeurs du State.
self._hudTimer = NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(
0.1, self, "refreshHud:", None, True)
# 2c) Timer webcam update (NSImageView fullscreen deja cree en 2a)
self._camTimer = NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(
1.0 / 15.0, self, "refreshCam:", None, True)
# 2d) Auto-engage du mode openpos (#9) quand des personnes sont
# detectees. Si l'utilisateur a force un mode au clavier dans
# les 8 dernieres secondes, on n'override pas (lock manuel).
self._user_viz_lock_t = 0.0 # set par _on_key, lu par autoOpenpos
self._autoOpenposTimer = NSTimer.scheduledTimerWithTimeInterval_target_selector_userInfo_repeats_(
0.5, self, "autoOpenpos:", None, True)
if self._opts.fullscreen:
self._window.toggleFullScreen_(None)
# 3) Listener OSC
self._listener.start()
LOG.info("ready — listening OSC :%d (sc -> :%d)",
self._opts.port, self._opts.sclang_port)
# 3b) Pose worker (optionnel, cv2 + YOLOv8-pose -> State)
if self._opts.pose:
self._request_camera_and_start_pose()
def _request_camera_and_start_pose(self):
"""macOS exige que la demande TCC camera (AVCaptureDevice.requestAccess)
vienne du MAIN THREAD AppKit. OpenCV depuis un worker thread plante
avec 'can not spin main run loop from other thread'. On demande ici,
sur le main thread, PUIS on lance le pose worker avec
OPENCV_AVFOUNDATION_SKIP_AUTH=1 pour qu'il ne tente pas une 2e demande."""
import os
os.environ["OPENCV_AVFOUNDATION_SKIP_AUTH"] = "1"
try:
from AVFoundation import (
AVCaptureDevice, AVMediaTypeVideo,
AVAuthorizationStatusAuthorized,
AVAuthorizationStatusNotDetermined,
)
status = AVCaptureDevice.authorizationStatusForMediaType_(AVMediaTypeVideo)
LOG.info("camera TCC status: %s", status)
if status == AVAuthorizationStatusAuthorized:
self._start_pose_worker()
return
if status == AVAuthorizationStatusNotDetermined:
LOG.info("requesting camera access via AVFoundation...")
def _handler(granted):
LOG.info("camera access granted=%s", granted)
if granted:
# Le handler tourne sur un thread arbitraire ; on dispatch
# vers le main pour creer le worker proprement.
from Foundation import NSOperationQueue
NSOperationQueue.mainQueue().addOperationWithBlock_(
self._start_pose_worker)
AVCaptureDevice.requestAccessForMediaType_completionHandler_(
AVMediaTypeVideo, _handler)
return
LOG.warning("camera access denied — Reglages > Confidentialite")
except Exception as e: # noqa: BLE001
LOG.warning("AVFoundation TCC check failed (%s) — tentative directe", e)
self._start_pose_worker()
def _start_pose_worker(self):
# Priorite (user feedback : on veut FACE + HANDS + BODY toujours
# disponibles pour le mapping sonore + viz openpos, donc MediaPipe
# Multi devient le default avec ses 33 body + 478 face + 42 hand
# par personne, plutot qu'Apple Vision body-only 13 joints) :
# 0. Multi-HMR (opt-in via --multi-hmr flag)
# 1. MediaPipe Multi (33+478+42 kp × 4 personnes) — DEFAUT
# 2. Apple Vision body pose (fallback si MediaPipe casse)
# 3. CoreML pose, DETRPose, Holistic, YOLO — fallbacks
import os as _os
# iPhone ARBodyTracker (option 2 LiDAR fusion) : always-on
# listener on :57128. Harmless if no iPhone is broadcasting ;
# state.persons_arkit_joints stays empty and the arkit_fuse
# stage no-ops. Activated via POSE_FILTER=...+arkit_fuse.
try:
from .iphone_osc_listener import IphoneOSCListener
self._iphone_osc = IphoneOSCListener(self._state)
self._iphone_osc.start()
LOG.info("worker: + iPhone OSC listener :57128")
except Exception as e: # noqa: BLE001
LOG.warning("iphone OSC listener start failed (%s)", e)
# ICP LiDAR fusion (opt-in via ICP_FUSION=1). Parallel to the
# ARKit pelvis fuse: ICP operates on SMPL-X dense vertices, not
# joints. Requires a calibrated extrinsic on disk (see
# scripts/calibrate_lidar.py) and an iPhone LiDAR stream
# broadcasting on ICP_LIDAR_HOST:ICP_LIDAR_PORT.
if _os.environ.get("ICP_FUSION", "0") == "1":
host = _os.environ.get("ICP_LIDAR_HOST")
if not host:
LOG.warning("ICP_FUSION=1 but ICP_LIDAR_HOST unset — "
"fusion disabled")
else:
try:
from .icp_fusion_worker import IcpFusionThread
self._icp_fusion = IcpFusionThread(
self._state,
host=host,
port=int(_os.environ.get("ICP_LIDAR_PORT", "5500")),
)
self._icp_fusion.start()
LOG.info("worker: + ICP LiDAR fusion -> %s:%s", host,
_os.environ.get("ICP_LIDAR_PORT", "5500"))
except Exception as e: # noqa: BLE001
LOG.warning("icp fusion start failed (%s)", e)
# 0. Multi-HMR (SMPL-X 10475 verts mesh dense) — opt-in via flag
if getattr(self._opts, "multi_hmr", False):
try:
from .multi_hmr_worker import MultiHMRWorker
from .smplx_osc_sender import SMPLXTCPSender
if MultiHMRWorker.is_available():
self._pose_worker = MultiHMRWorker(
self._state, num_persons=4,
target_fps=10.0,
device=getattr(self._opts, "pose_device", "mps"),
det_thresh=getattr(self._opts, "det_thresh", 0.15),
nms_kernel_size=getattr(
self._opts, "nms_kernel_size", 5),
motion_gate=getattr(
self._opts, "motion_gate", 5.0),
camera_index=getattr(self._opts, "camera_index", -1))
self._pose_worker.start()
# MESH_RIG=0 disables the 30 fps rigid translation
# rigger from mesh_rigger.py (used to debug deformation
# issues introduced by the hybrid rigging path).
self._smplx_tcp = SMPLXTCPSender(
self._state,
enable_rigging=os.environ.get("MESH_RIG", "1") != "0",
)
self._smplx_tcp.start()
LOG.info("worker: Multi-HMR + SMPL-X (mesh dense)")
# Secondary body-pose worker in parallel: AVLiveBody
# gets body keypoints on UDP :57126 alongside the mesh
# on TCP :57130. Default: Apple Vision (ANE-accel,
# body only 19 joints). Set AV_LIVE_PARALLEL_POSE=
# mediapipe to swap to MediaPipe Holistic (CPU
# XNNPACK but provides face + hand + 3D world).
# Defaut: lance BOTH Apple Vision (body 19 joints sur
# ANE, ~30 fps) ET MediaPipe Multi (face 468 + hands 21
# + pose 3D world sur CPU XNNPACK). Set
# AV_LIVE_PARALLEL_POSE=apple_vision pour ne garder que
# le path ANE (face/hand fin disparait), ou =mediapipe
# pour ne garder que CPU.
parallel = _os.environ.get(
"AV_LIVE_PARALLEL_POSE", "both")
if parallel in ("apple_vision", "both"):
try:
from .apple_vision_pose import AppleVisionPoseWorker
if AppleVisionPoseWorker.is_available():
self._av_worker = AppleVisionPoseWorker(
self._state, target_fps=30.0,
num_persons=4)
self._av_worker.start()
LOG.info("worker: + Apple Vision body pose "
"(ANE) in parallel")
else:
raise RuntimeError("apple_vision unavailable")
except Exception as e: # noqa: BLE001
LOG.warning("Apple Vision parallel start failed "
"(%s)", e)
if parallel in ("mediapipe", "both"):
try:
from .multi import MultiWorker
self._mp_worker = MultiWorker(
self._state, num_persons=4)
self._mp_worker.start()
LOG.info("worker: + MediaPipe Multi (3D pose "
"+ face + hand) in parallel")
except Exception as e: # noqa: BLE001
LOG.warning("MediaPipe parallel start failed "
"(%s)", e)
return
LOG.info("Multi-HMR indisponible (checkpoints manquants) "
"— voir scripts/setup_multihmr.sh")
except Exception as e: # noqa: BLE001
LOG.warning("Multi-HMR failed (%s) — fallback", e)
# 1. MediaPipe Multi : DEFAUT pour le mapping sonore + openpos
# (33 body + 478 face + 21x2 hands × 4 personnes). Skip via
# AV_LIVE_MEDIAPIPE=0 si on prefere body-only ANE-accelere.
if _os.environ.get("AV_LIVE_MEDIAPIPE") != "0":
try:
from .multi import MultiWorker
self._pose_worker = MultiWorker(self._state, num_persons=4)
self._pose_worker.start()
LOG.info("worker: MediaPipe Multi (Pose+Face+Hand × 4)")
return
except Exception as e: # noqa: BLE001
LOG.warning("MediaPipe Multi unavailable (%s) — fallback", e)
# 2. Apple Vision body pose : fallback si MediaPipe casse.
# Body only, 13 joints, pas de face/hands.
if _os.environ.get("AV_LIVE_APPLE_VISION") != "0":
try:
from .apple_vision_pose import AppleVisionPoseWorker
if AppleVisionPoseWorker.is_available():
self._pose_worker = AppleVisionPoseWorker(
self._state, target_fps=30.0, num_persons=4)
self._pose_worker.start()
LOG.info("worker: Apple Vision body pose "
"(ANE natif, body only, multi-personne)")
return
LOG.info("Apple Vision body pose indisponible "
"(macOS < 11 ?) — fallback")
except Exception as e: # noqa: BLE001
LOG.warning("Apple Vision pose indisponible (%s) — fallback", e)
if _os.environ.get("AV_LIVE_COREML") != "0":
try:
from .coreml_pose import CoreMLPoseWorker
if CoreMLPoseWorker.is_available():
self._pose_worker = CoreMLPoseWorker(
self._state, target_fps=30.0, num_persons=4)
self._pose_worker.start()
LOG.info("worker: CoreML pose natif "
"(AVFoundation + YOLO11n-pose ANE)")
return
LOG.info("CoreML pose .mlpackage absent — "
"lancer 'uv run python -m data_only_viz.scripts.convert_coreml' "
"puis relancer pour activer le pipeline ANE")
except Exception as e: # noqa: BLE001
LOG.warning("CoreML pose indisponible (%s) — fallback", e)
if _os.environ.get("AV_LIVE_DETRPOSE") == "1":
try:
from .detrpose import DETRPoseWorker, is_available
if is_available():
self._pose_worker = DETRPoseWorker(
self._state, num_persons=4,
model_size=getattr(self._opts, "detrpose_model_size", "n"))
self._pose_worker.start()
LOG.info("worker: DETRPose (transformer, body 17 kp × 4)")
return
LOG.info("DETRPose pas installe — fallback MediaPipe Multi "
"(voir data_only_viz/detrpose.py pour install)")
except Exception as e: # noqa: BLE001
LOG.warning("detrpose indisponible (%s) — fallback multi", e)
# MediaPipe Multi deja tente en priorite 1 ; on saute direct
# au fallback holistic puis YOLO.
try:
from .holistic import HolisticWorker
self._pose_worker = HolisticWorker(self._state)
self._pose_worker.start()
LOG.info("worker: MediaPipe Holistic (mono-personne ~553 lm)")
return
except Exception as e: # noqa: BLE001
LOG.warning("holistic unavailable (%s) — fallback YOLO", e)
from .pose import PoseWorker
self._pose_worker = PoseWorker(
self._state, device=self._opts.pose_device)
self._pose_worker.start()
# 4) Hook clavier : local (app au focus) + global (app au fond).
# Le global monitor est read-only mais permet de garder le pilotage
# quand l'utilisateur a une autre app au premier plan (IDE,
# browser). On ignore le retour pour le global (sinon double-trigger).
self._kb_monitor = NSEvent.addLocalMonitorForEventsMatchingMask_handler_(
NSEventMaskKeyDown, self._on_key)
self._kb_global = NSEvent.addGlobalMonitorForEventsMatchingMask_handler_(
NSEventMaskKeyDown, self._on_key_global)
# Force le focus initial pour que le local monitor reçoive
# tout de suite les touches sans nécessiter un clic.
NSApp().activateIgnoringOtherApps_(True)
self._window.makeKeyAndOrderFront_(None)
self._window.makeFirstResponder_(self._container)
_cam_log_count = 0
def refreshCam_(self, _timer): # noqa: N802
"""Lit la derniere frame JPEG du pose worker et l'affiche en fond."""
with self._state.lock():
jpg = self._state.last_webcam_jpeg
if not jpg:
return
try:
# NSData : on essaye 2 APIs ; la deuxieme est plus robuste
# avec pyobjc 12 pour les Python bytes.
data = NSData.dataWithBytes_length_(jpg, len(jpg))
except Exception:
data = NSData.alloc().initWithBytes_length_(jpg, len(jpg))
img = NSImage.alloc().initWithData_(data)
if img is not None and img.isValid():
self._cam.setImage_(img)
self.__class__._cam_log_count += 1
if self.__class__._cam_log_count in (1, 30, 150):
LOG.info("cam frame #%d displayed (size %dx%d)",
self.__class__._cam_log_count,
int(img.size().width), int(img.size().height))
elif self.__class__._cam_log_count == 0:
LOG.warning("cam: NSImage init failed (jpg %d bytes)", len(jpg))
self.__class__._cam_log_count = -1
def refreshHud_(self, _timer): # noqa: N802
"""Format le HUD avec la valeur courante des flux + correspondance
avec le rendu visuel. Appele a 10 Hz par NSTimer."""
s = self._state
with s.lock():
mode = s.viz_mode
mode_name = s.viz_mode_names[mode]
preset = s.active_preset
scene = s.active_scene
kp = s.swpc_kp
flare = s.swpc_flare_norm
wind = s.swpc_wind_speed
bz = s.swpc_bz
netz = s.netz_dev
lr = s.lightning_rate_min
quake = s.usgs_last_mag
planes = s.aviation_count
social = s.social_rate
pose_n = s.pose_count
pose_live = s.pose_alive()
rms = s.rms
bpm = s.bpm
beat = s.beat
bridge_age = 0.0
if s.last_heartbeat:
import time as _t
bridge_age = _t.monotonic() - s.last_heartbeat
# Marqueurs : ce que chaque donnee fait dans le rendu courant
kp_norm = min(1.0, kp / 9.0)
wind_norm = max(0.0, min(1.0, (wind - 280.0) / 600.0))
bz_norm = max(-1.0, min(1.0, bz / 15.0))
bridge_ok = "OK" if bridge_age < 15 else "DOWN"
bar = lambda v, w=14: "" * int(max(0.0, min(1.0, v)) * w) + \
"·" * (w - int(max(0.0, min(1.0, v)) * w))
bits = []
if preset: bits.append(f"source:{preset}")
if scene: bits.append(f"scène:{scene}")
active_line = (" " + " · ".join(bits)) if bits else ""
bridge_label = "OK" if bridge_age < 15 else "ARRÊTÉ"
pose_label = "OUI" if pose_live else "non"
txt = (
f"AV-Live data-only · vidéo:[{mode}]{mode_name}{active_line}\n"
f"────────────────────────────────────────\n"
f" Touches vidéo:azertyuiop audio:qsdfghjklm source:wxcvbn 0-9\n"
f"────────────────────────────────────────\n"
f" Audio rms {rms:5.2f} bpm {bpm:5.1f} beat {beat:>4}\n"
f" Pont {bridge_label} ({bridge_age:4.1f}s)\n"
f"────────────────────────────────────────\n"
f" Soleil Kp {kp:4.1f} {bar(kp_norm)} → palette orage\n"
f" éruption {flare:4.2f} {bar(flare)} → flash orange\n"
f" vent {wind:4.0f} {bar(wind_norm)} → vitesse fond\n"
f" Bz {bz:+5.1f} {bar((bz_norm+1)/2)} → reverb\n"
f" Réseau Δf {netz:+.3f}Hz {bar(abs(netz)*10)} → vibrato\n"
f" Foudre {lr:4.1f}/min → percussions\n"
f" Séisme M {quake:4.1f} → sub-bass\n"
f" Avions {planes:>4} → voix FM\n"
f" Social {social:4.1f}/s → kick\n"
f" Pose n={pose_n:>2} live={pose_label} → squelette\n"
)
self._hud.setString_(txt)
def autoOpenpos_(self, _timer): # noqa: N802
"""Si des personnes sont detectees ET l'utilisateur n'a pas pose
un mode au clavier dans les 8 dernieres secondes, on passe
automatiquement en mode openpos (#9) pour mettre la pose en
valeur. Lache la main si lock recent ou si plus personne."""
import time as _t
s = self._state
with s.lock():
has_persons = bool(s.persons_body)
mode = s.viz_mode
now = _t.monotonic()
if (now - self._user_viz_lock_t) < 8.0:
return # lock utilisateur recent
if has_persons and mode != 9:
with s.lock():
s.viz_mode = 9
LOG.info("[auto] openpos engaged (persons detectees)")
elif not has_persons and mode == 9:
# plus de personne, on revient au mode storm par defaut
with s.lock():
s.viz_mode = 0
LOG.info("[auto] storm engaged (plus de pose)")
def _on_key_global(self, ev):
# Global monitor : read-only, on appelle _on_key mais on ne
# retourne pas l'event (interdit par AppKit pour les globaux).
try:
self._on_key(ev)
except Exception as e: # noqa: BLE001
LOG.warning("global key handler failed: %s", e)
def _on_key(self, ev):
key = ev.charactersIgnoringModifiers()
if not key:
return ev
k = key.lower()
# Debug : log toutes les touches recues pour diagnostic
LOG.debug("[key] raw=%r lower=%r", key, k)
if key == "\x1b":
NSApp().terminate_(self); return None
if key == " ":
self._scClient.send_message("/control/doScene", ["stop"])
return None
# Cmd+F geree par macOS pour fullscreen ; on garde shift+F en raccourci
if key == "F":
self._window.toggleFullScreen_(None); return None
if key == "H":
self._hud.setHidden_(not self._hud.isHidden()); return None
if key == "C": # Shift+C : toggle webcam overlay
self._cam.setHidden_(not self._cam.isHidden()); return None
# azertyuiop -> video (viz mode)
for kk, name in KEYMAP_VIDEO:
if k == kk:
names = list(self._state.viz_mode_names)
if name in names:
idx = names.index(name)
with self._state.lock():
self._state.viz_mode = idx
# Lock l'auto-openpos pendant 8s : l'utilisateur a
# explicitement choisi un mode, on respecte.
import time as _t
self._user_viz_lock_t = _t.monotonic()
LOG.info("[video] viz -> %s (%d) (lock 8s)", name, idx)
return None
# qsdfghjklm -> audio (scene SC)
for kk, scene in KEYMAP_AUDIO:
if k == kk:
self._scClient.send_message("/control/doScene", [scene])
with self._state.lock():
self._state.active_scene = scene
LOG.info("[audio] scene -> %s", scene)
return None
# wxcvbn + 0-9 -> preset bundle (source + scene audio + viz video)
for kk, source, scene, viz in (*KEYMAP_SOURCE, *KEYMAP_SOURCE_NUM):
if key == kk or k == kk:
# Audio : envoie a sclang
self._scClient.send_message("/control/doScene", [scene])
# Video : applique le viz mode
names = list(self._state.viz_mode_names)
idx = names.index(viz) if viz in names else 0
with self._state.lock():
self._state.viz_mode = idx
self._state.active_preset = source
self._state.active_scene = scene
LOG.info("[preset] %s : scene=%s viz=%s", source, scene, viz)
return None
return ev
def applicationWillTerminate_(self, _): # noqa: N802
self._listener.stop()
if self._pose_worker is not None:
self._pose_worker.stop()
icp = getattr(self, "_icp_fusion", None)
if icp is not None:
try:
icp.stop()
except Exception as e: # noqa: BLE001
LOG.warning("icp fusion stop failed (%s)", e)
LOG.info("bye")
def main() -> int:
p = argparse.ArgumentParser(prog="data_only_viz")
p.add_argument("--port", type=int, default=57123,
help="UDP port OSC entrant (default 57123)")
p.add_argument("--sclang-port", type=int, default=57121,
help="UDP port SC sortant pour /control/doScene (default 57121)")
p.add_argument("--width", type=int, default=1280)
p.add_argument("--height", type=int, default=720)
p.add_argument("--fullscreen", action="store_true")
p.add_argument("--pose", action="store_true",
help="Active la captation pose YOLO (cv2 + ultralytics)")
p.add_argument("--multi-hmr", dest="multi_hmr", action="store_true",
help="Active Multi-HMR worker pour mesh SMPL-X dense "
"(necessite setup_multihmr.sh + SMPLX_NEUTRAL.npz)")
p.add_argument("--camera-index", dest="camera_index", type=int,
default=-1,
help="Index camera OpenCV (-1 = auto built-in Mac)")
p.add_argument("--pose-device", default="mps",
choices=("cpu", "mps", "cuda:0"),
help="Device YOLO inference (default mps)")
p.add_argument("--det-thresh", dest="det_thresh", type=float,
default=0.15,
help="Multi-HMR detection threshold (default 0.15)")
p.add_argument("--nms-kernel-size", dest="nms_kernel_size", type=int,
default=5,
help="Multi-HMR NMS kernel (odd, >=3; default 5)")
p.add_argument("--motion-gate", dest="motion_gate", type=float,
default=5.0,
help="Skip Multi-HMR si diff caméra <X (0-255 ; "
"0=desactive ; default 5.0)")
p.add_argument("--detrpose-model-size",
choices=["n", "s", "l"],
default="n",
help="DETRPose model size (default: n)")
p.add_argument("-v", "--verbose", action="store_true")
opts = p.parse_args()
logging.basicConfig(
level=logging.DEBUG if opts.verbose else logging.INFO,
format="%(asctime)s %(levelname)-7s %(name)s%(message)s",
datefmt="%H:%M:%S",
)
app = NSApplication.sharedApplication()
app.setActivationPolicy_(NSApplicationActivationPolicyRegular)
delegate = AppDelegate.alloc().initWithOpts_(opts)
app.setDelegate_(delegate)
# Ctrl-C en console termine proprement
signal.signal(signal.SIGINT, lambda *_: app.terminate_(None))
app.run()
return 0
if __name__ == "__main__":
sys.exit(main())