Same -/+ stepper controls duplicated on the MATRICE tab; the wiring is
factorized into bindStepGroup so both tabs' sliders and readouts stay
in sync while driving the same OSC addresses.
The web surface had no master control at all (SC-only via
/control/doMaster 0..2). Adds the slider next to Tempo with 0.05
steppers and a live readout, same pattern as the tempo control.
One-BPM steps beside the tempo slider (fine adjustment is hard on a
touch slider mid-performance) plus a live numeric readout; both paths
share the same /launch/tempo send.
User request live: right-hand index pinch fires breakdown instead of
relanceEvolve — SC default, generator default, web defaults, the 196
presets and the test pins all updated. relanceEvolve stays in the
assignable vocabulary.
7 style families x 8 melodic roles x 6 composition stages. Chord
progressions (i-VI-III-VII etc.) aligned across all voices per colour;
call-response via +4-step offset for melody/stab vs arp/acid. Pure
deterministic transforms: rotate, ornament, climax (+7 oct lift),
sparse, resolve-to-anchors. New MEL[role]["composed_<style>"] entries;
existing hand-authored families untouched.
SC's mod cache keeps the last received value when a hand vanishes, but
the gauge markers and wrist cross disappeared — the screen no longer
showed what the matrix was hearing. Markers now hold the last known
position (dim conf 0.45) and go full-bright when the hand is live.
Context: live sets needed variety; the single *_live.matrix per style
gave no choice during a performance.
Approach: 28 styles x 4 variants each produced by
generate_live_variant_sc() with deterministic era kits and family
overrides.
Changes:
- Add *_live-1..4.matrix for every style (112 files)
- v1 house/M1: m1_organ/m1_piano/dx7_epiano on chord/bells/melody
- v2 acid/hoover: tb303 on acid/arp, hoover on lead/reese
- v3 virus/nord: virus_saw on lead/arp, nord_lead on melody/arp
- v4 raw 909/rave: tr909_snare on snare, rave_stab/brass_stab on stab
- Each variant uses a distinct pattern family and busy-block profile
Impact: 196 total .matrix presets; SC engine discovers all via ls scan.
Add hoover/nord_lead/virus_saw to arp+lead, brass_stab/rave_stab to
stab, m1_piano/m1_organ/dx7_epiano to chord+sub+bells+melody,
reese_juno to sub+pad+reese, tr909_snare to clap+snare.
Mirror changes in web UI MATRIX_INST_CHOICES.
Metal lines are 1px and the skeleton was barely visible in
performance. Each ARKit bone is now drawn in concentric offset passes
(SKEL_LINE_WIDTH, default 4 approx px; 1 restores the thin look).
VIZ_DEFAULT_MODE=<name> boots the viz on that mode and disables the
auto-openpos switch entirely, so the mode is truly always active (user
request: the t-key voronoi effect permanent). Concert launcher defaults
to voronoi; "auto" keeps the historical behavior.
The module doc still mandated landscape (2026-06-30 ARKit constraint);
portrait was validated live 2026-07-02 and is now the concert default.
Document the rotation split (video ccw, 2D points already upright).
The X/Y voice mods were anchored to the 21-landmark centroid, so
opening or pinching the fingers shifted the mod values (gesture/mod
cross-talk). cx/cy now track the WRIST (kp 0). The renderer draws a
cross marker at each wrist on the video whenever features exist, so
the control point stays visible even for far hands the overlay
filters out (user request live 2026-07-02).
The deepen-bindings commit accidentally staged the preset generator's
__pycache__ directory (compiled .pyc artifacts). Untrack the files and
add a local .gitignore so preset regeneration never stages them again.
User request live: more audible depth on the hand X/Y mods. All 1848
lHand/rHand X/Y bindings across the 84 presets raised from 0.40-0.50
to 0.70 (per-voice editable in the web UI as before).
qsdfghjkl fired auto-synth scenes on stray keypresses during matrix
performance — same gate as wxcvbn (VIZ_AUDIO_KEYS=1 re-enables). The
m matrix-mode shortcut (scene stop + openpos) stays active.
m now stops the auto data-synth scene AND forces the openpos viz
(matrix unaffected — it is not a scene). Repairs the two test pins the
frozen pinch defaults broke (ext_min 1) and makes the skeleton path
safe for __new__-built renderers (hand gate getattr).
Live session outcomes: (1) pinch thresholds validated for inner
fingers frozen as defaults (ratio_on 0.50, margin 0.05, ext_min 1);
(2) panel frames + X/Y gauges now permanent (no early-return when
nothing is detected); (3) wxcvbn source-bundle shortcuts gated off by
default (VIZ_SOURCE_KEYS=1 re-enables) — a stray key mid-performance
switched the audio scene.
Add gesture_quality() pure helper (hand_display.py) that maps a slot's
plausibility, facing, and proximity to a score in [0,1]. Renderer uses
it to modulate panel frame brightness (0.25+0.75*q) and stroke count
(1..3 passes at q≥0, ≥0.5, ≥0.85 / status==3). Status still drives
hue (pid 7/8/9). New gesture_slot_quality field in State written
alongside gesture_slot_status each tick.
Add gauge_segments() for X/Y position gauges around each panel: a
horizontal rail below and a vertical rail on the outer side, each with
a bold two-tick notch at the hand's normalised cx/cy (mirror-aware for
X). Dim rails (conf 0.25) when slot absent. Renderer reads hand_feats
L/R directly from state.
11 new tests (6 gesture_quality + 5 gauge_segments); suite 430 passed.
Add gesture_slot_status [0..3] to State (absent/detected/armed/engaged).
PinchDetector.engaged_slots() exposes per-slot pinch state. Publisher
writes status after each tick via _update_gesture_slot_status().
Renderer draws panel frame always: dim (pid7 conf=0.3) when absent,
normal (pid7) detected, pid8 armed, pid9 double-stroke engaged.
Add hand_facing() palm-spread metric (dist(kp5,kp17)/hand_size).
Gate GestureSlotStabilizer slot activation on facing >= HAND_FACE_MIN
(default 0.5) with 0.8x hysteresis; side-on palms no longer arm
pinch/strike. Add VizConfig.hand_face_min + HAND_FACE_MIN env var.
Update test fixtures: _near_hand() now spreads kp[5]/kp[17] so
existing tests keep passing through both near+face gates.
Update globalActions in all 84 .matrix preset files from old
default to: [\fill, \doubleTime, \washReverb, \muteDrums,
\relanceEvolve, \halfTime, \next, \muteMelo].
Add \doubleTime and \halfTime as drum-only subdivision feel toggles
(\matDrumFeel \normal/\double/\half, mutually exclusive). Engaging a
feel re-sources only drum voices (matLastColor reset + matApplyBar).
The \relanceEvolve composite action seeks to bar 0 then fires
matRelanceEvolve. \matDrumFeelPush sends /matrix/drumfeel on toggle
and on preset load (resets to \normal). OSCdef mat_drumfeel_get added.
Drum feel stretch applied in matVariationOverlay + matColorStepPattern
via txStretch local var; non-drum voices use matTxStretch unchanged.
Web vocab: relanceEvolve added, evolve relabelled Évolue.
Test file extended with sections R (feel toggle) and S (relanceEvolve).
Live: skeleton still misrotated after the conjugation fix. Both app
2D streams (ARKit skeleton2D + Vision hands) arrive already uprighted
in portrait — the interface orientation drives the projection. Points
are no longer rotated by default; VIDEO_ROTATE only turns the video.
IPHONE_SKEL_ROTATE / IPHONE_HANDS_ROTATE remain as scan knobs.
Live report: skeleton straight but hands off the wrists. The app runs
Vision with a hardcoded .right orientation (ARBodySession.swift:261),
so hand coords live in an uprighted frame while ARKit skeleton2D lives
in the raw sensor frame. Hands get their own IPHONE_HANDS_ROTATE knob
(default none); VIDEO_ROTATE now only rotates the skeleton2D points.
SC handlers (edge detector, tick, OSCdefs, pose/hands calls) and web
UI (Poings section, sensibilite poing slider, fistprogress indicators)
removed. Python FIST_ENABLE=0 kill-switch was already in effect.
Old .matrix files with fistThresh key load without error (key silently
ignored). Test harness Q section removed from test_global_actions.scd.
Live report: video straight but skeleton rotated wrong. Flip and
90-degree rotation do not commute — the video is flipped THEN rotated
while points are rotated THEN display-flipped. Points now use the
mirror-conjugated mode (cw/ccw swapped under mirror), proven by a
composition test.
Portrait mode validated live: VIDEO_ROTATE now also rotates the
iPhone 2D landmarks (skeleton2D + Vision hands) at the decode layer so
overlays/panels/gestures stay aligned with the rotated video. ARKit
per-finger joints are masked from the wireframe (unreliable 2D, seen
as false hands); the display shows the filtered Vision hands instead.
Apply hand_plausible + HandPersistenceGate in _step_stab before
route_hands so gestures and /pose/hands features only see quality-
gated hands. Cache per-tick validated set (_tick_hands/_tick_chir/
_tick_mirror) so all three emitters share one acquisition. Gate
stepped at 30 Hz publisher cadence (audit R6). Non-protocol hands
(raw numpy arrays) pass through via try/except for backward compat.
The body-mesh triangles (solid cyan blob reported live) were the one
surface no filter covered: under iphone-usb persons_body is the ARKit
conversion with c=1.0, so garbage/stale joints painted full triangles.
Skip BODY_TRIANGLES whenever the ARKit topology is present (webcam
MediaPipe path unchanged); the body stays the freshness-gated ARKit
wireframe.
config.py: PINCH_EXT_MIN docs said "min frames for debounce"
— it is min extended fingers required to fire; fix wording.
HAND_PERSIST_FRAMES said "persist after disappearance" — it
is frames required before a hand is drawn; fix wording.
CLAUDE.md: MEDIAPIPE_DELEGATE default was "cpu" — real
default is "gpu" (config.py); fix the RC0.1+ table row.
Add missing PINCH_ENABLE / VIZ_HUD / CONCERT_MIRROR rows
to the main env-vars table.
iphone_usb_source.py: after stream ends, clear
persons_hands_iphone, persons_hands_chirality, and
persons_hands (when write_hands=True) under state.lock()
to prevent renderer display freeze.
test_iphone_usb_reconnect.py: add
test_hands_cleared_on_stream_drop to verify lists are
empty after simulated USB disconnect.
renderer.py: read ARKIT_FULL_SKELETON via VizConfig at
initWithState_ instead of module-level os.environ; store as
self._arkit_full, use getattr fallback for __new__ test bypass.
main.py: pass cfg.iphone_osc_port to IphoneOSCListener
instead of hardcoded 57128; log actual port.
hand_slots.py: import _finite/_coord/_clamp from hand_features
at module level (acyclic: hand_features imports hand_slots only
lazily inside step()); delete the duplicate copies.
User request live 2026-07-02: disable the fist gesture completely
(false held-fist actions during play). fist is forced to 0.0 at the
extractor unless FIST_ENABLE=1; continuous features unchanged. SC
held-fist actions (reload/next) go silent without touching SC.
iphone-usb path: camera decode + ARKit conversion is cheap (no
MediaPipe inference on Mac). Frame-based gesture calibrations
(debounce/hold/grace/persist) assume 30 Hz. Keep 18 Hz for the
webcam MediaPipe path (M5 CPU protection).
Tests: period formula + MultiWorker.__init__ signature check.
Read state.mirror_2d in _pick_hands and thread it through
_step_stab (route_hands mirror=) and _emit_hands
(HandFeatureExtractor.step mirror=) so the cx fallback slot
assignment matches the renderer panel layout on chirality dropout.
Add mirror kwarg to HandFeatureExtractor.step plumbed to route_hands.
Tests: extractor-level mirror flip + publisher _emit_hands mirror_2d.
Detection v3 review follow-ups: pin hand_persist_grace/near_off/
hold_frames defaults; use the module-level route_hands (restores the
monkeypatch seam, drops the dead local alias); document the grace
count-inheritance window in the gate docstring.
hand_slots module docstring still claimed inverted chirality (default
1) after the live A/B validated chirality correct (default 0); align
it. Pin the MESH_RIG ne0 bool convention (chantier 5 review minor).
61 tests: all VizConfig defaults, four bool conventions,
numeric cast fallback, monkeypatch round-trip, and Part B
gating (_ensure_model never called under --iphone-usb).
Fix hand_swap_lr from_env default: was True, now False.
Add data_only_viz/config.py: frozen dataclass VizConfig with typed
fields for all 56 env reads in data_only_viz. from_env() reads
os.environ at call time — no module-level singleton, test-compat.
Migrate action_head_pub._init_finger_piano and renderer.initWithState_
to consume VizConfig. Keep renderer.py ARKIT_FULL module-level (test
uses MetalRenderer.__new__ without calling initWithState_).
HAND_SWAP_LR default changed False (was True): Vision chirality
validated correct live 2026-07-02; swap knob now defaults off.
Review C1 (Critical): /pose/hands per-voice L/R mods routed without
the HAND_SWAP_LR swap while pinch/strike used it — the two paths
contradicted each other on an inverted-chirality source. step() now
takes swap, the publisher passes its env-read value, two tests pin
the contract. Also hoists the route_hands imports (M5) and fixes the
stale slot comment (M1).
New module hand_slots.route_hands():
- Chirality path: iPhone ARBodyTracker Vision 0/1 -> L/R slots,
alignment preserved before invalid-hand filtering.
- Fallback: ascending cx sort (mirror-aware).
- swap=True: swaps the two slots at the very end (safety knob,
HAND_SWAP_LR=1, if a source inverts Vision chirality).
- near_min gate: drops slots below wrist->middle-MCP distance.
18 new tests covering all branches.
Extract apply_skeleton_joints() as a pure module-level function that
copies prev_arr (or zeros) and writes a new array, never mutating the
old one. The TAG_SKELETON handler now stores the new array under the
state lock, so readers that captured a reference to the previous array
(multi.py shallow-copy + ArkitFuse.apply outside the lock) always see
a complete, consistent snapshot — eliminating the B4 tearing race.
6 new tests cover: shape, identity (is not), value preservation for
invalid joints, value application for valid joints, wrong-shape reset,
and no-mutation of prev_arr.
Review follow-up: make the v2 filter's rejection of legacy 0.707
diagonal fixtures explicit instead of silently retuned, and document
the close-up caveat on max_len (ARKIT_BONE_MAX raises it live).
HandPersistenceGate (min_frames=3, radius=0.15) suppresses ghost
hands that flash for fewer than min_frames consecutive frames.
Gate is called once per frame in the renderer on persons_hands;
result filters both the video overlay loop and the side panels.
hand_plausible now rejects wrists outside [-0.1, 1.1] (out-of-frame
ghost anchor check). panel_segments gains a conf_min param.
Env knobs read at renderer init (single parse, no per-frame cost):
HAND_CONF_MIN (default 0.45, raised from 0.3)
HAND_PERSIST_FRAMES (default 3)
ARKIT_BONE_MAX (default 0.5, passed to arkit_segments)
New tests: 12 gate tests + 4 wrist-bounds tests.
arkit_segments now accepts max_len and bounds params.
Bones whose endpoints fall outside [-bounds, 1+bounds] or whose
length exceeds max_len are skipped (tracking-loss garbage).
Defaults: max_len=0.5, bounds=0.1. Renderer will pass
max_len=ARKIT_BONE_MAX explicitly (Part 3).
Tests updated to use realistic bone lengths; new tests added
for the out-of-bounds and overlong-bone cases.
NSEvent keyboard monitors were placed at the END of
_start_pose_worker, unreachable: every backend path returns
early (lines 369/384/399/413/440) so monitors were never
installed. ESC, space (concertNext) and keymaps were silently
dead in all real concert paths.
Move the local + global monitor installation into
applicationDidFinishLaunching_, after window focus setup,
non-headless path only. The --multi-hmr headless early-return
(line 95) is unaffected.
_read_sources used .get("pid")/.get("v3d") (dict API) but
multi_hmr_worker emits SMPLXPerson dataclasses (field: vertices_3d).
AttributeError was swallowed by LOG.exception at 30 Hz.
- Dual-read via isinstance check: dataclass path uses p.pid /
p.vertices_3d / p.expression; dict path keeps p.get() unchanged
- 3 new tests: real SMPLXPerson emits OSC, expression field accepted,
legacy dict format still works
persons_arkit_2d_t was written by IphoneUSBSource but never
read. A USB drop left the frozen skeleton on screen indefinitely.
- Add arkit_2d_fresh(ts_by_pid, now, max_age=1.0) pure helper
in hand_display.py (8 TDD tests, all green)
- renderer._update_skeleton: use_arkit now requires at least one
pid younger than 1 s (replaces bool check)
- Per-pid freshness gate in draw loop skips stale pids
Add panel_rect/panel_segments/panel_frame helpers to hand_display.py
(PANEL_SIDE=0.30, PANEL_MARGIN=0.02, PANEL_INNER=0.10). Segments use
uniform pixel-space scale so the hand aspect ratio is preserved
regardless of view aspect. mirror=True flips X within the panel.
In renderer._update_skeleton add push_panel (no mirror flip) and a
panels block before the segs==0 epilogue: routes persons_hands to
left/right panel via chirality list when aligned, screen-cx fallback
otherwise. Draws frame (pid 7) + wireframe (pid 5 left / 6 right).
Runs on every path including use_arkit. Guards: _mp_bones None, empty
hands, misaligned chirality, height==0.
26 tests pass (10 new for panel helpers).
The AV-Live data-only text HUD hid the lower-left corner of the
performance display and would collide with the new left-hand panel.
VIZ_HUD=1 re-enables it; the H key toggle guards the absent view.
_decode_hands now returns (hands, chirality) tuple instead of a bare
list. Landmark .z keeps the raw wire value; .c = clamp(z, 0, 1).
Chirality byte (1=right, else 0=left) is decoded per hand and stored
in State.persons_hands_chirality under the same lock as
persons_hands_iphone.
ARKit-only-body mode no longer consumes face: the Mac dropped face
entirely (face=0, /pose/mouth gone), so VNDetectFaceLandmarksRequest
and the .face USB frames were pure wasted per-frame compute. Drop the
face request and its send.
The on-device ARView (RealityKit) rendered a black background behind
the SwiftUI skeleton overlay -- it never showed the camera -- so it
only spun up the RealityKit AR render pipeline (the engine:rematerial
console spam) to draw black. Remove the ARView; the screen stays
black + skeleton overlay, identical UX, minus the GPU/thermal cost.
Video (HEVC) and hands are kept: the Mac uses the iPhone video as the
visualizer backdrop and Vision hands drive pinch/fist gestures.
Add camera: ARCamera param to publishUSB and thread frame.camera
through from the session delegate. After the existing .skeleton
send, project all 91 joints to viewport 2D, normalize to [0,1],
and send as Skeleton2DPayload / FrameTag.skeleton2D (tag 6).
Under --iphone-usb, skip all three MediaPipe detectors
(pose, face, hand) and rebuild the MP-33 body from
state.persons_arkit_{2d,joints} via new arkit_body_2d /
arkit_body_3d builders. Add head anchor (ARKIT_HEAD_IDX=51,
needs live verify) as MP33 slot 0. Webcam path unchanged.
Add TAG_SKELETON2D=6, SKEL2D_BYTES=819, decode_skeleton2D() to
iphone_usb_bridge.py; handler in IphoneUSBSource writing
persons_arkit_2d / persons_arkit_2d_t to state; new state fields
in State dataclass. TDD: test_arkit_body.py (round-trip + wrong
length → None). Import check verified.
Add FrameTag.skeleton2D (raw 6) and Skeleton2DPayload for
2D-projected ARKit joints: 91×SIMD2<Float> + 91 valid bytes,
big-endian wire format mirroring SkeletonPayload style.
Note: raw value is 6 (not 5 as drafted) because face=5 is taken.
Add matFistThresh state (default 0.15) with localStorage persistence.
Wire a "Sensibilite poing" slider in Reglages, synced via fisthresh.
Add two fist indicator rows in #morceau-fists driven by fistprogress.
Context: Task 1 wired fistThresh into the SC matrix engine (default
0.15); the generator was not yet emitting this field, so every
.matrix was missing it and the engine fell back to its hardcoded
default.
Approach: TDD — added a failing check to test_global_actions.py,
then edited to_sc so pinchHoldMs ends with a comma and fistThresh
0.15 follows as the last field before the closing paren.
Regenerated 84 presets twice for determinism.
Changes:
- generate_presets.py: pinchHoldMs now ends with comma; fistThresh
0.15 emitted as the final key in every preset's SC Event literal
- test_global_actions.py: EXPECT_FTH block checks to_sc output and
all 84 on-disk .matrix files
- *.matrix (84 files): regenerated with fistThresh field
Impact: all four gates green (GLOBALACT-PY, test_sections,
test_families, GLOBALACT SC); fistThresh is now a first-class
preset field readable at load time by ~matLoadFile.
Held closed-fist (low openness, hysteresis+presence guard) per hand
fires an action after ~matPinchHoldMs: left hand reloads the current
preset, right hand advances to the next. Progress emitted on
/matrix/fistprogress.
- ~matFistEdge edge detector (engage/release/absent + hysteresis)
- ~matFistTick fires once past hold threshold (guarded by ~matFistFired)
- ~matFistReloadCurrent and ~matNextPreset wired to hand 0/1
- /matrix/fisthresh[/get] OSCdefs + ~matFistThreshPush
- fistThresh saved and restored per .matrix file
- 30 Hz poller now calls ~matFistTick alongside ~matPinchTick
- Harness group Q: A-Q all green
Add top-level 'pinchHoldMs': 300 to every generated .matrix so the SC
engine (Task 2) reads it instead of its hardcoded default.
- generate_presets.py: globalSettings ends with ',' and
'pinchHoldMs': 300 is the last field before the closing ')'
- test_global_actions.py: EXPECT_PHMS check added (TDD red then green)
- 84 .matrix files regenerated (28 morceaux x intro/live/outro)
PinchDetector now emits state=1 on engage and state=0 on release
(and on hand-disappear). send_pinch appends the state as the 4th
OSC arg so SC can distinguish hold from lift. Launcher switches
from FINGER_PIANO to PINCH_ENABLE to gate only the pinch detector.
Expand globalActions from 4 to 8 slots:
[\fill, \drop, \breakdown, \evolve,
\muteDrums, \muteMelo, \washReverb, \next]
Drop openThresh/closeThresh from globalSettings (now
engine-side constants via Task 1). Regenerate all 84
.matrix files deterministically.
All gates: GLOBALACT-PY PASS, TEST PASS x2, GLOBALACT PASS
Replace open/close hand detection with 8 pinch slots (hand*4+finger-1).
Global actions array grows from 4 to 8 entries (L/R x index..ring).
Remove ~matHandFlip, ~matHandState, openThresh/closeThresh entirely.
Add the Reglages group to the morceau panel: two threshold sliders
(open/close, the close one being the user-requested "sensibilite"),
a transient-duration slider (1-8 bars), and a breakdown kept-voices
checkbox grid covering all MATRIX_VOICES.
Wired over /matrix/globalsettings with inbound sync and onOpen
get-request. Imports consolidated into a single coherent block;
no duplicate identifiers. DOM container added to index.html;
styles appended to control.css.
Context: Phase 2 Task 1 made the SC engine load a globalSettings Event
from .matrix files (openThresh/closeThresh/transientBars/breakdownKeep)
and default it when absent. Generator was not yet writing the field.
Approach: append globalSettings after globalActions in to_sc() return
string; extend test_global_actions.py with EXPECT_SET check against
both the to_sc sample and all 84 .matrix files on disk.
Changes:
- generate_presets.py: to_sc now ends with globalActions (comma) then
globalSettings Event with default values matching SC defaults
- test_global_actions.py: add EXPECT_SET literal + two check() calls
- *.matrix (84 files): each preset now carries the globalSettings field
Impact: every preset ships with openThresh/closeThresh/transientBars/
breakdownKeep defaults; SC engine reads them on load; no more implicit
fallback needed at runtime for any preset in the library.
When the evolve toggle is OFF, ~matRelanceEvolve now snapshots
~matBaseGrid, mutates once, then restores it to nil. When ON it
keeps the persistent base and re-mutates from it. Fixes stale-
snapshot corruption across preset loads. Case I added to harness.
Add bottom-of-matrix panel with 4 gesture-slot dropdowns wired
over OSC (/matrix/globalaction). State lives in matrix-state.js;
syncs on connect and on incoming OSC echo.
Context: SC engine (Task 2) loads 'globalActions' from .matrix files
and defaults to [\fill, \drop, \evolve, \breakdown] when absent.
Generated presets did not write this field, causing implicit reliance
on the engine default.
Approach: extend to_sc() return string to include the globalActions
key so every preset explicitly carries the default mapping
(LH-open=fill, LH-closed=drop, RH-open=evolve, RH-closed=breakdown).
Changes:
- generate_presets.py: add 'globalActions' line in to_sc() return
- test_global_actions.py: new Python test verifying to_sc() output
and all 84 on-disk .matrix files carry the field
- 84 .matrix files: regenerated with globalActions field present
Impact: presets are self-documenting; SC engine reads the explicit
value rather than applying a silent default.
The live loop tiled the plateau into 4 identical blocks -- full but static.
LIVE_STYLE maps each morceau to a style group; LIVE_VAR holds per-style
profiles; _live_variation recolours active cells in some 16-bar blocks
(rhythmic 2/4, melodic 3 octave) and adds phrase-end fills (colour 6) so the
full loop breathes in a genre-appropriate way -- techno double-time hats +
fills, breaks snare rolls, ambient gentle bell octaves, etc. Active cells
only -> density unchanged, no voice dropped, no intro/breakdown back.
test_sections asserts the live still has no intro/breakdown AND varies (not
4 identical blocks). Regen 84 deterministic, VALIDATE 84/84 + DEFAULTS PASS.
Each morceau's _live section reused the standalone arc, which re-introduced
(near-empty intro -> rebuild -> plateau -> breakdown). Now make_live_continuous
tiles the plateau (P, bars 32-47) across all 64 bars: full groove from bar 0,
no intro ramp, no breakdown -- a clean loop the _intro builds to and the
_outro descends from. Applied post-apply_colors for the live section only via
to_sc(post=); intro/outro unchanged.
test_sections asserts the live is full-from-start + has no breakdown dip.
Regen 84 deterministic, VALIDATE 84/84 + DEFAULTS PASS, test_families PASS.
The SP-A default bound danse->gate on the fx and sweep voices, so the
action-head classifying the dancer as "danse" toggled those voices on/off
even with no fx/sweep cells in the grid -- poses triggering FX that the
user never placed. Pose->voice triggers are now opt-in (default_poses
returns []); continuous pose modulation (default_mods) is unchanged.
Updated test_default_mods.py + the validate_presets DEFAULTS assertion
(fx voicePoses now expected empty). Regenerated 84 presets. VALIDATE
84/84 + DEFAULTS PASS; test_families + test_sections PASS.
The Metal GPU MediaPipe delegate (set as default earlier to fix pose
choppiness) crashes under sustained load: fatal CVPixelBuffer alloc
failure (-6662) in gpu_buffer_storage_cv_pixel_buffer.cc /
Image::ConvertToGpu(), on M1 Max too. The choppiness it addressed was CPU
starvation (baby-brain + model servers), now stopped -- so the CPU
delegate has headroom and is both fluid and stable. Default to cpu,
overridable via MEDIAPIPE_DELEGATE=gpu for short sessions.
The iPhone USB source gave up permanently if the first connect_device()
failed (app not streaming on :7000 at startup -- phone locked, app not
launched/crashed, USB hiccup): start() returned False and the read thread
(which holds the reconnect loop) never ran. Auto-reconnect only covered a
mid-session drop, never a startup race.
Now start() launches the thread regardless and _run reconnects from the
top of its loop, so the initial connect retries with backoff exactly like
a mid-session drop -- the source connects whenever the iPhone app appears,
no data_only_viz restart needed.
Test: tests/test_iphone_usb_reconnect.py (start() keeps retrying when no
device). The 2 unrelated pre-existing collection errors are unchanged.
Emit /matrix/loaded <name> at each user-facing load point (the /matrix/load
OSCdef, the queued branch of ~matWrapStep firing at the wrap, and the
not-playing fallback in /matrix/loadnext) - not from the shared ~matLoad,
to avoid spamming during the boot autoload. The web fills #matrix-name from
it, so SAUVER updates the loaded matrix in place. For CHARGER SUIVANT the
name tracks the actual swap (at the wrap), never the merely-armed preset.
VALIDATE 84/84, test_matnames PASS, node --check + 29 web tests pass.
Add a SUIVANT button to the matrix persist bar that arms the selected
preset to load at the next loop wrap (/matrix/loadnext), with an armed
pulse + queued-name label driven by the /matrix/queued feedback.
No jsdom unit test: the project's web tests cover pure logic modules, not
main.js DOM wiring (jsdom is not a dependency); the button is verified by
the live smoke. node --check + the existing 29-test suite pass.
Expand the 7 breaks morceaux from the shared _breaks_intro/_breaks_outro
helper into 14 fully-independent arc bodies (one intro + one outro each),
per the "no shared helper" requirement. Each varies by morceau character
(bass_science reese-forward, jungle_amen amen-snare, footwork kick step2,
neuro_reese stab, tribal_perc tom/perc).
test_sections PASS, VALIDATE 84/84.
Wire the hand-authored families into the 84 presets and make the launcher
boot the live section.
Changes:
- patterns_for: strip the _intro/_live/_outro suffix before the
PRESET_FAMILY lookup, so all 84 section presets resolve to their
morceau's family.
- PRESET_FAMILY: retarget mel families to the new banks where they fit -
house (detroit/rave/trance/melodic/electro), psy (hypnotic/psytrance/
minimal), dub (dub_techno/melodic_deep), ambient (ambient_intro); drum
families now style ride/shaker/clap/perc per dfam.
- launch_concert.sh: MATRIX_PRESET default -> techno_drive_live.
Impact: genre presets now use the authored families. test_families PASS,
test_sections PASS, regen deterministic, VALIDATE PASS 84/84.
Hand-write the intro and outro arc for each of the 28 morceaux (56 new
independent arc functions). intro = voices enter across the 64 bars
(sparse->full, capped at the plateau with a light O lead-in into live);
outro = voices exit (full early, dropping out toward the break/outro).
Approach:
- one arc_intro_<m> / arc_outro_<m> per morceau, registered in
INTRO_ARCS/OUTRO_ARCS; composed from the primitive fill/hit grid API
over the I/B/R/D/P/K/O section codes.
- the 7 breaks morceaux share a _breaks_intro/_breaks_outro shape
parameterized per morceau (deviation from "fully independent" - flagged
for review).
- test_sections.py gate: intro builds, outro breaks down, live densest -
all 28 PASS.
Impact: 84 presets regenerated, VALIDATE PASS 84/84. Section feel is
shape-correct; musical review by ear pending.
Add explicit 6-colour style families to the pattern bank so genre presets
feel musically distinct (merged SP-C2). All colours hand-written following
the role convention (base/busier/broken/double/sparse/fill), velocity-
humanized; existing families and every default stay untouched.
Changes:
- MEL: dub (reese/pad/chord/bells/melody), psy (sub/reese/arp/lead/stab),
house (pad/chord/stab/bells/lead/melody), ambient (pad/bells/melody/
sub/reese) - 21 families x 6 motifs.
- DRUMS: percu-voice + style-completion families for ride/shaker/crash/
rim/fx/sweep/perc (techno/house/breaks/dnb/trance/ambient).
Impact: vocabulary for the genre presets; wired per preset in a later
task. test_families.py PASS (validity + >=4 distinct + coverage).
Each morceau is now decomposed into three sections (intro/live/outro)
as three separate .matrix files, replacing the single-grid presets.
Approach:
- MORCEAUX: lazy view over PRESETS; live = the existing arc, intro/outro
resolve from INTRO_ARCS/OUTRO_ARCS registries (fallback to live) so the
per-morceau arcs can be populated anywhere at module level.
- main() emits <morceau>_<section>.matrix (28 x 3 = 84).
- test_sections.py: section-shape gate (intro builds, outro breaks down,
live densest) - currently RED on outros (placeholders) to drive the
hand-written arc tasks.
- validate_presets.scd: count 28 -> 84, DEFAULTS check on peak_drop_live.
Changes:
- generate_presets.py: MORCEAUX/_sections/_Morceaux, main() loop.
- removed the 28 old single-name .matrix; wrote 84 section files.
Impact: structural base for hand-authored intro/outro arcs (Tasks 3-9)
and the new pattern families (Task 2). VALIDATE PASS 84/84.
Busier/sparse/broken in derive_drum_colors and derive_mel_colors
now operate on the base's own rest/hit lists, not fixed i%4
positions. Stab (all hits at i%4==2) went from 4 distinct colours
(busier==broken==base, sparse EMPTY) to 6. Test strengthened with
distinctness and sparse-not-empty asserts for stab/kick/hats/acid/
shaker/pad. 28 presets regenerated; determinism confirmed (md5 zero
diff on two consecutive runs). VALIDATE PASS 28/28.
Replace weak != property checks with equality assertions against
derive_drum_colors / derive_mel_colors. The old checks passed even
without wiring; equality fails if patterns_for reverts to raw authored
defaults.
Wire derive_drum_colors/derive_mel_colors into patterns_for:
default family colours 2-6 are now derived from colour 1
(busier/broken/double/sparse/fill); authored style families
pass through unchanged. Remove dead _pick/_DRUM_VELS.
Regenerate 28 .matrix files; sclang VALIDATE PASS 28/28.
Fix 1: ~matClear now mirrors the zero grid into ~matBaseGrid when
evolving, preventing resurrection on the next ~matEvolveStep cycle.
Fix 2: ~matSetEvolve.(true) guards the base snapshot with a
not-yet-evolving check; re-enabling while already evolving no longer
overwrites the original baseline.
Fix 3: ~matLoadFile now calls ~matEvolvePush after load so
reconnecting web clients see the correct toggle and rate.
RED->GREEN on two new test assertions (test_matrix.scd).
Position sources (lHandX/Y, rHandX/Y, bodyX/Y, headX/Y, bodySym)
now default to 0.5 (centre) when absent from cache instead of 0.
Magnitude sources (speed, accel, openness, span, danse, depth,
mouthOpen) keep 0 as correct rest. Fixes pitch-flat-at-boot-without-pose
and matching meter skew in matModValuesPush.
Regenerate 28 .matrix presets with role-adapted voiceMods/voicePoses:
soloists (acid/arp/lead/stab/bells/melody) get hand bindings (rHandX/Y,
lHandX/Y); drums get body bindings (bodyVitesse/bodyX); fx/sweep get
danse->gate pose. Update validate_presets.scd to assert new scheme via
peak_drop (acid 4 hand bindings, kick 2 body, fx danse->gate).
Remove 4 SC functions (matSetColorMod/Pose, matColorMod/PosePush),
2 web OSC handlers (/matrix/colormod /colorpose), and the test
blocks that called them. Superseded by per-voice system; OSCdefs
were already gone. Legacy field read paths and colorDef fields
preserved for preset migration.
/pose/finger and /pose/pinch carry a leading pid at msg[1]; hand is
msg[2] and finger is msg[3]. Previous code read pid as hand and hand as
finger, making 18/21 curated events unreachable.
Add OSCdef-level spy assertions in test_matrix.scd that would have
caught this: RED against old indices, GREEN against the fix.
Replace the old per-colour single-mod row in the MOD/POSE tab with a
per-voice additive list (matVoiceMods) and a curated pose-event
dropdown (matVoicePoses), both independent of cdColor.
Add OSC handlers for /matrix/voicemod and /matrix/voicepose echoes.
Update openFor to request voicemod/get + voicepose/get on expand.
Extend colordefs-parse test stub to export all new matrix-state symbols.
Add PoseSoundBridge.send_mouth(pid, mouth_open) mirroring send_kin
pattern; call it in ActionHeadPublisher._tick after send_kin using
the already-in-scope `mouth` scalar; add route to module docstring.
3 unit tests (address, VJ mirror, boundary values) all green.
- ~matSave payload gains voiceMods/voicePoses keys (one per voice)
- ~matLoadFile restores new keys; collapses legacy per-colour mod/pose
into voice lists (dedup) when new keys absent
- push ~matVoiceModPush/~matVoicePosePush for all voices after load
- round-trip test in test_matrix.scd (RED->GREEN)
Replace mat_colorpose/mat_colorpose_get OSCdefs with per-voice
mat_voicepose/mat_voicepose_get. Rewrite matPoseFire to iterate
matVoicePoses[name] using current bar color. Add matSetVoicePose,
matVoicePosePush, matEventFinger/Pinch mappers, mat_ev_finger/pinch
OSCdefs. Old color-pose functions left intact.
SC emits /matrix/stephead s (0-15) every 16th note via
~matStepRoutine (started/stopped with ~matPlay/~matStop).
Web highlights .seq-cell or .pr-cell[data-step] with
.playhead class; CSS adds white outline + brightness.
SC ~matColorDefPush sends 8 fields per colour (stretch, octave,
amp, inst, cutoff, pan, res, rev). The JS receiver used stride 6,
so colours 2-6 parsed from shifted indices and res/rev were never
read — edits snapped back on every echo.
- Extract pure parseColorDefs(args) (stride 8, all 4 sentinels
decoded: cutoff=-1, pan=-2, res=-3, rev=-4 → null); export it.
- Rewrite handler to use parseColorDefs; preserve mod/pose/steps.
- Fix stale CSS comment: 16x32 → 22x64.
- Add test/colordefs-parse.test.mjs (8 cases, guard against drift).
- node --test tests/*.mjs: 20/20 pass (was 12).
Split renderPattern into renderStepsTab, renderSynthFxTab,
renderModPoseTab behind a 3-tab bar. activeTab module state
resets to steps on open; colour selector stays above tabs.
Per-voice editor-row accent via borderLeftColor; mvoice-label
cursor moved to CSS.
Replace fullscreen #colordef-modal with a .voice-editor-row that
expands inline under each voice in the matrix grid. One voice open
at a time; ▷ toggle rotates to ▼ when expanded; clicking label or
button calls toggle(vi, containerEl). Editor content (colour 1-6,
var row, steps, mod, pose, AUDITION) is identical, only location
changes. CSS: drop .cd-modal/.cd-panel, add .voice-editor-row /
.voice-expand-btn.
Install --m1..--m6 on :root from PALETTE in main.js at bootstrap.
Replace triplicated hex in mcell/cd-row/seq-row with var(--mN).
Remove local COLORS array from renderColorSel; use imported PALETTE.
matrix-state.js stays DOM-free.
Deep refactor of the 28 matrix presets for style fidelity:
- fix 6 invalid instrument assignments (silent lp_ fallback)
- per-style melodies via colorDefs steps on 7 mono voices
- exploit all 6 variation colours (build rolls, drop lift,
half-time + octave-down break) instead of monochrome 1
- restore tribal_perc; validate kits against matInstChoices
Validated: 28/28 load via matLoadFile, all step patterns
stream clean on the per-color engine.
Add null guard to /matrix/colormod and /matrix/colorpose WS
handlers to prevent TypeError when legacy browser cache holds
null color slots. Remove dead CSS rules for .cd-tabs, .cd-tab,
.cd-tab.on, .cd-pane[hidden].
Rewrite the 16 classic presets and add 12 new (electro/EBM/
synthwave, DnB/jungle/footwork, melodic techno/progressive) as
22-voice x 64-bar arrangements with section arcs (intro/build/
rise/drop/plateau/break/outro) and per-preset machine kits
(909/808/303/juno/supersaw/minimoog/dx7/sp1200...). The generator
now emits the Event (grid+instruments) format. All load + valid.
Wire 6 new tracks (melody, chord, fx, snare, crash, shaker) into
~matVoices and extend the grid to 64 bars. Per-voice dicts (mod
targets, neutral cut, instrument choices) gain the new voices.
~matLoadFile auto-migrates any legacy grid (tile bars, pad voices)
so the old 16x32 presets still load. Web grid/mixer/timeline +
handlers de-hardcode 16/32/512 to MATRIX_VOICES.length/MATRIX_BARS.
Headless test assertions made dimension-dynamic.
The rev (reverb send) modulation target was dropped from ~matModPairs
when factoring capture effects. This branch now emits a 0..1 reverb
send scaled by mod depth, matching the previous engine behavior.
Add synthdefs (lp_snare, lp_crash, lp_shaker, lp_fx) and base
Pdefs (lp_melody, lp_chord, lp_fx, lp_snare, lp_crash, lp_shaker)
for 6 new matrix tracks. Foundation only -- not yet wired into
matVoices/the grid (the 22-voice x 64-bar core change + preset
migration + web overhaul comes next).
Web-researched the iconic machines per style and added 7 more:
Minimoog (ladder bass/lead), Roland JP-8000 supersaw (trance/
progressive), Juno-106 (chorus pad), Yamaha DX7 (FM), E-mu
SP-1200 (12-bit drum crunch), Akai sampler (lo-fi grit) and
Roland System-100. Standard UGens only (MoogFF/Latch/DelayC).
Wired per voice into the choices + web mirror; all kit/test
instrument targets kept in the choice lists.
Replace pose: [] with pose: Array.new to ensure fresh arrays per color.
Guard nil action in ~matSetColorPose with default \trigger.
Add test assertions for dedup, echo, and out-of-bounds safety.
Add faithful-ish TR-909 (kick/clap/hat/ride), TR-808 (kick/clap/
hat/cowbell), TB-303 (acid), Korg MS-20 and Roland SH-101 as extra
matrix instrument choices. All note-terminating (perc+doneAction
or gated ASR), Pbind-safe, with pan + reverb send; standard UGens
only. Wired into ~matInstChoices and the web mirror per voice.
Context: main added a richer capture-modulation palette (more mod sources
and targets in matrix.scd, control.js, launchpad) in parallel with the
complete per-instrument editor build. Both touch the mod/matrix surfaces.
Approach: git auto-merged cleanly (disjoint hunks); validated semantically.
Changes:
- matrix.scd unions the new mod-source/target palette with the step-sequence
engine, pose bindings, and colorDefs steps.
- control.js: the modal Assign pane iterates the enriched MATRIX_MOD_SOURCES,
so the new sources appear automatically; row consolidation and the step and
pose tabs are intact.
Impact: the per-instrument editor (instrument, colors, 16-step per-color
sequencer, capture assignment with the richer palette, pose bindings) coexists
with main's mod palette. SC test PASS, P:0 B:0, node --check OK.
Add horizontal position sources (bodyX, lHandX, rHandX) and make
pan + a per-voice reverb send (rev) modulatable on every voice,
plus cutoff on perc and reese. All lp_ synthdefs gain pan + rev
(send to the reverb bus, nil-safe); perc/reese gain a cutoff
control that keeps their freq-tracking timbre when unmodulated.
matrix.scd mod sources/targets/neutral-cut + a rev modPairs
branch (in the color-editor overlay); web mod mirror updated.
Consolidate per-row instrument select and mod controls into a
tabbed modal (Instrument, Colors, Sequence, Assign, Poses tabs).
Remove .minst/.mmod/.minst-head/.mmod-head from matrix rows;
.minst-edit stays as modal trigger. WS echo handlers repointed
to modal elements (#cd-inst-sel, .cd-mod-src/tgt/dep), updating
JS state regardless of modal visibility. Steps/poses panes are
empty stubs, requesting /matrix/steps/get and /matrix/poses/get
on open (wired in Tasks 6-7).
Adds nil-guard to ~matPoseFire's per-voice binding loop to prevent
raised when ~matPoseBindings[vi] is nil. Adds gate binding test in
test_matrix.scd to verify \gate action is stored correctly.
- pass base into matVariationOverlay (no double matBaseFor call)
- add nil fallback (? 1.0) on spec[\amp] in mod-amp Pfunc
- push colorDefs for all voices after matLoadFile
- guard Pdef.play to fire once in matAudition routine
- beforeunload stops audition if active (tab close)
- add color-1 identity assertion to test_matrix.scd
Context: main advanced in parallel (octave->freqRatio variation rework,
drone-instrument free fix, mixer/loop/seek) while the per-instrument
color-editor feature was built on an isolated branch. Both heavily edited
~matVariation, producing a semantic conflict.
Approach: keep the color-editor's per-voice editable ~matColorDefs as the
variation source, but adopt main's freq-multiply scheme for octave colors
so editable octaves transpose voices that compute \freq via Pfunc.
Changes:
- ~matVariationOverlay drops \octave; derives freqRatio = 2**octave and adds
a pitched-only \freq multiply, appended with instPair, cdPairs and modPairs
(mod last so live capture overrides static color cutoff/pan).
- ~matVariation reads ~matColorDefs[vi][color] (per-voice editable) instead
of main's hardcoded freqRatio table.
- test_matrix.scd unions the color-editor and loop-region test blocks.
- control.js/control.css/index.html union the mixer/timeline and color-editor
modal additions (auto-merged).
Impact: per-voice editable colors (variation, sound, audition) coexist with
main's freqRatio transpose, per-voice mixer volume, loop/seek, presets, glow,
instrument selection and capture-effect. SC test PASS, P:0 B:0, node --check OK.
The instrument-override / kit feature offered continuous \do_
drones (do_drone, do_body_drone, do_body_gran, do_weather,
do_geo) that have no gate and no doneAction. Driven per-note by
the Pbind they never free and pile up forever (observed 5+
do_body_drone stacked, surviving /matrix/stop).
Drop them from ~matInstChoices, ~matKits and the web mirror;
substitute gated \lp_ sustained voices. ~matApplyInstrument and
~matLoadFile validate against the choices, so saved presets and
kits referencing the drones now fall back to the default voice.
Repurpose AV-Live Concert.app to launch the matrix arranger:
boot SC with MATRIX_ONLY, keep the pose pipeline (feeds the
/pose/* capture-modulation sources), drop the morceau button,
and load a preset (MATRIX_PRESET, default techno_drive) + start
the playhead instead of switching to the concert scene.
Matrix colors 3/5 pushed a dead \octave key; every pitched Pdef
sets \freq directly so the transposition was inaudible. Replace
with a base-\freq multiplier (x2/x0.5) over the Pchain, gated to
pitched voices so drums and instrument overrides keep their pitch.
Add a dedicated sustained \lp_lead SynthDef (gated ASR) and point
Pdef(\lp_lead) at it with \legato so lead notes hold; arp and stab
stay on \lp_pluck.
Merge main (presets, loop/seek, per-voice mixer) into the SDD branch.
matVariation combines main per-voice volume amp with the instPair and
modPairs spread; volume is propagated into the mod amp formula. Both
OSCdef sets (mod/capture + loop/seek) and both test blocks are kept.
Bring the per-instrument mixer (web) and the matrix \amp respecting
per-voice volume (matrix.scd) from origin/main 6a14b3b. Disjoint from
local commits, clean auto-merge.
Add per-voice mod controls (source select + target select + depth range)
sticky-right in each matrix row. WS handlers for /matrix/mod (single)
and /matrix/mods (bulk 48-flat). Persist mod state in localStorage
alongside grid/inst. MATRIX_MOD_TARGETS mirrors SC ~matModTargets.
- Add 78px instHead spacer in renderMatrix() header row so bar
numbers align with voice row cells (fixes 78px left drift).
- Remove dead MATRIX_KITS constant (kit dispatch uses data-kit).
- Guard loadMatState grid assignment with length === 16 shape
check on both object and legacy bare-array branches.
Add per-voice instrument <select> (sticky-left) populated from
MATRIX_INST_CHOICES, kit buttons in the matrix transport, contextmenu
right-click cell-clear, and WS handlers for /matrix/instrument and
/matrix/instruments. Persist instrument choices alongside the grid in
localStorage with legacy bare-array backward-compat in loadMatState.
Bring the editable loop region/seek (matrix.scd) and the web timeline
header (origin/main f9da19f) into the feature branch. Disjoint from the
local commits, clean auto-merge.
Add 32-bar timeline header above the matrix grid in #tab-matrix.
- Tap a bar: sends /matrix/seek <bar>
- Drag across bars: sends /matrix/loop <min> <max>
- BOUCLE COMPLETE button: resets to /matrix/loop 0 31
- .in-loop highlight from /matrix/loop feedback
- .tl-head moves with /matrix/playhead (O(32))
- Pointer Events API (pointerdown/move/up + setPointerCapture +
elementFromPoint) — works on desktop mouse and iPad touch
Bring the MATRIX_ONLY boot gate (origin/main 46c8823) into the feature
branch. With MATRIX_ONLY set, boot skips the concert/body-play scene so
only the matrix arranger plays. Clean auto-merge.
Bring the 16 full-kit creative matrix presets (origin/main 6482baf)
into feat/matrix-instruments alongside the capture-effect design doc.
Disjoint changes, clean auto-merge. Loader reads the preset dir live,
so the new presets are listable without an SC reboot.
Replace the 14 sparse starter grids with 16 hand-authored arrangements
that use all 16 voices and evolve over the 32 bars via sections
(intro / build / drop / break / final) with fills and accents.
- Styles: techno_drive, acid_journey, trance_euphoria, psytrance_roll,
detroit_soul, industrial_grind, hardcore_punk, dub_techno, breakbeat,
minimal_hypno, ambient_intro, peak_drop, rave_stab, tribal_perc,
bass_science, melodic_builder.
- Every voice featured across the set (ride/rim/tom/reese/bells/sweep
now used, not empty); peak_drop activates the full kit on the drop.
- generate_presets.py reworked with a sections DSL; prints a voice-
coverage report (16/16).
Bring matrix file persistence + 14 morceau-inspired presets (origin/main
16adac4) into macm1 main which had the audio-reactive glow + an extra
matrix-instrument-selection design doc. Disjoint changes, clean auto-merge.
Result: glow + persistence + presets + save/load UI all together.
Track last-applied color per voice in ~matLastColor; ~matApplyBar
skips Pdef re-source and re-play when color is unchanged and non-zero,
eliminating envelope re-attacks on sustaining pads/subs/reese/sweeps.
~matStop resets ~matLastColor to -1 so subsequent play re-applies all.
~matVariation now returns nil for color 0 instead of the base pattern.
loadMatState validates each cell is an integer 0..6.
Adds step 6i to data_only/boot.scd: loads matrix.scd after
sections.scd, matching the additive load pattern used by every
other data_only feature file.
Remove the try-catch wrapper added around s.waitForBoot in
sound_algo/engine.scd — it swallowed real boot errors in
production. Move boot-failure tolerance into
sound_algo/tests/test_tempo.scd instead, where it belongs.
data_only/engine.scd was not affected (no wrapper added).
Add /launch/quant selector (0.5/1/4/8 beats, default 1 mes)
to the LIVE tab. Update /armed/ handler to read 4-state int
(0=off 1=playing 2=queued-launch 3=queued-stop), apply .armed
and .queued classes accordingly, sync clipState map. Pad tap
now toggles based on last-known state. CSS blink animation for
.queued (600 ms ease-in-out infinite).
Move ~lp[\clipGen] increment inside the four transitions that
schedule or cancel a closure (s==0 launch, s==3 cancel-stop,
s==1 stop, s==2 cancel-launch). No-op re-taps (s==1/2 on launch,
s==0/3 on stop) no longer clobber the pending generation counter.
Also clamp /launch/quant to clip(0.125, 64) instead of max(0.125)
to prevent a runaway large value from locking out launches.
~toscArmedPush now iterates all 16 pads plus melseq and rhythmseq,
sending 0/1/2/3 (off/playing/queued-launch/queued-stop) from clipState
with fallback to armed membership for concert-driven clips. Test
extended with clipState in dummy ~lp and clipState read assertion.
Add 4-state machine (off/playing/queued-launch/queued-stop) to ~lpArm.
Launch and stop are quantized to ~lp[\quant] beats (default 4) via
clock.schedAbs(clock.nextTimeOnGrid(q)). Per-clip generation counter
cancels superseded callbacks on re-tap. New env fields: ~lp[\quant],
~lp[\clipState], ~lp[\clipGen]. New /launch/quant OSCdef. ~lp[\armed]
semantics preserved so /launch/clear and concert logic keep working.
iPhone Vision hands (on-device, .right upright, y already
top-left, rotation-invariant) are stored in a dedicated state
field and used by the strike detector when fresh (FINGER_SOURCE
auto/iphone/mediapipe). More stable than MediaPipe on the
rotated downscaled video. Adds FINGER_DEBUG strike logging.
Reorganize control surface into LIVE / FX-HARMONIE-CONCERT /
SEQUENCEURS tabs with pure CSS/JS tab switching.
Default tab is LIVE.
Add engine feedback rendering via WS onmessage:
- /armed/<name>: reconcile armed CSS class from engine state
- /sync/beat: 80 ms flash on beat-dot indicator in header
- /sync/rms: drive master VU bar width from float 0..1
- /seq/rhythm/state: load k/s/h step grid + preset name; guard
suppressEmit to prevent echo back as /seq/rhythm/set
- /seq/melody/state: load degree inputs; same no-echo guard
All pre-existing OSC sends preserved unchanged.
data-only boot does not load palette/scales.scd, so
~scaleMinorPent was nil and ~fpResolve crashed on wrapAt.
Embed fallback scale arrays so notes resolve headless.
Add a second osc.UDPPort bound to FEEDBACK_PORT_IN (default
9000, matches ~toscPort in SC) that receives engine feedback
OSC (/armed/*, /sync/beat, /sync/rms, /seq/*/state) from SC
and relays each message to all WS clients using the existing
{address,args} JSON shape. Does not update the snapshot cache
(feedback is transient). Existing data port, control-send
path, and WS logic are untouched.
node --check: SYNTAX OK.
Replace single ~tosc NetAddr with ~toscClients Dictionary
(keyed by ip) so feedback OSC reaches all connected clients
(web bridge + iPad) simultaneously. ~toscTouch deduplicates
by IP and logs only on first connection. ~toscSend iterates
the dictionary; no-op when empty.
Test: add assertion that two distinct IPs produce size==2.
Balance: P:0 B:0 on both files. Headless TEST PASS.
Add /control/fp (on/off/scale/inst/sens/octave/debug) so the
air-piano is controllable on a headless sclang, plus a debug
strike log gated by ~fpDebug.
Move /armed/<name> receive msg from arm BUTTON to parent GROUP so
pad.lua onReceiveOSC fires on the node that owns the message (FIX 1).
Remove feedback.lua from master RMS fader; native receive already
drives the VU, onFrame decay was pure downside (FIX 3).
Add Lua 5.1/LuaJIT compat shim for table.unpack in melody_faders
(FIX 2). Document deferred sequencer playhead in README (FIX 4).
Regression test: 16 pad GROUPs carry /armed/* receive; 21 tests pass.
Rebuild dist/av-live-control.tosc.
Rotate the iPhone/webcam frame before MediaPipe and display
so detection, skeleton overlay and the shown video stay
aligned after the phone is physically rotated. ARKit 3D
world joints are orientation-invariant and untouched.
Adds sound_algo/data_only/touchosc_feedback.scd: captures the
controlling client IP from inbound /launch* OSC traffic, pushes
/armed/<name> for all 16 pads, /sync/beat each beat on ~lp[clock],
/sync/rms at 20 Hz (server-only, guarded), and echoes
/seq/rhythm/state + /seq/melody/state on preset select. Additive,
nil-guarded, safe to load headless without a booted audio server.
Also adds test/test_touchosc_feedback.scd: headless harness that
builds a dummy ~lp env, loads the feature, asserts the three public
API functions are wired, exercises ~toscArmedPush with nil ~tosc,
and prints TEST PASS.
Pre-existing failure: test_pose_bridge_hands builds the
bridge via __new__, so _vj raised AttributeError on _vdmx.
Defensive getattr makes the full suite green.
Wire finger_piano.scd into both data-only boot paths so
OSCdef /pose/finger and ~fp* API are available at runtime.
Call ~fpModTick on every /pose/hands frame (~30 Hz) for
continuous air-piano modulation. Document FINGER_PIANO env
vars in CLAUDE.md.
Three-page pager (LIVE / FX-HARM-CONCERT / SEQ) with corrected
OSC mapping: scale sends string name not index, octave is absolute
(0/1/2), body-play is /control/doScene "play", arm-mel/rhy send
/launch "melseq"/"rhythmseq". Per-voice VU strip deliberately
omitted (engine sums to bus 0 only). Lua scripts embedded from
gen/lua/*.lua. Dist file committed as the deliverable.
18 tests pass.
Add consti/constf/vali/valf to schema.py so numeric OSC args
carry the right wire type. Partial already supported scale_min/
scale_max; helpers expose them cleanly. Existing API unchanged.
Strengthen test_send_finger_emits_expected_osc to verify all seven OSC
arguments by value (z, tipx, tipy) and their types. Ensures OSC tags and
SC ~fpResolve reader get correct int/float types.
Hybrid architecture: Python detects finger strikes from
iPhone hand joints, SC owns musicality (scale, octave,
instrument) and continuous modulation from hand spacing.
Add melseq (1 Pdef, 8 melodic sequences, /seq/melody) and rhythmseq
(3 sub-Pdefs rsk/rss/rsh, 8 16-step patterns, /seq/rhythm) to the
launchpad OSC bank. ~lpArm special-cases "rhythmseq" to arm/disarm
all three voices together. Web surface gets a Séquenceurs section
with arm pads, selector rows (.sel highlight), and vol sliders.
Add SC_HOST/SC_PORT env config and outbound osc.UDPPort (scPort)
so inbound WS messages {address, args:[...]} are forwarded as typed
OSC packets to the SC engine. JSON.parse and scPort.send are each
guarded by try/catch so a down SC does not crash the bridge.
Add node:test integration test that spawns the server, opens a
loopback UDP listener standing in for SC, sends a WS message, and
asserts the OSC packet arrives with the correct address and args.
Receiver uses metadata:false so args unwrap to plain JS values.
Add 10 OSCdefs to launchpad.scd: /control/harmony/{root,scale,octave,
phrase} mutate ~ccHarmony; /control/fx/{filter,stutter,crash,kick,swell,
breakdown} dispatch to ~ccMaster / ~ccFire. All nil-guarded; scale name
validated against ~ccScales. P:0 B:0.
Add lp_sub/acid/pluck/pad SynthDefs plus lpNote helper and six
pitched Pdefs (sub, acid, arp, lead, stab, pad) armable by name
via the existing /launch OSCdef.
Add launchpad.scd with 4 drum SynthDefs (kick/hat/clap/perc) and an
OSC-armable Pdef bank controlled via /launch, /launch/tempo,
/launch/clear. Patterns run on a dedicated TempoClock (~lp[\clock]).
Load it last in boot.scd, after concert_gestures.scd.
The 6 pitched concert morceaux now source their notes from the
global ~ccNote / ~ccHarmony instead of baked roots and scale arrays,
so the body's harmonic conducting drives their pitch. Each morceau
keeps its own rhythm, structure, SynthDefs and degree sequence; only
the pitch source changes. Acid line/storm gain a 4-entry phrase table
selected by ~ccHarmony[phrase]; hypno drift's drone base cutoff tracks
~ccHarmony[pad] brightness.
Register 5 body-gesture detectors (jump/stomp/squat/spin/tpose) in
sound_algo/data_only/concert_gestures.scd wired to ccFire/ccMaster FX
sinks. boot.scd loads the file after the morceaux block (step 6e).
Add concert_fx.scd providing ~ccMaster (filterTo/dip/stutterOn),
one-shot event synths cc_ev_crash/kick/swell, and ~ccFire helper.
Loaded in boot.scd after engine.scd so ~doFilter/~doMaster exist.
Add /pose/skel OSC route emitting nose, shoulders, hips,
knees and ankles (18 floats) from _emit_person. Add
\poseSkel OSCdef in data_feeds.scd storing the 9 joint
positions per pid; cleanup on \poseLeave.
Extensible ~ccGesture registry (declarative gesture->event, engine
evaluates with hold/cooldown) so triggers are unlimited. Phase 1:
skeleton data layer (/pose/skel -> ~poseSkel), the framework, a master
event-FX layer (master filter + stutter + event synths), and a first
batch of FX/dramatic gestures (jump/stomp/squat/spin/tpose). Harmonic/
melodic triggers + ~ccHarmony are Phase 2 on the same framework.
At full-body concert distance MediaPipe cannot detect the hands
(/pose/hands all zero), muting every morceau hand mapping. ~ccPose now
fills pose[hands] from the body wrists (~poseWrist) when the hands are
not live, shaped exactly like ~handFeat (wrist height/x, arm spread ->
openness, per-tick wrist motion -> speed). All 8 morceaux are driven by
the arms at distance with NO per-morceau change (they read pose[hands]).
Verified headless.
The hand-based arms-crossed gesture could never fire: hand L/R is
position-sorted (rx>=lx always), and at full-body distance MediaPipe
doesnt detect the hands at all (crossing also occludes them). Detect
the advance from the body wrists instead: new /pose/wrist OSCdef ->
~poseWrist, and detectCross fires when the right wrist swaps clearly
left of the left wrist (a clear X), performer centered, wrists at chest
height. Verified headless (wrist-cross advance 1->2).
Add 07_dub_chords.scd: sustained sub (gated asr), dub chord
with CombL delay, deep kick on half notes. Register via
~concertAdd. Wire into boot.scd load list after 06_hard_rave.
Add 04_acid_storm.scd: overdriven 303 + hard kick + screech.
Right hand controls cutoff/res/drive; both arms high triggers
screech fill (once per raise). Registered in boot.scd load list.
inRange is not defined in the installed SuperCollider (Message not
understood) -> the arms-crossed detector errored at runtime and the
gesture never fired. Replace with explicit >= / <= comparisons.
Verified headless: the gesture now fires (advance/clamp logged).
Env-gated VDMX_OSC_HOST (+ VDMX_OSC_PORT, default 1234) forwards the
VJ-useful pose signals (/pose/hands, /pose/kin, /pose/center,
/pose/action, /pose/count) to VDMX OSC input so the body drives any
VDMX parameter. Off by default (no env = no mirror). Verified on macm1:
hands/kin/action received on the VDMX port; None without the env.
Sequential techno setlist engine with crossed-arms gesture to advance
tracks. Adds ~poseCenter OSCdef, scene_concert.scd engine, morceaux/
directory, first morceau (Hypno Drift), and wires into boot/scenes.
Engine first (Task 1: ~concert + ~poseCenter + Hypno Drift + integration),
then one morceau per task (8 self-contained .load files registering via
~concertAdd with their own \cc_* SynthDefs), then a live smoke. Complete
working SuperCollider per task; headless sclang load/advance tests on
GrosMac, audio tuning at the macm1 smoke.
Body-driven techno concert: a sequential setlist of 8 custom-mapped
morceaux (hard/dub/acid/hypnotic) advanced by an arms-crossed gesture.
Concert engine (~concert) + uniform morceau module interface
(start/update/tick/stop + ctx) + crossfade + /pose/center wiring. New
\concert scene alongside body-play. Plan decomposes: engine first, then
morceaux in batches.
When the iPhone USB stream drops (app backgrounds, USB hiccup, decode
end), the reader thread now reconnects with capped backoff instead of
spinning dead on read()=(False,None) forever. The stale frame is cleared
on drop so read() reports no-frame during the gap (no frozen pose).
Verified on macm1: app kill -> "reconnecting" -> relaunch -> detection
resumes (body=1). Unit tests still 10/10.
The data-only boot started the \all scene (all data-web scenes). Boot
directly to \play (body-play) so a body-play session has no data-web
synths at startup. \all and the individual data scenes remain available
via /control/doScene <name>.
The body3d / MediaPipe fallback feeding action-head silently failed:
getattr(kp, "x", kp[0]) eagerly evaluated the kp[0] fallback, which
raises TypeError on the non-subscriptable Kp3D dataclass, so
_kp_list_to_array returned None and every body was skipped. This
blocked /pose/kin and /pose/action for any MediaPipe-only mode (webcam
and iphone-usb), so SC body-play never saw a body (hasBody=false ->
no notes). Use attribute access for Kp3D, index fallback only for
indexable rows. Verified: /pose/kin + /pose/action now flow (97 vs 0).
Add --iphone-usb flag so the pose worker reads frames from
IphoneUSBSource instead of the Mac webcam. Skip Mac-camera TCC
when iphone_usb is set; IphoneUSBSource handles its own usbmuxd
connect. MediaPipe still runs on iPhone frames; arkit_fuse refines
body joints as before.
Handle TAG_SKELETON and TAG_HANDS in IphoneUSBSource._run.
Skeleton: decode_skeleton() -> (91,3) float32 array into
persons_arkit_joints[pid] + persons_arkit_last_t[pid].
Hands: _decode_hands() parses count:u8 + per-hand chirality:u8
+ 21x(x,y,z) BE f32 into list[list[PoseKp]] for persons_hands,
compatible with HandFeatureExtractor.step().
cv2-VideoCapture-shaped IphoneUSBSource backed by the iPhone
ARBodyTracker USB stream (usbmuxd). Decodes AVLiveWire HEVC
video to 640x480 BGR frames for MediaPipe. Thread-safe read().
Skeleton and hands (Task 2) dispatched via state but not yet
wired.
Three-task plan: IphoneUSBSource with HEVC video decode (cv2 .read
contract, reusing the de-risked decode), ARKit skeleton + Vision hands
into State, and --iphone-usb wiring in multi.py/main.py. Tests run on
macm1 against the live iPhone; plus a manual live smoke.
Drive data_only_viz entirely from the iPhone over USB: HEVC video ->
PyAV decode -> MediaPipe, ARKit skeleton -> arkit_fuse, Vision hands ->
hand features. IphoneUSBSource implements the cv2 .read() contract so it
drops into multi.py in place of cv2.VideoCapture; --iphone-usb flag.
HEVC decode de-risked (256 frames at ~28fps from real iPhone AUs).
The ARBodyTracker iOS app serves its ARKit 91-joint skeleton over the
device TCP :7000 in AVLiveWire framing, not OSC, so data_only_viz never
received it. This bridge tunnels to :7000 through usbmuxd (ListDevices +
Connect, port byte-swapped), demuxes the AVLiveWire stream (magic AVL1,
19-byte BE header), decodes SkeletonPayload (91 interleaved BE xyz floats
+ 91 validity bytes) and republishes each valid joint as OSC /body3d/kp
pid joint_idx x y z to :57128 for the arkit_fuse POSE_FILTER stage.
Verified on macm1: usbmux connect to device :7000, frames forwarded.
Final whole-branch review caught a cross-task defect: ~doScenePlay
started ~bpEngine but registered no cleanup, so ~doStopAll (run by
~doScene at every scene switch, and by \stop) never stopped the
engine -- body-play kept playing over any other scene, breaking the
"\stop coupe tout" contract. Register the stop via ~doRegisterPriv
(name, killFn) like every other scene. Verified on macm1: doScene(\play)
starts the engine, doScene(\stop) stops it.
- stop: call clock.stop + nil to prevent TempoClock leak per cycle
- remove unused ~doScene !? / ~bpRegisterPlay dead block
- clarify ~bpCtlDur comment: period is in beats, not seconds
Five-task TDD plan (headless sclang tests) for the body-play module:
bp_* SynthDef palette, modes table + scale quantizer, pure pose->music
mapping functions, the engine (clock/arp/lead/rhythm/crossfade), and
boot wiring with a live smoke step.
Design for body-driven musical generation in the data-only patch:
action-head label selects one of three complete modes (scale+tempo+
instruments), the right hand plays a gesture-triggered lead while the
left hand holds a chord an arpeggio plays in rhythm, and body speed
drives the tempo. New scene_pose_play.scd + synthdefs_play.scd
(bp_* palette), separate from the existing pose->FX module.
The data-only boot opened the OSC receive port 57121 but fell back
silently to the lang port when it was busy — so /pose (data_only_viz)
and /data (data_feeds bridge), both sent to 57121, never reached
sclang and the patch looked alive but reacted to nothing. The port is
typically squatted by a zombie: a scsynth or data_feeds bridge launched
by a previous boot via unixCmd inherits the 57121 socket FD and outlives
its sclang parent, so a fresh boot can't bind it. Instead of falling
back, free the stale holder (lsof -ti UDP:57121 | xargs kill -9) and
retry openUDPPort. Verified: a planted zombie on 57121 is killed and the
port recovered on boot ("57121 recupere — sclang listening on 57121").
scene_pose_action's Routine died on its first tick with "Non Boolean in
test", killing the whole pose->FX mapping (base scenes still played).
Root cause in ~mapPoseToFx: `avgProbs.collect { _ / n }`. The underscore
inside an explicit { } block does NOT build {|x| x/n} — it builds a
function that RETURNS that function, so collect fills avgProbs with
Functions instead of numbers. The next line, avgProbs.maxIndex, then
compares Functions with `>` (a BinaryOpFunction) inside a boolean test
-> MustBeBooleanError. Replace with the idiomatic element-wise
`avgProbs / n`. Verified in isolation (headless sclang): ~mapPoseToFx
and ~mapHandsToFx now pass with both empty and populated pose dicts.
scsynth failed to boot under the data-only patch with "AudioDeviceStart
failed" because it opened a default input device that was unavailable
(an iPhone Continuity mic showed up in the device list) and needed a mic
TCC grant. The patch is output-only, so pin numInputBusChannels = 0; the
server now boots cleanly with no mic prompt.
Also nil-guard the hand-coordinate reads in scene_pose_action's hand->FX
map (hf[\lx]/\ly/\rx/\ry could be nil when one hand is absent, matching
the existing ? 0 pattern on lopen/rspeed/dist).
Note: a separate "Non Boolean in test" error still kills the
scene_pose_action routine on its first tick (pose->FX mapping not yet
live); ~poseKin/~poseState/~handFeat are Dictionaries populated by
data_feeds OSCdefs — left to investigate.
DETRPoseWorker._load_py_config imported LazyConfig from
src.misc.lazy_config, which the DETRPose repo does not expose (it lives
in src.core, as tools/inference/torch_inf.py uses it). The import raised
ImportError with no fallback, dropping into a hand-rolled exec() that
cannot resolve the lazy config's relative imports ("from .include..."),
so model loading died with KeyError "'__name__' not in globals". Try
src.core first, then the old path, before the exec fallback. DETRPose
now loads on macm1 (HGNetV2 backbone auto-downloads, no exception).
The whole data_only_viz pytest run was aborted at collection because
test_multihmr_remote imported REQ_PAYLOAD_LEN, a constant that had been
dropped from multihmr_remote while RSP_PAYLOAD_LEN stayed — restore it
as the RAW-format request payload size. The test had also drifted from
the protocol: decode_request now returns a third timing field, so the
two unpack sites take (img, K, _); and the backend defaults to async
double-buffering (MULTIHMR_REMOTE_ASYNC), under which infer() returns
None when the out-queue is empty — the two loopback stub tests now pin
MULTIHMR_REMOTE_ASYNC=0 for a deterministic sync round-trip. Suite goes
from aborted-at-collection to 151 passed, 14 skipped.
Swift test runs were dropping default.profraw files and the data_feeds
venv symlink into the working tree, and the recovery backup dir plus
tooling state (.serena, agent-memory) showed up as untracked noise.
Ignore them so git status stays clean for day-to-day work.
The phase 2 code review surfaced four follow-ups left after merge.
The joint accessor indexed body.valid without checking its length,
so a short valid array could crash on out-of-range access; it now
guards i < body.valid.count. The viz mode name list was rebuilt on
every modeName() call, so it is hoisted to a static constant. A
Metal shader compile test is added to catch scene.metal regressions
in CI. The dead SceneView placeholder, superseded by LayeredSceneView,
is removed.
Drive HandFaceSkeleton.setDepth() from the body pelvis z each frame
(negated ARKit→RealityKit) so hand/face markers follow body depth
instead of rendering at the fixed default plane depth 0.
Add HandFaceSkeleton: pools of sphere markers for 21x2 hand
points (cyan) and Vision face landmarks (magenta), placed in
the RealityKit worldAnchor at normalized-to-scene-meter coords.
Wire via SceneController.updateHandFace + pushUniforms.
Introduce SceneUniformBuilder to derive Metal shader scalars from the
iPhone stream (body/hands) and wire it into ContentView via three
.onReceive handlers (skeletons, hands, face). Face channels stay 0
for v1 pending Vision region-index mapping.
Add MTKView (SceneRenderer, opaque black) behind a transparent
ARView in a new LayeredSceneView NSViewRepresentable. ARView
background set to .clear with isOpaque=false so the Metal shader
shows through. RenderSettings gains showScene and vizMode; applyAll
drives viz_mode onto the renderer.
Copy scene.metal from archive, add SceneRenderer.swift (Bundle.main),
stride-guard test (144 B). Xcode compiles .metal in Resources phase
even with buildPhase:resources, so use a postBuildScript with
UNLOCALIZED_RESOURCES_FOLDER_PATH to copy the source as plain text.
Add SettingsPanel SwiftUI view bound to all RenderSettings fields.
Wire ContentView: @StateObject settings, applyAll() on appear and
on settings change via objectWillChange, ZStack panel+toggle button.
Add RenderSettings ObservableObject with Published defaults for
layer visibility, mesh material, video opacity, light intensities,
camera FOV, and panel toggle. Add RenderSettingsTests asserting
all eleven defaults via TDD red-green cycle.
Route FrameTag.hands (=4) and FrameTag.face (=5) in USBSkeletonConsumer.
Decode each payload and publish on the main queue, mirroring the
existing .skeleton pattern. Add @Published hands/face properties so
SwiftUI views can observe them directly.
HandsPayload.encoded() wrote min(count,255) in the header but looped
over all hands; use hands.prefix(255) so header and body agree.
Hoist the float reader func f() to the outer hand loop, matching
FacePayload style; declared once per hand instead of per point.
Add FrameTag.hands (=4) and FrameTag.face (=5) without renumbering
existing tags. Implement HandsPayload (up to 255 hands, 21 SIMD3
points each, big-endian codec) and FacePayload (confidence + N
SIMD2 landmarks). Both follow the SkeletonPayload/VideoPayload
codec pattern. Five new XCTest round-trip and rejection tests pass.
Phase 0: capture hands (Vision hand pose) + face (Vision face landmarks) on the iPhone from AR rear-camera frames, stream over new AVLiveWire tags 3/4. Phase 1: port settings panel + 3 lights + mesh material into avlivebody-mac. Phase 2: 10 Metal shader modes + 3D hand/face skeleton, uniforms fed from the iPhone stream.
inferAsync runs the CoreML prediction on a background queue with drop-if-busy; wire() snapshots skeletons on main, infers off-main, then fuses + updates mesh on main. Makes AVBODY_MULTIHMR=1 safe (no run-loop saturation). Default stays skeleton-only.
CoreML Multi-HMR inference ran synchronously on the main thread per video frame, saturating the run loop (beachball). Gate it behind AVBODY_MULTIHMR=1; default to skeleton-only rendering from the iPhone 91-joint ARKit stream.
Add hand_height/openness/speed/dist to SceneUniforms (struct now
24 floats, matching Python pack order). bg_fragment: dist-based zoom
(all modes except hands3d) and post-dispatch color modulation
(luminosity, bloom, turbulence driven by hand pose).
Add OSCdef(\poseHands) to data_feeds.scd storing lx/ly/lopen/lspeed,
rx/ry/ropen/rspeed/dist into ~handFeat Dictionary (9 floats, pid=0).
Initialize ~handFeat alongside existing ~poseState/~poseKin.
Load scene_pose_action.scd in boot.scd after scenes.scd so ~mapPoseToFx
is defined at runtime (was silently dead before).
Document /pose/hands contract in data_only_viz/README.md.
- _emit_hands: replace bare `if h` with numpy-safe predicate
`h is not None and len(h) > 0` in single-person fallback
- test_hand_features: add numpy-row landmark test for
HandFeatureExtractor.step(), asserting finite cx/cy/openness
- test_action_head_pub: add fallback-path test and two-pid
once-per-tick invariant test for send_hands
Vertical slice: MediaPipe hand features drive both the data_only FX rack (SuperCollider) and the Metal scene via one shared signal. Modulation only, no note generation. Includes reliability fixes (boot loads scene_pose_action, FX stubs wired, finite guards).
* docs(plans): action-head v3 + branch sync notes
Update plan header :
- v2 (Task 18) + v3 (Task 19) extensions chronology
- Studio train validated, ckpt action_head_v3.pt landed
- Mesh NaN-guard debug trail (commit 4e7101c)
- Branch convergence main == feat/action-head
- Pointers to memories project_action_head_v3, etc.
* feat(av-live): openpos 3D + DINO reid + filter
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
* fix(av-live-body): restore face+hand+3D (f540158)
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
* feat(av-live): wireframe skel + face/hand filter
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
* deps(icp): add open3d optional extra + smoke test
Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.
Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.
Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
(scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
(PointCloud roundtrip + ICP convergence on translated copy),
gated by `pytest.importorskip("open3d")`
Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.
* feat(icp): LiDAR TCP frame decoder + tests
* feat(icp): LiDAR TCP socket reader with reconnect
* feat(icp): extrinsic dataclass + JSON persistence
* feat(icp): Kabsch + calibration CLI scaffold
* feat(state): persons_arkit_joints fields
* feat(viz): ARKit 91 -> MP 33 joint map
* feat(viz): iphone OSC listener :57128
* feat(viz): arkit_fuse stage overrides 14 slots
* feat(viz): arkit pelvis z locks cam translation
* feat(viz): iphone OSC listener auto-start
* docs: arkit fusion env vars
* feat(icp): point-to-plane register + reject gate
* feat(icp): partition LiDAR per pid by max-dist
* feat(icp): FusionWorker + State.lidar_points
* feat(icp): wire fusion thread behind ICP_FUSION
Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.
A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.
Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.
MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.
* test(icp): synthetic latency + convergence bench
* docs(icp): env vars + calibration procedure
* docs(plans): icp lidar mesh + arkit joints
Two complementary fusion plans landed in parallel on 2026-05-14:
- iphone-lidar-multihmr-fusion : ARKit 91 joints -> MP33 fuse stage +
pelvis z override (already implemented in 7 commits)
- icp-lidar-smplx-fusion : LiDAR mesh point-to-plane ICP onto SMPL-X
10475 verts (12 tasks executed via subagent-driven-development)
Both paths coexist; joints are sparse+fast (60 Hz), mesh is dense+slow
(5-10 Hz). See docs/ICP_FUSION.md for the integration topology.
* feat(icp): predict_once via CoreML backend
* feat(av-live-body): wire ArkitOSCListener :57129
Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.
* feat(ios): iphone ARBodyTracker swiftpm app
iOS 17+ Swift Package app (.swiftpm) streaming ARKit body
joints via OSC UDP to two destinations:
:57128 -> data_only_viz/iphone_osc_listener.py
:57129 -> launcher/AV-Live-Body ArkitOSCListener.swift
Features:
- ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported
- 91 joints per body, /body3d/kp pid joint_idx x y z
- 30 fps throttle
- SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second
- Inline OSC encoder (no external dep)
Env mesh (TCP :5500) NOT yet implemented; requires a separate
ARWorldTrackingConfiguration session. ICP fusion path runs on
bench data only until phase 2.
* feat(data-feeds): 10 open-data OSC publisher
* feat(viz): DataFeeds OSC listener + HUD
* chore: gitignore tweaks
* docs: network topology + mDNS hostnames
Add a "Network topology" section to top-level CLAUDE.md doc
the 3-host layout (GrosMac source, Supra-M1 sink via mDNS,
iPhone via Personal Hotspot DHCP).
mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST
accept hostname.local instead of IPs, so the cluster survives
DHCP rotations on iPhone hotspot (172.20.10.x).
* fix(ios): add NSLocalNetworkUsageDescription
iOS 14+ silently blocks UDP to LAN addresses without this key.
The first time the app tries to send to 192.168.0.159, iOS will
prompt the user to allow Local Network Access; the prompt must
be accepted or the OSC stream never reaches the Mac.
Also adds NSBonjourServices declaring _osc._udp so the system
treats the connection as a recognised service.
* docs: network topology + gitignore hygiene
- CLAUDE.md: add mDNS hostname table (grosmac.local, supra-m1.local,
iPhone hotspot 172.20.10.x). AVBODY_HOST / MULTIHMR_REMOTE_HOST
accept hostnames — resilient to DHCP rotation.
- .gitignore: exclude .remember/ tool state and iCloud '* 2'
collision artifacts.
* feat(ios): ARBody skeleton2D + overlay preview
ARBodySession: publish 2D-projected skeleton snapshot for live
overlay rendering on the iPhone screen alongside the camera feed.
ContentView: SkeletonOverlay drawing on top of the AR view, with
mock T-pose for Xcode previews (useMockBackground, useMockSkeleton).
* docs: iPhone USB body-tracking link design
Brainstormed design for replacing the OSC/network iPhone-Mac
link with a wired USB transport via usbmuxd. iPhone streams
ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML
and renders the mesh. Network-free, single native macOS app.
* docs: iPhone USB transport plan (1 of 3)
Bite-sized TDD plan for the network-free USB byte-pipe:
shared AVLiveWire frame format, native usbmux client,
iOS TCP frame server, incremental stream demuxer.
* feat(avlivewire): shared wire package skeleton
* feat(avlivewire): fixed 19-byte frame header codec
Add FrameHeader, a fixed-size binary record so the demuxer can
frame and resync the iPhone USB stream. Layout is big-endian:
4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32.
The magic prefix lets a reader detect and skip corrupt bytes.
Decoding rejects short buffers and bad magic by returning nil.
Big-endian append/parse helpers are added as Data/UInt extensions
to keep the codec self-contained.
* chore: ignore SwiftPM .build artifacts
Both AVLiveWire and AV-Live-Body produce .build/ on swift
test; ignore them so they never get accidentally staged.
* feat(avlivewire): skeleton and video codecs
Add SkeletonPayload (91 ARKit joints + per-joint validity) and
VideoPayload (one HEVC access unit + keyframe flag) with
big-endian encode/decode. Reuses Task 2 Data/UInt32 helpers.
* feat(avlivewire): incremental stream demuxer
Add StreamDemuxer that accepts arbitrary byte chunks from a
non-frame-aligned stream and emits complete (FrameHeader, Data)
frames, resyncing on the magic prefix after corruption.
* fix(avlivewire): cap demuxer payload length
A corrupt header with a huge UInt32 length made feed buffer
forever waiting for bytes that never arrive. Add an 8 MB max
payload cap; a header exceeding it is treated as corrupt, its
magic is skipped, and the demuxer resyncs on the next frame.
* feat(av-live-body): usbmux message codec
Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.
Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.
* feat(av-live-body): usbmux device client
Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.
Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.
* feat(av-live-body): usbmuxd unix socket transport
Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.
* fix(av-live-body): harden unix socket transport
Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.
* feat(ios): USB TCP frame server
Add USBServer: an NWListener on a fixed local TCP port that
usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames
and exposes a connection-state callback.
* build: depend on shared AVLiveWire package
Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.
* build(ios): add AVLiveWire package to xcodegen
The xcodegen project did not declare the shared AVLiveWire
package, so USBServer.swift would fail to import it in the
generated Xcode project. Add it as a local package dep.
* test(avlivewire): end-to-end chunked loopback
Feeds 20 framed skeleton payloads through StreamDemuxer in
7-byte chunks (worst-case TCP fragmentation). Fixed a split
range operator from the plan that did not parse.
* fix(ios): guard USBServer listener and payload
Report .idle (not .listening) when NWListener creation fails,
and drop payloads larger than the demuxer's 8 MB cap so the
receiver never silently discards an oversized frame.
* chore: ignore .swiftpm editor state dirs
swift test / Xcode create hidden .swiftpm dirs inside
packages; ignore them so they never get staged.
* docs: iPhone capture plan (2 of 3)
Plan for HEVC video capture (VideoToolbox) over the USB
transport and removal of the legacy OSC sender. Skeleton
USB path already exists; this adds the video half.
* feat(ios): HEVC video capture, drop OSC
Add VideoEncoder (VideoToolbox HEVC) and stream encoded
frames over USB as .video AVLiveWire frames alongside the
skeleton. Remove the legacy OSC/UDP fanout and its host/port
config UI — the iPhone link is now USB-only.
* docs(ios): refresh stale OSC references
ARBodySession header comment and Info.plist usage strings
still described the removed OSC/UDP path; update them to the
USB transport and drop the dead _osc._udp Bonjour service.
* docs: macOS USB consumer plan (3a of 3)
Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.
* feat(av-live-body): USB skeleton consumer
Background usbmux read loop feeding StreamDemuxer; republishes
.skeleton frames as 91-joint ArkitBodyFrames and forwards
.video payloads. Removed stale iCloud collision duplicate
source files that broke the build.
* fix(data-only): CoreML Multi-HMR usage bugs
The CoreML Multi-HMR model was fine; the "0 detections" bug
was caller-side. Add ImageNet normalization in infer() (the
DINOv2 backbone needs it; raw [0,1] input collapsed all
scores) and update stale hardcoded output var names to match
the re-converted mlpackage. Bump the latency test threshold
to the realistic ~140 ms full-model figure.
* feat(av-live-body): HEVC video decoder
VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.
* feat(av-live-body): render 91-joint USB skeleton
Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.
* feat(av-live-body): wire USB consumer to renderer
ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.
* docs: macOS Multi-HMR mesh plan (3b of 3)
Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML
Swift wrapper, BodyFusion (ARKit depth correction), mesh
pipeline wiring. Completes the spec.
* docs: AVLiveBody macOS rewrite design
Clean-rewrite spec: fresh native macOS Xcode app for the
iPhone-USB body pipeline. Reuses the tested USB components,
single RealityKit scene (video quad + skeleton + mesh),
drops all legacy MediaPipe/viz/data-feed code.
* docs: AVLiveBody macOS rewrite plan
10-task plan: scaffold the xcodegen app, migrate the USB
pipeline, build the RealityKit scene (video quad, skeleton,
mesh), wire it, archive the legacy app.
* feat(avlivebody-mac): scaffold xcode app
Add an empty buildable native macOS app generated via xcodegen,
sibling of iphone-arbody. Depends on the shared AVLiveWire package.
Later tasks add the USB pipeline and RealityKit scene.
* feat(avlivebody-mac): migrate usb transport
Context: the new native AVLiveBody app needs the proven iPhone-Mac
usbmux transport layer. These files are self-contained, depending
only on AVLiveWire plus Apple system frameworks, so they cross the
rewrite boundary unchanged.
Approach: copy the three transport files and their unit tests
byte-for-byte from launcher/AV-Live-Body, then make the test target
buildable.
Changes:
- Add usb/USBMuxProtocol.swift, usb/USBClient.swift and
usb/VideoDecoder.swift under Sources/AVLiveBody.
- Add USBMuxProtocolTests.swift and USBClientTests.swift under
Tests/AVLiveBodyTests.
- Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so
xcodebuild can code sign the now-populated test bundle.
Impact: the usbmux pipeline is available in the rewrite and its
six unit tests run green under xcodebuild test.
* feat(avlivebody-mac): usb skeleton consumer
Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload
keyed by pid and owns video decoding directly, dropping the legacy
ArkitOSCListener conversion layer.
* fix(avlivebody-mac): guard thread store with lock
Move the `thread` property write inside the stateLock-held
region in start(); t.start() stays outside since the thread
cannot run before start() is called. Removes a latent race.
* feat(avlivebody-mac): multi-hmr and body fusion
Context: Task 4 of the macOS rewrite needs the dense-mesh half of
the pipeline alongside the USB skeleton consumer landed in task 3.
Approach: Add a CoreML wrapper that mirrors the validated Python
reference (data_only_viz/multihmr_coreml.py) and a pure-logic
fusion stage that corrects the mesh pelvis depth using the
LiDAR-precise USB skeleton.
Changes:
- MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image
input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at
10475 vertices, det threshold 0.3.
- BodyFusion.swift: stateless fuse(persons, skeletons) overrides
the highest-score mesh translation.z with the skeleton pelvis Z
when available, passes through otherwise.
- BodyFusionTests.swift: pelvis override and pass-through cases.
Impact: Unlocks the mesh renderer wiring in later tasks and gives
the macOS app metrically-correct depth in front of the camera.
* fix(avlivebody-mac): load mlmodelc, clarify fusion
Xcode compiles .mlpackage resources to .mlmodelc at build time;
look up the compiled artifact directly and drop the redundant
MLModel.compileModel step. Also rewrite BodyFusion docstring to
match actual single-person pelvis-z override behaviour.
* feat(avlivebody-mac): scene controller + view
RealityKit scene plumbing: SceneController owns ARView, orbital
camera, and holders for VideoQuad/SkeletonEntity/MeshEntity.
SceneView is the SwiftUI NSViewRepresentable bridge.
Build intentionally deferred to T8 (holder types land in T6-T8).
* fix(avlivebody): orbit gesture + setUp guard
Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to
dispatch only on .changed, replacing the Task wrapper with
MainActor.assumeIsolated. Guard SceneController.setUp() with a
didSetUp flag so duplicate makeNSView calls do not re-install
gestures or re-add anchors.
* feat(avlivebody-mac): 91-joint skeleton entity
Yellow marker spheres pooled per pid; ARKit (x,y,z) ->
RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for
macOS RealityKit Material.Color. Build deferred to T8.
* feat(avlivebody-mac): video quad
Flat 1.6x0.9m plane at z=-2m, textured per-frame from
CVPixelBuffer via CIImage -> CGImage -> TextureResource.
Per-frame TextureResource creation is the known perf hot
spot, isolated here for later LowLevelTexture upgrade.
* fix(avlivebody-mac): appkit import for orbit
NSPanGestureRecognizer lives in AppKit on macOS; without the
import the AVLiveBody module failed to emit. T5 leftover
surfaced once T6/T7/T8 made the target compilable.
* feat(avlivebody-mac): smpl-x mesh entity
Render SMPL-X dense meshes (10475 verts) from Multi-HMR with
pooled ModelEntity per person. Triangle indices loaded from the
bundled smplx_faces.bin (flat UInt32 triplets, copied from the
legacy launcher target). xcodegen folder-scanning bundles the
.bin under Contents/Resources/ — no project.yml change needed.
* feat(avlivebody-mac): wire scene + status bar
Replace placeholder window with ContentView wiring
USBSkeletonConsumer, SceneController, MultiHMRCoreML and
BodyFusion per the T9 dataflow plan.
* chore: archive legacy AV-Live-Body
Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.
* fix(avlivebody): break onVideoFrame retain cycle
Capture consumer weakly in the onVideoFrame closure so the
USBSkeletonConsumer can be deallocated and its background thread
exits cleanly. Guard the mesh-fusion path when consumer is gone.
* fix(launcher): disable body spawn post-archive
Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).
* docs(avlivebody-mac): contributor setup README
Document prerequisites, mlpackage copy, signing xcconfig, and
xcodegen/xcodebuild commands. Points at design spec and plan.
* refactor(avlivebody): axis helper + cleanups
- Extract arkitToRealityKit helper, dedupe 3 call sites.
- Add onDisappear consumer.stop() to terminate USB read loop.
- Replace @State with let for SceneController (stable class id).
- Add NSLog diagnostics in VideoQuad+MeshEntity silent guards.
* fix(avlivebody): ad-hoc signing for local dev
Apple Development cert + Automatic signing makes Xcode demand a
Mac Development cert that no one has. Switch to manual ad-hoc
(CODE_SIGN_IDENTITY = -) so any contributor can build. Drop
hardened runtime; re-enable for distribution builds.
* feat(arbody): keep iphone awake while streaming
iOS auto-lock tears down the USBServer TCP listener within
seconds, breaking AVLiveBody Mac-side connect. Disable the
idle timer for the lifetime of ContentView, restore on exit.
* docs: add AGENTS.md skeleton
iOS auto-lock tears down the USBServer TCP listener within
seconds, breaking AVLiveBody Mac-side connect. Disable the
idle timer for the lifetime of ContentView, restore on exit.
Apple Development cert + Automatic signing makes Xcode demand a
Mac Development cert that no one has. Switch to manual ad-hoc
(CODE_SIGN_IDENTITY = -) so any contributor can build. Drop
hardened runtime; re-enable for distribution builds.
- Extract arkitToRealityKit helper, dedupe 3 call sites.
- Add onDisappear consumer.stop() to terminate USB read loop.
- Replace @State with let for SceneController (stable class id).
- Add NSLog diagnostics in VideoQuad+MeshEntity silent guards.
Legacy SwiftPM target archived to launcher/_archive-AV-Live-Body/.
New native Xcode app lives at avlivebody-mac/; no swift run path.
startBodyApp now logs + no-ops with FIXME(rewrite-2026-05-18).
Capture consumer weakly in the onVideoFrame closure so the
USBSkeletonConsumer can be deallocated and its background thread
exits cleanly. Guard the mesh-fusion path when consumer is gone.
Superseded by avlivebody-mac/ on 2026-05-18. See
docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md
for the rewrite design and rationale.
Render SMPL-X dense meshes (10475 verts) from Multi-HMR with
pooled ModelEntity per person. Triangle indices loaded from the
bundled smplx_faces.bin (flat UInt32 triplets, copied from the
legacy launcher target). xcodegen folder-scanning bundles the
.bin under Contents/Resources/ — no project.yml change needed.
NSPanGestureRecognizer lives in AppKit on macOS; without the
import the AVLiveBody module failed to emit. T5 leftover
surfaced once T6/T7/T8 made the target compilable.
Flat 1.6x0.9m plane at z=-2m, textured per-frame from
CVPixelBuffer via CIImage -> CGImage -> TextureResource.
Per-frame TextureResource creation is the known perf hot
spot, isolated here for later LowLevelTexture upgrade.
Yellow marker spheres pooled per pid; ARKit (x,y,z) ->
RealityKit (x,-y,-z). Adapted .systemYellow to NSColor for
macOS RealityKit Material.Color. Build deferred to T8.
Filter NSPanGestureRecognizer state in OrbitTarget.handlePan to
dispatch only on .changed, replacing the Task wrapper with
MainActor.assumeIsolated. Guard SceneController.setUp() with a
didSetUp flag so duplicate makeNSView calls do not re-install
gestures or re-add anchors.
RealityKit scene plumbing: SceneController owns ARView, orbital
camera, and holders for VideoQuad/SkeletonEntity/MeshEntity.
SceneView is the SwiftUI NSViewRepresentable bridge.
Build intentionally deferred to T8 (holder types land in T6-T8).
Xcode compiles .mlpackage resources to .mlmodelc at build time;
look up the compiled artifact directly and drop the redundant
MLModel.compileModel step. Also rewrite BodyFusion docstring to
match actual single-person pelvis-z override behaviour.
Context: Task 4 of the macOS rewrite needs the dense-mesh half of
the pipeline alongside the USB skeleton consumer landed in task 3.
Approach: Add a CoreML wrapper that mirrors the validated Python
reference (data_only_viz/multihmr_coreml.py) and a pure-logic
fusion stage that corrects the mesh pelvis depth using the
LiDAR-precise USB skeleton.
Changes:
- MultiHMRCoreML.swift: 1x3x672x672 ImageNet-normalized image
input, 1x3x3 cam_K input, K=4 SMPL-X person outputs at
10475 vertices, det threshold 0.3.
- BodyFusion.swift: stateless fuse(persons, skeletons) overrides
the highest-score mesh translation.z with the skeleton pelvis Z
when available, passes through otherwise.
- BodyFusionTests.swift: pelvis override and pass-through cases.
Impact: Unlocks the mesh renderer wiring in later tasks and gives
the macOS app metrically-correct depth in front of the camera.
Move the `thread` property write inside the stateLock-held
region in start(); t.start() stays outside since the thread
cannot run before start() is called. Removes a latent race.
Add a cleaned USBSkeletonConsumer that publishes SkeletonPayload
keyed by pid and owns video decoding directly, dropping the legacy
ArkitOSCListener conversion layer.
Context: the new native AVLiveBody app needs the proven iPhone-Mac
usbmux transport layer. These files are self-contained, depending
only on AVLiveWire plus Apple system frameworks, so they cross the
rewrite boundary unchanged.
Approach: copy the three transport files and their unit tests
byte-for-byte from launcher/AV-Live-Body, then make the test target
buildable.
Changes:
- Add usb/USBMuxProtocol.swift, usb/USBClient.swift and
usb/VideoDecoder.swift under Sources/AVLiveBody.
- Add USBMuxProtocolTests.swift and USBClientTests.swift under
Tests/AVLiveBodyTests.
- Set GENERATE_INFOPLIST_FILE=YES on the AVLiveBodyTests target so
xcodebuild can code sign the now-populated test bundle.
Impact: the usbmux pipeline is available in the rewrite and its
six unit tests run green under xcodebuild test.
Layer E draws two concentric wireframe shells of the sphere with
additive blending: an outer blue shell breathing with bass and an
inner magenta shell pulsing with kick, spinning opposite ways.
Add an empty buildable native macOS app generated via xcodegen,
sibling of iphone-arbody. Depends on the shared AVLiveWire package.
Later tasks add the USB pipeline and RealityKit scene.
10-task plan: scaffold the xcodegen app, migrate the USB
pipeline, build the RealityKit scene (video quad, skeleton,
mesh), wire it, archive the legacy app.
Clean-rewrite spec: fresh native macOS Xcode app for the
iPhone-USB body pipeline. Reuses the tested USB components,
single RealityKit scene (video quad + skeleton + mesh),
drops all legacy MediaPipe/viz/data-feed code.
A fourth layer draws the icosphere as a wireframe, displaced by the
same relief and rendered as bright unlit colormap lines just above
the skin. Toggled with key 4.
The spectrogram now extrudes the geometry alongside the waveform, the
skin is lit with a screen-space normal, and the sphere tumbles and
pulses with the kick and signal energy.
drawVertices on the indexed skin mesh gave a degenerate gl_PointCoord,
so the round-point discard killed every fragment. Use a dedicated
OF_PRIMITIVE_POINTS mesh, float it 5% outside the skin, enlarge points.
Final plan: bundle the validated FP32 mlpackage, MultiHMRCoreML
Swift wrapper, BodyFusion (ARKit depth correction), mesh
pipeline wiring. Completes the spec.
ContentView owns and starts a USBSkeletonConsumer, threaded
through BodyView into Skeleton3DRenderer.attach. The renderer
subscribes its $bodies into lastArkit, so the iPhone's USB
skeleton drives the on-screen 91-joint markers.
Complete the long-standing TODO: draw the 91 ARKit/USB
skeleton joints as yellow markers, fed from lastArkit. Spawn
entity trees for ARKit-only pids so the USB skeleton shows
without a MediaPipe pose.
Auto-rotation ran unconditionally, so it kept spinning during a
camera-orbit drag. Accumulate a frame-rate-independent angle only
when the mouse is not pressed.
VTDecompressionSession decoder for .video VideoPayloads.
Rebuilds the format description from the parameter sets
prepended to keyframe payloads by the iOS VideoEncoder.
The CoreML Multi-HMR model was fine; the "0 detections" bug
was caller-side. Add ImageNet normalization in infer() (the
DINOv2 backbone needs it; raw [0,1] input collapsed all
scores) and update stale hardcoded output var names to match
the re-converted mlpackage. Bump the latency test threshold
to the realistic ~140 ms full-model figure.
Plan for consuming the iPhone USB stream in AVLiveBody:
USBSkeletonConsumer, VideoDecoder, 91-joint skeleton render.
Multi-HMR dense mesh deferred to Plan 3b.
Add the project CLAUDE.md and README covering the build, the stereo
channel convention, the three layers, controls, and the 16 MS/s
dual-channel hardware constraint.
ARBodySession header comment and Info.plist usage strings
still described the removed OSC/UDP path; update them to the
USB transport and drop the dead _osc._udp Bonjour service.
Add VideoEncoder (VideoToolbox HEVC) and stream encoded
frames over USB as .video AVLiveWire frames alongside the
skeleton. Remove the legacy OSC/UDP fanout and its host/port
config UI — the iPhone link is now USB-only.
Plan for HEVC video capture (VideoToolbox) over the USB
transport and removal of the legacy OSC sender. Skeleton
USB path already exists; this adds the video half.
Upload the per-channel waveform as a texture and displace icosphere
vertices radially in the vertex shader: CH1 drives the northern
hemisphere, CH2 the southern.
ofShader::load returned bool was discarded, so a failed shader load
rendered black with no diagnostic. Log an error on failure and
document the global ofDisableArbTex state change.
Context: Task 5 of oscope-sphere - add the SphereViz layer A skin
and full ofApp wiring so the app produces a visual output.
Approach: Build an icosphere (5 subdivisions) skinned by a scrolling
2-channel spectrogram texture uploaded each frame via ofFloatPixels.
GLSL 150 shaders compute spherical UV from vertex direction so the
texture wraps seamlessly; scroll is handled by a fract() offset in
the fragment shader to avoid re-uploading geometry.
Changes:
- bin/data/shaders/sphere.vert: spherical UV projection from position
- bin/data/shaders/sphere.frag: magma/viridis colormaps + scroll
- src/SphereViz.h/.cpp: icosphere mesh, shader, texture management,
pushSpectrogramColumn() wraps SpectrogramBuffer for dual-channel
- src/ofApp.h: full member declarations (camera, analyzers, flags)
- src/ofApp.cpp: setup/update/draw/HUD/keyPressed wired end-to-end;
HantekDevice probed at startup, falls back to DemoSignal on failure
Impact: app renders a rotating spectrogram globe in demo mode with
orbit camera, colormap toggle, freeze, and per-layer visibility keys.
Report .idle (not .listening) when NWListener creation fails,
and drop payloads larger than the demuxer's 8 MB cap so the
receiver never silently discards an oversized frame.
Feeds 20 framed skeleton payloads through StreamDemuxer in
7-byte chunks (worst-case TCP fragmentation). Fixed a split
range operator from the plan that did not parse.
Copy oscope-of's Makefile and libusb-aware config.make, add a window
bootstrap and a placeholder ofApp so the GL build is verified before
any feature code lands.
The xcodegen project did not declare the shared AVLiveWire
package, so USBServer.swift would fail to import it in the
generated Xcode project. Add it as a local package dep.
Both ARBodyTracker (iOS) and AVLiveBody (macOS) now depend on
the local shared/AVLiveWire package so the wire format is
defined once. iOS USBServer imports it; macOS use lands in
Plan 3.
Add USBServer: an NWListener on a fixed local TCP port that
usbmuxd tunnels to the tethered Mac. Sends AVLiveWire frames
and exposes a connection-state callback.
The old test only checked two blocks differ, true of any non-DC
signal. Replace it with a split-equals-whole identity that genuinely
proves next() is deterministic and continuous across call boundaries.
Apply four code-review fixes to UnixMuxTransport:
- send() now loops on partial writes and retries on EINTR
instead of discarding write(2)'s return value.
- Add deinit and an fd = -1 sentinel so close() is
idempotent and the descriptor cannot leak.
- precondition guards strcpy against sun_path overflow.
- readN() distinguishes EOF from error and retries EINTR.
Pure-C++ continuous two-channel generator used when no scope is
connected: CH1 frequency sweep, CH2 steady tone plus noise. Adds the
run_tests.sh runner covering all three pure-C++ units.
A buffer of height 2 made half-1 zero, dividing by zero in pushColumn
and writing NaN into the column. Clamp the denominator and add a
regression test for the 2-row case.
Add UnixMuxTransport, the production MuxTransport that opens a
blocking AF_UNIX socket to /var/run/usbmuxd. Implements framed
packet reads (4-byte LE length prefix) and raw stream reads for
the tunneled byte stream after a successful Connect.
Add USBClient for usbmux device discovery and connect-to-port,
with an injectable MuxTransport so tests need no real device.
Harden USBMuxProtocol.readLE32 to return an optional with a
bounds check, avoiding an out-of-range crash on truncated data.
Pure-C++ store that log-resamples both channels' FFT magnitudes into
scrolling columns, CH1 north and CH2 south, high frequencies toward
the poles. No GL dependency, fully unit-tested.
Add USBMuxProtocol, a codec for Apple's usbmuxd request/response
protocol: a 16-byte little-endian header (length, version=1,
message=8 plist, tag) followed by an XML property list.
Wire an AVLiveBodyTests test target into Package.swift (none
existed) so swift test runs the round-trip and header coverage.
Context: oscope-sphere is a new openFrameworks app (sibling of
oscope-of) that will render audio analysis on a 3D sphere. This
commit establishes the project tree and reuses the mature capture
and analysis layer from oscope-of.
Approach: Copy four proven source files byte-for-byte from
oscope-of/src/ (no modifications), create the standard oF directory
layout, and add a standalone clang++ test harness that validates two
design assumptions without requiring openFrameworks installed.
Changes:
- Create oscope-sphere/{src,tests,bin/data/shaders} directories
- Copy FFT.h/cpp, AudioAnalyzer.h/cpp, ScopeData.h,
HantekDevice.h/cpp from oscope-of/src (unmodified)
- Add addons.make and .gitignore for oF build system
- Add tests/check.h: lightweight CHECK/REPORT macros
- Add tests/test_analysis.cpp: two tests (FFT peak at bin 64,
per-channel isolation via dual-analyzer trick)
Impact: Test harness compiles with plain clang++ -std=c++17 and
passes 5/5 checks, confirming FFT correctness and AudioAnalyzer
channel isolation before any oF integration work begins.
A corrupt header with a huge UInt32 length made feed buffer
forever waiting for bytes that never arrive. Add an 8 MB max
payload cap; a header exceeding it is treated as corrupt, its
magic is skipped, and the demuxer resyncs on the next frame.
Nine-task TDD plan: three pure-C++ units unit-tested with clang++
(spectrogram buffer, demo signal, reused FFT/analyzer checks) then
five openFrameworks GL units verified visually, building the stereo
sphere layer by layer.
Add StreamDemuxer that accepts arbitrary byte chunks from a
non-frame-aligned stream and emits complete (FrameHeader, Data)
frames, resyncing on the magic prefix after corruption.
Brainstorming output for a new AV-Live sibling project: a minimal
openFrameworks app rendering a dual-channel Hantek 6022 capture as a
single 3D sphere with three composable layers (spectrogram skin +
waveform displacement, orbit rings, point cloud). Reuses oscope-of's
capture and analysis layer.
Add FrameHeader, a fixed-size binary record so the demuxer can
frame and resync the iPhone USB stream. Layout is big-endian:
4-byte magic AVL1, tag u8, pid i16, timestamp f64, length u32.
The magic prefix lets a reader detect and skip corrupt bytes.
Decoding rejects short buffers and bad magic by returning nil.
Big-endian append/parse helpers are added as Data/UInt extensions
to keep the codec self-contained.
Bite-sized TDD plan for the network-free USB byte-pipe:
shared AVLiveWire frame format, native usbmux client,
iOS TCP frame server, incremental stream demuxer.
Brainstormed design for replacing the OSC/network iPhone-Mac
link with a wired USB transport via usbmuxd. iPhone streams
ARKit skeleton + HEVC video; macOS app runs Multi-HMR CoreML
and renders the mesh. Network-free, single native macOS app.
ARBodySession: publish 2D-projected skeleton snapshot for live
overlay rendering on the iPhone screen alongside the camera feed.
ContentView: SkeletonOverlay drawing on top of the AR view, with
mock T-pose for Xcode previews (useMockBackground, useMockSkeleton).
iOS 14+ silently blocks UDP to LAN addresses without this key.
The first time the app tries to send to 192.168.0.159, iOS will
prompt the user to allow Local Network Access; the prompt must
be accepted or the OSC stream never reaches the Mac.
Also adds NSBonjourServices declaring _osc._udp so the system
treats the connection as a recognised service.
Add a "Network topology" section to top-level CLAUDE.md doc
the 3-host layout (GrosMac source, Supra-M1 sink via mDNS,
iPhone via Personal Hotspot DHCP).
mDNS is canonical now : AVBODY_HOST and MULTIHMR_REMOTE_HOST
accept hostname.local instead of IPs, so the cluster survives
DHCP rotations on iPhone hotspot (172.20.10.x).
iOS 17+ Swift Package app (.swiftpm) streaming ARKit body
joints via OSC UDP to two destinations:
:57128 -> data_only_viz/iphone_osc_listener.py
:57129 -> launcher/AV-Live-Body ArkitOSCListener.swift
Features:
- ARBodyTrackingConfiguration + sceneDepth (LiDAR) when supported
- 91 joints per body, /body3d/kp pid joint_idx x y z
- 30 fps throttle
- SwiftUI UI: Host/Port fields, Start/Stop, live joints-per-second
- Inline OSC encoder (no external dep)
Env mesh (TCP :5500) NOT yet implemented; requires a separate
ARWorldTrackingConfiguration session. ICP fusion path runs on
bench data only until phase 2.
Receives /body3d/kp from iPhone ARBodyTracker on the diagnostic
port (57129, distinct from Python's 57128 fuse input). Plumbed
through ContentView -> BodyView -> Skeleton3DRenderer so the
ARKit joints can be overlayed alongside Multi-HMR mesh.
Task 9 of the ICP LiDAR plan: integrate the FusionWorker built in
earlier tasks into the live data_only_viz pipeline without
disturbing the existing ARKit pelvis fuse path or the Multi-HMR
worker thread.
A new IcpFusionThread pulls LiDAR frames from LidarTCPReader,
stages them into State, and applies in-place ICP registration on
state.persons_smplx[*].vertices_3d. It runs as a separate daemon
thread parallel to MultiHMRWorker rather than inline per frame —
the autonomous-worker architecture didn't fit the plan's
per-frame call site, so we adapted to a polling thread at 8 Hz.
Activation is opt-in via ICP_FUSION=1 plus ICP_LIDAR_HOST; the
default code path is untouched. Shutdown wired through
applicationWillTerminate_.
MultiHMRWorker.predict_once is added as a documented stub
(NotImplementedError) because the existing PyTorch run loop is
too coupled to the camera and MPS lifecycle for a clean
single-shot extraction. calibrate_lidar.py keeps its placeholder
until a follow-up refactor extracts a pure _infer(rgb) helper.
Context: Task 1 of the ICP LiDAR <-> SMPL-X fusion plan needs a
point-cloud library to align iPhone LiDAR scans with Multi-HMR
SMPL-X meshes. Open3D's CPU-only ICP is sufficient at the 5-10 Hz
LiDAR cadence.
Approach: Add a dedicated `lidar` optional-dep group so the heavy
dependency stays opt-in. Pin Python to 3.12 implicitly via the
regenerated uv.lock because open3d 0.18-0.19 only ships cp311/cp312
wheels (cp314 absent). Smoke test guards future regressions.
Changes:
- pyproject.toml: new `lidar` extra with `open3d>=0.18,<0.20`
- uv.lock: regenerated with open3d 0.19 + transitive deps
(scikit-learn, scipy, dash stack, etc.)
- tests/test_open3d_smoke.py: two-test smoke suite
(PointCloud roundtrip + ICP convergence on translated copy),
gated by `pytest.importorskip("open3d")`
Impact: Unlocks subsequent ICP fusion tasks (LiDAR ingest, mesh
alignment, transform publication) without forcing open3d on
contributors who only run the base pose pipeline.
Restore 166 lines of feed implementations (VELIB Paris, SYTADIN
trafic Ile-de-France, HUB'EAU water, GCN seismic, GDELT events,
SWPC space weather, RTE eco2mix) that existed in the local canon
but were missing from origin/main after the 2026-05-13 cherry-pick
recovery cycle. Verified unique via diff against fresh clone.
Two changes that take loop fps from 9.8 to 20+ without touching
the predict cost :
1. MultiHMRWorker default target_fps 10 -> 30. The Python loop was
self-capping at 10 fps via sleep(period - dt) regardless of the
backend. With the remote async pipeline (queue maxsize=2) the
actual fresh-predict rate stays at the macm1 server ceiling
(~9 fps under MLX contention), but the loop now submits frames
at 30 Hz and re-emits MeshRigger-interpolated state at the same
cadence -- the mesh becomes visually fluid at the loop rate even
when fresh predicts are slower. Tunable via MULTIHMR_LOOP_FPS.
2. Short-circuit reused-humans : when infer() returns None (queue
empty), skip dedup/tracker/One-Euro reprocessing and just sleep
to the next slot. Saves ~3-5 ms CPU per idle iteration and
avoids advancing the smoother on identical state.
Heartbeat log extended with (fresh=Y) so we can distinguish loop
rate (visualiser feed) from real predict rate. Bench remote macm1
under MLX contention : hb[coreml]: 25.0 fps (fresh=9.0).
Root cause for the predict gap 53 vs 87 ms : MLX servers
mlx_lm.server :8502/8503 + LoRA on macm1 burn ~30 ms of unified
memory bandwidth per Multi-HMR predict. Not a software fix.
Three perf optimizations stacked on top of the distributed pipeline:
1. multihmr_server.py ported to pyobjc CoreML.framework direct
(drops the ~30 ms coremltools.MLModel.predict overhead). Compiles
the mlpackage to .mlmodelc on load, then predicts via
MLDictionaryFeatureProvider + MLMultiArray ctypes memcpy. Toggle
MULTIHMR_SERVER_BACKEND=coremltools for fallback. setup script
installs pyobjc-framework-CoreML on the macm1 venv.
2. MediaPipe Holistic now uses GPU Metal delegate on M5. Required
wrapping camera frames as SRGBA (4-channel) instead of SRGB --
the Metal CVPixelBuffer path rejects 3-channel formats. Bench
M5 standalone : pose 6.7 -> 2.9 ms, face 4.0 -> 1.0 ms, hand
6.1 -> 3.2 ms. Frees ~10 ms CPU per frame for OSC + rigger.
3. Remote client queues bumped maxsize 1 -> 2/3 (in/out) to absorb
jitter without stalling capture.
Multi-HMR inference offloaded to a remote macm1 (M1 Max, 32-core
GPU) via a custom TCP protocol on :57140. JPEG q=80 compression
(~80 KB vs 1.35 MB raw) + double-buffer async pipeline (3-thread
server reader/worker/writer, single-buffer client) overlaps net
I/O with predict. Live: 9.8 fps Multi-HMR keyframe (vs 6.8 fps
M5 local CoreML), 27 fps perceived via MeshRigger.
Distributed display option : AVBODY_HOST env routes the TCP mesh
and UDP OSC streams to a remote AVLiveBody. The Swift launcher
runs on macm1 in the user GUI session (via osascript do shell
script over SSH), receiving the full mesh+skeleton+face+hand
stream from M5 capture.
Skeleton3DRenderer now bundles body 33 joints + dlib 68 face
landmarks + 21x2 hand landmarks into a single procedural
LowLevelMesh (line topology) per person : 143 vertices, 288 line
indices, 4 draw calls. Anatomical connectors body nose -> face
nose bridge and body wrists -> hand wrists.
MediaPipe delegate is selectable via MEDIAPIPE_DELEGATE=gpu|cpu
(default cpu after observed Metal GpuBuffer crashes on macOS).
Multi-HMR ViT-L checkpoint also tested : 350 ms on macm1 GPU
(2.9 fps) -- too slow to ship, ViT-S remains default. Conversion
script generalized via MULTIHMR_CKPT_NAME env.
Skeleton3DRenderer now renders a wireframe: joint radius 1 mm
(quasi-invisible), bone radius 3 mm (line-like). Replaces the
chunky bead armature with a clean filaire silhouette covering
body 33 joints + face 68 dlib + hands 21x2, all 3D.
FaceHandOverlay 2D Canvas removed from ContentView -- face and
hand landmarks now live in the same 3D RealityKit armature as
the body skeleton (Skeleton3DRenderer.applyFace / applyHands,
anchored on nose joint 0 + wrist joints 15/16).
pose_filter.py extended with FaceFilterChain (alpha-beta + 30 ms
lookahead) and HandFilterChain. multi.py wires them after the
2D smoothers, plus ghost rejection (POSE_GHOST_MIN_VISIBLE),
bbox NMS (POSE_NMS_IOU), and pid hysteresis. 10 new tests, all
green.
CoreML perf audit (bench_multihmr_coreml.py): predict() = 99% of
wall-time on FP32. ANE catastrophic for DINOv2 (1300 ms),
INT8 weight quant = no live gain (GPU compute-bound).
6.4-6.8 fps live is the hardware ceiling on this model.
quantize_multihmr_int8.py left in scripts/ for future trials.
Three regressions after recent merges, all restored to match the
original f540158 design:
1. FaceHandOverlay was no longer instantiated in ContentView.
Added back as a SwiftUI Canvas overlay (68 dlib face landmarks
with mouth slots 48-67, plus 21x2 hand landmarks cyan/magenta).
2. Skeleton3DRenderer was not attached. BodyView now creates an
AnchorEntity at (0,0,-2.5), instantiates Skeleton3DRenderer
and ties its visibility to vizMode==9 or showSkeleton toggle.
3. Joint and bone radii bumped to 4.5 cm / 2.2 cm so the 3D
skeleton actually reads as 3D instead of looking flat.
MeshRenderer exposes pelvisWorld map per pid for future
interconnect uses (not auto-applied -- design keeps mesh and
skeleton each in their own coord space per f540158).
Three improvements wired end-to-end:
1. Openpos 3D skeleton visible: Skeleton3DRenderer attached to a
RealityKit AnchorEntity in BodyView, toggled by showSkeleton
or vizMode==9. PoseOSCListener now parses /pose3d/count and
/pose3d/kp (plus restored /face/* and /hand/* paths).
2. DINO re-id (dinov2_vits14, ~9 ms ANE forward):
MeshRigger combines Hungarian IoU with cosine similarity over
a per-pid embedding history (deque maxlen=10), weighted by
MULTIHMR_REID_ALPHA (default 0.5). Falls back to pure IoU if
DINO mlpackage absent or scipy missing. state.last_frame_rgb
buffer added so the rigger can crop bbox regions for embedding.
3. PoseFilterChain on pose_world_landmarks:
median (anti-spike) -> Kalman constant-velocity ->
50 ms lookahead -> IK elbow/knee/ankle clamp. Configurable
via POSE_FILTER env (default median+kalman+lookahead+ik).
<2 ms per frame for typical 1-2 persons.
Tests: 5 new in test_dino_reid.py + 6 new in test_pose_filter.py,
all green. Live validated by user: skeleton spawns, mesh stays
stable.
Apres cherry-pick des 14 commits action-head, le working tree main
divergeait encore de feat/action-head sur 21 fichiers a cause des
resolutions modify/delete qui avaient pris la mauvaise version
(c52271e botched merge avait drop ces fichiers, le cherry-pick
voyait alors "modify in feat / deleted in main" et conservait la
mauvaise version dans certains cas).
Cette commit synchronise le working tree main vers feat/action-head
en checkout-ant tous les fichiers depuis feat. Equivalent fonctionnel
a un fast-forward merge sans toucher l'historique.
Files restored : Skeleton3DRenderer.swift, FaceHandOverlay.swift,
test_multihmr_coreml.py, test_pose_bridge_*.py (4 test files).
Files updated : action_head.py, action_head_pub.py, multi.py,
pose_bridge.py, state.py, extract_j3d_offline.py, training/*.py,
SettingsPanel.swift, data_feeds.scd, et plus.
Result : main == feat/action-head sur le working tree, historique
granulaire preserve via les cherry-picks precedents.
CATransform3D scale x=-1 on the AVCaptureVideoPreviewLayer so the
user sees themselves like in a mirror (left in reality = left on
screen). Overlays (skeleton/face/hand/mesh) keep their raw camera
coordinates for now; align via X-flip if user requests it.
MULTIHMR_AUTOCAST=1 enables MPS mixed precision for the ViT-S
backbone. Tested 2026-05-13: slower than fp32 baseline (400ms vs
270ms) -- overhead cast within forward exceeds matmul savings on
M5. Off by default; FP16 .half() crashes MPS matmul accumulator,
left out entirely.
apple_vision_pose face parser short-circuits to return immediately
since pyobjc 11 cannot dereference VNFaceLandmarkRegion2D
pointsInImageOfSize_ result. Removes ObjCPointerWarning spam at
30 fps (9 regions per face).
Add a parallel-pose worker selector (env AV_LIVE_PARALLEL_POSE)
defaulting to "both": runs Apple Vision body 2D on ANE alongside
MediaPipe Holistic on CPU XNNPACK, while Multi-HMR PyTorch streams
the dense mesh on MPS. Modes "apple_vision" or "mediapipe" to pick
one. Body keypoints land in state.persons_body either way.
Face landmark parser via pyobjc remains blocked: pointAtIndex_
selector arity confuses pyobjc 11 and pointsInImageOfSize_ returns
an opaque PyObjCPointer with no address handle. Keep the
ANE-detected count for logging, fall back to MediaPipe face/hand
fin landmarks until a Swift bridge is in place.
Adresse final review du feature action-head :
- action_head_pub.py + extract_j3d_offline.py : CoreMLArray
wraps numpy mais n'a pas __array__ ; unwrap via .numpy()
avant np.asarray pour eviter object-array silencieux
quand persons_smplx vient du backend CoreML. extract_j3d
ramene depuis main (manquait sur feat suite au merge c52271e).
- train_on_studio.sh : TRAIN_ARGS quote defensivement via
printf %q + reject single quotes pour eviter injection
via le payload single-quoted sur bastion.
CoreML mlpackage reverted to FLOAT32 compute_precision: the FP16
build, even with the roma branchless rotmat patch, visibly
degraded the mesh on extreme poses (atan2 near theta=pi + SMPL-X
decoder drift). FP32 stays 2.3x faster than PyTorch MPS (139 ms
vs 270 ms; 8 fps live with 3 workers).
MeshRigger pid matching now uses Hungarian assignment on bbox IoU
with sticky cache (new_iou >= 0.30 AND old_iou < 0.15 to switch).
Replaces fragile pelvis-distance heuristic. 5 new tests pass.
Multi-HMR PyTorch MPS sort occasionnellement des v3d avec NaN/Inf
ou vertices a magnitudes extremes (>5 m). Ces frames glissaient
sans guard jusqu'a AVLiveBody et apparaissaient comme pics/trous
sur le mesh.
Approche : np.isfinite check sur v3d + transl_np, skip person si
invalide. Plus sanity clamp sur |v3d|.max() > 5.0 m : humains ~2 m,
au-dela = garbage du modele, skip aussi. Le receiver Swift garde
le dernier good mesh pendant les frames skip via retain_window
500 ms (dc7de90).
Tradeoff : on perd la frame entiere du pid au lieu de juste les
verts NaN. Acceptable car corruption est globale au tensor v3d
quand elle arrive (jamais juste 2-3 verts isoles).
Multi-HMR PyTorch MPS sort occasionnellement des v3d avec NaN/Inf
ou vertices a magnitudes extremes (>5 m). Ces frames glissaient
sans guard jusqu'a AVLiveBody et apparaissaient comme pics/trous
sur le mesh.
Approche : np.isfinite check sur v3d + transl_np, skip person si
invalide. Plus sanity clamp sur |v3d|.max() > 5.0 m : humains ~2 m,
au-dela = garbage du modele, skip aussi. Le receiver Swift garde
le dernier good mesh pendant les frames skip via retain_window
500 ms (dc7de90).
Tradeoff : on perd la frame entiere du pid au lieu de juste les
verts NaN. Acceptable car corruption est globale au tensor v3d
quand elle arrive (jamais juste 2-3 verts isoles).
Permet de desactiver le rigger 30 fps (mesh_rigger.py, ajoute en
2c8094c) via env var MESH_RIG=0 pour debugger les deformations
mesh sans changer le code.
Root cause of v3d/transl NaN identified: roma.rotmat_to_rotvec
uses torch.empty + 8 index_put_ on a buffer that CoreML mlprogram
translates as scatter_nd over a garbage-initialised tensor. Cells
that the scatter chain misses keep NaN; the subsequent quat /
norm propagates NaN to every vertex.
Patch: branchless atan2 formulation (stack/clamp/norm/atan2 only),
no torch.empty, no index_put_. Precision drift vs roma original:
2.26e-6 L_inf on random batches. Mlpackage now validates all
outputs finite (1.27e-4 L_inf vs PyTorch eager on v3d).
Bench standalone: 65 ms median FP16 (15.3 fps, target met).
Live with 3 parallel workers: 8 fps Multi-HMR keyframe rate
(2.3x speedup vs PyTorch MPS baseline 3.5 fps); rigging still
ships at 15-20 fps perceived.
Output names shifted post-patch (var_2541 -> var_2412 etc) so
multihmr_coreml.py constants updated.
Add standalone MP4 extractor for action-head v3 training data via
MediaPipe HolisticLandmarker (VIDEO mode). Unlike extract_j3d_offline.py
(SMPL-X path), writes real hands_kp (42, 3) and mouth_open from face
landmarks instead of zeros.
- scripts/extract_mediapipe_offline.py: full pipeline (open video,
iterate frames, map body33/face/hands -> j3d32 + hands_kp42 +
mouth_open, write jsonl rows matching dataset.py schema)
- tests/test_extract_mediapipe_offline.py: 8 pure-numpy unit tests;
no mediapipe runtime required
Enables hand-aware training from recorded footage without SMPL-X
or GPU at extraction time.
CATransform3D scale x=-1 on the AVCaptureVideoPreviewLayer so the
user sees themselves like in a mirror (left in reality = left on
screen). Overlays (skeleton/face/hand/mesh) keep their raw camera
coordinates for now; align via X-flip if user requests it.
MULTIHMR_AUTOCAST=1 enables MPS mixed precision for the ViT-S
backbone. Tested 2026-05-13: slower than fp32 baseline (400ms vs
270ms) -- overhead cast within forward exceeds matmul savings on
M5. Off by default; FP16 .half() crashes MPS matmul accumulator,
left out entirely.
apple_vision_pose face parser short-circuits to return immediately
since pyobjc 11 cannot dereference VNFaceLandmarkRegion2D
pointsInImageOfSize_ result. Removes ObjCPointerWarning spam at
30 fps (9 regions per face).
MeshRigger module : entre deux keyframes Multi-HMR (~3.5 fps mesh
dense PyTorch MPS), on translate rigidement le mesh keyframe via le
delta pelvis 2D Apple Vision (30 fps body ANE) projete a profondeur
constante. SMPLXTCPSender bumpe a 30 fps target et applique le rig
sur chaque tick. Verifie live : 27 fps TCP soutenu, 100% rigged,
keyframe Multi-HMR a 3.2 fps -> ~8x speedup perceptuel dans la
fenetre RealityKit AVLiveBody.
Limitations connues :
- Translation seule (pas de rotation ni de LBS articule)
- Pelvis 2D delta projete a Z constant du keyframe
- Pas de matching d'identite robuste Vision <-> Multi-HMR : on prend
la personne Vision la plus proche du pelvis keyframe projete
Add a parallel-pose worker selector (env AV_LIVE_PARALLEL_POSE)
defaulting to "both": runs Apple Vision body 2D on ANE alongside
MediaPipe Holistic on CPU XNNPACK, while Multi-HMR PyTorch streams
the dense mesh on MPS. Modes "apple_vision" or "mediapipe" to pick
one. Body keypoints land in state.persons_body either way.
Face landmark parser via pyobjc remains blocked: pointAtIndex_
selector arity confuses pyobjc 11 and pointsInImageOfSize_ returns
an opaque PyObjCPointer with no address handle. Keep the
ANE-detected count for logging, fall back to MediaPipe face/hand
fin landmarks until a Swift bridge is in place.
Adresse final review du feature action-head :
- action_head_pub.py + extract_j3d_offline.py : CoreMLArray
wraps numpy mais n'a pas __array__ ; unwrap via .numpy()
avant np.asarray pour eviter object-array silencieux
quand persons_smplx vient du backend CoreML. extract_j3d
ramene depuis main (manquait sur feat suite au merge c52271e).
- train_on_studio.sh : TRAIN_ARGS quote defensivement via
printf %q + reject single quotes pour eviter injection
via le payload single-quoted sur bastion.
Merges face+hand+3D pose bridge and ActionHead training work into
main. Conflict resolution kept both:
- mains keyboard NSEvent monitor (no system beeps) + p alias
- feats Skeleton3DRenderer anchor wiring
- mains /pose/skel flat OSC path AND feats /face/*, /hand/*, /pose3d/*
Standalone publisher polls state at 30 Hz, extracts 22-joint
positions from either persons_smplx (vertex anchors) or
persons_body3d (MediaPipe 33→22 map), runs ActionHead.step()
per pid, and emits /pose/action + /pose/kin + lifecycle OSC.
- action_head_pub.py: ActionHeadPublisher thread with dedup
via smplx_last_t / pose_last_t; purges lost pids
- tests/test_action_head_pub.py: 4 unit tests (39 total pass)
- multi.py: import + instantiate + start publisher in __init__
Add comment in TracedMHMR.forward documenting that CoreML conversion
produces all-NaN on v3d/transl while PyTorch eager works. Tested FP32,
K_inv closed-form, simplified subtract+divide projection, nan_to_num
masking. Root cause is an op-level mistranslation in the v3d/transl
path; needs sub-wrapper bisection. Workaround: PyTorch backend.
Implement Task 10 of action-head plan. Records webcam frames +
timestamps for action-head training using OpenCV. Outputs MP4 video
+ timestamp text file to ~/.cache/av-live-action/raw/ with 672x672
square crop, configurable fps/session/camera, interactive q-quit.
Stream MediaPipe Holistic face landmarks (68 dlib subset of 478),
hand landmarks (21 left + 21 right), and pose world landmarks (33
3D xyz meters) over OSC :57126 to AVLiveBody. Launcher renders
face/hand as SwiftUI Canvas overlay and the 3D skeleton as a
RealityKit armature (sphere joints + cylinder bones, color per
chain) toggled via p / mode openpos.
Multi-HMR worker now also starts MediaPipe Multi in parallel so
both the dense SMPL-X mesh (TCP 57130, PyTorch backend) and the
skeleton/face/hand streams (OSC 57126) feed the launcher from
one Python process.
Launcher AppDelegate forces .regular activation so SwiftPM
binaries actually show their WindowGroup without a bundle.
Tests: 9 new pytest cases (4 body3d + 5 face/hand), all green.
CoreML conversion still produces NaN on v3d/transl; PyTorch
backend is the working path for now.
Add interactive console TUI for manual label review of
auto-labeled action datasets. Displays ASCII skeleton,
kinetics, and proposed label with confidence. User can
accept proposed label, choose manual override (1/2/3),
skip, or quit. Reads auto-labeled JSONL and writes
validated rows to reviewed dataset.
rsync excludes .venv/__pycache__/.pytest_cache + uv sync ajoute
extra multihmr (torch). End-to-end valide: smoke 160 windows
3 epochs MPS studio en ~4s, ckpt rsync back OK.
Le precedent printf %q sur-quotait $HOME -> mkdir recevait 0 arg.
On utilise des chemins absolus /Users/clems/av-live-action/* cote
studio et single-quotes pour proteger des bastion expansions.
Add OSCdef handlers for /pose/action, /pose/kin, /pose/enter,
/pose/leave routes emitted by data_only_viz pose_bridge. Store
person state and kinematics in ~poseState and ~poseKin dicts.
Ajoute 4 methodes OSC pour action_head et localisation cinetique:
send_action(pid, label_idx, probs, t_now, force) envoie /pose/action
avec [pid, label_idx, prob_0, prob_1, prob_2] pour les 3 classes.
send_kin(pid, kin, t_now, force) envoie /pose/kin avec [pid, kin[0],
kin[1], kin[2]] pour angles de poignets/coudes.
send_enter/send_leave envoient /pose/enter et /pose/leave pour cycle
vie des personnes.
Throttle reuse _period/_last_t existants; force=True bypass throttle.
Wrapper bash : rsync dataset+code grosmac->studio via bastion
electron-server, exec uv run train_action_head --device mps sur
M3 Ultra, rsync checkpoint back. SSH direct cassee depuis reboot
studio 2026-05-12 ; route via bastion documentee dans CLAUDE.md.
Add evaluation script to compute test accuracy, confusion matrix, and
inference latency on a trained action-head checkpoint. Reuses existing
WindowDataset and model infrastructure from training pipeline.
Falls back to evaluating on full dataset if test split is empty
(edge case with <4 sessions).
Context:
- User demande : faire afficher le mesh open-pos et le rendre
plus ameliore
- Signale aussi : les touches ne fonctionne pas j'ai le beep
- Phase 2 fusion AV-Live-Body : rendu skeleton 3D dans la
fenetre RealityKit, alimente par MediaPipe Multi via OSC
Approach:
- Etendre /pose/skel OSC pour shipper les 33 keypoints flat
- Rendu RealityKit pur (UnlitMaterial spheres + cylindres)
- Remplacer keyboardShortcut SwiftUI sans modifier (qui beep)
par NSEvent.addLocalMonitorForEvents qui consomme l'event
Changes:
- pose_bridge.py emet /pose/skel pid + 33 (x,y,c) vers :57126
- PoseOSCListener gagne PoseFrame.skeleton + parser /pose/skel
- SkeletonOverlay (nouveau fichier) : 33 spheres + 35 cylindres
bones BlazePose, palette par pid, conf < 0.3 cache joint/bone
- BodyView : Coordinator.skeletonOverlay attache au bodyAnchor,
update() visible si vizMode == 9 OU showSkeleton toggle
- BodyView : NSEvent monitor capture S C V M W 0-9, retour nil
- AVLiveBodyApp : keyboardShortcut modifiers [.command] partout
Impact:
- Mode openpos affiche le squelette 3D live, palette par pid
- Touches single-key (S / 0-9 / C / V / M / W) marchent sans
beep grace au NSEvent monitor qui consomme l'event
- Cmd+S Cmd+0-9 Cmd+C/V/M/W accessibles via menubar
Add train_action_head.py with WindowDataset, class-weighted
CrossEntropy, AdamW optimizer, per-epoch train/val loop, and
best-val-acc checkpoint saving. Add smoke tests verifying
2-epoch run and checkpoint loadability via ActionHead.
- WindowDataset computes position/velocity/accel features inline
- _class_weights balances imbalanced label distribution
- train() returns history dict (train/val loss and acc)
- CLI entry point for --device mps/cuda/cpu production runs
Convert Multi-HMR ViT-S 672 to CoreML mlpackage and wire as optional
worker backend via MULTIHMR_BACKEND=coreml env var.
Inference path uses pyobjc + native CoreML framework (Python 3.14 has
no libcoremlpython binding). Conversion done in a separate Py 3.12
venv; einsum cascade patched (camera intrinsics broadcast + smplx
landmarks) via setup_multihmr.sh, idempotent on re-clone.
Bench: 28 ms mock, 100-170 ms live (~13 fps, 4x PyTorch MPS). ANE
compile fails on this model; CPU+GPU is the sweet spot.
Brings 9ab82e4 (MediaPipe Multi default + rich pose OSC) +
96a326d (av-live-body scene Metal) + 214b154 (HUD + menus).
Conflicts on duplicate launcher Swift commits (667f63c vs
96a326d) resolved with -X theirs (prefer main version, since
214b154 builds on it).
Phase 1 fusion : AV-Live-Body absorbe la couche Metal viz et
ecoute les data pose via OSC.
Metal scene (10 viz modes : storm, tunnel, plasma, kaleido,
voronoi, metaballs, starfield, bars, hands3d, openpos) :
- Resources/scene.metal copie depuis data_only_viz, compile au
runtime via MTLLibrary.makeLibrary(source).
- SceneRenderer.swift : MTKViewDelegate qui rebuild
SceneUniforms (20 floats, miroir du struct Metal) et drive
bg_pipeline (full-screen triangle).
- BodyView : nouveau MTKView entre la cam preview et l'ARView,
zPosition intermediaire, alpha pour laisser passer la cam.
- RenderSettings : showScene + vizMode (0..9), picker 10
boutons numerotes dans SettingsPanel + libelle du mode actif
affiche dans la row 'Scene Metal (<name>)'.
Pose OSC :
- PoseOSCListener.swift : UDP listener :57126, parser OSC
minimal (i, f, s args), @MainActor dispatch des Published.
Stocke un PoseFrame par pid (center, head, wrists, sho_span,
yaw, pitch) avec GC 2 s.
- data_only_viz/pose_bridge.py : 2e SimpleUDPClient broadcast
vers 127.0.0.1:57126 (try/except OSError pour silencer si
AVLiveBody pas la). Throttle 30 Hz partage.
Phase 2 (futur) : rendu skeleton entities RealityKit (spheres +
cylindres) consommant PoseOSCListener.persons.
Package.swift : ajout Resources/scene.metal en .copy.
Phase 1 fusion : AV-Live-Body absorbe la couche Metal viz et
ecoute les data pose via OSC.
Metal scene (10 viz modes : storm, tunnel, plasma, kaleido,
voronoi, metaballs, starfield, bars, hands3d, openpos) :
- Resources/scene.metal copie depuis data_only_viz, compile au
runtime via MTLLibrary.makeLibrary(source).
- SceneRenderer.swift : MTKViewDelegate qui rebuild
SceneUniforms (20 floats, miroir du struct Metal) et drive
bg_pipeline (full-screen triangle).
- BodyView : nouveau MTKView entre la cam preview et l'ARView,
zPosition intermediaire, alpha pour laisser passer la cam.
- RenderSettings : showScene + vizMode (0..9), picker 10
boutons numerotes dans SettingsPanel + libelle du mode actif
affiche dans la row 'Scene Metal (<name>)'.
Pose OSC :
- PoseOSCListener.swift : UDP listener :57126, parser OSC
minimal (i, f, s args), @MainActor dispatch des Published.
Stocke un PoseFrame par pid (center, head, wrists, sho_span,
yaw, pitch) avec GC 2 s.
- data_only_viz/pose_bridge.py : 2e SimpleUDPClient broadcast
vers 127.0.0.1:57126 (try/except OSError pour silencer si
AVLiveBody pas la). Throttle 30 Hz partage.
Phase 2 (futur) : rendu skeleton entities RealityKit (spheres +
cylindres) consommant PoseOSCListener.persons.
Package.swift : ajout Resources/scene.metal en .copy.
3 fixes pour reduire le bruit de logs des sources publiques :
wikimedia.py : ajoute User-Agent descriptif. WMF EventStreams
refuse l'UA httpx par defaut avec 403 Forbidden. Le 1.0+URL
descriptif est obligatoire pour le rate-limiting et les abuse
reports.
netzfrequenz.py : backoff exponentiel cap a 5 min sur les ws
disconnect. La source (mainsfrequenz.de) est NXDOMAIN depuis
mai 2026 ; on logguait toutes les 3 s soit ~28 800/jour. Avec
backoff x1.6 et log uniquement attempt==1 ou attempt%10, on
descend a ~30 log/jour.
config.data-only.toml : opensky poll_seconds 20 -> 60. L'API
anonyme est credit-based (10 req/min budget). 60s reste
raisonnable pour le viz tout en restaurant le credit.
Comparatif backbone DINOv2 ViT-S 672 sur deux Apple Silicon :
Hardware | CoreML GPU | CoreML ALL | PyTorch MPS
M5 (16 GB) | 25.1 ms | 157.3 ms | 274.7 ms
M1 Max(32G) | 63.3 ms | 154.1 ms | 110.6 ms
M5 GPU 2.5x plus rapide en CoreML (per-core efficiency vs
32 cores M1 Max). M1 Max 2.5x plus rapide en PyTorch MPS
(M5 souffrait probablement de throttle accumule).
ANE handicape les deux (~155 ms vs 25-63 ms GPU) — DINOv2
ViT-S non ANE-friendly.
macM1 workers MLX pauses via SIGSTOP/SIGCONT pour bench
isole.
addLocalMonitorForEventsMatchingMask reagit uniquement quand
l'app a le focus, ce qui empechait les touches azertyuiop /
qsdfghjklm de marcher des qu'on cliquait dans un terminal.
Ajoute un addGlobalMonitor parallele (read-only, AppKit
contrainte) qui delegue a _on_key sans retourner l'event
(le local monitor reste en charge du discard).
Force aussi activateIgnoringOtherApps + makeKeyAndOrderFront
+ makeFirstResponder au boot, pour que le local monitor
recoive les events tout de suite sans necessiter un clic
initial dans la fenetre.
LOG.debug ajoute pour tracer chaque touche recue.
Ajoute un 10e viz_mode 'openpos' dans data_only_viz pour
afficher le squelette multi-personne sur un fond minimal qui
met en valeur les corps detectes.
state.py :
- viz_mode_names complete avec 'openpos' a l'index 9
- KEYMAP_VIDEO : touche 'p' -> openpos (slot precedemment libre)
shaders/scene.metal :
- nouveau mode_openpos : fond radial sombre avec une grille
de points discrete + pulsation legere sur ~rms. Le bg sert
juste a faire ressortir le skel_pipeline rendu PAR-DESSUS
par le renderer (qui beneficie deja du depth_fog + palette
par pid + epaisseur confiance-adaptee).
- bg_fragment dispatcher : mode == 9 -> mode_openpos
web/public/control.html :
- 10e bouton 'openpos' dans la grille #vizmodes.
Patch 10 cumule : aten::diagonal override pour supporter offset=0
dim1=1 dim2=2 sur tensor rank-3 (B, N, N). Utilise par
roma.rotmat_to_rotvec invoque dans Multi-HMR head.
Implementation : reshape (B, N, N) -> (B, N*N), gather indices
diagonale aplaties [0, N+1, 2N+2, ...] axis=1.
Cascade T3 status : 10 patches actifs, conversion progresse.
Blocker 11 = reshape size mismatch ([1, 51, 3] -> [204, 3, 1])
= incoherence model-side post topk (facteur 4 = K=num_persons).
Plus une op manquante, du model surgery requis.
T3 roadmap mis a jour avec investigation Step B1-B3 pour reprise.
Cascade T3 progression (9 patches cumules) :
6. tile no-op pour reps=[]
7. new_ones converter (aten::new_ones manquait)
8. concat 0d->1d auto-promote
9. clamp_min/max avec dtype promotion
Bloque maintenant a aten::diagonal avec offset/dim non-default.
Cascade encore en cours, on continue.
Web data-only fully wired contre les 20 feeds :
dashboard.js : 8 nouveaux handlers (gdelt, wikimedia, tides,
moon, atc, pose, mempool, github, rte_eco2mix). Sparklines +
seuils alert/warn/green coherents.
map.js : marker GDELT (purple) en plus des seismes/foudre/
avions/ISS/volcans, TTL 60s comme les autres ephemeres.
control.html : nouvelle section 'Modes visuels (oF / Metal)'
avec 9 boutons (storm/tunnel/plasma/kaleido/voronoi/metaballs/
stars/bars/hands3d). Le handler scene-btn est splitte par
container (#scenes vs #vizmodes) pour ne pas melanger
l'etat actif.
server.js : whitelist /control/viz* + routing par path. Les
/control/viz* partent vers oF (:57123), tout le reste vers
SC (:57121). Nouveau env OF_PORT_OUT.
Audit fixes :
- LaunchMode displayName Data-only : 'SC + oF + feeds, no web'
-> 'SC + Metal viz + feeds + web' (l'ancien libelle datait
d'avant le toggle Web UI dans dataOnly).
- SettingsPanel : toggle Squelette etiquete '(a venir)' tant
que le rendu n'est pas implemente cote MeshRenderer.
Deux plans complementaires post-cascade T3 :
1. hybrid-backbone : approche pragmatique (CoreML backbone seul
+ PyTorch head). 2.5 h cible, 2-3x speedup sans toucher au
head. Le probe v4 .mlpackage retourne cls_token mais on a
besoin de patch tokens (get_intermediate_layers) -> Task 1
re-convertit un wrapper adapte.
2. t3-roadmap : sous-plan dedies pour reprendre la cascade Task 3
en session multi-jour. Documente les 5 patches deja faits +
blocker actif (tile reps=0) + cascade probable estimee
(6-12 h focused).
Recommandation : tenter hybrid d'abord, T3 full si insuffisant.
Task 2 — apply_topk(K=4) validee comme drop-in remplacant
apply_threshold dans la tete Multi-HMR. Cosine sim = 1.000000
sur 4/4 detections image reelle, MAE 2-4 mm.
Script : scripts/probe_head_topk.py.
Task 3 — full Multi-HMR convert tente. Patches appliques :
1. apply_threshold -> apply_topk monkey-patch
2. backbone.encoder.interpolate_pos_encoding -> buffer fige
3. utils.camera.inverse_perspective_projection -> closed-form
jit.trace OK, mais conversion coremltools echoue sur op
suivantes en cascade. Estimation restante : 1-2 j.
Script : scripts/coreml_full_probe.py.
Le breakthrough probe v4 backbone seul 11.8x reste la victoire.
Task 4 worker integration bloquee tant que T3 pas complete.
Nouveau fichier control/web_bridge.scd, charge par
boot.data-only.scd apres data_feeds.scd. Ferme la boucle :
Entrants (port 57121, depuis data_only_viz/web /control.html) :
- /control/master_gain f master amp sur ~mainGroup
- /control/cutoff f LPF sur ~fxGroup
- /control/reso f resonance LPF
- /control/reverb_mix f mix reverb
- /control/delay_time f delay time
- /control/delay_feedback f delay feedback (clamp ~0.95)
- /control/tempo f TempoClock.default.tempo_(bpm/60)
- /scene/play s ~dataScene.(name) — meme contrat
que les keymaps qsdfghjklm
- /xy/x f /xy/y f pad XY -> ~fxGroup.set(\xyMod, ...)
Sortants (port 57125 -> server.js -> WS clients) a 4 Hz :
- /sync/bpm f TempoClock.default.tempo * 60
- /sync/beat i compteur monotone
- /sync/rms f ~rmsLevel.value si dispo
- /sync/voices i ~activeVoiceCount.value si dispo
State partage dans ~webState pour debug REPL.
Le pont est best-effort cote SC : si ~mainGroup / ~fxGroup ne
sont pas encore crees, les set() sont silencieusement skip.
Tentative T1 du plan smplerx-arm-port : install extra OK,
mmcv-lite Config shim OK, mais cascade de modules mmcv 1.x
manquants depasse largement le budget (Config, deprecated_api_warning,
17 funcs mmcv.cnn, bricks.registry 12 sub-reg, puis mmcv.runner
50+ classes, et la cascade continue).
C'est la migration mmcv 1.x -> 2.x complete a refaire a la
main : ~1-2 semaines effort propre, pas 5-6 h. Abandon final.
Shim partiel _smplerx_shims.py committe comme socle si reprise
future. Fork electron-rare/SMPLer-X + submodule restent en place.
Le serveur Express :3211 devient bidirectionnel :
- IN :57124 data_feeds bridge.py -> feeds vers WS broadcast
- IN :57125 SuperCollider sync (BPM/RMS/voies) -> WS broadcast
- OUT :57121 navigateur -> SC pour /control/* /scene/* /xy/*
Nouvelle page /control.html (3 colonnes) :
- Sliders synth/mix : master_gain, cutoff, reso, reverb_mix,
delay_time, delay_feedback, tempo. Chaque slider envoie son
path OSC en temps reel sur drag.
- 10 boutons scenes audio (cavity, geo, body, weather, flight,
pulse, quiet, all, full, stop) : meme contrat que les
keymaps qsdfghjklm du Metal viz. Toggle visuel actif.
- XY pad : drag/clic envoie /xy/x et /xy/y dans [0,1] (Y
inverse pour up=1). Cursor pink avec ombre + grille 4x4.
- Retour SC : labels BPM / Beat / RMS / Voies actives mis a
jour quand SC pousse /sync/* sur :57125.
Liens 'Controle' ajoutes dans dashboard.html et map.html.
Fork electron-rare/SMPLer-X (S-Lab License 1.0, non-comm
seulement) ajoute en git submodule a third_party/SMPLer-X.
Setup script utilise le submodule au lieu de cloner upstream.
pyproject : nouvel extra 'smplerx' (torch + mmcv-lite +
ultralytics + smplx + timm). NOTICE.md documente la posture
non-commerciale heritee de Multi-HMR et SMPLer-X.
Le portage inference ARM Mac est l'etape suivante : SMPLer-X
vendorise sa propre mmpose (transformer_utils/mmpose), donc
seul mmcv.Config a besoin d'un substitut (mmcv-lite). Le
detecteur mmdet sera remplace par YOLO Ultralytics.
NOT YET DONE: smpler_x_worker.py (model load + inference,
~1 j) ; bench camera ; integration main.py.
4 nouvelles sources open-data publiques, sans cle :
- gdelt : GDELT 2.0 events CSV.zip toutes les 15 min ; emet
/event lat lon tone country eid + /batch n countries avg_tone.
Dedup par GLOBALEVENTID, ring buffer 8192.
- wikimedia : EventStreams SSE firehose recentchange (toutes
langues). Sample ~5% des edits + rate per second toutes
les 5 s. Reconnect auto sur disconnect.
- tides : NOAA CO-OPS water level observe + predit pour une
station (Boston par defaut, station code configurable) +
phase lunaire calculee (Conway approx, sans dep). Poll 6 min.
- atc : LiveATC hub listeners scraping (KJFK/KLAX/KSFO/KORD/
EGLL/LFPG par defaut). Polite 500ms entre hubs, total
poll 5 min.
netzfrequenz, rte_eco2mix, mempool, github, pose passent a
enabled=true. Seul gcn reste off (exige auth NASA client_id +
client_secret obligatoires, pas de fallback anonyme).
Warnings connus apres ce toggle :
- netzfrequenz : ws mainsfrequenz.de down depuis mai 2026 -> spam
d'erreurs jusqu'a fallback gridradar/swissgrid
- pose : fonctionne uniquement quand bridge.py tourne dans le
bundle launcher (TCC camera prompt impossible en CLI pur)
- rte_eco2mix : credentials vides -> probable echec si l'API
data.rte-france.com exige OAuth
- github : sans token, rate-limit 60 req/h -> marche par bursts
A. CoreML : update probe v2 result (pos-embed surgery
suffit pas, blocker reste op 17/610). Recommande pivot.
B. Studio offload : plan ecrit puis MARQUE DEFERRED apres
contrainte user "<8 GB RAM sur M5, on-device only".
C. Modern body mesh survey : 10 modeles 2025-2026 filtres
sur M5 8 GB constraint. SMPLer-X-S = candidat #1
(claim 64 ms M5, 30M params, SMPL-X drop-in).
D. Studio comme rig training (deploy M5) : 4 voies, recos
en cas de besoin de retraining/distillation.
Skip Multi-HMR si diff thumbnail 112x112 < 5.0 (sur 0-255).
Mesure terrain : 74/78 frames skipped en 5s sur scene
statique = ~95% compute saved sans degrader le mesh visuel.
Flag CLI --motion-gate.
torch.compile teste, plante runtime sur shape arithmetic
(FakeTensor erreur), abandonne avec note.
CoreML Task 1 probe (DINOv2 ViT-S backbone seul) execute :
conversion echoue sur l'interpolate_pos_encoding (tensor
to Python scalar). Plan mis a jour avec verdict 2-3 jours
de chirurgie minimum, pas 1 jour. Recommandation pivot :
Studio M3 Ultra offload comme alternative.
alpha 0.5 -> 0.25 : lerp s'etale sur la frame Python
au lieu de converger en 100 ms puis figer. Persistance
entite 500 ms : une frame Multi-HMR ratee n'efface plus
le mesh. Skip recompute normales pendant interp ticks
(stale ~150 ms invisible vs 24 ms/tick economises).
Lerp SMPL-X vertices between Multi-HMR frames (Python at ~3 fps)
via SceneEvents.Update subscription. Throttle to 30 fps (1 tick
of 2) with alpha=0.5 to keep MainActor cost bounded. Skip normal
recompute during interp ticks — stale ~150 ms is invisible vs
the 24 ms/tick CPU cost it saves.
Pipeline web data-only complet :
Cote feeds, OSC target :57124 ajoute au profil data-only.toml
pour que le bridge.py diffuse vers : SC (57121) + oF (57123)
+ Web (57124).
Cote web (data_only_viz/web/) :
- server.js : Express :3211 + WS broadcast + osc UDP :57124
parse les paths /data/<feed>/<sub> en JSON et diffuse
- public/dashboard.html : grille de cards live avec sparklines
SVG vanilla (USGS, SWPC Kp/wind/Bz/X-ray, Blitz, OpenSky,
Bluesky, meteo, air, ISS, volcans, social_buzz, grid)
- public/map.html : Leaflet dark fullscreen avec markers
ephemeres (60s TTL) pour seismes/foudre/avions/volcans +
marker ISS persistant qui suit la position
Cote launcher :
- ProcessManager : startDataWeb / stopDataWeb + openDataDashboard
/ openDataMap, port :3211 hardcode, script attendu a
metalVizDir/web/server.js
- MenuBarContent : nouvelle row 'Dashboard data-only' en mode
.dataOnly et .bodyMesh + boutons Dashboard/Carte monde quand
le serveur tourne
- stopAll inclut dataWebProc
Ajoute 5 feeds open-data sans cle, branches sur le profil
config.data-only.toml :
- openmeteo : meteo locale (temp, vent, humidite, pression, pluie)
via api.open-meteo.com, poll 10 min, lat/lon configurables
- openaq : qualite de l'air autour d'une position (PM2.5, PM10,
NO2, O3) via openaq.org v3, poll 15 min, radius configurable
- iss : position ISS temps reel via wheretheiss.at, poll 5 s,
emet aussi un event /pass quand la station entre dans un
rayon configurable autour de l'observateur
- volcano : eruptions actives 7-jours via volcano.si.edu (GVP),
poll 1 h, dedup par eventid, emet /eruption + /active count
- social_buzz : Reddit /r/all hot + HackerNews top, poll 1 min,
emet score/comments moyens par source + /pulse normalise
Tous les feeds suivent le pattern Context.send(sub, *args).
Multi-HMR ViT-S is the only S variant published (672x672 only).
Camera bench on M5 MPS: 3.8 fps median (228 ms/frame), no speedup
vs ViT-L — bottleneck is SMPL-X head, not DINOv2 backbone.
Heartbeat moved before no-detect early-return so FPS metric stays
meaningful with empty camera view.
Le toggle 'Fil de fer' du panel etait un placeholder UI sans
rendu. Implementation maintenant fonctionnelle :
- loadFaces precompute les indices d'aretes : pour chaque
triangle (a,b,c) on emet 3 lignes (a,b)(b,c)(c,a). 62724
aretes total pour 20908 triangles SMPL-X.
- makeEntity cree un ModelEntity sibling avec un LowLevelMesh
en topologie .line, UnlitMaterial blanc, attache comme child.
- updateMeshVertices ecrit les memes positions dans le buffer
du wireframe (seulement si actif).
- applyWireframeSetting toggle entity.isEnabled sur tous les
wireframes vivants.
- BodyView.updateNSView appelle applyWireframeSetting avec
settings.showWireframe a chaque tick.
BASE PATH block (TLB:1) and CHARGER TOUT block (TLB:2) shared the
same ~base auto-detect preamble. Merged into one block so .load
evaluates the full loader, not just the base-path setter.
Ajoute une ProcessRow AV-Live-Body toujours visible dans le
menubar (sauf en mode bodyMesh ou la ligne est deja affichee).
L'utilisateur peut maintenant lancer la fenetre RealityKit a la
demande depuis n'importe quel mode pour acceder au panel de
reglages visuels (touche S).
Le guard 'useMultiHMR' dans startBodyApp est supprime : sans
worker Multi-HMR la fenetre affichera juste la cam et un fond
vide pour le mesh, mais le panel reste utilisable pour tester
les sliders / toggles.
/oscope/fx/<name> and /oscope/glitch handlers deleted from
OscClient.cpp — grep across sound_algo/, data_only_viz/,
launcher/, data_feeds/, web_realart/ (*.scd *.py *.swift
*.js *.html) found zero emitters.
OscClient::fx() now always returns its fallback; consumeGlitchPulse()
always returns false — same observable behaviour since these paths
were never populated at runtime.
Local build not verified: openFrameworks not symlinked on GrosMac
(../../../libs/openFrameworksCompiled missing). Change is a pure
handler removal with no logic side-effects.
Le projet est dirige en francais (CLAUDE.md root), donc l'UI
utilisateur suit. Traduction des libelles visibles :
- Reglages AV-Live-Body / Couches / Webcam / Maillage / Lumieres / Vue
- Maillage SMPL-X / Fil de fer / Squelette (articulations)
- Opacite / Metallique / Rugosite
- Principale / Remplissage / Contre-jour
- Champ de vision / Luminosite du fond
Les commentaires et identifiants Swift restent en anglais
conformement aux conventions du repo.
Restructures SettingsPanel into 5 sections : Layers / Webcam /
Mesh / Lights / View. Adds explicit toggles for each visual
layer with a pink icon when active : webcam, mesh fill,
wireframe, skeleton joints. The two new layers (wireframe and
skeleton) are wired into RenderSettings but not yet rendered ;
the toggles are the contract for the next renderer pass.
Panel is now scrollable + larger (320x620) to fit the extra
controls cleanly.
Replace 10475-iteration struct.pack loop with
np.ascontiguousarray(...).tobytes() — one C call per field.
SMPLXPerson fields now typed as np.ndarray; worker stops
converting to Python tuples. Wire format byte-identical.
Adds RenderSettings ObservableObject (cam opacity, bg brightness,
mesh metallic/roughness/visible, 3 light intensities, FOV, panel
visibility) and a SettingsPanel SwiftUI overlay with sliders and
toggles for each.
ContentView wraps BodyView in a ZStack with a corner button + an
on-receive keyboard handler ('S') that toggles the panel. BodyView
applies every setting in updateNSView : preview opacity, container
bg, ARView FOV, the three DirectionalLight intensities, and
MeshRenderer.applyMaterialSettings rebuilds SimpleMaterial with
the new metallic/roughness on each call.
BodyView also tightens the layer chain that was visually ambiguous :
arView.layer.isOpaque = false (clear background actually honored by
the compositor) and preview.zPosition = -100 so the cam stays
strictly behind the AR layer.
BodyView keeps the AVCaptureVideoPreviewLayer behind the ARView
but drops its opacity to 0.35 so the SMPL-X mesh stays the
dominant visual subject while the webcam frame provides spatial
context.
The ARView gets a three-light rig (key 4000 lm warm from upper
right, fill 1500 lm cool from upper left, rim 2000 lm white from
behind) so the per-vertex normals computed in MeshRenderer give
visible shading instead of flat color blobs.
Also captures two follow-up plans : body-mesh-perf-safety and
smplx-test-coverage, both stay as docs only for now.
- bodyMesh keeps SC + data_feeds (audio + sonification) in addition
to Multi-HMR worker headless + AV-Live-Body. Only oscope-of and
the web UI are stopped.
- dataOnly now keeps the web UI (the user wants Hydra/control panel
available even in the Metal-viz-only mode). Only oscope-of is
stopped on transition.
- MenuBar shows Web UI row in both .full and .dataOnly, and a
Data Feeds row in all three modes (with a distinct subtitle in
bodyMesh that flags it as sonification-only).
- ModePicker subtitles updated to reflect the new contract.
A third LaunchMode (.bodyMesh) joins .full and .dataOnly. It runs
the minimal pipeline : Multi-HMR worker in headless mode + the
Swift AV-Live-Body app, nothing else. No SuperCollider, no
oscope-of, no data feeds, no web UI.
applyModeTransition tears down everything else when switching in
and forces useMultiHMR = true so startMetalViz takes the headless
+ AV-Live-Body branch.
MenuBarContent renders a focused two-row layout in this mode :
'Multi-HMR worker' + 'AV-Live-Body'. ModePickerWindow gets a
pink figure.walk card alongside the two existing ones.
The mesh was rendered as flat color blobs because LowLevelMesh
had only a position attribute, leaving SimpleMaterial (which is
lit) without normals to compute shading. Each face ends up with
identical lighting, no depth cue.
createLowLevelMesh now declares a second vertex buffer for
normals (semantic .normal, bufferIndex 1, same float3 stride).
updateMeshVertices recomputes per-vertex normals each frame by
accumulating triangle normals and normalising, then writes the
buffer in place. Cost is O(faces + verts) per person : ~63 k
cross products for 10 475 verts / 20 908 triangles, comfortably
under a millisecond on M5.
Result : the SMPL-X mesh now has visible relief instead of a
chromatic silhouette.
Three coupled changes so the SMPL-X mesh actually shows up in the
RealityKit ARView when fed from Multi-HMR :
- Multi-HMR camera frame is y-down z-forward, RealityKit is
y-up z-back. We flip v.y and v.z in updatePersons, leaving
the entity at the origin since v3d is already absolute camera
coordinates (Python sender ships them post-translation).
- Flipping y and z reverses chirality, so the front-facing
triangles become back-facing and backface culling hides the
whole mesh. We swap each triangle's i1/i2 in loadFaces.
- A DirectionalLight is added in BodyView so SimpleMaterial,
which is lit, doesn't render the mesh as a black silhouette.
Logged once per pid when an entity is first created : the
bounding box of the converted vertices (helps spot scale or
sign issues in the future).
DETRPose ships in three sizes (n/s/l); previously switching required
editing the DEFAULT_MODEL_SIZE constant in source. Add _VALID_SIZES and
_configure_model_size() to DETRPoseWorker for validated size selection,
and wire --detrpose-model-size into the argparse CLI in main.py.
Adds the pytest dev-group used by the upcoming tech-debt cleanup
tasks (test_nlf_worker_bailout, test_renderer_allocations) that
will need 'uv run pytest' to work in CI / locally.
User wants a single visible window. data_only_viz now skips
NSWindow / Metal / HUD entirely when --multi-hmr is set, keeping
only the OSC listener and the Multi-HMR worker that streams
vertices to AV-Live-Body. AV-Live-Body keeps the camera preview
behind the SMPL-X mesh and stays the sole UI surface.
Tradeoff: the generative Metal scenes (storm, tunnel, plasma...)
and the multi-person skeleton overlay drawn by data_only_viz are
not visible in multi-hmr mode. They remain available by running
the launcher without the Multi-HMR toggle.
Both the Python AVCapture path and the Swift CameraPreviewLayer
now enforce the same two-step filter :
1. deviceType == .builtInWideAngleCamera
2. localizedName must NOT contain 'iPhone', 'GSM', 'Desk View',
or 'Continuity'
The second clause matters because under some macOS configurations
(Continuity Camera promoted to widescreen, iPhone mirroring),
externally tethered phone cameras can advertise themselves with
the same primary deviceType.
Swift side also logs via NSLog so the picked device shows up in
Console.app even when stdout is buffered.
Replace per-frame floats list + struct.pack in _update_skeleton
with a preallocated numpy.float32 staging buffer (_skel_cpu_buf),
eliminating the dynamic allocation on every draw call (~60/s).
Adds CameraPreviewLayer that runs its own AVCaptureSession on the
built-in webcam and exposes the AVCaptureVideoPreviewLayer.
BodyView now layers : an NSView with the preview layer as backing
+ an ARView on top with transparent background. The result is the
live webcam visible behind the SMPL-X meshes — useful for debug
and for the performative angle the user wants.
macOS lets multiple processes open the same camera simultaneously
(each with its own AVCaptureSession), so this coexists fine with
the Python Multi-HMR worker that consumes the camera too.
Replaces cv2.VideoCapture by a pyobjc AVFoundation wrapper that
selects the webcam by device type (BuiltInWideAngleCamera) rather
than by an opaque cv2 index, fixing the case where the Continuity
Camera (iPhone) was picked up because cv2's enumeration order does
not match AVCaptureDeviceDiscoverySession's.
_av_capture.AVCapture exposes a cv2-like .read(timeout_s) -> (ok,
BGR HxWx3 uint8). Internally :
- AVCaptureSession + AVCaptureVideoDataOutput
- delegate marked with @objc.python_method
- global GCD queue resolved via libdispatch.dylib (the historical
delegate-on-main-queue crash mentioned in apple_vision_pose is
avoided this way)
- CVPixelBufferGetBaseAddress returns an objc.varlist ; we use its
.as_buffer(n) -> memoryview for the zero-copy numpy view
Measured 30 fps on the MacBook Pro built-in at 1920x1080. The
camera_index flag still works but now means 'index in the
AVFoundation enumeration', which is deterministic.
Add FAIL_THRESHOLD=30 counter: after 30 consecutive inference
failures (NotImplementedError or any Exception), log once at ERROR
and exit the loop instead of spinning at full CPU. Reset on success.
4 findings : silent NLF inference loop, per-frame allocations in
the Metal renderer, unguarded top-level torch import, hardcoded
DETRPose model size. Out of scope here, kept for future cleanup.
cv2.VideoCapture indices on Mac don't follow the AVFoundation
discovery order, so the previous auto-pick still ended up on
the iPhone Continuity stream. resolve_camera_index now probes
each cv2 index, reads a frame, and prefers the one whose mean
luminance is >= 50 (rejects black / standby streams from a
locked iPhone or Desk View).
multi_hmr_worker emits a 5 s heartbeat with the observed fps
and the average detected persons per frame, so 'no detection'
becomes visible in the log instead of staying silent.
The cv2.VideoCapture index on Mac is unpredictable when multiple
cameras are plugged in (Continuity from iPhone, Desk View, external
USB). Adds _camera_select.py that enumerates AVFoundation devices
and picks BuiltInWideAngleCamera by default, mirroring the logic
already used in apple_vision_pose.
Wired through main.py with a --camera-index flag (default -1 = auto)
and MultiHMRWorker honors it. Existing logs now report the chosen
index so the picked device is visible in the launch trace.
Banner on the Multi-HMR plan flipped from OBSOLETE back to
IMPLEMENTED after the NLF re-pivot (NLF TorchScript bundles
a CUDA-hardcoded YOLO detector incompatible with Mac CPU/MPS).
NLF plan stays in tree for the historical analysis.
MULTIHMR_README.md documents the setup, the launch paths
(launcher toggle vs manual), the architecture diagram, the
deviations from the original plan (NLF abandoned, demo.py
bypassed, v3d used directly), expected FPS on M5 MPS, the
cache layout, and a debug table.
multi_hmr_scaffold.md gets a banner flagging the pipeline
as implemented and pointing at the new README.
Each frame the Python sender pushes ~125 KB of vertices per person.
The previous MeshRenderer rebuilt the MeshResource from scratch
every frame, costing a copy + driver upload for each entity.
LowLevelMesh (RealityKit, macOS 15+) keeps a persistent GPU vertex
buffer that we mutate in place via withUnsafeMutableBytes. The
faces buffer is written once at entity creation, and we tag each
ModelEntity with a PidComponent so updateMeshVertices can look up
its LowLevelMesh without touching the entities-by-pid dictionary
from the render path.
Package.swift bumped to swift-tools 6.0 / macOS 15 for the
LowLevelMesh.parts.replaceAll API ; the target stays in Swift 5
language mode so the existing closures don't need Sendable
annotations.
ProcessManager gains a 'useMultiHMR' UserDefault and a 'AV-Live-Body'
process pair (startBodyApp / stopBodyApp). When useMultiHMR is on,
startMetalViz appends '--multi-hmr' to the Python args and, 0.8 s
later, launches the SwiftPM AV-Live-Body executable from
launcher/AV-Live-Body so the RealityKit listener is ready when the
TCP sender connects on :57130.
MenuBarContent adds a Switch under Metal Viz (visible only in
data-only mode) plus a row for AV-Live-Body once it is running.
stopAll terminates the body app along with the rest of the stack.
Path to a working model load on Mac :
- demo.py imports multi_hmr_anny (needs private 'anny' pkg) and
utils.render (needs pyrender + OpenGL offscreen). We skip both
by stubbing the modules in sys.modules and loading Model
directly from model.py.
- utils/__init__.py pulls in roma/trimesh/pillow/tqdm ; added
to the multihmr extra.
- SMPLX_DIR and MEAN_PARAMS are relative paths in
utils/constants.py ; worker chdir's into the repo for the
load, restores cwd in finally. Setup script symlinks
multi-hmr/models -> ../models and downloads
smpl_mean_params.npz from openmmlab-share.
Smoke test : ViT-L checkpoint loads in 4 s, 317 M params,
48 missing / 16 unexpected state-dict keys (normal cross-arch
drift — SMPL_Layer-side, not the backbone).
Adds a standalone SwiftPM executable that renders SMPL-X meshes
in a RealityKit ARView and ingests vertices from the Python TCP
sender on :57130.
Components :
- AVLiveBodyApp + ContentView (SwiftUI App entry, @main)
- BodyView (NSViewRepresentable wrapper around ARView with a
fixed perspective camera 2 m back)
- MeshRenderer (@MainActor ObservableObject ; reads
smplx_faces.bin from Bundle.module ; maintains a ModelEntity
per pid ; rebuilds MeshResource each frame, immutable)
- OSCServer (Network.framework NWListener ; parses the SMPX
binary frame protocol and emits SMPLXPersonData)
Resources are bundled inside Sources/AVLiveBody/Resources/
because SwiftPM rejects '..' traversal in resource paths.
Smoke test : 'nc -zv 127.0.0.1 57130' succeeds after launch.
Wires the Multi-HMR + TCP sender path as priority 0 of
_start_pose_worker so '--multi-hmr' routes the webcam through
the SMPL-X mesh pipeline instead of the Apple Vision / MediaPipe
fallback chain.
The flag is opt-in : without it, behaviour stays identical to
the existing pose-detection chain.
Streams each frame's SMPL-X persons to a localhost listener on
:57130 using a length-prefixed binary frame (~126 KB per person:
magic, pid, confidence, transl, 10 betas, 10 expression, 10475
xyz floats).
UDP would split each frame across 90+ MTUs ; TCP fragmentation
is the right primitive. The sender retries the connection every
second and logs only once every 5 s while the Swift listener is
absent.
Multi-HMR forward pass on the Mac webcam ; multi-person SMPL-X
vertices flow into State.persons_smplx with persistent IDs.
Plan deviated from the real upstream API : load_model lives in the
demo module (not model.utils), the model expects camera intrinsics
K (not just an image tensor), and each human already exposes the
decoded SMPL-X vertices under 'v3d' / joints under 'j3d', so the
SMPLXDecoder is only used for tests now.
One Euro smoothing is applied on shape and expression coefficients
only ; vertices/joints are taken raw because Multi-HMR already gives
temporally consistent meshes per frame.
Adds SMPLXDecoder around smplx.SMPLXLayer with two modes : full decode
from Multi-HMR params (betas+pose+expression+transl) and decode_neutral
for T-pose smoke tests.
Two implementation notes from the test bring-up :
- model_path file -> use file.parent as model_folder (smplx tacks on
SMPLX_<GENDER>.<ext> internally, single level not double).
- pose tensors are rotation matrices (B, N, 3, 3) ; passing zeros
collapses the mesh. decode_neutral uses identity matrices.
Runtime port is 57123 (per bin/data/settings.json) and
all emitters (sound_algo bridge, data_feeds, data_only_viz,
launcher) target 57123. Several internal docs and the
default in ofApp.h still claimed 57122. Align them so the
default matches reality even if settings.json is missing.
Fixes:
- src/ofApp.h:267 default oscListenPort_
- src/OscClient.h:5 header comment
- README.md (3 occurrences, incl. settings.json sample)
- docs/OSC_PROTOCOL.md (5 occurrences incl. diagram)
Root CLAUDE.md (overview, where-to-look, conventions) plus
nested files for data_only_viz, oscope-of, launcher.
sound_algo already had its own nested tree.
Sized for the ~150-instruction budget: root 43 lines,
each nested 43-52 lines, telegraphic style.
Adds the SMPL-X person record (vertices, joints, translation, betas,
expression) and the corresponding list + last-timestamp fields on the
shared State. Mirrors the NLFPerson shape so renderers can switch
backends without touching the upstream pipeline.
User provided SMPLX_NEUTRAL.npz locally so we can generate the real
faces file. Adds the extraction script and the produced binary
(20908 triangles x 3 indices x int32 LE = 250896 bytes) consumed
by the upcoming AV-Live-Body RealityKit target.
Adds the `multihmr` extra (smplx, einops, torchgeometry, ...) and
a setup script that clones the Naver Multi-HMR repo and downloads
multiHMR_896_L (ViT-L, 1.28 GB) with a HuggingFace fallback.
NLF was abandoned because its bundled YOLO detector is TorchScript
traced with CUDA hardcoded (aten::empty_strided fails on CPU/MPS).
Multi-HMR is pure PyTorch and stays MPS-friendly.
SMPL-X NEUTRAL.npz still requires manual registration on
smpl-x.is.tue.mpg.de (MPII academic license).
Add favicon.svg. Refactor WebGL app.js with new scene
structure. Add OSC feed display to audio/hydra/index pages.
Update docker-compose ports and service config.
New standalone visualizer: OSC listener, pose bridge,
Apple Vision / CoreML / YOLO pose backends, Euro filter,
fine analysis, mesh topology, holistic renderer.
Metal shader (scene.metal) for GPU-accelerated drawing.
- Implemented Hydra visualizations in `web_realart/public/hydra/app.js` and `index.html`.
- Created WebGL visualizer using Three.js and WebGPU in `web_realart/public/webgl/app.js` and `index.html`.
- Developed main landing page in `web_realart/public/index.html` to showcase different visualizations.
- Set up WebSocket server in `web_realart/server.js` to relay OSC messages to web clients.
- Added health check API endpoint for monitoring server status.
- Implemented new data feeds for Blitzortung, Bluesky, GCN, GitHub, Mempool, Netzfrequenz, OpenSky, RTE éCO2mix, SWPC, USGS.
- Introduced utility functions for rate limiting and hashing.
- Established OSC communication for data reception and processing.
- Created SuperCollider definitions for handling incoming data and generating audio output based on real-time data.
- Added configuration management for API credentials and polling intervals.
2026-05-11 06:59:44 +02:00
792 changed files with 120886 additions and 170 deletions
-`AV-Live-corrupted-20260514/` — quarantined corrupted snapshot, do not touch
## Domain-specific gotchas
- **mDNS hostnames are required** (`grosmac.local`, `supra-m1.local`) for `AVBODY_HOST` / `MULTIHMR_REMOTE_HOST`. They resist DHCP changes (iPhone hotspot reassigns 172.20.10.x routinely).
- **`POSE_FILTER` chain ordering is load-bearing**: default is `median+kalman+lookahead+ik`. Extras must be inserted at the right stage — `one_euro_joints` BEFORE kalman, `one_euro_bones` AFTER SMPL-X fusion in `multi.py`. `arkit_fuse` overrides 14 body slots with ARKit ARSkeleton3D from iOS app via `/body3d/kp` on `:57128` (always-on listener).
- **`ICP_FUSION=1`** requires `ICP_LIDAR_HOST` (iPhone IP), `ICP_LIDAR_PORT` (default 5500, iPhone ARMesh TCP), and an extrinsic JSON at `~/.config/av-live/lidar_extrinsic.json`. See `docs/ICP_FUSION.md`.
- **iPhone OSC port `57128`** is hardcoded as the publish target for `/body3d/kp` — don't reassign.
- **`avlivebody-mac` requires ad-hoc signing for local dev** (fixed in `85589f2`). Don't strip the signing identity.
- **`onVideoFrame` retain cycle in avlivebody** was fixed in `3b5f29e` — when adding new frame callbacks, mind the strong-self capture.
- **AVLive-Body legacy** has been archived (`9e1482e`); the canonical client is `avlivebody-mac`. Don't reintroduce paths to the old project.
- **macm1 = sink** (Multi-HMR CoreML + Apple Vision ANE + SMPL-X TCP); GrosMac = source. Mind the direction when wiring new OSC topics.
- **Each major sub-project has its own `CLAUDE.md`** — closest wins. Put cross-cutting rules here, sub-project specifics in the nested file.
## When in doubt
- Read root `CLAUDE.md` and the nested `CLAUDE.md` of the sub-project you're editing.
Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout. **RC0.1+ (tag `v0.1.0-rc1`)** ajoute le pipeline Multi-HMR distribué M5 ↔ macm1 sur LAN gigabit + 10 scènes Metal pose-réactives + unified 3D armature wireframe (body+face+hands).
## Communication
Toujours répondre en français à l'utilisateur. Code, commentaires de code, commits et docs en anglais.
| `POSE_FILTER` | `median+kalman+lookahead+ik` | filter chain stages — extra: `one_euro_joints` (joint-space CHI 2012 One Euro, inserted before kalman), `one_euro_bones` (bone-vector One Euro applied after SMPL-X fusion in multi.py), `arkit_fuse` (overrides 14 body slots with ARKit ARSkeleton3D from the iOS app, expects /body3d/kp on :57128) |
| `IPHONE_OSC_PORT` | `57128` | UDP port the iPhone ARBodyTracker app pushes /body3d/kp to (always-on listener in data_only_viz) |
| `VIDEO_ROTATE` | `none` | rotate the source video frame before MediaPipe + display: `none`/`ccw`/`cw`/`180`. Applied at multi.py read; pose detection + skeleton overlay stay aligned. ARKit 3D joints (gravity-aligned world) are unaffected |
| `FINGER_STRIKE_VEL` | `0.02` | downward relative-velocity threshold for a strike (normalized image units/frame) |
| `FINGER_STRIKE_REFRACTORY_MS` | `120` | min ms between strikes per finger |
| `FINGER_SOURCE` | `auto` | hand source for finger gestures: `auto` (iPhone Vision hands if fresh, else MediaPipe), `iphone`, `mediapipe` |
| `FINGER_DEBUG` | `0` | `1` logs each detected strike/pinch (hand, finger) — for threshold tuning |
| `PINCH_RATIO_ON` | `0.50` | thumb-to-finger distance / hand size below which a pinch engages (0.50 validated live 2026-07-02 for ring/pinky reach) |
| `PINCH_RATIO_OFF` | `0.65` | distance/size above which the pinch re-arms (hysteresis) |
| `PINCH_REFRACTORY_MS` | `250` | min ms between pinches per finger |
| `PINCH_MARGIN` | `0.05` | closest-wins margin between nearest and runner-up fingertip (0.05 validated live: 0.20 rejected middle/ring/pinky pinches) |
| `PINCH_EXT_RATIO` | `1.35` | tip-to-wrist distance / hand size above which a finger counts as extended (open-hand pinch gate) |
| `PINCH_EXT_MIN` | `1` | min extended non-pinching fingers required for a pinch to engage; `0` disables the open-hand gate (1 validated live: inner-finger pinches curl neighbours) |
| `PINCH_DEBOUNCE_FRAMES` | `3` | consecutive qualifying frames before a pinch engage fires (release stays immediate); `1` disables |
| `HAND_CONF_MIN` | `0.45` | min Vision hand confidence to draw (raised from 0.3; applies to both overlay and side panels) |
| `HAND_PERSIST_FRAMES` | `3` | consecutive frames a hand track must match before being drawn (1 = off, all hands drawn immediately) |
| `HAND_SWAP_LR` | `0` | Safety knob: invert the Vision chirality interpretation when routing hands (gestures + panels). Chirality validated correct live 2026-07-02; default `0`. Set `1` if a future iPhone app build delivers inverted chirality |
| `HAND_NEAR_MIN` | `0.10` | Minimum wrist-to-middle-MCP distance (normalised) for a hand to trigger pinch/strike. Hands smaller than this are gated out. Display panels are unaffected |
| `HAND_PERSIST_GRACE` | `2` | grace frames before dropping a hand track (0 = strict per-frame) |
| `HAND_NEAR_OFF` | `0.08` | hand size hysteresis off-threshold for gesture slot stabilizer |
| `HAND_HOLD_FRAMES` | `2` | gesture slot: frames to hold last hand on Vision miss |
| `HAND_FACE_MIN` | `0.5` | palm-spread ratio threshold — gesture slot arms only when hand_facing >= this (palm-facing ~0.7-1.0, side-on < 0.4; 0.8× hysteresis for off) |
| `ARKIT_BONE_MAX` | `0.5` | max ARKit body bone length in normalized units; longer bones are skipped (tracking-loss garbage filter) |
- Pas d'emojis dans code/docs/commits sauf demande explicite.
- Commits : sujet ≤ 50 char, body ≤ 72 char/ligne, pas d'attribution AI, pas de `--no-verify`, pas d'underscore dans le scope (hooks enforcent).
-`*.pt`, `*.ckpt`, `*.safetensors`, `*.mlpackage` exclus par `.gitignore` racine.
## Agent Workflow
Explore localise → librarian lit (>500 lignes) → tu raisonnes → general-purpose implémente → validator vérifie. Lance les tâches indépendantes en parallèle.
## Guidance imbriquée
Chaque sous-projet majeur a son propre `CLAUDE.md`. Claude charge automatiquement le plus proche du fichier édité (« closest wins »). N'ajouter ici que ce qui s'applique à TOUS les sous-projets.
> **Live coding audio-visual performance system** built around a SuperCollider sound engine, an openFrameworks oscilloscope visualizer driven by a real Hantek 6022BL USB scope, and a macOS menubar launcher that ties it all together.
**Release : `v0.1.0-rc1` (2026-05-14)** — distributed Multi-HMR (M5 ↔ macm1 LAN), unified 3D armature openpos, hybrid mesh rigger 27 fps perceived, 10 pose-reactive Metal scenes. See [RC0.1+ architecture](#rc01-distributed-multi-hmr-architecture) below.
> **Live coding audio-visual performance system** built around a SuperCollider sound engine, an openFrameworks oscilloscope visualizer driven by a real Hantek 6022BL USB scope, a macOS menubar launcher, and a Metal-native pose / body-mesh visualizer that listens to the same audio bus.
>
> 15 scripted demoparties, 41 fullscreen visual backgrounds, 35 3D parametric meshes, 14 retro OS pixel-art tributes, 1099 SynthDefs across 345 tracks, all driven by FFT analysis of the actual audio physically passing through the scope probes.
> 15 scripted demoparties · ~47 fullscreen visuals (26 3D parametric meshes + 18 procedural shaders + 3 dedicated C++ scenes) · 5 OS pixel-art shaders · 14 retro OS logos · 33 GLSL shader pairs (~65 files) · 1099 SynthDefs across 368 tracks (23 albums × 16) · **20 real-world data feeds** · 7 pose-estimation backends · **SMPL-X body mesh** (10 475 vertices via Multi-HMR + RealityKit) · **3 launch modes** (Full / Data-only / Body Mesh) — all reactive to the audio physically passing through the scope probes.
Metal-native fullscreen visualizer (pyobjc, MTKView, ~60 fps) driven by webcam pose detection. Listens to the same OSC bus as `oscope-of` (`:57123`) so audio sync and data feeds drive its shaders identically, **and** ships pose features back to `sound_algo` on `:57121` to close the loop (the dancer steers the synths).
- **7 pose backends** (drop-in via `pyproject.toml` extras) :
| Backend | Module | Model | Status |
|---------|--------|-------|--------|
| MediaPipe Holistic | `holistic.py` | 33 body + 478 face + 42 hand kp | stable |
| Studio M3 Ultra (80c GPU, *training only*) | est. 35 ms (~30 fps) | — | — |
Hard ceiling on macm1 ≈ 18 fps fresh predict (unified memory bandwidth + CoreML sync overhead) ; further gains require moving MLX servers off macm1 or quantising the model.
### 🎬 Demoparties — 15 narrative demos
Each demoparty is a multi-act scripted show with unique scroller text, background sequence, scroller style, and SuperCollider album track :
@@ -66,7 +208,7 @@ Each demoparty is a multi-act scripted show with unique scroller text, backgroun
| `F9` | CUBE STORM | 8 | Q neurofunk_dnb |
| `F10` | GRAND FINAL | 8 | L future_garage |
Each act lasts 12-55 s, transitions between acts trigger flash + glitch postfx + audio swap. Demos loop on their first act after the last.
Each act lasts 12–55 s, transitions between acts trigger flash + glitch postfx + audio swap. Demos loop on their first act after the last.
### 🎛 Live FX control (Mac AZERTY FR friendly)
-`a z e r t y u i o p` — 10 background scenes (BoingBall, Tunnel, Metaballs, Voronoi, Twister, KIFS, Fire, Mode7, Octahedron, PlasmaC64)
@@ -78,8 +220,32 @@ Each act lasts 12-55 s, transitions between acts trigger flash + glitch postfx +
- **20 feed modules** in `data_feeds/feeds/` ingested by `bridge.py` and broadcast as OSC `/data/<source>/<sub>` to SC (`:57121`), oF (`:57123`) and the web bridge (`:57124`) :
- All three layers share `feeds_client.js` (`window.feeds`) over one WebSocket
- **Web data-only** `data_only_viz/web/` (port `:3211`, Express + WebSocket) — **bidirectional** OSC bridge for the Data-only mode :
- **`/dashboard.html`** — live cards + SVG sparklines for all 19 active feeds (USGS, GDELT, Wiki, ISS, tides + moon, ATC, ...) with severity classes (alert / warn / green)
- **`/map.html`** — Leaflet dark fullscreen with markers ephemeres for geocoded feeds (quakes, lightning, planes, volcanoes, GDELT events) + persistent ISS marker
- **`/control.html`** — 3-pane control surface : 7 synth/mix sliders (master, cutoff, reso, reverb, delay, tempo), 10 audio scene buttons (`/scene/play`), 9 visual mode buttons routed to oF (`/control/vizMode` → `:57123`), XY pad (`/xy/{x,y}`)
- **SC retour** : `sound_algo/control/web_bridge.scd` listens on `:57121` for `/control/*``/scene/*``/xy/*` and pushes `/sync/bpm|beat|rms|voices` to web on `:57125` at 4 Hz
Open the menubar icon for per-process Start/Stop, logs, and the "AV-Live-Body" launch button available from any mode.
### Without Hantek hardware
oscope-of works without the scope plugged in. It falls back to synthesized signals from the OSC `/sync/*` metadata sent by `sound_algo`. You lose the audio-derived FFT but the demoscene visuals + narrative demos all still play.
`oscope-of` works without the scope plugged in. It falls back to synthesized signals from the OSC `/sync/*` metadata sent by `sound_algo`. You lose the audio-derived FFT but the demoscene visuals + narrative demos all still play.`data_only_viz` runs identically since it only needs the OSC bus + a webcam.
## Demoparty mode
Press a digit (1-9, 0) or F-key (F6-F10) to launch a 1-3 minute scripted demoparty. Each one is a journey through audio + visual transitions, with greetings to the demoscene legends along the way (Fairlight, Razor 1911, ASD, Conspiracy, Farbrausch, TBL, Mercury, TPOLM, Loonies, Hokuto Force, IQ, Knighty, Smash, Gargaj…).
Press a digit (1–9, 0) or F-key (F6–F10) to launch a 1–3 minute scripted demoparty. Each one is a journey through audio + visual transitions, with greetings to the demoscene legends along the way (Fairlight, Razor 1911, ASD, Conspiracy, Farbrausch, TBL, Mercury, TPOLM, Loonies, Hokuto Force, IQ, Knighty, Smash, Gargaj…).
The `0` GREETINGS demo cycles through 18 acts including Boing Ball, plasma C64, dot tunnel, twister bars, copper bars, fractal KIFS, Möbius topology, then a parade of OS logos (Win 1.0, Lotus 1-2-3, Atari, Apple, OS/2 Warp, IRIX, ZX Spectrum), ending on a roll call of demoscene groups.
@@ -140,14 +351,21 @@ The `0` GREETINGS demo cycles through 18 acts including Boing Ball, plasma C64,
Visualiseur natif Metal (pyobjc) pour le mode data-only d'AV-Live : capture caméra → détection pose multi-personne → tracker → rendu Metal → OSC out vers `oscope-of`.
uv run python -m data_only_viz.main # lancement standard
```
Python **3.11+** requis. `pyproject.toml` est la source de vérité — ne jamais éditer `uv.lock` à la main.
## Backends pose disponibles
| Backend | Fichier | Statut |
|---------|---------|--------|
| MediaPipe Holistic | `holistic.py` | stable |
| MediaPipe multi (Pose+Face+Hand) | `multi.py` | stable ; `MEDIAPIPE_DELEGATE=gpu` (défaut) ou `cpu`. **GPU Metal exige SRGBA 4-ch** (3-ch SRGB crashe `gpu_buffer_storage_cv_pixel_buffer.cc`) — multi.py route auto vers `cv2.COLOR_BGR2RGBA` + `mp.ImageFormat.SRGBA` quand delegate=GPU. Bench M5 image-mode SRGBA : pose 2.9 vs 6.7 ms (GPU/CPU), face 1.0 vs 4.1, hand 3.2 vs 6.1 |
| Ultralytics YOLOv8-pose | `pose.py` | stable, modèle `yolov8n-pose.pt` à la racine repo |
uv run python -m data_only_viz.training.eval --ckpt ~/.cache/av-live-action/checkpoints/action_head.pt --dataset ~/.cache/av-live-action/dataset/dataset.jsonl
# Live : publisher déjà câblé dans multi.py, aucune action requise
```
Checkpoint par défaut : `~/.cache/av-live-action/checkpoints/action_head.pt`. Absent → random init (warmup retourne `debout`).
## Tests
```bash
uv run pytest tests/ -v
```
Tests TDD-first pour `nlf_worker.py` ; valider avant chaque commit qui touche un worker.
Suite action-head (8 fichiers, 39 tests) : `tests/test_action_head_*.py`, `tests/test_{dataset,autolabel,augment,training_smoke,pose_bridge_action}.py`. Tous doivent rester verts avant chaque commit qui touche `action_head*.py` ou `training/*.py`.
## Anti-patterns
- Ne pas charger un modèle ML sans guard `try/except ImportError` — les optional-extras peuvent manquer.
- Ne pas committer `*.pt`, `*.ckpt`, `*.safetensors`, `*.mlpackage` (gitignore racine).
- Ne pas appeler `state.persons_nlf = ...` hors `with state.lock():`.
- Ne pas hardcoder le device (`cuda`/`mps`/`cpu`) : détecter via `torch.backends.mps.is_available()` puis fallback.
- Pas de `print` dans la boucle de rendu — utiliser un logger conditionnel.
# scores: (1, N), keypoints: (1, N, 17, 2) en pixels image originale
scores0=scores[0].detach().cpu().numpy()
kps0=keypoints[0].detach().cpu().numpy()
idx=scores0>self.score_thresh
sel_scores=scores0[idx]
sel_kps=kps0[idx]
# Trier par score decroissant et limiter a num_persons
order=(-sel_scores).argsort()[:self.num_persons]
bodies:list[list[PoseKp]]=[]
foriinorder:
conf=float(sel_scores[i])
pts=sel_kps[i]# (17, 2) en pixels
kp_list=[]
forkx,kyinpts:
kp_list.append(PoseKp(
x=float(kx)/max(1,w),
y=float(ky)/max(1,h),
z=0.0,
c=conf,
))
bodies.append(kp_list)
# Encode webcam JPEG pour overlay
ok2,jpg=cv2.imencode(".jpg",frame_bgr,
[int(cv2.IMWRITE_JPEG_QUALITY),70])
jpg_bytes=bytes(jpg)ifok2elseNone
withself.state.lock():
self.state.persons_body=bodies
# DETRPose ne fournit pas face/hands — on vide pour
# eviter que le renderer dessine des anciennes valeurs.
self.state.persons_face=[]
self.state.persons_hands=[]
self.state.face_present=False
self.state.hands_present=False
ifbodies:
self.state.body_present=True
# Compat single-person : on remplit les 17 premiers
# slots du buffer body_kp (mediapipe en attend 33,
# le reste reste a zero — acceptable).
forkinrange(33):
ifk<17andk<len(bodies[0]):
self.state.body_kp[k]=bodies[0][k]
else:
self.state.body_kp[k]=PoseKp()
# On remplit aussi pose_kp[17] (legacy YOLO COCO).
forkinrange(17):
self.state.pose_kp[k]=(
bodies[0][k]ifk<len(bodies[0])elsePoseKp())
else:
self.state.body_present=False
self.state.pose_count=len(bodies)
self.state.pose_last_t=time.monotonic()
ifjpg_bytes:
self.state.last_webcam_jpeg=jpg_bytes
dt=time.monotonic()-tA
ifdt<self.period:
time.sleep(self.period-dt)
cap.release()
LOG.info("detrpose worker stopped")
def_load_py_config(path:str):
"""Charge une config DETRPose ecrite en .py (style detectron2/lazy)."""
fromomegaconfimportOmegaConf
# Les configs DETRPose sont des fichiers Python qui exposent un dict
# `model = LazyCall(...)`. On utilise le helper lazy_config si dispo.
# DETRPose exposes LazyConfig from src.core (see tools/inference/torch_inf.py);
# the older src.misc.lazy_config path is kept as a fallback. LazyConfig.load
# handles the config's relative imports (`from .include...`) that a raw exec
# cannot.
for_modin("src.core","src.misc.lazy_config"):
try:
mod=__import__(_mod,fromlist=["LazyConfig"])
returnmod.LazyConfig.load(path)
except(ImportError,AttributeError):
continue
# Fallback minimal : exec + recup des noms cles.
ns:dict={}
withopen(path)asf:
code=compile(f.read(),path,"exec")
exec(code,ns)
cfg=OmegaConf.create({
k:ns[k]forkin("model","postprocessor")
ifkinns
})
returncfg
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
L'électron rare
electron