Compare commits
360 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 L'électron rare
|
2908dc4c28 | ||
|
L'électron rare
|
042da7faa0 | ||
|
L'électron rare
|
7b458b8b58 | ||
|
L'électron rare
|
dc766fe896 | ||
|
clement
|
25ce6e0139 | ||
|
L'électron rare
|
c535367817 | ||
|
clement
|
a87044f698 | ||
|
clement
|
be02a71bde | ||
|
clement
|
74e43d6915 | ||
|
clement
|
8c7a1aee00 | ||
|
clement
|
2d64e8d6aa | ||
|
L'électron rare
|
68e5c5c8a8 | ||
|
clement
|
8e6434cf2a | ||
|
clement
|
25acb50e59 | ||
|
clement
|
d1c9485904 | ||
|
clement
|
74b1ccfa1a | ||
|
clement
|
07268dd8f3 | ||
|
clement
|
89d201b068 | ||
|
clement
|
3d38b1006e | ||
|
clement
|
296dc99435 | ||
|
clement
|
39ac9406a8 | ||
|
clement
|
8f6886f44e | ||
|
clement
|
9f7f8558d4 | ||
|
clement
|
3eb6426f61 | ||
|
clement
|
8a3e7fa5cb | ||
|
clement
|
162eab3d2e | ||
|
clement
|
2a03a40270 | ||
|
clement
|
0edaf7b2ba | ||
|
clement
|
05f16aa886 | ||
|
clement
|
d11addea4a | ||
|
clement
|
532b23735c | ||
|
L'électron rare
|
684ea5d4d3 | ||
|
clement
|
96ea70a645 | ||
|
clement
|
f5527369ca | ||
|
clement
|
b90d0ad802 | ||
|
clement
|
a95db5efd7 | ||
|
clement
|
a17e417fa3 | ||
|
clement
|
51f5bff1bf | ||
|
L'électron rare
|
ce46a27903 | ||
|
clement
|
b154d1cbd4 | ||
|
L'électron rare
|
7675c2ac11 | ||
|
clement
|
daeb1e6c1e | ||
|
clement
|
a6dce8e712 | ||
|
clement
|
1df8b89ac3 | ||
|
clement
|
210a224e68 | ||
|
clement
|
a01d09287b | ||
|
clement
|
8bb1b778ed | ||
|
L'électron rare
|
6a14b3b67d | ||
|
clement
|
fd60783047 | ||
|
L'électron rare
|
29e0df3378 | ||
|
clement
|
96bb1151a7 | ||
|
clement
|
97905fa2e6 | ||
|
clement
|
3ece3c9e8e | ||
|
clement
|
e33796c1b4 | ||
|
clement
|
1b922d446f | ||
|
clement
|
d2636d4a12 | ||
|
clement
|
38f1974455 | ||
|
clement
|
744ba6475a | ||
|
clement
|
528a9398b5 | ||
|
clement
|
2d767e8d98 | ||
|
L'électron rare
|
f9da19f547 | ||
|
clement
|
987df60b10 | ||
|
L'électron rare
|
529d6c5cd9 | ||
|
clement
|
3fb9e0eca0 | ||
|
clement
|
b8098de39e | ||
|
L'électron rare
|
46c88236f6 | ||
|
clement
|
2bfa846631 | ||
|
clement
|
128bc3f1b4 | ||
|
L'électron rare
|
6482baf22c | ||
|
clement
|
cff42d0863 | ||
|
clement
|
d5a1de0938 | ||
|
L'électron rare
|
16adac4bec | ||
|
clement
|
b65d16843e | ||
|
L'électron rare
|
b141127a13 | ||
|
L'électron rare
|
c81f3d8587 | ||
|
L'électron rare
|
2ed077bbf5 | ||
|
clement
|
9c8c271113 | ||
|
clement
|
b80ba66d4c | ||
|
clement
|
6c7cd45ecd | ||
|
clement
|
0069b82875 | ||
|
clement
|
c483977c34 | ||
|
clement
|
f9d64cd7a5 | ||
|
clement
|
df2bceca74 | ||
|
clement
|
a484ab943c | ||
|
clement
|
3ce4c38f1c | ||
|
clement
|
7ca9fd98ff | ||
|
L'électron rare
|
6843e5aafc | ||
|
clement
|
015374af38 | ||
|
L'électron rare
|
2d927107f1 | ||
|
L'électron rare
|
ee7f349359 | ||
|
L'électron rare
|
2250fe347f | ||
|
clement
|
53f9111ed8 | ||
|
clement
|
fe7b25e536 | ||
|
clement
|
8d95f3e455 | ||
|
clement
|
e9022fab7d | ||
|
clement
|
222c087610 | ||
|
clement
|
141ed0e122 | ||
|
L'électron rare
|
2f4a80ce2b | ||
|
L'électron rare
|
495c5680bf | ||
|
clement
|
90ded48a57 | ||
|
clement
|
62957f09d5 | ||
|
L'électron rare
|
7b90acf80e | ||
|
L'électron rare
|
1990fc692f | ||
|
clement
|
17d78127f6 | ||
|
L'électron rare
|
d7c1049661 | ||
|
L'électron rare
|
ab88b0f6df | ||
|
L'électron rare
|
ca21a4216d | ||
|
clement
|
562ac45d13 | ||
|
clement
|
2777243d5c | ||
|
clement
|
77249049c9 | ||
|
clement
|
fab35a5520 | ||
|
clement
|
2aac8f57f6 | ||
|
clement
|
9ad0ddda13 | ||
|
clement
|
bb11311720 | ||
|
L'électron rare
|
6ec864ed7d | ||
|
L'électron rare
|
16f68efe01 | ||
|
clement
|
57abc2ae0f | ||
|
clement
|
463bab7665 | ||
|
clement
|
b8f5cc0b2e | ||
|
L'électron rare
|
b071c31b86 | ||
|
L'électron rare
|
582d955522 | ||
|
L'électron rare
|
714c5988c2 | ||
|
L'électron rare
|
ebe15d0804 | ||
|
L'électron rare
|
0fa6cac2bf | ||
|
L'électron rare
|
91c4c5c599 | ||
|
clement
|
6a286b4f0e | ||
|
L'électron rare
|
6566e7e410 | ||
|
clement
|
154db324cf | ||
|
L'électron rare
|
8530f92c15 | ||
|
L'électron rare
|
2213b4e53d | ||
|
clement
|
217e41c632 | ||
|
L'électron rare
|
67fd253de5 | ||
|
clement
|
c1b567f116 | ||
|
L'électron rare
|
c90cbb103c | ||
|
L'électron rare
|
a74d3fb402 | ||
|
clement
|
5c0007f961 | ||
|
L'électron rare
|
1d62ff4347 | ||
|
L'électron rare
|
73c17d41f7 | ||
|
clement
|
e1cf2af339 | ||
|
clement
|
764bf02dc8 | ||
|
clement
|
4cabf12eac | ||
|
L'électron rare
|
427341b8a1 | ||
|
clement
|
858762fe39 | ||
|
clement
|
6b7cae277e | ||
|
L'électron rare
|
2ad47c705e | ||
|
L'électron rare
|
4414bbf168 | ||
|
clement
|
85379be922 | ||
|
clement
|
211ee2b323 | ||
|
clement
|
2b41516a20 | ||
|
L'électron rare
|
9e32e3afd8 | ||
|
L'électron rare
|
048403e92b | ||
|
clement
|
8ad309de33 | ||
|
clement
|
2e32dec51c | ||
|
L'électron rare
|
1101758719 | ||
|
L'électron rare
|
2d05ac9372 | ||
|
L'électron rare
|
7f44ee527c | ||
|
L'électron rare
|
82ac0181a3 | ||
|
L'électron rare
|
73432a0c96 | ||
|
L'électron rare
|
0f5a9cd098 | ||
|
L'électron rare
|
db2f331bd1 | ||
|
L'électron rare
|
c1b0cf07b6 | ||
|
L'électron rare
|
b145655892 | ||
|
L'électron rare
|
3ec306fdb2 | ||
|
L'électron rare
|
e23b97f048 | ||
|
L'électron rare
|
abfe950c25 | ||
|
L'électron rare
|
aa37b7832c | ||
|
L'électron rare
|
3d33719f2b | ||
|
L'électron rare
|
241b93046b | ||
|
L'électron rare
|
4adc118e2b | ||
|
L'électron rare
|
3b3b1fe51b | ||
|
L'électron rare
|
1a6503cb71 | ||
|
L'électron rare
|
5343913a7a | ||
|
L'électron rare
|
a98957ce46 | ||
|
L'électron rare
|
c7f98c24af | ||
|
L'électron rare
|
a88fc5e643 | ||
|
L'électron rare
|
24613cd809 | ||
|
L'électron rare
|
0d467da088 | ||
|
L'électron rare
|
c3fdb11e65 | ||
|
L'électron rare
|
cdee717d51 | ||
|
L'électron rare
|
031bb16e0f | ||
|
L'électron rare
|
abedc53a4c | ||
|
L'électron rare
|
245edacd4f | ||
|
L'électron rare
|
337f791b7a | ||
|
L'électron rare
|
a9ce3d58f5 | ||
|
L'électron rare
|
1660640d48 | ||
|
L'électron rare
|
114ebbd99e | ||
|
L'électron rare
|
6b6be7bff2 | ||
|
L'électron rare
|
46d912e526 | ||
|
L'électron rare
|
b487038f30 | ||
|
L'électron rare
|
15578ec60a | ||
|
L'électron rare
|
4285d160c3 | ||
|
L'électron rare
|
883ada7d61 | ||
|
L'électron rare
|
1cd8c2e259 | ||
|
L'électron rare
|
bcfb86ae53 | ||
|
L'électron rare
|
d8ea115a1b | ||
|
L'électron rare
|
66a16d263b | ||
|
L'électron rare
|
abccc1c7b3 | ||
|
L'électron rare
|
0a3798ea38 | ||
|
L'électron rare
|
79842c8bc0 | ||
|
L'électron rare
|
0080cc489f | ||
|
L'électron rare
|
0cfaa0451c | ||
|
L'électron rare
|
cc0d5270cb | ||
|
L'électron rare
|
c4e4511262 | ||
|
L'électron rare
|
5dcefb3420 | ||
|
L'électron rare
|
42460abf6f | ||
|
L'électron rare
|
04764b6d6f | ||
|
L'électron rare
|
0757450648 | ||
|
L'électron rare
|
3314b58b69 | ||
|
L'électron rare
|
57e3a200bc | ||
|
L'électron rare
|
8c4d9f4d7b | ||
|
L'électron rare
|
50772f3b9b | ||
|
L'électron rare
|
a79bc14dc3 | ||
|
L'électron rare
|
9e3cfc308a | ||
|
L'électron rare
|
b259f5437f | ||
|
L'électron rare
|
54d07aca27 | ||
|
L'électron rare
|
8d09ae555b | ||
|
L'électron rare
|
8954198648 | ||
|
L'électron rare
|
8c8de924db | ||
|
L'électron rare
|
2ec59571c9 | ||
|
L'électron rare
|
83ea2be4e6 | ||
|
L'électron rare
|
c2c9753c8c | ||
|
L'électron rare
|
2e484ed2ed | ||
|
L'électron rare
|
73f85aa276 | ||
|
L'électron rare
|
082bb19dd5 | ||
|
L'électron rare
|
83dc910624 | ||
|
L'électron rare
|
480aa79ef4 | ||
|
L'électron rare
|
8a4a0e688d | ||
|
L'électron rare
|
32a4ef2232 | ||
|
L'électron rare
|
f504ad4f41 | ||
|
L'électron rare
|
b40991f7a8 | ||
|
L'électron rare
|
48d019cc77 | ||
|
L'électron rare
|
b1e53da2bb | ||
|
L'électron rare
|
f9d5614d4d | ||
|
L'électron rare
|
0475ea0340 | ||
|
L'électron rare
|
28078dbeba | ||
|
L'électron rare
|
10873707e4 | ||
|
L'électron rare
|
131f6ed5a8 | ||
|
L'électron rare
|
8afc44b2ca | ||
|
L'électron rare
|
dac96481f2 | ||
|
L'électron rare
|
598f00ea31 | ||
|
L'électron rare
|
5629a7e859 | ||
|
L'électron rare
|
ca25732265 | ||
|
L'électron rare
|
5dfbfc7ae9 | ||
|
L'électron rare
|
e7b107a96e | ||
|
L'électron rare
|
b918eeea16 | ||
|
L'électron rare
|
01b0c56ea2 | ||
|
L'électron rare
|
b4b8139902 | ||
|
L'électron rare
|
242c8ec751 | ||
|
L'électron rare
|
1e9f711278 | ||
|
L'électron rare
|
c5d89e740c | ||
|
L'électron rare
|
cf8d24d629 | ||
|
L'électron rare
|
b01a35326b | ||
|
L'électron rare
|
5ed6aeb388 | ||
|
L'électron rare
|
a8c803403c | ||
|
L'électron rare
|
f106b35fa3 | ||
|
L'électron rare
|
fe69f99ba2 | ||
|
L'électron rare
|
cf6eb9c47e | ||
|
L'électron rare
|
ff2ef5dfb8 | ||
|
L'électron rare
|
886b90b84d | ||
|
L'électron rare
|
9ab238bd2d | ||
|
L'électron rare
|
2a3fdb1d38 | ||
|
L'électron rare
|
088b233a48 | ||
|
L'électron rare
|
de2748accb | ||
|
L'électron rare
|
ec2c387091 | ||
|
L'électron rare
|
7a4f14056a | ||
|
L'électron rare
|
71d0a47723 | ||
|
L'électron rare
|
52b902d3e3 | ||
|
L'électron rare
|
d7c8a3e687 | ||
|
L'électron rare
|
8657805eaf | ||
|
L'électron rare
|
c5941b9625 | ||
|
L'électron rare
|
df2b068c2b | ||
 electron
|
1b88da7053 | ||
|
Clément SAILLANT
|
5c61112826 | ||
|
L'électron rare
|
68a56057c4 | ||
|
L'électron rare
|
986d03c09d | ||
|
L'électron rare
|
85589f2627 | ||
|
L'électron rare
|
a22027edb7 | ||
|
L'électron rare
|
a46cbd471c | ||
|
L'électron rare
|
0ee7ddddcd | ||
|
L'électron rare
|
3b5f29e5c7 | ||
|
L'électron rare
|
9e1482e29d | ||
|
L'électron rare
|
ea9c96f716 | ||
|
L'électron rare
|
8cf3d90073 | ||
|
L'électron rare
|
6dab2bcf72 | ||
|
L'électron rare
|
6aea7e0230 | ||
|
L'électron rare
|
819c9688e7 | ||
|
L'électron rare
|
177214d913 | ||
|
L'électron rare
|
76985a2ae9 | ||
|
L'électron rare
|
010b5caaa9 | ||
|
L'électron rare
|
c0834ed7ab | ||
|
L'électron rare
|
87982efb0b | ||
|
L'électron rare
|
ddb4ee3e9d | ||
|
L'électron rare
|
db33e74c15 | ||
|
L'électron rare
|
d8aa727887 | ||
|
L'électron rare
|
950cb29628 | ||
|
L'électron rare
|
360a0745de | ||
|
L'électron rare
|
555efa1b4b | ||
|
L'électron rare
|
914e52deaf | ||
|
L'électron rare
|
7fc4f41f4e | ||
|
L'électron rare
|
e33375f7ac | ||
|
L'électron rare
|
af66ecaecd | ||
|
L'électron rare
|
73f8f75771 | ||
|
L'électron rare
|
8099b79ec8 | ||
|
L'électron rare
|
e06f53b11e | ||
|
L'électron rare
|
b6a85e24fa | ||
|
L'électron rare
|
3ec973d2ce | ||
|
L'électron rare
|
8c24db8e03 | ||
|
L'électron rare
|
131c93f885 | ||
|
L'électron rare
|
94124f974e | ||
|
L'électron rare
|
30fa7276c8 | ||
|
L'électron rare
|
a8be68adff | ||
|
L'électron rare
|
35b18f6ca5 | ||
|
L'électron rare
|
1af056dbf1 | ||
|
L'électron rare
|
9b82a90579 | ||
|
L'électron rare
|
39457dde41 | ||
|
L'électron rare
|
eeb5726103 | ||
|
L'électron rare
|
4cca03db9f | ||
|
L'électron rare
|
fddb31e3ee | ||
|
L'électron rare
|
5e58c0c4fe | ||
|
L'électron rare
|
4463ca0d43 | ||
|
L'électron rare
|
04fb49aae3 | ||
|
L'électron rare
|
c8139a2bea | ||
|
L'électron rare
|
921ae9cd42 | ||
|
L'électron rare
|
20d91c037f | ||
|
L'électron rare
|
4b6386ab5a | ||
|
L'électron rare
|
7211ca8393 | ||
|
L'électron rare
|
f79ce95f78 | ||
|
L'électron rare
|
9eff9daad8 | ||
|
L'électron rare
|
9fd856744d | ||
|
L'électron rare
|
549a6b53a6 | ||
|
L'électron rare
|
7949f50920 | ||
|
L'électron rare
|
ebed40c98a | ||
|
L'électron rare
|
b8d76ccb27 | ||
|
L'électron rare
|
e70595d81a | ||
|
L'électron rare
|
1848368bc2 | ||
|
L'électron rare
|
630b4aa6c0 | ||
|
L'électron rare
|
d7f3e3a2c1 | ||
|
L'électron rare
|
e7b3b63f5f | ||
|
L'électron rare
|
ec4be7bd1f | ||
|
L'électron rare
|
66ad49fe43 | ||
|
L'électron rare
|
0ab7c56036 | ||
|
L'électron rare
|
077a7f707f | ||
|
L'électron rare
|
b15dc75ad5 | ||
|
L'électron rare
|
d9fb4ac503 | ||
|
L'électron rare
|
bb06f349c7 | ||
|
L'électron rare
|
698fe9f7da | ||
|
L'électron rare
|
f951aacc9a | ||
|
L'électron rare
|
95c3bb5687 | ||
|
L'électron rare
|
a769bcd7ae | ||
|
L'électron rare
|
d122c93d19 | ||
|
L'électron rare
|
89e1be4803 | ||
|
L'électron rare
|
6848c798cc | ||
|
L'électron rare
|
d3e6297091 | ||
|
L'électron rare
|
7d0711b3f6 | ||
|
L'électron rare
|
d077f6564e | ||
|
L'électron rare
|
b6cd8cf4c7 | ||
|
L'électron rare
|
bd46f6e052 | ||
|
L'électron rare
|
1828d7ca3e | ||
|
L'électron rare
|
67302e7ca2 | ||
|
L'électron rare
|
9838da3137 |
+33
@@ -10,6 +10,19 @@ launcher/**/xcuserdata/
|
||||
launcher/**/*.xcuserstate
|
||||
launcher/**/Package.resolved
|
||||
|
||||
# iphone-arbody — xcodeproj is generated by xcodegen, Local.xcconfig
|
||||
# carries personal Apple Developer Team ID. Both are local-only.
|
||||
iphone-arbody/ARBodyTracker.xcodeproj/
|
||||
iphone-arbody/Config/Local.xcconfig
|
||||
iphone-arbody/**/xcuserdata/
|
||||
iphone-arbody/**/*.xcuserstate
|
||||
iphone-arbody/build/
|
||||
iphone-arbody/DerivedData/
|
||||
|
||||
# SwiftPM build + editor artifacts
|
||||
.build/
|
||||
.swiftpm/
|
||||
|
||||
# openFrameworks — on garde les shaders + settings.json pour qu'ils
|
||||
# arrivent sur les autres machines, mais on ignore les binaires.
|
||||
oscope-of/bin/*
|
||||
@@ -44,3 +57,23 @@ sound_algo/*.log
|
||||
.vscode/
|
||||
*.swp
|
||||
*~
|
||||
|
||||
# tool session state (claude-mem / remember skill)
|
||||
.remember/
|
||||
.serena/
|
||||
.claude/agent-memory/
|
||||
|
||||
# Swift coverage artifacts
|
||||
*.profraw
|
||||
|
||||
# local python venvs (dirs or symlinks)
|
||||
**/.venv
|
||||
|
||||
# corrupted recovery backups
|
||||
AV-Live-corrupted-*/
|
||||
|
||||
# macOS / iCloud collision artifacts (auto-created on rename)
|
||||
*\ 2
|
||||
*\ 2.*
|
||||
*\ 3
|
||||
*\ 3.*
|
||||
|
||||
+1
-1
@@ -1,3 +1,3 @@
|
||||
[submodule "third_party/SMPLer-X"]
|
||||
path = third_party/SMPLer-X
|
||||
url = https://github.com/electron-rare/SMPLer-X.git
|
||||
url = https://git.saillant.cc/electron-rare/SMPLer-X.git
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
# AGENTS.md
|
||||
|
||||
Guidance for AI coding agents (Claude Code, Aider, Cursor, etc.) working in this repo.
|
||||
|
||||
## Project
|
||||
|
||||
`AV-Live` — live-coding audio-visual performance system: SuperCollider sound engine, openFrameworks visualiser driven by a Hantek 6022BL oscilloscope, and a SwiftUI menubar launcher orchestrating everything. Public, GPL-3. Repo `electron-rare/AV-Live`, branch `main`. Multi-host: GrosMac (source), macm1 (sink / Multi-HMR + Apple Vision ANE), iPhone 16 Pro (ARKit/LiDAR pub).
|
||||
|
||||
## Tech stack (per sub-project)
|
||||
|
||||
| Sub-project | Stack |
|
||||
|-------------|-------|
|
||||
| `sound_algo/` | SuperCollider (sclang + scsynth), 1099 SynthDefs, 345 tracks |
|
||||
| `oscope-of/` | openFrameworks C++, libusb (Hantek bulk), GLSL 150 / GL 3.2 core |
|
||||
| `launcher/` | SwiftUI menubar app, Swift Package Manager |
|
||||
| `data_only_viz/` | Python 3.11+ via `uv`, native Metal (pyobjc), multi-backend pose |
|
||||
| `data_feeds/` | Python data ingestion |
|
||||
| `web_realart/` | Node.js, Express, OSC bridge |
|
||||
| `avlivebody-mac/` | SwiftUI body-tracking client (ARKit/SMPL-X mesh, ad-hoc signed for local dev) |
|
||||
| `iphone-arbody/` | iOS app, ARBodyTracker, publishes `/body3d/kp` via OSC |
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
# Python sub-projects (uv only)
|
||||
cd data_only_viz && uv sync && uv run python -m data_only_viz
|
||||
cd data_feeds && uv sync
|
||||
|
||||
# openFrameworks
|
||||
cd oscope-of && make -j
|
||||
|
||||
# Web bridge
|
||||
cd web_realart && npm install && npm start
|
||||
|
||||
# Swift
|
||||
open launcher/Package.swift # or xcodebuild from CLI
|
||||
open avlivebody-mac/avlivebody.xcodeproj
|
||||
```
|
||||
|
||||
## Conventions
|
||||
|
||||
- Commits: subject ≤ 50 chars, body ≤ 72, no underscore in scope, no AI attribution, never `--no-verify` (hooks enforce).
|
||||
- Branches: `feat/<name>`, `fix/<name>`, `docs/<name>`, `refactor/<name>`, `chore/<name>`.
|
||||
- Language: French to the user, English in code/comments/commits.
|
||||
- No emojis in code/docs/commits unless explicitly requested.
|
||||
- Python: **always `uv`** (never pip/poetry/conda directly).
|
||||
- `.gitignore` already excludes `*.pt`, `*.ckpt`, `*.safetensors`, `*.mlpackage` at root — don't commit weights.
|
||||
- License: GPL-3 (whole repo) — keep new files under a compatible license header when adding third-party code.
|
||||
|
||||
## File layout
|
||||
|
||||
- `sound_algo/` — SC sound engine (own `CLAUDE.md`)
|
||||
- `oscope-of/` — visualiser
|
||||
- `launcher/` — macOS menubar
|
||||
- `data_only_viz/` — pose / mesh / body tracking pipeline (Metal)
|
||||
- `data_feeds/` — data ingestion
|
||||
- `web_realart/` — web UI + OSC bridge
|
||||
- `avlivebody-mac/`, `iphone-arbody/` — body-tracking clients
|
||||
- `shared/` — cross-sub-project assets
|
||||
- `third_party/` — vendored deps (CHECK before adding to root deps)
|
||||
- `tools/` — helper scripts
|
||||
- `docs/superpowers/plans/` — in-flight plans/specs
|
||||
- `AV-Live-corrupted-20260514/` — quarantined corrupted snapshot, do not touch
|
||||
|
||||
## Domain-specific gotchas
|
||||
|
||||
- **mDNS hostnames are required** (`grosmac.local`, `supra-m1.local`) for `AVBODY_HOST` / `MULTIHMR_REMOTE_HOST`. They resist DHCP changes (iPhone hotspot reassigns 172.20.10.x routinely).
|
||||
- **`POSE_FILTER` chain ordering is load-bearing**: default is `median+kalman+lookahead+ik`. Extras must be inserted at the right stage — `one_euro_joints` BEFORE kalman, `one_euro_bones` AFTER SMPL-X fusion in `multi.py`. `arkit_fuse` overrides 14 body slots with ARKit ARSkeleton3D from iOS app via `/body3d/kp` on `:57128` (always-on listener).
|
||||
- **`ICP_FUSION=1`** requires `ICP_LIDAR_HOST` (iPhone IP), `ICP_LIDAR_PORT` (default 5500, iPhone ARMesh TCP), and an extrinsic JSON at `~/.config/av-live/lidar_extrinsic.json`. See `docs/ICP_FUSION.md`.
|
||||
- **iPhone OSC port `57128`** is hardcoded as the publish target for `/body3d/kp` — don't reassign.
|
||||
- **`avlivebody-mac` requires ad-hoc signing for local dev** (fixed in `85589f2`). Don't strip the signing identity.
|
||||
- **`onVideoFrame` retain cycle in avlivebody** was fixed in `3b5f29e` — when adding new frame callbacks, mind the strong-self capture.
|
||||
- **AVLive-Body legacy** has been archived (`9e1482e`); the canonical client is `avlivebody-mac`. Don't reintroduce paths to the old project.
|
||||
- **macm1 = sink** (Multi-HMR CoreML + Apple Vision ANE + SMPL-X TCP); GrosMac = source. Mind the direction when wiring new OSC topics.
|
||||
- **Each major sub-project has its own `CLAUDE.md`** — closest wins. Put cross-cutting rules here, sub-project specifics in the nested file.
|
||||
|
||||
## When in doubt
|
||||
|
||||
- Read root `CLAUDE.md` and the nested `CLAUDE.md` of the sub-project you're editing.
|
||||
- Recent commits: `git log --oneline -20`.
|
||||
- Plans: `docs/superpowers/plans/`.
|
||||
- Cluster context: `~/CLAUDE.md` (GrosMac / macm1 / iPhone topology).
|
||||
- For sound: read `sound_algo/CLAUDE.md` before touching SynthDefs.
|
||||
@@ -1,6 +1,6 @@
|
||||
# AV-Live
|
||||
|
||||
Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout.
|
||||
Live coding audio-visual performance system : moteur SuperCollider, visualiseur openFrameworks piloté par un oscilloscope Hantek 6022BL, app menubar macOS qui orchestre le tout. **RC0.1+ (tag `v0.1.0-rc1`)** ajoute le pipeline Multi-HMR distribué M5 ↔ macm1 sur LAN gigabit + 10 scènes Metal pose-réactives + unified 3D armature wireframe (body+face+hands).
|
||||
|
||||
## Communication
|
||||
|
||||
@@ -26,6 +26,37 @@ Toujours répondre en français à l'utilisateur. Code, commentaires de code, co
|
||||
| Détection pose / mesh / body tracking | `data_only_viz/` |
|
||||
| Bridge web / UI de live coding | `web_realart/` |
|
||||
| Plans / specs en cours | `docs/superpowers/plans/` |
|
||||
| Multi-HMR remote server pyobjc | `data_only_viz/scripts/multihmr_server.py` (tourne sur macm1) |
|
||||
| Mesh dense rigger 27 fps perçu | `data_only_viz/mesh_rigger.py` |
|
||||
| 3D wireframe armature (body+face+hands) | `launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift` |
|
||||
| Pose → Metal scenes uniforms | `launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift` + `Resources/scene.metal` |
|
||||
| Filter chain (median + Kalman + lookahead + IK) | `data_only_viz/pose_filter.py` |
|
||||
| DINO re-id pid matching | `data_only_viz/dino_reid.py` |
|
||||
|
||||
## RC0.1+ environment variables
|
||||
|
||||
| Env | Default | Effect |
|
||||
|-----|---------|--------|
|
||||
| `MULTIHMR_BACKEND` | `pytorch` | `pytorch`, `coreml`, `remote` |
|
||||
| `MULTIHMR_REMOTE_HOST` | `127.0.0.1` | macm1 IP for remote inference |
|
||||
| `MULTIHMR_REMOTE_JPEG` | `1` | JPEG q=80 on the wire |
|
||||
| `MULTIHMR_REMOTE_ASYNC` | `1` | client double-buffer queue |
|
||||
| `MULTIHMR_SERVER_BACKEND` | `pyobjc` | server: `pyobjc` or `coremltools` |
|
||||
| `MULTIHMR_LOOP_FPS` | `30` | Python worker loop target_fps |
|
||||
| `AVBODY_HOST` | `127.0.0.1` | route TCP mesh + OSC to remote AVLiveBody |
|
||||
| `MEDIAPIPE_DELEGATE` | `cpu` | `gpu` Metal SRGBA (faster, flake on M5) |
|
||||
| `POSE_FILTER` | `median+kalman+lookahead+ik` | filter chain stages |
|
||||
| `MULTIHMR_REID` | `dino` | DINO cosine matching, `iou` fallback |
|
||||
|
||||
## Network topology
|
||||
|
||||
| Host | mDNS | IP (DHCP) | Role |
|
||||
|------|------|-----------|------|
|
||||
| GrosMac M5 | `grosmac.local` | LAN | Source + visualisation (AVLiveBody + data_only_viz + data_feeds) |
|
||||
| macm1 M1 Max | `supra-m1.local` | `192.168.0.175` | Sink (Multi-HMR CoreML + Apple Vision ANE + SMPL-X TCP) |
|
||||
| iPhone 16 Pro | (Personal Hotspot) | DHCP | ARKit/LiDAR pub via OSC `/body3d/kp` |
|
||||
|
||||
`AVBODY_HOST` / `MULTIHMR_REMOTE_HOST` accept mDNS hostnames — résiste aux changements DHCP (notamment iPhone hotspot 172.20.10.x).
|
||||
|
||||
## Environment variables
|
||||
|
||||
@@ -33,10 +64,19 @@ Toujours répondre en français à l'utilisateur. Code, commentaires de code, co
|
||||
|-----|---------|--------|
|
||||
| `POSE_FILTER` | `median+kalman+lookahead+ik` | filter chain stages — extra: `one_euro_joints` (joint-space CHI 2012 One Euro, inserted before kalman), `one_euro_bones` (bone-vector One Euro applied after SMPL-X fusion in multi.py), `arkit_fuse` (overrides 14 body slots with ARKit ARSkeleton3D from the iOS app, expects /body3d/kp on :57128) |
|
||||
| `IPHONE_OSC_PORT` | `57128` | UDP port the iPhone ARBodyTracker app pushes /body3d/kp to (always-on listener in data_only_viz) |
|
||||
| `VIDEO_ROTATE` | `none` | rotate the source video frame before MediaPipe + display: `none`/`ccw`/`cw`/`180`. Applied at multi.py read; pose detection + skeleton overlay stay aligned. ARKit 3D joints (gravity-aligned world) are unaffected |
|
||||
| `ICP_FUSION` | `0` | `1` to enable LiDAR↔SMPL-X ICP fusion (cf. `docs/ICP_FUSION.md`) |
|
||||
| `ICP_LIDAR_HOST` | _(unset)_ | iPhone ARBodyTracker IP when `ICP_FUSION=1` |
|
||||
| `ICP_LIDAR_PORT` | `5500` | iPhone ARMesh TCP port |
|
||||
| `ICP_LIDAR_EXTRINSIC` | `~/.config/av-live/lidar_extrinsic.json` | extrinsic JSON path |
|
||||
| `FINGER_PIANO` | `0` | `1` enables air-piano finger strike emission from data_only_viz |
|
||||
| `FINGER_STRIKE_VEL` | `0.02` | downward relative-velocity threshold for a strike (normalized image units/frame) |
|
||||
| `FINGER_STRIKE_REFRACTORY_MS` | `120` | min ms between strikes per finger |
|
||||
| `FINGER_SOURCE` | `auto` | hand source for finger gestures: `auto` (iPhone Vision hands if fresh, else MediaPipe), `iphone`, `mediapipe` |
|
||||
| `FINGER_DEBUG` | `0` | `1` logs each detected strike/pinch (hand, finger) — for threshold tuning |
|
||||
| `PINCH_RATIO_ON` | `0.45` | thumb-to-finger distance / hand size below which a pinch engages (clip toggle) |
|
||||
| `PINCH_RATIO_OFF` | `0.65` | distance/size above which the pinch re-arms (hysteresis) |
|
||||
| `PINCH_REFRACTORY_MS` | `250` | min ms between pinches per finger |
|
||||
|
||||
## Conventions globales
|
||||
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# AV-Live
|
||||
|
||||
**Release : `v0.1.0-rc1` (2026-05-14)** — distributed Multi-HMR (M5 ↔ macm1 LAN), unified 3D armature openpos, hybrid mesh rigger 27 fps perceived, 10 pose-reactive Metal scenes. See [RC0.1+ architecture](#rc01-distributed-multi-hmr-architecture) below.
|
||||
|
||||
> **Live coding audio-visual performance system** built around a SuperCollider sound engine, an openFrameworks oscilloscope visualizer driven by a real Hantek 6022BL USB scope, a macOS menubar launcher, and a Metal-native pose / body-mesh visualizer that listens to the same audio bus.
|
||||
>
|
||||
> 15 scripted demoparties · ~47 fullscreen visuals (26 3D parametric meshes + 18 procedural shaders + 3 dedicated C++ scenes) · 5 OS pixel-art shaders · 14 retro OS logos · 33 GLSL shader pairs (~65 files) · 1099 SynthDefs across 368 tracks (23 albums × 16) · **20 real-world data feeds** · 7 pose-estimation backends · **SMPL-X body mesh** (10 475 vertices via Multi-HMR + RealityKit) · **3 launch modes** (Full / Data-only / Body Mesh) — all reactive to the audio physically passing through the scope probes.
|
||||
@@ -101,7 +103,89 @@ Metal-native fullscreen visualizer (pyobjc, MTKView, ~60 fps) driven by webcam p
|
||||
- **Renderer** : Metal pipelines compiled at runtime (`shaders/*.metal`), `bg_pipeline` (full-screen FBM) + `skel_pipeline` (skeleton lines). SMPL face topology shipped as binary (`mesh_topology.py`) for RealityKit-compatible mesh rendering.
|
||||
- **Tracker** : One Euro Filter on keypoints + IoU multi-person association (`scipy.linear_sum_assignment`, ByteTrack-like).
|
||||
- **OSC out → sclang** `:57121` : `/pose/count`, `/pose/center`, `/pose/wrist`, `/pose/head`, `/pose/sho_span`, `/pose/limb_span`.
|
||||
- **Thread-safe state** : `state.py` exposes `State.lock()` ; dataclasses `PoseKp`, `NLFPerson` (vertices_3d, joints_3d), and a multi-person container.
|
||||
- **OSC out → AVLiveBody** `:57126` UDP (mode openpos, mode 9 / touche `p`) : `/pose/skel`, `/face/kp` (68 dlib landmarks), `/hand/kp` (21 × 2 hands), `/pose3d/kp` (33 MediaPipe pose_world_landmarks 3D meters).
|
||||
- **TCP out → AVLiveBody** `:57130` : SMPL-X dense mesh (10475 verts) frame-packed binary, 30 Hz rigged interpolation between Multi-HMR keyframes.
|
||||
- **Thread-safe state** : `state.py` exposes `State.lock()` ; dataclasses `PoseKp`, `Kp3D`, `SMPLXPerson`, multi-person container.
|
||||
|
||||
## RC0.1+ distributed Multi-HMR architecture
|
||||
|
||||
`feat/action-head` + tag `v0.1.0-rc1` (2026-05-14). The body-mesh pipeline is now **distributed across two Apple-silicon Macs over LAN gigabit** :
|
||||
|
||||
```
|
||||
M5 (capture host) macm1 (compute host)
|
||||
┌─────────────────────────────────┐ ┌──────────────────────────────────┐
|
||||
│ Caméra MacBook Pro │ │ Multi-HMR server :57140 (pyobjc) │
|
||||
│ data_only_viz/ │ JPEG q80 │ multihmr_full_672_s.mlpackage │
|
||||
│ ├─ multi_hmr_worker ├──TCP────▶│ ├─ Pyobjc direct CoreML.fwk │
|
||||
│ │ backend=remote (async) │ │ ├─ ~87 ms predict (M1 Max GPU) │
|
||||
│ ├─ MultiHMRRemoteBackend │◀──RSP───┤ └─ 6 outputs (v3d 10475, │
|
||||
│ │ queue maxsize 2/3 │ │ transl, scores, betas, │
|
||||
│ │ JPEG encode q80 │ │ expression, joints 127) │
|
||||
│ │ │ │ │
|
||||
│ ├─ MediaPipe Holistic │ │ AVLiveBody display (RealityKit) │
|
||||
│ │ Metal GPU delegate │ │ ├─ MeshRenderer (TCP :57130) │
|
||||
│ │ pose 33 + face 478 + │ /pose/* │ │ SMPL-X dense, low-level │
|
||||
│ │ hand 21×2 ├──UDP────▶│ │ mesh, 30 fps rigged interp │
|
||||
│ ├─ pose_bridge (UDP :57126) │ /face/* │ │ │
|
||||
│ ├─ smplx_tcp (TCP :57130, 30 Hz) │ /hand/* │ ├─ Skeleton3DRenderer (OSC) │
|
||||
│ ├─ MeshRigger (Hungarian + DINO │ /pose3d/│ │ 1 fused LowLevelMesh / │
|
||||
│ │ re-id, sticky pid 0.30/0.15) │ │ │ person : body 33 + face │
|
||||
│ └─ PoseFilterChain │ │ │ 68 + hand 21×2 = 143 vts │
|
||||
│ (median + Kalman CV + │ │ │ 288 line indices │
|
||||
│ lookahead + IK clamps) │ │ └─ 10 Metal scenes (storm, │
|
||||
│ │ │ tunnel, plasma, kaleido, │
|
||||
│ TCP loop fps : 25 │ │ voronoi, metaballs, │
|
||||
│ Multi-HMR fresh fps : 9 │ │ starfield, bars, hands3d, │
|
||||
│ MeshRigger perceived : 27 │ │ openpos), all consume │
|
||||
│ │ │ pose uniforms (mouth, │
|
||||
└─────────────────────────────────┘ │ velocity, head_tilt, │
|
||||
│ arm_spread, eye_open, │
|
||||
│ finger_pinch, body_xyz) │
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Key technical wins (commits in `feat/action-head`)
|
||||
|
||||
| Commit | Win |
|
||||
|--------|-----|
|
||||
| `5800156` | Multi-HMR ViT-S/672 → CoreML mlpackage (FP32, 6 outputs) |
|
||||
| `52588b9` | roma.rotmat_to_rotvec → branchless atan2 (fixed all-NaN v3d/transl bug) |
|
||||
| `4e7101c` | NaN/Inf guard on v3d before TCP ship |
|
||||
| `2c8094c` | MeshRigger : 27 fps perceived via Hungarian pid match + Vision pelvis delta |
|
||||
| `1f623fe` | AVLiveBody mirror webcam preview (CATransform3D scale -1) |
|
||||
| `9838da3` | Remote inference protocol (TCP :57140) + multi-buffer async client |
|
||||
| `67302e7` | Pyobjc server (drops coremltools overhead) + MediaPipe Metal GPU SRGBA |
|
||||
| `1828d7c` | 12 new SceneUniforms (mouth, eye, head, finger, body) drive 5 scenes |
|
||||
| `bd46f6e` | Lift Python self-throttle 10 → 30 fps loop, fresh fps metric |
|
||||
| latest | 10 Metal scenes pose-reactive + SMPL-X 127 joints output (finger props) |
|
||||
|
||||
### Environment toggles
|
||||
|
||||
| Env var | Default | Effect |
|
||||
|---------|---------|--------|
|
||||
| `MULTIHMR_BACKEND` | `pytorch` | `pytorch`, `coreml`, `remote` |
|
||||
| `MULTIHMR_REMOTE_HOST` | `127.0.0.1` | macm1 IP (gigabit LAN) |
|
||||
| `MULTIHMR_REMOTE_JPEG` | `1` | JPEG q=80 compression on the wire |
|
||||
| `MULTIHMR_REMOTE_ASYNC` | `1` | client double-buffer queue (maxsize 2/3) |
|
||||
| `MULTIHMR_SERVER_BACKEND` | `pyobjc` | server : `pyobjc` or `coremltools` |
|
||||
| `MULTIHMR_LOOP_FPS` | `30` | Python loop target_fps (formerly capped at 10) |
|
||||
| `AVBODY_HOST` | `127.0.0.1` | route TCP mesh + OSC pose to a remote AVLiveBody |
|
||||
| `MEDIAPIPE_DELEGATE` | `cpu` | `gpu` Metal SRGBA (faster but IOSurface flake on M5) |
|
||||
| `POSE_FILTER` | `median+kalman+lookahead+ik` | toggle filter chain stages |
|
||||
| `MULTIHMR_REID` | `dino` if mlpackage present | `dino` (cosine match) or `iou` |
|
||||
| `MULTIHMR_REID_ALPHA` | `0.5` | IoU vs cosine weight (0=DINO only, 1=IoU only) |
|
||||
|
||||
### Hardware ceilings observed (M5 + M1 Max 32c GPU, LAN gigabit)
|
||||
|
||||
| Path | Predict | Live loop | Mesh rigged perceived |
|
||||
|------|---------|-----------|------------------------|
|
||||
| M5 local (PyTorch MPS) | 270 ms | 3.5 fps | 27 fps |
|
||||
| M5 local (CoreML FP32) | 139 ms | 6.8 fps | 27 fps |
|
||||
| Remote macm1 (idle GPU) | 53 ms | ~18 fps | 27 fps |
|
||||
| Remote macm1 (under MLX contention) | 87 ms | 25 fps loop / 9 fps fresh | 27 fps |
|
||||
| Studio M3 Ultra (80c GPU, *training only*) | est. 35 ms (~30 fps) | — | — |
|
||||
|
||||
Hard ceiling on macm1 ≈ 18 fps fresh predict (unified memory bandwidth + CoreML sync overhead) ; further gains require moving MLX servers off macm1 or quantising the model.
|
||||
|
||||
### 🎬 Demoparties — 15 narrative demos
|
||||
Each demoparty is a multi-act scripted show with unique scroller text, background sequence, scroller style, and SuperCollider album track :
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
*.xcodeproj/
|
||||
Config/Local.xcconfig
|
||||
.build/
|
||||
.swiftpm/
|
||||
@@ -0,0 +1,3 @@
|
||||
// Copy to Config/Local.xcconfig and set your Apple Developer Team ID.
|
||||
// Config/Local.xcconfig is gitignored.
|
||||
DEVELOPMENT_TEAM = YOUR_TEAM_ID
|
||||
@@ -0,0 +1,8 @@
|
||||
#include? "Local.xcconfig"
|
||||
|
||||
MACOSX_DEPLOYMENT_TARGET = 15.0
|
||||
SWIFT_VERSION = 5.10
|
||||
// Manual ad-hoc signing for local dev (no Apple Mac Development cert
|
||||
// required). Override here or via target settings for distribution.
|
||||
CODE_SIGN_STYLE = Manual
|
||||
CODE_SIGN_IDENTITY = -
|
||||
@@ -0,0 +1,62 @@
|
||||
# AVLiveBody (macOS)
|
||||
|
||||
Native macOS Xcode app that renders SMPL-X body meshes in RealityKit
|
||||
from the USB iPhone body-tracking pipeline (ARBodyTracker -> Multi-HMR
|
||||
worker -> AVLiveBody scene).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- macOS 15+
|
||||
- Xcode 16+
|
||||
- `xcodegen` — `brew install xcodegen`
|
||||
|
||||
## First-time setup
|
||||
|
||||
1. Copy the CoreML model into the app resources (required, gitignored,
|
||||
~195 MB). Without it the app degrades to skeleton-only rendering:
|
||||
|
||||
```
|
||||
cp -R ~/.cache/av-live-multihmr/multihmr_full_672_s.mlpackage \
|
||||
avlivebody-mac/Sources/AVLiveBody/Resources/
|
||||
```
|
||||
|
||||
2. Create your local xcconfig and set your signing team:
|
||||
|
||||
```
|
||||
cp Config/Local.xcconfig.example Config/Local.xcconfig
|
||||
# Edit Config/Local.xcconfig:
|
||||
# DEVELOPMENT_TEAM = <your Apple Developer Team ID>
|
||||
```
|
||||
|
||||
## Build
|
||||
|
||||
Generate the Xcode project (run after every `project.yml` change) then
|
||||
open or build from the CLI:
|
||||
|
||||
```
|
||||
cd avlivebody-mac
|
||||
xcodegen generate
|
||||
open AVLiveBody.xcodeproj
|
||||
```
|
||||
|
||||
CLI build / test:
|
||||
|
||||
```
|
||||
xcodebuild -project AVLiveBody.xcodeproj -scheme AVLiveBody \
|
||||
-destination 'platform=macOS' build
|
||||
|
||||
xcodebuild -project AVLiveBody.xcodeproj -scheme AVLiveBody \
|
||||
-destination 'platform=macOS' test
|
||||
```
|
||||
|
||||
## Runtime requirements
|
||||
|
||||
A tethered iPhone running the matching `ARBodyTracker` iOS app over USB
|
||||
is required for body input. See `iphone-arbody/` for the iOS side.
|
||||
|
||||
## Architecture
|
||||
|
||||
- Design spec:
|
||||
`docs/superpowers/specs/2026-05-18-avlivebody-macos-rewrite-design.md`
|
||||
- Implementation plan:
|
||||
`docs/superpowers/plans/2026-05-18-avlivebody-macos-rewrite.md`
|
||||
@@ -0,0 +1,139 @@
|
||||
import Cocoa
|
||||
import CoreVideo
|
||||
import SwiftUI
|
||||
|
||||
/// Forces a regular, keyboard-focusable foreground app.
|
||||
final class AppDelegate: NSObject, NSApplicationDelegate {
|
||||
func applicationDidFinishLaunching(_ notification: Notification) {
|
||||
NSApp.setActivationPolicy(.regular)
|
||||
NSApp.activate()
|
||||
}
|
||||
}
|
||||
|
||||
@main
|
||||
struct AVLiveBodyApp: App {
|
||||
@NSApplicationDelegateAdaptor(AppDelegate.self)
|
||||
private var appDelegate
|
||||
|
||||
var body: some Scene {
|
||||
WindowGroup {
|
||||
ContentView()
|
||||
.frame(minWidth: 900, minHeight: 600)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@MainActor
|
||||
struct ContentView: View {
|
||||
@StateObject private var consumer = USBSkeletonConsumer()
|
||||
@StateObject private var settings = RenderSettings()
|
||||
@State private var prevPelvis: SIMD3<Float>?
|
||||
private let controller = SceneController()
|
||||
private let renderer = SceneRenderer.make()
|
||||
/// Multi-HMR mesh inference (CoreML, ~150-300 ms/frame) is OFF by
|
||||
/// default: it would saturate the main thread and beachball the UI.
|
||||
/// Skeleton-only rendering (iPhone 91-joint ARKit stream) otherwise.
|
||||
/// Enable on a capable, properly-threaded host with AVBODY_MULTIHMR=1.
|
||||
private let multiHMR: MultiHMRCoreML? = {
|
||||
guard ProcessInfo.processInfo.environment["AVBODY_MULTIHMR"] == "1"
|
||||
else { return nil }
|
||||
return MultiHMRCoreML()
|
||||
}()
|
||||
/// Placeholder intrinsics until a `.meta` frame supplies real ones.
|
||||
private let cameraK: [Float] = [
|
||||
672, 0, 336, 0, 672, 336, 0, 0, 1,
|
||||
]
|
||||
|
||||
var body: some View {
|
||||
ZStack(alignment: .topTrailing) {
|
||||
LayeredSceneView(controller: controller, renderer: renderer,
|
||||
showScene: settings.showScene)
|
||||
StatusBar(consumer: consumer)
|
||||
.frame(maxWidth: .infinity, alignment: .top)
|
||||
HStack(alignment: .top) {
|
||||
Spacer()
|
||||
if settings.showPanel {
|
||||
SettingsPanel(settings: settings)
|
||||
.transition(.move(edge: .trailing))
|
||||
}
|
||||
Button {
|
||||
withAnimation { settings.showPanel.toggle() }
|
||||
} label: {
|
||||
Image(systemName: "slider.horizontal.3")
|
||||
.padding(8)
|
||||
}
|
||||
.buttonStyle(.borderedProminent)
|
||||
.padding(8)
|
||||
}
|
||||
}
|
||||
.onAppear { wire(); applyAll() }
|
||||
.onDisappear { consumer.stop() }
|
||||
.onReceive(consumer.$skeletons) { skeletons in
|
||||
controller.updateSkeleton(skeletons)
|
||||
pushUniforms()
|
||||
}
|
||||
.onReceive(consumer.$hands) { _ in pushUniforms() }
|
||||
.onReceive(consumer.$face) { _ in pushUniforms() }
|
||||
.onReceive(settings.objectWillChange) { _ in
|
||||
// objectWillChange fires before the value updates; apply on
|
||||
// the next runloop tick so we read the new values.
|
||||
DispatchQueue.main.async { applyAll() }
|
||||
}
|
||||
}
|
||||
|
||||
private func applyAll() {
|
||||
controller.setSkeletonVisible(settings.showSkeleton)
|
||||
controller.setMeshVisible(settings.showMesh)
|
||||
controller.setVideoVisible(settings.showVideo)
|
||||
controller.setVideoOpacity(settings.videoOpacity)
|
||||
controller.updateMeshMaterial(metallic: settings.meshMetallic,
|
||||
roughness: settings.meshRoughness)
|
||||
controller.setLightIntensities(key: settings.keyIntensity,
|
||||
fill: settings.fillIntensity,
|
||||
rim: settings.rimIntensity)
|
||||
controller.setFieldOfView(settings.fieldOfView)
|
||||
if let r = renderer {
|
||||
r.uniforms.viz_mode = Float(settings.vizMode)
|
||||
}
|
||||
}
|
||||
|
||||
private func pushUniforms() {
|
||||
guard let r = renderer else { return }
|
||||
r.uniforms.viz_mode = Float(settings.vizMode)
|
||||
SceneUniformBuilder.fill(&r.uniforms,
|
||||
skeletons: consumer.skeletons,
|
||||
hands: consumer.hands,
|
||||
face: consumer.face,
|
||||
prevPelvis: &prevPelvis)
|
||||
controller.updateHandFace(hands: consumer.hands,
|
||||
face: consumer.face,
|
||||
depth: Float(-(prevPelvis?.z ?? 0)))
|
||||
}
|
||||
|
||||
private func wire() {
|
||||
let controller = self.controller
|
||||
let multiHMR = self.multiHMR
|
||||
let cameraK = self.cameraK
|
||||
consumer.onVideoFrame = { [weak consumer] pixelBuffer in
|
||||
MainActor.assumeIsolated {
|
||||
controller.updateVideo(pixelBuffer)
|
||||
guard let consumer, let hmr = multiHMR else { return }
|
||||
// Snapshot skeletons on main; run the heavy CoreML
|
||||
// prediction off-main (drops frames if busy), then hop
|
||||
// back to main for the light fuse + mesh UI update.
|
||||
let skeletons = consumer.skeletons
|
||||
hmr.inferAsync(pixelBuffer, cameraK: cameraK) { raw in
|
||||
DispatchQueue.main.async {
|
||||
MainActor.assumeIsolated {
|
||||
let fused = BodyFusion.fuse(
|
||||
persons: raw,
|
||||
skeletons: skeletons)
|
||||
controller.updateMesh(fused)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
consumer.start()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
import Foundation
|
||||
import simd
|
||||
|
||||
/// ARKit/Multi-HMR world coords (y up, z back) -> RealityKit world
|
||||
/// coords (y up, z forward). Apply to every vertex/translation that
|
||||
/// crosses from source pipeline space into the scene.
|
||||
@inline(__always)
|
||||
func arkitToRealityKit(_ v: SIMD3<Float>) -> SIMD3<Float> {
|
||||
SIMD3<Float>(v.x, -v.y, -v.z)
|
||||
}
|
||||
@@ -0,0 +1,51 @@
|
||||
// avlivebody-mac/Sources/AVLiveBody/HandFaceSkeleton.swift
|
||||
import AVLiveWire
|
||||
import AppKit
|
||||
import RealityKit
|
||||
import simd
|
||||
|
||||
@MainActor
|
||||
final class HandFaceSkeleton {
|
||||
let root = Entity()
|
||||
private static let r: Float = 0.008
|
||||
private static let spanX: Float = 1.2 // maps [0,1] -> [-0.6,0.6] m
|
||||
private static let spanY: Float = 0.9
|
||||
|
||||
private let mesh = MeshResource.generateSphere(radius: r)
|
||||
private let handMat = SimpleMaterial(color: NSColor.cyan, isMetallic: false)
|
||||
private let faceMat = SimpleMaterial(color: NSColor.magenta, isMetallic: false)
|
||||
private var handPool: [ModelEntity] = []
|
||||
private var facePool: [ModelEntity] = []
|
||||
private var depth: Float = 0
|
||||
|
||||
func setDepth(_ z: Float) { depth = z }
|
||||
|
||||
func update(hands: HandsPayload?, face: FacePayload?) {
|
||||
let hpts = (hands?.hands ?? []).flatMap { $0.points }
|
||||
layout(&handPool, count: hpts.count, mat: handMat)
|
||||
for (i, p) in hpts.enumerated() { place(handPool[i], p.x, p.y) }
|
||||
for i in hpts.count..<handPool.count { handPool[i].isEnabled = false }
|
||||
|
||||
let fpts = face?.points ?? []
|
||||
layout(&facePool, count: fpts.count, mat: faceMat)
|
||||
for (i, p) in fpts.enumerated() { place(facePool[i], p.x, p.y) }
|
||||
for i in fpts.count..<facePool.count { facePool[i].isEnabled = false }
|
||||
}
|
||||
|
||||
private func place(_ e: ModelEntity, _ nx: Float, _ ny: Float) {
|
||||
// normalized top-left [0,1] -> centered meters, y up
|
||||
e.transform.translation = SIMD3<Float>(
|
||||
(nx - 0.5) * Self.spanX,
|
||||
(0.5 - ny) * Self.spanY,
|
||||
depth)
|
||||
e.isEnabled = true
|
||||
}
|
||||
|
||||
private func layout(_ pool: inout [ModelEntity], count: Int,
|
||||
mat: SimpleMaterial) {
|
||||
while pool.count < count {
|
||||
let e = ModelEntity(mesh: mesh, materials: [mat])
|
||||
root.addChild(e); pool.append(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>CFBundleName</key><string>AVLiveBody</string>
|
||||
<key>CFBundleIdentifier</key><string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
|
||||
<key>CFBundleExecutable</key><string>$(EXECUTABLE_NAME)</string>
|
||||
<key>CFBundlePackageType</key><string>APPL</string>
|
||||
<key>CFBundleShortVersionString</key><string>1.0</string>
|
||||
<key>CFBundleVersion</key><string>1</string>
|
||||
<key>LSMinimumSystemVersion</key><string>15.0</string>
|
||||
<key>NSCameraUsageDescription</key>
|
||||
<string>Receives the tethered iPhone camera over USB.</string>
|
||||
<key>NSLocalNetworkUsageDescription</key>
|
||||
<string>Connects to the tethered iPhone over USB (usbmuxd).</string>
|
||||
</dict>
|
||||
</plist>
|
||||
@@ -0,0 +1,51 @@
|
||||
// avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift
|
||||
import AppKit
|
||||
import MetalKit
|
||||
import RealityKit
|
||||
import SwiftUI
|
||||
|
||||
/// Layered container: MTKView (Metal shader bg, transparent) behind the
|
||||
/// SceneController's transparent ARView. The owner holds the SceneRenderer
|
||||
/// so it can push uniforms each frame.
|
||||
struct LayeredSceneView: NSViewRepresentable {
|
||||
let controller: SceneController
|
||||
let renderer: SceneRenderer?
|
||||
let showScene: Bool
|
||||
|
||||
func makeNSView(context: Context) -> NSView {
|
||||
controller.setUp()
|
||||
let container = NSView(frame: .zero)
|
||||
container.wantsLayer = true
|
||||
container.layer?.backgroundColor = NSColor.black.cgColor
|
||||
|
||||
if let renderer {
|
||||
let mtk = MTKView(frame: container.bounds,
|
||||
device: MTLCreateSystemDefaultDevice())
|
||||
mtk.delegate = renderer
|
||||
mtk.colorPixelFormat = .bgra8Unorm
|
||||
mtk.framebufferOnly = false
|
||||
mtk.layer?.isOpaque = true
|
||||
mtk.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 1)
|
||||
mtk.preferredFramesPerSecond = 60
|
||||
mtk.autoresizingMask = [.width, .height]
|
||||
container.addSubview(mtk)
|
||||
context.coordinator.mtk = mtk
|
||||
}
|
||||
|
||||
let arView = controller.arView
|
||||
arView.frame = container.bounds
|
||||
arView.autoresizingMask = [.width, .height]
|
||||
arView.wantsLayer = true
|
||||
arView.layer?.isOpaque = false
|
||||
arView.layer?.backgroundColor = NSColor.clear.cgColor
|
||||
container.addSubview(arView) // front
|
||||
return container
|
||||
}
|
||||
|
||||
func updateNSView(_ nsView: NSView, context: Context) {
|
||||
context.coordinator.mtk?.isHidden = !showScene
|
||||
}
|
||||
|
||||
func makeCoordinator() -> Coordinator { Coordinator() }
|
||||
final class Coordinator { var mtk: MTKView? }
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
import AppKit
|
||||
import Foundation
|
||||
import RealityKit
|
||||
import simd
|
||||
|
||||
/// Renders SMPL-X dense body meshes (10475 vertices) from Multi-HMR.
|
||||
/// Triangle indices come from the bundled `smplx_faces.bin`
|
||||
/// (flat UInt32 triplets).
|
||||
@MainActor
|
||||
final class MeshEntity {
|
||||
let root = Entity()
|
||||
|
||||
private static let vertexCount = 10475
|
||||
private let faces: [UInt32]
|
||||
private var pools: [Int: ModelEntity] = [:]
|
||||
private var material = SimpleMaterial(
|
||||
color: NSColor(white: 0.8, alpha: 1.0),
|
||||
roughness: 0.5, isMetallic: false)
|
||||
|
||||
init() {
|
||||
faces = MeshEntity.loadFaces()
|
||||
}
|
||||
|
||||
func update(_ persons: [MultiHMRPerson]) {
|
||||
for (idx, person) in persons.enumerated() {
|
||||
let entity = pools[idx] ?? {
|
||||
let e = ModelEntity()
|
||||
root.addChild(e)
|
||||
pools[idx] = e
|
||||
return e
|
||||
}()
|
||||
guard let mesh = buildMesh(person.vertices) else { continue }
|
||||
entity.model = ModelComponent(mesh: mesh,
|
||||
materials: [material])
|
||||
let t = person.translation
|
||||
entity.transform.translation = arkitToRealityKit(t)
|
||||
entity.isEnabled = true
|
||||
}
|
||||
for idx in pools.keys where idx >= persons.count {
|
||||
pools[idx]?.isEnabled = false
|
||||
}
|
||||
}
|
||||
|
||||
func setVisible(_ visible: Bool) { root.isEnabled = visible }
|
||||
|
||||
func setMaterial(metallic: Bool, roughness: Double) {
|
||||
material = SimpleMaterial(
|
||||
color: NSColor(white: 0.8, alpha: 1.0),
|
||||
roughness: .float(Float(roughness)), isMetallic: metallic)
|
||||
for entity in pools.values {
|
||||
entity.model?.materials = [material]
|
||||
}
|
||||
}
|
||||
|
||||
private func buildMesh(_ verts: [SIMD3<Float>])
|
||||
-> MeshResource? {
|
||||
guard verts.count == Self.vertexCount, !faces.isEmpty else {
|
||||
NSLog("MeshEntity: vertex count mismatch %d (expected %d), faces=%d",
|
||||
verts.count, Self.vertexCount, faces.count)
|
||||
return nil
|
||||
}
|
||||
var descriptor = MeshDescriptor(name: "smplx")
|
||||
descriptor.positions = MeshBuffer(verts.map(arkitToRealityKit))
|
||||
descriptor.primitives = .triangles(faces)
|
||||
return try? MeshResource.generate(from: [descriptor])
|
||||
}
|
||||
|
||||
private static func loadFaces() -> [UInt32] {
|
||||
guard let url = Bundle.main.url(
|
||||
forResource: "smplx_faces", withExtension: "bin"),
|
||||
let data = try? Data(contentsOf: url) else {
|
||||
NSLog("MeshEntity: smplx_faces.bin missing")
|
||||
return []
|
||||
}
|
||||
return data.withUnsafeBytes { raw in
|
||||
Array(raw.bindMemory(to: UInt32.self))
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,35 @@
|
||||
// avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift
|
||||
import SwiftUI
|
||||
|
||||
/// Live visual settings for AVLiveBody. ContentView observes this and
|
||||
/// pushes changes to SceneController. Defaults chosen so the scene is
|
||||
/// lit and everything visible out of the box.
|
||||
@MainActor
|
||||
final class RenderSettings: ObservableObject {
|
||||
// Layer visibility
|
||||
@Published var showSkeleton: Bool = true
|
||||
@Published var showMesh: Bool = true
|
||||
@Published var showVideo: Bool = true
|
||||
|
||||
// Mesh material
|
||||
@Published var meshMetallic: Bool = false
|
||||
@Published var meshRoughness: Double = 0.6
|
||||
|
||||
// Video quad
|
||||
@Published var videoOpacity: Double = 1.0
|
||||
|
||||
// Lights (RealityKit DirectionalLight intensities)
|
||||
@Published var keyIntensity: Double = 4000
|
||||
@Published var fillIntensity: Double = 1500
|
||||
@Published var rimIntensity: Double = 2000
|
||||
|
||||
// Camera
|
||||
@Published var fieldOfView: Double = 55
|
||||
|
||||
// Panel visibility
|
||||
@Published var showPanel: Bool = false
|
||||
|
||||
// Metal shader background (Phase 2)
|
||||
@Published var showScene: Bool = true
|
||||
@Published var vizMode: Int = 0 // 0..9
|
||||
}
|
||||
+56
-13
@@ -34,8 +34,34 @@ struct SceneUniforms {
|
||||
float hand_l_y;
|
||||
float hand_r_x;
|
||||
float hand_r_y;
|
||||
// ---- 2026-05-14 face / hand / body3d derivatives ----
|
||||
// mouth_open : |lip51.y - lip57.y| normalized 0..~0.15
|
||||
// eye_open_l/r : H/W ratio of eye bbox, ~0.15 (closed) .. 0.45 (open)
|
||||
// head_tilt : atan2(eye_R.y-eye_L.y, eye_R.x-eye_L.x) rad
|
||||
// head_yaw : nose.y - eyes_mid.y (proxy pitch, normalized)
|
||||
// finger_pinch_l/r: simd_distance(thumb_tip[4], index_tip[8]) px-norm
|
||||
// body_x/y/z : pelvis world meters (MediaPipe hip-relative)
|
||||
// body_height : |hip.y - head.y| world meters
|
||||
// arm_spread : |wristL.x - wristR.x| world meters
|
||||
// pose_velocity : EMA(|pelvis(t) - pelvis(t-1)|), alpha=0.3
|
||||
float mouth_open;
|
||||
float eye_open_l;
|
||||
float eye_open_r;
|
||||
float head_tilt;
|
||||
float head_yaw;
|
||||
float finger_pinch_l;
|
||||
float finger_pinch_r;
|
||||
float body_x;
|
||||
float body_y;
|
||||
float body_z;
|
||||
float body_height;
|
||||
float arm_spread;
|
||||
float pose_velocity;
|
||||
float _pad0;
|
||||
float _pad1;
|
||||
float _pad2;
|
||||
float _pad3;
|
||||
float _pad4;
|
||||
};
|
||||
|
||||
struct VsOut {
|
||||
@@ -120,21 +146,25 @@ float vignette(float2 p) {
|
||||
// ===== Modes =======================================================
|
||||
|
||||
// ---- 0 storm : tissu fbm reactif + bloom-fake ----
|
||||
// Pose mods : pose_velocity boost intensity, head_tilt shifts hue.
|
||||
float3 mode_storm(float2 p, constant SceneUniforms& U) {
|
||||
float storm = saturate(U.kp_norm * 1.0 + max(-U.bz_norm, 0.0) * 0.5);
|
||||
float speed = 0.08 + U.wind_norm * 1.5;
|
||||
float storm = saturate(U.kp_norm * 1.0 + max(-U.bz_norm, 0.0) * 0.5
|
||||
+ U.pose_velocity * 2.0);
|
||||
float speed = 0.08 + U.wind_norm * 1.5 + U.pose_velocity * 3.0;
|
||||
float zoom = 1.8 - U.rms * 1.2;
|
||||
float n = fbm(p * zoom + float2(U.time * speed, U.time * speed * 0.7));
|
||||
n = pow(n, 1.2 - U.rms * 0.5);
|
||||
float netz = sin(U.time * 50.0 + U.netz_dev * 800.0) * 0.06;
|
||||
float3 base = palIQ(n + storm * 0.5,
|
||||
float hue_shift = U.head_tilt * 0.25;
|
||||
float3 base = palIQ(n + storm * 0.5 + hue_shift,
|
||||
float3(0.10, 0.05, 0.20),
|
||||
float3(0.40, 0.30, 0.55),
|
||||
float3(1.0, 1.0, 1.0),
|
||||
float3(0.0, 0.33, 0.67));
|
||||
float bloom = smoothstep(0.7, 1.0, n);
|
||||
return base * (n * 1.4 + 0.3) + netz + U.rms * 1.2
|
||||
+ bloom * 0.7
|
||||
float velocity_boost = 1.0 + U.pose_velocity * 1.5;
|
||||
return (base * (n * 1.4 + 0.3) + netz + U.rms * 1.2
|
||||
+ bloom * 0.7) * velocity_boost
|
||||
+ float3(1.0, 0.55, 0.1) * U.flare * 1.4
|
||||
+ float3(U.lightning_flash * 0.7);
|
||||
}
|
||||
@@ -167,34 +197,40 @@ float3 mode_tunnel(float2 p, constant SceneUniforms& U) {
|
||||
}
|
||||
|
||||
// ---- 2 plasma : volumetric noise palette IQ ----
|
||||
// Pose mods : mouth_open modulates spatial frequency; head_yaw shifts hue.
|
||||
float3 mode_plasma(float2 p, constant SceneUniforms& U) {
|
||||
float t = U.time * (0.5 + U.rms * 1.5);
|
||||
float freq_boost = 1.0 + U.mouth_open * 12.0;
|
||||
// 3 octaves de sin/cos en composition
|
||||
float v = sin(p.x * 4.0 + t)
|
||||
+ sin(p.y * 5.0 - t * 1.2)
|
||||
float v = sin(p.x * 4.0 * freq_boost + t)
|
||||
+ sin(p.y * 5.0 * freq_boost - t * 1.2)
|
||||
+ sin((p.x + p.y) * 3.5 + t * 0.7)
|
||||
+ sin(length(p) * (8.0 + U.kp_norm * 4.0) - t * 1.8);
|
||||
v = v * 0.25 + 0.5;
|
||||
// Fake volumetric "depth" : repeat layers
|
||||
float layer2 = sin(p.x * 2.0 - t * 0.5) * sin(p.y * 2.5 + t * 0.7);
|
||||
v = mix(v, v * 0.5 + 0.5 * (layer2 + 1.0) * 0.5, 0.35);
|
||||
float3 col = palIQ(v,
|
||||
float hue_offset = U.head_yaw * 0.5;
|
||||
float3 col = palIQ(v + hue_offset,
|
||||
float3(0.5),
|
||||
float3(0.5),
|
||||
float3(1.0, 1.0, 1.0),
|
||||
float3(0.0, 0.33, 0.67));
|
||||
col *= 0.8 + U.kp_norm * 0.7 + U.social_rate * 0.5;
|
||||
col *= 0.8 + U.kp_norm * 0.7 + U.social_rate * 0.5
|
||||
+ U.mouth_open * 1.0;
|
||||
return col + float3(0.6, 0.3, 1.0) * U.lightning_flash * 0.5;
|
||||
}
|
||||
|
||||
// ---- 3 kaleido : KIFS fractal 6-fold avec rot 3D fake ----
|
||||
// Pose mods : arm_spread drives segment count (4..16).
|
||||
float3 mode_kaleido(float2 p, constant SceneUniforms& U) {
|
||||
float ang = U.time * 0.15 + U.flare * 2.0;
|
||||
float c = cos(ang), s = sin(ang);
|
||||
p = float2(c * p.x - s * p.y, s * p.x + c * p.y);
|
||||
float r = length(p);
|
||||
float a = atan2(p.y, p.x);
|
||||
float seg = 6.28318 / 6.0;
|
||||
float seg_count = clamp(ceil(4.0 + U.arm_spread * 8.0), 3.0, 16.0);
|
||||
float seg = 6.28318 / seg_count;
|
||||
a = abs(fmod(a + seg * 0.5, seg) - seg * 0.5);
|
||||
float2 q = float2(cos(a), sin(a)) * r;
|
||||
// Iteration KIFS-like
|
||||
@@ -303,14 +339,18 @@ float3 mode_metaballs(float2 p, constant SceneUniforms& U) {
|
||||
}
|
||||
|
||||
// ---- 6 starfield : galaxy spiral + parallax ----
|
||||
// Pose mods : finger_pinch L+R drives star density per layer.
|
||||
float3 mode_starfield(float2 p, constant SceneUniforms& U) {
|
||||
float warp = U.time * (1.5 + U.wind_norm * 6.0);
|
||||
float pinch = saturate((U.finger_pinch_l + U.finger_pinch_r) * 2.0);
|
||||
int stars_per_layer = 30 + int(pinch * 70.0); // 30..100
|
||||
// 3 layers of stars at different speeds
|
||||
float3 col = float3(0);
|
||||
for (int L = 0; L < 3; ++L) {
|
||||
float speed = (1.0 + float(L) * 0.5);
|
||||
float scale = 6.0 + float(L) * 4.0;
|
||||
for (int k = 0; k < 50; ++k) {
|
||||
for (int k = 0; k < 100; ++k) {
|
||||
if (k >= stars_per_layer) break;
|
||||
float fk = float(k + L * 50);
|
||||
float r0 = hash21(float2(fk, 7.0 + float(L)));
|
||||
float a0 = hash21(float2(fk, 17.0 + float(L))) * 6.28;
|
||||
@@ -352,8 +392,11 @@ float3 mode_bars(float2 p, constant SceneUniforms& U) {
|
||||
// Hauteur barre depend du bin "i" via hash + RMS
|
||||
float h0 = hash21(float2(float(i), 0.0));
|
||||
float h = sin(t * (0.5 + h0 * 4.0) + float(i)) * 0.5 + 0.5;
|
||||
h = h * (0.3 + U.rms * 1.5 + U.social_rate * 0.4);
|
||||
h = clamp(h, 0.02, 0.85);
|
||||
// body_height (0..~1.8 m) + eye_open avg modulate bar height
|
||||
float eyes = (U.eye_open_l + U.eye_open_r) * 0.5;
|
||||
h = h * (0.3 + U.rms * 1.5 + U.social_rate * 0.4
|
||||
+ U.body_height * 0.6 + eyes * 0.8);
|
||||
h = clamp(h, 0.02, 0.95);
|
||||
float bar_top = y_base + h * perspective * 0.3;
|
||||
// Largeur = 1 / nbars perspective
|
||||
float bx = (fi - 0.5) * perspective * 1.5;
|
||||
@@ -0,0 +1,172 @@
|
||||
import AppKit
|
||||
import Foundation
|
||||
import CoreVideo
|
||||
import RealityKit
|
||||
import simd
|
||||
import AVLiveWire
|
||||
|
||||
/// Owns the single RealityKit scene: the video quad, the body root,
|
||||
/// and an orbital camera. The app calls `updateVideo/updateSkeleton/
|
||||
/// updateMesh` from the main queue.
|
||||
@MainActor
|
||||
final class SceneController {
|
||||
let arView = ARView(frame: .zero)
|
||||
|
||||
private let cameraAnchor = AnchorEntity(world: .zero)
|
||||
private let camera = PerspectiveCamera()
|
||||
private let worldAnchor = AnchorEntity(world: .zero)
|
||||
|
||||
private let keyLight = DirectionalLight()
|
||||
private let fillLight = DirectionalLight()
|
||||
private let rimLight = DirectionalLight()
|
||||
|
||||
private(set) var videoQuad: VideoQuad?
|
||||
private(set) var skeleton: SkeletonEntity?
|
||||
private(set) var mesh: MeshEntity?
|
||||
private(set) var handFace: HandFaceSkeleton?
|
||||
|
||||
/// Orbital camera state.
|
||||
private var orbitYaw: Float = 0
|
||||
private var orbitPitch: Float = 0
|
||||
private var orbitRadius: Float = 3.0
|
||||
|
||||
private var didSetUp = false
|
||||
|
||||
func setUp() {
|
||||
guard !didSetUp else { return }
|
||||
didSetUp = true
|
||||
arView.environment.background = .color(.clear)
|
||||
arView.layer?.isOpaque = false
|
||||
arView.layer?.backgroundColor = NSColor.clear.cgColor
|
||||
arView.scene.addAnchor(worldAnchor)
|
||||
|
||||
keyLight.light.intensity = 4000
|
||||
keyLight.orientation = simd_quatf(angle: .pi / 6,
|
||||
axis: SIMD3(1, 0, 0))
|
||||
let keyA = AnchorEntity(world: SIMD3<Float>(1, 2, -1))
|
||||
keyA.addChild(keyLight)
|
||||
arView.scene.addAnchor(keyA)
|
||||
|
||||
fillLight.light.intensity = 1500
|
||||
fillLight.light.color = NSColor(red: 0.7, green: 0.8,
|
||||
blue: 1.0, alpha: 1.0)
|
||||
let fillA = AnchorEntity(world: SIMD3<Float>(-2, 1, -2))
|
||||
fillA.addChild(fillLight)
|
||||
arView.scene.addAnchor(fillA)
|
||||
|
||||
rimLight.light.intensity = 2000
|
||||
let rimA = AnchorEntity(world: SIMD3<Float>(0, 1, -5))
|
||||
rimA.addChild(rimLight)
|
||||
arView.scene.addAnchor(rimA)
|
||||
|
||||
camera.camera.fieldOfViewInDegrees = 55
|
||||
cameraAnchor.addChild(camera)
|
||||
arView.scene.addAnchor(cameraAnchor)
|
||||
applyCamera()
|
||||
|
||||
let q = VideoQuad()
|
||||
worldAnchor.addChild(q.entity)
|
||||
videoQuad = q
|
||||
|
||||
let s = SkeletonEntity()
|
||||
worldAnchor.addChild(s.root)
|
||||
skeleton = s
|
||||
|
||||
let m = MeshEntity()
|
||||
worldAnchor.addChild(m.root)
|
||||
mesh = m
|
||||
|
||||
let hf = HandFaceSkeleton()
|
||||
worldAnchor.addChild(hf.root)
|
||||
handFace = hf
|
||||
|
||||
installOrbitGestures()
|
||||
}
|
||||
|
||||
func updateVideo(_ pixelBuffer: CVPixelBuffer) {
|
||||
videoQuad?.update(pixelBuffer)
|
||||
}
|
||||
|
||||
func updateSkeleton(_ skeletons: [Int: SkeletonPayload]) {
|
||||
skeleton?.update(skeletons)
|
||||
}
|
||||
|
||||
func updateMesh(_ persons: [MultiHMRPerson]) {
|
||||
mesh?.update(persons)
|
||||
}
|
||||
|
||||
func updateHandFace(hands: HandsPayload?, face: FacePayload?,
|
||||
depth: Float) {
|
||||
handFace?.setDepth(depth)
|
||||
handFace?.update(hands: hands, face: face)
|
||||
}
|
||||
|
||||
func setSkeletonVisible(_ v: Bool) { skeleton?.setVisible(v) }
|
||||
func setMeshVisible(_ v: Bool) { mesh?.setVisible(v) }
|
||||
func setVideoVisible(_ v: Bool) { videoQuad?.setVisible(v) }
|
||||
func setVideoOpacity(_ v: Double) { videoQuad?.setOpacity(v) }
|
||||
|
||||
func updateMeshMaterial(metallic: Bool, roughness: Double) {
|
||||
mesh?.setMaterial(metallic: metallic, roughness: roughness)
|
||||
}
|
||||
|
||||
func setLightIntensities(key: Double, fill: Double, rim: Double) {
|
||||
keyLight.light.intensity = Float(key)
|
||||
fillLight.light.intensity = Float(fill)
|
||||
rimLight.light.intensity = Float(rim)
|
||||
}
|
||||
|
||||
func setFieldOfView(_ deg: Double) {
|
||||
camera.camera.fieldOfViewInDegrees = Float(deg)
|
||||
}
|
||||
|
||||
// MARK: - Orbital camera
|
||||
|
||||
private func applyCamera() {
|
||||
let cy = cos(orbitYaw), sy = sin(orbitYaw)
|
||||
let cp = cos(orbitPitch), sp = sin(orbitPitch)
|
||||
let pos = SIMD3<Float>(orbitRadius * cp * sy,
|
||||
orbitRadius * sp,
|
||||
orbitRadius * cp * cy)
|
||||
cameraAnchor.transform.translation = pos
|
||||
camera.look(at: .zero, from: pos, relativeTo: nil)
|
||||
}
|
||||
|
||||
private func installOrbitGestures() {
|
||||
let pan = NSPanGestureRecognizer(
|
||||
target: OrbitTarget.shared, action: #selector(
|
||||
OrbitTarget.handlePan(_:)))
|
||||
OrbitTarget.shared.controller = self
|
||||
arView.addGestureRecognizer(pan)
|
||||
}
|
||||
|
||||
fileprivate func orbit(dx: Float, dy: Float) {
|
||||
orbitYaw += dx * 0.01
|
||||
orbitPitch = max(-1.4, min(1.4, orbitPitch + dy * 0.01))
|
||||
applyCamera()
|
||||
}
|
||||
}
|
||||
|
||||
/// Bridges the AppKit pan gesture to `SceneController.orbit`.
|
||||
final class OrbitTarget: NSObject {
|
||||
static let shared = OrbitTarget()
|
||||
weak var controller: SceneController?
|
||||
private var last: CGPoint = .zero
|
||||
|
||||
@objc func handlePan(_ g: NSPanGestureRecognizer) {
|
||||
switch g.state {
|
||||
case .began:
|
||||
last = g.translation(in: g.view)
|
||||
case .changed:
|
||||
let p = g.translation(in: g.view)
|
||||
let dx = Float(p.x - last.x)
|
||||
let dy = Float(p.y - last.y)
|
||||
last = p
|
||||
MainActor.assumeIsolated {
|
||||
self.controller?.orbit(dx: dx, dy: -dy)
|
||||
}
|
||||
default:
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
import Foundation
|
||||
import Metal
|
||||
import MetalKit
|
||||
|
||||
/// Metal renderer for the 10 background viz modes (storm, tunnel,
|
||||
/// plasma, kaleido, voronoi, metaballs, starfield, bars, hands3d,
|
||||
/// openpos). Compiles the bundled scene.metal at runtime; sits as the
|
||||
/// backing layer under the transparent ARView.
|
||||
final class SceneRenderer: NSObject, MTKViewDelegate {
|
||||
/// Mirror of scene.metal SceneUniforms (36 floats, 144 B). Field
|
||||
/// order MUST match the shader exactly.
|
||||
struct SceneUniforms {
|
||||
var time: Float = 0
|
||||
var rms: Float = 0
|
||||
var kp_norm: Float = 0
|
||||
var netz_dev: Float = 0
|
||||
var lightning_flash: Float = 0
|
||||
var flare: Float = 0
|
||||
var wind_norm: Float = 0
|
||||
var bz_norm: Float = 0
|
||||
var social_rate: Float = 0
|
||||
var pose_alive: Float = 0
|
||||
var pose_count: Float = 0
|
||||
var width: Float = 1280
|
||||
var height: Float = 720
|
||||
var viz_mode: Float = 0
|
||||
var hand_l_x: Float = 0
|
||||
var hand_l_y: Float = 0
|
||||
var hand_r_x: Float = 0
|
||||
var hand_r_y: Float = 0
|
||||
var mouth_open: Float = 0
|
||||
var eye_open_l: Float = 0
|
||||
var eye_open_r: Float = 0
|
||||
var head_tilt: Float = 0
|
||||
var head_yaw: Float = 0
|
||||
var finger_pinch_l: Float = 0
|
||||
var finger_pinch_r: Float = 0
|
||||
var body_x: Float = 0
|
||||
var body_y: Float = 0
|
||||
var body_z: Float = 0
|
||||
var body_height: Float = 0
|
||||
var arm_spread: Float = 0
|
||||
var pose_velocity: Float = 0
|
||||
var _pad0: Float = 0
|
||||
var _pad1: Float = 0
|
||||
var _pad2: Float = 0
|
||||
var _pad3: Float = 0
|
||||
var _pad4: Float = 0
|
||||
}
|
||||
|
||||
private let device: MTLDevice
|
||||
private let commandQueue: MTLCommandQueue
|
||||
private let bgPipeline: MTLRenderPipelineState
|
||||
private let uniformsBuffer: MTLBuffer
|
||||
private var startTime: CFTimeInterval = CACurrentMediaTime()
|
||||
|
||||
var uniforms = SceneUniforms()
|
||||
|
||||
static func make() -> SceneRenderer? { SceneRenderer(failable: ()) }
|
||||
|
||||
private init?(failable: Void) {
|
||||
guard let dev = MTLCreateSystemDefaultDevice(),
|
||||
let queue = dev.makeCommandQueue() else { return nil }
|
||||
self.device = dev
|
||||
self.commandQueue = queue
|
||||
guard let url = Bundle.main.url(forResource: "scene",
|
||||
withExtension: "metal"),
|
||||
let source = try? String(contentsOf: url, encoding: .utf8) else {
|
||||
NSLog("SceneRenderer: scene.metal missing from bundle")
|
||||
return nil
|
||||
}
|
||||
let lib: MTLLibrary
|
||||
do { lib = try dev.makeLibrary(source: source,
|
||||
options: MTLCompileOptions()) }
|
||||
catch { NSLog("SceneRenderer: compile error %@",
|
||||
String(describing: error)); return nil }
|
||||
guard let vfn = lib.makeFunction(name: "bg_vertex"),
|
||||
let ffn = lib.makeFunction(name: "bg_fragment") else {
|
||||
NSLog("SceneRenderer: bg_vertex/bg_fragment missing")
|
||||
return nil
|
||||
}
|
||||
let pd = MTLRenderPipelineDescriptor()
|
||||
pd.vertexFunction = vfn
|
||||
pd.fragmentFunction = ffn
|
||||
pd.colorAttachments[0].pixelFormat = .bgra8Unorm
|
||||
do { self.bgPipeline =
|
||||
try dev.makeRenderPipelineState(descriptor: pd) }
|
||||
catch { NSLog("SceneRenderer: pipeline failed %@",
|
||||
String(describing: error)); return nil }
|
||||
guard let buf = dev.makeBuffer(
|
||||
length: MemoryLayout<SceneUniforms>.stride,
|
||||
options: .storageModeShared) else { return nil }
|
||||
self.uniformsBuffer = buf
|
||||
super.init()
|
||||
}
|
||||
|
||||
func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
|
||||
uniforms.width = Float(size.width)
|
||||
uniforms.height = Float(size.height)
|
||||
}
|
||||
|
||||
func draw(in view: MTKView) {
|
||||
uniforms.time = Float(CACurrentMediaTime() - startTime)
|
||||
uniformsBuffer.contents().bindMemory(
|
||||
to: SceneUniforms.self, capacity: 1).pointee = uniforms
|
||||
guard let rpd = view.currentRenderPassDescriptor,
|
||||
let drawable = view.currentDrawable,
|
||||
let cb = commandQueue.makeCommandBuffer(),
|
||||
let enc = cb.makeRenderCommandEncoder(descriptor: rpd)
|
||||
else { return }
|
||||
enc.setRenderPipelineState(bgPipeline)
|
||||
enc.setFragmentBuffer(uniformsBuffer, offset: 0, index: 0)
|
||||
enc.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
|
||||
enc.endEncoding()
|
||||
cb.present(drawable)
|
||||
cb.commit()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
// avlivebody-mac/Sources/AVLiveBody/SceneUniformBuilder.swift
|
||||
import AVLiveWire
|
||||
import simd
|
||||
|
||||
enum SceneUniformBuilder {
|
||||
// ARKit defaultBody3D indices (best-effort constants — ARSkeletonDefinition
|
||||
// is iOS-only and cannot be queried here; verify indices against the
|
||||
// iPhone's 91-joint order if channels read zero unexpectedly; the j()
|
||||
// guard degrades any out-of-range or invalid joint to 0 gracefully).
|
||||
private static let hipsIdx = 0
|
||||
private static let headIdx = 51
|
||||
private static let lWristIdx = 32
|
||||
private static let rWristIdx = 7
|
||||
|
||||
static func fill(_ u: inout SceneRenderer.SceneUniforms,
|
||||
skeletons: [Int: SkeletonPayload],
|
||||
hands: HandsPayload?,
|
||||
face: FacePayload?,
|
||||
prevPelvis: inout SIMD3<Float>?) {
|
||||
u.pose_count = Float(skeletons.count)
|
||||
u.pose_alive = skeletons.isEmpty ? 0 : 1
|
||||
|
||||
if let body = skeletons.values.first {
|
||||
func j(_ i: Int) -> SIMD3<Float>? {
|
||||
(i < body.joints.count && i < body.valid.count
|
||||
&& body.valid[i]) ? body.joints[i] : nil
|
||||
}
|
||||
if let hips = j(hipsIdx) {
|
||||
u.body_x = hips.x; u.body_y = hips.y; u.body_z = hips.z
|
||||
if let prev = prevPelvis {
|
||||
let v = simd_length(hips - prev)
|
||||
u.pose_velocity = u.pose_velocity * 0.7 + v * 0.3
|
||||
}
|
||||
prevPelvis = hips
|
||||
if let head = j(headIdx) {
|
||||
u.body_height = abs(head.y - hips.y)
|
||||
}
|
||||
}
|
||||
if let lw = j(lWristIdx), let rw = j(rWristIdx) {
|
||||
u.arm_spread = abs(lw.x - rw.x)
|
||||
}
|
||||
}
|
||||
|
||||
// Hands (Phase-0 Vision, normalized [0,1]).
|
||||
u.hand_l_x = 0; u.hand_l_y = 0; u.hand_r_x = 0; u.hand_r_y = 0
|
||||
u.finger_pinch_l = 0; u.finger_pinch_r = 0
|
||||
for hand in hands?.hands ?? [] {
|
||||
let c = centroid(hand.points) // (x,y)
|
||||
let pinch = dist(hand.points, 4, 8) // thumb_tip..index_tip
|
||||
if hand.isRight {
|
||||
u.hand_r_x = c.x; u.hand_r_y = c.y; u.finger_pinch_r = pinch
|
||||
} else {
|
||||
u.hand_l_x = c.x; u.hand_l_y = c.y; u.finger_pinch_l = pinch
|
||||
}
|
||||
}
|
||||
|
||||
// Face (Phase-0 Vision allPoints). Region-based derivation is
|
||||
// deferred to v2 — the Vision face-region index mapping is not yet
|
||||
// established. Leave all face scalars at 0 intentionally for v1.
|
||||
if let f = face, f.points.count > 8 {
|
||||
let ys = f.points.map { $0.y }, xs = f.points.map { $0.x }
|
||||
let h = (ys.max() ?? 0) - (ys.min() ?? 0)
|
||||
let w = (xs.max() ?? 0) - (xs.min() ?? 0)
|
||||
u.mouth_open = 0 // set from outer/inner-lip points at impl time
|
||||
u.eye_open_l = 0; u.eye_open_r = 0
|
||||
u.head_tilt = 0; u.head_yaw = 0
|
||||
_ = (h, w)
|
||||
}
|
||||
}
|
||||
|
||||
private static func centroid(_ p: [SIMD3<Float>]) -> SIMD2<Float> {
|
||||
guard !p.isEmpty else { return .zero }
|
||||
var s = SIMD2<Float>(0, 0)
|
||||
for v in p { s.x += v.x; s.y += v.y }
|
||||
return s / Float(p.count)
|
||||
}
|
||||
private static func dist(_ p: [SIMD3<Float>], _ a: Int, _ b: Int)
|
||||
-> Float {
|
||||
guard a < p.count, b < p.count else { return 0 }
|
||||
return simd_length(SIMD2(p[a].x, p[a].y) - SIMD2(p[b].x, p[b].y))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
// avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift
|
||||
import SwiftUI
|
||||
|
||||
/// Collapsible right-side control panel bound to RenderSettings.
|
||||
struct SettingsPanel: View {
|
||||
@ObservedObject var settings: RenderSettings
|
||||
|
||||
var body: some View {
|
||||
ScrollView {
|
||||
VStack(alignment: .leading, spacing: 14) {
|
||||
Text("Reglages").font(.headline)
|
||||
|
||||
group("Couches") {
|
||||
Toggle("Squelette", isOn: $settings.showSkeleton)
|
||||
Toggle("Mesh", isOn: $settings.showMesh)
|
||||
Toggle("Video", isOn: $settings.showVideo)
|
||||
}
|
||||
|
||||
group("Mesh") {
|
||||
Toggle("Metallique", isOn: $settings.meshMetallic)
|
||||
slider("Rugosite", $settings.meshRoughness,
|
||||
0...1)
|
||||
}
|
||||
|
||||
group("Lumieres") {
|
||||
slider("Principale", $settings.keyIntensity,
|
||||
0...10000)
|
||||
slider("Remplissage", $settings.fillIntensity,
|
||||
0...10000)
|
||||
slider("Contre-jour", $settings.rimIntensity,
|
||||
0...10000)
|
||||
}
|
||||
|
||||
group("Vue") {
|
||||
slider("Champ de vision", $settings.fieldOfView,
|
||||
20...120)
|
||||
slider("Opacite video", $settings.videoOpacity,
|
||||
0...1)
|
||||
}
|
||||
|
||||
group("Scene") {
|
||||
Toggle("Fond shader", isOn: $settings.showScene)
|
||||
Picker("Mode", selection: $settings.vizMode) {
|
||||
ForEach(0..<10, id: \.self) { i in
|
||||
Text(Self.modeName(i)).tag(i)
|
||||
}
|
||||
}
|
||||
.pickerStyle(.menu)
|
||||
}
|
||||
}
|
||||
.padding(16)
|
||||
}
|
||||
.frame(width: 280)
|
||||
.background(.ultraThinMaterial)
|
||||
}
|
||||
|
||||
private static let modeNames = ["storm","tunnel","plasma","kaleido",
|
||||
"voronoi","metaballs","starfield","bars","hands3d","openpos"]
|
||||
static func modeName(_ i: Int) -> String {
|
||||
modeNames.indices.contains(i) ? "\(i) \(modeNames[i])" : "\(i)"
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private func group(_ title: String,
|
||||
@ViewBuilder _ content: () -> some View)
|
||||
-> some View {
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
Text(title).font(.subheadline).foregroundStyle(.secondary)
|
||||
content()
|
||||
}
|
||||
}
|
||||
|
||||
private func slider(_ label: String,
|
||||
_ value: Binding<Double>,
|
||||
_ range: ClosedRange<Double>) -> some View {
|
||||
VStack(alignment: .leading, spacing: 2) {
|
||||
HStack {
|
||||
Text(label).font(.caption)
|
||||
Spacer()
|
||||
Text(String(format: "%.2f", value.wrappedValue))
|
||||
.font(.caption).foregroundStyle(.secondary)
|
||||
}
|
||||
Slider(value: value, in: range)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
import AVLiveWire
|
||||
import AppKit
|
||||
import Foundation
|
||||
import RealityKit
|
||||
import simd
|
||||
|
||||
/// Renders 91-joint skeletons as yellow marker spheres. One marker
|
||||
/// pool per pid. ARKit world coords -> RealityKit space (x, -y, -z).
|
||||
@MainActor
|
||||
final class SkeletonEntity {
|
||||
let root = Entity()
|
||||
|
||||
private static let jointCount = 91
|
||||
private static let markerRadius: Float = 0.012
|
||||
|
||||
private var pools: [Int: [ModelEntity]] = [:]
|
||||
private let mesh = MeshResource.generateSphere(radius: markerRadius)
|
||||
private let material = SimpleMaterial(
|
||||
color: NSColor.systemYellow, roughness: 0.6, isMetallic: false)
|
||||
|
||||
func setVisible(_ visible: Bool) { root.isEnabled = visible }
|
||||
|
||||
func update(_ skeletons: [Int: SkeletonPayload]) {
|
||||
// Drop pools for pids no longer present.
|
||||
for pid in pools.keys where skeletons[pid] == nil {
|
||||
pools[pid]?.forEach { $0.removeFromParent() }
|
||||
pools.removeValue(forKey: pid)
|
||||
}
|
||||
for (pid, payload) in skeletons {
|
||||
let pool = pools[pid] ?? makePool()
|
||||
pools[pid] = pool
|
||||
let n = min(Self.jointCount, payload.joints.count,
|
||||
payload.valid.count)
|
||||
for i in 0..<n {
|
||||
let marker = pool[i]
|
||||
if payload.valid[i] {
|
||||
let j = payload.joints[i]
|
||||
marker.transform.translation = arkitToRealityKit(j)
|
||||
marker.isEnabled = true
|
||||
} else {
|
||||
marker.isEnabled = false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private func makePool() -> [ModelEntity] {
|
||||
var pool: [ModelEntity] = []
|
||||
pool.reserveCapacity(Self.jointCount)
|
||||
for _ in 0..<Self.jointCount {
|
||||
let e = ModelEntity(mesh: mesh, materials: [material])
|
||||
e.isEnabled = false
|
||||
root.addChild(e)
|
||||
pool.append(e)
|
||||
}
|
||||
return pool
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
import SwiftUI
|
||||
|
||||
/// A thin overlay showing the USB connection state.
|
||||
struct StatusBar: View {
|
||||
@ObservedObject var consumer: USBSkeletonConsumer
|
||||
|
||||
var body: some View {
|
||||
HStack(spacing: 6) {
|
||||
Circle()
|
||||
.fill(consumer.connected ? Color.green : Color.orange)
|
||||
.frame(width: 9, height: 9)
|
||||
Text(consumer.connected
|
||||
? "iPhone connected (USB)"
|
||||
: "waiting for iPhone…")
|
||||
.font(.caption)
|
||||
.foregroundStyle(.white)
|
||||
Spacer()
|
||||
}
|
||||
.padding(8)
|
||||
.background(.black.opacity(0.5))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,60 @@
|
||||
import AppKit
|
||||
import CoreImage
|
||||
import CoreVideo
|
||||
import Foundation
|
||||
import RealityKit
|
||||
|
||||
/// A flat plane at the back of the scene, textured with the iPhone
|
||||
/// camera video. `update(_:)` is called on the main queue per frame.
|
||||
@MainActor
|
||||
final class VideoQuad {
|
||||
let entity = ModelEntity()
|
||||
|
||||
private let ciContext = CIContext()
|
||||
private var opacity: Double = 1.0
|
||||
/// Plane is 1.6 m wide, 16:9; positioned 2 m behind the body.
|
||||
private static let width: Float = 1.6
|
||||
private static let height: Float = 0.9
|
||||
private static let zBack: Float = -2.0
|
||||
|
||||
init() {
|
||||
let plane = MeshResource.generatePlane(
|
||||
width: Self.width, height: Self.height)
|
||||
var material = UnlitMaterial()
|
||||
material.color = .init(tint: .white)
|
||||
entity.model = ModelComponent(mesh: plane,
|
||||
materials: [material])
|
||||
entity.transform.translation =
|
||||
SIMD3<Float>(0, 0, Self.zBack)
|
||||
}
|
||||
|
||||
func setVisible(_ visible: Bool) { entity.isEnabled = visible }
|
||||
|
||||
func setOpacity(_ value: Double) {
|
||||
opacity = value
|
||||
// Re-tint current material; `update(_:)` will also use `opacity`.
|
||||
if var mat = entity.model?.materials.first as? UnlitMaterial {
|
||||
mat.color = .init(tint: NSColor(white: 1, alpha: CGFloat(value)))
|
||||
entity.model?.materials = [mat]
|
||||
}
|
||||
}
|
||||
|
||||
/// Replace the plane's texture from a decoded camera frame.
|
||||
func update(_ pixelBuffer: CVPixelBuffer) {
|
||||
let ci = CIImage(cvPixelBuffer: pixelBuffer)
|
||||
guard let cg = ciContext.createCGImage(
|
||||
ci, from: ci.extent) else { return }
|
||||
guard let texture = try? TextureResource(
|
||||
image: cg, options: .init(semantic: .color)) else {
|
||||
NSLog("VideoQuad: TextureResource creation failed (%dx%d)",
|
||||
CVPixelBufferGetWidth(pixelBuffer),
|
||||
CVPixelBufferGetHeight(pixelBuffer))
|
||||
return
|
||||
}
|
||||
var material = UnlitMaterial()
|
||||
material.color = .init(
|
||||
tint: NSColor(white: 1, alpha: CGFloat(opacity)),
|
||||
texture: .init(texture))
|
||||
entity.model?.materials = [material]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
import AVLiveWire
|
||||
import Foundation
|
||||
import simd
|
||||
|
||||
/// Overrides the highest-scoring Multi-HMR mesh's pelvis depth with
|
||||
/// the first valid USB skeleton pelvis z. Single-person assumption:
|
||||
/// with multiple skeletons in the dict the source pelvis is arbitrary
|
||||
/// (dict iteration order). Pure, stateless — unit-testable.
|
||||
enum BodyFusion {
|
||||
/// ARSkeleton3D joint 0 = root (hips), per ARSkeletonDefinition.defaultBody3D.
|
||||
static let pelvisJoint = 0
|
||||
|
||||
static func fuse(persons: [MultiHMRPerson],
|
||||
skeletons: [Int: SkeletonPayload])
|
||||
-> [MultiHMRPerson] {
|
||||
let pelvisZs: [Float] = skeletons.values.compactMap { s in
|
||||
guard pelvisJoint < s.valid.count,
|
||||
s.valid[pelvisJoint] else { return nil }
|
||||
return s.joints[pelvisJoint].z
|
||||
}
|
||||
guard !pelvisZs.isEmpty,
|
||||
let primaryIdx = persons.indices.max(by: {
|
||||
persons[$0].score < persons[$1].score
|
||||
}) else { return persons }
|
||||
var out = persons
|
||||
out[primaryIdx].translation.z = pelvisZs[0]
|
||||
return out
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
import CoreML
|
||||
import CoreVideo
|
||||
import CoreImage
|
||||
import Foundation
|
||||
|
||||
/// One detected SMPL-X body from Multi-HMR.
|
||||
struct MultiHMRPerson {
|
||||
var vertices: [SIMD3<Float>] // 10475 SMPL-X verts, model space
|
||||
var translation: SIMD3<Float> // pelvis translation
|
||||
var score: Float
|
||||
}
|
||||
|
||||
/// CoreML wrapper around the bundled `multihmr_full_672_s.mlpackage`.
|
||||
/// Mirrors `data_only_viz/multihmr_coreml.py`: two MLMultiArray inputs
|
||||
/// (`image` 1x3x672x672 ImageNet-normalized, `cam_K` 1x3x3), fixed
|
||||
/// K=4 person outputs.
|
||||
final class MultiHMRCoreML {
|
||||
static let inputSize = 672
|
||||
static let vertexCount = 10475
|
||||
static let maxPersons = 4
|
||||
private static let detThreshold: Float = 0.3
|
||||
private static let normMean: [Float] = [0.485, 0.456, 0.406]
|
||||
private static let normStd: [Float] = [0.229, 0.224, 0.225]
|
||||
|
||||
private let model: MLModel
|
||||
private let ciContext = CIContext()
|
||||
/// Serial background queue + busy flag for `inferAsync`: keep the
|
||||
/// ~150-300 ms CoreML prediction off the main thread and drop frames
|
||||
/// that arrive while one is in flight.
|
||||
private let inferQueue = DispatchQueue(
|
||||
label: "cc.saillant.avlivebody.multihmr", qos: .userInitiated)
|
||||
private let inferLock = NSLock()
|
||||
private var inferBusy = false
|
||||
|
||||
/// Loads the bundled model. Returns nil if the resource or load
|
||||
/// fails — callers fall back to skeleton-only rendering.
|
||||
init?() {
|
||||
guard let url = Bundle.main.url(
|
||||
forResource: "multihmr_full_672_s",
|
||||
withExtension: "mlmodelc") else {
|
||||
NSLog("MultiHMRCoreML: mlpackage resource missing")
|
||||
return nil
|
||||
}
|
||||
let cfg = MLModelConfiguration()
|
||||
cfg.computeUnits = .cpuAndGPU
|
||||
do {
|
||||
model = try MLModel(contentsOf: url, configuration: cfg)
|
||||
} catch {
|
||||
NSLog("MultiHMRCoreML: load failed %@",
|
||||
String(describing: error))
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
/// Async inference: runs `infer` on a background queue, DROPS the
|
||||
/// frame if a prior prediction is still in flight, and calls
|
||||
/// `completion` with the result ON THE BACKGROUND QUEUE (the caller
|
||||
/// hops back to the main thread for UI). Prevents the per-frame
|
||||
/// CoreML prediction from saturating the run loop.
|
||||
func inferAsync(_ pixelBuffer: CVPixelBuffer,
|
||||
cameraK: [Float],
|
||||
completion: @escaping ([MultiHMRPerson]) -> Void) {
|
||||
inferLock.lock()
|
||||
if inferBusy { inferLock.unlock(); return }
|
||||
inferBusy = true
|
||||
inferLock.unlock()
|
||||
inferQueue.async { [weak self] in
|
||||
guard let self else { return }
|
||||
let result = self.infer(pixelBuffer, cameraK: cameraK)
|
||||
self.inferLock.lock()
|
||||
self.inferBusy = false
|
||||
self.inferLock.unlock()
|
||||
completion(result)
|
||||
}
|
||||
}
|
||||
|
||||
/// Run inference on one camera frame. `cameraK` is the 3x3 camera
|
||||
/// intrinsics row-major.
|
||||
func infer(_ pixelBuffer: CVPixelBuffer,
|
||||
cameraK: [Float]) -> [MultiHMRPerson] {
|
||||
guard let image = makeImageInput(pixelBuffer),
|
||||
let k = makeKInput(cameraK) else { return [] }
|
||||
let inputs: [String: MLFeatureValue] = [
|
||||
"image": MLFeatureValue(multiArray: image),
|
||||
"cam_K": MLFeatureValue(multiArray: k),
|
||||
]
|
||||
guard let provider = try? MLDictionaryFeatureProvider(
|
||||
dictionary: inputs),
|
||||
let out = try? model.prediction(from: provider) else {
|
||||
return []
|
||||
}
|
||||
return parse(out)
|
||||
}
|
||||
|
||||
// MARK: - Input preprocessing
|
||||
|
||||
/// `CVPixelBuffer` -> [1,3,672,672] Float32, RGB, ImageNet-normed.
|
||||
private func makeImageInput(_ pb: CVPixelBuffer) -> MLMultiArray? {
|
||||
let n = Self.inputSize
|
||||
// Resize to n x n BGRA via CoreImage.
|
||||
let ci = CIImage(cvPixelBuffer: pb)
|
||||
let sx = CGFloat(n) / ci.extent.width
|
||||
let sy = CGFloat(n) / ci.extent.height
|
||||
let scaled = ci.transformed(
|
||||
by: CGAffineTransform(scaleX: sx, y: sy))
|
||||
var dst: CVPixelBuffer?
|
||||
CVPixelBufferCreate(kCFAllocatorDefault, n, n,
|
||||
kCVPixelFormatType_32BGRA, nil, &dst)
|
||||
guard let dst else { return nil }
|
||||
ciContext.render(scaled, to: dst)
|
||||
CVPixelBufferLockBaseAddress(dst, .readOnly)
|
||||
defer { CVPixelBufferUnlockBaseAddress(dst, .readOnly) }
|
||||
guard let base = CVPixelBufferGetBaseAddress(dst) else {
|
||||
return nil
|
||||
}
|
||||
let rowBytes = CVPixelBufferGetBytesPerRow(dst)
|
||||
let px = base.assumingMemoryBound(to: UInt8.self)
|
||||
guard let arr = try? MLMultiArray(
|
||||
shape: [1, 3, NSNumber(value: n), NSNumber(value: n)],
|
||||
dataType: .float32) else { return nil }
|
||||
let ptr = arr.dataPointer.assumingMemoryBound(to: Float.self)
|
||||
let plane = n * n
|
||||
for y in 0..<n {
|
||||
for x in 0..<n {
|
||||
let p = y * rowBytes + x * 4 // BGRA
|
||||
let b = Float(px[p]) / 255.0
|
||||
let g = Float(px[p + 1]) / 255.0
|
||||
let r = Float(px[p + 2]) / 255.0
|
||||
let idx = y * n + x
|
||||
ptr[idx] =
|
||||
(r - Self.normMean[0]) / Self.normStd[0]
|
||||
ptr[plane + idx] =
|
||||
(g - Self.normMean[1]) / Self.normStd[1]
|
||||
ptr[2 * plane + idx] =
|
||||
(b - Self.normMean[2]) / Self.normStd[2]
|
||||
}
|
||||
}
|
||||
return arr
|
||||
}
|
||||
|
||||
/// 9 row-major intrinsics -> [1,3,3] Float32.
|
||||
private func makeKInput(_ k: [Float]) -> MLMultiArray? {
|
||||
guard k.count == 9,
|
||||
let arr = try? MLMultiArray(
|
||||
shape: [1, 3, 3], dataType: .float32) else { return nil }
|
||||
let ptr = arr.dataPointer.assumingMemoryBound(to: Float.self)
|
||||
for i in 0..<9 { ptr[i] = k[i] }
|
||||
return arr
|
||||
}
|
||||
|
||||
// MARK: - Output parsing
|
||||
|
||||
private func parse(_ out: MLFeatureProvider) -> [MultiHMRPerson] {
|
||||
guard let v3d = out.featureValue(for: "var_2420")?
|
||||
.multiArrayValue,
|
||||
let transl = out.featureValue(for: "var_2423")?
|
||||
.multiArrayValue,
|
||||
let scores = out.featureValue(for: "var_2436")?
|
||||
.multiArrayValue else { return [] }
|
||||
var persons: [MultiHMRPerson] = []
|
||||
let vc = Self.vertexCount
|
||||
for k in 0..<Self.maxPersons {
|
||||
let score = scores[k].floatValue
|
||||
if score < Self.detThreshold { continue }
|
||||
var verts = [SIMD3<Float>](
|
||||
repeating: .zero, count: vc)
|
||||
let base = k * vc * 3
|
||||
for i in 0..<vc {
|
||||
let o = base + i * 3
|
||||
verts[i] = SIMD3(v3d[o].floatValue,
|
||||
v3d[o + 1].floatValue,
|
||||
v3d[o + 2].floatValue)
|
||||
}
|
||||
let tb = k * 3
|
||||
persons.append(MultiHMRPerson(
|
||||
vertices: verts,
|
||||
translation: SIMD3(transl[tb].floatValue,
|
||||
transl[tb + 1].floatValue,
|
||||
transl[tb + 2].floatValue),
|
||||
score: score))
|
||||
}
|
||||
return persons
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,135 @@
|
||||
import Foundation
|
||||
import Darwin
|
||||
|
||||
/// Transport abstraction over the usbmuxd Unix socket. The real
|
||||
/// implementation wraps a `socket(AF_UNIX)`; tests inject a mock.
|
||||
protocol MuxTransport {
|
||||
func send(_ data: Data)
|
||||
func receivePacket() -> Data?
|
||||
func close()
|
||||
}
|
||||
|
||||
/// usbmux client: device discovery + connect-to-port. After a
|
||||
/// successful `connect`, the same transport carries the raw tunneled
|
||||
/// byte stream from the device.
|
||||
final class USBClient {
|
||||
private let transport: MuxTransport
|
||||
private var tag: UInt32 = 0
|
||||
|
||||
init(transport: MuxTransport) {
|
||||
self.transport = transport
|
||||
}
|
||||
|
||||
func listDevices() -> [Int] {
|
||||
tag += 1
|
||||
transport.send(USBMuxProtocol.encode(
|
||||
plist: ["MessageType": "ListDevices"], tag: tag))
|
||||
guard let reply = transport.receivePacket(),
|
||||
let plist = USBMuxProtocol.decode(reply),
|
||||
let list = plist["DeviceList"] as? [[String: Any]]
|
||||
else { return [] }
|
||||
return list.compactMap { $0["DeviceID"] as? Int }
|
||||
}
|
||||
|
||||
/// Returns true once the transport is tunneled to `port` on the
|
||||
/// device. usbmux wants the TCP port in big-endian order.
|
||||
func connect(deviceID: Int, port: UInt16) -> Bool {
|
||||
tag += 1
|
||||
let swapped = Int((port << 8) | (port >> 8))
|
||||
transport.send(USBMuxProtocol.encode(plist: [
|
||||
"MessageType": "Connect",
|
||||
"DeviceID": deviceID,
|
||||
"PortNumber": swapped,
|
||||
], tag: tag))
|
||||
guard let reply = transport.receivePacket(),
|
||||
let plist = USBMuxProtocol.decode(reply),
|
||||
let number = plist["Number"] as? Int
|
||||
else { return false }
|
||||
return number == 0
|
||||
}
|
||||
}
|
||||
|
||||
/// Production transport: blocking AF_UNIX socket to usbmuxd.
|
||||
final class UnixMuxTransport: MuxTransport {
|
||||
private var fd: Int32 = -1
|
||||
|
||||
init?(path: String = "/var/run/usbmuxd") {
|
||||
fd = socket(AF_UNIX, SOCK_STREAM, 0)
|
||||
guard fd >= 0 else { return nil }
|
||||
var addr = sockaddr_un()
|
||||
addr.sun_family = sa_family_t(AF_UNIX)
|
||||
precondition(path.utf8.count < 104,
|
||||
"usbmuxd socket path exceeds sun_path limit")
|
||||
_ = path.withCString { src in
|
||||
withUnsafeMutablePointer(to: &addr.sun_path) {
|
||||
$0.withMemoryRebound(to: CChar.self, capacity: 104) {
|
||||
strcpy($0, src)
|
||||
}
|
||||
}
|
||||
}
|
||||
let size = socklen_t(MemoryLayout<sockaddr_un>.size)
|
||||
let ok = withUnsafePointer(to: &addr) {
|
||||
$0.withMemoryRebound(to: sockaddr.self, capacity: 1) {
|
||||
Darwin.connect(fd, $0, size)
|
||||
}
|
||||
}
|
||||
if ok != 0 { Darwin.close(fd); return nil }
|
||||
}
|
||||
|
||||
func send(_ data: Data) {
|
||||
guard fd >= 0 else { return }
|
||||
data.withUnsafeBytes { buf in
|
||||
guard let base = buf.baseAddress else { return }
|
||||
var off = 0
|
||||
while off < data.count {
|
||||
let w = Darwin.write(fd, base.advanced(by: off),
|
||||
data.count - off)
|
||||
if w <= 0 {
|
||||
if w < 0 && errno == EINTR { continue }
|
||||
break
|
||||
}
|
||||
off += w
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Read one usbmux packet: 4-byte LE length prefix then body.
|
||||
func receivePacket() -> Data? {
|
||||
guard let head = readN(4) else { return nil }
|
||||
guard let len = USBMuxProtocol.readLE32(head, 0) else { return nil }
|
||||
let total = Int(len)
|
||||
guard total >= 16, let rest = readN(total - 4) else { return nil }
|
||||
return head + rest
|
||||
}
|
||||
|
||||
/// Read raw tunneled bytes after a successful Connect.
|
||||
func readStream(max: Int = 65536) -> Data? {
|
||||
readN(max, exact: false)
|
||||
}
|
||||
|
||||
private func readN(_ n: Int, exact: Bool = true) -> Data? {
|
||||
var buf = [UInt8](repeating: 0, count: n)
|
||||
var got = 0
|
||||
while got < n {
|
||||
let r = buf.withUnsafeMutableBytes {
|
||||
Darwin.read(fd, $0.baseAddress!.advanced(by: got), n - got)
|
||||
}
|
||||
if r < 0 {
|
||||
if errno == EINTR { continue }
|
||||
return got > 0 && !exact ? Data(buf[0..<got]) : nil
|
||||
}
|
||||
if r == 0 { // EOF — peer closed
|
||||
return got > 0 && !exact ? Data(buf[0..<got]) : nil
|
||||
}
|
||||
got += r
|
||||
if !exact { break }
|
||||
}
|
||||
return Data(buf[0..<got])
|
||||
}
|
||||
|
||||
deinit { close() }
|
||||
|
||||
func close() {
|
||||
if fd >= 0 { Darwin.close(fd); fd = -1 }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
import Foundation
|
||||
|
||||
/// Codec for the usbmuxd request/response protocol. 16-byte
|
||||
/// little-endian header (length, version=1, message=8, tag) then an
|
||||
/// XML property list.
|
||||
enum USBMuxProtocol {
|
||||
static func encode(plist: [String: Any], tag: UInt32) -> Data {
|
||||
let body = (try? PropertyListSerialization.data(
|
||||
fromPropertyList: plist, format: .xml, options: 0))
|
||||
?? Data()
|
||||
var d = Data()
|
||||
appendLE32(&d, UInt32(16 + body.count)) // length
|
||||
appendLE32(&d, 1) // version
|
||||
appendLE32(&d, 8) // message: plist
|
||||
appendLE32(&d, tag)
|
||||
d.append(body)
|
||||
return d
|
||||
}
|
||||
|
||||
static func decode(_ packet: Data) -> [String: Any]? {
|
||||
guard packet.count >= 16 else { return nil }
|
||||
let body = packet.dropFirst(16)
|
||||
return (try? PropertyListSerialization.propertyList(
|
||||
from: body, options: [], format: nil)) as? [String: Any]
|
||||
}
|
||||
|
||||
static func appendLE32(_ d: inout Data, _ v: UInt32) {
|
||||
for i in 0..<4 { d.append(UInt8((v >> (8 * i)) & 0xFF)) }
|
||||
}
|
||||
|
||||
static func readLE32(_ d: Data, _ offset: Int) -> UInt32? {
|
||||
guard offset >= 0, d.count >= offset + 4 else { return nil }
|
||||
let b = [UInt8](d)
|
||||
var v: UInt32 = 0
|
||||
for i in 0..<4 { v |= UInt32(b[offset + i]) << (8 * i) }
|
||||
return v
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,123 @@
|
||||
import AVLiveWire
|
||||
import Combine
|
||||
import CoreVideo
|
||||
import Foundation
|
||||
|
||||
/// Connects to the tethered iPhone over USB (usbmuxd), demuxes the
|
||||
/// AVLiveWire stream, republishes skeleton payloads (keyed by pid)
|
||||
/// and forwards decoded camera frames. Blocking transport runs on a
|
||||
/// dedicated background thread; only `@Published` writes hop to main.
|
||||
final class USBSkeletonConsumer: ObservableObject {
|
||||
/// 91-joint skeleton payloads keyed by pid.
|
||||
@Published var skeletons: [Int: SkeletonPayload] = [:]
|
||||
@Published var connected = false
|
||||
/// Latest Vision hand landmarks from the iPhone (tag=4).
|
||||
@Published var hands: HandsPayload?
|
||||
/// Latest Vision face landmarks from the iPhone (tag=5).
|
||||
@Published var face: FacePayload?
|
||||
|
||||
/// Called on the main queue for every decoded camera frame.
|
||||
var onVideoFrame: ((CVPixelBuffer) -> Void)?
|
||||
|
||||
/// TCP port the iPhone `USBServer` listens on.
|
||||
static let devicePort: UInt16 = 7000
|
||||
|
||||
private let videoDecoder = VideoDecoder()
|
||||
private let stateLock = NSLock()
|
||||
private var running = false
|
||||
private var thread: Thread?
|
||||
|
||||
init() {
|
||||
videoDecoder.onFrame = { [weak self] pixelBuffer in
|
||||
DispatchQueue.main.async {
|
||||
self?.onVideoFrame?(pixelBuffer)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private var isRunning: Bool {
|
||||
stateLock.lock(); defer { stateLock.unlock() }
|
||||
return running
|
||||
}
|
||||
|
||||
func start() {
|
||||
stateLock.lock()
|
||||
if running { stateLock.unlock(); return }
|
||||
running = true
|
||||
let t = Thread { [weak self] in self?.loop() }
|
||||
t.name = "cc.avlive.usbconsumer"
|
||||
thread = t
|
||||
stateLock.unlock()
|
||||
t.start()
|
||||
}
|
||||
|
||||
func stop() {
|
||||
stateLock.lock(); running = false; stateLock.unlock()
|
||||
}
|
||||
|
||||
private func loop() {
|
||||
while isRunning {
|
||||
guard let transport = UnixMuxTransport() else {
|
||||
NSLog("USBSkeletonConsumer: no usbmuxd; retry")
|
||||
Thread.sleep(forTimeInterval: 1.0); continue
|
||||
}
|
||||
let client = USBClient(transport: transport)
|
||||
let devices = client.listDevices()
|
||||
guard let dev = devices.first,
|
||||
client.connect(deviceID: dev,
|
||||
port: Self.devicePort) else {
|
||||
NSLog("USBSkeletonConsumer: no device; retry")
|
||||
transport.close()
|
||||
Thread.sleep(forTimeInterval: 1.0); continue
|
||||
}
|
||||
NSLog("USBSkeletonConsumer: connected to device %d", dev)
|
||||
publishConnected(true)
|
||||
var demux = StreamDemuxer()
|
||||
while isRunning {
|
||||
guard let chunk = transport.readStream(),
|
||||
!chunk.isEmpty else { break }
|
||||
for frame in demux.feed(chunk) { route(frame) }
|
||||
}
|
||||
transport.close()
|
||||
publishConnected(false)
|
||||
NSLog("USBSkeletonConsumer: disconnected")
|
||||
if isRunning { Thread.sleep(forTimeInterval: 1.0) }
|
||||
}
|
||||
}
|
||||
|
||||
private func route(_ frame: StreamDemuxer.Frame) {
|
||||
switch frame.header.tag {
|
||||
case .skeleton:
|
||||
guard let payload =
|
||||
SkeletonPayload(decoding: frame.payload) else { return }
|
||||
let pid = Int(frame.header.pid)
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.skeletons[pid] = payload
|
||||
}
|
||||
case .video:
|
||||
guard let payload =
|
||||
VideoPayload(decoding: frame.payload) else { return }
|
||||
videoDecoder.decode(payload)
|
||||
case .hands:
|
||||
guard let payload = HandsPayload(decoding: frame.payload)
|
||||
else { return }
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.hands = payload
|
||||
}
|
||||
case .face:
|
||||
guard let payload = FacePayload(decoding: frame.payload)
|
||||
else { return }
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.face = payload
|
||||
}
|
||||
case .meta:
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
private func publishConnected(_ value: Bool) {
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.connected = value
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,184 @@
|
||||
import AVLiveWire
|
||||
import CoreMedia
|
||||
import CoreVideo
|
||||
import Foundation
|
||||
import VideoToolbox
|
||||
|
||||
/// HEVC decoder. Feed `VideoPayload`s in; receive `CVPixelBuffer`s via
|
||||
/// `onFrame`. Keyframe payloads must carry the VPS/SPS/PPS parameter
|
||||
/// sets prepended as 4-byte-length-prefixed NAL units (the layout the
|
||||
/// iOS `VideoEncoder` emits); the decoder (re)builds its format
|
||||
/// description from those.
|
||||
final class VideoDecoder {
|
||||
var onFrame: ((CVPixelBuffer) -> Void)?
|
||||
|
||||
private var session: VTDecompressionSession?
|
||||
private var formatDesc: CMVideoFormatDescription?
|
||||
|
||||
/// Decode one access unit.
|
||||
func decode(_ payload: VideoPayload) {
|
||||
var au = payload.data
|
||||
if payload.isKeyframe {
|
||||
let (params, rest) = Self.splitParameterSets(au)
|
||||
if !params.isEmpty {
|
||||
rebuildFormat(params)
|
||||
}
|
||||
au = rest
|
||||
}
|
||||
guard let fmt = formatDesc, !au.isEmpty else { return }
|
||||
if session == nil { makeSession(fmt) }
|
||||
guard let session, let block = Self.blockBuffer(au) else {
|
||||
return
|
||||
}
|
||||
var sample: CMSampleBuffer?
|
||||
var sampleSize = au.count
|
||||
guard CMSampleBufferCreateReady(
|
||||
allocator: kCFAllocatorDefault, dataBuffer: block,
|
||||
formatDescription: fmt, sampleCount: 1,
|
||||
sampleTimingEntryCount: 0, sampleTimingArray: nil,
|
||||
sampleSizeEntryCount: 1, sampleSizeArray: &sampleSize,
|
||||
sampleBufferOut: &sample) == noErr, let sample else {
|
||||
return
|
||||
}
|
||||
VTDecompressionSessionDecodeFrame(
|
||||
session, sampleBuffer: sample, flags: [],
|
||||
infoFlagsOut: nil) { [weak self] status, _, image, _, _ in
|
||||
guard status == noErr, let image else { return }
|
||||
self?.onFrame?(image)
|
||||
}
|
||||
}
|
||||
|
||||
func stop() {
|
||||
if let session { VTDecompressionSessionInvalidate(session) }
|
||||
session = nil
|
||||
formatDesc = nil
|
||||
}
|
||||
|
||||
deinit { stop() }
|
||||
|
||||
// MARK: - Helpers
|
||||
|
||||
/// Leading 4-byte-length-prefixed NAL units of HEVC parameter-set
|
||||
/// type (VPS=32, SPS=33, PPS=34) are split from the frame data.
|
||||
/// Returns (parameterSetData, frameData).
|
||||
private static func splitParameterSets(_ data: Data)
|
||||
-> (Data, Data) {
|
||||
let bytes = [UInt8](data)
|
||||
var offset = 0
|
||||
var paramEnd = 0
|
||||
while offset + 4 <= bytes.count {
|
||||
let len = (Int(bytes[offset]) << 24)
|
||||
| (Int(bytes[offset + 1]) << 16)
|
||||
| (Int(bytes[offset + 2]) << 8)
|
||||
| Int(bytes[offset + 3])
|
||||
let nalStart = offset + 4
|
||||
guard len > 0, nalStart + len <= bytes.count else { break }
|
||||
let nalType = (Int(bytes[nalStart]) >> 1) & 0x3F
|
||||
if nalType == 32 || nalType == 33 || nalType == 34 {
|
||||
offset = nalStart + len
|
||||
paramEnd = offset
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
return (data.prefix(paramEnd),
|
||||
data.suffix(from: data.startIndex
|
||||
.advanced(by: paramEnd)))
|
||||
}
|
||||
|
||||
private func rebuildFormat(_ paramData: Data) {
|
||||
var sets: [[UInt8]] = []
|
||||
let bytes = [UInt8](paramData)
|
||||
var offset = 0
|
||||
while offset + 4 <= bytes.count {
|
||||
let len = (Int(bytes[offset]) << 24)
|
||||
| (Int(bytes[offset + 1]) << 16)
|
||||
| (Int(bytes[offset + 2]) << 8)
|
||||
| Int(bytes[offset + 3])
|
||||
let start = offset + 4
|
||||
guard len > 0, start + len <= bytes.count else { break }
|
||||
sets.append(Array(bytes[start..<start + len]))
|
||||
offset = start + len
|
||||
}
|
||||
guard sets.count >= 3 else { return }
|
||||
var fmt: CMFormatDescription?
|
||||
let status = withParameterSetPointers(sets) { pBuf, sBuf in
|
||||
CMVideoFormatDescriptionCreateFromHEVCParameterSets(
|
||||
allocator: kCFAllocatorDefault,
|
||||
parameterSetCount: sets.count,
|
||||
parameterSetPointers: pBuf,
|
||||
parameterSetSizes: sBuf,
|
||||
nalUnitHeaderLength: 4, extensions: nil,
|
||||
formatDescriptionOut: &fmt)
|
||||
}
|
||||
if status == noErr, let fmt {
|
||||
formatDesc = fmt
|
||||
if let session { VTDecompressionSessionInvalidate(session) }
|
||||
session = nil
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the C-style parallel arrays of parameter-set pointers and
|
||||
/// sizes that `CMVideoFormatDescriptionCreateFromHEVCParameterSets`
|
||||
/// requires, keeping the backing storage alive for the call.
|
||||
private func withParameterSetPointers(
|
||||
_ sets: [[UInt8]],
|
||||
_ body: (UnsafePointer<UnsafePointer<UInt8>>,
|
||||
UnsafePointer<Int>) -> OSStatus) -> OSStatus {
|
||||
func recurse(_ index: Int,
|
||||
_ ptrs: inout [UnsafePointer<UInt8>],
|
||||
_ sizes: inout [Int]) -> OSStatus {
|
||||
if index == sets.count {
|
||||
return ptrs.withUnsafeBufferPointer { pBuf in
|
||||
sizes.withUnsafeBufferPointer { sBuf in
|
||||
body(pBuf.baseAddress!, sBuf.baseAddress!)
|
||||
}
|
||||
}
|
||||
}
|
||||
return sets[index].withUnsafeBufferPointer { buf in
|
||||
ptrs.append(buf.baseAddress!)
|
||||
sizes.append(buf.count)
|
||||
return recurse(index + 1, &ptrs, &sizes)
|
||||
}
|
||||
}
|
||||
var ptrs: [UnsafePointer<UInt8>] = []
|
||||
var sizes: [Int] = []
|
||||
ptrs.reserveCapacity(sets.count)
|
||||
sizes.reserveCapacity(sets.count)
|
||||
return recurse(0, &ptrs, &sizes)
|
||||
}
|
||||
|
||||
private func makeSession(_ fmt: CMVideoFormatDescription) {
|
||||
let attrs: [CFString: Any] = [
|
||||
kCVPixelBufferPixelFormatTypeKey:
|
||||
kCVPixelFormatType_32BGRA,
|
||||
]
|
||||
VTDecompressionSessionCreate(
|
||||
allocator: kCFAllocatorDefault, formatDescription: fmt,
|
||||
decoderSpecification: nil,
|
||||
imageBufferAttributes: attrs as CFDictionary,
|
||||
outputCallback: nil, decompressionSessionOut: &session)
|
||||
}
|
||||
|
||||
private static func blockBuffer(_ data: Data) -> CMBlockBuffer? {
|
||||
var block: CMBlockBuffer?
|
||||
guard CMBlockBufferCreateWithMemoryBlock(
|
||||
allocator: kCFAllocatorDefault, memoryBlock: nil,
|
||||
blockLength: data.count,
|
||||
blockAllocator: kCFAllocatorDefault,
|
||||
customBlockSource: nil, offsetToData: 0,
|
||||
dataLength: data.count, flags: 0,
|
||||
blockBufferOut: &block) == noErr, let block else {
|
||||
return nil
|
||||
}
|
||||
var ok = false
|
||||
data.withUnsafeBytes { raw in
|
||||
if let base = raw.baseAddress,
|
||||
CMBlockBufferReplaceDataBytes(
|
||||
with: base, blockBuffer: block,
|
||||
offsetIntoDestination: 0,
|
||||
dataLength: data.count) == noErr { ok = true }
|
||||
}
|
||||
return ok ? block : nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
import XCTest
|
||||
import AVLiveWire
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class BodyFusionTests: XCTestCase {
|
||||
private func skeleton(pelvisZ: Float) -> SkeletonPayload {
|
||||
var p = SkeletonPayload()
|
||||
p.joints[0] = SIMD3(0, 0, pelvisZ)
|
||||
p.valid[0] = true
|
||||
return p
|
||||
}
|
||||
|
||||
func testPelvisDepthOverride() {
|
||||
let mesh = MultiHMRPerson(
|
||||
vertices: [SIMD3<Float>](repeating: .zero, count: 1),
|
||||
translation: SIMD3(0, 0, -1.0), score: 0.9)
|
||||
let fused = BodyFusion.fuse(
|
||||
persons: [mesh], skeletons: [0: skeleton(pelvisZ: -2.5)])
|
||||
XCTAssertEqual(fused[0].translation.z, -2.5, accuracy: 1e-4)
|
||||
}
|
||||
|
||||
func testPassthroughWhenNoSkeleton() {
|
||||
let mesh = MultiHMRPerson(
|
||||
vertices: [SIMD3<Float>](repeating: .zero, count: 1),
|
||||
translation: SIMD3(0, 0, -1.0), score: 0.9)
|
||||
let fused = BodyFusion.fuse(persons: [mesh], skeletons: [:])
|
||||
XCTAssertEqual(fused[0].translation.z, -1.0, accuracy: 1e-4)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
// avlivebody-mac/Tests/AVLiveBodyTests/RenderSettingsTests.swift
|
||||
import XCTest
|
||||
@testable import AVLiveBody
|
||||
|
||||
@MainActor
|
||||
final class RenderSettingsTests: XCTestCase {
|
||||
func testDefaults() {
|
||||
let s = RenderSettings()
|
||||
XCTAssertTrue(s.showSkeleton)
|
||||
XCTAssertTrue(s.showMesh)
|
||||
XCTAssertTrue(s.showVideo)
|
||||
XCTAssertFalse(s.meshMetallic)
|
||||
XCTAssertEqual(s.meshRoughness, 0.6, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.videoOpacity, 1.0, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.keyIntensity, 4000, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.fillIntensity, 1500, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.rimIntensity, 2000, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.fieldOfView, 55, accuracy: 1e-9)
|
||||
XCTAssertFalse(s.showPanel)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
import XCTest
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class SceneUniformsTests: XCTestCase {
|
||||
func testStride() {
|
||||
XCTAssertEqual(MemoryLayout<SceneRenderer.SceneUniforms>.stride, 144)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,22 @@
|
||||
// avlivebody-mac/Tests/AVLiveBodyTests/ShaderCompileTests.swift
|
||||
import XCTest
|
||||
import Metal
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class ShaderCompileTests: XCTestCase {
|
||||
func testSceneMetalCompiles() throws {
|
||||
let here = URL(fileURLWithPath: #filePath) // .../Tests/AVLiveBodyTests/ShaderCompileTests.swift
|
||||
let shader = here
|
||||
.deletingLastPathComponent() // AVLiveBodyTests
|
||||
.deletingLastPathComponent() // Tests
|
||||
.deletingLastPathComponent() // avlivebody-mac
|
||||
.appendingPathComponent("Sources/AVLiveBody/Resources/scene.metal")
|
||||
let src = try String(contentsOf: shader, encoding: .utf8)
|
||||
guard let dev = MTLCreateSystemDefaultDevice() else {
|
||||
throw XCTSkip("no Metal device")
|
||||
}
|
||||
let lib = try dev.makeLibrary(source: src, options: MTLCompileOptions())
|
||||
XCTAssertNotNil(lib.makeFunction(name: "bg_vertex"))
|
||||
XCTAssertNotNil(lib.makeFunction(name: "bg_fragment"))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import XCTest
|
||||
@testable import AVLiveBody
|
||||
|
||||
/// In-memory stand-in for the usbmuxd Unix socket.
|
||||
final class MockMuxTransport: MuxTransport {
|
||||
var sent: [Data] = []
|
||||
var canned: [Data] = []
|
||||
func send(_ data: Data) { sent.append(data) }
|
||||
func receivePacket() -> Data? {
|
||||
canned.isEmpty ? nil : canned.removeFirst()
|
||||
}
|
||||
func close() {}
|
||||
}
|
||||
|
||||
final class USBClientTests: XCTestCase {
|
||||
func testListDevicesParsesDeviceIDs() {
|
||||
let mock = MockMuxTransport()
|
||||
mock.canned = [USBMuxProtocol.encode(plist: [
|
||||
"DeviceList": [
|
||||
["DeviceID": 42,
|
||||
"Properties": ["ConnectionType": "USB"]],
|
||||
]], tag: 0)]
|
||||
let client = USBClient(transport: mock)
|
||||
let devices = client.listDevices()
|
||||
XCTAssertEqual(devices, [42])
|
||||
}
|
||||
|
||||
func testConnectSendsConnectRequest() {
|
||||
let mock = MockMuxTransport()
|
||||
mock.canned = [USBMuxProtocol.encode(
|
||||
plist: ["MessageType": "Result", "Number": 0], tag: 0)]
|
||||
let client = USBClient(transport: mock)
|
||||
let ok = client.connect(deviceID: 42, port: 7000)
|
||||
XCTAssertTrue(ok)
|
||||
let req = USBMuxProtocol.decode(mock.sent.last!)
|
||||
XCTAssertEqual(req?["MessageType"] as? String, "Connect")
|
||||
XCTAssertEqual(req?["DeviceID"] as? Int, 42)
|
||||
XCTAssertEqual(req?["PortNumber"] as? Int,
|
||||
Int((UInt16(7000) << 8) | (UInt16(7000) >> 8)))
|
||||
}
|
||||
|
||||
func testConnectFailsOnNonZeroResult() {
|
||||
let mock = MockMuxTransport()
|
||||
mock.canned = [USBMuxProtocol.encode(
|
||||
plist: ["MessageType": "Result", "Number": 3], tag: 0)]
|
||||
let client = USBClient(transport: mock)
|
||||
XCTAssertFalse(client.connect(deviceID: 1, port: 7000))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
import XCTest
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class USBMuxProtocolTests: XCTestCase {
|
||||
func testEncodeWrapsPlistWith16ByteHeader() {
|
||||
let body: [String: Any] = ["MessageType": "ListDevices"]
|
||||
let packet = USBMuxProtocol.encode(plist: body, tag: 3)
|
||||
XCTAssertGreaterThan(packet.count, 16)
|
||||
XCTAssertEqual(USBMuxProtocol.readLE32(packet, 0).map(Int.init),
|
||||
packet.count)
|
||||
XCTAssertEqual(USBMuxProtocol.readLE32(packet, 4), 1)
|
||||
XCTAssertEqual(USBMuxProtocol.readLE32(packet, 8), 8)
|
||||
XCTAssertEqual(USBMuxProtocol.readLE32(packet, 12), 3)
|
||||
}
|
||||
|
||||
func testDecodeRoundTrip() {
|
||||
let packet = USBMuxProtocol.encode(
|
||||
plist: ["MessageType": "Result", "Number": 0], tag: 1)
|
||||
let decoded = USBMuxProtocol.decode(packet)
|
||||
XCTAssertEqual(decoded?["MessageType"] as? String, "Result")
|
||||
XCTAssertEqual(decoded?["Number"] as? Int, 0)
|
||||
}
|
||||
|
||||
func testDecodeRejectsShortPacket() {
|
||||
XCTAssertNil(USBMuxProtocol.decode(Data([0, 1, 2])))
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
name: AVLiveBody
|
||||
options:
|
||||
bundleIdPrefix: cc.saillant
|
||||
deploymentTarget:
|
||||
macOS: "15.0"
|
||||
createIntermediateGroups: true
|
||||
|
||||
configFiles:
|
||||
Debug: Config/Shared.xcconfig
|
||||
Release: Config/Shared.xcconfig
|
||||
|
||||
packages:
|
||||
AVLiveWire:
|
||||
path: ../shared/AVLiveWire
|
||||
|
||||
targets:
|
||||
AVLiveBody:
|
||||
type: application
|
||||
platform: macOS
|
||||
deploymentTarget: "15.0"
|
||||
sources:
|
||||
- path: Sources/AVLiveBody
|
||||
excludes:
|
||||
- Info.plist
|
||||
- Resources/scene.metal
|
||||
postBuildScripts:
|
||||
- script: |
|
||||
cp "${SRCROOT}/Sources/AVLiveBody/Resources/scene.metal" \
|
||||
"${BUILT_PRODUCTS_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}/scene.metal"
|
||||
name: "Copy scene.metal source to bundle"
|
||||
basedOnDependencyAnalysis: false
|
||||
dependencies:
|
||||
- package: AVLiveWire
|
||||
product: AVLiveWire
|
||||
configFiles:
|
||||
Debug: Config/Shared.xcconfig
|
||||
Release: Config/Shared.xcconfig
|
||||
settings:
|
||||
base:
|
||||
PRODUCT_NAME: AVLiveBody
|
||||
PRODUCT_BUNDLE_IDENTIFIER: cc.saillant.AVLiveBody
|
||||
INFOPLIST_FILE: Sources/AVLiveBody/Info.plist
|
||||
GENERATE_INFOPLIST_FILE: NO
|
||||
CODE_SIGN_STYLE: Manual
|
||||
CODE_SIGN_IDENTITY: "-"
|
||||
CODE_SIGNING_REQUIRED: NO
|
||||
CODE_SIGNING_ALLOWED: NO
|
||||
SWIFT_VERSION: "5.10"
|
||||
ENABLE_HARDENED_RUNTIME: NO
|
||||
AVLiveBodyTests:
|
||||
type: bundle.unit-test
|
||||
platform: macOS
|
||||
sources:
|
||||
- path: Tests/AVLiveBodyTests
|
||||
dependencies:
|
||||
- target: AVLiveBody
|
||||
- package: AVLiveWire
|
||||
product: AVLiveWire
|
||||
settings:
|
||||
base:
|
||||
GENERATE_INFOPLIST_FILE: YES
|
||||
@@ -0,0 +1,37 @@
|
||||
[osc]
|
||||
host = "127.0.0.1"
|
||||
port = 57127
|
||||
|
||||
[feeds.eco2mix]
|
||||
enabled = true
|
||||
interval_sec = 60
|
||||
|
||||
[feeds.velib]
|
||||
enabled = true
|
||||
interval_sec = 120
|
||||
station_codes = []
|
||||
|
||||
[feeds.hubeau]
|
||||
enabled = true
|
||||
interval_sec = 300
|
||||
codes = ["F050001001"]
|
||||
|
||||
[feeds.gbfs]
|
||||
enabled = false
|
||||
interval_sec = 120
|
||||
url = "https://velib-metropole-opendata.smoove.pro/opendata/Velib_Metropole/station_status.json"
|
||||
|
||||
[feeds.ais]
|
||||
enabled = false
|
||||
|
||||
[feeds.carburants]
|
||||
enabled = false
|
||||
|
||||
[feeds.prim]
|
||||
enabled = false
|
||||
|
||||
[feeds.sytadin]
|
||||
enabled = false
|
||||
|
||||
[feeds.teleray]
|
||||
enabled = false
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Registry of available feed classes (auto-discovery on import)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from .base import Feed
|
||||
from .eco2mix import Eco2MixFeed
|
||||
from .gbfs import GBFSFeed
|
||||
from .hubeau import HubeauFeed
|
||||
from .velib import VelibFeed
|
||||
from .ais import AISFeed
|
||||
from .carburants import CarburantsFeed
|
||||
from .prim import PRIMFeed
|
||||
from .sytadin import SytadinFeed
|
||||
from .teleray import TelerayFeed
|
||||
|
||||
REGISTRY: dict[str, type[Feed]] = {
|
||||
"eco2mix": Eco2MixFeed,
|
||||
"gbfs": GBFSFeed,
|
||||
"hubeau": HubeauFeed,
|
||||
"velib": VelibFeed,
|
||||
"ais": AISFeed,
|
||||
"carburants": CarburantsFeed,
|
||||
"prim": PRIMFeed,
|
||||
"sytadin": SytadinFeed,
|
||||
"teleray": TelerayFeed,
|
||||
}
|
||||
|
||||
__all__ = ["Feed", "REGISTRY"]
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
"""AIS vessel positions feed — STUB.
|
||||
|
||||
TODO: needs aisstream.io API key + websocket subscription.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.ais")
|
||||
|
||||
|
||||
class AISFeed(Feed):
|
||||
name = "ais"
|
||||
interval_sec = 60.0
|
||||
|
||||
def fetch(self):
|
||||
return None
|
||||
|
||||
def publish(self, payload) -> None:
|
||||
LOG.info("stub")
|
||||
@@ -0,0 +1,55 @@
|
||||
"""Abstract base class for data feeds."""
|
||||
from __future__ import annotations
|
||||
|
||||
import abc
|
||||
import logging
|
||||
import time
|
||||
import threading
|
||||
from typing import Any
|
||||
|
||||
LOG = logging.getLogger("data_feeds.base")
|
||||
|
||||
|
||||
class Feed(abc.ABC):
|
||||
name: str = "feed"
|
||||
interval_sec: float = 60.0
|
||||
|
||||
def __init__(self, osc_send, **cfg) -> None:
|
||||
self.osc_send = osc_send
|
||||
self.cfg = cfg
|
||||
self._stop = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
self.last_t: float = 0.0
|
||||
|
||||
def configure(self, **kwargs) -> None:
|
||||
self.cfg.update(kwargs)
|
||||
if "interval_sec" in kwargs:
|
||||
self.interval_sec = float(kwargs["interval_sec"])
|
||||
|
||||
@abc.abstractmethod
|
||||
def fetch(self) -> Any: ...
|
||||
|
||||
@abc.abstractmethod
|
||||
def publish(self, payload: Any) -> None: ...
|
||||
|
||||
def tick(self) -> None:
|
||||
try:
|
||||
payload = self.fetch()
|
||||
self.publish(payload)
|
||||
self.last_t = time.time()
|
||||
self.osc_send(f"/data/{self.name}/heartbeat", [self.last_t])
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("%s fetch failed: %s", self.name, e)
|
||||
|
||||
def start(self) -> None:
|
||||
self._thread = threading.Thread(
|
||||
target=self._run, name=f"feed-{self.name}", daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
|
||||
def _run(self) -> None:
|
||||
while not self._stop.is_set():
|
||||
self.tick()
|
||||
self._stop.wait(self.interval_sec)
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Prix carburants feed — STUB.
|
||||
|
||||
TODO: needs prix-carburants.gouv.fr GeoJSON cache + station selection.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.carburants")
|
||||
|
||||
|
||||
class CarburantsFeed(Feed):
|
||||
name = "carburants"
|
||||
interval_sec = 3600.0
|
||||
|
||||
def fetch(self):
|
||||
return None
|
||||
|
||||
def publish(self, payload) -> None:
|
||||
LOG.info("stub")
|
||||
@@ -0,0 +1,58 @@
|
||||
"""RTE eco2mix feed — France electricity production mix in MW.
|
||||
|
||||
Uses the public OpenDataSoft mirror of RTE eco2mix-national-tr (temps reel,
|
||||
15-min resolution). Stdlib HTTP only.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.eco2mix")
|
||||
|
||||
# OpenDataSoft public mirror — no key required.
|
||||
URL = (
|
||||
"https://odre.opendatasoft.com/api/records/1.0/search/"
|
||||
"?dataset=eco2mix-national-tr&rows=1&sort=-date_heure"
|
||||
)
|
||||
|
||||
|
||||
class Eco2MixFeed(Feed):
|
||||
name = "eco2mix"
|
||||
interval_sec = 60.0
|
||||
|
||||
def fetch(self) -> Any:
|
||||
req = urllib.request.Request(URL, headers={"User-Agent": "av-live/0.1"})
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
data = json.loads(r.read().decode("utf-8"))
|
||||
records = data.get("records") or []
|
||||
if not records:
|
||||
return None
|
||||
return records[0].get("fields") or {}
|
||||
|
||||
def publish(self, payload: Any) -> None:
|
||||
if not isinstance(payload, dict):
|
||||
return
|
||||
# Pick a representative subset (MW). Keys per eco2mix-national-tr.
|
||||
keys = [
|
||||
"consommation", "nucleaire", "gaz", "charbon", "fioul",
|
||||
"eolien", "solaire", "hydraulique", "bioenergies",
|
||||
"ech_physiques",
|
||||
]
|
||||
count = 0
|
||||
for k in keys:
|
||||
v = payload.get(k)
|
||||
if v is None:
|
||||
continue
|
||||
try:
|
||||
fv = float(v)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
self.osc_send(f"/data/{self.name}/sample", [k, fv])
|
||||
count += 1
|
||||
self.osc_send(f"/data/{self.name}/count", [count])
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Generic GBFS (General Bikeshare Feed Specification) reader.
|
||||
|
||||
Reads a `station_status.json` URL and publishes aggregate counters.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.gbfs")
|
||||
|
||||
|
||||
class GBFSFeed(Feed):
|
||||
name = "gbfs"
|
||||
interval_sec = 120.0
|
||||
|
||||
def fetch(self) -> Any:
|
||||
url = self.cfg.get("url")
|
||||
if not url:
|
||||
LOG.info("gbfs disabled: no url configured")
|
||||
return None
|
||||
req = urllib.request.Request(url, headers={"User-Agent": "av-live/0.1"})
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
return json.loads(r.read().decode("utf-8"))
|
||||
|
||||
def publish(self, payload: Any) -> None:
|
||||
if not isinstance(payload, dict):
|
||||
return
|
||||
stations = (payload.get("data") or {}).get("stations") or []
|
||||
if not stations:
|
||||
return
|
||||
codes = set(self.cfg.get("station_codes") or [])
|
||||
bikes = 0
|
||||
docks = 0
|
||||
operative = 0
|
||||
sampled = 0
|
||||
for s in stations:
|
||||
sid = str(s.get("station_id", ""))
|
||||
if codes and sid not in codes:
|
||||
continue
|
||||
bikes += int(s.get("num_bikes_available") or 0)
|
||||
docks += int(s.get("num_docks_available") or 0)
|
||||
if s.get("is_renting") or s.get("is_installed"):
|
||||
operative += 1
|
||||
sampled += 1
|
||||
self.osc_send(f"/data/{self.name}/sample", ["bikes_available", float(bikes)])
|
||||
self.osc_send(f"/data/{self.name}/sample", ["docks_available", float(docks)])
|
||||
self.osc_send(f"/data/{self.name}/sample", ["stations_active", float(operative)])
|
||||
self.osc_send(f"/data/{self.name}/count", [sampled])
|
||||
@@ -0,0 +1,66 @@
|
||||
"""Hub'Eau hydrometrie feed — water level and flow rate for French rivers.
|
||||
|
||||
API: https://hubeau.eaufrance.fr/api/v1/hydrometrie/observations_tr
|
||||
Open, no API key required.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from typing import Any
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.hubeau")
|
||||
|
||||
BASE = "https://hubeau.eaufrance.fr/api/v1/hydrometrie/observations_tr"
|
||||
|
||||
|
||||
class HubeauFeed(Feed):
|
||||
name = "hubeau"
|
||||
interval_sec = 300.0
|
||||
|
||||
def fetch(self) -> Any:
|
||||
codes = self.cfg.get("codes") or ["F050001001"]
|
||||
out: dict[str, dict[str, float]] = {}
|
||||
for code in codes:
|
||||
params = {
|
||||
"code_entite": code,
|
||||
"size": 1,
|
||||
"sort": "desc",
|
||||
"fields": "code_station,grandeur_hydro,resultat_obs,date_obs",
|
||||
}
|
||||
url = BASE + "?" + urllib.parse.urlencode(params)
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
url, headers={"User-Agent": "av-live/0.1"})
|
||||
with urllib.request.urlopen(req, timeout=10) as r:
|
||||
data = json.loads(r.read().decode("utf-8"))
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("hubeau %s failed: %s", code, e)
|
||||
continue
|
||||
for obs in data.get("data") or []:
|
||||
station = obs.get("code_station") or code
|
||||
gr = obs.get("grandeur_hydro") or "X"
|
||||
v = obs.get("resultat_obs")
|
||||
if v is None:
|
||||
continue
|
||||
try:
|
||||
fv = float(v)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
out.setdefault(station, {})[gr] = fv
|
||||
return out
|
||||
|
||||
def publish(self, payload: Any) -> None:
|
||||
if not isinstance(payload, dict) or not payload:
|
||||
return
|
||||
count = 0
|
||||
for station, vals in payload.items():
|
||||
for gr, v in vals.items():
|
||||
key = f"{station}_{gr}"
|
||||
self.osc_send(f"/data/{self.name}/sample", [key, float(v)])
|
||||
count += 1
|
||||
self.osc_send(f"/data/{self.name}/count", [count])
|
||||
@@ -0,0 +1,22 @@
|
||||
"""PRIM Ile-de-France Mobilites feed — STUB.
|
||||
|
||||
TODO: needs API key (https://prim.iledefrance-mobilites.fr/).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.prim")
|
||||
|
||||
|
||||
class PRIMFeed(Feed):
|
||||
name = "prim"
|
||||
interval_sec = 60.0
|
||||
|
||||
def fetch(self):
|
||||
return None
|
||||
|
||||
def publish(self, payload) -> None:
|
||||
LOG.info("stub")
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Sytadin Paris traffic feed — STUB.
|
||||
|
||||
TODO: needs sytadin.fr scraping / cumulative km of congestion.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.sytadin")
|
||||
|
||||
|
||||
class SytadinFeed(Feed):
|
||||
name = "sytadin"
|
||||
interval_sec = 300.0
|
||||
|
||||
def fetch(self):
|
||||
return None
|
||||
|
||||
def publish(self, payload) -> None:
|
||||
LOG.info("stub")
|
||||
@@ -0,0 +1,22 @@
|
||||
"""IRSN Teleray radiation feed — STUB.
|
||||
|
||||
TODO: needs https://teleray.irsn.fr/data endpoint research.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from .base import Feed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.teleray")
|
||||
|
||||
|
||||
class TelerayFeed(Feed):
|
||||
name = "teleray"
|
||||
interval_sec = 600.0
|
||||
|
||||
def fetch(self):
|
||||
return None
|
||||
|
||||
def publish(self, payload) -> None:
|
||||
LOG.info("stub")
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Velib Metropole feed — specialization of GBFS against the Paris URL."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
|
||||
from .gbfs import GBFSFeed
|
||||
|
||||
LOG = logging.getLogger("data_feeds.velib")
|
||||
|
||||
VELIB_URL = (
|
||||
"https://velib-metropole-opendata.smoove.pro/opendata/"
|
||||
"Velib_Metropole/station_status.json"
|
||||
)
|
||||
|
||||
|
||||
class VelibFeed(GBFSFeed):
|
||||
name = "velib"
|
||||
interval_sec = 120.0
|
||||
|
||||
def configure(self, **kwargs) -> None:
|
||||
# Force the URL if caller did not provide one.
|
||||
kwargs.setdefault("url", VELIB_URL)
|
||||
super().configure(**kwargs)
|
||||
if not self.cfg.get("url"):
|
||||
self.cfg["url"] = VELIB_URL
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Run all enabled feeds, publish OSC to AVLiveBody."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import sys
|
||||
import time
|
||||
import tomllib
|
||||
from pathlib import Path
|
||||
|
||||
from .feeds import REGISTRY
|
||||
from .osc_sender import OscSender
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--config", default="data_feeds/config.avlivedata.toml")
|
||||
p.add_argument("--osc-host")
|
||||
p.add_argument("--osc-port", type=int)
|
||||
p.add_argument("-v", "--verbose", action="store_true")
|
||||
args = p.parse_args(argv)
|
||||
logging.basicConfig(
|
||||
level=logging.INFO if args.verbose else logging.WARNING,
|
||||
format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
cfg = tomllib.loads(Path(args.config).read_text())
|
||||
osc_cfg = cfg.get("osc", {})
|
||||
host = args.osc_host or osc_cfg.get("host", "127.0.0.1")
|
||||
port = args.osc_port or osc_cfg.get("port", 57127)
|
||||
sender = OscSender(host, port)
|
||||
feeds = []
|
||||
for name, kwargs in (cfg.get("feeds") or {}).items():
|
||||
if not kwargs.get("enabled", False):
|
||||
continue
|
||||
cls = REGISTRY.get(name)
|
||||
if cls is None:
|
||||
logging.warning("Unknown feed: %s", name)
|
||||
continue
|
||||
f = cls(sender.send)
|
||||
f.configure(**kwargs)
|
||||
f.start()
|
||||
feeds.append(f)
|
||||
logging.info("started feed %s (interval %.0fs)", name, f.interval_sec)
|
||||
if not feeds:
|
||||
logging.warning("No feeds enabled. Exiting.")
|
||||
return 1
|
||||
try:
|
||||
while True:
|
||||
time.sleep(60)
|
||||
except KeyboardInterrupt:
|
||||
return 0
|
||||
finally:
|
||||
for f in feeds:
|
||||
f.stop()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Wrapper around python-osc SimpleUDPClient with per-route helpers."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from pythonosc.udp_client import SimpleUDPClient
|
||||
|
||||
LOG = logging.getLogger("data_feeds.osc")
|
||||
|
||||
|
||||
class OscSender:
|
||||
def __init__(self, host: str, port: int) -> None:
|
||||
self.host = host
|
||||
self.port = port
|
||||
self._client = SimpleUDPClient(host, port)
|
||||
|
||||
def send(self, addr: str, args: list[Any]) -> None:
|
||||
try:
|
||||
self._client.send_message(addr, args)
|
||||
except OSError as e:
|
||||
LOG.warning("send %s failed: %s", addr, e)
|
||||
@@ -20,6 +20,7 @@ Python **3.11+** requis. `pyproject.toml` est la source de vérité — ne jamai
|
||||
| Backend | Fichier | Statut |
|
||||
|---------|---------|--------|
|
||||
| MediaPipe Holistic | `holistic.py` | stable |
|
||||
| MediaPipe multi (Pose+Face+Hand) | `multi.py` | stable ; `MEDIAPIPE_DELEGATE=gpu` (défaut) ou `cpu`. **GPU Metal exige SRGBA 4-ch** (3-ch SRGB crashe `gpu_buffer_storage_cv_pixel_buffer.cc`) — multi.py route auto vers `cv2.COLOR_BGR2RGBA` + `mp.ImageFormat.SRGBA` quand delegate=GPU. Bench M5 image-mode SRGBA : pose 2.9 vs 6.7 ms (GPU/CPU), face 1.0 vs 4.1, hand 3.2 vs 6.1 |
|
||||
| Ultralytics YOLOv8-pose | `pose.py` | stable, modèle `yolov8n-pose.pt` à la racine repo |
|
||||
| Apple Vision (Core ML) | `apple_vision_pose.py`, `coreml_pose.py` | macOS uniquement |
|
||||
| DETRPose | `detrpose.py` | clone manuel + checkpoint, voir docstring |
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
# data_only_viz
|
||||
|
||||
Python + Metal visualiser for the AV-Live data-only patch.
|
||||
|
||||
## Hand features OSC
|
||||
|
||||
`/pose/hands [pid=0, lx, ly, lopen, lspeed, rx, ry, ropen, rspeed, dist]`
|
||||
|
||||
- Emitted once per pipeline tick by `pose_bridge.send_hands` to sclang `:57121`.
|
||||
- L = leftmost hand on screen, R = rightmost; absent hand -> its 4 floats are 0.
|
||||
- Ranges: positions/openness/dist in [0,1]; speed >= 0 (normalized units/frame).
|
||||
- Consumed by SuperCollider `~handFeat` (data_only) and the Metal renderer uniforms.
|
||||
- Multi-person is not yet carried (pid is always 0).
|
||||
@@ -98,6 +98,9 @@ class ActionHeadPublisher(threading.Thread):
|
||||
self._last_smplx_t = 0.0
|
||||
self._last_body_t = 0.0
|
||||
self._last_pids: set[int] = set()
|
||||
from data_only_viz.hand_features import HandFeatureExtractor
|
||||
self._hand_ext = HandFeatureExtractor()
|
||||
self._init_finger_piano()
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
@@ -142,6 +145,89 @@ class ActionHeadPublisher(threading.Thread):
|
||||
self.head.forget(gone)
|
||||
self.bridge.send_leave(pid=gone)
|
||||
self._last_pids = current_pids
|
||||
self._emit_hands(t_now)
|
||||
self._emit_fingers(t_now)
|
||||
self._emit_pinch(t_now)
|
||||
|
||||
def _emit_hands(self, t_now: float) -> None:
|
||||
"""Lock state, extract hand features, emit /pose/hands once per tick."""
|
||||
with self.state.lock():
|
||||
hands = list(getattr(self.state, "persons_hands", None) or [])
|
||||
if not hands and getattr(self.state, "hands_present", False):
|
||||
lkp = getattr(self.state, "left_hand_kp", None)
|
||||
rkp = getattr(self.state, "right_hand_kp", None)
|
||||
hands = [h for h in (lkp, rkp) if h is not None and len(h) > 0]
|
||||
feats = self._hand_ext.step(hands)
|
||||
with self.state.lock():
|
||||
self.state.hand_feats = feats
|
||||
self.bridge.send_hands(feats, t_now)
|
||||
|
||||
def _init_finger_piano(self) -> None:
|
||||
"""Read FINGER_* env config and build the strike + pinch detectors."""
|
||||
import os
|
||||
from data_only_viz.finger_strike import (
|
||||
FingerStrikeDetector, PinchDetector,
|
||||
)
|
||||
self._finger_enabled = os.environ.get("FINGER_PIANO", "0") not in (
|
||||
"0", "", "false", "False",
|
||||
)
|
||||
vel = float(os.environ.get("FINGER_STRIKE_VEL", "0.02"))
|
||||
refr = float(os.environ.get("FINGER_STRIKE_REFRACTORY_MS", "120"))
|
||||
self._finger_det = FingerStrikeDetector(
|
||||
vel_thresh=vel, refractory_ms=refr,
|
||||
)
|
||||
self._pinch_det = PinchDetector(
|
||||
ratio_on=float(os.environ.get("PINCH_RATIO_ON", "0.45")),
|
||||
ratio_off=float(os.environ.get("PINCH_RATIO_OFF", "0.65")),
|
||||
refractory_ms=float(os.environ.get("PINCH_REFRACTORY_MS", "250")),
|
||||
)
|
||||
self._finger_dbg = os.environ.get("FINGER_DEBUG", "0") not in (
|
||||
"0", "", "false", "False",
|
||||
)
|
||||
# auto = iPhone Vision hands if fresh, else MediaPipe. Or force
|
||||
# "iphone" / "mediapipe".
|
||||
self._finger_source = os.environ.get("FINGER_SOURCE", "auto").lower()
|
||||
self._fdbg_n = 0
|
||||
|
||||
def _pick_hands(self) -> tuple[list, str, bool]:
|
||||
"""Return (hands, source_label, ip_fresh) per FINGER_SOURCE policy."""
|
||||
with self.state.lock():
|
||||
mp_hands = list(getattr(self.state, "persons_hands", None) or [])
|
||||
ip_hands = list(getattr(self.state, "persons_hands_iphone", None) or [])
|
||||
ip_t = getattr(self.state, "persons_hands_iphone_t", 0.0)
|
||||
ip_fresh = bool(ip_hands) and (time.perf_counter() - ip_t) < 0.5
|
||||
src = getattr(self, "_finger_source", "auto")
|
||||
if src == "iphone":
|
||||
return ip_hands, "iphone", ip_fresh
|
||||
if src == "mediapipe":
|
||||
return mp_hands, "mediapipe", ip_fresh
|
||||
return (ip_hands, "iphone", ip_fresh) if ip_fresh else (
|
||||
mp_hands, "mediapipe", ip_fresh)
|
||||
|
||||
def _emit_fingers(self, t_now: float) -> None:
|
||||
"""Detect finger strikes (air-piano) and emit /pose/finger."""
|
||||
if not getattr(self, "_finger_enabled", False):
|
||||
return
|
||||
hands, used, ip_fresh = self._pick_hands()
|
||||
events = self._finger_det.step(hands, t_now)
|
||||
if getattr(self, "_finger_dbg", False):
|
||||
self._fdbg_n += 1
|
||||
if events or self._fdbg_n % 30 == 0:
|
||||
LOG.info("fingers dbg: src=%s hands=%d strikes=%d (ip_fresh=%s)",
|
||||
used, len(hands), len(events), ip_fresh)
|
||||
for ev in events:
|
||||
self.bridge.send_finger(ev)
|
||||
|
||||
def _emit_pinch(self, t_now: float) -> None:
|
||||
"""Detect thumb-to-finger pinches (clip toggles) and emit /pose/pinch."""
|
||||
if not getattr(self, "_finger_enabled", False):
|
||||
return
|
||||
hands, used, _ip = self._pick_hands()
|
||||
for ev in self._pinch_det.step(hands, t_now):
|
||||
if getattr(self, "_finger_dbg", False):
|
||||
LOG.info("pinch dbg: src=%s hand=%d finger=%d",
|
||||
used, ev.hand, ev.finger)
|
||||
self.bridge.send_pinch(ev)
|
||||
|
||||
def _read_sources(self) -> tuple[
|
||||
list[tuple[int, np.ndarray, np.ndarray, float, np.ndarray]] | None,
|
||||
@@ -292,22 +378,26 @@ class ActionHeadPublisher(threading.Thread):
|
||||
|
||||
@staticmethod
|
||||
def _kp_list_to_array(body: Any) -> np.ndarray | None:
|
||||
"""Best-effort conversion of a body keypoint list to (N, 3) array."""
|
||||
"""Best-effort conversion of a body keypoint list to (N, 3) array.
|
||||
|
||||
Handles attribute objects (the Kp3D dataclass exposes .x/.y/.z) and
|
||||
indexable rows. Kp3D is NOT subscriptable, so the index fallback must
|
||||
not be evaluated eagerly (the old `getattr(kp, "x", kp[0])` raised
|
||||
TypeError on Kp3D and silently returned None — breaking the body3d /
|
||||
MediaPipe-only path that feeds action-head when no SMPL-X is present)."""
|
||||
if body is None:
|
||||
return None
|
||||
if isinstance(body, np.ndarray):
|
||||
return body
|
||||
try:
|
||||
return np.asarray(
|
||||
[
|
||||
(
|
||||
getattr(kp, "x", kp[0]),
|
||||
getattr(kp, "y", kp[1]),
|
||||
getattr(kp, "z", kp[2] if len(kp) > 2 else 0.0),
|
||||
)
|
||||
for kp in body
|
||||
],
|
||||
dtype=np.float32,
|
||||
)
|
||||
except (TypeError, IndexError, AttributeError):
|
||||
rows = []
|
||||
for kp in body:
|
||||
if hasattr(kp, "x"):
|
||||
rows.append((kp.x, kp.y, getattr(kp, "z", 0.0)))
|
||||
else:
|
||||
try:
|
||||
rows.append((kp[0], kp[1], kp[2] if len(kp) > 2 else 0.0))
|
||||
except (TypeError, IndexError):
|
||||
return None
|
||||
if not rows:
|
||||
return None
|
||||
return np.asarray(rows, dtype=np.float32)
|
||||
|
||||
@@ -334,11 +334,16 @@ def _load_py_config(path: str):
|
||||
from omegaconf import OmegaConf
|
||||
# Les configs DETRPose sont des fichiers Python qui exposent un dict
|
||||
# `model = LazyCall(...)`. On utilise le helper lazy_config si dispo.
|
||||
try:
|
||||
from src.misc.lazy_config import LazyConfig # type: ignore
|
||||
return LazyConfig.load(path)
|
||||
except ImportError:
|
||||
pass
|
||||
# DETRPose exposes LazyConfig from src.core (see tools/inference/torch_inf.py);
|
||||
# the older src.misc.lazy_config path is kept as a fallback. LazyConfig.load
|
||||
# handles the config's relative imports (`from .include...`) that a raw exec
|
||||
# cannot.
|
||||
for _mod in ("src.core", "src.misc.lazy_config"):
|
||||
try:
|
||||
mod = __import__(_mod, fromlist=["LazyConfig"])
|
||||
return mod.LazyConfig.load(path)
|
||||
except (ImportError, AttributeError):
|
||||
continue
|
||||
# Fallback minimal : exec + recup des noms cles.
|
||||
ns: dict = {}
|
||||
with open(path) as f:
|
||||
|
||||
@@ -0,0 +1,188 @@
|
||||
"""Air-piano finger strike detection from raw MediaPipe hand joints.
|
||||
|
||||
Mirrors hand_features.py conventions (L/R slotting by cx, finite-guarded
|
||||
coordinate extraction). A "strike" is a fast downward motion of a fingertip
|
||||
RELATIVE to its base knuckle, so translating the whole hand does not fire all
|
||||
fingers. Output feeds the OSC /pose/finger route consumed by SuperCollider.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import math
|
||||
from dataclasses import dataclass
|
||||
|
||||
from data_only_viz.hand_features import _clamp, _coord, _finite
|
||||
|
||||
# MediaPipe 21-kp hand: fingertip and base-knuckle indices per finger
|
||||
# (thumb, index, middle, ring, pinky). Thumb base = ThumbMP (2).
|
||||
FINGERTIPS: tuple[int, ...] = (4, 8, 12, 16, 20)
|
||||
FINGER_BASES: tuple[int, ...] = (2, 5, 9, 13, 17)
|
||||
|
||||
# Pinch: thumb tip vs the 4 opposable finger tips (index..pinky).
|
||||
THUMB_TIP: int = 4
|
||||
WRIST: int = 0
|
||||
MIDDLE_MCP: int = 9
|
||||
PINCH_TIPS: tuple[int, ...] = (8, 12, 16, 20)
|
||||
|
||||
|
||||
def order_hands_by_cx(hands: list, max_slots: int) -> list:
|
||||
"""Return up to max_slots hand-landmark lists ordered leftmost-first
|
||||
(slot 0 = L), matching HandFeatureExtractor / FingerStrikeDetector."""
|
||||
cand = []
|
||||
for lm in hands:
|
||||
try:
|
||||
if lm is None or len(lm) < 21:
|
||||
continue
|
||||
except TypeError:
|
||||
continue
|
||||
xs = [_finite(_coord(p, "x", 0), 0.5) for p in lm[:21]]
|
||||
cx = _clamp(sum(xs) / len(xs), 0.0, 1.0)
|
||||
cand.append((cx, lm))
|
||||
cand.sort(key=lambda c: c[0])
|
||||
return [lm for _cx, lm in cand[:max_slots]]
|
||||
|
||||
|
||||
@dataclass
|
||||
class StrikeEvent:
|
||||
hand: int # 0 = left slot (leftmost cx), 1 = right slot
|
||||
finger: int # 0..4 = thumb, index, middle, ring, pinky
|
||||
strike_speed: float
|
||||
z: float
|
||||
tipx: float
|
||||
tipy: float
|
||||
|
||||
|
||||
class _FingerState:
|
||||
__slots__ = ("prev_rel", "armed", "last_t")
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.prev_rel: float | None = None
|
||||
self.armed: bool = True
|
||||
self.last_t: float = -1e9
|
||||
|
||||
|
||||
class FingerStrikeDetector:
|
||||
def __init__(self, vel_thresh: float = 0.02, refractory_ms: float = 120.0,
|
||||
speed_scale: float = 0.10, history_slots: int = 2) -> None:
|
||||
self.vel_thresh = vel_thresh
|
||||
self.refractory_s = refractory_ms / 1000.0
|
||||
self.speed_scale = max(1e-6, speed_scale)
|
||||
# state[slot][finger]
|
||||
self._state = [[_FingerState() for _ in range(5)]
|
||||
for _ in range(history_slots)]
|
||||
|
||||
def _slot_hands(self, hands: list) -> list:
|
||||
"""Validate and order hands leftmost-first (slot 0 = L), like
|
||||
HandFeatureExtractor. Returns up to 2 (cx, landmarks) ordered."""
|
||||
cand = []
|
||||
for lm in hands:
|
||||
try:
|
||||
if lm is None or len(lm) < 21:
|
||||
continue
|
||||
except TypeError:
|
||||
continue
|
||||
xs = [_finite(_coord(p, "x", 0), 0.5) for p in lm[:21]]
|
||||
cx = _clamp(sum(xs) / len(xs), 0.0, 1.0)
|
||||
cand.append((cx, lm))
|
||||
cand.sort(key=lambda c: c[0])
|
||||
return cand[:len(self._state)]
|
||||
|
||||
def step(self, hands: list, t_now: float) -> list[StrikeEvent]:
|
||||
ordered = self._slot_hands(hands)
|
||||
present = set(range(len(ordered)))
|
||||
events: list[StrikeEvent] = []
|
||||
for slot, (_cx, lm) in enumerate(ordered):
|
||||
for f in range(5):
|
||||
tip = lm[FINGERTIPS[f]]
|
||||
base = lm[FINGER_BASES[f]]
|
||||
tip_y = _finite(_coord(tip, "y", 1), 0.5)
|
||||
base_y = _finite(_coord(base, "y", 1), 0.5)
|
||||
rel = tip_y - base_y # +down (image y grows downward)
|
||||
st = self._state[slot][f]
|
||||
if st.prev_rel is None:
|
||||
st.prev_rel = rel
|
||||
continue
|
||||
vel = rel - st.prev_rel # +down velocity per frame
|
||||
st.prev_rel = rel
|
||||
if vel < 0.0: # lifting -> rearm
|
||||
st.armed = True
|
||||
if (vel > self.vel_thresh and st.armed
|
||||
and (t_now - st.last_t) >= self.refractory_s):
|
||||
st.armed = False
|
||||
st.last_t = t_now
|
||||
events.append(StrikeEvent(
|
||||
hand=slot, finger=f,
|
||||
strike_speed=_clamp(vel / self.speed_scale, 0.0, 1.0),
|
||||
z=_finite(_coord(tip, "z", 2, 0.0), 0.0),
|
||||
tipx=_finite(_coord(tip, "x", 0), 0.5),
|
||||
tipy=tip_y,
|
||||
))
|
||||
# reset slots not present this tick so re-entry does not spike
|
||||
for slot in range(len(self._state)):
|
||||
if slot not in present:
|
||||
for f in range(5):
|
||||
self._state[slot][f].prev_rel = None
|
||||
self._state[slot][f].armed = True
|
||||
return events
|
||||
|
||||
|
||||
@dataclass
|
||||
class PinchEvent:
|
||||
hand: int # 0 = left slot (leftmost cx), 1 = right slot
|
||||
finger: int # 1..4 = index, middle, ring, pinky (thumb is trigger)
|
||||
|
||||
|
||||
class _PinchState:
|
||||
__slots__ = ("engaged", "last_t")
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.engaged: bool = False
|
||||
self.last_t: float = -1e9
|
||||
|
||||
|
||||
class PinchDetector:
|
||||
"""Edge-triggered thumb-to-finger pinch with hysteresis.
|
||||
|
||||
Fires one PinchEvent when thumb tip contacts a finger tip (distance,
|
||||
normalized by hand size, drops below ratio_on). Re-arms only after the
|
||||
distance rises back above ratio_off, so one pinch = one event.
|
||||
"""
|
||||
|
||||
def __init__(self, ratio_on: float = 0.45, ratio_off: float = 0.65,
|
||||
refractory_ms: float = 250.0, history_slots: int = 2) -> None:
|
||||
self.ratio_on = ratio_on
|
||||
self.ratio_off = ratio_off
|
||||
self.refractory_s = refractory_ms / 1000.0
|
||||
self._state = [[_PinchState() for _ in range(4)]
|
||||
for _ in range(history_slots)]
|
||||
|
||||
def step(self, hands: list, t_now: float) -> list[PinchEvent]:
|
||||
ordered = order_hands_by_cx(hands, len(self._state))
|
||||
present = set(range(len(ordered)))
|
||||
events: list[PinchEvent] = []
|
||||
for slot, lm in enumerate(ordered):
|
||||
tx = _finite(_coord(lm[THUMB_TIP], "x", 0), 0.5)
|
||||
ty = _finite(_coord(lm[THUMB_TIP], "y", 1), 0.5)
|
||||
wx = _finite(_coord(lm[WRIST], "x", 0), 0.5)
|
||||
wy = _finite(_coord(lm[WRIST], "y", 1), 0.5)
|
||||
mx = _finite(_coord(lm[MIDDLE_MCP], "x", 0), 0.5)
|
||||
my = _finite(_coord(lm[MIDDLE_MCP], "y", 1), 0.5)
|
||||
size = math.hypot(mx - wx, my - wy)
|
||||
size = size if size > 1e-4 else 1e-4
|
||||
for i, tip_idx in enumerate(PINCH_TIPS):
|
||||
fx = _finite(_coord(lm[tip_idx], "x", 0), 0.5)
|
||||
fy = _finite(_coord(lm[tip_idx], "y", 1), 0.5)
|
||||
ratio = math.hypot(fx - tx, fy - ty) / size
|
||||
st = self._state[slot][i]
|
||||
if st.engaged:
|
||||
if ratio > self.ratio_off:
|
||||
st.engaged = False
|
||||
elif ratio < self.ratio_on and (
|
||||
t_now - st.last_t) >= self.refractory_s:
|
||||
st.engaged = True
|
||||
st.last_t = t_now
|
||||
events.append(PinchEvent(hand=slot, finger=i + 1))
|
||||
for slot in range(len(self._state)):
|
||||
if slot not in present:
|
||||
for i in range(4):
|
||||
self._state[slot][i].engaged = False
|
||||
return events
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Turn MediaPipe hand landmarks into a compact expressive feature vector.
|
||||
|
||||
Mirrors the kinematics pattern of action_head.FeatureExtractor: a small ring
|
||||
buffer per hand slot, finite-guarded, clamped to declared ranges. Output feeds
|
||||
both the OSC /pose/hands route (audio) and the Metal renderer uniforms (visual).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import math
|
||||
from collections import deque
|
||||
|
||||
WRIST, THUMB_TIP, MIDDLE_MCP, PINKY_TIP = 0, 4, 9, 20
|
||||
|
||||
NEUTRAL_HAND = {"cx": 0.5, "cy": 0.5, "openness": 0.0, "speed": 0.0}
|
||||
|
||||
# openness calibration: (span/size) maps fist~0.3 -> 0, open~2.0 -> 1
|
||||
_OPEN_LO, _OPEN_HI = 0.5, 2.0
|
||||
|
||||
|
||||
def _finite(v: float, fallback: float) -> float:
|
||||
return v if isinstance(v, float) and math.isfinite(v) else fallback
|
||||
|
||||
|
||||
def _coord(p, attr: str, idx: int, fallback: float = 0.5) -> float:
|
||||
"""Extract a coordinate from a landmark object or an indexable row."""
|
||||
if hasattr(p, attr):
|
||||
return float(getattr(p, attr))
|
||||
try:
|
||||
return float(p[idx])
|
||||
except (TypeError, IndexError):
|
||||
return fallback
|
||||
|
||||
|
||||
def _clamp(v: float, lo: float, hi: float) -> float:
|
||||
return lo if v < lo else hi if v > hi else v
|
||||
|
||||
|
||||
class HandFeatureExtractor:
|
||||
def __init__(self, history: int = 5):
|
||||
self._history = max(2, history)
|
||||
# two screen slots: 0 = leftmost (L), 1 = rightmost (R)
|
||||
self._buf = [deque(maxlen=self._history), deque(maxlen=self._history)]
|
||||
|
||||
def _features(self, lm: list) -> dict:
|
||||
xs = [_finite(_coord(p, "x", 0), 0.5) for p in lm[:21]]
|
||||
ys = [_finite(_coord(p, "y", 1), 0.5) for p in lm[:21]]
|
||||
cx = _clamp(sum(xs) / len(xs), 0.0, 1.0)
|
||||
cy = _clamp(sum(ys) / len(ys), 0.0, 1.0)
|
||||
size = math.hypot(xs[MIDDLE_MCP] - xs[WRIST], ys[MIDDLE_MCP] - ys[WRIST])
|
||||
size = size if size > 1e-4 else 1e-4
|
||||
span = math.hypot(xs[THUMB_TIP] - xs[PINKY_TIP],
|
||||
ys[THUMB_TIP] - ys[PINKY_TIP])
|
||||
openness = _clamp((span / size - _OPEN_LO) / (_OPEN_HI - _OPEN_LO),
|
||||
0.0, 1.0)
|
||||
return {"cx": cx, "cy": cy, "openness": openness}
|
||||
|
||||
def _speed(self, slot: int, cx: float, cy: float) -> float:
|
||||
buf = self._buf[slot]
|
||||
spd = 0.0
|
||||
if buf:
|
||||
px, py = buf[-1]
|
||||
spd = math.hypot(cx - px, cy - py)
|
||||
buf.append((cx, cy))
|
||||
return _clamp(spd, 0.0, 1.0)
|
||||
|
||||
def step(self, hands: list) -> dict:
|
||||
feats = []
|
||||
for lm in hands:
|
||||
try:
|
||||
valid = lm is not None and len(lm) >= 21
|
||||
except TypeError:
|
||||
continue
|
||||
if valid:
|
||||
feats.append(self._features(lm))
|
||||
if not feats:
|
||||
self._buf[0].clear()
|
||||
self._buf[1].clear()
|
||||
return {"L": None, "R": None, "dist": 0.0}
|
||||
feats.sort(key=lambda f: f["cx"]) # leftmost first
|
||||
left = feats[0]
|
||||
right = feats[-1] if len(feats) > 1 else None
|
||||
left["speed"] = self._speed(0, left["cx"], left["cy"])
|
||||
out_l = left
|
||||
out_r = None
|
||||
dist = 0.0
|
||||
if right is not None:
|
||||
right["speed"] = self._speed(1, right["cx"], right["cy"])
|
||||
out_r = right
|
||||
dist = _clamp(math.hypot(right["cx"] - left["cx"],
|
||||
right["cy"] - left["cy"]), 0.0, 1.0)
|
||||
else:
|
||||
self._buf[1].clear()
|
||||
return {"L": out_l, "R": out_r, "dist": dist}
|
||||
@@ -0,0 +1,205 @@
|
||||
"""IphoneUSBSource — a cv2-VideoCapture-shaped frame source backed by the
|
||||
iPhone ARBodyTracker USB stream. Decodes the AVLiveWire HEVC video to BGR
|
||||
frames for MediaPipe, and (Task 2) writes ARKit skeleton + Vision hands into
|
||||
State. Substitutes for cv2.VideoCapture in multi.py under --iphone-usb."""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import socket
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
|
||||
try:
|
||||
import av
|
||||
except ImportError: # pragma: no cover — av missing is caught at runtime by caller
|
||||
av = None # type: ignore[assignment]
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
from data_only_viz.scripts.iphone_usb_bridge import (
|
||||
connect_device, decode_skeleton, iter_frames, TAG_SKELETON,
|
||||
)
|
||||
from data_only_viz.state import PoseKp
|
||||
|
||||
TAG_VIDEO = 2
|
||||
TAG_HANDS = 4
|
||||
LOG = logging.getLogger("iphone_usb_source")
|
||||
|
||||
_HAND_BYTES = 1 + 21 * 12 # chirality:u8 + 21 × (x,y,z) × 4 bytes BE f32
|
||||
_ARKIT_JOINTS = 91
|
||||
|
||||
|
||||
def _make_point(x: float, y: float, z: float) -> PoseKp:
|
||||
"""Wrap raw floats from the wire into a PoseKp landmark."""
|
||||
return PoseKp(x=x, y=y, z=z, c=1.0)
|
||||
|
||||
|
||||
def _decode_hands(payload: bytes):
|
||||
"""Decode TAG_HANDS payload → list[list[PoseKp]] (may be empty) or None.
|
||||
|
||||
Wire layout: count:u8, then per hand: chirality:u8 (1=right) followed by
|
||||
21 × (x, y, z) big-endian f32. x,y are normalized image coords; z≈confidence.
|
||||
Returns None only on empty payload; an empty list when count==0.
|
||||
"""
|
||||
if not payload:
|
||||
return None
|
||||
n = payload[0]
|
||||
if n == 0:
|
||||
return []
|
||||
off = 1
|
||||
hands: list[list[PoseKp]] = []
|
||||
for _ in range(n):
|
||||
if off + _HAND_BYTES > len(payload):
|
||||
break
|
||||
off += 1 # skip chirality byte (not used by HandFeatureExtractor)
|
||||
pts = struct.unpack(">" + "f" * 63, payload[off:off + 21 * 12])
|
||||
off += 21 * 12
|
||||
hand = [_make_point(pts[i * 3], pts[i * 3 + 1], pts[i * 3 + 2])
|
||||
for i in range(21)]
|
||||
hands.append(hand)
|
||||
return hands
|
||||
|
||||
|
||||
def _to_annexb(data: bytes) -> bytes:
|
||||
out = bytearray(); i = 0
|
||||
while i + 4 <= len(data):
|
||||
n = int.from_bytes(data[i:i + 4], "big"); i += 4
|
||||
if i + n > len(data):
|
||||
break
|
||||
out += b"\x00\x00\x00\x01" + data[i:i + n]; i += n
|
||||
return bytes(out)
|
||||
|
||||
|
||||
class IphoneUSBSource:
|
||||
def __init__(self, state=None, target_size=(640, 480),
|
||||
write_hands: bool = True, mirror: bool = True) -> None:
|
||||
self.state = state
|
||||
self.target_w, self.target_h = target_size
|
||||
self._write_hands = write_hands
|
||||
# Mirror the video horizontally so the performer facing the camera
|
||||
# interacts naturally (move right -> goes right; raise right arm ->
|
||||
# the right side responds). Env CONCERT_MIRROR=0 disables it.
|
||||
import os as _os
|
||||
self._mirror = mirror and (_os.environ.get("CONCERT_MIRROR", "1") != "0")
|
||||
self._codec = av.codec.CodecContext.create("hevc", "r") if av is not None else None
|
||||
self._lock = threading.Lock()
|
||||
self._frame = None # latest BGR np.ndarray
|
||||
self._stop = threading.Event()
|
||||
self._thread = None
|
||||
self._sock = None # stored for shutdown() in release()
|
||||
self._opened = False
|
||||
|
||||
def start(self) -> bool:
|
||||
sock = connect_device()
|
||||
if sock is None:
|
||||
LOG.error("iphone usb: no device / connect failed")
|
||||
return False
|
||||
self._sock = sock
|
||||
self._opened = True
|
||||
self._thread = threading.Thread(
|
||||
target=self._run, args=(sock,), name="iphone_usb_src", daemon=True)
|
||||
self._thread.start()
|
||||
return True
|
||||
|
||||
def isOpened(self) -> bool:
|
||||
return self._opened
|
||||
|
||||
def _run(self, sock) -> None:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
for tag, pid, payload in iter_frames(sock):
|
||||
if self._stop.is_set():
|
||||
break
|
||||
if tag == TAG_VIDEO and len(payload) > 1:
|
||||
annexb = _to_annexb(payload[1:])
|
||||
if self._codec is not None and av is not None:
|
||||
try:
|
||||
for fr in self._codec.decode(av.Packet(annexb)):
|
||||
img = fr.to_ndarray(format="bgr24")
|
||||
img = cv2.resize(img, (self.target_w, self.target_h))
|
||||
if self._mirror:
|
||||
img = cv2.flip(img, 1)
|
||||
with self._lock:
|
||||
self._frame = img
|
||||
except Exception as e: # av.AVError is a removal-prone alias
|
||||
LOG.debug("hevc decode: %s", e)
|
||||
elif tag == TAG_SKELETON and self.state is not None:
|
||||
joints = decode_skeleton(payload)
|
||||
if joints is not None:
|
||||
with self.state.lock():
|
||||
arr = self.state.persons_arkit_joints.get(pid)
|
||||
if arr is None or arr.shape != (_ARKIT_JOINTS, 3):
|
||||
arr = np.zeros((_ARKIT_JOINTS, 3), dtype=np.float32)
|
||||
for i, (x, y, z, valid) in enumerate(joints):
|
||||
if valid:
|
||||
arr[i] = (x, y, z)
|
||||
self.state.persons_arkit_joints[pid] = arr
|
||||
self.state.persons_arkit_last_t[pid] = time.perf_counter()
|
||||
elif tag == TAG_HANDS:
|
||||
hands = _decode_hands(payload)
|
||||
if hands is not None and self.state is not None:
|
||||
with self.state.lock():
|
||||
# Always expose iPhone Vision hands (stable,
|
||||
# rotation-invariant) for the air-piano.
|
||||
self.state.persons_hands_iphone = hands
|
||||
self.state.persons_hands_iphone_t = \
|
||||
time.perf_counter()
|
||||
if self._write_hands:
|
||||
self.state.persons_hands = hands
|
||||
except OSError as e:
|
||||
LOG.warning("iphone usb stream error: %s", e)
|
||||
finally:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
# Stream ended. Drop the stale frame so read() reports no-frame,
|
||||
# then reconnect (unless stopping) so a transient USB/app drop
|
||||
# doesn't freeze the pose pipeline (the iPhone backgrounding or a
|
||||
# USB hiccup ends iter_frames; we re-establish :7000).
|
||||
with self._lock:
|
||||
self._frame = None
|
||||
self._opened = False
|
||||
if self._stop.is_set():
|
||||
break
|
||||
LOG.info("iphone usb stream ended — reconnecting...")
|
||||
sock = self._reconnect()
|
||||
if sock is None:
|
||||
break
|
||||
self._sock = sock
|
||||
self._opened = True
|
||||
LOG.info("iphone usb reconnected")
|
||||
self._opened = False
|
||||
|
||||
def _reconnect(self):
|
||||
"""Retry connect_device() with capped backoff until it succeeds or
|
||||
release() is called. Returns the new socket, or None if stopping."""
|
||||
delay = 0.5
|
||||
while not self._stop.is_set():
|
||||
sock = connect_device()
|
||||
if sock is not None:
|
||||
return sock
|
||||
self._stop.wait(delay)
|
||||
delay = min(delay * 2, 3.0)
|
||||
return None
|
||||
|
||||
def read(self):
|
||||
with self._lock:
|
||||
if self._frame is None or not self._opened:
|
||||
return False, None
|
||||
return True, self._frame.copy()
|
||||
|
||||
def set(self, *args) -> bool:
|
||||
return True # cv2 CAP_PROP_* no-op
|
||||
|
||||
def release(self) -> None:
|
||||
self._stop.set()
|
||||
if self._sock is not None:
|
||||
try:
|
||||
self._sock.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=2.0)
|
||||
self._opened = False
|
||||
+23
-5
@@ -207,8 +207,16 @@ class AppDelegate(NSObject):
|
||||
vienne du MAIN THREAD AppKit. OpenCV depuis un worker thread plante
|
||||
avec 'can not spin main run loop from other thread'. On demande ici,
|
||||
sur le main thread, PUIS on lance le pose worker avec
|
||||
OPENCV_AVFOUNDATION_SKIP_AUTH=1 pour qu'il ne tente pas une 2e demande."""
|
||||
OPENCV_AVFOUNDATION_SKIP_AUTH=1 pour qu'il ne tente pas une 2e demande.
|
||||
|
||||
Exception: when --iphone-usb is set, IphoneUSBSource connects via
|
||||
usbmuxd — no Mac-camera TCC needed — so we skip straight to the worker.
|
||||
"""
|
||||
import os
|
||||
if getattr(self._opts, "iphone_usb", False):
|
||||
LOG.info("iphone-usb mode — skipping camera TCC request")
|
||||
self._start_pose_worker()
|
||||
return
|
||||
os.environ["OPENCV_AVFOUNDATION_SKIP_AUTH"] = "1"
|
||||
try:
|
||||
from AVFoundation import (
|
||||
@@ -289,9 +297,15 @@ class AppDelegate(NSObject):
|
||||
from .multi_hmr_worker import MultiHMRWorker
|
||||
from .smplx_osc_sender import SMPLXTCPSender
|
||||
if MultiHMRWorker.is_available():
|
||||
# target_fps=30 : the worker loop used to self-throttle
|
||||
# at 10 fps (sleep(period - dt)). With the async remote
|
||||
# backend (drop-newest in / latest out queue), we want
|
||||
# the loop to spin at camera rate so we always submit
|
||||
# the freshest frame and drain the freshest result.
|
||||
self._pose_worker = MultiHMRWorker(
|
||||
self._state, num_persons=4,
|
||||
target_fps=10.0,
|
||||
target_fps=float(_os.environ.get(
|
||||
"MULTIHMR_LOOP_FPS", "30.0")),
|
||||
device=getattr(self._opts, "pose_device", "mps"),
|
||||
det_thresh=getattr(self._opts, "det_thresh", 0.15),
|
||||
nms_kernel_size=getattr(
|
||||
@@ -360,7 +374,9 @@ class AppDelegate(NSObject):
|
||||
if _os.environ.get("AV_LIVE_MEDIAPIPE") != "0":
|
||||
try:
|
||||
from .multi import MultiWorker
|
||||
self._pose_worker = MultiWorker(self._state, num_persons=4)
|
||||
self._pose_worker = MultiWorker(
|
||||
self._state, num_persons=4,
|
||||
iphone_usb=getattr(self._opts, "iphone_usb", False))
|
||||
self._pose_worker.start()
|
||||
LOG.info("worker: MediaPipe Multi (Pose+Face+Hand × 4)")
|
||||
return
|
||||
@@ -567,8 +583,8 @@ class AppDelegate(NSObject):
|
||||
LOG.debug("[key] raw=%r lower=%r", key, k)
|
||||
if key == "\x1b":
|
||||
NSApp().terminate_(self); return None
|
||||
if key == " ":
|
||||
self._scClient.send_message("/control/doScene", ["stop"])
|
||||
if key == " ": # espace : morceau suivant (concert)
|
||||
self._scClient.send_message("/control/concertNext", [1])
|
||||
return None
|
||||
# Cmd+F geree par macOS pour fullscreen ; on garde shift+F en raccourci
|
||||
if key == "F":
|
||||
@@ -662,6 +678,8 @@ def main() -> int:
|
||||
choices=["n", "s", "l"],
|
||||
default="n",
|
||||
help="DETRPose model size (default: n)")
|
||||
p.add_argument("--iphone-usb", dest="iphone_usb", action="store_true",
|
||||
help="drive pose from the iPhone USB stream (no Mac camera)")
|
||||
p.add_argument("-v", "--verbose", action="store_true")
|
||||
opts = p.parse_args()
|
||||
|
||||
|
||||
+72
-13
@@ -19,6 +19,8 @@ import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
from .action_head_pub import ActionHeadPublisher
|
||||
from .euro_filter import SkeletonFilter
|
||||
from .pose_bridge import PoseSoundBridge
|
||||
@@ -29,6 +31,25 @@ from .tracker import IoUTracker
|
||||
|
||||
LOG = logging.getLogger("multi")
|
||||
|
||||
# Rotation cosmetique + detection de la frame video (env VIDEO_ROTATE).
|
||||
# Appliquee AVANT MediaPipe : la detection tourne sur l'image redressee et
|
||||
# l'overlay reste aligne. Les joints ARKit (monde 3D, gravity-aligned) sont
|
||||
# invariants a l'orientation du device, donc non concernes.
|
||||
_ROTATE_K = {"none": 0, "ccw": 1, "180": 2, "cw": 3}
|
||||
|
||||
|
||||
def _apply_video_rotate(frame, mode: str):
|
||||
"""Rotate a BGR frame by mode (none/ccw/cw/180) via numpy rot90.
|
||||
|
||||
Returns a C-contiguous array (MediaPipe / cv2 need contiguous input).
|
||||
Unknown modes are treated as 'none' (no rotation).
|
||||
"""
|
||||
k = _ROTATE_K.get(mode, 0)
|
||||
if k == 0:
|
||||
return frame
|
||||
return np.ascontiguousarray(np.rot90(frame, k))
|
||||
|
||||
|
||||
MODELS = {
|
||||
"pose": (
|
||||
"https://storage.googleapis.com/mediapipe-models/pose_landmarker/"
|
||||
@@ -78,9 +99,11 @@ class MultiWorker:
|
||||
target_fps: float = 18.0,
|
||||
num_persons: int = 4,
|
||||
min_conf: float = 0.4,
|
||||
iphone_usb: bool = False,
|
||||
) -> None:
|
||||
self.state = state
|
||||
self.camera_index = camera_index
|
||||
self.iphone_usb = iphone_usb
|
||||
self.period = 1.0 / max(1.0, target_fps)
|
||||
self.num_persons = num_persons
|
||||
self.min_conf = min_conf
|
||||
@@ -303,8 +326,20 @@ class MultiWorker:
|
||||
LOG.error("download models failed: %s", e)
|
||||
return
|
||||
|
||||
# GPU delegate (Metal sur macOS) : libere le CPU pour OSC, state,
|
||||
# mesh_rigger. Multi-HMR remote macm1 + MediaPipe GPU M5 =
|
||||
# workload distribue. Toggle via MEDIAPIPE_DELEGATE=cpu si plante.
|
||||
import os as _os
|
||||
_deleg_name = _os.environ.get("MEDIAPIPE_DELEGATE", "gpu").lower()
|
||||
_deleg = (BaseOptions.Delegate.GPU if _deleg_name == "gpu"
|
||||
else BaseOptions.Delegate.CPU)
|
||||
LOG.info("MediaPipe delegate = %s (env MEDIAPIPE_DELEGATE)",
|
||||
_deleg.name)
|
||||
_rot = _os.environ.get("VIDEO_ROTATE", "none").lower()
|
||||
LOG.info("video rotate = %s (env VIDEO_ROTATE: none/ccw/cw/180)", _rot)
|
||||
pose = PoseLandmarker.create_from_options(PoseLandmarkerOptions(
|
||||
base_options=BaseOptions(model_asset_path=str(pose_p)),
|
||||
base_options=BaseOptions(model_asset_path=str(pose_p),
|
||||
delegate=_deleg),
|
||||
running_mode=RunningMode.VIDEO,
|
||||
num_poses=self.num_persons,
|
||||
min_pose_detection_confidence=self.min_conf,
|
||||
@@ -312,7 +347,8 @@ class MultiWorker:
|
||||
min_tracking_confidence=self.min_conf,
|
||||
))
|
||||
face = FaceLandmarker.create_from_options(FaceLandmarkerOptions(
|
||||
base_options=BaseOptions(model_asset_path=str(face_p)),
|
||||
base_options=BaseOptions(model_asset_path=str(face_p),
|
||||
delegate=_deleg),
|
||||
running_mode=RunningMode.VIDEO,
|
||||
num_faces=self.num_persons,
|
||||
min_face_detection_confidence=self.min_conf,
|
||||
@@ -320,22 +356,32 @@ class MultiWorker:
|
||||
min_tracking_confidence=self.min_conf,
|
||||
))
|
||||
hand = HandLandmarker.create_from_options(HandLandmarkerOptions(
|
||||
base_options=BaseOptions(model_asset_path=str(hand_p)),
|
||||
base_options=BaseOptions(model_asset_path=str(hand_p),
|
||||
delegate=_deleg),
|
||||
running_mode=RunningMode.VIDEO,
|
||||
num_hands=self.num_persons * 2,
|
||||
min_hand_detection_confidence=self.min_conf,
|
||||
min_hand_presence_confidence=self.min_conf,
|
||||
min_tracking_confidence=self.min_conf,
|
||||
))
|
||||
LOG.info("3 landmarkers prets (num=%d)", self.num_persons)
|
||||
LOG.info("3 landmarkers prets (num=%d, delegate=%s)",
|
||||
self.num_persons, _deleg.name)
|
||||
|
||||
cap = cv2.VideoCapture(self.camera_index)
|
||||
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
|
||||
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
|
||||
if not cap.isOpened():
|
||||
LOG.error("camera index %d indisponible (TCC ?)", self.camera_index)
|
||||
return
|
||||
LOG.info("camera ouverte (index %d)", self.camera_index)
|
||||
if self.iphone_usb:
|
||||
from .iphone_usb_source import IphoneUSBSource # noqa: PLC0415
|
||||
cap = IphoneUSBSource(self.state, write_hands=False)
|
||||
if not cap.start():
|
||||
LOG.error("iphone USB source unavailable (app running? phone unlocked?)")
|
||||
return
|
||||
LOG.info("iphone USB source")
|
||||
else:
|
||||
cap = cv2.VideoCapture(self.camera_index)
|
||||
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 640)
|
||||
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 480)
|
||||
if not cap.isOpened():
|
||||
LOG.error("camera index %d indisponible (TCC ?)", self.camera_index)
|
||||
return
|
||||
LOG.info("camera ouverte (index %d)", self.camera_index)
|
||||
|
||||
t0_ms = int(time.monotonic() * 1000)
|
||||
while not self._stop.is_set():
|
||||
@@ -344,9 +390,22 @@ class MultiWorker:
|
||||
if not ok or frame_bgr is None:
|
||||
time.sleep(self.period)
|
||||
continue
|
||||
# Redresse la frame (iPhone tourne physiquement) AVANT MediaPipe
|
||||
# et l'encodage JPEG : detection + overlay + affichage coherents.
|
||||
frame_bgr = _apply_video_rotate(frame_bgr, _rot)
|
||||
h, w = frame_bgr.shape[:2]
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
mp_img = mp.Image(image_format=mp.ImageFormat.SRGB, data=frame_rgb)
|
||||
# MediaPipe GPU delegate on macOS uploads via CVPixelBuffer
|
||||
# which only accepts 4-channel formats. SRGB (3ch) crashes
|
||||
# in gpu_buffer_storage_cv_pixel_buffer.cc with
|
||||
# "unsupported ImageFrame format: 1". Use SRGBA when on GPU.
|
||||
if _deleg == BaseOptions.Delegate.GPU:
|
||||
frame_rgba = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGBA)
|
||||
mp_img = mp.Image(image_format=mp.ImageFormat.SRGBA,
|
||||
data=frame_rgba)
|
||||
else:
|
||||
frame_rgb = cv2.cvtColor(frame_bgr, cv2.COLOR_BGR2RGB)
|
||||
mp_img = mp.Image(image_format=mp.ImageFormat.SRGB,
|
||||
data=frame_rgb)
|
||||
ts = int(time.monotonic() * 1000) - t0_ms
|
||||
try:
|
||||
pose_res = pose.detect_for_video(mp_img, ts)
|
||||
|
||||
@@ -100,12 +100,22 @@ class MultiHMRWorker:
|
||||
# (cf tracker.py) pour resister aux occlusions et au mouvement
|
||||
# rapide. Multi-HMR a 3 fps -> 30 frames = 10s de survie.
|
||||
self._tracker = IoUTracker(iou_threshold=0.15, max_miss=30)
|
||||
# Lazily-loaded CoreML backend for predict_once (single-shot,
|
||||
# off-thread). Independent of the worker thread's _run_coreml
|
||||
# backend instance — predict_once must work even without start().
|
||||
self._coreml_backend_singleshot = None
|
||||
|
||||
@staticmethod
|
||||
def is_available() -> bool:
|
||||
backend = os.environ.get("MULTIHMR_BACKEND", "pytorch").strip().lower()
|
||||
if backend == "coreml":
|
||||
return COREML_MLPACKAGE.exists()
|
||||
if backend == "remote":
|
||||
try:
|
||||
from .multihmr_remote import MultiHMRRemoteBackend
|
||||
return MultiHMRRemoteBackend.is_available()
|
||||
except Exception: # noqa: BLE001
|
||||
return False
|
||||
return CKPT.exists() and SMPLX_PATH.exists() and MULTIHMR_REPO.exists()
|
||||
|
||||
def start(self) -> None:
|
||||
@@ -116,23 +126,89 @@ class MultiHMRWorker:
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
|
||||
def _get_or_load_coreml_backend(self):
|
||||
"""Lazily load the CoreML backend for single-shot inference.
|
||||
|
||||
Returns the cached `MultiHMRCoreMLBackend` instance, or None if
|
||||
the backend cannot be imported / the .mlpackage is missing.
|
||||
Thread-safe enough for our use (calibration CLI is single-
|
||||
threaded; the worker thread uses its own backend in _run_coreml).
|
||||
"""
|
||||
if self._coreml_backend_singleshot is not None:
|
||||
return self._coreml_backend_singleshot
|
||||
try:
|
||||
from .multihmr_coreml import MultiHMRCoreMLBackend
|
||||
backend = MultiHMRCoreMLBackend(COREML_MLPACKAGE)
|
||||
except (ImportError, FileNotFoundError) as e:
|
||||
LOG.info("predict_once: CoreML backend unavailable: %s", e)
|
||||
return None
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("predict_once: CoreML backend init failed: %s", e)
|
||||
return None
|
||||
self._coreml_backend_singleshot = backend
|
||||
return backend
|
||||
|
||||
def predict_once(self, rgb_image):
|
||||
"""Single-shot SMPL-X prediction on one RGB image.
|
||||
|
||||
Used by calibrate_lidar.py to acquire a pelvis vertex without
|
||||
spinning the worker thread. The current PyTorch path is
|
||||
deeply coupled to the run loop (model lifecycle, camera, MPS
|
||||
setup) so this is left as a stub — calibrate_lidar.py keeps
|
||||
its placeholder until a follow-up refactor extracts a pure
|
||||
``_infer(rgb) -> humans`` helper.
|
||||
Args:
|
||||
rgb_image: (H, W, 3) uint8 RGB array. Will be center-
|
||||
cropped + resized to 672x672 internally.
|
||||
|
||||
Returns:
|
||||
First `SMPLXPerson` detection (pid=0) or None if no
|
||||
humans pass the detection threshold.
|
||||
|
||||
Raises:
|
||||
NotImplementedError: if the CoreML backend is unavailable
|
||||
(PyTorch single-shot path is TBD).
|
||||
"""
|
||||
raise NotImplementedError(
|
||||
"MultiHMRWorker.predict_once is not wired yet — see "
|
||||
"scripts/calibrate_lidar.py for the placeholder it gates")
|
||||
backend = self._get_or_load_coreml_backend()
|
||||
if backend is None:
|
||||
raise NotImplementedError(
|
||||
"CoreML backend unavailable; PyTorch single-shot path TBD")
|
||||
|
||||
try:
|
||||
import cv2
|
||||
except ImportError as e:
|
||||
raise NotImplementedError(
|
||||
"opencv-python required for predict_once: %s" % e)
|
||||
|
||||
rgb = np.asarray(rgb_image)
|
||||
if rgb.ndim != 3 or rgb.shape[2] != 3:
|
||||
raise ValueError(
|
||||
f"rgb_image must be (H,W,3), got {rgb.shape}")
|
||||
h, w = rgb.shape[:2]
|
||||
if (h, w) != (IMG_SIZE, IMG_SIZE):
|
||||
side = min(h, w)
|
||||
y0 = (h - side) // 2
|
||||
x0 = (w - side) // 2
|
||||
rgb = rgb[y0:y0 + side, x0:x0 + side]
|
||||
rgb = cv2.resize(rgb, (IMG_SIZE, IMG_SIZE))
|
||||
|
||||
img = rgb.transpose(2, 0, 1).astype(np.float32) / 255.0
|
||||
focal = float(IMG_SIZE)
|
||||
K_np = np.array([[focal, 0.0, IMG_SIZE / 2.0],
|
||||
[0.0, focal, IMG_SIZE / 2.0],
|
||||
[0.0, 0.0, 1.0]], dtype=np.float32)
|
||||
|
||||
humans = backend.infer(img, K_np, det_thresh=self.det_thresh)
|
||||
if not humans:
|
||||
return None
|
||||
|
||||
hh = humans[0]
|
||||
v3d = hh["v3d"].detach().cpu().numpy()
|
||||
return SMPLXPerson(
|
||||
pid=0,
|
||||
vertices_3d=np.ascontiguousarray(v3d, dtype=np.float32),
|
||||
)
|
||||
|
||||
def _run(self) -> None:
|
||||
if self.backend == "coreml":
|
||||
self._run_coreml()
|
||||
self._run_coreml(remote=False)
|
||||
return
|
||||
if self.backend == "remote":
|
||||
self._run_coreml(remote=True)
|
||||
return
|
||||
self._run_pytorch()
|
||||
|
||||
@@ -481,20 +557,34 @@ class MultiHMRWorker:
|
||||
# ------------------------------------------------------------------
|
||||
# CoreML backend
|
||||
# ------------------------------------------------------------------
|
||||
def _run_coreml(self) -> None:
|
||||
def _run_coreml(self, remote: bool = False) -> None:
|
||||
"""CoreML inference path (ANE+GPU+CPU via Apple's framework).
|
||||
|
||||
Mirrors _run_pytorch but loads the .mlpackage via pyobjc + the
|
||||
CoreML.framework, bypassing torch/MPS entirely. ~3-4x faster
|
||||
on M5 (28.8ms median vs ~100ms with MPS)."""
|
||||
on M5 (28.8ms median vs ~100ms with MPS).
|
||||
|
||||
If ``remote=True``, the local CoreML backend is swapped for a
|
||||
TCP client (``MultiHMRRemoteBackend``) that talks to a server
|
||||
running the same mlpackage on a faster Mac (macm1, M1 Max).
|
||||
"""
|
||||
try:
|
||||
import cv2
|
||||
except ImportError as e:
|
||||
LOG.error("opencv-python missing: %s", e)
|
||||
return
|
||||
try:
|
||||
from .multihmr_coreml import MultiHMRCoreMLBackend
|
||||
backend = MultiHMRCoreMLBackend(COREML_MLPACKAGE)
|
||||
if remote:
|
||||
from .multihmr_remote import MultiHMRRemoteBackend
|
||||
host = os.environ.get(
|
||||
"MULTIHMR_REMOTE_HOST", "192.168.0.175")
|
||||
port = int(os.environ.get(
|
||||
"MULTIHMR_REMOTE_PORT", "57140"))
|
||||
backend = MultiHMRRemoteBackend(host=host, port=port)
|
||||
LOG.info("Multi-HMR remote backend (%s:%d)", host, port)
|
||||
else:
|
||||
from .multihmr_coreml import MultiHMRCoreMLBackend
|
||||
backend = MultiHMRCoreMLBackend(COREML_MLPACKAGE)
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.error("CoreML backend init failed: %s", e)
|
||||
return
|
||||
@@ -522,12 +612,14 @@ class MultiHMRWorker:
|
||||
if not cap.start():
|
||||
LOG.error("AVCapture start failed pour %s", info["name"])
|
||||
return
|
||||
LOG.info("camera ouverte %s (%s) [coreml backend]",
|
||||
info["name"], info["type"])
|
||||
LOG.info("camera ouverte %s (%s) [%s backend]",
|
||||
info["name"], info["type"],
|
||||
"remote" if remote else "coreml")
|
||||
|
||||
frame_count = 0
|
||||
persons_count = 0
|
||||
skipped_static = 0
|
||||
fresh_count = 0
|
||||
next_heartbeat = time.monotonic() + 5.0
|
||||
prev_thumb: np.ndarray | None = None
|
||||
|
||||
@@ -575,18 +667,34 @@ class MultiHMRWorker:
|
||||
time.sleep(self.period)
|
||||
continue
|
||||
|
||||
# Async remote backend may return None when no fresh result
|
||||
# is ready yet — reuse the previous frame's humans so the
|
||||
# visualiser keeps drawing instead of clearing.
|
||||
if humans is None:
|
||||
humans = getattr(self, "_last_humans", []) or []
|
||||
reused_humans = True
|
||||
else:
|
||||
self._last_humans = humans
|
||||
reused_humans = False
|
||||
fresh_count += 1
|
||||
|
||||
t_post_start = time.monotonic()
|
||||
t_now = time.monotonic()
|
||||
frame_count += 1
|
||||
persons_count += len(humans) if humans else 0
|
||||
if reused_humans:
|
||||
LOG.debug("hb[remote]: reusing %d cached humans "
|
||||
"(no fresh result)", len(humans))
|
||||
if t_now >= next_heartbeat:
|
||||
fps = frame_count / 5.0
|
||||
fresh_fps = fresh_count / 5.0
|
||||
avg = persons_count / max(1, frame_count)
|
||||
LOG.info(
|
||||
"hb[coreml]: %.1f fps, %.2f persons/frame, %d skipped",
|
||||
fps, avg, skipped_static)
|
||||
"hb[coreml]: %.1f fps (fresh=%.1f), %.2f persons/frame, "
|
||||
"%d skipped", fps, fresh_fps, avg, skipped_static)
|
||||
frame_count = 0
|
||||
persons_count = 0
|
||||
fresh_count = 0
|
||||
skipped_static = 0
|
||||
next_heartbeat = t_now + 5.0
|
||||
|
||||
@@ -596,6 +704,16 @@ class MultiHMRWorker:
|
||||
time.sleep(self.period)
|
||||
continue
|
||||
|
||||
# If async backend reused last humans, keep state untouched and
|
||||
# spin to the next frame without re-running dedup/tracker/
|
||||
# smoothing (saves ~3-5 ms CPU per loop iteration and avoids
|
||||
# walking the One-Euro filter forward on stale data).
|
||||
if reused_humans:
|
||||
dt = time.monotonic() - t_cap_start
|
||||
if dt < self.period:
|
||||
time.sleep(self.period - dt)
|
||||
continue
|
||||
|
||||
# Dedup intra-frame (same logic as pytorch path).
|
||||
cand: list[tuple[
|
||||
float, float, float, float, float,
|
||||
|
||||
@@ -39,12 +39,24 @@ DEFAULT_MLPACKAGE = (
|
||||
N_PERSONS_FIXED = 4
|
||||
N_VERTS = 10475
|
||||
|
||||
# CoreML output names from the exported .mlpackage.
|
||||
OUT_V3D = "var_2412" # (4, 10475, 3)
|
||||
OUT_TRANSL = "var_2415" # (4, 1, 3)
|
||||
OUT_SCORES = "var_2428" # (4,)
|
||||
OUT_BETAS = "var_2431" # (4, 10)
|
||||
OUT_EXPR = "var_2434" # (4, 10)
|
||||
# CoreML output names from the exported .mlpackage. The exported
|
||||
# `multihmr_full_672_s.mlpackage` (2026-05-14 re-convert) renumbered
|
||||
# the MIL vars; verified against the on-disk artifact's spec.
|
||||
OUT_V3D = "var_2420" # (4, 10475, 3)
|
||||
OUT_TRANSL = "var_2423" # (4, 1, 3)
|
||||
OUT_SCORES = "var_2436" # (4,)
|
||||
OUT_BETAS = "var_2439" # (4, 10)
|
||||
OUT_EXPR = "var_2442" # (4, 10)
|
||||
# var_2445 (4, 127, 3) = j3d joints — present but unused here.
|
||||
|
||||
# DINOv2 backbone was trained on ImageNet-normalized RGB; the public
|
||||
# `infer()` contract takes [0,1] CHW input and applies this here so
|
||||
# every caller stays normalization-agnostic. Feeding raw [0,1] to the
|
||||
# model collapses all detection scores to ~0.01 ("0 detections" bug).
|
||||
_IMG_NORM_MEAN = np.array([0.485, 0.456, 0.406],
|
||||
dtype=np.float32).reshape(1, 3, 1, 1)
|
||||
_IMG_NORM_STD = np.array([0.229, 0.224, 0.225],
|
||||
dtype=np.float32).reshape(1, 3, 1, 1)
|
||||
|
||||
# MLMultiArrayDataType raw values (from CoreML headers).
|
||||
ML_DTYPE_FLOAT32 = 65568
|
||||
@@ -245,7 +257,8 @@ class MultiHMRCoreMLBackend:
|
||||
"""Run a forward pass and return list of humans dicts.
|
||||
|
||||
Args:
|
||||
image_chw_float32: (3, 672, 672) or (1, 3, 672, 672) in [0,1].
|
||||
image_chw_float32: (3, 672, 672) or (1, 3, 672, 672), RGB in
|
||||
[0,1]. ImageNet normalization is applied internally.
|
||||
K_33: (3, 3) or (1, 3, 3) camera intrinsics.
|
||||
det_thresh: scores threshold; CoreML forwards K=4 always.
|
||||
|
||||
@@ -264,6 +277,7 @@ class MultiHMRCoreMLBackend:
|
||||
if K.shape != (1, 3, 3):
|
||||
raise ValueError(f"K shape {K.shape}, expected (1,3,3)")
|
||||
|
||||
img = (img - _IMG_NORM_MEAN) / _IMG_NORM_STD
|
||||
raw = self._predict(img, K)
|
||||
v3d = raw.get(OUT_V3D)
|
||||
transl = raw.get(OUT_TRANSL)
|
||||
|
||||
@@ -0,0 +1,463 @@
|
||||
"""Multi-HMR remote backend: drop-in replacement of MultiHMRCoreMLBackend
|
||||
that delegates inference to a remote TCP server (see
|
||||
``scripts/multihmr_server.py``).
|
||||
|
||||
Protocol (little-endian, persistent connection):
|
||||
|
||||
Request:
|
||||
[4B uint32 payload_len]
|
||||
[4B magic "REQ\x01"]
|
||||
[1B uint8 format_id] # 1 = raw RGB uint8 HWC, 2 = JPEG (variable length)
|
||||
[3B padding]
|
||||
[variable image bytes] # IMG_BYTES if format=1, else JPEG bytes
|
||||
[9 float32 K = 36 bytes]
|
||||
|
||||
The K block is *always* the last 36 bytes of the payload, regardless of
|
||||
``format_id`` — the server slices it off before treating the rest as the
|
||||
image.
|
||||
|
||||
Response:
|
||||
[4B uint32 payload_len]
|
||||
[4B magic "RSP\x01"]
|
||||
[4B int32 status]
|
||||
[v3d : 4*10475*3 f32]
|
||||
[transl: 4*1*3 f32]
|
||||
[scores: 4 f32]
|
||||
[betas: 4*10 f32]
|
||||
[expr : 4*10 f32]
|
||||
|
||||
Two extra features over the bare RPC:
|
||||
|
||||
* JPEG compression (``MULTIHMR_REMOTE_JPEG=1``, default ON, quality 80).
|
||||
Cuts wire bytes from ~1.35 MB to ~50-150 KB.
|
||||
|
||||
* Asynchronous double-buffer (``MULTIHMR_REMOTE_ASYNC=1``, default ON).
|
||||
``infer()`` is decoupled from the I/O round-trip via a dedicated worker
|
||||
thread and two ``Queue(maxsize=1)`` slots. When the out-queue is empty
|
||||
``infer()`` returns ``None`` — the worker loop reuses its last humans
|
||||
list so the visualiser keeps drawing.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import socket
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
LOG = logging.getLogger("multihmr_remote")
|
||||
|
||||
IMG_SIZE = 672
|
||||
N_PERSONS_FIXED = 4
|
||||
N_VERTS = 10475
|
||||
|
||||
MAGIC_REQ = b"REQ\x01"
|
||||
MAGIC_RSP = b"RSP\x01"
|
||||
|
||||
FORMAT_RAW = 1
|
||||
FORMAT_JPEG = 2
|
||||
|
||||
IMG_BYTES = IMG_SIZE * IMG_SIZE * 3
|
||||
K_BYTES = 9 * 4
|
||||
REQ_HEADER = 4 + 1 + 3 # magic + format_id + 3-byte pad
|
||||
# Fixed RAW-format request payload (mirrors RSP_PAYLOAD_LEN). The JPEG
|
||||
# path is variable-length, so this is the upper-bound / RAW case only.
|
||||
REQ_PAYLOAD_LEN = REQ_HEADER + IMG_BYTES + K_BYTES
|
||||
|
||||
V3D_BYTES = N_PERSONS_FIXED * N_VERTS * 3 * 4
|
||||
TRANSL_BYTES = N_PERSONS_FIXED * 1 * 3 * 4
|
||||
SCORES_BYTES = N_PERSONS_FIXED * 4
|
||||
BETAS_BYTES = N_PERSONS_FIXED * 10 * 4
|
||||
EXPR_BYTES = N_PERSONS_FIXED * 10 * 4
|
||||
RSP_HEADER = 4 + 4
|
||||
RSP_PAYLOAD_LEN = (RSP_HEADER + V3D_BYTES + TRANSL_BYTES
|
||||
+ SCORES_BYTES + BETAS_BYTES + EXPR_BYTES)
|
||||
|
||||
|
||||
def _env_flag(name: str, default: bool) -> bool:
|
||||
raw = os.environ.get(name)
|
||||
if raw is None:
|
||||
return default
|
||||
return raw.strip().lower() in ("1", "true", "yes", "on")
|
||||
|
||||
|
||||
def _recv_exact(sock: socket.socket, n: int) -> bytes:
|
||||
buf = bytearray(n)
|
||||
view = memoryview(buf)
|
||||
pos = 0
|
||||
while pos < n:
|
||||
got = sock.recv_into(view[pos:])
|
||||
if got == 0:
|
||||
raise ConnectionError("peer closed mid-stream")
|
||||
pos += got
|
||||
return bytes(buf)
|
||||
|
||||
|
||||
def encode_request_raw(image_uint8_hwc: np.ndarray,
|
||||
K_33: np.ndarray) -> bytes:
|
||||
"""Raw uint8 HWC request (format_id=1, fixed payload length)."""
|
||||
if image_uint8_hwc.shape != (IMG_SIZE, IMG_SIZE, 3):
|
||||
raise ValueError(
|
||||
f"image shape {image_uint8_hwc.shape} != "
|
||||
f"({IMG_SIZE},{IMG_SIZE},3)")
|
||||
if image_uint8_hwc.dtype != np.uint8:
|
||||
raise ValueError(f"image dtype {image_uint8_hwc.dtype} != uint8")
|
||||
K = np.ascontiguousarray(K_33, dtype="<f4").reshape(9)
|
||||
img = np.ascontiguousarray(image_uint8_hwc, dtype=np.uint8)
|
||||
img_bytes = img.tobytes()
|
||||
header_after_magic = bytes([FORMAT_RAW, 0, 0, 0])
|
||||
payload_len = REQ_HEADER + len(img_bytes) + K_BYTES
|
||||
return b"".join([
|
||||
struct.pack("<I", payload_len),
|
||||
MAGIC_REQ,
|
||||
header_after_magic,
|
||||
img_bytes,
|
||||
K.tobytes(),
|
||||
])
|
||||
|
||||
|
||||
def encode_request_jpeg(jpeg_bytes: bytes, K_33: np.ndarray) -> bytes:
|
||||
"""JPEG request (format_id=2, variable payload length)."""
|
||||
K = np.ascontiguousarray(K_33, dtype="<f4").reshape(9)
|
||||
header_after_magic = bytes([FORMAT_JPEG, 0, 0, 0])
|
||||
payload_len = REQ_HEADER + len(jpeg_bytes) + K_BYTES
|
||||
return b"".join([
|
||||
struct.pack("<I", payload_len),
|
||||
MAGIC_REQ,
|
||||
header_after_magic,
|
||||
jpeg_bytes,
|
||||
K.tobytes(),
|
||||
])
|
||||
|
||||
|
||||
def decode_response(payload: bytes) -> tuple[
|
||||
np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray, int]:
|
||||
if len(payload) != RSP_PAYLOAD_LEN:
|
||||
raise ValueError(
|
||||
f"rsp payload {len(payload)} != {RSP_PAYLOAD_LEN}")
|
||||
if payload[:4] != MAGIC_RSP:
|
||||
raise ValueError(f"bad rsp magic {payload[:4]!r}")
|
||||
status = struct.unpack("<i", payload[4:8])[0]
|
||||
off = 8
|
||||
v3d = np.frombuffer(payload, dtype="<f4",
|
||||
count=N_PERSONS_FIXED * N_VERTS * 3,
|
||||
offset=off).reshape(N_PERSONS_FIXED, N_VERTS, 3)
|
||||
off += V3D_BYTES
|
||||
transl = np.frombuffer(payload, dtype="<f4",
|
||||
count=N_PERSONS_FIXED * 1 * 3,
|
||||
offset=off).reshape(N_PERSONS_FIXED, 1, 3)
|
||||
off += TRANSL_BYTES
|
||||
scores = np.frombuffer(payload, dtype="<f4",
|
||||
count=N_PERSONS_FIXED,
|
||||
offset=off).reshape(N_PERSONS_FIXED)
|
||||
off += SCORES_BYTES
|
||||
betas = np.frombuffer(payload, dtype="<f4",
|
||||
count=N_PERSONS_FIXED * 10,
|
||||
offset=off).reshape(N_PERSONS_FIXED, 10)
|
||||
off += BETAS_BYTES
|
||||
expr = np.frombuffer(payload, dtype="<f4",
|
||||
count=N_PERSONS_FIXED * 10,
|
||||
offset=off).reshape(N_PERSONS_FIXED, 10)
|
||||
return (v3d.copy(), transl.copy(), scores.copy(),
|
||||
betas.copy(), expr.copy(), int(status))
|
||||
|
||||
|
||||
# Back-compat shim — old call sites used encode_request(img, K) for raw.
|
||||
def encode_request(image_uint8_hwc: np.ndarray, K_33: np.ndarray) -> bytes:
|
||||
return encode_request_raw(image_uint8_hwc, K_33)
|
||||
|
||||
|
||||
class _Tensorlike:
|
||||
"""Mimics CoreMLArray to avoid a hard import on multihmr_coreml."""
|
||||
__slots__ = ("_arr",)
|
||||
|
||||
def __init__(self, arr: np.ndarray) -> None:
|
||||
self._arr = arr
|
||||
|
||||
def detach(self) -> "_Tensorlike":
|
||||
return self
|
||||
|
||||
def cpu(self) -> "_Tensorlike":
|
||||
return self
|
||||
|
||||
def numpy(self) -> np.ndarray:
|
||||
return self._arr
|
||||
|
||||
def item(self) -> float:
|
||||
return float(self._arr.reshape(-1)[0])
|
||||
|
||||
@property
|
||||
def shape(self) -> tuple[int, ...]:
|
||||
return tuple(self._arr.shape)
|
||||
|
||||
|
||||
def _humans_from_arrays(v3d: np.ndarray, transl: np.ndarray,
|
||||
scores: np.ndarray, betas: np.ndarray,
|
||||
expr: np.ndarray, det_thresh: float
|
||||
) -> list[dict[str, Any]]:
|
||||
humans: list[dict[str, Any]] = []
|
||||
for k in range(N_PERSONS_FIXED):
|
||||
sc = float(scores[k])
|
||||
if sc < det_thresh:
|
||||
continue
|
||||
humans.append({
|
||||
"v3d": _Tensorlike(v3d[k]),
|
||||
"transl_pelvis": _Tensorlike(transl[k]),
|
||||
"scores": _Tensorlike(np.array([sc], dtype=np.float32)),
|
||||
"shape": _Tensorlike(betas[k]),
|
||||
"expression": _Tensorlike(expr[k]),
|
||||
})
|
||||
return humans
|
||||
|
||||
|
||||
class MultiHMRRemoteBackend:
|
||||
"""TCP client backend mirroring ``MultiHMRCoreMLBackend.infer`` API.
|
||||
|
||||
JPEG compression and async double-buffering are toggleable via env
|
||||
(``MULTIHMR_REMOTE_JPEG``, ``MULTIHMR_REMOTE_ASYNC``).
|
||||
"""
|
||||
|
||||
def __init__(self, host: str = "192.168.0.175", port: int = 57140,
|
||||
connect_timeout: float = 3.0,
|
||||
io_timeout: float = 5.0) -> None:
|
||||
self.host = host
|
||||
self.port = port
|
||||
self.connect_timeout = connect_timeout
|
||||
self.io_timeout = io_timeout
|
||||
self._sock: socket.socket | None = None
|
||||
self._lock = threading.Lock()
|
||||
|
||||
self.use_jpeg = _env_flag("MULTIHMR_REMOTE_JPEG", True)
|
||||
self.jpeg_quality = int(os.environ.get(
|
||||
"MULTIHMR_REMOTE_JPEG_QUALITY", "80"))
|
||||
self.use_async = _env_flag("MULTIHMR_REMOTE_ASYNC", True)
|
||||
|
||||
# Async pipeline state.
|
||||
# Multi-buffer queues (2 in / 3 out) absorb jitter without
|
||||
# stalling capture. Drop-oldest semantics on overflow.
|
||||
self._in_q: queue.Queue[tuple[bytes, float, float]] = queue.Queue(
|
||||
maxsize=2)
|
||||
self._out_q: queue.Queue[
|
||||
tuple[list[dict[str, Any]], dict[str, float]]
|
||||
] = queue.Queue(maxsize=3)
|
||||
self._stop = threading.Event()
|
||||
self._async_det_thresh = 0.3
|
||||
self._worker_thread: threading.Thread | None = None
|
||||
self._last_stats: dict[str, float] = {}
|
||||
|
||||
if self.use_jpeg:
|
||||
try:
|
||||
import cv2 # noqa: F401
|
||||
except ImportError:
|
||||
LOG.warning("cv2 unavailable client-side, disabling JPEG")
|
||||
self.use_jpeg = False
|
||||
|
||||
if self.use_async:
|
||||
self._start_worker()
|
||||
LOG.info(
|
||||
"MultiHMRRemoteBackend %s:%d (jpeg=%s q=%d, async=%s)",
|
||||
host, port, self.use_jpeg, self.jpeg_quality, self.use_async)
|
||||
|
||||
# -- connection management -------------------------------------------
|
||||
|
||||
def _connect(self) -> socket.socket:
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
sock.settimeout(self.connect_timeout)
|
||||
sock.connect((self.host, self.port))
|
||||
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
|
||||
sock.settimeout(self.io_timeout)
|
||||
LOG.info("connected to %s:%d", self.host, self.port)
|
||||
return sock
|
||||
|
||||
def _ensure_sock(self) -> socket.socket:
|
||||
if self._sock is None:
|
||||
self._sock = self._connect()
|
||||
return self._sock
|
||||
|
||||
def _drop_sock(self) -> None:
|
||||
if self._sock is not None:
|
||||
try:
|
||||
self._sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
self._sock = None
|
||||
|
||||
@staticmethod
|
||||
def is_available(host: str | None = None, port: int | None = None
|
||||
) -> bool:
|
||||
host = host or os.environ.get(
|
||||
"MULTIHMR_REMOTE_HOST", "192.168.0.175")
|
||||
port = port or int(os.environ.get(
|
||||
"MULTIHMR_REMOTE_PORT", "57140"))
|
||||
try:
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
s.settimeout(1.0)
|
||||
s.connect((host, port))
|
||||
s.close()
|
||||
return True
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
# -- request encoding ------------------------------------------------
|
||||
|
||||
def _encode_request_from_chw(
|
||||
self, image_chw_float32: np.ndarray, K_33: np.ndarray
|
||||
) -> tuple[bytes, float]:
|
||||
"""Return (request bytes, encode_ms)."""
|
||||
img = np.asarray(image_chw_float32, dtype=np.float32)
|
||||
if img.ndim == 4 and img.shape[0] == 1:
|
||||
img = img[0]
|
||||
if img.shape != (3, IMG_SIZE, IMG_SIZE):
|
||||
raise ValueError(
|
||||
f"image shape {img.shape} != (3,{IMG_SIZE},{IMG_SIZE})")
|
||||
img_hwc = np.clip(img.transpose(1, 2, 0) * 255.0, 0.0, 255.0
|
||||
).astype(np.uint8)
|
||||
K = np.asarray(K_33, dtype=np.float32)
|
||||
if K.ndim == 3 and K.shape[0] == 1:
|
||||
K = K[0]
|
||||
if K.shape != (3, 3):
|
||||
raise ValueError(f"K shape {K.shape} != (3,3)")
|
||||
|
||||
t0 = time.monotonic()
|
||||
if self.use_jpeg:
|
||||
import cv2 # local import to keep optional
|
||||
# cv2.imencode wants BGR for nicest JPEG perceptually but the
|
||||
# server decodes back to RGB ; encode RGB->BGR once for parity.
|
||||
bgr = cv2.cvtColor(img_hwc, cv2.COLOR_RGB2BGR)
|
||||
ok, enc = cv2.imencode(
|
||||
".jpg", bgr,
|
||||
[int(cv2.IMWRITE_JPEG_QUALITY), self.jpeg_quality])
|
||||
if not ok:
|
||||
raise RuntimeError("cv2.imencode failed")
|
||||
req = encode_request_jpeg(bytes(enc), K)
|
||||
else:
|
||||
req = encode_request_raw(img_hwc, K)
|
||||
enc_ms = (time.monotonic() - t0) * 1e3
|
||||
return req, enc_ms
|
||||
|
||||
# -- synchronous fallback -------------------------------------------
|
||||
|
||||
def _send_recv(self, req: bytes) -> bytes:
|
||||
attempts = 0
|
||||
last_err: Exception | None = None
|
||||
while attempts < 2:
|
||||
attempts += 1
|
||||
try:
|
||||
sock = self._ensure_sock()
|
||||
sock.sendall(req)
|
||||
len_buf = _recv_exact(sock, 4)
|
||||
payload_len = struct.unpack("<I", len_buf)[0]
|
||||
if payload_len != RSP_PAYLOAD_LEN:
|
||||
raise ValueError(
|
||||
f"unexpected rsp len {payload_len}")
|
||||
return _recv_exact(sock, payload_len)
|
||||
except (ConnectionError, BrokenPipeError, OSError,
|
||||
socket.timeout) as e:
|
||||
LOG.warning("rpc failed (try %d): %s", attempts, e)
|
||||
self._drop_sock()
|
||||
last_err = e
|
||||
raise RuntimeError(f"remote inference failed: {last_err}")
|
||||
|
||||
# -- async worker ---------------------------------------------------
|
||||
|
||||
def _start_worker(self) -> None:
|
||||
self._worker_thread = threading.Thread(
|
||||
target=self._async_loop, name="multihmr-remote",
|
||||
daemon=True)
|
||||
self._worker_thread.start()
|
||||
|
||||
def _async_loop(self) -> None:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
req, t_submit, det_thresh = self._in_q.get(timeout=0.5)
|
||||
except queue.Empty:
|
||||
continue
|
||||
t_send = time.monotonic()
|
||||
try:
|
||||
with self._lock:
|
||||
payload = self._send_recv(req)
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("async send_recv failed: %s", e)
|
||||
continue
|
||||
t_recv = time.monotonic()
|
||||
try:
|
||||
v3d, transl, scores, betas, expr, status = decode_response(
|
||||
payload)
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("decode_response failed: %s", e)
|
||||
continue
|
||||
if status != 0:
|
||||
humans: list[dict[str, Any]] = []
|
||||
else:
|
||||
humans = _humans_from_arrays(
|
||||
v3d, transl, scores, betas, expr, det_thresh)
|
||||
stats = {
|
||||
"queue_wait_ms": (t_send - t_submit) * 1e3,
|
||||
"rpc_ms": (t_recv - t_send) * 1e3,
|
||||
}
|
||||
# Drop any pending stale output before pushing.
|
||||
try:
|
||||
self._out_q.get_nowait()
|
||||
except queue.Empty:
|
||||
pass
|
||||
try:
|
||||
self._out_q.put_nowait((humans, stats))
|
||||
except queue.Full:
|
||||
pass
|
||||
|
||||
# -- public API -----------------------------------------------------
|
||||
|
||||
def infer(
|
||||
self,
|
||||
image_chw_float32: np.ndarray,
|
||||
K_33: np.ndarray,
|
||||
det_thresh: float = 0.3,
|
||||
) -> list[dict[str, Any]] | None:
|
||||
"""In sync mode returns the humans list (possibly empty).
|
||||
|
||||
In async mode, submits the new frame (non-blocking, drop-newest
|
||||
if previous frame still in flight) and returns whatever output
|
||||
is ready in the out-queue. Returns ``None`` if nothing is ready
|
||||
yet — caller must reuse its last humans list.
|
||||
"""
|
||||
req, _enc_ms = self._encode_request_from_chw(
|
||||
image_chw_float32, K_33)
|
||||
|
||||
if not self.use_async:
|
||||
with self._lock:
|
||||
payload = self._send_recv(req)
|
||||
v3d, transl, scores, betas, expr, status = decode_response(
|
||||
payload)
|
||||
if status != 0:
|
||||
return []
|
||||
return _humans_from_arrays(
|
||||
v3d, transl, scores, betas, expr, det_thresh)
|
||||
|
||||
# Async path.
|
||||
self._async_det_thresh = det_thresh
|
||||
# drop-newest semantics: keep the freshest pending frame
|
||||
try:
|
||||
self._in_q.get_nowait()
|
||||
except queue.Empty:
|
||||
pass
|
||||
try:
|
||||
self._in_q.put_nowait((req, time.monotonic(), det_thresh))
|
||||
except queue.Full:
|
||||
pass
|
||||
|
||||
try:
|
||||
humans, stats = self._out_q.get_nowait()
|
||||
except queue.Empty:
|
||||
return None
|
||||
self._last_stats = stats
|
||||
return humans
|
||||
|
||||
def close(self) -> None:
|
||||
self._stop.set()
|
||||
with self._lock:
|
||||
self._drop_sock()
|
||||
@@ -85,10 +85,28 @@ class PoseSoundBridge:
|
||||
self._client = SimpleUDPClient(sclang_host, sclang_port)
|
||||
# Broadcast secondaire vers AV-Live-Body (Swift) pour overlay
|
||||
# skeleton dans la fenetre RealityKit. Silent si pas connecte.
|
||||
self._avbody = SimpleUDPClient("127.0.0.1", 57126)
|
||||
import os as _os
|
||||
_avbody_host = _os.environ.get("AVBODY_HOST", "127.0.0.1")
|
||||
self._avbody = SimpleUDPClient(_avbody_host, 57126)
|
||||
# Optional VDMX (VJ) OSC mirror. Off by default; when VDMX_OSC_HOST is
|
||||
# set, the VJ-useful pose signals (hands/kin/center/action/count) are
|
||||
# forwarded to VDMX's OSC input so the body can drive any VDMX param.
|
||||
_vdmx_host = _os.environ.get("VDMX_OSC_HOST")
|
||||
self._vdmx = (
|
||||
SimpleUDPClient(_vdmx_host, int(_os.environ.get("VDMX_OSC_PORT", "1234")))
|
||||
if _vdmx_host else None
|
||||
)
|
||||
self._period = 1.0 / max(1.0, throttle_hz)
|
||||
self._last_t = 0.0
|
||||
|
||||
def _vj(self, addr: str, args: list) -> None:
|
||||
"""Mirror a pose message to VDMX if configured (silent if not)."""
|
||||
if getattr(self, "_vdmx", None) is not None:
|
||||
try:
|
||||
self._vdmx.send_message(addr, args)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def send(self, persons_body: list, persons_body_ids: list, t_now: float,
|
||||
*,
|
||||
persons_face: Sequence[Sequence[Any]] | None = None,
|
||||
@@ -109,6 +127,7 @@ class PoseSoundBridge:
|
||||
self._client.send_message("/pose/count", [int(n)])
|
||||
try: self._avbody.send_message("/pose/count", [int(n)])
|
||||
except OSError: pass
|
||||
self._vj("/pose/count", [int(n)])
|
||||
except OSError:
|
||||
return # SC pas la, on continue silencieusement
|
||||
|
||||
@@ -139,6 +158,7 @@ class PoseSoundBridge:
|
||||
cli.send_message("/pose/center", [pid, float(cx), float(cy)])
|
||||
try: self._avbody.send_message("/pose/center", [pid, float(cx), float(cy)])
|
||||
except OSError: pass
|
||||
self._vj("/pose/center", [pid, float(cx), float(cy)])
|
||||
|
||||
# Nez (visage) — important pour piloter une voix
|
||||
if len(body) > NOSE and body[NOSE].c > 0.3:
|
||||
@@ -174,6 +194,17 @@ class PoseSoundBridge:
|
||||
try: self._avbody.send_message("/pose/limb_span", [pid, float(span)])
|
||||
except OSError: pass
|
||||
|
||||
if len(body) >= 29:
|
||||
def _xy(i):
|
||||
kp = body[i]
|
||||
return [float(getattr(kp, "x", 0.0)), float(getattr(kp, "y", 0.0))]
|
||||
skel = [pid]
|
||||
for idx in (0, 11, 12, 23, 24, 25, 26, 27, 28): # nose, sh L/R, hip L/R, knee L/R, ank L/R
|
||||
skel += _xy(idx)
|
||||
cli.send_message("/pose/skel", skel)
|
||||
try: self._avbody.send_message("/pose/skel", skel)
|
||||
except OSError: pass
|
||||
|
||||
# ------------------------------------------------------------------
|
||||
def send_face(self, persons_face: Sequence[Sequence[Any]],
|
||||
persons_face_ids: Sequence[int], t_now: float,
|
||||
@@ -305,6 +336,7 @@ class PoseSoundBridge:
|
||||
return
|
||||
p = [float(probs[0]), float(probs[1]), float(probs[2])]
|
||||
self._client.send_message("/pose/action", [int(pid), int(label_idx), *p])
|
||||
self._vj("/pose/action", [int(pid), int(label_idx), *p])
|
||||
|
||||
def send_kin(self, pid: int, kin,
|
||||
t_now: float, force: bool = False) -> None:
|
||||
@@ -314,10 +346,55 @@ class PoseSoundBridge:
|
||||
"""
|
||||
if not force and (t_now - self._last_t) < self._period:
|
||||
return
|
||||
self._client.send_message(
|
||||
"/pose/kin",
|
||||
[int(pid), float(kin[0]), float(kin[1]), float(kin[2])],
|
||||
)
|
||||
kin_args = [int(pid), float(kin[0]), float(kin[1]), float(kin[2])]
|
||||
self._client.send_message("/pose/kin", kin_args)
|
||||
self._vj("/pose/kin", kin_args)
|
||||
|
||||
@staticmethod
|
||||
def _hand_slot(h):
|
||||
if not h:
|
||||
return [0.0, 0.0, 0.0, 0.0]
|
||||
return [float(h["cx"]), float(h["cy"]),
|
||||
float(h["openness"]), float(h["speed"])]
|
||||
|
||||
def send_hands(self, feats, t):
|
||||
"""Emit /pose/hands [0, lx,ly,lopen,lspeed, rx,ry,ropen,rspeed, dist]."""
|
||||
args = [0]
|
||||
args += self._hand_slot(feats.get("L"))
|
||||
args += self._hand_slot(feats.get("R"))
|
||||
args.append(float(feats.get("dist", 0.0)))
|
||||
try:
|
||||
self._client.send_message("/pose/hands", args)
|
||||
except OSError:
|
||||
pass
|
||||
self._vj("/pose/hands", args)
|
||||
|
||||
def send_finger(self, ev) -> None:
|
||||
"""Emit one air-piano strike event. Not throttled: the detector's
|
||||
refractory window already rate-limits these."""
|
||||
args = [
|
||||
0, # pid (single performer)
|
||||
int(ev.hand),
|
||||
int(ev.finger),
|
||||
float(ev.strike_speed),
|
||||
float(ev.z),
|
||||
float(ev.tipx),
|
||||
float(ev.tipy),
|
||||
]
|
||||
try:
|
||||
self._client.send_message("/pose/finger", args)
|
||||
except OSError:
|
||||
pass
|
||||
self._vj("/pose/finger", args)
|
||||
|
||||
def send_pinch(self, ev) -> None:
|
||||
"""Emit one thumb-to-finger pinch event (clip toggle trigger)."""
|
||||
args = [0, int(ev.hand), int(ev.finger)]
|
||||
try:
|
||||
self._client.send_message("/pose/pinch", args)
|
||||
except OSError:
|
||||
pass
|
||||
self._vj("/pose/pinch", args)
|
||||
|
||||
def send_enter(self, pid: int) -> None:
|
||||
"""Send lifecycle event when person enters frame."""
|
||||
|
||||
@@ -73,6 +73,10 @@ multihmr = [
|
||||
# via git submodule third_party/SMPLer-X (fork electron-rare).
|
||||
# mmcv-lite suffit pour Config (le repo vendorise sa propre mmpose).
|
||||
# Le detector mmdet est remplace par YOLO Ultralytics (extras pose).
|
||||
iphone-usb = [
|
||||
"av>=12.0",
|
||||
"opencv-python>=4.10",
|
||||
]
|
||||
smplerx = [
|
||||
"torch>=2.4",
|
||||
"torchvision>=0.19",
|
||||
|
||||
@@ -119,8 +119,8 @@ MESH_MAX_VERTS = 10475 # SMPL-X is the larger family; SMPL (6890) fits inside
|
||||
# de 16, regle Metal). On y stocke (time, rms, kp_norm, netz_dev,
|
||||
# lightning_flash, flare, wind_norm, bz_norm, social_rate, pose_alive,
|
||||
# pose_count, width, height, viz_mode, _pad0, _pad1).
|
||||
UNIFORM_FLOATS = 20 # +4 floats : hand_l_x/y, hand_r_x/y
|
||||
UNIFORM_SIZE = UNIFORM_FLOATS * 4 # 80 octets, aligne 16
|
||||
UNIFORM_FLOATS = 24 # +4 floats : hand_l_x/y, hand_r_x/y + hand_height, hand_openness, hand_speed, hand_dist
|
||||
UNIFORM_SIZE = UNIFORM_FLOATS * 4 # 96 octets, aligne 16
|
||||
|
||||
|
||||
class MetalRenderer(NSObject):
|
||||
@@ -262,6 +262,14 @@ class MetalRenderer(NSObject):
|
||||
hly = 1 - (lh_wrist.y if lh_wrist else 0.5) * 2
|
||||
hrx = (rh_wrist.x if rh_wrist else 0.5) * 2 - 1
|
||||
hry = 1 - (rh_wrist.y if rh_wrist else 0.5) * 2
|
||||
# Expressive hand features from HandFeatureExtractor
|
||||
hf = getattr(s, "hand_feats", None) or {}
|
||||
hl, hr = hf.get("L"), hf.get("R")
|
||||
ys = [h["cy"] for h in (hl, hr) if h]
|
||||
hand_height = (1.0 - min(ys)) if ys else 0.0
|
||||
hand_open = max([h["openness"] for h in (hl, hr) if h] or [0.0])
|
||||
hand_speed = max([h["speed"] for h in (hl, hr) if h] or [0.0])
|
||||
hand_dist = float(hf.get("dist", 0.0))
|
||||
uniforms = struct.pack(
|
||||
f"{UNIFORM_FLOATS}f",
|
||||
s.elapsed(), # 1
|
||||
@@ -278,8 +286,10 @@ class MetalRenderer(NSObject):
|
||||
float(s.width), # 12
|
||||
float(s.height), # 13
|
||||
float(s.viz_mode), # 14
|
||||
hlx, hly, hrx, hry, # 15-18 (mains)
|
||||
0.0, 0.0, # 19-20 pad
|
||||
hlx, hly, hrx, hry, # 15-18 wrist xy (kept)
|
||||
hand_height, hand_open, # 19-20
|
||||
hand_speed, hand_dist, # 21-22
|
||||
0.0, 0.0, # 23-24 pad (16-byte align)
|
||||
)
|
||||
n_segs = self._update_skeleton(s)
|
||||
n_tris = self._update_mesh(s)
|
||||
|
||||
@@ -14,7 +14,10 @@ import torch
|
||||
import torch.nn as nn
|
||||
|
||||
CACHE = Path.home() / ".cache" / "av-live-multihmr"
|
||||
CKPT = CACHE / "checkpoints" / "multiHMR_672_S.pt"
|
||||
_CKPT_NAME = os.environ.get("MULTIHMR_CKPT_NAME", "multiHMR_672_S.pt")
|
||||
CKPT = CACHE / "checkpoints" / _CKPT_NAME
|
||||
_OUT_NAME = os.environ.get("MULTIHMR_OUT_NAME",
|
||||
_CKPT_NAME.replace(".pt", ".mlpackage").lower())
|
||||
MULTIHMR_REPO = CACHE / "multi-hmr"
|
||||
|
||||
sys.path.insert(0, str(MULTIHMR_REPO))
|
||||
@@ -276,7 +279,8 @@ class TracedMHMR(nn.Module):
|
||||
zeros_p = torch.zeros(K_PERSONS, 3)
|
||||
zeros_s = torch.zeros(K_PERSONS)
|
||||
zeros_b = torch.zeros(K_PERSONS, 10)
|
||||
return zeros, zeros_p, zeros_s, zeros_b, zeros_b
|
||||
zeros_j = torch.zeros(K_PERSONS, 127, 3)
|
||||
return zeros, zeros_p, zeros_s, zeros_b, zeros_b, zeros_j
|
||||
v3d = torch.stack([h["v3d"] for h in humans])
|
||||
transl = torch.stack([h["transl_pelvis"] for h in humans])
|
||||
scores = torch.stack([
|
||||
@@ -285,13 +289,15 @@ class TracedMHMR(nn.Module):
|
||||
]).squeeze(-1)
|
||||
shape = torch.stack([h["shape"] for h in humans])
|
||||
expr = torch.stack([h["expression"] for h in humans])
|
||||
# NOTE: CoreML mlprogram conversion currently produces all-NaN
|
||||
# outputs for v3d and transl while PyTorch eager produces valid
|
||||
# finite values from the same trace. nan_to_num here masks the
|
||||
# symptom but yields all-zero meshes (no information). Leave
|
||||
# raw outputs and let downstream decide; investigation tracked
|
||||
# in task #2 (op-by-op bisection needed).
|
||||
return v3d, transl, scores, shape, expr
|
||||
# Joints (SMPL-X). smplx.create(use_pca=False) populates
|
||||
# output.joints of shape (B, 127, 3) which is then carried as
|
||||
# 'j3d' in smpl_layer.py:148 already in camera space (same
|
||||
# transl_up applied as v3d). The first 55 are the standard
|
||||
# SMPL-X joints (22 body + jaw + 2 eyes + 30 fingers); the
|
||||
# remaining 72 are face/landmark anchors. Downstream code can
|
||||
# slice [..., :55, :] if it only needs the skeleton.
|
||||
j3d = torch.stack([h["j3d"] for h in humans])
|
||||
return v3d, transl, scores, shape, expr, j3d
|
||||
|
||||
|
||||
wrapper = TracedMHMR(model).eval()
|
||||
@@ -306,10 +312,10 @@ example_x = torch.rand(1, 3, IMG_SIZE, IMG_SIZE)
|
||||
|
||||
print("==> Sanity forward")
|
||||
with torch.no_grad():
|
||||
v3d, transl, scores, shape, expr = wrapper(example_x, example_K)
|
||||
v3d, transl, scores, shape, expr, joints = wrapper(example_x, example_K)
|
||||
print(f" v3d: {tuple(v3d.shape)}, transl: {tuple(transl.shape)},")
|
||||
print(f" scores: {tuple(scores.shape)}, shape: {tuple(shape.shape)},")
|
||||
print(f" expr: {tuple(expr.shape)}")
|
||||
print(f" expr: {tuple(expr.shape)}, joints: {tuple(joints.shape)}")
|
||||
|
||||
print("==> torch.jit.trace")
|
||||
try:
|
||||
@@ -529,9 +535,19 @@ try:
|
||||
# teste 2026-05-14 : aucun gain sur GPU compute-bound.
|
||||
compute_precision=ct.precision.FLOAT32,
|
||||
)
|
||||
out_path = "/tmp/multihmr_full_672_s.mlpackage"
|
||||
out_path = f"/tmp/{_OUT_NAME}"
|
||||
mlmodel.save(out_path)
|
||||
print(f" CONVERT OK -> {out_path}")
|
||||
# Dump output names + shapes so we can wire OUT_* constants.
|
||||
try:
|
||||
spec = mlmodel.get_spec()
|
||||
print("==> mlpackage outputs:")
|
||||
for o in spec.description.output:
|
||||
mt = o.type.multiArrayType
|
||||
shape = list(mt.shape) if mt is not None else []
|
||||
print(f" {o.name} shape={shape}")
|
||||
except Exception as e: # noqa: BLE001
|
||||
print(f" spec dump failed: {e}")
|
||||
except Exception as e:
|
||||
print(f" CONVERT FAILED: {type(e).__name__}: {e}")
|
||||
raise
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
#!/usr/bin/env python3
|
||||
"""iPhone ARBodyTracker USB skeleton -> OSC /body3d/kp bridge.
|
||||
|
||||
The ARBodyTracker iOS app serves its ARKit 91-joint skeleton over the
|
||||
device's TCP :7000 in the AVLiveWire framing (it does NOT publish OSC).
|
||||
AVLiveBody (Swift) consumes that over usbmuxd; data_only_viz instead
|
||||
listens for OSC /body3d/kp on UDP :57128 and fuses it via the `arkit_fuse`
|
||||
POSE_FILTER stage. This script is the missing link: it tunnels to the
|
||||
device's :7000 through usbmuxd, demuxes the AVLiveWire stream, decodes the
|
||||
skeleton frames, and republishes each valid joint as OSC.
|
||||
|
||||
Run it on the Mac the iPhone is plugged into, alongside:
|
||||
POSE_FILTER=median+kalman+lookahead+ik+arkit_fuse \
|
||||
python -m data_only_viz.main --pose
|
||||
|
||||
Wire formats (reverse-engineered from shared/AVLiveWire + avlivebody-mac):
|
||||
- usbmux packet: 16-byte LE header (length=16+body, version=1, message=8,
|
||||
tag) + XML plist. ListDevices -> DeviceList[].DeviceID; Connect with
|
||||
PortNumber byte-swapped to big-endian, Number==0 means success.
|
||||
- AVLiveWire frame header (19 B, big-endian): magic "AVL1", tag u8
|
||||
(skeleton=1), pid i16, timestamp f64, length u32.
|
||||
- SkeletonPayload (1183 B): 91 joints * (x,y,z) big-endian f32 interleaved,
|
||||
then 91 validity bytes.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import plistlib
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import time
|
||||
|
||||
from pythonosc.udp_client import SimpleUDPClient
|
||||
|
||||
USBMUXD_PATH = "/var/run/usbmuxd"
|
||||
DEVICE_PORT = 7000
|
||||
OSC_HOST = "127.0.0.1"
|
||||
OSC_PORT = 57128
|
||||
|
||||
MAGIC = b"AVL1"
|
||||
HEADER_LEN = 19
|
||||
TAG_SKELETON = 1
|
||||
JOINT_COUNT = 91
|
||||
SKEL_FLOAT_BYTES = JOINT_COUNT * 3 * 4 # 1092
|
||||
SKEL_BYTES = SKEL_FLOAT_BYTES + JOINT_COUNT # 1183
|
||||
MAX_PAYLOAD = 8 * 1024 * 1024
|
||||
|
||||
LOG = logging.getLogger("iphone_usb_bridge")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# usbmux client
|
||||
# --------------------------------------------------------------------------
|
||||
def _recv_exact(sock: socket.socket, n: int) -> bytes | None:
|
||||
buf = bytearray()
|
||||
while len(buf) < n:
|
||||
chunk = sock.recv(n - len(buf))
|
||||
if not chunk:
|
||||
return None
|
||||
buf += chunk
|
||||
return bytes(buf)
|
||||
|
||||
|
||||
def _usbmux_send(sock: socket.socket, plist: dict, tag: int) -> None:
|
||||
body = plistlib.dumps(plist, fmt=plistlib.FMT_XML)
|
||||
sock.sendall(struct.pack("<IIII", 16 + len(body), 1, 8, tag) + body)
|
||||
|
||||
|
||||
def _usbmux_recv(sock: socket.socket) -> dict | None:
|
||||
head = _recv_exact(sock, 4)
|
||||
if head is None:
|
||||
return None
|
||||
(length,) = struct.unpack("<I", head)
|
||||
if length < 16:
|
||||
return None
|
||||
rest = _recv_exact(sock, length - 4)
|
||||
if rest is None:
|
||||
return None
|
||||
packet = head + rest # full `length` bytes
|
||||
return plistlib.loads(packet[16:]) # plist body after 16-byte header
|
||||
|
||||
|
||||
def connect_device() -> socket.socket | None:
|
||||
"""usbmux ListDevices + Connect to the first device's :7000.
|
||||
|
||||
On success returns a socket whose stream is the tunneled device bytes.
|
||||
"""
|
||||
try:
|
||||
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
sock.connect(USBMUXD_PATH)
|
||||
except OSError as e:
|
||||
LOG.warning("no usbmuxd (%s)", e)
|
||||
return None
|
||||
try:
|
||||
_usbmux_send(sock, {"MessageType": "ListDevices"}, 1)
|
||||
reply = _usbmux_recv(sock)
|
||||
devices = [d.get("DeviceID") for d in (reply or {}).get("DeviceList", [])]
|
||||
devices = [d for d in devices if isinstance(d, int)]
|
||||
if not devices:
|
||||
LOG.warning("no iOS device attached")
|
||||
sock.close()
|
||||
return None
|
||||
device_id = devices[0]
|
||||
swapped = ((DEVICE_PORT << 8) | (DEVICE_PORT >> 8)) & 0xFFFF
|
||||
_usbmux_send(
|
||||
sock,
|
||||
{"MessageType": "Connect", "DeviceID": device_id,
|
||||
"PortNumber": swapped},
|
||||
2,
|
||||
)
|
||||
reply = _usbmux_recv(sock)
|
||||
if not reply or reply.get("Number") != 0:
|
||||
LOG.warning("Connect to :%d refused (%s)", DEVICE_PORT, reply)
|
||||
sock.close()
|
||||
return None
|
||||
LOG.info("connected to device %d port %d", device_id, DEVICE_PORT)
|
||||
return sock
|
||||
except OSError as e:
|
||||
LOG.warning("usbmux handshake failed: %s", e)
|
||||
sock.close()
|
||||
return None
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# AVLiveWire stream demux + skeleton decode
|
||||
# --------------------------------------------------------------------------
|
||||
def iter_frames(sock: socket.socket):
|
||||
"""Yield (tag, pid, payload) for each complete AVLiveWire frame."""
|
||||
buf = bytearray()
|
||||
while True:
|
||||
chunk = sock.recv(65536)
|
||||
if not chunk:
|
||||
return
|
||||
buf += chunk
|
||||
while True:
|
||||
idx = buf.find(MAGIC)
|
||||
if idx < 0:
|
||||
if len(buf) > 3: # keep a partial trailing magic
|
||||
del buf[:-3]
|
||||
break
|
||||
if idx > 0:
|
||||
del buf[:idx]
|
||||
if len(buf) < HEADER_LEN:
|
||||
break
|
||||
tag = buf[4]
|
||||
(length,) = struct.unpack(">I", buf[15:19])
|
||||
if length > MAX_PAYLOAD: # corrupt header, skip the magic
|
||||
del buf[:len(MAGIC)]
|
||||
continue
|
||||
total = HEADER_LEN + length
|
||||
if len(buf) < total:
|
||||
break
|
||||
(pid,) = struct.unpack(">h", buf[5:7])
|
||||
payload = bytes(buf[HEADER_LEN:total])
|
||||
del buf[:total]
|
||||
yield tag, pid, payload
|
||||
|
||||
|
||||
def decode_skeleton(payload: bytes):
|
||||
"""Return [(x, y, z, valid), ...] of length 91, or None if malformed."""
|
||||
if len(payload) != SKEL_BYTES:
|
||||
return None
|
||||
floats = struct.unpack(">" + "f" * (JOINT_COUNT * 3), payload[:SKEL_FLOAT_BYTES])
|
||||
valid = payload[SKEL_FLOAT_BYTES:]
|
||||
return [
|
||||
(floats[i * 3], floats[i * 3 + 1], floats[i * 3 + 2], valid[i] != 0)
|
||||
for i in range(JOINT_COUNT)
|
||||
]
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# main loop
|
||||
# --------------------------------------------------------------------------
|
||||
def main() -> int:
|
||||
logging.basicConfig(
|
||||
level=logging.INFO,
|
||||
format="%(asctime)s %(levelname)s %(name)s — %(message)s",
|
||||
)
|
||||
osc = SimpleUDPClient(OSC_HOST, OSC_PORT)
|
||||
LOG.info("iphone_usb_bridge: usbmuxd :%d -> OSC %s:%d",
|
||||
DEVICE_PORT, OSC_HOST, OSC_PORT)
|
||||
frames = 0
|
||||
last_log = time.monotonic()
|
||||
while True:
|
||||
sock = connect_device()
|
||||
if sock is None:
|
||||
time.sleep(1.0)
|
||||
continue
|
||||
try:
|
||||
for tag, pid, payload in iter_frames(sock):
|
||||
if tag != TAG_SKELETON:
|
||||
continue
|
||||
joints = decode_skeleton(payload)
|
||||
if joints is None:
|
||||
continue
|
||||
n = 0
|
||||
for i, (x, y, z, v) in enumerate(joints):
|
||||
if v:
|
||||
osc.send_message("/body3d/kp", [pid, i, x, y, z])
|
||||
n += 1
|
||||
osc.send_message("/body3d/count", [1 if n else 0])
|
||||
frames += 1
|
||||
now = time.monotonic()
|
||||
if now - last_log > 3.0:
|
||||
LOG.info("forwarding skeleton frames (last: pid=%d, %d/%d valid joints)",
|
||||
pid, n, JOINT_COUNT)
|
||||
last_log = now
|
||||
except (OSError, struct.error) as e:
|
||||
LOG.warning("stream error: %s — reconnecting", e)
|
||||
finally:
|
||||
try:
|
||||
sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
time.sleep(1.0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -0,0 +1,607 @@
|
||||
"""Multi-HMR inference server (TCP, coremltools backend).
|
||||
|
||||
Runs on a remote Mac (macm1 in the AV-Live cluster), loads the
|
||||
mlpackage via coremltools (Python 3.12), and serves frames over TCP.
|
||||
|
||||
Protocol (little-endian, persistent connection):
|
||||
|
||||
Request:
|
||||
[4 bytes uint32 payload_len]
|
||||
[4 bytes magic "REQ\\x01"]
|
||||
[1 byte uint8 format_id] # 1 = raw RGB uint8 HWC, 2 = JPEG
|
||||
[3 bytes padding]
|
||||
[variable image bytes] # IMG_BYTES if format=1, else JPEG bytes
|
||||
[9 float32 LE = 36 bytes K] # always last 36 bytes
|
||||
|
||||
Response:
|
||||
[4 bytes uint32 payload_len]
|
||||
[4 bytes magic "RSP\\x01"]
|
||||
[4 bytes int32 status] # 0 = OK, 1 = error
|
||||
[v3d: 4*10475*3 float32]
|
||||
[transl: 4*1*3 float32]
|
||||
[scores: 4 float32]
|
||||
[betas: 4*10 float32]
|
||||
[expr: 4*10 float32]
|
||||
|
||||
Connection handler runs a 3-thread pipeline: reader -> worker -> writer.
|
||||
While the worker predicts frame N, the reader has already buffered frame
|
||||
N+1 so the next predict can start the instant the previous response is
|
||||
handed to the writer. Queue depth is 2 to absorb network jitter.
|
||||
|
||||
Bench mode (--bench): synthetic frames against the loaded backend.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import queue
|
||||
import signal
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
LOG = logging.getLogger("multihmr_server")
|
||||
|
||||
IMG_SIZE = 672
|
||||
N_PERSONS_FIXED = 4
|
||||
N_VERTS = 10475
|
||||
|
||||
MAGIC_REQ = b"REQ\x01"
|
||||
MAGIC_RSP = b"RSP\x01"
|
||||
|
||||
FORMAT_RAW = 1
|
||||
FORMAT_JPEG = 2
|
||||
|
||||
IMG_BYTES = IMG_SIZE * IMG_SIZE * 3 # 1_354_752
|
||||
K_BYTES = 9 * 4 # 36
|
||||
REQ_HEADER = 4 + 1 + 3 # magic + fmt u8 + 3 pad
|
||||
|
||||
V3D_BYTES = N_PERSONS_FIXED * N_VERTS * 3 * 4
|
||||
TRANSL_BYTES = N_PERSONS_FIXED * 1 * 3 * 4
|
||||
SCORES_BYTES = N_PERSONS_FIXED * 4
|
||||
BETAS_BYTES = N_PERSONS_FIXED * 10 * 4
|
||||
EXPR_BYTES = N_PERSONS_FIXED * 10 * 4
|
||||
RSP_HEADER = 4 + 4
|
||||
RSP_PAYLOAD_LEN = (RSP_HEADER + V3D_BYTES + TRANSL_BYTES
|
||||
+ SCORES_BYTES + BETAS_BYTES + EXPR_BYTES)
|
||||
|
||||
|
||||
DEFAULT_MLPACKAGE = Path(
|
||||
os.environ.get("MULTIHMR_MLPACKAGE")
|
||||
or str(Path.home() / ".cache" / "av-live-multihmr"
|
||||
/ "multihmr_full_672_s.mlpackage"))
|
||||
|
||||
OUT_V3D = "var_2420"
|
||||
OUT_TRANSL = "var_2423"
|
||||
OUT_SCORES = "var_2436"
|
||||
OUT_BETAS = "var_2439"
|
||||
OUT_EXPR = "var_2442"
|
||||
OUT_JOINTS = "var_2445" # (4, 127, 3) SMPL-X joints incl fingers
|
||||
N_JOINTS = 127
|
||||
|
||||
|
||||
def recv_exact(sock: socket.socket, n: int) -> bytes:
|
||||
buf = bytearray(n)
|
||||
view = memoryview(buf)
|
||||
pos = 0
|
||||
while pos < n:
|
||||
got = sock.recv_into(view[pos:])
|
||||
if got == 0:
|
||||
raise ConnectionError("peer closed")
|
||||
pos += got
|
||||
return bytes(buf)
|
||||
|
||||
|
||||
def encode_response(v3d: np.ndarray, transl: np.ndarray,
|
||||
scores: np.ndarray, betas: np.ndarray,
|
||||
expr: np.ndarray, status: int = 0) -> bytes:
|
||||
parts = [
|
||||
struct.pack("<I", RSP_PAYLOAD_LEN),
|
||||
MAGIC_RSP,
|
||||
struct.pack("<i", status),
|
||||
np.ascontiguousarray(v3d, dtype=np.float32).tobytes(),
|
||||
np.ascontiguousarray(transl, dtype=np.float32).tobytes(),
|
||||
np.ascontiguousarray(scores, dtype=np.float32).tobytes(),
|
||||
np.ascontiguousarray(betas, dtype=np.float32).tobytes(),
|
||||
np.ascontiguousarray(expr, dtype=np.float32).tobytes(),
|
||||
]
|
||||
return b"".join(parts)
|
||||
|
||||
|
||||
def decode_request(payload: bytes) -> tuple[np.ndarray, np.ndarray, float]:
|
||||
"""Decode a request payload (without the leading 4-byte length).
|
||||
|
||||
Returns (image_uint8_hwc, K_33_f32, decode_ms_overhead).
|
||||
"""
|
||||
if len(payload) < REQ_HEADER + K_BYTES:
|
||||
raise ValueError(f"req payload too short: {len(payload)}")
|
||||
magic = payload[:4]
|
||||
if magic != MAGIC_REQ:
|
||||
raise ValueError(f"bad magic {magic!r}")
|
||||
fmt = payload[4]
|
||||
# payload[5:8] reserved.
|
||||
img_end = len(payload) - K_BYTES
|
||||
img_bytes = payload[REQ_HEADER:img_end]
|
||||
K = np.frombuffer(payload, dtype="<f4", count=9,
|
||||
offset=img_end).reshape(3, 3).astype(np.float32)
|
||||
t0 = time.monotonic()
|
||||
if fmt == FORMAT_RAW:
|
||||
if len(img_bytes) != IMG_BYTES:
|
||||
raise ValueError(
|
||||
f"raw img bytes {len(img_bytes)} != {IMG_BYTES}")
|
||||
img = np.frombuffer(img_bytes, dtype=np.uint8).reshape(
|
||||
IMG_SIZE, IMG_SIZE, 3)
|
||||
elif fmt == FORMAT_JPEG:
|
||||
import cv2
|
||||
arr = np.frombuffer(img_bytes, dtype=np.uint8)
|
||||
bgr = cv2.imdecode(arr, cv2.IMREAD_COLOR)
|
||||
if bgr is None:
|
||||
raise ValueError("cv2.imdecode failed")
|
||||
if bgr.shape[:2] != (IMG_SIZE, IMG_SIZE):
|
||||
bgr = cv2.resize(bgr, (IMG_SIZE, IMG_SIZE))
|
||||
img = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
|
||||
else:
|
||||
raise ValueError(f"unknown format_id {fmt}")
|
||||
decode_ms = (time.monotonic() - t0) * 1e3
|
||||
return img, K, decode_ms
|
||||
|
||||
|
||||
ML_DTYPE_FLOAT16 = 65552
|
||||
ML_DTYPE_FLOAT32 = 65568
|
||||
ML_DTYPE_DOUBLE = 65600
|
||||
|
||||
|
||||
def _np_to_mlarray(arr: np.ndarray, MLMultiArray):
|
||||
"""Create a contiguous float32 MLMultiArray from a numpy array."""
|
||||
import ctypes
|
||||
arr = np.ascontiguousarray(arr, dtype=np.float32)
|
||||
shape = [int(s) for s in arr.shape]
|
||||
ml = MLMultiArray.alloc().initWithShape_dataType_error_(
|
||||
shape, ML_DTYPE_FLOAT32, None)
|
||||
if ml is None:
|
||||
raise RuntimeError("MLMultiArray alloc failed")
|
||||
ptr = ml.dataPointer()
|
||||
addr = int(ptr) if isinstance(ptr, int) else ctypes.cast(
|
||||
ptr, ctypes.c_void_p).value
|
||||
if addr is None:
|
||||
raise RuntimeError("MLMultiArray dataPointer null")
|
||||
ctypes.memmove(addr, arr.ctypes.data, arr.nbytes)
|
||||
return ml
|
||||
|
||||
|
||||
def _mlarray_to_np(ml) -> np.ndarray:
|
||||
"""Copy an MLMultiArray (FLOAT16/32/64) to numpy float32."""
|
||||
import ctypes
|
||||
shape = tuple(int(s) for s in ml.shape())
|
||||
dtype_id = int(ml.dataType())
|
||||
count = 1
|
||||
for s in shape:
|
||||
count *= s
|
||||
ptr = ml.dataPointer()
|
||||
addr = int(ptr) if isinstance(ptr, int) else ctypes.cast(
|
||||
ptr, ctypes.c_void_p).value
|
||||
if addr is None:
|
||||
raise RuntimeError("MLMultiArray dataPointer null")
|
||||
if dtype_id == ML_DTYPE_FLOAT16:
|
||||
raw = (ctypes.c_uint16 * count).from_address(addr)
|
||||
arr = np.ctypeslib.as_array(raw).view(np.float16).astype(np.float32)
|
||||
elif dtype_id == ML_DTYPE_FLOAT32:
|
||||
raw = (ctypes.c_float * count).from_address(addr)
|
||||
arr = np.ctypeslib.as_array(raw).copy()
|
||||
elif dtype_id == ML_DTYPE_DOUBLE:
|
||||
raw = (ctypes.c_double * count).from_address(addr)
|
||||
arr = np.ctypeslib.as_array(raw).astype(np.float32)
|
||||
else:
|
||||
raise RuntimeError(f"unsupported MLMultiArray dtype {dtype_id}")
|
||||
return arr.reshape(shape)
|
||||
|
||||
|
||||
class CoreMLModel:
|
||||
"""pyobjc-direct CoreML wrapper. Drops the ~30 ms coremltools.MLModel.predict
|
||||
overhead by using CoreML.framework directly (MLDictionaryFeatureProvider
|
||||
+ MLMultiArray ctypes memcpy). Fallback to coremltools if pyobjc missing,
|
||||
via MULTIHMR_SERVER_BACKEND=coremltools env."""
|
||||
|
||||
def __init__(self, mlpackage_path: Path) -> None:
|
||||
self.path = Path(mlpackage_path)
|
||||
if not self.path.exists():
|
||||
raise FileNotFoundError(f"mlpackage missing: {self.path}")
|
||||
backend = os.environ.get(
|
||||
"MULTIHMR_SERVER_BACKEND", "pyobjc").strip().lower()
|
||||
cu_env = os.environ.get(
|
||||
"COREML_COMPUTE_UNITS", "cpu_and_gpu").strip().lower()
|
||||
if backend == "pyobjc":
|
||||
self._use_pyobjc = True
|
||||
self._init_pyobjc(cu_env)
|
||||
else:
|
||||
self._use_pyobjc = False
|
||||
self._init_coremltools(cu_env)
|
||||
|
||||
def _init_pyobjc(self, cu_env: str) -> None:
|
||||
import objc
|
||||
from Foundation import NSURL
|
||||
ns: dict = {}
|
||||
objc.loadBundle("CoreML", ns,
|
||||
"/System/Library/Frameworks/CoreML.framework")
|
||||
cu_map = {"cpu_only": 0, "cpu_and_gpu": 1, "all": 2,
|
||||
"cpu_and_ne": 3}
|
||||
cu = cu_map.get(cu_env, 1)
|
||||
MLModel = ns["MLModel"]
|
||||
MLModelConfiguration = ns["MLModelConfiguration"]
|
||||
cfg = MLModelConfiguration.alloc().init()
|
||||
try:
|
||||
cfg.setComputeUnits_(cu)
|
||||
except Exception: # noqa: BLE001
|
||||
pass
|
||||
url = NSURL.fileURLWithPath_(str(self.path))
|
||||
compiled_url = MLModel.compileModelAtURL_error_(url, None)
|
||||
if compiled_url is None:
|
||||
raise RuntimeError(f"compileModelAtURL failed for {self.path}")
|
||||
model = MLModel.modelWithContentsOfURL_configuration_error_(
|
||||
compiled_url, cfg, None)
|
||||
if model is None:
|
||||
raise RuntimeError(f"MLModel load failed for {compiled_url}")
|
||||
self._model = model
|
||||
self._ns = ns
|
||||
LOG.info("loading mlpackage %s via pyobjc (computeUnit=%s)",
|
||||
self.path.name, cu_env)
|
||||
|
||||
def _init_coremltools(self, cu_env: str) -> None:
|
||||
import coremltools as ct
|
||||
from coremltools.models import MLModel as CTMLModel
|
||||
cu_map = {
|
||||
"cpu_only": ct.ComputeUnit.CPU_ONLY,
|
||||
"cpu_and_gpu": ct.ComputeUnit.CPU_AND_GPU,
|
||||
"all": ct.ComputeUnit.ALL,
|
||||
"cpu_and_ne": ct.ComputeUnit.CPU_AND_NE,
|
||||
}
|
||||
cu = cu_map.get(cu_env, ct.ComputeUnit.CPU_AND_GPU)
|
||||
LOG.info("loading mlpackage %s via coremltools (computeUnit=%s)",
|
||||
self.path.name, cu_env)
|
||||
self.model = CTMLModel(str(self.path), compute_units=cu)
|
||||
|
||||
def predict(self, image_uint8_hwc: np.ndarray, K_33: np.ndarray
|
||||
) -> dict[str, np.ndarray]:
|
||||
img_chw = image_uint8_hwc.transpose(2, 0, 1).astype(np.float32) / 255.0
|
||||
img4 = img_chw[np.newaxis, ...]
|
||||
K = K_33.astype(np.float32)
|
||||
if K.ndim == 2:
|
||||
K = K[np.newaxis, ...]
|
||||
if self._use_pyobjc:
|
||||
return self._predict_pyobjc(img4, K)
|
||||
return self.model.predict({"image": img4, "cam_K": K})
|
||||
|
||||
def _predict_pyobjc(self, image_4d: np.ndarray, K_33: np.ndarray
|
||||
) -> dict[str, np.ndarray]:
|
||||
ns = self._ns
|
||||
MLMultiArray = ns["MLMultiArray"]
|
||||
MLDictionaryFeatureProvider = ns["MLDictionaryFeatureProvider"]
|
||||
MLFeatureValue = ns["MLFeatureValue"]
|
||||
img_ml = _np_to_mlarray(image_4d, MLMultiArray)
|
||||
k_ml = _np_to_mlarray(K_33, MLMultiArray)
|
||||
feats = {
|
||||
"image": MLFeatureValue.featureValueWithMultiArray_(img_ml),
|
||||
"cam_K": MLFeatureValue.featureValueWithMultiArray_(k_ml),
|
||||
}
|
||||
provider = MLDictionaryFeatureProvider.alloc(
|
||||
).initWithDictionary_error_(feats, None)
|
||||
if provider is None:
|
||||
raise RuntimeError("MLDictionaryFeatureProvider alloc failed")
|
||||
out = self._model.predictionFromFeatures_error_(provider, None)
|
||||
if out is None:
|
||||
raise RuntimeError("MLModel predict failed")
|
||||
names = [str(n) for n in out.featureNames()]
|
||||
result: dict[str, np.ndarray] = {}
|
||||
for n in names:
|
||||
fv = out.featureValueForName_(n)
|
||||
if fv is None:
|
||||
continue
|
||||
ml = fv.multiArrayValue()
|
||||
if ml is None:
|
||||
continue
|
||||
result[n] = _mlarray_to_np(ml)
|
||||
return result
|
||||
|
||||
|
||||
def _zero_outputs() -> tuple[np.ndarray, ...]:
|
||||
return (
|
||||
np.zeros((N_PERSONS_FIXED, N_VERTS, 3), dtype=np.float32),
|
||||
np.zeros((N_PERSONS_FIXED, 1, 3), dtype=np.float32),
|
||||
np.zeros((N_PERSONS_FIXED,), dtype=np.float32),
|
||||
np.zeros((N_PERSONS_FIXED, 10), dtype=np.float32),
|
||||
np.zeros((N_PERSONS_FIXED, 10), dtype=np.float32),
|
||||
)
|
||||
|
||||
|
||||
def _extract_outputs(raw: dict[str, np.ndarray]
|
||||
) -> tuple[np.ndarray, ...]:
|
||||
v3d = np.asarray(raw[OUT_V3D], dtype=np.float32).reshape(
|
||||
N_PERSONS_FIXED, N_VERTS, 3)
|
||||
transl = np.asarray(raw[OUT_TRANSL], dtype=np.float32).reshape(
|
||||
N_PERSONS_FIXED, 1, 3)
|
||||
scores = np.asarray(raw[OUT_SCORES], dtype=np.float32).reshape(
|
||||
N_PERSONS_FIXED)
|
||||
betas = np.asarray(raw[OUT_BETAS], dtype=np.float32).reshape(
|
||||
N_PERSONS_FIXED, 10)
|
||||
expr = np.asarray(raw[OUT_EXPR], dtype=np.float32).reshape(
|
||||
N_PERSONS_FIXED, 10)
|
||||
return v3d, transl, scores, betas, expr
|
||||
|
||||
|
||||
class Server:
|
||||
def __init__(self, model: CoreMLModel, host: str, port: int) -> None:
|
||||
self.model = model
|
||||
self.host = host
|
||||
self.port = port
|
||||
self._stop = threading.Event()
|
||||
self._sock: socket.socket | None = None
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
if self._sock is not None:
|
||||
try:
|
||||
self._sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
def serve(self) -> None:
|
||||
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
sock.bind((self.host, self.port))
|
||||
sock.listen(4)
|
||||
sock.settimeout(1.0)
|
||||
self._sock = sock
|
||||
LOG.info("listening %s:%d", self.host, self.port)
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
conn, addr = sock.accept()
|
||||
except socket.timeout:
|
||||
continue
|
||||
except OSError:
|
||||
break
|
||||
LOG.info("client connected %s", addr)
|
||||
try:
|
||||
self._handle_pipelined(conn)
|
||||
except (ConnectionError, BrokenPipeError, OSError) as e:
|
||||
LOG.info("client disconnected: %s", e)
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except OSError:
|
||||
pass
|
||||
LOG.info("server stopped")
|
||||
|
||||
# -- pipelined per-connection handler -----------------------------
|
||||
|
||||
def _handle_pipelined(self, conn: socket.socket) -> None:
|
||||
conn.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
|
||||
conn_stop = threading.Event()
|
||||
|
||||
# raw requests in, encoded responses out.
|
||||
req_q: queue.Queue[bytes] = queue.Queue(maxsize=2)
|
||||
rsp_q: queue.Queue[bytes] = queue.Queue(maxsize=2)
|
||||
|
||||
# stats
|
||||
served = {"n": 0, "t0": time.monotonic(),
|
||||
"sum_decode": 0.0, "sum_pred": 0.0,
|
||||
"sum_encode": 0.0}
|
||||
|
||||
def reader() -> None:
|
||||
try:
|
||||
while not conn_stop.is_set() and not self._stop.is_set():
|
||||
len_buf = recv_exact(conn, 4)
|
||||
payload_len = struct.unpack("<I", len_buf)[0]
|
||||
if payload_len > 8 * 1024 * 1024:
|
||||
raise ValueError(f"reqlen too big {payload_len}")
|
||||
payload = recv_exact(conn, payload_len)
|
||||
req_q.put(payload)
|
||||
except (ConnectionError, BrokenPipeError, OSError) as e:
|
||||
LOG.info("reader exit: %s", e)
|
||||
finally:
|
||||
conn_stop.set()
|
||||
# poison-pill the worker
|
||||
try:
|
||||
req_q.put_nowait(b"")
|
||||
except queue.Full:
|
||||
pass
|
||||
|
||||
def worker() -> None:
|
||||
try:
|
||||
while not conn_stop.is_set() and not self._stop.is_set():
|
||||
try:
|
||||
payload = req_q.get(timeout=0.5)
|
||||
except queue.Empty:
|
||||
continue
|
||||
if payload == b"":
|
||||
break
|
||||
try:
|
||||
img, K, decode_ms = decode_request(payload)
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("decode failed: %s", e)
|
||||
v3d, transl, scores, betas, expr = _zero_outputs()
|
||||
rsp_q.put(encode_response(
|
||||
v3d, transl, scores, betas, expr, status=1))
|
||||
continue
|
||||
t_pred = time.monotonic()
|
||||
try:
|
||||
raw = self.model.predict(img, K)
|
||||
v3d, transl, scores, betas, expr = _extract_outputs(
|
||||
raw)
|
||||
status = 0
|
||||
except Exception as e: # noqa: BLE001
|
||||
LOG.warning("predict failed: %s", e)
|
||||
v3d, transl, scores, betas, expr = _zero_outputs()
|
||||
status = 1
|
||||
t_pred_end = time.monotonic()
|
||||
t_enc = time.monotonic()
|
||||
rsp = encode_response(
|
||||
v3d, transl, scores, betas, expr, status=status)
|
||||
t_enc_end = time.monotonic()
|
||||
pred_ms = (t_pred_end - t_pred) * 1e3
|
||||
encode_ms = (t_enc_end - t_enc) * 1e3
|
||||
served["n"] += 1
|
||||
served["sum_decode"] += decode_ms
|
||||
served["sum_pred"] += pred_ms
|
||||
served["sum_encode"] += encode_ms
|
||||
rsp_q.put(rsp)
|
||||
now = time.monotonic()
|
||||
if served["n"] % 30 == 0:
|
||||
dt = now - served["t0"]
|
||||
fps = served["n"] / max(1e-6, dt)
|
||||
LOG.info(
|
||||
"served %d frames at %.1f fps over %.1f s "
|
||||
"(decode=%.1fms pred=%.1fms encode=%.1fms)",
|
||||
served["n"], fps, dt,
|
||||
served["sum_decode"] / served["n"],
|
||||
served["sum_pred"] / served["n"],
|
||||
served["sum_encode"] / served["n"])
|
||||
finally:
|
||||
conn_stop.set()
|
||||
try:
|
||||
rsp_q.put_nowait(b"")
|
||||
except queue.Full:
|
||||
pass
|
||||
|
||||
def writer() -> None:
|
||||
try:
|
||||
while not conn_stop.is_set() and not self._stop.is_set():
|
||||
try:
|
||||
rsp = rsp_q.get(timeout=0.5)
|
||||
except queue.Empty:
|
||||
continue
|
||||
if rsp == b"":
|
||||
break
|
||||
conn.sendall(rsp)
|
||||
except (ConnectionError, BrokenPipeError, OSError) as e:
|
||||
LOG.info("writer exit: %s", e)
|
||||
finally:
|
||||
conn_stop.set()
|
||||
|
||||
t_r = threading.Thread(target=reader, name="srv-reader", daemon=True)
|
||||
t_w = threading.Thread(target=worker, name="srv-worker", daemon=True)
|
||||
t_x = threading.Thread(target=writer, name="srv-writer", daemon=True)
|
||||
t_r.start()
|
||||
t_w.start()
|
||||
t_x.start()
|
||||
t_r.join()
|
||||
t_w.join()
|
||||
t_x.join()
|
||||
dt = time.monotonic() - served["t0"]
|
||||
if served["n"] > 0:
|
||||
LOG.info("connection closed: served %d frames at %.1f fps "
|
||||
"over %.1f s", served["n"],
|
||||
served["n"] / max(1e-6, dt), dt)
|
||||
|
||||
|
||||
def run_bench(model: CoreMLModel, n: int = 30) -> None:
|
||||
"""Local synthetic bench (no socket)."""
|
||||
rng = np.random.default_rng(0)
|
||||
K = np.array([[672.0, 0.0, 336.0],
|
||||
[0.0, 672.0, 336.0],
|
||||
[0.0, 0.0, 1.0]], dtype=np.float32)
|
||||
img0 = rng.integers(0, 256, (IMG_SIZE, IMG_SIZE, 3), dtype=np.uint8)
|
||||
model.predict(img0, K)
|
||||
times = []
|
||||
for _ in range(n):
|
||||
img = rng.integers(0, 256, (IMG_SIZE, IMG_SIZE, 3), dtype=np.uint8)
|
||||
t0 = time.monotonic()
|
||||
model.predict(img, K)
|
||||
times.append((time.monotonic() - t0) * 1e3)
|
||||
ts = sorted(times)
|
||||
median = ts[len(ts) // 2]
|
||||
mean = sum(times) / len(times)
|
||||
p90 = ts[int(len(ts) * 0.9)]
|
||||
LOG.info("bench n=%d median=%.1fms mean=%.1fms p90=%.1fms (%.1f fps)",
|
||||
n, median, mean, p90, 1000.0 / median)
|
||||
|
||||
|
||||
def run_bench_async(model: CoreMLModel, host: str, port: int,
|
||||
n: int = 60) -> None:
|
||||
"""End-to-end pipeline bench via real socket loopback."""
|
||||
import threading
|
||||
server = Server(model, host, port)
|
||||
th = threading.Thread(target=server.serve, daemon=True)
|
||||
th.start()
|
||||
time.sleep(0.5)
|
||||
try:
|
||||
from data_only_viz.multihmr_remote import MultiHMRRemoteBackend
|
||||
except ImportError:
|
||||
# When the server runs standalone, the client package may not be
|
||||
# importable. Skip with a friendly message.
|
||||
LOG.warning("data_only_viz package not importable on this host, "
|
||||
"skipping --bench-async")
|
||||
server.stop()
|
||||
return
|
||||
os.environ.setdefault("MULTIHMR_REMOTE_HOST", host)
|
||||
os.environ.setdefault("MULTIHMR_REMOTE_PORT", str(port))
|
||||
be = MultiHMRRemoteBackend(host=host, port=port)
|
||||
rng = np.random.default_rng(0)
|
||||
K = np.array([[672.0, 0.0, 336.0],
|
||||
[0.0, 672.0, 336.0],
|
||||
[0.0, 0.0, 1.0]], dtype=np.float32)
|
||||
t0 = time.monotonic()
|
||||
got = 0
|
||||
for _ in range(n):
|
||||
img = (rng.random((3, IMG_SIZE, IMG_SIZE), dtype=np.float32))
|
||||
out = be.infer(img, K)
|
||||
if out is not None:
|
||||
got += 1
|
||||
time.sleep(0.01)
|
||||
dt = time.monotonic() - t0
|
||||
LOG.info("bench-async submitted=%d got=%d in %.2fs (%.1f fps submit)",
|
||||
n, got, dt, n / max(1e-6, dt))
|
||||
be.close()
|
||||
server.stop()
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
ap = argparse.ArgumentParser(description="Multi-HMR TCP server")
|
||||
ap.add_argument("--mlpackage", type=Path, default=DEFAULT_MLPACKAGE)
|
||||
ap.add_argument("--host", default=os.environ.get(
|
||||
"MULTIHMR_SERVER_HOST", "0.0.0.0"))
|
||||
ap.add_argument("--port", type=int, default=int(os.environ.get(
|
||||
"MULTIHMR_SERVER_PORT", "57140")))
|
||||
ap.add_argument("--bench", action="store_true",
|
||||
help="local synthetic bench, no socket")
|
||||
ap.add_argument("--bench-async", action="store_true",
|
||||
help="loopback pipeline bench through real sockets")
|
||||
ap.add_argument("--bench-n", type=int, default=30)
|
||||
ap.add_argument("--log-level", default="INFO")
|
||||
args = ap.parse_args(argv)
|
||||
|
||||
logging.basicConfig(
|
||||
level=args.log_level.upper(),
|
||||
format="%(asctime)s %(levelname)s %(name)s %(message)s")
|
||||
|
||||
model = CoreMLModel(args.mlpackage)
|
||||
|
||||
if args.bench:
|
||||
run_bench(model, n=args.bench_n)
|
||||
return 0
|
||||
if args.bench_async:
|
||||
run_bench_async(model, "127.0.0.1", args.port, n=args.bench_n)
|
||||
return 0
|
||||
|
||||
server = Server(model, args.host, args.port)
|
||||
|
||||
def _sigint(*_a):
|
||||
LOG.info("SIGINT received, stopping")
|
||||
server.stop()
|
||||
|
||||
signal.signal(signal.SIGINT, _sigint)
|
||||
signal.signal(signal.SIGTERM, _sigint)
|
||||
try:
|
||||
server.serve()
|
||||
except KeyboardInterrupt:
|
||||
server.stop()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# Push the Multi-HMR mlpackage + server to macm1 (M1 Max, 32-core GPU)
|
||||
# and launch the inference server in the background.
|
||||
#
|
||||
# Prereqs on macm1 :
|
||||
# * passwordless ssh (Tailscale alias 'macm1' or LAN)
|
||||
# * uv installed
|
||||
# * Python 3.12 available via uv (uv pulls it)
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/setup_remote_macm1.sh
|
||||
# MACM1_HOST=clems@192.168.0.175 ./scripts/setup_remote_macm1.sh
|
||||
set -euo pipefail
|
||||
|
||||
HOST="${MACM1_HOST:-macm1}"
|
||||
PORT="${MULTIHMR_SERVER_PORT:-57140}"
|
||||
MLPACKAGE_LOCAL="${MLPACKAGE_LOCAL:-$HOME/.cache/av-live-multihmr/multihmr_full_672_s.mlpackage}"
|
||||
REMOTE_TMP="/tmp/av-live-multihmr"
|
||||
REMOTE_VENV="/tmp/av-live-multihmr/venv"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
echo "==> Target host : $HOST"
|
||||
echo "==> mlpackage : $MLPACKAGE_LOCAL"
|
||||
|
||||
if [ ! -d "$MLPACKAGE_LOCAL" ]; then
|
||||
echo "ERROR: mlpackage missing at $MLPACKAGE_LOCAL" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "==> Creating remote tmp dir"
|
||||
ssh "$HOST" "mkdir -p $REMOTE_TMP"
|
||||
|
||||
echo "==> rsync mlpackage (~70 MB, may take a moment first time)"
|
||||
rsync -a --delete \
|
||||
"$MLPACKAGE_LOCAL/" \
|
||||
"$HOST:$REMOTE_TMP/multihmr_full_672_s.mlpackage/"
|
||||
|
||||
echo "==> rsync multihmr_server.py"
|
||||
rsync -a "$SCRIPT_DIR/multihmr_server.py" \
|
||||
"$HOST:$REMOTE_TMP/multihmr_server.py"
|
||||
|
||||
echo "==> Provision Python 3.12 venv with uv (idempotent)"
|
||||
ssh "$HOST" "bash -lc 'set -e
|
||||
if [ ! -x $REMOTE_VENV/bin/python ]; then
|
||||
uv venv --python 3.12 $REMOTE_VENV --quiet
|
||||
fi
|
||||
uv pip install --python $REMOTE_VENV/bin/python --quiet \
|
||||
coremltools numpy opencv-python-headless \
|
||||
pyobjc-core pyobjc-framework-Cocoa pyobjc-framework-CoreML
|
||||
'"
|
||||
|
||||
echo "==> Killing any stale server on :$PORT"
|
||||
ssh "$HOST" "bash -lc 'pkill -f multihmr_server.py 2>/dev/null || true; sleep 0.3'"
|
||||
|
||||
echo "==> Launching server (background)"
|
||||
ssh "$HOST" "bash -lc 'cd $REMOTE_TMP && \
|
||||
MULTIHMR_SERVER_PORT=$PORT \
|
||||
nohup $REMOTE_VENV/bin/python multihmr_server.py \
|
||||
--mlpackage $REMOTE_TMP/multihmr_full_672_s.mlpackage \
|
||||
--port $PORT \
|
||||
>> $REMOTE_TMP/server.log 2>&1 &
|
||||
echo \$! > $REMOTE_TMP/server.pid
|
||||
disown || true'"
|
||||
|
||||
echo "==> Waiting for server to be ready"
|
||||
REMOTE_ADDR=$(ssh "$HOST" 'echo $SSH_CONNECTION' | awk '{print $3}')
|
||||
# Fallback to host alias if SSH_CONNECTION trick fails.
|
||||
if [ -z "${REMOTE_ADDR:-}" ]; then REMOTE_ADDR="$HOST"; fi
|
||||
|
||||
for i in $(seq 1 30); do
|
||||
if ssh "$HOST" "bash -lc 'nc -z 127.0.0.1 $PORT 2>/dev/null'"; then
|
||||
echo "==> Server up on $HOST:$PORT (probed via localhost on host)"
|
||||
echo "==> Reachable from this Mac at $REMOTE_ADDR:$PORT"
|
||||
ssh "$HOST" "tail -n 20 $REMOTE_TMP/server.log" || true
|
||||
exit 0
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
|
||||
echo "ERROR: server did not come up within 30s. Last log lines:" >&2
|
||||
ssh "$HOST" "tail -n 60 $REMOTE_TMP/server.log" || true
|
||||
exit 1
|
||||
@@ -34,6 +34,10 @@ struct SceneUniforms {
|
||||
float hand_l_y;
|
||||
float hand_r_x;
|
||||
float hand_r_y;
|
||||
float hand_height;
|
||||
float hand_openness;
|
||||
float hand_speed;
|
||||
float hand_dist;
|
||||
float _pad0;
|
||||
float _pad1;
|
||||
};
|
||||
@@ -454,6 +458,10 @@ fragment float4 bg_fragment(VsOut in [[stage_in]],
|
||||
p.x *= U.width / U.height;
|
||||
|
||||
int mode = int(U.viz_mode + 0.5);
|
||||
|
||||
// Mains ecartees -> leger zoom (sauf mode 8 qui pilote sa propre camera).
|
||||
if (mode != 8) { p *= (1.0 - U.hand_dist * 0.2); }
|
||||
|
||||
float3 color;
|
||||
if (mode == 1) color = mode_tunnel(p, U);
|
||||
else if (mode == 2) color = mode_plasma(p, U);
|
||||
@@ -466,6 +474,11 @@ fragment float4 bg_fragment(VsOut in [[stage_in]],
|
||||
else if (mode == 9) color = mode_openpos(p, U);
|
||||
else color = mode_storm(p, U);
|
||||
|
||||
// Modulation expressive pilotee par les mains (subtile, tous les modes).
|
||||
color *= (1.0 + U.hand_height * 0.6); // luminosite
|
||||
color += U.hand_openness * 0.5 * color; // bloom / dispersion
|
||||
color += U.hand_speed * 0.05 * sin(U.time * 8.0 + p.x * 10.0); // turbulence
|
||||
|
||||
// Flash global + vignette
|
||||
color += float3(U.lightning_flash * 1.2);
|
||||
color *= vignette(p);
|
||||
|
||||
@@ -45,8 +45,9 @@ class SMPLXTCPSender:
|
||||
def __init__(self, state: State, host: str = "127.0.0.1",
|
||||
port: int = PORT, target_fps: float = 30.0,
|
||||
enable_rigging: bool = True) -> None:
|
||||
import os as _os
|
||||
self.state = state
|
||||
self.host = host
|
||||
self.host = _os.environ.get("AVBODY_HOST", host)
|
||||
self.port = port
|
||||
self.period = 1.0 / max(1.0, target_fps)
|
||||
self._stop = threading.Event()
|
||||
|
||||
@@ -102,6 +102,12 @@ class State:
|
||||
persons_body: list[list[PoseKp]] = field(default_factory=list)
|
||||
persons_face: list[list[PoseKp]] = field(default_factory=list)
|
||||
persons_hands: list[list[PoseKp]] = field(default_factory=list)
|
||||
# iPhone Vision hands (on-device, 21 kp MediaPipe order, .right upright,
|
||||
# y already top-left/down). Stored separately from MediaPipe persons_hands
|
||||
# so the air-piano can prefer this stabler, rotation-invariant source.
|
||||
persons_hands_iphone: list[list[PoseKp]] = field(default_factory=list)
|
||||
persons_hands_iphone_t: float = 0.0
|
||||
hand_feats: dict | None = None
|
||||
# MediaPipe pose_world_landmarks per person : 33 keypoints in meters,
|
||||
# relative to the hip-center. Optional companion of persons_body
|
||||
# (image-space xy). Empty if no detection or backend doesn't emit it.
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
"""ActionHeadPublisher emits /pose/finger when FINGER_PIANO is enabled."""
|
||||
from __future__ import annotations
|
||||
|
||||
import threading
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from data_only_viz.finger_strike import FINGERTIPS, FINGER_BASES
|
||||
|
||||
|
||||
class _FakeState:
|
||||
def __init__(self) -> None:
|
||||
self.persons_hands = []
|
||||
self._lock = threading.RLock()
|
||||
|
||||
def lock(self):
|
||||
return self._lock
|
||||
|
||||
|
||||
def _hand(index_tip_y: float, base_y: float = 0.4):
|
||||
lm = [[0.3, base_y, 0.0] for _ in range(21)]
|
||||
lm[FINGERTIPS[1]] = [0.3, index_tip_y, 0.0]
|
||||
lm[FINGER_BASES[1]] = [0.3, base_y, 0.0]
|
||||
return lm
|
||||
|
||||
|
||||
def _pub(monkeypatch, enabled: bool):
|
||||
monkeypatch.setenv("FINGER_PIANO", "1" if enabled else "0")
|
||||
monkeypatch.setenv("FINGER_STRIKE_VEL", "0.02")
|
||||
monkeypatch.setenv("FINGER_STRIKE_REFRACTORY_MS", "120")
|
||||
from data_only_viz.action_head_pub import ActionHeadPublisher
|
||||
# bypass __init__ heavy ckpt load: construct then set only what we touch
|
||||
pub = ActionHeadPublisher.__new__(ActionHeadPublisher)
|
||||
pub.state = _FakeState()
|
||||
pub.bridge = MagicMock()
|
||||
pub._init_finger_piano() # helper that reads env + builds detector
|
||||
return pub
|
||||
|
||||
|
||||
def test_emit_fingers_fires_when_enabled(monkeypatch):
|
||||
pub = _pub(monkeypatch, enabled=True)
|
||||
pub.state.persons_hands = [_hand(0.40)]
|
||||
pub._emit_fingers(0.00) # prime
|
||||
pub.state.persons_hands = [_hand(0.46)] # downward strike
|
||||
pub._emit_fingers(0.04)
|
||||
assert pub.bridge.send_finger.called
|
||||
|
||||
|
||||
def test_emit_fingers_noop_when_disabled(monkeypatch):
|
||||
pub = _pub(monkeypatch, enabled=False)
|
||||
pub.state.persons_hands = [_hand(0.40)]
|
||||
pub._emit_fingers(0.00)
|
||||
pub.state.persons_hands = [_hand(0.46)]
|
||||
pub._emit_fingers(0.04)
|
||||
assert not pub.bridge.send_finger.called
|
||||
@@ -152,3 +152,91 @@ def test_publisher_passes_hands_kp_to_step() -> None:
|
||||
assert len(captured_hands) == 1
|
||||
assert captured_hands[0] is not None
|
||||
assert captured_hands[0].shape == (42, 3)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Task-3 tests: _emit_hands wired into _tick
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _fake_hand_21(seed: int = 0):
|
||||
"""Return a list of 21 objects with .x and .y float attributes."""
|
||||
import types
|
||||
rng = __import__("random").Random(seed)
|
||||
return [
|
||||
types.SimpleNamespace(x=rng.random(), y=rng.random())
|
||||
for _ in range(21)
|
||||
]
|
||||
|
||||
|
||||
def test_emit_hands_called_once_per_tick_with_hands() -> None:
|
||||
"""`bridge.send_hands` must be called once per tick with L/R/dist keys."""
|
||||
from data_only_viz.action_head_pub import ActionHeadPublisher
|
||||
state = _FakeState()
|
||||
bridge = MagicMock()
|
||||
pub = ActionHeadPublisher(state, bridge, ckpt_path=None)
|
||||
|
||||
state.persons_hands = [_fake_hand_21(0)]
|
||||
state.persons_smplx = [_make_smplx_person(0)]
|
||||
state.smplx_last_t = 1.0
|
||||
pub._tick(t_now=0.0)
|
||||
|
||||
bridge.send_hands.assert_called_once()
|
||||
feats, _t = bridge.send_hands.call_args[0]
|
||||
assert set(feats.keys()) >= {"L", "R", "dist"}
|
||||
# One hand present → L is populated, R is None
|
||||
assert feats["L"] is not None
|
||||
assert feats["R"] is None
|
||||
assert isinstance(feats["dist"], float)
|
||||
|
||||
|
||||
def test_emit_hands_called_once_per_tick_empty() -> None:
|
||||
"""`bridge.send_hands` still called once when persons_hands is empty."""
|
||||
from data_only_viz.action_head_pub import ActionHeadPublisher
|
||||
state = _FakeState()
|
||||
bridge = MagicMock()
|
||||
pub = ActionHeadPublisher(state, bridge, ckpt_path=None)
|
||||
|
||||
state.persons_hands = []
|
||||
state.persons_smplx = [_make_smplx_person(0)]
|
||||
state.smplx_last_t = 1.0
|
||||
pub._tick(t_now=0.0)
|
||||
|
||||
bridge.send_hands.assert_called_once()
|
||||
feats, _t = bridge.send_hands.call_args[0]
|
||||
assert set(feats.keys()) >= {"L", "R", "dist"}
|
||||
assert feats["L"] is None
|
||||
assert feats["R"] is None
|
||||
assert feats["dist"] == 0.0
|
||||
|
||||
|
||||
def test_emit_hands_single_person_fallback_sends_once() -> None:
|
||||
"""Single-person fallback (hands_present=True + left/right_hand_kp) → send_hands once."""
|
||||
from data_only_viz.action_head_pub import ActionHeadPublisher
|
||||
state = _FakeState()
|
||||
bridge = MagicMock()
|
||||
pub = ActionHeadPublisher(state, bridge, ckpt_path=None)
|
||||
|
||||
# No per-person hands list; exercise the single-person fallback branch.
|
||||
state.persons_hands = []
|
||||
state.hands_present = True
|
||||
state.left_hand_kp = _fake_hand_21(0)
|
||||
state.right_hand_kp = _fake_hand_21(1)
|
||||
state.persons_smplx = [_make_smplx_person(0)]
|
||||
state.smplx_last_t = 1.0
|
||||
pub._tick(t_now=0.0)
|
||||
|
||||
bridge.send_hands.assert_called_once()
|
||||
|
||||
|
||||
def test_emit_hands_once_for_two_pids() -> None:
|
||||
"""Two pids in persons_smplx → send_hands called exactly once, not once per pid."""
|
||||
from data_only_viz.action_head_pub import ActionHeadPublisher
|
||||
state = _FakeState()
|
||||
bridge = MagicMock()
|
||||
pub = ActionHeadPublisher(state, bridge, ckpt_path=None)
|
||||
|
||||
state.persons_smplx = [_make_smplx_person(0), _make_smplx_person(1)]
|
||||
state.smplx_last_t = 1.0
|
||||
pub._tick(t_now=0.0)
|
||||
|
||||
assert bridge.send_hands.call_count == 1
|
||||
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Tests for FingerStrikeDetector (air-piano strike detection)."""
|
||||
from __future__ import annotations
|
||||
|
||||
from data_only_viz.finger_strike import (
|
||||
FingerStrikeDetector,
|
||||
PinchDetector,
|
||||
StrikeEvent,
|
||||
FINGERTIPS,
|
||||
FINGER_BASES,
|
||||
)
|
||||
|
||||
|
||||
def _hand(tip_y_by_finger: dict[int, float], base_y: float = 0.4,
|
||||
cx: float = 0.3) -> list[list[float]]:
|
||||
"""Build a 21-landmark hand. Every base knuckle sits at base_y; each
|
||||
fingertip sits at base_y unless overridden in tip_y_by_finger (keyed by
|
||||
finger 0..4). x is set near cx so L/R slotting is deterministic."""
|
||||
lm = [[cx, base_y, 0.0] for _ in range(21)]
|
||||
for f, base_idx in enumerate(FINGER_BASES):
|
||||
lm[base_idx] = [cx, base_y, 0.0]
|
||||
for f, tip_idx in enumerate(FINGERTIPS):
|
||||
ty = tip_y_by_finger.get(f, base_y)
|
||||
lm[tip_idx] = [cx, ty, 0.0]
|
||||
return lm
|
||||
|
||||
|
||||
def test_downward_spike_fires_exactly_one_strike():
|
||||
det = FingerStrikeDetector(vel_thresh=0.02, refractory_ms=120.0)
|
||||
# frame 0: neutral (primes prev), index tip level with base
|
||||
det.step([_hand({1: 0.40})], t_now=0.00)
|
||||
# frame 1: index tip drops 0.06 below -> downward velocity 0.06 > thresh
|
||||
e1 = det.step([_hand({1: 0.46})], t_now=0.04)
|
||||
# frame 2: tip stays down -> velocity ~0, must NOT refire
|
||||
e2 = det.step([_hand({1: 0.46})], t_now=0.08)
|
||||
strikes = e1 + e2
|
||||
assert len(strikes) == 1
|
||||
assert strikes[0].finger == 1
|
||||
assert strikes[0].hand == 0
|
||||
assert strikes[0].strike_speed > 0.0
|
||||
|
||||
|
||||
def test_whole_hand_translation_does_not_fire():
|
||||
det = FingerStrikeDetector(vel_thresh=0.02)
|
||||
out = []
|
||||
# tip and base move down together each frame -> relative y constant
|
||||
for i, by in enumerate((0.40, 0.50, 0.60, 0.70)):
|
||||
out += det.step([_hand({1: by}, base_y=by)], t_now=i * 0.04)
|
||||
assert out == []
|
||||
|
||||
|
||||
def test_refractory_blocks_second_strike():
|
||||
det = FingerStrikeDetector(vel_thresh=0.02, refractory_ms=120.0)
|
||||
det.step([_hand({1: 0.40})], t_now=0.00) # prime
|
||||
a = det.step([_hand({1: 0.46})], t_now=0.02) # strike 1
|
||||
det.step([_hand({1: 0.40})], t_now=0.04) # lift -> rearm
|
||||
b = det.step([_hand({1: 0.46})], t_now=0.06) # within 120 ms -> blocked
|
||||
assert len(a) == 1
|
||||
assert b == []
|
||||
|
||||
|
||||
def test_strike_speed_scales_with_velocity():
|
||||
soft = FingerStrikeDetector(vel_thresh=0.02, speed_scale=0.10)
|
||||
hard = FingerStrikeDetector(vel_thresh=0.02, speed_scale=0.10)
|
||||
soft.step([_hand({1: 0.40})], t_now=0.0)
|
||||
hard.step([_hand({1: 0.40})], t_now=0.0)
|
||||
s = soft.step([_hand({1: 0.44})], t_now=0.04) # delta 0.04
|
||||
h = hard.step([_hand({1: 0.50})], t_now=0.04) # delta 0.10
|
||||
assert h[0].strike_speed > s[0].strike_speed
|
||||
|
||||
|
||||
def _pinch_hand(thumb_xy, index_xy):
|
||||
"""21-kp hand with fixed wrist & middle-MCP (hand size = 0.3). Middle/
|
||||
ring/pinky tips are parked far from the thumb so only the thumb-index
|
||||
pair can pinch."""
|
||||
lm = [[0.5, 0.5, 0.0] for _ in range(21)]
|
||||
lm[0] = [0.5, 0.8, 0.0] # WRIST
|
||||
lm[9] = [0.5, 0.5, 0.0] # MIDDLE_MCP -> size 0.3
|
||||
lm[4] = [thumb_xy[0], thumb_xy[1], 0.0] # THUMB_TIP
|
||||
lm[8] = [index_xy[0], index_xy[1], 0.0] # INDEX_TIP
|
||||
lm[12] = [0.9, 0.5, 0.0] # MIDDLE_TIP (far)
|
||||
lm[16] = [0.9, 0.6, 0.0] # RING_TIP (far)
|
||||
lm[20] = [0.9, 0.7, 0.0] # LITTLE_TIP (far)
|
||||
return lm
|
||||
|
||||
|
||||
_OPEN = ((0.2, 0.5), (0.8, 0.5)) # thumb-index dist 0.6 -> ratio 2.0
|
||||
_PINCH = ((0.5, 0.5), (0.52, 0.5)) # dist 0.02 -> ratio 0.067
|
||||
|
||||
|
||||
def test_pinch_fires_on_thumb_index_contact():
|
||||
det = PinchDetector(ratio_on=0.45, ratio_off=0.65)
|
||||
assert det.step([_pinch_hand(*_OPEN)], 0.0) == []
|
||||
ev = det.step([_pinch_hand(*_PINCH)], 0.1)
|
||||
assert len(ev) == 1
|
||||
assert ev[0].finger == 1 # index = finger 1
|
||||
assert ev[0].hand == 0
|
||||
|
||||
|
||||
def test_pinch_does_not_refire_while_held():
|
||||
det = PinchDetector(ratio_on=0.45, ratio_off=0.65, refractory_ms=0)
|
||||
det.step([_pinch_hand(*_OPEN)], 0.0)
|
||||
a = det.step([_pinch_hand(*_PINCH)], 0.1) # fire
|
||||
b = det.step([_pinch_hand(*_PINCH)], 0.2) # still held -> no refire
|
||||
assert len(a) == 1
|
||||
assert b == []
|
||||
|
||||
|
||||
def test_pinch_rearms_after_release():
|
||||
det = PinchDetector(ratio_on=0.45, ratio_off=0.65, refractory_ms=0)
|
||||
det.step([_pinch_hand(*_OPEN)], 0.0)
|
||||
a = det.step([_pinch_hand(*_PINCH)], 0.1) # fire
|
||||
det.step([_pinch_hand(*_OPEN)], 0.2) # release (ratio > off)
|
||||
c = det.step([_pinch_hand(*_PINCH)], 0.3) # pinch again -> fire
|
||||
assert len(a) == 1
|
||||
assert len(c) == 1
|
||||
|
||||
|
||||
def test_pinch_refractory_blocks():
|
||||
det = PinchDetector(ratio_on=0.45, ratio_off=0.65, refractory_ms=200)
|
||||
det.step([_pinch_hand(*_OPEN)], 0.0)
|
||||
a = det.step([_pinch_hand(*_PINCH)], 0.05) # fire t=0.05
|
||||
det.step([_pinch_hand(*_OPEN)], 0.08) # release
|
||||
b = det.step([_pinch_hand(*_PINCH)], 0.10) # within 200 ms -> blocked
|
||||
assert len(a) == 1
|
||||
assert b == []
|
||||
@@ -0,0 +1,102 @@
|
||||
import math
|
||||
import pytest
|
||||
from data_only_viz.hand_features import HandFeatureExtractor, NEUTRAL_HAND
|
||||
|
||||
|
||||
class LM:
|
||||
"""Minimal landmark stub (PoseKp-shaped)."""
|
||||
def __init__(self, x, y, z=0.0, c=1.0):
|
||||
self.x, self.y, self.z, self.c = x, y, z, c
|
||||
|
||||
|
||||
def _hand(cx, cy, span):
|
||||
"""21 landmarks centered at (cx,cy); thumb/pinky separated by `span`.
|
||||
|
||||
Index 0 wrist, 4 thumb_tip, 9 middle_mcp, 20 pinky_tip.
|
||||
hand_size (wrist->middle_mcp) fixed at 0.10 so openness ~ span/size.
|
||||
"""
|
||||
pts = [LM(cx, cy) for _ in range(21)]
|
||||
pts[0] = LM(cx, cy + 0.05) # wrist below center
|
||||
pts[9] = LM(cx, cy - 0.05) # middle_mcp above -> size 0.10
|
||||
pts[4] = LM(cx - span / 2, cy) # thumb_tip
|
||||
pts[20] = LM(cx + span / 2, cy) # pinky_tip
|
||||
return pts
|
||||
|
||||
|
||||
def test_open_hand_high_openness():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.5, 0.5, span=0.20)]) # span/size = 2.0
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["openness"] > 0.8
|
||||
|
||||
|
||||
def test_fist_low_openness():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.5, 0.5, span=0.03)]) # span/size = 0.3
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["openness"] < 0.2
|
||||
|
||||
|
||||
def test_static_hand_zero_speed():
|
||||
ext = HandFeatureExtractor()
|
||||
h = _hand(0.5, 0.5, span=0.1)
|
||||
ext.step([h])
|
||||
out = ext.step([h])
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["speed"] == pytest.approx(0.0, abs=1e-6)
|
||||
|
||||
|
||||
def test_moving_hand_positive_speed():
|
||||
ext = HandFeatureExtractor()
|
||||
ext.step([_hand(0.2, 0.5, span=0.1)])
|
||||
out = ext.step([_hand(0.6, 0.5, span=0.1)])
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["speed"] > 0.0
|
||||
|
||||
|
||||
def test_left_right_by_screen_x():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.8, 0.5, span=0.1), _hand(0.2, 0.5, span=0.1)])
|
||||
assert out["L"]["cx"] < out["R"]["cx"] # L is leftmost on screen
|
||||
assert out["dist"] > 0.0
|
||||
|
||||
|
||||
def test_single_hand_zero_dist():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.5, 0.5, span=0.1)])
|
||||
assert out["dist"] == 0.0
|
||||
|
||||
|
||||
def test_no_hands_returns_none():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([])
|
||||
assert out["L"] is None and out["R"] is None and out["dist"] == 0.0
|
||||
|
||||
|
||||
def test_nan_landmark_falls_back_no_exception():
|
||||
ext = HandFeatureExtractor()
|
||||
h = _hand(0.5, 0.5, span=0.1)
|
||||
h[0] = LM(float("nan"), float("nan"))
|
||||
out = ext.step([h]) # must not raise
|
||||
hand = out["L"] or out["R"]
|
||||
assert math.isfinite(hand["cx"]) and math.isfinite(hand["openness"])
|
||||
|
||||
|
||||
def test_numpy_row_landmarks_finite_features():
|
||||
"""Landmarks as numpy rows (indexable [x, y, ...]) must yield finite cx/cy/openness."""
|
||||
import numpy as np
|
||||
arr = np.zeros((21, 3), dtype=np.float32)
|
||||
# replicate _hand(0.5, 0.5, span=0.20) geometry via index access
|
||||
arr[:] = [0.5, 0.5, 0.0]
|
||||
arr[0] = [0.5, 0.55, 0.0] # wrist
|
||||
arr[9] = [0.5, 0.45, 0.0] # middle_mcp → size 0.10
|
||||
arr[4] = [0.4, 0.5, 0.0] # thumb_tip
|
||||
arr[20] = [0.6, 0.5, 0.0] # pinky_tip → span 0.20
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([arr]) # must not raise
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand is not None
|
||||
assert math.isfinite(hand["cx"])
|
||||
assert math.isfinite(hand["cy"])
|
||||
assert math.isfinite(hand["openness"])
|
||||
assert hand["openness"] > 0.8 # span/size = 2.0, same as test_open_hand
|
||||
@@ -0,0 +1,124 @@
|
||||
"""Pure (no-device) unit tests for iphone_usb_source helpers.
|
||||
|
||||
Tests _to_annexb and _decode_hands without any hardware or network connection.
|
||||
"""
|
||||
import struct
|
||||
|
||||
import pytest
|
||||
|
||||
from data_only_viz.iphone_usb_source import _to_annexb, _decode_hands, _HAND_BYTES
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _avcc(nals: list[bytes]) -> bytes:
|
||||
"""Build an AVCC-style buffer with 4-byte big-endian length prefixes."""
|
||||
out = bytearray()
|
||||
for nal in nals:
|
||||
out += struct.pack(">I", len(nal)) + nal
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def _make_hands_payload(hands: list[list[tuple[float, float, float]]]) -> bytes:
|
||||
"""Build a synthetic HandsPayload matching the wire layout.
|
||||
|
||||
count:u8, then per hand: chirality:u8 (1=right) + 21 × (x,y,z) big-endian f32.
|
||||
"""
|
||||
out = bytearray()
|
||||
out += bytes([len(hands)])
|
||||
for hand_kps in hands:
|
||||
out += bytes([1]) # chirality = right
|
||||
for x, y, z in hand_kps:
|
||||
out += struct.pack(">fff", x, y, z)
|
||||
return bytes(out)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _to_annexb
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_to_annexb_two_nals():
|
||||
nal1 = b"\x67\x01\x02\x03" # SPS-like
|
||||
nal2 = b"\x41\xAB\xCD"
|
||||
avcc = _avcc([nal1, nal2])
|
||||
result = _to_annexb(avcc)
|
||||
# First unit: Annex-B start code + nal1
|
||||
assert result[:4] == b"\x00\x00\x00\x01"
|
||||
assert result[4:4 + len(nal1)] == nal1
|
||||
# Second unit: Annex-B start code + nal2
|
||||
rest = result[4 + len(nal1):]
|
||||
assert rest[:4] == b"\x00\x00\x00\x01"
|
||||
assert rest[4:4 + len(nal2)] == nal2
|
||||
assert len(result) == 4 + len(nal1) + 4 + len(nal2)
|
||||
|
||||
|
||||
def test_to_annexb_payload_bytes_preserved():
|
||||
nal = bytes(range(32))
|
||||
result = _to_annexb(_avcc([nal]))
|
||||
assert result[4:] == nal
|
||||
|
||||
|
||||
def test_to_annexb_empty():
|
||||
assert _to_annexb(b"") == b""
|
||||
|
||||
|
||||
def test_to_annexb_truncated_nal_is_dropped():
|
||||
# Length prefix claims 10 bytes but only 3 are present
|
||||
buf = struct.pack(">I", 10) + b"\x00" * 3
|
||||
result = _to_annexb(buf)
|
||||
assert result == b""
|
||||
|
||||
|
||||
def test_to_annexb_single_nal():
|
||||
nal = b"\x65\xB8"
|
||||
result = _to_annexb(_avcc([nal]))
|
||||
assert result == b"\x00\x00\x00\x01" + nal
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# _decode_hands
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def test_decode_hands_empty_payload():
|
||||
assert _decode_hands(b"") is None
|
||||
|
||||
|
||||
def test_decode_hands_count_zero():
|
||||
assert _decode_hands(bytes([0])) == []
|
||||
|
||||
|
||||
def test_decode_hands_two_hands_length_and_coords():
|
||||
hand_a = [(float(i) * 0.1, float(i) * 0.2, float(i) * 0.05) for i in range(21)]
|
||||
hand_b = [(1.0 - float(i) * 0.03, 0.5, float(i) * 0.01) for i in range(21)]
|
||||
payload = _make_hands_payload([hand_a, hand_b])
|
||||
result = _decode_hands(payload)
|
||||
assert isinstance(result, list)
|
||||
assert len(result) == 2
|
||||
for hand in result:
|
||||
assert len(hand) == 21
|
||||
# x/y spot-checks for hand A
|
||||
assert abs(result[0][0].x - hand_a[0][0]) < 1e-5
|
||||
assert abs(result[0][5].y - hand_a[5][1]) < 1e-5
|
||||
# x/y spot-checks for hand B
|
||||
assert abs(result[1][3].x - hand_b[3][0]) < 1e-5
|
||||
assert abs(result[1][20].y - hand_b[20][1]) < 1e-5
|
||||
|
||||
|
||||
def test_decode_hands_one_hand():
|
||||
kps = [(0.1 * i, 0.2 * i, 0.0) for i in range(21)]
|
||||
result = _decode_hands(_make_hands_payload([kps]))
|
||||
assert isinstance(result, list)
|
||||
assert len(result) == 1
|
||||
assert len(result[0]) == 21
|
||||
|
||||
|
||||
def test_decode_hands_truncated_does_not_raise():
|
||||
hand_kps = [(0.1, 0.2, 0.3)] * 21
|
||||
full = _make_hands_payload([hand_kps, hand_kps])
|
||||
# Truncate to half — second hand will be incomplete
|
||||
truncated = full[:len(full) // 2]
|
||||
# Must not raise
|
||||
result = _decode_hands(truncated)
|
||||
assert isinstance(result, (list, type(None)))
|
||||
@@ -51,3 +51,16 @@ def test_state_mutations_are_all_under_lock():
|
||||
f"line {lineno} mutates persons_smplx without a nearby `state.lock()` context:\n"
|
||||
f"{lines[lineno - 1]}"
|
||||
)
|
||||
|
||||
|
||||
def test_predict_once_returns_none_when_coreml_unavailable(monkeypatch):
|
||||
from data_only_viz.multi_hmr_worker import MultiHMRWorker
|
||||
from data_only_viz.state import State
|
||||
# Force CoreML loader to return None
|
||||
state = State()
|
||||
worker = MultiHMRWorker(state, num_persons=1)
|
||||
monkeypatch.setattr(worker, "_get_or_load_coreml_backend", lambda: None)
|
||||
import pytest, numpy as np
|
||||
rgb = np.zeros((480, 640, 3), dtype=np.uint8)
|
||||
with pytest.raises(NotImplementedError):
|
||||
worker.predict_once(rgb)
|
||||
|
||||
@@ -80,8 +80,12 @@ def test_infer_latency_under_target():
|
||||
times.sort()
|
||||
median_ms = times[n // 2]
|
||||
print(f"median latency: {median_ms:.1f} ms (n={n})")
|
||||
# Target 50ms = 20fps. M5 bench shows ~29ms. Generous margin.
|
||||
assert median_ms < 80.0, f"median {median_ms:.1f}ms > 80ms target"
|
||||
# Full Multi-HMR CoreML on M5: ~120-140 ms standalone (7-8 fps),
|
||||
# see scripts/bench_multihmr_coreml.py and multihmr_coreml.py
|
||||
# docstring. The earlier 80 ms target was a backbone-only probe
|
||||
# estimate that does not hold for the full model. 250 ms gives
|
||||
# headroom for thermal/contention without masking a regression.
|
||||
assert median_ms < 250.0, f"median {median_ms:.1f}ms > 250ms target"
|
||||
|
||||
|
||||
def test_filter_threshold():
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
"""Tests for the Multi-HMR remote TCP backend (client-side).
|
||||
|
||||
A loopback server is spun up in a background thread; it returns
|
||||
deterministic stub outputs so we can exercise the byte-for-byte
|
||||
protocol without depending on a live mlpackage.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import struct
|
||||
import threading
|
||||
import time
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
from data_only_viz.multihmr_remote import (
|
||||
IMG_SIZE,
|
||||
MAGIC_REQ,
|
||||
MAGIC_RSP,
|
||||
N_PERSONS_FIXED,
|
||||
N_VERTS,
|
||||
REQ_PAYLOAD_LEN,
|
||||
RSP_PAYLOAD_LEN,
|
||||
MultiHMRRemoteBackend,
|
||||
decode_response,
|
||||
encode_request,
|
||||
)
|
||||
from data_only_viz.scripts.multihmr_server import (
|
||||
decode_request,
|
||||
encode_response,
|
||||
)
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Pure protocol roundtrip tests (no socket, no model).
|
||||
# -----------------------------------------------------------------------
|
||||
|
||||
def _make_K() -> np.ndarray:
|
||||
return np.array([[672.0, 0.0, 336.0],
|
||||
[0.0, 672.0, 336.0],
|
||||
[0.0, 0.0, 1.0]], dtype=np.float32)
|
||||
|
||||
|
||||
def test_request_encode_decode_roundtrip():
|
||||
rng = np.random.default_rng(0)
|
||||
img = rng.integers(0, 256, (IMG_SIZE, IMG_SIZE, 3), dtype=np.uint8)
|
||||
K = _make_K()
|
||||
req = encode_request(img, K)
|
||||
# length prefix + payload
|
||||
assert len(req) == 4 + REQ_PAYLOAD_LEN
|
||||
payload_len = struct.unpack("<I", req[:4])[0]
|
||||
assert payload_len == REQ_PAYLOAD_LEN
|
||||
assert req[4:8] == MAGIC_REQ
|
||||
img_back, K_back, _ = decode_request(req[4:])
|
||||
assert img_back.shape == (IMG_SIZE, IMG_SIZE, 3)
|
||||
np.testing.assert_array_equal(img_back, img)
|
||||
np.testing.assert_array_equal(K_back, K)
|
||||
|
||||
|
||||
def test_response_encode_decode_roundtrip():
|
||||
rng = np.random.default_rng(1)
|
||||
v3d = rng.standard_normal(
|
||||
(N_PERSONS_FIXED, N_VERTS, 3)).astype(np.float32)
|
||||
transl = rng.standard_normal(
|
||||
(N_PERSONS_FIXED, 1, 3)).astype(np.float32)
|
||||
scores = rng.random(N_PERSONS_FIXED).astype(np.float32)
|
||||
betas = rng.standard_normal(
|
||||
(N_PERSONS_FIXED, 10)).astype(np.float32)
|
||||
expr = rng.standard_normal(
|
||||
(N_PERSONS_FIXED, 10)).astype(np.float32)
|
||||
resp = encode_response(v3d, transl, scores, betas, expr, status=0)
|
||||
assert len(resp) == 4 + RSP_PAYLOAD_LEN
|
||||
assert resp[4:8] == MAGIC_RSP
|
||||
v3d2, transl2, scores2, betas2, expr2, status = decode_response(resp[4:])
|
||||
assert status == 0
|
||||
np.testing.assert_array_equal(v3d2, v3d)
|
||||
np.testing.assert_array_equal(transl2, transl)
|
||||
np.testing.assert_array_equal(scores2, scores)
|
||||
np.testing.assert_array_equal(betas2, betas)
|
||||
np.testing.assert_array_equal(expr2, expr)
|
||||
|
||||
|
||||
def test_request_rejects_wrong_dtype():
|
||||
img = np.zeros((IMG_SIZE, IMG_SIZE, 3), dtype=np.float32)
|
||||
with pytest.raises(ValueError):
|
||||
encode_request(img, _make_K())
|
||||
|
||||
|
||||
def test_request_rejects_wrong_shape():
|
||||
img = np.zeros((100, 100, 3), dtype=np.uint8)
|
||||
with pytest.raises(ValueError):
|
||||
encode_request(img, _make_K())
|
||||
|
||||
|
||||
# -----------------------------------------------------------------------
|
||||
# Loopback mock server: full client.infer() end-to-end.
|
||||
# -----------------------------------------------------------------------
|
||||
|
||||
class _StubServer:
|
||||
"""Minimal TCP server that replies with deterministic stub outputs."""
|
||||
|
||||
def __init__(self, host: str = "127.0.0.1", port: int = 0) -> None:
|
||||
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
self.sock.bind((host, port))
|
||||
self.sock.listen(1)
|
||||
self.sock.settimeout(2.0)
|
||||
self.host, self.port = self.sock.getsockname()
|
||||
self._stop = threading.Event()
|
||||
self._thread: threading.Thread | None = None
|
||||
self.requests_seen: list[tuple[np.ndarray, np.ndarray]] = []
|
||||
|
||||
def start(self) -> None:
|
||||
self._thread = threading.Thread(target=self._run, daemon=True)
|
||||
self._thread.start()
|
||||
|
||||
def stop(self) -> None:
|
||||
self._stop.set()
|
||||
try:
|
||||
self.sock.close()
|
||||
except OSError:
|
||||
pass
|
||||
if self._thread is not None:
|
||||
self._thread.join(timeout=2.0)
|
||||
|
||||
@staticmethod
|
||||
def _stub_outputs() -> bytes:
|
||||
# Distinct values per person so the test can assert order.
|
||||
v3d = np.zeros((N_PERSONS_FIXED, N_VERTS, 3), dtype=np.float32)
|
||||
transl = np.zeros((N_PERSONS_FIXED, 1, 3), dtype=np.float32)
|
||||
betas = np.zeros((N_PERSONS_FIXED, 10), dtype=np.float32)
|
||||
expr = np.zeros((N_PERSONS_FIXED, 10), dtype=np.float32)
|
||||
for k in range(N_PERSONS_FIXED):
|
||||
v3d[k] = float(k + 1)
|
||||
transl[k] = float(k + 1) * 0.1
|
||||
betas[k] = float(k + 1) * 0.01
|
||||
expr[k] = float(k + 1) * 0.02
|
||||
scores = np.array([0.9, 0.8, 0.2, 0.1], dtype=np.float32)
|
||||
return encode_response(v3d, transl, scores, betas, expr, status=0)
|
||||
|
||||
def _run(self) -> None:
|
||||
try:
|
||||
conn, _addr = self.sock.accept()
|
||||
except OSError:
|
||||
return
|
||||
conn.settimeout(2.0)
|
||||
try:
|
||||
while not self._stop.is_set():
|
||||
try:
|
||||
len_buf = conn.recv(4)
|
||||
except socket.timeout:
|
||||
continue
|
||||
if not len_buf or len(len_buf) < 4:
|
||||
return
|
||||
payload_len = struct.unpack("<I", len_buf)[0]
|
||||
buf = bytearray()
|
||||
while len(buf) < payload_len:
|
||||
chunk = conn.recv(payload_len - len(buf))
|
||||
if not chunk:
|
||||
return
|
||||
buf.extend(chunk)
|
||||
img, K, _ = decode_request(bytes(buf))
|
||||
self.requests_seen.append((img.copy(), K.copy()))
|
||||
conn.sendall(self._stub_outputs())
|
||||
finally:
|
||||
try:
|
||||
conn.close()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def stub_server():
|
||||
srv = _StubServer()
|
||||
srv.start()
|
||||
try:
|
||||
yield srv
|
||||
finally:
|
||||
srv.stop()
|
||||
|
||||
|
||||
def test_remote_backend_infer_against_stub(stub_server: _StubServer, monkeypatch):
|
||||
monkeypatch.setenv("MULTIHMR_REMOTE_ASYNC", "0") # sync path for determinism
|
||||
backend = MultiHMRRemoteBackend(
|
||||
host=stub_server.host, port=stub_server.port,
|
||||
connect_timeout=2.0, io_timeout=2.0)
|
||||
rng = np.random.default_rng(7)
|
||||
img = rng.random((3, IMG_SIZE, IMG_SIZE), dtype=np.float32)
|
||||
K = _make_K()
|
||||
humans = backend.infer(img, K, det_thresh=0.3)
|
||||
backend.close()
|
||||
|
||||
# det_thresh 0.3 keeps scores 0.9 and 0.8 only.
|
||||
assert len(humans) == 2
|
||||
h0 = humans[0]
|
||||
v = h0["v3d"].detach().cpu().numpy()
|
||||
assert v.shape == (N_VERTS, 3)
|
||||
assert float(v[0, 0]) == pytest.approx(1.0)
|
||||
t = h0["transl_pelvis"].detach().cpu().numpy()
|
||||
assert t.shape == (1, 3)
|
||||
assert float(h0["scores"].item()) == pytest.approx(0.9, abs=1e-5)
|
||||
assert h0["shape"].detach().cpu().numpy().shape == (10,)
|
||||
assert h0["expression"].detach().cpu().numpy().shape == (10,)
|
||||
|
||||
h1 = humans[1]
|
||||
assert float(h1["scores"].item()) == pytest.approx(0.8, abs=1e-5)
|
||||
|
||||
# Server saw exactly one request, and the image round-tripped as
|
||||
# uint8 (we lose [0,1] precision but the shape and dtype are right).
|
||||
assert len(stub_server.requests_seen) == 1
|
||||
img_seen, K_seen = stub_server.requests_seen[0]
|
||||
assert img_seen.shape == (IMG_SIZE, IMG_SIZE, 3)
|
||||
assert img_seen.dtype == np.uint8
|
||||
np.testing.assert_array_equal(K_seen, K)
|
||||
|
||||
|
||||
def test_remote_backend_threshold_filters_all(stub_server: _StubServer, monkeypatch):
|
||||
monkeypatch.setenv("MULTIHMR_REMOTE_ASYNC", "0") # sync path for determinism
|
||||
backend = MultiHMRRemoteBackend(
|
||||
host=stub_server.host, port=stub_server.port)
|
||||
img = np.zeros((3, IMG_SIZE, IMG_SIZE), dtype=np.float32)
|
||||
humans = backend.infer(img, _make_K(), det_thresh=1.5)
|
||||
backend.close()
|
||||
assert humans == []
|
||||
|
||||
|
||||
def test_is_available_returns_true_for_live_stub(stub_server: _StubServer):
|
||||
assert MultiHMRRemoteBackend.is_available(
|
||||
host=stub_server.host, port=stub_server.port) is True
|
||||
|
||||
|
||||
def test_is_available_false_for_dead_port():
|
||||
# Bind+release a port to get a guaranteed-closed one.
|
||||
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
s.bind(("127.0.0.1", 0))
|
||||
_, port = s.getsockname()
|
||||
s.close()
|
||||
# Tiny race window: port may be reused by something else, but very
|
||||
# unlikely in unit-test scope.
|
||||
time.sleep(0.05)
|
||||
assert MultiHMRRemoteBackend.is_available(
|
||||
host="127.0.0.1", port=port) is False
|
||||
@@ -0,0 +1,36 @@
|
||||
"""Tests for PoseSoundBridge.send_finger."""
|
||||
from __future__ import annotations
|
||||
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
from data_only_viz.finger_strike import StrikeEvent
|
||||
from data_only_viz.pose_bridge import PoseSoundBridge
|
||||
|
||||
|
||||
def _bridge() -> PoseSoundBridge:
|
||||
b = PoseSoundBridge()
|
||||
b._client = MagicMock() # stub the UDP client
|
||||
return b
|
||||
|
||||
|
||||
def test_send_finger_emits_expected_osc():
|
||||
b = _bridge()
|
||||
ev = StrikeEvent(hand=1, finger=2, strike_speed=0.5, z=0.1,
|
||||
tipx=0.7, tipy=0.6)
|
||||
b.send_finger(ev)
|
||||
b._client.send_message.assert_called_once()
|
||||
addr, args = b._client.send_message.call_args[0]
|
||||
assert addr == "/pose/finger"
|
||||
assert args[0] == 0 # pid
|
||||
assert args[1] == 1 # hand
|
||||
assert args[2] == 2 # finger
|
||||
assert abs(args[3] - 0.5) < 1e-6
|
||||
assert len(args) == 7
|
||||
# Verify z, tipx, tipy values
|
||||
assert abs(args[4] - 0.1) < 1e-6 # z
|
||||
assert abs(args[5] - 0.7) < 1e-6 # tipx
|
||||
assert abs(args[6] - 0.6) < 1e-6 # tipy
|
||||
# types matter for OSC tags / the SC ~fpResolve reader
|
||||
assert isinstance(args[0], int) and not isinstance(args[0], bool)
|
||||
assert isinstance(args[1], int) and isinstance(args[2], int)
|
||||
assert all(isinstance(a, float) for a in args[3:7])
|
||||
@@ -0,0 +1,48 @@
|
||||
from data_only_viz.pose_bridge import PoseSoundBridge
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self):
|
||||
self.sent = []
|
||||
|
||||
def send_message(self, addr, args):
|
||||
self.sent.append((addr, args))
|
||||
|
||||
|
||||
def _bridge():
|
||||
b = PoseSoundBridge.__new__(PoseSoundBridge) # bypass __init__/sockets
|
||||
b._client = FakeClient()
|
||||
return b
|
||||
|
||||
|
||||
def test_send_hands_both_present():
|
||||
b = _bridge()
|
||||
feats = {
|
||||
"L": {"cx": 0.2, "cy": 0.4, "openness": 0.9, "speed": 0.1},
|
||||
"R": {"cx": 0.8, "cy": 0.3, "openness": 0.5, "speed": 0.0},
|
||||
"dist": 0.6,
|
||||
}
|
||||
b.send_hands(feats, t=1.0)
|
||||
addr, args = b._client.sent[-1]
|
||||
assert addr == "/pose/hands"
|
||||
assert args[0] == 0
|
||||
assert args[1:5] == [0.2, 0.4, 0.9, 0.1]
|
||||
assert args[5:9] == [0.8, 0.3, 0.5, 0.0]
|
||||
assert args[9] == 0.6
|
||||
assert len(args) == 10
|
||||
|
||||
|
||||
def test_send_hands_left_only_zero_pads_right():
|
||||
b = _bridge()
|
||||
feats = {"L": {"cx": 0.5, "cy": 0.5, "openness": 0.7, "speed": 0.2},
|
||||
"R": None, "dist": 0.0}
|
||||
b.send_hands(feats, t=1.0)
|
||||
_, args = b._client.sent[-1]
|
||||
assert args[5:9] == [0.0, 0.0, 0.0, 0.0]
|
||||
|
||||
|
||||
def test_send_hands_none_emits_all_zero():
|
||||
b = _bridge()
|
||||
b.send_hands({"L": None, "R": None, "dist": 0.0}, t=1.0)
|
||||
_, args = b._client.sent[-1]
|
||||
assert args == [0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
|
||||
@@ -0,0 +1,25 @@
|
||||
"""Verify Metal uniform buffer layout constants after hand-feature expansion.
|
||||
|
||||
The renderer imports pyobjc/Metal which is unavailable headless, so we test
|
||||
the struct layout independently and import the constants with a fallback.
|
||||
"""
|
||||
|
||||
import struct
|
||||
|
||||
|
||||
def test_uniform_layout_struct():
|
||||
"""24 floats pack to exactly 96 bytes (16-byte aligned for Metal)."""
|
||||
assert struct.calcsize("24f") == 96
|
||||
|
||||
|
||||
def test_uniform_constants():
|
||||
"""UNIFORM_FLOATS == 24 and UNIFORM_SIZE == 96 from renderer module."""
|
||||
try:
|
||||
from data_only_viz.renderer import UNIFORM_FLOATS, UNIFORM_SIZE
|
||||
assert UNIFORM_FLOATS == 24, f"expected 24, got {UNIFORM_FLOATS}"
|
||||
assert UNIFORM_SIZE == 96, f"expected 96, got {UNIFORM_SIZE}"
|
||||
assert struct.calcsize(f"{UNIFORM_FLOATS}f") == 96
|
||||
except ImportError as exc: # pyobjc/Metal unavailable headless
|
||||
# Fallback: at minimum the struct math is correct
|
||||
import pytest
|
||||
pytest.skip(f"renderer import skipped headless ({exc}); struct layout verified by test_uniform_layout_struct")
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Tests for the VIDEO_ROTATE frame rotation helper in multi.py."""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
|
||||
from data_only_viz.multi import _apply_video_rotate
|
||||
|
||||
|
||||
def _frame():
|
||||
"""(2, 3, 3) BGR frame with a marker at the top-right corner (row0, col2)."""
|
||||
f = np.zeros((2, 3, 3), dtype=np.uint8)
|
||||
f[0, 2] = (9, 9, 9)
|
||||
return f
|
||||
|
||||
|
||||
def test_none_is_identity():
|
||||
f = _frame()
|
||||
out = _apply_video_rotate(f, "none")
|
||||
assert out is f # no copy when not rotating
|
||||
|
||||
|
||||
def test_unknown_mode_is_identity():
|
||||
f = _frame()
|
||||
out = _apply_video_rotate(f, "garbage")
|
||||
assert out is f
|
||||
|
||||
|
||||
def test_ccw_swaps_dims_and_moves_corner():
|
||||
f = _frame() # h=2, w=3
|
||||
out = _apply_video_rotate(f, "ccw")
|
||||
assert out.shape == (3, 2, 3) # w and h swapped
|
||||
# CCW: top-right corner -> top-left
|
||||
assert tuple(out[0, 0]) == (9, 9, 9)
|
||||
assert out.flags["C_CONTIGUOUS"]
|
||||
|
||||
|
||||
def test_cw_swaps_dims_and_moves_corner():
|
||||
f = _frame()
|
||||
out = _apply_video_rotate(f, "cw")
|
||||
assert out.shape == (3, 2, 3)
|
||||
# CW: top-right corner -> bottom-right
|
||||
assert tuple(out[2, 1]) == (9, 9, 9)
|
||||
assert out.flags["C_CONTIGUOUS"]
|
||||
|
||||
|
||||
def test_180_moves_corner_opposite():
|
||||
f = _frame()
|
||||
out = _apply_video_rotate(f, "180")
|
||||
assert out.shape == (2, 3, 3) # dims preserved
|
||||
# 180: top-right corner -> bottom-left
|
||||
assert tuple(out[1, 0]) == (9, 9, 9)
|
||||
assert out.flags["C_CONTIGUOUS"]
|
||||
Generated
+35
-1
@@ -71,6 +71,34 @@ wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "av"
|
||||
version = "17.1.0"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/5e/e3/477fa20578c284abeda08d91b63ee9abaebc93445d8feeb989d3d444bae1/av-17.1.0.tar.gz", hash = "sha256:7f1e71ff621b66253333926f948e00faae11d855b2442133c65128bca64cdeb3", size = 4288546, upload-time = "2026-06-07T05:52:55.999Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/ec/87/8036b5c781bc3639ea04ef42d4e26da253bd4bd4311d8705b6a1c8824047/av-17.1.0-cp311-abi3-macosx_11_0_x86_64.whl", hash = "sha256:ad7b4aa011093324b7118245f50ac6db244cfe9900d4072508a5245a2b0d3f41", size = 22460847, upload-time = "2026-06-07T05:52:04.261Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6d/af/dfdf6fc7b17814b50d0aa9e7a7e37b87be91be3890f44b0d525433cd1fd1/av-17.1.0-cp311-abi3-macosx_14_0_arm64.whl", hash = "sha256:43ebbe977f19a7f2d2bd1a4e119675a0b15e05852cf7309846b6ab922ba7ffe9", size = 18159115, upload-time = "2026-06-07T05:52:06.64Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ad/13/64f6c466471cea225b8b2f4cdc51a571f8a286984b55a08d169b932fda5d/av-17.1.0-cp311-abi3-manylinux_2_28_aarch64.whl", hash = "sha256:6a20658ec7d96a70e14b1196eff00b7cdd8831ac3b99868e16b8ba8b24090847", size = 33224427, upload-time = "2026-06-07T05:52:09.165Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/77/43/96b35170bf2e64e00a41748c6400ff73232dc0fc62ded283679fb07c7fe0/av-17.1.0-cp311-abi3-manylinux_2_28_x86_64.whl", hash = "sha256:f9a65d1f48b818323fb411e80358f89d77dec340b01d27c6b2dfbb9cbf4b779f", size = 35370183, upload-time = "2026-06-07T05:52:11.959Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2e/b3/8e8b4b6498731bfbd88e8399a756543f8088f1bd33d08eab678b5aebe728/av-17.1.0-cp311-abi3-manylinux_2_31_armv7l.whl", hash = "sha256:58f7593726437cda5bd19793027e027768450b5c4a594777bf487798a33db702", size = 24459265, upload-time = "2026-06-07T05:52:14.66Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/14/ac/ceb84b7553db21f1143d817245c560d9267168e1e58b1a8eeae2b62c4d04/av-17.1.0-cp311-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:bbab058bd965309f39962e53caac8126987c68c0be094fc4f9427e5615b0218f", size = 34283709, upload-time = "2026-06-07T05:52:17.389Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/59/f9/4115fd84148c9a1cf365096694be6ac882fd3cd3cdb7a2f35e71fecf1631/av-17.1.0-cp311-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:9514cfda85180554c430695282faf4be3ffdf95775d8519733821244eecb58e0", size = 25397573, upload-time = "2026-06-07T05:52:20.012Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e2/ac/92e52d5ed0e0b84d9d93e52b4338c2713d8a44082b8696e6516fdae7c4e4/av-17.1.0-cp311-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:e1c90f85cd7431ede95b11e8e711571a896ebea433f298849c2c0f1594c8d86e", size = 36451495, upload-time = "2026-06-07T05:52:22.581Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/6b/f2/53a7cd34adb6a971d7e6d99663e74db286966c9db8afdca17472fdf0f98e/av-17.1.0-cp311-abi3-win_amd64.whl", hash = "sha256:5df5c1172ef1cf65a1529d612f7da7798ce2cf82c1ff7212466b538a6cc7214c", size = 28036393, upload-time = "2026-06-07T05:52:25.657Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/66/47/cd9ae0edf2206351c1251bb94b5ec58728e42c5f6ee16c03c412f3a1bb3e/av-17.1.0-cp311-abi3-win_arm64.whl", hash = "sha256:ee98534242a74da847af78624779ac5a3177dc7c69f956a4da9e6f0fdb37d7f6", size = 21174601, upload-time = "2026-06-07T05:52:28.077Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/36/90/b5668cddb3c401fcf22553bc495d5b0c6d8a01d118624b26f0db1d0b8653/av-17.1.0-cp314-cp314t-macosx_11_0_x86_64.whl", hash = "sha256:5327807c1219293803ef0c5d1578ff3ae1cf638c09e5998962026e1a554ec240", size = 22699499, upload-time = "2026-06-07T05:52:30.335Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/e0/7e/7be6bfddb823d045ff9fd5d4deb922ee3847605e162c3882e6c45b4c35ff/av-17.1.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:6c9b71fe5c0c5a8d303b1588d4d8ce9397d6b023f467cfef95000ba1f75507fa", size = 18366696, upload-time = "2026-06-07T05:52:32.645Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a2/23/391dcfa75c1ae1977efca44b753a11b929399b558826670c16a8808dd0e3/av-17.1.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:f997e3351bdf51127c07a74e21741a2996e9230cbeb2d81c14acde761b116c9c", size = 36582649, upload-time = "2026-06-07T05:52:35.218Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/fb/32/7312854868b318b9d1b1dcbd1bddb460aaaeac7d57f816e11efec3bef5b1/av-17.1.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:efe9b1397300b67b644ad220c89df4892a76f2debe70f16bae1749fa20526e63", size = 38479390, upload-time = "2026-06-07T05:52:37.968Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/2a/72/af47f59b4458e81ca7d89f477698dbfb3d5a0cd8ae6c1e4441d01074af8a/av-17.1.0-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:fa64e1f1500d01c4a98e7a41dc1a9a35fb4dfe71f5de0389264ec1192200c76a", size = 27127432, upload-time = "2026-06-07T05:52:40.371Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/88/85/c2e6861baf0f8c7d21c4ce811d4d424fedac915e3910d3570ce4377717dc/av-17.1.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ffbd78d73d2c9bf31e9a007c992faec3991428b2941a3b085b84fb82e8c32d19", size = 37406592, upload-time = "2026-06-07T05:52:43.215Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/ba/40/3cc13125aea976101c0858af99ac47257c0654411aa199b5d8e81eea7002/av-17.1.0-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:bff8896454b38fcb785a70e5ae0485d7021cb776303a5849393128a30b8f850b", size = 28336228, upload-time = "2026-06-07T05:52:46.134Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a2/38/c7d9c3e746209a1a695c13e3aa7d817229e84a85d0a84271f313d1befdd3/av-17.1.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:1284addf3c0dd939887a9722dc30df2241a97471ad52c3c507e31583ae22ff02", size = 39490680, upload-time = "2026-06-07T05:52:48.887Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a1/25/9d42da561b7b8f7dabdfaebba07b52977bee58c5c7e4285ac991abcfaa72/av-17.1.0-cp314-cp314t-win_amd64.whl", hash = "sha256:ec630be6321b04e317862f6082e84812bbd801e55a3c2298312e3fc8a0a4af4f", size = 28355673, upload-time = "2026-06-07T05:52:51.614Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/a8/41/562a61d5a61fba3ffb273a115e249f1d8471b9515c59fcc38b4b9deda238/av-17.1.0-cp314-cp314t-win_arm64.whl", hash = "sha256:b41647e42884bf543b8e8d0a1dabd4d1b006c99183eb1a2d7afc5b01f73eeff4", size = 21324700, upload-time = "2026-06-07T05:52:53.972Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "av-live-data-only-viz"
|
||||
version = "0.1.0"
|
||||
@@ -100,6 +128,10 @@ detrpose = [
|
||||
{ name = "transformers" },
|
||||
{ name = "xtcocotools" },
|
||||
]
|
||||
iphone-usb = [
|
||||
{ name = "av" },
|
||||
{ name = "opencv-python" },
|
||||
]
|
||||
lidar = [
|
||||
{ name = "open3d" },
|
||||
]
|
||||
@@ -154,6 +186,7 @@ dev = [
|
||||
|
||||
[package.metadata]
|
||||
requires-dist = [
|
||||
{ name = "av", marker = "extra == 'iphone-usb'", specifier = ">=12.0" },
|
||||
{ name = "cloudpickle", marker = "extra == 'detrpose'", specifier = ">=3.0" },
|
||||
{ name = "coremltools", marker = "extra == 'pose'", specifier = ">=9.0" },
|
||||
{ name = "einops", marker = "extra == 'multihmr'", specifier = ">=0.8" },
|
||||
@@ -170,6 +203,7 @@ requires-dist = [
|
||||
{ name = "omegaconf", marker = "extra == 'detrpose'", specifier = ">=2.3" },
|
||||
{ name = "open3d", marker = "extra == 'lidar'", specifier = ">=0.18,<0.20" },
|
||||
{ name = "opencv-python", marker = "extra == 'detrpose'", specifier = ">=4.10" },
|
||||
{ name = "opencv-python", marker = "extra == 'iphone-usb'", specifier = ">=4.10" },
|
||||
{ name = "opencv-python", marker = "extra == 'multihmr'", specifier = ">=4.10" },
|
||||
{ name = "opencv-python", marker = "extra == 'nlf'", specifier = ">=4.10" },
|
||||
{ name = "opencv-python", marker = "extra == 'pose'", specifier = ">=4.10" },
|
||||
@@ -210,7 +244,7 @@ requires-dist = [
|
||||
{ name = "xtcocotools", marker = "extra == 'detrpose'", specifier = ">=1.14" },
|
||||
{ name = "yacs", marker = "extra == 'smplerx'", specifier = ">=0.1.8" },
|
||||
]
|
||||
provides-extras = ["pose", "detrpose", "lidar", "nlf", "multihmr", "smplerx"]
|
||||
provides-extras = ["pose", "detrpose", "lidar", "nlf", "multihmr", "iphone-usb", "smplerx"]
|
||||
|
||||
[package.metadata.requires-dev]
|
||||
dev = [{ name = "pytest", specifier = ">=9.0.3" }]
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,430 @@
|
||||
# iPhone Capture Implementation Plan (Plan 2 of 3)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make the iOS `ARBodyTracker` app stream the camera RGB video (HEVC) over the USB transport alongside the ARKit skeleton, and retire the legacy OSC/UDP sender — so the iPhone is a self-contained, network-free capture source.
|
||||
|
||||
**Architecture:** `ARBodySession` already captures the ARKit 91-joint skeleton and sends it as `AVLiveWire` `.skeleton` frames through `USBServer` (built in Plan 1). This plan adds a `VideoEncoder` (VideoToolbox hardware HEVC) that encodes each `ARFrame.capturedImage` and sends it as `.video` frames through the same `USBServer`. The OSC/UDP fanout (`/body3d/kp` to `host:57128/57129`) and its `ContentView` config fields are removed.
|
||||
|
||||
**Tech Stack:** Swift 5.10, ARKit, VideoToolbox, CoreMedia, `AVLiveWire` (local package), iOS 17. Build verification via `xcodebuild`.
|
||||
|
||||
**Companion spec:** `docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md`
|
||||
**Prerequisite:** Plan 1 (`docs/superpowers/plans/2026-05-18-iphone-usb-transport.md`) — merged.
|
||||
|
||||
---
|
||||
|
||||
## Verification note
|
||||
|
||||
The iOS app is an iOS-only target; it cannot be built with `swift build`
|
||||
on a macOS host. The verification command for every task is:
|
||||
|
||||
```bash
|
||||
cd iphone-arbody && xcodegen generate && \
|
||||
xcodebuild -project ARBodyTracker.xcodeproj -scheme ARBodyTracker \
|
||||
-sdk iphonesimulator -destination 'generic/platform=iOS Simulator' \
|
||||
-configuration Debug build
|
||||
```
|
||||
|
||||
Expected: `** BUILD SUCCEEDED **`. VideoToolbox HEVC encoding and ARKit
|
||||
body tracking only run fully on a physical device — runtime behavior is
|
||||
an owner on-device check, out of this plan's automated scope.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility |
|
||||
|------|----------------|
|
||||
| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/VideoEncoder.swift` | NEW. VideoToolbox HEVC hardware encoder: `CVPixelBuffer` → `VideoPayload` via callback |
|
||||
| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift` | MODIFY. Add video encoding in `session(_:didUpdate:)`; remove OSC fanout |
|
||||
| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift` | MODIFY. Remove OSC host/port config UI |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: VideoEncoder
|
||||
|
||||
`VideoEncoder` wraps a `VTCompressionSession` configured for HEVC. It
|
||||
accepts `CVPixelBuffer`s and invokes `onPayload` with a `VideoPayload`
|
||||
(keyframe flag + the access-unit bytes; for keyframes the HEVC
|
||||
parameter sets are prepended so the Mac decoder is self-sufficient).
|
||||
|
||||
**Files:**
|
||||
- Create: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/VideoEncoder.swift`
|
||||
|
||||
- [ ] **Step 1: Create the file**
|
||||
|
||||
```swift
|
||||
import AVLiveWire
|
||||
import CoreMedia
|
||||
import CoreVideo
|
||||
import Foundation
|
||||
import VideoToolbox
|
||||
|
||||
/// Hardware HEVC encoder. Feed `CVPixelBuffer`s from ARKit frames in;
|
||||
/// receive one `VideoPayload` per encoded access unit via `onPayload`.
|
||||
/// Keyframe payloads carry the VPS/SPS/PPS parameter sets prepended,
|
||||
/// each as a 4-byte-length-prefixed NAL unit, so the Mac decoder can
|
||||
/// build its format description without a side channel.
|
||||
final class VideoEncoder {
|
||||
var onPayload: ((VideoPayload) -> Void)?
|
||||
|
||||
private var session: VTCompressionSession?
|
||||
private let lock = NSLock()
|
||||
|
||||
/// Create the compression session for a given frame size.
|
||||
func start(width: Int32, height: Int32) {
|
||||
stop()
|
||||
var s: VTCompressionSession?
|
||||
let status = VTCompressionSessionCreate(
|
||||
allocator: kCFAllocatorDefault,
|
||||
width: width, height: height,
|
||||
codecType: kCMVideoCodecType_HEVC,
|
||||
encoderSpecification: nil,
|
||||
imageBufferAttributes: nil,
|
||||
compressedDataAllocator: nil,
|
||||
outputCallback: nil,
|
||||
refcon: nil,
|
||||
compressionSessionOut: &s)
|
||||
guard status == noErr, let s else {
|
||||
NSLog("VideoEncoder: VTCompressionSessionCreate failed %d",
|
||||
status)
|
||||
return
|
||||
}
|
||||
VTSessionSetProperty(s, key: kVTCompressionPropertyKey_RealTime,
|
||||
value: kCFBooleanTrue)
|
||||
VTSessionSetProperty(s,
|
||||
key: kVTCompressionPropertyKey_AllowFrameReordering,
|
||||
value: kCFBooleanFalse)
|
||||
VTSessionSetProperty(s,
|
||||
key: kVTCompressionPropertyKey_MaxKeyFrameInterval,
|
||||
value: 30 as CFNumber)
|
||||
VTCompressionSessionPrepareToEncodeFrames(s)
|
||||
session = s
|
||||
}
|
||||
|
||||
/// Encode one frame. `pts` is the capture timestamp in seconds.
|
||||
func encode(_ pixelBuffer: CVPixelBuffer, pts: Double) {
|
||||
lock.lock(); let s = session; lock.unlock()
|
||||
guard let s else { return }
|
||||
let time = CMTime(seconds: pts, preferredTimescale: 1_000_000)
|
||||
VTCompressionSessionEncodeFrame(
|
||||
s, imageBuffer: pixelBuffer, presentationTimeStamp: time,
|
||||
duration: .invalid, frameProperties: nil,
|
||||
infoFlagsOut: nil) { [weak self] status, _, sample in
|
||||
guard status == noErr, let sample else { return }
|
||||
self?.handle(sample)
|
||||
}
|
||||
}
|
||||
|
||||
func stop() {
|
||||
lock.lock(); let s = session; session = nil; lock.unlock()
|
||||
if let s {
|
||||
VTCompressionSessionInvalidate(s)
|
||||
}
|
||||
}
|
||||
|
||||
deinit { stop() }
|
||||
|
||||
// MARK: - Sample → VideoPayload
|
||||
|
||||
private func handle(_ sample: CMSampleBuffer) {
|
||||
let isKeyframe = !Self.notSync(sample)
|
||||
var out = Data()
|
||||
if isKeyframe, let fmt = CMSampleBufferGetFormatDescription(sample) {
|
||||
out.append(Self.parameterSets(fmt))
|
||||
}
|
||||
if let block = CMSampleBufferGetDataBuffer(sample) {
|
||||
var lengthOut = 0
|
||||
var ptr: UnsafeMutablePointer<Int8>?
|
||||
if CMBlockBufferGetDataPointer(
|
||||
block, atOffset: 0, lengthAtOffsetOut: nil,
|
||||
totalLengthOut: &lengthOut,
|
||||
dataPointerOut: &ptr) == noErr, let ptr {
|
||||
out.append(UnsafeBufferPointer(
|
||||
start: UnsafeRawPointer(ptr)
|
||||
.assumingMemoryBound(to: UInt8.self),
|
||||
count: lengthOut))
|
||||
}
|
||||
}
|
||||
guard !out.isEmpty else { return }
|
||||
onPayload?(VideoPayload(isKeyframe: isKeyframe, data: out))
|
||||
}
|
||||
|
||||
/// True if the sample is NOT a sync (key) frame.
|
||||
private static func notSync(_ sample: CMSampleBuffer) -> Bool {
|
||||
guard let arr = CMSampleBufferGetSampleAttachmentsArray(
|
||||
sample, createIfNecessary: false),
|
||||
CFArrayGetCount(arr) > 0 else { return false }
|
||||
let dict = unsafeBitCast(CFArrayGetValueAtIndex(arr, 0),
|
||||
to: CFDictionary.self)
|
||||
let key = Unmanaged.passUnretained(
|
||||
kCMSampleAttachmentKey_NotSync).toOpaque()
|
||||
return CFDictionaryContainsKey(dict, key)
|
||||
}
|
||||
|
||||
/// Concatenate the HEVC VPS/SPS/PPS parameter sets, each as a
|
||||
/// 4-byte big-endian length prefix followed by the NAL bytes.
|
||||
private static func parameterSets(
|
||||
_ fmt: CMFormatDescription) -> Data {
|
||||
var count = 0
|
||||
CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(
|
||||
fmt, parameterSetIndex: 0, parameterSetPointerOut: nil,
|
||||
parameterSetSizeOut: nil, parameterSetCountOut: &count,
|
||||
nalUnitHeaderLengthOut: nil)
|
||||
var data = Data()
|
||||
for i in 0..<count {
|
||||
var ptr: UnsafePointer<UInt8>?
|
||||
var size = 0
|
||||
guard CMVideoFormatDescriptionGetHEVCParameterSetAtIndex(
|
||||
fmt, parameterSetIndex: i,
|
||||
parameterSetPointerOut: &ptr,
|
||||
parameterSetSizeOut: &size,
|
||||
parameterSetCountOut: nil,
|
||||
nalUnitHeaderLengthOut: nil) == noErr,
|
||||
let ptr else { continue }
|
||||
var be = UInt32(size).bigEndian
|
||||
withUnsafeBytes(of: &be) { data.append(contentsOf: $0) }
|
||||
data.append(UnsafeBufferPointer(start: ptr, count: size))
|
||||
}
|
||||
return data
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify it compiles**
|
||||
|
||||
Run the verification command from the "Verification note" section above.
|
||||
Expected: `** BUILD SUCCEEDED **` (the new file compiles within the
|
||||
target).
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/VideoEncoder.swift
|
||||
git commit -m "feat(ios): VideoToolbox HEVC encoder"
|
||||
```
|
||||
|
||||
(subject ≤50 chars; add a short body — the commit hook rejects
|
||||
subject-only messages; no AI attribution.)
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Stream video from ARBodySession
|
||||
|
||||
Wire `VideoEncoder` into the ARKit frame loop. On each `didUpdate`
|
||||
frame already processed for skeletons, also encode `capturedImage` and
|
||||
send the resulting `VideoPayload` over the existing `USBServer`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift`
|
||||
|
||||
- [ ] **Step 1: Add the encoder property and its payload wiring**
|
||||
|
||||
In `ARBodySession`, next to `private let usb = USBServer()` (currently
|
||||
line 45), add:
|
||||
|
||||
```swift
|
||||
private let videoEncoder = VideoEncoder()
|
||||
private var videoStarted = false
|
||||
```
|
||||
|
||||
In `init()`, after the `usb.onState = { ... }` block, add the encoder
|
||||
output wiring:
|
||||
|
||||
```swift
|
||||
videoEncoder.onPayload = { [weak self] payload in
|
||||
Task { @MainActor in
|
||||
guard let self, self.usbState == .connected else {
|
||||
return
|
||||
}
|
||||
self.usb.send(tag: .video, pid: -1,
|
||||
timestamp: self.lastFrameTime,
|
||||
payload: payload.encoded())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Encode the captured image in the frame loop**
|
||||
|
||||
In `session(_:didUpdate:)`, inside the `Task { @MainActor in ... }`
|
||||
block, after `self.lastFrameTime = t` and before the anchor loop,
|
||||
add video encoding:
|
||||
|
||||
```swift
|
||||
// Start the encoder lazily once the first frame size is
|
||||
// known, then encode every (throttled) frame.
|
||||
let img = frame.capturedImage
|
||||
let w = Int32(CVPixelBufferGetWidth(img))
|
||||
let h = Int32(CVPixelBufferGetHeight(img))
|
||||
if !self.videoStarted, w > 0, h > 0 {
|
||||
self.videoEncoder.start(width: w, height: h)
|
||||
self.videoStarted = true
|
||||
}
|
||||
if self.videoStarted {
|
||||
self.videoEncoder.encode(img, pts: t)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Stop the encoder on stop()**
|
||||
|
||||
In `stop()`, after `usb.stop()`, add:
|
||||
|
||||
```swift
|
||||
videoEncoder.stop()
|
||||
videoStarted = false
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify it compiles**
|
||||
|
||||
Run the verification command. Expected: `** BUILD SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift
|
||||
git commit -m "feat(ios): stream HEVC video over USB"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Remove the legacy OSC sender
|
||||
|
||||
The OSC/UDP fanout is the network dependency the autonomous USB design
|
||||
removes. Delete it from `ARBodySession`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift`
|
||||
|
||||
- [ ] **Step 1: Delete OSC members and methods**
|
||||
|
||||
In `ARBodySession.swift`, delete:
|
||||
- the stored properties `host`, `pythonPort`, `swiftPort` (currently
|
||||
lines 39-41) and `conns` (line 44);
|
||||
- the `configure(host:pythonPort:swiftPort:sendEnvMesh:)` method —
|
||||
replace it with a parameterless `configure(sendEnvMesh:)`:
|
||||
```swift
|
||||
func configure(sendEnvMesh: Bool) {
|
||||
self.sendEnvMesh = sendEnvMesh
|
||||
}
|
||||
```
|
||||
- the call `openUDP()` in `start()`;
|
||||
- in `stop()`, the lines `for c in conns { c.cancel() }` and
|
||||
`conns.removeAll()`;
|
||||
- the entire `// MARK: - UDP fanout` section: `openUDP()` and
|
||||
`sendDatagram(_:)`;
|
||||
- the `publishJoints(pid:body:)` method and its call site in
|
||||
`session(_:didUpdate:)` (`self.publishJoints(pid: count, body: body)`);
|
||||
- the `sendOSC(addr:args:)` call for `/body3d/count` in
|
||||
`session(_:didUpdate:)`;
|
||||
- the `// MARK: - OSC minimal encoder` section: the `OSCArg` enum,
|
||||
`sendOSC(addr:args:)`, and `appendOSCString(_:into:)`.
|
||||
|
||||
After deletion, `Network` is still needed (`USBServer` uses it
|
||||
indirectly — actually `USBServer` imports its own `Network`). Remove
|
||||
`import Network` from `ARBodySession.swift` only if no symbol from it
|
||||
remains; if `NWConnection`/`NWEndpoint` no longer appear in the file,
|
||||
remove the import.
|
||||
|
||||
- [ ] **Step 2: Verify it compiles**
|
||||
|
||||
Run the verification command. Expected: `** BUILD SUCCEEDED **`. If the
|
||||
build reports an unused `import` or an unresolved symbol, fix it
|
||||
minimally (remove the dead import, or keep it if still referenced).
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift
|
||||
git commit -m "refactor(ios): drop legacy OSC sender"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Simplify ContentView
|
||||
|
||||
`ContentView` exposes OSC host/port text fields that no longer have a
|
||||
backing. Remove them; keep the USB status indicator and Start/Stop.
|
||||
|
||||
**Files:**
|
||||
- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift`
|
||||
|
||||
- [ ] **Step 1: Remove OSC state and UI**
|
||||
|
||||
In `ContentView`:
|
||||
- delete the `@State` properties `host`, `pythonPort`, `swiftPort`
|
||||
(currently lines 7-9);
|
||||
- in `controlPanel`, delete the `HStack { Text("Host") ... }` block and
|
||||
the `HStack { Text("Py") ... Text("Swift") ... }` block (the two
|
||||
rows of OSC text fields, currently lines 85-102);
|
||||
- in the Start/Stop button action, replace the `session.configure(
|
||||
host:pythonPort:swiftPort:sendEnvMesh:)` call with
|
||||
`session.configure(sendEnvMesh: sendEnvMesh)`.
|
||||
|
||||
Keep: `sendEnvMesh` toggle, Start/Stop button, status text, the USB
|
||||
status dot/label, and the bodies/frames/jointsPerSec line.
|
||||
|
||||
- [ ] **Step 2: Verify it compiles**
|
||||
|
||||
Run the verification command. Expected: `** BUILD SUCCEEDED **`. The
|
||||
three `#Preview` blocks at the end of the file construct
|
||||
`ContentView(useMockBackground:useMockSkeleton:)` — those parameters
|
||||
are unaffected; the previews must still compile.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift
|
||||
git commit -m "refactor(ios): drop OSC config from ContentView"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Final build verification
|
||||
|
||||
- [ ] **Step 1: Full clean build**
|
||||
|
||||
```bash
|
||||
cd iphone-arbody && xcodegen generate && \
|
||||
xcodebuild -project ARBodyTracker.xcodeproj -scheme ARBodyTracker \
|
||||
-sdk iphonesimulator -destination 'generic/platform=iOS Simulator' \
|
||||
-configuration Debug clean build
|
||||
```
|
||||
|
||||
Expected: `** BUILD SUCCEEDED **`, zero errors.
|
||||
|
||||
- [ ] **Step 2: Confirm no OSC references remain**
|
||||
|
||||
```bash
|
||||
grep -rn -E "OSC|57128|57129|openUDP|sendDatagram" \
|
||||
iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/
|
||||
```
|
||||
|
||||
Expected: no matches in `ARBodySession.swift` or `ContentView.swift`.
|
||||
(`USBServer.swift` and `VideoEncoder.swift` never had OSC.) Comments
|
||||
mentioning history are acceptable; live OSC code is not.
|
||||
|
||||
- [ ] **Step 3: Commit any cleanup** (only if Step 2 found stragglers)
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- **Spec coverage:** This plan implements the spec's `VideoEncoder`
|
||||
unit and the `ARBodySession` "exposes video frames / OSC sender
|
||||
removed" requirement. `ARBodySession` already builds and sends
|
||||
`SkeletonPayload` over `USBServer` (delivered via the recovery
|
||||
branch + Plan 1), so no skeleton-path task is needed. `ContentView`
|
||||
simplification follows from OSC removal.
|
||||
- **Placeholders:** none — every step has concrete code or an exact
|
||||
command and expected output.
|
||||
- **Type consistency:** `VideoPayload`, `FrameTag.video`,
|
||||
`USBServer.send(tag:pid:timestamp:payload:)` are used consistently
|
||||
with their Plan 1 / `AVLiveWire` definitions. `VideoEncoder.start`
|
||||
takes `Int32` width/height matching `CVPixelBufferGetWidth`'s `Int`
|
||||
cast to `Int32`.
|
||||
- **Known risk:** the `VideoEncoder` VideoToolbox code compiles on the
|
||||
simulator but HEVC hardware encoding and the exact access-unit /
|
||||
parameter-set byte layout can only be validated on a physical
|
||||
device. Plan 3's `VideoDecoder` must agree with the framing chosen
|
||||
here (length-prefixed parameter sets prepended to keyframe payloads);
|
||||
this is the integration seam to verify when Plan 3 is built.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,549 @@
|
||||
# macOS Multi-HMR Mesh Implementation Plan (Plan 3b of 3)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add the dense-mesh half of the macOS pipeline — run Multi-HMR (CoreML) on the USB video stream inside `AVLiveBody`, fuse the result with the ARKit skeleton, and render the SMPL-X body mesh.
|
||||
|
||||
**Architecture:** `VideoDecoder` (Plan 3a) already turns `.video` frames into `CVPixelBuffer`s. This plan adds `MultiHMRCoreML`, a Swift wrapper around the bundled `multihmr_full_672_s.mlpackage`: it preprocesses a pixel buffer into the model's two `MLMultiArray` inputs, runs inference, and parses up to 4 detected persons (10475-vertex SMPL-X meshes). `BodyFusion` associates each mesh with the ARKit skeleton from `USBSkeletonConsumer` and corrects pelvis depth. The existing `MeshRenderer` (which already renders 10475-vertex SMPL-X meshes from its OSC server) is fed from the fusion output.
|
||||
|
||||
**Tech Stack:** Swift 5, macOS 15, CoreML, CoreVideo/CoreImage, RealityKit, `AVLiveWire`, `XCTest`. Build verifies on the host with `swift build` / `swift test`.
|
||||
|
||||
**Companion spec:** `docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md`
|
||||
**Prerequisites:** Plan 1, 2, 3a (merged); the working CoreML model (voie 2).
|
||||
|
||||
---
|
||||
|
||||
## The model — exact I/O contract
|
||||
|
||||
The reference implementation is `data_only_viz/multihmr_coreml.py` (Python, validated). The Swift wrapper must mirror it:
|
||||
|
||||
- **File:** `~/.cache/av-live-multihmr/multihmr_full_672_s.mlpackage` (204 MB, FP32). Not in git (`*.mlpackage` is gitignored).
|
||||
- **Load:** an `.mlpackage` must be compiled to `.mlmodelc` (`MLModel.compileModel(at:)`) before `MLModel(contentsOf:configuration:)`. Use `MLComputeUnits.cpuAndGPU` (benched best: ~139 ms standalone).
|
||||
- **Inputs** (an `MLDictionaryFeatureProvider` with two `MLMultiArray`s):
|
||||
- `"image"` — shape `[1, 3, 672, 672]`, Float32, RGB, **ImageNet-normalized**: `(v - mean) / std`, mean `[0.485, 0.456, 0.406]`, std `[0.229, 0.224, 0.225]` per channel. Feeding raw `[0,1]` collapses all scores (the "0 detections" bug).
|
||||
- `"cam_K"` — shape `[1, 3, 3]`, Float32, camera intrinsics.
|
||||
- **Outputs** (fixed K=4 persons):
|
||||
- `var_2420` — v3d `[4, 10475, 3]` vertices
|
||||
- `var_2423` — transl `[4, 1, 3]` pelvis translation
|
||||
- `var_2436` — scores `[4]`
|
||||
- `var_2439` — betas `[4, 10]`, `var_2442` — expression `[4, 10]` (unused here)
|
||||
- **Detection:** keep person `k` when `scores[k] >= 0.3`.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility |
|
||||
|------|----------------|
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/Resources/multihmr_full_672_s.mlpackage` | NEW (build input, gitignored). Copied from `~/.cache/av-live-multihmr/` by a setup step |
|
||||
| `launcher/AV-Live-Body/Package.swift` | MODIFY. Declare the `.mlpackage` as a `.copy` resource |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/MultiHMRCoreML.swift` | NEW. Load the model; `CVPixelBuffer` → inputs → inference → `[MultiHMRPerson]` |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/BodyFusion.swift` | NEW. Associate ARKit skeleton ↔ Multi-HMR person; pelvis-depth correction |
|
||||
| `launcher/AV-Live-Body/Tests/AVLiveBodyTests/BodyFusionTests.swift` | NEW. Pure association/correction logic tests |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift` | MODIFY. Drive `VideoDecoder` → `MultiHMRCoreML` → `BodyFusion` → `MeshRenderer` |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/MeshRenderer.swift` | REFERENCE — reuse its existing `updatePersons`-style entry point for 10475-vertex meshes |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Bundle the model + loader
|
||||
|
||||
**Files:**
|
||||
- Create (copy): `launcher/AV-Live-Body/Sources/AVLiveBody/Resources/multihmr_full_672_s.mlpackage`
|
||||
- Modify: `launcher/AV-Live-Body/Package.swift`
|
||||
|
||||
- [ ] **Step 1: Copy the model into the package resources**
|
||||
|
||||
The model is a build input that cannot live in git. Copy it:
|
||||
|
||||
```bash
|
||||
mkdir -p launcher/AV-Live-Body/Sources/AVLiveBody/Resources
|
||||
cp -R ~/.cache/av-live-multihmr/multihmr_full_672_s.mlpackage \
|
||||
launcher/AV-Live-Body/Sources/AVLiveBody/Resources/
|
||||
```
|
||||
|
||||
Verify it is gitignored (root `.gitignore` has `*.mlpackage`):
|
||||
|
||||
```bash
|
||||
git check-ignore launcher/AV-Live-Body/Sources/AVLiveBody/Resources/multihmr_full_672_s.mlpackage
|
||||
```
|
||||
|
||||
Expected: the path is printed (it is ignored — it must NOT be committed).
|
||||
|
||||
If the source file is absent, STOP — Plan 3b is blocked until voie 2's
|
||||
`.mlpackage` is regenerated (`data_only_viz/scripts/coreml_full_probe.py`).
|
||||
|
||||
- [ ] **Step 2: Declare the resource in Package.swift**
|
||||
|
||||
In `launcher/AV-Live-Body/Package.swift`, add to the `AVLiveBody`
|
||||
executable target's `resources:` array (next to the existing
|
||||
`smplx_faces.bin` / `scene.metal` copies):
|
||||
|
||||
```swift
|
||||
.copy("Resources/multihmr_full_672_s.mlpackage"),
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify the build still resolves resources**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift build`
|
||||
Expected: build succeeds; the `.mlpackage` is copied into the bundle.
|
||||
|
||||
- [ ] **Step 4: Commit (Package.swift only — the model is gitignored)**
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Package.swift
|
||||
git commit -m "build(av-live-body): bundle Multi-HMR mlpackage"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: MultiHMRCoreML
|
||||
|
||||
`MultiHMRCoreML` loads the bundled model, preprocesses a `CVPixelBuffer`
|
||||
into the two model inputs, runs inference, and returns detected persons.
|
||||
|
||||
**Files:**
|
||||
- Create: `launcher/AV-Live-Body/Sources/AVLiveBody/MultiHMRCoreML.swift`
|
||||
|
||||
- [ ] **Step 1: Write the implementation**
|
||||
|
||||
`launcher/AV-Live-Body/Sources/AVLiveBody/MultiHMRCoreML.swift`:
|
||||
|
||||
```swift
|
||||
import CoreML
|
||||
import CoreVideo
|
||||
import CoreImage
|
||||
import Foundation
|
||||
|
||||
/// One detected SMPL-X body from Multi-HMR.
|
||||
struct MultiHMRPerson {
|
||||
var vertices: [SIMD3<Float>] // 10475 SMPL-X verts, model space
|
||||
var translation: SIMD3<Float> // pelvis translation
|
||||
var score: Float
|
||||
}
|
||||
|
||||
/// CoreML wrapper around the bundled `multihmr_full_672_s.mlpackage`.
|
||||
/// Mirrors `data_only_viz/multihmr_coreml.py`: two MLMultiArray inputs
|
||||
/// (`image` 1x3x672x672 ImageNet-normalized, `cam_K` 1x3x3), fixed
|
||||
/// K=4 person outputs.
|
||||
final class MultiHMRCoreML {
|
||||
static let inputSize = 672
|
||||
static let vertexCount = 10475
|
||||
static let maxPersons = 4
|
||||
private static let detThreshold: Float = 0.3
|
||||
private static let normMean: [Float] = [0.485, 0.456, 0.406]
|
||||
private static let normStd: [Float] = [0.229, 0.224, 0.225]
|
||||
|
||||
private let model: MLModel
|
||||
private let ciContext = CIContext()
|
||||
|
||||
/// Loads the bundled model. Returns nil if the resource or load
|
||||
/// fails — callers fall back to skeleton-only rendering.
|
||||
init?() {
|
||||
guard let url = Bundle.module.url(
|
||||
forResource: "multihmr_full_672_s",
|
||||
withExtension: "mlpackage") else {
|
||||
NSLog("MultiHMRCoreML: mlpackage resource missing")
|
||||
return nil
|
||||
}
|
||||
let cfg = MLModelConfiguration()
|
||||
cfg.computeUnits = .cpuAndGPU
|
||||
do {
|
||||
let compiled = try MLModel.compileModel(at: url)
|
||||
model = try MLModel(contentsOf: compiled, configuration: cfg)
|
||||
} catch {
|
||||
NSLog("MultiHMRCoreML: load failed %@",
|
||||
String(describing: error))
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
/// Run inference on one camera frame. `cameraK` is the 3x3 camera
|
||||
/// intrinsics row-major.
|
||||
func infer(_ pixelBuffer: CVPixelBuffer,
|
||||
cameraK: [Float]) -> [MultiHMRPerson] {
|
||||
guard let image = makeImageInput(pixelBuffer),
|
||||
let k = makeKInput(cameraK) else { return [] }
|
||||
let inputs: [String: MLFeatureValue] = [
|
||||
"image": MLFeatureValue(multiArray: image),
|
||||
"cam_K": MLFeatureValue(multiArray: k),
|
||||
]
|
||||
guard let provider = try? MLDictionaryFeatureProvider(
|
||||
dictionary: inputs),
|
||||
let out = try? model.prediction(from: provider) else {
|
||||
return []
|
||||
}
|
||||
return parse(out)
|
||||
}
|
||||
|
||||
// MARK: - Input preprocessing
|
||||
|
||||
/// `CVPixelBuffer` -> [1,3,672,672] Float32, RGB, ImageNet-normed.
|
||||
private func makeImageInput(_ pb: CVPixelBuffer) -> MLMultiArray? {
|
||||
let n = Self.inputSize
|
||||
// Resize to n x n BGRA via CoreImage.
|
||||
let ci = CIImage(cvPixelBuffer: pb)
|
||||
let sx = CGFloat(n) / ci.extent.width
|
||||
let sy = CGFloat(n) / ci.extent.height
|
||||
let scaled = ci.transformed(
|
||||
by: CGAffineTransform(scaleX: sx, y: sy))
|
||||
var dst: CVPixelBuffer?
|
||||
CVPixelBufferCreate(kCFAllocatorDefault, n, n,
|
||||
kCVPixelFormatType_32BGRA, nil, &dst)
|
||||
guard let dst else { return nil }
|
||||
ciContext.render(scaled, to: dst)
|
||||
CVPixelBufferLockBaseAddress(dst, .readOnly)
|
||||
defer { CVPixelBufferUnlockBaseAddress(dst, .readOnly) }
|
||||
guard let base = CVPixelBufferGetBaseAddress(dst) else {
|
||||
return nil
|
||||
}
|
||||
let rowBytes = CVPixelBufferGetBytesPerRow(dst)
|
||||
let px = base.assumingMemoryBound(to: UInt8.self)
|
||||
guard let arr = try? MLMultiArray(
|
||||
shape: [1, 3, NSNumber(value: n), NSNumber(value: n)],
|
||||
dataType: .float32) else { return nil }
|
||||
let ptr = arr.dataPointer.assumingMemoryBound(to: Float.self)
|
||||
let plane = n * n
|
||||
for y in 0..<n {
|
||||
for x in 0..<n {
|
||||
let p = y * rowBytes + x * 4 // BGRA
|
||||
let b = Float(px[p]) / 255.0
|
||||
let g = Float(px[p + 1]) / 255.0
|
||||
let r = Float(px[p + 2]) / 255.0
|
||||
let idx = y * n + x
|
||||
ptr[idx] =
|
||||
(r - Self.normMean[0]) / Self.normStd[0]
|
||||
ptr[plane + idx] =
|
||||
(g - Self.normMean[1]) / Self.normStd[1]
|
||||
ptr[2 * plane + idx] =
|
||||
(b - Self.normMean[2]) / Self.normStd[2]
|
||||
}
|
||||
}
|
||||
return arr
|
||||
}
|
||||
|
||||
/// 9 row-major intrinsics -> [1,3,3] Float32.
|
||||
private func makeKInput(_ k: [Float]) -> MLMultiArray? {
|
||||
guard k.count == 9,
|
||||
let arr = try? MLMultiArray(
|
||||
shape: [1, 3, 3], dataType: .float32) else { return nil }
|
||||
let ptr = arr.dataPointer.assumingMemoryBound(to: Float.self)
|
||||
for i in 0..<9 { ptr[i] = k[i] }
|
||||
return arr
|
||||
}
|
||||
|
||||
// MARK: - Output parsing
|
||||
|
||||
private func parse(_ out: MLFeatureProvider) -> [MultiHMRPerson] {
|
||||
guard let v3d = out.featureValue(for: "var_2420")?
|
||||
.multiArrayValue,
|
||||
let transl = out.featureValue(for: "var_2423")?
|
||||
.multiArrayValue,
|
||||
let scores = out.featureValue(for: "var_2436")?
|
||||
.multiArrayValue else { return [] }
|
||||
var persons: [MultiHMRPerson] = []
|
||||
let vc = Self.vertexCount
|
||||
for k in 0..<Self.maxPersons {
|
||||
let score = scores[k].floatValue
|
||||
if score < Self.detThreshold { continue }
|
||||
var verts = [SIMD3<Float>](
|
||||
repeating: .zero, count: vc)
|
||||
let base = k * vc * 3
|
||||
for i in 0..<vc {
|
||||
let o = base + i * 3
|
||||
verts[i] = SIMD3(v3d[o].floatValue,
|
||||
v3d[o + 1].floatValue,
|
||||
v3d[o + 2].floatValue)
|
||||
}
|
||||
let tb = k * 3
|
||||
persons.append(MultiHMRPerson(
|
||||
vertices: verts,
|
||||
translation: SIMD3(transl[tb].floatValue,
|
||||
transl[tb + 1].floatValue,
|
||||
transl[tb + 2].floatValue),
|
||||
score: score))
|
||||
}
|
||||
return persons
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify it compiles**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift build`
|
||||
Expected: build succeeds. `Bundle.module` exists because the target
|
||||
has resources. If a CoreML signature differs on this SDK, fix
|
||||
minimally; the I/O contract (two named MLMultiArray inputs, the three
|
||||
named outputs) must be preserved.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/MultiHMRCoreML.swift
|
||||
git commit -m "feat(av-live-body): Multi-HMR CoreML wrapper"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: BodyFusion
|
||||
|
||||
`BodyFusion` is pure logic: given the ARKit 91-joint skeleton frames
|
||||
(from `USBSkeletonConsumer`) and the Multi-HMR persons, associate each
|
||||
mesh with the nearest skeleton and lock the mesh pelvis depth to the
|
||||
ARKit pelvis Z (the LiDAR-anchored, metrically-correct depth).
|
||||
|
||||
**Files:**
|
||||
- Create: `launcher/AV-Live-Body/Sources/AVLiveBody/BodyFusion.swift`
|
||||
- Test: `launcher/AV-Live-Body/Tests/AVLiveBodyTests/BodyFusionTests.swift`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`launcher/AV-Live-Body/Tests/AVLiveBodyTests/BodyFusionTests.swift`:
|
||||
|
||||
```swift
|
||||
import XCTest
|
||||
import AVLiveWire
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class BodyFusionTests: XCTestCase {
|
||||
private func skeleton(pelvisZ: Float)
|
||||
-> ArkitOSCListener.ArkitBodyFrame {
|
||||
var f = ArkitOSCListener.ArkitBodyFrame()
|
||||
f.pid = 0
|
||||
// ARKit body skeleton joint 0 is the hips/pelvis root.
|
||||
f.joints[0] = SIMD3(0, 0, pelvisZ)
|
||||
f.hasJoint[0] = true
|
||||
return f
|
||||
}
|
||||
|
||||
func testPelvisDepthOverride() {
|
||||
let mesh = MultiHMRPerson(
|
||||
vertices: [SIMD3<Float>](repeating: .zero, count: 1),
|
||||
translation: SIMD3(0, 0, -1.0), score: 0.9)
|
||||
let fused = BodyFusion.fuse(
|
||||
persons: [mesh], skeletons: [0: skeleton(pelvisZ: -2.5)])
|
||||
XCTAssertEqual(fused.count, 1)
|
||||
XCTAssertEqual(fused[0].translation.z, -2.5, accuracy: 1e-4)
|
||||
}
|
||||
|
||||
func testPassthroughWhenNoSkeleton() {
|
||||
let mesh = MultiHMRPerson(
|
||||
vertices: [SIMD3<Float>](repeating: .zero, count: 1),
|
||||
translation: SIMD3(0, 0, -1.0), score: 0.9)
|
||||
let fused = BodyFusion.fuse(persons: [mesh], skeletons: [:])
|
||||
XCTAssertEqual(fused[0].translation.z, -1.0, accuracy: 1e-4)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift test --filter BodyFusionTests`
|
||||
Expected: FAIL — `BodyFusion` undefined.
|
||||
|
||||
- [ ] **Step 3: Write the implementation**
|
||||
|
||||
`launcher/AV-Live-Body/Sources/AVLiveBody/BodyFusion.swift`:
|
||||
|
||||
```swift
|
||||
import AVLiveWire
|
||||
import Foundation
|
||||
import simd
|
||||
|
||||
/// Associates Multi-HMR meshes with ARKit skeletons and corrects the
|
||||
/// mesh pelvis depth. Pure, stateless — unit-testable.
|
||||
enum BodyFusion {
|
||||
/// ARKit body skeleton root (hips) joint index.
|
||||
static let pelvisJoint = 0
|
||||
|
||||
/// Returns the persons with `translation.z` of each replaced by
|
||||
/// the matching ARKit skeleton's pelvis Z when one is available.
|
||||
/// Association is nearest-translation; with a single skeleton and
|
||||
/// a single dominant person this is exact.
|
||||
static func fuse(persons: [MultiHMRPerson],
|
||||
skeletons: [Int: ArkitOSCListener.ArkitBodyFrame])
|
||||
-> [MultiHMRPerson] {
|
||||
// Collect candidate ARKit pelvis depths.
|
||||
let pelvisZs: [Float] = skeletons.values.compactMap { s in
|
||||
guard pelvisJoint < s.hasJoint.count,
|
||||
s.hasJoint[pelvisJoint] else { return nil }
|
||||
return s.joints[pelvisJoint].z
|
||||
}
|
||||
guard !pelvisZs.isEmpty else { return persons }
|
||||
// Highest-scoring person is the primary; lock its depth to the
|
||||
// single ARKit skeleton (ARKit tracks one body). Others pass
|
||||
// through unchanged.
|
||||
guard let primaryIdx = persons.indices.max(by: {
|
||||
persons[$0].score < persons[$1].score
|
||||
}) else { return persons }
|
||||
var out = persons
|
||||
out[primaryIdx].translation.z = pelvisZs[0]
|
||||
return out
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the test to verify it passes**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift test --filter BodyFusionTests`
|
||||
Expected: PASS, 2 tests.
|
||||
|
||||
- [ ] **Step 5: Run the full suite + commit**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift test` — Expected: all pass
|
||||
(9: prior 7 + 2).
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/BodyFusion.swift launcher/AV-Live-Body/Tests/AVLiveBodyTests/BodyFusionTests.swift
|
||||
git commit -m "feat(av-live-body): ARKit-to-mesh body fusion"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Wire the mesh pipeline
|
||||
|
||||
Drive the chain: `USBSkeletonConsumer.onVideo` → `VideoDecoder` →
|
||||
`MultiHMRCoreML` → `BodyFusion` → `MeshRenderer`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift`
|
||||
- Reference: `launcher/AV-Live-Body/Sources/AVLiveBody/MeshRenderer.swift`
|
||||
|
||||
- [ ] **Step 1: Read `MeshRenderer.swift`**
|
||||
|
||||
Identify the method that ingests SMPL-X persons (the OSC `SMPX` server
|
||||
path calls it — likely `updatePersons(_:)` taking per-person 10475
|
||||
vertex arrays). Note its exact signature and the vertex/coordinate
|
||||
convention it expects.
|
||||
|
||||
- [ ] **Step 2: Add the mesh pipeline to `USBSkeletonConsumer`**
|
||||
|
||||
Give `USBSkeletonConsumer` an optional mesh pipeline. Add stored
|
||||
properties:
|
||||
|
||||
```swift
|
||||
private let videoDecoder = VideoDecoder()
|
||||
private let multiHMR = MultiHMRCoreML()
|
||||
/// Set by the app to receive fused mesh persons on the main queue.
|
||||
var onMeshPersons: (([MultiHMRPerson]) -> Void)?
|
||||
/// Camera intrinsics (row-major 3x3) for Multi-HMR; a sane default
|
||||
/// is the iPhone main-camera focal at 672 px until a `.meta` frame
|
||||
/// supplies the real values.
|
||||
private var cameraK: [Float] = [
|
||||
672, 0, 336,
|
||||
0, 672, 336,
|
||||
0, 0, 1,
|
||||
]
|
||||
```
|
||||
|
||||
In `init()` (or `start()`), wire the decoder to the model:
|
||||
|
||||
```swift
|
||||
videoDecoder.onFrame = { [weak self] pixelBuffer in
|
||||
guard let self else { return }
|
||||
guard let hmr = self.multiHMR else { return }
|
||||
let raw = hmr.infer(pixelBuffer, cameraK: self.cameraK)
|
||||
let latestSkeletons = self.bodies
|
||||
let fused = BodyFusion.fuse(
|
||||
persons: raw, skeletons: latestSkeletons)
|
||||
DispatchQueue.main.async {
|
||||
self.onMeshPersons?(fused)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Change the `.video` branch of `route(_:)` so it feeds the decoder
|
||||
instead of only forwarding the payload:
|
||||
|
||||
```swift
|
||||
case .video:
|
||||
guard let payload =
|
||||
VideoPayload(decoding: frame.payload) else { return }
|
||||
videoDecoder.decode(payload)
|
||||
```
|
||||
|
||||
(`onVideo` may be kept for diagnostics or removed — keeping it is
|
||||
harmless; if removed, delete its declaration too.)
|
||||
|
||||
- [ ] **Step 3: Feed `MeshRenderer` from the app**
|
||||
|
||||
In `AVLiveBodyApp.swift`'s `ContentView` `.onAppear` (or where the
|
||||
renderers are wired), set `usbConsumer.onMeshPersons` to call the
|
||||
`MeshRenderer` ingest method identified in Step 1, converting
|
||||
`[MultiHMRPerson]` (vertices + fused translation) into whatever shape
|
||||
that method expects. The translation from `BodyFusion` positions each
|
||||
mesh; the 10475 vertices are the SMPL-X surface.
|
||||
|
||||
If `MeshRenderer`'s ingest method is not reachable from `ContentView`
|
||||
(it may be owned by `BodyView`), thread an `onMeshPersons` closure the
|
||||
same way `usbConsumer` itself was threaded in Plan 3a Task 4.
|
||||
|
||||
- [ ] **Step 4: Verify build + tests**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift build && swift test`
|
||||
Expected: build succeeds; all tests pass (9).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift
|
||||
git commit -m "feat(av-live-body): wire Multi-HMR mesh pipeline"
|
||||
```
|
||||
|
||||
(Include `BodyView.swift` in the commit if Step 3 threaded a closure
|
||||
through it.)
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Final verification
|
||||
|
||||
- [ ] **Step 1: Clean build + full test suite**
|
||||
|
||||
```bash
|
||||
cd launcher/AV-Live-Body && swift build && swift test
|
||||
```
|
||||
|
||||
Expected: build succeeds; all 9 tests pass.
|
||||
|
||||
- [ ] **Step 2: Confirm the model is bundled, not committed**
|
||||
|
||||
```bash
|
||||
git status --porcelain | grep mlpackage || echo "model not staged — correct"
|
||||
ls -d launcher/AV-Live-Body/Sources/AVLiveBody/Resources/multihmr_full_672_s.mlpackage
|
||||
```
|
||||
|
||||
Expected: the model directory exists on disk but is NOT staged in git.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- **Spec coverage:** This plan implements the spec's `MultiHMRCoreML`,
|
||||
`BodyFusion`, and the mesh-render wiring — the dense-mesh half
|
||||
deferred from Plan 3a. With Plan 3b done, the full spec
|
||||
(`USBClient`/`StreamDemuxer`/`VideoDecoder`/`MultiHMRCoreML`/
|
||||
`BodyFusion` + renderers) is covered.
|
||||
- **Placeholders:** none — new files carry complete code; modify tasks
|
||||
cite exact files and instruct reading `MeshRenderer.swift` for the
|
||||
one signature this plan cannot reproduce blind.
|
||||
- **Type consistency:** `MultiHMRPerson` is produced by
|
||||
`MultiHMRCoreML.infer` and consumed by `BodyFusion.fuse` and
|
||||
`onMeshPersons`. The model I/O names (`image`, `cam_K`, `var_2420`,
|
||||
`var_2423`, `var_2436`) match `multihmr_coreml.py` exactly.
|
||||
- **Known risks:**
|
||||
1. **Bundling 204 MB** — `swift build` copies the `.mlpackage` into
|
||||
the app bundle; build is slower and the app is large. Acceptable
|
||||
per the owner's decision (FP32, validated).
|
||||
2. **`CVPixelBuffer` → tensor** — the CoreImage resize + manual
|
||||
BGRA→normalized-CHW packing is the most error-prone code here and
|
||||
needs on-device validation against `multihmr_coreml.py`'s output
|
||||
on the same frame. It also runs per-frame on the CPU — a perf
|
||||
hotspot; revisit with `vImage`/Metal if frame rate suffers.
|
||||
3. **~7.6 fps** — Multi-HMR is far below 30 fps; the mesh layer is
|
||||
slow while the skeleton (Plan 3a) stays real-time. `MeshRenderer`
|
||||
already interpolates meshes to ~60 fps between worker frames —
|
||||
reuse that, do not block the USB read loop on inference (the
|
||||
`videoDecoder.onFrame` callback already runs off the main queue).
|
||||
4. **`cameraK`** — a placeholder intrinsics matrix is used until a
|
||||
`.meta` frame carries the real values; absolute depth scale will
|
||||
be approximate until then. A future iteration should send camera
|
||||
intrinsics from the iPhone in a `.meta` frame.
|
||||
@@ -0,0 +1,655 @@
|
||||
# macOS USB Consumer Implementation Plan (Plan 3a of 3)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Make the macOS `AVLiveBody` app consume the iPhone's USB stream — connect via `usbmuxd`, demux `AVLiveWire` frames, render the 91-joint skeleton on screen, and HEVC-decode the video — without the Multi-HMR dense-mesh step (deferred to Plan 3b).
|
||||
|
||||
**Architecture:** A new `USBSkeletonConsumer` runs the blocking `UnixMuxTransport`/`USBClient` read loop on a dedicated background thread, feeds bytes through `StreamDemuxer`, and republishes `.skeleton` frames as `@Published` ARKit-shaped body frames plus a `.video` callback. `Skeleton3DRenderer`'s long-standing `// TODO: render yellow ARKit markers` (line 138) is completed so the 91-joint USB skeleton actually draws. A new `VideoDecoder` turns `.video` `VideoPayload`s into `CVPixelBuffer`s via `VTDecompressionSession`.
|
||||
|
||||
**Tech Stack:** Swift 5 (language mode v5), macOS 15, RealityKit, VideoToolbox, `AVLiveWire` (already a dependency of `AV-Live-Body`), `XCTest`.
|
||||
|
||||
**Companion spec:** `docs/superpowers/specs/2026-05-18-iphone-usb-body-link-design.md`
|
||||
**Prerequisites:** Plan 1 (transport, merged), Plan 2 (iOS capture, merged).
|
||||
**Out of scope:** `MultiHMRCoreML`, `BodyFusion`, dense-mesh rendering — Plan 3b, gated on a confirmed CoreML Multi-HMR `.mlpackage`.
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
`AV-Live-Body` is a macOS target — it builds on the host:
|
||||
|
||||
```bash
|
||||
cd launcher/AV-Live-Body && swift build
|
||||
cd launcher/AV-Live-Body && swift test
|
||||
```
|
||||
|
||||
Each task ends with `swift build` (and `swift test` where a test was
|
||||
added) succeeding.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Responsibility |
|
||||
|------|----------------|
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift` | NEW. Background USB read loop → `StreamDemuxer` → `@Published` body frames + video callback |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift` | NEW. `VTDecompressionSession` HEVC decode: `VideoPayload` → `CVPixelBuffer` |
|
||||
| `launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift` | NEW. Unit test for the `SkeletonPayload` → `ArkitBodyFrame` mapping |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift` | MODIFY. Complete the line-138 TODO: draw 91 USB-skeleton joint markers |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/ArkitOSCListener.swift` | REFERENCE only — reuse its nested `ArkitBodyFrame` type |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift` | MODIFY. Own a `USBSkeletonConsumer`, start it in `.onAppear` |
|
||||
| `launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift` | MODIFY. Thread the consumer into `Skeleton3DRenderer.attach` |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: USBSkeletonConsumer
|
||||
|
||||
`USBSkeletonConsumer` owns the blocking USB read loop on a background
|
||||
`Thread`. It reconnects on drop. It republishes `.skeleton` frames as
|
||||
`ArkitOSCListener.ArkitBodyFrame` (the existing 91-joint body type, so
|
||||
`Skeleton3DRenderer` can consume them with no new type) and forwards
|
||||
`.video` payloads via a callback. It is **not** `@MainActor`: the loop
|
||||
runs off-main and hops to main only for `@Published` writes — the same
|
||||
pattern as `ArkitOSCListener`.
|
||||
|
||||
**Files:**
|
||||
- Create: `launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift`
|
||||
- Test: `launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift`
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
`launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift`:
|
||||
|
||||
```swift
|
||||
import XCTest
|
||||
import AVLiveWire
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class USBSkeletonConsumerTests: XCTestCase {
|
||||
func testSkeletonPayloadMapsToBodyFrame() {
|
||||
var p = SkeletonPayload()
|
||||
p.joints[0] = SIMD3(1, 2, 3)
|
||||
p.valid[0] = true
|
||||
p.joints[90] = SIMD3(-4, 5, -6)
|
||||
p.valid[90] = true
|
||||
let frame = USBSkeletonConsumer.bodyFrame(pid: 7, from: p)
|
||||
XCTAssertEqual(frame.pid, 7)
|
||||
XCTAssertEqual(frame.joints.count, 91)
|
||||
XCTAssertEqual(frame.hasJoint.count, 91)
|
||||
XCTAssertEqual(frame.joints[0], SIMD3(1, 2, 3))
|
||||
XCTAssertTrue(frame.hasJoint[0])
|
||||
XCTAssertEqual(frame.joints[90], SIMD3(-4, 5, -6))
|
||||
XCTAssertFalse(frame.hasJoint[1])
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the test to verify it fails**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests`
|
||||
Expected: FAIL — `USBSkeletonConsumer` undefined.
|
||||
|
||||
- [ ] **Step 3: Write the implementation**
|
||||
|
||||
`launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift`:
|
||||
|
||||
```swift
|
||||
import AVLiveWire
|
||||
import Combine
|
||||
import Foundation
|
||||
|
||||
/// Connects to the tethered iPhone over USB (usbmuxd), demuxes the
|
||||
/// AVLiveWire stream, and republishes skeleton frames (as the existing
|
||||
/// 91-joint `ArkitOSCListener.ArkitBodyFrame`) plus video payloads.
|
||||
/// The blocking transport runs on a dedicated background thread; only
|
||||
/// `@Published` writes hop to the main queue.
|
||||
final class USBSkeletonConsumer: ObservableObject {
|
||||
/// 91-joint body frames keyed by pid — same shape `Skeleton3DRenderer`
|
||||
/// already consumes from `ArkitOSCListener`.
|
||||
@Published var bodies: [Int: ArkitOSCListener.ArkitBodyFrame] = [:]
|
||||
@Published var connected = false
|
||||
|
||||
/// Called (on the main queue) for every decoded `.video` frame.
|
||||
var onVideo: ((VideoPayload) -> Void)?
|
||||
|
||||
/// TCP port the iPhone `USBServer` listens on (must match the iOS
|
||||
/// app's `USBServer.port`).
|
||||
static let devicePort: UInt16 = 7000
|
||||
|
||||
private let stateLock = NSLock()
|
||||
private var running = false
|
||||
private var thread: Thread?
|
||||
|
||||
private var isRunning: Bool {
|
||||
stateLock.lock(); defer { stateLock.unlock() }
|
||||
return running
|
||||
}
|
||||
|
||||
func start() {
|
||||
stateLock.lock()
|
||||
if running { stateLock.unlock(); return }
|
||||
running = true
|
||||
stateLock.unlock()
|
||||
let t = Thread { [weak self] in self?.loop() }
|
||||
t.name = "cc.avlive.usbconsumer"
|
||||
t.start()
|
||||
thread = t
|
||||
}
|
||||
|
||||
func stop() {
|
||||
stateLock.lock(); running = false; stateLock.unlock()
|
||||
}
|
||||
|
||||
/// Pure mapping `SkeletonPayload` -> `ArkitBodyFrame`. Static so it
|
||||
/// is unit-testable without a transport.
|
||||
static func bodyFrame(pid: Int, from p: SkeletonPayload)
|
||||
-> ArkitOSCListener.ArkitBodyFrame {
|
||||
var f = ArkitOSCListener.ArkitBodyFrame()
|
||||
f.pid = pid
|
||||
f.joints = p.joints
|
||||
f.hasJoint = p.valid
|
||||
f.seenAt = CFAbsoluteTimeGetCurrent()
|
||||
return f
|
||||
}
|
||||
|
||||
// MARK: - Background read loop
|
||||
|
||||
private func loop() {
|
||||
while isRunning {
|
||||
guard let transport = UnixMuxTransport() else {
|
||||
Thread.sleep(forTimeInterval: 1.0); continue
|
||||
}
|
||||
let client = USBClient(transport: transport)
|
||||
guard let dev = client.listDevices().first,
|
||||
client.connect(deviceID: dev,
|
||||
port: Self.devicePort) else {
|
||||
transport.close()
|
||||
Thread.sleep(forTimeInterval: 1.0); continue
|
||||
}
|
||||
publishConnected(true)
|
||||
var demux = StreamDemuxer()
|
||||
while isRunning {
|
||||
guard let chunk = transport.readStream(),
|
||||
!chunk.isEmpty else { break }
|
||||
for frame in demux.feed(chunk) { route(frame) }
|
||||
}
|
||||
transport.close()
|
||||
publishConnected(false)
|
||||
if isRunning { Thread.sleep(forTimeInterval: 1.0) }
|
||||
}
|
||||
}
|
||||
|
||||
private func route(_ frame: StreamDemuxer.Frame) {
|
||||
switch frame.header.tag {
|
||||
case .skeleton:
|
||||
guard let payload =
|
||||
SkeletonPayload(decoding: frame.payload) else { return }
|
||||
let pid = Int(frame.header.pid)
|
||||
let body = Self.bodyFrame(pid: pid, from: payload)
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.bodies[pid] = body
|
||||
}
|
||||
case .video:
|
||||
guard let payload =
|
||||
VideoPayload(decoding: frame.payload) else { return }
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.onVideo?(payload)
|
||||
}
|
||||
case .meta:
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
private func publishConnected(_ value: Bool) {
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.connected = value
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the test to verify it passes**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift test --filter USBSkeletonConsumerTests`
|
||||
Expected: PASS, 1 test.
|
||||
|
||||
If `ArkitOSCListener.ArkitBodyFrame` has no memberwise mutability or a
|
||||
different field set than `pid`/`joints`/`hasJoint`/`seenAt`, read
|
||||
`ArkitOSCListener.swift` and adjust `bodyFrame` to match the actual
|
||||
struct (it is a `struct ArkitBodyFrame: Equatable` with `var pid`,
|
||||
`var joints: [SIMD3<Float>]`, `var hasJoint: [Bool]`, `var seenAt`).
|
||||
|
||||
- [ ] **Step 5: Run the full suite + commit**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift test`
|
||||
Expected: PASS, all tests (7: prior 6 + this 1).
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift launcher/AV-Live-Body/Tests/AVLiveBodyTests/USBSkeletonConsumerTests.swift
|
||||
git commit -m "feat(av-live-body): USB skeleton consumer"
|
||||
```
|
||||
|
||||
(subject ≤50 chars; add a short body — the hook rejects subject-only.)
|
||||
|
||||
---
|
||||
|
||||
## Task 2: VideoDecoder
|
||||
|
||||
`VideoDecoder` turns `.video` `VideoPayload`s into `CVPixelBuffer`s. A
|
||||
keyframe payload carries the HEVC parameter sets prepended (each as a
|
||||
4-byte big-endian length prefix + NAL bytes — the format Plan 2's iOS
|
||||
`VideoEncoder` produces); the decoder builds its
|
||||
`CMVideoFormatDescription` from those, then decodes subsequent access
|
||||
units.
|
||||
|
||||
**Files:**
|
||||
- Create: `launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift`
|
||||
|
||||
- [ ] **Step 1: Write the implementation**
|
||||
|
||||
`launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift`:
|
||||
|
||||
```swift
|
||||
import AVLiveWire
|
||||
import CoreMedia
|
||||
import CoreVideo
|
||||
import Foundation
|
||||
import VideoToolbox
|
||||
|
||||
/// HEVC decoder. Feed `VideoPayload`s in; receive `CVPixelBuffer`s via
|
||||
/// `onFrame`. Keyframe payloads must carry the VPS/SPS/PPS parameter
|
||||
/// sets prepended as 4-byte-length-prefixed NAL units (the layout the
|
||||
/// iOS `VideoEncoder` emits); the decoder (re)builds its format
|
||||
/// description from those.
|
||||
final class VideoDecoder {
|
||||
var onFrame: ((CVPixelBuffer) -> Void)?
|
||||
|
||||
private var session: VTDecompressionSession?
|
||||
private var formatDesc: CMVideoFormatDescription?
|
||||
|
||||
/// Decode one access unit.
|
||||
func decode(_ payload: VideoPayload) {
|
||||
var au = payload.data
|
||||
if payload.isKeyframe {
|
||||
// Split the prepended parameter sets from the frame data.
|
||||
let (params, rest) = Self.splitParameterSets(au)
|
||||
if !params.isEmpty {
|
||||
rebuildFormat(params)
|
||||
}
|
||||
au = rest
|
||||
}
|
||||
guard let fmt = formatDesc, !au.isEmpty else { return }
|
||||
if session == nil { makeSession(fmt) }
|
||||
guard let session else { return }
|
||||
guard let block = Self.blockBuffer(au) else { return }
|
||||
var sample: CMSampleBuffer?
|
||||
var sampleSize = au.count
|
||||
guard CMSampleBufferCreateReady(
|
||||
allocator: kCFAllocatorDefault, dataBuffer: block,
|
||||
formatDescription: fmt, sampleCount: 1, sampleTimingEntryCount: 0,
|
||||
sampleTimingArray: nil, sampleSizeEntryCount: 1,
|
||||
sampleSizeArray: &sampleSize,
|
||||
sampleBufferOut: &sample) == noErr, let sample else { return }
|
||||
VTDecompressionSessionDecodeFrame(
|
||||
session, sampleBuffer: sample, flags: [],
|
||||
infoFlagsOut: nil) { [weak self] status, _, image, _, _ in
|
||||
guard status == noErr, let image else { return }
|
||||
self?.onFrame?(image)
|
||||
}
|
||||
}
|
||||
|
||||
func stop() {
|
||||
if let session { VTDecompressionSessionInvalidate(session) }
|
||||
session = nil
|
||||
formatDesc = nil
|
||||
}
|
||||
|
||||
deinit { stop() }
|
||||
|
||||
// MARK: - Helpers
|
||||
|
||||
/// Parameter sets are 4-byte-length-prefixed NAL units at the head
|
||||
/// of a keyframe payload. The first NAL whose type is a VCL slice
|
||||
/// marks the start of frame data — but to stay simple and robust,
|
||||
/// we treat every leading NAL as a parameter set until the running
|
||||
/// concatenation can build a valid HEVC format description; the
|
||||
/// remainder is the frame. Returns (parameterSetData, frameData).
|
||||
private static func splitParameterSets(_ data: Data)
|
||||
-> (Data, Data) {
|
||||
// Parameter set NALs for HEVC: VPS=32, SPS=33, PPS=34
|
||||
// (nal_unit_type = (firstByte >> 1) & 0x3F).
|
||||
var offset = 0
|
||||
let bytes = [UInt8](data)
|
||||
var paramEnd = 0
|
||||
while offset + 4 <= bytes.count {
|
||||
let len = (Int(bytes[offset]) << 24)
|
||||
| (Int(bytes[offset + 1]) << 16)
|
||||
| (Int(bytes[offset + 2]) << 8)
|
||||
| Int(bytes[offset + 3])
|
||||
let nalStart = offset + 4
|
||||
guard len > 0, nalStart + len <= bytes.count else { break }
|
||||
let nalType = (Int(bytes[nalStart]) >> 1) & 0x3F
|
||||
if nalType == 32 || nalType == 33 || nalType == 34 {
|
||||
offset = nalStart + len
|
||||
paramEnd = offset
|
||||
} else {
|
||||
break
|
||||
}
|
||||
}
|
||||
return (data.prefix(paramEnd),
|
||||
data.suffix(from: data.startIndex
|
||||
.advanced(by: paramEnd)))
|
||||
}
|
||||
|
||||
private func rebuildFormat(_ paramData: Data) {
|
||||
var sets: [[UInt8]] = []
|
||||
let bytes = [UInt8](paramData)
|
||||
var offset = 0
|
||||
while offset + 4 <= bytes.count {
|
||||
let len = (Int(bytes[offset]) << 24)
|
||||
| (Int(bytes[offset + 1]) << 16)
|
||||
| (Int(bytes[offset + 2]) << 8)
|
||||
| Int(bytes[offset + 3])
|
||||
let start = offset + 4
|
||||
guard len > 0, start + len <= bytes.count else { break }
|
||||
sets.append(Array(bytes[start..<start + len]))
|
||||
offset = start + len
|
||||
}
|
||||
guard sets.count >= 3 else { return }
|
||||
let pointers = sets.map { UnsafePointer<UInt8>($0) }
|
||||
let sizes = sets.map { $0.count }
|
||||
var fmt: CMFormatDescription?
|
||||
let status = pointers.withUnsafeBufferPointer { pBuf in
|
||||
sizes.withUnsafeBufferPointer { sBuf in
|
||||
CMVideoFormatDescriptionCreateFromHEVCParameterSets(
|
||||
allocator: kCFAllocatorDefault,
|
||||
parameterSetCount: sets.count,
|
||||
parameterSetPointers: pBuf.baseAddress!,
|
||||
parameterSetSizes: sBuf.baseAddress!,
|
||||
nalUnitHeaderLength: 4, extensions: nil,
|
||||
formatDescriptionOut: &fmt)
|
||||
}
|
||||
}
|
||||
if status == noErr, let fmt {
|
||||
formatDesc = fmt
|
||||
if let session { VTDecompressionSessionInvalidate(session) }
|
||||
session = nil
|
||||
}
|
||||
}
|
||||
|
||||
private func makeSession(_ fmt: CMVideoFormatDescription) {
|
||||
let attrs: [CFString: Any] = [
|
||||
kCVPixelBufferPixelFormatTypeKey:
|
||||
kCVPixelFormatType_32BGRA,
|
||||
]
|
||||
VTDecompressionSessionCreate(
|
||||
allocator: kCFAllocatorDefault, formatDescription: fmt,
|
||||
decoderSpecification: nil,
|
||||
imageBufferAttributes: attrs as CFDictionary,
|
||||
outputCallback: nil, decompressionSessionOut: &session)
|
||||
}
|
||||
|
||||
private static func blockBuffer(_ data: Data) -> CMBlockBuffer? {
|
||||
var block: CMBlockBuffer?
|
||||
guard CMBlockBufferCreateWithMemoryBlock(
|
||||
allocator: kCFAllocatorDefault, memoryBlock: nil,
|
||||
blockLength: data.count, blockAllocator: kCFAllocatorDefault,
|
||||
customBlockSource: nil, offsetToData: 0,
|
||||
dataLength: data.count, flags: 0,
|
||||
blockBufferOut: &block) == noErr, let block else {
|
||||
return nil
|
||||
}
|
||||
var ok = false
|
||||
data.withUnsafeBytes { raw in
|
||||
if CMBlockBufferReplaceDataBytes(
|
||||
with: raw.baseAddress!, blockBuffer: block,
|
||||
offsetIntoDestination: 0,
|
||||
dataLength: data.count) == noErr { ok = true }
|
||||
}
|
||||
return ok ? block : nil
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify it compiles**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift build`
|
||||
Expected: build succeeds. If a VideoToolbox/CoreMedia signature differs
|
||||
on this SDK, fix minimally — the behavior (build a format description
|
||||
from the prepended parameter sets, decode the rest) must be preserved.
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/VideoDecoder.swift
|
||||
git commit -m "feat(av-live-body): HEVC video decoder"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Render the 91-joint USB skeleton
|
||||
|
||||
`Skeleton3DRenderer` already subscribes to a 91-joint ARKit body
|
||||
publisher into `lastArkit` but never draws it — `Skeleton3DRenderer.swift:138`
|
||||
is `// TODO: render yellow ARKit markers from lastArkit in update()`.
|
||||
Complete it: draw the 91 joints as small yellow spheres.
|
||||
|
||||
**Files:**
|
||||
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift`
|
||||
|
||||
- [ ] **Step 1: Read the renderer**
|
||||
|
||||
Read `Skeleton3DRenderer.swift` fully. Note: `PersonEntities` (the
|
||||
per-pid entity struct), `lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame]`,
|
||||
`makePerson(pid:parent:)`, the `update(frames:)` 30 fps tick, and the
|
||||
RealityKit space conversion used for MediaPipe joints
|
||||
(`SIMD3(k.x, -k.y, -k.z)`).
|
||||
|
||||
- [ ] **Step 2: Add 91 ARKit marker entities to `PersonEntities`**
|
||||
|
||||
In the `PersonEntities` struct, add a field:
|
||||
|
||||
```swift
|
||||
var arkitMarkers: [ModelEntity] // 91 yellow ARKit joint spheres
|
||||
```
|
||||
|
||||
In `makePerson(pid:parent:)`, after the hand spheres are built, create
|
||||
91 yellow marker spheres (reuse the `jointRadius`-sized sphere mesh, a
|
||||
yellow `SimpleMaterial`), parent them to `root`, start them disabled,
|
||||
and include `arkitMarkers:` in the returned `PersonEntities(...)`:
|
||||
|
||||
```swift
|
||||
let arkitMat = SimpleMaterial(
|
||||
color: .systemYellow, roughness: 0.6, isMetallic: false)
|
||||
var arkitMarkers: [ModelEntity] = []
|
||||
arkitMarkers.reserveCapacity(91)
|
||||
for _ in 0..<91 {
|
||||
let e = ModelEntity(mesh: sphereMesh, materials: [arkitMat])
|
||||
e.isEnabled = false
|
||||
root.addChild(e)
|
||||
arkitMarkers.append(e)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Draw the ARKit markers each tick**
|
||||
|
||||
Replace the line `// TODO: render yellow ARKit markers from lastArkit in update()`
|
||||
(`Skeleton3DRenderer.swift:138`) — leave the comment removed — and add,
|
||||
at the end of `update(frames:)` (after the existing per-pid loop), a
|
||||
call to a new private method `applyArkit()`. Then add the method:
|
||||
|
||||
```swift
|
||||
/// Draw the 91-joint ARKit/USB skeletons as yellow joint markers.
|
||||
/// ARKit joints are world-space metric; convert to RealityKit
|
||||
/// space (x, y, z) -> (x, -y, -z) like the MediaPipe path.
|
||||
private func applyArkit() {
|
||||
for (pid, entities) in persons {
|
||||
guard let frame = lastArkit[pid] else {
|
||||
for m in entities.arkitMarkers { m.isEnabled = false }
|
||||
continue
|
||||
}
|
||||
let n = min(91, entities.arkitMarkers.count,
|
||||
frame.joints.count)
|
||||
for i in 0..<n {
|
||||
let marker = entities.arkitMarkers[i]
|
||||
if frame.hasJoint[i] {
|
||||
let j = frame.joints[i]
|
||||
marker.transform.translation =
|
||||
SIMD3<Float>(j.x, -j.y, -j.z)
|
||||
marker.isEnabled = true
|
||||
} else {
|
||||
marker.isEnabled = false
|
||||
}
|
||||
}
|
||||
for i in n..<entities.arkitMarkers.count {
|
||||
entities.arkitMarkers[i].isEnabled = false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note: `applyArkit()` iterates `persons`, which is only populated for
|
||||
pids seen in the MediaPipe `frames`. If the USB skeleton must show
|
||||
when there is no MediaPipe pose, also create a `PersonEntities` for
|
||||
each pid present in `lastArkit`. To keep Task 3 minimal, in
|
||||
`update(frames:)` before `applyArkit()`, ensure entities exist for
|
||||
ARKit-only pids:
|
||||
|
||||
```swift
|
||||
for pid in lastArkit.keys where persons[pid] == nil {
|
||||
persons[pid] = makePerson(pid: pid, parent: anchor)
|
||||
lastSeenAt[pid] = now
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Verify build + tests**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift build` — Expected: succeeds.
|
||||
Run: `cd launcher/AV-Live-Body && swift test` — Expected: all tests
|
||||
still pass (no regression).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/Skeleton3DRenderer.swift
|
||||
git commit -m "feat(av-live-body): render 91-joint USB skeleton"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Wire the consumer into the app
|
||||
|
||||
Construct `USBSkeletonConsumer` in the app, start/stop it with the
|
||||
scene, and feed it into `Skeleton3DRenderer` in place of (or alongside)
|
||||
`ArkitOSCListener`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift`
|
||||
- Modify: `launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift`
|
||||
|
||||
- [ ] **Step 1: Read the two files**
|
||||
|
||||
Read `AVLiveBodyApp.swift` and `BodyView.swift`. Identify: where the
|
||||
`@StateObject` listeners are declared in `ContentView`, where `.onAppear`
|
||||
starts them, how `ArkitOSCListener` is passed into `BodyView`, and where
|
||||
`BodyView.makeNSView` calls `skel3d.attach(to:listener:arkitListener:)`.
|
||||
|
||||
- [ ] **Step 2: Own and start the consumer**
|
||||
|
||||
In `AVLiveBodyApp.swift`'s `ContentView`, add a `@StateObject`:
|
||||
|
||||
```swift
|
||||
@StateObject private var usbConsumer = USBSkeletonConsumer()
|
||||
```
|
||||
|
||||
In `.onAppear`, alongside the existing listener `.start()` calls, add
|
||||
`usbConsumer.start()`. If there is an `.onDisappear`, add
|
||||
`usbConsumer.stop()`.
|
||||
|
||||
- [ ] **Step 3: Thread the consumer to the renderer**
|
||||
|
||||
`Skeleton3DRenderer.attach` currently takes
|
||||
`arkitListener: ArkitOSCListener?`. The simplest correct change: give
|
||||
`USBSkeletonConsumer` the same role. Add an overload / extra parameter
|
||||
so `attach` can subscribe to `usbConsumer.$bodies` exactly as it
|
||||
subscribes to `arkitListener.$bodies` (both publish
|
||||
`[Int: ArkitOSCListener.ArkitBodyFrame]`). Concretely, in
|
||||
`Skeleton3DRenderer.attach`, accept `usbConsumer: USBSkeletonConsumer?`
|
||||
and, if non-nil, subscribe its `$bodies` into `lastArkit` with the same
|
||||
sink already used for `arkitListener` (the `arkitSub` Combine
|
||||
subscription). Pass `usbConsumer` from `ContentView` → `BodyView` →
|
||||
`makeNSView` → `skel3d.attach(...)`, mirroring how `arkitListener` is
|
||||
already threaded.
|
||||
|
||||
If `arkitListener` (the OSC one) is now redundant, it may be passed as
|
||||
`nil`; do not delete `ArkitOSCListener` in this plan (other code or
|
||||
Plan 3b cleanup may still reference it).
|
||||
|
||||
- [ ] **Step 4: Verify build**
|
||||
|
||||
Run: `cd launcher/AV-Live-Body && swift build` — Expected: succeeds.
|
||||
Run: `cd launcher/AV-Live-Body && swift test` — Expected: no regression.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add launcher/AV-Live-Body/Sources/AVLiveBody/AVLiveBodyApp.swift launcher/AV-Live-Body/Sources/AVLiveBody/BodyView.swift
|
||||
git commit -m "feat(av-live-body): wire USB consumer to renderer"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Final verification
|
||||
|
||||
- [ ] **Step 1: Clean build + full test suite**
|
||||
|
||||
```bash
|
||||
cd launcher/AV-Live-Body && swift build && swift test
|
||||
```
|
||||
|
||||
Expected: build succeeds; all tests pass (7: prior 6 + Task 1's).
|
||||
|
||||
- [ ] **Step 2: Confirm the integration seam**
|
||||
|
||||
`USBSkeletonConsumer.devicePort` (7000) must equal the iOS app's
|
||||
`USBServer.port`. Verify:
|
||||
|
||||
```bash
|
||||
grep -rn "port.*7000\|devicePort" \
|
||||
launcher/AV-Live-Body/Sources/AVLiveBody/USBSkeletonConsumer.swift \
|
||||
iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/USBServer.swift
|
||||
```
|
||||
|
||||
Expected: both sides use `7000`.
|
||||
|
||||
- [ ] **Step 3: Commit any fix** (only if Step 2 found a mismatch).
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
- **Spec coverage:** This plan implements the spec's `USBClient`
|
||||
consumption inside `AVLiveBody`, the `VideoDecoder` unit, and the
|
||||
skeleton render path. `MultiHMRCoreML`, `BodyFusion`, and dense-mesh
|
||||
rendering are explicitly Plan 3b (gated on a confirmed CoreML
|
||||
Multi-HMR `.mlpackage`).
|
||||
- **Placeholders:** none — new files have complete code; modify tasks
|
||||
cite exact files and the line-138 TODO, and instruct the implementer
|
||||
to read exact context for `AVLiveBodyApp.swift`/`BodyView.swift`
|
||||
(whose current line numbers are not reproduced here).
|
||||
- **Type consistency:** `USBSkeletonConsumer.bodyFrame` returns
|
||||
`ArkitOSCListener.ArkitBodyFrame`; `Skeleton3DRenderer` already
|
||||
stores `lastArkit: [Int: ArkitOSCListener.ArkitBodyFrame]`, so the
|
||||
consumer is type-compatible with the existing `arkitSub` path.
|
||||
`VideoDecoder` consumes `VideoPayload` exactly as Plan 2's
|
||||
`VideoEncoder` produces it (parameter sets prepended, 4-byte
|
||||
big-endian length prefixes).
|
||||
- **Known risks:** (1) `BodyView` owns `Skeleton3DRenderer`, so Task 4
|
||||
threads a new object through `ContentView` → `BodyView` → `attach` —
|
||||
multi-file, follow the existing `arkitListener` threading exactly.
|
||||
(2) `StreamDemuxer.findMagic` copies the whole buffer per `feed()`;
|
||||
for HEVC video this is a perf risk — acceptable for Plan 3a, revisit
|
||||
if frame rate suffers. (3) The HEVC parameter-set split in
|
||||
`VideoDecoder` assumes the iOS encoder's exact prepend layout —
|
||||
this is the Plan 2 ↔ Plan 3a integration seam; validate on real
|
||||
device data.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,443 @@
|
||||
# AVLiveBody Phase 0 — Hands + Face over AVLiveWire (Implementation Plan)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Capture hands (Vision hand pose) and face (Vision face landmarks) on the iPhone from the AR rear-camera frames and stream them to the Mac over two new AVLiveWire frame tags, with the Mac consumer exposing them as published state.
|
||||
|
||||
**Architecture:** Three layers. (1) `shared/AVLiveWire` gains `FrameTag.hands`/`.face` and `HandsPayload`/`FacePayload` codecs (pure Swift, unit-tested). (2) The iPhone `ARBodySession` runs `VNDetectHumanHandPoseRequest` + `VNDetectFaceLandmarksRequest` on `ARFrame.capturedImage` on a background queue (throttled, drop-if-busy) and sends the new frames. (3) The Mac `USBSkeletonConsumer` routes the new tags into `@Published` properties.
|
||||
|
||||
**Tech Stack:** Swift, SwiftPM (AVLiveWire), Xcode projects (iphone-arbody iOS, avlivebody-mac macOS), Apple Vision, ARKit, usbmux/AVLiveWire framing.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- AVLiveWire framing is fixed: 19-byte header (`magic 'AVL1' | tag u8 | pid i16 | timestamp f64 | length u32`), big-endian. Do not change the header.
|
||||
- `FrameTag` already has `skeleton=1, video=2, meta=3` — the NEW tags are `hands = 4` and `face = 5`. (The design doc said 3/4; 3 is taken by `meta`.)
|
||||
- Encode floats as `appendBE(f.bitPattern)` (UInt32 BE); decode as `Float(bitPattern: UInt32(bigEndianBytes: b[o..<o+4]))` — mirror the existing `SkeletonPayload`/`VideoPayload` codecs in `WirePayloads.swift`.
|
||||
- Coordinates on the wire are **normalized image coordinates [0,1]** (convert Vision points before sending).
|
||||
- No emojis. Commit subject ≤ 50 chars, body ≤ 72/line, no AI attribution, no `--no-verify`, no underscore in commit scope (use hyphens).
|
||||
- Vision must run OFF the main thread on the iPhone with drop-if-busy; never block the AR/encode/send path (same discipline as the Multi-HMR off-main fix). `usb.send` is invoked from `@MainActor` elsewhere — hop sends back to `@MainActor`.
|
||||
- Branch: work on `main` (trunk-based, per this repo's workflow).
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `shared/AVLiveWire/Sources/AVLiveWire/FrameHeader.swift` | Modify | add `hands=4`, `face=5` to `FrameTag` |
|
||||
| `shared/AVLiveWire/Sources/AVLiveWire/WirePayloads.swift` | Modify | `HandsPayload`, `FacePayload` codecs |
|
||||
| `shared/AVLiveWire/Tests/AVLiveWireTests/WirePayloadsTests.swift` | Modify | roundtrip tests for both payloads |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/usb/USBSkeletonConsumer.swift` | Modify | route `.hands`/`.face` → `@Published hands/face` |
|
||||
| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift` | Modify | Vision hand+face on AR frames, send tags 4/5 |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: AVLiveWire protocol — tags + HandsPayload + FacePayload
|
||||
|
||||
Pure-Swift protocol layer, fully unit-tested via `swift test`.
|
||||
|
||||
**Files:**
|
||||
- Modify: `shared/AVLiveWire/Sources/AVLiveWire/FrameHeader.swift`
|
||||
- Modify: `shared/AVLiveWire/Sources/AVLiveWire/WirePayloads.swift`
|
||||
- Test: `shared/AVLiveWire/Tests/AVLiveWireTests/WirePayloadsTests.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces:
|
||||
- `FrameTag.hands` (= 4), `FrameTag.face` (= 5)
|
||||
- `HandsPayload` with nested `HandsPayload.Hand { isRight: Bool; points: [SIMD3<Float>] }` (21 points, each `(x, y, confidence)`), `static pointsPerHand = 21`, `encoded() -> Data`, `init?(decoding: Data)`
|
||||
- `FacePayload { confidence: Float; points: [SIMD2<Float>] }`, `encoded() -> Data`, `init?(decoding: Data)`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
Append to `shared/AVLiveWire/Tests/AVLiveWireTests/WirePayloadsTests.swift` (inside the existing `XCTestCase` subclass — match the file's existing class name):
|
||||
|
||||
```swift
|
||||
func testHandsPayloadRoundTrip() {
|
||||
let left = HandsPayload.Hand(
|
||||
isRight: false,
|
||||
points: (0..<21).map { SIMD3(Float($0) * 0.01,
|
||||
Float($0) * 0.02, 0.9) })
|
||||
let right = HandsPayload.Hand(
|
||||
isRight: true,
|
||||
points: (0..<21).map { SIMD3(Float($0) * 0.03,
|
||||
Float($0) * 0.04, 0.5) })
|
||||
let p = HandsPayload(hands: [left, right])
|
||||
let decoded = HandsPayload(decoding: p.encoded())
|
||||
XCTAssertEqual(decoded, p)
|
||||
XCTAssertEqual(decoded?.hands.count, 2)
|
||||
XCTAssertEqual(decoded?.hands[0].isRight, false)
|
||||
XCTAssertEqual(decoded?.hands[1].isRight, true)
|
||||
}
|
||||
|
||||
func testHandsPayloadEmpty() {
|
||||
let p = HandsPayload(hands: [])
|
||||
let decoded = HandsPayload(decoding: p.encoded())
|
||||
XCTAssertEqual(decoded?.hands.count, 0)
|
||||
}
|
||||
|
||||
func testHandsPayloadRejectsTruncated() {
|
||||
let p = HandsPayload(hands: [
|
||||
HandsPayload.Hand(isRight: false,
|
||||
points: Array(repeating: .zero, count: 21))])
|
||||
let bad = p.encoded().dropLast()
|
||||
XCTAssertNil(HandsPayload(decoding: bad))
|
||||
}
|
||||
|
||||
func testFacePayloadRoundTrip() {
|
||||
let pts = (0..<76).map { SIMD2(Float($0) * 0.001,
|
||||
Float($0) * 0.002) }
|
||||
let p = FacePayload(confidence: 0.87, points: pts)
|
||||
let decoded = FacePayload(decoding: p.encoded())
|
||||
XCTAssertEqual(decoded, p)
|
||||
XCTAssertEqual(decoded?.points.count, 76)
|
||||
XCTAssertEqual(decoded?.confidence, 0.87)
|
||||
}
|
||||
|
||||
func testFacePayloadRejectsTruncated() {
|
||||
let p = FacePayload(confidence: 0.5,
|
||||
points: [SIMD2(0.1, 0.2)])
|
||||
XCTAssertNil(FacePayload(decoding: p.encoded().dropLast()))
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live/shared/AVLiveWire" && swift test 2>&1 | tail -20`
|
||||
Expected: FAIL — `cannot find 'HandsPayload' in scope` / `cannot find 'FacePayload' in scope`.
|
||||
|
||||
- [ ] **Step 3: Add the new frame tags**
|
||||
|
||||
In `shared/AVLiveWire/Sources/AVLiveWire/FrameHeader.swift`, extend the enum:
|
||||
|
||||
```swift
|
||||
public enum FrameTag: UInt8 {
|
||||
case skeleton = 1
|
||||
case video = 2
|
||||
case meta = 3
|
||||
case hands = 4
|
||||
case face = 5
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Add the payload codecs**
|
||||
|
||||
Append to `shared/AVLiveWire/Sources/AVLiveWire/WirePayloads.swift`:
|
||||
|
||||
```swift
|
||||
/// Up to N hands; each = chirality + 21 (x, y, confidence) points in
|
||||
/// normalized image coordinates [0,1].
|
||||
public struct HandsPayload: Equatable {
|
||||
public static let pointsPerHand = 21
|
||||
|
||||
public struct Hand: Equatable {
|
||||
public var isRight: Bool
|
||||
public var points: [SIMD3<Float>]
|
||||
public init(isRight: Bool, points: [SIMD3<Float>]) {
|
||||
self.isRight = isRight; self.points = points
|
||||
}
|
||||
}
|
||||
|
||||
public var hands: [Hand]
|
||||
public init(hands: [Hand]) { self.hands = hands }
|
||||
|
||||
public func encoded() -> Data {
|
||||
var d = Data()
|
||||
d.append(UInt8(min(hands.count, 255)))
|
||||
for hand in hands {
|
||||
d.append(hand.isRight ? 1 : 0)
|
||||
for i in 0..<Self.pointsPerHand {
|
||||
let p = i < hand.points.count ? hand.points[i] : .zero
|
||||
d.appendBE(p.x.bitPattern)
|
||||
d.appendBE(p.y.bitPattern)
|
||||
d.appendBE(p.z.bitPattern)
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
public init?(decoding data: Data) {
|
||||
let b = [UInt8](data)
|
||||
guard let n = b.first else { return nil }
|
||||
let per = 1 + Self.pointsPerHand * 12
|
||||
guard b.count == 1 + Int(n) * per else { return nil }
|
||||
var o = 1
|
||||
var result: [Hand] = []
|
||||
for _ in 0..<Int(n) {
|
||||
let isRight = b[o] != 0; o += 1
|
||||
var pts: [SIMD3<Float>] = []
|
||||
for _ in 0..<Self.pointsPerHand {
|
||||
func f() -> Float {
|
||||
let v = Float(bitPattern:
|
||||
UInt32(bigEndianBytes: b[o..<o+4]))
|
||||
o += 4; return v
|
||||
}
|
||||
pts.append(SIMD3(f(), f(), f()))
|
||||
}
|
||||
result.append(Hand(isRight: isRight, points: pts))
|
||||
}
|
||||
hands = result
|
||||
}
|
||||
}
|
||||
|
||||
/// One face: observation confidence + N landmark points in normalized
|
||||
/// image coordinates [0,1] (Vision `allPoints`, ~76).
|
||||
public struct FacePayload: Equatable {
|
||||
public var confidence: Float
|
||||
public var points: [SIMD2<Float>]
|
||||
public init(confidence: Float, points: [SIMD2<Float>]) {
|
||||
self.confidence = confidence; self.points = points
|
||||
}
|
||||
|
||||
public func encoded() -> Data {
|
||||
var d = Data()
|
||||
d.appendBE(confidence.bitPattern)
|
||||
d.appendBE(UInt16(min(points.count, 65535)))
|
||||
for p in points {
|
||||
d.appendBE(p.x.bitPattern); d.appendBE(p.y.bitPattern)
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
public init?(decoding data: Data) {
|
||||
let b = [UInt8](data)
|
||||
guard b.count >= 6 else { return nil }
|
||||
var o = 0
|
||||
func f() -> Float {
|
||||
let v = Float(bitPattern: UInt32(bigEndianBytes: b[o..<o+4]))
|
||||
o += 4; return v
|
||||
}
|
||||
confidence = f()
|
||||
let n = Int(UInt16(bigEndianBytes: b[o..<o+2])); o += 2
|
||||
guard b.count == 6 + n * 8 else { return nil }
|
||||
var pts: [SIMD2<Float>] = []
|
||||
for _ in 0..<n { pts.append(SIMD2(f(), f())) }
|
||||
points = pts
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Run tests to verify they pass**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live/shared/AVLiveWire" && swift test 2>&1 | tail -20`
|
||||
Expected: PASS (all existing tests + the 5 new ones).
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add shared/AVLiveWire/Sources/AVLiveWire/FrameHeader.swift \
|
||||
shared/AVLiveWire/Sources/AVLiveWire/WirePayloads.swift \
|
||||
shared/AVLiveWire/Tests/AVLiveWireTests/WirePayloadsTests.swift
|
||||
git commit -m "feat(avlivewire): hands and face frame payloads"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Mac consumer routes hands/face
|
||||
|
||||
**Files:**
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/usb/USBSkeletonConsumer.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `FrameTag.hands`/`.face`, `HandsPayload`, `FacePayload` (Task 1).
|
||||
- Produces: on `USBSkeletonConsumer`, `@Published var hands: HandsPayload?` and `@Published var face: FacePayload?`, updated on the main queue for each received frame.
|
||||
|
||||
- [ ] **Step 1: Read the current routing**
|
||||
|
||||
Read `avlivebody-mac/Sources/AVLiveBody/usb/USBSkeletonConsumer.swift`, the `route(_:)` method (switch on `frame.header.tag`, currently `.skeleton`/`.video`, plus how `.meta` is handled) and the `@Published` properties near the top (`skeletons`, `connected`).
|
||||
|
||||
- [ ] **Step 2: Add published properties**
|
||||
|
||||
Near the existing `@Published var skeletons` / `@Published var connected`, add:
|
||||
|
||||
```swift
|
||||
/// Latest Vision hand landmarks from the iPhone (tag=4).
|
||||
@Published var hands: HandsPayload?
|
||||
/// Latest Vision face landmarks from the iPhone (tag=5).
|
||||
@Published var face: FacePayload?
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Route the new tags**
|
||||
|
||||
In `route(_:)`, add cases alongside `.skeleton`/`.video`. Keep the `switch` exhaustive over `FrameTag` (it now has `.skeleton/.video/.meta/.hands/.face`); if a `.meta` case or `default` already exists, leave it and insert these before it:
|
||||
|
||||
```swift
|
||||
case .hands:
|
||||
guard let payload = HandsPayload(decoding: frame.payload)
|
||||
else { return }
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.hands = payload
|
||||
}
|
||||
case .face:
|
||||
guard let payload = FacePayload(decoding: frame.payload)
|
||||
else { return }
|
||||
DispatchQueue.main.async { [weak self] in
|
||||
self?.face = payload
|
||||
}
|
||||
```
|
||||
|
||||
If the existing switch has no `.meta` case and no `default`, adding the enum cases in Task 1 will have made it non-exhaustive — add a `case .meta: break` too so it compiles.
|
||||
|
||||
- [ ] **Step 4: Build to verify it compiles**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/avbody-dd" && \
|
||||
xcodebuild -project AVLiveBody.xcodeproj -scheme AVLiveBody -configuration Debug \
|
||||
-derivedDataPath "$DD" build 2>&1 | grep -E "error:|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
Expected: `** BUILD SUCCEEDED **`, no `error:` lines. (The exhaustive-switch requirement is compiler-enforced — a missing case fails the build.)
|
||||
|
||||
Note: decode correctness is already unit-tested in Task 1 (payload roundtrips). Live frame flow is verified on-device in Task 3 / the user's run.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/usb/USBSkeletonConsumer.swift
|
||||
git commit -m "feat(avlivebody): consume hands and face frames"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: iPhone captures hands + face via Vision and sends them
|
||||
|
||||
**Files:**
|
||||
- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `FrameTag.hands`/`.face`, `HandsPayload`, `FacePayload` (Task 1); the existing `usb.send(tag:pid:timestamp:payload:)`.
|
||||
- Produces: per qualifying AR frame, at most one `.hands` and one `.face` AVLiveWire frame sent over USB.
|
||||
|
||||
- [ ] **Step 1: Add Vision import and worker state**
|
||||
|
||||
At the top of `ARBodySession.swift`, add `import Vision`. Add stored properties to `ARBodySession` (near `lastFrameTime`):
|
||||
|
||||
```swift
|
||||
private let visionQueue = DispatchQueue(
|
||||
label: "cc.saillant.arbody.vision", qos: .userInitiated)
|
||||
private var visionBusy = false
|
||||
private var lastVisionTime: TimeInterval = 0
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the Vision extraction + send helper**
|
||||
|
||||
Add this method to `ARBodySession` (it runs OFF the main actor; mark `nonisolated`). It converts Vision points to normalized image coordinates and sends the two frames. `usb.send` is hopped back to the main actor:
|
||||
|
||||
```swift
|
||||
nonisolated func runVision(on pixelBuffer: CVPixelBuffer,
|
||||
timestamp t: TimeInterval) {
|
||||
let handReq = VNDetectHumanHandPoseRequest()
|
||||
handReq.maximumHandCount = 2
|
||||
let faceReq = VNDetectFaceLandmarksRequest()
|
||||
let handler = VNImageRequestHandler(
|
||||
cvPixelBuffer: pixelBuffer, orientation: .right, options: [:])
|
||||
try? handler.perform([handReq, faceReq])
|
||||
|
||||
// ---- hands ----
|
||||
// Fixed 21-joint order matching MediaPipe-style indexing.
|
||||
let order: [VNHumanHandPoseObservation.JointName] = [
|
||||
.wrist,
|
||||
.thumbCMC, .thumbMP, .thumbIP, .thumbTip,
|
||||
.indexMCP, .indexPIP, .indexDIP, .indexTip,
|
||||
.middleMCP, .middlePIP, .middleDIP, .middleTip,
|
||||
.ringMCP, .ringPIP, .ringDIP, .ringTip,
|
||||
.littleMCP, .littlePIP, .littleDIP, .littleTip,
|
||||
]
|
||||
var handsOut: [HandsPayload.Hand] = []
|
||||
for obs in (handReq.results ?? []).prefix(2) {
|
||||
guard let pts = try? obs.recognizedPoints(.all) else { continue }
|
||||
let p = order.map { name -> SIMD3<Float> in
|
||||
if let rp = pts[name] {
|
||||
// Vision: normalized, bottom-left origin -> flip y.
|
||||
return SIMD3(Float(rp.location.x),
|
||||
Float(1.0 - rp.location.y),
|
||||
Float(rp.confidence))
|
||||
}
|
||||
return .zero
|
||||
}
|
||||
handsOut.append(HandsPayload.Hand(
|
||||
isRight: obs.chirality == .right, points: p))
|
||||
}
|
||||
|
||||
// ---- face (most prominent) ----
|
||||
var facePayload: FacePayload?
|
||||
if let face = (faceReq.results ?? []).first,
|
||||
let all = face.landmarks?.allPoints {
|
||||
let bb = face.boundingBox
|
||||
let pts = all.normalizedPoints.map { np -> SIMD2<Float> in
|
||||
// landmark points are relative to the face bbox.
|
||||
let x = bb.origin.x + Double(np.x) * bb.size.width
|
||||
let y = bb.origin.y + Double(np.y) * bb.size.height
|
||||
return SIMD2(Float(x), Float(1.0 - y))
|
||||
}
|
||||
facePayload = FacePayload(
|
||||
confidence: face.confidence, points: pts)
|
||||
}
|
||||
|
||||
let handsData = HandsPayload(hands: handsOut).encoded()
|
||||
let faceData = facePayload?.encoded()
|
||||
Task { @MainActor in
|
||||
guard self.usbState == .connected else { return }
|
||||
self.usb.send(tag: .hands, pid: -1,
|
||||
timestamp: t, payload: handsData)
|
||||
if let faceData {
|
||||
self.usb.send(tag: .face, pid: -1,
|
||||
timestamp: t, payload: faceData)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Call it from the frame handler (throttled, drop-if-busy)**
|
||||
|
||||
In `session(_:didUpdate:)`, inside the existing `Task { @MainActor in ... }` block, after the video-encode section and before/after `publishUSB`, add a throttled off-main Vision dispatch (≈15 fps):
|
||||
|
||||
```swift
|
||||
// Vision hands+face at ~15 fps, off-main, drop if busy.
|
||||
if t - self.lastVisionTime >= 1.0 / 15.0, !self.visionBusy {
|
||||
self.lastVisionTime = t
|
||||
self.visionBusy = true
|
||||
let buf = img // CVPixelBuffer captured for the worker
|
||||
self.visionQueue.async {
|
||||
self.runVision(on: buf, timestamp: t)
|
||||
Task { @MainActor in self.visionBusy = false }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
(`img` is the `frame.capturedImage` already bound earlier in the block.)
|
||||
|
||||
- [ ] **Step 4: Build the iPhone app to verify it compiles**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/iphone-arbody" && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/arbody-dd" && \
|
||||
xcodebuild -project ARBodyTracker.xcodeproj -scheme ARBodyTracker -configuration Debug \
|
||||
-destination 'generic/platform=iOS' -derivedDataPath "$DD" -allowProvisioningUpdates build 2>&1 \
|
||||
| grep -E "error:|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
Expected: `** BUILD SUCCEEDED **`, no `error:` lines.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift
|
||||
git commit -m "feat(ios): vision hands and face over usb"
|
||||
```
|
||||
|
||||
- [ ] **Step 6: On-device smoke (manual, user)**
|
||||
|
||||
Deploy to the iPhone (`xcodebuild` build → `xcrun devicectl device install/launch`, device `00008140-000845660163001C`). On the Mac, run a small AVLiveWire listener (or AVLiveBody with a log) and confirm `tag=4`/`tag=5` frames arrive when a hand / face is in the iPhone camera. Tune the `orientation` (.right vs .up/.left) if hand/face coordinates look rotated. This step is verified by the user on hardware.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review notes
|
||||
|
||||
- **Spec coverage:** §4.2 protocol (tags + payloads) → Task 1; §4.3 consumer routing → Task 2; §4.1 iPhone Vision capture → Task 3. Testing §8 Phase 0 (roundtrip unit tests, consumer build, iPhone build, on-device) → Tasks 1/2/3.
|
||||
- **Correction vs spec:** spec said tags 3/4; `meta=3` pre-exists, so tags are **4 (hands) / 5 (face)** — recorded in Global Constraints.
|
||||
- **Type consistency:** `HandsPayload.Hand{isRight,points}`, `FacePayload{confidence,points}` used identically in Tasks 1→2→3. `usb.send(tag:pid:timestamp:payload:)` matches the existing call site.
|
||||
- **Out of scope (later phases):** consuming hands/face for shader uniforms + 3D hand/face skeleton (Phase 2); multi-person association; depth (2D only).
|
||||
@@ -0,0 +1,472 @@
|
||||
# AVLiveBody Phase 1 — Settings panel + lights + mesh material (Implementation Plan)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Give `avlivebody-mac` an interactive control surface: a `RenderSettings` state object, a SwiftUI settings panel, 3 directional lights (the scene is currently unlit/black), live mesh material (metallic/roughness), camera FOV, and show/hide toggles for skeleton/mesh/video.
|
||||
|
||||
**Architecture:** Port the archived app's `RenderSettings` + `SettingsPanel` pattern into B, adapted to B's RealityKit `SceneController`. `RenderSettings` (ObservableObject) is the single UI-state source; `ContentView` overlays the panel in a `ZStack` and pushes changes to `SceneController` via `.onReceive`/`.onChange`; `SceneController` applies them to its entities (lights, mesh material, visibility, camera FOV).
|
||||
|
||||
**Tech Stack:** Swift, SwiftUI, RealityKit, Xcode project (avlivebody-mac, macOS 15).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Base app `avlivebody-mac/` is an xcodegen Xcode project; build/test with `xcodebuild` (scheme `AVLiveBody`, configuration Debug). It is unsigned (`CODE_SIGNING_ALLOWED=NO`).
|
||||
- `SceneController`, `MeshEntity`, `SkeletonEntity`, `VideoQuad` are all `@MainActor`. UI runs on the main actor.
|
||||
- ARKit→RealityKit coordinate mapping is the existing `arkitToRealityKit` helper — do not change it.
|
||||
- No emojis. Commit subject ≤ 50 chars, body ≤ 72/line, no AI attribution, no `--no-verify`, no underscore in commit scope (use hyphens).
|
||||
- French labels in user-facing panel text (matches the archive); code/comments in English.
|
||||
- Branch: `main` (trunk-based).
|
||||
- SourceKit single-file diagnostics are noise; the whole-module `xcodebuild` result is the source of truth.
|
||||
|
||||
### Scope note (vs spec §5)
|
||||
**Wireframe is deferred to Phase 2.** RealityKit `MeshDescriptor` has no reliable line primitive; a correct wireframe needs a custom mesh/material technique best built alongside the Phase 2 Metal work. Phase 1 ships metallic/roughness + visibility + lights + FOV (all certain). `showWireframe` is NOT added to `RenderSettings` here.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift` | Create | `@Published` UI state |
|
||||
| `avlivebody-mac/Tests/AVLiveBodyTests/RenderSettingsTests.swift` | Create | default-value test |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SceneController.swift` | Modify | 3 lights + apply methods (visibility/material/FOV/light) |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/MeshEntity.swift` | Modify | mutable material + `setMaterial`/`setVisible` |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SkeletonEntity.swift` | Modify | `setVisible` |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/VideoQuad.swift` | Modify | `setVisible`/`setOpacity` |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift` | Create | SwiftUI control panel |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift` | Modify | own RenderSettings; ZStack panel; push to controller |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: RenderSettings state object
|
||||
|
||||
**Files:**
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift`
|
||||
- Test: `avlivebody-mac/Tests/AVLiveBodyTests/RenderSettingsTests.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `@MainActor final class RenderSettings: ObservableObject` with `@Published`:
|
||||
`showSkeleton: Bool`, `showMesh: Bool`, `showVideo: Bool`, `meshMetallic: Bool`,
|
||||
`meshRoughness: Double`, `videoOpacity: Double`, `keyIntensity: Double`,
|
||||
`fillIntensity: Double`, `rimIntensity: Double`, `fieldOfView: Double`,
|
||||
`showPanel: Bool`.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```swift
|
||||
// avlivebody-mac/Tests/AVLiveBodyTests/RenderSettingsTests.swift
|
||||
import XCTest
|
||||
@testable import AVLiveBody
|
||||
|
||||
@MainActor
|
||||
final class RenderSettingsTests: XCTestCase {
|
||||
func testDefaults() {
|
||||
let s = RenderSettings()
|
||||
XCTAssertTrue(s.showSkeleton)
|
||||
XCTAssertTrue(s.showMesh)
|
||||
XCTAssertTrue(s.showVideo)
|
||||
XCTAssertFalse(s.meshMetallic)
|
||||
XCTAssertEqual(s.meshRoughness, 0.6, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.videoOpacity, 1.0, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.keyIntensity, 4000, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.fillIntensity, 1500, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.rimIntensity, 2000, accuracy: 1e-9)
|
||||
XCTAssertEqual(s.fieldOfView, 55, accuracy: 1e-9)
|
||||
XCTAssertFalse(s.showPanel)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && \
|
||||
xcodebuild test -project AVLiveBody.xcodeproj -scheme AVLiveBody \
|
||||
-destination 'platform=macOS' 2>&1 | grep -E "error:|Cannot find 'RenderSettings'|Test Suite.*(passed|failed)|** TEST"
|
||||
```
|
||||
Expected: FAIL — `Cannot find 'RenderSettings' in scope`.
|
||||
|
||||
- [ ] **Step 3: Create RenderSettings**
|
||||
|
||||
```swift
|
||||
// avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift
|
||||
import SwiftUI
|
||||
|
||||
/// Live visual settings for AVLiveBody. ContentView observes this and
|
||||
/// pushes changes to SceneController. Defaults chosen so the scene is
|
||||
/// lit and everything visible out of the box.
|
||||
@MainActor
|
||||
final class RenderSettings: ObservableObject {
|
||||
// Layer visibility
|
||||
@Published var showSkeleton: Bool = true
|
||||
@Published var showMesh: Bool = true
|
||||
@Published var showVideo: Bool = true
|
||||
|
||||
// Mesh material
|
||||
@Published var meshMetallic: Bool = false
|
||||
@Published var meshRoughness: Double = 0.6
|
||||
|
||||
// Video quad
|
||||
@Published var videoOpacity: Double = 1.0
|
||||
|
||||
// Lights (RealityKit DirectionalLight intensities)
|
||||
@Published var keyIntensity: Double = 4000
|
||||
@Published var fillIntensity: Double = 1500
|
||||
@Published var rimIntensity: Double = 2000
|
||||
|
||||
// Camera
|
||||
@Published var fieldOfView: Double = 55
|
||||
|
||||
// Panel visibility
|
||||
@Published var showPanel: Bool = false
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && \
|
||||
xcodebuild test -project AVLiveBody.xcodeproj -scheme AVLiveBody \
|
||||
-destination 'platform=macOS' 2>&1 | grep -E "error:|Test Suite 'RenderSettingsTests'|** TEST"
|
||||
```
|
||||
Expected: `Test Suite 'RenderSettingsTests' ... passed`, `** TEST SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift \
|
||||
avlivebody-mac/Tests/AVLiveBodyTests/RenderSettingsTests.swift
|
||||
git commit -m "feat(avlivebody): render settings state object"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: SceneController apply methods (lights, material, visibility, FOV)
|
||||
|
||||
**Files:**
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/SceneController.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/MeshEntity.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/SkeletonEntity.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/VideoQuad.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `RenderSettings` values (Task 1).
|
||||
- Produces on `SceneController`:
|
||||
`setSkeletonVisible(_: Bool)`, `setMeshVisible(_: Bool)`, `setVideoVisible(_: Bool)`,
|
||||
`setVideoOpacity(_: Double)`, `updateMeshMaterial(metallic: Bool, roughness: Double)`,
|
||||
`setLightIntensities(key: Double, fill: Double, rim: Double)`, `setFieldOfView(_: Double)`.
|
||||
On `MeshEntity`: `setVisible(_: Bool)`, `setMaterial(metallic: Bool, roughness: Double)`.
|
||||
On `SkeletonEntity`: `setVisible(_: Bool)`. On `VideoQuad`: `setVisible(_: Bool)`, `setOpacity(_: Double)`.
|
||||
|
||||
- [ ] **Step 1: Add visibility/opacity to the entities**
|
||||
|
||||
In `MeshEntity.swift`, change the material from `let` to `var` and add methods (place after `update(_:)`):
|
||||
|
||||
```swift
|
||||
func setVisible(_ visible: Bool) { root.isEnabled = visible }
|
||||
|
||||
func setMaterial(metallic: Bool, roughness: Double) {
|
||||
material = SimpleMaterial(
|
||||
color: NSColor(white: 0.8, alpha: 1.0),
|
||||
roughness: .float(Float(roughness)), isMetallic: metallic)
|
||||
for entity in pools.values {
|
||||
entity.model?.materials = [material]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Change the property declaration from:
|
||||
```swift
|
||||
private let material = SimpleMaterial(
|
||||
color: NSColor(white: 0.8, alpha: 1.0),
|
||||
roughness: 0.5, isMetallic: false)
|
||||
```
|
||||
to:
|
||||
```swift
|
||||
private var material = SimpleMaterial(
|
||||
color: NSColor(white: 0.8, alpha: 1.0),
|
||||
roughness: 0.5, isMetallic: false)
|
||||
```
|
||||
|
||||
In `SkeletonEntity.swift`, add:
|
||||
```swift
|
||||
func setVisible(_ visible: Bool) { root.isEnabled = visible }
|
||||
```
|
||||
|
||||
In `VideoQuad.swift`, add (the quad uses `UnlitMaterial`; opacity via tint alpha — note the texture is re-applied every frame in `update`, so store the opacity and apply it there too):
|
||||
|
||||
```swift
|
||||
private var opacity: Double = 1.0
|
||||
|
||||
func setVisible(_ visible: Bool) { entity.isEnabled = visible }
|
||||
|
||||
func setOpacity(_ value: Double) {
|
||||
opacity = value
|
||||
// Re-tint current material; `update(_:)` will also use `opacity`.
|
||||
if var mat = entity.model?.materials.first as? UnlitMaterial {
|
||||
mat.color = .init(tint: NSColor(white: 1, alpha: CGFloat(value)))
|
||||
entity.model?.materials = [mat]
|
||||
}
|
||||
}
|
||||
```
|
||||
And in `VideoQuad.update(_:)`, change the tint to carry `opacity`:
|
||||
```swift
|
||||
material.color = .init(
|
||||
tint: NSColor(white: 1, alpha: CGFloat(opacity)),
|
||||
texture: .init(texture))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add the 3 lights in SceneController.setUp()**
|
||||
|
||||
Add stored properties to `SceneController` (near the other `private let` anchors):
|
||||
```swift
|
||||
private let keyLight = DirectionalLight()
|
||||
private let fillLight = DirectionalLight()
|
||||
private let rimLight = DirectionalLight()
|
||||
```
|
||||
|
||||
In `setUp()`, after `arView.scene.addAnchor(worldAnchor)`, add (ported from the archive's 3-light rig):
|
||||
```swift
|
||||
keyLight.light.intensity = 4000
|
||||
keyLight.orientation = simd_quatf(angle: .pi / 6,
|
||||
axis: SIMD3(1, 0, 0))
|
||||
let keyA = AnchorEntity(world: SIMD3<Float>(1, 2, -1))
|
||||
keyA.addChild(keyLight)
|
||||
arView.scene.addAnchor(keyA)
|
||||
|
||||
fillLight.light.intensity = 1500
|
||||
fillLight.light.color = NSColor(red: 0.7, green: 0.8,
|
||||
blue: 1.0, alpha: 1.0)
|
||||
let fillA = AnchorEntity(world: SIMD3<Float>(-2, 1, -2))
|
||||
fillA.addChild(fillLight)
|
||||
arView.scene.addAnchor(fillA)
|
||||
|
||||
rimLight.light.intensity = 2000
|
||||
let rimA = AnchorEntity(world: SIMD3<Float>(0, 1, -5))
|
||||
rimA.addChild(rimLight)
|
||||
arView.scene.addAnchor(rimA)
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Add the apply methods to SceneController**
|
||||
|
||||
Add (after `updateMesh(_:)`):
|
||||
```swift
|
||||
func setSkeletonVisible(_ v: Bool) { skeleton?.setVisible(v) }
|
||||
func setMeshVisible(_ v: Bool) { mesh?.setVisible(v) }
|
||||
func setVideoVisible(_ v: Bool) { videoQuad?.setVisible(v) }
|
||||
func setVideoOpacity(_ v: Double) { videoQuad?.setOpacity(v) }
|
||||
|
||||
func updateMeshMaterial(metallic: Bool, roughness: Double) {
|
||||
mesh?.setMaterial(metallic: metallic, roughness: roughness)
|
||||
}
|
||||
|
||||
func setLightIntensities(key: Double, fill: Double, rim: Double) {
|
||||
keyLight.light.intensity = Float(key)
|
||||
fillLight.light.intensity = Float(fill)
|
||||
rimLight.light.intensity = Float(rim)
|
||||
}
|
||||
|
||||
func setFieldOfView(_ deg: Double) {
|
||||
camera.camera.fieldOfViewInDegrees = Float(deg)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Build to verify it compiles**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/avbody-dd" && \
|
||||
xcodebuild -project AVLiveBody.xcodeproj -scheme AVLiveBody -configuration Debug \
|
||||
-derivedDataPath "$DD" build 2>&1 | grep -E "error:|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
Expected: `** BUILD SUCCEEDED **`, no `error:` lines. (If `SimpleMaterial` roughness init rejects `.float(...)`, use the scalar form `roughness: MaterialScalarParameter(floatLiteral: Float(roughness))` — but `.float(_)` is the standard `MaterialScalarParameter` case.)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/SceneController.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/MeshEntity.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/SkeletonEntity.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/VideoQuad.swift
|
||||
git commit -m "feat(avlivebody): lights material and visibility apply"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Settings panel + ContentView wiring
|
||||
|
||||
**Files:**
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `RenderSettings` (Task 1), `SceneController` apply methods (Task 2).
|
||||
- Produces: a `SettingsPanel` SwiftUI view bound to `RenderSettings`; `ContentView` owns a `@StateObject RenderSettings`, overlays the panel, and pushes every change to the controller.
|
||||
|
||||
- [ ] **Step 1: Create the SettingsPanel view**
|
||||
|
||||
```swift
|
||||
// avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift
|
||||
import SwiftUI
|
||||
|
||||
/// Collapsible right-side control panel bound to RenderSettings.
|
||||
struct SettingsPanel: View {
|
||||
@ObservedObject var settings: RenderSettings
|
||||
|
||||
var body: some View {
|
||||
ScrollView {
|
||||
VStack(alignment: .leading, spacing: 14) {
|
||||
Text("Reglages").font(.headline)
|
||||
|
||||
group("Couches") {
|
||||
Toggle("Squelette", isOn: $settings.showSkeleton)
|
||||
Toggle("Mesh", isOn: $settings.showMesh)
|
||||
Toggle("Video", isOn: $settings.showVideo)
|
||||
}
|
||||
|
||||
group("Mesh") {
|
||||
Toggle("Metallique", isOn: $settings.meshMetallic)
|
||||
slider("Rugosite", $settings.meshRoughness,
|
||||
0...1)
|
||||
}
|
||||
|
||||
group("Lumieres") {
|
||||
slider("Principale", $settings.keyIntensity,
|
||||
0...10000)
|
||||
slider("Remplissage", $settings.fillIntensity,
|
||||
0...10000)
|
||||
slider("Contre-jour", $settings.rimIntensity,
|
||||
0...10000)
|
||||
}
|
||||
|
||||
group("Vue") {
|
||||
slider("Champ de vision", $settings.fieldOfView,
|
||||
20...120)
|
||||
slider("Opacite video", $settings.videoOpacity,
|
||||
0...1)
|
||||
}
|
||||
}
|
||||
.padding(16)
|
||||
}
|
||||
.frame(width: 280)
|
||||
.background(.ultraThinMaterial)
|
||||
}
|
||||
|
||||
@ViewBuilder
|
||||
private func group(_ title: String,
|
||||
@ViewBuilder _ content: () -> some View)
|
||||
-> some View {
|
||||
VStack(alignment: .leading, spacing: 6) {
|
||||
Text(title).font(.subheadline).foregroundStyle(.secondary)
|
||||
content()
|
||||
}
|
||||
}
|
||||
|
||||
private func slider(_ label: String,
|
||||
_ value: Binding<Double>,
|
||||
_ range: ClosedRange<Double>) -> some View {
|
||||
VStack(alignment: .leading, spacing: 2) {
|
||||
HStack {
|
||||
Text(label).font(.caption)
|
||||
Spacer()
|
||||
Text(String(format: "%.2f", value.wrappedValue))
|
||||
.font(.caption).foregroundStyle(.secondary)
|
||||
}
|
||||
Slider(value: value, in: range)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Wire RenderSettings + panel into ContentView**
|
||||
|
||||
In `AVLiveBodyApp.swift` `ContentView`, add the settings object and overlay. Add the property:
|
||||
```swift
|
||||
@StateObject private var settings = RenderSettings()
|
||||
```
|
||||
Replace the `body` with (keeps existing SceneView + StatusBar; adds a panel toggle button and the panel; pushes settings to the controller via `applyAll` on appear/change):
|
||||
```swift
|
||||
var body: some View {
|
||||
ZStack(alignment: .topTrailing) {
|
||||
SceneView(controller: controller)
|
||||
StatusBar(consumer: consumer)
|
||||
.frame(maxWidth: .infinity, alignment: .top)
|
||||
HStack(alignment: .top) {
|
||||
Spacer()
|
||||
if settings.showPanel {
|
||||
SettingsPanel(settings: settings)
|
||||
.transition(.move(edge: .trailing))
|
||||
}
|
||||
Button {
|
||||
withAnimation { settings.showPanel.toggle() }
|
||||
} label: {
|
||||
Image(systemName: "slider.horizontal.3")
|
||||
.padding(8)
|
||||
}
|
||||
.buttonStyle(.borderedProminent)
|
||||
.padding(8)
|
||||
}
|
||||
}
|
||||
.onAppear { wire(); applyAll() }
|
||||
.onDisappear { consumer.stop() }
|
||||
.onReceive(consumer.$skeletons) { skeletons in
|
||||
controller.updateSkeleton(skeletons)
|
||||
}
|
||||
.onReceive(settings.objectWillChange) { _ in
|
||||
// objectWillChange fires before the value updates; apply on
|
||||
// the next runloop tick so we read the new values.
|
||||
DispatchQueue.main.async { applyAll() }
|
||||
}
|
||||
}
|
||||
|
||||
private func applyAll() {
|
||||
controller.setSkeletonVisible(settings.showSkeleton)
|
||||
controller.setMeshVisible(settings.showMesh)
|
||||
controller.setVideoVisible(settings.showVideo)
|
||||
controller.setVideoOpacity(settings.videoOpacity)
|
||||
controller.updateMeshMaterial(metallic: settings.meshMetallic,
|
||||
roughness: settings.meshRoughness)
|
||||
controller.setLightIntensities(key: settings.keyIntensity,
|
||||
fill: settings.fillIntensity,
|
||||
rim: settings.rimIntensity)
|
||||
controller.setFieldOfView(settings.fieldOfView)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Build to verify it compiles**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/avbody-dd" && \
|
||||
xcodebuild -project AVLiveBody.xcodeproj -scheme AVLiveBody -configuration Debug \
|
||||
-derivedDataPath "$DD" build 2>&1 | grep -E "error:|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
Expected: `** BUILD SUCCEEDED **`, no `error:` lines.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift
|
||||
git commit -m "feat(avlivebody): settings panel and wiring"
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Manual visual check (user)**
|
||||
|
||||
Launch the built `.app`. Click the slider button → panel slides in. Toggle Squelette/Mesh/Video → entities show/hide. Move the light sliders → the scene lighting changes (it was black/unlit before). Change Rugosite/Metallique → the mesh material updates (when a mesh is present, i.e. `AVBODY_MULTIHMR=1`). Change Champ de vision → camera zoom changes. This visual confirmation is the user's.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review notes
|
||||
|
||||
- **Spec coverage (§5):** RenderSettings → Task 1; SettingsPanel → Task 3; 3 lights + intensity → Task 2; mesh material → Task 2; SceneController apply methods → Task 2; ContentView ZStack + panel → Task 3. **Wireframe deferred to Phase 2** (RealityKit line-primitive limitation) — documented in the scope note.
|
||||
- **Type consistency:** apply methods named identically in Task 2 (definitions) and Task 3 (calls): `setSkeletonVisible/setMeshVisible/setVideoVisible/setVideoOpacity/updateMeshMaterial/setLightIntensities/setFieldOfView`. `RenderSettings` field names identical across tasks.
|
||||
- **Placeholder scan:** none; the `.float(...)` fallback note in Task 2 Step 4 is a concrete alternative, not a placeholder.
|
||||
- **Out of scope (Phase 2):** wireframe, Metal shader modes + viz picker, 3D hand/face skeleton, `showScene`/`vizMode` settings.
|
||||
@@ -0,0 +1,717 @@
|
||||
# AVLiveBody Phase 2 — Metal shader backgrounds + 3D hand/face skeleton (Implementation Plan)
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Bring the archived app's 10 Metal shader background modes and 3D hand/face skeleton into `avlivebody-mac`, composited behind the RealityKit scene, with the shader uniforms fed from the iPhone stream (body + Phase-0 hands/face).
|
||||
|
||||
**Architecture:** A layered `NSView` container hosts an `MTKView` (running `scene.metal` via a ported `SceneRenderer`) behind a transparent `ARView` (the existing `SceneController` content). `RenderSettings` gains `showScene`/`vizMode`; the panel gets a mode picker. A per-frame uniform feed derives the `SceneUniforms` from `USBSkeletonConsumer`'s skeleton + hands + face. The 3D hand/face skeleton renders the Phase-0 Vision landmarks in the ARView at the body's depth.
|
||||
|
||||
**Tech Stack:** Swift, SwiftUI, AppKit (`NSViewRepresentable`), RealityKit, Metal/MetalKit, the bundled `scene.metal`.
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Base app `avlivebody-mac/` (xcodegen Xcode project, macOS 15, unsigned). Build with `xcodebuild`. New source/resource files need `xcodegen generate` before build (project.yml is directory-glob for sources; `scene.metal` must be declared as a resource).
|
||||
- All UI/scene types are `@MainActor`.
|
||||
- `SceneUniforms` Swift struct field order MUST exactly match `scene.metal`'s struct (36 floats, stride 144). The shader is ported verbatim from `launcher/_archive-AV-Live-Body/Sources/AVLiveBody/Resources/scene.metal`.
|
||||
- Shader must compile offline: `xcrun -sdk macosx metal -c scene.metal -o /tmp/x.air` (exit 0).
|
||||
- Vision coords from Phase 0 are normalized image coords [0,1], y already flipped to top-left on the iPhone (`HandsPayload`/`FacePayload`).
|
||||
- No emojis. Commit subject ≤ 50 chars, no AI attribution, no `--no-verify`, no underscore in commit scope.
|
||||
- Branch: `main` (trunk-based). SourceKit single-file diagnostics are noise; `xcodebuild` is the source of truth.
|
||||
|
||||
### Interaction with the "max compute on iPhone" track
|
||||
Task 3 (uniform feed) derives the shader scalars **on the Mac** from the raw
|
||||
iPhone landmarks. This derivation is deliberately isolated in ONE function
|
||||
(`SceneUniformBuilder`) so option A (derive on the iPhone, send a compact
|
||||
derivatives frame) can later replace its body with a passthrough without
|
||||
touching the renderer or compositing. Do not scatter derivation logic.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `avlivebody-mac/Sources/AVLiveBody/Resources/scene.metal` | Create (copy A) | shader (10 modes + bg_vertex/bg_fragment) |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SceneRenderer.swift` | Create (port A) | MTKView delegate + SceneUniforms + pipeline |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift` | Create | NSViewRepresentable: MTKView back + ARView front |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SceneUniformBuilder.swift` | Create | derive SceneUniforms from skeleton+hands+face |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/HandFaceSkeleton.swift` | Create | 3D hand/face landmark entities |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift` | Modify | add `showScene`, `vizMode` |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift` | Modify | Scene toggle + viz-mode picker |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/SceneController.swift` | Modify | expose hands/face update + provide ARView for layering |
|
||||
| `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift` | Modify | use LayeredSceneView; route hands/face + uniforms |
|
||||
| `avlivebody-mac/project.yml` | Modify | bundle `scene.metal`; ensure MetalKit |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Port scene.metal + SceneRenderer
|
||||
|
||||
**Files:**
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/Resources/scene.metal`
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/SceneRenderer.swift`
|
||||
- Modify: `avlivebody-mac/project.yml`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces: `final class SceneRenderer: NSObject, MTKViewDelegate` with `static func make() -> SceneRenderer?`, a public `var uniforms: SceneRenderer.SceneUniforms`, and a nested `struct SceneUniforms` (36 floats, exact `scene.metal` order).
|
||||
|
||||
- [ ] **Step 1: Copy the shader**
|
||||
|
||||
```bash
|
||||
cp "/Users/electron/Documents/Projets/AV-Live/launcher/_archive-AV-Live-Body/Sources/AVLiveBody/Resources/scene.metal" \
|
||||
"/Users/electron/Documents/Projets/AV-Live/avlivebody-mac/Sources/AVLiveBody/Resources/scene.metal"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify the shader compiles offline**
|
||||
|
||||
Run: `xcrun -sdk macosx metal -c "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac/Sources/AVLiveBody/Resources/scene.metal" -o /tmp/scene.air`
|
||||
Expected: exit 0 (warnings OK, no errors).
|
||||
|
||||
- [ ] **Step 3: Declare scene.metal as a bundled resource in project.yml**
|
||||
|
||||
In `avlivebody-mac/project.yml`, under the `AVLiveBody` target, add a `resources`/sources entry so `scene.metal` is bundled (it must be copied as a resource, not compiled into the default Metal library, because `SceneRenderer` compiles it at runtime from source). Add to the target:
|
||||
```yaml
|
||||
sources:
|
||||
- path: Sources/AVLiveBody
|
||||
excludes:
|
||||
- Info.plist
|
||||
- path: Sources/AVLiveBody/Resources/scene.metal
|
||||
buildPhase: resources
|
||||
```
|
||||
(If `Sources/AVLiveBody` already globs the file as a Metal compile input, also add `excludes: [Resources/scene.metal]` to the first entry so it is ONLY a resource, never compiled into `default.metallib`.)
|
||||
|
||||
- [ ] **Step 4: Create SceneRenderer (port from the archive, Bundle.main)**
|
||||
|
||||
Create `SceneRenderer.swift` as a verbatim port of
|
||||
`launcher/_archive-AV-Live-Body/Sources/AVLiveBody/SceneRenderer.swift` with ONE
|
||||
change: the resource lookup uses `Bundle.main` (the Xcode app bundle), not
|
||||
`Bundle.module` (SwiftPM). The full file:
|
||||
|
||||
```swift
|
||||
import Foundation
|
||||
import Metal
|
||||
import MetalKit
|
||||
|
||||
/// Metal renderer for the 10 background viz modes (storm, tunnel,
|
||||
/// plasma, kaleido, voronoi, metaballs, starfield, bars, hands3d,
|
||||
/// openpos). Compiles the bundled scene.metal at runtime; sits as the
|
||||
/// backing layer under the transparent ARView.
|
||||
final class SceneRenderer: NSObject, MTKViewDelegate {
|
||||
/// Mirror of scene.metal SceneUniforms (36 floats, 144 B). Field
|
||||
/// order MUST match the shader exactly.
|
||||
struct SceneUniforms {
|
||||
var time: Float = 0
|
||||
var rms: Float = 0
|
||||
var kp_norm: Float = 0
|
||||
var netz_dev: Float = 0
|
||||
var lightning_flash: Float = 0
|
||||
var flare: Float = 0
|
||||
var wind_norm: Float = 0
|
||||
var bz_norm: Float = 0
|
||||
var social_rate: Float = 0
|
||||
var pose_alive: Float = 0
|
||||
var pose_count: Float = 0
|
||||
var width: Float = 1280
|
||||
var height: Float = 720
|
||||
var viz_mode: Float = 0
|
||||
var hand_l_x: Float = 0
|
||||
var hand_l_y: Float = 0
|
||||
var hand_r_x: Float = 0
|
||||
var hand_r_y: Float = 0
|
||||
var mouth_open: Float = 0
|
||||
var eye_open_l: Float = 0
|
||||
var eye_open_r: Float = 0
|
||||
var head_tilt: Float = 0
|
||||
var head_yaw: Float = 0
|
||||
var finger_pinch_l: Float = 0
|
||||
var finger_pinch_r: Float = 0
|
||||
var body_x: Float = 0
|
||||
var body_y: Float = 0
|
||||
var body_z: Float = 0
|
||||
var body_height: Float = 0
|
||||
var arm_spread: Float = 0
|
||||
var pose_velocity: Float = 0
|
||||
var _pad0: Float = 0
|
||||
var _pad1: Float = 0
|
||||
var _pad2: Float = 0
|
||||
var _pad3: Float = 0
|
||||
var _pad4: Float = 0
|
||||
}
|
||||
|
||||
private let device: MTLDevice
|
||||
private let commandQueue: MTLCommandQueue
|
||||
private let bgPipeline: MTLRenderPipelineState
|
||||
private let uniformsBuffer: MTLBuffer
|
||||
private var startTime: CFTimeInterval = CACurrentMediaTime()
|
||||
|
||||
var uniforms = SceneUniforms()
|
||||
|
||||
static func make() -> SceneRenderer? { SceneRenderer(failable: ()) }
|
||||
|
||||
private init?(failable: Void) {
|
||||
guard let dev = MTLCreateSystemDefaultDevice(),
|
||||
let queue = dev.makeCommandQueue() else { return nil }
|
||||
self.device = dev
|
||||
self.commandQueue = queue
|
||||
guard let url = Bundle.main.url(forResource: "scene",
|
||||
withExtension: "metal"),
|
||||
let source = try? String(contentsOf: url, encoding: .utf8) else {
|
||||
NSLog("SceneRenderer: scene.metal missing from bundle")
|
||||
return nil
|
||||
}
|
||||
let lib: MTLLibrary
|
||||
do { lib = try dev.makeLibrary(source: source,
|
||||
options: MTLCompileOptions()) }
|
||||
catch { NSLog("SceneRenderer: compile error %@",
|
||||
String(describing: error)); return nil }
|
||||
guard let vfn = lib.makeFunction(name: "bg_vertex"),
|
||||
let ffn = lib.makeFunction(name: "bg_fragment") else {
|
||||
NSLog("SceneRenderer: bg_vertex/bg_fragment missing")
|
||||
return nil
|
||||
}
|
||||
let pd = MTLRenderPipelineDescriptor()
|
||||
pd.vertexFunction = vfn
|
||||
pd.fragmentFunction = ffn
|
||||
pd.colorAttachments[0].pixelFormat = .bgra8Unorm
|
||||
do { self.bgPipeline =
|
||||
try dev.makeRenderPipelineState(descriptor: pd) }
|
||||
catch { NSLog("SceneRenderer: pipeline failed %@",
|
||||
String(describing: error)); return nil }
|
||||
guard let buf = dev.makeBuffer(
|
||||
length: MemoryLayout<SceneUniforms>.stride,
|
||||
options: .storageModeShared) else { return nil }
|
||||
self.uniformsBuffer = buf
|
||||
super.init()
|
||||
}
|
||||
|
||||
func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {
|
||||
uniforms.width = Float(size.width)
|
||||
uniforms.height = Float(size.height)
|
||||
}
|
||||
|
||||
func draw(in view: MTKView) {
|
||||
uniforms.time = Float(CACurrentMediaTime() - startTime)
|
||||
uniformsBuffer.contents().bindMemory(
|
||||
to: SceneUniforms.self, capacity: 1).pointee = uniforms
|
||||
guard let rpd = view.currentRenderPassDescriptor,
|
||||
let drawable = view.currentDrawable,
|
||||
let cb = commandQueue.makeCommandBuffer(),
|
||||
let enc = cb.makeRenderCommandEncoder(descriptor: rpd)
|
||||
else { return }
|
||||
enc.setRenderPipelineState(bgPipeline)
|
||||
enc.setFragmentBuffer(uniformsBuffer, offset: 0, index: 0)
|
||||
enc.drawPrimitives(type: .triangle, vertexStart: 0, vertexCount: 3)
|
||||
enc.endEncoding()
|
||||
cb.present(drawable)
|
||||
cb.commit()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Layout-guard test**
|
||||
|
||||
Add `avlivebody-mac/Tests/AVLiveBodyTests/SceneUniformsTests.swift`:
|
||||
```swift
|
||||
import XCTest
|
||||
@testable import AVLiveBody
|
||||
|
||||
final class SceneUniformsTests: XCTestCase {
|
||||
func testStride() {
|
||||
XCTAssertEqual(MemoryLayout<SceneRenderer.SceneUniforms>.stride, 144)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Regenerate + build + test**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && xcodegen generate && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/avbody-dd" && \
|
||||
xcodebuild test -project AVLiveBody.xcodeproj -scheme AVLiveBody -destination 'platform=macOS' \
|
||||
-derivedDataPath "$DD" 2>&1 | grep -E "error:|SceneUniformsTests|** TEST"
|
||||
```
|
||||
Expected: `SceneUniformsTests` passes (`stride == 144`), `** TEST SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/Resources/scene.metal \
|
||||
avlivebody-mac/Sources/AVLiveBody/SceneRenderer.swift \
|
||||
avlivebody-mac/Tests/AVLiveBodyTests/SceneUniformsTests.swift \
|
||||
avlivebody-mac/project.yml
|
||||
git commit -m "feat(avlivebody): port metal scene renderer"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Layered compositing (MTKView behind ARView)
|
||||
|
||||
**Files:**
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/SceneController.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `SceneRenderer` (Task 1), `SceneController.arView`.
|
||||
- Produces: `LayeredSceneView: NSViewRepresentable` exposing the created
|
||||
`SceneRenderer` to the caller (via a binding or a passed-in instance) and
|
||||
hosting MTKView (back, transparent) + `SceneController.arView` (front,
|
||||
transparent). `RenderSettings` gains `@Published var showScene: Bool = true`
|
||||
and `@Published var vizMode: Int = 0`.
|
||||
|
||||
- [ ] **Step 1: Add showScene/vizMode to RenderSettings**
|
||||
|
||||
In `RenderSettings.swift`, add:
|
||||
```swift
|
||||
// Metal shader background (Phase 2)
|
||||
@Published var showScene: Bool = true
|
||||
@Published var vizMode: Int = 0 // 0..9
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Make the ARView transparent in SceneController**
|
||||
|
||||
In `SceneController.setUp()`, change the background from black to clear so the
|
||||
MTKView shows through:
|
||||
```swift
|
||||
arView.environment.background = .color(.clear)
|
||||
arView.layer?.isOpaque = false
|
||||
arView.layer?.backgroundColor = NSColor.clear.cgColor
|
||||
```
|
||||
(Replace the existing `arView.environment.background = .color(.black)` line.)
|
||||
|
||||
- [ ] **Step 3: Create LayeredSceneView**
|
||||
|
||||
```swift
|
||||
// avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift
|
||||
import AppKit
|
||||
import MetalKit
|
||||
import RealityKit
|
||||
import SwiftUI
|
||||
|
||||
/// Layered container: MTKView (Metal shader bg, transparent) behind the
|
||||
/// SceneController's transparent ARView. The owner holds the SceneRenderer
|
||||
/// so it can push uniforms each frame.
|
||||
struct LayeredSceneView: NSViewRepresentable {
|
||||
let controller: SceneController
|
||||
let renderer: SceneRenderer?
|
||||
|
||||
func makeNSView(context: Context) -> NSView {
|
||||
controller.setUp()
|
||||
let container = NSView(frame: .zero)
|
||||
container.wantsLayer = true
|
||||
container.layer?.backgroundColor = NSColor.black.cgColor
|
||||
|
||||
if let renderer {
|
||||
let mtk = MTKView(frame: container.bounds,
|
||||
device: MTLCreateSystemDefaultDevice())
|
||||
mtk.delegate = renderer
|
||||
mtk.colorPixelFormat = .bgra8Unorm
|
||||
mtk.framebufferOnly = false
|
||||
mtk.layer?.isOpaque = true
|
||||
mtk.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 1)
|
||||
mtk.preferredFramesPerSecond = 60
|
||||
mtk.autoresizingMask = [.width, .height]
|
||||
container.addSubview(mtk)
|
||||
context.coordinator.mtk = mtk
|
||||
}
|
||||
|
||||
let arView = controller.arView
|
||||
arView.frame = container.bounds
|
||||
arView.autoresizingMask = [.width, .height]
|
||||
arView.wantsLayer = true
|
||||
arView.layer?.isOpaque = false
|
||||
arView.layer?.backgroundColor = NSColor.clear.cgColor
|
||||
container.addSubview(arView) // front
|
||||
return container
|
||||
}
|
||||
|
||||
func updateNSView(_ nsView: NSView, context: Context) {
|
||||
context.coordinator.mtk?.isHidden = false // visibility via showScene handled by caller setting renderer.uniforms / hidden
|
||||
}
|
||||
|
||||
func makeCoordinator() -> Coordinator { Coordinator() }
|
||||
final class Coordinator { var mtk: MTKView? }
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Wire LayeredSceneView into ContentView**
|
||||
|
||||
In `AVLiveBodyApp.swift` `ContentView`, add a renderer and swap `SceneView` for
|
||||
`LayeredSceneView`:
|
||||
```swift
|
||||
private let renderer = SceneRenderer.make()
|
||||
```
|
||||
Replace `SceneView(controller: controller)` in the `body` with:
|
||||
```swift
|
||||
LayeredSceneView(controller: controller, renderer: renderer)
|
||||
```
|
||||
In `applyAll()`, drive scene visibility + mode onto the renderer:
|
||||
```swift
|
||||
if let r = renderer {
|
||||
r.uniforms.viz_mode = Float(settings.vizMode)
|
||||
}
|
||||
```
|
||||
(Show/hide of the MTKView when `!settings.showScene` is handled in Task 4 via
|
||||
the panel toggle; for now the scene is always shown.)
|
||||
|
||||
- [ ] **Step 5: Regenerate + build**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/avlivebody-mac" && xcodegen generate && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/avbody-dd" && \
|
||||
xcodebuild -project AVLiveBody.xcodeproj -scheme AVLiveBody -configuration Debug -derivedDataPath "$DD" build 2>&1 | grep -E "error:|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
Expected: `** BUILD SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/RenderSettings.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/SceneController.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift \
|
||||
avlivebody-mac/AVLiveBody.xcodeproj/project.pbxproj
|
||||
git commit -m "feat(avlivebody): layered metal plus arview"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: SceneUniforms feed (Mac-side derivation, isolated)
|
||||
|
||||
**Files:**
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/SceneUniformBuilder.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `USBSkeletonConsumer.skeletons` (`[Int: SkeletonPayload]`),
|
||||
`.hands` (`HandsPayload?`), `.face` (`FacePayload?`); the `SceneRenderer`.
|
||||
- Produces: `enum SceneUniformBuilder { static func fill(_ u: inout SceneRenderer.SceneUniforms, skeletons:, hands:, face:, prevPelvis: inout SIMD3<Float>?) }` — the SINGLE place deriving shader scalars (so option A can later swap it for an iPhone-derivatives passthrough).
|
||||
|
||||
- [ ] **Step 1: Create SceneUniformBuilder**
|
||||
|
||||
Derives the body/hand/face channels; leaves data-feed channels at 0. (Indices
|
||||
follow ARKit body joints — confirm the wrist/head/pelvis indices against
|
||||
`ARSkeletonDefinition.defaultBody3D` jointNames during implementation; the
|
||||
constants below are the documented ARKit body3D ordering.)
|
||||
|
||||
```swift
|
||||
// avlivebody-mac/Sources/AVLiveBody/SceneUniformBuilder.swift
|
||||
import AVLiveWire
|
||||
import simd
|
||||
|
||||
enum SceneUniformBuilder {
|
||||
// ARKit defaultBody3D indices (verify against jointNames at impl time).
|
||||
private static let hipsIdx = 0
|
||||
private static let headIdx = 51
|
||||
private static let lWristIdx = 32
|
||||
private static let rWristIdx = 7
|
||||
|
||||
static func fill(_ u: inout SceneRenderer.SceneUniforms,
|
||||
skeletons: [Int: SkeletonPayload],
|
||||
hands: HandsPayload?,
|
||||
face: FacePayload?,
|
||||
prevPelvis: inout SIMD3<Float>?) {
|
||||
u.pose_count = Float(skeletons.count)
|
||||
u.pose_alive = skeletons.isEmpty ? 0 : 1
|
||||
|
||||
if let body = skeletons.values.first {
|
||||
func j(_ i: Int) -> SIMD3<Float>? {
|
||||
(i < body.joints.count && body.valid[i]) ? body.joints[i] : nil
|
||||
}
|
||||
if let hips = j(hipsIdx) {
|
||||
u.body_x = hips.x; u.body_y = hips.y; u.body_z = hips.z
|
||||
if let prev = prevPelvis {
|
||||
let v = simd_length(hips - prev)
|
||||
u.pose_velocity = u.pose_velocity * 0.7 + v * 0.3
|
||||
}
|
||||
prevPelvis = hips
|
||||
if let head = j(headIdx) {
|
||||
u.body_height = abs(head.y - hips.y)
|
||||
}
|
||||
}
|
||||
if let lw = j(lWristIdx), let rw = j(rWristIdx) {
|
||||
u.arm_spread = abs(lw.x - rw.x)
|
||||
}
|
||||
}
|
||||
|
||||
// Hands (Phase-0 Vision, normalized [0,1]).
|
||||
u.hand_l_x = 0; u.hand_l_y = 0; u.hand_r_x = 0; u.hand_r_y = 0
|
||||
u.finger_pinch_l = 0; u.finger_pinch_r = 0
|
||||
for hand in hands?.hands ?? [] {
|
||||
let c = centroid(hand.points) // (x,y)
|
||||
let pinch = dist(hand.points, 4, 8) // thumb_tip..index_tip
|
||||
if hand.isRight {
|
||||
u.hand_r_x = c.x; u.hand_r_y = c.y; u.finger_pinch_r = pinch
|
||||
} else {
|
||||
u.hand_l_x = c.x; u.hand_l_y = c.y; u.finger_pinch_l = pinch
|
||||
}
|
||||
}
|
||||
|
||||
// Face (Phase-0 Vision allPoints). Region-based derivation is
|
||||
// refined at impl time; here a robust bbox-ratio proxy.
|
||||
if let f = face, f.points.count > 8 {
|
||||
let ys = f.points.map { $0.y }, xs = f.points.map { $0.x }
|
||||
let h = (ys.max() ?? 0) - (ys.min() ?? 0)
|
||||
let w = (xs.max() ?? 0) - (xs.min() ?? 0)
|
||||
u.mouth_open = 0 // set from outer/inner-lip points at impl time
|
||||
u.eye_open_l = 0; u.eye_open_r = 0
|
||||
u.head_tilt = 0; u.head_yaw = 0
|
||||
_ = (h, w)
|
||||
}
|
||||
}
|
||||
|
||||
private static func centroid(_ p: [SIMD3<Float>]) -> SIMD2<Float> {
|
||||
guard !p.isEmpty else { return .zero }
|
||||
var s = SIMD2<Float>(0, 0)
|
||||
for v in p { s.x += v.x; s.y += v.y }
|
||||
return s / Float(p.count)
|
||||
}
|
||||
private static func dist(_ p: [SIMD3<Float>], _ a: Int, _ b: Int)
|
||||
-> Float {
|
||||
guard a < p.count, b < p.count else { return 0 }
|
||||
return simd_length(SIMD2(p[a].x, p[a].y) - SIMD2(p[b].x, p[b].y))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Drive the uniforms each frame from ContentView**
|
||||
|
||||
In `AVLiveBodyApp.swift`, add a `@State private var prevPelvis: SIMD3<Float>?`
|
||||
and feed the renderer when skeleton/hands/face change. Add `.onReceive`
|
||||
handlers (alongside the existing skeleton one):
|
||||
```swift
|
||||
.onReceive(consumer.$hands) { _ in pushUniforms() }
|
||||
.onReceive(consumer.$face) { _ in pushUniforms() }
|
||||
```
|
||||
and extend the existing skeleton `.onReceive` to also `pushUniforms()`. Add:
|
||||
```swift
|
||||
@State private var prevPelvis: SIMD3<Float>?
|
||||
private func pushUniforms() {
|
||||
guard let r = renderer else { return }
|
||||
r.uniforms.viz_mode = Float(settings.vizMode)
|
||||
SceneUniformBuilder.fill(&r.uniforms,
|
||||
skeletons: consumer.skeletons,
|
||||
hands: consumer.hands,
|
||||
face: consumer.face,
|
||||
prevPelvis: &prevPelvis)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Regenerate + build**
|
||||
|
||||
Run the same `xcodegen generate && xcodebuild ... build` as Task 2 Step 5.
|
||||
Expected: `** BUILD SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/SceneUniformBuilder.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift \
|
||||
avlivebody-mac/AVLiveBody.xcodeproj/project.pbxproj
|
||||
git commit -m "feat(avlivebody): feed scene uniforms from stream"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Viz-mode picker + Scene toggle in the panel
|
||||
|
||||
**Files:**
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `RenderSettings.showScene`/`vizMode` (Task 2).
|
||||
- Produces: panel controls; MTKView hidden when `!showScene`; `vizMode` reflected
|
||||
in the renderer.
|
||||
|
||||
- [ ] **Step 1: Add a Scene section to SettingsPanel**
|
||||
|
||||
In `SettingsPanel.swift`, add after the "Vue" group:
|
||||
```swift
|
||||
group("Scene") {
|
||||
Toggle("Fond shader", isOn: $settings.showScene)
|
||||
Picker("Mode", selection: $settings.vizMode) {
|
||||
ForEach(0..<10, id: \.self) { i in
|
||||
Text(Self.modeName(i)).tag(i)
|
||||
}
|
||||
}
|
||||
.pickerStyle(.menu)
|
||||
}
|
||||
```
|
||||
Add the mode-name helper:
|
||||
```swift
|
||||
static func modeName(_ i: Int) -> String {
|
||||
let n = ["storm","tunnel","plasma","kaleido","voronoi",
|
||||
"metaballs","starfield","bars","hands3d","openpos"]
|
||||
return n.indices.contains(i) ? "\(i) \(n[i])" : "\(i)"
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Expose MTKView hide to the controller path**
|
||||
|
||||
In `LayeredSceneView`, store the renderer/MTKView so visibility can be toggled.
|
||||
Add to `updateNSView`:
|
||||
```swift
|
||||
func updateNSView(_ nsView: NSView, context: Context) {
|
||||
context.coordinator.mtk?.isHidden = !showScene
|
||||
}
|
||||
```
|
||||
and add `let showScene: Bool` to `LayeredSceneView` (passed from ContentView).
|
||||
|
||||
- [ ] **Step 3: Pass showScene from ContentView**
|
||||
|
||||
In `AVLiveBodyApp.swift`, update the `LayeredSceneView(...)` call to
|
||||
`LayeredSceneView(controller: controller, renderer: renderer, showScene: settings.showScene)`
|
||||
and ensure `pushUniforms()` already sets `viz_mode` (Task 3). The `.onReceive(settings.objectWillChange)` from Phase 1 re-renders the representable so `updateNSView` runs with the new `showScene`.
|
||||
|
||||
- [ ] **Step 4: Regenerate + build**
|
||||
|
||||
Same build command. Expected: `** BUILD SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/SettingsPanel.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/LayeredSceneView.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift \
|
||||
avlivebody-mac/AVLiveBody.xcodeproj/project.pbxproj
|
||||
git commit -m "feat(avlivebody): viz mode picker and scene toggle"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: 3D hand/face skeleton
|
||||
|
||||
**Files:**
|
||||
- Create: `avlivebody-mac/Sources/AVLiveBody/HandFaceSkeleton.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/SceneController.swift`
|
||||
- Modify: `avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `consumer.hands` (`HandsPayload?`), `consumer.face` (`FacePayload?`);
|
||||
the body pelvis depth from `SceneController`'s skeleton.
|
||||
- Produces: `HandFaceSkeleton` (marker pools for 21×2 hand points + face points)
|
||||
added to the worldAnchor; `SceneController.updateHandFace(hands:face:)`.
|
||||
|
||||
- [ ] **Step 1: Create HandFaceSkeleton**
|
||||
|
||||
Renders normalized 2D Vision points as small spheres on a plane at a fixed depth
|
||||
(the body's pelvis z if available, else 0). Normalized [0,1] → scene meters via a
|
||||
simple span mapping.
|
||||
|
||||
```swift
|
||||
// avlivebody-mac/Sources/AVLiveBody/HandFaceSkeleton.swift
|
||||
import AVLiveWire
|
||||
import AppKit
|
||||
import RealityKit
|
||||
import simd
|
||||
|
||||
@MainActor
|
||||
final class HandFaceSkeleton {
|
||||
let root = Entity()
|
||||
private static let r: Float = 0.008
|
||||
private static let spanX: Float = 1.2 // maps [0,1] -> [-0.6,0.6] m
|
||||
private static let spanY: Float = 0.9
|
||||
|
||||
private let mesh = MeshResource.generateSphere(radius: r)
|
||||
private let handMat = SimpleMaterial(color: .cyan, isMetallic: false)
|
||||
private let faceMat = SimpleMaterial(color: .magenta, isMetallic: false)
|
||||
private var handPool: [ModelEntity] = []
|
||||
private var facePool: [ModelEntity] = []
|
||||
private var depth: Float = 0
|
||||
|
||||
func setDepth(_ z: Float) { depth = z }
|
||||
|
||||
func update(hands: HandsPayload?, face: FacePayload?) {
|
||||
let hpts = (hands?.hands ?? []).flatMap { $0.points }
|
||||
layout(&handPool, count: hpts.count, mat: handMat)
|
||||
for (i, p) in hpts.enumerated() { place(handPool[i], p.x, p.y) }
|
||||
for i in hpts.count..<handPool.count { handPool[i].isEnabled = false }
|
||||
|
||||
let fpts = face?.points ?? []
|
||||
layout(&facePool, count: fpts.count, mat: faceMat)
|
||||
for (i, p) in fpts.enumerated() { place(facePool[i], p.x, p.y) }
|
||||
for i in fpts.count..<facePool.count { facePool[i].isEnabled = false }
|
||||
}
|
||||
|
||||
private func place(_ e: ModelEntity, _ nx: Float, _ ny: Float) {
|
||||
// normalized top-left [0,1] -> centered meters, y up
|
||||
e.transform.translation = SIMD3<Float>(
|
||||
(nx - 0.5) * Self.spanX,
|
||||
(0.5 - ny) * Self.spanY,
|
||||
depth)
|
||||
e.isEnabled = true
|
||||
}
|
||||
|
||||
private func layout(_ pool: inout [ModelEntity], count: Int,
|
||||
mat: SimpleMaterial) {
|
||||
while pool.count < count {
|
||||
let e = ModelEntity(mesh: mesh, materials: [mat])
|
||||
root.addChild(e); pool.append(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add HandFaceSkeleton to SceneController**
|
||||
|
||||
In `SceneController`, add `private(set) var handFace: HandFaceSkeleton?`, create
|
||||
it in `setUp()` (`let hf = HandFaceSkeleton(); worldAnchor.addChild(hf.root); handFace = hf`), and add:
|
||||
```swift
|
||||
func updateHandFace(hands: HandsPayload?, face: FacePayload?) {
|
||||
handFace?.update(hands: hands, face: face)
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Route hands/face from ContentView**
|
||||
|
||||
In `AVLiveBodyApp.swift`, in the `pushUniforms()` (or the `.onReceive` for
|
||||
hands/face), also call:
|
||||
```swift
|
||||
controller.updateHandFace(hands: consumer.hands, face: consumer.face)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Regenerate + build**
|
||||
|
||||
Same build command. Expected: `** BUILD SUCCEEDED **`.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add avlivebody-mac/Sources/AVLiveBody/HandFaceSkeleton.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/SceneController.swift \
|
||||
avlivebody-mac/Sources/AVLiveBody/AVLiveBodyApp.swift \
|
||||
avlivebody-mac/AVLiveBody.xcodeproj/project.pbxproj
|
||||
git commit -m "feat(avlivebody): 3d hand and face skeleton"
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Manual visual check (user, on-device end-to-end)**
|
||||
|
||||
With the iPhone streaming (body + hands + face) to the Mac: the Metal shader
|
||||
renders behind the body; selecting modes 0–9 in the panel switches it; the
|
||||
scene reacts to body/hand movement; cyan hand points + magenta face points track
|
||||
in 3D. Confirm visually.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review notes
|
||||
|
||||
- **Spec coverage (§6):** scene.metal + SceneRenderer → Task 1; layered MTKView+ARView → Task 2; uniform feed (§6.1 mapping) → Task 3; viz-mode picker → Task 4; 3D hand/face skeleton → Task 5.
|
||||
- **A-track isolation:** all derivation lives in `SceneUniformBuilder.fill` (Task 3) so option A (iPhone-derived uniforms) replaces only that function body.
|
||||
- **Placeholder honesty:** Task 3's face derivations (mouth_open/eye_open/head_tilt/yaw) are seeded to 0 with a note to derive from Vision face regions at implementation time — these need the live Vision landmark indices, which the implementer confirms on-device; the body+hand channels are fully derived. The ARKit joint indices in `SceneUniformBuilder` are marked "verify against jointNames" — the implementer must confirm them (ARKit body3D ordering) during Task 3.
|
||||
- **xcodegen:** every task with a new file runs `xcodegen generate`; the pbxproj is regenerated (the project is directory-glob; `git add` of the pbxproj is included where the prior phase tracked it — if `.gitignore` excludes it, the add is a no-op and a clean checkout relies on `xcodegen generate`, per the README).
|
||||
- **Risk:** Task 2 (NSView layering of MTKView + ARView transparency) is the highest-risk; the archive's working compositing is the reference. If the ARView does not composite transparently over the MTKView, consult `_archive .../BodyView.swift:36-63`.
|
||||
|
||||
> NOTE (open, needs decision before/with Task 3): the exact ARKit `defaultBody3D`
|
||||
> joint indices for hips/head/wrists, and the Vision face-region derivations for
|
||||
> mouth/eyes/head pose, must be pinned during implementation. Two genuinely
|
||||
> unverified spots are flagged inline; treat them as implement-time confirmations,
|
||||
> not guesses to ship.
|
||||
@@ -0,0 +1,924 @@
|
||||
# Hand-driven `data_only` Modulation Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Turn MediaPipe hand landmarks into a compact expressive feature vector that modulates the `data_only` SuperCollider FX rack and visibly drives the Metal scene, via one shared signal.
|
||||
|
||||
**Architecture:** A new Python `HandFeatureExtractor` reads the already-captured hand landmarks from `state`, computes per-hand position/openness/speed plus two-hand distance, and emits them (a) over OSC `/pose/hands` to SuperCollider and (b) into the Metal renderer uniforms. SuperCollider stores them in `~handFeat` and maps them to a new global filter / reverb / width / drive. The Metal shader reacts to the same features. Modulation only — no note generation, no dense hand mesh.
|
||||
|
||||
**Tech Stack:** Python 3.11+ (uv), MediaPipe, python-osc (`SimpleUDPClient`), pyobjc + Metal (`scene.metal`), SuperCollider (sclang/scsynth).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- Python: use **uv** for all runs (`uv run ...`); never pip/poetry/conda directly.
|
||||
- No emojis in code/docs/commits.
|
||||
- Commits: subject ≤ 50 chars, body ≤ 72 chars/line, no AI attribution, no `--no-verify`, no underscore in the commit scope (use hyphens, e.g. `data-only`).
|
||||
- `.scd` files: parentheses/brackets must balance (P:0 B:0); files loaded via `.load` need exactly one top-level block. Run the `validating-scd-files` skill after every `.scd` edit.
|
||||
- SuperCollider SynthDefs in `sound_algo`: lagged controls, `EnvGen.kr(..., doneAction: 2)` for any voiced synth, `Pan2.ar`/`ReplaceOut.ar` on the master path, no zombie synths. Use the `adding-synthdef` skill for new SynthDefs.
|
||||
- Stay inside the `data_only` subsystem; do not touch the main live engine (tracks A–W, `01_live.scd`).
|
||||
|
||||
### v1 contract decision (refines the spec)
|
||||
|
||||
The OSC route is a **single global message** `pid=0` carrying the two most
|
||||
prominent hands, labelled **L = leftmost on screen, R = rightmost** (by centroid
|
||||
x). This avoids the fragile pid-parity handedness proxy in the existing
|
||||
`_send_hand`. Multi-person hand modulation is a deliberate later widening.
|
||||
|
||||
```
|
||||
/pose/hands [0, lx, ly, lopen, lspeed, rx, ry, ropen, rspeed, dist]
|
||||
```
|
||||
- `lx,ly,rx,ry` ∈ [0,1] (normalized; y down, MediaPipe convention)
|
||||
- `lopen,ropen` ∈ [0,1] (0 = fist, 1 = open hand)
|
||||
- `lspeed,rspeed` ≥ 0 (centroid speed, normalized units/frame)
|
||||
- `dist` ∈ [0,1] (0 = one/zero hands or hands together, 1 = arms wide)
|
||||
- A hand absent → its 4 floats are 0.0.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `data_only_viz/hand_features.py` | Create | `HandFeatureExtractor`: landmarks → feature dict |
|
||||
| `data_only_viz/tests/test_hand_features.py` | Create | Unit tests for the extractor |
|
||||
| `data_only_viz/pose_bridge.py` | Modify | `send_hands(feats, t)` OSC emitter |
|
||||
| `data_only_viz/tests/test_pose_bridge_hands.py` | Create | Unit test for message construction |
|
||||
| `data_only_viz/action_head_pub.py` | Modify | Per-tick: extract + send hand features |
|
||||
| `data_only_viz/renderer.py` | Modify | Pack hand features into Metal uniforms |
|
||||
| `data_only_viz/shaders/scene.metal` | Modify | Expressive hand uniforms + scene reaction |
|
||||
| `sound_algo/control/data_feeds.scd` | Modify | OSCdef `/pose/hands` → `~handFeat` |
|
||||
| `sound_algo/data_only/engine.scd` | Modify | `\do_master_filter` (cutoff/width/drive) |
|
||||
| `sound_algo/data_only/scene_pose_action.scd` | Modify | `~mapHandsToFx` + wire FX stubs |
|
||||
| `sound_algo/data_only/boot.scd` | Modify | Load `scene_pose_action.scd` |
|
||||
| `data_only_viz/README.md` / `sound_algo/data_only/` notes | Modify | Document `/pose/hands` contract |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: HandFeatureExtractor (Python, pure)
|
||||
|
||||
**Files:**
|
||||
- Create: `data_only_viz/hand_features.py`
|
||||
- Test: `data_only_viz/tests/test_hand_features.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: a list of hands, each a list of ≥21 objects with float attributes `.x`, `.y` (MediaPipe `PoseKp` shape; `state.persons_hands` items).
|
||||
- Produces:
|
||||
- `HandFeatureExtractor(history: int = 5)`
|
||||
- `.step(hands: list[list]) -> dict` returning
|
||||
`{"L": {"cx","cy","openness","speed"} | None, "R": {...} | None, "dist": float}`
|
||||
- module constant `NEUTRAL_HAND = {"cx":0.5,"cy":0.5,"openness":0.0,"speed":0.0}`
|
||||
|
||||
Landmark index constants (MediaPipe hand): `WRIST=0`, `THUMB_TIP=4`, `MIDDLE_MCP=9`, `PINKY_TIP=20`.
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```python
|
||||
# data_only_viz/tests/test_hand_features.py
|
||||
import math
|
||||
import pytest
|
||||
from data_only_viz.hand_features import HandFeatureExtractor, NEUTRAL_HAND
|
||||
|
||||
|
||||
class LM:
|
||||
"""Minimal landmark stub (PoseKp-shaped)."""
|
||||
def __init__(self, x, y, z=0.0, c=1.0):
|
||||
self.x, self.y, self.z, self.c = x, y, z, c
|
||||
|
||||
|
||||
def _hand(cx, cy, span):
|
||||
"""21 landmarks centered at (cx,cy); thumb/pinky separated by `span`.
|
||||
|
||||
Index 0 wrist, 4 thumb_tip, 9 middle_mcp, 20 pinky_tip.
|
||||
hand_size (wrist->middle_mcp) fixed at 0.10 so openness ~ span/size.
|
||||
"""
|
||||
pts = [LM(cx, cy) for _ in range(21)]
|
||||
pts[0] = LM(cx, cy + 0.05) # wrist below center
|
||||
pts[9] = LM(cx, cy - 0.05) # middle_mcp above -> size 0.10
|
||||
pts[4] = LM(cx - span / 2, cy) # thumb_tip
|
||||
pts[20] = LM(cx + span / 2, cy) # pinky_tip
|
||||
return pts
|
||||
|
||||
|
||||
def test_open_hand_high_openness():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.5, 0.5, span=0.20)]) # span/size = 2.0
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["openness"] > 0.8
|
||||
|
||||
|
||||
def test_fist_low_openness():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.5, 0.5, span=0.03)]) # span/size = 0.3
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["openness"] < 0.2
|
||||
|
||||
|
||||
def test_static_hand_zero_speed():
|
||||
ext = HandFeatureExtractor()
|
||||
h = _hand(0.5, 0.5, span=0.1)
|
||||
ext.step([h])
|
||||
out = ext.step([h])
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["speed"] == pytest.approx(0.0, abs=1e-6)
|
||||
|
||||
|
||||
def test_moving_hand_positive_speed():
|
||||
ext = HandFeatureExtractor()
|
||||
ext.step([_hand(0.2, 0.5, span=0.1)])
|
||||
out = ext.step([_hand(0.6, 0.5, span=0.1)])
|
||||
hand = out["L"] or out["R"]
|
||||
assert hand["speed"] > 0.0
|
||||
|
||||
|
||||
def test_left_right_by_screen_x():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.8, 0.5, span=0.1), _hand(0.2, 0.5, span=0.1)])
|
||||
assert out["L"]["cx"] < out["R"]["cx"] # L is leftmost on screen
|
||||
assert out["dist"] > 0.0
|
||||
|
||||
|
||||
def test_single_hand_zero_dist():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([_hand(0.5, 0.5, span=0.1)])
|
||||
assert out["dist"] == 0.0
|
||||
|
||||
|
||||
def test_no_hands_returns_none():
|
||||
ext = HandFeatureExtractor()
|
||||
out = ext.step([])
|
||||
assert out["L"] is None and out["R"] is None and out["dist"] == 0.0
|
||||
|
||||
|
||||
def test_nan_landmark_falls_back_no_exception():
|
||||
ext = HandFeatureExtractor()
|
||||
h = _hand(0.5, 0.5, span=0.1)
|
||||
h[0] = LM(float("nan"), float("nan"))
|
||||
out = ext.step([h]) # must not raise
|
||||
hand = out["L"] or out["R"]
|
||||
assert math.isfinite(hand["cx"]) and math.isfinite(hand["openness"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live" && uv run python -m pytest data_only_viz/tests/test_hand_features.py -v`
|
||||
Expected: FAIL — `ModuleNotFoundError: No module named 'data_only_viz.hand_features'`
|
||||
|
||||
- [ ] **Step 3: Write the implementation**
|
||||
|
||||
```python
|
||||
# data_only_viz/hand_features.py
|
||||
"""Turn MediaPipe hand landmarks into a compact expressive feature vector.
|
||||
|
||||
Mirrors the kinematics pattern of action_head.FeatureExtractor: a small ring
|
||||
buffer per hand slot, finite-guarded, clamped to declared ranges. Output feeds
|
||||
both the OSC /pose/hands route (audio) and the Metal renderer uniforms (visual).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
import math
|
||||
from collections import deque
|
||||
|
||||
WRIST, THUMB_TIP, MIDDLE_MCP, PINKY_TIP = 0, 4, 9, 20
|
||||
|
||||
NEUTRAL_HAND = {"cx": 0.5, "cy": 0.5, "openness": 0.0, "speed": 0.0}
|
||||
|
||||
# openness calibration: (span/size) maps fist~0.3 -> 0, open~2.0 -> 1
|
||||
_OPEN_LO, _OPEN_HI = 0.5, 2.0
|
||||
|
||||
|
||||
def _finite(v: float, fallback: float) -> float:
|
||||
return v if isinstance(v, float) and math.isfinite(v) else fallback
|
||||
|
||||
|
||||
def _clamp(v: float, lo: float, hi: float) -> float:
|
||||
return lo if v < lo else hi if v > hi else v
|
||||
|
||||
|
||||
class HandFeatureExtractor:
|
||||
def __init__(self, history: int = 5):
|
||||
self._history = max(2, history)
|
||||
# two screen slots: 0 = leftmost (L), 1 = rightmost (R)
|
||||
self._buf = [deque(maxlen=self._history), deque(maxlen=self._history)]
|
||||
|
||||
def _features(self, lm: list) -> dict:
|
||||
xs = [_finite(float(p.x), 0.5) for p in lm[:21]]
|
||||
ys = [_finite(float(p.y), 0.5) for p in lm[:21]]
|
||||
cx = _clamp(sum(xs) / len(xs), 0.0, 1.0)
|
||||
cy = _clamp(sum(ys) / len(ys), 0.0, 1.0)
|
||||
size = math.hypot(xs[MIDDLE_MCP] - xs[WRIST], ys[MIDDLE_MCP] - ys[WRIST])
|
||||
size = size if size > 1e-4 else 1e-4
|
||||
span = math.hypot(xs[THUMB_TIP] - xs[PINKY_TIP],
|
||||
ys[THUMB_TIP] - ys[PINKY_TIP])
|
||||
openness = _clamp((span / size - _OPEN_LO) / (_OPEN_HI - _OPEN_LO),
|
||||
0.0, 1.0)
|
||||
return {"cx": cx, "cy": cy, "openness": openness}
|
||||
|
||||
def _speed(self, slot: int, cx: float, cy: float) -> float:
|
||||
buf = self._buf[slot]
|
||||
spd = 0.0
|
||||
if buf:
|
||||
px, py = buf[-1]
|
||||
spd = math.hypot(cx - px, cy - py)
|
||||
buf.append((cx, cy))
|
||||
return _clamp(spd, 0.0, 1.0)
|
||||
|
||||
def step(self, hands: list) -> dict:
|
||||
feats = []
|
||||
for lm in hands:
|
||||
if lm and len(lm) >= 21:
|
||||
feats.append(self._features(lm))
|
||||
if not feats:
|
||||
self._buf[0].clear()
|
||||
self._buf[1].clear()
|
||||
return {"L": None, "R": None, "dist": 0.0}
|
||||
feats.sort(key=lambda f: f["cx"]) # leftmost first
|
||||
left = feats[0]
|
||||
right = feats[-1] if len(feats) > 1 else None
|
||||
left["speed"] = self._speed(0, left["cx"], left["cy"])
|
||||
out_l = left
|
||||
out_r = None
|
||||
dist = 0.0
|
||||
if right is not None:
|
||||
right["speed"] = self._speed(1, right["cx"], right["cy"])
|
||||
out_r = right
|
||||
dist = _clamp(math.hypot(right["cx"] - left["cx"],
|
||||
right["cy"] - left["cy"]), 0.0, 1.0)
|
||||
else:
|
||||
self._buf[1].clear()
|
||||
return {"L": out_l, "R": out_r, "dist": dist}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live" && uv run python -m pytest data_only_viz/tests/test_hand_features.py -v`
|
||||
Expected: PASS (8 passed)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add data_only_viz/hand_features.py data_only_viz/tests/test_hand_features.py
|
||||
git commit -m "feat(viz): hand feature extractor"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: OSC sender `send_hands`
|
||||
|
||||
**Files:**
|
||||
- Modify: `data_only_viz/pose_bridge.py` (class `PoseSoundBridge`, near `send_kin` ~line 311)
|
||||
- Test: `data_only_viz/tests/test_pose_bridge_hands.py`
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the dict from `HandFeatureExtractor.step()` (`{"L","R","dist"}`).
|
||||
- Produces: `PoseSoundBridge.send_hands(self, feats: dict, t: float) -> None` sending OSC `"/pose/hands"` with `[0, lx,ly,lopen,lspeed, rx,ry,ropen,rspeed, dist]` (1 int + 9 floats) via `self._client`. Absent hand → its 4 floats are 0.0. Silent on send failure.
|
||||
|
||||
- [ ] **Step 1: Write the failing test**
|
||||
|
||||
```python
|
||||
# data_only_viz/tests/test_pose_bridge_hands.py
|
||||
from data_only_viz.pose_bridge import PoseSoundBridge
|
||||
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self):
|
||||
self.sent = []
|
||||
|
||||
def send_message(self, addr, args):
|
||||
self.sent.append((addr, args))
|
||||
|
||||
|
||||
def _bridge():
|
||||
b = PoseSoundBridge.__new__(PoseSoundBridge) # bypass __init__/sockets
|
||||
b._client = FakeClient()
|
||||
return b
|
||||
|
||||
|
||||
def test_send_hands_both_present():
|
||||
b = _bridge()
|
||||
feats = {
|
||||
"L": {"cx": 0.2, "cy": 0.4, "openness": 0.9, "speed": 0.1},
|
||||
"R": {"cx": 0.8, "cy": 0.3, "openness": 0.5, "speed": 0.0},
|
||||
"dist": 0.6,
|
||||
}
|
||||
b.send_hands(feats, t=1.0)
|
||||
addr, args = b._client.sent[-1]
|
||||
assert addr == "/pose/hands"
|
||||
assert args[0] == 0
|
||||
assert args[1:5] == [0.2, 0.4, 0.9, 0.1]
|
||||
assert args[5:9] == [0.8, 0.3, 0.5, 0.0]
|
||||
assert args[9] == 0.6
|
||||
assert len(args) == 10
|
||||
|
||||
|
||||
def test_send_hands_left_only_zero_pads_right():
|
||||
b = _bridge()
|
||||
feats = {"L": {"cx": 0.5, "cy": 0.5, "openness": 0.7, "speed": 0.2},
|
||||
"R": None, "dist": 0.0}
|
||||
b.send_hands(feats, t=1.0)
|
||||
_, args = b._client.sent[-1]
|
||||
assert args[5:9] == [0.0, 0.0, 0.0, 0.0]
|
||||
|
||||
|
||||
def test_send_hands_none_emits_all_zero():
|
||||
b = _bridge()
|
||||
b.send_hands({"L": None, "R": None, "dist": 0.0}, t=1.0)
|
||||
_, args = b._client.sent[-1]
|
||||
assert args == [0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live" && uv run python -m pytest data_only_viz/tests/test_pose_bridge_hands.py -v`
|
||||
Expected: FAIL — `AttributeError: 'PoseSoundBridge' object has no attribute 'send_hands'`
|
||||
|
||||
- [ ] **Step 3: Add the method**
|
||||
|
||||
Add to `PoseSoundBridge` in `data_only_viz/pose_bridge.py` (next to `send_kin`):
|
||||
|
||||
```python
|
||||
@staticmethod
|
||||
def _hand_slot(h):
|
||||
if not h:
|
||||
return [0.0, 0.0, 0.0, 0.0]
|
||||
return [float(h["cx"]), float(h["cy"]),
|
||||
float(h["openness"]), float(h["speed"])]
|
||||
|
||||
def send_hands(self, feats, t):
|
||||
"""Emit /pose/hands [0, lx,ly,lopen,lspeed, rx,ry,ropen,rspeed, dist]."""
|
||||
args = [0]
|
||||
args += self._hand_slot(feats.get("L"))
|
||||
args += self._hand_slot(feats.get("R"))
|
||||
args.append(float(feats.get("dist", 0.0)))
|
||||
try:
|
||||
self._client.send_message("/pose/hands", args)
|
||||
except OSError:
|
||||
pass
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live" && uv run python -m pytest data_only_viz/tests/test_pose_bridge_hands.py -v`
|
||||
Expected: PASS (3 passed)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add data_only_viz/pose_bridge.py data_only_viz/tests/test_pose_bridge_hands.py
|
||||
git commit -m "feat(viz): /pose/hands OSC sender"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Wire extraction + send into the per-tick loop
|
||||
|
||||
**Files:**
|
||||
- Modify: `data_only_viz/action_head_pub.py` (around the `send_kin` call, ~line 138)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `HandFeatureExtractor` (Task 1), `PoseSoundBridge.send_hands` (Task 2), and the live `state` object exposing `persons_hands` (and `left_hand_kp`/`right_hand_kp` as single-person fallback).
|
||||
- Produces: a `/pose/hands` message emitted once per tick alongside `/pose/kin`.
|
||||
|
||||
- [ ] **Step 1: Locate the tick and the state handle**
|
||||
|
||||
Read `data_only_viz/action_head_pub.py` around the publisher `__init__` and the
|
||||
`_tick`/`step` method that calls `self.bridge.send_kin(...)` (~line 138).
|
||||
Confirm how the shared `state` snapshot is obtained there (the same object whose
|
||||
`persons_hands` field Task 1 consumes).
|
||||
|
||||
- [ ] **Step 2: Instantiate the extractor in `__init__`**
|
||||
|
||||
Add to the publisher constructor (after `self.bridge = ...`):
|
||||
|
||||
```python
|
||||
from data_only_viz.hand_features import HandFeatureExtractor
|
||||
self._hand_ext = HandFeatureExtractor()
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Extract + send each tick**
|
||||
|
||||
Immediately after the existing `self.bridge.send_kin(pid, kin, t_now, force=True)`
|
||||
call, add (using the same state snapshot variable already in scope — replace
|
||||
`state` below with the local name found in Step 1):
|
||||
|
||||
```python
|
||||
hands = list(getattr(state, "persons_hands", None) or [])
|
||||
if not hands and getattr(state, "hands_present", False):
|
||||
hands = [state.left_hand_kp, state.right_hand_kp]
|
||||
hands = [h for h in hands if h]
|
||||
hand_feats = self._hand_ext.step(hands)
|
||||
self.bridge.send_hands(hand_feats, t_now)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Smoke test the wire**
|
||||
|
||||
In one terminal, listen for the route:
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
uv run python - <<'PY'
|
||||
from pythonosc import dispatcher, osc_server
|
||||
d = dispatcher.Dispatcher()
|
||||
d.map("/pose/hands", lambda a, *args: print("HANDS", args))
|
||||
osc_server.BlockingOSCUDPServer(("127.0.0.1", 57121), d).serve_forever()
|
||||
PY
|
||||
```
|
||||
|
||||
In another, run the pose pipeline as usual (the data_only_viz entry that drives
|
||||
`action_head_pub`) with a hand in frame.
|
||||
Expected: `HANDS (0, lx, ly, lopen, lspeed, rx, ry, ropen, rspeed, dist)` lines
|
||||
streaming; values change when the hand moves/opens. Stop the listener (Ctrl-C)
|
||||
before booting SuperCollider on 57121.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add data_only_viz/action_head_pub.py
|
||||
git commit -m "feat(viz): emit hand features each tick"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: SuperCollider ingestion `~handFeat` + boot fix + docs
|
||||
|
||||
**Files:**
|
||||
- Modify: `sound_algo/control/data_feeds.scd` (next to pose OSCdefs, ~line 464-496)
|
||||
- Modify: `sound_algo/data_only/boot.scd` (load sequence, ~line 133)
|
||||
- Modify: `data_only_viz/README.md` (document the route)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: OSC `/pose/hands` from Task 2.
|
||||
- Produces: global `~handFeat` Dictionary with keys
|
||||
`lx, ly, lopen, lspeed, rx, ry, ropen, rspeed, dist` (Floats), refreshed each
|
||||
message. Read by Task 6.
|
||||
|
||||
- [ ] **Step 1: Add the OSCdef**
|
||||
|
||||
In `sound_algo/control/data_feeds.scd`, after the `\poseLeave` OSCdef block, add:
|
||||
|
||||
```supercollider
|
||||
~handFeat = ~handFeat ? Dictionary.new;
|
||||
|
||||
OSCdef(\poseHands, { |msg|
|
||||
~handFeat[\lx] = msg[2];
|
||||
~handFeat[\ly] = msg[3];
|
||||
~handFeat[\lopen] = msg[4];
|
||||
~handFeat[\lspeed] = msg[5];
|
||||
~handFeat[\rx] = msg[6];
|
||||
~handFeat[\ry] = msg[7];
|
||||
~handFeat[\ropen] = msg[8];
|
||||
~handFeat[\rspeed] = msg[9];
|
||||
~handFeat[\dist] = msg[10];
|
||||
}, '/pose/hands');
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Load `scene_pose_action.scd` at boot**
|
||||
|
||||
In `sound_algo/data_only/boot.scd`, in the load sequence (after the line that
|
||||
loads `scenes.scd`, ~line 133), add a load of the pose/hand mapping file. Match
|
||||
the existing load idiom used a few lines above (same `(thisProcess... .resolveRelative)`
|
||||
or `.loadRelative` form already used for `scenes.scd`); for example:
|
||||
|
||||
```supercollider
|
||||
(thisProcess.nowExecutingPath.dirname +/+ "scene_pose_action.scd").load;
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Validate the `.scd` files**
|
||||
|
||||
Invoke the `validating-scd-files` skill on `sound_algo/control/data_feeds.scd`
|
||||
and `sound_algo/data_only/boot.scd`.
|
||||
Expected: P:0 B:0 for both; boot.scd keeps exactly one top-level block (it is
|
||||
`.load`-ed). Fix any imbalance before continuing.
|
||||
|
||||
- [ ] **Step 4: Document the contract**
|
||||
|
||||
In `data_only_viz/README.md`, add a short "Hand features OSC" subsection:
|
||||
|
||||
```markdown
|
||||
### Hand features OSC
|
||||
|
||||
`/pose/hands [pid=0, lx, ly, lopen, lspeed, rx, ry, ropen, rspeed, dist]`
|
||||
|
||||
- Emitted once per pipeline tick by `pose_bridge.send_hands` to sclang `:57121`.
|
||||
- L = leftmost hand on screen, R = rightmost; absent hand -> its 4 floats are 0.
|
||||
- Ranges: positions/openness/dist in [0,1]; speed >= 0 (normalized units/frame).
|
||||
- Consumed by SuperCollider `~handFeat` (data_only) and the Metal renderer uniforms.
|
||||
- Multi-person is not yet carried (pid is always 0).
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Smoke test ingestion**
|
||||
|
||||
Boot the data_only subsystem (`sclang sound_algo/data_only/boot.scd`), then from
|
||||
a separate shell inject one message and confirm storage:
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
uv run python -c "from pythonosc.udp_client import SimpleUDPClient; \
|
||||
SimpleUDPClient('127.0.0.1',57121).send_message('/pose/hands',[0,0.2,0.4,0.9,0.1,0.8,0.3,0.5,0.0,0.6])"
|
||||
```
|
||||
|
||||
In the sclang console run: `~handFeat[\dist].postln;`
|
||||
Expected: `0.6` printed.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add sound_algo/control/data_feeds.scd sound_algo/data_only/boot.scd data_only_viz/README.md
|
||||
git commit -m "feat(data-only): ingest /pose/hands into handFeat"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: FX rack global filter (cutoff / width / drive)
|
||||
|
||||
**Files:**
|
||||
- Modify: `sound_algo/data_only/engine.scd` (FX rack, ~line 34-59)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the master audio path (bus 0) and the existing FX group `~doMasterGroup`.
|
||||
- Produces: a Synth `~doFilter` placed before `~doMaster` on bus 0 with set-able
|
||||
args `cutoff` (Hz, default 20000 = open), `width` (0..1, default 1 = full
|
||||
stereo), `drive` (1..4, default 1 = unity). All lagged.
|
||||
|
||||
- [ ] **Step 1: Add the SynthDef (use the adding-synthdef skill)**
|
||||
|
||||
Invoke the `adding-synthdef` skill, then add to `sound_algo/data_only/engine.scd`
|
||||
near the other rack SynthDefs:
|
||||
|
||||
```supercollider
|
||||
SynthDef(\do_master_filter, { |out=0, cutoff=20000, width=1.0, drive=1.0|
|
||||
var sig, mid, side; // all vars first (sclang rule)
|
||||
sig = In.ar(out, 2);
|
||||
sig = RLPF.ar(sig, cutoff.clip(80, 20000).lag(0.08), 0.7);
|
||||
sig = (sig * drive.clip(1, 4).lag(0.05)).tanh;
|
||||
// stereo width via mid/side
|
||||
mid = (sig[0] + sig[1]) * 0.5;
|
||||
side = (sig[0] - sig[1]) * 0.5 * width.clip(0, 1).lag(0.08);
|
||||
sig = [mid + side, mid - side];
|
||||
sig = LeakDC.ar(sig);
|
||||
ReplaceOut.ar(out, sig);
|
||||
}).add;
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Instantiate before the master**
|
||||
|
||||
In the rack boot section (where `~doRev` and `~doMaster` are created via
|
||||
`Synth.tail(~doMasterGroup, ...)`, ~line 56-59), add `~doFilter` so its node
|
||||
sits *before* `~doMaster` on bus 0. Insert it immediately ahead of the
|
||||
`~doMaster` creation:
|
||||
|
||||
```supercollider
|
||||
~doFilter = Synth.before(~doMaster, \do_master_filter, [\out, 0]);
|
||||
```
|
||||
|
||||
If `~doMaster` is created after this line, instead create `~doFilter` with
|
||||
`Synth.tail(~doMasterGroup, \do_master_filter, [\out, 0])` *before* the
|
||||
`~doMaster` `Synth.tail` call so node order is filter → master.
|
||||
|
||||
- [ ] **Step 3: Validate the `.scd`**
|
||||
|
||||
Invoke the `validating-scd-files` skill on `sound_algo/data_only/engine.scd`.
|
||||
Expected: P:0 B:0.
|
||||
|
||||
- [ ] **Step 4: Smoke test audibly**
|
||||
|
||||
Boot data_only with a scene playing sound (`~doScene.(\all)`), then in sclang:
|
||||
|
||||
```supercollider
|
||||
~doFilter.set(\cutoff, 400); // should audibly darken
|
||||
~doFilter.set(\cutoff, 20000); // back to open
|
||||
~doFilter.set(\drive, 3); // should audibly saturate
|
||||
~doFilter.set(\drive, 1);
|
||||
~doFilter.set(\width, 0); // should collapse to mono
|
||||
~doFilter.set(\width, 1);
|
||||
```
|
||||
|
||||
Expected: each change is audible; no errors; no runaway level (limiter in
|
||||
`~doMaster` still after the filter).
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add sound_algo/data_only/engine.scd
|
||||
git commit -m "feat(data-only): global filter cutoff width drive"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 6: Map `~handFeat` to the FX rack
|
||||
|
||||
**Files:**
|
||||
- Modify: `sound_algo/data_only/scene_pose_action.scd` (~line 26-83)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: `~handFeat` (Task 4), `~doFilter`/`~doRev` (Task 5 / engine).
|
||||
- Produces: `~mapHandsToFx` (a function) and a clock Routine that calls it at
|
||||
`~paConfig[\rate]` Hz, driving cutoff/reverb/width/drive from hand features.
|
||||
|
||||
Mapping (approved): height `1-cy` → cutoff 200..8000 Hz; `openness` → reverb mix
|
||||
0..0.8; `speed` → drive 1..3; `dist` → width 0..1. Use the higher hand's height
|
||||
and the max openness/speed across the two hands.
|
||||
|
||||
- [ ] **Step 1: Implement the FX stubs against real targets**
|
||||
|
||||
At the top of `sound_algo/data_only/scene_pose_action.scd` (before
|
||||
`~mapPoseToFx`), wire the stubs the existing pose mapping also calls:
|
||||
|
||||
```supercollider
|
||||
~fxCut = ~fxCut ? { |hz| if(~doFilter.notNil) { ~doFilter.set(\cutoff, hz) } };
|
||||
~fxRev = ~fxRev ? { |mix| if(~doRev.notNil) { ~doRev.set(\mix, mix) } };
|
||||
~fxSt = ~fxSt ? { |w| if(~doFilter.notNil) { ~doFilter.set(\width, w) } };
|
||||
~fxDrive = ~fxDrive ? { |amt| if(~doFilter.notNil) { ~doFilter.set(\drive, amt.linlin(0,1,1,3)) } };
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Add `~mapHandsToFx`**
|
||||
|
||||
Add after `~mapPoseToFx`:
|
||||
|
||||
```supercollider
|
||||
~mapHandsToFx = {
|
||||
var hf = ~handFeat;
|
||||
var hasL, hasR, ys, height, openMax, spdMax; // all vars first (sclang rule)
|
||||
if (hf.notNil and: { hf[\ly].notNil }) {
|
||||
hasL = (hf[\lx] + hf[\ly]) > 0.001;
|
||||
hasR = (hf[\rx] + hf[\ry]) > 0.001;
|
||||
if (hasL or: { hasR }) {
|
||||
// higher hand = smaller y; height = 1 - min(y)
|
||||
ys = [];
|
||||
if (hasL) { ys = ys.add(hf[\ly]) };
|
||||
if (hasR) { ys = ys.add(hf[\ry]) };
|
||||
height = 1 - ys.minItem;
|
||||
openMax = max(hf[\lopen] ? 0, hf[\ropen] ? 0);
|
||||
spdMax = max(hf[\lspeed] ? 0, hf[\rspeed] ? 0);
|
||||
~fxCut.(height.linexp(0, 1, 200, 8000));
|
||||
~fxRev.(openMax.linlin(0, 1, 0.1, 0.8));
|
||||
~fxDrive.(spdMax.linlin(0, 0.3, 0, 1));
|
||||
~fxSt.((hf[\dist] ? 0).linlin(0, 1, 0.2, 1.0));
|
||||
};
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
> Note: `~mapPoseToFx` (accel→cutoff) and `~mapHandsToFx` (height→cutoff) both
|
||||
> drive `~fxCut` on the same clock; hands run second so they take precedence —
|
||||
> intended for this slice (hands are the new expressive control).
|
||||
|
||||
- [ ] **Step 3: Drive it on the config clock**
|
||||
|
||||
Find where `~mapPoseToFx` is scheduled (a Routine/clock at `~paConfig[\rate]`).
|
||||
If a scheduling Routine already exists, add `~mapHandsToFx.value;` next to the
|
||||
`~mapPoseToFx.value;` call. If none exists in this file, add one at the end:
|
||||
|
||||
```supercollider
|
||||
~paHandRoutine.notNil.if { ~paHandRoutine.stop };
|
||||
~paHandRoutine = Routine({
|
||||
inf.do {
|
||||
~mapPoseToFx.value;
|
||||
~mapHandsToFx.value;
|
||||
(1 / (~paConfig[\rate] ? 10)).wait;
|
||||
};
|
||||
}).play(AppClock);
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Validate the `.scd`**
|
||||
|
||||
Invoke the `validating-scd-files` skill on
|
||||
`sound_algo/data_only/scene_pose_action.scd`.
|
||||
Expected: P:0 B:0.
|
||||
|
||||
- [ ] **Step 5: Smoke test end-to-end (audio)**
|
||||
|
||||
Boot data_only with sound playing. Inject a high open hand then a low fist:
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
uv run python -c "from pythonosc.udp_client import SimpleUDPClient as C; \
|
||||
C('127.0.0.1',57121).send_message('/pose/hands',[0,0.5,0.1,0.9,0.0,0.0,0.0,0.0,0.0,0.0])"
|
||||
# high + open -> bright + reverby
|
||||
uv run python -c "from pythonosc.udp_client import SimpleUDPClient as C; \
|
||||
C('127.0.0.1',57121).send_message('/pose/hands',[0,0.5,0.9,0.05,0.0,0.0,0.0,0.0,0.0,0.0])"
|
||||
# low + fist -> dark + dry
|
||||
```
|
||||
|
||||
Expected: first message brightens + adds reverb; second darkens + dries. No
|
||||
errors in the sclang console.
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add sound_algo/data_only/scene_pose_action.scd
|
||||
git commit -m "feat(data-only): map hand features to FX"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 7: Metal uniforms — expressive hand features
|
||||
|
||||
**Files:**
|
||||
- Modify: `data_only_viz/renderer.py` (`UNIFORM_FLOATS` ~line 118; `_update_uniforms` ~line 251-299)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the `HandFeatureExtractor.step()` output for the current frame
|
||||
(obtain from the shared state the renderer already locks, or store the latest
|
||||
feature dict on `state`).
|
||||
- Produces: 4 extra uniform floats appended after the existing pad slots:
|
||||
`hand_height`, `hand_openness`, `hand_speed`, `hand_dist`. `UNIFORM_FLOATS`
|
||||
becomes 24 (still 16-byte aligned: 24*4 = 96 bytes).
|
||||
|
||||
- [ ] **Step 1: Make the latest hand features available to the renderer**
|
||||
|
||||
In `action_head_pub.py` (Task 3), after computing `hand_feats`, also publish them
|
||||
onto the shared state so the renderer can read them under its lock:
|
||||
|
||||
```python
|
||||
state.hand_feats = hand_feats
|
||||
```
|
||||
|
||||
And ensure `state.py` has a default. Add to the state dataclass/fields:
|
||||
|
||||
```python
|
||||
hand_feats: dict | None = None
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Bump the uniform count**
|
||||
|
||||
In `data_only_viz/renderer.py`, change the uniform size constants:
|
||||
|
||||
```python
|
||||
UNIFORM_FLOATS = 24 # 20 + hand_height, hand_openness, hand_speed, hand_dist
|
||||
UNIFORM_SIZE = UNIFORM_FLOATS * 4 # 96 bytes, 16-byte aligned
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Compute and pack the features**
|
||||
|
||||
In `_update_uniforms`, after the existing wrist extraction (~line 259-264), derive
|
||||
the four expressive scalars from `state.hand_feats` (guard for None), and append
|
||||
them to the `struct.pack` call (replace the two trailing pad floats with the four
|
||||
new values; keep total = `UNIFORM_FLOATS`):
|
||||
|
||||
```python
|
||||
hf = getattr(s, "hand_feats", None) or {}
|
||||
hl, hr = hf.get("L"), hf.get("R")
|
||||
ys = [h["cy"] for h in (hl, hr) if h]
|
||||
hand_height = (1.0 - min(ys)) if ys else 0.0
|
||||
hand_open = max([h["openness"] for h in (hl, hr) if h] or [0.0])
|
||||
hand_speed = max([h["speed"] for h in (hl, hr) if h] or [0.0])
|
||||
hand_dist = float(hf.get("dist", 0.0))
|
||||
```
|
||||
|
||||
Update the `struct.pack(...)` (currently ending `hlx, hly, hrx, hry, 0.0, 0.0`)
|
||||
to end with:
|
||||
|
||||
```python
|
||||
hlx, hly, hrx, hry, # 15-18 wrist xy (kept)
|
||||
hand_height, hand_open, # 19-20
|
||||
hand_speed, hand_dist, # 21-22
|
||||
0.0, 0.0, # 23-24 pad (16-byte align)
|
||||
```
|
||||
|
||||
(Confirm the final argument count to `struct.pack(f"{UNIFORM_FLOATS}f", ...)`
|
||||
equals 24.)
|
||||
|
||||
- [ ] **Step 4: Verify packing length**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live" && uv run python -c "import struct; print(len(struct.pack('24f', *([0.0]*24))))"`
|
||||
Expected: `96`
|
||||
|
||||
Then launch the renderer briefly with a hand in frame and confirm no crash and
|
||||
no buffer-size assertion (the Metal buffer is allocated from `UNIFORM_SIZE`).
|
||||
Expected: window renders; no `as_buffer` size error.
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add data_only_viz/renderer.py data_only_viz/state.py data_only_viz/action_head_pub.py
|
||||
git commit -m "feat(viz): pack hand features into uniforms"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 8: Metal shader reaction across scene modes
|
||||
|
||||
**Files:**
|
||||
- Modify: `data_only_viz/shaders/scene.metal` (`SceneUniforms` ~line 18-39; mode dispatch ~line 465-476)
|
||||
|
||||
**Interfaces:**
|
||||
- Consumes: the 4 new uniform floats from Task 7.
|
||||
- Produces: a subtle, shared reaction in scene modes 0-7 (brightness/scale ←
|
||||
height, bloom ← openness, turbulence ← speed, zoom ← dist), without adding a
|
||||
new mode.
|
||||
|
||||
- [ ] **Step 1: Extend the uniform struct**
|
||||
|
||||
In `scene.metal`, add the four fields to `SceneUniforms` (after `hand_r_y`,
|
||||
replacing/augmenting the two `_pad` floats so the struct matches the 24-float
|
||||
Python layout — order must match Task 7's pack order: indices 19-22):
|
||||
|
||||
```metal
|
||||
float hand_l_x; // 14 (0-based 14 in struct; matches pack idx 15)
|
||||
float hand_l_y; // 15
|
||||
float hand_r_x; // 16
|
||||
float hand_r_y; // 17
|
||||
float hand_height; // 18
|
||||
float hand_openness;// 19
|
||||
float hand_speed; // 20
|
||||
float hand_dist; // 21
|
||||
float _pad0; // 22
|
||||
float _pad1; // 23
|
||||
};
|
||||
```
|
||||
|
||||
(The struct field order must equal the `struct.pack` argument order. Verify
|
||||
against Task 7 Step 3.)
|
||||
|
||||
- [ ] **Step 2: Apply a shared, subtle reaction**
|
||||
|
||||
In the fragment dispatcher where modes 0-7 produce `color` (near line 465-476,
|
||||
where `alpha = mix(0.55, 0.25, U.pose_alive)` is applied), fold in the hand
|
||||
features multiplicatively so the active scene visibly reacts:
|
||||
|
||||
```metal
|
||||
// hand-driven expressive modulation (subtle, all modes)
|
||||
float hb = 1.0 + U.hand_height * 0.6; // brightness/scale
|
||||
float bloom = U.hand_openness * 0.5; // dispersion
|
||||
float turb = U.hand_speed * 2.0; // motion energy
|
||||
color.rgb *= hb;
|
||||
color.rgb += bloom * color.rgb; // simple bloom lift
|
||||
color.rgb += turb * 0.05 * sin(U.time * 8.0 + p.x * 10.0);
|
||||
```
|
||||
|
||||
(Use the existing fragment-local variable names for `color` and the normalized
|
||||
coordinate; adjust `p.x` to whatever the file names the uv/coordinate var. Keep
|
||||
`hand_dist` available for a zoom term in modes that compute a camera/uv scale —
|
||||
apply `uv *= 1.0 - U.hand_dist * 0.2;` where the mode samples space, if trivial;
|
||||
otherwise leave `hand_dist` consumed only by the audio side for now.)
|
||||
|
||||
- [ ] **Step 3: Visual smoke test**
|
||||
|
||||
Run the renderer with the data_only pipeline. Inject hand messages (as in Task 6
|
||||
Step 5) or move a real hand:
|
||||
- raise hand → scene brightens;
|
||||
- open hand → scene blooms/disperses;
|
||||
- move hand fast → visible turbulence.
|
||||
|
||||
Expected: the active mode (not just mode 8) reacts; no shader compile error in
|
||||
the console.
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add data_only_viz/shaders/scene.metal
|
||||
git commit -m "feat(viz): scene reacts to hand features"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 9: Full-loop verification
|
||||
|
||||
**Files:** none (verification only)
|
||||
|
||||
- [ ] **Step 1: Run the Python test suite**
|
||||
|
||||
Run: `cd "/Users/electron/Documents/Projets/AV-Live" && uv run python -m pytest data_only_viz/tests/test_hand_features.py data_only_viz/tests/test_pose_bridge_hands.py -v`
|
||||
Expected: all PASS.
|
||||
|
||||
- [ ] **Step 2: Boot the full chain**
|
||||
|
||||
Boot data_only SuperCollider (`sclang sound_algo/data_only/boot.scd`) and the
|
||||
data_only_viz pipeline + renderer. Perform with both hands in frame.
|
||||
Expected: audio (cutoff/reverb/width/drive) and the Metal scene both react to the
|
||||
same gestures, frame-aligned; raising/opening/spreading hands reads in sound and
|
||||
image together.
|
||||
|
||||
- [ ] **Step 3: Graceful degradation check**
|
||||
|
||||
Kill SuperCollider while the pipeline runs.
|
||||
Expected: the Python pipeline keeps running, the renderer still reacts to hands,
|
||||
no traceback from `send_hands` (silent on `OSError`).
|
||||
|
||||
- [ ] **Step 4: Final commit (if any cleanup)**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git status # expect clean; commit only if verification surfaced a fix
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review notes
|
||||
|
||||
- **Spec coverage:** §3 architecture → Tasks 1/3/7; §4.1 extractor → Task 1;
|
||||
§4.2 OSC contract → Tasks 2/4; §4.3 mapping + stubs → Task 6; §4.4 FX rack →
|
||||
Task 5; §4.5 Metal → Tasks 7/8; §5 reliability (boot fix, finite guards, OSC
|
||||
robustness, docs) → Tasks 1/2/4; §7 testing → Tasks 1/2 + smokes + Task 9.
|
||||
- **Deferred (spec §8):** extractor home = new file `hand_features.py` (chosen);
|
||||
`~mapHandsToFx` = new closure (chosen); filter = `RLPF` + mid/side width
|
||||
(chosen); ring buffer = 5 frames (chosen).
|
||||
- **v1 simplification vs spec:** single global `pid=0` message, L/R by screen x
|
||||
(documented in Global Constraints and Task 4 docs); multi-person deferred.
|
||||
@@ -0,0 +1,221 @@
|
||||
# iPhone Compute-Budget Overlay Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Show ARKit fps, Vision latency, Vision drop rate, and thermal state on the iPhone screen so we can measure how much compute headroom the device has.
|
||||
|
||||
**Architecture:** `ARBodySession` (already owns the frame loop + Vision worker) computes four `@Published` metrics; `ContentView`'s control panel shows them in one line, color-coded by thermal state.
|
||||
|
||||
**Tech Stack:** Swift, SwiftUI, ARKit, Apple Vision, Foundation (`ProcessInfo.thermalState`).
|
||||
|
||||
## Global Constraints
|
||||
|
||||
- iPhone app `iphone-arbody/` (ARBodyTracker). Build via `xcodebuild` for `generic/platform=iOS`, `-allowProvisioningUpdates`. Source lives in the `.swiftpm/Sources` dir, built through `ARBodyTracker.xcodeproj`.
|
||||
- `ARBodySession` is `@MainActor`. Metrics computed off-main (in `runVision`) hop back to `@MainActor` for the `@Published` write (mirror the existing `usb.send` hop).
|
||||
- Vision must stay off-main + drop-if-busy; the metrics must NOT change capture behavior, only observe it.
|
||||
- No emojis. Commit subject ≤ 50 chars, no AI attribution, no `--no-verify`, no underscore in commit scope.
|
||||
- Build is the automated gate; the real deliverable is the on-device reading (operator).
|
||||
- Branch: `main` (trunk-based).
|
||||
- SourceKit single-file diagnostics are noise; the `xcodebuild` result is the source of truth.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
| File | Action | Responsibility |
|
||||
|------|--------|----------------|
|
||||
| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift` | Modify | compute + publish arkitFps / visionMs / visionDropRate / thermalLabel |
|
||||
| `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift` | Modify | metrics line in the control panel, thermal color-coded |
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Budget metrics + overlay
|
||||
|
||||
**Files:**
|
||||
- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift`
|
||||
- Modify: `iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift`
|
||||
|
||||
**Interfaces:**
|
||||
- Produces on `ARBodySession`: `@Published var arkitFps: Double`, `@Published var visionMs: Double`, `@Published var visionDropRate: Double`, `@Published var thermalLabel: String`, and `nonisolated static func thermalString(_:) -> String`.
|
||||
|
||||
- [ ] **Step 1: Add the published metrics + counters + thermal observer to ARBodySession**
|
||||
|
||||
Add near the other `@Published` properties:
|
||||
```swift
|
||||
@Published var arkitFps: Double = 0
|
||||
@Published var visionMs: Double = 0
|
||||
@Published var visionDropRate: Double = 0
|
||||
@Published var thermalLabel: String = "nominal"
|
||||
```
|
||||
Add private counters near `lastVisionTime`:
|
||||
```swift
|
||||
private var visionAttempts: Int = 0
|
||||
private var visionDrops: Int = 0
|
||||
```
|
||||
Add the thermal helper (place after the properties, before `start()`):
|
||||
```swift
|
||||
nonisolated static func thermalString(
|
||||
_ s: ProcessInfo.ThermalState) -> String {
|
||||
switch s {
|
||||
case .nominal: return "nominal"
|
||||
case .fair: return "fair"
|
||||
case .serious: return "serious"
|
||||
case .critical: return "critical"
|
||||
@unknown default: return "?"
|
||||
}
|
||||
}
|
||||
```
|
||||
In `init()` (after the existing `videoEncoder.onPayload` setup), seed + observe thermal:
|
||||
```swift
|
||||
thermalLabel = Self.thermalString(
|
||||
ProcessInfo.processInfo.thermalState)
|
||||
NotificationCenter.default.addObserver(
|
||||
forName: ProcessInfo.thermalStateDidChangeNotification,
|
||||
object: nil, queue: .main) { [weak self] _ in
|
||||
Task { @MainActor in
|
||||
self?.thermalLabel = Self.thermalString(
|
||||
ProcessInfo.processInfo.thermalState)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Compute ARKit fps in the frame handler**
|
||||
|
||||
In `session(_:didUpdate:)`, inside the `Task { @MainActor in ... }` block, the throttle currently reads:
|
||||
```swift
|
||||
if t - self.lastFrameTime < 1.0 / 30.0 { return }
|
||||
self.lastFrameTime = t
|
||||
```
|
||||
Replace it with a version that also derives fps from the inter-frame delta:
|
||||
```swift
|
||||
let dt = t - self.lastFrameTime
|
||||
if dt < 1.0 / 30.0 { return }
|
||||
self.lastFrameTime = t
|
||||
if dt > 0, dt < 1.0 {
|
||||
let inst = 1.0 / dt
|
||||
self.arkitFps = self.arkitFps == 0
|
||||
? inst : self.arkitFps * 0.9 + inst * 0.1
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Count Vision attempts/drops in the throttled dispatch**
|
||||
|
||||
The Phase-0 Vision dispatch currently looks like:
|
||||
```swift
|
||||
if t - self.lastVisionTime >= 1.0 / 15.0, !self.visionBusy {
|
||||
self.lastVisionTime = t
|
||||
self.visionBusy = true
|
||||
let buf = img
|
||||
self.visionQueue.async {
|
||||
self.runVision(on: buf, timestamp: t)
|
||||
Task { @MainActor in self.visionBusy = false }
|
||||
}
|
||||
}
|
||||
```
|
||||
Replace it with a version that counts attempts and drops over a 60-sample
|
||||
rolling window:
|
||||
```swift
|
||||
if t - self.lastVisionTime >= 1.0 / 15.0 {
|
||||
self.visionAttempts += 1
|
||||
if self.visionBusy {
|
||||
self.visionDrops += 1
|
||||
} else {
|
||||
self.lastVisionTime = t
|
||||
self.visionBusy = true
|
||||
let buf = img
|
||||
self.visionQueue.async {
|
||||
self.runVision(on: buf, timestamp: t)
|
||||
Task { @MainActor in self.visionBusy = false }
|
||||
}
|
||||
}
|
||||
if self.visionAttempts >= 60 {
|
||||
self.visionDropRate =
|
||||
Double(self.visionDrops) / Double(self.visionAttempts)
|
||||
self.visionAttempts = 0
|
||||
self.visionDrops = 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Time the Vision pass in runVision**
|
||||
|
||||
In `runVision(on:timestamp:)`, wrap the `handler.perform(...)` with timing and publish an EMA. Replace:
|
||||
```swift
|
||||
let handler = VNImageRequestHandler(
|
||||
cvPixelBuffer: pixelBuffer, orientation: .right, options: [:])
|
||||
try? handler.perform([handReq, faceReq])
|
||||
```
|
||||
with:
|
||||
```swift
|
||||
let handler = VNImageRequestHandler(
|
||||
cvPixelBuffer: pixelBuffer, orientation: .right, options: [:])
|
||||
let t0 = Date()
|
||||
try? handler.perform([handReq, faceReq])
|
||||
let ms = Date().timeIntervalSince(t0) * 1000.0
|
||||
Task { @MainActor in
|
||||
self.visionMs = self.visionMs == 0
|
||||
? ms : self.visionMs * 0.8 + ms * 0.2
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Show the metrics line in ContentView**
|
||||
|
||||
In `ContentView.swift` `controlPanel`, after the existing `HStack` that shows
|
||||
`"bodies: ... frames: ... j/s: ..."`, add a metrics line:
|
||||
```swift
|
||||
Text("fps \(String(format: "%.1f", session.arkitFps)) "
|
||||
+ "vis \(Int(session.visionMs)) ms "
|
||||
+ "drop \(Int(session.visionDropRate * 100))% "
|
||||
+ session.thermalLabel)
|
||||
.font(.caption2)
|
||||
.foregroundColor(thermalColor)
|
||||
```
|
||||
Add a computed color property to `ContentView` (near `usbDotColor`):
|
||||
```swift
|
||||
private var thermalColor: Color {
|
||||
switch session.thermalLabel {
|
||||
case "serious": return .orange
|
||||
case "critical": return .red
|
||||
default: return .green
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Build to verify it compiles**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live/iphone-arbody" && \
|
||||
DD="/private/tmp/claude-501/-Users-electron-Documents-Projets-AV-Live/8a95bb35-b732-48ee-a725-0e61d6b41061/scratchpad/arbody-dd" && \
|
||||
xcodebuild -project ARBodyTracker.xcodeproj -scheme ARBodyTracker -configuration Debug \
|
||||
-destination 'generic/platform=iOS' -allowProvisioningUpdates -derivedDataPath "$DD" build 2>&1 \
|
||||
| grep -E "error:|BUILD (SUCCEEDED|FAILED)"
|
||||
```
|
||||
Expected: `** BUILD SUCCEEDED **`, no `error:` lines.
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
cd "/Users/electron/Documents/Projets/AV-Live"
|
||||
git add iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ARBodySession.swift \
|
||||
iphone-arbody/ARBodyTracker.swiftpm/Sources/ARBodyTracker/ContentView.swift
|
||||
git commit -m "feat(ios): compute budget metrics overlay"
|
||||
```
|
||||
|
||||
- [ ] **Step 8: On-device read (operator)**
|
||||
|
||||
Deploy to the iPhone (build → `xcrun devicectl device install/launch`, device
|
||||
`00008140-000845660163001C`). Tap Start, stand in frame with hands + face
|
||||
visible ~1 min, and read the overlay line: `fps`, `vis ms`, `drop %`,
|
||||
`thermal`. Report these numbers — they decide how far to push A (derive uniforms
|
||||
on-device) and B (mesh on-device). This reading is the deliverable.
|
||||
|
||||
---
|
||||
|
||||
## Self-Review notes
|
||||
|
||||
- **Spec coverage:** §4.1 metrics (arkitFps/visionMs/visionDropRate/thermalLabel) → Steps 1-4; §4.2 overlay → Step 5; §5 build gate → Step 6, on-device read → Step 8.
|
||||
- **Placeholder scan:** none; all code is concrete.
|
||||
- **Type consistency:** `arkitFps/visionMs/visionDropRate/thermalLabel` (Double/Double/Double/String) used identically in ARBodySession (Steps 1-4) and ContentView (Step 5). `thermalString` switch covers all `ProcessInfo.ThermalState` cases.
|
||||
- **Behavior safety:** the Vision dispatch keeps drop-if-busy (only counts added); fps/drop counters mutate on `@MainActor`; `visionMs` write hops to `@MainActor`. Capture behavior unchanged.
|
||||
- **Out of scope:** sending metrics to the Mac; A/B work (gated on these numbers).
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user