diff --git a/README.md b/README.md index 81ad0d6..b0c24c8 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ > **Live coding audio-visual performance system** built around a SuperCollider sound engine, an openFrameworks oscilloscope visualizer driven by a real Hantek 6022BL USB scope, a macOS menubar launcher, and a Metal-native pose / body-mesh visualizer that listens to the same audio bus. > -> 15 scripted demoparties · ~47 fullscreen visuals (26 3D parametric meshes + 18 procedural shaders + 3 dedicated C++ scenes) · 5 OS pixel-art shaders · 14 retro OS logos · 33 GLSL shader pairs (~65 files) · 1099 SynthDefs across 368 tracks (23 albums × 16) · 11 real-world data feeds · 7 pose-estimation backends — all reactive to the audio physically passing through the scope probes. +> 15 scripted demoparties · ~47 fullscreen visuals (26 3D parametric meshes + 18 procedural shaders + 3 dedicated C++ scenes) · 5 OS pixel-art shaders · 14 retro OS logos · 33 GLSL shader pairs (~65 files) · 1099 SynthDefs across 368 tracks (23 albums × 16) · **20 real-world data feeds** · 7 pose-estimation backends · **SMPL-X body mesh** (10 475 vertices via Multi-HMR + RealityKit) · **3 launch modes** (Full / Data-only / Body Mesh) — all reactive to the audio physically passing through the scope probes. ## Top-level architecture @@ -136,9 +136,15 @@ Each act lasts 12–55 s, transitions between acts trigger flash + glitch postfx - `Espace` — manual glitch pulse - `F1`–`F5` — fullscreen / GUI / postFx / autoglitch / reload shaders -### 🌐 Real-world data feeds — `data_feeds/` + `web_realart/` -- **11 feed modules** in `data_feeds/feeds/` ingested by `bridge.py` and broadcast as OSC `/data//` to SC (`:57121`), oF (`:57123`) and the web bridge (`:57124`) : - USGS quakes · NOAA SWPC (solar wind, Bz IMF, Kp, X-ray flares) · Mainsfrequenz.de · RTE eCO2mix · Blitzortung lightning · OpenSky ADS-B · Bluesky firehose · Bitcoin mempool · GitHub events · GCN astrophysics · YOLOv8 webcam pose +### 🌐 Real-world data feeds — `data_feeds/` + `web_realart/` + `data_only_viz/web/` +- **20 feed modules** in `data_feeds/feeds/` ingested by `bridge.py` and broadcast as OSC `/data//` to SC (`:57121`), oF (`:57123`) and the web bridge (`:57124`) : + - **Geophysique** : USGS quakes · Smithsonian GVP volcanoes · Blitzortung lightning + - **Meteo / Air** : Open-Meteo (temp/wind/rain) · OpenAQ (PM2.5/PM10/NO2/O3) + - **Espace** : NOAA SWPC (solar wind, Bz IMF, Kp, X-ray) · ISS position (wheretheiss.at) · GCN astrophysics + - **Mobilite** : OpenSky ADS-B · LiveATC listeners + - **Energie / reseau** : Mainsfrequenz.de · RTE eCO2mix · NOAA tides + moon phase + - **Social / numerique** : Bluesky firehose · Reddit /r/all + HackerNews top · Wikipedia EventStream · GDELT 2.0 events · GitHub · Bitcoin mempool + - **Pose / webcam** : YOLOv8-pose - **SC presets** `sound_algo/examples/16_data_feeds.scd` and `17_data_feeds_more.scd` map each source to synthesis : Schumann cavity drone (foudre), aurora additive pad (Bz/wind/Kp), Netzfrequenz pulse kick, RTE 8-op carbon FM, OpenSky granular swarm - **Web standalone** `web_realart/` ports the visualizers and synths to the browser for `real.art.saillant.cc` : - **WebGPU + three.js TSL** (r171) globe with quake/strike/flight particles (auto-fallback WebGL2) @@ -146,6 +152,12 @@ Each act lasts 12–55 s, transitions between acts trigger flash + glitch postfx - **Hydra** with 7 data-driven patches (aurora, quake, lightning, flightmap, gridpulse, solarwind, bskyrain) - All three layers share `feeds_client.js` (`window.feeds`) over one WebSocket +- **Web data-only** `data_only_viz/web/` (port `:3211`, Express + WebSocket) — **bidirectional** OSC bridge for the Data-only mode : + - **`/dashboard.html`** — live cards + SVG sparklines for all 19 active feeds (USGS, GDELT, Wiki, ISS, tides + moon, ATC, ...) with severity classes (alert / warn / green) + - **`/map.html`** — Leaflet dark fullscreen with markers ephemeres for geocoded feeds (quakes, lightning, planes, volcanoes, GDELT events) + persistent ISS marker + - **`/control.html`** — 3-pane control surface : 7 synth/mix sliders (master, cutoff, reso, reverb, delay, tempo), 10 audio scene buttons (`/scene/play`), 9 visual mode buttons routed to oF (`/control/vizMode` → `:57123`), XY pad (`/xy/{x,y}`) + - **SC retour** : `sound_algo/control/web_bridge.scd` listens on `:57121` for `/control/*` `/scene/*` `/xy/*` and pushes `/sync/bpm|beat|rms|voices` to web on `:57125` at 4 Hz + ### 📡 Audio reactivity pipeline ASCII : @@ -231,7 +243,13 @@ uv run python -m data_only_viz.main open AV-Live/launcher/build/AVLiveLauncher.app ``` -The launcher autostarts `sclang` + `oscope-of` + `node` (web UI) and can spawn `data_only_viz` from the menubar mode picker. Open the menubar icon for status, logs, restart controls. +The launcher autostarts `sclang` + the relevant visualizers + the web bridges depending on the **3 launch modes** offered by the picker at startup : + +- **Full AV-Live** — sclang + oscope-of + sound_algo web UI (`:3000`) + data_feeds (full profile) +- **Data-only** — sclang + `data_only_viz` Metal viz + data_feeds (data-only profile) + sound_algo web UI + **data-only dashboard** (`:3211` with `/dashboard.html`, `/map.html`, `/control.html`) +- **Body Mesh** — sclang + `data_only_viz` (headless, Multi-HMR worker only) + **`AVLiveBody`** Swift app (RealityKit, SMPL-X mesh + webcam overlay + live `RenderSettings` panel `S`) + data_feeds + dashboard data-only + +Open the menubar icon for per-process Start/Stop, logs, and the "AV-Live-Body" launch button available from any mode. ### Without Hantek hardware @@ -252,7 +270,10 @@ The `0` GREETINGS demo cycles through 18 acts including Boing Ball, plasma C64, | Visualizer | `oscope-of/` | openFrameworks 0.12, GLSL 150 GL 3.2 core, ~6k LOC C++ | | Pose / body mesh | `data_only_viz/` | Python 3.11+ / `uv`, Metal natif via pyobjc, 7 pose backends | | Launcher | `launcher/` | SwiftUI universal app, sentinel-based restart, ProcessManager `@MainActor` | -| Real-world feeds | `data_feeds/` | Python `uv`, 11 async feed modules, `bridge.py` OSC fan-out | +| Real-world feeds | `data_feeds/` | Python `uv`, 20 async feed modules, `bridge.py` OSC fan-out (3 targets) | +| Data-only web | `data_only_viz/web/` | Node Express `:3211` + WS bidir bridge OSC, dashboard / map / control | +| Body mesh app | `launcher/AV-Live-Body/` | Swift 6 + RealityKit, SMPL-X 10 475 verts, LowLevelMesh + wireframe | +| Multi-HMR worker | `data_only_viz/multi_hmr_worker.py` | PyTorch MPS, AVCaptureSession native, TCP sender :57130 | | Web standalone | `web_realart/` | Node Express + ws, WebGPU three.js r171 + Web Audio + Hydra | | Audio FFT | `oscope-of/src/AudioAnalyzer.{h,cpp}` | downsample + Hann + FFT 2048 | | 3D models | `oscope-of/bin/data/models/` | 27 PLY parametric meshes | diff --git a/data_feeds/README.md b/data_feeds/README.md index 04604dc..6854b87 100644 --- a/data_feeds/README.md +++ b/data_feeds/README.md @@ -1,10 +1,12 @@ # data_feeds — Pont flux temps réel → OSC -Worker Python asynchrone qui aspire des sources publiques (sismique, -géophysique, réseau électrique, foudre, aviation, social, blockchain…) -et les rebalance en OSC vers SuperCollider (`:57121`) et openFrameworks -(`:57123`). Le but : nourrir l'engine audio et le visualizer avec des -**signaux du monde réel**, sans bricoler du networking dans `sclang`. +Worker Python asynchrone qui aspire **20 sources publiques** (sismique, +geophysique, meteo, qualite de l'air, espace, energie, foudre, aviation, +social, blockchain, evenements monde…) et les rebalance en OSC vers +SuperCollider (`:57121`), openFrameworks (`:57123`) et le dashboard web +data-only (`:57124`). Le but : nourrir l'engine audio et le visualizer +avec des **signaux du monde reel**, sans bricoler du networking dans +`sclang`. ## Architecture @@ -72,16 +74,26 @@ complet. | `opensky` | `count`, `plane` | 15 s | | `bluesky` | `post`, `rate` | event-based | | `mempool` | `tx`, `block` | event-based | -| `rte_eco2mix` | `mix` | 15 min | -| `github` | `event` | 30 s | +| `rte_eco2mix` | `mix`, `co2` | 15 min | +| `github` | `event`, `rate` | 30 s | | `gcn` | `alert` | rare | | `pose` | `count`, `person`, `skel`, `bone` | ~20 fps | +| `openmeteo` | `now` (temp, hum, wind, press, rain) | 10 min | +| `openaq` | `now` (PM2.5, PM10, NO2, O3) | 15 min | +| `iss` | `pos` (lat, lon, alt, vel), `pass` | 5 s | +| `volcano` | `active`, `eruption` | 1 h | +| `social_buzz` | `reddit`, `hn`, `pulse` | 1 min | +| `gdelt` | `batch`, `event` (lat, lon, tone, country) | 15 min | +| `wikimedia` | `edit`, `rate` | streaming | +| `tides` | `level` (obs/pred/residual), `moon` | 6 min | +| `atc` | `hub` (icao, listeners), `total` | 5 min | ## Configuration Éditer `config.toml` : -- `osc.targets` : liste `{host, port}` à arroser (par défaut SC + oF). +- `osc.targets` : liste `{host, port}` à arroser. Profil data-only + par défaut : SC `:57121` + oF `:57123` + web data-only `:57124`. - `feeds..enabled` : booléen. - `feeds..poll_seconds` : période pour les feeds HTTP. - `feeds.opensky.bbox` : `[lamin, lomin, lamax, lomax]` (Lyon par défaut). diff --git a/data_only_viz/web/README.md b/data_only_viz/web/README.md new file mode 100644 index 0000000..6fa15e6 --- /dev/null +++ b/data_only_viz/web/README.md @@ -0,0 +1,132 @@ +# data_only_viz/web — Dashboard data-only + +Pont **bidirectionnel** Express + WebSocket pour le mode Data-only +(et Body Mesh) d'AV-Live. Sert 3 pages temps reel et fait le bridge +entre : + +- les 20 feeds Python (`data_feeds/bridge.py`) sur OSC `:57124` +- le navigateur via WebSocket +- SuperCollider (`sclang`) sur OSC `:57121` (out) et `:57125` (sync in) +- openFrameworks (`oscope-of`) sur OSC `:57123` pour les `vizMode` + +## Architecture + +```text + data_feeds bridge.py + | OSC :57124 + v + server.js (Express :3211 + WS) <----- SC sync :57125 (/sync/*) + | + +---> dashboard.html (cards + sparklines SVG vanilla) + +---> map.html (Leaflet dark + markers temps reel) + +---> control.html (sliders + scenes + viz modes + XY pad) + | + v WS -> OSC out + :57121 SC (/control/* /scene/play /xy/*) + :57123 oF (/control/vizMode) +``` + +## Demarrage + +```sh +cd data_only_viz/web +npm install # une fois +HTTP_PORT=3211 node server.js +# ou via le launcher : mode dataOnly ou bodyMesh -> "Dashboard data-only" +``` + +URLs : + +- `http://127.0.0.1:3211/dashboard.html` +- `http://127.0.0.1:3211/map.html` +- `http://127.0.0.1:3211/control.html` + +## Pages + +### `/dashboard.html` + +Grille responsive de 18 cards live, une par source de donnees. +Chaque card affiche : + +- valeur courante (numerique formate) +- ligne sub (contexte : unite, timestamp, info secondaire) +- sparkline SVG vanilla 100x30, dernieres 64-128 valeurs +- classe `.alert` / `.warn` / `.green` selon seuils (ex: Kp >= 6 + alert, PM2.5 > 35 alert, tone GDELT < -5 alert) + +Cards couvertes : usgs, swpc (kp/wind/bz/xray), blitzortung, +opensky, bluesky, openmeteo, openaq, iss (pos + pass), +volcano, social_buzz (reddit/hn/pulse), netzfrequenz, gdelt, +wikimedia (rate + edits), tides + moon, atc, pose, mempool, +github, rte_eco2mix (mix + co2). + +### `/map.html` + +Leaflet dark fullscreen (CartoDB) avec : + +- markers ephemeres (TTL 60 s) : seismes (pink), foudre (jaune), + avions (vert), volcans (orange), GDELT events (purple) +- 1 marker ISS persistant (bleu, contour blanc) qui suit la + position temps reel +- HUD coin haut-gauche : statut connexion + compteur d'events + + liens vers dashboard et control + +### `/control.html` + +Surface de controle 3 colonnes : + +| Colonne | Contenu | Endpoint OSC | +|---------|---------|--------------| +| Synthes & mix | 7 sliders : master_gain, cutoff, reso, reverb_mix, delay_time, delay_feedback, tempo | `/control/` -> SC :57121 | +| Scenes audio + retour SC | 10 boutons scenes (cavity/geo/body/weather/flight/pulse/quiet/all/full/stop) + labels BPM/Beat/RMS/Voies retournes par SC | `/scene/play ` -> SC :57121 | +| Modes visuels + XY pad | 9 boutons (storm/tunnel/plasma/kaleido/voronoi/metaballs/stars/bars/hands3d) + XY pad pour FX | `/control/vizMode ` -> oF :57123
`/xy/{x,y} ` -> SC :57121 | + +Les sliders envoient sur drag (input event). Les boutons scenes +et viz toggle visuellement actif. Le XY pad envoie sur +pointerdown/pointermove (Y inverse pour avoir up=1). + +## Variables d'environnement + +```sh +HTTP_PORT=3211 # port HTTP + WS +OSC_DATA_IN=57124 # feeds IN +OSC_SYNC_IN=57125 # SC sync IN +SC_HOST=127.0.0.1 +SC_PORT_OUT=57121 # /control/* /scene/* /xy/* +OF_PORT_OUT=57123 # /control/vizMode +``` + +## Protocole WebSocket + +Tous les messages WS sont du JSON : + +```js +// server -> browser (feed) : +{ t: 1715600000000, kind: "feed", feed: "usgs", sub: "event", + args: [lat, lon, mag] } + +// server -> browser (SC sync) : +{ t: 1715600000000, kind: "sync", sub: "bpm", args: [120.0] } + +// browser -> server (control) : +{ path: "/control/master_gain", args: [0.8] } +{ path: "/scene/play", args: ["body"] } +{ path: "/xy/x", args: [0.42] } +{ path: "/control/vizMode", args: [3] } +``` + +Le serveur replay les ~100 derniers events a chaque nouvelle +connexion WS (ring buffer 500 entrees). + +## Cote SC : `sound_algo/control/web_bridge.scd` + +Charge automatiquement par `boot.data-only.scd` apres +`data_feeds.scd`. Installe les OSCdef `/control/*` `/scene/play` +`/xy/*` sur :57121, et pousse `/sync/bpm|beat|rms|voices` vers +`NetAddr("127.0.0.1", 57125)` a 4 Hz via une `Routine` `AppClock`. + +Variable d'env `~webState` partagee pour debug REPL : + +```supercollider +~webState.postln; // -> ( masterGain: 0.8, cutoff: 1200, ... ) +``` diff --git a/launcher/README.md b/launcher/README.md index 074cc58..9e025fa 100644 --- a/launcher/README.md +++ b/launcher/README.md @@ -1,44 +1,96 @@ # AVLiveLauncher -macOS menubar launcher for AV-Live. Starts and stops `sclang` (which -auto-loads `sound_algo/00_load.scd` and serves the web UI), the -`oscope-of` visualizer, and the `data_feeds` Python bridge (USGS, -SWPC, grid frequency, lightning, pose, etc. → OSC) with one click -each, and aggregates their logs in one window. +macOS menubar launcher for AV-Live. Coordonne le lancement / arret de +toutes les briques (sclang, oscope-of, data_only_viz, data_feeds, web +UI, dashboard data-only, AV-Live-Body RealityKit) selon le **mode** +choisi au demarrage. ## Build -Requires Swift 5.7+ and Xcode Command Line Tools. +Requires Swift 6.0+ and Xcode Command Line Tools. ```sh cd launcher -./build.sh +./build.sh # universal arm64+x86_64 open build/AVLiveLauncher.app + +# Build AV-Live-Body (RealityKit, app SwiftPM independante) +cd AV-Live-Body +swift build -c release ``` -The first run shows a `waveform.path.ecg` icon in the menubar. Click it -to open the popover. +Le premier lancement affiche un `waveform.path.ecg` dans la menubar. +Clic = popover. -## Configuration +## Les 3 modes (picker au demarrage) -Three paths are stored in `UserDefaults` (`~/Library/Preferences/cc.saillant.AVLiveLauncher.plist`) : +| Mode | Process lances | Use case | +|------|---------------|----------| +| **Full AV-Live** | SC + oscope-of + web UI `:3000` + data_feeds | Studio complet avec Hantek + Hydra | +| **Data-only** | SC + Metal Viz Python + data_feeds + web UI + dashboard `:3211` | Performance autonome sans Hantek, scenes data-driven | +| **Body Mesh** | SC + Multi-HMR headless + AVLiveBody (RealityKit) + data_feeds + dashboard | Mocap performatif, mesh SMPL-X 10 475 verts overlay cam | -| Default | Override | +Le picker peut etre desactive (`Ne plus afficher`) ; il revient au +dernier mode utilise. Le mode peut etre change a tout moment depuis le +menubar — `applyModeTransition` arrete proprement les process +incompatibles (ex: oF coupe en `.dataOnly`). + +## AV-Live-Body — panneau de reglages live + +L'app SwiftPM `launcher/AV-Live-Body/` ouvre une fenetre RealityKit +qui affiche la webcam built-in + le mesh SMPL-X recu en TCP `:57130`. +Touche **`S`** ou le bouton coin haut-droit ouvre un panel a 5 +sections : + +| Section | Reglages | |---------|----------| +| Couches | toggles webcam / mesh / fil de fer / squelette | +| Webcam | opacite 0..1 | +| Maillage | metallique, rugosite | +| Lumieres | key / fill / rim (0..10 000) | +| Vue | FOV (20..120°), luminosite du fond | + +Tous appliques en live dans `BodyView.updateNSView`. L'app est +lancable depuis n'importe quel mode via la row "AV-Live-Body" du +menubar. + +## Configuration des chemins + +Paths persistes dans `UserDefaults` (`~/Library/Preferences/cc.saillant.AVLiveLauncher.plist`) : + +| Defaut | Override | +|--------|----------| | `/Applications/SuperCollider.app/Contents/MacOS/sclang` | Paths… → sclang binary | -| `~/Documents/Projets/AV-Live/sound_algo/00_load.scd` | Paths… → load file | -| `~/Documents/Projets/AV-Live/oscope-of/bin/oscope-of` | Paths… → oscope binary | +| `~/Documents/Projets/AV-Live/sound_algo/boot.scd` | Paths… → load file | +| `~/Documents/Projets/AV-Live/oscope-of/bin/oscope-of.app/.../oscope-of` | Paths… → oscope binary | | `/opt/homebrew/bin/uv` (or `~/.local/bin/uv`) | Paths… → uv binary | | `~/Documents/Projets/AV-Live/data_feeds` | Paths… → data_feeds directory | +| `~/Documents/Projets/AV-Live/data_only_viz` | Paths… → data_only_viz dir | -## What it does +## Process spawnes — table de routage -- Spawns `sclang ` with stdout/stderr captured into the log - window. `sclang` itself boots `scsynth`, loads the 1099-SynthDef - palette, and serves the web bridge on `:8080`. -- Spawns the `oscope-of` binary with cwd set to its parent directory - (so oF can find `bin/data/`). -- Spawns `uv run python bridge.py -v` in `data_feeds/` for the real-world - flux → OSC bridge. Off by default (Auto-start data_feeds toggle in - Settings to enable). uv auto-syncs the venv on first run. -- Quitting the menubar app sends `SIGTERM` to all children. +``` +sclang : Process sclang + boot.scd (mode-dependent boot file) +oscope-of : Process .app bundle (capture Hantek + visuals) +metalviz : uv run python -m data_only_viz.main [--multi-hmr] + [--fullscreen] +web : node sound_algo/web/server.js :3000 (control + Hydra) +dataweb : node data_only_viz/web/server.js :3211 (dashboard data-only) +data_feeds : uv run python bridge.py [-c config.data-only.toml] +body : swift run AVLiveBody (RealityKit, :57130 TCP listener) +``` + +Chaque process a son terminationHandler avec restart sentinel +(`metalVizWantsRestart` etc.) — pas de zombies. Tous les stdouts sont +captures dans `LogView`. + +## Ports OSC + +| Port | Direction | Role | +|------|-----------|------| +| 57121 | UDP | sclang OSCdef IN (data_feeds + web control) | +| 57122 | UDP | sclang -> web sound_algo (BPM/sync) | +| 57123 | UDP | oscope-of OSC IN + web /control/viz* | +| 57124 | UDP | data_only_viz/web dashboard OSC IN (data_feeds) | +| 57125 | UDP | data_only_viz/web sync IN (SC -> web BPM/RMS) | +| 57130 | TCP | AVLiveBody (RealityKit) - SMPL-X vertices binaire |