ambisonic-360-player

Static web player for 360° equirectangular video with first-order ambisonic (ambiX) audio, rendered binaurally in the browser and steered by phone gyroscope (magic window), WebXR headset, or mouse drag. Three.js + Omnitone. No backend.

Design spec: docs/superpowers/specs/2026-08-06-ambisonic-360-player-design.md

Dev

npm install
npm run fixtures   # generates test content (needs ffmpeg, python3, Pillow)
npm run dev        # HTTPS dev server, reachable over LAN

Setup: brew install ffmpeg python3 && pip install Pillow

Manual test matrix (before any release)

Platform	Checks
iOS Safari (real device)	decode-test.html PASS; motion permission prompt inside Enter; magic window pans, horizon level; denied permission → touch drag + hint
Android Chrome	magic window; VR button visible; Cardboard session renders stereo, audio follows head
Desktop Chrome + Firefox	drag look; play/pause/mute/fullscreen; back and re-enter without leaks; 3+ min playback → no audible desync (fixture tone stays aligned with SOUND captions)
Binaural direction (headphones, any platform)	during "SOUND FRONT" the tone is centered; during "SOUND LEFT" it is in the left ear; dragging (or panning) the view 90° right while "SOUND FRONT" is showing moves the tone to the LEFT ear — validates the sound field counter-rotates against the view instead of turning with it

Content contract

Each scene folder: equirect.mp4 (H264 equirect 2:1) + ambix.wav (4-ch PCM, ambiX ACN/SN3D) + thumb.jpg, declared in content/manifest.json (id, title, video, audio, thumbnail). Content preparation (stitching + sync + transcode) is covered below — see "Preparing real content". No A→B ambisonic conversion is needed; see that section for why.

Preparing real content

The player never touches raw captures — it only reads the prepared content/ layout described above. Turning a raw Insta360 X5 + Zoom H3-VR take into that layout is a separate offline chain: stitch → measure sync offset → build the scene → rebuild the site.

1. Stitch the video

Raw Insta360 .insv pairs are stitched into equirectangular H264 with a patched fork of insv-stitch (installed locally, outside this repo). The X5 writes both lenses as two streams inside one file; older/pre-X5 cameras (e.g. a 2024-era model) instead split the two lenses across two separate files, ..._00_... and ..._10_... — use insv-stitch's x3-dual-file branch for those, not the default X5 path.

Stitching is CPU-bound and single-threaded: roughly 3.4 s/frame at 2880 px input resolution, 5.8 s/frame at 3840 px — i.e. hours per take, not minutes. Budget accordingly; this is the long pole in the whole chain, not the audio work below.

CUDA engine (optional): the fork gained a --engine {cpu,cuda} backend. cuda runs the remap and the spatial-gain blur on the GPU via cv2.cuda and encodes on NVENC — it needs an OpenCV built with CUDA_ENABLED=ON and ffmpeg with h264_nvenc (see the fork's gpu_engine.py). Pass it through this driver:

python3 scripts/stitch-parallel.py \
  --insv "/path/VID_..._00_...insv" \
  --out  "/path/stitched-equirect.mp4" \
  --width 3840 --engine cuda --workers 4

2. Measure the audio/video sync offset

uv run --with numpy,scipy python scripts/sync-offset.py \
  --h3vr /path/to/H3VR/260805_002.WAV \
  --video /path/to/stitched/piezzano-017-equirect.mp4

Why this step exists: dual-system sound. The H3-VR has no reliable shared clock with the camera (its internal clock is wrong — file timestamps are meaningless for pairing or offsets), so which H3-VR file goes with which take, and by how much they're offset, has to be recovered by cross-correlating the camera's own embedded audio against the H3-VR's W channel. sync-offset.py does this and prints an offset_s plus a confidence (Pearson r) over several windows and the measured clock drift. Reject a pairing if the correlation is garbage (r ≈ 0.02–0.07, as opposed to a real match's r ≈ 0.4–0.7) — that's a wrong pairing (e.g. a false-start recording), not a bad offset.

Two facts worth knowing before the next capture, because they save the most time:

• The Zoom H3-VR WAVs are already B-format ambiX — no A→B conversion is needed. The recorder's bext/iXML metadata reads Rec Mode=AmbiX, TRACK_LIST=W,Y,Z,X (ACN order), confirmed by signal analysis (strong negative Z↔X correlation, which is impossible for raw A-format capsule signals). This matches the player's content contract directly — just trim/pad and transcode, covered by prepare-scene.py below.
• The camera's own embedded 4-channel AAC track is NOT ambisonics. It shows up as channel_layout=4.0, handler_name=Ambarella AAC, with balanced per-channel RMS — a discrete mic array, not an encoded soundfield. It is only ever useful as the sync reference for sync-offset.py above; never feed it to the player as spatial audio.

3. Build the scene

python3 scripts/prepare-scene.py \
  --id piezzano-017 \
  --video /path/to/stitched/piezzano-017-equirect.mp4 \
  --audio /path/to/H3VR/260805_002.WAV \
  --offset 0.1653 \
  --title "Piezzano — prise 017 (1:20)"

--offset takes the value sync-offset.py printed, unmodified — the sign convention is shared between the two tools: it's the position in the H3-VR file that corresponds to video t=0. Positive means the H3-VR started first (front of the wav gets trimmed); negative means it started later (silence gets padded onto the front). prepare-scene.py trims/pads all 4 channels identically via a single ffmpeg filter chain (never per-channel — that would break the soundfield), then transcodes the stitched video down from its raw ~55 Mbps to a web-playable ~14 Mbps H264, extracts a thumbnail, verifies the result (4-channel/48 kHz, at least as long as the video), and upserts the scene into public/content/manifest.json without touching any other entry. Safe to re-run for the same --id.

4. Rebuild the site

npm run build

Vite copies public/ (now including the new scene) into dist/ — dist/content/ is the deployable payload.

Worked example: piezzano

Three takes were paired against the H3-VR recordings by cross-correlation and processed with the commands above:

Take	H3-VR file	offset_s	r (mean)	Action
piezzano-017	260805_002.WAV	+0.1653	0.63	trim 0.1653 s off front
piezzano-018	260805_003.WAV	−1.4984	0.40	pad 1.4984 s of silence
piezzano-019	260805_004.WAV	+6.0285	0.69	trim 6.0285 s off front

260805_001.WAV was a false start — correlation was garbage (r ≈ 0.02–0.07 against every take) — and was rejected rather than paired. Measured clock drift across all three takes was −0.04 to −1.06 ms over the take length, far under both the ~40 ms lip-sync tolerance and the player's 80 ms resync threshold, so no resampling was needed to correct drift.

Known limits

• iOS Safari memory ceiling. A 3½-minute take yields a ~127 MB 24-bit ambix.wav; decodeAudioData expands that to ~160 MB of raw PCM in memory. That's the practical ceiling to watch on iOS Safari for take length/scene count kept resident at once.
• Payload size. Per scene: video at ~14 Mbps (from prepare-scene.py's transcode) plus an uncompressed WAV — for four scenes like the piezzano set, dist/ comes out to roughly 1 GB.

Deploy

npm run build

Outputs static assets to dist/. HTTPS is required in production — DeviceOrientationEvent.requestPermission() (iOS motion sensors) and WebXR both require a secure context and simply refuse to run without one; this is not just a dev-server convenience, TLS must terminate wherever dist/ is served.

dist/ as built from this repo bundles the dev content/manifest.json, which points at the gitignored fixture media under content/scenes/test-orbit/ — those files are never committed and won't exist on a fresh checkout. Before the first real deploy, either replace/provide content/ next to the deployed dist/ with real scenes (matching the content contract above) or run npm run fixtures to regenerate the fixture scene.

Browser floor: iOS Safari 15+ and modern evergreen browsers — the code uses top-level await.

License

GPL-3.0-or-later — © 2026 Clément Saillant