Fork notice. This is a fork of BenjaminHenriksson/insv-stitch (MIT, © 2026 Benjamin Henriksson). Local addition on branch x3-dual-file: support for X3-generation dual-file .insv input — see X3 dual-file support (fork addition) below. Upstream code, attribution, and license are otherwise unmodified.

Insta360 X5 Stitching Pipeline

Linux stitcher for raw Insta360 X5 footage. Reads .insv files (two H.265 fisheye streams, IMU samples, a protobuf calibration sidecar) and produces stabilized equirectangular stills or video. No Insta360 Studio required.

PSNR against Studio's own output: 22.5 to 22.9 dB at 7680×3840.

Insta360 Studio is closed source and Windows/macOS only. Reproducing its output on Linux meant working out the .insv container, the protobuf-encoded MEI calibration, the IMU axis convention, and the stitching and blending math. The result is a single-file pipeline of about 1,200 lines that matches the reference within a dB.

Files

• x5_pipeline.py. The pipeline.
• PIPELINE.md. Architecture: the twelve stages, the MEI model, the blending math, the known limitations.
• x5_pipeline.md. Longer notes: container format, IMU axis calibration, rolling shutter, optical flow experiments.
• old/. First implementation, plus FINDINGS.md with the reverse-engineering notes the rewrite is built on. See old/README.md.

Architecture

Everything fuses into one backward remap per output pixel (following the pattern from the Insta360 SDK and Qualcomm's stabilization patent):

1. Parse the .insv into two H.265 streams and an IMU track.
2. Parse the .pb sidecar for MEI calibration (xi = 2.0, 13 distortion coefficients per lens, per-lens extrinsics).
3. Derive per-frame stabilization from IMU gravity.
4. Derive per-scanline rolling-shutter rotations, 32 SLERP keyframes across a 21 ms readout.
5. For each output pixel: ray, stabilize, transform into the lens frame, MEI-project, distort, sample.
6. Blend on longitude preference times coverage depth. No hardcoded feather width.
7. Symmetric per-channel gain across the seam.
8. Optional DIS optical flow for close-range parallax. Optional bilateral denoise.

Full treatment in PIPELINE.md.

Install

Python 3.12+, with ffmpeg and ffprobe on PATH.

uv sync
# or
pip install -e .

Usage

# single frame, full resolution, with denoising
uv run python x5_pipeline.py input.insv -o output.jpg -w 7680 --denoise

# full video
uv run python x5_pipeline.py input.insv -o output.mp4 -w 3840 --video

# stabilization off (required on un-calibrated hardware, see below)
uv run python x5_pipeline.py input.insv --no-stab -o output.jpg

# PSNR against a Studio-rendered reference
uv run python x5_pipeline.py input.insv --gt studio_render.mp4 -o output.jpg

# CUDA backend (remap + gain blur on GPU, NVENC encode)
uv run python x5_pipeline.py input.insv -o output.mp4 -w 3840 --video --engine cuda

The .insv needs to sit inside the camera's default layout:

DCIM/Camera01/VID_xxx_00_001.insv
MISC/Camera01/VID_xxx_00_001.insv.pb

X3 dual-file support (fork addition)

Upstream targets the X5: one .insv file muxes both fisheye lenses as two H.265 video streams (-map 0:0 front, -map 0:1 back). A 2024-era X3-generation camera instead writes one H.264 video stream per file (plus a 4-channel AAC audio track) and splits the two lenses across two sibling files, e.g. VID_20241128_..._00_017.insv (front) and VID_20241128_..._10_017.insv (back). Feeding a single X3 file into upstream's decoder hits -map 0:1, which lands on the audio stream and crashes with Decode failed: got 0 bytes.

This branch (x3-dual-file) detects and handles that layout automatically:

• count_video_streams() probes the container with ffprobe. Two video streams → X5 single-file layout (unchanged upstream path). One video stream → X3 dual-file layout.
• find_dual_file_sibling() locates the other lens's file by swapping the _00_/_10_ marker in the filename.
• X5Pipeline.__init__ resolves front_path/back_path (and a per-file track index) once, instead of assuming a single path with two tracks. stitch_frame decodes from those resolved paths.
• Metadata/IMU/calibration is always parsed from the _00_ file — on this camera telemetry_parser can only open that one; the _10_ file raises OSError: Unsupported file format. The _00_ telemetry already covers both lenses.
• The two sibling files run about 33 ms (one frame) apart in total duration, since the two lens modules don't start/stop recording in perfect sync. frame_count is clamped to min(front_frames, back_frames) so a video run never decodes past EOF. Lenses are aligned by start (frame 0 == frame 0 on both files) — no per-frame offset was found in the metadata; this is an assumption, not a measurement.

Camera support

Insta360 X3-generation bodies that write the dual-file, single-H.264- stream-per-file layout described above. X5 single-file dual-stream footage still goes through the unchanged upstream path.

Per-camera empirical constant — re-derive on your own unit

Like upstream's existing IMU_TO_CAM caveat below, this fork adds one constant that was fit empirically against one specific X3 unit and is not guaranteed to hold on a different camera body:

• X3_FRONT_MARKER = '_00_' (class attribute on X5Pipeline) — which filename marker carries the lens assigned to lens_params[0] ("front"). Determined by rendering a test frame with both assignments and comparing seam quality: the wrong assignment shows visible doubling/ghosting at the stitch line, the correct one gives a clean seam. Re-derive the same way on your own unit if _00_/_10_ turn out to be swapped; flip the class attribute if so.

mei_forward fix (likely upstream-relevant, not X3-specific)

For lenses with xi > 1 in the unified-sphere (MEI) model, the ray→plane mapping stops being injective past theta_max = arccos(-1/xi). On this camera's fallback calibration, xi ≈ 1.95, so theta_max ≈ 121°: rays more off-axis than that fold back and land on the same fisheye pixel coordinates as legitimate on-axis content, producing a duplicate "ghost island" per lens after remapping. The existing denom > 1e-6 check doesn't catch this — the denominator stays positive over the whole sphere whenever xi > 1. Fixed with the geometric cutoff Zs > -1/xi.

This never triggered on X5 clips because those ship a .pb sidecar and take the extended 13-coefficient calibration path; it only surfaced here because the X3 clip has no sidecar and falls back to the 5-coefficient Gyroflow calibration, whose xi/k1-k3 combination folds back well inside the valid pixel circle. Since the bug is in the projection math itself (not camera-specific), it's latent in upstream too whenever a clip takes that fallback path — flagging in case it's worth upstreaming independently of the dual-file support.

Throughput

Measured single-threaded on an M-series laptop:

Output width	s/frame	Full clip (13,920 frames)
2880 px	3.41	≈ 13 h
3840 px	5.77	≈ 22 h

Per-frame ffmpeg spawn (see upstream's own note below) plus the Lanczos4 remap dominate. Batched/piped decode and multiprocessing across frames are the obvious next levers — not implemented in this branch; treat the numbers above as current-state, not a ceiling.

Validation

Three real takes (80 s, 210 s, 174 s) stitched successfully at 1920×960 with clean seams and no ghosting, using the mei_forward fix above. IMU-based leveling tracks the camera's actual orientation correctly: checked at t≈20 s and t≈60 s in two different takes, the nadir/zenith placement is geometrically correct (piano at the nadir as a symmetric band, standing people upright, people leaning over the camera inverted near the zenith) — see the smoke-test note below for why frame 0 alone is not a reliable place to judge this.

Known limitations (fork-specific)

• Don't judge orientation from a single early frame. The first seconds of a handheld take often show the camera being carried, set down, or otherwise off the tripod/mount — a genuinely tilted frame 0 looks like a leveling bug but isn't one. Sample a frame well into the take (e.g. t≈20 s) before drawing any conclusion about orientation or IMU calibration.
• X3_FRONT_MARKER and the dual-file layout assumptions were validated on one X3 camera body and one set of test clips; re-verify on other units.
• The ~33 ms front/back start offset is unmeasured; frames are assumed aligned at index 0 rather than time-corrected.
• Full-resolution full-length runs take hours (see Throughput); no batching or multiprocessing yet.

CUDA engine (--engine cuda)

The default backend is pure CPU (NumPy + OpenCV, pip wheels have no CUDA kernels). The cuda engine (gpu_engine.py) pushes the two heaviest per-frame stages onto a NVIDIA GPU, keeping the seam-band logic on CPU:

• remap (both lenses): cv2.cuda.remap, tables uploaded once per map build, invalid-pixel zeroing on the device.
• spatial gain blur: the sigma=80 weighted Gaussian blurs of compute_spatial_gain run through cv2.cuda.GaussianBlur.
• encode: --encoder nvenc (h264_nvenc) is selected automatically with --engine cuda.

Requirements: OpenCV built with CUDA_ENABLED=ON (the pip wheel only shows the cv2.cuda framework — GpuMat/Stream — with no kernels), and ffmpeg compiled with h264_nvenc. On a build without kernels the engine raises CudaUnavailable with a precise message at construction, so CPU runs are unaffected.

The seam/DIS, seam-DP and multiband-blend paths intentionally stay on the CPU (narrow ~30° bands; DIS has no CUDA kernel). The final blend and the remap-mask semantics match the CPU path pixel-for-pixel, so the --gt PSNR comparison stays meaningful.

Limitations

IMU calibration is camera-specific. The IMU_TO_CAM rotation in x5_pipeline.py was solved via Wahba's method against ground-truth gravity on one X5 unit. Unit-to-unit PCB mounting variation will degrade stabilization on other cameras. Pass --no-stab, or re-solve against a Studio render from your own hardware.

Close-object parallax. Around 18 px of ghosting at the stitch line for objects under 3 m, a function of the 30 mm inter-lens baseline. DIS flow helps but does not match Insta360's learned ai_stitch_model_v2.ins on repetitive patterns like fence mesh or foliage.

Per-frame ffmpeg decode. Each frame spawns its own ffmpeg process, about 2 s of overhead. Piped batch decoding is the obvious next step for video throughput.

License

MIT. See LICENSE.