NVIDIA Lyra 2.0: one photo → a video → a 3D world

The whole journey of the Schönbrunn Palm House photo through Lyra 2.0, run on rented H100 GPUs on Modal on 9 October 2026. Lyra ran twice on the same inputs: once in distilled (fast) mode and once at full quality. Each run is covered separately below, with its own time and cost.

1Input (the same for both runs)

Lyra takes three things: a starting image, a camera path and a text caption. Nothing else about the Palm House is given to it.

Camera paths

• A · zoom: push in to 0.5× the distance to the stair (81 frames), then pull back 1.5× behind the photographer (241 frames). Built-in horizontal_zoom.
• B · orbit: push in 0.35× (81 frames), then circle about 69° (1.2 rad) around the point at the centre of the photo, the stair (241 frames). Built-in orbit_horizontal.

Shared settings

• Resolution 480×832, 16 fps, seed 1, 322 frames per video.
• NVIDIA's realism_boost and detail_enhancer LoRAs at 0.4 each.
• The only difference between the runs is the sampler (see each run below).

Caption (written in the style of NVIDIA's own examples)

Loading…

Run B used the same description reworded for an orbit: “a smooth orbit around the staircase that reveals its far side… the same continuous helix of steps and the same filigree pattern are seen from every side”.

2How Lyra works (both runs)

• Generation. Lyra builds the video in chunks of 80 frames. Before each new frame it projects everything generated so far into the new camera position using per-frame depth. That is its 3D memory. The video model, a 14-billion-parameter Wan 2.1, then paints only what that memory cannot explain. The intermediate clips below show this: top half, the reprojected memory (black means never seen); bottom half, the generated frame.
• Joining. The zoom-out phase is played in reverse and joined to the zoom-in phase into one 322-frame clip. Your photo sits at frame 240.
• Reconstruction. A separate stage lifts the video to 3D. VIPE runs visual SLAM (DROID-SLAM) over all 322 frames to recover the camera path. Depth-Anything-3, with a Gaussian head that NVIDIA fine-tuned on Lyra videos, predicts Gaussians from 128 keyframes. These are fused into one scene of 11,153,408 Gaussians (758 MB PLY).

Full qualityRun 2: full mode

Lyra's recommended setting: 50 sampling steps with classifier-free guidance (scale 5), no distillation LoRA. About 7× slower than distilled in our runs.

Improving the 3D model: original vs refined

Lyra's 3D model is softer than its own video; NVIDIA's own render of it shows the same, and others report it too (lyra#59). As an experiment we refined the full-quality orbit model (B) with standard 3D Gaussian Splatting optimisation (gsplat): start from Lyra's 11.2 M Gaussians and camera poses, then adjust them so renders match the video frames.

• Fair test: every 10th frame (32 of 322) was held out and never trained on; scores below are on those frames. Our render of Lyra's original model matched NVIDIA's own render at 38.8 dB PSNR, close to pixel-identical, so the “original” here is exactly Lyra's output.
• Recipe: 3,000 steps, 0.8 × L1 + 0.2 × D-SSIM loss (the original 3DGS loss), low learning rates, no new Gaussians added; the best checkpoint by held-out score was kept (the last one).

Scored against Lyra's video on 32 held-out frames	PSNR (higher = closer)	SSIM (higher = closer)
Lyra original 3D model	15.67 dB	0.365
Refined 3D model	20.92 dB (+5.3)	0.662 (+81%)
Refined, on training frames (overfitting check)	21.16 dB	0.685

Held-out frames improved almost as much as training frames, so this is a better 3D model, not memorised frames. It still cannot add detail no frame showed, and views far off the camera path are unchanged.

Held-out frames: video · original · refined

Whole camera path: original (left) vs refined (right)

Open 3D · original (full)Open 3D · refined

Refinement step (one H100)	Duration	Cost
Load model + video, render check, baseline scores	27 s	$0.03
3,000 refinement steps (6 held-out evaluations included)	2 m 16 s	$0.15
Total	2 m 43 s	≈ $0.18

3Distilled vs full quality, side by side

The same moment of each orbit walkthrough. Use the 3D viewer buttons in each run section to compare the reconstructed models.

4One-time setup (shared by both runs)

Rates: H100 $3.95/h ($0.0010972 per second), L4 $0.80/h, CPU $0.00003942 per core-second, memory $0.00000667 per GiB-second (modal.com/pricing). Run costs below are H100 time × $3.95/h, using container lifetimes from Modal's records and Lyra's logs (so they include idle seconds). Actual bill, from the Modal dashboard: $7.29 for setup plus all four runs (2 distilled + 2 full quality), covered by Modal's free $30 monthly credit. The GPU-time estimate above comes to $7.51, which matches within the setup estimates. The volumes were deleted afterwards, so there is no ongoing storage cost.

5What it took to get running

• transformer-engine wouldn't compile: missing nvtx3 headers. Fixed with the include paths from lyra#47.
• transformer-engine required cuDNN ≥ 9.12. Releases from 21 Sep 2026 raised the minimum; pinned to v2.13.0, which was current when Lyra shipped.
• Depth-Anything-3 failed to install. hatchling was installed with --no-deps, so its own dependencies were missing.
• addict missing at runtime. Found by importing every inference module on a cheap L4 before using H100s.
• Reconstruction crashed on an old gdown. The DROID-SLAM weights are now baked into the image, so no Google Drive download happens at runtime.