Representations Before Pixels:
Semantics-Guided Hierarchical Video Prediction
Abstract
Accurate future video prediction requires both high visual fidelity and consistent scene semantics, particularly in complex dynamic environments such as autonomous driving. We present Re2Pix, a hierarchical video prediction framework that decomposes forecasting into two stages: semantic representation prediction and representation-guided visual synthesis. Instead of directly predicting future RGB frames, our approach first forecasts future scene structure in the feature space of a frozen vision foundation model, and then conditions a latent diffusion model on these predicted representations to render photorealistic frames. This decomposition enables the model to focus first on scene dynamics and then on appearance generation. A key challenge arises from the train–test mismatch between ground-truth representations available during training and predicted ones used at inference. To address this, we introduce two conditioning strategies, nested dropout and mixed supervision, that improve robustness to imperfect autoregressive predictions. Experiments on challenging driving benchmarks demonstrate that the proposed semantics-first design significantly improves temporal semantic consistency, perceptual quality, and training efficiency (up to 7× faster generation and 14× faster segmentation convergence) compared to strong diffusion baselines.
Overview
Re2Pix decomposes video prediction into two stages. In Stage 1 – Semantic Feature Prediction, a frozen vision foundation model (VFM) encoder — DINOv2 — extracts dense semantic features \(h_{1:M}\) from the context frames, and a lightweight masked feature transformer autoregressively forecasts the future semantic features \(\hat{h}_{M+1:K}\), one frame at a time. These representations capture high-level scene structure while abstracting away low-level appearance. In Stage 2 – Semantics-Guided Video Generation, a latent video diffusion transformer operating in the WAN2.1 VAE latent space is conditioned on both the past and the predicted semantic features to denoise the future latents \(\hat{z}_{M+1:K}\), which the VAE decoder renders into photorealistic RGB frames. Separating what happens from how it looks lets each stage specialize — temporal reasoning first, appearance synthesis second.
Architecture & Robust Semantic Conditioning
Our diffusion transformer follows the video-prediction DiT design of Cosmos-Predict, retaining 3D-factorized RoPE, query/key normalization, RMSNorm and AdaLN-LoRA noise conditioning, while removing text cross-attention. Semantic guidance is injected by early fusion: the VAE latents are patchified (2×2), the VFM features are resized to the same grid, both are embedded independently and combined by channel-wise summation at the input — strong conditioning without increasing the token count.
During training the diffusion model sees clean, ground-truth future features, but at inference it must rely on autoregressively predicted ones that accumulate errors. Naively training on perfect features causes a severe train–test mismatch — sharp semantics but blurry pixels. We close this gap with two complementary strategies:
- Nested dropout. The 1152 PCA channels of the DINOv2 features form a coarse-to-fine hierarchy. With equal probability we keep only the first \(c \in \{8, 16, 32, 64, 128, 256, 512, 1152\}\) channels and zero the rest, forcing the generator to rely on robust, coarse semantics rather than memorizing fine-scale feature values.
- Mixed supervision. Each batch mixes 90% ground-truth and 10% Stage-1-predicted features, directly exposing the generator to imperfect conditioning. The 90/10 ratio best trades off semantic consistency against perceptual quality.
Comparison with Baselines
We compare against a standard latent video-diffusion baseline, two representation-alignment methods (REPA and VideoREPA), and a larger-capacity Baseline-Large. Re2Pix improves both semantic-consistency metrics (segmentation mIoU, depth) and generation quality (FID, FVD) — e.g. +2.98 mIoU(A), +4.65 IoU(M), −2.96 FID and −8.04 FVD over the baseline — while a purely parameter-scaled Baseline-Large stays well behind, confirming the gains come from the hierarchical design rather than model size. Re2Pix (Stage-1) reports the feature-space upper bound on semantic consistency.
Accelerated Training Convergence
Training both models with a constant learning rate for 140k iterations, Re2Pix dominates the baseline throughout. It reaches FID 15 in 20k iterations versus 140k for the baseline — a 7× speed-up — with a matching 7× on FVD and a 14× speed-up on segmentation mIoU. This convergence boost comes at negligible cost: inference is 3.2s / 28.5 GB per scene (vs. 3.1s / 27 GB for the baseline), far cheaper than Baseline-Large.
Scaling & Cross-Dataset Generalization
Trained on a larger mix of Cityscapes, nuScenes and CoVLA, Re2Pix consistently beats the baselines in-domain (Cityscapes, nuScenes) and zero-shot on KITTI. It also surpasses Vista — a 2.5B-parameter driving world model — and is competitive with Cosmos-Predict-2, despite both being pretrained with orders of magnitude more data and compute before being fine-tuned in our setting.
Ablation Study
All ablations are on Cityscapes with DINOv2-Reg ViT-B/14 features unless noted.
Qualitative Results
Five representative Cityscapes scenes. Top: context frames with their DINOv2 PCA features. Bottom rows: ground truth, the diffusion baseline, Re2Pix, and the semantic features forecast by Stage 1 that guide Re2Pix. Re2Pix preserves scene geometry and object boundaries more faithfully and produces smoother, more stable future rollouts.
Cityscapes — scene 60
Cityscapes — scene 228
Cityscapes — scene 289
Cityscapes — scene 456
Cityscapes — scene 485
BibTeX
@inproceedings{karypidis2026re2pix,
title = {Representations Before Pixels: Semantics-Guided Hierarchical Video Prediction},
author = {Karypidis, Efstathios and Gidaris, Spyros and Komodakis, Nikos},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}