Anonymous Authors
Using image models naively for solving inverse video problems often suffers from flickering, texture-sticking, and temporal inconsistency in generated videos. To tackle these problems, in this paper, we view frames as continuous functions in the 2D space and videos as a sequence of continuous transformations, i.e., warpings, between different frames. Given this perspective, we propose to train function space diffusion models for solving inverse problems only in \textit{images}. We then show that to apply these models to videos, the model should be equivariant with respect to the transformation observed in input videos. To ensure equivariance, we introduce two simple approaches including post-hoc test-time consistency guidance and training-time consistency augmentation. Our method allows us to use state-of-the-art latent diffusion models such as Stable Diffusion XL to solve video inverse problems. We demonstrate the effectiveness of our method for video inpainting and $8\times$ video super-resolution, outperforming state-of-the-art techniques based on noise transformation.