TimeRewind: Rewinding Time with Image-and-Events Video Diffusion

Jingxi Chen1, Brandon Y. Feng2, Haoming Cai1, Mingyang Xie1, Christopher Metzler1, Cornelia Fermüller1, Yiannis Aloimonos1
1University of Maryland, College Park, 2Massachusetts Institute of Technology
arXiv Code

Rewinding Time with Image-and-Events Video Diffusion

methods

TimeRewind synthesizes the video backward into the pre-capture time with image-and-events video diffusion.

Comparison with Standard SVD and Baselines

Comparison with Standard SVD and baselines, E2VID+* [48], EVDI* [59], and REFID* [48].

Below are backward-time videos synthesized from the reference video's last frame and events.

Sequence 1:

Sequence 2:

Experimental Results

Simple Motion Scenarios

The motion in before-the-capture time mostly comes from a few rigid body objects.

Note that since our task is backward-time video synthesis, the motion in the video is the reverse of original.

Below are backward-time videos synthesized from the reference video's last frame and events.

Sequence 1:

Sequence 2:

Sequence 3:

Sequence 4:

Moderately Complex Motion Scenarios

The motion in before-the-capture time is either due to animals and humans performing actions or scenarios with both the camera and the objects moving.

Note that since our task is backward-time video synthesis, the motion in the video is the reverse of original.

Below are backward-time videos synthesized from the reference video's last frame and events.

Sequence 1:

Sequence 2:

Sequence 3:

Sequence 4:

Physically Complex Motion Scenarios

The motion in before-the-capture time is either due to a dynamic fluid (water stream, fire) or complex particle motion under force .

Note that since our task is backward-time video synthesis, the motion in the video is the reverse of original.

Below are backward-time videos synthesized from the reference video's last frame and events.

Sequence 1:

Sequence 2:

Sequence 3:

Sequence 4:

Method

Illustration of our proposed TimeRewind approach as an adaptor for the general Img2Vid architectures. The components shown in shades of blue (both dark and light) represent the elements of the original pre-trained model, which remain unchanged during our training process. The orange-colored components are specific to our TimeRewind and are being optimized throughout the training.

methods