Improve Temporal Reasoning in MLLMs via Video Contrastive Decoding

A major distinction between video and image understanding is that the former
requires reasoning over time. Existing Video Large Language Models (VLLMs)
demonstrate promising performance in general video understanding, such as brief
captioning or object recognition within individual frames. However, they often
struggle with temporal reasoning such as understanding continuous actions or track
ing object transformations over time—which typically demands the integration
of multiple frames in a temporally coherent manner. We first explore and explain
such failures in Video LLMs from the perspective of language and “image” priors.
While existing research has attempted to enhance the temporal understanding of
VLLMs through various training strategies, the demand for expensive computa
tional resources and training data often presents significant barriers. To this end,
we further propose a simple yet novel idea for improving temporal reasoning in
videos at no additional training cost. Specifically, to better capture the temporal
structure across multiple frames—the key to effective temporal reasoning—we
distort the temporal consistency in key frames during the decoding phase. Such
corruption induces time-insensitive wrong responses from the model, which are
then contrastively avoided when generating the final correct output. In this way, the
model is encouraged to perform more temporally coherent reasoning. Our method
yields consistent improvements across both temporal-specific and general video
understanding benchmarks, demonstrating its effectiveness and generalizability.