HuggingFace Papers Jul 21, 2026 8/10 signal

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

modelsresearcheval

Summary

This paper introduces TimeLens2, a generalist video temporal grounding model that treats evidence as an interval set throughout supervision and optimization. The authors create TimeLens2-93K, a dataset with reliable multi-span supervision for long videos, using a staged evidence-verification pipeline. They also propose a temporal Wasserstein reward that provides dense, matching-free feedback for non-overlapping and unequal-cardinality predictions. The 2B, 4B, and 8B TimeLens2 models achieve state-of-the-art performance on seven benchmarks, with the 4B model outperforming open-source models up to 397B parameters by an average of 7.5 mIoU points.

Problem

Generalist video temporal grounding, which aims to localize evidence for a query across diverse videos and query types, faces two structural mismatches. First, supervision for long videos is often unreliable, as one-pass annotation can miss repeated evidence, confuse similar events, or produce imprecise boundaries. Second, existing optimization objectives, like reinforcement learning with temporal IoU (tIoU), provide zero feedback for non-overlapping predictions, making it difficult to learn from near misses, and are fragile for multi-span targets.

Method

TimeLens2 addresses supervision and optimization mismatches. For supervision, it introduces TimeLens2-93K, a dataset of 93,232 instances built via a pipeline of caption-derived proposals, independent localization by dual agents, cross-agent consensus, semantic verification, and boundary refinement. For optimization, it uses a two-stage training process: long-context supervised fine-tuning (SFT) on TimeLens2-93K, TimeLens-100K, and Ego4D-NLQ, followed by reinforcement learning (GRPO) with a novel temporal Wasserstein reward that complements tIoU by providing dense, geometry-aware feedback for interval sets.

Details

State-of-the-Art Comparison:

TimeLens2 models outperform prior open-source and proprietary models across seven benchmarks. The 2B, 4B, and 8B variants improve over their Qwen3-VL backbones by 14.2, 13.0, and 18.1 average mIoU points, respectively. The 4B model surpasses the Qwen3.5-397B-A17B model on every benchmark.

Model	Size	Charades TL (mIoU)	ActivityNet TL (mIoU)	QVHighlights TL (mIoU)	VUE-TR (mIoU)	VUE-TR-V2 (mIoU)	MomentSeeker (mIoU)	Ego4D-NLQ (mIoU)
TimeLens2-2B	2B	53.4	55.4	67.0	51.1	43.8	24.3	16.8
Qwen3-VL-2B (Backbone)	2B	43.4	39.9	52.2	31.4	24.1	12.4	9.3
TimeLens2-4B	4B	57.7	59.0	69.3	53.2	48.1	27.9	18.6
Qwen3-VL-4B (Backbone)	4B	49.4	50.7	62.3	33.6	20.3	15.3	11.4
TimeLens2-8B	8B	58.6	58.6	70.2	53.5	47.7	28.5	19.0
Qwen3-VL-8B (Backbone)	8B	47.2	48.0	59.4	25.5	14.0	9.8	5.3
Gemini 2.5 Pro	Proprietary	52.8	58.1	70.4	21.9	-	-	-
Qwen3.5-397B-A17B	397B	47.5	53.8	65.8	42.2	34.5	23.3	14.5

Ablation on TimeLens2-93K Data:

Training a Qwen3-VL-4B model on the full TimeLens2-93K dataset improves average mIoU from 34.7 (backbone) to 45.8.
The first 5% of TimeLens2-93K provides the largest gain, raising average mIoU to 42.8.
Despite containing only declarative queries, TimeLens2-93K improves performance on the question-form MomentSeeker benchmark from 15.3 to 25.8 mIoU.
The final SFT data mixture includes TimeLens2-93K, TimeLens-100K, and Ego4D-NLQ, reaching an average mIoU of 46.4.

Ablation on Label Curation Pipeline:

Each stage of the data curation pipeline progressively improves label quality and model performance, demonstrating a quality-over-quantity effect. The full cascade improves mIoU by 3.8 points over raw labels from the Qwen annotator, using only 1/8th of the data.

Curation Stage (Qwen-anchored)	QA Turns (K)	Average mIoU
Raw Qwen3-VL-30B-A3B labels	735.4	42.0
+ Temporal consensus	174.2	43.4 (+1.4)
+ Semantic verification	93.2	44.1 (+0.6)
+ Boundary refinement	93.2	45.8 (+1.7)

Ablation on Reward Design:

The proposed temporal Wasserstein reward (R_TW) with uniform support consistently improves over a tIoU-only reward, raising average mIoU from 47.0 to 47.7.
R_TW outperforms matched NGIoU, which achieves 47.1 average mIoU, with the largest gaps on multi-interval benchmarks VUE-TR (1.4 points) and VUE-TR-V2 (1.0 point).
R_TW is most effective on predictions with zero tIoU. It helps recover positive overlap for 21.9% of near misses (normalized gap ≤ 1x) vs. only 5.7% of far misses (gap > 5x).
Adding R_TW to the reward function rescues 75.8% of GRPO groups that would otherwise have an all-zero tIoU reward, creating a non-constant learning signal. It reduces constant-reward groups from 13.8% to 3.6% and nearly doubles the mean absolute advantage.

Ablations on Training Recipe:

Increasing the maximum packed sequence length for SFT from 16K to 100K tokens improves average mIoU from 44.4 to 46.4, with the largest gains on long-video benchmarks.
Using diverse instructions and response formats during SFT improves average mIoU by 0.9 points over a fixed format, from 44.9 to 45.8.

What's new

The paper introduces two main contributions. First is TimeLens2-93K, a dataset created with a novel staged evidence-verification pipeline that produces reliable single- and multi-span temporal grounding supervision for long videos. Second is a matching-free temporal Wasserstein reward, based on the exact one-dimensional W1 distance over merged interval support, which provides graded, fragmentation-invariant feedback for disjoint and unequal-cardinality predictions during reinforcement learning.

Conclusion

The authors conclude that treating temporal evidence as an interval set from supervision through optimization is key to making video MLLMs reliable. The TimeLens2-93K dataset, created via a verifiable annotation pipeline, and the temporal Wasserstein reward, which provides geometry-aware feedback, enable compact models (2B, 4B, 8B) to achieve state-of-the-art performance across diverse grounding tasks. This approach elevates temporal grounding from a specialized task to a native, auditable capability for generalist video MLLMs.

Read the source →

Don't read this site daily. Get it in your inbox.

The daily brief and Sunday deep dive — distilled, scored, and opinionated. For builders only.