Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity… Read More
TrajTok: Learning Trajectory Tokens enables better Video Understanding
Tokenization in video models, typically through patchification, generates an excessive and redundant number of tokens. This severely limits video efficiency and scalability. While recent trajectory-based tokenizers offer a promising solution by decoupling video duration from token count, they rely on complex external segmentation and tracking pipelines that are slow and task-agnostic. We propose TrajTok, an end-to-end video tokenizer module that is fully integrated and co-trained with video models for a downstream objective, dynamically adapting its token granularity to semantic complexity…
Related Posts
Evaluating Perplexity on Language Models
This article is divided into two parts; they are: • What Is Perplexity and How…
TASER: Translation Assessment via Systematic Evaluation and Reasoning
We introduce TASER (Translation Assessment via Systematic Evaluation and Reasoning), a metric that uses Large…
Train Your Large Model on Multiple GPUs with Pipeline Parallelism
This article is divided into six parts; they are: • Pipeline Parallelism Overview • Model…