BROWSE
Detail
Conference Paper
Sink-Token-Aware Pruning for Fine-Grained Video Understanding in Efficient Video LLMs
Cited - time in 
Share 
Share
- Authors
- Kibum Kim, Jiwan Kim, Kyle Min, Yueqi Wang, Jinyoung Moon, Julian McAuley, Chanyoung Park
- Issue Date
- 2026-09
- Citation
- European Conference on Computer Vision (ECCV) 2026, pp.1-18
- Publisher
- Springer
- Language
- English
- Type
- Conference Paper
- Abstract
- Video Large Language Models (Video LLMs) incur high in-ference latency due to a large number of visual tokens provided to LLMs.To address this, training-free visual token pruning has emerged as asolution to reduce computational costs; however, existing methods areprimarily validated on Multiple-Choice Question Answering (MCQA)benchmarks, where coarse-grained cues often suffice. In this work, wereveal that these methods suffer a sharp performance collapse on fine-grained understanding tasks requiring precise visual grounding, such ashallucination evaluation. To explore this gap, we conduct a systematicanalysis and identify sink tokens–semantically uninformative tokens thatattract excessive attention–as a key obstacle to fine-grained video un-derstanding. When these sink tokens survive pruning, they distort themodel’s visual evidence and hinder fine-grained understanding. Moti-vated by these insights, we propose Sink-Token-aware Pruning (SToP),a simple yet effective plug-and-play method that introduces a sink scoreto quantify each token’s tendency to behave as a sink and applies thisscore to existing spatial and temporal pruning methods to suppress them,thereby enhancing video understanding. To validate the effectivenessof SToP, we apply it to state-of-the-art pruning methods (VisionZip,FastVid, and Holitom) and evaluate it across diverse benchmarks cover-ing hallucination, open-ended generation, compositional reasoning, andMCQA. Our results demonstrate that SToP significantly boosts perfor-mance, even when pruning up to 90% of visual tokens. Our code is avail-able at https://github.com/rlqja1107/SToP
- Keyword
- Video LLMs, Visual Token Pruning, Attention Sink
- KSP Keywords
- Compositional Reasoning, Fine grained(FG), In-ference, Language Models, Open-ended, Plug-and-Play, Pruning method, Question Answering, Spatial and temporal, Video understanding, Visual evidence