BROWSE
Detail
Conference Paper
Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment
- Authors
- Hyeju Shin, Chorwon Kim, Ryangsoo Kim, Hark Yoo, Jaein Kim
- Issue Date
- 2026-07
- Citation
- International Conference on Machine Learning (ICML) 2026 Workshop : Hypothesis Testing, pp.1-14
- Language
- English
- Type
- Conference Paper
- Abstract
- The emergence of vision language models with fewer than 3 billion parameters has accelerated the implementation of on-device multimodal intelligence. However, a detailed understanding of component-wise quantization remains a bottleneck for optimal deployment. This paper presents a systematic evaluation framework for empirically validating five hypotheses across six quantization configurations on the Jetson Orin NX and AGX. By separating the vision encoder, projector, and large language model backbone yields the following results: (1) Quantization sensitivity is governed by the structural paradigm (MoE vs. dense) rather than scale alone, with MoE backbones mitigating INT4 noise where dense backbones degrade; (2) SigLIP encoders incur disproportionate INT8 latency on Jetson Ampere--a deployment-specific encoder-kernel-hardware interaction, not a SigLIP flaw; (3) Although INT4 quantization of LLMs greatly reduces VRAM consumption, it also causes slower token generation due to dequantization overhead; (4) Composite quantization errors are largely additive, except along the modality-alignment path, which is architecture-dependent; (5) The intelligence-per-joule profile varies significantly across platforms owing to memory bandwidth constraints.
- Keyword
- vision-language models, post-training quantization, component-wise analysis, edge deployment, energy-efficient inference
- KSP Keywords
- Bandwidth constraints, Component-Wise, Language Models, Memory bandwidth, Optimal deployment, Quantization error(QE), Systematic evaluation, energy-efficient, evaluation framework
This work is distributed under the term of Creative Commons License (CCL)
(CC BY)
(CC BY)
