BROWSE
Detail
Journal Article
Length-Based Dispatching of Small Language Models On Heterogeneous On-Device Platforms
Cited 0 time in 
Share 
Share
- Authors
- Seongjin Yun, Jeong-Si Kim
- Citation
- IEEE Embedded Systems Letters, Early Access
- ISSN
- 1943-0663
- Publisher
- IEEE
- Language
- English
- Type
- Journal Article
- Abstract
- Heterogeneous on-device platforms integrate accelerators with distinct energy and latency characteristics, so the cost-optimal inference device depends on runtime conditions. In small language model inference, input and output lengths induce length-dependent efficiency regimes, yet the output length is unknown before execution because autoregressive generation yields prompt-conditioned length variability. Consequently, dispatch under dynamic energy–latency preferences requires tracking a preference-dependent crossover boundary rather than predicting an exact output length. We propose a lightweight length-based dispatcher that combines a hardware-calibrated weighted Energy–Delay Product (EDP) boundary with a τ-conditioned classifier for stable boundary tracking. On a GPU–NPU platform, the proposed method maintains stable near-optimal efficiency across a preference sweep without retuning, achieving about a 2% weighted EDP optimality gap with under 1% latency and about 0.1% energy overhead. Compared with recent output-length prediction strategies adapted from LLM scheduling literature, the proposed method achieves the lowest optimality gap while requiring 3–13× less dispatch latency. The existing prediction-based methods degrade under preference shifts, confirming that explicit τ conditioning is essential for stable dispatch.
- Keyword
- Heterogeneous on-device platforms, runtime dispatch, small language model inference
- KSP Keywords
- Boundary tracking, Cost-optimal, Input and output(I/O), Language Models, Model Inference, Optimal efficiency, Optimality Gap, Prediction-based, Stable boundary, dynamic energy, energy overhead