BROWSE
Detail
Journal Article
GPU 및 NPU 기반 다중 노드 클러스터링 환경에서 Ray-vLLM 분산 시스템 성능분석
Cited - time in 
Share 
Share
- Issue Date
- 2026-08
- Citation
- 대한임베디드공학회논문지, v.21, no.4, pp.237-243
- ISSN
- 1975-5066
- Publisher
- 대한임베디드공학회
- Language
- Korean
- Type
- Journal Article
- Abstract
- Large Language Model (LLM) services require distributed inference systems that sustain high throughput under limited memory capacity and increasing request concurrency. Ray, a distributed processing framework, enables cluster-level distributed execution, while vLLM improves node-level inference efficiency. This paper presents a Ray–vLLM based distributed inference architecture that directly maps RAG-composed requests to Actor-level execution. We evaluate performances of four configurations with and without vLLM and Ray Serve on both GPU and NPU clusters. The proposed Ray Actor with vLLM structure achieves the best overall performance on both platforms, reducing total latency by 27.7% and increasing throughput by 38.2% on GPU, and reducing total latency by 20.3% while increasing throughput by 25.5% on NPU compared to the Ray Serve-vLLM baseline. The results show that vLLM is the key factor in performance improvements. It improves both request-level latency and throughput in the GPU environment, whereas it mainly enhances batch-level execution efficiency and overall throughput in the NPU environment. The results further show that the Actor-direct execution path enables efficient distributed LLM inference and is well suited for high-throughput RAG-based services.
- Keyword
- Distributed Inference System, Large Language Model, vLLM, Ray
- KSP Keywords
- Batch-level, Cluster-level, Distributed Execution, Distributed inference, Execution efficiency, High throughput, Inference System, Key factor, Language Models, Overall performance, distributed processing