ETRI-Knowledge Sharing Plaform

ENGLISH

성과물

논문 검색
구분 SCI
연도 ~ 키워드

상세정보

학술대회 Performance Study of Distributed Big Data Analysis in YARN Cluster
Cited 3 time in scopus Download 14 time Share share facebook twitter linkedin kakaostory
저자
안후영, 김현재, 유웅식
발행일
201810
출처
International Conference on Information and Communication Technology Convergence (ICTC) 2018, pp.1261-1266
DOI
https://dx.doi.org/10.1109/ICTC.2018.8539474
초록
In the 4-th Industrial Revolution era, various intelligent solutions and services have been emerging recently. To provide high quality service in those intelligent applications, the big data should be collected without any loss and comprehensively analyzed. Especially, when using machine and deep learning techniques, the big data processing delays should be minimized in order to guarantee the freshness of models. In this paper, we evaluate the performance of Apache Spark which is one of the most popular big data processing and analysis frameworks. Beyond the performance analysis of Spark in distributed cluster environment, we evaluate the performance of TensorFlowOnSpark which is the promising distributed deep learning framework designed to handle big data efficiently. From the experimental results, we can conclude that Spark on YARN is a solid underlying framework that guarantees the performance and scalability of distributed machine and deep learning by efficiently processing its data and algorithms in a parallel and distributed manner.
KSP 제안 키워드
Apache spark, Big Data Processing, Big Data analysis, Deep learning framework, Distributed cluster, Industrial Revolution, Intelligent Solutions, Performance analysis, Performance and scalability, deep learning(DL), distributed manner