상세정보
학술대회
Performance Study of Distributed Big Data Analysis in YARN Cluster
- 발행일
- 201810
- 출처
- International Conference on Information and Communication Technology Convergence (ICTC) 2018, pp.1261-1266
- 초록
- In the 4-th Industrial Revolution era, various intelligent solutions and services have been emerging recently. To provide high quality service in those intelligent applications, the big data should be collected without any loss and comprehensively analyzed. Especially, when using machine and deep learning techniques, the big data processing delays should be minimized in order to guarantee the freshness of models. In this paper, we evaluate the performance of Apache Spark which is one of the most popular big data processing and analysis frameworks. Beyond the performance analysis of Spark in distributed cluster environment, we evaluate the performance of TensorFlowOnSpark which is the promising distributed deep learning framework designed to handle big data efficiently. From the experimental results, we can conclude that Spark on YARN is a solid underlying framework that guarantees the performance and scalability of distributed machine and deep learning by efficiently processing its data and algorithms in a parallel and distributed manner.
- KSP 제안 키워드
- Apache spark, Big Data Processing, Big Data analysis, Deep learning framework, Distributed cluster, Industrial Revolution, Intelligent Solutions, Performance analysis, Performance and scalability, deep learning(DL), distributed manner