BROWSE
Detail
Conference Paper
Performance Study of Distributed Big Data Analysis in YARN Cluster
Cited 4 time in 
Share 
Share
- Authors
- HooYoung Ahn, Hyunjae Kim, WoongShik You
- Issue Date
- 2018-10
- Citation
- International Conference on Information and Communication Technology Convergence (ICTC) 2018, pp.1261-1266
- Publisher
- IEEE
- Language
- English
- Type
- Conference Paper
- Abstract
- In the 4-th Industrial Revolution era, various intelligent solutions and services have been emerging recently. To provide high quality service in those intelligent applications, the big data should be collected without any loss and comprehensively analyzed. Especially, when using machine and deep learning techniques, the big data processing delays should be minimized in order to guarantee the freshness of models. In this paper, we evaluate the performance of Apache Spark which is one of the most popular big data processing and analysis frameworks. Beyond the performance analysis of Spark in distributed cluster environment, we evaluate the performance of TensorFlowOnSpark which is the promising distributed deep learning framework designed to handle big data efficiently. From the experimental results, we can conclude that Spark on YARN is a solid underlying framework that guarantees the performance and scalability of distributed machine and deep learning by efficiently processing its data and algorithms in a parallel and distributed manner.
- KSP Keywords
- Apache spark, Big data analysis, Deep learning framework, Distributed cluster, Industrial Revolution, Intelligent Solutions, Performance analysis, Performance and scalability, big data processing, deep learning(DL), distributed manner