BROWSE
Detail
Conference Paper
An Implementation of a High Throughput Data Ingestion System for Machine Logs in Manufacturing Industry
Cited 11 time in 
Share 
Share
- Authors
- Jaehui Park, Su-young Chi
- Issue Date
- 2016-07
- Citation
- International Conference on Ubiquitous and Future Networks (ICUFN) 2016, pp.117-120
- Publisher
- IEEE
- Language
- English
- Type
- Conference Paper
- Abstract
- This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.
- KSP Keywords
- Apache Flume, Apache Hadoop, Apache Kafka, Case studies, Data ingestion, Data stream, File System, High throughput(HTP), Milling machines, Open source, Performance evaluation