상세정보
학술대회
An Implementation of a High Throughput Data Ingestion System for Machine Logs in Manufacturing Industry
- 발행일
- 201607
- 출처
- International Conference on Ubiquitous and Future Networks (ICUFN) 2016, pp.117-120
- 협약과제
- 15PC5200, 중소 제조산업의 4M (Man, Machine, Materiel, Method) 데이터 통합 분석을 활용한 프리틱디브 매뉴펙춰링 시스템 개발 , 지수영
- 초록
- This paper aims at presenting a case study of designing and implementing a data ingestion system for manufacturers. In our implementation, clustered server architecture for high throughput data ingestion is proposed with regard to following factors: receiving stream data, i.e., machine logs, from a set of milling machines, storing them in a centralized messaging queue, and sinking to external systems with ease. Especially, we leverage the power of the open sources frameworks, Apache Kafka, Apache Hadoop File System and Apache Flume to cope with the data streams from a large number of machines in the factory floors. As this is an on-going study, we only illustrate our implementation details with structural diagrams, but exclude the theoretical study and the performance evaluation results in this paper.
- KSP 제안 키워드
- Apache Flume, Apache Hadoop, Apache Kafka, Case studies, Data ingestion, Data stream, File System, High throughput(HTP), Open source, Performance evaluation, Stream Data