상세정보
학술대회
Real-Time Data Flow Language Processing System for Handling Streams of Data
- 발행일
- 201409
- 출처
- International Conference on Scalable Information Systems (INFOSCALE) 2014, pp.97-106
- 협약과제
- 14MI9200, (통합)스마트 네트워킹 핵심 기술 개발, 이병선
- 초록
- Apache Pig system generates MapReduce jobs by compiling program scripts written in Pig Latin to process large data sets in parallel on distributed computing nodes. There are inefficient features in Pig due to the limitation of the MapReduce, e.g., the MapReduce is used only for batch processing. As various smart devices are extensively utilized recently, streams of data are generated explosively and the need to process streams of data in real-time is required. In this paper, we propose a data flow language processing system, called LAMA-CEP, by generating DAG-based stream processing services to process unbounded streams of data in real-time continuously. We present a stream processing language, called Pig Latin Stream extended from Pig Latin. Programs written in Pig Latin Stream are translated into distributed stream processing jobs and then the jobs are executed on a highly scalable distributed stream processing system to process large streams of data in real-time.
- KSP 제안 키워드
- Apache Pig, Batch processing, Language processing, Large data sets, Real-time data flow, Smart devices, distributed computing, distributed stream processing system