ETRI-Knowledge Sharing Plaform

ENGLISH

성과물

논문 검색
구분 SCI
연도 ~ 키워드

상세정보

학술지 Ensemble Three-Stream RGB-S Deep Neural Network for Human Behavior Recognition Under Intelligent Home Service Robot Environments
Cited 2 time in scopus Download 40 time Share share facebook twitter linkedin kakaostory
저자
변영현, 김도형, 이재연, 곽근창
발행일
202105
출처
IEEE Access, v.9, pp.73240-73250
ISSN
2169-3536
출판사
IEEE
DOI
https://dx.doi.org/10.1109/ACCESS.2021.3077487
협약과제
21HS1500, 고령 사회에 대응하기 위한 실환경 휴먼케어 로봇 기술 개발, 이재연
초록
This paper presents a method for recognizing behaviors in videos based on the ensemble RGB-S deep neural network, which combines RGB images and skeleton features from an action recognition database built in intelligent home service robot environments. The ensemble model is designed using the three-stream approach. The first stream classifies behaviors in videos using a convolutional neural network (CNN) based on a pre-trained ResNet101 model, which uses two-dimensional (2D) sequence images of actions as its input, and training a long short-term memory (LSTM) neural network with the sequence (RGB 2D-CNN + LSTM). The second stream directly manages the video and uses a three-dimensional (3D) CNN to include both temporal and spatial information. The 3D CNN is based on a pre-trained R3D-18 model (RGB 3D-CNN). The last stream uses the pose evolution image (PEI) method, which converts the skeleton sequence into a single-color image. The converted images are used as the input for the CNN (Skeleton PEI-2D-CNN). This approach not only reflects the spatial and temporal features of the behaviors in videos, but also includes all characteristics of the 2D sequence images, 3D videos, and skeleton sequences. Finally, a large-scale database for behavior recognition in videos, known as ETRI-Activity3D, is used in this study to verify the performance of the proposed deep neural network. A recognition performance of 93.2% is achieved in a cross-subject experiment, verifying the superiority of this method over models from previous studies.
키워드
Ensemble RGB-S deep neural network, ETRI-Activity3D database, human behavior recognition, transfer learning
KSP 제안 키워드
3D CNN, 3D Video, Action recognition, Color images, Convolution neural network(CNN), Deep neural network(DNN), Ensemble models, Human behavior recognition, Large-scale database, Long short-term memory, RGB image
본 저작물은 크리에이티브 커먼즈 저작자 표시 (CC BY) 조건에 따라 이용할 수 있습니다.
저작자 표시 (CC BY)