상세정보
학술대회
Deep Neural Network Using Trainable Activation Functions
- 발행일
- 201607
- 출처
- International Joint Conference on Neural Networks (IJCNN) 2016, pp.348-352
- 협약과제
- 16MS1700, 언어학습을 위한 자유발화형 음성대화처리 원천기술 개발, 이윤근
- 초록
- This paper proposes trainable activation functions for deep neural network (DNN). A DNN is a feed-forward neural network composed of more than one hidden nonlinear layer. It is characterized by a set of weight matrices, bias vectors, and a nonlinear activation function. In model parameter training, weight matrices and bias vectors are updated using an error back-propagation algorithm but activation functions is not. It is just fixed empirically. Many rectifier-type nonlinear functions have been proposed as activation functions, but the best nonlinear functions for any given task domain remain unknown. In order to address the issue, we propose a trainable activation function. In the proposed approach, conventional nonlinear activation functions were approximated for a Taylor series, and the coefficients were retrained simultaneously with other parameters. The effectiveness of the proposed approach was evaluated for MNIST handwritten digit recognition domain.
- KSP 제안 키워드
- Back Propagation(BP), Deep neural network(DNN), Error back-propagation, Feedforward neural networks, Model parameter, Taylor Series, backpropagation algorithm, handwritten digit recognition, nonlinear activation function, nonlinear function, weight matrix