BROWSE
Detail
Conference Paper
Deep Neural Network Using Trainable Activation Functions
Cited 50 time in 
Share 
Share
- Authors
- Hoon Chung, Sung Joo Lee, Jeon Gue Park
- Issue Date
- 2016-07
- Citation
- International Joint Conference on Neural Networks (IJCNN) 2016, pp.348-352
- Language
- English
- Type
- Conference Paper
- Abstract
- This paper proposes trainable activation functions for deep neural network (DNN). A DNN is a feed-forward neural network composed of more than one hidden nonlinear layer. It is characterized by a set of weight matrices, bias vectors, and a nonlinear activation function. In model parameter training, weight matrices and bias vectors are updated using an error back-propagation algorithm but activation functions is not. It is just fixed empirically. Many rectifier-type nonlinear functions have been proposed as activation functions, but the best nonlinear functions for any given task domain remain unknown. In order to address the issue, we propose a trainable activation function. In the proposed approach, conventional nonlinear activation functions were approximated for a Taylor series, and the coefficients were retrained simultaneously with other parameters. The effectiveness of the proposed approach was evaluated for MNIST handwritten digit recognition domain.
- KSP Keywords
- Back Propagation(BP), Back Propagation Algorithm, Deep neural network(DNN), Error back-propagation, Feedforward neural networks, Model parameter, Taylor Series, handwritten digit recognition, nonlinear activation function, nonlinear function, weight matrix