A Study on the Performance Improvement of Adaptive Learning-Rate Algorithm in Neural Network

Abstract

In this thesis, improving the performance of adaptive learning-rate algorithms in neural network for predicting time-series data is studied.

In Chapter 1, background of artificial neural network for predicting time-series data is introduced to help understand the thesis. First, the basic architecture of artificial neural network is given. Next, the conventional adaptive learning-rate algorithms applied to artificial neural network are briefly described.

In Chapter 2, a robust adaptive learning-rate algorithm based on the mean-square-deviation analysis (ALR-MSD) is introduced. The ALR-MSD algorithm updates the learning rate in the direction of minimizing MSD at each hidden layer and it shows that minimizing the MSD can effectively reduce the overall error of the neural network for the first time. The problem that exact value of the MSD at each hidden layer is not feasible is solved by setting the upper bound of the MSD and reinterpreting the backpropagation error as the perspective of the weight deviation. In addition, to be robustness to the impulsive noises, the ALR-MSD algorithm adopts the variance of the disturbance at each hidden layer through the variance of the error signals. The results of the two simulations show how robust and excellent the ALR-MSD algorithm was even in the case of the impulsive noise.

In Chapter 3, another ALR-MSD algorithm based on the novel analysis of the MSD is introduced to improve the performance in terms of prediction accuracy, convergence rate, and robustness to the missing data. As in the Chapter 2, the backpropagation errors are also interpreted as the perspective of the weight deviations, and a recursive equation of the upper bound MSD in the hidden layer is derived by this interpretation. However, in the Chapter 3, the final update equation of the learning rate is differently derived by analyzing the MSD in a new way that can remove the variance of the disturbance generated in the deriving process of the previous MSD. The experimental results show that the network with the ALR-MSD algorithm outperforms not only the accuracy and convergence rate but also the robustness to the missing data than the networks with the comparison algorithms.

In Chapter 4, for robustness against defective time-series data, gradient scaler-based optimization algorithms are introduced. The novel ${log}$ cost function is proposed to derive the gradient scaler (GS), which instantly scales down the gradient whenever the training undergoes with the outlier or missing samples. The GS is easily applied to various gradient-based optimization algorithms such as the SGD, RMSProp and Adam. The experiment results show that the GS-based optimization algorithms outperform the original gradient-based optimization algorithms for all datasets.