$\zeta$ and $dP$ Parameters for AM-LM

In the GD and other learning algorithms, the value of the learning rate $\eta$ has to be chosen. The convergence time and the accuracy depend on this choice. The optimal value of this parameter varies from one problem to another and from network architecture to another; however, most of the implementations set its value to 0.1. The AM-LM has two parameters ($\zeta$ and $dP$) to be determined, instead of one. In [87], the authors studied this problem and they determined through simulations ``ideal'' ranges for these parameters: $\zeta \in [0.85 \, ,0.95]$ and $dP \in [0.1 \, , 0.6]$. We will show the performance of this algorithm when adapted to RNN for the first problem and for these ranges. To do that, for each pair of values, we counted the number of times the algorithm converges to a predefined mean square error MSE value ( $5\times 10^{-5}$). For each case, the total number of tests is fixed to 100. We show in Figure 10.4 the results for the first problem. As we can see from these Figures, the values of these parameters effectively affect the performance of the algorithm. The best range for $dP$ was [0.4, 0.6]. However, the impact of $\zeta$ appears to be random and not easy to characterize. In addition, we can note that the impact of $dP$ is more significant than that of $\zeta$. Thus, we propose that this parameter must be fixed to 0.90 and that only $dP$ must be changed.

Figure 10.3: The RNN network architecture used to solve the XOR problem.

  

Figure 10.4: The impact of the two variables $\zeta$ and $dP$ on the performance of the adaptive momentum LM training algorithm for RNN. The results for the first problem.