Performance Evaluation of the Proposed Algorithms

In this Section, we provide some experimental results that we obtained in evaluating the proposed algorithms. We compare these results with those obtained for the gradient descent algorithm. For this purpose, we used two problems that have been already used in [1] for validation purposes. In addition, the XOR problem is also considered. The description of these problems is as follows: $\bullet$

• Pattern classification: in this problem, the goal is to classify two patterns [-1 1 -1 -1 1 1 1] and [1 -1 1 1 -1 -1 -1]. The desired outputs for these patterns are [1 0] and [0 1] respectively. As stated in [1], even though this problem may seem trivial, it is used in applications in decoding data coded with special sequences as in [2]. For this problem, we used the three-layer feedforward network consisting of 7 neurons in the input layer, 5 neurons in the output layer and 2 neurons in the output layer as shown in Figure 10.1. It should be mentioned that this problem has been used to show that RNN performs better than ANN. Even if ANN can converge in less time and iterations to the solution, RNN is the only one that can generalize well when a perturbation of the input is applied (inversion of one bit or week signal amplitude). For more details, refer to [1].
• Fully recurrent network: this problem concerns a 9-neuron single-layer fully recurrent network as shown in Figure 10.2. The goal is to use the NN to store the analog pattern [0.1 0.3 0.9 0.1 0.3 0.5 0.9 0.1] as a fixed attractor for the network. One of the possible applications of such a problem is the recognition of the normalized gray levels of some texture images. Hopfield recurrent training algorithm cannot be used in this case as the patterns are not binary. Further analysis of this problem by RNN is given in [1].
• The XOR problem: this problem is widely used in the literature by many researchers to evaluate the performance of the proposed training algorithms. This is because the binary-XOR problem is very hard for NN to learn. Here, the NN architecture we use for this problem is shown in Figure 10.3.

We have implemented the proposed algorithms in MATLAB. During their evaluation, we have noticed that when we allow negative values for the weights, we obtain better results. Thus, to deal with this point, we have considered three different procedures when a negative weight is produced. In the first one, whenever there is a negative value of the weight, we simply put it to zero. The second one is as proposed in [87]. To ensure that the weights are always positive, the relation $p^2=w_{i,j}$ is used. In this case, the above equations are slightly modified to take into account the derivatives with respect to the weights. The third case is to allow negative weights. We provide a comparative study for these cases. In the following analysis, Figures and Tables, we use the acronyms given bellow:

GD:

the basic gradient descent training algorithm, as proposed in [1].

LM:

the Levenberg-Marquardt training algorithm; in this case, we allow negative weights to take place during the training.

LM1:

the Levenberg-Marquardt training algorithm, but in this case, no negative weights are allowed: whenever a negative value of any weight is produced, we simply put it to zero.

LM2:

the Levenberg-Marquardt training algorithm, no negative weights allowed, by using $p^2=w_{i,j}$ and by modifying the derivatives of the equations.

AM-LM:

the Levenberg-Marquardt with adaptive momentum training algorithm. We allow negative weights to occur.

Figure 10.1: The 7-5-2 feedforward RNN network architecture.

  

Figure 10.2: The fully-connected recurrent RNN network architecture.