Analytical Formulation of the Algorithm

Following Gelenbe's notations, let us write

$$N_i=\lambda^+_i+\sum_j \varrho_jw^+_{i,j},$$ (1010)

$$D_i=r_i+\lambda^-_i+\sum_j \varrho_jw^-_{i,j},$$ (1011)

$$\varrho_i=N_i/D_i,$$ (1012)

where $i\in 1,\ldots, n$ represents the neuron index, $w^-, w^+$ the weights, $\lambda^+_i, \lambda^-_i$ the excitation and inhibition external signals for neuron $i$, and $\varrho_i$ the output of the neuron $i$. Define

$$E=\sum_{k=1}^K E^{(k)},$$ (1013)

recalling that

$$E^{(k)}= \textstyle \frac{1}{2} \displaystyle \sum_{i=1}^n a_i(\varrho_i^{(k)}-y_i^{(k)})^2, \quad a_i \geq 0.$$ (1014)

The mathematical formulation of LM method applied to RNN is as follows: $\bullet$

• We define a generic vector $p$, of $M$ elements, containing the adjustable parameters $w^+_{i,j}$ and $w^-_{i,j}$; $p=(p_1, \ldots, p_M)$; $p^{(k)}$ is the parameter vector at step $(k)$ of the training process.
• We define also $g=(g_1, \ldots, g_M)$ the gradient vector, where $g_l=\partial E/ \partial p_l$ for $l=1, \ldots, M$. Denote by $g^{(k)}$ the gradient vector at point $p^{(k)}$.
• The weights update based on Newton method is as follows:
  

  $$p^{(k+1)}=p^{(k)} +s^{(k)},$$ (1015)

  
  
  where $s^{(k)}$ is the Newton's direction obtained by solving the system
  

  $$\mathbf{H}^{(k)}s^{(k)} = -g ^{(k)},$$ (1016)

  
  
  $\mathbf{H}^{(k)}$ being the Hessian matrix at step $k$. For LM, the Hesssian matrix $\mathbf{H}^{(k)}$ is approximated by:
  

  $$\mathbf{H}^{(k)}=\mathbf{J}^{{(k)}^T}\mathbf{J}^{(k)}+\mu^{(k)}\mathbf{I}.$$ (1017)

  
  
  Here, $\mu^{(k)}>0$ and $J^{(k)}$ is the Jacobian matrix at step $k$, given by
  

  $$\mathbf{J}=\left[ \begin{array}{ccc} \frac{\partial e... ...frac{\partial e_N}{\partial p_M} \\ \end{array} \right],$$ (1018)

  
  
  where $e_i=y_i-\varrho_i$ is the prediction error of neuron $i$, $\varrho_i$ is the output of neuron $i$ at the output layer, $y_i$ is the desired output of neuron $i$ at the output layer, and $N$ is the number of outputs multiplied by the number of training examples $K$. Each element of matrix $J$ is computed using the following equations:
  

  $$\mathbf{J}_{l,m}=\partial e_l / \partial p_m =- \partial ... ...ad\mbox{ where } l=1, \ldots, N \mbox{ and } m=1, \ldots, M,$$ (1019)

  
  
  
  

  $$\partial \mathbf{\varrho} / \partial p_m= \left\{ \begi... ...{-1} & \mbox{if} & p_m=w^-_{u,v},\\ \end{array}. \right.$$ (1020)

  
  
  From Eq. 10.16, we obtain:
  

  $$s^{(k)}=-[\mathbf{H}^{(k)}]^{-1} g^{(k)}.$$ (1021)

  
  
  Equations 10.17 and 10.21 give:
  

  $$s^{(k)}=-[\mathbf{J}^{{(k)}^T}\mathbf{J}^{(k)}+\mu^{(k)}\mathbf{I}]^{-1} g^{(k)}.$$ (1022)

  
  
  By grouping Equations 10.15 and 10.22 we obtain:
  

  $$p^{(k+1)}=p^{(k)}-[\mathbf{J}^{{(k)}^T}\mathbf{J}^{(k)}+\mu^{(k)}\mathbf{I}]^{-1} g^{(k)}.$$ (1023)

  
  
  To compute vector $g$, we use:
  

  $$g=\mathbf{J}^T e,$$ (1024)

  
  
  where $e=(e_1, \ldots, e_N)$.