What to remember: How memory order affects the performance of NARX neural networks

It has been shown that gradient-descent learning can be more effective in NARX networks than in other recurrent neural networks that have `hidden states' on problems such as grammatical inference and nonlinear system identification. For these problems, NARX neural networks can converge faster and generalize better. Part of the reason can be attributed to the embedded memory of NARX networks, which can reduce the network's sensitivity to long-term dependencies. In this paper, we explore experimentally the effect of the order of embedded memory of NARX networks on learning ability and generalization performance for the problems above. We show that the embedded memory plays a crucial role in learning and generalization. In particular, generalization performance could be seriously deficient if the embedded memory is either inadequate or unnecessary prodigal but is quite good if the order of the network is similar to that of the problem.