Two-Phase Pattern Search-based Learning Method for Multi-layer Neural Network

A new multi-layer artificial neural network learning algorithm based on pattern search method is proposed. The learning model has two phases-a pattern search phase, and a local minimum-escaping phase. In the pattern search phase, our method performs local search iteratively and minimize the error measure function along with the set of descent directions of the error measure directly and finds the nearest minima efficiently. When the network gets stuck in local minima, the local minimum-escaping phase attempts to fill up the valley by modifying temperature parameters in ascent direction of the error measure. Thus, the two phases are repeated until the network gets out of local minima. The learning model is designed to provide a very simple and effective means of searching the minima of objective function directly without any knowledge of its derivatives. We test this algorithm on benchmark problems, such as exclusive-or (XOR), parity, Arabic numerals recognition, function approximation problems and a real world classification task. For all problems, the systems are shown be trained efficiently by our method. As a simple direct search method, it can be applied in hardware implementations easily.