| C++ Neural Networks and Fuzzy Logic |
by Valluru B. Rao
M&T Books, IDG Books Worldwide, Inc.
ISBN: 1558515526 Pub Date: 06/01/95
|Previous||Table of Contents||Next|
In the last chapter, we presented an overview of different neural network models. In this chapter, we continue the broad discussion of neural networks with two important topics: Learning and Training. Here are key questions that we would like to answer:
There are many varieties of neural networks. In the final analysis, as we have discussed briefly in Chapter 4 on network modeling, all neural networks do one or more of the following :
A neural network, in any of the previous tasks, maps a set of inputs to a set of outputs. This nonlinear mapping can be thought of as a multidimensional mapping surface. The objective of learning is to mold the mapping surface according to a desired response, either with or without an explicit training process.
A network can learn when training is used, or the network can learn also in the absence of training. The difference between supervised and unsupervised training is that, in the former case, external prototypes are used as target outputs for specific inputs, and the network is given a learning algorithm to follow and calculate new connection weights that bring the output closer to the target output. Unsupervised learning is the sort of learning that takes place without a teacher. For example, when you are finding your way out of a labyrinth, no teacher is present. You learn from the responses or events that develop as you try to feel your way through the maze. For neural networks, in the unsupervised case, a learning algorithm may be given but target outputs are not given. In such a case, data input to the network gets clustered together; similar input stimuli cause similar responses.
When a neural network model is developed and an appropriate learning algorithm is proposed, it would be based on the theory supporting the model. Since the dynamics of the operation of the neural network is under study, the learning equations are initially formulated in terms of differential equations. After solving the differential equations, and using any initial conditions that are available, the algorithm could be simplified to consist of an algebraic equation for the changes in the weights. These simple forms of learning equations are available for your neural networks.
At this point of our discussion you need to know what learning algorithms are available, and what they look like. We will now discuss two main rules for learningHebbian learning, used with unsupervised learning and the delta rule, used with supervised learning. Adaptations of these by simple modifications to suit a particular context generate many other learning rules in use today. Following the discussion of these two rules, we present variations for each of the two classes of learning: supervised learning and unsupervised learning.
Learning algorithms are usually referred to as learning rules. The foremost such rule is due to Donald Hebb. Hebbs rule is a statement about how the firing of one neuron, which has a role in the determination of the activation of another neuron, affects the first neurons influence on the activation of the second neuron, especially if it is done in a repetitive manner. As a learning rule, Hebbs observation translates into a formula for the difference in a connection weight between two neurons from one iteration to the next, as a constant [mu] times the product of activations of the two neurons. How a connection weight is to be modified is what the learning rule suggests. In the case of Hebbs rule, it is adding the quantity [mu]aiaj, where ai is the activation of the ith neuron, and aj is the activation of the jth neuron to the connection weight between the ith and jth neurons. The constant [mu] itself is referred to as the learning rate. The following equation using the notation just described, states it succinctly:
[Delta]wij = [mu]aiaj
As you can see, the learning rule derived from Hebbs rule is quite simple and is used in both simple and more involved networks. Some modify this rule by replacing the quantity ai with its deviation from the average of all as and, similarly, replacing aj by a corresponding quantity. Such rule variations can yield rules better suited to different situations.
For example, the output of a neural network being the activations of its output layer neurons, the Hebbian learning rule in the case of a perceptron takes the form of adjusting the weights by adding [mu] times the difference between the output and the target. Sometimes a situation arises where some unlearning is required for some neurons. In this case a reverse Hebbian rule is used in which the quantity [mu]aiaj is subtracted from the connection weight under question, which in effect is employing a negative learning rate.
In the Hopfield network of Chapter 1, there is a single layer with all neurons fully interconnected. Suppose each neurons output is either a + 1 or a 1. If we take [mu] = 1 in the Hebbian rule, the resulting modification of the connection weights can be described as follows: add 1 to the weight, if both neuron outputs match, that is, both are +1 or 1. And if they do not match (meaning one of them has output +1 and the other has 1), then subtract 1 from the weight.
|Previous||Table of Contents||Next|
Copyright © IDG Books Worldwide, Inc.