Ch02 Neural network.docx

CHAPTER TWO NEURAL NETWORKS 2.1 Overview This chapter presents an explanation of Artificial Neural Networks (ANNs). The importance of ANNs in the technological development and how it works. Inspiration of the system from the human brain. ANNs architectural and components. The process of training and testing for the ANNs. Their algorithms and laws of training will be discussed. The backpropagation learning algorithm which is used to train the dental identification system will be described. 2.2 Artificial Neural Networks (ANNs) An ANN is a mathematical model designed for simulation of nerves system in the human brain. It is a machine that is designed to model the way in the brain performs a particular task or function of interest. The network is usually implemented by using electronic components or is simulated in software on a digital computer. Each cell of this mathematical model is simple, but when it met together it will be able to produce complex results [12]. During the time with the development of human knowledge of the human brain and try to make a mathematical model that simulates the brain system. Create a system to solve the problems of artificial intelligence without necessarily need to creating of a real biological system. Here is the beginning of work on artificial neural networks. McCullouch and Pitts in 1943, created a computational model for neural networks based on mathematics and algorithms. They called this model threshold logic. The model paved the way for neural network research to split into two distinct approaches. One approach focused on biological processes in the brain and the other focused on the application of neural networks to artificial intelligence [13]. In the late 1940s psychologist Donald Hebb created a hypothesis of learning based on the mechanism of neural plasticity that is now known as Hebbian learning. Hebbian learning is considered to be a 'typical' unsupervised learning rule and its later variants were early models for long term potentiation. These ideas started being applied to computational models in 1948 with Turing's B-type machines [14]. Farley and Clark in 1954, first used computational machines, and then called calculators, to simulate a Hebbian network at Massachusetts Institute of Technology (MIT). Other neural network computational machines were created by Rochester, Holland, Habit, and Duda in 1956 [15] [16]. Rosenblatt in 1962, created the perceptron, an algorithm for pattern recognition based on a two-layer learning computer network using simple addition and subtraction. With mathematical notation, Rosenblatt also described circuitry not in the basic perceptron, such as the exclusive-or circuit. A circuit whose mathematical computation could not be processed until after the backpropagation algorithm was created by Werbos in 1975 [15] [17]. Neural network research stagnated after the publication of machine learning research by Minsky and Papert in 1969. They discovered two key issues with the computational machines that processed neural networks. The first issue was that single-layer neural networks were incapable of processing the exclusive-or circuit. The second significant issue was that computers were not sophisticated enough to handle effectively the long run time required by large neural networks. Neural network research slowed until computers achieved greater processing power. Also key in later advances was the backpropagation algorithm which effectively solved the exclusive-or problem [18]. The cognition in 1975, designed by Kunihiko Fukushima was an early multilayered neural network with a training algorithm. The actual structure of the network and the methods used to set the interconnection weights change from one neural strategy to another, each with its advantages and disadvantages. Networks can propagate information in one direction only, or they can bounce back and forth until self-activation at a node occurs and the network settles on a final state. The ability for bidirectional flow of inputs between neurons/nodes was produced with the Hopfield's network in 1982, and specialization of these node layers for specific purposes was introduced through the first hybrid network [19]. The parallel distributed processing of the mid-1980s became popular under the name connectionism. The text by Rummelhart and McClelland in 1986 provided a full exposition on the use of connectionism in computers to simulate neural processes [20]. The rediscovery of the backpropagation algorithm was probably the main reason behind the repopularisation of neural networks after the publication of "Learning Internal Representations by Error Propagation" in 1986 (Though backpropagation itself dates from 1969). Training was done by a form of stochastic gradient descent. The employment of the chain rule of differentiation in deriving the appropriate parameter updates results in an algorithm that seems to 'backpropagate errors', hence the nomenclature. However it is essentially a form of gradient descent. Determining the optimal parameters in a model of this type is not trivial. And local numerical optimization methods such as gradient descent can be sensitive to initialization because of the presence of local minima of the training criterion. In recent times, networks with the same architecture as the backpropagation network are referred to as multilayer perceptrons (MLP). This name does not impose any limitations on the type of algorithm used for learning [20]. The back propagation network generated much enthusiasm at the time and there was much controversy about whether such learning could be implemented in the brain or not. Partially because a mechanism for reverse signaling was not obvious at the time, but most importantly because there was no plausible source for the 'teaching' or 'target' signal. However, since 2006, several unsupervised learning procedures have been proposed for neural networks with one or more layers, using "deep learning" algorithms. These algorithms can be used to learn intermediate representations, with or without a target signal, that capture the salient features of the distribution of sensory signals arriving at each layer of the neural network [21]. 2.3 Human Brain The sonar of a bat is an active echo-location system, in additional to providing information about how far away a target is. A bate sonar load information about the relative speed of the target, the size of the target, the size of different features of the target, the azimuth and elevation of the target. The complex neural computations needed to extract all this information from the target echoes occur within a brain the size of a berry. The bat sonar system can achieve a success rate accuracy excellence and larger than all the radar and sonar systems that made by engineering. The human nervous system can be viewed and represented as a system consists of three stages. The first stage is the central brain, which is the main combination of the nervous system in humans, which receives information continuously and responds to them and take the appropriate decisions. While the second stage, which is a set of lines and lanes, which represents the transfer of data from sensors in all parts of the body to the brain as an electrical pulse. And feed the information through the system, vice versa in the process of system response the lines represent the reaction of the system to a particular input and response quickly. While the third stage is the sensors system deployed in the all over the body and which are converting external stimuli to an electrical pulse that transmits information to the brain, and effectors shift electrical impulses generated by the neural networks as a significant response to the output of the system as shown in figure 2.1 [22]. -114300184785Receptors Effectors Stimulus Response Trained Neural Net 00Receptors Effectors Stimulus Response Trained Neural Net Figure 2.1 Block Diagram Representing Nervous System [22]. The human brain is made up of a wide network of computing elements called (neurons), coupled with sensory receptors and effectors. The average human brain approximately contains about 100 billion cell of various types. Neurons are a special cell that conducts an electrical signal. There are about 10 billion neurons in the human brain. The remaining 90 billion on cells are called glial or glue cell and these serve as support cells for the neurons. Neurons interact through contact links called synapses. Each synapse spans a gap about a millionth of an inch wide. On the average each neuron receives signals via thousands of synapses [19] [22]. In fact, the neurons are five or six times slower than current of silicon gates. The modern computer easily outperforms the human in reprogrammable, repetitive computations quickly than human do, but in real applications such as understanding, perception, speech, which human being almost effortlessly implements, are still beyond the reach of serial digital computers even after allowing for an increase of speed. This task is difficult for the serial digital computers because the load requires speed not realizable with existing technology currently [22]. Many of the facts are still unknown about how the brain works and how it trains itself to process information. The learning process occurs by changing the effectiveness of the synapses between nerve cells, and by adding or deleting connections between neurons. 2.4 Artificial Neural Network Components An Artificial Neural Network (ANN) is a massively parallel distributing processor made up of simple processing unites which has natural propensity for storing experiential knowledge and making it available for use. It resembles the brain in two respects [23]: Knowledge is acquired by the network from its environment through a learning process. Inter neuron connection strengths known as synaptic weights are used to store the acquired knowledge. 2.5 Models of Neurons A neuron is an information- processing unit that is fundamental to the operation of a neural network. The first attempts to simulate neural networks in the nervous system are to draw the main features of neurons and how interconnection and programmed it by a computer program to simulate these features. But our knowledge of nerves system is not great enough and our potential technological are limited, in figure 2.2 below the mathematical model to simulate neurons [24] [25]. Figure 2.2 Neurons Mathematical Model [22]. There are three basic elements of the neurons model as shown in figure 2.4 [24]: 1. A set of synapses of connecting links, each of which characterized by a weight or strength of its own. 2. An adder for summing the input signal, weighted by the respective synapses of the neuron, the operations described here constitute a linear combiner. 3. An activation function for limiting the amplitude of the output of a neuron. The activation function is also referred to in the literature as a squashing function in that it squashed (limits) the permissible amplitude range of the output signal to some finite value. Typically the normalized amplitude rang of the output of a neuron is written as the closed unit interval [0 1] or alternatively [-1 1]. -5715090170 X1 X2 Xp Input Signals Synaptic Weights Wk1 Wk2 Wkp Summing Junction Activation Function Threshold Output 00 X1 X2 Xp Input Signals Synaptic Weights Wk1 Wk2 Wkp Summing Junction Activation Function Threshold Output Figure 2.3 Neurons Network Elements [22]. 2.5.1 Types of Activation Function The activation function, defines the output of a neuron in terms of the activity level at its input. There are three basic types of activation function [22] [24]: Threshold function: in Engineering literature this form of a threshold function is commonly referred to as a Heaviside function. Correspondingly, the output of neuron K employing such a threshold function is expressed as (2.1) McCulloch-Pitts model, create a model in recognition of the pioneering work done by McCulloch and Pitts (1943). In this model the output of a neuron takes on the value of 1 if the induced local field of that neuron is nonnegative and 0 otherwise as shown in figure 2.4. Figure 2.4 Threshold Activation Function [22]. Piecewise linear function: The amplification factor inside the linear region is supposed to be unity. This type of an activation function can be viewed as approximation to a nonlinear amplifier. The following two situations may be viewed as special forms of the piecewise linear function [22]: A linear combiner arises if the linear region of operation is maintained without running into saturation. The piecewise linear function reduces to a threshold function if the amplification doctor of the linear region is made infinitely large. The output of neuron that uses the piecewise linear function is expressed as (2.2) Figure 2.5 shows the Piecewise linear activation function Figure 2.5 Piecewise Linear Activation Function [2]. Sigmoid function: the sigmoid function is the most commonly form of activation function that used in the construction of artificial neural networks. It can be defined as accurate increase function of increasing and accurate. There is a balance between linear and nonlinear behavior [24]. There is a slop parameter of the sigmoid function (a). By varying the parameter a, we obtain sigmoid function of different slops. When we make the slop parameter approaches infinity, the sigmoid function become a threshold function. Where the threshold function range value is 0 or 1, the sigmoid function has a continuous range of value from 0 to 1. The equation below shows the sigmoid activation function representation and how the output value calculated. (2.3) The sigmoid activation function rang is between 0 to +1, but sometimes the range that used in this function is between -1 to +1, in some cases, which is assumes an anti symmetric with respect to the origin. The activation function is an odd function the induces local field. Figure 2.6 shows the sigmoid activation function. Figure 2.6 Sigmoid Activation Function [2]. 2.6 Neural Network Architectures First we must define the perceptron and what it represents. The perceptron can be seen as the simplest kind of artificial neural network it basically consist of a single neuron with the adjustable synaptic weight as shown in figure 2.7 [19] [24]. Figure 2.7 The Perceptron [26]. We can classify the neural network architectures into three classes: Single layer perceptron feed forward networks (SLP). Multi layer perceptron feed forward networks (MLP). Recurrent Networks. 2.6.1 Single Layer Perceptron (SLP) Artificial Neural Networks (ANNs) are organized the neurons in the form of successive layers of artificial neurons. The simplest form of these classes that we have is the input layer, which is the source layer and followed by the output layer or as it is called the computation nodes. In this network there is just a process of feeding forward but not vice versa and it cannot be feed backward networks. Figure 2.8 below showed the single layer perceptron neural network [27]. Figure 2.8 Single Layer Perceptron SLP [28]. 2.6.2 Multi Layer Perceptron (MLP) The second class of feed forward neural network distinguishes itself by one or more hidden node or hidden layers. Which is a computation node called the hidden neurons or hidden units. The objective and the function of hidden neurons are to intervene between the external inputs and the network outputs in some useful manner. By adding one or more hidden layers the neural networks can activate the industrial capacities of great learning and pattern recognition and many other tasks. This process depending on the number of hidden nodes by increase the size of hidden layer by add or decrease by delete hidden layers as shown in figure 2.9 [22] [27]. Figure 2.9 Multi Layer Perceptron MLP with Two Hidden Layers [22]. The source node in the input layer of the network supply elements to the computation neurons in the first hidden neurons in the next layer, which is her first hidden layer. Output of the second layer (first hidden layer) represents the input of the next layer, and so on for the rest of the neural network layers until the output layer which is the last computation layer in the neural network. All neurons in the neural network have inputs and also have outputs at the same time. Outputs of the last layer neurons (final layer) in artificial neural networks are the output of the network by processing the supplied input source node in the input (first) layer. The neural network said fully connected in the sense that every node in each layer of the network is connected to every other node in the adjacent forward layer. If some of the communication links (synaptic connections) are missing from the network, the network is partially connected [22]. 2.6.3 Recurrent Networks A recurrent neural network distinguishes itself from a feed forward neural network and it has at least one feedback loop. That means the output of the feed forward neural networks due to be input in the rest of the neurons and feedback in the same network but not to the same neurons itself. The presence of feedback loops in artificial neural network gives it the ability to learn from the network and affect its performance, the figure below shows the recurrent neural networks. Figure 2.10 Recurrent Networks [22]. 2.7 Artificial Neural Network Learning The significance of artificial neural network is the ability of this network to learn from its environment. The process of learning in artificial neural networks and also referred to as training process depends mainly on the pattern recognition. Where the artificial neural networks like human it learns through examples. The neural networks learn about the environment through adjusting the synaptic weight value and bias levels. There are two types of learning artificial neural networks first one called the supervised learning. In this type of learning, the learning process is under the supervision of the teacher, in this case the programmer. Where the teacher is determined the desired output or as called target in terms of inputs the learning rate depends on the response of artificial neural networks to the training, and how closes the actual output resulting from the training process of the desired output (target). The second type of learning is unsupervised learning. In this type the neural network doesn't need to a teacher to determine the output of the network [29] [30]. There are a large number of training algorithms that used to train artificial neural networks. The most important and best known and most widely used algorithm is Backpropagation training algorithm. This algorithm that was adopted to train the artificial neural network system in this thesis. 2.7.1 Learning Paradigms To make the learning process easier to understand. Consider a neural network consisting of one input neurons in input layer and one computation neurons (output layer) and one or more hidden layers. When the input node feed the network with the input signal it will be operations computation in the hidden layers to the arrival to the output layer. Then we have an actual output, represent the output of the neural network. We make a comparison between the actual output obtained from the neural network with the desired output or as called the target. This learning method is known as learning by error correction learning. The objective of this method is to create a sequence of adjustment on the synaptic weight of the network to make the actual output signal of the neural network come closer to the desired output or target. There are two types of learning paradigms [30]: 2.7.1.1 Supervised Learning Supervised leaning and also referred to learning with a teacher. In these paradigms the teacher must have knowledge of the environment surrounding the neural network. The teacher represented this knowledge as a sequence of input output examples. Suppose that the teacher wants to teach the neural network a particular process, the teacher must provide the neural network with the desired outputs according with the inputs in the training process. Certainly as we have already explain the training process done by modifying the values ??of the synaptic weights in the neural network then reduce the value of the actual output and compare it with desired outputs (target). Thus, step by step, the neural networks are simulating the teacher. In the training process knowledge transfer from the teacher to the neural networks. When the neural network gets to this stage the neural network dispenses the teacher and start working and deal with the environment completely by itself [30] [31]. 2.7.1.2 Unsupervised Learning (Self Organization Learning) Unsupervised learning or as called learning without teacher or self organization learning. Through the name of this learning paradigms that there is no teacher to supervise the learning process. That mean it has no examples to illustrate the environment surrounding the neural networks. Neural networks without any outside help, it response to the input signal that feeds and automatically respond to the features of these inputs, and make an adjustment of the synaptic weight , because of that, these networks have a different response for each set of input [32] [33]. 2.7.2 Learning Rule Neural networks are adaptive statistical devices. This means that they can change iteratively the values of the synaptic weights as a function of their performance. These changes are made according to learning rules which can be characterized as supervised (when a desired output is known and used to compute an error signal) or unsupervised (when no such error signal is used). Many learning rules are in common use. Most of these rules are some sort of variation of the best known and oldest learning rule, Hebb's Rule. Research into different learning functions continues as new ideas routinely show up in trade publications. Some researchers have the modeling of biological learning as their main objective. Others are experimenting with adaptations of their perceptions of how nature handles learning. Either way, researcher's understanding of how neural processing actually works is very limited. A few of the main learning rules are presented as examples: Hebb's Rule The first, and undoubtedly the best known, learning rule were introduced by Donald Hebb. The description appeared in his book The Organization of Behavior in 1949. His basic rule is: If a neuron receives an input from another neuron and if both are highly active (mathematically have the same sign), the weight between two neurons increase if the two neurons activate simultaneously and reduce if they activate separately [14] [15]. Hopfield Rule It is similar to Hebb's rule with the exception that it specifies the magnitude of the strengthening or weakening. It states, if the desired and actual output are either active or both inactive, increment the connection weight by the learning rate, and otherwise decrement the weight by the learning rate [20]. The Widrow-Hoff Learning Rule This rule is a further variation of Hebb's Rule. It is one of the most widely known supervised learning rules. This rule is based on the simple idea of continuously modifying the strengths of the input connections to reduce the difference (delta) between the desired output value and the actual output of a neural network in the output layer. This rule changes the synaptic weights in the way that minimizes the mean squared error (MSE) of the network. This rule is also referred to as the Least Mean Square (LMS) Learning Rule. The way that the delta rule works are that the delta error in the output layer is transformed by the derivative of the transfer function and is then used in the previous neural layer to adjust input connection weights. In other words, this error is back-propagated into previous layers one layer at a time. The process of back-propagating the network error continues until the first layer is reached and start feed forward process again. The network type called Feed forward, Back-propagation derives its name from this method of computing the error term [31] [36]. The Gradient Descent Rule The whole idea behind gradient descent is to gradually, but consistently, decreases the output error by adjusting the weights. The trick is to find out how to adjust the weights. Intuitively, we know that if a change in a weight will increase (decrease) the error, then we want to decrease (increase) that weight mathematically [31] [36]. Kohonen's Learning Rule This procedure, developed by Teuvo Kohonen, was inspired by learning in biological systems. The Kohonen rule allows the weights of a neuron to learn an input vector, and because of that, it is useful in recognition applications. Thus, the neuron whose weight vector was closest to the input vector is updated to be even closer. The result is that the winning neuron is more likely to win the competition the next time a similar vector is presented, and less likely to win when a very different input vector is presented. As more and more inputs are presented, each neuron in the layer closest to a group of input vectors soon adjusts its weight vector toward those input vectors [37]. Eventually, if there are enough neurons, every cluster of similar input vectors will have a neuron that output 1 when a vector in the cluster is presented, while outputting a 0 at all other times. Thus, the competitive network learns to categorize the input vectors it sees. 2.8 Advantages of Artificial Neural Networks Neural networks, with their remarkable ability to derive meaning from complicated or imprecise data, can be used to extract patterns and detect trends that are too complex to be noticed by either humans or other computer techniques. A trained neural network can be thought of as an "expert" in the category of information it has been given to analyze. This expert can then be used to provide projections given new situations of interest. Other advantages include [38]: Adaptive learning: An ability to learn how to do tasks based on the data given for training or initial experience. Self-Organization: An ANN can create its own organization or representation of the information it receives during learning time. Real Time Operation: ANN computations may be carried out in parallel, and special hardware devices are being designed and manufactured which take advantage of this capability. Fault Tolerance via Redundant Information Coding: Partial destruction of a network leads to the corresponding degradation of performance. However, some network capabilities may be retained even with major network damage. That what makes artificial neural networks (ANNs) the best choice to design an identification system for identifying human from their dental radiography images. 2.9 Back Propagation Algorithm (BP) Back propagation (BP) is the most common method of training artificial neural networks and it used to minimize the objective function. Arthur E. Bryson and Yu-Chi Ho described it as a multi-stage dynamic system optimization method in 1969 [39]. Then Minsky and Papert showed in 1969 that a two layer feed-forward network can overcome many restrictions, but did not present a solution to the problem of how to adjust the weights from input to hidden units. Then in 1974 when applied in the context of neural networks. And through the work of Paul Werbos, and David E. Rumelhart, Geoffrey E. Hinton and Ronald J.Williams, in 1986, that it gained recognition, and it led to a “renaissance” in the field of artificial neural network research [19] [21] [40] . It is a supervised learning method. It requires a dataset of the desired output for many inputs, making up the training set. It is most useful to train the neural network with the feed-forward networks and feed backward networks. The term is an abbreviation for "backward propagation of errors". Backpropagation requires that the activation function used by the artificial neurons be differentiable. The backpropagation algorithm knew as error-back propagation algorithm and this algorithm based on error-correction learning rule. The backpropagation algorithm consist of two basic passes through the network layers, forward pass and backward pass as shown in figure 2.11. Output Layer Forward Pass Hidden Layers Input Layer Backward Pass Output Layer Forward Pass Hidden Layers Input Layer Backward Pass Figure 2.11 Back Propagation Network Architecture. BP algorithm, calculated the output signal from forward network  depending on the error-correction rule, at this stage the synaptic weight does not change and remains constant, after comparing the output error signal or actual output with the desired output (target) and then propagate the error signal backward through the network and adjust the synaptic weight. Hence the name of the backpropagation algorithm comes from. This process will continues to make the actual output closer to the desired output. The steps of back propagation algorithm are: Feed Forward: Each input pattern in a training set is applied to the input unites and then propagated forward. Initialize hidden and output weights to small random values. Calculate outputs of hidden neurons. Calculate outputs of output neurons. Make differences between the actual outputs of the output neurons and targets to get the error signal. If the actual output is equal or less than the target then the neural network was trained otherwise it must propagating the error signal backward. Feed Backward: The error signal for each output pattern is back-propagated from the outputs to the input in order to adjust the synaptic weights in each hidden and output layers of the network. Repeat the above two passes to make the actual output closer to the desired output and repeated it many times until reaching the error goal. After a back propagation learning algorithm was learned the network. The network can be tested with a second set of input to see how well it trained and calculate the network accuracy. 2.9.1 Feed Forward Network Stage When the feedforward process started to training artificial neural network by using backpropagation learning algorithm, the input signal of the input units are passed forward through the network to the output layer through the connecting weights. The starting point for most neural networks is a source neuron, as shown in figure 2.12. 362585458470x1 x2 xi Neuron Sigmoid Function wji wj1 wj2 netj Oj 00x1 x2 xi Neuron Sigmoid Function wji wj1 wj2 netj Oj Feed Forward Propagation Feed Forward Propagation Figure 2.12 Feed Forward Neuron Structure. This neuron consists of multiple inputs and a single output. Each input is modified by a weight, which multiplies with the input value. The neuron will combine these weighted inputs with the activation function determines its output. The idea of feed forward network is passing inputs forward, all the outputs are calculated through the activation function, in most cases it was usually a sigmoid function, as shown in figure 2.6. The output of any neuron in any layer is described by two sets of equations: (2.4) (2.5) For every neuron, xi, in a layer, each input neuron is multiplied by a previously established weight wxi. All the result of all the neurons of the same layer is summed together, resulting in the internal value of this operation, net. Then this value was feed through an activation function, fth. The resulting output, Oj, is an input to the next layer or it is an actual output of the neural network if it is the last layer. The output of the neuron with sigmoid activation function is given by equation (2.6) 2.9.2 Error Signal The calculation of the feed forward network during the training process is combined with backward propagation the error signal to weight adjustment calculations that represent the network's learning. The concept of training a neural network is the definition of network error. Rumelhart and McClelland define an error term that depends on the difference between the actual output values of the network and the desired output that is supposed to have, called the target. The actual output is the result of the feed forward calculations. The error term represents a measure of how well a network is training on a particular training set [21]. To minimize the total error on the patterns in the training set, gradient descent method is used; weights are changed in proportion to the negative of an error derivative as shown in the equation: (2.7) Where is the learning rate and is the total error. In the next algorithm step the output signal of the network which is found in training data set, is compared with the desired output value (target). The difference is called error signal of output layer neuron. This signal feed backward through the network for adjust the weight. 2.9.3 Mean Squared Error Function Mean Squared Error Function (MSE) is the most commonly error function used in Back Propagation learning algorithm. The error is calculated using the gradient descent method. It is used to minimize the MSE between the actual output computed by the network and the desired output, for all possible input. The MSE function is defined as: (2.8) Where tkj is the target value from output node (k) to hidden node (j) Okj is the actual network value from output node (k) to hidden node (j) 2.9.4 Learning Rate and Momentum Factor These parameters are effecting to the learning capability of the neural network. First the learning rate coefficient () which defines the network teaching speed. It is factoring in considering the size of weights adjustment which done in each iteration and contributes to the convergence rate between the actual output and the target. If the learning rate is large number, huge change in weight make a fluctuate in the search path and convergence more slowly than a direct descent. When the steps were small which significantly will boost the total time to converge if the learning rate is too small. Second is the momentum factor () which defines the speed at which the neural network learns. This can be adjusted to a certain value in order to prevent the neural network from getting caught in what is called local energy minima and it could help in smoothing out the descent path by preventing the extreme change in weight. Both the learning rate and momentum factor can have a value between 0 and 1 [40] [41]. 2.9.5 Weight Adjusting The weight has set to a random initially value as usually performed. Weight adjustment is performed in stages, starting at the output layer the end of the feed forward phase, and going backward to the inputs through the hidden layer. Suppose initial small random values ??of the weights that used in the neural network, where the values ??must be random and non-symmetrical that makes the neurons do deferent operation. If all neurons have the same weight values, they respond the same for any input pattern. The gradient, which updates the weights, would be the same for each neuron, that mean the weights would remain the same even after the update and this means there is no learning. The weights that feed the output layer and the hidden layers are updated using below equation: (2.9) 2.9.6 Back Propagation Training The back-propagation learning algorithm is a supervised learning for artificial neural networks (ANNs). It extends the weight update rule for adjust the weight between the neurons in the network layers in multilayer feed forward artificial neural network. In Figure 2.13 the block diagram of back propagation learning algorithm is shown. Desired Output Network Inputs Back Propagation Algorithm Actual Outputs Training Dataset Input Layer Hidden Layer Output Layer Feed Forward Pass Feed Backward Pass ded backward pass Desired Output Network Inputs Back Propagation Algorithm Actual Outputs Training Dataset Input Layer Hidden Layer Output Layer Feed Forward Pass Feed Backward Pass ded backward pass Figure 2.13 Block Diagram of Back Propagation Network. In back propagation neural networks (BPNNs) the learning algorithm carried out the training process. Training input patterns are presented to the network input layer. The input pattern is propagated forward through the network to the output layer, get an actual error that is different from the desired output. An error is calculated and then propagated backwards through the network. The weights are modified as the error is propagated. These training phases can be described as shown in figure 2.14. 3905259525Begin Set Random Values for Weight Start Training Feed Forward Propagation Actual Output?Desired Output Set Desired Output Back Propagation Error Signal Actual Output?Desired Output No Yes No Yes Trained Neural Net 00Begin Set Random Values for Weight Start Training Feed Forward Propagation Actual Output?Desired Output Set Desired Output Back Propagation Error Signal Actual Output?Desired Output No Yes No Yes Trained Neural Net Figure 2.14 Back Propagation Flow Chart. The advantages that make the backpropagation learning algorithm is the most common used to train the artificial neural network with respect to the simplicity of use and tends to be significantly faster for training artificial neural networks than general-purpose optimization techniques such as evolutionary optimization. 2.10 Summary This chapter presents a general review of Artificial Neural Networks (ANNs) and their importance in the technological development, the inspiration of the system of the human brain, their architectural and component, training and testing process. Their algorithms and laws of learning, and an explanation of back propagation learning algorithm which is used to train the dental identification system in this thesis. The dental identification system will be described in details next chapter.