Neural Networks

 

Neural Networks

By Engineer: Firas Faham

 

Introduction to Neural Network

Neural Networks (NN) describe machine behavior mapping and are used to perform a wide variety of computational mechanisms in order to model natural or living systems. This system is a manifestation of the survivability of the animal kingdom, health of economy, insects, human, and other living systems.

Backpropagation algorithm is used in embedded Neural Networks to propagate the resultant error produced by the Neural Networks while executing a training set. The training set is a set of rules describing the desired NN behavior according to a preset Input Output set of rules or relationships. The error is used to alter the action outcome performed by the NN according to a desired outcome leading to a fast and accurate adaptive useful behavior regardless of the current or unforeseen environment. This process is similar to feeding back a processed System Output Signal to the input producing an Error Signal and is used to stabilize the overall system response. Among the concerns (Network Definitions and parameters) of this process are:

  1. Speed of the Backpropagation algorithm.
  2. The time it takes to stabilize to the desired outcome.
  3. Deviations from intended results.
  4. Ability to adapt to sudden environmental changes such as temperature, pressure, noise, and false signals.
  5. Ability to recognize a wide range of variables at the Input nodes.
  6. Ability to poses high degree of resolution at both of Inputs and Outputs and when needed.
  7. Interoperability with different systems, Interfacing, and Conversion.
  8. Forward and Backward Scalability.
  9. Self-Activation, Initialization, and Calibration.
  10. Power requirements and mobility.

 

Among of the different applications of Neural Networks are:

NN network behavior is observed from the human brain that carries approximately 100,000,000,000 neuron. Neuron is the basic structure of the brain cells, each neuron connects to approximately 1000 adjacent neuron in certain areas inside the brain. The figure below illustrates the basic structure of the neuron cell.

Figure 1. The Neuron is the basic structure of the brain cells.

The neuron is a nerve cell constituting the structure and function of the nerve tissue. The learning process occurs when more Neuron Synapses convolute around one another forming new behavioral and learning patterns. Each neuron has many axons branching from it. Axons end with synapses. Synapses could be inhibitory (turn off) or excitory (turn on). Neighboring neurons communicate via their synapses. Synapses can send or receive signals to or from the neighboring neurons. Signal flow into & from synapses manifests the muscle movement and other human biological activities such as learning, speaking, emotions, walking, sensing, and other activities and interactions.

All of these activities are based on the basic system Input Output functionality.

 

Figure 2. Systematic approach to the Input Output mechanism of the NN network.

 

Neural Network Models

There are two common models for representing the Neural Networks, the single layer model and the multiple layer models.

A Single Layer Model (SLM) usually contains one processing neuron layer having multiple inputs and one output while the Multiple Layer Model (MLM) is having multiple neuron layers with multiple inputs and one output. The following diagram illustrates this principle:

Figure 3. Two Models of the NN networks, Single Layer and Multiple Layer Models.

 

Each input Ui represents a sensor reading or an output from a previous network. The output O from any stage equals the input Ui multiplied by the weighting coefficient Wi. Positive weights represent activation signal while negative weights represent inhibiting signal.

For single layer Model the Activation or Inhibition function Gamma g defines the NN net function as in equation 1 below:

 ------------------------------------------------------------ (Equation 1)

The W factor represents a learning mechanism or memory effect. If gamma is less than 0 then gamma triggers to the value –1, while if it is greater than 0 it triggers to 1 as follows in equation 2:

------------------------------------------------------------ (Equation 2)

Among the various uses of the Single Layer Model NN networks are the logic and basic decision making structures such as the logic gates AND, OR, and NOT as follows:

 Figure 4. Examples of the NN network logical functions.

 
Single Layer Model is though unable to solve for XOR and thus is said to have a Separability Problem, a problem that is not existing in the Multiple Layer NN Networks.
The Multiple Layer NN Network is capable of performing more complex processes. It contains multiple inputs such as u1 and u2 as shown above, a hidden layer/s such as shown above the cells u3 and u4, and the output layer u5. There exist, for each connection, a special weighting coefficient W between layers. Inputs u1 and u2 provide input to the NN network while u3 and u4 represent the fabric of the network behavior. For every layer there exist a similar equation to equation 1 above describing the relation of inputs and outputs. For the entire NN network equation 1 above is expanded. This expansion provides similar relation between inputs and outputs of the entire NN network. This process results in a Sum Of Product (SOP) function called the Sigmoid Squashing Function as shown below: To compute the total NN response the Hidden Layer response is first evaluated and then used to compute the total NN system response.

  Figure 5. The Sigmoid Function can be used to represent the Input [u] - Output [S(u)] NN network behavior.

 

Microcontroller based NN Network

 It is of vital importance for the Artificial Intelligence (AI) system to provide a valuable response for both of predictable and unpredictable input or unforeseen environmental changes and conditions. At the same time it is almost impossible for the AI programmer to predict the unforeseen. However the AI machine is programmed with the largest possible set of predictable inputs and conditions and is also given the ability to learn its own environment and extrapolate an as decent as possible output or solution from the given set of inputs and the desired output behavior. The Neural Network thus has the following parameters:

      1. A character embracing the algorithm embedded in the NN set of software codes.
      2. Various combinations of perceived environmental inputs.
      3. Action selection and extrapolation.
        through altering the environmental inputs and feedback them into the system

(Back Propagation).

 The back propagation reaction takes into account the speed of attaining the desired behavior given certain stepping resolution and a starting point. Training the NN network is achieved by finding the proper weight coefficients w that lead to the desired response. A backward pass is performed by the embedded Micro Controller Unit (MCU) algorithm on the output cells and the intermediate cells to calculate the associated output cell error d o and the intermediate cells error d i as follows in equation 3 below:

-------------------------------- (Equations 3)

 Where:

d

o :Output cell error

d

i : Input cell error

C : Correct response

w : Weight coefficient between cells

U : Activation cell

i : Denotes the intermediate cell

m : Denotes cells connected to hidden node

 

The weight coefficients connecting the hidden layer to the input are then updated as follows in equation 4 below:

--------------------------------------------- (Equation 4)

 And the weight coefficients connecting the hidden layer to the output are then updated as follows in equation 5 below:

--------------------------------------------- (Equation 5)

Where r is the learning rate. The initial value and step size of the learning rate both determine how fast the NN network function converges to the desired outcome. A typical start value for the learning rate is 0.1 and the increment can be selected depending on how fast and how critical is convergence and is application dependent. If the learning rate was too high then a convergence may annihilate. If the convergence is too low then the output will not overshoot with time.

 TBC…

 

By: Engr. Firas Faham.

September 2024