Types of activation functions

The purpose of each node in the neural network is to accept input values and to enter an output value in the next layer. The input nodes insert the values of the variables in the hidden layer without modifying them. Each neuron adds the weighted values of all of the neurons connected to it and adds a bias value. To this result, an activation function is applied, which does nothing but transform the value mathematically before passing it to the next layer. In this way, the input values are propagated through the network up to the output neurons. The goal is to adjust weights and bias in order to achieve the desired result. The following diagram shows the scheme of a single neuron with the activation function:

A neural network without an activation function is simply equivalent to a regression model; this means that it tries to approximate the distribution of data with a straight line (linear trend). On the contrary, the purpose of neural networks is to be able to approximate any function, and to do this it's necessary to introduce a non-linearity factor—this term is represented by the activation function.

The activation function must satisfy some criteria:

• It must have output values in the range {0, 1}
• It must provide an output value close to 1 when sufficiently stimulated (threshold effect), to propagate activity within the network

There're many activation functions available for a neural network to use. We'll try to analyze some of them.