The computational graph

A basic neural network consists of forward propagation followed by a backward propagation. As a result, it consists of a series of steps that includes the values of different nodes, weights, and biases, as well as derivatives of cost function with regards to all the weights and biases. In order to keep track of these processes, the computational graph comes into the picture. The computational graph also keeps track of chain rule differentiation irrespective of the depth of the neural network.