What do node gradients represent in a neural network?
neural-network
Solution
Let first explain gradient descent. GRADIENT DESCENT is an optimization algorithm for minimizing a cost function.
Consider the following example:
Where f(t) is the function we want to minimize, t has some initial value t1 but we want to find one such that f(t) gets its minimal value.
This is the formula for the gradient descent algorithm:
t = t - α d/dt f(t),
where α ** is a learning rate and d/dt f(t) is the derivative of the function. And a derivative is simply the slope of the line that is tangent to the function.
We keep applying this formula until we reach the minimum.
Looking at the picture above, the gradient descent will update t in the following fashion: the slope (derivative) is positive, α is positive, so t value will decrease now, hence minimizing f(t). We repeat this until we get for d/dt f(t) == 0 (the slope of any cont. function at its minimum (and max.) is zero).
We can now apply the idea of gradient descent in our backpropagation algorithm in order to adjust properly our weights.
Given a training example e, we define the error function as
E_e (w ⃗ )= 1/2 ∑_(k ∈Outputs) (d_k- o_k )^2 ,
where k is the number of outputs in the neural network, d is desired output, and o is observed output.
Observation: If this function E equals 0, that means that for all k, d_k == o_k, which means the output of the neural network was the same as the desired one and there is no work to be done, e.i. our NN is very smart. Initially of course, the weights are assigned randomly and its never the case, but we want to achieve that (or nearly that hopefully).
Since we now have an error function which we want to minimize, does it click now what can we apply? The gradient descent yes! ^^ The idea is to modify the weights according to the negative of the gradient of the error function to get a fast reduction of error on this example (meaning e), so we revise the weights according to gradient information as
∆w_ji= α (-∂E(wij)/∂wij )
If you compare with the above example, this does exactly the same, the only difference is that in the latter case the error function is a multivariate function, e.i. for a given weight we find the partial derivative for that particular weight.)
Applying this error correction (gradient descent) to the weights over and over again, we would achieve a lever where the error function is minimized and our NN well trained.
*Note: there are many issues involved in this problem, for example, if the learning rate is too large, the gradient descent can even overshoot the minimum, but to avoid confusions, don’t care about this too much now. :)
Problem
I am following along (code is a mess, I'm just messing around) with Introduction to the math of neural networks with this simple 3-layer neural net: My calculations are coming out pretty much the same as the book (attributing difference to rounding): ``` o1 delta: 0.04518482993361776 h1 delta: -0.0023181625149143255 h2 delta: 0.005031782661407674 h1 -> o1: 0.01674174257328656 h2 -> o1: 0.033471787838638474 b2 -> o1: 0.04518482993361776 // didn't calculate layer 1 gradients but would use the same approach ``` But what exactly are the gradients? Are they the individual node's contribution to the error of o1?