MLP neural network won't learn
I went through all my code, and if this is indeed a problem, I'm not sure how this escaped me. Too long to post, so I'll tell you about my problem, which is what I was looking for, and if you have any ideas what else I might find, I'd really appreciate it!
So first, the weights are initialized with a mean zero and variance equal to 1 / square root of the number of inputs for this neuron, as specified with haykin.
I fed this with a simple sine wave to learn first. The weights in the hidden layer seem to converge to give the same result for every neuron within that layer ... which makes the output neuron almost fixed.
So what could be the reason? First, I checked to see if it was causing it to loop on local lows by increasing it, and also tried without momentum. I found this fixed the problem somewhat as the network creates a sine wave. However, not correct! :(
The network output has an amplitude of about a third of the height from the central axis upwards and does not go below. It's like you pick a sine wave, swallow a third, and lift it up to sit on it with the smallest peaks on the axis. Also, the top peaks are all flat ...
Since then, I've tried changing the topology of the network, with the result that if you add another hidden neuron (3 in total), it suddenly only gives a fixed output.
source to share
Sine wave is not an easy problem for a neural network with sigmoid activation functions. 3 hidden neurons are usually not enough. Take a look at this example : 200 hidden nodes are approaching a sine wave within [0, 2 * pi]. The activation functions in this example are logistic sigmoid in the hidden layer and identification in the output layer. You should always use the ID in the output layer for regression.
When you don't get a good result, it can also be benificial to decrease your learning rate. Sometimes, gradient descent oscillates between steep regions of the error function because the learning rate is too fast.
source to share