Outputs always equal in a network trained with pybrain to approximate a function

neural-network, pybrain, python

Solution

I see a few problems with your code sample and they cause your net not learning anything meaningful. Here are some problems with your code:

- Your data is not normalized! Neural networks are sensitive to the range of data fed into them. In your case, you supplied a range with a min of ~2500 and a max of ~36000, when your input and targets should be normalized between the -1,1 or 0,1 range (depending on if you used Tanh or Sigmoid).

- You chose `BackpropTrainer` which generally needs to be trained a lot more than 10 epochs. Try 100 or 1000. (Or try `RPromMinusTrainer`, which I personally like.)

Both of these problems caused your net to just learn to spit out a single value.

I changed your code a little to include normalization and the `RPropMinusTrainer` with a decent amount of iterations (RProp- needs a lot fewer epochs):

from pybrain.supervised.trainers import *
from pybrain.tools.shortcuts import buildNetwork
from pybrain.datasets import SupervisedDataSet, SequentialDataSet
from pybrain.structure import RecurrentNetwork
from pybrain.structure import TanhLayer
import numpy as np

ds = SupervisedDataSet(1, 1)

tf = open('defl_07h.csv','r')

for line in tf.readlines():
    data = [float(x) for x in line.strip().split(';') if x != '']
    indata =  tuple(data[:1])
    outdata = tuple(data[1:])
    ds.addSample(indata,outdata)

i = np.array([d[0] for d in ds])
i /= np.max(np.abs(i),axis=0)
o = np.array([d[1] for d in ds])
o /= np.max(np.abs(o),axis=0)

nds = SupervisedDataSet(1, 1)
for ix in range(len(ds)):
    nds.addSample( i[ix], o[ix])

# train with normalized
net = buildNetwork(nds.indim,5,nds.outdim,recurrent=True)
t = RPropMinusTrainer(net,verbose=True)
t.trainOnDataset(nds,30)
t.testOnData(nds, verbose=True)

I get the following output:

out:     [  0.234]
correct: [  0.168]
error:  0.00217289
out:     [  0.209]
correct: [  0.168]
error:  0.00083736
...
out:     [  0.986]
correct: [  0.914]
error:  0.00258013
out:     [  1.006]
correct: [  0.916]
error:  0.00405199
out:     [  0.985]
correct: [  0.924]
error:  0.00187248

Not perfect, but the predictions increase with the actual normalized values, just like your original (pre-normalized) data did.

PS: I know this is an old question, but I've hit this problem before in the past and thought it might help some people. So if you're in the NN trenches and getting nowhere, remember to ask yourself: is my neural net expecting normalized data? Am I training enough?

Problem

Original source