Q-Learning in combination with neural-networks (rewarding understanding)

Question

As far as my understanding is, it's possible to replace a look-up-table for Q-values (state-action-pair-evaluation) by a neural network for estimating these state-action pairs. I programmed a small library, which is able to propagate and backpropagate through a self-built neural network for learning wanted target-values for a certain in-out-put. So I also found this site while googling, and googling through the whole web (as it felt for me): http://www.cs.indiana.edu/~gasser/Salsa/nn.html where the Q-learning combined with a neural network is shortly explained. For each action, there's an extra output neuron, and the activation-value of one of these output-"units" tells me, the estimated Q-value. (One question: Is the activation value the same as the "output" of the neuron or something different?) I used the standard sigmoid-function as activation-function, so the range of the function-values x is ``` 0<x<1 ``` So I thought, my target value should always be from 0.0 to 1.0 -> Question…

Accepted Answer

Using a neural-network to store q-value is a good extension of table lookup. This makes it possible to use q-learning when the state space is continuous.

```
  input layer     ......

|/  \ |  \|
  output layer  a1   a2   a3
                0.1  0.2  0.9
```

Suppose you have 3 actions available. Above shows the outputs from the neural network using current state and learned weights. So you know `a3` is the best action to go with.

Now the questions you have:

One question: Is the activation value the same as the "output" of the neuron or something different?

Yes, I think so. In the referred link, the author said:

Some of the units may also be designated output units; their activations represent the network's response.

So I thought, my target value should always be from 0.0 to 1.0 -> Question: Is that point of my understanding correct? Or did I missunderstand something about that?

If you choose `sigmoid` as your activation function, for sure you output will be from 0.0 to 1.0. There are different choices of activation functions, e.g., here. `Sigmoid` is one of the most popular choices though. I think the output value being from 0.0 to 1.0 is not a problem here. if at current time, you have only two available actions, `Q(s,a1) = 0.1, Q(s,a2) = 0.9`, you know that action `a2` is much better than `a1` with respective to q-value.

So how do I perform this equation to get the right target for the neural network, if targets should be from 0.0 to 1.0?! How do I calculate good reward-values?

I am not sure for this, but you can try to clamp the new target q-value to be between 0.0 and 1.0, i.e.,

```
q(s,a) = min(max(0.0, q(s,a) + learningrate * (reward + discountfactor * q'(s,a) - q(s,a))), 1.0)
```

Try to do some experiments for finding a proper reward value.

Is moving toward the aim more worth it, than going away from it? (more +reward when nearing the aim than -reward for bigger distance to aim?)

Normally you should give more reward when it's close to the aim if you use the classical update equation, so that the new q-value gets increased.

Q-Learning in combination with neural-networks (rewarding understanding)

Solution

Problem

Related problems