Tensorflow Sampled Softmax Loss Correct Usage

Question

In a classification problem with many classes, tensorflow docs suggests using sampled_softmax_loss over a simple softmax to reduce training runtime. According to the docs and source (line 1180), the call pattern for sampled_softmax_loss is: ``` tf.nn.sampled_softmax_loss(weights, # Shape (num_classes, dim) - floatXX biases, # Shape (num_classes) - floatXX labels, # Shape (batch_size, num_true) - int64 inputs, # Shape (batch_size, dim) - floatXX num_sampled, # - int num_classes, # - int num_true=1, sampled_values=None, remove_accidental_hits=True, partition_strategy="mod", name="sampled_softmax_loss") ``` It's unclear (at least to me) how to convert a real world problem into the shapes that this loss function requires. I think the 'inputs' field is the problem. Here is a copy-paste-ready minimum working example that throws a matrix multiplication shape error when calling the loss function. ``` import tensorflow as tf # Network Parameters n_hidden_1 = 256 # 1st layer number of features…

Accepted Answer

In your softmax layer you are multiplying your network predictions, which have dimension `(num_classes,)` by your `w` matrix which has dimension `(num_classes, num_hidden_1)`, so you end up trying to compare your target labels of size `(num_classes,)` to something that is now size `(num_hidden_1,)`. Change your tiny perceptron to output `layer_1` instead, then change the definition of your cost. The code below might do the trick.

```
def tiny_perceptron(x, weights, biases):
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.nn.relu(layer_1)
    return layer_1

layer_1 = tiny_perceptron(x, weights, biases)
loss_function = tf.reduce_mean(tf.nn.sampled_softmax_loss(
                     weights=weights['h1'],
                     biases=biases['b1'],
                     labels=labels,
                     inputs=layer_1,
                     num_sampled=num_sampled,
                     num_true=num_true,
                     num_classes=num_classes))
```

When you train your network with some optimizer, you will tell it to minimize `loss_function`, which should mean that it will adjust both sets of weights and biases.

Tensorflow Sampled Softmax Loss Correct Usage

Solution

Problem