Multi GPU architecture, gradient averaging - less accurate model?
neural-network, tensorflow
Solution
In the code you linked, using the function `average_gradient` with 2 GPUs is exactly equivalent (1) to simply using 1 GPU with twice the batch size.
You can see it in the definition:
grad = tf.concat(axis=0, values=grads)
grad = tf.reduce_mean(grad, 0)
Using a larger batch size (given the same number of epochs) can have any kind of effect on your results.
Therefore, if you want to do exactly equivalent (1) calculations in 1-GPU or 2-GPU cases, you may want to halve the batch size in the latter case. (People sometimes avoid doing it, because smaller batch sizes may also make the computation on each GPU slower, in some cases)
Additionally, one needs to be careful with learning rate decay here. If you use it, you want to make sure the learning rate is the same in the `n`th epoch in both 1-GPU and 2-GPU cases -- I'm not entirely sure this code is doing the right thing here. I tend to print the learning rate in the logs, something like
print sess.run(lr)
should work here.
(1) Ignoring issues related to pseudo-random numbers, finite precision or data set sizes not divisible by the batch size.
Problem
When I execute the cifar10 model as described at https://www.tensorflow.org/tutorials/deep_cnn I achieve 86% accuracy after approx 4 hours using a single GPU , when I utilize 2 GPU's the accuracy drops to 84% but reaching 84% accuracy is faster on 2 GPU's than 1. My intuition is that average_gradients function as defined at https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py returns a less accurate gradient value as an average of gradients will be less accurate than the actual gradient value. If the gradients are less accurate then the parameters than control the function that is learned as part of training is less accurate. Looking at the code (https://github.com/tensorflow/models/blob/master/tutorials/image/cifar10/cifar10_multi_gpu_train.py) why is averaging the gradients over multiple GPU's less accurate than computing the gradient on a single GPU ? Is my intuition of averaging the gradients producing a less accurate value correct ? Randomness in the model is described as : ``` The images are processed as follows: They are cropped to 24 x 24 pixels, centrally for evaluation or randomly for training. They are approximately whitened to make the model insensitive to dynamic range. For training, we additionally apply a series of random distortions to artificially increase the data set size: Randomly flip the image from left to right. Randomly distort the image brightness. Randomly distort the image contrast. ``` src : https://www.tensorflow.org/tutorials/deep_cnn Does this have an effect on training accuracy ? Update : Attempting to investigate this further, the loss function value training with different number of GPU's. ``` Training with 1 GPU : loss value : .7 , Accuracy : 86% Training with 2 GPU's : loss value : .5 , Accuracy : 84% ``` Shouldn't the loss value be lower for higher for higher accuracy, not vice versa ?