Why does Keras LSTM batch size used for prediction have to be the same as fitting batch size?

Question

When using a Keras LSTM to predict on time series data I've been getting errors when I'm trying to train the model using a batch size of 50, while then trying to predict on the same model using a batch size of 1 (ie just predicting the next value). Why am I not able to train and fit the model with multiple batches at once, and then use that model to predict for anything other than the same batch size. It doesn't seem to make sense, but then I could easily be missing something about this. Edit: this is the model. `batch_size` is 50, `sl` is sequence length, which is set at 20 currently. ``` model = Sequential() model.add(LSTM(1, batch_input_shape=(batch_size, 1, sl), stateful=True)) model.add(Dense(1)) model.compile(loss='mean_squared_error', optimizer='adam') model.fit(trainX, trainY, epochs=epochs, batch_size=batch_size, verbose=2) ``` here is the line for predicting on the training set for RMSE ``` # make predictions trainPredict = model.predict(trainX, batch_size=batch_size) ``` he…

Accepted Answer

Unfortunately what you want to do is impossible with Keras ... I've also struggle a lot of time on this problems and the only way is to dive into the rabbit hole and work with Tensorflow directly to do LSTM rolling prediction.

First, to be clear on terminology, `batch_size` usually means number of sequences that are trained together, and `num_steps` means how many time steps are trained together. When you mean `batch_size=1` and "just predicting the next value", I think you meant to predict with `num_steps=1`.

Otherwise, it should be possible to train and predict with `batch_size=50` meaning you are training on 50 sequences and make 50 predictions every time step, one for each sequence (meaning training/prediction `num_steps=1`).

However, I think what you mean is that you want to use stateful LSTM to train with `num_steps=50` and do prediction with `num_steps=1`. Theoretically this make senses and should be possible, and it is possible with Tensorflow, just not Keras.

The problem: Keras requires an explicit batch size for stateful RNN. You must specify batch_input_shape (batch_size, num_steps, features).

The reason: Keras must allocate a fixed-size hidden state vector in the computation graph with shape (batch_size, num_units) in order to persist the values between training batches. On the other hand, when `stateful=False`, the hidden state vector can be initialized dynamically with zeroes at the beginning of each batch so it does not need to be a fixed size. More details here: http://philipperemy.github.io/keras-stateful-lstm/

Possible work around: Train and predict with `num_steps=1`. Example: https://github.com/keras-team/keras/blob/master/examples/lstm_stateful.py. This might or might not work at all for your problem as the gradient for back propagation will be computed on only one time step. See: https://github.com/fchollet/keras/issues/3669

My solution: use Tensorflow: In Tensorflow you can train with `batch_size=50, num_steps=100`, then do predictions with `batch_size=1, num_steps=1`. This is possible by creating a different model graph for training and prediction sharing the same RNN weight matrices. See this example for next-character prediction: https://github.com/sherjilozair/char-rnn-tensorflow/blob/master/model.py#L11 and blog post http://karpathy.github.io/2015/05/21/rnn-effectiveness/. Note that one graph can still only work with one specified `batch_size`, but you can setup multiple model graphs sharing weights in Tensorflow.

Why does Keras LSTM batch size used for prediction have to be the same as fitting batch size?

Solution

Problem