How to keep tensorflow session open between predictions? Loading from SavedModel

Question

I trained a tensorflow model that i'd like to run predictions on from numpy arrays. This is for image processing within videos. I will pass the images to the model as they happen. Not every frame is passed. I reload my SavedModel within a session like so ``` def run(self): with tf.Session(graph=tf.Graph()) as sess: tf.saved_model.loader.load(sess, [tf.saved_model.tag_constants.SERVING], "model") ``` My code works perfectly if I pass a list of images (self.tfimages) to the prediction. Condensed to: ``` softmax_tensor = sess.graph.get_tensor_by_name('final_ops/softmax:0') predictions = sess.run(softmax_tensor, {'Placeholder:0': self.tfimages}) ``` But i won't have all the images at once. Do I really have to reload the model from file each time (takes 2+ minutes). I thought to do something like this ``` class tensorflow_model: def __init__(self): with tf.Session(graph=tf.Graph()) as self.sess: tf.saved_model.loader.load(self.sess, [tf.saved_model.tag_constants.SERVING], "model") def pred…

Accepted Answer

Others have explained why you can't put your session in a `with` statement in the constructor.

The reason you see different behavior when using the context manager vs. not is because `tf.saved_model.loader.load` has some weird interactions between the default graph and the graph that is part of the session.

The solution is simple; don't pass a graph to session if you're not using it in a `with` block:

```
sess=tf.Session()
tf.saved_model.loader.load(sess,[tf.saved_model.tag_constants.SERVING], "model")
```

Here's some example code for a class to do predictions:

```
class Model(object):

def __init__(self, model_path):
    # Note, if you don't want to leak this, you'll want to turn Model into
    # a context manager. In practice, you probably don't have to worry
    # about it.
    self.session = tf.Session()

tf.saved_model.loader.load(
        self.session,
        [tf.saved_model.tag_constants.SERVING],
        model_path)

self.softmax_tensor = self.session.graph.get_tensor_by_name('final_ops/softmax:0')

def predict(self, images):
    predictions = self.session.run(self.softmax, {'Placeholder:0': images})
    # TODO: convert to human-friendly labels
    return predictions

images = [tf.gfile.FastGFile(f, 'rb').read() for f in glob.glob("*.jpg")]
model = Model('model_path')
print(model.predict(images))

# Alternatively (uses less memory, but has lower throughput):
for f in glob.glob("*.jpg"):
  print(model.predict([tf.gfile.FastGFile(f, 'rb').read()]))
```

How to keep tensorflow session open between predictions? Loading from SavedModel

Solution

Problem