Android Asynctask to process live video frames

android, android-asynctask, image-processing

Solution

I would do something like that :

// Example value for a timeout.
private static final long TIMEOUT = 1000L;

private BlockingQueue<Mat> frames = new LinkedBlockingQueue<Mat>();

Thread worker = new Thread() {
    @Override
    public void run() {
        while (running) {
            Mat inputFrame = frames.poll(TIMEOUT, TimeUnit.MILLISECONDS);
            if (inputFrame == null) {
                // timeout. Also, with a try {} catch block poll can be interrupted via Thread.interrupt() so not to wait for the timeout.
                continue;
            }
            cropToCenter(inputFrame);
            String result = doImgProcessing(inputFrame);
        }
    }
};
worker.start();

public Mat onCameraFrame(Mat inputFrame) {
    inputFrame.copyTo(mRgba);//this will be used for the live stream
    frames.put(inputFrame);
    return mRgba;
}

The onCameraFrame puts the frame on the Queue, the worker Thread polls from the Queue.

This decorelate the reception and the treatment of the frame. You can monitor the growth of the Queue using `frames.size()`.

This is a typical producer-consumer example.

Problem

I am using OpenCV to attempt to do some live video processing. Since the processing is fairly heavy, it delays the output frames significantly, making the live stream look choppy. I'd like to offload some of the processing into an AsyncTask. I've tried it and it actually makes the video much smoother. However, it ends up starting a large amount of Tasks at once, and then they will slowly start returning with some results. Is there any way to slow this down, and wait for a result, either by using Synchronize statements, or some other method? On each camera frame, I start one of these tasks. DoImgProcessing does the long processing and returns a string result. ``` private class LongOperation extends AsyncTask<Mat, Void, String> { @Override protected String doInBackground(Mat... params) { Mat inputFrame = params[0]; cropToCenter(inputFrame); return doImgProcessing(inputFrame); } @Override protected void onPostExecute(String result) { Log.d(TAG, "on post execute: "+result); } @Override protected void onPreExecute() { Log.d(TAG, "on pre execute"); } } public Mat onCameraFrame(Mat inputFrame) { inputFrame.copyTo(mRgba);//this will be used for the live stream LongOperation op = new LongOperation(); op.execute(inputFrame); return mRgba; } ```

Original source