OpenCV Android - color issue using CameraBridgeViewBase

android, android-emulator, java, macos, opencv

Solution

So, after drilling into the problem I've found the root of the issue.

Let's take a look at the OpenCV `JavaCameraView` class and its `CameraBridgeViewBase` base class. The problem was that camera frames received as `byte[]` array in `onPreviewFrame` method were decoded incorrectly.

The exact place of code where the decoding process takes place is an implementation of the `Mat rgba()` method in inner `JavaCameraFrame` class of the `JavaCameraView`:

    public Mat rgba() {
        Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
        return mRgba;
    }

As we see, Imgproc.cvtColor(...) method is used in order to convert frame from YUV to RGBA. `NV21 YUV -> RGBA` conversion takes place there. During the initialization process we set the format to NV21, so this should be right. Moreover, every Android device should support NV21. Also, we can check whether device accepted the format using debugger:

protected boolean initializeCamera(int width, int height) {
    ...
    params.setPreviewFormat(ImageFormat.NV21);
    ...
    mCamera.setParameters(params);
    ...
    params = mCamera.getParameters();
    Log.d(TAG, String.format("Actual preview format is 0x%X", params.getPreviewFormat()));
}

Both phone (HTC Sensation) and emulator reported to be using NV21 indeed.

However, if we change `COLOR_YUV2RGBA_NV21` to `COLOR_YUV2RGB_I420` (YV12 and I420 is the same thing, just with Y and V inverted;) we'll see that emulator will get a proper color space finally. Changing NV21 to YV12 in `params.setPreviewFormat(ImageFormat.NV21);` we'll get similar results. Looks like there's bug either in Imgproc.cvtColor, or in Android.

Here comes the solution. Change `public Mat rgba()` the following way:

    public Mat rgba() {
        if (previewFormat == ImageFormat.NV21) {
            Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGBA_NV21, 4);
        }
        else if (previewFormat == ImageFormat.YV12) {
            Imgproc.cvtColor(mYuvFrameData, mRgba, Imgproc.COLOR_YUV2RGB_I420, 4);  // COLOR_YUV2RGBA_YV12 produces inverted colors
        }
        return mRgba;
    }

`previewFormat` is a new `int` variable, it's declared this way:

private int previewFormat = ImageFormat.NV21;

Add following changes to the initialization:

protected boolean initializeCamera(int width, int height) {
        ...
                params.setPreviewFormat(ImageFormat.NV21);
                // "generic" = android emulator
                if (Build.BRAND.equalsIgnoreCase("generic")) {
                    params.setPreviewFormat(ImageFormat.YV12);
                }
                ...
                mCamera.setParameters(params);
                params = mCamera.getParameters();
                previewFormat = params.getPreviewFormat();
        ...
}

Important: Please note: this is just a temporary solution in order to make OpenCV usable with emulator in my case. Further research should be done. It's quite easy to check whether device uses correct image format in onPreviewFrame. I'll get back to this when I have some time.

Problem

Original source