How to make the sample run Open CV

augmented-reality, ios, iphone, opencv, xcode

Solution

You have three interrelated questions here:

1/ how to get openCV framework to run in an iOS project 2/ how to get the Template Matching c++ sample code to run in an iOS project 3/ how to do live template matching with the camera view

1/ how to get openCV framework to run in an iOS project

- Download and import the openCV framework as you describe

- Change the .pch file as you describe

check that c++ standard library is set to libc++ in your target build settings (this is the default for new projects)

Don't just import demo.cpp without making changes as described below (it is a 'raw' c++ progam with it's own `main` function, and needs alterations to work as part of an iOS/Cocoa project).

Don't mess with header search paths, other linker flags etc, this isn't necessary if you have imported the prebuilt framework from openCV.org.

Don't change your .m files to .mm unless you know that you need to. My advice is to keep your c++ code separate from your objective-C code as far as practicable, so most of you files should be .m files (objective-C) or .cpp files (c++). You only need .mm prefixes for "objective-C++" where you intend to mix objective-C and c++ in the same file.

2/ how to get the Template Matching c++ sample code to run in an iOS project

We are going to set this up so that your iOS viewController - and the bulk of your iOS code - does not need to know that the image is processed using openCV/C++, and likewise the C++ code doesn't need to know where it's input or output image data is being routed to. We do this by making a small wrapper class between the two that translates objective-C method calls to c++ class member functions and back. We will also set up a category on UIImage to translate image formats from iOS-friendly UIImage to openCV-native cv::Mat.

UIImage+OpenCV Category

You need some utility methods to convert from UIImage to cv::Mat and back. A good place to put these is in a UIImage category. In XCode: File>New FIle>Cocoa Touch>Objective-C category will set you up. Call the category OpenCV and make it a category on UIImage. This .m file you will want to change to .mm as it will need to understand c++ types from the openCV framework.

The header should look something like this:

#import <UIKit/UIKit.h>

@interface UIImage (OpenCV)

    //cv::Mat to UIImage
+ (UIImage *)imageWithCVMat:(const cv::Mat&)cvMat;

    //UIImage to cv::Mat
- (cv::Mat)cvMat;

@end

The .mm file should implement these methods by closely following this openCV.org code sample adapted to work as category methods (eg you don't pass a UIImage into the instance method, but refer to it using `self`).

You can use the category methods as if they are UIImage class and instance methods like this:

UIImage* image = [UIImage imageWithCVMat:matImage];  //class method

cv::Mat matImage = [image cvMat];  //instance method

openCV wrapper class

Make a wrapper class to convert your objective-C method (called from a viewController) to a c++ function

header something like this

//  CVWrapper.h
#import <Foundation/Foundation.h>

@interface CVWrapper : NSObject

+ (NSImage*) templateMatchImage:(UIImage*)image
                          patch:(UIImage*)patch
                         method:(int)method;
@end

We send in the template image, patch image and template matching method, and get back an image showing the match

implementation (.mm file)

//  CVWrapper.mm
#import "CVWrapper.h"
#import "CVTemplateMatch.h"
#import "UIImage+OpenCV.h"

@implementation CVWrapper


+ (UIImage*) templateMatchImage:(UIImage *)image
                          patch:(UIImage *)patch
                         method:(int)method
{
    cv::Mat imageMat = [image cvMat];
    cv::Mat patchMat = [patch cvMat];

    cv::Mat matchImage = 
           CVTemplateMatch::matchImage(imageMat, 
                                        patchMat,
                                        method);

    UIImage* result =  [UIImage imageWithCVMat:matchImage];
    return result;
}

We are effectively taking a standard objective-C method and UIImage types and translating them into a call to a C++ member function with c++(openCV framework) types, and translating the result back to a UIImage.

C++ TemplateMatch class

Header:

//  TemplateMatch.h

#ifndef __CVOpenTemplate__CVTemplateMatch__
#define __CVOpenTemplate__CVTemplateMatch__

class CVTemplateMatch
{
public:
    static cv::Mat matchImage (cv::Mat imageMat,
                               cv::Mat patchMat, 
                               int method);
};

#endif /* defined(__CVOpenTemplate__CVTemplateMatch__) */


@end

Implementation:

This is the Template Match openCV example code, reworked as a class implementation:

//  TemplateMatch.cpp
/*
 Alterations for use in iOS project
 [1] remove GUI code (iOS supplies the GUI)
 [2] change main{} to static member function 
     with appropriate inputs and return value
 [3] change MatchingMethod{} signature 
     to return Mat value
 */

#include "CVTemplateMatch.h"

    //[1] #include "opencv2/highgui/highgui.hpp"
#include "opencv2/imgproc/imgproc.hpp"
#include <iostream>
#include <stdio.h>

using namespace std;
using namespace cv;

    /// Global Variables
Mat img; Mat templ; Mat result;
    //[1] char* image_window = "Source Image";
    //[1] char* result_window = "Result window";

int match_method; 
    //[1]  int max_Trackbar = 5; 

    /// Function Headers
Mat MatchingMethod( int, void* );  //[3] (added return value to function)

    // [2] /** @function main */
    // [2] int main( int argc, char** argv )

Mat CVTemplateMatch::matchImage (Mat image,Mat patch, int method)
    // [2]
{
        /// Load image and template
        //[2]  img = imread( argv[1], 1 );
        //[2] templ = imread( argv[2], 1 );

    img = image;           //[2]
    templ = patch;         //[2]
    match_method = method; //[2]

        /// Create windows
        //[1] namedWindow( image_window, CV_WINDOW_AUTOSIZE );
        //[1] namedWindow( result_window, CV_WINDOW_AUTOSIZE );

        /// Create Trackbar
        //[1] char* trackbar_label = "Method: \n 0: SQDIFF \n 1: SQDIFF NORMED \n 2: TM CCORR \n 3: TM CCORR NORMED \n 4: TM COEFF \n 5: TM COEFF NORMED";
        //[1] createTrackbar( trackbar_label, image_window, &match_method, max_Trackbar, MatchingMethod );

    Mat result = MatchingMethod( 0, 0 );

        //[1] waitKey(0);
        //[2] return 0;
    return result;  //[2]
}


    //[3] void MatchingMethod( int, void* )
Mat MatchingMethod( int, void* )

{
        /// Source image to display
    Mat img_display;
    img.copyTo( img_display );

        /// Create the result matrix
    int result_cols =  img.cols - templ.cols + 1;
    int result_rows = img.rows - templ.rows + 1;

    result.create( result_cols, result_rows, CV_32FC1 );

        /// Do the Matching and Normalize
    matchTemplate( img, templ, result, match_method );
    normalize( result, result, 0, 1, NORM_MINMAX, -1, Mat() );

        /// Localizing the best match with minMaxLoc
    double minVal; double maxVal; Point minLoc; Point maxLoc;
    Point matchLoc;

    minMaxLoc( result, &minVal, &maxVal, &minLoc, &maxLoc, Mat() );

        /// For SQDIFF and SQDIFF_NORMED, the best matches are lower values. For all the other methods, the higher the better
    if( match_method  == CV_TM_SQDIFF || match_method == CV_TM_SQDIFF_NORMED )
        { matchLoc = minLoc; }
    else
        { matchLoc = maxLoc; }

        /// Show me what you got
    rectangle( img_display, matchLoc, Point( matchLoc.x + templ.cols , matchLoc.y + templ.rows ), Scalar::all(0), 2, 8, 0 );
    rectangle( result, matchLoc, Point( matchLoc.x + templ.cols , matchLoc.y + templ.rows ), Scalar::all(0), 2, 8, 0 );

        //[1]  imshow( image_window, img_display );  
        //[1] imshow( result_window, result );       

    return img_display; //[3] add return value
}

Now in your `viewController` you just need to call this method:

UIImage* matchedImage = 
       [CVWrapper templateMatchImage:self.imageView.image
                               patch:self.patchView.image
                              method:0];

with no c++ in sight.

3/ Template Matching with live camera view

The short answer: `matchTemplate` is not going to work too well in a live camera context. The algorithm is looking for a match in the image with the same scale and orientation as the patch: it slides the patch tile across the image at it's original orientation and size comparing for best match. This is not going to yield great results if the image is perspective-skewed, a different size or rotated to a different orientation.

You could look instead at OpenCV's Feature Detection algorithms, some of which have been moved to non-free. Here is a nice description of SIFT to give you the idea. For video capture you might also want to look at `cap_ios.h` in `opencv2/highgui`: here is a tutorial.

Problem

Original source