Python openCV matchTemplate on halftone with masking

You have already decided on the first questions, but I'll break them down a bit:

For a binary mask, it must be of a type uint8

where the values ​​are simply zero or non-zero. Dots with zero are ignored and included in the mask if they are nonzero. You can pass float32

as a mask instead, in which case it allows you to weigh the pixels; so a value of 0 is ignored, 1 includes, and .5 includes, but only gives half the weight compared to another pixel. Note that mask is only supported for TM_SQDIFF

and TM_CCORR_NORMED

, which is fine since you are using the latter. Masks for matchTemplate

are single channel only. And as you know, mask

it is not a positional argument, so it must be called with the key in the argument mask=your_mask

. This is all covered in some detail on this page in the OpenCV docs .

Now to a new problem:

It has to do with the method you are using and the fact that you are using jpg

s. Look at the formulas for the normalized methods . If the image is completely zero, you will get erroneous results because you will be dividing by zero. But this is not an exact problem --- because it returns nan

and np.nan > value

always returns false, so you will never draw a square from values nan

.

Instead, the problem is true in edge cases where you get a hint of a non-zero value; and since you are using jpg

images, not all black values ​​are 0; in fact, many do not. Pay attention to the formula you are diving on the averages, and the averages will be extremely small if there are values ​​like 1, 2, 5, etc. in your image window, so it will blow up the correlation value. Should be used instead TM_SQDIFF

(because this is the only other method that allows a mask to be used). Also, since you are using jpg

, most of your masks are useless, as any non-zero value (even 1) is considered an inclusion. You must use png

for masks. As long as the templates have a proper mask, it doesn't matter if you usejpg

or png

for templates.

FROM TM_SQDIFF

instead of looking for maximum values, you are looking for minimum - you want the smallest difference between the template and the image patch. You know the difference has to be very small - exactly 0 for a perfect pixel match, which you probably won't get. You can play with the threshold for a bit. Note that you will always get pretty close values ​​for each rotation, because the nature of your template --- the small arrow is unlikely to add many positive values, and this does not necessarily guarantee that the single-duplex sampling will be exactly like this if you did not take the image. thus). But even an arrow pointing in a completely wrong direction will still be very close, since there are many overlaps; and an arrow next to the correct directionwill be really close to the values ​​with exactly the right direction.

Review what the result of the square difference is when you use the code:

res = cv2.matchTemplate(img_gray, tmpl, cv2.TM_SQDIFF, mask=mask)
cv2.imshow("result", res.astype(np.uint8))
if cv2.waitKey(0) & 0xFF == ord('q'):
    break

      

        
        
        
      

    

You can see that basically every orientation of the template is the same.

In any case, it seems that threshold 8 nailed it:

The only thing I changed in your code changed to png

for all images, switching to TM_SQDIFF

, making sure to loc

look for values ​​less than the threshold, not more, and using MATCH_THRESH

8. At least I think that's all I changed. Look just in case:

import numpy as np
import cv2
import os
from scipy import misc, ndimage

STRIPPED_DIR = ...
TMPL_DIR = ...
MATCH_THRESH = 8
MATCH_RES = 1  #specifies degree-interval at which to match

def make_templates():
    base = misc.imread(os.path.join(TMPL_DIR,'base.jpg')) # The templ that I rotate to make 360 templates
    for deg in range(360):
        print('making template: ' + str(deg))
        tmpl = ndimage.rotate(base, deg)
        misc.imsave(os.path.join(TMPL_DIR, 'tmp' + str(deg) + '.png'), tmpl)

def make_masks():
    for deg in range(360):
        tmpl = cv2.imread(os.path.join(TMPL_DIR, 'tmp' + str(deg) + '.png'), 0)
        ret2, mask = cv2.threshold(tmpl, 0, 255, cv2.THRESH_BINARY+cv2.THRESH_OTSU)
        cv2.imwrite(os.path.join(TMPL_DIR, 'mask' + str(deg) + '.png'), mask)

def match(img_name):
    img_rgb = cv2.imread(os.path.join(STRIPPED_DIR, img_name))
    img_gray = cv2.cvtColor(img_rgb, cv2.COLOR_BGR2GRAY)

    for deg in range(0, 360, MATCH_RES):
        tmpl = cv2.imread(os.path.join(TMPL_DIR, 'tmp' + str(deg) + '.png'), 0)
        mask = cv2.imread(os.path.join(TMPL_DIR, 'mask' + str(deg) + '.png'), 0)
        w, h = tmpl.shape[::-1]
        res = cv2.matchTemplate(img_gray, tmpl, cv2.TM_SQDIFF, mask=mask)

        loc = np.where(res < MATCH_THRESH)
        for pt in zip(*loc[::-1]):
            cv2.rectangle(img_rgb, pt, (pt[0] + w, pt[1] + h), (0,0,255), 2)
        cv2.imwrite('res.png',img_rgb)