Clever Geek Handbook

How to pass a numpy array to openCV without saving the file as png or jpeg?

I am trying to take a screenshot and then convert it to a numpy array. Then I want to run cv2.matchTemplate using the screenshot. So far, the only way I've gotten this is to save an image: cv2.imwrite ('temp.png', imcv) and then use that image in cv2.matchTemplate. This seems terribly wrong. How can I convert the numpy array correctly to avoid saving and just pass it straight to the cv2.matchTemplate function?

I am doing this project in Ubuntu btw.

import pyscreenshot as ImageGrab
import PIL
import cv2
import numpy as np
from matplotlib import pyplot as plt

# part of the screen
im=ImageGrab.grab(bbox=(65,50,835,725)) # X1,Y1,X2,Y2

#convert to numpy array
im=im.convert('RGB')
imcv = np.array(im) 
imcv = imcv[:, :, ::-1].copy() 
cv2.imwrite('temp.png',imcv)

img = cv2.imread('temp.png',0)
template = cv2.imread('fight.png',0)
w, h = template.shape[::-1]

# Apply template Matching
res = cv2.matchTemplate(img,template,cv2.TM_CCOEFF)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
print(min_val)
print(max_val)
print(min_loc)
print(max_loc)
if(max_loc == (484,125)):
    print("True!")

top_left = max_loc
bottom_right = (top_left[0] + w, top_left[1] + h)

cv2.rectangle(img,top_left, bottom_right, 255, 2)

plt.subplot(121),plt.imshow(res,cmap = 'gray')
plt.title('Matching Result'), plt.xticks([]), plt.yticks([])
plt.subplot(122),plt.imshow(img,cmap = 'gray')
plt.title('Detected Point'), plt.xticks([]), plt.yticks([])
plt.suptitle(cv2.TM_CCOEFF)

plt.show()

This is the easiest thing I can handle: I will also post the error after the code.

import pyscreenshot as ImageGrab
import PIL
import cv2
import numpy

im=ImageGrab.grab(bbox=(65,50,835,725)) # X1,Y1,X2,Y2
print type(im)
im=im.convert('RGB')
print type(im)
im = numpy.array(im)
print type(im)
im = im[:, :, ::-1].copy() 
print type(im)
cv2.cv.fromarray(im)
print type(im) 

template = cv2.imread('fight.png',0)

templateTest = cv2.matchTemplate(im,template,cv2.TM_CCOEFF)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
print(min_val)
print(max_val)
print(min_loc)
print(max_loc)

<type 'instance'>
<type 'instance'>
<type 'numpy.ndarray'>
<type 'numpy.ndarray'>
<type 'numpy.ndarray'>
OpenCV Error: Assertion failed ((img.depth() == CV_8U || img.depth() == CV_32F) && img.type() == templ.type()) in matchTemplate, file /home/kninja/Downloads/opencv-2.4.9/modules/imgproc/src/templmatch.cpp, line 249
Traceback (most recent call last):
  File "StartOVer.py", line 32, in <module>
    res = cv2.matchTemplate(im,template,cv2.TM_CCOEFF)
cv2.error: /home/kninja/Downloads/opencv-2.4.9/modules/imgproc/src/templmatch.cpp:249: error: (-215) (img.depth() == CV_8U || img.depth() == CV_32F) && img.type() == templ.type() in function matchTemplate
+3


source to share


4 answers 


import pyscreenshot as ImageGrab
import PIL
import cv2
import numpy

im=ImageGrab.grab(bbox=(65,50,835,725)) # X1,Y1,X2,Y2
print type(im)
im=im.convert('RGB')
print type(im)
im = numpy.array(im)
print type(im) 

cv_img = im.astype(np.uint8)
cv_gray = cv2.cvtColor(cv_img, cv2.COLOR_RGB2GRAY)

template = cv2.imread("filename.png", cv2.IMREAD_GRAYSCALE)


+1


source


PIL images support an array interface, so you can use fromarray

. Try the following:



cv2.cv.fromarray(imcv)

+2


source


I am working on a similar project. I used the pyautogui library for automation, but was unhappy with the image matching functions provided by the library due to the long time and inflexibility of exact image matching, so I switched to opencv for pattern matching. I found this post to create a series of grayscale images as quickly as possible. Froyo's answer is not being written to the hard drive, but I found that I have everything faster in place. I am also working on Ubuntu and I believe the screenshot taken by pyautogui is being called to the backend using the popular linux tool. The following code snippet is modified from the example code provided in the opencv documentation: http://docs.opencv.org/3.1.0/d4/dc6/tutorial_py_template_matching.html

#!/usr/bin/env python
import cv2
import numpy as np
import pyautogui
import PIL
from time import time, sleep
import pyscreenshot as ImageGrab

def click_image(template_filename):
    start = time()
    '''
    im=ImageGrab.grab()
    im=im.convert('RGB')
    im = np.array(im)
    cv_img = im.astype(np.uint8)
    screen = cv2.cvtColor(cv_img, cv2.COLOR_RGB2GRAY)
    '''
    pyautogui.screenshot('current_screen.png')
    screen = cv2.imread('current_screen.png',cv2.IMREAD_GRAYSCALE)
    template = cv2.imread(template_filename,cv2.IMREAD_GRAYSCALE)
    if template is None:
        print("failed to load template.")
        quit()
    w, h = template.shape[::-1]
    method = 'cv2.TM_CCOEFF'
    meth = eval(method)
    # Apply template Matching
    res = cv2.matchTemplate(screen,template,meth)
    #get min/max values to match
    min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(res)
    top_left = max_loc
    bottom_right = (top_left[0] + w, top_left[1] + h)
    center =((top_left[0]+bottom_right[0])/2(top_left[1]+bottom_right[1])/2)
    print(center)
    pyautogui.moveTo(center)
    pyautogui.click(center,button="right")
    end = time()
    print("clicked in "+str(int(1000*(end-start)))+"ms")
click_image("files.png")
+1


source


Use the following code to convert:

import cv2
import PIL
import numpy as np

img = ImageGrab.grab(bbox=(x1, y1, x2, y2))
img = cv2.cvtColor(np.array(img), cv2.COLOR_BGR2RGB)
0


source






More articles:


All Articles