Applying homography to a non-planar surface
As I know, Homography (projective transformation) in computer vision can be used to detect an object in images, but all the examples I saw were on flat objects. Does homography only work on a flat surface of an object? Or can He detect any objects ? I ask because I tried to find the object below (This is a nonplanar surface object) with no success:
In this link you can see the code. I used it by simply updating the name of the image, so we do the following:
- Getting key points from two images using SURF
- Describe key points using the SURF descriptor
- Try to match key points between two images
- Use a list of matching cue points to compute the Homography matrix.
- Get the coordinates of the corner points of the object.
- Apply a viewTransform to get their matches in the scene image.
- Draw lines between the points of the results.
Note that the green lines drawn inside the large circles in the images represent the line drawn to represent the result points.
Based on what I showed above, it seems to me that there is something incomprehensible in my understanding of homography, and where it can be applied, because this example is quite simple and does not work. I am currently researching the OpenCV code to understand exactly how they are evaluating it, but it is not that fast. So, does anyone know how OpenCV calculates this conversion? Or any link that can help with this situation?
Editorial:
Here's another example:
I have applied homography on the object and a yellow border that contains only the tool that I need. The results are even worse because now it is somehow a dot, as you can see in the green dot surrounded by a red circle. Also, I cannot take images for objects from the scene because I have a lot of videos, so what I do is take separate images for each instrument and try to find them in the scene videos.
source to share
Strictly speaking, you're right, homographies only represent observations of flat objects. It is not very clear in your post, but I am assuming that you are showing the matches found findHomography
. As you said, this approach works well for planar objects. In the case of nonplanar but rigid objects, the equivalent will be the smaller matches found findFundamentalMat
(see the OpenCV doc and the Wikipedia page ).
In practice, however, the use of homography should at least provide an approximate solution.
In my opinion your problem is more with bad SURF matches and not with the choice of homography transform. This is quite obvious when you look at a couple of images that you show: only a couple of dots correspond to the object you want to detect, while most of them correspond to different things in the scene.
One of the main problems with the approach you choose is that you are not dealing with rigid objects, but deformable: the servo handle can move, there is a non-linear deformation in the appearance due to the fluid inside the seringue, etc. Such deformations can make the SURF descriptors extracted in the target image completely different from those extracted in the reference image and therefore cannot be matched. Take a look at [1], they give a good idea of why the descriptors are the same or not.
For your problem, alternative approaches might be local matching (with small correlation fixes, for example), color matching, shape matching, deep learning, etc.
[1]: Vondrick, Karl, et al. "Hoggles: Visualizing Object Detection Functions." Computer Vision (ICCV), 2013 International IEEE Conference. IEEE, 2013. ( link )
source to share
SURF works on any object, I tested the same code you are using on my image and here are the results . In my opinion, the algorithm does not always support finding objects in another scene. It works well if the object image is extracted from the scene image.
Try these different options depending on your needs:
(1) If you have different series of source images, you can extract the desired object from the scene image and find its position in other images in the scene.
(2) Try to crop the object image without any preparation, it is not always successful, but it will work even if the angle changes.
(3) Sometimes changing the minHessian value, and depending on the number of cue points, where only smaller cue points will be detected, you will be able to detect your image.
source to share