How to determine the distance of a (distorted) rectangular target from the camera

I have a photo that contains several rectangles of different sizes and orientations. I am currently trying to find the distance from the camera to any rectangles present in the image. What's the best way to do this?

For example, a sample photo might look something like this (although it's probably very out of proportion):

I can find the pixel coordinates of the corners of any of the rectangles in the image, as well as the camera's FOV and resolution. I also know in advance the length and width of any rectangle that may be in the image (but not the angle at which they face the camera). The length-to-width ratio of each rectangular target that may appear in the image is guaranteed to be unique. The rectangles and camera will always be parallel to the ground.

What I have tried:

I hacked into a solution based on some sample code I found on the internet. I basically iterate over each rectangle and find the average pixel length and height.

Then I use that to find the ratio of length to height and compare it to the list of ratio of all known rectangular targets, so I can find the actual height of the target in inches. Then I use this information to find the distance:

... where actual_height

is the real height of the target in inches, IMAGE_HEIGHT

corresponds to how tall the image is (in pixels), pixel_height

is the average height of the rectangle in the image (in pixels), and VERTICAL_FOV

is the angle that the camera sees along the vertical axis in degrees (about 39, 75 degrees on my camera).

I found this formula on the internet, and while it seems to work okay, I really don't understand how it works and it always seems a little underestimating the actual distance.

Also, I'm not sure how to change the formula to deal with rectangles that are very distorted from viewing them at an angle. Since my algorithm works, if you find the proportion of length and height, it works fine for rectangles 1 and 2 (which are not too distorted), but does not work for rectangle 3 as it is very distorted, discarding the relationship entirely.

I decided to find the relationship using the method described in this StackOverflow question regarding the aspect ratio of a perspective-warped rectangle , but I wasn't sure how well that would work with what I have and wondered if it could survive or if it does a simpler solution i could try.