We will be referring to the primitive picture above. The original rectangle had horizontal side $a$ and vertical side $b$. It was rotated through an angle $\theta$ ($t$ in the picture).
We start with an $a \times b$ rectangle, with horizontal and vertical sides, such that the sides of length $a$ are horizontal. We rotate the rectangle counterclockwise about its center, through an angle $\theta\le \pi/2$.
How much horizontal and vertical space is occupied by the rotated rectangle? (In general, the minimal containing rectangle will not be a square.)
The diagram above can be used to see that the horizontal space occupied is
$$a\cos\theta+b\sin\theta,$$
and that the vertical space occupied is
$$a\sin\theta+b\cos\theta.$$
For suppose that the side labelled $a$ has length $a$, and the side labelled $b$ has length $b$. Then the horizontal side $PR$ of the containing rectangle is made up of two parts. By basic trigonometry, the part $QR$ has length $a\cos\theta$, and the part $PQ$ has length $b\sin\theta$. Add up. A similar argument deals with the vertical side of the containing rectangle.
If we want to fit the rotated rectangle into a square of side $s$, then the least $s$ that will work is given by
$$s=\max(a\cos\theta+b\sin\theta, a\sin\theta+b\cos\theta).$$
If we have a square of fixed side $d$ (in the post, $d=100$), we may want to scale the original rectangle to make the rotated version fit. Let $\lambda$ be the (common) scaling factor of the sides of the original $a\times b$ rectangle that will make the fit in the $d\times d$ square snug horizontally and/or vertically. Then we need
$$\lambda \max(a\cos\theta+b\sin\theta, a\sin\theta+b\cos\theta)=d,$$
and now we can compute $\lambda$.
Comment: The same basic formulas can be used to solve some of the more general problems that you mentioned. We obtained separate formulas for the horizontal and vertical space occupied by the rotated rectangle. Suppose that our target rectangle has given horizontal and vertical sides. We can compute the scaling factor $\lambda_h$ that will give a horizontal fit, and the scaling factor $\lambda_v$ that will give a vertical fit. Then the desired scaling factor for both a horizontal fit and a vertical fit is $\min(\lambda_h, \lambda_v)$. (Here as usual we are working on the assumption that the ratio of the sides must be maintained, so a common scaling factor must be applied to each.)
The problem with using
"Calibration constant, number of pixels per cm at 50cm distance"
is that not all of the scene is at 50cm depth or distance from the camera.
I think you need to remove the perspective correction first like in this example...
http://opencv-code.com/tutorials/automatic-perspective-correction-for-quadrilateral-objects/
Your green points would be the red and green ones in this example. Now obviously you want your blue points to be the white and blue in this example, which you don't have because they are outside the image. But if you work out where the left and right edges will coincide with the bottom edge of the image (the last row) you can still perform the perspective correction.
Now the correction is removed you can map pixels to cm and estimate where the bottom edge will appear.
Best Answer
Let the eye of the viewer be at $(0, 0, e)$
Let the square have side $a$ and its center be at $(0, 0, -d)$
Let the projection plane be the $xy$ plane, and the plane of the square have a unit normal vector of $N = (\sin t, 0, \cos t)$. Spanning vectors of the plane of the square are: $(\cos t, 0, -\sin t)$ and $(0, 1, 0)$.
Top Corners of the square:
Top Left corner at: $(0, 0, -d) + (a/2) ( - \cos t, 1, \sin t )$
Top Right corner at: $(0, 0, -d) + (a/2) ( \cos t , 1, -\sin t )$
Ray from eye to top left corner is:
$r = (0, 0, e) + s ( (0, 0, -d - e) + (a/2) ( - \cos t, 1, \sin t ) )$
at $z = 0$:
$s = \dfrac{- e}{ - d - e + (a/2) \sin t } = \dfrac{e}{d + e - (a/2) \sin t} $
In the image, the top left corner has coordinates $(x_1, y_1)$ with
$x_1 = - \dfrac{(ae/2) \cos t}{ d + e - (a/2) \sin t }$
$y_1 = \dfrac{a e/2 } {d + e - (a/2) \sin t }$
Similarly, in the image, the top right corner has coordinates $(x_2, y_2)$ with
$x_2 = \dfrac{a e / 2 \cos t}{ d + e + (a/2) \sin t }$
$y_2 = \dfrac{a e / 2}{ d + e + (a/2) \sin t }$
If follows that
$- x_1 / y_1 = \cos t = x_2 / y_2 $
Hence,
$x_1 = -\cos t \ y_1$
$x_2 = \cos t \ y_2$
The horizontal width of the image (the distance between the two vertical lines) is
$ W = x_2 - x_1 = \cos t (y_2 + y_1 ) $
But $y_1 = \dfrac{h_1}{2} $ and $y_2 = \dfrac{h_2}{2} $
Hence,
$\cos t = \dfrac{ W }{\overline{h}}$
where $\overline{h} = \dfrac{1}{2} ( h_1 + h_2 ) $