trigonometry
I'm trying to figure out if it's possible to calculate the width & height of the gray rectangle if I only know the width & height of the dotted red rectangle and the angle of the rotation.
I've tried substituting values for the width and height and using $x \cos(\alpha) + y \sin(\alpha)$ and attempting to figure out a scaling factor, but I'm not really getting anywhere.
The formula you want in this case is as follows: $$ k = \cos\theta+\frac{W}{H}\sin\theta $$ Where: $H$ is the height, $W$ is the width, $\theta$ is the angle of rotation (from $0$ to $90$ degrees), and $k$ is the scaling factor.
Interestingly, this means that the "worst case" (largest scaling factor) corresponds to $$ \theta = \tan^{-1}\left(\frac{W}{H}\right) $$
Proof:
In fact, I overlooked what happens when $W<H$. In fact, we should have
$$ k = \begin{cases} \cos\theta+\frac{W}{H}\sin\theta & W\geq H\\ \cos\theta+\frac{H}{W}\sin\theta & W < H \end{cases} $$
For the second case, we could draw the above and still have overlap. The more mathematically concise (but computationally expensive) way to write this would be $$k = \cos\theta + \max\left\{\frac{W}{H},\frac{H}{W}\right\}\sin\theta$$
In this picture,
$$
a = \arctan MF/ML
$$
and since $LF = LF'$,
$$
c = \arccos LF"/LF
$$
so the counterclockwise rotation angle is
$$
b = \pi/2 - a - c.
$$
You can do the same kind of calculation for the other corners of the small rectangle and find the smallest $b$.
In code you will have several cases to manage. Can the small rectangle be so small that it rotates all the way around?
Best Answer
$$x\cos\alpha+y\sin\alpha=w\\x\sin\alpha+y\cos\alpha=h$$ Multiply first equation by $\cos\alpha$, the second by $\sin\alpha$ and subtract: $$x(\cos^2\alpha-\sin^2\alpha)=w\cos\alpha-h\sin\alpha$$or $$x=\frac{w\cos\alpha-h\sin\alpha}{\cos^2\alpha-\sin^2\alpha}$$ Similarly, multiply the first equation by $\sin\alpha$, the second by $\cos\alpha$ and subtract: $$y(\sin^2\alpha-\cos^2\alpha)=w\sin\alpha-h\cos\alpha\\y=\frac{w\sin\alpha-h\cos\alpha}{\sin^2\alpha-\cos^2\alpha}$$ Notice that the denominator might be $0$ for $\alpha=45^\circ$. Then you must have the numerator equal to $0$ as well, or $w=h$, so you have a square inside a square $x=y$ and $$2x\frac{\sqrt 2}2=w$$ or $$x=w\frac{\sqrt 2}2$$