We will be referring to the primitive picture above. The original rectangle had horizontal side $a$ and vertical side $b$. It was rotated through an angle $\theta$ ($t$ in the picture).
We start with an $a \times b$ rectangle, with horizontal and vertical sides, such that the sides of length $a$ are horizontal. We rotate the rectangle counterclockwise about its center, through an angle $\theta\le \pi/2$.
How much horizontal and vertical space is occupied by the rotated rectangle? (In general, the minimal containing rectangle will not be a square.)
The diagram above can be used to see that the horizontal space occupied is
$$a\cos\theta+b\sin\theta,$$
and that the vertical space occupied is
$$a\sin\theta+b\cos\theta.$$
For suppose that the side labelled $a$ has length $a$, and the side labelled $b$ has length $b$. Then the horizontal side $PR$ of the containing rectangle is made up of two parts. By basic trigonometry, the part $QR$ has length $a\cos\theta$, and the part $PQ$ has length $b\sin\theta$. Add up. A similar argument deals with the vertical side of the containing rectangle.
If we want to fit the rotated rectangle into a square of side $s$, then the least $s$ that will work is given by
$$s=\max(a\cos\theta+b\sin\theta, a\sin\theta+b\cos\theta).$$
If we have a square of fixed side $d$ (in the post, $d=100$), we may want to scale the original rectangle to make the rotated version fit. Let $\lambda$ be the (common) scaling factor of the sides of the original $a\times b$ rectangle that will make the fit in the $d\times d$ square snug horizontally and/or vertically. Then we need
$$\lambda \max(a\cos\theta+b\sin\theta, a\sin\theta+b\cos\theta)=d,$$
and now we can compute $\lambda$.
Comment: The same basic formulas can be used to solve some of the more general problems that you mentioned. We obtained separate formulas for the horizontal and vertical space occupied by the rotated rectangle. Suppose that our target rectangle has given horizontal and vertical sides. We can compute the scaling factor $\lambda_h$ that will give a horizontal fit, and the scaling factor $\lambda_v$ that will give a vertical fit. Then the desired scaling factor for both a horizontal fit and a vertical fit is $\min(\lambda_h, \lambda_v)$. (Here as usual we are working on the assumption that the ratio of the sides must be maintained, so a common scaling factor must be applied to each.)
First of all, there may be multiple rectangles that satisfy your conditions, e.g.
if you want specific angle of rectangle, or
if you want to specify a point on the boundary.
However, there is a special case where there is at most only once rectangle, i.e. if you assume that it has to touch both of your boundaries. In such case it is easy to compute it. Place the origin (point $(0,0)$) in the center of the boundary (the center of the circle), and let $(C_x,C_y)$ be the top right boundary corner (the boundary rectangle has size $2C_x \times 2C_y$ ). Let $(x,y)$ be the top right vertex of small rectangle, then it satisfies conditions ($\alpha > 0$ means counterclockwise rotation):
\begin{align*}
\left[\begin{matrix}x'\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\y\end{matrix}\right] \\\
\left[\begin{matrix}C_x\\\ y'\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\ -y\end{matrix}\right]
\end{align*}
where $x'$ and $y'$ are just placeholders. Extracting appropriate rows from those formulae, we can transform that into one equation (notice the lack of minus sign in the matrix):
\begin{align*}
\left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\ y\end{matrix}\right]
\end{align*}
with solution being:
\begin{align*}
\left[\begin{matrix}\cos\alpha&\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]^{-1}
\left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}x\\\ y\end{matrix}\right] \\\
\sec{2\alpha}\left[\begin{matrix}\cos\alpha&-\sin\alpha\\\ -\sin\alpha&\cos\alpha\end{matrix}\right]
\left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}x\\\ y\end{matrix}\right]
\end{align*}
Please note, that this may not have a proper solution if $\alpha$ is to big!
Edit: Ok, I missed the comment about maximizing the area. Then again, consider this example:
The gray figure is a rhombus (it was created by rotating the black rectangle by $2\alpha$ ). To get the inscribed rectangle with the greatest area, consider the case when the rhombus would be a square--the greatest area would be when each rectangle vertex splits the rhombus edge in half (because then it is also a square and that is the rectangle with greatest area and given perimeter).
But we can scale our rhombus (that may not be a square) so that is a square, obtain the solution there, and then scale back (the area will scale accordingly)! In conclusion the rectangle of greatest area will split the rhombus edges in half.
How to compute it? You could do it using the same approach:
\begin{align*}
\left[\begin{matrix}x'\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\y\end{matrix}\right] \\\
\left[\begin{matrix}x''\\\ -C_y\end{matrix}\right] &=
\left[\begin{matrix}\cos(-\alpha)&-\sin(-\alpha)\\\sin(-\alpha)&\cos(-\alpha)\end{matrix}\right]
\left[\begin{matrix}x\\\ -y\end{matrix}\right]
\end{align*}
However, one can do it simpler: the vertex of the inscribed rectangle splits the
gray edge in half, so $$2x\sin\alpha = C_y = 2y\cos\alpha\,.$$ This works if $C_x > C_y$, otherwise you need to do the same for $C_x$ instead. Moreover, even if $C_x > C_y$, you still need to check if the rotated rectangle fits into the boundary (because it may be that $C_x = C_y + \varepsilon $ ), if not, the solution from previous part will do.
Hope that helps ;-)
Best Answer
The formula you want in this case is as follows: $$ k = \cos\theta+\frac{W}{H}\sin\theta $$ Where: $H$ is the height, $W$ is the width, $\theta$ is the angle of rotation (from $0$ to $90$ degrees), and $k$ is the scaling factor.
Interestingly, this means that the "worst case" (largest scaling factor) corresponds to $$ \theta = \tan^{-1}\left(\frac{W}{H}\right) $$
Proof:
In fact, I overlooked what happens when $W<H$. In fact, we should have
$$ k = \begin{cases} \cos\theta+\frac{W}{H}\sin\theta & W\geq H\\ \cos\theta+\frac{H}{W}\sin\theta & W < H \end{cases} $$
For the second case, we could draw the above and still have overlap. The more mathematically concise (but computationally expensive) way to write this would be $$k = \cos\theta + \max\left\{\frac{W}{H},\frac{H}{W}\right\}\sin\theta$$