First of all, there may be multiple rectangles that satisfy your conditions, e.g.
if you want specific angle of rectangle, or
if you want to specify a point on the boundary.
However, there is a special case where there is at most only once rectangle, i.e. if you assume that it has to touch both of your boundaries. In such case it is easy to compute it. Place the origin (point $(0,0)$) in the center of the boundary (the center of the circle), and let $(C_x,C_y)$ be the top right boundary corner (the boundary rectangle has size $2C_x \times 2C_y$ ). Let $(x,y)$ be the top right vertex of small rectangle, then it satisfies conditions ($\alpha > 0$ means counterclockwise rotation):
\begin{align*}
\left[\begin{matrix}x'\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\y\end{matrix}\right] \\\
\left[\begin{matrix}C_x\\\ y'\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\ -y\end{matrix}\right]
\end{align*}
where $x'$ and $y'$ are just placeholders. Extracting appropriate rows from those formulae, we can transform that into one equation (notice the lack of minus sign in the matrix):
\begin{align*}
\left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\ y\end{matrix}\right]
\end{align*}
with solution being:
\begin{align*}
\left[\begin{matrix}\cos\alpha&\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]^{-1}
\left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}x\\\ y\end{matrix}\right] \\\
\sec{2\alpha}\left[\begin{matrix}\cos\alpha&-\sin\alpha\\\ -\sin\alpha&\cos\alpha\end{matrix}\right]
\left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}x\\\ y\end{matrix}\right]
\end{align*}
Please note, that this may not have a proper solution if $\alpha$ is to big!
Edit: Ok, I missed the comment about maximizing the area. Then again, consider this example:
The gray figure is a rhombus (it was created by rotating the black rectangle by $2\alpha$ ). To get the inscribed rectangle with the greatest area, consider the case when the rhombus would be a square--the greatest area would be when each rectangle vertex splits the rhombus edge in half (because then it is also a square and that is the rectangle with greatest area and given perimeter).
But we can scale our rhombus (that may not be a square) so that is a square, obtain the solution there, and then scale back (the area will scale accordingly)! In conclusion the rectangle of greatest area will split the rhombus edges in half.
How to compute it? You could do it using the same approach:
\begin{align*}
\left[\begin{matrix}x'\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\y\end{matrix}\right] \\\
\left[\begin{matrix}x''\\\ -C_y\end{matrix}\right] &=
\left[\begin{matrix}\cos(-\alpha)&-\sin(-\alpha)\\\sin(-\alpha)&\cos(-\alpha)\end{matrix}\right]
\left[\begin{matrix}x\\\ -y\end{matrix}\right]
\end{align*}
However, one can do it simpler: the vertex of the inscribed rectangle splits the
gray edge in half, so $$2x\sin\alpha = C_y = 2y\cos\alpha\,.$$ This works if $C_x > C_y$, otherwise you need to do the same for $C_x$ instead. Moreover, even if $C_x > C_y$, you still need to check if the rotated rectangle fits into the boundary (because it may be that $C_x = C_y + \varepsilon $ ), if not, the solution from previous part will do.
Hope that helps ;-)
Suppose that the unknown width and height are $x$ and $y$, and you’re given a diagonal $d$ and a ratio $m:n$ of width to height. That ratio means that the width is $\frac{m}{n}$ times the height, so you know that $x=\frac{m}{n}y$. You get a second relationship between $x$ and $y$ from the Pythagorean theorem: $x$, $y$, and $d$ are the lengths of the two legs and the hypotenuse of a right triangle, so $x^2+y^2=d^2$.
Now substitute $\frac{m}{n}y$ for $x$ in this second equation to get $\displaystyle\left(\frac{m}{n}y\right)^2 + y^2 = d^2$. Simplifying this, you get in turn: $$\frac{m^2}{n^2}y^2 + y^2 = d^2,$$ $$\left(\frac{m^2}{n^2}+1\right)y^2 = d^2,$$ $$\left(\frac{m^2+n^2}{n^2}\right)y^2=d^2,$$ and $$(m^2+n^2)y^2=d^2n^2.$$ Finally, solve for $y$: $\displaystyle y^2 = \frac{d^2n^2}{m^2+n^2}$, so $y=\displaystyle\frac{dn}{\sqrt{m^2+n^2}}$.
Once you have a numerical value for $y$, you can plug it into $x=\frac{m}{n}y$ to get a value for $x$.
(Or you can do that symbolically: $\displaystyle x=\frac{m}{n}\cdot \frac{dn}{\sqrt{m^2+n^2}} =$ $\displaystyle\frac{dm}{\sqrt{m^2+n^2}}$.)
Best Answer
It's probably bugging out because $\cos 45^\circ = \sin 45^\circ$, so the denominator becomes zero. I can't imagine why it would return exactly $64$ though.
In any case, there probably isn't any way to do it for exactly $45^\circ$. For any given square, there are lots of rectangles of different sizes that fit exactly in it at $45^\circ$. So just from knowing the size of the square, you can't recover the dimensions of the rectangle. You can do it if you know the aspect ratio of the rectangle, though.