trigonometry
A given square is rotated on its center point by 'z' degrees. A new square is added inside this at no angle, whose size is based on the perimeter of the containing square.
Is there a way to calculate my black square's dimensions, given the angle that the blue square was rotated by and blue's dimensions?
Let's say that the origin of a coordinate system is at the center of the original black rectangle. This is the center of the rotation and also the center of each of the green rectangles. Consider only the first quadrant (from the origin up and to the right). Exactly 1/4 of the black rectangle and each green rectangle is in the first quadrant. Let's further assume that, as in your picture, the angle of rotation is relatively small (say, less than 45°) and in the clockwise direction, and that $y_1<x_1$.
Let the vertex of the black rectangle that is in the first quadrant be $(a,b)=(\frac{1}{2}x_1,\frac{1}{2}y_1)$. Call the magnitude of the angle of rotation $\alpha$ ($0°<\alpha<45°$). The coordinates of the vertex of the red rectangle in the first quadrant are $(a\cos\alpha+b\sin\alpha,-a\sin\alpha+b\cos\alpha)$. The slope of the upper side of the red rectangle is $-\tan\alpha$. An equation for the upper side of the red rectangle in the first quadrant is $y-(-a\sin\alpha+b\cos\alpha)=-\tan\alpha(x-(a\cos\alpha+b\sin\alpha))$ for $0\le x\le a\cos\alpha+b\sin\alpha$. An equivalent simplified equation is $y=\sec\alpha(b-x\sin\alpha)$. This red upper side intersects the original black upper side at $x=b\cot\alpha(\sec\alpha-1)$.
So:
Assuming the first case:
edit: I originally forgot the factors of 4 on the area, moving from my picture of just one-fourth of things back to the whole rectangles. Similarly, note that my $x$- and $y$-values are half of your $x_2$ and $y_2$, respectively. The maximum possible area for the second case (where the vertex is at $(x,b)$) is $4b^2\cot\alpha(\sec\alpha-1)$, which Mathematica seems to be telling me cannot be greater than the maximum area for the first case under my assumptions about the problem.
First of all, there may be multiple rectangles that satisfy your conditions, e.g.
if you want specific angle of rectangle, or
if you want to specify a point on the boundary.
However, there is a special case where there is at most only once rectangle, i.e. if you assume that it has to touch both of your boundaries. In such case it is easy to compute it. Place the origin (point $(0,0)$) in the center of the boundary (the center of the circle), and let $(C_x,C_y)$ be the top right boundary corner (the boundary rectangle has size $2C_x \times 2C_y$ ). Let $(x,y)$ be the top right vertex of small rectangle, then it satisfies conditions ($\alpha > 0$ means counterclockwise rotation):
\begin{align*} \left[\begin{matrix}x'\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\y\end{matrix}\right] \\\ \left[\begin{matrix}C_x\\\ y'\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\ -y\end{matrix}\right] \end{align*}
where $x'$ and $y'$ are just placeholders. Extracting appropriate rows from those formulae, we can transform that into one equation (notice the lack of minus sign in the matrix):
\begin{align*} \left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\ y\end{matrix}\right] \end{align*}
with solution being:
\begin{align*} \left[\begin{matrix}\cos\alpha&\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]^{-1} \left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}x\\\ y\end{matrix}\right] \\\ \sec{2\alpha}\left[\begin{matrix}\cos\alpha&-\sin\alpha\\\ -\sin\alpha&\cos\alpha\end{matrix}\right] \left[\begin{matrix}C_x\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}x\\\ y\end{matrix}\right] \end{align*}
Please note, that this may not have a proper solution if $\alpha$ is to big!
Edit: Ok, I missed the comment about maximizing the area. Then again, consider this example:
The gray figure is a rhombus (it was created by rotating the black rectangle by $2\alpha$ ). To get the inscribed rectangle with the greatest area, consider the case when the rhombus would be a square--the greatest area would be when each rectangle vertex splits the rhombus edge in half (because then it is also a square and that is the rectangle with greatest area and given perimeter). But we can scale our rhombus (that may not be a square) so that is a square, obtain the solution there, and then scale back (the area will scale accordingly)! In conclusion the rectangle of greatest area will split the rhombus edges in half.
How to compute it? You could do it using the same approach:
\begin{align*} \left[\begin{matrix}x'\\\ C_y\end{matrix}\right] &= \left[\begin{matrix}\cos\alpha&-\sin\alpha\\\sin\alpha&\cos\alpha\end{matrix}\right]\left[\begin{matrix}x\\\y\end{matrix}\right] \\\ \left[\begin{matrix}x''\\\ -C_y\end{matrix}\right] &= \left[\begin{matrix}\cos(-\alpha)&-\sin(-\alpha)\\\sin(-\alpha)&\cos(-\alpha)\end{matrix}\right] \left[\begin{matrix}x\\\ -y\end{matrix}\right] \end{align*}
However, one can do it simpler: the vertex of the inscribed rectangle splits the gray edge in half, so $$2x\sin\alpha = C_y = 2y\cos\alpha\,.$$ This works if $C_x > C_y$, otherwise you need to do the same for $C_x$ instead. Moreover, even if $C_x > C_y$, you still need to check if the rotated rectangle fits into the boundary (because it may be that $C_x = C_y + \varepsilon $ ), if not, the solution from previous part will do.
Hope that helps ;-)
Best Answer
The answer is surprisingly simple: $$s = \frac{L}{\cos \theta + \sin \theta},$$ for a enclosing square's side length of $L$ and an angle of rotation $\theta$ between $0$ and $\pi/2$ radians. But I will leave it to you to obtain the derivation of this result.