This is not a trivial problem and requires a lot of effort. In other words, you need a computer. I will also assume that you are familiar with elementary calculations from analytical geometry. In particular you should be able to:
- Calculate coordinates of all corner points of both rectangles.
- Determine if two segments AB and CD intersect or not. Basically, you have equation of line AB and equation of line CD; if the intersection point M is between A and B and also between C and D, the segments do intersect.
- Find a line $p$ through point A perpendicular to line BC and find the intersection point D of line $p$ and line BC. You should also be able to determine if the point D lies between points B and C.
All of this is fairly trivial and I won't go into details.
Step 1: You have four segements of the first rectange and four segments of the second rectangle. By using (2) find out if any segment from the first rectangle intersects with any segment from the second rectangle. If any pair of segments has an intersection point, the rectangles overlap (or touch) and their distance is zero. If not, go to step 2.
Step 2: For every corner point from the first rectangle find distances to all segments from the second rectangle by applying (3). You have two particular cases:
Case 1: Line through corner point A of the first rectangle perpendicular to segment BC from the second rectangle intersects line BC at point D between points B and C. In this case:
$$d=AD$$
Case 2: Line through corner point A of the first rectangle perpendicular to segment BC from the second rectangle intersects line BC at point D which is not between points B and C. In this case:
$$d=\min(AB,AC)$$
This gives you four distances for every corner point of the first rectangle. For the whole first rectangle you will get a set of 4x4=16 different distances $(d_1,d_2,...,d_{16})$.
Step 3: Repeat step 2, but this time consider distances from corner points of the second rectangle to segments of the first rectangle. This gives another set of 16 distances: $(d_{17},d_{18},...,d_{32})$.
The distance between rectangles is:
$$D=\min(d_1,d_2,...,d_{32})$$
If I understand your problem correctly, the solution would be to find coordinates of all the corners of rectangle in a new other coordinate system where:
- Origin of the coordinate system is at the bottom left corner of the bottom left rectangle.
- The angle between x-axis of new coordinate system with respect to horizontal is $\omega$
As shown in above image you just need to get $(a, b)$ in that new coordinate system.
Now if you assume $(a, b)$ (all the corners of rectangle in new system) in orthogonal Cartesian system ($x'$-axis and $y$-axis with the same origin $(0, 0)$) as shown in figure you get your desired result with width and height preserved.
So the set of coordinates in this system is your final coordinates you want.
Shifting Origin and finding coordinates is left to you as an exercise!
Best Answer
I'll assume $0<a<b$ and $0\leq\phi\leq\frac\pi2$. All other cases can be reduced to this.
Case 1: $\phi < 2\arctan\left(\frac ab\right)$
In this case you have an octagonal intersection, with an area of
$$2\frac{2ab\sin(\phi) + \bigl(a^2 + b^2\bigr)\bigl(\cos(\phi) - 1\bigr)} {\sin(\phi)\,\cos(\phi)}$$
Case 2: $\phi \geq 2\arctan\left(\frac ab\right)$
In this case you get a quadrilateral intersection, with an area of
$$\frac{4a^2}{\sin(\phi)}$$
I obtained the above results using symbolic expressions for all coordinates, performing the computation using sage.