[Math] Rectify image from congruent planar shape objects

Question

I am implementing an algorithm to remove projective distortions on the following image.

I understand this is possible by applying the following transformation:

$$
\begin{matrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
l_1 & l_2 & l_3 \\
\end{matrix}
$$

Where $$ l_\infty=(\begin{matrix}l_1 & l_2 & l_3 \end{matrix})^T $$ is the line at the infinity. In a perspective image of a plane, the line at infinity on the world plane is imaged as the vanishing line of the plane.

The vanishing line can be computed by intersecting two vanishing points which can be computed in the following ways:

1. From the intersection of two sets of imaged parallel lines. But seems there are no two sets of imaged parallel lines on the image.
2. Given two intervals on a imaged line $$\lt0,a^\prime,a^\prime+b^\prime\gt$$ with a known length ratio $$ d(0,a^\prime):d(a^\prime,a^\prime+b^\prime)=0:a^\prime $$ Where I need to solve the system (up to scale) $$\left(\begin{matrix}0 \\ 1\end{matrix}\right)=\left(\begin{matrix}h11 & h12 \\ h21 & h22\end{matrix}\right) \left(\begin{matrix}0 \\ 1\end{matrix}\right)$$ $$\left(\begin{matrix}a \\ 1\end{matrix}\right)=\left(\begin{matrix}h11 & h12 \\ h21 & h22\end{matrix}\right) \left(\begin{matrix}a^\prime \\ 1\end{matrix}\right)$$ $$\left(\begin{matrix}a+b \\ 1\end{matrix}\right)=\left(\begin{matrix}h11 & h12 \\ h21 & h22\end{matrix}\right) \left(\begin{matrix}a^\prime+b^\prime \\ 1\end{matrix}\right)$$ And compute the vanishing point as $$x^\prime=\left(\begin{matrix}h11 & h12 \\ h21 & h22\end{matrix}\right) \left(\begin{matrix}0 \\ 1\end{matrix}\right)$$ But I don't understand this approach as I don't have the world points $$\lt0,a,a+b\gt$$ and I don't know what serves me for knowing the length ratio.
3. Using the cross ratio. Which I totally don't understand how could be possible to use in this case.

I would appreciate any insight about this.

Edit: isn't necessary to follow any particular approach just remind that the planar objects are irregular (no orthogonal angles in real world) congruent shapes (they have the same shape in real world)

Answer 1

I will assume that the two depicted bright polygons are congruent in the world plane. If they are not, then I guess you won't have enough information to reconstruct anything.

The first step is finding a projective transformation which maps one of the polygons onto the other. That transformation is uniquely defined using four points and their images, and I've described this computation in detail in another post. So choosing any four pairs of matching corners will give you that matrix. Originally I had assumed that this is the transformation you were interested in, but I had not read the question carefully enough. The transformation you just found describes the rotation of one polygon onto the other in the image plane.

Next, you look for fixed points of the transformation. You find these as eigenvectors of the transformation matrix. In $\mathbb C\mathrm P^2$ you should get three fixed points, one of them real and two complex and conjugate to one another. The real one is your center of rotation. The complex fixed points correspond to the ideal circle points $I=(1,i,0)^T$ and $J=(1,-i,0)^T$, which remain fixed under every similarity transformation.

By the way: The join of the two complex points corresponds to the vanishing line. Which in your image is way outside the picture area. But knowing the vanishing line is less useful than knowing the locations of $I$ and $J$ as we do, since these points can be used to define angles, so they will help you avoid skewing.

Now you can again choose four points and their images to define a projective transformation. This time, you map the complex fixed points to $I$ and $J$, and you map the center of rotation to wherever you want it to lie in your reconstructed image, e.g. the origin. You still have one point which you can choose arbitrarily. Its image will fix the scale and orientation of the reconstructed image. Strictly speaking, you don't have to take the center of rotation as one preimage point either: as long as you correctly map $I$ and $J$ from image coordinates to world coordinates, you can choose any two pairs of real points to uniquely define the projective transformation.

Counting degrees of freedom, the two arbitrarily chosen points above amount to four real degrees of freedom, matching the four degrees of freedom of a similarity transformation. Everything else is fixed. In particular, not only parallel lines but also angles are reconstructed.

---

I've just implemented the above description as a proof of concept using Cinderella. The center of rotation, drawn in green, appears to coincide with the upper boundary of your picture. The points P1, P2 and R2 were chosen arbitrarily.