Your image is 2D, so we need only work with 2x2 matrices. Notice how all your other matrices have that 1 floating around? Unnecessary in the 2-D case.
A rotation matrix is a matrix of the form $\begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta\end{pmatrix}$.
A pixel given by $(x,y)$ can be rotated simply by multiplying:
$$\begin{pmatrix} x' \\ y' \end{pmatrix} = \begin{pmatrix} \cos \theta & -\sin \theta \\ \sin \theta & \cos \theta\end{pmatrix}\begin{pmatrix} x \\ y \end{pmatrix}.$$
Multiplying this out, you get
$$\begin{align*}
x' &= \cos \theta x - \sin \theta y, \\
y' &= \sin \theta x + \cos \theta y.
\end{align*}$$
Even though we have 1 unknown ($\theta$), let's treat it as if we have two: $\alpha = \cos \theta, \beta = \sin \theta$.
This gives us a new linear system:
$$\begin{pmatrix} x & -y \\ y & x\end{pmatrix}\begin{pmatrix} \alpha \\ \beta \end{pmatrix} = \begin{pmatrix} x' \\ y'\end{pmatrix}.$$
Fill in your values $x,y, x', y'$ and solve:
A = [x -y; y x];
b = [x1; y1];
v = A\b;
alpha = v(1);
beta = v(2);
theta = acos(alpha); % or theta = asin(beta)
As I googled, I think I realized what you need. Every row is a 2D point. So "rotation" and "translation" are performed to EVERY row. So given two sets of points written in the matrices $A$ and $B$ you want to find a rotation matrix $R = \left(\begin{matrix} \cos{T} & -\sin{T} \\ \sin{T} & \cos{T} \end{matrix}\right)$ and a translation vector $tr = (t_x, t_y)^t$ such that for every row $B_i$ of $B$ and every row $A_i$ of $A$ you have $B_i^t = R.A_i^t + tr$.
Now, if $R$ and $tr$ exist and if you have two distinct rows from $B$ and $A$ you will have that
$$ B_1^t = R.A_1^t+tr, \qquad B_2^t = R.A_2^t+tr$$
from where after subtraction you will obtain $B_2^t-B_1^t = R.(A_2^t-A_1^t)$ which is a $2\times 2$ linear system for the "variables" $\cos{T}$ and $\sin{T}$. If your points are distinct, you will obtain unique solution for $R$ (and you know that an angle $T$ is determined uniquely by its $\sin$ and $\cos$). Once you have $R$ determined you obtain $tr$ from the first equation: $tr = B_1^t-R.A_1^t$.
Best Answer
You can certainly use whatever you'd like. However, this won't produce an even-looking image. Why? Look at a table of what input maps to what output: $$\begin{array}{|c|c|}\hline \text{Input} & \text{Output} \\ \hline 1 & 1 \\ \hline \vdots & \vdots \\ \hline 4 & 2 \\ \hline \vdots & \vdots \\ \hline 9 & 3 \\ \hline \vdots & \vdots \\ \hline 16 & 4 \\ \hline \vdots & \vdots \\ \hline 25 & 5 \\\hline \end{array}$$
Ok. So that was interesting and trivial. What does this mean? Notice that the input maps $3$ digits to $1$, $5$ digits to $2$, $7$ digits to $3$, etc. This indicates that the image will be more weighted in the lighter/higher colors.
Another way to look at this is to say that the ideal image transformation would have the midpoint be the same on both. That is, if you plug $\frac{2^{16}}{2}$ into the transformation, you'd get back $\frac{2^8}{2}$. The same goes for all other denominators: $\frac{2^{16}}{4} \to \frac{2^8}{4}$, etc.
You may want to look at using the $\log_2(x)$ function... (not sure how this would pan out, but it could work).