Whenever I'm solving the hill cipher's key the final matrix is not in the original form. When I do the one from Wikipedia and also the one that I made myself neither comes back in the original form for the encryption or decryption key.
First Wikipedia example.
C=Cipher-text Matrix.
P=Plain-text Matrix.
C=
$\begin{bmatrix}
7&8& \\
0&19 \\
\end{bmatrix}$
P=$\begin{bmatrix}
7&4\\
11&15 \\
\end{bmatrix}$
To calculate the decryption key I have to setup the formula like so.
$D = [C]^{-1} ~\cdot P$
Then I calculate the modular multiplicative inverse of C as follows.
$[C]^{-1} = \det[C]^{-1} \cdot adj([C])$
$det[C]= (ad – bc) \mod 26$
$7*19 – 8*0 \mod 26 = 133 \mod 26 = 3 $
mod inverse of 3 mod 26 is 9.
$adj([C]) = \begin{bmatrix}19&-8\\0&7 \end{bmatrix}$
$[C]^{-1} =9 \cdot \begin{bmatrix}19&-8\\0&7 \end{bmatrix} \mod 26 \Rightarrow \begin{bmatrix}15&6\\0&11\end{bmatrix} $
$D=\begin{bmatrix}15&6 \\ 0&11 \end{bmatrix} \cdot \begin{bmatrix}7&4\\11&15\end{bmatrix} \mod 26 \Rightarrow \begin{bmatrix}129&255\\44&165\end{bmatrix} \mod 26 \Rightarrow D = \begin{bmatrix} 15&20\\17&9\end{bmatrix}$
Wikipedia's decryption matrix though is $\begin{bmatrix}15&17\\20&9\end{bmatrix}$
It seems to hold true for all matricies that I calculate that the end result matrix is $\begin{bmatrix}a&c\\b&d\end{bmatrix}$ every single time. I don't know if this is normal or not but I don't get it.
My own matricies.
$K=\begin{bmatrix}7&11\\8&11\end{bmatrix}$
$P=\begin{bmatrix}7&11\\4&11\end{bmatrix}$
$C=\begin{bmatrix}15&16\\22&1\end{bmatrix}$
$D=\begin{bmatrix}25&1\\22&23\end{bmatrix}$
If I convert my $C^{-1}$ and rotate it to instead be $\begin{bmatrix}a&c\\b&d\end{bmatrix}$ then I get back the encryption key correctly. I don't know what's going on with it though as the vectors are setup like wikipedia.
P.S.
I'm writing a lab to show the rest of the students how I solved a hill cipher CTF challenge utilizing the KPA against it but it seems like I've forgotten how in the world I solved it. My goal is to make it so that everyone at my community college who's interested in doing such events has the knowledge as to how to do such events. Somehow, somewhere I'm rotating things and I don't know how/why/where. I had it working Tuesday Morning at 01:30 when I woke up with the answer and did it on my phone's calculator but I didn't write it down and now I'm back into the same boat again.
Best Answer
Call the encryption matrix $E$. Then $E \cdot P = C$, so $E = C \cdot P^{-1}$ (so $P$ has to be invertible, which it is), or, if you really want the decryption matrix $D$ only, start from $D \cdot C = P$ and get $D = P \cdot C^{-1}$, which is in another order as you have.
$$C= \begin{bmatrix} 7&0 \\ 8&19 \\ \end{bmatrix}$$
(Using the Wikipedia preferred way of having the plain/cipher text in column vectors and we multiply $E$ on the right by what we're encrypting. $P$ is also transposed because of that)
And indeed $\det(C)=7 \cdot 19 = 3 \pmod{26}$ and its inverse is $9$.
So $$C^{-1} =9 \cdot \begin{bmatrix}19& 0 \\ -8 & 7 \end{bmatrix} \pmod {26} = \begin{bmatrix}15&0 \\6 &11\end{bmatrix} $$
which checks out.
So the decryption matrix becomes $$D = P \cdot C^{-1}= \begin{bmatrix} 7&11 \\ 4&15 \\ \end{bmatrix} \cdot \begin{bmatrix}15&0\\6 &11\end{bmatrix} = \begin{bmatrix} 15 &17\\ 20 & 9 \end{bmatrix}$$