I want to prove:
$B$ is similar to $A \Leftrightarrow m_A(x) = m_B(x)$ and $P_A(x) = P_B(x)$, where $m,P$ are the minimal and characteristic polynomial, respectively.
"$\Rightarrow$" Let $A$ to be similar to $B$, then they have the same rational canonical form. On the other hand, the characteristic polynomial is the product of the invariant factors, but the invariant factors of an $n \times n$ matrix over a field $F$ are the invariant factors of its canonical form. Since $A,B$ have the same rational canonical form, then the characteristic polynomials are the same, hence: $P_A(x) = P_B(x)$, but the minimal polynomial is the largest invariant factor, so the minimal polynomials are the same and every other invariant factor divides the minimal polynomial, hence: $m_A(x) = m_B(x)$. (This comes from the fact that similar matrices have the same determinant)
"$\Leftarrow$"
Let $A, B \in M_3(F)$ with the property that they have the same characteristic and minimal polynomial, that is:
$P_A(x) = P_B(x) = f(x),$
$m_A(x) = m_B(x) = g(x).$
we want to prove that $A$ and $B$ are similar. First, recall that since we have a $3 \times 3$ matrix, this implies that the characteristic polynomial will have degree $3$, $g | f$ and every irreducible factor of $f(x)$ appears in $g(x)$. Moreover the degree of $g(x)$ is at most $3$. To prove similarity, it is enough to prove that they have the same set of invariant factors, which implies that the companion matrices will be the same.
- $\deg(g(x)) = 3$
If the degree of $g(x)$ is $3$, basically we cannot do a lot, so, we have that: $f(x) = g(x)$, so in this case the minimal polynomial is just $f(x)$ which implies that they are similar. - $\deg(g(x)) = 2$
Notice that $f(x) = (x-a)g(x)$ for some $a \in F$ and $(x-a)$ is irreducible. On the other hand, every irreducible factor of the characteristic polynomial must appear in $g(x)$, which implies that: $g(x) = (x-a)(x-b)$ for some $b \in F$ and $f(x) = (x-a)^2 (x-b)$. Again, in this case the set of invariant factors is given by
$\{(x-a),(x-a)(x-b)\}$
Then, they share the same invariant factors, so the companion matrices are the same, hence there are similar. - $\deg(g(x)) = 1$
clearly, we have that: $g(x) = (x-a)$ for some $a \in F$. Then, the only possibility is that the invariant factors have the following form:
$$\{(x-a),(x-a),(x-a)\}$$
Then, the matrices, $A,B$ have the same set of invariant factors and the same companion matrix, which implies that they are similar
Best Answer
About your proof. The "only if" ($\Rightarrow$) direction is quite simple. The characteristic polynomial is (as you say) the product of the invariant factors, and the minimal polynomial is similarly the largest invariant factor; both then are determined by the rational canonical form (RCF). So your work for the minimal polynomial is not really needed; moreover it seems to go off in an completely wrong direction (what is the point of focussing on a single root $\lambda$ in $F$ of the characteristic polynomial, which you don't even know exists?), and I cannot follow much of it.
Your proof of the "if" direction is entirely wrong. The characteristic polynomial is certainly not the same as the RCF (they aren't even the same kind of thing), and matrices with the same characteristic polynomials can very well not be similar (for instance one could be diagonalisable but the other not), ever for the $3\times3$ case.
What you do know is that the characteristic polynomial is the product of the invariant factors, and the minimal polynomial is largest of them (which all the others divide). From this one cannot in general conclude that all invariant factors must correspond (so that the matrices will be similar), but now the $3\times3$ condition comes to the rescue. This means the characteristic polynomial$~\chi$ has degree$~3$, and this makes it impossible to obtain equal minimal polynomials yet distinct invariant factors. To be precise, if the common minimal polynomial$~\mu$ has degree at least$~2$, then any remaining invariant factors have as product the quotient $\chi/\mu$ which has degree $1$ at most; either (if $\deg \chi/\mu=0$) there are no other invariant factors, or (if $\deg \chi/\mu=1$) there is one remaining invariant equal to $\chi/\mu$. But this leaves only the case $\deg\mu=1$, in which case the remaining invariant factors must all be monic polynomials dividing$~\mu$, which forces them to be equal to$~\mu$.
A similar argument for the $4\times4$ or larger cases fails. For $n=4$, one could have $\deg\mu=2$ and then the remaining factor $\chi/\mu$ of degree $2$ could either itself be$~\mu$, or could split into two identical invariant factors of degree$~1$. For such a minimal counterexample one needs $\chi$ to be of the form $(X-\lambda)^4$ and $\mu=(X-\lambda)^2$; then the remaining invariant factors are either $(X-\lambda)^2$ or twice $X-\lambda$. You can see here an example where this happens (with $\lambda=0$).