When drawing Feynman diagrams, it is important to fix the incoming and outgoing particles and their momenta. For the inexperienced, this is ideally done before drawing the rest of the diagram in order to avoid confusions like yours. So, let's assign each state some momentum: Let's give the electron in the upper left corner (4-)momentum $p_1$, lower left $p_2$, upper right $p_3$ and lower right $p_4$. Let us consider all momenta as 'flowing' from left to right (this is an arbitrary choice that makes no difference to the actual calculations).
Then, it immediately becomes clear why the two diagrams are different. For instance, check out the momentum on the photon line (call it $q$, and imagine it flowing top to bottom) that connects both vertices. Conservation of (4-)momentum forces always applies, so we can write it in terms of the momenta flowing into the vertices. For the $t$-channel diagram, it is clear that $q=p_1-p_3=p_2-p_4\equiv \sqrt{t}$. However, for the $u$-channel diagram $q=p_1-p_4=p_2-p_3\equiv \sqrt u$. Thus, the photon carries a different momentum. When carrying through the calculation, it will be seen that this makes a serious difference.
There is a more general point to be made here: Feynman diagrams can be very misleading, especially to those that are not (all too) familiar with the actual calculations that they visually represent. When it really comes down to it, it is of vital importance to realize that the diagrams should always be considered as secondary aids in performing calculations, which can just as well be done without drawing anything (it just takes a lot longer): Always be very careful when interpreting the diagrams.
Crossing symmetry tells you that you should not only exchange $$p_2\leftrightarrow -p_4$$
in the amputated matrix elements, but also replace the wavefunction polarizations
$$
u^{\pm}(p_2)\rightarrow v^{\mp}(p_4)
$$
(where the spin polarizations have been reversed), and finally multiply the amplitude for a factor $-1$ (Since you are crossing a fermionic pair).
All of this is equivalent to the simple $s\leftrightarrow u$ only for the forward scattering at $t=0$ or for the scattering averaged over all the polarizations.
Best Answer
You only need to include $u$ and $t$ channels when the final particles are indistinguishable (classically thinking, you don't know which blip in your detector came from which initial particle). Since $e^-$ and $e^+$ are distinguishable, you will always know which one is which and only $t$ channel contributes to the scattering amplitude.