A short answer, is that to estimate interaction energy (which says if same charges attract or repel), you use propagators. Propagators come from the expression of Lagrangians. Finally, the time derivative part for dynamical freedom degrees in the action must be positive, and this has a consequence on the sign of the Lagrangian.
Choose a metrics $(1,-1,-1,-1)$
For instance, for scalar field (spin-0), we have ($i=1,2,3$ representing the spatial coordinate) the : $$S = \int d^4x ~(\partial_0 \Phi\partial^0 \Phi+\partial_i \Phi\partial^i \Phi)$$
Here, the time derivative part of the action is positive (because $g_{00}=1$), so all is OK.
When we calculate energy interaction for particles wich interact via a spin-0 field, one finds that same charges attract each other.
Now, take a spin-1 Lagrangian (electromagnetism):
$$S \sim \int d^4x ~(\partial_\mu A_\nu - \partial_\nu A_\mu) (\partial^\mu A^\nu - \partial^\nu A^\mu)$$
The dynamical degrees of freedom are (some of) the spatial components $A_i$, so the time derivative of the dynamical degrees of freedom is :
$$S \sim \int d^4x ~\partial_0 A_i \partial^0 A^i$$
Now, there is a problem, because this is negative (because $g_{ii} = -1$), so to have the correct action, you must add a minus sign :
$$S \sim -\int d^4x ~ (\partial_\mu A_\nu - \partial_\nu A_\mu) (\partial^\mu A^\nu - \partial^\nu A^\mu)$$
This sign has a direct consequence on the propagators, and it has a direct consequence on interaction energy, which is calculated from propagators.
This explains while same charges interacting via a spin-0 (or spin-2) field attract, while same charges interacting via a spin-1 field repel.
See Zee (Quantum Field Theory in a nutshell), Chapter 1.5, for a complete discussion.
I believe you are assuming the magnetic field to have the same direction as the magnetic force. That is not correct.
An electric field $\vec E$ is in the same direction as the electric force $\vec F_e$. We can see that directly from the formula that ties them together:
$$\vec F_e=q\vec E$$
$q$ is charge. This is a direct linear relationship. The two are proportional, and the direction of one is the same as the other.
A magnetic field $\vec B$ on the other hand is not in the same direction as the magnetic force $\vec F_m$. In fact, it is perpendicular. We again see that from the formula:
$$\vec F_m=q\vec v \times \vec B$$
$q$ is a charge with speed $\vec v$. This is a cross-product; the result from a cross is always perpendicular to both vectors. $\vec F_m$ is perpendicular to both the motion direction $\vec v$ of this charge, and to the magnetic field $\vec B$. This is a 3D-consideration. Sometimes you might see another version of this formula, though, which is the case when many electrons moving - in other words, in the case of a current $\vec I$:
$$\vec F_m=l\vec I \times \vec B$$
$l$ is length of the wire that carries the current. Directions are the same, as you can see; force is perpendicular to these moving electrons (the current) and to the field.
Have a look at this illustration (source):
Charge moving through a wire as a current, and they set up a magnetic field. In each case, the magnetic force is perpendicularly towards the wire or blue line. Materials being attracted are not being attracted "around" the wire, but directly into it. The force is perpendicular, even though the field is around the wire.
This is the way, people in physics use the concept of magnetic field lines. It was chosen so, since it describes the situation well - for example, you see the cirular manner, they are drawn in; magnetic field lines are never-ending, but circular. This namely describes that the magnetic force is equal in size everywhere on this circular path, which is the case in reality. But even though it is the same size on the line, it is not the same direction. That is a fundamental difference from the maybe more intuitive concept of electric field lines.
Many such field lines can be googled for objects, wires, setups, magnets etc. There are many examples out there. And every time you see them, think of the force as being perpendicular to those lines. Not along them!
I hope this explanation gives some insight and helps out the confusion a bit.
Best Answer
You can't use the formula in Coulomb's law to compute the force between two magnets