conventionselectromagnetismmagnetic fieldsVector Fields
According to right hand rule, If I put my thumb in the direction of the current flow and encircle myt other fingers, the direction of those finger will refer to the direction of the magnetic field. But why does this work?
I mean, why is the magnetic field created in that direcrion?
It comes from the cross product. Every situation in which you have to use the right-hand rule corresponds to some mathematical equation that involves a cross product.
In this case, the relevant equation is the Biot-Savart law,
$$\vec{B} = \frac{\mu_0}{4\pi}\int \frac{I\,\mathrm{d}\vec{l}\times\vec{r}}{r^3}$$
If you use the right-hand rule - the version you know to use for cross products - to compute the cross product of $\mathrm{d}\vec{l}$ and $\vec{r}$, like this (apologies for the crude drawing):
you will get the same result as from the curling-fingers form of the right-hand rule that you've shown in your question. That's by design, and in fact the curling-fingers form of the rule is just a shortcut for an infinite number of applications of the cross-product form.
In case you're curious, there is a deeper reason the right hand rule is needed for cross products. When you take the cross product of a vector and another vector, you get a slightly different mathematical object called a pseudovector or axial vector. The magnetic field at a point is the best-known example of a pseudovector. Despite looking just like a vector, pseudovectors actually represent a magnitude and an oriented plane, whereas an ordinary vector represents a magnitude and direction. Now, if you have a plane, and you want to represent it with an arrow, in a sense you can do that by picking the arrow to be perpendicular to the plane, and then your convention is that an arrow represents the plane that is perpendicular to it. But there are two arrows perpendicular to the plane; how do you choose which one to use? That's where the right-hand rule comes in. It plays a role in mapping a pseudovector (the thing that e.g. magnetic field really is) to a vector (the thing we use to represent e.g. magnetic field).
When current flows in a circle, we can use the right hand rule to find that the magnetic field points in one direction inside the loop, and in the other direction outside the loop (to be precise, the magnetic field lines wrap around the wire, as shown in the second diagram below). Thus, in the case of current carrying loops, if we curl the fingers of our right hand in the direction of the current flow, our right thumb will point in the direction of the loop's magnetic dipole moment, which is also the direction of the magnetic field at the center of the loop. The two pictures below show a current carrying loop in a top-down view and a side cross-sectional view. Hope they help. Thus, this other right hand rule is derived from the first one, it's just a quick short cut we can use for current carrying loops (or spinning charges).
Best Answer
It's an arbitrary choice, because the direction of $\vec B$ is not actually an observable.
Whenever you compute observables in electromagnetism --- for instance, whether two parallel currents are attracted or repelled, or whether two skewed currents experience an aligning torque or an anti-aligning torque --- you always find yourself using the right-hand rule an even number of times. For instance, you use the right-hand rule to find the direction of $\vec B$, then use the right-hand rule again to find the direction of $\vec v \times \vec B$. If you were to consistently use your left hand in every circumstance, you'd disagree with other people about the direction of $\vec B$, but you'd predict all of the same dynamics.
This property of electromagnetism, where it doesn't matter whether you use your right or left hand to compute the direction of a vector product, is known as "conservation of parity." While electromagnetism doesn't change under a parity transformation (which transforms your right hand into a left hand), that's not a generally true statement about the world: in the weak nuclear interaction, there are different rules for interacting particles with spin, depending on whether their spin axis is parallel to their momentum (i.e. "north pole forward") or antiparallel ("south pole forward").