Your question consists of three parts so let's answer them one by one.
1) Surface tension between liquid-solid and air-solid
why isn't all strange surface tension phenomenon seen at those surfaces?
Just a short note upfront: I don't know what you mean by 'strange surface tension phenomenon' so I will explain in general how surface tension is also present for the liquid-solid and the air-solid interface.
You are absolutely correct in saying that these interactions also exist at other interfaces, so also at the interface of the liquid and the container (solid) and at the interface of the air and the container. The main reason why this is not easy to see is that the solid is extremely rigid as compared to the liquid and gas and therefore doesn't significantly deform because of surface tension acting on it.
You can, however, see a signature of the surface tensions of gas-solid and liquid-solid in the case of a sessile drop. What happens there is that the droplet while adjust it's shape to a certain contact angle ($\theta$). According to the balance of there 3 surface tensions. This can be visualized as follows:
If you balance the surface tensions you get an equation which is called Young's equation: $\gamma_{sg}=\gamma_{sl}+\gamma_{lg}\cos\theta$. If, for example, the surface tension between the liquid and the solid is extremely high, this will result in a contact angle that is close to 0.
2) Direction of surface tension
how can surface tension be described as perpendicular forces to an imaginary line at the surface, isn't it supposed to act in all direction?
This is actually not true. Surface tension is a force parallel to the surface. The reason comes from the molecular origin of surface tension. Let's explain it in terms of the liquid-air interface. Liquid molecules prefer to have a certain amount of interaction with their surrounding liquid molecules. If a molecule is located at the air-liquid interface, it doesn't have sufficient interaction with the air molecules. To 'solve' this, the molecules at the surface will pull more strongly on the neighbors that are also on the surface, thus exerting a force parallel to the surface.
3) Capillarity
how can we assume the meniscus formed will be spherical and not defined by some other function?
I assume you meant to write the equation for capillary action: $h = \frac{2\gamma \cos \theta}{\rho g r}$. The assumption there is indeed that the meniscus is spherical. This is necessary to simplify the mathematics and be able to solve the liquid height in a capillary tube analytically.
You can estimate when it is still a reasonable assumption by calculating the capillary length: $\lambda_c=\sqrt{\frac{\gamma}{\rho g}}$ and comparing this to the radius of the tube. If the radius of the tube is below this length then surface tension is 'sufficiently dominant' to make sure that the meniscus is a spherical cap. You can see this capillary length in the equation for capillary action:
$$h = \frac{2\gamma \cos \theta}{\rho g r}= \frac{2 \lambda_c^2 \cos \theta}{r} $$
The loop has a curvature. As a result, when you take a small element of the loop, the tension force applied to it from both the sides is at an angle, which results in a force component to the inside.
Since your question asks for a rationale behind the tension force, I'll leave it here. You can now try yourself to find the tension by equating the outward force with the inward force. Or, if you are given the total outward force, you can integrate the tension force over the loop and then equate the total outward and inward forces.
Best Answer
Even though you may not have seen this before, but the string will not necessarily always behave the same.
If it is just the string and the liquid then you will get an effect called the Cheerios effect (Am. J. Phys. 73, 817 (2005)) that causes the string to clot. This is caused by the string locally deforming the surface due to a balance of buoyancy and wetting adhesion, which causes the surface energy to go up. Minimization of the surface energy drives the string to clot together such that it deforms the surface as little as possible.
Note that this clotting will only occur if the energy gain for by clotting is not balanced by an energetic contribution for the bending of the string i.e. if it would be a rubber like material it will probably not (fully) clot, but with a normal piece of cotton string I think it will.
A nice circular loop can form if Marangoni effects play as suggested by Mark Rovetta. In this case adding a detergent or a bit of oil inside the string will cause the surface tension to lower such that it is energetically favorable to create more surface area with this low surface tension, causing the liquid with oil/detergent to flow outwards and open up the string into a loop.
I have explained this in terms of energies (because that is so much easier when thinking about surface tension effects), but to answer the part of your question on the direction of the force: it acts perpendicular to thread because that is the direction of the energy gradient when clotting or expanding. The tension in the thread itself is indeed parallel to the thread as explanation by Mark.
Footnote: I just found out that this question is strongly related to Why does a cork float to the side of a glass?, in particular the answer by Pulsar is useful in this context.