I have seen similar questions, but none of the answers relate to my difficulty, which I will now proceed to convey.
Let $(M,g)$ be a Riemannian manifolds. The Levi-Civita connection is the unique connection that satisfies two conditions: agreeing with the metric, and being torsion-free.
Agreeing with the metric is easy to understand. This is equivalent to the parallel transport associated with the connection to satisfy that the isomorphism between tangent spaces at different points along a path are isometries. Makes sense.
Let's imagine for a second what happens if we stop with this condition, and take the case of $M=\mathbb{R}^2$, with $g$ being the usual metric. Then it's easy to think of non-trivial ways to define parallel transport other than the one induced by the Levi-Civita connection.
For example, imagine the following way to do parallel transport: if $\gamma$ is a path in $\mathbb{R}^2$, then the associated map from $TM_{\gamma(s)}$ to $TM_{\gamma(t)}$ will be a rotation based with angle $p_2(\gamma(s))-p_2(\gamma(t))$, where $p_i$ is the projection of $\mathbb{R}^2$ onto the $i^\text{th}$ coordinate.
So I guess torsion-free-ness is supposed to rule this kind of example out.
Now I'm somewhat confused. One of the answers to a similar question that any two connections that satisfy that they agree with the metric satisfy that they have the same geodesics, and in that case choosing a torsion-free one is just a way of choosing a canonical one. That seems incorrect, as $\gamma(t)=(0,t)$ is a geodesic of $\mathbb{R}^2$ with the Levi-Civita connection but not the one I just described…
Let's think from a different direction. In the case of $\mathbb{R}^2$, if $\nabla$ is the usual (and therefore Levi-Civita) connection then $\nabla_XY$ is just $XY$, and $\nabla_YX$ is just $YX$. So of course we have torsion-free-ness.
So I guess one way to think of torsion-free-ness is saying that you want the parallel transport induced by the connection to be the one associated with $\mathbb{R}^n$ via the local trivializations.
Except that this seems over-simplistic: torsion-free-ness is weaker than the condition that $\nabla_XY=XY$ and $\nabla_YX=YX$. So why this crazy weaker condition that $\nabla_XY-\nabla_YX=[X, Y]$? What does that even mean geometrically? Why is this sensible? How would say that in words that are similar to "it means that the connection is the connection induced from the trivializations" except more correct than that?
Best Answer
I think that the literal answer is that the Levi-Civita connection of $g$ is trying to describe the metric $g$ and nothing else. It is the only connection-assignment that is uniquely defined by the metric and its first derivatives and nothing else, in the sense that, if you have a diffeomorphism-equivariant assignment $g\to C(g)$ where $C(g)$ is a connection that depends only on $g$ and its first derivatives, then $C(g)$ is the Levi-Civita connection.
Note that the restriction to first derivatives is necessary. For example, there is a unique connection on $TM$ that is compatible with $g$ and satisfies $$ \nabla_XY -\nabla_YX - [X,Y] = \mathrm{d}S(X)\,Y - \mathrm{d}S(Y)\,X, $$ where $S= S(g)$ is the scalar curvature of $g$. However, this canonical connection depends on three derivatives of $g$.
Meanwhile, connections with torsion can arise naturally from other structures: For example, on a Lie group, there is a unique connection for which the left-invariant vector fields are parallel and a unique connection for which the right-invariant vector fields are parallel. When the identity component of the group is nonabelian, these are distinct connections with nonvanishing torsion, while their average is a canonical connection that is torsion-free. (This latter connection need not be metric compatible, of course.) A more well-known example is the unique connection associated to an Hermitian metric on a complex manifold that is compatible with both the metric and the complex structure and whose torsion is of type (0,2).
It's not unreasonable to ask whether imposing the torsion-free condition, just because you can, right out of the gate is too restrictive. Einstein tried for years to devise a 'unified field theory' that would geometrize all of the known forces of nature by considering connections compatible with the metric (i.e., the gravitational field) that had torsion. There is a book containing the correspondence between Einstein and Élie Cartan (Letters on absolute parallelism) in which Einstein would propose a set of field equations that would constrain the torsion so that they describe the other known forces (just as the Einstein equations constrain the gravitational field) and Cartan would analyze them to determine whether they had the necessary 'flexibility' to describe the known phenomena without being so 'flexible' that they couldn't make predictions. It's very interesting reading.
This tradition of seeking a physical interpretation of torsion has continued, off and on, since then, with several attempts to generalize Einstein's theory of gravity (aka, 'general relativity'). Some of these are described in Misner, Thorne, and Wheeler, and references are given to others. In fact, quite recently, Thibault Damour (IHÉS), famous for his work on black holes, and a collaborator have been working on a gravitational theory-with-torsion, which they call 'torsion bigravity'. (See arXiv:1906.11859 [gr-qc] and arXiv:2007.08606 [gr-qc].) [To be frank, though, I'm not aware that any of these alternative theories have made any predictions that disagree with GR that have been verified by experiment. I think we all would have heard about that.]
I guess the point is that 'why impose torsion-free?' is actually a very reasonable question to ask, and, indeed, it has been asked many times. One answer is that, if you are only trying to understand the geometry of a metric, you might as well go with the most natural connection, and the Levi-Civita connection is the best one of those in many senses. Another answer is that, if you have some geometric or physical phenomenon that can be captured by a metric and another tensor that can be interpreted as (part of) the torsion of the connection, then, sure, go ahead and incorporate that information into the connection and see where it leads you.
Remark on connections with the same geodesics: I realize that I didn't respond to the OP's confusion about connections with the same geodesics vs. compatible with a metric $g$ but with torsion. (I did respond in a comment that turned out to be wrong, so I deleted it. Hopefully, this will be better.)
First, about torsion (of a connection on TM). The torsion $T^\nabla$ of a (linear) connection on $TM$ is a section of the bundle $TM\otimes\Lambda^2(T^*M)$. Here is an (augmented) Fundamental Lemma of (pseudo-)Riemannian geometry:
Lemma 1: If $g$ is a (nondegenerate) pseudo-Riemannian metric on $M$ and $\tau$ is a section of $TM\otimes\Lambda^2(T^*M)$, then there is a unique linear connection $\nabla$ on $TM$ such that $\nabla g = 0$ and $T^\nabla = \tau$.
(The usual FLRG is the special case $\tau=0$.) Note that this $\nabla$ depends algebraically on $\tau$ and the $1$-jet of $g$. The proof of Lemma 1 is the usual linear algebra.
Second, if $\nabla$ and $\nabla^*$ are two linear connections on $TM$, their difference is well-defined and is a section of $TM\otimes T^*M\otimes T^*M$. Specifically $\nabla^* - \nabla:TM\times TM\to TM$ has the property that, on vector fields $X$ and $Y$, we have $$ \left({\nabla^*} - \nabla\right)(X,Y) = {\nabla^*}_XY-\nabla_XY. $$
Lemma 2: Two linear connections, $\nabla$ and $\nabla^*$ have the same geodesics (i.e., each curve $\gamma$ is a geodesic for one if and only if it is a geodesic for the other) if and only if $\tilde\nabla - \nabla$ is a section of the subbundle $TM\otimes\Lambda^2(T^*M)\subset TM\otimes T^*M\otimes T^*M$.
Proof: In local coordinates $x = (x^i)$, let $\Gamma^i_{jk}$ (respectively, $\tilde\Gamma^i_{jk}$) be the coefficients of $\nabla$0 (respectively, $\tilde\nabla$). Then $$ \tilde\nabla-\nabla = (\tilde\Gamma^i_{jk}-\Gamma^i_{jk})\ \partial_i\otimes \mathrm{d}x^j\otimes\mathrm{d}x^k. $$ Meanwhile, a curve $\gamma$ in the $x$-coordinates is a $\nabla$-geodesic (respectively, a $\tilde\nabla$-geodesic) iff $$ \ddot x^i + \Gamma^i_{jk}(x)\,\dot x^j\dot x^k = 0\qquad (\text{respectively},\ \ddot x^i + \tilde\Gamma^i_{jk}(x)\,\dot x^j\dot x^k = 0). $$ These are the same equations iff $(\tilde\Gamma^i_{jk}(x)-\Gamma^i_{jk}(x))\,y^jy^k\equiv0$ for all $y^i$, i.e., iff $$ {\tilde\nabla}-\nabla = \tfrac12({\tilde\Gamma}^i_{jk}-\Gamma^i_{jk})\ \partial_i\otimes \mathrm{d}x^j\wedge\mathrm{d}x^k.\quad \square $$
Finally, we examine when two $g$-compatible connections have the same geodesics:
Lemma 3: If $g$ is a nondegenerate (pseudo-)Riemannian metric, and $\nabla$ and $\nabla^*$ are linear connections on $TM$ that satisfy $\nabla g = \nabla^*g = 0$, then they have the same geodesics if and only if the expression $$ \phi(X,Y,Z) = g\bigl( X,(\nabla^*{-}\nabla)(Y,Z)\bigr) $$ is skew-symmetric in $X$, $Y$, and $Z$.
Proof: $\nabla g = \nabla^* g = 0$ implies $\phi(X,Y,Z)+\phi(Z,Y,X)=0$, while they have the same geodesics if and only if $\phi(X,Y,Z)+\phi(X,Z,Y)=0$.
Corollary: If $g$ is a nondegenerate (pseudo-)Riemannian metric, then the space of linear connections $\nabla$ on $TM$ that satisfy $\nabla g = 0$ and have the same geodesics as $\nabla^g$, the Levi-Civita connection of $g$, is a vector space naturally isomorphic to $\Omega^3(M)$, the space of $3$-forms on $M$.