This is an excellent question. As indicated by the MathOverflow link in the comments, there are many ways to think about torsion and torsion-freeness. At the risk of being repetitive, allow me to summarize some of these, adding my own thoughts.
Throughout, we let $M$ be a smooth manifold, $\nabla$ a connection on $TM$, and $$T^\nabla(X,Y) = \nabla_XY - \nabla_YX - [X,Y]$$
its torsion tensor field. We let $X$, $Y$ denote vector fields.
Initial Observations
(1) Parallel coordinates
Torsion (at a point) can be seen as the obstruction to the existence of parallel coordinates (at that point):
Fact: Let $p \in M$. Then $T^\nabla|_p = 0$ if and only if there exists a coordinate system $(x^i)$ centered at $p$ such that $\nabla \partial_i |_p = 0$.
The point here is that if $T^\nabla = 0$, then any parallel frame is commuting (i.e.: $\nabla E_i = 0$ $\forall i$ $\implies$ $[E_i, E_j] = 0$ $\forall i,j$), hence is a coordinate frame (by the "Flowbox Coordinate Theorem").
(2) Commuting of second partials
The following two facts indicate that torsion can be thought of as the obstruction to (certain types of) second partial derivatives commuting.
For a smooth function $f \colon M \to \mathbb{R}$, recall that its covariant Hessian (or second covariant derivative) is the covariant $2$-tensor field defined by
$$\text{Hess}(f) := \nabla \nabla f = \nabla df.$$
Explicitly, $\text{Hess}(f)(X,Y) = (\nabla_X df)(Y) = X(Yf) - (\nabla_XY)(f)$.
Fact [Lee]: The following are equivalent:
(i) $T^\nabla = 0$
(ii) The Christoffel symbols of $\nabla$ with respect to any coordinate system are symmetric: $$\Gamma^k_{ij} = \Gamma^k_{ji}$$
(iii) The covariant Hessian of any smooth function $f$ is symmetric: $$\text{Hess}(f)(X,Y) = \text{Hess}(f)(Y,X)$$
Torsion-freeness also implies another kind of symmetry of second partials:
Symmetry Lemma [Lee]: If $T^\nabla = 0$, then for every smooth family of curves $\Gamma \colon (-\epsilon, \epsilon) \times [a,b] \to M$, we have
$$\frac{D}{ds} \frac{d}{dt} \Gamma(s,t) = \frac{D}{dt} \frac{d}{ds} \Gamma(s,t).$$
I don't know for certain whether the converse to the Symmetry Lemma is true, but I imagine it is.
Some Heuristic Interpretations
(i) "Twisting" of parallel vector fields along geodesics
Suppose we have a connection $\nabla$ on $\mathbb{R}^n$ whose geodesics are lines, but that has torsion. One could then imagine that parallel translating a vector along a line results in the vector "spinning" along the line, as if one were holding each end of a string and rolling it between our fingers.
An explicit example of such a connection is in the MathOverflow answer linked in the comments.
The justification for why this interpretation should be believed in general will be discussed below in (B).
On the MO thread, Igor Belegradek points out two related facts:
Fact [Spivak]:
(1) Two connections $\nabla^1$, $\nabla^2$ on $TM$ are equal if and only if they have the same geodesics and torsion tensors.
(2) For every connection on $TM$, there is a unique torsion-free connection with the same geodesics.
(ii) Closing of geodesic parallelograms (to second order)
Let $v, w \in T_pM$ be tangent vectors. Let $\gamma_v$ and $\gamma_w$ be the geodesics whose initial tangent vectors are $v$, $w$, respectively. Consider parallel translating the vector $w$ along $\gamma_v$, and also the vector $v$ along $\gamma_w$. Then the tips of the resulting two vectors agree to second order if and only if $T^\nabla|_p = 0$.
Heuristic reasons for this (and a picture!) are given in this excellent answer by Sepideh Bakhoda.
A precise proof of this fact is outlined by Robert Bryant at the end of this MO answer of his.
More Reasons We Like $T^\nabla = 0$
(A) Submanifolds of $\mathbb{R}^N$ come with torsion-free connections
Suppose $(M,g)$ is isometrically immersed into $\mathbb{R}^N$.
As hinted in the comments, the euclidean connection $\overline{\nabla}$ on $\mathbb{R}^N$ is torsion-free. It is a fact that the tangential component of $\overline{\nabla} = \nabla^\top + \nabla^\perp$ defines an induced connection on $M \subset \mathbb{R}^N$. This induced connection on $M$ will then also be torsion-free (and compatible with the induced metric).
Point: If $(M,g) \subset \mathbb{R}^N$ is an isometrically immersed submanifold, then its induced connection is torsion-free.
This example is more general than it seems: by the Nash Embedding Theorem, every Riemannian manifold $(M,g)$ can be isometrically embedded in some $\mathbb{R}^N$.
(B) $T = d^\nabla(\text{Id})$
[I'll add this another time.]
(C) Simplification of identities
Finally, I should mention that $T^\nabla = 0$ greatly simplifies many identities.
First, we have the Ricci Formula
$$\nabla^2_{X,Y}Z - \nabla^2_{Y,X}Z = R(X,Y)Z - \nabla_{T^\nabla(X,Y)}Z.$$
Thus, in the case where $T^\nabla = 0$, we can interpret the curvature $R(X,Y)$ as the obstruction to commuting second covariant derivatives of vector fields.
In the presence of torsion, the First and Second Bianchi Idenities read, respectively,
$$\mathfrak{S}(R(X,Y)Z) = \mathfrak{S}[ T(T(X,Y),Z) + (\nabla_XT)(Y,Z)],$$
$$\mathfrak{S}[(\nabla_XR)(Y,Z) + R(T(X,Y),Z)] = 0,$$
where $\mathfrak{S}$ denotes the cyclic sum over $X,Y,Z$.
References
[Lee] "Riemannian Manifolds: An Introduction to Curvature"
[Spivak] "A Comprehensive Introduction to Differential Geometry: Volume II"
Best Answer
$\newcommand{\R}{\mathbb{R}}$ The directional derivative of a function is well-defined on a manifold without any additional assumptions. In particular, given a smooth function $f: M \rightarrow \R$, $p \in M$, and tangent vector $v \in T_pM$, we can define the directional derivative to be $$ D_vf(p)= \left.\frac{d}{dt}\right|_{t=0}f(c(t)) = \lim_{t\rightarrow 0}\frac{f(c(t))-f(c(0))}{t},$$ where $c$ is a smooth parameterized curve such that $c(0) = p$ and $c'(0) = v$. It is straightforward, using local coordinates and the chain rule, to show that this is well-defined, independent of the curve $c$ and the local coordinates.
If we tryto define the directional derivative of a vector field $X$ the same way, we get something like $$ D_vX(p)= \left.\frac{d}{dt}\right|_{t=0}X(c(t)) = \lim_{t\rightarrow 0}\frac{X(c(t))-X(c(0))}{t}.$$ This, however, doesn't work, because $X(c(t)) \in T_{c(t)}M$ and $X(c(0)) \in T_{c(0)}M$ lie in different vector spaces, so it is not possible to subtract one from the other. You have to "transport" one of them to be in the same tangent space as the other. The process of doing this is a long story. In short, there is not a unique way to do it, and involves setting up a so-called connection on the tangent bundle of the manifold.