In the 1957 paper, On the differentiability of isometries, Richard S. Palais gives a way to construct the tangent spaces of a Riemannian manifold using only its metric space structure (Theorem, p.1).
Specifically, given a Riemannian manifold $M$ and a point $p \in M$, the tangent space $T_p M$ equals the group of germs of pointed local similitudes $\mathbb{R} \to M$, $0 \mapsto p$. (Definitions below.)
Question: What are the metric space properties of Riemannian manifolds which allow this construction to work for Riemannian manifolds but not for arbitrary metric manifolds?
This construction can't work for arbitrary metric manifolds because any second-countable, Hausdorff, locally Euclidean space is metrizable, and yet there exist such spaces which are not homeomorphic to any smooth manifold, and in particular have no good definition of tangent spaces. If this construction worked generally, these manifolds would have tangent spaces; contradiction.
Definitions: Given metric spaces $(M_1,d_1)$ and $(M_2, d_2)$, a similitude is a bijective map $f:M_1 \to M_2$ such that, for a particular $r > 0$, for all $x,y \in M_1$, $d_2(f(x),f(y))= r \cdot d_1(x,y)$. An isometry is just the special case when $r = 1$.
A local similitude is the obvious analog of a local isometry (see Burago, Burago, Ivanov, Metric Geometry, Definition 3.4.1., p.78). Namely, a map $f: M_1 \to M_2$ is called a (pointed) local similitude at $x \in M_1$ (to $p \in M_2$) if $x \in M_1$ has an open neighborhood $U_x \subseteq M_1$ such that the (restriction of) $f$ is a similitude $U_x \to V_p$, where $V_p$ is some open neighborhood of $p$.
(Pointed here is solely meant to denote here that the point $p \in M_2$ under consideration is fixed in advance, as opposed to a "non-pointed" local similitude at $x \in M_1$, which could map onto any open set in $M_2$, not necessarily a neighborhood of the specified point $p \in M_2$. The terminology admittedly probably makes more sense when $f: M_1 \to M_1$ is a self-map, and $p = x$.)
Germ here is meant to denote the usual equivalence relation, where two functions belong to the same germ if and only if they agree on some open neighborhood of the point in question.
Note: Needless to say, Richard S. Palais uses different terminology. I am fairly confident, but not 100% confident, that my characterization is accurate/equivalent. (See here.) It is at the very least superficially similar to the definition of tangent spaces for smooth manifolds (germs of smooth maps $\mathbb{R} \to M$, $0 \mapsto p$ under an equivalence relation of smooth jets, see here). Tangent vectors are used to formalize notions of direction (see), and I had already been lead to germs of pointed local similitudes when trying to formalize the intuition of "direction" from Euclidean space (albeit ones $M \to M, p \mapsto p$ rather than $\mathbb{R} \to M, 0 \mapsto p$), see here (although it needs to be revised further).
Even if my attempted characterization of his result is not equivalent, I am still interested in the construction Richard S. Palais mentions for the tangent space of a Riemannian manifold, and would still like to know which metric space properties (e.g. strictly intrinsic metric? length space? existence of geodesics? etc.) allow the construction to work for Riemannian manifolds but not for arbitrary metric space manifolds. It would be interesting in particular to observe whether those metric space properties also hold for, e.g., Finsler manifolds.
Note: This website is for "research-level" mathematics. However, this question is with reference to a fairly old paper (1957), so I am not sure if it is on-topic. If not, please tell me. Note that there are at least two other questions (here and here) on MathOverflow which mention/discuss the paper.
This question seems at least tangentially related.
Best Answer
Pretty much the papers to read are
Nikolaev, I.G., Smoothness of the metric of spaces with two-sided bounded Aleksandrov curvature, Sib. Math. J. 24, 247-263 (1983); translation from Sib. Mat. Zh. 24, No.2(138), 114-132 (1983). ZBL0547.53011.
Nikolaev, I.G., A metric characterization of Riemannian spaces, Sib. Adv. Math. 9, No.4, 1-58 (1999). ZBL0956.53027. See also MR2142160
The point is that the Riemannian distance function $d$ (when we do not even know in advance that the underlying space topological space $X$ admits structure of a smooth manifold) satisfies a number of properties (which are discussed in the "Metric Geometry" book that you are reading), such as:
$(X,d)$ is a path-metric and (locally) a geodesic metric space.
$(X,d)$ has (locally) curvature bounded above and below (where the upper and lower bounds are in the sense of the comparison geometry).
$(X,d)$ has (locally) extendible geodesics.
... (local compactness, etc.)
Among other things, the (local) curvature bounds guarantee that geodesics between two points are (locally) unique. The point of the two examples I suggested (in my comment) is to see what goes wrong if one drops these assumptions, in particular, what happens to the "tangent space" that you defined (e.g. it may fail to have structure of a vector space; its dimension might become larger than the topological dimension of $X$, e.g. infinite). A cone is just a regular cone of revolution in $E^3$, obtained by rotating a ray $R$ along another ray $A$ (the axis of rotation), such that $R$ and $A$ share the initial point (the apex of the cone); you equip this cone with the path-metric induced from $E^3$. In both examples, the underlying space is a topological manifold, but something goes wrong with its "tangent space" that you defined: In the first example it is infinite-dimensional, in the second example it is not a vector space.
What Nikolaev proved is quite remarkable: The above properties imply that $(X,d)$ is isometric to a manifold equipped with a Riemannian distance function, in particular, the tangent space (at every point) indeed has natural structure of a vector space of the expected dimension. (The difference between the two papers is in the degree of smoothness of the Riemannian metric: The first paper gives a $C^{1,\alpha}$-smooth metric, while the second paper gives a $C^{\infty}$ metric, under a stronger hypothesis.) This result was conjectured by A.D.Alexandrov in the 1940s. Nikolaev's papers are a culmination of a long chain of results, including, for instance, the one by Berestovsky who proved the existence of a continuous metric, etc.