There is a condition which classifies when two permutations commute. However, it is easier to state if we write the permutations using disjoint cyclic notation (see wikipedia).
So, write $\sigma=(a_{1, 1} \: a_{2, 1}\: \ldots \: a_{n_1, 1})\cdots (a_{1, i} \: a_{2, i}\: \ldots \: a_{n_i, i})$, then $\sigma$ and $\tau$ commute if and only if $(\tau(a_{1, 1}) \: \tau(a_{2, 1})\: \ldots \: \tau(a_{n_1, 1}))\cdots (\tau(a_{1, i}) \: \tau(a_{2, i})\: \ldots \: \tau(a_{n_i, i}))=\sigma$ (so apply $\tau$ to each of the "numbers" in $\sigma$).
For example, if $\sigma$ is a single cycle $(a_{1} \: a_{2}\: \ldots \: a_{n})$ then $\sigma$ and $\tau$ commute if and only if $(\tau(a_{1}) \: \tau(a_{2})\: \ldots \: \tau(a_{n}))$ is a cyclic shift of $(a_{1} \: a_{2}\: \ldots \: a_{n})$.
This follows from the fact that, in general, $$\tau^{-1}(a_{1, 1} \: a_{2, 1}\: \ldots \: a_{n_1, 1})\cdots (a_{1, i} \: a_{2, i}\: \ldots \: a_{n_i, i})\tau$$$$=(\tau(a_{1, 1}) \: \tau(a_{2, 1})\: \ldots \: \tau(a_{n_1, 1}))\cdots (\tau(a_{1, i}) \: \tau(a_{2, i})\: \ldots \: \tau(a_{n_i, i}))$$ which is quite well-known but I will leave proving it to the reader (it is in most advanced undergraduate texts). I will also leave interpreting this criterion to non-disjoint-cycle notation to the OP or any other interester person. I am hungry and have to run home for my dinner, so I have no time to do this myself!
As stated in the comments, once you show that your map is a homomorphism $D_{2n}\to \operatorname{GL}_2(\mathbf{R})$, then you have a 2-dimensional representation (since the matrices are 2x2). Thus all you need to show is that the matrices for $\sigma$ and $\tau$ satisfy the relations in the group presentation.
Here is another way to get the 2-dimensional reps using induced representations, just for fun.
There is an obvious subgroup $\{1, \sigma\, \ldots, \sigma^{n-1}\}$ which is a cyclic group of order $n$, let's call it $C_n < D_{2n}$. Since $C_n$ is abelian, it has $n$ irreducible 1-dimensional representations over $\mathbf{C}$, namely
$$\sigma\mapsto e^{2\pi ki/n}\qquad 0\leq k < n$$ which captures the idea of rotating by an angle of $2\pi k/n$. The trick is to induce these easily-described representations to $D_{2n}$ in order to find some possibly new representations.
Recall that if we have a representation $W$ of a subgroup $H\leq G$ (i.e. an H-linear action on $W$), the induced representation of $W$ is $$\bigoplus_{g\in G/H}g\cdot W$$ where $g$ ranges over a set of representatives of $G/H$.
The induced representation of $C_n$ to $D_{2n}$ for fixed $k$ is not hard to describe then since $D_{2n}/C_n$ has representatives $\{1, \tau\}$. So we just need to describe the $D_{2n}$-vector space $\mathbf{C}\oplus\tau\cdot\mathbf{C}$ where $\mathbf{C}$ has basis consisting only of $w_1$. The neat part is seeing how the $D_{2n}$ action turns into an actual matrix representation.
Specifically, we can find out how $\sigma$ acts on each summand using our representation of $C_n$: $$\sigma\cdot w_1 = e^{2\pi ki/n}w_1, \text{ and}$$
$$\sigma\cdot(\tau\cdot w_1) = \sigma\tau\cdot w_1 = \tau\sigma^{-1}\cdot w_1 = e^{-2\pi ki/n}\tau\cdot w_1$$ which tells us that $\sigma$ acts by the matrix $\pmatrix{e^{2\pi ki/n}&0\\0&e^{-2\pi ki/n}}$!
We can also figure out how $\tau$ acts. $\tau$ obviously takes $w_1$ to $\tau\cdot w_1$, and $\tau$ takes $\tau\cdot w_1$ to $\tau^2 w_1=w_1$, so $\tau$ simply interchanges the two summands. This tells us that $\tau$ acts by the matrix $\pmatrix{0&1\\1&0}$.
So that's another way to get 2-dimensional representations of $D_{2n}$. Note that the $\sigma$ matrix I described with $k=1$ is the diagonalized form of the rotation matrix you mention.
Best Answer
Although he denoted it by $D_{4n},$ Lang never claimed this group was the dihedral group of order $4n.$ Actually, it cannot, since $\tau$ is of order $4,$ whereas the exponent of the dihedral group of order $4n$ is $2n,$ and $n$ is assumed to be odd.
Denoting by $\sigma,\tau$ (like you did) Lang's two matrices and letting $$a=\sigma\tau^2\quad\text{and}\quad x=\tau,$$ we recognize his "$D_{4n}$" as the dicyclic group $$Q_{4n}=\operatorname{Dic}_n = \left\langle a, x \mid a^{2n} = 1,\ x^2 = a^n,\ x^{-1}ax = a^{-1}\right\rangle.$$