When $p = 2$, Lucas' Theorem states that ${n \choose m} \equiv 0 \pmod{2}$ if and only if in the binary expansion of $n = (\overline{ \cdots n_1n_0})_2$ and $m = (\overline{ \cdots m_1m_0})_2$, some binary digit (say the $d$th binary digit) satisfies $n_d = 0, m_d = 1$. In particular, this shows that ${2^k + n \choose m} \equiv {2^k + n \choose 2^k + m} \equiv {n \choose m} \pmod{2}$ when $2^k > n > m$, which describes the recursive nature of the Sierpinski triangle that can be found in Pascal's triangle by highlighting the odd elements (which it seems is the geometric interpretation you referred to). However, this is a fairly restricted application of Lucas' Theorem and can be seen by easier means.
For prime $m=p$ you have Lucas' theorem. It might not be obvious, but this gives a fractal structure to the binomial coefficients, modulo $p$.
(From here out, $m$ is an arbitrary integer, not $p$, because this is the statement from Wikipedia.)
It says that if $m=\sum_{i=0}^k m_i p^i$ and $n=\sum_{i=0}^k n_ip_i$ are the base $p$ representations (so $0\leq m_i,n_i<p$) then:
$$\binom{m}{n}\equiv \prod_{i=0}^k \binom{m_i}{n_i} \pmod p$$
Specifically, then, this means that if $0\leq a< b<p^k$ and and $M,N$ are any non-negative integers, then:
$$\binom{Mp^k + a}{Np^k+b} \equiv 0\pmod p$$
This gives big blocks where Pascal's triangle is zero, as in Sierpinski.
More generally, we might want to ask about modulo prime powers.
This answer gives some details on Andrew Granville's result for modulo prime powers. Don't know how "fractal" that result is. It seems to be not as "fractal" is the result
for primes, but it is hard to tell.
If it is, then modulo any number $M$, Chinese Remainder Theorem says that Pascal's triangle looks like the overlay of the fractals for each of the prime powers in the unique factorization.
Without Granville, if $M$ is square-free, there will be a fractal structure, modulo $M$.
Best Answer
Yes, it's true. The identity ${m \choose n} = \frac{m}{n} {m-1 \choose n-1}$ can be rearranged as $n {m \choose n} = m {m-1 \choose n-1}$. If ${m \choose n}$ is prime it follows that it must divide either $m$ or ${m-1 \choose n-1}$. In the first case we can only have $n = 1, n = m-1$, as you have already observed, and in the second case, the quotient $\frac{n}{m}$ cannot be an integer unless $n = m$ or $n = 0$, and neither of these cases gives a prime.