I am reading Ian Goodfellow's Deep Learning. I am in chapter 1 about Linear Algebra. There is the section called Linear Dependence and Span, they say that for a linear equation $\bf Ax = b$ there can be only one solution, no solution or infinitely many solutions. But there can't be more than one solution and less than infinitely many solutions. How so?
Here, $\bf A$ is a matrix, whereas $\bf b$ and $\bf x$ are vectors.
Best Answer
It is easy to find some examples for the case with $0$, $1$, or $\infty$ many solutions. We will rule out all other cases by using proof via contradiction.
Now, let us assume that we have at least two distinct but not infinitely many solutions for the given equation. Then let us pick $\boldsymbol{x}_1$ and $\boldsymbol{x}_2$ as solutions. Then we construct $\boldsymbol{z} = \boldsymbol{x}_1 +\mu \left[ \boldsymbol{x}_1-\boldsymbol{x}_2\right]$ and show that $\boldsymbol{z}$ allows us to construct infinitely many solutions by only using the two solutions $\boldsymbol{x}_1$ and $\boldsymbol{x}_2$. We can check this by plugging in $\boldsymbol{z}$ into the equation
$$\boldsymbol{A}\boldsymbol{z}=\boldsymbol{A}\boldsymbol{x}_1+\mu\boldsymbol{A}\left[ \boldsymbol{x}_1-\boldsymbol{x}_2\right]=\boldsymbol{b}+\mu \left[\boldsymbol{b}-\boldsymbol{b} \right]=\boldsymbol{b}.$$
We have used $\boldsymbol{Ax}_1=\boldsymbol{Ax}_2=\boldsymbol{b}$ in this derivation. But that is a contradiction that we only have finitely many solutions. Hence, we can conclude that we can only have $0$, $1$ or $\infty$ many solutions.