I think it's fairly straight-forward actually, naturally depending on how much you want to show. Do you take the following as given?
If $\mathbf{A}$ is non-invertible, then there exists a non-zero vector
$\mathbf{x}$ such that $\mathbf{Ax}=\boldsymbol{0}$.
If you do, then it shouldn't be too problematic.
If $\mathbf{A}$ is not invertible, then that means there is a vector such that $\mathbf{Ax}=\boldsymbol{0}$. Thus, $\mathbf{x}^\prime \mathbf{Ax}=\mathbf{x}^\prime \boldsymbol{0}=0$. But, if $\mathbf{A}$ is positive definite, then $\mathbf{x}^\prime \mathbf{Ax}>0$. So $\mathbf{A}$ cannot be both non-invertible and positive definite. Hence, it must be invertible if it is positive definite.
Edit: This only addresses the second part.
Your source talks about matrices over the complex numbers and $M$ is supposed to be Hermitian, that is, equal to its “conjugate transpose”.
I'll denote by $M^*$ the conjugate transpose of the (generic) matrix $M$. A couple of definitions; let $M$ be a square matrix:
The source also talks about the spectral theorem, which says that every normal matrix $M$ can be written as
$$
M=\sum_{k=1}^r \lambda_k P_k\tag{*}
$$
where $\lambda_1,\dots,\lambda_r$ are the distinct eigenvalues of $M$ and $P_1,\dots,P_r$ are the projections matrices to the eigenspaces; in particular they satisfy $P_k^2=P_k$ and $P_k^*=P_k$. Moreover $P_kP_l$ is the null matrix when $k\ne l$.
Clearly any Hermitian matrix is normal and it's easy to deduce that if $M$ is Hermitian then its eigenvalues are real: indeed, using its spectral decomposition (*), we have, from $M=M^*$,
$$
\sum_{k=1}^r \lambda_kP_k=\sum_{k=1}^r \lambda_k^*P_k
$$
and upon multiplying this by $P_l$ we get $\lambda_l=\lambda_l^*$.
The statement that any Hermitian matrix has real eigenvalues can be proved also without the spectral theorem, by directly using the definition: let $v$ be an eigenvector relative to the eigenvalue $\lambda$; then
$$
\lambda(v^*v)=v^*(\lambda v)=v^*Av=v^*A^*v=(Av)^*v=(\lambda v)^*v=\lambda^*v^*v
$$
Since $v^*v\ne0$, we get $\lambda=\lambda^*$.
Another important fact about normal matrices is that they can be diagonalized with a unitary matrix; this is actually an equivalent condition.
A square matrix $M$ is normal if and only if there exists a unitary matrix $U$ (that is, $U^{-1}=U^*$) such that $D=U^{-1}MU$ is diagonal.
Now, suppose $M$ is (Hermitian) and positive definite, that is, for $v\ne0$, $v^*Mv>0$. Consider $e_k$, the $k$-th vector of the canonical basis and consider $v_k=Ue_k$; then
$$
0<v_k^*Mv_k=e_k^*U^*UDU^*Ue_k=e_k^*De_k
$$
and, clearly, $e_k^*De_k$ is the entry at place $(k,k)$ in $D$. Since $D$ has on its diagonal the eigenvalues of $M$, we are done.
Conversely, if $D$ has all its diagonal entries positive, then, for every $v\ne0$ we have
$$
v^*Mv=(Uv)^*D(Uv)>0
$$
because clearly $D$ is positive definite.
Best Answer
I think this is false. Let $A = \begin{pmatrix} 1 & -3 \\ 0 & 1 \end{pmatrix}$ be a 2x2 matrix, in the canonical basis of $\mathbb R^2$. Then A has a double eigenvalue b=1. If $v=\begin{pmatrix}1\\1\end{pmatrix}$, then $\langle v, Av \rangle < 0$.
The point is that the matrix can have all its eigenvalues strictly positive, but it does not follow that it is positive definite.