I get the answer finally.
I see the question about how to determine the equation of a circle defined by three points. Zaz's answer mentioned this page, which can solve my problem indirectly.
Notice that:
$D = -p_x M_{41} + p_y M_{42} - (p_x^2 +p_y^2) M_{43} + M_{44}$
and if we regard $p$ as a moving point, then $D=0$ is actually an equation of a circle:
$(p_x+\frac{1}{2}\frac{M_{41}}{M_{43}})^2+(p_y-\frac{1}{2}\frac{M_{42}}{M_{43}})^2=(\frac{1}{2}\frac{M_{41}}{M_{43}})^2+(\frac{1}{2}\frac{M_{42}}{M_{43}})^2+\frac{M_{44}}{M_{43}}$
i.e. the circle locate at $(x_0,y_0)=(\frac{1}{2}\frac{M_{41}}{M_{43}},-\frac{1}{2}\frac{M_{42}}{M_{43}})$
with radius $r_0^2=x_0^2+y_0^2+\frac{M_{44}}{M_{43}}$.
When $D>0$ the circle equation become $(x-x_0)^2+(y-y_0)^2<r_0^2$
So it's clear that if $(p_x,p_y)$ fall inside that circle, then $D>0$ and vice versa.
Note: According to Cramer's rule, $(-\frac{M_{41}}{M_{43}},\frac{M_{42}}{M_{43}},\frac{M_{44}}{M_{43}})$ is exactly the solution of the equation
$
\begin{pmatrix}
a_x & a_y & 1 \\
b_x & b_y & 1 \\
c_x & c_y & 1
\end{pmatrix}
\boldsymbol{x}=\begin{pmatrix}a_x^2+a_y^2 \\ b_x^2+b_y^2 \\c_x^2+c_y^2\end{pmatrix}
$
And this 3 parameters perfectly define a circle pass through $a,b,c$.One can easily verify when $p$ is $a,b$ or $c$, $D=0$ by the equation above.
Best Answer
Following the hints from J.M., I was able to get the answer.
$$\det \begin{bmatrix} p_x & p_y & p_x^2+p_y^2 & 1 \\ q_x & q_y & q_x^2+q_y^2 & 1 \\ r_x & r_y & r_x^2+r_y^2 & 1 \\ s_x & s_y & s_x^2+s_y^2 & 1 \\ \end{bmatrix} $$
$$ =-a(s_x^2+s_y^2)-bs_x+cs_y+d\\ =-a(s_x^2+s_y^2+\frac{b}{a}s_x-\frac{c}{a}s_y+\frac{d}{a})\\ =-a((s_x+\frac{b}{2a})^2+(s_y-\frac{c}{2a})^2-\frac{b^2+a^2}{4a^2}+\frac{d}{a} )\\ =-a((s_x+\frac{b}{2a})^2+(s_y-\frac{c}{2a})^2-r^2 )\\ $$
where $a=\det \begin{bmatrix} p_x & p_y & 1 \\ q_x & q_y & 1 \\ r_x & r_y & 1 \\ \end{bmatrix}$ and $r=\frac{\sqrt{b^2+c^2-4ad}}{2a}$. Since $p,q,r$ are in clockwise order, $a>0$ and $d<0$. Therefore, the determinant in question is positiver if and only if $s$ lies inside the circumcircle.