$$\left | \begin{matrix} x^2 + y^2 + z^2 & x & y & z & 1 \\
x_1^2 + y_1^2 + z_1^2 & x_1 & y_1 & z_1 & 1 \\
x^2_2 + y_2^2 + z_2^2 & x_2 & y_2 & z_2 & 1 \\
x_3^2 + y_3^2 + z_3^2 & x_3 & y_3 & z_3 & 1 \\
x_4^2 + y_4^2 + z_4^2 & x_4 & y_4 & z_4 & 1 \end{matrix} \right | = 0 $$
i.e. $-128(x^2+y^2+z^2-4x-\frac{29}{2}y-4z)=0$
Actually, the method that you’re trying to apply would not produce the correct answers for single point. The method finds the tangent plane to a surface at a point on the surface, but neither of the given points lies on the surface (well, $A$ would be on the hyperboloid if there weren’t a slice missing along the plane $x=0$). It makes no sense to compute $\vec n$ at these points. Instead, what you need to do is construct the equation of the tangent plane at the generic point $P_0=(x_0,y_0,z_0)$ on the surface and then plug in the coordinates of the two known points to solve for $P_0$. You’ve already worked out that $$\vec n_{P_0} = \left[{1-y_0^2\over x_0^2},2{y_0\over x_0},-1\right]$$ so the equation of the unknown plane is $\vec n_{P_0}\cdot(P-P_0)=0$. Substitute the coordinates of $A$ and $B$ for $P$ and solve the resulting system of equations for $P_0$.
Since you’ve tagged this question with “linear algebra,” here’s a linear-algebraic way to solve this. Rewrite the equation of the surface as $xz=y^2-1$ and keep in mind that the domain excludes $x=0$. This is the equation of a quadric surface which can be written in matrix form as $$\mathbf x^TQ\mathbf x = \begin{bmatrix}x&y&z&1\end{bmatrix} \begin{bmatrix} 0&0&-\frac12&0 \\ 0&1&0&0 \\ -\frac12&0&0&0 \\ 0&0&0&-1 \end{bmatrix} \begin{bmatrix}x\\y\\z\\1\end{bmatrix} = 0.$$ The equation $ax+by+cz+d=0$ of a plane can also be written as $(a,b,c,d)^T(x,y,z,1)=0$, so this plane can be represented by the vector $\mathbf\pi = (a,b,c,d)^T$. It turns out that all of the planes tangent to a nondegenerate quadric $Q$ satisfy the dual equation $\mathbf\pi^TQ^{-1}\mathbf\pi=0$ and that if $\mathbf\pi$ is tangent to $Q$, then its point of tangency is $\mathbf p = Q^{-1}\mathbf\pi$. (These equations can be derived from pole-polar relationships.)
Now, by substituting the coordinates of $A$ and $B$ into the generic plane equation, we see that the coefficients in the equations of any plane that passes through these points satisfies the system $$b+d=0 \\ a+3b+4c+d=0,$$ i.e., the representative vectors of this family of planes are the null space of $$\begin{bmatrix}0&1&0&1\\1&3&4&1\end{bmatrix}.$$ Since multiplying both sides of an equation by a nonzero constant doesn’t change the solution set, vectors that are multiples of each other represent the same plane, so to describe the family of planes through $A$ and $B$ it suffices to consider only convex combinations of basis vectors for this null space. Computing a basis for the null space via row-reduction produces $\mathbf\pi(\lambda) = (2-6\lambda,\lambda-1,\lambda,1-\lambda)^T$. So, finding the tangent planes through $A$ and $B$ becomes a matter of solving $\mathbf\pi(\lambda)^TQ^{-1}\mathbf\pi(\lambda)=0$ for $\lambda$. For this particular problem, inverting $Q$ is quite simple. This equation is quadratic in $\lambda$, so there might be two solutions. If you check where they are tangent to the surface, however, you’ll find that one of the points of tangency has the forbidden $x=0$, so you must reject that potential solution.
Best Answer
$C_1(6,1,-1);\;r_1=1$
$C_2(0,5,4);\;r_2=3$
A generic plane $\pi$ has equation $$\pi:ax+by+cz+d=0$$ The distance from the plane to the center of the spheres must be equal to the respective radius. Furthermore the plane must pass through $P(5,2,0)$
$$\begin{cases} \frac{|6a+b-c+d|}{\sqrt{a^2+b^2+c^2}}=1\\ \frac{|5b+4c+d|}{\sqrt{a^2+b^2+c^2}}=3\\ 5a+2b+d=0&\\ \end{cases}$$
$$\begin{cases} (6 a+b-c+d)^2=\frac{1}{9} (5 b+4 c+d)^2\\ (6 a+b-c+d)^2=a^2+b^2+c^2\\ d=-5a-2b\\ \end{cases}$$
Two solutions $a=0,b=-d/2,c=0;\;a=-d/9,b=-2 d/9,c=-2 d/9$
Two planes
$-d/2 y + d = 0\to y-2=0$
$-d/9 x -2d/9 y -2 d/9 z + d=0 \to x+2 y+2 z-9=0$