I think you are missing something. If you are treating the payload as a point-mass and applying all the cables to the same point, then as @ja72 says, three cables will suffice. Mathematically within your model, you would be able to set any one force arbitrarily and solve for the other three.
I think what you are neglecting is that the payload is a physical object with finite dimension; it can rotate in three-space. You don't have to connect all the cables at the same position as each other. In addition to balancing the linear forces, you have to also balance the torque the cables apply to the payload to prevent it from rotating. From the photos in the Wikipedia article, it looks to me like the real Skycam uses three cables plus gravity to set the position and constrain the rotation to a single axis, and the fourth cable to rotate around that axis.
EDIT: The way you have things set up, you are taking as given the angles and points of connection of the cables, and solving for the tensions. That gives you the freedom to connect the cables anywhere on the payload that you wish. You are using that freedom to connect them all to the center of mass, which makes the calculations easier, but doesn't give you any torque. I'm going to use that freedom to say that the cables are going to pass through the z-axis, which I am taking to be the vertical line running through the center of mass of the payload. This isn't necessary*, but it makes some calculations simpler.
Now let's compute the torque around the center of mass. Gravity runs through the COM, so it provides no torque. You can use plane geometry to compute the torque of the nth cable as $\vec{\tau_n} = \vec{r_n}\times \vec{F_n} = r_nF_n\sin\phi_n (\pm\hat\theta)$. $\phi_n$ is as you defined it. Choose the sign on whether the cable crosses the +z-axis before it connects to the payload, or if it would cross the -z-axis if it were allowed to pass through the payload. $\hat\theta$ is a unit vector pointing in the direction of increasing $\theta$, and will depend on the cable. It will have components in both the x direction and the y direction, so that's two more equations. That over-determines your problem: you then have five equations in four unknowns. I take that to mean that you can't fully arbitrarily pick where you connect the cables to the payload. Either the x or y coordinate of one of them will be determined for you.
*The complication is that the torque then has a component in the z direction. That gives you another equation, which means the connection point of one of your cables will be fully determined by the connection points and angles of the other three.
Here is an idea, borrowed from linear elastic theory. Solve all the forces in terms of an unknown force (I chose $f_{10}$) and construct a long vector $f$
$$ \boldsymbol f = \begin{bmatrix} f_1 & f_2 & \dots & f_9 \end{bmatrix}^\top $$
where each component is a function of $P$, $\beta$ and $f_{10}$.
Now assemble something resembling the total potential energy by doing this
$$ U = \frac{L}{2\,E\,A} \, \boldsymbol f^\top \boldsymbol f $$
Now by minimizing this with
$$ \frac{{\rm d}U}{{\rm d}f_{10}} = 0 $$
will produce a result for $f_{10}$ and hence all of the values in $\boldsymbol{f}$.
This works because for each element $f_i = \frac{E_i,A_i}{L_i} \delta_i$ where $\delta_i$ is the deformation, and the energy is $U_i = \frac{1}{2} \left(\frac{E_i,A_i}{L_i}\right) \delta_i^2 = \frac{L_i}{2 E_i A_i} f_i^2 $.
So $U = \sum U_i = \frac{L}{2\,E\,A} \, \boldsymbol f^\top \boldsymbol f $ except that not all the elements have the same length. So my method will produce an incorrect result. I just realized this.
To correct this you have to construct the total energy as
$$ U = \sum_{i} \frac{L_i}{2 E_i A_i} f_i^2 $$ and then minimize it with the derivative.
Illustrative Example
The force equilibrium on points A, B and C is
$$\begin{array}{cc}
A_{y}-f_{1}=0\\
-P+f_{1}-f_{2}=0\\
C_{y}+f_{2}=0
\end{array}\biggr\}\begin{array}{cc}
A_{y}=P+f_{2}\\
C_{y}=-f_{2}\\
f_{1}=P+f_{2}
\end{array}$$
which is indeterminate. The total strain energy is
$$U=\frac{L_{1}}{2EA}f_{1}^{2}+\frac{L_{2}}{2EA}f_{2}^{2} \\
=\frac{L_{1}}{2EA}\left(P+f_{2}\right)^{2}+\frac{L_{2}}{2EA}f_{2}^{2}$$
which is minimized by
$$\frac{{\rm d}U}{{\rm d}f_{2}}=\frac{1}{2E\, A}\left[\frac{{\rm d}}{{\rm d}f_{2}}\left(L_{1}\left(P+f_{2}\right)^{2}\right)+\frac{{\rm d}}{{\rm d}f_{2}}\left(L_{2}f_{2}^{2}\right)\right]=0 \\
=\frac{1}{E\, A}\left(L_{1}\left(P+f_{2}\right)+L_{2}f_{2}\right)=0 \\
f_{2}=\mbox{-}\frac{L_{1}}{L_{1}+L_{2}}P$$
back substituting into
$$ A_y = \frac{L_2}{L_1+L_2} P \\ C_y = \frac{L_1}{L_1+L_2} P \\ f_1 = \frac{L_2}{L_1+L_2} P $$
Of course in your case, you need to add the member weights into the node equilibrium equations.
Best Answer
As you have noticed yourself, your system is simply underdetermined. In order to find a unique solution you need to add some extra constraints in addition to Newton's equations. Imagine a table with more than four legs: the more legs you add, the more unknown forces you have. But the number of equations does not change. If we instead remove a leg we find a unique static solution.
See also the Wikipedia page about statically indeterminate systems.