Answers to many questions
is a rectangle a plane?
No, it is not. A plane is infinite in two dimensions, a rectangle has finite size.
I thought planes were defined by 3 points
That is correct but not useful for your application.
my rectangle is defined by four.
As you claim that your rectangle is aligned to the horizontal and vertical directions, two points suffice. For a general rectangle, a third point would be required, or in fact half of a third point (5 real degrees of freedom in total). But this, too, is irrelevant to your question.
the equation of a rectangle
There is no equation for a rectangle, there is only a set of inequalities. Which don't integrate well into the whole setup of dealing with multiple equations.
Is either solution #1 or solution #2 valid in solving collision detection?
Each one will work for a particular use case.
Which solution (#1 or #2) is preferable for what reasons?
As your rectangle is formed by four line segments, solution #2 is the one to take, as your edit in response to the comment by jshin47 already indicates.
The core question
How can I calculate the intersection between a 2D rectangle and a 2D circle (without splitting up the rectangle into four separate lines)?
For the general case of an arbitrary circle and rectangle, I'd go for the intersections between circle and line segments. Avoiding that will only make things more complicated.
Solution #2 sounds like a good starting point. It isn't too hard: If the line segment starts at $(x_1,y_1)$ and ends at $(x_2,y_2)$ and your circle is centered at $(x_C, y_C)$ and has radius $r$, then you have the single non-linear equation
$$ ((1-t)\cdot x_1 + t\cdot x_2 - x_C)^2 +
((1-t)\cdot y_1 + t\cdot y_2 - y_C)^2 = r^2 $$
Solving this equation for $t$, you might find up to two solutions. If these satisfy $0\le t\le1$ then there is an intersection along the line segment. Do this for all four line segments, and if any intersects the circle, then the rectangle intersects the circle.
The above equation originated from the equation of the circle, with a generic point on the line (segment) substituted for $x$ and $y$.
\begin{align*}
x &= (1-t)\cdot x_1 + t\cdot x_2 \\
y &= (1-t)\cdot y_1 + t\cdot y_2 \\
r^2 &= (x-x_C)^2 + (y-y_C)^2
\end{align*}
In contrast to this solution, I use $t=0$ to represent the preceived starting point $(x_1,y_1)$ of the line, and $t=1$ for the end point $(x_2,y_2)$. This choice has no impact on your problem, as the direction of your line segments is irrelevant. But it is more useful or common for other applications.
If you have trouble understanding these parametric equations for $x$ and $y$, I suggest you simply choose end points with simple coordinates, and plug in some values of $t$, both from within and outside the specified range. You will see how all of these lie on the connecting line, and the range $0\le t\le 1$ corresponds to the line segment.
Note that all of the above won't deal with the scenario where the circle is fully contained within the rectangle, or vice versa. So you might want to check whether the center of the circle lies within the rectangle, or whether any rectangle corner lies within the circle.
Aligned rectangle
If your rectangle is aligned to the horizontal and vertical axes, you can simplify the intersection checks somewhat. I'd still examine every edge separately, but using a simpler decision making process.
As an example, let's take a horizontal edge, from $(x_1, y)$ to $(x_2, y)$. The circle is still centered at $(x_C, y_C)$ with radius $r$.
- If $x_1 \le x_C \le x_2$, then the circle intersects the line if $\lvert y - y_C\rvert \le r$. This means that the circle center and the line are no more than $r$ apart.
- Otherwise, they intersect if $(x_1-x_C)^2+(y-y_C)^2\le r^2$ or $(x_2-x_C)^2+(y-y_C)^2\le r^2$. This means that either endpoint must lie within the circle.
Best Answer
Line
You already have a solution for that.
Circle
A circle is described by an equation like this:
$$ 0 = (x-c_x)^2 + (y-c_y)^2 - r = x^2 + y^2 + (-2c_x)x + (-2c_y)y + (c_x^2+c_y^2-r) $$
Multiplying that equation by a real factor and renaming the coefficients, you obtain an equation of the following form:
$$\alpha(x^2+y^2) + \beta x + \gamma y + \delta=0$$
Three points in the plane define a circle. They define the parameters $\alpha$ through $\delta$ up to a common scale factor. Adding a fourth point, you have two options: Either that point lies on the circle, then it fits the same 1d space of parameters. Or it does not lie on the same circle, then the only way to satisfy all four equations is by choosing all four parameters equal to zero.
So you can check cocircularity by evaluating the determinant of this system of equations:
$$\begin{vmatrix} x_1^2+y_1^2 & x_1 & y_1 & 1 \\ x_2^2+y_2^2 & x_2 & y_2 & 1 \\ x_3^2+y_3^2 & x_3 & y_3 & 1 \\ x_4^2+y_4^2 & x_4 & y_4 & 1 \end{vmatrix}=0$$
If you write this out, you get something pretty bulky:
\begin{align*} x_{2}^{2} x_{3} y_{1} - x_{2} x_{3}^{2} y_{1} - x_{2}^{2} x_{4} y_{1} + x_{3}^{2} x_{4} y_{1} + x_{2} x_{4}^{2} y_{1} - x_{3} x_{4}^{2} y_{1} - x_{1}^{2} x_{3} y_{2} + x_{1} x_{3}^{2} y_{2} \\{}+ x_{1}^{2} x_{4} y_{2} - x_{3}^{2} x_{4} y_{2} - x_{1} x_{4}^{2} y_{2} + x_{3} x_{4}^{2} y_{2} - x_{3} y_{1}^{2} y_{2} + x_{4} y_{1}^{2} y_{2} + x_{3} y_{1} y_{2}^{2} - x_{4} y_{1} y_{2}^{2} \\{}+ x_{1}^{2} x_{2} y_{3} - x_{1} x_{2}^{2} y_{3} - x_{1}^{2} x_{4} y_{3} + x_{2}^{2} x_{4} y_{3} + x_{1} x_{4}^{2} y_{3} - x_{2} x_{4}^{2} y_{3} + x_{2} y_{1}^{2} y_{3} - x_{4} y_{1}^{2} y_{3} \\{}- x_{1} y_{2}^{2} y_{3} + x_{4} y_{2}^{2} y_{3} - x_{2} y_{1} y_{3}^{2} + x_{4} y_{1} y_{3}^{2} + x_{1} y_{2} y_{3}^{2} - x_{4} y_{2} y_{3}^{2} - x_{1}^{2} x_{2} y_{4} + x_{1} x_{2}^{2} y_{4} \\{}+ x_{1}^{2} x_{3} y_{4} - x_{2}^{2} x_{3} y_{4} - x_{1} x_{3}^{2} y_{4} + x_{2} x_{3}^{2} y_{4} - x_{2} y_{1}^{2} y_{4} + x_{3} y_{1}^{2} y_{4} + x_{1} y_{2}^{2} y_{4} - x_{3} y_{2}^{2} y_{4} \\{}- x_{1} y_{3}^{2} y_{4} + x_{2} y_{3}^{2} y_{4} + x_{2} y_{1} y_{4}^{2} - x_{3} y_{1} y_{4}^{2} - x_{1} y_{2} y_{4}^{2} + x_{3} y_{2} y_{4}^{2} + x_{1} y_{3} y_{4}^{2} - x_{2} y_{3} y_{4}^{2} &=0 \end{align*}
Note that four points on a line will also satisfy this equation, and the solution will have $\alpha=0$. If you don't want to include this case, check for lines first.
Rectangle
Two vectors are orthogonal if their dot product is zero. Do this for three pairs of edge vectors and you have checked three corner angles. If they are all right angles, then so is the fourth and you have the corners of a rectangle.
For points on the edge of a rectangle: you will always find a rectangle which has the four points as elements of its edges as long as the points are in convex position. Start by connecting two points with a line, then extend that to the smallest bounding rectangle of the orientation indicated by that line. There are examples where the first choice of line would lead to one of the other points inside the resulting rectangle, but as far as my mental experiments go (although this is no proof), you can always find a different starting edge such that things work out.
Triangle
For four points to be the corners of a triangle, two of them have to be the same.
You will always find a triangle which has the four points on its edges as long as the points are in convex position. Simply take the convex hull, which is a quadrilateral in this case, and extend two of its edges to form a triangle.