I think that you need to generate all pairs of points that correspond to a crossing of each quadrant boundary. Then include a linear constraint that there is only one active pair of each type in the solution.
As documented, when using a custom population type, "ga ignores all constraints, including bounds, linear constraints, and nonlinear constraints." So your custom creation, crossover, and mutation functions must enforce your constraints.
Best Answer