What you are talking about are basic inteaction effets. You could think forexample of simple model:
y = a + b1*x1 + b2*x2 + e
If we add and interaction-term to the model depicted, the following term consequences.
y = a + b1*x1 + b2*x2 + b3*x1*x2 + e
By simply reordering, we find
y = (a + b1*x1) + x2*(b2+b3*x1) + e
The first parenthesis depicts the intercept and the second the regression’s slope. However, both elements are subject to the level of x1. Despite term appearing rather econometrically tame, Braumöller (2004) shows that interpreting models with multiplicative terms often goes askew especially when turning to the lower-order coefficients b1 and b2.
The regression’s slope will equal b1 only if x2 equals zero. Thus b1
denotes the simple effect of x1 on y. Similar, x2 denotes the simple effect on y since y = a + b2*x2 + e ; for x1=0. Similarly, a is the intercept when both x1 anx2 are zero.
b3 finally depicts the interactions between x1and x2. And here lies the crux:
First, you can only interpret x3if you incorporate both x1and x2in your regression. For a detailed analysis of this matter see Whisman, Mark A., and Gary H. McClelland. "Designing, testing, and interpreting interactions and moderator effects in family research." Journal of Family Psychology 19.1 (2005): 111. Second, you need "meaniful" values for x1=0and x2=0. One idea might be a centering at meaningful values or at means. For a more detailed discussion of this matter check here: When conducting multiple regression, when should you center your predictor variables & when should you standardize them?
Statistic packages like Stata and R usually do everything for you. Practically, you can incorporate x1, x2 and a third variable x1*x2. But usually it is better to merely incorporate x1 and x2 and "tell" R/Stata to gauge an interactio effect too ( x1##x2 or x1#x2 in Stata). R is a little different. Check here: Different ways to write interaction terms in lm?
This is an instance of (multi)collinearity.
As you've noted in "my thoughts so far," ordinary linear regression can "distribute" its weight between the two columns as it pleases without changing the objective. Then fits by actually inverting a matrix may fail; if you fit by some general-purpose optimization algorithm, you'll get some arbitrary breakdown between them. This can cause serious problems if you're trying to infer regression weights, but if you just want to learn a predictive model it doesn't really matter (unless your matrix inverse fails).
If you use a regularized linear model, say ridge regression or LASSO, then the weight will be distributed between the two components in a way that corresponds to the choice of regularization.
If you're fitting a decision tree, then it'll split based on one or the other arbitrarily, and it won't really matter unless you're looking at feature importance, where it'll mess you up in the same way as weight inference in linear models.
If you're doing a method based on distances between points, e.g. nearest neighbors, kernel smoothing, or many types of Gaussian processes, then it'll effectively make that component "count more" in the distance.
Best Answer
Your data is not linearly separable in the original space.
But it seems like it actually is separable with a circle/ellipse (let's say it's inside a circle to simplify the problem): it seems reasonable to have hypothesis that, for some $c$ if $x^2 + y^2< c$ then a point is blue.
That means that if you use $x^2, y^2$ as features, you can fit a linear classifier to these data points and actually separate the classes linearly.