In plain English: If your classifier misclassifies some data, train another copy of it mainly on this misclassified part with hope that it will discover something subtle. And then, as usual, iterate. On the way there are some voting schemes that allow to combine all those classifiers' predictions in sensible way.
Because sometimes it is impossible (the noise is just hiding some of the information, or it is not even present in the data); on the other hand, boosting too much may lead to overfitting.
Welcome Chukwudi. The definitions can be a bit confusing, but it looks like you are thinking of bias and variance as properties of the numerical estimates themselves instead of as properties of the estimator (i.e. of the function or algorithm that produces the numerical value of the estimates). I suggest you look at the mathematical definitions of these terms again before continuing on, paying attention to the notation. It's super important in stats to understand the notation and what is random and what is not. Notation will usually indicate what the "truth" (not random... for now, at least) is, what is observed data (random and not random), and what is an estimator (most always random, because they are functions of random variables).
- It might be more intuitive to think of bias as "on average, how wrong will my numerical estimate be, compared to the true value if I use this method/estimator" and variance as "on average, how much will my numerical estimate change if I use this method/estimator on different datasets."
- Again, remember that both bias and variance are expected values (i.e. averages over all datasets), and that bias is the average of the true value minus the estimate. In your examples above, you have not assumed any true values that I can see.
- Even if you had assumed true values, your examples are not a fair comparison. It's not a fair comparison to look at the bias of one estimator (your 1-NN example) versus the average bias of three estimators (your 3-NN example). To do this kind of numerical comparison fairly, you need to 1) apply both algorithms to the same dataset (y and x values), compare each estimate to the estimate from the other algorithm, then generate new data with the same true values and variance, and do it again, and again...
A few quick examples to clarify and develop intuition:
For the following examples, assume that the true data-generating mechanism is $y_i = f(x_i) + \epsilon_i$, where $\epsilon$ is just noise (i.e. mean zero, but nonzero variance $\sigma^2$). Pretend that I am trying to estimate/predict a new $y_{new}$ associated with a new $x_{new}$. Bias then is defined as $E[\hat{y}_{new} - f(x_{new})]$.
Consider a (dumb) estimating function/estimator, $\hat{y_{new}}=6$. That is, I don't care what the surrounding y values are, I don't care what the x value is, I just think it's 6. This will always be biased except when the underlying truth is coincidentally that $f(x_{new})=6$. But this estimator has 0 variance, because no matter what my data is, I still just estimate $\hat{y_{new}}=6$.
On the other hand, consider the estimator $\hat{y_{new}}=\bar{y}$ (n-NN regression). Now at least I'm learning a tiny bit from my data, so I'm probably less biased than having just chosen a random number such as 6.... but my bias is still large: for any prediction I try to make, I'm probably very biased unless somehow the true mean value of what I am trying to predict ($f(x_{new})$ is close to the overall true mean value of the observations in my data (average of all $f(x_i)$ for all $x_i$ in my data). And the variance of my estimator is now non-zero ($\sigma^2/n$).
Notice in both of the above cases, I only get small bias in particular locations or by dumb luck. I'm very likely to be doing a bad job of fitting most of the data. (i.e. underfitting / oversmoothing from oversimplified models).
Now consider 1-NN regression. Now I probably have pretty small prediction bias in most places - if I have any datapoint close to what I am predicting, I should do fairly well in terms of bias (assuming the function is reasonably smooth). However, the variance of my prediction is $\sigma^2$, since my prediction is just the value of the closest observation (which has variance $\sigma^2$, i.e. the variance is $n$ times larger than the n-NN case, and 2 times larger than the 2-NN case, 3 times larger than 3-NN, and so on).
Hope this helps!
Best Answer
1) and 2) use different models as reference.
1) Compared to the simple base learner (e.g. a shallow tree), boosting increases variance and reduces bias.
2) If you boost a simple base learner, the resulting model will have lower variance compared to some high variance reference like a too deep decision tree.