I know what Variance is. But what is Bias? I just have problems to understand this what is written!
Solved – what is bias and variance of an estimator
biasestimationvariance
Related Solutions
If I've understood the question correctly, the goal is to evaluate an estimator for $y$. If so, the problem is that you're counting the irreducible error twice. In mathematical notation, letting the estimate be denoted by $Z$, we have $$\begin{split}\textrm{MSE}(Z) &= \mathbb{E} ((Z-y)^2 ) \\ &=\mathbb{E} ((Z-\mathbb{E} (Z) )^2 ) +\mathbb{E} ((\mathbb{E}(Z) -y)^2 ) \\ &=\mathbb{E} ((Z-\mathbb{E} (Z) )^2 ) +\mathbb{E}((\mathbb{E} (Z) -\mathbb{E} (y) )^2 ) +\mathbb{E} ((\mathbb{E}(y) -y)^2 ) \\ &=\underbrace{\mathrm{Var} (Z) }_{\textrm{variance} } +\underbrace{(\mathbb{E} (Z) -\mathbb{E} (y) )^2 }_{\textrm{bias} ^2} +\underbrace{\mathrm{Var} (y) }_{\textrm{irreducible error} } . \end{split}$$ The important thing to notice here is the definition of the bias: it's the expectation of the estimate, minus the true posterior expectation of $y$. However, in the code, you're calculating the bias as $\mathbb{E} (Z) -y$. That means you've calculated the MSE as an empirical estimate of $$\textrm{MSE} (Z) =\textrm{Var} (Z) +\mathbb{E} ((\mathbb{E} (Z) -y)^2 ) ,$$ so the second term is the square bias plus the irreducible error, as in the second line of the equation above. This means the pink line in your plot is superfluous, because the irreducible error is already reflected in the orange square-bias line.
Therefore, you should be able to add the blue and orange lines, and have it roughly match the red MSE line. It won't be exact, because of the bootstrap issues mentioned by hitchpy, but it'll be pretty close. An example with major overlap is below, where the green line is the sum.
All are metrics to find the best model: You would like to have an unbiased minimum variance estimator related to the validation interval - and thus neither over nor underfitted. But how to balance these metrics is up to your application / context.
Bias: The error you have for certain even if you can use an infinite number of cases / records. Usually you try to get unbiased models.
Variance / Significance: Relates to the probability that the true relationships between your variables are trivial (e.g. zero) and your model sees an accidental and purely randomly generated data pattern. Variance and bias are usually independent.
Overfitting: Is related to the variance, but it's not the same. If you have a large data matrix you may fit a model with a large number of covariats and many parameters may have a small variance. Nevertheless if you split off 20% of the training material and make a prediction with your model on that validation data, the model may predict worse. That's overfitting: Your model fitted relationships, which aren't randomly within your full data set, but aren't systematic and stable for extrapolations outside the training data set.
Underfitting goes usually with a biased distribution of your residuals: Your model is insufficiently specified. (Relationship between target and covariat is quadratic, but you fit linear). Underfitting can result in a bias, but doesn't have to do. In any case it results in a non-minimal variance. Hope that helps a little bit.
Best Answer
It may be that the math notation is somehow intimidating, but it's not that daunting. Take a look at what happens with an un-biased estimator, such as the sample mean:
The difference between the expectation of the means of the samples we get from a population with mean $\theta$ and that population parameter, $\theta$, itself is zero, because the sample means will be all distributed around the population mean. None of them will be the population mean exactly, but the mean of all the sample means will be exactly the population mean.
This is not the case for other parameters, such as the variance, for which the variance observed in the sample tends to be too small in comparison to the true variance. So if we want to estimate the population variance from the sample we divide by $n-1$, instead of $n$ (Bassel's correction) to correct the bias of the sample variance as an estimator of the population variance:
In both instances, the sample is governed by the population parameter $\theta$, explaining the part in red in the defining equation: $\text{Bias}E[\bar\theta]=E_\color{red}{{p(X|\theta)}}[\bar\theta]-\theta$. However, the $\bar\theta$ steers away from $\theta$ when the estimator is biased.