...the expected [squared error] loss can be decomposed into a squared
bias term (which describes how far the average predictions are from
the true model), a variance term (which describes the spread of the
predictions around the average), and a noise term (which gives the
intrinsic noise of the data).
When looking at the squared error loss decomposition
$$\mathbb{E}_\theta[(\theta-\delta(X_{1:n}))^2]=(\theta-\mathbb{E}_\theta[\delta(X_{1:n})])^2+\mathbb{E}_\theta[(\mathbb{E}_\theta[\delta(X_{1:n})]-\delta(X_{1:n}))^2]$$
I only see two terms: one for the bias and another one for the variance of the estimator or predictor, $\delta(X_{1:n})$. There is no additional noise term in the expected loss. As should be since the variability is the variability of $\delta(X_{1:n})$, not of the sample itself.
- Can bias-variance decomposition be performed with loss functions other
than squared loss?
My interpretation of the squared bias+variance decomposition [and the way I teach it] is that this is the statistical equivalent of Pythagore's Theorem, namely that the squared distance between an estimator and a point within a certain set is the sum of the squared distance between an estimator and the set, plus the squared distance between the orthogonal projection on the set and the point within the set. Any loss based on a distance with a notion of orthogonal projection, i.e., an inner product, i.e., essentially Hilbert spaces, satisfies this decomposition.
- For a given model dataset, is there more than one model whose expected
loss is the minimum over all models, and if so, does that mean that
there could be different combinations of bias and variance that yield
the same minimum expected loss?
The question is unclear: if by minimum over models, you mean
$$\min_\theta \mathbb{E}_\theta[(\theta-\delta(X_{1:n}))^2]$$
then there are many examples of statistical models and associated decisions with a constant expected loss (or risk). Take for instance the MLE of a Normal mean.
- How can you calculate bias if you don't know the true model?
In a generic sense, the bias is the distance between the true model and the closest model within the assumed family of distributions. If the true model is unknown, the bias can be ascertained by bootstrap.
- Are there situations in which it makes more sense to minimize bias or variance rather than expected loss (the sum of squared bias and
variance)?
When considering another loss function like
$$(\theta-\mathbb{E}_\theta[\delta(X_{1:n})])^2+\alpha[(\mathbb{E}_\theta[\delta(X_{1:n})]-\delta(X_{1:n}))^2]\qquad 0<\alpha$$
pushing $\alpha$ to zero puts most of the evaluation on the bias while pushing $\alpha$ to infinity switches the focus on the variance.
why such a penalty is used
It is used to penalize the complexity of model (function), or equivalently to encourage the model to be as simple as possible (roughly has fewer weights involved). This way, over-fitting is avoided and generalization of model to unseen data is improved.
Why does penalizing larger values of W affect the "bias/variance
trade-off" of a model?
By forcing the parameters to be close to zero, the variance of model which is a function of its parameters would decrease, since parameters are less free to change compared to when their value is not penalized. On the other hand, the true function may need a wide range of weights to be estimated correctly, i.e. more flexible/complex model. However, by this penalty, we are allowing less flexibility (less complexity) thus moving the model away from the best configuration of weights required to estimate the true function, i.e. more bias.
Best Answer
When noise is "large" then learning is not pointless, but it's "expensive" in some sense. For instance, you know the expression "house always wins". It means that the odds favor the casino against the gambler. However, the odds can be very close to 1:1, they may only so slightly be tilted towards the "house", e.g. 0.5%. Hence, you may call the outcomes series very noisy in some cases, yet the casinos make a ton of money in a long run. So, the fact that the data is "noisy" doesn't mean in isolation that the learning will be pointless or useless or unprofitable.