**Questions**

- Does it depend on whether the tree is shallow or deep? Or can we say this irrespective of the depth/levels of the tree?
- Why is bias low & variance high? Please explain intuitively and mathematically

Skip to content
# Solved – Why does a decision tree have low bias & high variance

###### Related Question

biascartcovariancemachine learningvariance

**Questions**

- Does it depend on whether the tree is shallow or deep? Or can we say this irrespective of the depth/levels of the tree?
- Why is bias low & variance high? Please explain intuitively and mathematically

- Solved – How will one determine a classifier to be of high bias or high variance
- Machine Learning – Bias and Variance of a Decision Tree for Classification Explained
- Solved – Why does increasing K increase bias and reduce variance
- Machine Learning – What Does Low Bias and High Variance Mean in a Model?
- Machine Learning – Understanding the Different Curves of Test Error and Variance in Bias-Variance Trade-Off

## Best Answer

A bit late to the party but i feel that this question could use answer with concrete examples.

I will write summary of this

excellentarticle: bias-variance-trade-off, which helped me understand the topic.The prediction error for any machine learning algorithm can be broken down into three parts:

Irreducible errorAs the name implies, is an error component that we cannot correct, regardless of algorithm and it's parameter selection. Irreducible error is due to complexities which are simply

notcaptured in the training set. This could be attributes which we don't have in a learning set but they affect the mapping to outcome regardless.Bias errorBias error is due to our assumptions about target function. The more assumptions(restrictions) we make about target functions, the more bias we introduce. Models with high bias are less flexible because we have imposed more rules on the target functions.

Variance errorVariance error is variability of a target function's form with respect to different training sets. Models with small variance error will not change much if you replace couple of samples in training set. Models with high variance might be affected even with small changes in training set.

Consider simple linear regression:

Obviously, this is a fairly restrictive definition of a target function and therefore this model has a high bias.

On the other hand, due to low variance if you change couple of data samples, it's unlikely that this will cause major changes in the overall mapping the target function performs. On the other hand, algorithm such as k-nearest-neighbors have high variance and low bias. It's easy to imagine how different samples might affect K-N-N decision surface.

Generally, parametric algorithms have a high bias and low variance, and vice versa.

One of the challenges of machine learning is finding the right balance of bias error and variance error.

Decision treeNow that we have these definitions in place, it's also straightforward to see that decision trees are example of model with low bias and high variance. The tree makes almost no assumptions about target function but it is highly susceptible to variance in data.

There are ensemble algorithms, such as bootstrapping aggregation and random forest, which aim to reduce variance at the small cost of bias in decision tree.