Perhaps this question is too broad, but I would like to know – how does one use a kernel density estimate in practice? I know of course that one can use it to draw pretty pictures on top of histograms, but what in particular can a KDE be used for that the empirical distribution itself cannot be used for, especially with the whole notion of needing to properly determine the correct bandwith?
Practical Uses of Kernel Density Estimators – Density Estimation Techniques
density-estimationkernel-smoothing
Related Solutions
I would recommend you to read this beautiful article by Racine and Li published in the Journal of Econometrics in 2004. They develop a framework to estimate regression functions nonparametrically using kernel methods, with mixed types of covariates (categorical or continuous regressors). Among other results, they show consistency of the cross-validated estimates. This is a classical article in the nonparametric econometrics literature.
The main method for choosing bandwidth parameters is, undoubtfully, the cross-validation procedure. However, other methods exist such as bootstraping (a quick google search gives: a PhD thesis about choosing bandwidths for np kernel regression.
If you have a large sample, leave-one-out CV may not be the way to go for computational reasons. Moreover, if the data consist of time-series, the CV method may not be valid anymore. What you can do is proceed with hold-out validation. Assume you have $T$ observations.
- Split the sample into two parts: the estimation sample (observation 1 to $T-k$) and a hold-out sample (obs $T-k+1$ to $T$).
- Compute the estimator using the estimation sample (first $T-k$ obs) as a function of $h$.
- Compute the out-of-sample prediction for the hold-out sample (last $k$ obs) as a function of $h$.
- Minimize the squared prediction error with respect to $h$.
- Recompute $h$ using the same procedure as new data come in.
Also, if the Nadaraya-Watson estimator is indeed a np kernel estimator, this is not the case for Lowess, which is a local polynomial regression method. You could also fit your regression function using the Sieves (i.e. through a basis expansion of the function) based on wavelets for example given the structure of your data.
Finally, the np kernel estimation of densities is extremely similar to that of the conditional mean, which is what you have in mind when you talk about 'regression'. A np kernel regression considers estimating $E(Y|X)$, where $Y$ is the dependent variable and $X$ is a (hopefully) exogenous predictor. Replacing Y by $I(Y\leq y)$ - where $I$ denotes the indicator function that equates $1$ when the event inside the brackets occurs - gives the conditional mean expression $E(I(Y\leq y)|X)$. Now, run a bunch of np kernel regression on $E(I(Y\leq y)|X)$ for various values of $y$. This will provide an estimate the conditional cumulative distribution of $Y$ given $X$. Now take a derivative with respect to $y$, you have a density. So what's the difference after all? Just a matter of choosing the dependent variable. The method is the same, unless you want to re-scale and impose restrictions on the CDF maybe...
Best Answer
As noticed by others, empirical distribution is not smooth, it's a step function. Say that you observed three datapoints: 2.56, 4.17, and 4.89 and I ask you "what would be the probability density of observing the value 3.14?". Given empirical distribution, there is no nice answer, you need to make some rather arbitrary decision on how to proceed. It's not really that empirical distribution "cannot be used" in such cases, but that it would serve as a very rough approximation of the underlying distribution.
There are many uses of kernel densities, for example:
etc.
Yes, choosing the bandwidth is somehow arbitrary, but it often does not hurt us that much to pick imperfect bandwidth and there are already many available algorithms for picking it. Notice however that when using other methods (empirical distribution, $k$NN) you run into other problems, like deciding on how to interpolate and extrapolate from the estimated distributions, or picking other kinds of hyperparameters.