I am confused with the definition of non-parametric model after reading this link Parametric vs Nonparametric Models and Answer comments of my another question.
Originally I thought "parametric vs non-parametric" means if we have distribution assumptions on the model (similar to parametric or non-parametric hypothesis testing). But both of the resources claim "parametric vs non-parametric" can be determined by if number of parameters in the model is depending on number of rows in the data matrix.
For kernel density estimation (non-parametric) such a definition can be applied. But under this definition how can a neural network be a non-parametric model, as the number of parameters in the model is depending on the neural network structure and not on the number of rows in the data matrix?
What exactly is the difference between parametric and a non-parametric model?
Best Answer
In a parametric model, the number of parameters is fixed with respect to the sample size. In a nonparametric model, the (effective) number of parameters can grow with the sample size.
In an OLS regression, the number of parameters will always be the length of $\beta$, plus one for the variance.
A neural net with fixed architecture and no weight decay would be a parametric model.
But if you have weight decay, then the value of the decay parameter selected by cross-validation will generally get smaller with more data. This can be interpreted as an increase in the effective number of parameters with increasing sample size.