I understand the premise of kNN algorithm for spatial data. And I know I can extend that algorithm to be used on any continuous data variable (or nominal data with Hamming Distance). However, what strategies are used when dealing with higher dimensional data?
For example, say I have a table of data (x[1], x[2], x[3], …, x[n]) and I want to build a set of classifiers to predict one of those columns (say x[n]). Using kNN algorithm I would pick any two columns from the remaining columns (x[1]-x[n-1]) to train against. So say I could pick x[1] and x[2] and build a classifier off those. Or I could pick x[1] and x[4], or I could pick x[5] and x[8], etc. I could even pick just a single column and build a classifiers off that, or 3 columns and build a classifiers off that. Is there an advantage to using higher dimensions (2D, 3D, etc) or should you just build x-1 single dimension classifiers and aggregate their predictions in some way?
Since building all of these classifiers from all potential combinations of the variables would be computationally expensive. How could I optimize this search to find the the best kNN classifiers from that set? And, once I find a series of classifiers what's the best way to combine their output to a single prediction? Voting might be the simplest answer to this question. Or weighting each vote by error rates from the training data for each classifier.
How do most implementations apply kNN to a more generalized learning?
Best Answer
This depends on whether your features are informative or not. Do you suspect that some features will not be useful in your classification task? To gain a better idea of your data, you can also try to compute pairwise correlation or mutual information between the response variable and each of your features.
To combine all (or a subset) of your features, you can try computing the L1 (Manhattan), or L2 (Euclidean) distance between the query point and each 'training' point as a starting point.
This is the problem of feature subset selection. There is a lot of academic work in this area (see Guyon, I., & Elisseeff, A. (2003). An Introduction to Variable and Feature Selection. Journal of Machine Learning Research, 3, 1157-1182. for a good overview).
This will depend on whether or not the selected features are independent or not. In the case that features are independent, you can weight each feature by its mutual information (or some other measure of informativeness) with the response variable (whatever you are classifying on). If some features are dependent, then a single classification model will probably work best.
By allowing the user to specify their own distance matrix between the set of points. kNN works well when an appropriate distance metric is used.