I just started an intro machine learning course, and to get things better organized in my head, I was trying to come up with exactly what is needed to completely specify a classification algorithm. I understand a precise mathematical definition may not be possible. Here's what I have:
Let $\{X, Y\}$ denoted the data set (sample, label), $\theta$ the parameters.
A classification algorithm is a decision function $f(x; \theta)$ together with a cost/risk function $C(f(\cdot; \theta), X, Y)$. Specifying the functional form of $f$ and $C$ completely specifies the classification algorithm. $\hat{\theta} = \arg\min_\theta C(\theta)$ defines the classifier.
Examples I'm thinking of are maybe logistic regression, where $f$ is the logistic function and $C$ is the cross entropy cost, or linear regression, where $f$ is the linear function and $C$ is the sum of squares cost, or the perception algorithm, where the function $f$ is the linear function and $C$ is the sum of individual losses consisting of the kinked function.
Is the above correct? Roughly? Where is it wrong?
Best Answer
A classifier is a method that maps from inputs x to outputs l , where l are instances of a set of labels L.
There are many methods to build a classifier, an approach is:
define a variable y with value 1 when l is a label l' and 0 when is not that label.
In this way, we can translate the mapping in estimating a function f(x;θ ) such that y=f(x;θ ) where f is user defined and the parameters θ are selected to meet user stated requirements ( for example low classification errors , few parameters,....).
Logistic Regression is an example, using this method allows to leverage on theory, algorithms, ..