machine learning
What is the difference between a classifier, model and estimator?
From what I can tell:
But from looking online, it appears that I may have these definitions mixed up. So, what the true defintions in the context of machine learning?
Obviously, in a streaming context you cannot split your data into train and test sets to perform cross-validation. Using only the metrics calculated on the initial train set sounds even worse, as you assume that your data changes and your model will adapt to the changes--that is why you are using the online learning mode in the first place.
What you could do is to use the kind of cross-validation that is used in time-series (see Hyndman and Athanasopoulos, 2018). To assess accuracy of time-series models, you could use a sequential method, where the model is trained on $k$ observations to predict on the $k+1$ "future" timepoint. This could be applied one point at a time, or in batches, and the procedure is repeated until you have traversed all your data (see the figure below, taken from Hyndman and Athanasopoulos, 2018).
At the end, you somehow average (usually arithmetic mean, but you could use something like exponential smoothing as well) the error metrics to obtain the overall accuracy estimate.
In an online scenario this would mean that you start at timepoint 1 and test on timepoint 2, next re-train on timepoint 2, to test on timepoint 3 etc.
Notice that such cross-validation methodology lets you account for the changing nature of your models performance. Obviously, as your model adapts to the data and the data may change, you would need to monitor the error metrics regularly: otherwise it wouldn't differ much from using fixed-size train and test sets.
A classifier is a method that maps from inputs x to outputs l , where l are instances of a set of labels L.
There are many methods to build a classifier, an approach is:
define a variable y with value 1 when l is a label l' and 0 when is not that label.
In this way, we can translate the mapping in estimating a function f(x;θ ) such that y=f(x;θ ) where f is user defined and the parameters θ are selected to meet user stated requirements ( for example low classification errors , few parameters,....).
Logistic Regression is an example, using this method allows to leverage on theory, algorithms, ..
Best Answer