In machine learning, people usually refer to the "performance of a model" or "performance of an optimizer". What is the exact definition of "performance"? What would be the "performance of an optimizer"?
I know that there are ways of measuring how far away current predictions of machine learning models are from the expected ones: for example, you can measure the accuracy of a model, perplexity, etc. Is this what the performance of a ML model refers to? Is it a name to refer to any way of measuring the correctness of the predictions (or, in general, outputs) of the model? Or is performance actually the time it takes to perform the prediction? Or something else?
Best Answer
In the absence of any specific clarifying context, "performance" is just a synonym for "quality."
The sentence "I want a model that performs better" is essentially the same as the sentence "I want a higher-quality model." Readers understand that the speaker is not satisfied with how well the model solves some particular problem, but the reader does not know, precisely, what about the model is dissatisfactory. Does the model predict too many false positives? Or false negatives? Does it predict incorrect classes for images that have a tilted horizon, or are taken on cloudy days? Understanding what about the model needs improvement would require further, specific elaboration.
Likewise, if someone says that Adam has better performance than another optimizer, they're making a claim that Adam does better at some task, which they would have to specify for it to be possible to assess the truthfulness of the claim. One way to assesses performance of an optimizer is how many iterations it takes to reach some neighborhood around a minimum; another, which is particular to machine learning classifiers, is how well the solutions obtained by an optimizer generalize to out-of-sample data.