I'm asking this out of curiosity.
In the past I have thought of an heuristic as a "quick and dirty" rule not based on data analysis, as opposed to a solution which uses machine learning or statistical models.
For example imagine I have the following classification problem: are products listed in different e-commerce websites the same physical product?
One could simply define an initial heuristic that if product titles similarity is below an arbitrary level than products are considered different otherwise they are considered the same. This would be what I call an heuristic.
On the other hand, tagging data, training a model and cross validate to get best threshold to classify products is NOT an heuristic.
However if I read Wikipedia definition of Heuristic it says that it is a practical method that does not guarantee to be optimal or perfect. This seems very general and it seems that it could be extended to machine learning.
Can anyone help me understand the distinction a bit better?
Best Answer
The problem with this kind of definition is that it is ambiguous and can be understood differently by different people and in different contexts. Wikipedia says that,
How do you know that the solution is optimal or perfect? When you are dealing with random phenomena, then you cannot get "perfect" results (i.e., always correct). What machine learning algorithms give you, is the "best you can get" results, given certain conditions are met. Moreover, each of the algorithms that are commonly used gives you some guarantees for optimality in certain scenarios (if they didn't, we wouldn't use them).
Heuristics have very similar, though more precise, meaning in computer science, tl;dr: they are algorithms that seek an approximate, opinionated solution rather than the exact one. In machine learning, there is usually no exact solutions, so it is not achievable by any algorithm.