I understand the view that ML is a big optimization problem where we are trying to minimize the loss function and achieve the most optimal solution given the input. To achieve that we are feeding a loss function (let's say accuracy) and an optimizer (let's say stochastic gradient descent) which is helpful for the model to tune the parameters if it is a parametric learner, unlike kNN. But after all, the loss function, optimizer, decision boundary shape in SVM, hidden layer count in NN, maximum depth of a tree in bagging or the base estimator in boosting are all hyperparameters that the user needs to tune considering bias and variance trade-off.
Assuming that we have unlimited resources, can't we just find the strongest model by using a large GridSearchCV with many hyperparameter combinations? It boils down to this: Is ML all about hyperparameter tuning? If not, what am I missing?
I am also asking for cases where interpretability is not that important and the sole purpose is achieving the highest testing score.
Thanks in advance!
Best Answer
Indeed, a large part of ML is hyperparameter tuning, besides finding the appropriate method/model for your task, but I guess you could argue that this is part of hyperparameter tuning, too. Especially for the user that just needs to apply models to get answers, there is not much more than hyperparameter tuning. (OK, to get your job done in practice, you usually have to do lots of data preprocessing and create some kind of ML pipeline (i.e. all the data engineering stuff). That usually takes up 99% of your time, but this is arguably not really part of ML.)
For the designers of e.g. models, algorithms, and model selection methods, ML is more than this. It's a lot about stochastics, optimization, functional analysis, geometry, all kinds of math.
And it is an undeniable fact that you often do a much better job in hyperparameter tuning if you know the inner workings of the models.