In my seminar work I used the following sentence:
Overfitting stands out as the most important aspect of machine
learning and statistics.
Here, I want to replace "statistics" with either Inductive Reasoning or Statistical Inference. The terminology is a bit confusing and I am not sure which one to take.
Could someone clarify the difference (if possible from a machine learning perspective) between the two so I know which one to pick.
Best Answer
Definitions:
Lots of definitions abound, here is two I found that basically speak for the others I read
‘Inductive reasoning is the opposite of deductive reasoning. Inductive reasoning makes broad generalizations from specific observations.’ (https://www.livescience.com/21569-deduction-vs-induction.html)
‘Statistical inference means drawing conclusions based on data’ (http://www2.stat.duke.edu/~fab2/inference_talk.pdf)
The way I understand it is that Statistical Inference is a specialised type of Inductive Reasoning that specifically uses statistical tools to support and guide the reasoning/inference in a rigorous way.
It is the difference between looking at the data for two samples and saying,
Comments on Overfitting
You particularly reference overfitting which I see as one of the biggest challenges for machine learning. However, as you indicate by your statement it is not exclusive to machine learning by any stretch, and is a fundamental problem across science and is the very heart of the dangers of statistical inference. Statistical inference is by definition taking the results of applying some sort of construct or model to your specific data and then speculating that it would continue to apply in comparable or even reasoning how it may perform in expanded (extrapolation) situations.
Machine Learning example
So in machine learning the inductive reasoning could be simple as:
‘Model A showed good performance when we calibrated it and maintained strong performance in the validation set. We therefore expect that when we deploy the model in the final use case it will continue to provide similar levels of performance. This is based on the assumption that the subjects in the final use case fit a similar profile to those in the validation set. Specific exclusions are A, B C and D.'
A reasonably thorough statistical inference would be:
‘Model A worked with X (X$_1$-X$_2$) % accuracy in the calibration and Y (Y$_1$-Y$_2$)% in the validation set, therefore we expect to achieve an accuracy of Z (Z$_1$-Z$_2$) % when the model is deployed to its final use case’. This is based on the assumption that the subjects in the final use case fit a similar profile to those in the validation set, which a set of characteristics M(M$_1$-M$_2$), N(N$_1$-N$_2$), O(O$_1$-O$_2$). Specific exclusions are A, B C and D. If the final use case exposes the model to subjects with M$<$M$_1$ then the accuracy will likely be affected by V% (etc.)’