Is there an ideal "inertia" for K-mean convergence. For example I'm trying to cluster to 64 clusters using sci-kit. the output is
Initialization complete
Iteration 0, inertia 186.351
Iteration 1, inertia 143.837
Iteration 2, inertia 140.469
Iteration 3, inertia 138.978
Iteration 4, inertia 138.230
Iteration 5, inertia 137.721
Iteration 6, inertia 137.331
Iteration 7, inertia 137.061
Iteration 8, inertia 136.897
Iteration 9, inertia 136.791
Iteration 10, inertia 136.721
Iteration 11, inertia 136.655
Iteration 12, inertia 136.611
Iteration 13, inertia 136.577
Iteration 14, inertia 136.549
Iteration 15, inertia 136.524
Iteration 16, inertia 136.500
Iteration 17, inertia 136.475
Iteration 18, inertia 136.455
Iteration 19, inertia 136.439
Iteration 20, inertia 136.426
Iteration 21, inertia 136.413
Iteration 22, inertia 136.402
Iteration 23, inertia 136.393
Iteration 24, inertia 136.382
Iteration 25, inertia 136.374
Iteration 26, inertia 136.369
Iteration 27, inertia 136.364
Iteration 28, inertia 136.359
Iteration 29, inertia 136.350
Iteration 30, inertia 136.338
Iteration 31, inertia 136.330
Iteration 32, inertia 136.325
Iteration 33, inertia 136.320
Iteration 34, inertia 136.314
Iteration 35, inertia 136.300
Iteration 36, inertia 136.288
Iteration 37, inertia 136.278
Iteration 38, inertia 136.270
Iteration 39, inertia 136.264
Iteration 40, inertia 136.258
Iteration 41, inertia 136.255
Iteration 42, inertia 136.249
Iteration 43, inertia 136.248
Iteration 44, inertia 136.248
Iteration 45, inertia 136.248
How do I Know this is the correct convergence? I guess the bigger question how do I know it converged to a global and not a local minimum?
Best Answer
Inertia is sensitive to evrerything. Scale, number of clusters, ...
It does not allow for objective comparisons. The only value that allows absolute conclusions is 0: then every object is equal to a centroid.
Finding the global optimum is too hard. You cannot do this on larger data and larger k. You have to rely on luck. Try running k-means multiple times, and see how different the values are.
Beware to check all parameters. Many implementations have a threshold, and stop before converging, if the improvements get too small, or after a certain number of iterations. Then you don't even have a local optimum yet.