Kmeans clustering time on scikit
k-means, machine-learning, python-2.7, scikit-learn
Solution
If you haven't tried it yet, use `sklearn.cluster.MiniBatchKMeans` instead of `sklearn.cluster.KMeans`
E.g., if `X.shape = (100000, 2048)`, then write
from sklearn.cluster import MiniBatchKMeans
mbkm = MiniBatchKMeans(n_clusters=200) # Take a good look at the docstring and set options here
mbkm.fit(X)
`MiniBatchKMeans` finds slightly different clusters from normal `KMeans`, but has the huge advantage that it is an online algorithm which doesn't need all the data at every iteration and still gives useful results.
Problem
How much time should it take to cluster a set of 100'000 L2 normalized 2048-dim feature vectors using k-means with 200 clusters? I have all my data in a huge numpy array, maybe there's a more appropriate data structure? It didn't seem to do any progress in an hour. I'm also inclining to use the threshold stopping criteria, but it seems to take more than 5 minutes for just 2 iterations. Is there some sort of verbose command I can use to check in on the progress for kmeans clustering on scikit-learn? Does anyone suggest any other approach? Maybe something like dimensionality reduction, or PCA and then kmeans? (I'm just throwing random ideas out there)