large scale clustering library possibly with python bindings
cluster-analysis, data-mining, python
Solution
50000 instances and 7 dimensions isn't really big, and should not kill an implementation.
Although it doesn't have python binding, give ELKI a try. The benchmark set they use on their homepage is 110250 instances in 8 dimensions, and they run k-means on it in 60 seconds apparently, and the much more advanced OPTICS in 350 seconds.
Avoid hierarchical clustering. It's really only for small data sets. The way it is commonly implemented on matrix operations is `O(n^3)`, which is really bad for large data sets. So I'm not surprised these two timed out for you.
DBSCAN and OPTICS when implemented with index support are `O(n log n)`. When implemented naively, they are in `O(n^2)`. K-means is really fast, but often the results are not satisfactory (because it always splits in the middle). It should run in `O(n * k * iter)` which usually converges in not too many iterations (`iter<<100`). But it will only work with Euclidean distance, and just doesn't work well with some data (high-dimensional, discrete, binary, clusters with different sizes, ...)
Problem
I've been trying to cluster some larger dataset. consisting of 50000 measurement vectors with dimension 7. I'm trying to generate about 30 to 300 clusters for further processing. I've been trying the following clustering implementations with no luck: - Pycluster.kcluster (gives only 1-2 non-empty clusters on my dataset) - scipy.cluster.hierarchy.fclusterdata (runs too long) - scipy.cluster.vq.kmeans (runs out of memory) - sklearn.cluster.hierarchical.Ward (runs too long) Are there any other implementations which I might miss?