Distances between rankings
pandas, python, scikit-learn, scipy
Solution
You're looking for Normalized Discounted Cumulative Gain (NDGC). It's a metric commonly used in search engine rankings to test the quality of the result ranking.
The idea is that you test your ranking (in your case the two methods) against user feedback through clicks (in your cast the true rank). NDGC will tell you the quality of your ranking relative to the truth.
Python has RankEval based module that implements this metric (and some others if you want to try them). The repo is here and there is a nice IPython NB with examples
Problem
I have two methods that rank a list of strings differently, and what we can consider to be the "right" ranking of the list (i.e. a gold standard). In other words: ``` ranked_list_of_strings_1 = method_1(list_of_strings) ranked_list_of_strings_2 = method_2(list_of_strings) correctly_ranked_list_of_strings # Some permutation of list_of_strings ``` How can I determine which method is better considering that `method_1` and `method_2` are black boxes? Are there any methods to measure this available either in `SciPy` or `scikit-learn` or similar libraries? In my specific case, I actually have a dataframe, and each method outputs a score. What matters is not the difference in score between the methods and the true scores, but that the methods get the ranking right (higher score means higher ranking for all columns). ``` strings scores_method_1 scores_method_2 true_scores 5714 aeSeOg 0.54 0.1 0.8 5741 NQXACs 0.15 0.3 0.4 5768 zsFZQi 0.57 0.7 0.2 ```