Pairwise Kullback Leibler (or Jensen-Shannon) divergence distance matrix in Python
distance, matrix, metrics, python
Solution
Note that KL divergence is essentially a dot product of P(i) and log(P(i)/Q(i)). So, one option is to form a list of numpy arrays for P(i) and another for log(P(i)/Q(i)), one row for each KL divergence you want to calculate), then perform dot-products.
Problem
I have two matrices X and Y (in most of my cases they are similar) Now I want to calculate the pairwise KL divergence between all rows and output them in a matrix. E.g: ``` X = [[0.1, 0.9], [0.8, 0.2]] ``` The function should then take `kl_divergence(X, X)` and compute the pairwise Kl divergence distance for each pair of rows of both X matrices. The output would be a 2x2 matrix. Is already some implementation for this in Python? If not, this should be quite simple to calculate. I'd like some kind of matrix implementation for this, because I have a lot of data and need to keep the runtime as low as possible. Alternatively the Jensen-Shannon entropy is also fine. Eventually this would even be a better solution for me.