Why does Weka RandomForest gives me a different result than Scikit RandomForestClassifier?
python, random-forest, scikit-learn, weka
Solution
After posting the question at scikit-learn issue tracker, I got feedback that the problem is in the "predict" function I used. It should have been "pred_test = clr.predict_proba(X_test)[:, 1]" instead of "pred_test = clr.predict(X_test)", since the classification problem is binary: either 0 or 1.
After implementing the change, the results turned out to be the same for WEKA's and scikit's random forest :)
Problem
I am getting peculiar differences in results between WEKA and scikit while using the same RandomForest technique and the same dataset. With scikit I am getting an AUC around 0.62 (all the time, for I did extensive testing). However, with WEKA, im getting results close to 0.79. Thats a huge difference! The dataset I tested the algorithms on is KC1.arff, of which I put a copy in my public dropbox folder https://dl.dropbox.com/u/30688032/KC1.arff. For WEKA, I simply downloaded the .jar file from http://www.cs.waikato.ac.nz/ml/weka/downloading.html. In WEKA, I set the cross-validation parameter as 10-fold, the dataset as KC1.arff, the algorithm as "RandomForest -l 19 -K 0 -S 1". Then ran the code! Once you generate the results in WEKA, it should be saved as a file, .csv or .arff. Read that file and check the column 'Area_under_ROC', it should be somewhat close to 0.79. Below is the code for the scikit's RandomForest ``` import numpy as np from pandas import * from sklearn.ensemble import RandomForestClassifier def read_arff(f): from scipy.io import arff data, meta = arff.loadarff(f) return DataFrame(data) def kfold(clr,X,y,folds=10): from sklearn.cross_validation import StratifiedKFold from sklearn import metrics auc_sum=0 kf = StratifiedKFold(y, folds) for train_index, test_index in kf: X_train, X_test = X[train_index], X[test_index] y_train, y_test = y[train_index], y[test_index] clr.fit(X_train, y_train) pred_test = clr.predict(X_test) print metrics.auc_score(y_test,pred_test) auc_sum+=metrics.auc_score(y_test,pred_test) print 'AUC: ', auc_sum/folds print "----------------------------" #read the dataset X=read_arff('KC1.arff') y=X['Defective'] #changes N, and Y to 0, and 1 respectively s = np.unique(y) mapping = Series([x[0] for x in enumerate(s)], index = s) y=y.map(mapping) del X['Defective'] #initialize random forests (by defualt it is set to 10 trees) rf=RandomForestClassifier() #run algorithm kfold(rf,np.array(X),y) #You will get an average AUC around 0.62 as opposed to 0.79 in WEKA ``` Please keep in mind that the real auc value, as shown in relevant papers' experimental results, is around 0.79, so the problem lies on my implementation that uses the scikit random forests. Your kind help will be highly appreciated!! Thank you very much!