Caret Binary Classification with RMSE

r, r-caret

Solution

I don't know why you would want to do this but you can make your own summary function:

library(caret)

set.seed(1)
dat <- twoClassSim(100)

foo <- function(data, lev = NULL, model = NULL) {
  probs <- data[, lev[1]]
  c(rmse = RMSE(pred = probs,
                obs = ifelse(data$obs == lev[1], 1, 0)))
}

ctrl <- trainControl(classProbs = TRUE,
                     summaryFunction = foo)
set.seed(2)
mod <- train(Class ~ ., data = dat,
             method = "lda",
             metric = "rmse",
             minimize = TRUE,
             trControl = ctrl)

Max

Problem

Is there a way to get caret to use RMSE with a binary classification problem? If you try to use `metric = "RMSE"` with a classification problem you will receive the message: ``` Error in train.default(x, y, weights = w, ...) : Metric RMSE not applicable for classification models ``` Which makes sense. But is there a way to define a custom metric? For example, if your outcome is `0` or `1`, you can define the error as `outcome - p` where `p` is the probability predicted by the model. EDIT ==================== To give this some context and for some reasoning behind wanting to use this measure, see 2.7.1 in An Experimental Analysis of Classifier Ensembles for Learning Drifting Concepts Over Time in Autonomous Outdoor Robot Navigation by Michael J. Procopio, or the paper on softclassval

Original source