R - Removing all outliers from a data set

r

Solution

Factors are ints, just not atomic ints.

# Remove outliers from a column
remove_outliers <- function(x, na.rm = TRUE, ...) {
  qnt <- quantile(x, probs=c(.25, .75), na.rm = na.rm, ...)
  H <- 1.5 * IQR(x, na.rm = na.rm)
  y <- x
  y[x < (qnt[1] - H)] <- NA
  y[x > (qnt[2] + H)] <- NA
  y
}

You can replace the columns by index so you don't need to create separate data sets. Just make sure you pass the same data to `lapply`, eg, you don't want to do `data[, 1:3] <- lapply(data, FUN)` which I have done many times.

# Removes all outliers from a data set
remove_all_outliers1 <- function(df){
  # We only want the numeric columns
  df[,sapply(df, is.numeric)] <- lapply(df[,sapply(df, is.numeric)], remove_outliers)
  df
}

Similar to above (and slightly easier I think), you can pass the entire data set to `lapply`. Also making sure not to

data <- lapply(data, if (x) something else anotherthing)

or

data[] <- lapply(data, if (x) something)

Which are also mistakes I have made many times

remove_all_outliers2 <- function(df){
  df[] <- lapply(df, function(x) if (is.numeric(x))
    remove_outliers(x) else x)
  df
}

## test
mt <- within(mtcars, {
  mpg <- factor(mpg)
  gear <- letters[1:2]
})
head(mt)

identical(remove_all_outliers1(mt), remove_all_outliers2(mt))
# [1] TRUE

Your ideas can work with a few minor adjustments. `!is.numeric` can work as either `Negate(is.numeric)` or the more verbose `function(x) !is.numeric(x)` or `!sapply(x, is.numeric)`. Generally, `function(function)` doesn't work in r out of the box.

# Removes all outliers from a data set
remove_all_outliers <- function(df){
  # We only want the numeric columns

  ## drop = FALSE in case only one column for either
  a<-df[,sapply(df, is.numeric), drop = FALSE]
  b<-df[,sapply(df, Negate(is.numeric)), drop = FALSE]

  ## note brackets
  a[]<-lapply(a, function(x) remove_outliers(x))

  ## stack them back together, not merge
  ## you could merge if you had a unique id, one id per row
  ## then make sure the columns are returned in the original order
  d<-cbind(a,b)
  d[, names(df)]
}

identical(remove_all_outliers2(mt), remove_all_outliers(mt))
# [1] TRUE

Problem

Original source