I'm trying to figure out a simple way to do something like this with dplyr (data set = dat, variable = x):
dat$x[dat$x<0]=NA Should be simple but this is the best I can do at the moment. Is there an easier way?
dat = dat %>% mutate(x=ifelse(x<0,NA,x)) 5 Answers
You can use replace which is a bit faster than ifelse:
dat <- dat %>% mutate(x = replace(x, x<0, NA)) You can speed it up a bit more by supplying an index to replace using which:
dat <- dat %>% mutate(x = replace(x, which(x<0L), NA)) On my machine, this cut the time to a third, see below.
Here's a little comparison of the different answers, which is only indicative of course:
set.seed(24) dat <- data.frame(x=rnorm(1e6)) system.time(dat %>% mutate(x = replace(x, x<0, NA))) User System elapsed 0.03 0.00 0.03 system.time(dat %>% mutate(x=ifelse(x<0,NA,x))) User System elapsed 0.30 0.00 0.29 system.time(setDT(dat)[x<0,x:=NA]) User System elapsed 0.01 0.00 0.02 system.time(dat$x[dat$x<0] <- NA) User System elapsed 0.03 0.00 0.03 system.time(dat %>% mutate(x = "is.na<-"(x, x < 0))) User System elapsed 0.05 0.00 0.05 system.time(dat %>% mutate(x = NA ^ (x < 0) * x)) User System elapsed 0.01 0.00 0.02 system.time(dat %>% mutate(x = replace(x, which(x<0), NA))) User System elapsed 0.01 0.00 0.01 (I'm using dplyr_0.3.0.2 and data.table_1.9.4)
Since we're always very interested in benchmarking, especially in the course of data.table-vs-dplyr discussions I provide another benchmark of 3 of the answers using microbenchmark and the data by akrun. Note that I modified dplyr1 to be the updated version of my answer:
set.seed(285) dat1 <- dat <- data.frame(x=sample(-5:5, 1e8, replace=TRUE), y=rnorm(1e8)) dtbl1 <- function() {setDT(dat)[x<0,x:=NA]} dplr1 <- function() {dat1 %>% mutate(x = replace(x, which(x<0L), NA))} dplr2 <- function() {dat1 %>% mutate(x = NA ^ (x < 0) * x)} microbenchmark(dtbl1(), dplr1(), dplr2(), unit='relative', times=20L) #Unit: relative # expr min lq median uq max neval # dtbl1() 1.091208 4.319863 4.194086 4.162326 4.252482 20 # dplr1() 1.000000 1.000000 1.000000 1.000000 1.000000 20 # dplr2() 6.251354 5.529948 5.344294 5.311595 5.190192 20 The most natural approach in dplyr is to use the na_if function.
For one variable:
dat %<>% mutate(x = na_if(x, x < 0)) For all variables:
dat %<>% mutate_all(~ na_if(., . < 0)) If interested in replacing a specific value, instead of a range for all variables:
dat %<>% mutate_all(na_if, 0) Note that I am using the %<>% operator from the magrittr package.
You can use the is.na<- function:
dat %>% mutate(x = "is.na<-"(x, x < 0)) Or you can use mathematical operators:
dat %>% mutate(x = NA ^ (x < 0) * x) If you are using data.table, the below code is faster
library(data.table) setDT(dat)[x<0,x:=NA] Benchmarks
Using data.table_1.9.5 and dplyr_0.3.0.9000
library(microbenchmark) set.seed(285) dat <- data.frame(x=sample(-5:5, 1e7, replace=TRUE), y=rnorm(1e7)) dtbl1 <- function() {as.data.table(dat)[x<0,x:=NA]} dplr1 <- function() {dat %>% mutate(x = replace(x, x<0, NA))} microbenchmark(dtbl1(), dplr1(), unit='relative', times=20L) #Unit: relative #expr min lq mean median uq max neval cld #dtbl1() 1.00000 1.000000 1.000000 1.000000 1.000000 1.000000 20 a #dplr1() 2.06654 2.064405 1.927762 1.795962 1.881821 1.885655 20 b Updated Benchmarks
Using data.table_1.9.5 and dplyr_0.4.0. I used a slightly bigger dataset and replaced as.data.table with setDT (Included @Sven Hohenstein's faster function as well.)
set.seed(285) dat <- data.frame(x=sample(-5:5, 1e8, replace=TRUE), y=rnorm(1e8)) dat1 <- copy(dat) dtbl1 <- function() {setDT(dat)[x<0,x:=NA]} dplr1 <- function() {dat1 %>% mutate(x = replace(x, x<0, NA))} dplr2 <- function() {dat1 %>% mutate(x = NA ^ (x < 0) * x)} microbenchmark(dtbl1(), dplr1(), dplr2(), unit='relative', times=20L) #Unit: relative # expr min lq mean median uq max neval cld #dtbl1() 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 20 a #dplr1() 2.523945 2.542412 2.536255 2.579379 2.518336 2.486757 20 b #dplr2() 1.139216 1.089992 1.088753 1.058653 1.093906 1.100690 20 a Updated Benchmarks2
At the request of @docendo discimus, benchmarking again his "new" version of dplyrusing data.table_1.9.5 and dplyr_0.4.0.
NOTE: Because there is a change in @docendo discimus code, I changed 0 to 0L for the data.table`
set.seed(285) dat <- data.frame(x=sample(-5:5, 1e8, replace=TRUE), y=rnorm(1e8)) dat1 <- copy(dat) dtbl1 <- function() {setDT(dat)[x<0L, x:= NA]} dplr1 <- function() {dat1 %>% mutate(x = replace(x, which(x<0L), NA))} dplr2 <- function() {dat1 %>% mutate(x = NA ^ (x < 0) * x)} microbenchmark(dtbl1(), dplr1(), dplr2(), unit='relative', times=20L) #Unit: relative #expr min lq mean median uq max neval cld #dtbl1() 1.000000 1.000000 1.000000 1.000000 1.000000 1.000000 20 a #dplr1() 2.186055 2.183432 2.142293 2.222458 2.194450 1.442444 20 b #dplr2() 2.919854 2.925795 2.852528 2.942700 2.954657 1.904249 20 c data
set.seed(24) dat <- data.frame(x=sample(-5:5, 25, replace=TRUE), y=rnorm(25)) Using replace directly on the x column and not using mutate also works.
dat$x <- replace(dat$x, dat$x<0, NA) 
