how to remove correlated or duplicated variable or individuals in r

Question

I have the following type (however very large number of variables and ind) data: ``` mydf <- data.frame (Inv = 1:6, varA = c(1,1,1, 0,1,1), varB = c(1,0,1, 0, 1,1), varC = c(1,0,0, 0,1,1), varD = c(1,1,1, 0,1,1), varE = c(1,0,1, 0, 1,1), varF = c(1,1,1, 0, 1,1)) mydf Inv varA varB varC varD varE varF 1 1 1 1 1 1 1 1 2 2 1 0 0 1 0 1 3 3 1 1 0 1 1 1 4 4 0 0 0 0 0 0 5 5 1 1 1 1 1 1 6 6 1 1 1 1 1 1 ``` I want to do all one to one comparsion (both variables and individuals / subjects) and keep only one if they are duplicated and name of duplicated individuals / variables to different file as log: For example in above data: Among variables: ``` varA is exactly same as varD and varF - so I will just keep varA only in new data mydf$varA == mydf$varE [1] TRUE TRUE TRUE TRUE TRUE TRUE varB and varE has exactly same data - so I will just keep varB varC is unique ``` Among Inv( i.e. subjects): ``` 1, 5 and 6 are same -> so just keep 1 ``` Thus the resulting output file is ``` mydf <- data.frame (…

Accepted Answer

This should do the trick:

```
dat <- mydf[-1]
cMat <- abs(cor(dat)) >= (1 - .Machine$double.eps^0.5)
whichKeep <- which(rowSums(lower.tri(cMat) * cMat) == 0)
cbind(mydf[1], mydf[whichKeep + 1])

Inv varA varB varC
1   1    1    1    1
2   2    1    0    0
3   3    1    1    0
4   4    0    0    0
5   5    1    1    1
6   6    1    1    1
```

how to remove correlated or duplicated variable or individuals in r

Solution

Problem