Removing an "empty" character item from a corpus of documents in R?

Question

I am using the `tm` and `lda` packages in R to topic model a corpus of news articles. However, I am getting a "non-character" problem represented as `""` that is messing up my topics. Here is my workflow: ``` text <- Corpus(VectorSource(d$text)) newtext <- lapply(text, tolower) sw <- c(stopwords("english"), "ahram", "online", "egypt", "egypts", "egyptian") newtext <- lapply(newtext, function(x) removePunctuation(x)) newtext <- lapply(newtext, function(x) removeWords(x, sw)) newtext <- lapply(newtext, function(x) removeNumbers(x)) newtext <- lapply(newtext, function(x) stripWhitespace(x)) d$processed <- unlist(newtext) corpus <- lexicalize(d$processed) k <- 40 result <-lda.collapsed.gibbs.sampler(corpus$documents, k, corpus$vocab, 500, .02, .05, compute.log.likelihood = TRUE, trace = 2L) ``` Unfortunately, when I train the lda model, everything looks great except the most frequently occurring word is "". I try to remedy this by removing it from the vocab as given below and reestimating…

Accepted Answer

I deal with text a lot but not tm so this is 2 approaches to get rid of the "" you have. likely the extra "" characters are because of a double space bar between sentences. You can treat this condition before or after you turn the text into a bag of words. You could replace all " "x2 with " "x1 before the strsplit or you could do it afterward (you have to unlist after strsplit).

```
x <- "I like to ride my bicycle.  Do you like to ride too?"

#TREAT BEFORE(OPTION):
a <- gsub(" +", " ", x)
strsplit(a,  " ")

#TREAT AFTER OPTION:
y <- unlist(strsplit(x, " "))
y[!y%in%""]
```

You might also try:

```
newtext <- lapply(newtext, function(x) gsub(" +", " ", x))
```

Again I don't use tm so this may not be of help but this post hadn't seen any action so I figured I'd share possibilities.

Removing an "empty" character item from a corpus of documents in R?

Solution

Problem