How do I speed up text searches in R?
optimization, r
Solution
If you do need regular expressions, you can generally get a performance increase over the default regular expression engine by using the PCRE library (by setting `perl=TRUE`). There are other performance tips in `?grep`:
Performance considerations:
If you are doing a lot of regular expression matching, including on very long strings, you will want to consider the options used. Generally PCRE will be faster than the default regular expression engine, and ‘fixed = TRUE’ faster still (especially when each pattern is matched only a few times).
If you are working in a single-byte locale and have marked UTF-8 strings that are representable in that locale, convert them first as just one UTF-8 string will force all the matching to be done in Unicode, which attracts a penalty of around 3x for the default POSIX 1003.2 mode.
If you can make use of ‘useBytes = TRUE’, the strings will not be checked before matching, and the actual matching will be faster. Often byte-based matching suffices in a UTF-8 locale since byte patterns of one character never match part of another.
Problem
I have a large text vector I would like to search for a particular character or phrase. Regular expressions are taking forever. How do I search it quickly? Sample data: ``` R <- 10^7 garbage <- replicate( R, paste0(sample(c(letters[1:5]," "),10,replace=TRUE),collapse="") ) ```