When are n-grams (n>3) important as opposed to just bigrams or trigrams?
data-mining, n-gram, nlp, nltk
Solution
I'm not familiar with a good deal of the tags listed here, however n-grams (the abstract concept) are often useful related to statistical models. As a result, here's some applications which aren't restricted merely to bigrams and trigrams:
- Compression algorithms (the PPM variety especially) where the length of the grams depends on how much data is available for providing specific contexts.
- Approximate string matching (e.g. BLAST for genetic sequence matching)
- Predictive models (e.g. name generators)
- Speech recognition (phonemes grams are used to help evaluate the likelihood of possibilities for the current phoneme undergoing recognition)
Those are the ones off the top of my head, but there's much more listed on Wikipedia.
As far as "state-of-the-art" n-gram extraction, no idea. N-gram "extraction" is an adhoc attempt to speed up certain processes while still maintaining the benefits of n-gram style modeling. In short, "state-of-the-art" depends on what you're trying to do. If you're looking at fuzzy matching or fuzzy grouping, it depends on what kind of data you're matching/grouping. (E.g. street addresses are going to be very different to fuzzy match than first names.)
Problem
I am just wondering what is the use of n-grams (n>3) (and their occurrence frequency) considering the computational overhead in computing them. Are there any applications where bigrams or trigrams are simply not enough? If so, what is the state-of-the-art in n-gram extraction? Any suggestions? I am aware of the following: - A new method of n-gram statistics for large number of n and automatic extraction of words and phrases from large text data of Japanese - Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus - Word association norms, mutual information, and lexicography - Retrieving collocations from text: Xtract