How to find recurring word groups in text with C#?

c#, regex, text

Solution

I think that this works fairly well.

var text = @"The green algae (singular: green alga) are ..."; // include all your text

var remove = "().,:[]0123456789".Select(x => x.ToString()).ToArray();

var words =
    Regex
        .Matches(text, @"(\S+)")
        .Cast<Match>()
        .SelectMany(x => x.Captures.Cast<Capture>())
        .Select(x => remove.Aggregate(x.Value, (t, r) => t.Replace(r, "")))
        .Select(x => x.Trim().ToLowerInvariant())
        .Where(x => !String.IsNullOrWhiteSpace(x))
        .ToArray();

var groups =
    from n1 in Enumerable.Range(0, words.Length)
    from n2 in Enumerable.Range(1, words.Length - n1)
    select String.Join(" ", words.Skip(n1).Take(n2));

var frequencies =
    groups
        .GroupBy(x => x)
        .Select(x => new { wordgroup = x.Key, count = x.Count() })
        .OrderByDescending(x => x.count)
        .ThenBy(x => x.wordgroup.Count(y => y == ' '))
        .ThenBy(x => x.wordgroup)
        .ToArray();

This gives me the frequency of every single word grouping of contiguous sequences of words including up to a single word group of all the words.

The number of words is 288. The total number of word groups is `288 x (288 + 1) / 2 = 41,616`. The final number of word groups (after grouping duplicate word groups and removing empty/whitespace strings) is 41,449.

Here are the first 100 of these 41,449:

20 x "the", 13 x "and", 12 x "algae", 12 x "in", 11 x "green", 10 x "of", 9 x "green algae", 8 x "are", 6 x "as", 6 x "species", 5 x "a", 4 x "is", 4 x "or", 4 x "to", 3 x "embryophytes", 3 x "form", 3 x "found", 3 x "lichens", 3 x "live", 3 x "on", 3 x "plants", 3 x "that", 3 x "algae and", 3 x "and in", 3 x "as the", 3 x "in the", 3 x "of the", 2 x "alga", 2 x "can", 2 x "clade", 2 x "class", 2 x "colonial", 2 x "filamentous", 2 x "from", 2 x "higher", 2 x "macroscopic", 2 x "most", 2 x "other", 2 x "seaweeds", 2 x "their", 2 x "trentepohlia", 2 x "while", 2 x "with", 2 x "algae are", 2 x "are a", 2 x "green alga", 2 x "higher plants", 2 x "in lichens", 2 x "of green", 2 x "species of", 2 x "the clade", 2 x "the green", 2 x "green algae and", 2 x "green algae are", 2 x "of green algae", 2 x "species of green", 2 x "the green algae", 2 x "species of green algae", 1 x "about", 1 x "acquired", 1 x "algal", 1 x "also", 1 x "associations", 1 x "bark", 1 x "be", 1 x "both", 1 x "cannot", 1 x "cell", 1 x "cells", 1 x "cellular", 1 x "charales", 1 x "charophyte", 1 x "charophytes", 1 x "chlorarachniophytes", 1 x "chlorophyte", 1 x "chloroplasts", 1 x "ciliate", 1 x "closest", 1 x "coccoid", 1 x "coenobia", 1 x "colonies", 1 x "conduct", 1 x "consisting", 1 x "differentiated", 1 x "differentiation", 1 x "divisions", 1 x "emerged", 1 x "euglenids", 1 x "excluded", 1 x "family", 1 x "few", 1 x "filaments", 1 x "flagella", 1 x "flagellates", 1 x "flatworms", 1 x "for", 1 x "forms", 1 x "full", 1 x "fungal", 1 x "fungi"

Problem

I'm getting recurring word counts in StringBuilder(sb) with this code which i've found on internet and according to writer it's really consistent like Word's word counter. ``` StringBuilder wordBuffer = new StringBuilder(); int wordCount = 0; // 1. Build the list of words used. Consider ''' (apostrophe) and '-' (hyphen) a word continuation character. Dictionary<string, int> wordList = new Dictionary<string, int>(); foreach (char c in sb.ToString()) { if (char.IsLetter(c) || c == '\'' || c == '-') { wordBuffer.Append(char.ToLower(c)); } else { if (wordBuffer.Length > 3) { int count = 0; string word = wordBuffer.ToString(); wordList.TryGetValue(word, out count); wordList[word] = ++count; wordBuffer.Clear(); wordCount++; } } } ``` This is my sample text: The green algae (singular: green alga) are a large, informal grouping of algae consisting of the Chlorophyte and Charophyte algae, which are now placed in separate Divisions. The land plants or Embryophytes (higher plants) are thought to have emerged from the Charophytes.[1] As the embryophytes are not algae, and are therefore excluded, green algae are a paraphyletic group. However, the clade that includes both green algae and embryophytes is monophyletic and is referred to as the clade Viridiplantae and as the kingdom Plantae. The green algae include unicellular and colonial flagellates, most with two flagella per cell, as well as various colonial, coccoid and filamentous forms, and macroscopic, multicellular seaweeds. In the Charales, the closest relatives of higher plants, full cellular differentiation of tissues occurs. There are about 8,000 species of green algae.[2] Many species live most of their lives as single cells, while other species form coenobia (colonies), long filaments, or highly differentiated macroscopic seaweeds. A few other organisms rely on green algae to conduct photosynthesis for them. The chloroplasts in euglenids and chlorarachniophytes were acquired from ingested green algae,[1] and in the latter retain a nucleomorph (vestigial nucleus). Green algae are also found symbiotically in the ciliate Paramecium, and in Hydra viridissima and in flatworms. Some species of green algae, particularly of genera Trebouxia of the class Trebouxiophyceae and Trentepohlia (class Ulvophyceae), can be found in symbiotic associations with fungi to form lichens. In general the fungal species that partner in lichens cannot live on their own, while the algal species is often found living in nature without the fungus. Trentepohlia is a filamentous green alga that can live independently on humid soil, rocks or tree bark or form the photosymbiont in lichens of the family Graphidaceae. With my sample text, I'm getting green and algae words in the first lines as expected. Problem is, I don't need only single words, I need word groups too. With this example text, I want green algae words too, together with green and algae words. And my optional problem is: I need to do it with high performance, because texts can be very long. As i researched it's not high performance to use RegEx with this case, but I'm not sure about if there is a second way to make it possible. Thanks in advance. UPDATE If you got what I'm asking about, you don't need to read these lines. As I see too many comments about my "group" definiton is not clear, I think I need to state my point with more detail and I wished write these lines on comments section but it's a little narrow area for this update. Firstly, I know StackOverflow is not a coding service. I'm trying to find the most used word groups in an article and trying to decide what's article about, we can call it tag generator too. For this purpose I tried to find most used words and it was okay at the beginning. Then i realized it's not a good way to decide about topic because I can't assume the article is about only first or second word. In my example I can't say this article is only about green or algae because they mean something together here, not alone. If i try this with an article about a three named celebrity like "Helena Bonham Carter" (if I assume it's written full name along article, not only surname), I want to take these words together not one by one. I'm trying to achieve more clever algorithm which is guessing the topic in most accurate way and with one shot. I don't want to limit the word count because article may be about "United Nations Industrial Development Organization" (again I assume it's now written like "UNIDO" in article). And I can achieve this by trying to get every word group starting from any index to the end of text with any length. Okay it's not a good way really, especially with long texts but it's not impossible right? But i was looking for a better way to do this and I just asked about a better algorithm idea and best tool to use, I can write the code by myself. I hope I stated my goal clear finally.

Original source