Large RegEx Match causing program hang

c#, parsing, regex, wpf

Solution

Does the Regex work when omitting the `RegexOptions.Compiled` flag?

Reply was yes.

So why?

It seems the Regex compiler is slow with (some?) large patterns.

It's a trade-off you have to make.

Problem

I tried asking this the other day, and admittedly did not phrase the question well or post code at first, and the answer was closed. So here I am trying again, because honestly this is driving me insane very quickly. :) I am trying to implement this Address Parser, which is originally a console-based c# program. I have successfully converted it into a standalone WPF program which consists solely of a `TextBox` for input, a `Button` to activate the parsing, and a `TextBlock` to display the results. In writing this, I did truncate the output to what I will need in my main program, and still it works fine. I have included the entire code behind for this below. My next step was to graft this into my main program, which I did by literally using copy/paste. Upon running this however, the program hangs after the button press. Eventually VS gives an error that the process has gone too long without pumping out a message, and the memory usage in TaskManager gradually increases from ~70k to 3,000,000. In response to this, I assigned the `Parsing` method to a background worker, hoping to alleviate the workload on the main process. This did solve the program freezing up, but the background thread just did the same thing, raising the RAM usage and returning nothing. So now I'm kind of at an impasse. I know that the problem is somewhere in the `var result = parser.ParseAddress(input);` statement, as when using breakpoints for every line of code this is the last one to fire. But basically I'm at a loss to understand why this would cause a problem in one WPF program and not another. I would be more than happy to post the full source code for the main program somewhere if it's necessary, but I can't imagine it would be a good idea to post ~20 different class files and projects worth of code here. :) Stand-Alone WPF App ``` namespace AddressParseWPF { /// <summary> /// Interaction logic for MainWindow.xaml /// </summary> public partial class MainWindow : Window { public MainWindow() { InitializeComponent(); } public void Execute() { AddressParser.AddressParser parser = new AddressParser.AddressParser(); var input = inputTextBox.Text; var result = parser.ParseAddress(input); if (result == null) { outputTextBlock.Text = "ERROR. Input could not be parsed."; } else { outputTextBlock.Text = (result.StreetLine + ", " + result.City + ", " + result.State + " " + result.Zip); } } private void actionButton_Click(object sender, RoutedEventArgs e) { Execute(); } } } ``` Main Program to graft Parser into ``` public void ExecuteAddressParse() { AddressParser.AddressParser parser = new AddressParser.AddressParser(); var input = inputTextBox.Text; var result = parser.ParseAddress(input); if (result == null) { outputTextBlock.Text = "ERROR. Input could not be parsed."; } else { outputTextBlock.Text = (result.StreetLine + ", " + result.City + ", " + result.State + " " + result.Zip); } } private void actionButton_Click(object sender, RoutedEventArgs e) { ExecuteAddressParse(); } ``` ParseAddress method ``` public AddressParseResult ParseAddress(string input) { if (!string.IsNullOrWhiteSpace(input)) { var match = addressRegex.Match(input.ToUpperInvariant()); if (match.Success) { var extracted = GetApplicableFields(match); return new AddressParseResult(Normalize(extracted)); } } return null; } ``` RegEx Match method ``` private static void InitializeRegex() { var suffixPattern = new Regex( string.Join( "|", new [] { string.Join("|", suffixes.Keys), string.Join("|", suffixes.Values.Distinct()) }), RegexOptions.Compiled); var statePattern = @"\b(?:" + string.Join( "|", new [] { string.Join("|", states.Keys.Select(x => Regex.Escape(x))), string.Join("|", states.Values) }) + @")\b"; var directionalPattern = string.Join( "|", new [] { string.Join("|", directionals.Keys), string.Join("|", directionals.Values), string.Join("|", directionals.Values.Select(x => Regex.Replace(x, @"(\w)", @"$1\."))) }); var zipPattern = @"\d{5}(?:-?\d{4})?"; var numberPattern = @"( ((?<NUMBER>\d+)(?<SECONDARYNUMBER>(-[0-9])|(\-?[A-Z]))(?=\b)) # Unit-attached |(?<NUMBER>\d+[\-\ ]?\d+\/\d+) # Fractional |(?<NUMBER>\d+-?\d*) # Normal Number |(?<NUMBER>[NSWE]\ ?\d+\ ?[NSWE]\ ?\d+) # Wisconsin/Illinois )"; var streetPattern = string.Format( CultureInfo.InvariantCulture, @" (?: # special case for addresses like 100 South Street (?:(?<STREET>{0})\W+ (?<SUFFIX>{1})\b) | (?:(?<PREDIRECTIONAL>{0})\W+)? (?: (?<STREET>[^,]*\d) (?:[^\w,]*(?<POSTDIRECTIONAL>{0})\b) | (?<STREET>[^,]+) (?:[^\w,]+(?<SUFFIX>{1})\b) (?:[^\w,]+(?<POSTDIRECTIONAL>{0})\b)? | (?<STREET>[^,]+?) (?:[^\w,]+(?<SUFFIX>{1})\b)? (?:[^\w,]+(?<POSTDIRECTIONAL>{0})\b)? ) ) ", directionalPattern, suffixPattern); var rangedSecondaryUnitPattern = @"(?<SECONDARYUNIT>" + string.Join("|", rangedSecondaryUnits.Keys) + @")(?![a-z])"; var rangelessSecondaryUnitPattern = @"(?<SECONDARYUNIT>" + string.Join( "|", string.Join("|", rangelessSecondaryUnits.Keys)) + @")\b"; var allSecondaryUnitPattern = string.Format( CultureInfo.InvariantCulture, @" ( (:? (?: (?:{0} \W*) | (?<SECONDARYUNIT>\#)\W* ) (?<SECONDARYNUMBER>[\w-]+) ) |{1} ),? ", rangedSecondaryUnitPattern, rangelessSecondaryUnitPattern); var cityAndStatePattern = string.Format( CultureInfo.InvariantCulture, @" (?: (?<CITY>[^\d,]+?)\W+ (?<STATE>{0}) ) ", statePattern); var placePattern = string.Format( CultureInfo.InvariantCulture, @" (?:{0}\W*)? (?:(?<ZIP>{1}))? ", cityAndStatePattern, zipPattern); var addressPattern = string.Format( CultureInfo.InvariantCulture, @" ^ # Special case for APO/FPO/DPO addresses ( [^\w\#]* (?<STREETLINE>.+?) (?<CITY>[AFD]PO)\W+ (?<STATE>A[AEP])\W+ (?<ZIP>{4}) \W* ) | # Special case for PO boxes ( \W* (?<STREETLINE>(P[\.\ ]?O[\.\ ]?\ )?BOX\ [0-9]+)\W+ {3} \W* ) | ( [^\w\#]* # skip non-word chars except # (eg unit) ( {0} )\W* {1}\W+ (?:{2}\W+)? {3} \W* # require on non-word chars at end ) $ # right up to end of string ", numberPattern, streetPattern, allSecondaryUnitPattern, placePattern, zipPattern); addressRegex = new Regex( addressPattern, RegexOptions.Compiled | RegexOptions.Singleline | RegexOptions.IgnorePatternWhitespace); } ```

Original source