Multiple Grok Filters not storing first filter match record

logstash, logstash-grok

Solution

You need to protect the 2nd grok block -- ie don't execute it if the first one succeeds.

if ("BOUNCED" not in [tags]) {
  grok {
    patterns_dir => "patterns"
    match => [
        "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}"            
    ]
    add_tag => ["INTIALIZATION"]
    remove_tag => ["_grokparsefailure"]
    named_captures_only => true
  }
}

Problem

I am using Logstash to parse postfix logs. I am mainly focused to get bounced email logs from postfix logs, and store it in database. In order to get logs, first I need to find ID generated by postfix corresponding to my message-id, and using that Id, I need to find status of an email. For following configuation, I am able to get the logs. ``` grok { patterns_dir => "patterns" match => [ "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}", "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}" ] named_captures_only => true } ``` I am using following if condition to store logs that match the patterns: ``` if "_grokparsefailure" not in [tags] { #database call } ``` As you have seen, I am using two patterns to find corresponding two different logs from one log file. Now, I want to differentiate both pattern based on tags. So I have modified my configuration as follows: ``` grok { patterns_dir => "patterns" match => [ "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}" ] add_tag => ["BOUNCED"] remove_tag => ["_grokparsefailure"] named_captures_only => true } grok { patterns_dir => "patterns" match => [ "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}" ] add_tag => ["INTIALIZATION"] remove_tag => ["_grokparsefailure"] named_captures_only => true } ``` Now, it only store %{POSTFIXCLEANUP} pattern logs. If I reverse the order, it only store %{POSTFIXBOUNCE} pattern. so, after removing that if condition, I found that message being parsed from first filter have "_grokparsefailure" tag and first filter tag, and because of that it is not storing that record. Can anybody tell me what need to be done to rectify this? Am I am making any mistake?

Original source