have a grok filter create nested fields as a result

logstash, logstash-grok, syslog

Solution

Yes, this is expected. I don't think there's a way to produce nested fields with grok. I suspect you'll have to use the mutate filter to move them into place.

mutate {
    rename => {
      "date" => "[drupal][date]"
      "instigator" => "[drupal][instigator]"
      ...
    }
  }

If you have a lot of fields it might be more convenient to use a ruby filter. This is especially true if you prefix Drupal fields with e.g. "drupal." – then you'd write a filter to move all fields with that prefix into a subfield with the same name.

Problem

I have a drupal watchdog syslog file that I want to parse into essentially two nested fields, the syslog part and the message part so that I get this result ``` syslogpart: { timestamp: "", host: "", ... }, messagepart:{ parsedfield1: "", parsedfield2: "", ... } ``` I tried making a custom pattern that looks like this: `DRUPALSYSLOG (%{SYSLOGTIMESTAMP:date} %{SYSLOGHOST:logsource} %{WORD:program}: %{URL:domain}\|%{EPOCH:epoch}\|%{WORD:instigator}\|%{IP:ip}\|%{URL:referrer}\|%{URL:request}\|(?<user_id>\d+)\|\|)` and then run `match => ['message', '%{DRUPALSYSLOG:drupal}'}` but I don't get a nested response, I get a textblock `drupal: "ALL THE MATCHING FIELDS IN ONE STRING"` and then all the matches separately as well but not nested under drupal but rather on the same level.

Original source