Logstash multiline codec for Celery stacktraces

celery, logstash, python

Solution

After some more Googling I found the answer, apparently the multiline codec doesn't play nice with the stdin input. Using the file input, and now it works.

See: https://logstash.jira.com/browse/LOGSTASH-1629

Problem

Trying to create a configuration for logstash that correctly parses Celery's log format, including multiline stacktraces. An example of logging output might look like this: ``` [2014-04-15 15:11:27,350: DEBUG/Worker-4] Doing some work. [2014-04-15 15:11:27,362: ERROR/MainProcess] Task core.tasks.sometask[92dc34bd-8139-4f98-b359-d78caf68381d] raised unexpected: ValueError('Foobar',) Traceback (most recent call last): File "./venvs/backend-33/lib/python3.4/site-packages/celery/app/trace.py", line 238, in trace_task R = retval = fun(*args, **kwargs) File "./venvs/backend-33/lib/python3.4/site-packages/celery/app/trace.py", line 416, in __protected_call__ return self.run(*args, **kwargs) File "util.py", line 151, in wrapper rv = func(self, *args, **kwargs) File "tasks.py", line 104, in do_something_useful raise ValueError('Foobar') ValueError: Foobar ``` At some point I got Grok ready to parse the single line output, but the multiline is a problem. This configuration: ``` input { stdin { codec => multiline { 'negate' => true 'pattern' => '^\[' 'what' => 'previous' } } } output { stdout { codec => rubydebug } } ``` Is taken more or less straight from the Logstash documentation for the multiline codec, when I run it, it produces no output at all. I tried using a multiline filter instead, but it yielded the same result, I also tried running logstash with --verbose but it didn't give any useful information. What's going on here?

Original source