Include monotonically increasing value in logstash field?

logstash

Solution

here's nothing built into logstash to do it.

You can build a filter to do it pretty easily

Just drop something like this into lib/logstash/filters/seq.rb

# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"
require "set"
#
# This filter will adds a sequence number to a log entry
#
# The config looks like this:
#
#     filter {
#       seq {
#         field => "seq"
#       }
#     }
#
# The `field` is the field you want added to the event.
class LogStash::Filters::Seq < LogStash::Filters::Base

  config_name "seq"
  milestone 1

  config :field, :validate => :string, :required => false, :default => "seq"

  public
  def register
    # Nothing
  end # def register

  public
  def initialize(config = {})
    super

    @threadsafe = false

    # This filter needs to keep state.
    @seq=1
  end # def initialize

  public
  def filter(event)
    return unless filter?(event)
    event[@field] = @seq
    @seq = @seq + 1
    filter_matched(event)
  end # def filter
end # class LogStash::Filters::Seq

This will start at 1 every time Logstash is restarted, but for most situations, this would be ok. If you need something that is persistent across restarts, you need to do a bit more work to persist it somewhere

Problem

I know there's no built in "line count" functionality while processing files through logstash (for various, understandable and documented reasons). But - there should be a mechanism, within any given logstash instance - to have an monotonically increasing variable / count for every parsed line. I don't want to go the metrics route since it's a continuous polling mechanism (every n-seconds). Alternatives include pre-processing of log files which given my particular use case - is unacceptable. Again, let me reiterate - I need the ability to generate/read a monotonically increasing variable that I can store during in a logstash filter. Thoughts?

Original source