Fluentd does not flush any data to elastic search - but does flush upon shutdown
elasticsearch, fluentd
Solution
After much troubleshooting I found that there was a bug in the specific version of ruby installed that caused a thread to deadlock when starting up. The issue did not appear when shutting down as it was being called a differently. This is a specific bug re ruby 2.0.0-p353 ; the next release ruby 2.0.0-p451 fixes the issue.
https://groups.google.com/forum/#!msg/fluentd/t2uKfttiYCw/P8zy5kZpEIIJ
Problem
I have a fluentd + elastic search + kibana setup. Fluentd is configured to output to elastic search with this plugin: https://github.com/uken/fluent-plugin-elasticsearch . The problem is that fluentd will never flush its buffers to elastic search whilst it is running, it just stores the data in the memory buffer. Upon issuing a shutdown the buffer is flushed and elastic search is updated. This issue occurs despite setting `flush_interval 60s` and `buffer_chunk_limit 25k` . Default values also give the same problem. Fluentd collector log file output and cfg: ``` 2014-07-11 11:45:44 +0100 [info]: fluent/supervisor.rb:202:supervise: starting fluentd-0.10.48 2014-07-11 11:45:44 +0100 [info]: fluent/supervisor.rb:304:read_config: reading config file path="/etc/fluent/fluent.conf" 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered buffer plugin 'file' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered buffer plugin 'memory' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'debug_agent' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'exec' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'forward' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'gc_stat' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'http' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'monitor_agent' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'object_space' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'status' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'tcp' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'unix' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'syslog' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered input plugin 'tail' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'copy' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'exec' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'exec_filter' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'file' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'forward' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'null' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'roundrobin' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'stdout' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'tcp' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'unix' 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'test' 2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:87:block in configure: gem 'fluent-plugin-elasticsearch' version '0.3.0' 2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:87:block in configure: gem 'fluentd' version '0.10.48' 2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:91:configure: using configuration file: <ROOT> <source> type forward port 24224 bind 0.0.0.0 </source> <source> type syslog port 5544 protocol_type tcp bind 0.0.0.0 tag index.syslog </source> <match index.**> type elasticsearch logstash_format true host localhost port 9200 include_tag_key true tag_key tag </match> </ROOT> 2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:101:block in configure: adding source type="forward" 2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:101:block in configure: adding source type="syslog" 2014-07-11 11:45:44 +0100 [info]: fluent/engine.rb:117:block in configure: adding match pattern="index.**" type="elasticsearch" 2014-07-11 11:45:44 +0100 [trace]: fluent/plugin.rb:72:register_impl: registered output plugin 'elasticsearch' 2014-07-11 11:45:44 +0100 [info]: plugin/in_forward.rb:75:listen: listening fluent socket on 0.0.0.0:24224 2014-07-11 11:45:45 +0100 [debug]: plugin/in_syslog.rb:183:listen: listening syslog socket on 0.0.0.0:5544 with tcp 2014-07-11 11:45:45 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57119': object_id=70254255494700 2014-07-11 11:45:45 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254255494700 2014-07-11 11:45:46 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57120': object_id=70254256323480 2014-07-11 11:45:46 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254256323480 2014-07-11 11:45:47 +0100 [trace]: plugin/in_forward.rb:189:initialize: accepted fluent socket from '10.18.80.20:57121': object_id=70254256319280 2014-07-11 11:45:47 +0100 [trace]: plugin/in_forward.rb:245:on_close: closed fluent socket object_id=70254256319280 ....... ``` Could anyone help explain why fluentd is not flushing to elastic search? Thanks.