fluent codec doesn't work with latest version of fluentd

[ggillies]

Hi,

Using logstash with a tcp input and the fluent coded, with fluentd configured to send log messages to it, actually doesn't work. It looks like the fluent message format might have been changed at some point.

This is testing with fluentd 0.12

If I modify fluent.rb decode method to look something like

  def decode(data)
    @decoder.feed_each(data) do |obj|
      tag = obj[0].to_s
      puts(obj[1].class)
      entries = obj[1]
      if entries.class == String
        puts('PackedForward')
        entry_decoder = MessagePack::Unpacker.new
        entry_decoder.feed_each(entries) do |entry|
          puts(entry)
          event = LogStash::Event.new(entry[1].merge(
            "@timestamp" => Time.at(entry[0]),
            "tags" => tag
          ))
          yield event
        end
      elsif entries.class == Array
        puts('Forward')
        entries do |entry|
          puts(entry)
          event = LogStash::Event.new(entry[1].merge(
            "@timestamp" => Time.at(entry[0]),
            "tags" => tag
          ))
          yield event
        end
      else
        puts('Message')
      end
    end
  end # def decode

Then logastash does work correctly again

[jordansissel]

Thanks for working on a patch!

To be honest, I kind of wrote this thing assuming fluentd's protocol wasn't going to change much. I don't remember much about writing it other than it worked when I wrote and tested it.

If there's interest in improving this plugin, keep in mind that we'll want backwards compatibility with the existing codec.

[mynameisrufus]

I'm trying to figure this out, are you saying that on the latest version the data is like this:

[
  "foobar",
  [
    [1437717520, { "foo" => "bar" }],
    [1437717520, { "foo" => "bar" }]
  ]
]

I can't see how this is possible here for example https://github.com/fluent/fluent-logger-ruby/blob/master/lib/fluent/logger/logger_base.rb#L25

[fluggo]

One issue is that the msgpack format has changed; there is a new code for the type STR8, and your msgpack reference is out of date.

[jalberto]

Is there any progress here?

I have similar issue, trying to send logs from docker (using docker log driver) to LS

seems like fluent codec is not usable, and this issue is 2yr old

[cosmo0920]

I've registered patches to handle EventTime msgpack extension which is used in Fluentd v0.14/v1.0.
Does anyone want to evaluate #18?