Memory leak with Fluentd 1.19.2 due to cool.io >= 1.9.2

[2ZZ]

Describe the bug

Fluentd 1.19.2 comes bundled with cool.io 1.9.3, which introduced a memory leak affecting in_forward under connection churn (clients connecting without keepalive).

Worker RSS grows linearly (~20MB/min at ~10 conn/sec) until OOM.

Downgrading to cool.io 1.9.1 fixes it. Upstream issue: socketry/cool.io#89

To Reproduce

in_forward + @type null output, send traffic without keepalive. Ruby 3.4, jemalloc, Linux.

Expected behavior

No memory leak.

Your Environment

Fluentd version: 1.19.2
Package version: gem
Operating system: Debian GNU/Linux 13 (trixie)
Kernel version: 6.12.68

Your Configuration

<source>
  @type forward
  port 24224
</source>

<match **>
  @type null
</match>

Your Error Log

N/A

Additional context

No response

[Watson1978]

I tried to reproduce the memory leak locally by generating a high volume of connection churn using fluent-logger and parallel.

However, in my environment, I could not reproduce the linear memory growth.
In fact, cool.io 1.9.3 performed better and stabilized at a lower memory footprint compared to 1.9.1 and 1.9.2.

My Benchmark Results (Running for 600 seconds)

cool.io 1.9.1	cool.io 1.9.2	cool.io 1.9.3
124.11 MB	124.14 MB	79.5 MB

Fluentd configuration

<source>
  @type forward
  tag incoming
  bind "0.0.0.0"
  port 24224
</source>

<match incoming.**>
  @type null
</match>

Client script

require 'bundler/inline'
gemfile do
  source 'https://rubygems.org'
  gem 'fluent-logger'
  gem 'parallel'
end

Parallel.each(1..16, in_processes: 16) do |worker_id|
  loop do
    logger = Fluent::Logger::FluentLogger.new(nil, host: '127.0.0.1', port: 24224)

    str = (worker_id % 26 + 65).chr
    logger.post('test', { message: str * 100 })

    logger.close
  end
end

Environment

• OS: Linux 6.19.2-1-MANJARO
• Fluentd HEAD of master branch
• ruby 4.0.1 (2026-01-13 revision e04267a14b) +PRISM [x86_64-linux]

[Watson1978]

@2ZZ I've been investigating this extensively,
but so far, I haven't been able to reproduce the linear memory growth in my environment (Linux / Ruby 4.0 / cool.io 1.9.3).

I've already tested:

• High-volume connection churn using fluent-logger.
• Race condition loops (simulating the issue TypeError: wrong argument type nil (expected Data) when processing a detached watcher socketry/cool.io#87 scenario).
• Empty TCP connection churns (simulating k8s tcpSocket probes).

In all cases, memory remained stable. To help us narrow this down, could you share a bit more detail?

• What specific tool, script, or library are you using to generate the "traffic without keepalive"? (If it's a custom script, sharing a snippet would be amazing).
• Is the client sending partial data, huge payloads, or disconnecting abruptly during the transfer?
• Do you see the exact same 20MB/min growth if you run it with standard malloc instead of jemalloc?

Any reproducing script or specific configuration details would be extremely helpful!

[Watson1978]

@2ZZ Could you also provide the internal state dump from Fluentd?
Since Fluentd has sigdump built-in, you can generate a detailed log (including GC.stat and object counts per class) by sending a SIGCONT signal to the Fluentd process.

kill -CONT <fluentd_pid>

The output will be saved to /tmp/sigdump-<pid>.log by default.

It would be ideal if you could take two dumps with a few minutes interval (e.g., when the memory has grown by another 20-40MB) and share the logs. This will help us identify exactly which Ruby objects are accumulating in the heap.

Ref. https://docs.fluentd.org/deployment/signals#sigcont

[2ZZ]

Hi @Watson1978 , thanks for looking into this.

Here is a test setup that appears to reproduce it:

reproduce.sh

#!/bin/bash
set -e

IMAGE="${1:?Usage: $0 <image>}"
VERSION=$(echo "$IMAGE" | awk -F: '{print $2}')
RECEIVER="receiver-$VERSION"
SENDER="sender-$VERSION"
NETWORK="net-$VERSION"
DURATION=600
INTERVAL=10
DIR="$(dirname "$(realpath "$0")")"

cleanup() {
  echo "[$VERSION] Cleaning up..."
  docker rm -f $RECEIVER 2>/dev/null
  docker rm -f $SENDER 2>/dev/null
  docker network rm $NETWORK 2>/dev/null
  rm -f "$DIR/sender-$VERSION.conf"
}
trap cleanup EXIT

docker network create $NETWORK 2>/dev/null || true

echo "[$VERSION] Starting receiver..."
docker run -d --name $RECEIVER \
  --network $NETWORK \
  -v "$DIR/fluent.conf:/fluentd/etc/fluent.conf" \
  $IMAGE

sleep 5

sed "s/RECEIVER_HOST/$RECEIVER/" "$DIR/sender.conf" > "$DIR/sender-$VERSION.conf"

echo "[$VERSION] Starting sender..."
docker run -d --name $SENDER \
  --network $NETWORK \
  -v "$DIR/sender-$VERSION.conf:/fluentd/etc/fluent.conf" \
  $IMAGE

sleep 3

echo "[$VERSION] Monitoring receiver memory for ${DURATION}s (every ${INTERVAL}s)..."
echo "timestamp,mem_usage,version"

START=$(date +%s)
while true; do
  NOW=$(date +%s)
  ELAPSED=$((NOW - START))
  [ $ELAPSED -ge $DURATION ] && break

  MEM=$(docker stats --no-stream --format '{{.MemUsage}}' $RECEIVER 2>/dev/null | awk '{print $1}')
  echo "$(date +%H:%M:%S),$MEM,$VERSION"
  sleep $INTERVAL
done

echo ""
echo "[$VERSION] Done."

fluent.conf (receiver)

<system>
  log_level debug
</system>

<source>
  @type forward
  port 24224
  <security>
    self_hostname fluentd-receiver
    shared_key testkey
  </security>
</source>

<match **>
  @type stdout
</match>

sender.conf

<source>
  @type sample
  tag test
  rate 100
  sample {"message": "hello world from fluentd sender for memory leak reproduction"}
</source>

<match **>
  @type forward
  keepalive false
  <server>
    host RECEIVER_HOST
    port 24224
  </server>
  <security>
    self_hostname fluentd-sender
    shared_key testkey
  </security>
  <buffer>
    flush_mode immediate
  </buffer>
</match>

Usage

Compare two versions side by side:

chmod +x reproduce.sh
./reproduce.sh docker.io/fluent/fluentd:v1.19.2-2.0 &
./reproduce.sh docker.io/fluent/fluentd:v1.19.1-2.0 &

Results

Time	v1.19.1-2.0	v1.19.2-2.0
+0s	71.04 MiB	70.98 MiB
+30s	75.63 MiB	79.66 MiB
+60s	76.04 MiB	88.69 MiB
+90s	76.33 MiB	91.00 MiB
+120s	76.43 MiB	102.60 MiB
+150s	76.66 MiB	106.20 MiB
+180s	76.91 MiB	111.50 MiB
+210s	77.09 MiB	117.00 MiB

[2ZZ]

It looks like the <security> section is required to trigger the leak — without it, both versions behave the same.

[Watson1978]

@2ZZ Thanks!! I will try that.

[Watson1978]

This issue has been fixed in cool.io v1.9.4.
For a quick workaround, please update the gem by running the following command:

fluent-gem install cool.io -v 1.9.4

We are also planning an early release of fluent-package v6.0.3, which will include this fix out of the box.