engine_dispatch: task is never released when flb_input_chunk_flush() fails, pinning its task-map slot and input chunk until restart #12252

Bug Report

Summary

flb_engine_dispatch_retry() destroys a retry without releasing its task when flb_input_chunk_flush() returns NULL. The task survives with no users and no retries, and nothing in the engine ever reaps a task in that state, so it holds its task-map slot and keeps its input chunk busy until the process restarts. I confirmed the leak by fault injection, and the fix below re-schedules the retry instead of dropping it, matching the failure case a few lines above. I am happy to send it as a PR.

What goes wrong

flowchart TD
    A["flb_engine_dispatch_retry()"] --> B["flb_input_chunk_flush()"]
    B -->|"returns content"| C["task is flushed and released normally"]
    B -->|"returns NULL"| D["flb_task_retry_destroy(retry)"]
    D --> E["return -1"]
    E --> F["task survives: users == 0, retries list empty"]
    F --> G["task-map slot held until restart"]
    F --> H["input chunk stays busy until restart"]

• The leaked task holds a slot in config->task_map, so the map fills up over time.
• The leaked task keeps its input chunk marked busy, so the chunk is never re-dispatched.
• No further retry is ever scheduled for that chunk.
• No error, drop, or retry-failure counter is incremented, so nothing in metrics or logs reports it.
• Once enough of these accumulate to exhaust the task map, the pipeline stops creating new tasks for that output while inputs keep ingesting normally.
• The buffered data itself stays intact, so a restart flushes the whole backlog with zero drops.

The only visible symptom is fluentbit_input_storage_chunks_busy rising and never coming back down.

Affected code

https://github.com/fluent/fluent-bit/blob/v5.1.0/src/flb_engine_dispatch.c#L66-L73

    /* There is a match, get the buffer */
    buf_data = (char *) flb_input_chunk_flush(task->ic, &buf_size);
    if (!buf_data) {
        /* Could not retrieve chunk content */
        flb_error("[engine_dispatch] could not retrieve chunk content, removing retry");
        flb_task_retry_destroy(retry);
        return -1;
    }

• flb_task_retry_destroy() unlinks the retry from task->retries and frees it, but leaves the task itself behind.
• The sibling path handling the same situation does check before returning, so the convention already exists in the codebase. See https://github.com/fluent/fluent-bit/blob/v5.1.0/src/flb_task.c#L296-L306

    if (seconds == -1) {
        flb_warn("[task] retry for task %i could not be re-scheduled", task->id);
        flb_task_retry_destroy(retry);
        if (task->users == 0 && mk_list_size(&task->retries) == 0) {
            flb_task_destroy(task, FLB_TRUE);
        }
        return -1;
    }

flb_input_chunk_flush() can return NULL on three paths, none of which is limited to allocation failure:

• cio_chunk_up() failure. See https://github.com/fluent/fluent-bit/blob/v5.1.0/src/flb_input_chunk.c#L3686-L3691
• cio_chunk_get_content() failure. See https://github.com/fluent/fluent-bit/blob/v5.1.0/src/flb_input_chunk.c#L3706-L3711
• Empty content. See https://github.com/fluent/fluent-bit/blob/v5.1.0/src/flb_input_chunk.c#L3713-L3716

Versions affected

• The branch was introduced in 7f771e9 ("engine_dispatch: always validate if a 'retry' is possible", fluent-bit crashes with SIGSEGV in template_execute() #1734, 2019-11-19) and first released in v1.4.0.
• It has been unchanged since, and is byte-identical in v3.2.3, v5.1.0 (fe293d4270b27bc078e0bb82e7aa2afe08e723f8) and current master (a1d6fb1ba85e84f160c3fcc948aff26157e3a6dc).

To Reproduce

The trigger is an I/O failure that is hard to induce on demand, so I confirmed the leak by fault injection with gdb. The injection forces buf_data to NULL exactly 20 times, and then the pipeline is allowed to recover fully.

1. Run the official debug image, which ships gdb and debug symbols.

docker run -d --name flb --cap-add NET_ADMIN --cap-add SYS_PTRACE \
  -v $PWD/flb.conf:/flb.conf:ro -v flbstore:/flb-storage \
  --entrypoint /fluent-bit/bin/fluent-bit \
  fluent/fluent-bit:5.1.0-debug -c /flb.conf

flb.conf uses 8 inputs and 8 forward outputs pointing at a reachable fluentd with TLS enabled. Any output that produces retries works. I used forward because Require_ack_response makes every flush a full round trip.

[SERVICE]
    flush                     0.2
    log_level                 info
    storage.path              /flb-storage/
    storage.sync              normal
    storage.checksum          off
    storage.metrics           on
    http_server               on
    http_listen               0.0.0.0
    http_port                 2020

# repeat for tags g1..g8
[INPUT]
    name         dummy
    tag          g1
    rate         500
    storage.type filesystem

# repeat for tags g1..g8
[OUTPUT]
    name                      forward
    match                     g1
    host                      <fluentd host>
    port                      24224
    tls                       on
    tls.verify                off
    Require_ack_response      true
    net.io_timeout            15s
    net.keepalive_max_recycle 50
    Compress                  gzip
    Retry_Limit               no_limits
    storage.total_limit_size  10G

2. Make the destination unreachable so retries start piling up. Blocking only the inbound direction keeps established connections half-open, which is what I used.

docker exec flb iptables -A INPUT -p tcp -s <fluentd ip> --sport 24224 -j DROP

3. Once retries are flowing, inject 20 failures with this force.gdb.

set pagination off
set confirm off
set $n = 0
break flb_engine_dispatch.c:68
commands
  silent
  if $n < 20
    set var buf_data = 0
    set $n = $n + 1
    printf "FORCED_NULL %d\n", $n
  end
  continue
end
continue

docker exec -d flb sh -c 'gdb -p 1 -x /force.gdb -batch > /force.out 2>&1'
# wait until 20 hits, then detach
docker exec flb sh -c 'grep -c FORCED_NULL /force.out; pkill gdb'

After this step, [engine_dispatch] could not retrieve chunk content, removing retry appears exactly 20 times in the log.

4. Restore connectivity and let the pipeline drain completely.

docker exec flb iptables -F INPUT

5. Watch the busy chunk gauge. Note that it lives on /api/v2/, not /api/v1/.

curl -s localhost:2020/api/v2/metrics/prometheus \
  | grep '^fluentbit_input_storage_chunks_busy{' | awk '{s+=$2} END {print s}'

Observed

Sampled every 30s for 8.5 minutes after connectivity was restored:

06:47:27 busy=85 proc=1288712 retries=1261 errors=0 dropped=0
06:47:57 busy=20 proc=1417756 retries=1261 errors=0 dropped=0
06:48:27 busy=28 proc=1535571 retries=1261 errors=0 dropped=0
06:48:57 busy=20 proc=1658311 retries=1261 errors=0 dropped=0
06:49:28 busy=20 proc=1777015 retries=1261 errors=0 dropped=0
06:49:58 busy=28 proc=1895868 retries=1261 errors=0 dropped=0
06:50:28 busy=20 proc=2014425 retries=1261 errors=0 dropped=0
06:50:58 busy=20 proc=2133245 retries=1261 errors=0 dropped=0
06:51:28 busy=20 proc=2251765 retries=1261 errors=0 dropped=0
06:51:58 busy=20 proc=2370247 retries=1261 errors=0 dropped=0
06:52:29 busy=28 proc=2492865 retries=1261 errors=0 dropped=0
06:52:59 busy=20 proc=2611053 retries=1261 errors=0 dropped=0
06:53:29 busy=20 proc=2729892 retries=1261 errors=0 dropped=0
06:53:59 busy=28 proc=2848541 retries=1261 errors=0 dropped=0
06:54:29 busy=20 proc=2967085 retries=1261 errors=0 dropped=0
06:54:59 busy=28 proc=3089602 retries=1261 errors=0 dropped=0
06:55:30 busy=20 proc=3207533 retries=1261 errors=0 dropped=0
06:56:00 busy=20 proc=3326101 retries=1261 errors=0 dropped=0

• The floor is exactly 20, matching the 20 injected failures. busy oscillates between 20 and 28, and the 8 above the floor are the in-flight chunks of the 8 inputs.
• The pipeline is completely healthy throughout. proc advanced by roughly 2M records while those 20 chunks were never released.
• retries, errors and dropped do not move at all.
• A control run without fault injection ran 12 consecutive block and unblock cycles on the same setup, and every cycle returned to busy == 0. The floor of 20 is therefore attributable to the injected failures rather than to the backpressure itself.
• Restarting the container with the buffer volume preserved flushes all 20 with dropped == 0.

Expected behavior

The task should be released when destroying its last retry leaves it with no users and no retries, matching flb_task_retry_reschedule().

Proposed fix

Re-schedule the retry instead of dropping it, which is what the failure case a few lines above already does:

--- a/src/flb_engine_dispatch.c
+++ b/src/flb_engine_dispatch.c
@@ -66,10 +66,21 @@ int flb_engine_dispatch_retry(struct flb_task_retry *retry,
     /* There is a match, get the buffer */
     buf_data = (char *) flb_input_chunk_flush(task->ic, &buf_size);
     if (!buf_data) {
-        /* Could not retrieve chunk content */
-        flb_error("[engine_dispatch] could not retrieve chunk content, removing retry");
-        flb_task_retry_destroy(retry);
-        return -1;
+        /*
+         * Destroying the retry here would leave the task with no users and no
+         * retries, a state nothing reaps: it would hold its task-map slot and
+         * keep the chunk busy until a restart. Re-schedule like the case above.
+         */
+        flb_error("[engine_dispatch] could not retrieve chunk content, "
+                  "re-scheduling retry");
+
+        ret = flb_task_retry_reschedule(retry, config);
+        if (ret == -1) {
+            return -1;
+        }
+
+        /* Just return because it has been re-scheduled */
+        return 0;
     }

• The chunk stays on disk, so no data is discarded. This matters because the current leak, bad as it is, at least preserves the records for a restart to pick up.
• The task keeps a pending retry, so it is no longer in the unreachable state.
• flb_task_retry_reschedule() already handles the terminal case internally: if the scheduler cannot take the request it destroys the retry and releases the task.
• Returning 0 lets flb_sched_event_handler() destroy the old scheduler request, matching how the branch above is handled.

I first considered simply calling flb_task_users_release() after flb_task_retry_destroy(). That is wrong: it reaches flb_task_destroy(task, FLB_TRUE), whose del argument makes cio_chunk_close() delete the chunk file, so it would trade the leak for permanent data loss.

Two caveats worth your judgement:

• A chunk that can never be read will now retry forever under Retry_Limit no_limits, with backoff capped at FLB_SCHED_CAP. The branch above has the same property today, so this is consistent rather than new, but it is a behaviour change for this path.
• flb_input_chunk_flush() also returns NULL for genuinely empty content, and re-scheduling that would not terminate. I could not construct a case where a task chunk becomes empty, since a task is created from a chunk with records, but separating that case would require changing the function's return contract and felt out of scope here.

How this was found

I hit this while investigating a production stall, and the details may be useful for judging severity.

• An out_forward output stopped delivering while its inputs kept ingesting normally.
• A network authorization rule had blocked the destination for about an hour. Most processes recovered on their own once it was lifted.
• A few processes never resumed until the container was restarted, at which point the entire backlog flushed with zero drops.
• Throughout the stall, output_proc_records_total and output_retries_total were completely flat.
• output_errors_total, output_retries_failed_total and output_dropped_records_total all stayed at zero.
• input_storage_chunks_busy sat pinned at a constant equal to the task map size.

I could not confirm that this bug is what caused that particular stall. Its trigger is an I/O failure whose log line never appeared in the production logs I have. The bug is real and reachable and produces exactly that signature, so I am reporting it on its own merits rather than as a root cause.

Additional context

Two things made this hard to notice, and they may be worth addressing separately.

• The task map grows silently. flb_config_task_map_grow() logs nothing when it grows or when it reaches FLB_CONFIG_DEFAULT_TASK_MAP_SIZE_LIMIT, so exhaustion never shows up in logs. See https://github.com/fluent/fluent-bit/blob/v5.1.0/src/flb_config.c#L1642-L1650
• fluentbit_input_storage_chunks_busy is exposed only on /api/v2/metrics/prometheus and not on /api/v1/, so a scraper pointed at the v1 endpoint cannot see the one signal that reveals this.

Your Environment

• Version used: v5.1.0, with v3.2.3 and master a1d6fb1ba also reviewed
• Configuration: see above
• Environment name and version: Docker, official fluent/fluent-bit:5.1.0-debug image
• Server type and version: fluentd 1.18 as the forward destination
• Operating System and version: Linux, arm64
• Filters and plugins: in_dummy, out_forward, filesystem storage