Bug Report
Describe the bug
in_kubernetes_events crashes with a SIGSEGV every few hours while reading the Kubernetes watch stream.
The crash is in process_http_chunk(), which calls strpbrk(c->resp.payload, "\r\n") without checking resp.payload first. By the time it runs, that pointer can be pointing at memory that has already been freed.
Why it happens: after a watch chunk has been fully read, resp.payload_size is set to 0 but resp.payload is left pointing at the old buffer. The next read grows the buffer with flb_http_buffer_increase(), and the realloc inside it can move the buffer and free the old one. The code that updates resp.payload to the new location only runs when payload_size is greater than 0 (flb_http_client.c:1225 and :1245), so when it is 0 the pointer never gets updated and is left pointing at the freed memory. process_http_chunk() then reads through that stale pointer and crashes.
To Reproduce
- Rubular link if applicable: N/A
- Example log message if applicable:
[engine] caught signal (SIGSEGV)
#0 process_http_chunk() at plugins/in_kubernetes_events/kubernetes_events.c:752
#1 k8s_events_collect() at plugins/in_kubernetes_events/kubernetes_events.c:920
#2 co_init() at lib/monkey/deps/flb_libco/amd64.c:117
- Steps to reproduce the problem:
- Run in_kubernetes_events against a Kubernetes API server and leave it watching events for a long time.
- The crash happens after a watch chunk is fully read (so payload_size becomes 0) and the next read grows the buffer and moves it in memory, which leaves resp.payload pointing at freed memory. It depends on timing, so in practice it shows up about once every few hours per pod.
- You can also hit it on demand: call process_http_chunk() with an empty response (resp.payload == NULL, payload_size == 0), which is the state a stream starts in. Running that under AddressSanitizer reports a heap-use-after-free read in process_http_chunk, pointing back at the realloc in flb_http_buffer_increase.
Expected behavior
in_kubernetes_events processes the watch stream without crashing; an empty or already-consumed response is a no-op, not a segfault.
Screenshots
N/A
Your Environment
Version used: 5.0.9 and 5.1.0 (also present on current master)
Configuration:
[INPUT]
Name kubernetes_events
Tag k8s_events
DB /var/log/flb_kube_k8s_events.db
interval_sec 30
storage.type filesystem
Kube_Request_Limit 200
Additional context
We've seen this happen on and off across several production Kubernetes clusters. Each time, the fluent-bit-k8s-events pod dies (exit 134) and stops shipping Kubernetes events until it restarts. It looks like the same problem reported in #9543 ("Kubernetes Events Input Segfault"), which was closed without a fix.
Bug Report
Describe the bug
in_kubernetes_events crashes with a SIGSEGV every few hours while reading the Kubernetes watch stream.
The crash is in process_http_chunk(), which calls strpbrk(c->resp.payload, "\r\n") without checking resp.payload first. By the time it runs, that pointer can be pointing at memory that has already been freed.
Why it happens: after a watch chunk has been fully read, resp.payload_size is set to 0 but resp.payload is left pointing at the old buffer. The next read grows the buffer with flb_http_buffer_increase(), and the realloc inside it can move the buffer and free the old one. The code that updates resp.payload to the new location only runs when payload_size is greater than 0 (flb_http_client.c:1225 and :1245), so when it is 0 the pointer never gets updated and is left pointing at the freed memory. process_http_chunk() then reads through that stale pointer and crashes.
To Reproduce
Expected behavior
in_kubernetes_events processes the watch stream without crashing; an empty or already-consumed response is a no-op, not a segfault.
Screenshots
N/A
Your Environment
Version used: 5.0.9 and 5.1.0 (also present on current master)
Configuration:
[INPUT]
Name kubernetes_events
Tag k8s_events
DB /var/log/flb_kube_k8s_events.db
interval_sec 30
storage.type filesystem
Kube_Request_Limit 200
Additional context
We've seen this happen on and off across several production Kubernetes clusters. Each time, the fluent-bit-k8s-events pod dies (exit 134) and stops shipping Kubernetes events until it restarts. It looks like the same problem reported in #9543 ("Kubernetes Events Input Segfault"), which was closed without a fix.