Should I put a Kafka/Redis buffer between Fluent Bit and OpenSearch when shipping logs?

Short answer: yes, a buffer is the right move — but the choice between a Redis list and Kafka comes down to the guarantee level you’re willing to accept.

What you’re hitting is classic backpressure: once OpenSearch’s indexing rate drops below your production rate, the chain clogs backwards until the app container’s stdout blocks. The fix is a durable layer that decouples the producer (your app) from the consumer (OpenSearch).

1. Turn on Fluent Bit’s own disk buffer first. With storage.type filesystem and storage.total_limit_size on the output, Fluent Bit spools to disk instead of RAM when OpenSearch slows down, so no pressure reaches the app. Measure this before adding an external queue — it’s often all you need.
2. Redis list: cheap but risky. In-memory; RPUSH/LPOP is simple, but with AOF off you lose data on a restart and a growing list eats Redis’s RAM. Fine for logs you don’t mind losing, not for financial/audit logs.
3. Kafka: 40k/s is its home turf. Disk-based, with retention, replayable, horizontal consumption via consumer groups. It gives you at-least-once naturally. The cost is operational — you have to stand up and feed a cluster. If the volume is permanent, it’s the right tool.
4. Make at-least-once idempotent at the consumer. When writing to OpenSearch, give each document a deterministic _id (e.g. a hash of source+offset). A message delivered twice overwrites the same _id instead of creating a duplicate.

Bottom line: start with Fluent Bit filesystem buffering plus bulk tuning and ISM (rollover) on the OpenSearch side. If that’s not enough, move to Kafka — keep Redis for logs you can afford to lose. And the golden rule: never let the app container block while logging. Logging should be fire-and-forget; don’t carry critical data (payments, audit) over the log pipeline — run that on a separate, guaranteed path.