ngaut/benthos — explained in plain English
Analysis updated 2026-07-18 · repo last pushed 2018-08-15
Clean up, filter, or route messages flowing through Kafka, HTTP, or RabbitMQ pipelines.
Connect data from one service like Kafka to another like Elasticsearch without writing custom code.
Deduplicate a stream of data, such as Twitter data, before writing it into Kafka.
Monitor stream health by sending metrics to Statsd or Prometheus.
| ngaut/benthos | 42wim/fabio | 42wim/go-xmpp | |
|---|---|---|---|
| Language | Go | Go | Go |
| Last pushed | 2018-08-15 | 2018-02-04 | 2020-01-24 |
| Maintenance | Dormant | Dormant | Dormant |
| Setup difficulty | moderate | moderate | moderate |
| Complexity | 3/5 | 3/5 | 3/5 |
| Audience | ops devops | ops devops | developer |
Figures from each repo's GitHub metadata at analysis time.
Configured entirely through YAML files, stronger delivery guarantees require adding an optional buffer layer.
Benthos is a tool for moving and transforming data streams between different services. Think of it as a reliable middleman that sits between your data sources and destinations, reading messages from one place, optionally modifying them, and sending them somewhere else, all while being rock-solid about not losing anything. The typical use case: you have messages flowing through your system (maybe from Kafka, an HTTP endpoint, or RabbitMQ), and you need to clean them up, filter them, or route them to multiple places. Benthos handles all that plumbing. It connects to a wide variety of popular services, Kafka, AWS S3, Redis, Elasticsearch, MQTT, and many others, so you can plug it into almost any existing pipeline. You write a simple configuration file describing where data comes from, what to do with it, and where it should go, then Benthos runs that configuration reliably. What makes Benthos stand out is resilience. By default, it's built to handle crashes gracefully. If your application restarts, Benthos won't lose messages, it ensures everything gets delivered at least once, even without extra infrastructure. If you need even stronger guarantees, you can add a buffer layer. You also get monitoring built in, with the ability to send metrics to Statsd or Prometheus so you can track what's happening in your stream. The project is straightforward to deploy. You can run it as a standalone binary, drop it in a Docker container, or use it as a framework if you're building custom stream processing in Go. Configuration is done through YAML files that you can parameterize with environment variables, making it easy to adjust settings for different environments without changing code. The README includes concrete examples and even a cookbook section with real-world use cases like deduplicating Twitter data into Kafka.
A reliable data-streaming tool that reads, transforms, and routes messages between services like Kafka, S3, and Redis without losing data.
Mainly Go. The stack also includes Go, Kafka, YAML.
Dormant — no commits in 2+ years (last push 2018-08-15).
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly ops devops.
This repo across BitVibe Labs
double-check against the repo, no cap.