blob-stream is a Kafka-like streaming system for high-throughput workloads that prioritizes total
cost of ownership over ultra-low latency. Brokers accept producer batches, persist compressed
payload segments in blob storage, and publish metadata indexes for pull-based consumers.
Delivery is at least once. Producers can retry ambiguous requests and consumers can replay records after interrupted commits or rebalances, so applications must tolerate duplicates.
- Zero cross-AZ traffic.
- Stateless brokers with zero local storage.
- Brokers can autoscale at will without downtime.
- No independent control plane or cluster manager. Producers, brokers, and consumers coordinate entirely either through K8s service discovery and the DynamoDB metadata store.
- Excellent observability via metrics, OTLP traces, and admin HTTP state snapshots.
- Producers hash record keys into logical partitions and map them to virtual partitions by
writer_id. - Producers send
ProduceBatchesrequests to the locally discovered broker selected by rendezvous hashing. - Brokers hold producer-partition leases, reserve monotonic sequence ranges, and flush compressed segment blobs with metadata indexes.
- Consumers scan metadata windows, range-read selected blob batches, and commit progress per virtual partition.
- Blob-stream requires a shared accurate clock between brokers and consumers. See the FAQ for more information. This system was designed and implemented around the AWS Time Sync Service and assumes accurate clocks.
blob-stream-broker/: broker binary and write pathblob-stream-producer/: producer client libraryblob-stream-consumer/: consumer iterator and coordinator libraryblob-stream-broker-discovery/: static and Kubernetes service discoveryblob-stream-blob-store/: blob store trait with in-memory and S3 backendsblob-stream-metadata-store/: metadata and lease stores with in-memory and DynamoDB backendsblob-stream-proto/: protobuf API and configuration schemasblob-stream-types/: shared wire and storage typesblob-stream-integration-tests/: end-to-end and deterministic fault-injection tests
- Local end-to-end walkthrough: run static-discovery brokers, interactive text producers, and consumer groups against local Docker Compose dependencies
- Dashboard references: broker, producer, and consumer CUE definitions to adapt for an observability configuration
- Kubernetes deployment reference: per-cluster Terraform for a broker writer, service discovery, and runtime configuration
- System design and correctness contract
- Infrastructure setup: configuration, DynamoDB, S3, IAM, and Kubernetes discovery RBAC
- Operations guide: runtime controls, metrics, diagnostic endpoints, and troubleshooting
- Producer and consumer integration: library entry points, configuration contracts, lifecycles, and application-mounted diagnostics
- Metrics reference: producer, consumer, and broker metric definitions
- FAQ
- Cost analysis
- Contributions must include a Developer Certificate of Origin sign-off. See DCO.
- Development guide: standalone Cargo and monorepo Bazel workflows, local dependencies, stress runner, and Rust API documentation
- Integration-test audit: deterministic test requirements and hardening work
- TLA+ model: finite model of lease, sequence, publication, and cursor safety
The protobuf configuration schema in blobstream/v1/config.proto is the canonical field-level reference for broker, producer, and consumer configuration.
For async response open GitHub issues and I will try to respond when I can. For lack of a better option right now, I've also created a #blob-stream channel in Envoy Slack. I will try to answer questions there as well.