Skip to content

About

Blob stream

Resources

Code of conduct

Stars

55 stars

Watchers

1 watching

Forks

Latest commit

 

History

166 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

blob-stream

blob-stream

blob-stream is a Kafka-like streaming system for high-throughput workloads that prioritizes total cost of ownership over ultra-low latency. Brokers accept producer batches, persist compressed payload segments in blob storage, and publish metadata indexes for pull-based consumers.

Delivery is at least once. Producers can retry ambiguous requests and consumers can replay records after interrupted commits or rebalances, so applications must tolerate duplicates.

Goals

  • Zero cross-AZ traffic.
  • Stateless brokers with zero local storage.
  • Brokers can autoscale at will without downtime.
  • No independent control plane or cluster manager. Producers, brokers, and consumers coordinate entirely either through K8s service discovery and the DynamoDB metadata store.
  • Excellent observability via metrics, OTLP traces, and admin HTTP state snapshots.

System At A Glance

  • Producers hash record keys into logical partitions and map them to virtual partitions by writer_id.
  • Producers send ProduceBatches requests to the locally discovered broker selected by rendezvous hashing.
  • Brokers hold producer-partition leases, reserve monotonic sequence ranges, and flush compressed segment blobs with metadata indexes.
  • Consumers scan metadata windows, range-read selected blob batches, and commit progress per virtual partition.

Requirements

  • Blob-stream requires a shared accurate clock between brokers and consumers. See the FAQ for more information. This system was designed and implemented around the AWS Time Sync Service and assumes accurate clocks.

Project Layout

  • blob-stream-broker/: broker binary and write path
  • blob-stream-producer/: producer client library
  • blob-stream-consumer/: consumer iterator and coordinator library
  • blob-stream-broker-discovery/: static and Kubernetes service discovery
  • blob-stream-blob-store/: blob store trait with in-memory and S3 backends
  • blob-stream-metadata-store/: metadata and lease stores with in-memory and DynamoDB backends
  • blob-stream-proto/: protobuf API and configuration schemas
  • blob-stream-types/: shared wire and storage types
  • blob-stream-integration-tests/: end-to-end and deterministic fault-injection tests

Documentation

Deploy And Operate

Contribute And Verify

  • Contributions must include a Developer Certificate of Origin sign-off. See DCO.
  • Development guide: standalone Cargo and monorepo Bazel workflows, local dependencies, stress runner, and Rust API documentation
  • Integration-test audit: deterministic test requirements and hardening work
  • TLA+ model: finite model of lease, sequence, publication, and cursor safety

The protobuf configuration schema in blobstream/v1/config.proto is the canonical field-level reference for broker, producer, and consumer configuration.

Contact

For async response open GitHub issues and I will try to respond when I can. For lack of a better option right now, I've also created a #blob-stream channel in Envoy Slack. I will try to answer questions there as well.

About

Blob stream

Resources

Code of conduct

Stars

55 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages