Skip to content

Add support for pulsar 5.x and scalable topics - #152

Draft
efcasado wants to merge 10 commits into
mainfrom
pulsar-5
Draft

Add support for pulsar 5.x and scalable topics#152
efcasado wants to merge 10 commits into
mainfrom
pulsar-5

Conversation

@efcasado

@efcasado efcasado commented Jun 26, 2026

Copy link
Copy Markdown
Owner

https://pulsar.apache.org/blog/2026/06/23/announcing-apache-pulsar-5-0-m1/

  • Extended wire protocol
  • New abstractions (e.g. topology watchers)
  • Scalable Consumers
    • Stream
    • Queue
    • Checkpoint
  • Scalable Producers
  • End of topic support
  • Cumulative ACKing support

Stream abstraction feels like a step backwards? Whilst it preserves ordering across a changing number of segments (~partitions), we cannot spin up an arbitrary number of parallel consumers using a Key-Shared subscription. That is, orderly parallel consumption is now much more limited than before. The number of parallel consumers is now dictated by the number of segments (~partitions) in the topic.

The scalable producer is somewhat disappointing. Given that the broker knows about the DAG topology and is responsible for coordinating segments, etc. I would have expected for the producer flow to be as simple as sending a keyed message to the broker and let the broker handle it internally. It turns out routing has to be implemented in the client.

@efcasado
efcasado force-pushed the pulsar-5 branch 3 times, most recently from ffc2a6c to fd5a133 Compare June 27, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant