Skip to content

Redistribute SMB transforms with side inputs - #5991

Draft
yolgun wants to merge 1 commit into
spotify:mainfrom
yolgun:fix/smb-side-input-rebalance
Draft

yolgun wants to merge 1 commit into
spotify:mainfrom
yolgun:fix/smb-side-input-rebalance

Conversation

@yolgun

@yolgun yolgun commented Aug 6, 2026

Copy link
Copy Markdown

Summary

  • redistribute SMB bucket offsets with Reshuffle.viaRandomKey when TransformWithSides is used
  • keep TransformWithNoSides topology unchanged
  • add pipeline topology assertions for both paths

This prevents runner forward chaining from pinning side-input SMB work to workers available when processing starts, allowing workers that join later to receive work.

Tests

  • scio-smb/test: 173 passed, 0 failed
  • scio-smb/Compile/javafmtCheck: passed
  • scio-smb/Test/javafmtCheck: passed
  • git diff --check: passed
  • repository-wide sbt test: SMB and other applicable suites passed, but the aggregate command exited non-zero for unrelated local-environment failures:
    • Annoy tests load an x86_64 native library on this arm64 machine
    • scio-snowflake/update expects a missing local Maven artifact at org/bouncycastle/bcprov-jdk15on/1.70

The PR remains in draft pending CI and Flink validation.

Co-Authored-By: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant