[VOTE] Add invalidated_fragments to IndexMetadata

[westonpace]

There is a discussion thread here and a PR here.

The basic problem occurs when an existing indexed column is modified (on one or more fragments). Previously we removed those fragments from the index's fragment bitmap. However, the index might still be searched if it contains other valid (unmodified) fragments. The results must be masked out to block the stale results referring to invalidated fragments.

One approach could be to always apply an allow-list for the fragment bitmap of the index. However, this would add considerable cost to the happy path (indexed column is not updated after index is created) because we would need to apply this allow list on every search.

A simpler approach is to apply the block list to fragments removed from the fragment bitmap, only if fragments have actually been removed. To do this we need to know which fragments were removed from the index.

[jackye1995]

+1

[BubbleCal]

+1

[sinianluoye]

+1

[Xuanwo]

+1 binding