Exclude decomissioning nodes when opening new shards

[ncoiffier-celonis]

Description

Attempt to fix #6158

This PR:

• broadcast the ingester status through chitchat
• enrich the ControlPlaneModel to maintain a list of decomissioning indexer
• filter out the decomissioning nodes when opening new shards, rebalancing or scaling up shards

With this approach, even if we have some propagation delay before decomissioning, it is still possible to fail to ingest some documents if the chitchat takes longer than expected to gossip the ingester state to the control-plane.

I am wondering if this could conflict with the approach implemented here though #6163

Any feedback is welcome!!

How was this PR tested?

In addition of the unit and integration tests, I've run it against a local cluster with 2 indexer and observed that the number of errors reported in #6158 decreases from a few 100 to less than 10.

Other considerations

I also considered these 2 approaches:

• re-using the indexer state (i.e. READY/NOT_READY, by adding a DRAINING state), but an indexer needs to be ready to successfully completed the decomission process
• using the shard status itself in the decomissioning routine, but the changes were much more "spaghetti", and I couldnt quite make them working.
• using gRPC call to have the indexer call the control-plane when decomissioning doesn't seem to fit the rest of the codebase and doesn't seem to be that robust to failure and control-plane restarts.

If we want to de-riskify this change, we could put it behind a feature-flag/config property.