fix(controller): Always reconcile pod metadata #4477
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.


Fixes #4256
This addresses an issue we observe often: pod metadata is not consistent.
Example
Rollout is done.
A single ReplicaSet has replicas:
With the correct spec:
Yet, some pods don't have the expected labels:
Reconciliation happens and does nothing to fix this:
Why does it matter?
We have PDBs on
role=stableand the mismtach between the RS replica count and the Pods actually matching the label leads to PDB preventing any change.Root Cause
This is because the code bails if the ReplicaSet metadata matches what it should be. So if for any reason: pod update failure, crash, OOM, etc... one of the pod is not updated after the ReplicaSet is updated the first time, the controller will never try to rectify this and the inconsistent state will stay.
Also, the race condition is still present. Between the ReplicaSet update and the Pod listing. It is entirely possible that the ReplicaSet controller is reconciling on the old spec and creates a Pod with the old spec, after the Pod listing.
This ensure the logic to update pod metadata always runs, which is mostly fine efficiency-wise because the current code already doesn't patch pod in the correct state, so it would be a Pod list call at most in the standard case.
Testing
Asserting on reads doesn't seem to be a pattern and has little value. It would require 80 tests to be updated, makes more sense to ignore it.
Duplication gate is broken, tests should be ignored, but also how does removing code lead to 100% duplication 🤦

Checklist:
"fix(controller): Updates such and such. Fixes #1234".