Repository navigation
fix(extproc): don't serve before the first filter config is loaded - #2992
Open
abdulaziz-dev-lab wants to merge 1 commit into
Open
abdulaziz-dev-lab wants to merge 1 commit into
abdulaziz-dev-lab wants to merge 1 commit into
Conversation
The config bundle watcher tolerates two errors on the initial load and retries on the next tick: a part that doesn't match the index checksum, and a listed part that is missing. Both happen when a pod starts while the bundle is being rewritten. The external processor then served with no config. Check reported SERVING, so the readiness probe passed and the pod received traffic. The first request dereferenced the nil config, and the panic killed the process. Envoy failed closed with an empty 500, and the sidecar crash-looped until the bundle was consistent again and the back-off expired. Until the first config is loaded: - Check and List report NOT_SERVING, so the readiness probe keeps the pod out of rotation. - Streams for a registered path get an immediate 503 with a JSON error body and gRPC status Unavailable. Before, they crashed the process. An unregistered path still gets its 404. - The MCP proxy answers 503. Before, it panicked in its handlers and dropped the connection. Health checks now read the config while the watcher stores it, so Server.config becomes an atomic.Pointer. The MCP proxy sets a loaded flag after storing its config. New tests start the external processor on an incomplete bundle and check behaviour before the first load at four levels: the server, the MCP proxy, the extproc binary's /health, and end to end through Envoy (data-plane and data-plane-mcp). In each, requests are refused until the bundle is repaired, and then served once the watcher loads it. A pod that starts on a bundle that stays inconsistent now stays NotReady, which also takes its non-AI routes out of rotation. Before, it served them until the first AI request crashed the external processor. Signed-off-by: Abdulaziz Alharbi <calamitygaming64@gmail.com>
✅ Deploy Preview for theagentrouter canceled.
|
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
When a pod starts while the config bundle is being rewritten, the first load can fail on a checksum mismatch or a missing part, and the watcher retries on every tick. Meanwhile the extproc reported SERVING, and the first AI request crashed it on the nil config. The sidecar then crash-looped and clients got empty 500s.
Until the first config is loaded:
CheckandListreport NOT_SERVING, so the pod stays out of rotation.Unavailable). Unknown paths still get 404.Server.configis now anatomic.Pointer, since health checks read it concurrently.Related Issues/PRs (if applicable)
Fixes #2991
Special notes for reviewers (if applicable)
atomic.Pointer) and feat: protectedResourceMetadata.resource from the request instead of requiring it to be configured #2691 (MCP handler). Whichever lands second rebases.