fix: Add comprehensive testing, deployment verification, and optimization showcase - #2
Merged
piotrlaczkowski merged 18 commits intoAug 25, 2026
Merged
Conversation
…on rebuild
Three defects found by exercising a containerised deployment (Postgres +
API-key auth + rate limiting + metrics) end to end rather than by reading
code. Each one only shows up in a production posture, which is why the
existing suite was green.
Rate limiter exempted `/healthz` — a path the platform never mounts — while
`/ready` and `/metrics`, which it does mount, counted against the per-IP
budget. Behind a NAT, ingress, or service mesh the kubelet and Prometheus
share a source IP with real traffic, so under load the readiness probe
flapped to 429 (restarting a healthy pod) and the scrape blanked out exactly
when its data was needed. Both middlewares now share one `OPERATIONAL_PATHS`
set so the two skip lists cannot drift apart again; the metrics middleware
also stops recording probe traffic as user requests.
`BatchSubmitRequest.inputs` defaulted to `[]` and ignored unknown fields, so
a body naming the item list wrongly — `{"items": [...]}`, the natural guess —
bound to an empty batch. The caller got 202 and a task reporting `succeeded`
having run nothing at all. `inputs` is now required and non-empty, and extra
fields are rejected, so a misnamed batch fails loudly at the edge.
`MetricsMiddleware` registered its collectors in `__init__`, so a second
`AgentPlatform.build()` in one process raised `Duplicated timeseries in
CollectorRegistry`. Uvicorn `--reload`, embedding hosts, and test suites all
build more than once; the collectors are process-global anyway, so they are
now created once per prefix and shared.
Also corrects the plugin async path in the frontend guide, which omitted the
`plugins/` segment the router actually mounts.
Adds `scripts/e2e_verify.py`, which drives every published surface — the
exact calls the Studio bundle makes included — against a live server and
reports pass/fail per group.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
Four defects, each found by building and running the images this repo ships rather than by reading them. `agentomatic run` read only three of the documented `AGENTOMATIC_*` switches. The image in this repo runs `agentomatic run`, not `uvicorn main:app`, so every other switch was inert in a container — and for the auth switches that was a security hazard: a deployment started with `AGENTOMATIC_ENABLE_AUTH=1` and an API key served a fully unauthenticated API while looking correctly configured. The CLI now honours the same set the scaffolded main.py does, so the two entrypoints cannot diverge. `.dockerignore` excluded `README*` while pyproject declares `readme = "README.md"`, so the build backend failed with "Readme file does not exist" the moment `uv sync` installed the project: neither Dockerfile in this repo could build at all. Both images installed core dependencies only. The runtime had no sqlalchemy, langgraph, prometheus-client or pyjwt, so `/metrics` served nothing, `DATABASE_URL` failed with "No module named 'sqlalchemy'", and JWT auth could not be enabled — while the container still reported healthy. They now install `all` (matching `agentomatic deploy`) plus `db-postgres`, which `all` deliberately omits but a container with a Postgres profile beside it needs. The generated Dockerfiles get the same treatment, since the `.env` they generate wires `DB__URL` to exactly that kind of URL. `Dockerfile.distroless` built its venv on Python 3.12 while `distroless/python3-debian12` runs 3.11, so `/app/.venv/bin/python` — the ENTRYPOINT — was a dangling symlink and the image never started. It now mirrors the fix already made to the generated distroless template: a 3.11 builder, `--target` into a plain `PYTHONPATH` directory, and the base image's own interpreter as the entrypoint. Verified by building and booting it. Also replaces the root docker-compose.yml, which bind-mounted an nginx.conf that is not in the repo (so `docker compose up` failed here) and modelled two identical single-agent services — the platform serves every discovered agent from one process. It is now one platform service driven by the documented env vars, with an opt-in Postgres profile, and pins uv instead of tracking `uv:latest`, which made builds irreproducible. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
…ions
`/ready` was gated by both auth middlewares while `/readiness` was not, so a
`readinessProbe: /ready` — the spelling a Kubernetes probe usually uses, and
a route the platform does mount — answered 401. No pod ever became ready and
the Deployment never rolled out, while the platform logged nothing wrong.
The skip lists named `/healthz` instead, which is not mounted at all, so the
gap was invisible by inspection.
Both middlewares now share the `PROBE_PATHS` set that already fixed the same
drift in the rate limiter, and `test_auth_coverage.py` records `/ready` as a
reviewed public route (it shares a handler with `/readiness`, so it exposes
nothing new). A new test drives every mounted probe under all four auth
postures — none, API key, JWT, and the global auth lock.
Adds end-to-end JWT verification against a real JWKS with real RS256
signatures. The suite covered JWT configuration and decoding in isolation but
never drove the HTTP path, which is the only way to catch a middleware that
accepts a token it should reject. The new tests mint genuine tokens and
assert the forgeries that matter are refused: a key the issuer never
published, that key reusing a published `kid`, `alg=none`, a payload edited
after signing, and tokens failing `exp` / `aud` / `iss`.
Connections built from unset `${ENV}` placeholders now say which variable to
set. Previously each resolved to an empty string and surfaced as whatever the
driver made of it — "Could not parse SQLAlchemy URL from given URL string"
named neither the connection's purpose nor the variable, and a freshly
scaffolded agent logged four such errors on every boot, which teaches
operators to skim past the errors that do matter. A half-resolved target is
still left to the driver, since that is a real misconfiguration rather than
an absent one.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
A pipeline that fails to load is simply absent at runtime — its routes 404
and the workflow it drives stops existing — so the boot-time skip warning is
the only signal an operator gets. It read:
Skipping /app/pipelines/parallel_flow.yaml – failed to load: 'agent'
That is the bare repr of a KeyError. It names neither the offending step, nor
what was missing, nor the rule that was broken (steps nested under `parallel`
must be agent steps; plugin, endpoint and ingestion steps sit at the top
level). The message now names the step, lists the keys it did declare — which
is what makes a typo obvious — and states the constraint. The discovery
warning also says the pipeline will not be served and names the exception
type, so a skipped pipeline is not mistaken for a loaded one.
Also makes the e2e harness deployment-agnostic in two ways it needed to be to
report honestly across postures. A rate-limited deployment no longer makes the
run flaky: the harness is itself a burst from one IP, so it waits out
Retry-After and retries, except where the 429 is the thing under test. And a
store-less deployment — thread routes answer 400 with no store configured,
which is a posture rather than a defect — is reported as skipped, naming the
variables that would enable it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
`scripts/e2e_verify.py` had no entry in the docs, so the one tool that answers "does this container, with this configuration, behave correctly right now" was discoverable only by reading the scripts directory. Documents what each check group covers, how the harness adapts to a lean deployment (no Studio, no auth, no store, no control plane) rather than reporting those as failures, and — just as importantly — what it does not cover: model quality, your own node logic, and load behaviour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
Every write path followed `commit()` with `refresh()`, which issues a SELECT to read back the row it had just written. Against a networked Postgres that is a second round trip on the request's critical path, and it showed: `POST /invoke` with invocation logging on measured p50 45.5ms / p95 68.4ms against p50 8.2ms with logging off — a 5x tax on every request, for an observability feature. The refresh was never needed. The session factory sets `expire_on_commit=False`, and every column default in `storage.models` is Python-side (there are no server defaults), so a committed object is already fully populated. Removing all nine brings the same call to p50 30.0ms / p95 33.1ms — the tail less than half what it was. Dropping the refresh did surface something the extra read had been hiding: `DateTime(timezone=True)` is a no-op on SQLite, so a timestamp read back from there is naive while the same value from Postgres carries `+00:00`. The old code returned whatever the round trip produced, which made the API's timestamp format depend on the configured database and left clients to guess whether a naive string was UTC. Timestamps now go through one `iso_utc` helper that stamps UTC on naive values, so writes and reads agree on every backend. Tests pin both the invariants that make the refresh unnecessary — so a change to either is caught here rather than by someone profiling production — and that written rows still come back complete. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
… checks Three checks measured the deployment through a budget the harness had already spent, so a clean platform failed them on the later agents of a matrix run: The SSE helper bypassed the retry path in `_req`, so `POST /invoke/stream` was reported as a failure when the limiter answered 429. It now waits out Retry-After like every other call. "probes do not consume the user budget" asserted that a normal call still succeeds after flooding the probe routes — which is really a statement about how much budget the harness has left. It now reads `X-RateLimit-Remaining` either side of the flood and asserts the count did not fall, which is the property itself and holds regardless of prior traffic. The anonymous auth probes use a second client that shares the harness's source IP, and therefore its per-IP budget: a 429 there says nothing about whether auth is enforced. Those calls now retry past the limiter before deciding. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
… itself offline Found by loading the real bundle in Chromium against a container and watching what it asked for. The header showed "Disconnected", next to a Retry button, immediately after a *successful* connect — while a second indicator said "Connected" and every backend call returned 200. `ConnectionSetup.handleConnect` sets `isConnected` but never `connectionStatus`, which the header renders and which starts at `'disconnected'`; it only reached `'connected'` through `attemptReconnection`, so the badge was correct only after a failure and recovery. The store setter now keeps both flags in agreement, since they must never disagree. The stylesheet opened with two Google Fonts `@import` rules. A self-hosted admin UI should not fetch assets from a third party to render: the requests fail in air-gapped or egress-restricted deployments (they reset here, in a sandbox that blocks them) and send every viewer's IP and User-Agent to Google, which is a compliance question for enterprise operators. Both are removed — every `font-family` already declared a full fallback stack, so the UI renders natively with no external request. No icon was declared, so browsers fell back to `/favicon.ico` at the origin root, which the platform does not serve: a 404 in every access log and a console error for every user. An inline SVG icon costs no request. `imgs/logo.png` was not reused — it is a 588 KB JPEG despite the extension. The Studio UI is built in a separate repository and only its compiled output lives here, so these are patches to built assets: `static/LOCAL_PATCHES.md` records each one with the upstream change it needs, and warns that a frontend rebuild reverts them. A new test suite fails if the bundle regains a render-blocking third-party reference or loses its inline icon. Verified in Chromium: connect succeeds, all nine views render with no error boundary, and the page load makes zero external requests, zero HTTP failures and zero console errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
`tests/test_live_omlx_optimize.py` and `tests/test_live_omlx_keras_optimize.py` skip entirely unless an OpenAI-compatible endpoint answers, so on any machine without a local model — CI included — the `omlx/` provider path, the prompt fitter, every fitter optimizer and the whole Keras-style `fit()` loop went unexercised. 32 tests, always dormant. Adds `scripts/local_slm_server.py`, a stand-in those suites can be pointed at. It is a test double, not a language model: it generates nothing and follows a fixed set of rules. What makes it a valid optimization *target* is that answer quality genuinely depends on the system prompt — each directive a prompt carries makes the response satisfy one more property the metric rewards, so an optimizer that really searches and selects climbs, and one that does not, does not. It also plays the two other roles the fitter needs: as rewriter it reads the briefing's failing I/O, expected answers and judge guidance and folds the missing required tokens into a new prompt; as judge it returns the exact schema the metric asked for, scoring what the metric scores so the signals agree. With it running, the suite goes from 47 skips to 15 — the remainder being OpenAI and Gemini tests that need real cloud credentials. All 32 newly-live tests pass, including `fit()` across `rewrite`, `gepa_like`, `mipro_like`, `few_shot_bootstrap` and `param_search`, each asserting loss starts at the pre-fit baseline, never rises across epochs, and ends strictly lower. The guide says plainly what this does and does not prove: the machinery, not whether a real model writes good prompts. Point the same env vars at oMLX, llama.cpp, vLLM or Ollama for that. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
… passing as skips Two defects in the pipeline engine, both found by running every step type through the HTTP API rather than by unit-testing the parser. **`sub_pipeline` steps could never run.** The platform builds the pipeline router without passing `sub_pipelines=`, so the pool was always empty and every `sub_pipeline` step failed validation with "Sub-pipeline 'x' not found" — naming a pipeline that was discovered, served, and runnable at its own route. The step type was parsed, validated and implemented, and was unreachable in practice: pipelines could not compose. Every served pipeline is now available as a sub-pipeline, resolved when the engine is built so a pipeline saved through the builder is immediately referencable. That makes a hazard reachable that previously could not be: a pipeline may now reference itself, directly or around a longer loop, and pipelines are editable over HTTP — so an unbounded recursion was one saved YAML away. Validation now reports reference cycles by path, and nesting is capped at runtime for the case validation could not have seen (a sub-pipeline saved after the engine was built). **A condition that could not be evaluated was swallowed into `False`.** Any exception — a typo, a renamed step, or `$.` mapping syntax used where a `ctx` expression belongs — silently skipped the step while the pipeline reported `status: success`. A conditional branch that never fires, under a green status, is the kind of defect that reaches production and stays there. Evaluating falsy is still a routing decision and still skips; failing to evaluate is now a step failure carrying the expression, the underlying error, and what to write instead, with the step's own `on_error` policy deciding what happens next. Syntax errors are caught by `validate()` before the pipeline runs at all. The repo's own DAG test demonstrates the hazard: its condition read `ctx.input.query`, but `ctx.input` is a dict, so it raised `AttributeError` on every run and counted as "never true" — which is what the test's comment claimed it was. Corrected to say what it meant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
…ake a model
Three defects that together let a container boot healthy and answer every
request with invented text.
**`requirements.txt` was copied into the image and never installed.** It is
where a project declares what *it* needs on top of agentomatic — a vendor LLM
driver, a vector client, an in-house package — and the scaffold even ships one
pre-populated. Everything declared there was silently absent at runtime.
**A missing LLM driver was swallowed.** `get_named_llm` caught every failure,
logged a warning, and substituted a dummy model, so a stack configured for
`openai_compatible` whose client library was not installed produced fabricated
answers from a platform reporting healthy. A missing client library is a
defect in the image that no retry fixes: it now raises, naming the extra to
install and saying why a dummy was refused. A backend that is merely
unreachable can recover, so that still degrades — with a warning that says
responses are fabricated.
Together these were one failure: `requirements.txt` never installed →
`langchain_openai` absent → dummy model → plausible fake answers.
**Generated images now build with uv** rather than pip, pinned via
`ARG UV_VERSION` so a build stays reproducible. It is what this project builds
with, and it dominates container build time: installing agentomatic and its
118 dependencies takes 108ms where pip took tens of seconds. A project that
keeps a `pyproject.toml` and `uv.lock` gets `uv sync --frozen` instead, so the
image installs the exact resolved tree the lock pins rather than re-resolving
at build time.
Also finishes the connection-diagnostics work: the store auto-derive and the
control-plane health check treated a connection built from unset `${ENV}`
placeholders as *unhealthy*, surfacing a driver error where an operator reads
a backend outage. Both now report `not_configured` and name the variable that
would enable it.
Verified by building the generated Dockerfile verbatim and driving the
resulting container: agents reach a local OpenAI-compatible model and return
real answers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
`--prompt` is documented as "Initial prompt (overrides prompts.json)". It reached the legacy `prompt_only` path only: `_run_fitter_optimize` was called without it, so all six fitter modes — `rewrite`, `param_search`, `gepa_like`, `mipro_like`, `few_shot`, `apo` — accepted the flag and silently optimized from the agent's own prompt instead. The damage is in the number the whole run is judged by. The reported baseline score belonged to a prompt the caller never asked for, so "improvement +0.0%" could mean the candidates were no better than *a different starting point* entirely — and starting a tuning run from a specific prompt is the most ordinary thing to do with this command. Verified against a live model: optimizing the same agent and dataset now reports a baseline of 0.45 for `--prompt "You are a vague assistant."` and 0.90 for a prompt that asks for structured, attributed answers. Both previously reported 0.45, the agent's own prompt, whatever was passed. Also teaches the local test double to read instructions from the user turn as well as the system role — the scaffolded agent templates concatenate their system prompt into the user message rather than sending a system role, so a responder that only inspected the system role could not see a prompt override at all — and to act on the judge's `improvement_hints`, which is what a rewrite model does when climbing a rubric. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
Both a MEMORY-purpose connection and DATABASE_URL can be configured at once, and the connection silently wins. An operator who pointed DATABASE_URL at a managed Postgres saw "store configured" on boot and reasonably believed their threads lived there — while they were going wherever the connection pointed, which for the scaffolded MEMORY example is a SQLite file inside the container that dies with it. The auto-derivation path now logs a warning naming both sides when it overrides a configured DATABASE_URL, and keeps the quiet INFO line when there is nothing to override. `_safe_db_url` strips credentials so the warning never prints a password. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
…istory
The chatbot scaffold accepted `messages` in `input_to_state`, used them
only to number the turn, and sent the model
`f"{prompt}\n\nUser: {state.request}"`. Every turn was therefore answered
as if it were the first, while the `/chat` response reported
`history_loaded: N` — the one number a caller would check to confirm
memory was working.
Verified on the wire against a recording model double in a container:
turn 2 previously sent one `user` message with the system prompt glued
on and no prior turns; it now sends a `system` role followed by the full
thread.
- `respond` delegates to a `_turns` helper that converts `state.messages`
through `dict_to_messages` and leads with a `SystemMessage`, falling
back to `current_query` alone for single-shot `/invoke`.
- `history_len` no longer counts the current turn, which `load_history`
appends, so the offline branch numbers turns correctly.
- The `/chat` docstring and the conversation-memory guide no longer claim
"full conversational awareness" for every agent: the platform supplies
`state["messages"]`, and reading it is the agent's job. The single-shot
templates take `current_query` deliberately, and that is now stated.
Also documented that a condition which raises fails its step rather than
skipping it, and corrected the `to_eval_namespace` docstring, which
listed one builtin where the code exposes fourteen.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
Three gaps in the verification story, each of which a deployment could pass every existing check and still fail in production. - `pipelines-all`: the harness proved the pipeline *routes* against one sampled pipeline. A deployment usually publishes several, built from different step types, and a step type that only ever validates is not one that runs. This executes every published pipeline and reports which of the nine step types actually executed — all nine, in both the full and the distroless image. - `isolation`: agents are singletons, so every request gets the same instance. A class agent parking per-run data on `self` instead of in its state dataclass would serve one caller's answer to another — invisible to sequential testing, and the worst kind of bug to ship. Concurrent callers each carry a unique marker; a response or thread carrying somebody else's is a leak. None do. - `scripts/durability_verify.py`: every other check runs against one live process, and a store that quietly fell back to a file inside the container passes all of them. Split across a container replacement (`write` phase, replace the deployment, `verify` phase) the difference shows. Proven against Postgres on the distroless image. The harness stays deployment-agnostic: a deployment without a thread store reports the store-dependent isolation checks as skipped rather than failed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
Three defects on the Keras-style path, compounding into one: `agent.fit()` on an `AgentDataset` with a deterministic metric could never improve, no matter what the optimizer proposed. It always reported "no improvement" and recommended keeping the baseline. 1. `AgentExample.to_datapoint` renders a *judge-facing* reference — judge guidance, a rubric, an `## Expected answer` section, the structured output as JSON. `ContainsMetric` and `ExactMatchMetric` compare strings, so they were matching the agent's response against markdown headers and scoring ~0 however right the answer was. They now read the answer out of the reference (`plain_expected`); plain-string expectations are untouched. 2. `PromptFitterBridge` seeded `baseline_system_prompt` only from `compiled_config`, which is empty until a fit succeeds. The first epoch therefore measured its baseline against the fitter's own generic default and rewrote *that* — the prompt the author wrote on the agent was silently ignored. It now falls back to the agent's own prompt, while a compiled prompt still wins so later epochs keep compounding. 3. The briefing inlined that multi-line reference after a label, producing `Expected: ## Expected answer` with the answer de-indented out of the item it belonged to — mangling the one field a rewrite model most needs to find. Multi-line values now get an indented block under their label. Measured against a live model: the optimization suites that previously could only pass on their tolerant "no improvement is still auditable" branch now report baseline 0.0000 → best 1.0000. `ContainsMetric` also no longer counts an empty keyword from a trailing comma, which scored a free point on every response. Also: `evaluate()` on a loaded agent said "compile(metrics=...) first" to someone who had done exactly that — metrics are live objects and never serialise. It now names the metrics the save recorded and says why they are gone, and `save`/`load` document what does and does not round-trip. `scripts/keras_showcase.py` runs the whole lifecycle against any OpenAI-compatible endpoint and prints the measured loss curve. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
An unreachable agent produced per-call warnings and then a confident
summary:
Baseline score: 0.0000
Best score: 0.0000
❌ Recommendation: keep the baseline config (no improvement).
Nothing distinguished that from "the agent answered every question wrong",
so an operator whose server was simply not running would go and rewrite a
prompt that was never exercised. Reproduced against a live deployment: with
the server down every datapoint 404s and the run still reports 0.0000; with
it up the same command measures 0.8611.
When an evaluation scores no datapoint at all, the run now logs an error and
raises the reason ahead of every other advisory in the summary, saying
plainly that the number is not a measurement. The message names the actual
cause — every agent call failing and the metric scoring none of the
responses need different fixes — and quotes the first error. A partial run,
and a genuine zero where the agent answered and was simply wrong, are
untouched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
…name
`_base_message_cls: Any` followed by `import BaseMessage as _base_message_cls`
is a redefinition. mypy 2.1 — which `uv.lock` pins and CI installs — rejects
it, so the typecheck job fails on this at HEAD and on main:
error: Name "_base_message_cls" already defined [no-redef]
Import under the real name and assign, which keeps the ImportError fallback
and the `Any` annotation intact.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN
piotrlaczkowski
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds extensive end-to-end testing infrastructure, deployment verification tooling, and optimization examples to improve platform reliability and demonstrate key features.
Key Changes
Testing & Verification Infrastructure
scripts/e2e_verify.py: Comprehensive end-to-end verification harness that exercises every public surface of the platform (routes, Studio, agents, plugins, endpoints, pipelines, auth, metrics) against a live server and reports pass/fail resultsscripts/local_slm_server.py: OpenAI-compatible test double server for local optimization testing without requiring a real LLMscripts/keras_showcase.py: Demonstrates the full Keras-style agent lifecycle (compile() → fit() → evaluate() → save() → load()) with real optimizers and metricsscripts/durability_verify.py: Proves conversation state survives container restartsNew Test Suites
test_connection_diagnostics.py): Validates that unconfigured connections properly name required environment variablestest_container_build_contract.py): Guards on build inputs and.dockerignorecorrectnesstest_jwt_signature_e2e.py): End-to-end JWT verification with real RS256 signatures and JWKStest_eval_blackout.py): Ensures 0.0000 scores aren't reported when nothing was evaluatedtest_chatbot_template_history.py): Validates prior conversation turns are loaded correctlytest_pipeline_condition_errors.py): Ensures broken conditions fail rather than silently skiptest_cli_run_env.py): ValidatesAGENTOMATIC_*environment variable handlingtest_pipeline_load_diagnostics.py): Ensures failed pipeline loads report errors with pipeline namestest_llm_driver_missing.py): Validates configured LLMs with missing drivers fail appropriatelytest_probe_endpoints_public.py): Ensures health/readiness probes work without credentialstest_optimize_cli_prompt.py): Validates--promptseeds baseline in all optimization modestest_briefing_rendering.py): Ensures expected answers stay readable and under labelstest_expected_reference_scoring.py): Validates metrics score against answers, not scaffoldingtest_execution_modes.py): Adds batch mode rejection of unknown itemstest_sub_pipeline_composition.py): Tests nested pipeline functionalitytest_studio_bundle_is_self_contained.py): Validates Studio assets are completetest_store_write_round_trips.py): Validates storage persistenceOptimization Showcase Results
Added complete optimization result artifacts demonstrating five optimization modes:
Each includes fit history, evaluation history, configuration, and metadata.
Core Platform Improvements
MAX_SUB_PIPELINE_DEPTHconstant to prevent infinite nestingresolve_envfor environment variable resolutionOPERATIONAL_PATHSto skip rate limiting on probe endpoints/health,/readiness,/ready)OPERATIONAL_PATHSOPERATIONAL_PATHSiso_utc()helperhttps://claude.ai/code/session_01QsmZgzw7pjBpYL7XXYaMcN