docsy: make examples test runner async (concurrent runs)

[ppiegaze]

What

Makes test/test_runner.py run example scripts concurrently instead of sequentially, for both the cloud path (run_tests) and the local path (run_tests_local). Addresses DOC-1098 ("make the test runner async so it spawns all test runs at once and receives results as they complete; clean up the runner generally").

How

• The runner spawns every test at once via a concurrent.futures.ThreadPoolExecutor and collects results with as_completed, so results are handled the moment each run lands.
• Why threads, not asyncio: test submission is a blocking subprocess call (uv run <script> for cloud, flyte run --local for local) — there is no async client to await. A thread pool lets all runs wait on their cloud/local executions in parallel (subprocess I/O releases the GIL).
• A new --concurrency N flag (default 8, also settable via TestConfig.concurrency / config JSON) caps in-flight runs so the cloud backend isn't overwhelmed. --concurrency 1 restores the old sequential behavior.

Contract preserved (no behavior change to what the CI workflow sees)

Confirmed against .github/workflows/test-examples.yml → make {test,test-local,test-preview} → python test/test_runner.py [--local] [--preview] [--file|--filter] [--verbose]:

• Same CLI/entrypoint and flags (only an optional --concurrency added).
• Same per-test timeout, same pass/fail rules (including the "Flyte failure in output despite exit 0" override), same TestResult model.
• Same log files (test/reports/logs/<script>.log, _local suffix for local) and same test_report.json / historical_results.json / index.html reports.
• Same exit-code semantics (sys.exit(1) on any failure/timeout).

Cleanup included

• run_single_test / run_single_test_local are now pure workers (run → return TestResult); all console + log output happens in the single-threaded completion loop, so concurrent runs don't interleave output. Non-passing tests print their captured stdout/stderr so failures stay diagnosable in the Actions log.
• Removed the os.environ["FLYTECTL_CONFIG"] global mutation (unsafe under threads) — the config path is passed explicitly via each subprocess env.
• Bugfix (latent, now load-bearing): isolated local venvs were named by script stem, so two scripts sharing a filename (e.g. multiple main.py) collided on the same venv dir — a race under concurrency. Now named by full relative path.
• Parse the Flyte config once per run instead of once per test; replaced the FakeResult hack with a plain returncode override; removed the unused tempfile import; deterministic (sorted) result ordering.

Verification

Done without a live Flyte backend:

• python -m py_compile clean; --help shows all original args + --concurrency; --preview --filter hello discovers and lists scripts correctly.
• Concurrency machinery tested with a stubbed worker: 6 tests × 0.5s ran in 0.51s concurrently vs 3.02s with --concurrency 1; per-test result attribution correct (no cross-wiring); log files written with correct names/content; deterministic ordering; sequential fallback works.
• Verified the venv-name fix yields unique names for same-stem scripts.

Needs a live backend to confirm (QA)

• An actual concurrent cloud run (make test) against the Flyte backend — verify real concurrent submissions, correct per-run result attribution, execution-URL extraction, and that the default concurrency (8) respects backend quotas/rate limits (tune the default if needed).
• An actual concurrent local run (make test-local) — verify parallel uv venv provisioning + flyte run --local behaves under load.

🤖 Generated with Claude Code