[BugFix][Autotune] Drain timed-out benchmark calls

[morluto]

Fixes #2838.

Problem

A worker reported a benchmark timeout then immediately accepted another configuration while the timed-out daemon thread continued to run. That allowed CUDA work and measurement state to overlap on the same worker.

Change

After reporting a timeout, the worker drains the still-running benchmark thread before reading its next task.

timeout reported -> timed-out call drains -> next configuration starts

This does not claim to force-cancel arbitrary Python/CUDA code; an uncooperative call can still block that worker. Other workers remain independent.

Regression coverage

The new test queues a slow call and a fast call, verifies the timeout arrives promptly, and verifies the fast call cannot start before the slow one completes.

Validation

• python -m pytest -q testing/python/autotune/test_tilelang_autotune_timeout.py testing/python/autotune/test_tilelang_autotune_atomic_save.py testing/python/autotune/test_autotune_cache_key.py — 8 passed
• pre-commit, formatting, and git diff --check — passed

Summary

• Drain timed-out benchmark threads before a worker accepts another configuration.
• Prevent overlapping CUDA work and measurement state on the same worker.
• Preserve worker and device independence because the fix does not force thread cancellation.
• Add a regression test for timeout ordering and delayed execution.

Validation

• Targeted tests passed.
• Pre-commit checks passed.
• Formatting checks passed.
• git diff --check passed.

[github-actions]

👋 Hi! Thank you for contributing to the TileLang project.

Please remember to run pre-commit run --all-files in the root directory of the project to ensure your changes are properly linted and formatted. This will help ensure your contribution passes the format check.

We appreciate you taking this step! Our team will review your contribution, and we look forward to your awesome work! 🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3794dcda-d2ce-4b0e-808f-813472da8694

📥 Commits

Reviewing files that changed from the base of the PR and between e56c9f2 and 6e9be9e.

📒 Files selected for processing (1)

• testing/python/autotune/test_tilelang_autotune_timeout.py

💤 Files with no reviewable changes (1)

• testing/python/autotune/test_tilelang_autotune_timeout.py

---

📝 Walkthrough

Walkthrough

The benchmark worker now drains a timed-out benchmark thread before starting another configuration. A threaded regression test verifies timeout ordering, delayed execution, and clean worker shutdown.

Changes

Autotuner timeout handling

Layer / File(s)	Summary
Drain timed-out benchmarks before reuse tilelang/autotuner/tuner.py, testing/python/autotune/test_tilelang_autotune_timeout.py	The worker waits for a timed-out benchmark thread before processing the next configuration. The timeout warning documents this behavior. The regression test verifies ordering, delayed execution, and clean shutdown.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the autotuner bug fix: draining timed-out benchmark calls.
Linked Issues check	✅ Passed	The worker drains timed-out benchmark threads before starting another configuration, and the regression test verifies completion ordering as required by issue #2838.
Out of Scope Changes check	✅ Passed	The implementation and regression test directly support issue #2838, with no unrelated code changes identified.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4b95a44c6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d47bee3ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e56c9f2fe6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Wait indefinitely for the explicit release event

When a loaded CI worker does not reschedule the test thread for more than one second after slow_call_started.set(), this timed wait releases the slow call automatically, allowing the worker to publish its result and start the fast call before the assertions on lines 49–50. The fresh event-based revision therefore still has the original scheduling race; wait without a timeout here, since the finally block already guarantees that the event is released.

Useful? React with 👍 / 👎.