[kimseunghyun-kr]

What does this PR do?

Fixes #533: in streaming runs, a sample's sample.complete could be stamped before its own sample.recv_first, which gave a negative TPOT.

Cause. The ZMQ transport decodes every waiting message of one drain before the session handles any of them. QueryResult.completed_at is set at decode, but RECV_FIRST / RECV_NON_FIRST are stamped when the session handles each chunk. So when a response's chunks and its final result arrive in the same drain, COMPLETE is older than the chunks.

Fix. COMPLETE now uses the same per-response monotonic_ns() read (now_ns in _receive_responses) as the chunk events. Responses are handled in arrival order, so a sample's COMPLETE can no longer precede its own chunks. The source change is one line, plus a comment. QueryResult.completed_at itself is unchanged.

The design doc (docs/load_generator/DESIGN.md, "Timestamp fidelity" and the sequence diagram) is updated to match.

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Closes #533

Testing

• Tests added/updated
  • New test_streaming_complete_not_before_chunks_of_one_drain. It runs a real session in which the QueryResult is created first (as a decode in the same drain would), then the chunks and the result are delivered. It checks that RECV_FIRST ≤ RECV_NON_FIRST ≤ COMPLETE. Without the fix it fails: COMPLETE is about 11 ms before the chunks.
  • The _handle_response test now expects COMPLETE to carry the caller's arrival stamp, as the chunk events already do.
• All tests pass locally
  • pytest -m unit: 2,255 passed, 6 skipped, 1 failed on macOS: test_token_metrics.py::TestRayonCaps::test_init_worker_overrides_inherited_cap_with_block_size. It fails the same way on main, because os.sched_setaffinity is Linux-only.
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass. ruff, ruff-format, prettier, license headers and the PR size check pass. On macOS, mypy reports only the Linux-only os.sched_*affinity attributes in cpu_affinity.py and token_metrics.py. Those files are not touched here, and main gives the same errors.
• Documentation updated (if needed)

Note for reviewers. The issue lists a second option: stamp chunks at decode time too, so the client's own backlog is also removed from TTFT. That needs a new decode-time field on StreamChunk (more bytes on every chunk), so it is not in this focused fix. I can do it as a follow-up if you prefer it.

AI assistance: prepared with Claude Code. I reviewed every change.

🤖 Generated with Claude Code

[github-actions]

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
1 out of 2 committers have signed the MLCommons CLA.
✅ @arekay-nv
❌ @kimseunghyun-kr
_{You can retrigger this bot by commenting recheck in this Pull Request}

[Copilot]

🟡 Changes recommended

The design document still contains a contradictory description of the old completion timestamp behavior.

1 open finding

• Update duplicate timestamp documentation to match current behavior · New

What changed in this PR

Fixes negative streaming TPOT by timestamping COMPLETE when responses are handled.

Changes:

• Uses arrival timestamps for completion events.
• Adds streaming ordering regression tests.
• Updates timestamp documentation and diagrams.

File	Summary
tests/​unit/​load_generator/​test_async_session.py	Adds regression coverage and updates timestamp expectations.
src/​inference_endpoint/​load_generator/​session.py	Uses handling-time timestamps for COMPLETE.
docs/​load_generator/​DESIGN.md	Documents the revised timestamp behavior.

🧠 Review effort: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[kimseunghyun-kr]

recheck

[Copilot]

🟡 Changes recommended

Update the design documentation’s timestamp explanation and remaining _handle_response example.

1 open finding

• Update _receive_responses example for now_ns argument · New

1 resolved since last review

• Update duplicate timestamp documentation to match current behavior

🧠 Review effort: Lite

[kimseunghyun-kr]

recheck

[Copilot]

🟢 Approval recommended

The focused fix, tests, and documentation updates address the reported timing issue with no unresolved review concerns.

0 open findings

1 resolved since last review

• Update _receive_responses example for now_ns argument

🧠 Review effort: Lite

[arekay-nv]

@kimseunghyun-kr Please complete the CLA process:

Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@d3c57f2). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #536   +/-   ##
=======================================
  Coverage        ?   81.37%           
=======================================
  Files           ?      157           
  Lines           ?    22472           
  Branches        ?        0           
=======================================
  Hits            ?    18286           
  Misses          ?     4186           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[arekay-nv]

@kimseunghyun-kr please complete the CLA so we can review the PR.

MLCommons CLA bot: Thank you very much for your submission; we really appreciate it. Before we can accept your contribution, we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA. If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact support@mlcommons.org.1 out of 2 committers have signed the MLCommons CLA.✅ @arekay-nv❌ @kimseunghyun-krYou can retrigger this bot by commenting recheck in this Pull Request

[Copilot]

🔵 Needs a closer look

The ordering fix is correct, but it also changes the reported COMPLETE-based latency and OSL/s for all runs, and maintainers need to decide whether to accept or document that shift.

2 open issues

• Document or limit handling-time stamps to streaming responses · New
• Update latency bias documentation for client-side backlog · New

_{Review effort: Lite}