Skip to content

Commit b9a4f00

Browse files
owenpearsonclaude
andcommitted
docs: record the proxy tier and the httpRequestTimeout unit defect
The suite's README gains what a reader needs about the proxy package: where the binary comes from, the two environment variables that change how it is obtained, and the shape of a client built against a session. The translation skill gains what a writer needs — the harness table, a worked example and the traps the derivation hit, among them that a parent package's pytest timeout marker wins over a subpackage's unless the subpackage prepends its own, and that pytest-asyncio runs a session-scoped async fixture on a different event loop from the tests. deviations.md carries the one gated test and its measurement. The unit mismatch on http_request_timeout was already recorded twice, as something readable only off the options object; the proxy measures it reaching the wire, so both rows are corrected and the defect is counted once, as a root cause, where the gated test is. Its blast radius is bounded by the defaults coming out right by coincidence — 4 and 10 seconds being TO3l3's and TO3l4's 4000 and 10000 milliseconds — so only a client that configures the option is affected. Issue #709 is already filed against the same line for a different defect and the two are cross-referenced, since both change what one attempt may spend of the RSC15 retry budget. The counts are recomputed throughout: 1059 Test IDs, 1068 derived tests, 1139 pytest cases, and 67 gated root causes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent c763fa7 commit b9a4f00

3 files changed

Lines changed: 306 additions & 26 deletions

File tree

‎.claude/skills/uts-to-python/SKILL.md‎

Lines changed: 145 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,10 +16,13 @@ gh api repos/ably/specification/contents/uts/docs/integration-testing.md --jq '.
1616
gh api repos/ably/specification/contents/uts/rest/unit/<spec>.md --jq '.content' | base64 -d
1717
gh api repos/ably/specification/contents/uts/realtime/unit/<spec>.md --jq '.content' | base64 -d
1818
gh api repos/ably/specification/contents/uts/rest/integration/<spec>.md --jq '.content' | base64 -d
19+
gh api repos/ably/specification/contents/uts/docs/proxy.md --jq '.content' | base64 -d
20+
gh api repos/ably/specification/contents/uts/rest/integration/proxy/<spec>.md --jq '.content' | base64 -d
1921
```
2022

2123
`writing-derived-tests.md` governs, and `integration-testing.md` alongside it for
22-
`uts/rest/integration`. This file covers only what is particular to ably-python.
24+
`uts/rest/integration`, with `proxy.md` governing the proxy package within it. This
25+
file covers only what is particular to ably-python.
2326

2427
## Layout
2528

@@ -285,9 +288,10 @@ All in `test.uts.helpers.clock`.
285288
## The integration tier
286289

287290
`uts/rest/integration/<name>.md` becomes `test/uts/rest/integration/<name>_test.py` and
288-
runs against the real Ably sandbox — eleven specifications, 76 tests. There is no mock
289-
and no `_test_options`: the client reaches the network. `test/uts/README.md` covers the
290-
same ground for someone reading the suite; this is what someone writing a test needs.
291+
runs against the real Ably sandbox — twelve specifications, 84 tests, one of them the
292+
proxy package the section below covers. There is no mock and no `_test_options`: the
293+
client reaches the network. `test/uts/README.md` covers the same ground for someone
294+
reading the suite; this is what someone writing a test needs.
291295

292296
What differs from the mock-backed tiers:
293297

@@ -333,6 +337,86 @@ async def test_rsl2a_history_returns_messages(sandbox, use_binary_protocol):
333337
history = await wall_clock_poll_until(one_message, description='the message to reach history')
334338
```
335339

340+
## The proxy tier
341+
342+
`uts/rest/integration/proxy/<name>.md` becomes
343+
`test/uts/rest/integration/proxy/<name>_test.py`, and routes its traffic through
344+
[ably/uts-proxy](https://github.com/ably/uts-proxy) on the way to the sandbox. The
345+
proxy binds a port per session, takes plain HTTP on it, speaks TLS onwards, applies the
346+
session's rules and records what crosses it. `uts/docs/proxy.md` governs the tier and
347+
is worth reading in full before deriving one of these; fetch it the same way as the
348+
other governing docs.
349+
350+
What differs from the rest of the integration tier:
351+
352+
- **The client points at the session, not at the sandbox.** `endpoint='localhost'`,
353+
`port=session.proxy_port`, `tls=False`, `use_binary_protocol=False`. `endpoint` and
354+
`port` set the primary host, and `fallback_hosts=['localhost']` sets the fallback to
355+
the same session, so both attempts land in one event log. Leave `fallback_hosts` out
356+
where the spec does: `endpoint='localhost'` disables fallbacks by itself (REC2c2).
357+
- **Authentication is a callback.** A plain connection cannot carry basic auth, so
358+
every client takes `auth_callback=`; see the traps below.
359+
- **A fault is a rule, and everything else passes through.** `times: 1` faults the
360+
first request that matches and lets the retry reach the sandbox, which is the shape
361+
every fallback scenario wants.
362+
- **The proxy is the second witness.** The SDK's own result answers half the question
363+
and `session.get_log()` the other half — how many requests were made, in what order,
364+
and what the proxy answered them with.
365+
- **The per-test timeout is 300 seconds**, set by the package's own `conftest.py`.
366+
367+
| Name | Is |
368+
|---|---|
369+
| `proxy_session` fixture, in `rest/integration/proxy/conftest.py` | a specification's `create_proxy_session(...)`, together with its `AFTER EACH TEST: session.close()`. `session = await proxy_session(rules=[...])`, as many times as a test needs, and every session is closed afterwards |
370+
| `proxy_control` fixture | the running control API, session-scoped. `proxy_session` asks for it, so a test does not have to |
371+
| `create_proxy_session(endpoint=SANDBOX_ENDPOINT, port=None, rules=None, timeout_ms=SESSION_TIMEOUT_MS)` | the function itself, in `test.uts.helpers.proxy`, for the rare case that wants a session the fixture will not close |
372+
| `session.session_id`, `session.proxy_host`, `session.proxy_port` | what a spec reads off its `session`. The host is always `localhost` |
373+
| `session.add_rules(rules, position='append')` | rules added while the session runs. `position='prepend'` puts them ahead of the ones already there, which is how a spec faults traffic only once the client has reached some state |
374+
| `session.trigger_action(action)` | the imperative half — `{'type': 'disconnect'}`, `{'type': 'close', 'closeCode': 1000}`, `inject_to_client`. 409 from the control API when no connection is open |
375+
| `session.get_log()` | every event, in order, as the dictionaries the control API sends: `type`, `method`, `path`, `status`, `direction`, `queryParams`, `message`, `ruleMatched`. The field names are the ones a specification's assertions are written against, so they are not renamed |
376+
| `session.close()` | best effort and never raises; the fixture calls it for you |
377+
| `ensure_proxy()` / `stop_proxy()` | start the control process and reap it. Only the `proxy_control` fixture should need them |
378+
| `PROXY_VERSION`, `ARCHIVE_CHECKSUMS`, `RELEASE_URL` | the pinned release and the sha256 of each platform's archive. Moving the pin means new checksums, copied from the release's own `checksums.txt` |
379+
| `SESSION_TIMEOUT_MS` (120000) | the session's **idle** auto-cleanup timer, passed as `timeoutMs` |
380+
| `SUITE_TIMEOUT` (300) in the package `conftest.py` | the per-test timeout for this package |
381+
| `UTS_PROXY_LOCAL_PATH` | a locally built binary, or a `.tar.gz` holding one, in place of the pinned release |
382+
| `UTS_PROXY_CONTROL_URL` | a control API already running, which is then neither started nor stopped by the suite |
383+
384+
Everything but the two fixtures is in `test.uts.helpers.proxy`. `http_requests` and
385+
`http_responses` in the example below are the derived file's own log filters, defined
386+
at the top of it, because every test in the file reads the log the same two ways.
387+
388+
```python
389+
# UTS: rest/proxy/RSC15l4/cloudfront-header-fallback-0
390+
async def test_rsc15l4_cloudfront_header_fallback(sandbox, proxy_session):
391+
session = await proxy_session(rules=[{
392+
'match': {'type': 'http_request', 'pathContains': '/time'},
393+
'action': {
394+
'type': 'http_respond',
395+
'status': 403,
396+
'body': {'error': {'message': 'Forbidden', 'code': 40300, 'statusCode': 403}},
397+
'headers': {'Server': 'CloudFront'},
398+
},
399+
'times': 1,
400+
'comment': 'RSC15l4: CloudFront 403 on first /time request',
401+
}])
402+
403+
client = sandbox_rest_client(
404+
auth_callback=token_auth_callback(sandbox.key_str),
405+
endpoint='localhost',
406+
fallback_hosts=['localhost'],
407+
port=session.proxy_port,
408+
tls=False,
409+
use_binary_protocol=False,
410+
)
411+
412+
result = await client.time()
413+
assert isinstance(result, (int, float))
414+
415+
log = await session.get_log()
416+
assert len(http_requests(log, '/time')) >= 2
417+
assert http_responses(log)[0]['status'] == 403
418+
```
419+
336420
## Traps that cost the most time
337421

338422
Ordered by how much they cost, not by subject. Every one was hit for real while
@@ -594,6 +678,61 @@ that **passes while proving nothing**, rather than one that fails.
594678
Use `wall_clock_poll_until`, and no `FakeClock`. `writing-derived-tests.md` has a
595679
section on this ("Integration timeouts are wall-clock").
596680

681+
## Traps found while deriving the proxy tier
682+
683+
Established against the real proxy and the real sandbox while deriving
684+
`rest/integration/proxy/rest_fallback.md`.
685+
686+
- **Basic auth cannot be used through the proxy at all.** The session speaks plain
687+
HTTP, and `tls=False` makes the SDK raise `40103 "Cannot use Basic Auth over non-TLS
688+
connections"` (RSC18) before a request is written, so a client built with `key=`
689+
never reaches a rule. `client.time()` is the one call that works, because it is
690+
`skip_auth`. Every test in the specification therefore authenticates with
691+
`authCallback`, including the `/time` ones, and a derived test should keep that even
692+
where it looks unnecessary.
693+
- **The token callback's own client must go straight to the sandbox.** A callback that
694+
asks for a token through a client pointed at the session puts a request in front of
695+
the waiting rule and an extra `http_request` in the log, which breaks every
696+
assertion that counts requests exactly (`== 1` for RSC15l's 4xx test, `>= 2` for the
697+
fallback ones). Build an inner `AblyRest(key=api_key, endpoint=SANDBOX_ENDPOINT)`,
698+
request the token through that, and close it.
699+
- **`http_response` events carry no `path`.** They have `status` and `ruleMatched`
700+
only, so "the injected response fired" is read off the response events **in order**
701+
— `http_responses(log)[0]['status'] == 403` — rather than by filtering to an
702+
endpoint. `http_request` events do carry `method` and `path`. A rule with no
703+
`comment` appears as `ruleMatched: "rule-0"`.
704+
- **The parent package's timeout marker wins unless the subpackage prepends its
705+
own.** `pytest-timeout` reads the first of an item's own markers, and
706+
`rest/integration/conftest.py` marks everything beneath it with 120 seconds. The
707+
proxy package's `pytest_collection_modifyitems` adds its 300 with
708+
`append=False`; without that the marker order decides the timeout and a test that
709+
waits out a twenty-second delay on a cold cache is cut off. Anyone adding a further
710+
sub-tier under `test/uts/rest/integration/` hits this.
711+
- **A session-scoped async fixture runs on a different event loop from the tests.**
712+
pytest-asyncio 0.23 gives the session fixture its own loop, so an object bound to
713+
the loop that created it — an `httpx.AsyncClient`, say — must not be held across the
714+
yield: reusing it from a test raises `RuntimeError: Event loop is closed` or attaches
715+
to the wrong loop. `helpers/proxy.py` opens a client per control call for exactly
716+
this reason, the way `sandbox.py` does.
717+
- **The session's `timeoutMs` is an idle timer, not a deadline.** The proxy's default
718+
is 30000, measured from the last traffic through the session, and a test that has
719+
the proxy delay a response by twenty seconds and then reads the log spends longer
720+
than that idle, which is long enough for the session to be collected out from under
721+
the test. The harness passes 120000.
722+
- **Every request the test's client makes lands in the same log.** The log is
723+
per-session, not per-endpoint, so a verification step that reads history through the
724+
same client adds its own `http_request` events. In `RSL1k4` the log is read
725+
**before** the history call, and the history read is a `wall_clock_poll_until` rather
726+
than a single fetch, because a published message does not reach history at once.
727+
- **`httpRequestTimeout` is milliseconds in the specification and seconds in
728+
ably-python.** `ably/http/http.py:193` hands `(http_open_timeout,
729+
http_request_timeout)` to `httpx`, which reads seconds, so the specification's
730+
`http_request_timeout=3000` is a 3000-second deadline: measured, the request sat out
731+
the proxy's whole 20-second delay and then succeeded on the primary host, and no
732+
fallback was attempted. Passing `3` makes the same test pass in 3.1s, so only the
733+
unit is wrong. The test is written as the specification has it and gated with
734+
`@deviation`; `test/uts/deviations.md` carries the entry.
735+
597736
## Timers
598737

599738
Three regimes; pick by tier.
@@ -617,8 +756,8 @@ silence against the same clock it arms its timer on. See the fake-time section o
617756
a server the tests exist to talk to, and shortening a timeout would only make the tier
618757
flaky. Poll with `wall_clock_poll_until` rather than sleeping a guess.
619758

620-
The pytest timeout is 30 seconds for the suite and 120 for `rest/integration`, so keep
621-
waits well under whichever applies.
759+
The pytest timeout is 30 seconds for the suite, 120 for `rest/integration` and 300 for
760+
`rest/integration/proxy`, so keep waits well under whichever applies.
622761

623762
## Deviations
624763

‎test/uts/README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,54 @@ Tests here are given 120 seconds each, per `uts/docs/integration-testing.md`, ra
145145
than the 30 seconds `pyproject.toml` sets for the suite as a whole. The marker covers
146146
the integration package alone.
147147

148+
`rest/integration/proxy/` routes its traffic through
149+
[ably/uts-proxy](https://github.com/ably/uts-proxy), a programmable proxy standing
150+
between the client and the sandbox. Those specifications are about what the SDK does
151+
when a request goes wrong — a connection dropped mid-request, a 503, a CloudFront 403,
152+
a response held past the request timeout — and the sandbox answers correctly, so the
153+
fault is injected in front of it. The proxy binds a port per session, takes plain HTTP
154+
on it and speaks TLS onwards to the sandbox, applies the rules the session was opened
155+
with, and records every request and response that crosses it.
156+
157+
The binary is a pinned `uts-proxy` release. The first run that needs it fetches the
158+
release archive for the machine, checks it against the sha256 the release publishes,
159+
and extracts the binary into `~/.cache/uts-proxy/<version>/`, where every run
160+
afterwards finds it; the download is serialised on a lock file, so several Python
161+
versions starting at once on an empty cache fetch it once between them.
162+
`UTS_PROXY_LOCAL_PATH` names a locally built binary, or a `.tar.gz` holding one, to be
163+
used in place of the release, and `UTS_PROXY_CONTROL_URL` names a control API someone
164+
is already running, which the suite uses as it stands and leaves running. Otherwise one
165+
control process is started for the test session on a free port and reaped at the end of
166+
it; it serves every session the run opens.
167+
168+
`proxy_session` is a specification's `create_proxy_session(...)`, and closes every
169+
session it hands out when the test ends. A client reaches its session by naming
170+
`localhost` and the session's port with TLS off, which disables fallback hosts (REC2c2);
171+
a scenario about a retry names the same session again as its fallback, so both attempts
172+
arrive at the one port and appear in the one event log, which `session.get_log()`
173+
returns:
174+
175+
```python
176+
async def test_rsc15l_connection_drop_fallback(sandbox, proxy_session):
177+
session = await proxy_session(rules=[{
178+
'match': {'type': 'http_request', 'pathContains': '/time'},
179+
'action': {'type': 'http_drop'},
180+
'times': 1,
181+
}])
182+
client = sandbox_rest_client(
183+
auth_callback=token_auth_callback(sandbox.key_str),
184+
endpoint='localhost', fallback_hosts=['localhost'],
185+
port=session.proxy_port, tls=False, use_binary_protocol=False)
186+
```
187+
188+
A plain connection rules basic auth out, since RSC18 refuses it, so every client here
189+
authenticates through a callback whose own request goes straight to the sandbox and
190+
stays out of the event log.
191+
192+
Tests in this package are given 300 seconds each: a cold cache downloads the binary
193+
before the first of them runs, and a specification that provokes a timeout sits through
194+
the delay it asked the proxy for.
195+
148196
## Running
149197

150198
```

0 commit comments

Comments
 (0)