Skip to content

fix(filesv2): end upload requests that never answer instead of freezing the batch - #1257

Merged
ValentinKolb merged 7 commits into
mainfrom
fix/upload-request-timeouts
Oct 16, 2026
Merged

ValentinKolb merged 7 commits into
mainfrom
fix/upload-request-timeouts

Conversation

@ValentinKolb

Copy link
Copy Markdown
Collaborator

Part of #1163. Upstream: k2b-dev/filegate#5 (the Filegate SDK has no request deadline; Filegate itself is unchanged).

Outcome

No upload request can wait forever any more: Files workspace, public inbox, folder preparation and cld filesv2 put.

  • A transfer request the file server never answers ends after a bounded deadline. The row fails with a plain reason ("No response from the file server" / "Keine Antwort vom Dateiserver", or "The file server is busy" / "Der Dateiserver ist ausgelastet" for HTTP 408, 429, 502, 503 and 504), and the batch continues with the next file. Other 5xx answers such as 500 or 501 are final, because a retry fails the same way.
  • A Cloud request in the browser that hits its deadline (for example folder preparation) says "No response from the server" / "Keine Antwort vom Server", so the row names the component that did not answer.
  • One deadline ends the transfer. Before, a hung 8 MiB PUT would have run through the SDK's three attempts (about 9.4 min). Automatic retries of stalled transfers belong to the later slice upload-wait-and-retry.
  • cld filesv2 put: target resolution, --parents folder creation (also mkdir --parents), file-ID resolution, session opening and commit have a 60 s deadline, also when the response body stalls after its headers (including a non-success status). Transfer failures map to plain localized reasons without lease URLs. In text mode, progress goes to stderr: on a terminal one updating line (acknowledged bytes and percent, at most once per second, with a long file name shortened to the terminal width), otherwise only phase lines (uploading, finishing, done). Stdout and --json/--jsonl are unchanged.

Root cause

DirectSession (Filegate SDK 7.0.0) calls fetch without a deadline, and Cloud passed none. A status GET, segment-page GET or segment PUT that never answered left the row "working" with no message, and the queue waited behind it. A plain per-request timeout is not enough: the SDK retries a thrown fetch error unless its own signal is aborted.

Design

  • transferUpload passes a Cloud-owned request function to new DirectSession(url, request). Deadlines: 30 s for control calls; 60 s plus body size at 64 KiB/s (about 0.5 Mbit/s) for segment PUTs. A browser fetch cannot observe request-body bytes, so a true inactivity timer is impossible; the size-proportional deadline is the honest substitute and is named a deadline in code and docs.
  • The deadline also covers the response body (it is read inside the request function). A non-success response whose body cannot be read keeps its status and headers, so the SDK still honors Retry-After.
  • On a deadline, the request function aborts a per-transfer controller that is combined with the caller's cancel signal for the SDK call, so the SDK rethrows instead of retrying. The SDK keeps its quick retries for fetch TypeErrors and for 429/502/503/504 with a short Retry-After. A caller cancel keeps its own reason and is never reported as a stall.
  • UploadTransferError carries a kind (stalled, network, busy for 408/429/502/503/504, final for everything else) and the Retry-After hint (seconds or HTTP date).
  • Seat decision: "A browser fetch cannot observe request-body bytes, so a true inactivity timer for a PUT is impossible; the size-proportional deadline is the honest substitute. Name it a deadline in code and docs." This implements the lane plan's "per-request inactivity timeout via a fetch wrapper for DirectSession".
  • A test keeps every new DirectSession( inside upload-transfer.ts.
  • CLI Ctrl-C: outside uploadFile there is no SIGINT handler, so Ctrl-C still ends the process during resolution and folder creation; those requests only needed the deadline. During the transfer, Ctrl-C keeps aborting the session as before.
  • browser-upload.behavior.test.tsx flushes leftover work in afterEach: without it, a fire-and-forget abort from the previous test leaked into "a folder request with no response fails its group visibly" (reproduced).

How do we prevent this next time?

Guards: upload-transfer.test.ts expects exactly one attempt per expired deadline (status, segments, PUT, stalled response body), keeps Retry-After when an error body breaks, checks that caller cancel stays distinct, and classifies 500/501 as final after one attempt. upload-messages.test.ts maps a Cloud TimeoutError to "No response from the server" and a stalled transfer to "No response from the file server". A CLI test keeps every TTY progress line within the terminal width. filegate-transfer.integration.test.ts runs a real session against the test Filegate with a hung second segment PUT and expects stalled within the shrunken deadline, one hung attempt, and the session still open with segment 0 received (on main the call never settles). CLI tests stall resolution, put --parents, mkdir --parents, file-ID resolution and session opening with headers that never come, a body that stalls, and a 503 whose body stalls, and expect the localized deadline error; on main they hang. The construction test keeps DirectSession behind the deadline. The lesson is recorded on the lessons-learned page.

Docs

  • docs-site/apps-content/en/filesv2.md: deadline sized for the slowest supported uplink (about 0.5 Mbit/s); CLI progress on stderr.
  • docs-site/apps-content/en/filesv2.md also names the two browser deadline messages and the busy statuses.
  • packages/filesv2/src/cli-references/index.md: deadlines, messages, busy statuses (408, 429, 502, 503, 504; other failures final), progress on stderr, name shortened to the terminal width.
  • Help EN/DE filesv2-start: put shows progress; a file that gets no answer fails with a reason and the upload continues.
  • docs-site/docs/en/contributing/lessons-learned.md: new lesson and silent-failure guard row.
  • cloud-dev skill: unchanged; its bounded-work invariant already covers this, and no public platform contract changed.

Verification

All on the final HEAD, rebased on current main.

  • bun run --cwd packages/filesv2 typecheck: passed.
  • bun run test --filter packages/filesv2 (Chromium): unit 382 passed (109 integration-only skipped), render 5 passed, behavior 113 passed.
  • WebKit (TEST_BROWSER=webkit against a Playwright 1.63.0 WebKit server): bun run test --browser --filter packages/filesv2 5 files, 63 tests passed; behavior 113 passed. In the plain bun run test run (browser files sharing one process on a loaded machine), the unchanged StorageOverview.browser.test.ts timed out on its first test once (5 s); alone it passed 3 of 3 runs. This change does not touch it.
  • bun run test --integration --filter packages/filesv2 against the test infrastructure (incl. test Filegate): 519 passed, 0 failed.
  • Guards fail without their fix: reverting the 5xx classification fails 3 tests in upload-transfer.test.ts; reverting the TimeoutError mapping fails upload-messages.test.ts; disabling the name truncation fails the TTY width test in cli-upload.test.ts. The new CLI deadline tests hang on main until the test timeout.
  • bun scripts/check.ts help-writing localization biome boundaries module-mocks: 5/5 passed. git diff --check origin/main...HEAD: clean.
  • Codex review (gpt-6.1-sol): earlier rounds found two issues (Retry-After lost on a broken error body; CLI deadline message lost on a stalled 503 body), fixed with tests. A final review of the whole rebased branch found no actionable issues.
  • Not run: harper-cli is not installed on this machine, so the changed English Markdown had no Harper lint.

Operations

No configuration, migration or deployment change. Uploads on very slow links (below about 0.5 Mbit/s) may now fail with "No response from the file server" instead of hanging.

…ng the batch

The Filegate SDK has no request deadline (k2b-dev/filegate#5). A status
GET, segment-page GET or segment PUT that never answered froze the whole
upload batch: the bar stood still, the row stayed "working", and no
message appeared. The CLI sat silent in the same way, and target
resolution and folder creation for put --parents had no deadline either.

transferUpload now hands DirectSession a Cloud-owned request function
with a deadline per attempt: 30 s for control requests, 60 s plus the
body at 64 KiB/s (about 0.5 Mbit/s) for segment PUTs. A browser fetch
cannot observe request-body bytes, so a size-proportional deadline is
the honest substitute for an inactivity timer. The deadline also covers
the response body. When it expires, the request function aborts the
signal the SDK upload runs under, so the SDK stops instead of retrying
the same hung request up to three times (about 9.4 min for an 8 MiB
segment). The SDK keeps its quick retries for network errors and short
Retry-After answers. A caller cancel keeps its own reason.

Failures become one UploadTransferError with a kind (stalled, network,
busy, final) and the Retry-After hint; the Files workspace and public
inbox show "No response from the file server" or "The file server is
busy" in English and German, and the batch continues with the next
file. Folder preparation uses the same 60 s control deadline.

The CLI gives target resolution and folder creation a 60 s deadline,
maps transfer failures to plain localized reasons without lease URLs,
and writes phase and acknowledged-byte progress to stderr in text mode:
one updating line at most once per second on a terminal, phase lines
otherwise. Standard output and --json stay unchanged.

Guard: tests keep every new DirectSession( inside upload-transfer.ts,
expect one attempt per expired deadline, and an integration test against
a real Filegate checks that a hung second segment ends as "stalled" with
the session still open and segment 0 kept.

Part of #1163
… body breaks

The Cloud request function buffers each Filegate response. When a 429
with a long Retry-After lost its connection while the body was read,
the read error reached the SDK as a transport failure, and the SDK
retried after 250 ms instead of honoring Retry-After. A non-success
response whose body cannot be read now keeps its status and headers,
so the SDK's own status handling applies.

In the CLI, readJson turns an unreadable non-success body into a status
error, so a deadline that stalled a 503 body showed "503 Service
Unavailable" instead of the localized deadline message. readWithin now
checks its signal when reading fails.

Guard: transfer tests for a 429 and a 503 with a broken body, and the
CLI deadline tests gain a stalled 503 body for every request.
…file server errors final

A Cloud request that hits its deadline in the browser said the file server did not answer, so the row blamed the wrong component. It now says the server did not answer. Any Filegate 5xx was called busy, which invited a retry that fails the same way for 500 or 501; busy now means only the transient 408, 429, 502, 503 and 504. On a terminal, a long file name could wrap the one updating progress line and leave fragments; the name is shortened to fit the terminal width.
… in the CLI reference

The CLI reference did not say which file server answers count as busy or that a long file name is shortened on a terminal, so readers could not tell a retryable failure from a final one.
@ValentinKolb
ValentinKolb added this pull request to the merge queue Oct 16, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 16, 2026
…meouts

# Conflicts:
#	docs-site/docs/en/contributing/lessons-learned.md
@ValentinKolb
ValentinKolb added this pull request to the merge queue Oct 16, 2026
Merged via the queue into main with commit 3564daf Oct 16, 2026
21 checks passed
@ValentinKolb
ValentinKolb deleted the fix/upload-request-timeouts branch October 16, 2026 16:56
@github-actions github-actions Bot mentioned this pull request Oct 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant