feat(resources): the managed-resource upload streams into the multipart uploader, so a large file stores against a backend that bounds a single PUT (#1631)

[cjimti]

Closes #1631

The upload path streams

A managed resource is carried from the request into the multipart uploader and never assembled anywhere. resource.S3Client gains PutObjectStream beside the []byte write that thumbnails still use, NewResource.Content and RevisionUpload.Content are io.Reader, and the size a record carries is the count the write reported — a streamed body declares no length, so what reached storage is the only account of it.

Both write routes walk the form with r.MultipartReader() instead of parsing it. Metadata parts are read into their fields as they arrive; the file part is handed to the uploader unread. Content detection buffers contenttype.StructuredSniffLen (8 KiB) and puts it back in front of the rest with io.MultiReader, so a type detected off a stream is the type detected off the bytes, and the object the backend receives starts at its first byte.

Three things follow.

A backend that bounds a single PUT takes the file. MinIO refuses a PutObject above 16 MiB under the aws-chunked encoding the AWS SDK uses over HTTPS, which is what a 150 MB reference file met as chunk too big: choose chunk size <= 16MiB behind the message "The storage backend did not accept the file. Nothing was saved." The write is multipart now, so the bound does not apply.

No writable temporary directory is needed. ParseMultipartForm spooled any part past its memory budget to os.CreateTemp (mime/multipart/formdata.go:177). The published image is built FROM scratch (Dockerfile:6) and holds only the binary and the CA bundle, so that spool could not be created there. A walk spools nothing.

max_upload_bytes stops being resident heap. It bounds bytes streamed, enforced on the reader: the read that passes the ceiling fails, the uploader aborts the multipart upload it had begun, and the route answers 400 naming this deployment's number. The request body keeps its separate MaxBytesReader backstop at the ceiling plus the multipart framing headroom (#1628).

The corrected-CSV write both table revisers make streams for the same reason: a corrected CSV is as large as the file it corrects.

The file part goes last

Both routes read the form in order and stop at the file part, because that part is handed to the uploader where it is found. Every other field has to arrive before it. A form that puts a part behind the file is refused with the file part must be the last part of the form, and because the refusal happens while the file is being read, neither an object nor a record survives it — the alternative was storing the file and dropping the field without a word.

The portal's upload dialog, dev/seed-resources.sh and #1628's acceptance helper each sent the file first and now send it last. A script posting to POST /api/v1/resources or POST /api/v1/resources/{id}/content appends its file part last.

Storage refusals answer the same way on every route

ReviseContent reported a refused write as storing revision: ..., which writeError truncates at its first colon, so a storage outage reached the caller as the bare fragment "storing revision" with a 500 beside it. Replace-content and restore now answer 503 with the account the create route gives: nothing was saved, retry. One classification serves all three, and it is what lets the ceiling travel out as a 400 rather than being flattened into a storage failure.

The dev stack carries a backend that bounds a single PUT

#1631's criteria cannot be executed against a backend that accepts an unbounded single PUT. SeaweedFS does: probed at 40 MB, and it auto-chunks rather than refusing under -filer.maxMB and -master.volumeSizeLimitMB. MinIO refuses one, over HTTPS only, because the SDK sends a header checksum over plain HTTP and a trailing one over TLS.

So MinIO over TLS holds managed resources, SeaweedFS keeps portal assets, and the certificate is generated into gitignored dev/.tls/minio beside the api-test one. The platform trusts it through AWS_CA_BUNDLE, which config.LoadDefaultConfig reads; Go on darwin ignores SSL_CERT_FILE, so that is the only way to trust a local certificate without touching the system keychain.

A Hive catalog reads its metastore and its tables through one S3 client, and a registration points Trino at the file's own object, so two object stores means two scratch catalogs: scratch over SeaweedFS, scratch_resources over MinIO with its own metastore bucket, reached by connections acme-scratch and acme-scratch-resources. Trino trusts the same certificate through a truststore its container builds at start, which is why the certificate names minio as well as localhost. A deployment has one object store and needs one of these.

The platform runs with TMPDIR pointing at a path that does not exist, so the dev stack runs under the published image's condition rather than passing locally and failing once deployed. Nothing in production Go creates a temporary file.

What was executed

Against MinIO, through the real routes:

24 MiB create, above the 16 MiB single-PUT bound	201, size_bytes matches
24 MiB replacement	200, version recorded
Content read back	byte-identical by sha256
12 MiB upload with the server's TMPDIR absent	201
A part behind the file	400, no object and no record
250 MB at the deployment's ceiling (#1628)	stored
251 MB past it (#1628)	refused, naming 250 MB

In pkg/resource: a 64 MiB upload allocates under an eighth of the file, which no path holding the object can do; detection off a stream matches detection off the bytes for content longer than the sniff prefix, across four declarations; the ceiling refuses at the boundary between it and the body bound; and a form whose body runs out before the file part is reached is still refused by the size rather than as a form that would not parse.

make verify passes. Patch coverage 171/181 changed lines (94.5%).

Not covered

The read path. GetObject returns a []byte and pkg/blobserve serves from it, so serving a large managed resource still holds it — a real cost with its own blast radius, and out of scope here. The default ceiling stays 100 MB.

The acceptance suite carries failures this branch does not fix. Registrations over managed resources have not had a clean run since the catalog was renamed to scratch_resources, which is dev-stack configuration and reaches no released artifact; build/1546/acceptance.md records that rather than claiming a pass. Separately, two tests assert contracts #1604 and #1588 replaced, tracked in #1632 along with the reason nothing caught them: acceptance-check verifies a transcript exists, not that the run in it passed, and neither CI nor the release workflow runs the suite.

[codecov]

Codecov Report

❌ Patch coverage is 92.46575% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.52%. Comparing base (4e9f9fa) to head (d76e02b).

Files with missing lines	Patch %	Lines
pkg/resource/handler.go	92.78%	4 Missing and 3 partials ⚠️
pkg/resource/create.go	80.95%	2 Missing and 2 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1633      +/-   ##
==========================================
+ Coverage   90.45%   91.52%   +1.07%     
==========================================
  Files         776      776              
  Lines       95498    77760   -17738     
==========================================
- Hits        86385    71173   -15212     
+ Misses       6841     4315    -2526     
  Partials     2272     2272              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.