Skip to content

Curl backend: send and receive binary bodies correctly - #3047

Merged
adamw merged 4 commits into
masterfrom
curl-binary-bodies
Nov 2, 2026
Merged

adamw merged 4 commits into
masterfrom
curl-binary-bodies

Conversation

@adamw

@adamw adamw commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Part of #3043 (point 1).

Request bodies were turned into a String and then into a C string. Bytes that are not valid UTF-8 were changed, and the body was cut at the first NUL byte. As the Content-Length header still had the original size, the server kept waiting for the rest of the body, and the request timed out.

Response bodies had the same problem: they were read with fromCString.

Now:

  • request bodies (and multipart parts) are passed to curl as bytes, with an explicit size
  • response bodies are read as bytes, using the length tracked by the write callback
  • string bodies use their declared encoding instead of always UTF-8

Also fixed in the HttpClient backends (found by the new shared test): string bodies were always sent as UTF-8, ignoring their encoding, while Content-Length used the declared encoding.

Tests (shared HttpTest, mirrored in the native SyncHttpTest):

  • "as a byte array exact" now echoes all 256 byte values
  • the multipart binary part contains a NUL byte
  • new: a string body with a non-UTF-8 encoding
  • byte buffer and multipart encoding tests are now also run on native

adamw added 4 commits October 28, 2026 23:23
The request body options were set in a discarded effect, so with cats-effect
and ZIO the body was never sent, and curl read it from stdin.

The new shared non-UTF-8 test also showed that the HttpClient backends sent
string bodies as UTF-8, ignoring their encoding.
# Conflicts:
#	core/src/main/scalanative/sttp/client4/curl/AbstractCurlBackend.scala
@adamw
adamw merged commit 9198da0 into master Nov 2, 2026
27 checks passed
@adamw
adamw deleted the curl-binary-bodies branch November 2, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant