Streaming writes via pgduck_server RECEIVE protocol

[timmclaughlin]

Streaming writes via pgduck_server RECEIVE protocol

What this PR adds

An opt-in path for pg_lake's bulk-write operations (INSERT, UPDATE,
DELETE, COPY FROM STDIN, iceberg metadata uploads) to push bytes
directly to pgduck_server over libpq COPY-IN, instead of writing
them to a shared filesystem under $PGDATA/pgsql_tmp and asking
pgduck to read them back via read_csv().

Activated by setting pg_lake_engine.streaming_writes = on
(default off).

Why

The existing pg_lake design assumes postgres and pgduck_server
share a filesystem — the standard sidecar deployment puts both in
the same pod, talking via a Unix socket on a shared
$PGDATA-relative path. That layout breaks every bulk-write path
under deployment models where postgres and pgduck don't share a
filesystem:

• Operator-managed Kubernetes deployments (e.g. CloudNativePG, where
  the Cluster CR overrides the pod's command: and doesn't allow
  arbitrary sidecar containers).
• Multi-host setups where postgres and pgduck run on different
  machines.
• Anywhere a co-located sidecar isn't practical for ops or security
  reasons.

In all of these, today's pg_lake hits a "no such file or directory"
in pgduck_server when the bulk-write path's CSV temp lives on a
filesystem the server can't see.

This PR makes pg_lake work in those topologies by routing the bulk-
write data through libpq COPY-IN to a server-local sink on
pgduck_server, decoupling the two processes' filesystems.

How it works

pgduck_server learns a new RECEIVE <inner-query> query prefix.
When the server sees RECEIVE …:

1. Picks a server-local sink path under --recv_dir.
2. Substitutes the '@@PG_LAKE_RECV@@' token in the inner query
   (which appears exactly once, already wrapped in single quotes
   so it stands in for a string-literal argument — e.g.
   read_csv('@@PG_LAKE_RECV@@', …)) with a single-quoted SQL
   literal containing the sink path. The sink path is generated
   server-side and never contains a single quote; the substitution
   defensively rejects any path that does, rather than attempting
   to escape, which keeps the SQL-quoting story trivial to audit.
3. Sends CopyInResponse to the client; accepts CopyData chunks
   into the sink.
4. On CopyDone, runs the deferred inner query (which
   read_csv()'s from the sink) and streams its results back via the
   normal PGresult flow.

On the client side, pg_lake_engine adds a CSVStreamWriter
primitive that builds the same COPY (…) TO … command pg_lake
already uses for the file-based path, prefixes it with RECEIVE,
opens a libpq COPY-IN, and exposes a DestReceiver that the FDW
callers drive with rows.

The INSERT, UPDATE, DELETE, COPY-FROM-STDIN, and iceberg-metadata
upload paths in pg_lake_table / pg_lake_copy / pg_lake_engine
all switch transparently to the streaming path when the GUC is on.

Structure

This PR is two commits:

1. pgduck_server: add RECEIVE protocol for streaming COPY-IN sink.
   Self-contained: TCP listener, RECEIVE prefix handling,
   recv_sink module, SSL-negotiate-byte handling, explicit flush
   after CopyInResponse. Reviewable independently — nothing in
   pg_lake_* changes here.
2. pg_lake: streaming writes for INSERT, UPDATE, DELETE, COPY,
   and iceberg metadata. Adds CSVStreamWriter plumbing in
   pg_lake_engine, then adopts it in the pg_lake_table /
   pg_lake_copy / iceberg-metadata callsites. Gated behind the
   new pg_lake_engine.streaming_writes GUC.

Patch 2 builds and runs cleanly against a tree with only patch 1
applied — though it doesn't do much without patch 1's server-side
support.

Compatibility

• Default off. Existing deployments are unaffected.
• No data layout or catalog changes. Iceberg tables produced
  via the streaming path are byte-identical to those produced via
  the file-based path. The streaming path is a transport choice,
  not a representation choice.
• No SQL surface changes. Same INSERT / UPDATE / DELETE /
  COPY / CREATE TABLE syntax.
• Existing file-based paths unchanged. No regressions for
  shared-filesystem topologies.
• Unix-socket transport remains supported. pgduck_server only
  starts a TCP listener if --listen_addresses is set; the
  Unix-socket default behavior is preserved.

Testing

Tested against a CDC-shape workload (200 mixed INSERT/UPDATE/DELETE
events with COMMIT per event, ~10s per event, ~50 minutes of
continuous write activity) on a GKE deployment with pgduck_server
in a separate pod from postgres. Final verification: 10 back-to-back
soak runs with full instrumentation across the writer's entire
lifetime — zero correctness issues detected.

The companion cdc-stream.sql / cdc-stream-polaris.sql regression
fixtures in our local stack exercise the I+D+U mix that the
streaming path needs to handle correctly. Happy to upstream those
test fixtures separately if useful for CI.

Operational notes

• The RECEIVE query prefix is only recognized in libpq's simple-
  query protocol (because it needs the server-side prefix detection
  that the extended-query protocol doesn't surface). Clients using
  PQsendQueryParams keep going through the existing path.
• This PR does not add a per-stream byte cap on sink size;
  disk usage is currently bounded only by the underlying filesystem.
  Adding a --recv_max_bytes cap would be a small follow-up.
• WaitForResult is used (not raw PQgetResult) so that
  statement_timeout / SIGINT / postmaster-death fire promptly
  while the backend is waiting on pgduck — typical libpq calls
  don't observe statement_timeout because they sit below
  CHECK_FOR_INTERRUPTS.

[bkanuka]

Great job wrangling this into 2 commits! I've been following your progress because I'm also really interested in running this in Kubernetes. This is so much more readable - but it still looks like WAY more work separating these than I expected. Good job 👏🏼

[sfc-gh-mslot]

I think the general concept makes sense, for TRANSMIT to have a RECEIVE counter-part. I previously tried something like this where we read from the socket via a special DuckDB FileSystem, though the problem was that we lose parallelism and this was kind of slow. Using an intermediate file is probably better.

One high-level question though: It seems part of the goal is to simplify deployment on Kubernetes. If Postgres could start pgduck_server as a background process in the same container and funnel logs into the Postgres log, would that help?

[sfc-gh-mslot]

these are not included

[timmclaughlin]

One high-level question though: It seems part of the goal is to simplify deployment on Kubernetes. If Postgres could start pgduck_server as a background process in the same container and funnel logs into the Postgres log, would that help?

It doesn't help that much for our k8s use case because we want redundant PG pods for HA heap storage and stateless decoupled duckdb compute pod(s) that can be scaled up or down through a LB behind those PG pods. In our case we have 2x PG pods sharing a single duckdb pod but it could be 3x PG and 10x Duckdb depending on resource needs. I messed with doing it in the same pod or even the same node but liked it less and this was just more flexible.

[sfc-gh-mslot]

Alright, it makes sense. I think this has legs, but will need a bit of time to review the details.

Would you mind including the missing files, rebasing, and doing the DCO thing mentioned when opening the PR? (commit -s)

[timmclaughlin]

Yeah straightening out the PR. I built it as patches so I can actually use it knowing that it’d take a while for a review if ever. To also be up front (because some will be annoyed) it is mostly Claude with critique from GPT and me as architect :). So don’t get your hopes up for super tight code.

…

On Thu, May 14, 2026 at 3:25 AM Marco Slot ***@***.***> wrote: *sfc-gh-mslot* left a comment (Snowflake-Labs/pg_lake#345) <#345 (comment)> Alright, it makes sense. I think this has legs, but will need a bit of time to review the details. Would you mind including the missing files, rebasing, and doing the DCO thing mentioned when opening the PR? (commit -s) — Reply to this email directly, view it on GitHub <#345 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA2EKVU7IEWG4ENIO2MU53T42VYGLAVCNFSM6AAAAACYYGMYS2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DINBYGYYDEOJXHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[timmclaughlin]

@sfc-gh-mslot it should all be up with that last push. seems to be working well enough for us in our usage so far. let me know if you see any issues.