Add thresholded client-side compression for Redis payloads

[lan17]

Priority

P3 — Low — demand-driven Redis memory and bandwidth optimization; not required for a production trial.

Status

Deliberately punted. Keep this issue as a design record and revisit only when production payload-size observations show that Redis memory, bandwidth, replication traffic, or latency justify the added format and rollout complexity.

Problem

DialCache stores serializer output in Redis without compression. Large payloads therefore consume their full serialized size in application-to-Redis bandwidth, Redis memory, replication traffic, and read responses.

Compression should reduce both bandwidth and Redis memory while avoiding CPU overhead for small values. Redis must remain unaware of the compression format: Lua, the existing Redis frame, DialCacheRedisClient, and the node-redis and Valkey GLIDE adapters should continue treating serializer output as opaque string | Buffer data.

Applications own any hard payload-size policy through their serializers, Redis/deployment controls, and alerts. #36 was closed as not planned; compression is an optimization, not a replacement safety ceiling.

Related work:

• Add a repeatable performance and scale benchmark suite #35 owns repeatable performance baselines.
• Benchmark lower-copy Redis protocol alternatives for large payloads #37 owns lower-copy Redis protocol research.
• Add application value-schema versioning to cache identity #17 owns application value-schema identity and rolling-deployment isolation.

Confirmed design direction

• Perform compression in the DialCache client through a composable serializer wrapper or equivalent post-serialization helper, not in Redis Lua or a Redis module.
• Apply compression only to the Redis representation. Request-local and process-local values remain native.
• Measure the exact UTF-8 or Buffer byte length after the inner serializer runs.
• Attempt compression only at or above a configurable byte threshold.
• When compression support is enabled for a use case, always emit a binary private serializer payload. A small versioned client-owned header distinguishes identity/uncompressed bodies from compressed bodies; Redis does not interpret it.
• Preserve whether the inner serializer produced a string or Buffer before calling its load method.
• Keep compressed output only when it is smaller by the configured minimum savings after accounting for header overhead.
• Preserve fail-open behavior: encoding failures skip the Redis write, while invalid headers or decompression failures become refreshable Redis misses.

Initial rollout assumption

Simultaneous legacy and compression-format reads are not required initially. An unrecognized legacy payload may fail open as a miss and be refreshed.

The implementation and documentation must still call out mixed-fleet behavior: old and new processes using the same key can alternately reject and rewrite each other's values. Decide whether that churn is acceptable for controlled rollouts or whether compression format identity should participate in the application schema/key version from #17.

Open decisions

• Public API shape: an explicit withCompression(serializer, options) helper, a Redis-level policy applied after every selected serializer, or both.
• First supported codec and dependency policy.
• Default threshold, minimum savings, and whether compression is opt-in with no defaults.
• Exact private header layout, codec/version allocation, and decoded-length representation.
• How to expose both original serialized bytes and final stored bytes, plus compression latency and bounded outcomes, through the backend-neutral metrics contract.
• Defensive decompression-output validation as format safety without introducing a universal cache-value ceiling.
• Whether compression configuration participates directly in cache identity or relies on the schema-version mechanism.

Acceptance criteria

• Add a composable, typed compression path that works with the default JSON serializer and arbitrary custom serializers.
• Keep compression metadata entirely inside the client-owned serializer payload; do not require Redis-side compression or interpretation.
• Measure thresholds using exact post-serialization bytes, including multibyte UTF-8 strings and Buffers.
• Emit an unambiguous versioned binary header for both identity and compressed bodies.
• Restore the inner serializer's original string/Buffer representation on load.
• Skip compression below the threshold and for payloads that do not meet the minimum savings requirement.
• Reject malformed or unsafe decompression output before passing data to the inner serializer.
• Preserve DialCache's fail-open read/write behavior and bounded-cardinality observability.
• Cover exact threshold boundaries, compressible and incompressible data, empty and binary payloads, malformed/truncated/unknown headers, codec failures, tracked and untracked Redis paths, and both bundled Redis adapters.
• Benchmark representative payload sizes and codecs under supported Node versions before choosing defaults; report latency, CPU/worker-pool impact, compression ratio, Redis memory, and wire bytes.
• Document rollout, rollback, mixed-fleet churn, application-owned payload policy, and application schema identity.

Out of scope

• Redis Lua or Redis Function compression.
• A custom Redis module or server deployment requirement.
• Compression of request-local or process-local values.
• Changing the current Redis wire frame solely to expose compression metadata.
• A universal DialCache payload-size policy.