Detect and reproduce external-compression level

site/docs/commandline.md

@@ -76,6 +76,8 @@ compression options:
    -N           disable small string-matching compression
    -D           disable external decompression (encode/decode)
    -R           disable external recompression (decode)
+   -G           omit detected compression level from app-header
+                (encode; emits a legacy header older versions read)
    -n           disable checksum (encode/decode)
    -a           disable armor (whole-file BLAKE3 verification,
                 on by default; requires a seekable source)
@@ -106,6 +108,14 @@ with a clear message. It is **on by default** and requires a seekable
 (regular) source file. Pass `-a` to disable it and restore the legacy
 streaming behavior. See [Armor mode](armor.md).
 
+### `-G` compression level in the application header
+
+When xdelta3 auto-decompresses an externally compressed input, it records the
+detected compression level in the application header so decode can recompress
+the output to the same level. Pass `-G` to omit the level and emit a legacy
+header that older xdelta3 versions can still recompress. See
+[External compression](external-compression.md).
+
 ### `-A` application header
 
 The `-A` flag sets application-specific data in the VCDIFF header (view it with

site/docs/external-compression.md

@@ -15,13 +15,37 @@ external compression to the output.
 Xdelta decompresses the inputs by piping them through the external compression
 program. Recognition of externally-compressed inputs can be disabled with `-D`.
 
-External compression has a well-known pitfall: xdelta3 does not know the
-original compression settings, so when re-applying the external compression it
-uses default settings, which may produce different compressed bytes and break
-checksum verification of the compressed data. External recompression of the
-output can be disabled with `-R`. If you know the settings needed to reproduce
-the exact output (for example `gzip -9`), set the corresponding environment
-variable to control the external command (for example `export GZIP=-9`).
+External compression has a well-known pitfall: the *exact* compressed bytes
+depend on the compression program, its version, and its settings, so
+recompressing the decoded output may not reproduce the original file byte for
+byte. To narrow this gap, xdelta3 detects the original compression **level**
+from the input's header at encode time and records it in the application header,
+then passes it back to the compressor on decode:
+
+- **bzip2** — the block-size level (`1`–`9`) is recovered exactly.
+- **gzip** and **xz** — the level is a best-effort guess from the stream header
+  and may fall back to the format default when it cannot be determined exactly.
+
+The level is stored by appending a digit to the single-character compressor
+identifier in the application header (for example `B` becomes `B9`); view it with
+`xdelta3 printhdr`. Detection only narrows the level, not other version- or
+program-specific differences, so byte-identical recompression is reliable for
+bzip2 and best-effort for gzip/xz.
+
+External recompression of the output can be disabled entirely with `-R`. If you
+know the settings needed to reproduce the exact output (for example `gzip -9`),
+you can also set the corresponding environment variable to control the external
+command (for example `export GZIP=-9`).
+
+## Compatibility: `-G`
+
+Recording the level changes the application-header compressor field (`B` →
+`B9`). Older xdelta3 versions match the compressor identifier exactly and do not
+recognize the trailing level digit, so they decline to recompress such a delta's
+output (it is left decompressed, with a warning). Decoding the patch data itself
+is unaffected. Pass `-G` at encode time to emit a legacy application header
+(the bare identifier, no level) that older versions can still recompress. Deltas
+produced by older versions decode unchanged.
 
 ```sh
 gzip release-1.tar

xdelta3/CMakeLists.txt

@@ -321,6 +321,18 @@ if(XD3_BUILD_TESTS)
   add_test(NAME xdelta3_builtin_test COMMAND xdelta3 test)
   set_tests_properties(xdelta3_builtin_test PROPERTIES TIMEOUT 600)
 
+  # External-compression level detection round-trip.  Only meaningful where the
+  # POSIX external-compression path is compiled (i.e. not the Windows
+  # decode-only target); the script itself SKIPs when bzip2 is unavailable.
+  if(NOT WIN32)
+    add_test(NAME xdelta3_recompress_level_test
+      COMMAND ${CMAKE_COMMAND} -E env
+        sh ${CMAKE_CURRENT_SOURCE_DIR}/testing/recompress_level_test.sh
+        $<TARGET_FILE:xdelta3>)
+    set_tests_properties(xdelta3_recompress_level_test
+      PROPERTIES TIMEOUT 120)
+  endif()
+
   # Consumer smoke test: links the core library via its public API and runs a
   # real encode/decode round-trip, verifying the library is usable as built.
   if(XD3_BUILD_LIB)

xdelta3/testing/recompress_level_test.sh

@@ -0,0 +1,113 @@
+#!/bin/sh
+#
+# Regression test for external-compression level detection (see
+# main_detect_level_* and main_recompress_output in xdelta3-main.h).
+#
+# When xdelta3 automatically decompresses a compressed input and later
+# automatically recompresses the decoded output, it must reproduce the
+# *original* compression level so the recompressed bytes match the original
+# compressed file.  The level is detected from the input header at encode time,
+# carried in the VCDIFF application header as "<ident><level>" (e.g. "B9"), and
+# passed back to the recompressor at decode time.
+#
+# bzip2 is used because its output is fully deterministic (no timestamp,
+# filename, or OS fields) and its block-size level (1..9) is exactly recoverable
+# from the "BZh<n>" header, so a byte-identical round-trip is a reliable check
+# across environments.  Without level detection, the level-1 and level-5 cases
+# would be recompressed at bzip2's default level and would not match.
+#
+# Usage: recompress_level_test.sh [path-to-xdelta3]
+
+set -eu
+
+XD=${1:-}
+if [ -z "$XD" ]; then
+  for cand in ./build/xdelta3 ./xdelta3 build/xdelta3; do
+    if [ -x "$cand" ]; then XD=$cand; break; fi
+  done
+fi
+if [ -z "$XD" ] || [ ! -x "$XD" ]; then
+  echo "recompress_level_test: cannot find xdelta3 binary (pass it as \$1)" >&2
+  exit 2
+fi
+
+skip() {
+  echo "recompress_level_test: SKIP: $1"
+  exit 0
+}
+
+if ! command -v bzip2 >/dev/null 2>&1; then
+  skip "bzip2 not available"
+fi
+
+WORK=$(mktemp -d "${TMPDIR:-/tmp}/xd3-recomp.XXXXXX")
+trap 'rm -rf "$WORK"' EXIT INT TERM
+
+# --- Build deterministic inputs ----------------------------------------------
+# A base file and a target that differs from it, both compressible.  The exact
+# contents do not matter as long as they are reproducible within the run.
+i=0
+while [ "$i" -lt 4000 ]; do
+  printf 'xdelta3 recompress level regression line %d\n' "$i"
+  i=$((i + 1))
+done > "$WORK/base.bin"
+cp "$WORK/base.bin" "$WORK/target.bin"
+printf 'appended changes that make the target differ from the base\n' \
+  >> "$WORK/target.bin"
+
+fail=0
+for L in 1 5 9; do
+  bzip2 -"$L" -c "$WORK/base.bin"   > "$WORK/base.bz2"
+  bzip2 -"$L" -c "$WORK/target.bin" > "$WORK/target.bz2"
+
+  # Encode a delta from the compressed source to the compressed target.  The
+  # encoder transparently decompresses both and records the level.
+  "$XD" -f -e -s "$WORK/base.bz2" "$WORK/target.bz2" "$WORK/delta.xd3" \
+    2>"$WORK/enc.log"
+
+  # Decode: xdelta3 must recompress the output with bzip2 at the same level.
+  "$XD" -f -d -s "$WORK/base.bz2" "$WORK/delta.xd3" "$WORK/out.bz2" \
+    2>"$WORK/dec.log"
+
+  if cmp -s "$WORK/target.bz2" "$WORK/out.bz2"; then
+    echo "recompress_level_test: level $L byte-identical OK"
+  else
+    echo "recompress_level_test: level $L MISMATCH (recompressed output differs)" >&2
+    fail=1
+  fi
+done
+
+# --- -G opt-out: legacy header, default-level recompression -----------------
+# bzip2 defaults to -9, so a level-1 input encoded with -G must NOT round-trip
+# byte-identically (the level was intentionally dropped), yet the decoded
+# content must still be correct and decodable by older xdelta3 versions.
+bzip2 -1 -c "$WORK/base.bin"   > "$WORK/base.bz2"
+bzip2 -1 -c "$WORK/target.bin" > "$WORK/target.bz2"
+
+"$XD" -G -f -e -s "$WORK/base.bz2" "$WORK/target.bz2" "$WORK/deltaG.xd3" \
+  2>"$WORK/encG.log"
+"$XD" -f -d -s "$WORK/base.bz2" "$WORK/deltaG.xd3" "$WORK/outG.bz2" \
+  2>"$WORK/decG.log"
+
+if cmp -s "$WORK/target.bz2" "$WORK/outG.bz2"; then
+  echo "recompress_level_test: -G unexpectedly preserved the level" >&2
+  fail=1
+else
+  echo "recompress_level_test: -G dropped the level (legacy header) OK"
+fi
+
+bunzip2 -c "$WORK/outG.bz2" > "$WORK/outG.bin"
+if cmp -s "$WORK/target.bin" "$WORK/outG.bin"; then
+  echo "recompress_level_test: -G decoded content correct OK"
+else
+  echo "recompress_level_test: -G decoded content WRONG" >&2
+  fail=1
+fi
+
+if [ "$fail" -ne 0 ]; then
+  echo "recompress_level_test: FAIL" >&2
+  exit 1
+fi
+
+echo "recompress_level_test: PASS"
+exit 0

xdelta3/xdelta3-internal.h

@@ -107,6 +107,10 @@ struct _main_file {
   const char *realname;           /* File name or /dev/stdin,
                                    * /dev/stdout, /dev/stderr. */
   const main_extcomp *compressor; /* External compression struct. */
+  int compression_level;          /* Detected external compression level
+                                   * (0..9), or -1 if unknown.  Carried in the
+                                   * appheader so recompression reproduces the
+                                   * original bytes. */
   int flags;                      /* RD_FIRST, RD_NONEXTERNAL, ... */
   xoff_t nread;                   /* for input position */
   xoff_t nwrite;                  /* for output position */