diff --git a/site/docs/commandline.md b/site/docs/commandline.md index f4474cc..d95327b 100644 --- a/site/docs/commandline.md +++ b/site/docs/commandline.md @@ -76,6 +76,8 @@ compression options: -N disable small string-matching compression -D disable external decompression (encode/decode) -R disable external recompression (decode) + -G omit detected compression level from app-header + (encode; emits a legacy header older versions read) -n disable checksum (encode/decode) -a disable armor (whole-file BLAKE3 verification, on by default; requires a seekable source) @@ -106,6 +108,14 @@ with a clear message. It is **on by default** and requires a seekable (regular) source file. Pass `-a` to disable it and restore the legacy streaming behavior. See [Armor mode](armor.md). +### `-G` compression level in the application header + +When xdelta3 auto-decompresses an externally compressed input, it records the +detected compression level in the application header so decode can recompress +the output to the same level. Pass `-G` to omit the level and emit a legacy +header that older xdelta3 versions can still recompress. See +[External compression](external-compression.md). + ### `-A` application header The `-A` flag sets application-specific data in the VCDIFF header (view it with diff --git a/site/docs/external-compression.md b/site/docs/external-compression.md index c603741..132f993 100644 --- a/site/docs/external-compression.md +++ b/site/docs/external-compression.md @@ -15,13 +15,37 @@ external compression to the output. Xdelta decompresses the inputs by piping them through the external compression program. Recognition of externally-compressed inputs can be disabled with `-D`. -External compression has a well-known pitfall: xdelta3 does not know the -original compression settings, so when re-applying the external compression it -uses default settings, which may produce different compressed bytes and break -checksum verification of the compressed data. External recompression of the -output can be disabled with `-R`. If you know the settings needed to reproduce -the exact output (for example `gzip -9`), set the corresponding environment -variable to control the external command (for example `export GZIP=-9`). +External compression has a well-known pitfall: the *exact* compressed bytes +depend on the compression program, its version, and its settings, so +recompressing the decoded output may not reproduce the original file byte for +byte. To narrow this gap, xdelta3 detects the original compression **level** +from the input's header at encode time and records it in the application header, +then passes it back to the compressor on decode: + +- **bzip2** — the block-size level (`1`–`9`) is recovered exactly. +- **gzip** and **xz** — the level is a best-effort guess from the stream header + and may fall back to the format default when it cannot be determined exactly. + +The level is stored by appending a digit to the single-character compressor +identifier in the application header (for example `B` becomes `B9`); view it with +`xdelta3 printhdr`. Detection only narrows the level, not other version- or +program-specific differences, so byte-identical recompression is reliable for +bzip2 and best-effort for gzip/xz. + +External recompression of the output can be disabled entirely with `-R`. If you +know the settings needed to reproduce the exact output (for example `gzip -9`), +you can also set the corresponding environment variable to control the external +command (for example `export GZIP=-9`). + +## Compatibility: `-G` + +Recording the level changes the application-header compressor field (`B` → +`B9`). Older xdelta3 versions match the compressor identifier exactly and do not +recognize the trailing level digit, so they decline to recompress such a delta's +output (it is left decompressed, with a warning). Decoding the patch data itself +is unaffected. Pass `-G` at encode time to emit a legacy application header +(the bare identifier, no level) that older versions can still recompress. Deltas +produced by older versions decode unchanged. ```sh gzip release-1.tar diff --git a/xdelta3/CMakeLists.txt b/xdelta3/CMakeLists.txt index c18fba0..b01c906 100644 --- a/xdelta3/CMakeLists.txt +++ b/xdelta3/CMakeLists.txt @@ -321,6 +321,18 @@ if(XD3_BUILD_TESTS) add_test(NAME xdelta3_builtin_test COMMAND xdelta3 test) set_tests_properties(xdelta3_builtin_test PROPERTIES TIMEOUT 600) + # External-compression level detection round-trip. Only meaningful where the + # POSIX external-compression path is compiled (i.e. not the Windows + # decode-only target); the script itself SKIPs when bzip2 is unavailable. + if(NOT WIN32) + add_test(NAME xdelta3_recompress_level_test + COMMAND ${CMAKE_COMMAND} -E env + sh ${CMAKE_CURRENT_SOURCE_DIR}/testing/recompress_level_test.sh + $) + set_tests_properties(xdelta3_recompress_level_test + PROPERTIES TIMEOUT 120) + endif() + # Consumer smoke test: links the core library via its public API and runs a # real encode/decode round-trip, verifying the library is usable as built. if(XD3_BUILD_LIB) diff --git a/xdelta3/testing/recompress_level_test.sh b/xdelta3/testing/recompress_level_test.sh new file mode 100755 index 0000000..939842a --- /dev/null +++ b/xdelta3/testing/recompress_level_test.sh @@ -0,0 +1,113 @@ +#!/bin/sh +# +# Regression test for external-compression level detection (see +# main_detect_level_* and main_recompress_output in xdelta3-main.h). +# +# When xdelta3 automatically decompresses a compressed input and later +# automatically recompresses the decoded output, it must reproduce the +# *original* compression level so the recompressed bytes match the original +# compressed file. The level is detected from the input header at encode time, +# carried in the VCDIFF application header as "" (e.g. "B9"), and +# passed back to the recompressor at decode time. +# +# bzip2 is used because its output is fully deterministic (no timestamp, +# filename, or OS fields) and its block-size level (1..9) is exactly recoverable +# from the "BZh" header, so a byte-identical round-trip is a reliable check +# across environments. Without level detection, the level-1 and level-5 cases +# would be recompressed at bzip2's default level and would not match. +# +# Usage: recompress_level_test.sh [path-to-xdelta3] + +set -eu + +XD=${1:-} +if [ -z "$XD" ]; then + for cand in ./build/xdelta3 ./xdelta3 build/xdelta3; do + if [ -x "$cand" ]; then XD=$cand; break; fi + done +fi +if [ -z "$XD" ] || [ ! -x "$XD" ]; then + echo "recompress_level_test: cannot find xdelta3 binary (pass it as \$1)" >&2 + exit 2 +fi + +skip() { + echo "recompress_level_test: SKIP: $1" + exit 0 +} + +if ! command -v bzip2 >/dev/null 2>&1; then + skip "bzip2 not available" +fi + +WORK=$(mktemp -d "${TMPDIR:-/tmp}/xd3-recomp.XXXXXX") +trap 'rm -rf "$WORK"' EXIT INT TERM + +# --- Build deterministic inputs ---------------------------------------------- +# A base file and a target that differs from it, both compressible. The exact +# contents do not matter as long as they are reproducible within the run. +i=0 +while [ "$i" -lt 4000 ]; do + printf 'xdelta3 recompress level regression line %d\n' "$i" + i=$((i + 1)) +done > "$WORK/base.bin" +cp "$WORK/base.bin" "$WORK/target.bin" +printf 'appended changes that make the target differ from the base\n' \ + >> "$WORK/target.bin" + +fail=0 +for L in 1 5 9; do + bzip2 -"$L" -c "$WORK/base.bin" > "$WORK/base.bz2" + bzip2 -"$L" -c "$WORK/target.bin" > "$WORK/target.bz2" + + # Encode a delta from the compressed source to the compressed target. The + # encoder transparently decompresses both and records the level. + "$XD" -f -e -s "$WORK/base.bz2" "$WORK/target.bz2" "$WORK/delta.xd3" \ + 2>"$WORK/enc.log" + + # Decode: xdelta3 must recompress the output with bzip2 at the same level. + "$XD" -f -d -s "$WORK/base.bz2" "$WORK/delta.xd3" "$WORK/out.bz2" \ + 2>"$WORK/dec.log" + + if cmp -s "$WORK/target.bz2" "$WORK/out.bz2"; then + echo "recompress_level_test: level $L byte-identical OK" + else + echo "recompress_level_test: level $L MISMATCH (recompressed output differs)" >&2 + fail=1 + fi +done + +# --- -G opt-out: legacy header, default-level recompression ----------------- +# bzip2 defaults to -9, so a level-1 input encoded with -G must NOT round-trip +# byte-identically (the level was intentionally dropped), yet the decoded +# content must still be correct and decodable by older xdelta3 versions. +bzip2 -1 -c "$WORK/base.bin" > "$WORK/base.bz2" +bzip2 -1 -c "$WORK/target.bin" > "$WORK/target.bz2" + +"$XD" -G -f -e -s "$WORK/base.bz2" "$WORK/target.bz2" "$WORK/deltaG.xd3" \ + 2>"$WORK/encG.log" +"$XD" -f -d -s "$WORK/base.bz2" "$WORK/deltaG.xd3" "$WORK/outG.bz2" \ + 2>"$WORK/decG.log" + +if cmp -s "$WORK/target.bz2" "$WORK/outG.bz2"; then + echo "recompress_level_test: -G unexpectedly preserved the level" >&2 + fail=1 +else + echo "recompress_level_test: -G dropped the level (legacy header) OK" +fi + +bunzip2 -c "$WORK/outG.bz2" > "$WORK/outG.bin" +if cmp -s "$WORK/target.bin" "$WORK/outG.bin"; then + echo "recompress_level_test: -G decoded content correct OK" +else + echo "recompress_level_test: -G decoded content WRONG" >&2 + fail=1 +fi + +if [ "$fail" -ne 0 ]; then + echo "recompress_level_test: FAIL" >&2 + exit 1 +fi + +echo "recompress_level_test: PASS" +exit 0 diff --git a/xdelta3/xdelta3-internal.h b/xdelta3/xdelta3-internal.h index 6a559d9..c5400a6 100644 --- a/xdelta3/xdelta3-internal.h +++ b/xdelta3/xdelta3-internal.h @@ -107,6 +107,10 @@ struct _main_file { const char *realname; /* File name or /dev/stdin, * /dev/stdout, /dev/stderr. */ const main_extcomp *compressor; /* External compression struct. */ + int compression_level; /* Detected external compression level + * (0..9), or -1 if unknown. Carried in the + * appheader so recompression reproduces the + * original bytes. */ int flags; /* RD_FIRST, RD_NONEXTERNAL, ... */ xoff_t nread; /* for input position */ xoff_t nwrite; /* for output position */ diff --git a/xdelta3/xdelta3-main.h b/xdelta3/xdelta3-main.h index 6886622..bbd1fba 100644 --- a/xdelta3/xdelta3-main.h +++ b/xdelta3/xdelta3-main.h @@ -222,6 +222,11 @@ struct _main_extcomp { const char *magic; usize_t magic_size; int flags; + + /* Optionally recover the compression level (0..9) from the first bytes of a + * compressed stream, returning -1 when it cannot be determined. NULL when + * the format carries no recoverable level. */ + int (*detect_level)(const uint8_t *data, usize_t len); }; /* Merge state: */ @@ -283,6 +288,10 @@ static int num_subprocs = 0; static int option_force2 = 0; static int option_decompress_inputs = 1; static int option_recompress_outputs = 1; +/* Embed the detected external-compression level in the app-header so the + * decoder reproduces it (on by default). -G disables it to emit a legacy + * app-header that older xdelta3 versions can still recompress. */ +static int option_use_comp_level = 1; #endif /* This is for comparing "printdelta" output without attention to @@ -332,15 +341,86 @@ static char armor_merge_prev_target[XD3_BLAKE3_HEXBUF]; static int armor_merge_have_last_target = 0; static char armor_merge_last_target[XD3_BLAKE3_HEXBUF]; #endif +/* Recover the bzip2 block-size level (1..9) from the "BZh" header. This is + * exact: the digit is part of the magic and maps directly to the -N flag. */ +static int main_detect_level_bzip2(const uint8_t *data, usize_t len) { + if (len < 4) { + return -1; + } + if (data[3] >= '1' && data[3] <= '9') { + return data[3] - '0'; + } + return -1; +} + +/* Best-effort recovery of the gzip level from the XFL byte (RFC 1952 + * sec 2.3.1). Only the extremes are encoded: 2 == best, 4 == fastest. Any + * other value maps to the default (6), which is correct for the common case but + * cannot recover intermediate levels exactly. */ +static int main_detect_level_gzip(const uint8_t *data, usize_t len) { + if (len < 9) { + return -1; + } + switch (data[8]) { + case 2: + return 9; + case 4: + return 1; + default: + return 6; + } +} + +/* Best-effort recovery of the xz preset from the LZMA2 dictionary-size byte in + * the filter-flags. The dictionary size maps to a preset only approximately: + * a couple of sizes are shared by two presets, in which case the more common + * (default-ish) preset is chosen. Returns -1 when no LZMA2 filter is found in + * the expected header range. */ +static int main_detect_level_xz(const uint8_t *data, usize_t len) { + usize_t offs; + /* The LZMA2 filter flags follow the stream and block headers; their position + * varies with optional fields, so scan a small window for the filter ID. */ + for (offs = 14; offs + 2 < len && offs < 26; offs += 1) { + if (data[offs] != 0x21 || data[offs + 1] != 0x01) { + continue; /* not the LZMA2 filter (ID 0x21, props size 1) */ + } + switch (data[offs + 2]) { + case 12: + return 0; + case 16: + return 1; + case 18: + return 2; + case 20: + return 3; /* shared with preset 4 */ + case 22: + return 6; /* shared with preset 5; 6 is the default */ + case 24: + return 7; + case 26: + return 8; + case 28: + return 9; + default: + return -1; + } + } + return -1; +} + /* This array of compressor types is compiled even if EXTERNAL_COMPRESSION is * false just so the program knows the mapping of IDENT->NAME. */ static main_extcomp extcomp_types[] = { - {"bzip2", "-c", "bzip2", "-dc", "B", "BZh", 3, 0}, - {"gzip", "-c", "gzip", "-dc", "G", "\037\213", 2, 0}, - {"compress", "-c", "uncompress", "-c", "Z", "\037\235", 2, 0}, + {"bzip2", "-c", "bzip2", "-dc", "B", "BZh", 3, 0, main_detect_level_bzip2}, + /* gzip -n suppresses the stored name/mtime so recompression is + * reproducible byte-for-byte. */ + {"gzip", "-cn", "gzip", "-dc", "G", "\037\213", 2, 0, + main_detect_level_gzip}, + {"compress", "-c", "uncompress", "-c", "Z", "\037\235", 2, 0, NULL}, /* Xz is lzma with a magic number http://tukaani.org/xz/format.html */ - {"xz", "-c", "xz", "-dc", "Y", "\xfd\x37\x7a\x58\x5a\x00", 2, 0}, + {"xz", "-c", "xz", "-dc", "Y", "\xfd\x37\x7a\x58\x5a\x00", 2, 0, + main_detect_level_xz}, }; static int main_input(xd3_cmd cmd, main_file *ifile, main_file *ofile, @@ -467,6 +547,7 @@ static void reset_defaults(void) { option_force2 = 0; option_decompress_inputs = 1; option_recompress_outputs = 1; + option_use_comp_level = 1; num_subprocs = 0; #endif #if VCDIFF_TOOLS @@ -795,6 +876,8 @@ static int main_atou(const char *arg, usize_t *uo, usize_t low, usize_t high, void main_file_init(main_file *xfile) { memset(xfile, 0, sizeof(*xfile)); + xfile->compression_level = -1; + #if XD3_POSIX xfile->file = -1; #endif @@ -2374,6 +2457,11 @@ static int main_secondary_decompress_check(main_file *file, uint8_t *input_buf, } if (decompressor != NULL) { + if (decompressor->detect_level != NULL) { + file->compression_level = + decompressor->detect_level(check_buf, (usize_t)check_nread); + } + if (!option_quiet) { XPR(NT "externally compressed input: %s %s%s < %s\n", decompressor->decomp_cmdname, decompressor->decomp_options, @@ -2445,6 +2533,22 @@ static int main_recompress_output(main_file *ofile) { /* The child runs the recompression process: */ if (recomp_id == 0) { + char level_arg[8]; + const char *argv[5]; + int argi = 0; + + argv[argi++] = recomp->recomp_cmdname; + argv[argi++] = recomp->recomp_options; + if (ofile->compression_level >= 0 && ofile->compression_level <= 9) { + snprintf_func(level_arg, sizeof(level_arg), "-%d", + ofile->compression_level); + argv[argi++] = level_arg; + } + if (option_force2) { + argv[argi++] = "-f"; + } + argv[argi] = NULL; + if (option_verbose > 2) { XPR(NT "external recompression pid %d\n", getpid()); } @@ -2453,8 +2557,7 @@ static int main_recompress_output(main_file *ofile) { if (dup2(XFNO(ofile), STDOUT_FILENO) < 0 || dup2(pipefd[PIPE_READ_FD], STDIN_FILENO) < 0 || close(pipefd[PIPE_READ_FD]) || close(pipefd[PIPE_WRITE_FD]) || - execlp(recomp->recomp_cmdname, recomp->recomp_cmdname, - recomp->recomp_options, option_force2 ? "-f" : NULL, NULL)) { + execvp(recomp->recomp_cmdname, (char *const *)argv)) { XPR(NT "child process %s failed to execute: %s\n", recomp->recomp_cmdname, xd3_mainerror(get_errno())); } @@ -2803,6 +2906,25 @@ static const char *main_apphead_string(const char *x) { return (y = strrchr(x, '/')) == NULL ? x : y + 1; } +/* Format the appheader compressor field as "" or "" (e.g. + * "B9") so the decoder can reproduce the original compression level. Returns a + * pointer into buf, or the bare ident / "" when no level is known. */ +static const char *main_apphead_comp(const main_extcomp *comp, int level, + char *buf, size_t bufsize) { + if (comp == NULL) { + return ""; + } + if (level >= 0 && level <= 9 +#if EXTERNAL_COMPRESSION + && option_use_comp_level +#endif + ) { + snprintf_func(buf, bufsize, "%s%d", comp->ident, level); + return buf; + } + return comp->ident; +} + static int main_set_appheader(xd3_stream *stream, main_file *input, main_file *sfile) { /* The user may disable the application header. Once the appheader @@ -2820,6 +2942,8 @@ static int main_set_appheader(xd3_stream *stream, main_file *input, const char *icomp; const char *sname; const char *scomp; + char icomp_buf[8]; + char scomp_buf[8]; usize_t len; #if XD3_ARMOR /* Armored name fields carry "name#<64hex>". Computed below when armor @@ -2834,12 +2958,14 @@ static int main_set_appheader(xd3_stream *stream, main_file *input, #endif iname = main_apphead_string(input->filename); - icomp = (input->compressor == NULL) ? "" : input->compressor->ident; + icomp = main_apphead_comp(input->compressor, input->compression_level, + icomp_buf, sizeof(icomp_buf)); len = (usize_t)strlen(iname) + (usize_t)strlen(icomp) + 2; if (sfile->filename != NULL) { sname = main_apphead_string(sfile->filename); - scomp = (sfile->compressor == NULL) ? "" : sfile->compressor->ident; + scomp = main_apphead_comp(sfile->compressor, sfile->compression_level, + scomp_buf, sizeof(scomp_buf)); len += (usize_t)strlen(sname) + (usize_t)strlen(scomp) + 2; } else { sname = scomp = ""; @@ -2943,10 +3069,27 @@ static void main_get_appheader_params(main_file *file, char **parsed, } } - /* Set the compressor, initiate de/recompression later. */ + /* Set the compressor, initiate de/recompression later. The compressor field + * is "" or "" (e.g. "B9"); split the trailing decimal + * level, if present, so recompression can reproduce the original level. */ if (file->compressor == NULL && *parsed[1] != 0) { + char ident[8]; + const char *comp = parsed[1]; + usize_t n = 0; + + while (comp[n] != 0 && !(comp[n] >= '0' && comp[n] <= '9') && + n + 1 < sizeof(ident)) { + ident[n] = comp[n]; + n += 1; + } + ident[n] = 0; + + if (comp[n] >= '0' && comp[n] <= '9') { + file->compression_level = comp[n] - '0'; + } + file->flags |= RD_DECOMPSET; - file->compressor = main_get_compressor(parsed[1]); + file->compressor = main_get_compressor(ident); } } @@ -3086,9 +3229,15 @@ static int main_open_output(xd3_stream *stream, main_file *ofile) { /* Do output recompression. */ if (ofile->compressor != NULL && option_recompress_outputs == 1) { if (!option_quiet) { - XPR(NT "externally compressed output: %s %s%s > %s\n", + char level_str[8]; + level_str[0] = 0; + if (ofile->compression_level >= 0 && ofile->compression_level <= 9) { + snprintf_func(level_str, sizeof(level_str), " -%d", + ofile->compression_level); + } + XPR(NT "externally compressed output: %s %s%s%s > %s\n", ofile->compressor->recomp_cmdname, ofile->compressor->recomp_options, - (option_force2 ? " -f" : ""), ofile->filename); + level_str, (option_force2 ? " -f" : ""), ofile->filename); } if ((ret = main_recompress_output(ofile))) { @@ -3792,7 +3941,7 @@ int main(int argc, char **argv) #endif { static const char *flags = - "0123456789acdefhnqvDFJNORVs:m:B:C:E:I:L:O:M:P:W:A::S::"; + "0123456789acdefhnqvDFGJNORVs:m:B:C:E:I:L:O:M:P:W:A::S::"; xd3_cmd cmd; main_file ifile; main_file ofile; @@ -4088,6 +4237,16 @@ int main(int argc, char **argv) } #else option_recompress_outputs = 0; +#endif + break; + case 'G': +#if EXTERNAL_COMPRESSION == 0 + if (option_verbose > 0) { + XPR(NT "warning: -G option ignored, " + "external compression support was not compiled\n"); + } +#else + option_use_comp_level = 0; #endif break; case 's': @@ -4296,6 +4455,9 @@ static int main_help(void) { XPR(NTR " -N disable small string-matching compression\n"); XPR(NTR " -D disable external decompression (encode/decode)\n"); XPR(NTR " -R disable external recompression (decode)\n"); + XPR(NTR " -G omit detected compression level from app-header\n"); + XPR(NTR + " (encode; emits a legacy header older versions read)\n"); XPR(NTR " -n disable checksum (encode/decode)\n"); #if XD3_ARMOR XPR(NTR " -a disable armor (whole-file BLAKE3 verification,\n"); diff --git a/xdelta3/xdelta3.1 b/xdelta3/xdelta3.1 index 064d131..1799d3b 100644 --- a/xdelta3/xdelta3.1 +++ b/xdelta3/xdelta3.1 @@ -104,6 +104,16 @@ disable external decompression (encode/decode) .BI \-R disable external recompression (decode) .TP +.BI \-G +omit the detected external-compression level from the application header +(encode). By default +.B xdelta3 +records the level so decode recompresses the output to byte-identical bytes; +.B \-G +emits a legacy application header that older +.B xdelta3 +versions can still recompress. +.TP .BI \-n disable checksum (encode/decode) .TP @@ -156,7 +166,30 @@ patch is already applied and nothing was done. any other error. .SH NOTES -The +When +.B xdelta3 +automatically decompresses an externally compressed input (bzip2, gzip, xz, or +compress) it records the detected compression level in the application header +and passes it to the recompressor on decode, so the recompressed output +reproduces the original compression level. The bzip2 block-size level is +recovered exactly; the gzip and xz levels are best-effort guesses from the +stream header and may select the format default when the exact level cannot be +determined. This makes the application-header compressor field +.RB ( B , +.BR G , +.BR Z , +.BR Y ) +optionally carry a trailing level digit, e.g.\& +.IR B9 . +Use +.B \-R +to disable recompression, +.B \-D +to disable decompression entirely, or +.B \-G +to omit the level from the header for compatibility with older versions. +.PP +The .B XDELTA environment variable may contain extra args: