Skip to content

Decode download chunks without the redundant scrub and copy - #20

Open
tas50 wants to merge 1 commit into
chef:mainfrom
tas50:perf/write-file-decode
Open

tas50 wants to merge 1 commit into
chef:mainfrom
tas50:perf/write-file-decode

Conversation

@tas50

@tas50 tas50 commented Aug 28, 2026

Copy link
Copy Markdown

Summary

_write_file runs on every downloaded chunk and makes three passes over it where one will do:

contents = output.stdout.gsub('\n\r', "")      # full copy of the base64 payload
out = Base64.decode64(contents)                # decode
out = out[0, out.length - 1] if out.end_with? "\x00"   # another copy, to drop one byte

The gsub is redundant twice over. It is single-quoted, so the pattern is the four-character sequence \,n,\,r — not CRLF. That sequence cannot appear in base64 output, because backslash is not in the base64 alphabet:

b64.include?('\n\r')   #=> false

And even if it could, base64 decoding already discards every byte outside the alphabet:

Base64.decode64("QUJD\n\r=\t QUJD") == Base64.decode64("QUJDQUJD")   #=> true

So it never matched — it just allocated a copy of the entire payload on every chunk to establish that.

The trailing-NUL strip removes exactly the one byte that download.ps1.erb's $chunk[0..$bytesRead] over-reads, so delete_suffix! does it in place rather than copying the whole decoded chunk to drop its last byte.

Side effect: removes an undeclared dependency

This drops the gem's only Base64 reference. Nothing in chef-winrm-fs ever required base64file_manager.rb used the constant and it resolved only because something else in the dependency tree happened to load it first. String#unpack1("m") is precisely what Base64.decode64 calls internally, so the unrequired dependency disappears rather than needing a new require added. Worth knowing given base64 became a bundled gem in Ruby 3.4.

Verification

Output is identical for every case I could think of to break it:

input identical
empty payload yes
1 byte yes
1 byte + the NUL over-read yes
all 256 byte values yes
all 256 byte values + NUL over-read yes
data legitimately ending in one NUL yes
data ending in two NULs yes
embedded NULs mid-stream yes
1 MiB binary + NUL over-read yes
text containing CRLF yes
line-wrapped base64 (pack("m")) yes

End-to-end, a 32 MB download against a stub host that serves a real file over the same protocol download.ps1.erb speaks is byte-identical to the source before and after.

bundle exec rspec spec/unit                          # 3 examples, 0 failures
bundle exec cookstyle --chefstyle -c .rubocop.yml    # 16 files, no offenses

Results

Per 1 MiB chunk, 300 iterations (Ruby 4.0.6, macOS arm64):

                            user     system      total        real
_write_file (before)    0.521091   0.002282   0.523373 (  0.523541)
_write_file (after)     0.263882   0.000754   0.264636 (  0.264794)

1.98x faster

Transient String bytes per 1 MiB chunk drop from 2.00 MB to 1.00 MB (measured with ObjectSpace.memsize_of_all(String) under GC.disable; less than the naive 3x you'd expect from the source, because Ruby serves gsub-with-no-match and the leading substring as copy-on-write shared strings).

Local CPU time for a full 32 MB download drops from 99.4 ms to 81.5 ms. That is small next to the network, but it is per-chunk work on the client during a transfer that is already CPU-heavy from base64.


Note: #19 also touches file_manager.rb (different methods — download/ps_run/download_dir vs _write_file) and adds spec/unit/file_manager_spec.rb. They should merge cleanly in either order; happy to rebase whichever lands second.

_write_file did three passes over every chunk where one will do:

  output.stdout.gsub('\n\r', "")   # full copy of the base64 payload
  Base64.decode64(contents)        # decode
  out[0, out.length - 1]           # another full copy to drop one byte

The gsub is redundant twice over. Single-quoted '\n\r' is the four-character
sequence backslash-n-backslash-r, which cannot appear in base64 output because
backslash is not in the base64 alphabet; and base64 decoding already discards
any byte outside that alphabet, so even a real CR/LF would not matter. It
never matched, and allocated a copy of the whole payload every chunk to prove
it.

The trailing-NUL strip only removes the single byte that download.ps1.erb's
$chunk[0..$bytesRead] over-reads, so delete_suffix! does it in place instead of
copying the decoded chunk.

This also drops the gem's only Base64 reference. Nothing in chef-winrm-fs ever
required "base64" -- the constant resolved only because something else in the
dependency tree happened to load it. String#unpack1("m") is what
Base64.decode64 calls anyway, so the unrequired dependency goes away.

Verified identical output for: empty payload, 1 byte, 1 byte plus the NUL
over-read, all 256 byte values, data legitimately ending in one or two NULs,
embedded NULs mid-stream, 1 MiB binary, text containing CRLF, and line-wrapped
base64. A full 32 MB download against a stub host is byte-identical to the
source before and after.

    per 1 MiB chunk, 300 iterations (Ruby 4.0.6, macOS arm64):
      before  523.5 ms
      after   264.8 ms      1.98x faster

    transient String bytes per 1 MiB chunk: 2.00 MB -> 1.00 MB

Signed-off-by: Tim Smith <tsmith84@proton.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant