Conversation
_write_file did three passes over every chunk where one will do:
output.stdout.gsub('\n\r', "") # full copy of the base64 payload
Base64.decode64(contents) # decode
out[0, out.length - 1] # another full copy to drop one byte
The gsub is redundant twice over. Single-quoted '\n\r' is the four-character
sequence backslash-n-backslash-r, which cannot appear in base64 output because
backslash is not in the base64 alphabet; and base64 decoding already discards
any byte outside that alphabet, so even a real CR/LF would not matter. It
never matched, and allocated a copy of the whole payload every chunk to prove
it.
The trailing-NUL strip only removes the single byte that download.ps1.erb's
$chunk[0..$bytesRead] over-reads, so delete_suffix! does it in place instead of
copying the decoded chunk.
This also drops the gem's only Base64 reference. Nothing in chef-winrm-fs ever
required "base64" -- the constant resolved only because something else in the
dependency tree happened to load it. String#unpack1("m") is what
Base64.decode64 calls anyway, so the unrequired dependency goes away.
Verified identical output for: empty payload, 1 byte, 1 byte plus the NUL
over-read, all 256 byte values, data legitimately ending in one or two NULs,
embedded NULs mid-stream, 1 MiB binary, text containing CRLF, and line-wrapped
base64. A full 32 MB download against a stub host is byte-identical to the
source before and after.
per 1 MiB chunk, 300 iterations (Ruby 4.0.6, macOS arm64):
before 523.5 ms
after 264.8 ms 1.98x faster
transient String bytes per 1 MiB chunk: 2.00 MB -> 1.00 MB
Signed-off-by: Tim Smith <tsmith84@proton.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_write_fileruns on every downloaded chunk and makes three passes over it where one will do:The
gsubis redundant twice over. It is single-quoted, so the pattern is the four-character sequence\,n,\,r— not CRLF. That sequence cannot appear in base64 output, because backslash is not in the base64 alphabet:And even if it could, base64 decoding already discards every byte outside the alphabet:
So it never matched — it just allocated a copy of the entire payload on every chunk to establish that.
The trailing-NUL strip removes exactly the one byte that
download.ps1.erb's$chunk[0..$bytesRead]over-reads, sodelete_suffix!does it in place rather than copying the whole decoded chunk to drop its last byte.Side effect: removes an undeclared dependency
This drops the gem's only
Base64reference. Nothing inchef-winrm-fsever requiredbase64—file_manager.rbused the constant and it resolved only because something else in the dependency tree happened to load it first.String#unpack1("m")is precisely whatBase64.decode64calls internally, so the unrequired dependency disappears rather than needing a newrequireadded. Worth knowing givenbase64became a bundled gem in Ruby 3.4.Verification
Output is identical for every case I could think of to break it:
pack("m"))End-to-end, a 32 MB download against a stub host that serves a real file over the same protocol
download.ps1.erbspeaks is byte-identical to the source before and after.Results
Per 1 MiB chunk, 300 iterations (Ruby 4.0.6, macOS arm64):
Transient String bytes per 1 MiB chunk drop from 2.00 MB to 1.00 MB (measured with
ObjectSpace.memsize_of_all(String)underGC.disable; less than the naive 3x you'd expect from the source, because Ruby servesgsub-with-no-match and the leading substring as copy-on-write shared strings).Local CPU time for a full 32 MB download drops from 99.4 ms to 81.5 ms. That is small next to the network, but it is per-chunk work on the client during a transfer that is already CPU-heavy from base64.
Note: #19 also touches
file_manager.rb(different methods —download/ps_run/download_dirvs_write_file) and addsspec/unit/file_manager_spec.rb. They should merge cleanly in either order; happy to rebase whichever lands second.