Run OpenSSL's EVP vectors through the aws-lc-provider in CI - #3489
Open
geedo0 wants to merge 1 commit into
Open
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## feat/provider-errors #3489 +/- ##
========================================================
- Coverage 78.34% 78.14% -0.20%
========================================================
Files 700 700
Lines 125598 125595 -3
Branches 17378 17370 -8
========================================================
- Hits 98396 98144 -252
- Misses 26331 26580 +249
Partials 871 871 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
geedo0
force-pushed
the
feat/ossl-evp-test
branch
from
September 4, 2026 20:00
e8feabc to
147ccd0
Compare
Add provider/test/run_evp_vectors.sh, which drives OpenSSL's own known-answer corpus through the provider using the pinned tree's evp_test binary over its stock data files, and wire it into tests/ci/run_aws_lc_provider_tests.sh after the unit suites. evp_test reports no per-case attribution, so nothing here establishes which provider served a given vector; the unit suites' reachability inventory and config-route test own that.
geedo0
force-pushed
the
feat/ossl-evp-test
branch
from
September 4, 2026 20:12
147ccd0 to
3ac88c5
Compare
justsmth
self-requested a review
September 8, 2026 16:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context and motivation
Everything the provider is checked against so far is a test this project wrote. OpenSSL ships its own
known-answer corpus and the
evp_testdriver that consumes it, so the provider can be held to the samevectors OpenSSL holds itself to, over the real 3.x fetch path with both providers loaded and
awslcpreferred.
Description of changes
provider/test/run_evp_vectors.shdrives the OpenSSL source tree'sevp_testbinary over itsstock data files, using the provider's own test config.
Per data file it asserts a case count, a skip count, and a fetch count per algorithm name, not merely that
the run exited 0. That distinction is the point: a case whose init fails abandons the rest of the file, so
0 errorsalso describes a run that stopped after the first case. A skipped case is coverage lost ratherthan a wrong answer, so it is named separately and also fails the file. Algorithms the provider does not
back are left unconstrained, because they are the default provider's to serve.
One row today:
evpmd_sha.txt, 74 cases, 0 skips, with the fetch counts for the six SHA-2 names pinned. Afamily arriving later adds rows and moves counts from unclaimed to claimed rather than vendoring vectors.
Curated files rather than the whole corpus. This avoids pulling in large amounts of coverage which AWS-LC does not support and the associated maintenance liabilities.
CI.
tests/ci/run_aws_lc_provider_tests.shruns the vectors after the two unit suites and the linkagechecks. It tees the runner's output and requires the runner's final summary line, because exit 0 alone does
not show the runner reached the end; a runner that returned early would look identical.
Testing
The change is a test gate. It is green on both Linux axes, FIPS and non-FIPS, at 74 cases, 0 errors, 0
skips, with every pinned fetch count matching, run alongside the unit suites in the same job.
Control run in place: perturbing one expectation (
SHA512-224at 6 rather than its actual 7) exits 1 withSHA512-224 fetched 7 times, expected 6. An earlier attempt at that control was invalid because it copiedthe script to
/tmp, which broke its self-location ofprovider.cnf, and the script refused rather thanrunning.
Review considerations
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache
2.0 license and the ISC license.