Skip to content

Run OpenSSL's EVP vectors through the aws-lc-provider in CI - #3489

Open
geedo0 wants to merge 1 commit into
feat/provider-errorsfrom
feat/ossl-evp-test
Open

Run OpenSSL's EVP vectors through the aws-lc-provider in CI#3489
geedo0 wants to merge 1 commit into
feat/provider-errorsfrom
feat/ossl-evp-test

Conversation

@geedo0

@geedo0 geedo0 commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Context and motivation

Everything the provider is checked against so far is a test this project wrote. OpenSSL ships its own
known-answer corpus and the evp_test driver that consumes it, so the provider can be held to the same
vectors OpenSSL holds itself to, over the real 3.x fetch path with both providers loaded and awslc
preferred.

Description of changes

provider/test/run_evp_vectors.sh drives the OpenSSL source tree's evp_test binary over its
stock data files, using the provider's own test config.

Per data file it asserts a case count, a skip count, and a fetch count per algorithm name, not merely that
the run exited 0. That distinction is the point: a case whose init fails abandons the rest of the file, so
0 errors also describes a run that stopped after the first case. A skipped case is coverage lost rather
than a wrong answer, so it is named separately and also fails the file. Algorithms the provider does not
back are left unconstrained, because they are the default provider's to serve.

One row today: evpmd_sha.txt, 74 cases, 0 skips, with the fetch counts for the six SHA-2 names pinned. A
family arriving later adds rows and moves counts from unclaimed to claimed rather than vendoring vectors.

Curated files rather than the whole corpus. This avoids pulling in large amounts of coverage which AWS-LC does not support and the associated maintenance liabilities.

CI. tests/ci/run_aws_lc_provider_tests.sh runs the vectors after the two unit suites and the linkage
checks. It tees the runner's output and requires the runner's final summary line, because exit 0 alone does
not show the runner reached the end; a runner that returned early would look identical.

Testing

The change is a test gate. It is green on both Linux axes, FIPS and non-FIPS, at 74 cases, 0 errors, 0
skips, with every pinned fetch count matching, run alongside the unit suites in the same job.

Control run in place: perturbing one expectation (SHA512-224 at 6 rather than its actual 7) exits 1 with
SHA512-224 fetched 7 times, expected 6. An earlier attempt at that control was invalid because it copied
the script to /tmp, which broke its self-location of provider.cnf, and the script refused rather than
running.

Review considerations

  • There is no way to strongly require AWS-LC as the provider for this test suite without breaking the test suite. In other words, if we set AWS-LC as required, the test suite will error out on the first algorithm/vector that is unsupported. There is no easy way to filter out unsupported algorithms.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache
2.0 license and the ISC license.

@geedo0
geedo0 requested a review from a team as a code owner September 4, 2026 12:28
@codecov-commenter

codecov-commenter commented Sep 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.14%. Comparing base (fc6f113) to head (3ac88c5).

Additional details and impacted files
@@                   Coverage Diff                    @@
##           feat/provider-errors    #3489      +/-   ##
========================================================
- Coverage                 78.34%   78.14%   -0.20%     
========================================================
  Files                       700      700              
  Lines                    125598   125595       -3     
  Branches                  17378    17370       -8     
========================================================
- Hits                      98396    98144     -252     
- Misses                    26331    26580     +249     
  Partials                    871      871              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@geedo0
geedo0 force-pushed the feat/ossl-evp-test branch from e8feabc to 147ccd0 Compare September 4, 2026 20:00
Add provider/test/run_evp_vectors.sh, which drives OpenSSL's own known-answer
corpus through the provider using the pinned tree's evp_test binary over its
stock data files, and wire it into tests/ci/run_aws_lc_provider_tests.sh after
the unit suites.

evp_test reports no per-case attribution, so nothing here establishes which
provider served a given vector; the unit suites' reachability inventory and
config-route test own that.
@geedo0
geedo0 force-pushed the feat/ossl-evp-test branch from 147ccd0 to 3ac88c5 Compare September 4, 2026 20:12
@justsmth
justsmth self-requested a review September 8, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants