Honor restricted PSS min saltlen for AUTO verify

[justsmth]

Context and motivation

An EVP_PKEY_RSA_PSS key can restrict the minimum salt length. That minimum was enforced everywhere except RSA_PSS_SALTLEN_AUTO, where verify recovered the salt length from the signature and accepted any value, including zero. EVP_PKEY_CTX_dup also dropped min_saltlen, so a duplicated context lost the restriction entirely.

Description of changes

RSA_PSS_SALTLEN_AUTO remains settable on a restricted key, but verify now checks the recovered salt length against the key's minimum, plumbed through new module-internal variants of RSA_verify_pss_mgf1 and RSA_verify_PKCS1_PSS_mgf1. The public functions pass "no minimum" and are unchanged. pkey_rsa_copy now propagates min_saltlen. The rest is readability: a public RSA_PSS_SALTLEN_AUTO (-2) alongside the existing RSA_PSS_SALTLEN_DIGEST, replacing the bare -2 literals and comments.

Two non-obvious points:

• This differs from OpenSSL on purpose. OpenSSL rejects AUTO at EVP_PKEY_CTX_set_rsa_pss_saltlen time for a restricted verify; we accept it and enforce the floor on the recovered length instead, keeping AUTO's meaning intact. So we accept any salt at or above the minimum where OpenSSL requires exactly it, and the failure surfaces from EVP_PKEY_verify rather than the setter. Documented there.
• Signing is unaffected: AUTO maximizes the salt and pkey_pss_init already rejects a key whose minimum exceeds that maximum, so signatures we produce still verify.

Testing

New EVPExtraTest.RestrictedPssAutoSaltlenHonorsMinimum uses kExampleRSAPSSKeyPKCS8, which omits saltLength and so carries the RFC 4055 default minimum of 20. It signs at the RSA layer with salt lengths 0, 1, 19, 20 and 24, then verifies each through EVP_PKEY_verify with AUTO set, on both the original and a duplicated context: below the minimum must fail, at or above must pass.

Each half of the fix was reverted in turn to confirm the test catches it -- disabling the floor check lets the short-salt signatures verify, dropping the min_saltlen copy breaks the duplicated context, and turning the floor into an equality test breaks the salt-24 case.

Review considerations

• The behavior change is limited to restricted EVP_PKEY_RSA_PSS keys verifying with AUTO, where a verify can now fail that previously succeeded. That is the intent, but it is a tightening.
• Inside the FIPS module boundary: both new functions are module-internal, called only from crypto/fipsmodule, and service indicator handling is unchanged.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and the ISC license.

[github-actions]

clang-tidy made some suggestions

[github-actions]

warning: call to undeclared function 'FIPS_service_indicator_lock_state'; ISO C99 and later do not support implicit function declarations [clang-diagnostic-implicit-function-declaration]

  FIPS_service_indicator_lock_state();
  ^

[github-actions]

🔒 Security Review — View Report

Please review before merging.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 78.19%. Comparing base (f8f6fb5) to head (26a6bb1).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3470      +/-   ##
==========================================
- Coverage   78.36%   78.19%   -0.17%     
==========================================
  Files         700      700              
  Lines      125744   125784      +40     
  Branches    17388    17388              
==========================================
- Hits        98540    98360     -180     
- Misses      26332    26554     +222     
+ Partials      872      870       -2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.