openssl error `digest not allowed` on RSA PSS signature in fips mode

[rcatolino]

Description

If openssl is built/configured in fips mode, when trying to do a signature with an RSA key using the PSS padding format, openssl returns the error 1C8000AE:Provider routines::digest not allowed

Reproduction Steps

• build openssl 3.0.9 in fips mode, install it, and use the configuration provided above
• call the SignHash method using hash algorithm HashAlgorithmName.SHA256 (or sha384/sah512, doesn't matter) and padding RSASignaturePadding.Pss

Expected behavior

We get a valid signature and no error.

Actual behavior

We get the error 1C8000AE:Provider routines::digest not allowed from openssl.

Regression?

No response

Known Workarounds

No response

Configuration

• linux x86_64
• .NET 9.0.202
• openssl 3.0.9 built in fips mode
  Using the following openssl configuration:

openssl_conf = default_conf

[default_conf]
providers = provider_sect

[provider_sect]
fips = fips_sect
base = base_sect

[fips_sect]
activate = 1
conditional-errors = 1
security-checks = 1

[base_sect]
activate = 1

Other information

This issue stems from the ConfigureSignature function in the CryptoNative wrapper which calls EVP_PKEY_CTX_set_rsa_padding before setting the digest used for mgf1.

In this situation openssl defaults to SHA1, see here. In fips mode SHA1 is forbidden and the set_rsa_padding call fails.