FIPS 140-3 mode on RHEL 9 and RSA validation of <2048 keys

[Petr Menšík]

Hello,

I have sent already a notification about SHA-1 not validated in default
configuration. However that was not end of the story.

A new and even more severe issue has arisen. Our crypto team is
responsible for preparing RHEL 9 for FIPS 140-3 certification. They said
there is legal obligation to stop using all RSA signatures with keys
shorter than 2048 bits. I expect many of you already know especially
Zone Signing Keys often use 1280 or even 1024b keys without issues.

Current RHEL 9 validates such signatures well, but there is a bug [1]
filled to change that and start enforcing longer keys usage only.
Because it would be enforced at crypto library level (openssl), common
DNSSEC validation software would start failing. On quite a lot of
domains. And failing to resolve such names.

I am looking for the best way out of this. Once the above happens
without any changes to used DNS(SEC) software, the only way to keep
working DNS servers working would be either turning off FIPS more or
DNSSEC validation. I know the result seems quite ridiculous, because all
this has been done in order to increase the security. The only
alternative without changes would be disabling all RSA altogether and
providing a long list of ECDSA trust anchors for TLD domains, where at
least ECDSA would offer protection.

I think the only good way would be starting considering shorter keys as
insecure in FIPS mode. Anything above those domains would be without
DNSSEC protection, but at least single root key could be used. Do you
think it would be safe once the DS digest is checked on the key, that
key is then can be marked as insecure instead of bogus. Just as if the
algorithm were unsupported, but after checking the length of key.
Because the crypto library would refuse any operation, including
signature validation, on the given key. I think the library would not
even allow loading that key.

I don't think described behavior is possible in any supported version of
BIND9. I expect it would require not only small and self-contained change.
Would you know any blockers, which would prevent behavior described in
the above paragraph? Is it possible to consider all keys with short keys
to become unsupported, but keys with long enough keys to be still 
validated as usual?

Note: some companies and agencies have to use only FIPS 140-3 approved
algorithms. Enabling FIPS mode in RHEL 9 is not optional for them. Fixing
the problem by disabling FIPS mode is not possible for everyone.

Any comments or suggestions welcome.

Best Regards,
Petr Menšík

-- 
Petr Menšík
Software Engineer
Red Hat, http://www.redhat.com/
email: pemensik at redhat.com
PGP: DFCF908DB7C87E8E529925BC4931CA5B6C9FC5CB