FIPS 140-3 mode on RHEL 9 and RSA validation of <2048 keys

[Petr Špaček]

On 25. 04. 22 11:49, Petr Menšík wrote:
> Forgot to add the bug link.
> 
> - openssl: https://bugzilla.redhat.com/show_bug.cgi?id=2077884
> - bind: https://bugzilla.redhat.com/show_bug.cgi?id=2077906
> 
> On 4/25/22 11:39, Petr Menšík wrote:
>> Hello,
>>
>> I have sent already a notification about SHA-1 not validated in default
>> configuration. However that was not end of the story.
>>
>> A new and even more severe issue has arisen. Our crypto team is
>> responsible for preparing RHEL 9 for FIPS 140-3 certification. They said
>> there is legal obligation to stop using all RSA signatures with keys
>> shorter than 2048 bits. I expect many of you already know especially
>> Zone Signing Keys often use 1280 or even 1024b keys without issues.
>>
>> Current RHEL 9 validates such signatures well, but there is a bug [1]
>> filled to change that and start enforcing longer keys usage only.
>> Because it would be enforced at crypto library level (openssl), common
>> DNSSEC validation software would start failing. On quite a lot of
>> domains. And failing to resolve such names.
>>
>> I am looking for the best way out of this. Once the above happens
>> without any changes to used DNS(SEC) software, the only way to keep
>> working DNS servers working would be either turning off FIPS more or
>> DNSSEC validation. I know the result seems quite ridiculous, because all
>> this has been done in order to increase the security. The only
>> alternative without changes would be disabling all RSA altogether and
>> providing a long list of ECDSA trust anchors for TLD domains, where at
>> least ECDSA would offer protection.
>>
>> I think the only good way would be starting considering shorter keys as
>> insecure in FIPS mode. Anything above those domains would be without
>> DNSSEC protection, but at least single root key could be used. Do you
>> think it would be safe once the DS digest is checked on the key, that
>> key is then can be marked as insecure instead of bogus. Just as if the
>> algorithm were unsupported, but after checking the length of key.
>> Because the crypto library would refuse any operation, including
>> signature validation, on the given key. I think the library would not
>> even allow loading that key.
>>
>> I don't think described behavior is possible in any supported version of
>> BIND9. I expect it would require not only small and self-contained change.
>> Would you know any blockers, which would prevent behavior described in
>> the above paragraph? Is it possible to consider all keys with short keys
>> to become unsupported, but keys with long enough keys to be still
>> validated as usual?
>>
>> Note: some companies and agencies have to use only FIPS 140-3 approved
>> algorithms. Enabling FIPS mode in RHEL 9 is not optional for them. Fixing
>> the problem by disabling FIPS mode is not possible for everyone.
>>
>> Any comments or suggestions welcome.

I have my doubts about FIPS wisdom, but in general I think feature to 
"treat keys as insecure if their length falls below bit length threshold 
for a given algorithm" is a sensible request.

Even if FIPS was not on face of Earth it makes sense to me to treat RSA 
keys with 512 bits as insecure.

The threshold could go even higher...
https://en.wikipedia.org/wiki/RSA_Factoring_Challenge

-- 
Petr Špaček