Skip to contentSkip to conversation

Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

PATCH In review Stephan Mueller 3 participants 9 messages Aug 7, 2024 – Aug 24, 2024 1 message not in the archive Crypto API

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): DM4PR21MB360932816FA7B848D7D8F7B0C7B82@DM4PR21MB3609.namprd21.prod.outlook.com

#1 lore raw

Am Mittwoch, 7. August 2024, 14:50:32 MESZ schrieb Jeff Barnes:

Hi Jeff,

Quoted 22 lines

Hello,

We are currently migrating to kernel 6.6.14 and encountering intermittent EHEALTH errors that cause a kernel panic in initrd (FIPS mode). The error occurs in the following section of the code:

crypto/jitterentropy.c
722 /* Validate health test result */
723 if (jent_health_failure(&ec))
724 return JENT_EHEALTH;

This is called from jent_mod_init():

337         ret = jent_entropy_init(desc);
338         shash_desc_zero(desc);
339         crypto_free_shash(tfm);
340         if (ret) {

341 /* Handle permanent health test error */
342 if (fips_enabled)
343 panic("jitterentropy: Initialization failed with
host not compliant with requirements: %d\n", ret);

Quoted 5 lines

We are experiencing up to a 90% failure rate.

In my troubleshooting efforts, I followed the call to jent_condition_data() and attempted to increase the SHA3_HASH_LOOP to give the CPU more work, hoping to collect more entropy:

The proper way to handle it is the following: set CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the default is 1). The higher you set it the slower the collection will get as more samples are collected.

Quoted 9 lines

356

(quoted diff) +1 -1
-#define SHA3_HASH_LOOP (1<<3)
+#define SHA3_HASH_LOOP (1<<4)

This adjustment reduced the failure rate to 40-50%, but the issue persists. It is intermittent. It is also intermittent without the change. Sometimes I get a 90% failure rate on 10 reboots, sometimes 0%.

Quoted 7 lines

Given the difficulty in reproducing the kernel panic consistently, is there a more effective workaround or solution for this problem?

Your assistance is greatly appreciated.

Best regards,
Jeff Barnes

Ciao
Stephan

maintainer
#2 lore raw
Quoted 5 lines · Stephan Mueller

The proper way to handle it is the following: set CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the default is 1). The higher you set it the slower the collection will get as more samples are collected.

Stephan, it's not acceptable to rely on adjusting a Kconfig knob to obtain a working system. This needs to work out of the box.

Thanks,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#3 lore raw

Am Donnerstag, 8. August 2024, 07:17:01 MESZ schrieb Herbert Xu:

Hi Herbert,

Quoted 8 lines · Stephan Mueller
Quoted 4 lines

The proper way to handle it is the following: set CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the default is 1). The higher you set it the slower the collection will get as more samples are collected.

Stephan, it's not acceptable to rely on adjusting a Kconfig knob to obtain a working system. This needs to work out of the box.

The user-space version uses an OSR of 3. Using this value, I have not heard of any problems. I will prepare a patch.

Quoted 2 lines

Thanks,

Ciao
Stephan

maintainer
#4 lore raw
Quoted 3 lines · Stephan Mueller

The user-space version uses an OSR of 3. Using this value, I have not heard of any problems. I will prepare a patch.

Thanks Stephan. Jeff, could you please let us know if the value of 3 fixes your problems?

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#5 lore raw

Am Donnerstag, 8. August 2024, 08:10:15 MESZ schrieb Herbert Xu:

Hi Herbert,

Quoted 6 lines · Stephan Mueller
Quoted 2 lines

The user-space version uses an OSR of 3. Using this value, I have not heard of any problems. I will prepare a patch.

Thanks Stephan. Jeff, could you please let us know if the value of 3 fixes your problems?

However, the heart of the problem is the following: This failure mode is probabilistic in nature. A number of folks trying to push rules that the failure does not need to be handled with a panic.

A changed OSR only changes the probability, but that probability is always strictly higher than zero.

Quoted 2 lines

Cheers,

Ciao
Stephan

maintainer
#6 lore raw
Quoted 7 lines · Stephan Mueller

However, the heart of the problem is the following: This failure mode is probabilistic in nature. A number of folks trying to push rules that the failure does not need to be handled with a panic.

A changed OSR only changes the probability, but that probability is always strictly higher than zero.

That's fine. There are many places in the kernel that will fail with a probably that is non-zero. It is considered to be acceptable as long as the value is negligible (e.g., equal or less than the probablility of cosmic rays hitting DRAM).

But if it happens reproducibly it clearly is not acceptable.

Cheers,

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
#7 lore raw

By setting the configuration option
+CONFIG_CRYPTO_JITTERENTROPY_OSR=3,

I ran the following ad hoc test.

50 consecutive boots with fips=1 on the command line with the same image.

No jitterentropy health check failure. Booted successfully.

Thanks for the help. I plan to try a value of 2 to see if that will fix it too.

Jeff

________________________________________
From: Herbert Xu <[email protected]>
Sent: Thursday, August 8, 2024 2:20 AM
To: Stephan Mueller
Cc: [email protected]; Jeff Barnes; Vladis Dronov; [email protected]; Tyler Hicks; Shyam Saini
Subject: [EXTERNAL] Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

Quoted 7 lines · Stephan Mueller

However, the heart of the problem is the following: This failure mode is probabilistic in nature. A number of folks trying to push rules that the failure does not need to be handled with a panic.

A changed OSR only changes the probability, but that probability is always strictly higher than zero.

That's fine. There are many places in the kernel that will fail with a probably that is non-zero. It is considered to be acceptable as long as the value is negligible (e.g., equal or less than the probablility of cosmic rays hitting DRAM).

But if it happens reproducibly it clearly is not acceptable.

Cheers,
--
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

#8 lore raw

PATCH 1/1 crypto: JENT - set default OSR to 3

In reply to Herbert Xu

The user space Jitter RNG library uses the oversampling rate of 3 which implies that each time stamp is credited with 1/3 bit of entropy. To obtain 256 bits of entropy, 768 time stamps need to be sampled. The increase in OSR is applied based on a report where the Jitter RNG is used on a system exhibiting a challenging environment to collect entropy.

This OSR default value is now applied to the Linux kernel version of the Jitter RNG as well.

The increase in the OSR from 1 to 3 also implies that the Jitter RNG is now slower by default.

---
 crypto/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/Kconfig +1 -1
index 72e2decb8c6a..a779cab668c2 100644
@@ -1305,7 +1305,7 @@ config CRYPTO_JITTERENTROPY_MEMORY_BLOCKSIZE
 config CRYPTO_JITTERENTROPY_OSR
 	int "CPU Jitter RNG Oversampling Rate"
 	range 1 15
-	default 1
+	default 3
 	help
 	  The Jitter RNG allows the specification of an oversampling rate (OSR).
 	  The Jitter RNG operation requires a fixed amount of timing
Signature · 1 line
2.46.0
maintainer
#9 lore raw

Re: [PATCH] crypto: JENT - set default OSR to 3

Quoted 10 lines · Stephan Mueller

The user space Jitter RNG library uses the oversampling rate of 3 which implies that each time stamp is credited with 1/3 bit of entropy. To obtain 256 bits of entropy, 768 time stamps need to be sampled. The increase in OSR is applied based on a report where the Jitter RNG is used on a system exhibiting a challenging environment to collect entropy.

This OSR default value is now applied to the Linux kernel version of the Jitter RNG as well.

Quoted 8 lines

The increase in the OSR from 1 to 3 also implies that the Jitter RNG is now slower by default.

---
 crypto/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Patch applied. Thanks.

Signature · 3 lines
Email: Herbert Xu <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt