Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14
A message that is not in the archive (sent off-list or not yet received): DM4PR21MB360932816FA7B848D7D8F7B0C7B82@DM4PR21MB3609.namprd21.prod.outlook.com
Am Mittwoch, 7. August 2024, 14:50:32 MESZ schrieb Jeff Barnes:
Hi Jeff,
Quoted 22 lines
Hello,
We are currently migrating to kernel 6.6.14 and encountering intermittent EHEALTH errors that cause a kernel panic in initrd (FIPS mode). The error occurs in the following section of the code:
crypto/jitterentropy.c
722 /* Validate health test result */
723 if (jent_health_failure(&ec))
724 returnJENT_EHEALTH;This is called from
jent_mod_init():337 ret = jent_entropy_init(desc); 338 shash_desc_zero(desc); 339 crypto_free_shash(tfm); 340 if (ret) {341 /* Handle permanent health test error */
342 if (fips_enabled)
343 panic("jitterentropy: Initialization failed with
host not compliant with requirements: %d\n", ret);
Quoted 5 lines
We are experiencing up to a 90% failure rate.
In my troubleshooting efforts, I followed the call to
jent_condition_data()and attempted to increase theSHA3_HASH_LOOPto give the CPU more work, hoping to collect more entropy:
The proper way to handle it is the following: set CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the default is 1). The higher you set it the slower the collection will get as more samples are collected.
Quoted 9 lines
356
(quoted diff) +1
-1-#define SHA3_HASH_LOOP (1<<3) +#define SHA3_HASH_LOOP (1<<4)This adjustment reduced the failure rate to 40-50%, but the issue persists. It is intermittent. It is also intermittent without the change. Sometimes I get a 90% failure rate on 10 reboots, sometimes 0%.
Quoted 7 lines
Given the difficulty in reproducing the kernel panic consistently, is there a more effective workaround or solution for this problem?
Your assistance is greatly appreciated.
Best regards,
Jeff Barnes
Ciao
Stephan
Herbert Xu
maintainerQuoted 5 lines · Stephan Mueller
The proper way to handle it is the following: set
CONFIG_CRYPTO_JITTERENTROPY_OSRto a higer value as it is - like 3 (the default is 1). The higher you set it the slower the collection will get as more samples are collected.
Stephan, it's not acceptable to rely on adjusting a Kconfig knob to obtain a working system. This needs to work out of the box.
Thanks,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Am Donnerstag, 8. August 2024, 07:17:01 MESZ schrieb Herbert Xu:
Hi Herbert,
Quoted 8 lines · Stephan Mueller
Quoted 4 lines
The proper way to handle it is the following: set
CONFIG_CRYPTO_JITTERENTROPY_OSRto a higer value as it is - like 3 (the default is 1). The higher you set it the slower the collection will get as more samples are collected.Stephan, it's not acceptable to rely on adjusting a
Kconfigknob to obtain a working system. This needs to work out of the box.
The user-space version uses an OSR of 3. Using this value, I have not heard of any problems. I will prepare a patch.
Quoted 2 lines
Thanks,
Ciao
Stephan
Herbert Xu
maintainerQuoted 3 lines · Stephan Mueller
The user-space version uses an OSR of 3. Using this value, I have not heard of any problems. I will prepare a patch.
Thanks Stephan. Jeff, could you please let us know if the value of 3 fixes your problems?
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Am Donnerstag, 8. August 2024, 08:10:15 MESZ schrieb Herbert Xu:
Hi Herbert,
Quoted 6 lines · Stephan Mueller
Quoted 2 lines
The user-space version uses an OSR of 3. Using this value, I have not heard of any problems. I will prepare a patch.
Thanks Stephan. Jeff, could you please let us know if the value of 3 fixes your problems?
However, the heart of the problem is the following: This failure mode is probabilistic in nature. A number of folks trying to push rules that the failure does not need to be handled with a panic.
A changed OSR only changes the probability, but that probability is always strictly higher than zero.
Quoted 2 lines
Cheers,
Ciao
Stephan
Herbert Xu
maintainerQuoted 7 lines · Stephan Mueller
However, the heart of the problem is the following: This failure mode is probabilistic in nature. A number of folks trying to push rules that the failure does not need to be handled with a panic.
A changed OSR only changes the probability, but that probability is always strictly higher than zero.
That's fine. There are many places in the kernel that will fail with a probably that is non-zero. It is considered to be acceptable as long as the value is negligible (e.g., equal or less than the probablility of cosmic rays hitting DRAM).
But if it happens reproducibly it clearly is not acceptable.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
By setting the configuration option
+CONFIG_CRYPTO_JITTERENTROPY_OSR=3,
I ran the following ad hoc test.
50 consecutive boots with fips=1 on the command line with the same image.
No jitterentropy health check failure. Booted successfully.
Thanks for the help. I plan to try a value of 2 to see if that will fix it too.
Jeff
________________________________________
From: Herbert Xu <[email protected]>
Sent: Thursday, August 8, 2024 2:20 AM
To: Stephan Mueller
Cc: [email protected]; Jeff Barnes; Vladis Dronov; [email protected]; Tyler Hicks; Shyam Saini
Subject: [EXTERNAL] Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14
[You don't often get email from [email protected]. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
Quoted 7 lines · Stephan Mueller
However, the heart of the problem is the following: This failure mode is probabilistic in nature. A number of folks trying to push rules that the failure does not need to be handled with a panic.
A changed OSR only changes the probability, but that probability is always strictly higher than zero.
That's fine. There are many places in the kernel that will fail with a probably that is non-zero. It is considered to be acceptable as long as the value is negligible (e.g., equal or less than the probablility of cosmic rays hitting DRAM).
But if it happens reproducibly it clearly is not acceptable.
Cheers,
--
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
PATCH 1/1 crypto: JENT - set default OSR to 3
In reply to Herbert XuThe user space Jitter RNG library uses the oversampling rate of 3 which implies that each time stamp is credited with 1/3 bit of entropy. To obtain 256 bits of entropy, 768 time stamps need to be sampled. The increase in OSR is applied based on a report where the Jitter RNG is used on a system exhibiting a challenging environment to collect entropy.
This OSR default value is now applied to the Linux kernel version of the Jitter RNG as well.
The increase in the OSR from 1 to 3 also implies that the Jitter RNG is now slower by default.
--- crypto/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
crypto/Kconfig +1 -1
@@ -1305,7 +1305,7 @@ config CRYPTO_JITTERENTROPY_MEMORY_BLOCKSIZE config CRYPTO_JITTERENTROPY_OSR int "CPU Jitter RNG Oversampling Rate" range 1 15 - default 1 + default 3 help The Jitter RNG allows the specification of an oversampling rate (OSR). The Jitter RNG operation requires a fixed amount of timing
Signature · 1 line
2.46.0
Herbert Xu
maintainerRe: [PATCH] crypto: JENT - set default OSR to 3
Quoted 10 lines · Stephan Mueller
The user space Jitter RNG library uses the oversampling rate of 3 which implies that each time stamp is credited with 1/3 bit of entropy. To obtain 256 bits of entropy, 768 time stamps need to be sampled. The increase in OSR is applied based on a report where the Jitter RNG is used on a system exhibiting a challenging environment to collect entropy.
This OSR default value is now applied to the Linux kernel version of the Jitter RNG as well.
Quoted 8 lines
The increase in the OSR from 1 to 3 also implies that the Jitter RNG is now slower by default.
Reported-by Jeff Barnes <[email protected]>Signed-off-by Stephan Mueller <[email protected]>--- crypto/Kconfig | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Patch applied. Thanks.
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt