crypto: x86/aes-gcm: Disable FPU around skcipher_walk_done().
Subsystems (2)
PATCH 1/1 crypto: x86/aes-gcm: Disable FPU around skcipher_walk_done().
kernel_fpu_begin() disables preemption. gcm_crypt() has a skcipher_walk_done() invocation within a preempt disabled section. skcipher_walk_done() can invoke kfree() which requires sleeping locks on PREEMPT_RT and must not be invoked with disabled preemption.
Keep FPU access enabled while skcipher_walk_done() is invoked.
--- arch/x86/crypto/aesni-intel_glue.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/crypto/aesni-intel_glue.c +2 -0
@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end (); err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();/* * The low word of the counter isn't used by the * finalize, so there's no need to increment it here.
Signature · 1 line
2.45.2
Hi Sebastian,
Quoted 18 lines · Sebastian Andrzej Siewior
kernel_fpu_begin()disables preemption.gcm_crypt()has askcipher_walk_done()invocation within a preempt disabled section.skcipher_walk_done()can invokekfree()which requires sleeping locks onPREEMPT_RTand must not be invoked with disabled preemption.Keep FPU access enabled while
skcipher_walk_done()is invoked.Fixes b06affb1cb580 ("crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM")Signed-off-by Sebastian Andrzej Siewior <[email protected]>--- arch/x86/crypto/aesni-intel_glue.c | 2 ++ 1 file changed, 2 insertions(+)arch/x86/crypto/aesni-intel_glue.c +0
-0
arch/x86/crypto/aesni-intel_glue.c gcm_crypt(struct aead_request *req, int flags) · line 1411
Quoted 9 lines
arch/x86/crypto/aesni-intel_glue.c +2
-0@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end (); err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();/* * The low word of the counter isn't used by the * finalize, so there's no need to increment it here.
Can you make this conditional on CONFIG_PREEMPT_RT so that it doesn't hurt performance for everyone else?
Note that kfree() lacks a might_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide. Is it really necessary that kfree() takes a sleepable lock on PREEMPT_RT?
- Eric
Quoted 34 lines · Eric Biggers
Hi Sebastian,
Quoted 18 lines · Sebastian Andrzej Siewior
kernel_fpu_begin()disables preemption.gcm_crypt()has askcipher_walk_done()invocation within a preempt disabled section.skcipher_walk_done()can invokekfree()which requires sleeping locks onPREEMPT_RTand must not be invoked with disabled preemption.Keep FPU access enabled while
skcipher_walk_done()is invoked.Fixes b06affb1cb580 ("crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM")Signed-off-by Sebastian Andrzej Siewior <[email protected]>--- arch/x86/crypto/aesni-intel_glue.c | 2 ++ 1 file changed, 2 insertions(+)arch/x86/crypto/aesni-intel_glue.c +0
-0arch/x86/crypto/aesni-intel_glue.c gcm_crypt(struct aead_request *req, int flags) · line 1411
Quoted 9 lines
arch/x86/crypto/aesni-intel_glue.c +2
-0@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end (); err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();/* * The low word of the counter isn't used by the * finalize, so there's no need to increment it here. Can you make this conditional on
CONFIG_PREEMPT_RTso that it doesn't hurt performance for everyone else?
Quoted 5 lines
Note that
kfree()lacks amight_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide. Is it really necessary thatkfree()takes a sleepable lock onPREEMPT_RT?
This would work too, I think:
arch/x86/crypto/aesni-intel_glue.c +4 -1
@@ -1401,11 +1401,12 @@ gcm_crypt(struct aead_request *req, int flags) }else {/* Last segment: process all remaining data. */ aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); - err =skcipher_walk_done (&walk,0 ); + err =0 ; +break ;/* * The low word of the counter isn't used by the * finalize, so there's no need to increment it here. */ }
@@ -1439,10 +1440,12 @@ gcm_crypt(struct aead_request *req, int flags) datalen, tag, taglen, flags)) err = -EBADMSG ; } out:kernel_fpu_end (); +if (nbytes) +skcipher_walk_done (&walk,0 );return err; }
Herbert Xu
maintainerQuoted 2 lines · Eric Biggers
This would work too, I think:
Yes, and we can go a bit further like this:
arch/x86/crypto/aesni-intel_glue.c +19 -16
@@ -1381,8 +1381,9 @@ gcm_crypt(struct aead_request *req, int flags)gcm_process_assoc (key, ghash_acc, req->src, assoclen, flags);/* En/decrypt the data and pass the ciphertext through GHASH. */ -while ((nbytes = walk.nbytes) !=0 ) { -if (unlikely (nbytes < walk.total)) { + nbytes = walk.nbytes; +if (nbytes) { +while (unlikely (nbytes < walk.total)) {/* * Non-last segment. In this case, the assembly * function requires that the length be a multiple of 16
@@ -1397,21 +1398,24 @@ gcm_crypt(struct aead_request *req, int flags) le_ctr[0 ] += nbytes /AES_BLOCK_SIZE ;kernel_fpu_end (); err =skcipher_walk_done (&walk, walk.nbytes - nbytes); +if (err) +return err; + nbytes = walk.nbytes;kernel_fpu_begin (); - }else { -/* Last segment: process all remaining data. */ -aes_gcm_update (key, le_ctr, ghash_acc, - walk.src.virt.addr, walk.dst.virt.addr, - nbytes, flags); - err =skcipher_walk_done (&walk,0 ); -/* -* The low word of the counter isn't used by the -* finalize, so there's no need to increment it here. -*/ } + +/* Last segment: process all remaining data. */ +aes_gcm_update (key, le_ctr, ghash_acc, + walk.src.virt.addr, walk.dst.virt.addr, + nbytes, flags); +/* +* The low word of the counter isn't used by the +* finalize, so there's no need to increment it here. +*/ + }else if (err) { +kernel_fpu_end (); +return err; } -if (err) - goto out;/* Finalize */ taglen =crypto_aead_authsize (tfm);
@@ -1439,9 +1443,8 @@ gcm_crypt(struct aead_request *req, int flags) datalen, tag, taglen, flags)) err = -EBADMSG ; } -out:kernel_fpu_end (); -return err; +return skcipher_walk_done (&walk,0 ); } #define DEFINE_GCM_ALGS(suffix, flags, generic_driver_name, rfc_driver_name, \
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Sebastian Andrzej Siewior
Tested-byQuoted 5 lines · Herbert Xu
Quoted 2 lines · Eric Biggers
This would work too, I think:
Yes, and we can go a bit further like this:
Yes,
Sebastian
Herbert Xu
maintainerQuoted 5 lines · Eric Biggers
Note that
kfree()lacks amight_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide. Is it really necessary thatkfree()takes a sleepable lock onPREEMPT_RT?
Agreed. kfree() gets called in all sorts of places under softirq context in the network stack which would presumably have the same issue.
Please give a bit of context of when kfree started doing this.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Quoted 7 lines · Herbert Xu
Quoted 5 lines · Eric Biggers
Note that
kfree()lacks amight_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide. Is it really necessary thatkfree()takes a sleepable lock onPREEMPT_RT?
Quoted 5 lines
Agreed.
kfree()gets called in all sorts of places under softirq context in the network stack which would presumably have the same issue.Please give a bit of context of when kfree started doing this.
I added "under PREEMPT_RT" in the patch description. The softirq under PREEMPT_RT does not disable preemption so it is fully preemptible. The other accelerated versions drop exclusive FPU access/ enable preemption during these operations even on ARM/ ARM64.
Quoted 1 line
Cheers,
Sebastian
Quoted 1 line · Eric Biggers
Hi Sebastian,
Hi Eric,
Quoted 9 lines
Quoted 9 lines
arch/x86/crypto/aesni-intel_glue.c +1
-0@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end ();
Quoted 8 lines
arch/x86/crypto/aesni-intel_glue.c gcm_crypt(struct aead_request *req, int flags) · line 1411
Quoted 5 lines
arch/x86/crypto/aesni-intel_glue.c +1
-0err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();/* * The low word of the counter isn't used by the * finalize, so there's no need to increment it here. Can you make this conditional on
CONFIG_PREEMPT_RTso that it doesn't hurt performance for everyone else?
Every other instance in this file had a kernel_fpu_end/ begin() before skcipher_walk_done() so I though was just missed by chance.
Quoted 3 lines
Note that
kfree()lacks amight_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide.
kfree() can't have a might_sleep() because it does not qualify for this since you can use it in softirq context for instance with an acquired spinlockt_t on !RT which would trigger it.
On PREEMPT_RT interrupts are threaded, softirq is preemptible, spintlock_t is a sleeping lock so all these things where a kfree() would have been invoked in preempt-disable context on !PREEMPT_RT is actually preemptible on PREEMPT_RT.
This is of course not true in cases where preemption is explicitly disabled like in this case.
Quoted 2 lines
Is it really necessary that
kfree()takes
a sleepable lock onPREEMPT_RT?
Yes. The locking in SLUB and page allocator is spinlock_t.
Quoted 1 line
- Eric
Sebastian
Quoted 22 lines · Sebastian Andrzej Siewior
Quoted 1 line · Eric Biggers
Hi Sebastian,
Hi Eric,
Quoted 9 lines
Quoted 9 lines
arch/x86/crypto/aesni-intel_glue.c +1
-0@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end ();Quoted 8 lines
arch/x86/crypto/aesni-intel_glue.c gcm_crypt(struct aead_request *req, int flags) · line 1411
Quoted 5 lines
arch/x86/crypto/aesni-intel_glue.c +1
-0err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();/* * The low word of the counter isn't used by the * finalize, so there's no need to increment it here. Can you make this conditional on
CONFIG_PREEMPT_RTso that it doesn't hurt performance for everyone else?
Quoted 2 lines
Every other instance in this file had a kernel_fpu_end/
begin()beforeskcipher_walk_done()so I though was just missed by chance.
No, it was intentional. See the comment above the first kernel_fpu_begin() in gcm_crypt():
/*
* Since the AES-GCM assembly code requires that at least three assembly
* functions be called to process any message (this is needed to support
* incremental updates cleanly), to reduce overhead we try to do all
* three calls in the same kernel FPU section if possible. We close the
* section and start a new one if there are multiple data segments or if
* rescheduling is needed while processing the associated data. */
kernel_fpu_begin();
Quoted 4 lines
Quoted 3 lines
Note that
kfree()lacks amight_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide.
Quoted 9 lines
kfree()can't have amight_sleep()because it does not qualify for this since you can use it in softirq context for instance with an acquiredspinlockt_ton !RT which would trigger it.
OnPREEMPT_RTinterrupts are threaded, softirq is preemptible,spintlock_tis a sleeping lock so all these things where akfree()would have been invoked in preempt-disable context on !PREEMPT_RT is actually preemptible onPREEMPT_RT.
This is of course not true in cases where preemption is explicitly disabled like in this case.
WARN_ON(!preemptible()) then?
If I add that to kfree(), it triggers from lots of other places. Are those problems on PREEMPT_RT too?
What I am trying to get at is what debugging options do I need to detect issues like this. Is there really no option other than actually running a PREEMPT_RT kernel?
I had tested this code with lots of debug options pre-merge and nothing came up.
If there was something in CONFIG_SLUB_DEBUG, for example, I would have seen that, and you would never have had to deal with this issue at all as it would never have been introduced.
- Eric
Quoted 12 lines · Eric Biggers
No, it was intentional. See the comment above the first
kernel_fpu_begin()ingcm_crypt():/*
* Since the AES-GCM assembly code requires that at least three assembly
* functions be called to process any message (this is needed to support
* incremental updates cleanly), to reduce overhead we try to do all
* three calls in the same kernel FPU section if possible. We close the
* section and start a new one if there are multiple data segments or if
* rescheduling is needed while processing the associated data. */
kernel_fpu_begin();
Okay.
Quoted 14 lines
Quoted 4 lines
Quoted 3 lines
Note that
kfree()lacks amight_sleep(), and its kerneldoc does not say that it can sleep. Have you checked for other instances of this same problem? It seems it would be quite common kernel-wide.Quoted 9 lines
kfree()can't have amight_sleep()because it does not qualify for this since you can use it in softirq context for instance with an acquiredspinlockt_ton !RT which would trigger it.
OnPREEMPT_RTinterrupts are threaded, softirq is preemptible,spintlock_tis a sleeping lock so all these things where akfree()would have been invoked in preempt-disable context on !PREEMPT_RT is actually preemptible onPREEMPT_RT.
This is of course not true in cases where preemption is explicitly disabled like in this case.
Quoted 4 lines
WARN_ON(!preemptible()) then?
If I add that to
kfree(), it triggers from lots of other places. Are those problems onPREEMPT_RTtoo?
Nope, shouldn't. On PREEMPT_RT you can only invoke kfree() or kmalloc() from preemptible context. This is not a problem since interrupts are threaded, softirqs are preemptbile,…
Quoted 3 lines
What I am trying to get at is what debugging options do I need to detect issues like this. Is there really no option other than actually running a
PREEMPT_RTkernel?
There is PROVE_RAW_LOCK_NESTING but this only catches something like raw_spinlock_t -> spinlock_t or using a spinlock_t in an interrupt handler that won't be threaded. It won't find anything where you disable interrupts or preemption on purpose.
Quoted 5 lines
I had tested this code with lots of debug options pre-merge and nothing came up.
If there was something in
CONFIG_SLUB_DEBUG, for example, I would have seen that, and you would never have had to deal with this issue at all as it would never have been introduced.
Don't worry. I didn't think for a second that there was lack of testing on your side.
Quoted 1 line
- Eric
Sebastian
Herbert Xu
maintainerQuoted 18 lines · Sebastian Andrzej Siewior
kernel_fpu_begin()disables preemption.gcm_crypt()has askcipher_walk_done()invocation within a preempt disabled section.skcipher_walk_done()can invokekfree()which requires sleeping locks onPREEMPT_RTand must not be invoked with disabled preemption.Keep FPU access enabled while
skcipher_walk_done()is invoked.Fixes b06affb1cb580 ("crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM")Signed-off-by Sebastian Andrzej Siewior <[email protected]>--- arch/x86/crypto/aesni-intel_glue.c | 2 ++ 1 file changed, 2 insertions(+)arch/x86/crypto/aesni-intel_glue.c +0
-0
arch/x86/crypto/aesni-intel_glue.c gcm_crypt(struct aead_request *req, int flags) · line 1408
Quoted 6 lines
arch/x86/crypto/aesni-intel_glue.c +2
-0@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end (); err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();
What if the user already did a preempt_disable()? This would still be buggy, right?
The Crypto API allows this to be called with preemption disabled.
Cheers,
Signature · 3 lines
Email: Herbert Xu <[email protected]> Home Page: http://gondor.apana.org.au/~herbert/ PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
Quoted 14 lines · Herbert Xu
Quoted 13 lines · Sebastian Andrzej Siewior
kernel_fpu_begin()disables preemption.gcm_crypt()has askcipher_walk_done()invocation within a preempt disabled section.skcipher_walk_done()can invokekfree()which requires sleeping locks onPREEMPT_RTand must not be invoked with disabled preemption.Keep FPU access enabled while
skcipher_walk_done()is invoked.Fixes b06affb1cb580 ("crypto: x86/aes-gcm - add VAES and AVX512 / AVX10 optimized AES-GCM")Signed-off-by Sebastian Andrzej Siewior <[email protected]>--- arch/x86/crypto/aesni-intel_glue.c | 2 ++ 1 file changed, 2 insertions(+)
Quoted 14 lines
arch/x86/crypto/aesni-intel_glue.c gcm_crypt(struct aead_request *req, int flags) · line 1408
Quoted 11 lines
arch/x86/crypto/aesni-intel_glue.c +2
-0@@ -1403,7 +1403,9 @@ gcm_crypt(struct aead_request *req, int flags)aes_gcm_update (key, le_ctr, ghash_acc, walk.src.virt.addr, walk.dst.virt.addr, nbytes, flags); +kernel_fpu_end (); err =skcipher_walk_done (&walk,0 ); +kernel_fpu_begin ();What if the user already did a
preempt_disable()? This would still be buggy, right?
Yes if it has been done explicitly by preempt_disable(). And I am looking into explicit case of disabling preemption and trying to get rid of it if I stumble upon one. This one just popped up on one of my boxes.
Quoted 3 lines
The Crypto API allows this to be called with preemption disabled.
Cheers,
Sebastian