crypto: arm/aes-ce - work around Cortex-A72 erratum #1655431
Subsystems (2)
PATCH 1/1 crypto: arm/aes-ce - work around Cortex-A72 erratum #1655431
ARM Cortex-A72 cores running in 32-bit mode are affected by a silicon erratum (1655431: ELR recorded incorrectly on interrupt taken between cryptographic instructions in a sequence [0]) where the second instruction of a AES instruction pair may execute twice if an interrupt is taken right after the first instruction consumes an input register of which a single 32-bit lane has been updated the last time it was modified.
This is not such a rare occurrence as it may seem: in counter mode, only the least significant 32-bit word is incremented in the absence of a carry, which makes our counter mode implementation susceptible to the erratum.
So let's shuffle the counter assignments around a bit so that the most recent updates when the AES instruction pair executes are 128-bit wide.
[0] ARM-EPM-012079 v11.0 Cortex-A72 MPCore Software Developers Errata Notice
--- arch/arm/crypto/aes-ce-core.S | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)
arch/arm/crypto/aes-ce-core.S +10 -10
@@ -386,20 +386,20 @@ ENTRY(ce_aes_ctr_encrypt) .Lctrloop4x: subs r4, r4, #4 bmi .Lctr1x - add r6, r6, #1 + add ip, r6, #1 vmov q0, q7 + rev ip, ip + add lr, r6, #2 + vmov s31, ip + add ip, r6, #3 + rev lr, lr vmov q1, q7 - rev ip, r6 - add r6, r6, #1 + vmov s31, lr + rev ip, ip vmov q2, q7 - vmov s7, ip - rev ip, r6 - add r6, r6, #1 + vmov s31, ip + add r6, r6, #4 vmov q3, q7 - vmov s11, ip - rev ip, r6 - add r6, r6, #1 - vmov s15, ip vld1.8 {q4-q5}, [r1]! vld1.8 {q6}, [r1]! vld1.8 {q15}, [r1]!
Signature · 1 line
2.17.1
Quoted 48 lines · Ard Biesheuvel
ARM Cortex-A72 cores running in 32-bit mode are affected by a silicon erratum (1655431: ELR recorded incorrectly on interrupt taken between cryptographic instructions in a sequence [0]) where the second instruction of a AES instruction pair may execute twice if an interrupt is taken right after the first instruction consumes an input register of which a single 32-bit lane has been updated the last time it was modified.
This is not such a rare occurrence as it may seem: in counter mode, only the least significant 32-bit word is incremented in the absence of a carry, which makes our counter mode implementation susceptible to the erratum.
So let's shuffle the counter assignments around a bit so that the most recent updates when the AES instruction pair executes are 128-bit wide.
[0] ARM-EPM-012079 v11.0 Cortex-A72 MPCore Software Developers Errata Notice
Cc <[email protected]> # v5.4+Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm/crypto/aes-ce-core.S | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)arch/arm/crypto/aes-ce-core.S +8
-6@@ -386,20 +386,20 @@ ENTRY(ce_aes_ctr_encrypt) .Lctrloop4x: subs r4, r4, #4 bmi .Lctr1x - add r6, r6, #1 + add ip, r6, #1 vmov q0, q7 + rev ip, ip + add lr, r6, #2 + vmov s31, ip + add ip, r6, #3 + rev lr, lr vmov q1, q7 - rev ip, r6 - add r6, r6, #1 + vmov s31, lr + rev ip, ip vmov q2, q7 - vmov s7, ip - rev ip, r6 - add r6, r6, #1
arch/arm/crypto/aes-ce-core.S ENTRY(ce_aes_ctr_encrypt) · line 405
Quoted 10 lines
arch/arm/crypto/aes-ce-core.S +2
-4+ vmov s31, ip + add r6, r6, #4 vmov q3, q7 - vmov s11, ip - rev ip, r6 - add r6, r6, #1 - vmov s15, ip vld1.8 {q4-q5}, [r1]! vld1.8 {q6}, [r1]! vld1.8 {q15}, [r1]!
Seems like this could use a comment that explains that things need to be done in a certain way to avoid an erratum.
- Eric
Quoted 55 lines · Eric Biggers
Quoted 53 lines · Ard Biesheuvel
ARM Cortex-A72 cores running in 32-bit mode are affected by a silicon erratum (1655431: ELR recorded incorrectly on interrupt taken between cryptographic instructions in a sequence [0]) where the second instruction of a AES instruction pair may execute twice if an interrupt is taken right after the first instruction consumes an input register of which a single 32-bit lane has been updated the last time it was modified.
This is not such a rare occurrence as it may seem: in counter mode, only the least significant 32-bit word is incremented in the absence of a carry, which makes our counter mode implementation susceptible to the erratum.
So let's shuffle the counter assignments around a bit so that the most recent updates when the AES instruction pair executes are 128-bit wide.
[0] ARM-EPM-012079 v11.0 Cortex-A72 MPCore Software Developers Errata Notice
Cc <[email protected]> # v5.4+Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm/crypto/aes-ce-core.S | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-)arch/arm/crypto/aes-ce-core.S +10
-8@@ -386,20 +386,20 @@ ENTRY(ce_aes_ctr_encrypt) .Lctrloop4x: subs r4, r4, #4 bmi .Lctr1x - add r6, r6, #1 + add ip, r6, #1 vmov q0, q7 + rev ip, ip + add lr, r6, #2 + vmov s31, ip + add ip, r6, #3 + rev lr, lr vmov q1, q7 - rev ip, r6 - add r6, r6, #1 + vmov s31, lr + rev ip, ip vmov q2, q7 - vmov s7, ip - rev ip, r6 - add r6, r6, #1 + vmov s31, ip + add r6, r6, #4 vmov q3, q7 - vmov s11, ip - rev ip, r6
Quoted 9 lines
arch/arm/crypto/aes-ce-core.S ENTRY(ce_aes_ctr_encrypt) · line 405
Quoted 5 lines
arch/arm/crypto/aes-ce-core.S +0
-2- add r6, r6, #1 - vmov s15, ip vld1.8 {q4-q5}, [r1]! vld1.8 {q6}, [r1]! vld1.8 {q15}, [r1]!Seems like this could use a comment that explains that things need to be done in a certain way to avoid an erratum.
Fair enough, I'll add that. Note that A57 is equally affected, so I need to respin this anyway to add a mention of that.