arm64: save movk instructions in mov_q when the lower 16|32 bits are all zero
PATCH 1/1 arm64: save movk instructions in mov_q when the lower 16|32 bits are all zero
Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:
/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS
Before the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0
After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32
--- arch/arm64/include/asm/assembler.h | 4 ++++ 1 file changed, 4 insertions(+)
arch/arm64/include/asm/assembler.h +4 -0
@@ -568,9 +568,13 @@ alternative_endif movz \reg, :abs_g3:\val movk \reg, :abs_g2_nc:\val .endif + .if ((((\val) >> 16) & 0xffff) != 0) movk \reg, :abs_g1_nc:\val .endif + .endif + .if (((\val) & 0xffff) != 0) movk \reg, :abs_g0_nc:\val + .endif .endm/*
Signature · 1 line
2.34.1
Will Deacon
maintainerQuoted 31 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32Signed-off-by Jisheng Zhang <[email protected]>--- arch/arm64/include/asm/assembler.h | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/include/asm/assembler.h +0
-0
arch/arm64/include/asm/assembler.h alternative_endif · line 577
Quoted 10 lines
arch/arm64/include/asm/assembler.h +4
-0@@ -568,9 +568,13 @@ alternative_endif movz \reg, :abs_g3:\val movk \reg, :abs_g2_nc:\val .endif + .if ((((\val) >> 16) & 0xffff) != 0) movk \reg, :abs_g1_nc:\val .endif + .endif + .if (((\val) & 0xffff) != 0) movk \reg, :abs_g0_nc:\val + .endif
Please provide some numbers showing that this is worthwhile.
Will
Quoted 36 lines · Will Deacon
Quoted 35 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32Signed-off-by Jisheng Zhang <[email protected]>--- arch/arm64/include/asm/assembler.h | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/include/asm/assembler.h +1
-0@@ -568,9 +568,13 @@ alternative_endif movz \reg, :abs_g3:\val movk \reg, :abs_g2_nc:\val .endif + .if ((((\val) >> 16) & 0xffff) != 0)
Quoted 9 lines
arch/arm64/include/asm/assembler.h alternative_endif · line 577
Quoted 6 lines
arch/arm64/include/asm/assembler.h +3
-0movk \reg, :abs_g1_nc:\val .endif + .endif + .if (((\val) & 0xffff) != 0) movk \reg, :abs_g0_nc:\val + .endifPlease provide some numbers showing that this is worthwhile.
No, I have no performance numbers, but here are my opnion about this patch: the two checks doesn't add maintaince effort, its readability is good, if the two checks can save two movk instructions, it's worthwhile to add the checks.
Thanks
Will Deacon
maintainerQuoted 47 lines · Jisheng Zhang
Quoted 36 lines · Will Deacon
Quoted 35 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32Signed-off-by Jisheng Zhang <[email protected]>--- arch/arm64/include/asm/assembler.h | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/include/asm/assembler.h +1
-0@@ -568,9 +568,13 @@ alternative_endif movz \reg, :abs_g3:\val movk \reg, :abs_g2_nc:\val .endif + .if ((((\val) >> 16) & 0xffff) != 0)Quoted 9 lines
arch/arm64/include/asm/assembler.h alternative_endif · line 577
Quoted 6 lines
arch/arm64/include/asm/assembler.h +3
-0movk \reg, :abs_g1_nc:\val .endif + .endif + .if (((\val) & 0xffff) != 0) movk \reg, :abs_g0_nc:\val + .endifPlease provide some numbers showing that this is worthwhile.
Quoted 4 lines
No, I have no performance numbers, but here are my opnion about this patch: the two checks doesn't add maintaince effort, its readability is good, if the two checks can save two movk instructions, it's worthwhile to add the checks.
Not unless you can measure a performance increase, no. The code is always going to be more readable without this stuff added so we shouldn't clutter our low-level assembly macros with nested conditionals just for fun.
Will
Quoted 18 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0
Quoted 5 lines
After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32Signed-off-by Jisheng Zhang <[email protected]>
This is broken for constants that have 0xffff in the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to 0xffff not 0x0.
Quoted 30 lines
--- arch/arm64/include/asm/assembler.h | 4 ++++ 1 file changed, 4 insertions(+)arch/arm64/include/asm/assembler.h +4
-1@@ -568,9 +568,13 @@ alternative_endif movz \reg, :abs_g3:\val movk \reg, :abs_g2_nc:\val .endif + .if ((((\val) >> 16) & 0xffff) != 0) movk \reg, :abs_g1_nc:\val .endif + .endif + .if (((\val) & 0xffff) != 0) movk \reg, :abs_g0_nc:\val + .endif .endm/* -- 2.34.1
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 23 lines · Ard Biesheuvel
Quoted 22 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32
Quoted 5 lines
Quoted 1 line
Signed-off-by Jisheng Zhang <[email protected]>This is broken for constants that have
0xffffin the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to0xffffnot0x0.
Thanks so much for this hint. I think you are right about the 0xffff in top 16bits case.
Quoted 24 lines · Jisheng Zhang
Quoted 23 lines · Ard Biesheuvel
Quoted 22 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32
Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by Jisheng Zhang <[email protected]>This is broken for constants that have
0xffffin the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to0xffffnot0x0.Thanks so much for this hint. I think you are right about the
0xffffin top 16bits case.
the patch breaks below usage case:mov_q x0, 0xffffffff00000000
I think the reason is mov_q starts from high bits, if we change the macro to start from LSB, then that could solve the breakage. But this needs a rewrite of mov_q
Quoted 36 lines · Jisheng Zhang
Quoted 24 lines · Jisheng Zhang
Quoted 23 lines · Ard Biesheuvel
Quoted 22 lines · Jisheng Zhang
Currently
mov_qis used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, themov_qemits one or two useless movk instructions. If themov_qmacro is used in hot code path, we want to save the movk instructions as much as possible. For example, whenCONFIG_ARM64_MTEis 'Y' andCONFIG_KASAN_HW_TAGSis 'N', the following code in__cpu_setup()routine is the pontential optimization target:/* set the
TCR_EL1bits */
mov_qx10,TCR_MTE_FLAGSBefore the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32 movk x10, #0x0, lsl #16 movk x10, #0x0After the patch:
mov x10, #0x10000000000000 movk x10, #0x40, lsl #32Quoted 9 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by Jisheng Zhang <[email protected]>This is broken for constants that have
0xffffin the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to0xffffnot0x0.Thanks so much for this hint. I think you are right about the
0xffffin top 16bits case.
Quoted 6 lines
the patch breaks below usage case:
mov_qx0,0xffffffff00000000I think the reason is
mov_qstarts from high bits, if we change the macro to start from LSB, then that could solve the breakage. But this needs a rewrite ofmov_q
No it has nothing to do with that.
The problem is that the use of MOVN changes the implicit value of the 16-bit fields that are left unspecified, and assigning them in a different order is not going to make any difference.
I don't think we should further complicate mov_q, and I would argue that the existing optimization (which I added myself) is premature
already: in the grand scheme of things, one or two instructions more or less are not going to make a difference anyway, given how rarely this macro is used. And even if any of these occurrences are on a hot path, it is not a given that shorter sequences of MOVZ/MOVN/MOVK are going to execute any faster, as the canonical MOVZ/MOVK/MOVK/MOVK might well decode to fewer uops.
So in summary, let's leave this code be.