Skip to contentSkip to conversation

arm64: save movk instructions in mov_q when the lower 16|32 bits are all zero

PATCH In review Jisheng Zhang 3 participants 8 messages Jul 9, 2022 – Jul 28, 2022 arm64

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 arm64: save movk instructions in mov_q when the lower 16|32 bits are all zero

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
	movk	x10, #0x0, lsl #16
	movk	x10, #0x0

After the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
---
 arch/arm64/include/asm/assembler.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/assembler.h +4 -0
index 8c5a61aeaf8e..09f408424cae 100644
@@ -568,9 +568,13 @@ alternative_endif
 	movz	\reg, :abs_g3:\val
 	movk	\reg, :abs_g2_nc:\val
 	.endif
+	.if ((((\val) >> 16) & 0xffff) != 0)
 	movk	\reg, :abs_g1_nc:\val
 	.endif
+	.endif
+	.if (((\val) & 0xffff) != 0)
 	movk	\reg, :abs_g0_nc:\val
+	.endif
 	.endm
 
 /*
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 31 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
	movk	x10, #0x0, lsl #16
	movk	x10, #0x0

After the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
---
 arch/arm64/include/asm/assembler.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/assembler.h +0 -0
index 8c5a61aeaf8e..09f408424cae 100644

arch/arm64/include/asm/assembler.h alternative_endif · line 577

Quoted 10 lines
arch/arm64/include/asm/assembler.h +4 -0
@@ -568,9 +568,13 @@ alternative_endif
 	movz	\reg, :abs_g3:\val
 	movk	\reg, :abs_g2_nc:\val
 	.endif
+	.if ((((\val) >> 16) & 0xffff) != 0)
 	movk	\reg, :abs_g1_nc:\val
 	.endif
+	.endif
+	.if (((\val) & 0xffff) != 0)
 	movk	\reg, :abs_g0_nc:\val
+	.endif

Please provide some numbers showing that this is worthwhile.

Will

#2 lore raw
Quoted 36 lines · Will Deacon
Quoted 35 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
	movk	x10, #0x0, lsl #16
	movk	x10, #0x0

After the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
---
 arch/arm64/include/asm/assembler.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/assembler.h +1 -0
index 8c5a61aeaf8e..09f408424cae 100644
@@ -568,9 +568,13 @@ alternative_endif
 	movz	\reg, :abs_g3:\val
 	movk	\reg, :abs_g2_nc:\val
 	.endif
+	.if ((((\val) >> 16) & 0xffff) != 0)
Quoted 9 lines

arch/arm64/include/asm/assembler.h alternative_endif · line 577

Quoted 6 lines
arch/arm64/include/asm/assembler.h +3 -0
 	movk	\reg, :abs_g1_nc:\val
 	.endif
+	.endif
+	.if (((\val) & 0xffff) != 0)
 	movk	\reg, :abs_g0_nc:\val
+	.endif

Please provide some numbers showing that this is worthwhile.

No, I have no performance numbers, but here are my opnion about this patch: the two checks doesn't add maintaince effort, its readability is good, if the two checks can save two movk instructions, it's worthwhile to add the checks.

Thanks

maintainer
#3 lore raw
Quoted 47 lines · Jisheng Zhang
Quoted 36 lines · Will Deacon
Quoted 35 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
	movk	x10, #0x0, lsl #16
	movk	x10, #0x0

After the patch:

	mov	x10, #0x10000000000000
	movk	x10, #0x40, lsl #32
---
 arch/arm64/include/asm/assembler.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/assembler.h +1 -0
index 8c5a61aeaf8e..09f408424cae 100644
@@ -568,9 +568,13 @@ alternative_endif
 	movz	\reg, :abs_g3:\val
 	movk	\reg, :abs_g2_nc:\val
 	.endif
+	.if ((((\val) >> 16) & 0xffff) != 0)
Quoted 9 lines

arch/arm64/include/asm/assembler.h alternative_endif · line 577

Quoted 6 lines
arch/arm64/include/asm/assembler.h +3 -0
 	movk	\reg, :abs_g1_nc:\val
 	.endif
+	.endif
+	.if (((\val) & 0xffff) != 0)
 	movk	\reg, :abs_g0_nc:\val
+	.endif

Please provide some numbers showing that this is worthwhile.

Quoted 4 lines

No, I have no performance numbers, but here are my opnion about this patch: the two checks doesn't add maintaince effort, its readability is good, if the two checks can save two movk instructions, it's worthwhile to add the checks.

Not unless you can measure a performance increase, no. The code is always going to be more readable without this stuff added so we shouldn't clutter our low-level assembly macros with nested conditionals just for fun.

Will

#4 lore raw
In reply to Jisheng Zhang
Quoted 18 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
        movk    x10, #0x0, lsl #16
        movk    x10, #0x0
Quoted 5 lines

After the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32

This is broken for constants that have 0xffff in the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to 0xffff not 0x0.

Quoted 30 lines
---
 arch/arm64/include/asm/assembler.h | 4 ++++
 1 file changed, 4 insertions(+)
arch/arm64/include/asm/assembler.h +4 -1
index 8c5a61aeaf8e..09f408424cae 100644
@@ -568,9 +568,13 @@ alternative_endif
        movz    \reg, :abs_g3:\val
        movk    \reg, :abs_g2_nc:\val
        .endif
+       .if ((((\val) >> 16) & 0xffff) != 0)
        movk    \reg, :abs_g1_nc:\val
        .endif
+       .endif
+       .if (((\val) & 0xffff) != 0)
        movk    \reg, :abs_g0_nc:\val
+       .endif
        .endm
 
 /*
--

2.34.1

_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

#5 lore raw
Quoted 23 lines · Ard Biesheuvel
Quoted 22 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
        movk    x10, #0x0, lsl #16
        movk    x10, #0x0

After the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
Quoted 5 lines
Quoted 1 line

This is broken for constants that have 0xffff in the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to 0xffff not 0x0.

Thanks so much for this hint. I think you are right about the 0xffff in top 16bits case.

#6 lore raw
Quoted 24 lines · Jisheng Zhang
Quoted 23 lines · Ard Biesheuvel
Quoted 22 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
        movk    x10, #0x0, lsl #16
        movk    x10, #0x0

After the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
Quoted 9 lines
Quoted 5 lines
Quoted 1 line

This is broken for constants that have 0xffff in the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to 0xffff not 0x0.

Thanks so much for this hint. I think you are right about the 0xffff in top 16bits case.

the patch breaks below usage case:
mov_q x0, 0xffffffff00000000

I think the reason is mov_q starts from high bits, if we change the macro to start from LSB, then that could solve the breakage. But this needs a rewrite of mov_q

#7 lore raw
Quoted 36 lines · Jisheng Zhang
Quoted 24 lines · Jisheng Zhang
Quoted 23 lines · Ard Biesheuvel
Quoted 22 lines · Jisheng Zhang

Currently mov_q is used to move a constant into a 64-bit register, when the lower 16 or 32bits of the constant are all zero, the mov_q emits one or two useless movk instructions. If the mov_q macro is used in hot code path, we want to save the movk instructions as much as possible. For example, when CONFIG_ARM64_MTE is 'Y' and CONFIG_KASAN_HW_TAGS is 'N', the following code in __cpu_setup() routine is the pontential optimization target:

/* set the TCR_EL1 bits */
mov_q x10, TCR_MTE_FLAGS

Before the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
        movk    x10, #0x0, lsl #16
        movk    x10, #0x0

After the patch:

        mov     x10, #0x10000000000000
        movk    x10, #0x40, lsl #32
Quoted 9 lines
Quoted 5 lines
Quoted 1 line

This is broken for constants that have 0xffff in the top 16 bits, as in that case, we will emit a MOVN/MOVK/MOVK sequence, and omitting the MOVKs will set the corresponding field to 0xffff not 0x0.

Thanks so much for this hint. I think you are right about the 0xffff in top 16bits case.

Quoted 6 lines

the patch breaks below usage case:
mov_q x0, 0xffffffff00000000

I think the reason is mov_q starts from high bits, if we change the macro to start from LSB, then that could solve the breakage. But this needs a rewrite of mov_q

No it has nothing to do with that.

The problem is that the use of MOVN changes the implicit value of the 16-bit fields that are left unspecified, and assigning them in a different order is not going to make any difference.

I don't think we should further complicate mov_q, and I would argue that the existing optimization (which I added myself) is premature
already: in the grand scheme of things, one or two instructions more or less are not going to make a difference anyway, given how rarely this macro is used. And even if any of these occurrences are on a hot path, it is not a given that shorter sequences of MOVZ/MOVN/MOVK are going to execute any faster, as the canonical MOVZ/MOVK/MOVK/MOVK might well decode to fewer uops.

So in summary, let's leave this code be.