ARM: p2v: reduce min alignment to 2 MiB

When running in BE mode on LPAE hardware with a PA-to-VA translation that exceeds 4 GB, we patch bits 39:32 of the offset into the wrong byte of the opcode. So fix that, by rotating the offset in r0 to the right by 8 bits, which will put the 8-bit immediate in bits 31:24.

Note that this will also move bit #22 in its correct place when applying the rotation to the constant #0x400000.

Signed-off-by Ard Biesheuvel <[email protected]>

---
 arch/arm/kernel/head.S | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

arch/arm/kernel/head.S +1 -5

index f8904227e7fd..98c1e68bdfcb 100644
@@ -671,12 +671,8 @@ ARM_BE8(rev16	ip, ip)
 	ldrcc	r7, [r4], #4	@ use branch for delay slot
 	bcc	1b
 	bx	lr
-#else
-#ifdef CONFIG_CPU_ENDIAN_BE8
-	moveq	r0, #0x00004000	@ set bit 22, mov to mvn instruction
 #else
 	moveq	r0, #0x400000	@ set bit 22, mov to mvn instruction
-#endif
 	b	2f
 1:	ldr	ip, [r7, r3]
 #ifdef CONFIG_CPU_ENDIAN_BE8

@@ -685,7 +681,7 @@ ARM_BE8(rev16	ip, ip)
 	tst	ip, #0x000f0000	@ check the rotation field
 	orrne	ip, ip, r6, lsl #24 @ mask in offset bits 31-24
 	biceq	ip, ip, #0x00004000 @ clear bit 22
-	orreq	ip, ip, r0      @ mask in offset bits 7-0
+	orreq	ip, ip, r0, ror #8  @ mask in offset bits 7-0
 #else
 	bic	ip, ip, #0x000000ff
 	tst	ip, #0xf00	@ check the rotation field

Signature · 1 line

2.17.1

Quoted 5 lines · Ard Biesheuvel

When running in BE mode on LPAE hardware with a PA-to-VA translation that exceeds 4 GB, we patch bits 39:32 of the offset into the wrong byte of the opcode. So fix that, by rotating the offset in r0 to the right by 8 bits, which will put the 8-bit immediate in bits 31:24.

Quoted 4 lines

Note that this will also move bit #22 in its correct place when applying the rotation to the constant #0x400000.

Signed-off-by Ard Biesheuvel <[email protected]>

Should this have a fixes tag?

Quoted 31 lines

---
 arch/arm/kernel/head.S | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

arch/arm/kernel/head.S +1 -6

index f8904227e7fd..98c1e68bdfcb 100644
@@ -671,12 +671,8 @@ ARM_BE8(rev16    ip, ip)
      ldrcc   r7, [r4], #4    @ use branch for delay slot
      bcc     1b
      bx      lr
-#else
-#ifdef CONFIG_CPU_ENDIAN_BE8
-     moveq   r0, #0x00004000 @ set bit 22, mov to mvn instruction
 #else
      moveq   r0, #0x400000   @ set bit 22, mov to mvn instruction
-#endif
      b       2f
 1:   ldr     ip, [r7, r3]
 #ifdef CONFIG_CPU_ENDIAN_BE8

@@ -685,7 +681,7 @@ ARM_BE8(rev16     ip, ip)
      tst     ip, #0x000f0000 @ check the rotation field
      orrne   ip, ip, r6, lsl #24 @ mask in offset bits 31-24
      biceq   ip, ip, #0x00004000 @ clear bit 22
-     orreq   ip, ip, r0      @ mask in offset bits 7-0
+     orreq   ip, ip, r0, ror #8  @ mask in offset bits 7-0
 #else
      bic     ip, ip, #0x000000ff
      tst     ip, #0xf00      @ check the rotation field
--

Quoted 3 lines

2.17.1

--
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

When running in BE mode on LPAE hardware with a PA-to-VA translation that exceeds 4 GB, we patch bits 39:32 of the offset into the wrong byte of the opcode. So fix that, by rotating the offset in r0 to the right by 8 bits, which will put the 8-bit immediate in bits 31:24.

Note that this will also move bit #22 in its correct place when applying the rotation to the constant #0x400000.

Signed-off-by Ard Biesheuvel <[email protected]>

Like arm64, ARM supports position independent code sequences that produce symbol references with a greater reach than the ordinary adr/ldr instructions. Since on ARM, the adrl pseudo-instruction is only supported in ARM mode (and not at all when using Clang), having a adr_l macro like we do on arm64 is useful, and increases symmetry as well.

Currently, we use open coded instruction sequences involving literals and arithmetic operations. Instead, we can use movw/movt pairs on v7 CPUs, circumventing the D-cache entirely.

E.g., on v7+ CPUs, we can emit a PC-relative reference as follows:

       movw         <reg>, #:lower16:<sym> - (1f + 8)
       movt         <reg>, #:upper16:<sym> - (1f + 8)
  1:   add          <reg>, <reg>, pc

For older CPUs, we can emit the literal into a subsection, allowing it to be emitted out of line while retaining the ability to perform arithmetic on label offsets.

E.g., on pre-v7 CPUs, we can emit a PC-relative reference as follows:

       ldr          <reg>, 2f
  1:   add          <reg>, <reg>, pc
       .subsection  1
  2:   .long        <sym> - (1b + 8)

.previous

This is allowed by the assembler because, unlike ordinary sections, subsections are combined into a single section in the object file, and so the label references are not true cross-section references that are visible as relocations. (Subsections have been available in binutils since 2004 at least, so they should not cause any issues with older toolchains.)

So use the above to implement the macros mov_l, adr_l, ldr_l and str_l, all of which will use movw/movt pairs on v7 and later CPUs, and use PC-relative literals otherwise.

Reviewed-by Nicolas Pitre <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

Move the phys2virt patching code into a separate .S file before doing some work on it.

Suggested-by Nicolas Pitre <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

The ARM and Thumb2 versions of the p2v patching loop have some overlap at the end of the loop, so factor that out.

Signed-off-by Ard Biesheuvel <[email protected]>

Quoted 4 lines · Ard Biesheuvel

The ARM and Thumb2 versions of the p2v patching loop have some overlap at the end of the loop, so factor that out.

Signed-off-by Ard Biesheuvel <[email protected]>

Would add to commit message: "changes numeric labels to symbolic labels while we're at it".

Reviewed-by Linus Walleij <[email protected]>

The big and little endian versions of the ARM p2v patching routine only differ in the values of the constants, so factor those out into macros so that we only have one version of the logic sequence to maintain.

Signed-off-by Ard Biesheuvel <[email protected]>

Quoted 4 lines · Ard Biesheuvel

The big and little endian versions of the ARM p2v patching routine only differ in the values of the constants, so factor those out into macros so that we only have one version of the logic sequence to maintain.

Quoted 1 line

Signed-off-by Ard Biesheuvel <[email protected]>

Since you had patch 1 fixing up BE on LPAE I suppose it is implicit in the commit that this was tested with BE on classic MMU and LPAE. Very nice patch!

Reviewed-by Linus Walleij <[email protected]>

We always pass the same value for 'type' so pull it into the __pv_stub macro itself.

Signed-off-by Ard Biesheuvel <[email protected]>

Free up a register in the p2v patching code by switching to relative references, which don't require keeping the phys-to-virt displacement live in a register.

Signed-off-by Ard Biesheuvel <[email protected]>

Quoted 5 lines · Ard Biesheuvel

Free up a register in the p2v patching code by switching to relative references, which don't require keeping the phys-to-virt displacement live in a register.

Signed-off-by Ard Biesheuvel <[email protected]>

Neat!

Reviewed-by Linus Walleij <[email protected]>

Declutter the code in __fixup_pv_table() by using the new adr_l/str_l macros to take PC relative references to external symbols, and by using the value of PHYS_OFFSET passed in r8 to calculate the p2v offset.

Signed-off-by Ard Biesheuvel <[email protected]>

In preparation for reducing the phys-to-virt minimum relative alignment from 16 MiB to 2 MiB, switch to patchable sequences involving MOVW instructions that can more easily be manipulated to carry a 12-bit immediate. Note that the non-LPAE ARM sequence is not updated: MOVW may not be supported on non-LPAE platforms, and the sequence itself can be updated more easily to apply the 12 bits of displacement.

For Thumb2, which has many more versions of opcodes, switch to a sequence that can be patched by the same patching code for both versions. Note that the Thumb2 opcodes for MOVW and MVN are unambiguous, and have no rotation bits in their immediate fields, so there is no need to use placeholder constants in the asm blocks.

While at it, drop the 'volatile' qualifiers from the asm blocks: the code does not have any side effects that are invisible to the compiler, so it is free to omit these sequences if the outputs are not used.

Suggested-by Russell King <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

---
 arch/arm/include/asm/memory.h |  44 ++++--
 arch/arm/kernel/phys2virt.S   | 147 +++++++++++++++-----
 2 files changed, 148 insertions(+), 43 deletions(-)

arch/arm/include/asm/memory.h +32 -12

index 4121662dea5a..ccf55cef6ab9 100644
@@ -183,6 +183,7 @@ extern const void *__pv_table_begin, *__pv_table_end;
 #define PHYS_OFFSET	((phys_addr_t)__pv_phys_pfn_offset << PAGE_SHIFT)
 #define PHYS_PFN_OFFSET	(__pv_phys_pfn_offset)
 
+#ifndef CONFIG_THUMB2_KERNEL
 #define __pv_stub(from,to,instr)			\
 	__asm__("@ __pv_stub\n"				\
 	"1:	" instr "	%0, %1, %2\n"		\

@@ -192,25 +193,45 @@ extern const void *__pv_table_begin, *__pv_table_end;
 	: "=r" (to)					\
 	: "r" (from), "I" (__PV_BITS_31_24))
 
-#define __pv_stub_mov_hi(t)				\
-	__asm__ volatile("@ __pv_stub_mov\n"		\
-	"1:	mov	%R0, %1\n"			\
+#define __pv_add_carry_stub(x, y)			\
+	__asm__("@ __pv_add_carry_stub\n"		\
+	"0:	movw	%R0, #0\n"			\
+	"	adds	%Q0, %1, %R0, lsl #24\n"	\
+	"1:	mov	%R0, %2\n"			\
+	"	adc	%R0, %R0, #0\n"			\
 	"	.pushsection .pv_table,\"a\"\n"		\
-	"	.long	1b - .\n"			\
+	"	.long	0b - ., 1b - .\n"		\
 	"	.popsection\n"				\
-	: "=r" (t)					\
-	: "I" (__PV_BITS_7_0))
+	: "=&r" (y)					\
+	: "r" (x), "I" (__PV_BITS_7_0)			\
+	: "cc")
+
+#else
+#define __pv_stub(from,to,instr)			\
+	__asm__("@ __pv_stub\n"				\
+	"0:	movw	%0, #0\n"			\
+	"	lsl	%0, #24\n"			\
+	"	" instr " %0, %1, %0\n"			\
+	"	.pushsection .pv_table,\"a\"\n"		\
+	"	.long	0b - .\n"			\
+	"	.popsection\n"				\
+	: "=&r" (to)					\
+	: "r" (from))
 
 #define __pv_add_carry_stub(x, y)			\
-	__asm__ volatile("@ __pv_add_carry_stub\n"	\
-	"1:	adds	%Q0, %1, %2\n"			\
+	__asm__("@ __pv_add_carry_stub\n"		\
+	"0:	movw	%R0, #0\n"			\
+	"	lsls	%R0, #24\n"			\
+	"	adds	%Q0, %1, %R0\n"			\
+	"1:	mvn	%R0, #0\n"			\
 	"	adc	%R0, %R0, #0\n"			\
 	"	.pushsection .pv_table,\"a\"\n"		\
-	"	.long	1b - .\n"			\
+	"	.long	0b - ., 1b - .\n"		\
 	"	.popsection\n"				\
-	: "+r" (y)					\
-	: "r" (x), "I" (__PV_BITS_31_24)		\
+	: "=&r" (y)					\
+	: "r" (x)					\
 	: "cc")
+#endif
 
 static inline phys_addr_t __virt_to_phys_nodebug(unsigned long x)
 {

@@ -219,7 +240,6 @@ static inline phys_addr_t __virt_to_phys_nodebug(unsigned long x)
 	if (sizeof(phys_addr_t) == 4) {
 		__pv_stub(x, t, "add");
 	} else {
-		__pv_stub_mov_hi(t);
 		__pv_add_carry_stub(x, t);
 	}
 	return t;

arch/arm/kernel/phys2virt.S +116 -31

index be8fb0d89877..a4e364689663 100644
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 /*
  *  Copyright (C) 1994-2002 Russell King
- *  Copyright (c) 2003 ARM Limited
+ *  Copyright (c) 2003, 2020 ARM Limited
  *  All Rights Reserved
  */
 

@@ -58,55 +58,140 @@ __fixup_a_pv_table:
 	mov	r6, r6, lsr #24
 	cmn	r0, #1
 #ifdef CONFIG_THUMB2_KERNEL
+	@
+	@ The Thumb-2 versions of the patchable sequences are
+	@
+	@ phys-to-virt:			movw	<reg>, #offset<31:24>
+	@				lsl	<reg>, #24
+	@				sub	<VA>, <PA>, <reg>
+	@
+	@ virt-to-phys (non-LPAE):	movw	<reg>, #offset<31:24>
+	@				lsl	<reg>, #24
+	@				add	<PA>, <VA>, <reg>
+	@
+	@ virt-to-phys (LPAE):		movw	<reg>, #offset<31:24>
+	@				lsl	<reg>, #24
+	@				adds	<PAlo>, <VA>, <reg>
+	@				mov	<PAhi>, #offset<39:32>
+	@				adc	<PAhi>, <PAhi>, #0
+	@
+	@ In the non-LPAE case, all patchable instructions are MOVW
+	@ instructions, where we need to patch in the offset into the
+	@ second halfword of the opcode (the 16-bit immediate is encoded
+	@ as imm4:i:imm3:imm8)
+	@
+	@       15       11 10  9           4 3    0  15  14  12 11 8 7    0
+	@      +-----------+---+-------------+------++---+------+----+------+
+	@ MOVW | 1 1 1 1 0 | i | 1 0 0 1 0 0 | imm4 || 0 | imm3 | Rd | imm8 |
+	@      +-----------+---+-------------+------++---+------+----+------+
+	@
+	@ In the LPAE case, we also need to patch in the high word of the
+	@ offset into the immediate field of the MOV instruction, or patch it
+	@ to a MVN instruction if the offset is negative. In this case, we
+	@ need to inspect the first halfword of the opcode, to check whether
+	@ it is MOVW or MOV/MVN, and to perform the MOV to MVN patching if
+	@ needed. The encoding of the immediate is rather complex for values
+	@ of i:imm3 != 0b0000, but fortunately, we never need more than 8 lower
+	@ order bits, which can be patched into imm8 directly (and i:imm3
+	@ cleared)
+	@
+	@      15       11 10  9        5         0  15  14  12 11 8 7    0
+	@     +-----------+---+---------------------++---+------+----+------+
+	@ MOV | 1 1 1 1 0 | i | 0 0 0 1 0 0 1 1 1 1 || 0 | imm3 | Rd | imm8 |
+	@ MVN | 1 1 1 1 0 | i | 0 0 0 1 1 0 1 1 1 1 || 0 | imm3 | Rd | imm8 |
+	@     +-----------+---+---------------------++---+------+----+------+
+	@
 	moveq	r0, #0x200000		@ set bit 21, mov to mvn instruction
-	lsls	r6, #24
-	beq	.Lnext
-	clz	r7, r6
-	lsr	r6, #24
-	lsl	r6, r7
-	bic	r6, #0x0080
-	lsrs	r7, #1
-	orrcs	r6, #0x0080
-	orr	r6, r6, r7, lsl #12
-	orr	r6, #0x4000
 	b	.Lnext
 .Lloop:	add	r7, r4
-	adds	r4, #4
-	ldrh	ip, [r7, #2]
-ARM_BE8(rev16	ip, ip)
-	tst	ip, #0x4000
-	and	ip, #0x8f00
-	orrne	ip, r6			@ mask in offset bits 31-24
-	orreq	ip, r0			@ mask in offset bits 7-0
-ARM_BE8(rev16	ip, ip)
-	strh	ip, [r7, #2]
-	bne	.Lnext
+	adds	r4, #4			@ clears Z flag
+#ifdef CONFIG_ARM_LPAE
 	ldrh	ip, [r7]
 ARM_BE8(rev16	ip, ip)
-	bic	ip, #0x20
-	orr	ip, ip, r0, lsr #16
+	tst	ip, #0x200		@ MOVW has bit 9 set, MVN has it clear
+	bne	0f			@ skip to MOVW handling (Z flag is clear)
+	bic	ip, #0x20		@ clear bit 5 (MVN -> MOV)
+	orr	ip, ip, r0, lsr #16	@ MOV -> MVN if offset < 0
 ARM_BE8(rev16	ip, ip)
 	strh	ip, [r7]
+	@ Z flag is set
+0:
+#endif
+	ldrh	ip, [r7, #2]
+ARM_BE8(rev16	ip, ip)
+	and	ip, #0xf00		@ clear everything except Rd field
+	orreq	ip, r0			@ Z flag set -> MOV/MVN -> patch in high bits
+	orrne	ip, r6			@ Z flag clear -> MOVW -> patch in low bits
+ARM_BE8(rev16	ip, ip)
+	strh	ip, [r7, #2]
 #else
 #ifdef CONFIG_CPU_ENDIAN_BE8
 @ in BE8, we load data in BE, but instructions still in LE
-#define PV_BIT22	0x00004000
+#define PV_BIT24	0x00000001
 #define PV_IMM8_MASK	0xff000000
-#define PV_ROT_MASK	0x000f0000
 #else
-#define PV_BIT22	0x00400000
+#define PV_BIT24	0x01000000
 #define PV_IMM8_MASK	0x000000ff
-#define PV_ROT_MASK	0xf00
 #endif
 
+	@
+	@ The ARM versions of the patchable sequences are
+	@
+	@ phys-to-virt:			sub	<VA>, <PA>, #offset<31:24>, lsl #24
+	@
+	@ virt-to-phys (non-LPAE):	add	<PA>, <VA>, #offset<31:24>, lsl #24
+	@
+	@ virt-to-phys (LPAE):		movw	<reg>, #offset<31:24>
+	@				adds	<PAlo>, <VA>, <reg>, lsl #24
+	@				mov	<PAhi>, #offset<39:32>
+	@				adc	<PAhi>, <PAhi>, #0
+	@
+	@ In the non-LPAE case, all patchable instructions are ADD or SUB
+	@ instructions, where we need to patch in the offset into the
+	@ immediate field of the opcode, which is emitted with the correct
+	@ rotation value. (The effective value of the immediate is imm12<7:0>
+	@ rotated right by [2 * imm12<11:8>] bits)
+	@
+	@      31   28 27      23 22  20 19  16 15  12 11    0
+	@      +------+-----------------+------+------+-------+
+	@  ADD | cond | 0 0 1 0 1 0 0 0 |  Rn  |  Rd  | imm12 |
+	@  SUB | cond | 0 0 1 0 0 1 0 0 |  Rn  |  Rd  | imm12 |
+	@  MOV | cond | 0 0 1 1 1 0 1 0 |  Rn  |  Rd  | imm12 |
+	@  MVN | cond | 0 0 1 1 1 1 1 0 |  Rn  |  Rd  | imm12 |
+	@      +------+-----------------+------+------+-------+
+	@
+	@ In the LPAE case, we use a MOVW instruction to carry the low offset
+	@ word, and patch in the high word of the offset into the immediate
+	@ field of the subsequent MOV instruction, or patch it to a MVN
+	@ instruction if the offset is negative. We can distinguish MOVW
+	@ instructions based on bits 23:22 of the opcode, and ADD/SUB can be
+	@ distinguished from MOV/MVN (all using the encodings above) using
+	@ bit 24.
+	@
+	@      31   28 27      23 22  20 19  16 15  12 11    0
+	@      +------+-----------------+------+------+-------+
+	@ MOVW | cond | 0 0 1 1 0 0 0 0 | imm4 |  Rd  | imm12 |
+	@      +------+-----------------+------+------+-------+
+	@
 	moveq	r0, #0x400000		@ set bit 22, mov to mvn instruction
 	b	.Lnext
 .Lloop:	ldr	ip, [r7, r4]
+#ifdef CONFIG_ARM_LPAE
+	tst	ip, #PV_BIT24		@ ADD/SUB have bit 24 clear
+	beq	1f
+ARM_BE8(rev	ip, ip)
+	tst	ip, #0xc00000		@ MOVW has bits 23:22 clear
+	bic	ip, ip, #0x400000	@ clear bit 22
+	bfc	ip, #0, #12		@ clear imm12 field of MOV[W] instruction
+	orreq	ip, ip, r6		@ MOVW -> mask in offset bits 31-24
+	orrne	ip, ip, r0		@ MOV  -> mask in offset bits 7-0 (or bit 22)
+ARM_BE8(rev	ip, ip)
+	b	2f
+1:
+#endif
 	bic	ip, ip, #PV_IMM8_MASK
-	tst	ip, #PV_ROT_MASK		@ check the rotation field
-	orrne	ip, ip, r6 ARM_BE8(, lsl #24)	@ mask in offset bits 31-24
-	biceq	ip, ip, #PV_BIT22		@ clear bit 22
-	orreq	ip, ip, r0 ARM_BE8(, ror #8)	@ mask in offset bits 7-0 (or bit 22)
+	orr	ip, ip, r6 ARM_BE8(, lsl #24)	@ mask in offset bits 31-24
+2:
 	str	ip, [r7, r4]
 	add	r4, r4, #4
 #endif

Signature · 1 line

2.17.1

Quoted 6 lines · Ard Biesheuvel

In preparation for reducing the phys-to-virt minimum relative alignment from 16 MiB to 2 MiB, switch to patchable sequences involving MOVW instructions that can more easily be manipulated to carry a 12-bit immediate. Note that the non-LPAE ARM sequence is not updated: MOVW may not be supported on non-LPAE platforms, and the sequence itself can be updated more easily to apply the 12 bits of displacement.

I'm still wondering what is the advantage of a movw+lsl vs two adds? I think I remember but I'd have to go back in the ARM ARM to be sure. You could elaborate a bit more on the "more easily be manipulated" please. No need to resend just for this.

Quoted 9 lines

For Thumb2, which has many more versions of opcodes, switch to a sequence that can be patched by the same patching code for both versions. Note that the Thumb2 opcodes for MOVW and MVN are unambiguous, and have no rotation bits in their immediate fields, so there is no need to use placeholder constants in the asm blocks.

While at it, drop the 'volatile' qualifiers from the asm blocks: the code does not have any side effects that are invisible to the compiler, so it is free to omit these sequences if the outputs are not used.

Indeed. Weird that the volatile was there in the first place.

Quoted 273 lines

Suggested-by Russell King <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

---
 arch/arm/include/asm/memory.h |  44 ++++--
 arch/arm/kernel/phys2virt.S   | 147 +++++++++++++++-----
 2 files changed, 148 insertions(+), 43 deletions(-)

arch/arm/include/asm/memory.h +32 -12

index 4121662dea5a..ccf55cef6ab9 100644
@@ -183,6 +183,7 @@ extern const void *__pv_table_begin, *__pv_table_end;
 #define PHYS_OFFSET  ((phys_addr_t)__pv_phys_pfn_offset << PAGE_SHIFT)
 #define PHYS_PFN_OFFSET      (__pv_phys_pfn_offset)
 
+#ifndef CONFIG_THUMB2_KERNEL
 #define __pv_stub(from,to,instr)                     \
      __asm__("@ __pv_stub\n"                         \
      "1:     " instr "       %0, %1, %2\n"           \

@@ -192,25 +193,45 @@ extern const void *__pv_table_begin, *__pv_table_end;
      : "=r" (to)                                     \
      : "r" (from), "I" (__PV_BITS_31_24))
 
-#define __pv_stub_mov_hi(t)                          \
-     __asm__ volatile("@ __pv_stub_mov\n"            \
-     "1:     mov     %R0, %1\n"                      \
+#define __pv_add_carry_stub(x, y)                    \
+     __asm__("@ __pv_add_carry_stub\n"               \
+     "0:     movw    %R0, #0\n"                      \
+     "       adds    %Q0, %1, %R0, lsl #24\n"        \
+     "1:     mov     %R0, %2\n"                      \
+     "       adc     %R0, %R0, #0\n"                 \
      "       .pushsection .pv_table,\"a\"\n"         \
-     "       .long   1b - .\n"                       \
+     "       .long   0b - ., 1b - .\n"               \
      "       .popsection\n"                          \
-     : "=r" (t)                                      \
-     : "I" (__PV_BITS_7_0))
+     : "=&r" (y)                                     \
+     : "r" (x), "I" (__PV_BITS_7_0)                  \
+     : "cc")
+
+#else
+#define __pv_stub(from,to,instr)                     \
+     __asm__("@ __pv_stub\n"                         \
+     "0:     movw    %0, #0\n"                       \
+     "       lsl     %0, #24\n"                      \
+     "       " instr " %0, %1, %0\n"                 \
+     "       .pushsection .pv_table,\"a\"\n"         \
+     "       .long   0b - .\n"                       \
+     "       .popsection\n"                          \
+     : "=&r" (to)                                    \
+     : "r" (from))
 
 #define __pv_add_carry_stub(x, y)                    \
-     __asm__ volatile("@ __pv_add_carry_stub\n"      \
-     "1:     adds    %Q0, %1, %2\n"                  \
+     __asm__("@ __pv_add_carry_stub\n"               \
+     "0:     movw    %R0, #0\n"                      \
+     "       lsls    %R0, #24\n"                     \
+     "       adds    %Q0, %1, %R0\n"                 \
+     "1:     mvn     %R0, #0\n"                      \
      "       adc     %R0, %R0, #0\n"                 \
      "       .pushsection .pv_table,\"a\"\n"         \
-     "       .long   1b - .\n"                       \
+     "       .long   0b - ., 1b - .\n"               \
      "       .popsection\n"                          \
-     : "+r" (y)                                      \
-     : "r" (x), "I" (__PV_BITS_31_24)                \
+     : "=&r" (y)                                     \
+     : "r" (x)                                       \
      : "cc")
+#endif
 
 static inline phys_addr_t __virt_to_phys_nodebug(unsigned long x)
 {

@@ -219,7 +240,6 @@ static inline phys_addr_t __virt_to_phys_nodebug(unsigned long x)
      if (sizeof(phys_addr_t) == 4) {
              __pv_stub(x, t, "add");
      } else {
-             __pv_stub_mov_hi(t);
              __pv_add_carry_stub(x, t);
      }
      return t;

arch/arm/kernel/phys2virt.S +116 -32

index be8fb0d89877..a4e364689663 100644
@@ -1,7 +1,7 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 /*
  *  Copyright (C) 1994-2002 Russell King
- *  Copyright (c) 2003 ARM Limited
+ *  Copyright (c) 2003, 2020 ARM Limited
  *  All Rights Reserved
  */
 

@@ -58,55 +58,140 @@ __fixup_a_pv_table:
      mov     r6, r6, lsr #24
      cmn     r0, #1
 #ifdef CONFIG_THUMB2_KERNEL
+     @
+     @ The Thumb-2 versions of the patchable sequences are
+     @
+     @ phys-to-virt:                 movw    <reg>, #offset<31:24>
+     @                               lsl     <reg>, #24
+     @                               sub     <VA>, <PA>, <reg>
+     @
+     @ virt-to-phys (non-LPAE):      movw    <reg>, #offset<31:24>
+     @                               lsl     <reg>, #24
+     @                               add     <PA>, <VA>, <reg>
+     @
+     @ virt-to-phys (LPAE):          movw    <reg>, #offset<31:24>
+     @                               lsl     <reg>, #24
+     @                               adds    <PAlo>, <VA>, <reg>
+     @                               mov     <PAhi>, #offset<39:32>
+     @                               adc     <PAhi>, <PAhi>, #0
+     @
+     @ In the non-LPAE case, all patchable instructions are MOVW
+     @ instructions, where we need to patch in the offset into the
+     @ second halfword of the opcode (the 16-bit immediate is encoded
+     @ as imm4:i:imm3:imm8)
+     @
+     @       15       11 10  9           4 3    0  15  14  12 11 8 7    0
+     @      +-----------+---+-------------+------++---+------+----+------+
+     @ MOVW | 1 1 1 1 0 | i | 1 0 0 1 0 0 | imm4 || 0 | imm3 | Rd | imm8 |
+     @      +-----------+---+-------------+------++---+------+----+------+
+     @
+     @ In the LPAE case, we also need to patch in the high word of the
+     @ offset into the immediate field of the MOV instruction, or patch it
+     @ to a MVN instruction if the offset is negative. In this case, we
+     @ need to inspect the first halfword of the opcode, to check whether
+     @ it is MOVW or MOV/MVN, and to perform the MOV to MVN patching if
+     @ needed. The encoding of the immediate is rather complex for values
+     @ of i:imm3 != 0b0000, but fortunately, we never need more than 8 lower
+     @ order bits, which can be patched into imm8 directly (and i:imm3
+     @ cleared)
+     @
+     @      15       11 10  9        5         0  15  14  12 11 8 7    0
+     @     +-----------+---+---------------------++---+------+----+------+
+     @ MOV | 1 1 1 1 0 | i | 0 0 0 1 0 0 1 1 1 1 || 0 | imm3 | Rd | imm8 |
+     @ MVN | 1 1 1 1 0 | i | 0 0 0 1 1 0 1 1 1 1 || 0 | imm3 | Rd | imm8 |
+     @     +-----------+---+---------------------++---+------+----+------+
+     @
      moveq   r0, #0x200000           @ set bit 21, mov to mvn instruction
-     lsls    r6, #24
-     beq     .Lnext
-     clz     r7, r6
-     lsr     r6, #24
-     lsl     r6, r7
-     bic     r6, #0x0080
-     lsrs    r7, #1
-     orrcs   r6, #0x0080
-     orr     r6, r6, r7, lsl #12
-     orr     r6, #0x4000
      b       .Lnext
 .Lloop:      add     r7, r4
-     adds    r4, #4
-     ldrh    ip, [r7, #2]
-ARM_BE8(rev16        ip, ip)
-     tst     ip, #0x4000
-     and     ip, #0x8f00
-     orrne   ip, r6                  @ mask in offset bits 31-24
-     orreq   ip, r0                  @ mask in offset bits 7-0
-ARM_BE8(rev16        ip, ip)
-     strh    ip, [r7, #2]
-     bne     .Lnext
+     adds    r4, #4                  @ clears Z flag
+#ifdef CONFIG_ARM_LPAE
      ldrh    ip, [r7]
 ARM_BE8(rev16        ip, ip)
-     bic     ip, #0x20
-     orr     ip, ip, r0, lsr #16
+     tst     ip, #0x200              @ MOVW has bit 9 set, MVN has it clear
+     bne     0f                      @ skip to MOVW handling (Z flag is clear)
+     bic     ip, #0x20               @ clear bit 5 (MVN -> MOV)
+     orr     ip, ip, r0, lsr #16     @ MOV -> MVN if offset < 0
 ARM_BE8(rev16        ip, ip)
      strh    ip, [r7]
+     @ Z flag is set
+0:
+#endif
+     ldrh    ip, [r7, #2]
+ARM_BE8(rev16        ip, ip)
+     and     ip, #0xf00              @ clear everything except Rd field
+     orreq   ip, r0                  @ Z flag set -> MOV/MVN -> patch in high bits
+     orrne   ip, r6                  @ Z flag clear -> MOVW -> patch in low bits
+ARM_BE8(rev16        ip, ip)
+     strh    ip, [r7, #2]
 #else
 #ifdef CONFIG_CPU_ENDIAN_BE8
 @ in BE8, we load data in BE, but instructions still in LE
-#define PV_BIT22     0x00004000
+#define PV_BIT24     0x00000001
 #define PV_IMM8_MASK 0xff000000
-#define PV_ROT_MASK  0x000f0000
 #else
-#define PV_BIT22     0x00400000
+#define PV_BIT24     0x01000000
 #define PV_IMM8_MASK 0x000000ff
-#define PV_ROT_MASK  0xf00
 #endif
 
+     @
+     @ The ARM versions of the patchable sequences are
+     @
+     @ phys-to-virt:                 sub     <VA>, <PA>, #offset<31:24>, lsl #24
+     @
+     @ virt-to-phys (non-LPAE):      add     <PA>, <VA>, #offset<31:24>, lsl #24
+     @
+     @ virt-to-phys (LPAE):          movw    <reg>, #offset<31:24>
+     @                               adds    <PAlo>, <VA>, <reg>, lsl #24
+     @                               mov     <PAhi>, #offset<39:32>
+     @                               adc     <PAhi>, <PAhi>, #0
+     @
+     @ In the non-LPAE case, all patchable instructions are ADD or SUB
+     @ instructions, where we need to patch in the offset into the
+     @ immediate field of the opcode, which is emitted with the correct
+     @ rotation value. (The effective value of the immediate is imm12<7:0>
+     @ rotated right by [2 * imm12<11:8>] bits)
+     @
+     @      31   28 27      23 22  20 19  16 15  12 11    0
+     @      +------+-----------------+------+------+-------+
+     @  ADD | cond | 0 0 1 0 1 0 0 0 |  Rn  |  Rd  | imm12 |
+     @  SUB | cond | 0 0 1 0 0 1 0 0 |  Rn  |  Rd  | imm12 |
+     @  MOV | cond | 0 0 1 1 1 0 1 0 |  Rn  |  Rd  | imm12 |
+     @  MVN | cond | 0 0 1 1 1 1 1 0 |  Rn  |  Rd  | imm12 |
+     @      +------+-----------------+------+------+-------+
+     @
+     @ In the LPAE case, we use a MOVW instruction to carry the low offset
+     @ word, and patch in the high word of the offset into the immediate
+     @ field of the subsequent MOV instruction, or patch it to a MVN
+     @ instruction if the offset is negative. We can distinguish MOVW
+     @ instructions based on bits 23:22 of the opcode, and ADD/SUB can be
+     @ distinguished from MOV/MVN (all using the encodings above) using
+     @ bit 24.
+     @
+     @      31   28 27      23 22  20 19  16 15  12 11    0
+     @      +------+-----------------+------+------+-------+
+     @ MOVW | cond | 0 0 1 1 0 0 0 0 | imm4 |  Rd  | imm12 |
+     @      +------+-----------------+------+------+-------+
+     @
      moveq   r0, #0x400000           @ set bit 22, mov to mvn instruction
      b       .Lnext
 .Lloop:      ldr     ip, [r7, r4]
+#ifdef CONFIG_ARM_LPAE
+     tst     ip, #PV_BIT24           @ ADD/SUB have bit 24 clear
+     beq     1f
+ARM_BE8(rev  ip, ip)
+     tst     ip, #0xc00000           @ MOVW has bits 23:22 clear
+     bic     ip, ip, #0x400000       @ clear bit 22
+     bfc     ip, #0, #12             @ clear imm12 field of MOV[W] instruction
+     orreq   ip, ip, r6              @ MOVW -> mask in offset bits 31-24
+     orrne   ip, ip, r0              @ MOV  -> mask in offset bits 7-0 (or bit 22)
+ARM_BE8(rev  ip, ip)
+     b       2f
+1:
+#endif
      bic     ip, ip, #PV_IMM8_MASK
-     tst     ip, #PV_ROT_MASK                @ check the rotation field
-     orrne   ip, ip, r6 ARM_BE8(, lsl #24)   @ mask in offset bits 31-24
-     biceq   ip, ip, #PV_BIT22               @ clear bit 22
-     orreq   ip, ip, r0 ARM_BE8(, ror #8)    @ mask in offset bits 7-0 (or bit 22)
+     orr     ip, ip, r6 ARM_BE8(, lsl #24)   @ mask in offset bits 31-24
+2:
      str     ip, [r7, r4]
      add     r4, r4, #4
 #endif
--

2.17.1

In preparation for reducing the phys-to-virt minimum relative alignment from 16 MiB to 2 MiB, switch to patchable sequences involving MOVW instructions that can more easily be manipulated to carry a 12-bit immediate. Note that the non-LPAE ARM sequence is not updated: MOVW may not be supported on non-LPAE platforms, and the sequence itself can be updated more easily to apply the 12 bits of displacement.

For Thumb2, which has many more versions of opcodes, switch to a sequence that can be patched by the same patching code for both versions. Note that the Thumb2 opcodes for MOVW and MVN are unambiguous, and have no rotation bits in their immediate fields, so there is no need to use placeholder constants in the asm blocks.

While at it, drop the 'volatile' qualifiers from the asm blocks: the code does not have any side effects that are invisible to the compiler, so it is free to omit these sequences if the outputs are not used.

Suggested-by Russell King <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

Update the p2v patching code so we can deal with displacements that are not a multiple of 16 MiB but of 2 MiB, to prevent wasting of up to 14 MiB of physical RAM when running on a platform where the start of memory is not correctly aligned.

For the ARM code path, this simply comes down to using two add/sub instructions instead of one for the carryless version, and patching each of them with the correct immediate depending on the rotation field. For the LPAE calculation, it patches the MOVW instruction with up to 12 bits of offset.

For the Thumb2 code path, patching more than 11 bits off displacement is somewhat cumbersome, and given that 11 bits produce a minimum alignment of 2 MiB, which is also the granularity for LPAE block mappings, it makes sense to stick to 2 MiB for the new p2v requirement.

Suggested-by Zhen Lei <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

Quoted 11 lines · Ard Biesheuvel

Update the p2v patching code so we can deal with displacements that are not a multiple of 16 MiB but of 2 MiB, to prevent wasting of up to 14 MiB of physical RAM when running on a platform where the start of memory is not correctly aligned.

For the ARM code path, this simply comes down to using two add/sub instructions instead of one for the carryless version, and patching each of them with the correct immediate depending on the rotation field. For the LPAE calculation, it patches the MOVW instruction with up to 12 bits of offset.

Quoted 7 lines

For the Thumb2 code path, patching more than 11 bits off displacement is somewhat cumbersome, and given that 11 bits produce a minimum alignment of 2 MiB, which is also the granularity for LPAE block mappings, it makes sense to stick to 2 MiB for the new p2v requirement.

Suggested-by Zhen Lei <[email protected]>

Signed-off-by Ard Biesheuvel <[email protected]>

My understanding of what is going on is limited to the high level of things, and being able to do this is just a great thing so FWIW:

Acked-by Linus Walleij <[email protected]>

If you or Russell need more thorough review I can sit down and try to understand at the bit granularity what is going on but it requires a bunch of time. Just tell me if you need this.

Quoted 1 line

8 == 16 MiB of granularity, and so if the start of physical RAM is

By changing the patchable sequences and the patching logic to carry more bits of offset, we can improve this: 11 bits gives us 4 GiB >> 11 == 2 MiB granularity, and so you never waste more than that amount by rounding up the physical start of DRAM to the next multiple of 2 MiB. (Note that 2 MiB granularity guarantees that the linear mapping can be created efficiently, whereas less than 2 MiB may result in the linear mapping needing another level of page tables)

Quoted 7 lines · Ard Biesheuvel

By changing the patchable sequences and the patching logic to carry more bits of offset, we can improve this: 11 bits gives us 4 GiB >> 11 == 2 MiB granularity, and so you never waste more than that amount by rounding up the physical start of DRAM to the next multiple of 2 MiB. (Note that 2 MiB granularity guarantees that the linear mapping can be created efficiently, whereas less than 2 MiB may result in the linear mapping needing another level of page tables)

That is IMHO the top argument for limiting it to 2 MiB. It would be worth making this explicit in the commit log.

This series is inspired by Zhei Len's series [0], which updates the ARM p2v patching code to optionally support p2v relative alignments of as little as 64 KiB.

Reducing this alignment is necessary for some specific Huawei boards, but given that reducing this minimum alignment will make the boot sequence more robust for all platforms, especially EFI boot, which no longer relies on the 128 MB masking of the decompressor load address, but uses firmware memory allocation routines to find a suitable spot for the decompressed kernel.

This series is not based on Zhei Len's code, but addresses the same problem, and takes some feedback given in the review into account:
- use of a MOVW instruction to avoid two adds/adcs sequences when dealing with the carry on LPAE
- add support for Thumb2 kernels as well
- make the change unconditional - it will bit rot otherwise, and has value for other platforms as well.

Patch #1 fixes a BE8 bug in the existing code.

Patch #2 incorporates a patch from a different series [1] that introduces the adr_l and str_l macros, for use in subsequent patches.

Patch #3 moves the p2v patching code into a separate assembler source file, as suggested by Nico.

Patches #4 to #8 perform some preparatory clean up to make the real changes easier to review, and more maintainable going forward.

Patch #9 switches the ARM/LPAE and Thumb2 p2v patchable sequences to MOVW, so that we can extend the size of the immediate field in a later patch.

Patch #10 increases the size of the low order immediate to 11 bits, so that we can deal with any physical to virtual displacement that is a multiple of 2 MiB.

Tested in QEMU using various permutations of the the following configuration options:
- ARM mode vs Thumb2 mode
- LE vs BE8
- LPAE vs non-LPAE
- 3/1 split, 2/2 split, 3/1 opt split

with the following diff applied to decompress the kernel at an address which is not 16 MiB aligned:

arch/arm/boot/compressed/head.S +1 -0

@@ -275,6 +275,7 @@ not_angel:
                and     r4, r4, #0xf8000000
                /* Determine final kernel image address. */
                add     r4, r4, #TEXT_OFFSET
+               add     r4, r4, #SZ_2M
 #else
                ldr     r4, =zreladdr
 #endif

Changes since RFC/RFT:
- new patches #1 - #3 and #8
- drop bogus patching of ADD/SUB instructions, which is unnecessary, and only happens to work when the p2v displacement is exactly 2 GiB like on QEMU when running a 3/1 split
- use a copious amount of comments to make the code more understandable and maintainable

Cc Zhen Lei <[email protected]>

Cc Russell King <[email protected]>

Cc Linus Walleij <[email protected]>

Cc Nicolas Pitre <[email protected]>

[0] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/

Ard Biesheuvel (10):
ARM: p2v: fix handling of LPAE translation in BE mode
ARM: assembler: introduce adr_l, ldr_l and str_l macros
ARM: p2v: move patching code to separate assembler source file
ARM: p2v: factor out shared loop processing
ARM: p2v: factor out BE8 handling
ARM: p2v: drop redundant 'type' argument from __pv_stub
ARM: p2v: use relative references in patch site arrays
ARM: p2v: simplify __fixup_pv_table()
ARM: p2v: switch to MOVW for Thumb2 and ARM/LPAE
ARM: p2v: reduce p2v alignment requirement to 2 MiB

 arch/arm/Kconfig                 |   2 +-
 arch/arm/include/asm/assembler.h |  84 +++++++
 arch/arm/include/asm/memory.h    |  57 +++--
 arch/arm/kernel/Makefile         |   1 +
 arch/arm/kernel/head.S           | 142 ------------
 arch/arm/kernel/phys2virt.S      | 238 ++++++++++++++++++++
 6 files changed, 364 insertions(+), 160 deletions(-)
 create mode 100644 arch/arm/kernel/phys2virt.S

Signature · 1 line

2.17.1

This series is inspired by Zhei Len's series [0], which updates the ARM p2v patching code to optionally support p2v relative alignments of as little as 64 KiB.