Skip to contentSkip to conversation

arm64: mm: force write fault for atomic RMW instructions

PATCH Superseded Yang Shi 3 participants 5 messages Jun 5, 2024 – Jun 26, 2024 arm64

v4 of 5 Superseded Latest is v5 →

lore
Share & tools
lore

Yang Shi

author
#0 lore raw

PATCH v4 1/1 arm64: mm: force write fault for atomic RMW instructions

The atomic RMW instructions, for example, ldadd, actually does load + add + store in one instruction, it will trigger two page faults per the ARM64 architecture spec, the first fault is a read fault, the second fault is a write fault.

Some applications use atomic RMW instructions to populate memory, for example, openjdk uses atomic-add-0 to do pretouch (populate heap memory at launch time) between v18 and v22 in order to permit use of memory concurrently with pretouch.

But the double page fault has some problems:

1. Noticeable TLB overhead. The kernel actually installs zero page with readonly PTE for the read fault. The write fault will trigger a write-protection fault (CoW). The CoW will allocate a new page and make the PTE point to the new page, this needs TLB invalidations. The tlb invalidation and the mandatory memory barriers may incur significant overhead, particularly on the machines with many cores.

2. Break up huge pages. If THP is on the read fault will install huge zero pages. The later CoW will break up the huge page and allocate base pages instead of huge page. The applications have to rely on khugepaged (kernel thread) to collapse huge pages asynchronously. This also incurs noticeable performance penalty.

3. 512x page faults with huge page. Due to #2, the applications have to have page faults for every 4K area for the write, this makes the speed up by using huge page actually gone.

So it sounds pointless to have two page faults since we know the memory will be definitely written very soon. Forcing write fault for atomic RMW instruction makes some sense and it can solve the aforementioned problems:

Firstly, it just allocates zero'ed page, no tlb invalidation and memory barriers anymore.
Secondly, it can populate writable huge pages in the first place and don't break them up. Just one page fault is needed for 2M area instrad of 512 faults and also save cpu time by not using khugepaged.

A simple micro benchmark which populates 1G memory shows the number of page faults is reduced by half and the time spent by system is reduced by 60% on a VM running on Ampere Altra platform.

And the benchmark for anonymous read fault on 1G memory, file read fault on 1G file (cold page cache and warm page cache) don't show noticeable regression.

Exclude unallocated instructions and LD64B/LDAPR instructions.

Some other architectures also have code inspection in page fault path, for example, SPARC and x86.

---
 arch/arm64/include/asm/insn.h | 20 +++++++++++++++++++
 arch/arm64/mm/fault.c         | 36 +++++++++++++++++++++++++++++++++++
 2 files changed, 56 insertions(+)

v4: 1. Fixed the comments from Catalin.
2. Rebased to v6.10-rc2.
3. Collected the review tag from Christopher Lameter.
v3: Exclude unallocated insns and LD64B/LDAPR per Catalin. And thanks
for D Scott help figure out the minimum conditions.
v2: 1. Made commit log more precise per Anshuman and Catalin
2. Made pagefault_disable/enable window narrower per Anshuman
3. Covered CAS and CASP variants per Catalin
4. Put instruction fetching and decoding into a helper function and take into account endianess per Catalin
5. Don't fetch and decode insn for 32 bit mode (compat) per Catalin
6. More performance tests and exec-only test per Anshuman and Catalin

arch/arm64/include/asm/insn.h +20 -0
index 8c0a36f72d6f..efcc8b2050db 100644
@@ -325,6 +325,7 @@ static __always_inline u32 aarch64_insn_get_##abbr##_value(void)	\
  * "-" means "don't care"
  */
 __AARCH64_INSN_FUNCS(class_branch_sys,	0x1c000000, 0x14000000)
+__AARCH64_INSN_FUNCS(class_atomic,	0x3b200c00, 0x38200000)
 
 __AARCH64_INSN_FUNCS(adr,	0x9F000000, 0x10000000)
 __AARCH64_INSN_FUNCS(adrp,	0x9F000000, 0x90000000)
@@ -345,6 +346,7 @@ __AARCH64_INSN_FUNCS(ldeor,	0x3F20FC00, 0x38202000)
 __AARCH64_INSN_FUNCS(ldset,	0x3F20FC00, 0x38203000)
 __AARCH64_INSN_FUNCS(swp,	0x3F20FC00, 0x38208000)
 __AARCH64_INSN_FUNCS(cas,	0x3FA07C00, 0x08A07C00)
+__AARCH64_INSN_FUNCS(casp,	0xBFA07C00, 0x08207C00)
 __AARCH64_INSN_FUNCS(ldr_reg,	0x3FE0EC00, 0x38606800)
 __AARCH64_INSN_FUNCS(signed_ldr_reg, 0X3FE0FC00, 0x38A0E800)
 __AARCH64_INSN_FUNCS(ldr_imm,	0x3FC00000, 0x39400000)
@@ -549,6 +551,24 @@ static __always_inline bool aarch64_insn_uses_literal(u32 insn)
 	       aarch64_insn_is_prfm_lit(insn);
 }
 
+static __always_inline bool aarch64_insn_is_class_cas(u32 insn)
+{
+	return aarch64_insn_is_cas(insn) ||
+	       aarch64_insn_is_casp(insn);
+}
+
+/*
+ * Exclude unallocated atomic instructions and LD64B/LDAPR.
+ * The masks and values were generated by using Python sympy module.
+ */
+static __always_inline bool aarch64_atomic_insn_has_wr_perm(u32 insn)
+{
+	return ((insn & 0x3f207c00) == 0x38200000) ||
+	       ((insn & 0x3f208c00) == 0x38200000) ||
+	       ((insn & 0x7fe06c00) == 0x78202000) ||
+	       ((insn & 0xbf204c00) == 0x38200000);
+}
+
 enum aarch64_insn_encoding_class aarch64_get_insn_class(u32 insn);
 u64 aarch64_insn_decode_immediate(enum aarch64_insn_imm_type type, u32 insn);
 u32 aarch64_insn_encode_immediate(enum aarch64_insn_imm_type type,
arch/arm64/mm/fault.c +36 -0
index 451ba7cbd5ad..3dbeb6069fbe 100644
@@ -500,6 +500,34 @@ static bool is_write_abort(unsigned long esr)
 	return (esr & ESR_ELx_WNR) && !(esr & ESR_ELx_CM);
 }
 
+static bool is_el0_atomic_instr(struct pt_regs *regs)
+{
+	u32 insn;
+	__le32 insn_le;
+	unsigned long pc = instruction_pointer(regs);
+
+	if (compat_user_mode(regs))
+		return false;
+
+	pagefault_disable();
+	if (get_user(insn_le, (__le32 __user *)pc)) {
+		pagefault_enable();
+		return false;
+	}
+	pagefault_enable();
+
+	insn = le32_to_cpu(insn_le);
+
+	if (aarch64_insn_is_class_atomic(insn) &&
+	    aarch64_atomic_insn_has_wr_perm(insn))
+		return true;
+
+	if (aarch64_insn_is_class_cas(insn))
+		return true;
+
+	return false;
+}
+
 static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 				   struct pt_regs *regs)
 {
@@ -511,6 +539,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 	unsigned long addr = untagged_addr(far);
 	struct vm_area_struct *vma;
 	int si_code;
+	bool may_force_write = false;
 
 	if (kprobe_page_fault(regs, esr))
 		return 0;
@@ -547,6 +576,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 		/* If EPAN is absent then exec implies read */
 		if (!alternative_has_cap_unlikely(ARM64_HAS_EPAN))
 			vm_flags |= VM_EXEC;
+		may_force_write = true;
 	}
 
 	if (is_ttbr0_addr(addr) && is_el1_permission_fault(addr, esr, regs)) {
@@ -568,6 +598,12 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 	if (!vma)
 		goto lock_mmap;
 
+	if (may_force_write && (vma->vm_flags & VM_WRITE) &&
+	    is_el0_atomic_instr(regs)) {
+		vm_flags = VM_WRITE;
+		mm_flags |= FAULT_FLAG_WRITE;
+	}
+
 	if (!(vma->vm_flags & vm_flags)) {
 		vma_end_read(vma);
 		fault = 0;
Signature · 1 line
2.41.0
Acked-by on v4 1/1
#1 lore raw
Quoted 25 lines · Yang Shi

The atomic RMW instructions, for example, ldadd, actually does load + add + store in one instruction, it will trigger two page faults per the ARM64 architecture spec, the first fault is a read fault, the second fault is a write fault.

Some applications use atomic RMW instructions to populate memory, for example, openjdk uses atomic-add-0 to do pretouch (populate heap memory at launch time) between v18 and v22 in order to permit use of memory concurrently with pretouch.

But the double page fault has some problems:

1. Noticeable TLB overhead. The kernel actually installs zero page with readonly PTE for the read fault. The write fault will trigger a write-protection fault (CoW). The CoW will allocate a new page and make the PTE point to the new page, this needs TLB invalidations. The tlb invalidation and the mandatory memory barriers may incur significant overhead, particularly on the machines with many cores.

2. Break up huge pages. If THP is on the read fault will install huge zero pages. The later CoW will break up the huge page and allocate base pages instead of huge page. The applications have to rely on khugepaged (kernel thread) to collapse huge pages asynchronously. This also incurs noticeable performance penalty.

Quoted 3 lines

3. 512x page faults with huge page. Due to #2, the applications have to have page faults for every 4K area for the write, this makes the speed up by using huge page actually gone.

All interesting and valid points.

As raised, the app likely really should be using MADV_POPULATE_WRITE.

Signature · 3 lines
Cheers,

David / dhildenb
maintainer
#2 lore raw
In reply to Yang Shi
Quoted 7 lines · Yang Shi
arch/arm64/include/asm/insn.h +7 -0
+static __always_inline bool aarch64_insn_is_class_cas(u32 insn)
+{
+	return aarch64_insn_is_cas(insn) ||
+	       aarch64_insn_is_casp(insn);
+}
+
+/*

arch/arm64/include/asm/insn.h static __always_inline bool aarch64_insn_uses_literal(u32 insn) · line 570

Quoted 10 lines
arch/arm64/include/asm/insn.h +10 -0
+ * Exclude unallocated atomic instructions and LD64B/LDAPR.
+ * The masks and values were generated by using Python sympy module.
+ */
+static __always_inline bool aarch64_atomic_insn_has_wr_perm(u32 insn)
+{
+	return ((insn & 0x3f207c00) == 0x38200000) ||
+	       ((insn & 0x3f208c00) == 0x38200000) ||
+	       ((insn & 0x7fe06c00) == 0x78202000) ||
+	       ((insn & 0xbf204c00) == 0x38200000);
+}

This is still pretty opaque if we want to modify it in the future. I guess we could add more tests on top but it would be nice to have a way to re-generate these masks. I'll think about, for now these tests would do.

Quoted 17 lines
arch/arm64/mm/fault.c +2 -0
@@ -511,6 +539,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 	unsigned long addr = untagged_addr(far);
 	struct vm_area_struct *vma;
 	int si_code;
+	bool may_force_write = false;
 
 	if (kprobe_page_fault(regs, esr))
 		return 0;
@@ -547,6 +576,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 		/* If EPAN is absent then exec implies read */
 		if (!alternative_has_cap_unlikely(ARM64_HAS_EPAN))
 			vm_flags |= VM_EXEC;
+		may_force_write = true;
 	}
 
 	if (is_ttbr0_addr(addr) && is_el1_permission_fault(addr, esr, regs)) {

arch/arm64/mm/fault.c static int __kprobes do_page_fault(unsigned long far, unsigned long esr, · line 605

Quoted 8 lines
arch/arm64/mm/fault.c +5 -0
@@ -568,6 +598,12 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
 	if (!vma)
 		goto lock_mmap;
 
+	if (may_force_write && (vma->vm_flags & VM_WRITE) &&
+	    is_el0_atomic_instr(regs)) {
+		vm_flags = VM_WRITE;
+		mm_flags |= FAULT_FLAG_WRITE;
+	}

I think we can get rid of may_force_write and just test (vm_flags & VM_READ).

Signature · 1 line
Catalin
author
#3 lore raw
Quoted 14 lines · Catalin Marinas
Quoted 13 lines · Yang Shi
arch/arm64/include/asm/insn.h +13 -0
+static __always_inline bool aarch64_insn_is_class_cas(u32 insn)
+{
+	return aarch64_insn_is_cas(insn) ||
+	       aarch64_insn_is_casp(insn);
+}
+
+/*
+ * Exclude unallocated atomic instructions and LD64B/LDAPR.
+ * The masks and values were generated by using Python sympy module.
+ */
+static __always_inline bool aarch64_atomic_insn_has_wr_perm(u32 insn)
+{
+	return ((insn & 0x3f207c00) == 0x38200000) ||
Quoted 8 lines

arch/arm64/include/asm/insn.h static __always_inline bool aarch64_insn_uses_literal(u32 insn) · line 570

Quoted 4 lines
arch/arm64/include/asm/insn.h +4 -0
+	       ((insn & 0x3f208c00) == 0x38200000) ||
+	       ((insn & 0x7fe06c00) == 0x78202000) ||
+	       ((insn & 0xbf204c00) == 0x38200000);
+}

This is still pretty opaque if we want to modify it in the future. I guess we could add more tests on top but it would be nice to have a way to re-generate these masks. I'll think about, for now these tests would do.

Sorry for the late reply, just came back from vacation and tried to catch up all the emails and TODOs. We should be able to share the tool used by us to generate the tests. But it may take some time.

Quoted 21 lines
Quoted 21 lines
arch/arm64/mm/fault.c +2 -0
@@ -511,6 +539,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
  	unsigned long addr = untagged_addr(far);
  	struct vm_area_struct *vma;
  	int si_code;
+	bool may_force_write = false;
 
  	if (kprobe_page_fault(regs, esr))
  		return 0;
@@ -547,6 +576,7 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
  		/* If EPAN is absent then exec implies read */
  		if (!alternative_has_cap_unlikely(ARM64_HAS_EPAN))
  			vm_flags |= VM_EXEC;
+		may_force_write = true;
  	}
 
  	if (is_ttbr0_addr(addr) && is_el1_permission_fault(addr, esr, regs)) {
@@ -568,6 +598,12 @@ static int __kprobes do_page_fault(unsigned long far, unsigned long esr,
  	if (!vma)
  		goto lock_mmap;
 
Quoted 7 lines

arch/arm64/mm/fault.c static int __kprobes do_page_fault(unsigned long far, unsigned long esr, · line 605

Quoted 5 lines
arch/arm64/mm/fault.c +5 -0
+	if (may_force_write && (vma->vm_flags & VM_WRITE) &&
+	    is_el0_atomic_instr(regs)) {
+		vm_flags = VM_WRITE;
+		mm_flags |= FAULT_FLAG_WRITE;
+	}

I think we can get rid of may_force_write and just test (vm_flags & VM_READ).

Yes, will fix it in v5.

Quoted 1 line
author
#4 lore raw
Quoted 26 lines · Yang Shi
Quoted 14 lines · Catalin Marinas
Quoted 13 lines · Yang Shi
arch/arm64/include/asm/insn.h +13 -0
+static __always_inline bool aarch64_insn_is_class_cas(u32 insn)
+{
+    return aarch64_insn_is_cas(insn) ||
+           aarch64_insn_is_casp(insn);
+}
+
+/*
+ * Exclude unallocated atomic instructions and LD64B/LDAPR.
+ * The masks and values were generated by using Python sympy module.
+ */
+static __always_inline bool aarch64_atomic_insn_has_wr_perm(u32 insn)
+{
+    return ((insn & 0x3f207c00) == 0x38200000) ||
Quoted 8 lines
Quoted 4 lines
+           ((insn & 0x3f208c00) == 0x38200000) ||
+           ((insn & 0x7fe06c00) == 0x78202000) ||

+           ((insn & 0xbf204c00) == 0x38200000);
+}

This is still pretty opaque if we want to modify it in the future. I guess we could add more tests on top but it would be nice to have a way to re-generate these masks. I'll think about, for now these tests would do.

Quoted 1 line

Sorry for the late reply, just came back from vacation and tried to catch up all the emails and TODOs. We should be able to share the tool used by us to generate the tests. But it may take some time.

D Scott made the tool available publicly. Please refer to https://gitlab.com/scott-ph/arm64-insn-group-minimizer

We can re-generate the tests with this tool in the future.

Quoted 33 lines
Quoted 21 lines
Quoted 21 lines
arch/arm64/mm/fault.c +2 -0
@@ -511,6 +539,7 @@ static int __kprobes do_page_fault(unsigned long 
 far, unsigned long esr,
       unsigned long addr = untagged_addr(far);
       struct vm_area_struct *vma;
       int si_code;
+    bool may_force_write = false;
         if (kprobe_page_fault(regs, esr))
           return 0;
@@ -547,6 +576,7 @@ static int __kprobes do_page_fault(unsigned long 
 far, unsigned long esr,
           /* If EPAN is absent then exec implies read */
           if (!alternative_has_cap_unlikely(ARM64_HAS_EPAN))
               vm_flags |= VM_EXEC;
+        may_force_write = true;
       }
         if (is_ttbr0_addr(addr) && is_el1_permission_fault(addr, esr, regs)) {
@@ -568,6 +598,12 @@ static int __kprobes do_page_fault(unsigned 
 long far, unsigned long esr,
       if (!vma)
Quoted 8 lines
Quoted 6 lines
arch/arm64/mm/fault.c +0 -0
           goto lock_mmap;

  +    if (may_force_write && (vma->vm_flags & VM_WRITE) && +        is_el0_atomic_instr(regs)) {
+        vm_flags = VM_WRITE;
+        mm_flags |= FAULT_FLAG_WRITE;
+    }

I think we can get rid of may_force_write and just test (vm_flags & VM_READ).

Yes, will fix it in v5.

Quoted 1 line