Skip to contentSkip to conversation

Fix TLB invalidation on arm64

PATCH In review Will Deacon 4 participants 13 messages Aug 27, 2019 – Aug 30, 2019

Subsystems (4)
lore
Share & tools
lore

Will Deacon

author maintainer
#0 lore raw

cover Fix TLB invalidation on arm64

Hi all,

[+linux-arch since the end of this may be applicable to other architectures]

Commit 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud") removed ISB instructions immediately following updates to the page table, on the grounds that they are not required by the architecture and a DSB alone is sufficient to ensure that subsequent data accesses use the new translation:

DDI0487E_a, B2-128:

  | ... no instruction that appears in program order after the DSB instruction
  | can alter any state of the system or perform any part of its functionality
  | until the DSB completes other than:
  |
  | * Being fetched from memory and decoded
  | * Reading the general-purpose, SIMD and floating-point, Special-purpose, or
  |   System registers that are directly or indirectly read without causing
  |   side-effects.

However, the same document also states the following:

DDI0487E_a, B2-125:

  | DMB and DSB instructions affect reads and writes to the memory system
  | generated by Load/Store instructions and data or unified cache maintenance
  | instructions being executed by the PE. Instruction fetches or accesses
  | caused by a hardware translation table access are not explicit accesses.

which appears to claim that the DSB alone is insufficient. Unfortunately, some CPU designers have followed the second clause above, whereas in Linux we've been relying on the first. This means that our mapping sequence:

MOV X0, <valid pte>
STR X0, [Xptep] // Store new PTE to page table
DSB ISHST
LDR X1, [X2] // Translates using the new PTE

can actually raise a translation fault on the load instruction because the translation can be performed speculatively before the page table update and then marked as "faulting" by the CPU. For user PTEs, this is ok because we can handle the spurious fault, but for kernel PTEs and intermediate table entries this results in a panic().

We can fix this by reverting 24fe1b0efad4fcdd, but the fun doesn't stop there. If we consider the unmap case, then a similar constraint applies to ordering subsequent memory accesses after the completion of the TLB invalidation, so we also need to add an ISB instruction to __flush_tlb_kernel_pgtable(). For user addresses, the exception return provides the necessary context synchronisation.

This then raises an interesting question: if an ISB is required after a TLBI instruction to prevent speculative translation of subsequent instructions, how is this speculation prevented on concurrent CPUs that receive the broadcast TLB invalidation message? Sending and completing a broadcast TLB invalidation message does not imply execution of an ISB on the remote CPU, however it /does/ require that the remote CPU will no longer make use of any old translations because otherwise we wouldn't be able to guarantee that an unmapped page could no longer be modified. In this regard, receiving a TLB invalidation is in some ways stronger than sending one (where you need the ISB).

So far, so good, but the final piece of the puzzle isn't quite so rosy.

*** Other architecture maintainers -- start here! ***

In the case that one CPU maps a page and then sets a flag to tell another CPU:

	CPU 0
	-----

	MOV	X0, <valid pte>
	STR	X0, [Xptep]	// Store new PTE to page table
	DSB	ISHST
	ISB
	MOV	X1, #1
	STR	X1, [Xflag]	// Set the flag

	CPU 1
	-----

loop: LDAR X0, [Xflag] // Poll flag with Acquire semantics
CBZ X0, loop
LDR X1, [X2] // Translates using the new PTE

then the final load on CPU 1 can raise a translation fault for the same reasons as mentioned at the start of this description. In reality, code such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	*ptr = vmalloc(size);		if (*ptr)
	spin_unlock(&lock);			foo = **ptr;
					spin_unlock(&lock);

will not trigger the fault because there is an address dependency on CPU1 which prevents the speculative translation. However, more exotic code where the virtual address is known ahead of time, such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	set_fixmap(0, paddr, prot);	if (mapped)
	mapped = true;				foo = *fix_to_virt(0);
	spin_unlock(&lock);		spin_unlock(&lock);

could fault. This can be avoided by any of:

* Introducing broadcast TLB maintenance on the map path
* Adding a DSB;ISB sequence after checking a flag which indicates that a virtual address is now mapped
* Handling the spurious fault

Given that we have never observed a problem in the concurrent case under Linux and future revisions of the architecture are being tightened so that translation table walks are effectively ordered in the same way as explicit memory accesses, we no longer treat spurious kernel faults as fatal if the page table indicates that the access was valid.

Anyway, this patch series attempts to implement some of this and I plan to queue it for 5.4.

Will

Cc Catalin Marinas <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Peter Zijlstra <[email protected]>

--->8

Will Deacon (6):
Revert "arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}"
arm64: tlb: Ensure we execute an ISB following walk cache invalidation
arm64: mm: Add ISB instruction to set_pgd()
arm64: sysreg: Add some field definitions for PAR_EL1
arm64: mm: Ignore spurious translation faults taken from the kernel
arm64: kvm: Replace hardcoded '1' with SYS_PAR_EL1_F

 arch/arm64/include/asm/pgtable.h  | 13 ++++++++++---
 arch/arm64/include/asm/sysreg.h   |  3 +++
 arch/arm64/include/asm/tlbflush.h |  1 +
 arch/arm64/kvm/hyp/switch.c       |  2 +-
 arch/arm64/mm/fault.c             | 33 +++++++++++++++++++++++++++++++++
 5 files changed, 48 insertions(+), 4 deletions(-)
Signature · 1 line
2.11.0
author maintainer
#1 lore raw

PATCH 1/6 Revert "arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}"

This reverts commit 24fe1b0efad4fcdd32ce46cffeab297f22581707.

Commit 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}") removed ISB instructions immediately following updates to the page table, on the grounds that they are not required by the architecture and a DSB alone is sufficient to ensure that subsequent data accesses use the new translation:

DDI0487E_a, B2-128:

  | ... no instruction that appears in program order after the DSB
  | instruction can alter any state of the system or perform any part of
  | its functionality until the DSB completes other than:
  |
  | * Being fetched from memory and decoded
  | * Reading the general-purpose, SIMD and floating-point,
  |   Special-purpose, or System registers that are directly or indirectly
  |   read without causing side-effects.

However, the same document also states the following:

DDI0487E_a, B2-125:

  | DMB and DSB instructions affect reads and writes to the memory system
  | generated by Load/Store instructions and data or unified cache
  | maintenance instructions being executed by the PE. Instruction fetches
  | or accesses caused by a hardware translation table access are not
  | explicit accesses.

which appears to claim that the DSB alone is insufficient. Unfortunately, some CPU designers have followed the second clause above, whereas in Linux we've been relying on the first. This means that our mapping sequence:

MOV X0, <valid pte>
STR X0, [Xptep] // Store new PTE to page table
DSB ISHST
LDR X1, [X2] // Translates using the new PTE

can actually raise a translation fault on the load instruction because the translation can be performed speculatively before the page table update and then marked as "faulting" by the CPU. For user PTEs, this is ok because we can handle the spurious fault, but for kernel PTEs and intermediate table entries this results in a panic().

Revert the offending commit to reintroduce the missing barriers.

Fixes 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}")
---
 arch/arm64/include/asm/pgtable.h | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
arch/arm64/include/asm/pgtable.h +9 -3
index 5fdcfe237338..feda7294320c 100644
@@ -220,8 +220,10 @@ static inline void set_pte(pte_t *ptep, pte_t pte)
 	 * Only if the new pte is valid and kernel, otherwise TLB maintenance
 	 * or update_mmu_cache() have the necessary barriers.
 	 */
-	if (pte_valid_not_user(pte))
+	if (pte_valid_not_user(pte)) {
 		dsb(ishst);
+		isb();
+	}
 }
 
 extern void __sync_icache_dcache(pte_t pteval);
@@ -481,8 +483,10 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)
 
 	WRITE_ONCE(*pmdp, pmd);
 
-	if (pmd_valid(pmd))
+	if (pmd_valid(pmd)) {
 		dsb(ishst);
+		isb();
+	}
 }
 
 static inline void pmd_clear(pmd_t *pmdp)
@@ -540,8 +544,10 @@ static inline void set_pud(pud_t *pudp, pud_t pud)
 
 	WRITE_ONCE(*pudp, pud);
 
-	if (pud_valid(pud))
+	if (pud_valid(pud)) {
 		dsb(ishst);
+		isb();
+	}
 }
 
 static inline void pud_clear(pud_t *pudp)
Signature · 1 line
2.11.0
author maintainer
#3 lore raw

PATCH 2/6 arm64: tlb: Ensure we execute an ISB following walk cache invalidation

In reply to Will Deacon

05f2d2f83b5a ("arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable") added a new TLB invalidation helper which is used when freeing intermediate levels of page table used for kernel mappings, but is missing the required ISB instruction after completion of the TLBI instruction.

Add the missing barrier.

Fixes 05f2d2f83b5a ("arm64: tlbflush: Introduce __flush_tlb_kernel_pgtable")
---
 arch/arm64/include/asm/tlbflush.h | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/include/asm/tlbflush.h +1 -0
index 8af7a85f76bd..bc3949064725 100644
@@ -251,6 +251,7 @@ static inline void __flush_tlb_kernel_pgtable(unsigned long kaddr)
 	dsb(ishst);
 	__tlbi(vaae1is, addr);
 	dsb(ish);
+	isb();
 }
 #endif
 
Signature · 1 line
2.11.0
author maintainer
#4 lore raw

PATCH 3/6 arm64: mm: Add ISB instruction to set_pgd()

In reply to Will Deacon

Commit 6a4cbd63c25a ("Revert "arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}"") reintroduced ISB instructions to some of our page table setter functions in light of a recent clarification to the Armv8 architecture. Although 'set_pgd()' isn't currently used to update a live page table, add the ISB instruction there too for consistency with the other macros and to provide some future-proofing if we use it on live tables in the future.

---
 arch/arm64/include/asm/pgtable.h | 1 +
 1 file changed, 1 insertion(+)
arch/arm64/include/asm/pgtable.h +1 -0
index feda7294320c..2faa77635942 100644
@@ -605,6 +605,7 @@ static inline void set_pgd(pgd_t *pgdp, pgd_t pgd)
 
 	WRITE_ONCE(*pgdp, pgd);
 	dsb(ishst);
+	isb();
 }
 
 static inline void pgd_clear(pgd_t *pgdp)
Signature · 1 line
2.11.0
author maintainer
#5 lore raw

PATCH 4/6 arm64: sysreg: Add some field definitions for PAR_EL1

In reply to Will Deacon

PAR_EL1 is a mysterious creature, but sometimes it's necessary to read it when translating addresses in situations where we cannot walk the page table directly.

Add a couple of system register definitions for the fault indication field ('F') and the fault status code ('FST').

---
 arch/arm64/include/asm/sysreg.h | 3 +++
 1 file changed, 3 insertions(+)
arch/arm64/include/asm/sysreg.h +3 -0
index 06ebcfef73df..2b229c23f3c1 100644
@@ -212,6 +212,9 @@
 #define SYS_FAR_EL1			sys_reg(3, 0, 6, 0, 0)
 #define SYS_PAR_EL1			sys_reg(3, 0, 7, 4, 0)
 
+#define SYS_PAR_EL1_F			BIT(1)
+#define SYS_PAR_EL1_FST			GENMASK(6, 1)
+
 /*** Statistical Profiling Extension ***/
 /* ID registers */
 #define SYS_PMSIDR_EL1			sys_reg(3, 0, 9, 9, 7)
Signature · 1 line
2.11.0
author maintainer
#6 lore raw

PATCH 5/6 arm64: mm: Ignore spurious translation faults taken from the kernel

In reply to Will Deacon

Thanks to address translation being performed out of order with respect to loads and stores, it is possible for a CPU to take a translation fault when accessing a page that was mapped by a different CPU.

For example, in the case that one CPU maps a page and then sets a flag to tell another CPU:

	CPU 0
	-----

	MOV	X0, <valid pte>
	STR	X0, [Xptep]	// Store new PTE to page table
	DSB	ISHST
	ISB
	MOV	X1, #1
	STR	X1, [Xflag]	// Set the flag

	CPU 1
	-----

loop: LDAR X0, [Xflag] // Poll flag with Acquire semantics
CBZ X0, loop
LDR X1, [X2] // Translates using the new PTE

then the final load on CPU 1 can raise a translation fault because the translation can be performed speculatively before the read of the flag and marked as "faulting" by the CPU. This isn't quite as bad as it sounds since, in reality, code such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	*ptr = vmalloc(size);		if (*ptr)
	spin_unlock(&lock);			foo = **ptr;
					spin_unlock(&lock);

will not trigger the fault because there is an address dependency on CPU 1 which prevents the speculative translation. However, more exotic code where the virtual address is known ahead of time, such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	set_fixmap(0, paddr, prot);	if (mapped)
	mapped = true;				foo = *fix_to_virt(0);
	spin_unlock(&lock);		spin_unlock(&lock);

could fault. This can be avoided by any of:

* Introducing broadcast TLB maintenance on the map path
* Adding a DSB;ISB sequence after checking a flag which indicates that a virtual address is now mapped
* Handling the spurious fault

Given that we have never observed a problem due to this under Linux and future revisions of the architecture are being tightened so that translation table walks are effectively ordered in the same way as explicit memory accesses, we no longer treat spurious kernel faults as fatal if an AT instruction indicates that the access does not trigger a translation fault.

---
 arch/arm64/mm/fault.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
arch/arm64/mm/fault.c +33 -0
index cfd65b63f36f..9808da29a653 100644
@@ -8,6 +8,7 @@
  */
 
 #include <linux/acpi.h>
+#include <linux/bitfield.h>
 #include <linux/extable.h>
 #include <linux/signal.h>
 #include <linux/mm.h>
@@ -242,6 +243,34 @@ static inline bool is_el1_permission_fault(unsigned long addr, unsigned int esr,
 	return false;
 }
 
+static bool __kprobes is_spurious_el1_translation_fault(unsigned long addr,
+							unsigned int esr,
+							struct pt_regs *regs)
+{
+	unsigned long flags;
+	u64 par, dfsc;
+
+	if (ESR_ELx_EC(esr) != ESR_ELx_EC_DABT_CUR ||
+	    (esr & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT)
+		return false;
+
+	local_irq_save(flags);
+	asm volatile("at s1e1r, %0" :: "r" (addr));
+	isb();
+	par = read_sysreg(par_el1);
+	local_irq_restore(flags);
+
+	if (!(par & SYS_PAR_EL1_F))
+		return false;
+
+	/*
+	 * If we got a different type of fault from the AT instruction,
+	 * treat the translation fault as spurious.
+	 */
+	dfsc = FIELD_PREP(SYS_PAR_EL1_FST, par);
+	return (dfsc & ESR_ELx_FSC_TYPE) != ESR_ELx_FSC_FAULT;
+}
+
 static void die_kernel_fault(const char *msg, unsigned long addr,
 			     unsigned int esr, struct pt_regs *regs)
 {
@@ -270,6 +299,10 @@ static void __do_kernel_fault(unsigned long addr, unsigned int esr,
 	if (!is_el1_instruction_abort(esr) && fixup_exception(regs))
 		return;
 
+	if (WARN_RATELIMIT(is_spurious_el1_translation_fault(addr, esr, regs),
+	    "Ignoring spurious kernel translation fault at virtual address %016lx\n", addr))
+		return;
+
 	if (is_el1_permission_fault(addr, esr, regs)) {
 		if (esr & ESR_ELx_WNR)
 			msg = "write to read-only memory";
Signature · 1 line
2.11.0
author maintainer
#7 lore raw

PATCH 6/6 arm64: kvm: Replace hardcoded '1' with SYS_PAR_EL1_F

In reply to Will Deacon

Now that we have a definition for the 'F' field of PAR_EL1, use that instead of coding the immediate directly.

---
 arch/arm64/kvm/hyp/switch.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kvm/hyp/switch.c +1 -1
index adaf266d8de8..bd978ad71936 100644
@@ -264,7 +264,7 @@ static bool __hyp_text __translate_far_to_hpfar(u64 far, u64 *hpfar)
 	tmp = read_sysreg(par_el1);
 	write_sysreg(par, par_el1);
 
-	if (unlikely(tmp & 1))
+	if (unlikely(tmp & SYS_PAR_EL1_F))
 		return false; /* Translation failed, back to guest */
 
 	/* Convert PAR to HPFAR format */
Signature · 1 line
2.11.0
reviewer Reviewed-by on the v1 series
#8 lore raw
In reply to Will Deacon
Quoted 1 line · Will Deacon

Hi all,

Hi Will,

For the series:

Thanks,
Mark.

Quoted 151 lines

[+linux-arch since the end of this may be applicable to other architectures]

Commit 24fe1b0efad4fcdd ("arm64: Remove unnecessary ISBs from set_{pte,pmd,pud") removed ISB instructions immediately following updates to the page table, on the grounds that they are not required by the architecture and a DSB alone is sufficient to ensure that subsequent data accesses use the new translation:

DDI0487E_a, B2-128:

  | ... no instruction that appears in program order after the DSB instruction
  | can alter any state of the system or perform any part of its functionality
  | until the DSB completes other than:
  |
  | * Being fetched from memory and decoded
  | * Reading the general-purpose, SIMD and floating-point, Special-purpose, or
  |   System registers that are directly or indirectly read without causing
  |   side-effects.

However, the same document also states the following:

DDI0487E_a, B2-125:

  | DMB and DSB instructions affect reads and writes to the memory system
  | generated by Load/Store instructions and data or unified cache maintenance
  | instructions being executed by the PE. Instruction fetches or accesses
  | caused by a hardware translation table access are not explicit accesses.

which appears to claim that the DSB alone is insufficient. Unfortunately, some CPU designers have followed the second clause above, whereas in Linux we've been relying on the first. This means that our mapping sequence:

MOV X0, <valid pte>
STR X0, [Xptep] // Store new PTE to page table
DSB ISHST
LDR X1, [X2] // Translates using the new PTE

can actually raise a translation fault on the load instruction because the translation can be performed speculatively before the page table update and then marked as "faulting" by the CPU. For user PTEs, this is ok because we can handle the spurious fault, but for kernel PTEs and intermediate table entries this results in a panic().

We can fix this by reverting 24fe1b0efad4fcdd, but the fun doesn't stop there. If we consider the unmap case, then a similar constraint applies to ordering subsequent memory accesses after the completion of the TLB invalidation, so we also need to add an ISB instruction to __flush_tlb_kernel_pgtable(). For user addresses, the exception return provides the necessary context synchronisation.

This then raises an interesting question: if an ISB is required after a TLBI instruction to prevent speculative translation of subsequent instructions, how is this speculation prevented on concurrent CPUs that receive the broadcast TLB invalidation message? Sending and completing a broadcast TLB invalidation message does not imply execution of an ISB on the remote CPU, however it /does/ require that the remote CPU will no longer make use of any old translations because otherwise we wouldn't be able to guarantee that an unmapped page could no longer be modified. In this regard, receiving a TLB invalidation is in some ways stronger than sending one (where you need the ISB).

So far, so good, but the final piece of the puzzle isn't quite so rosy.

*** Other architecture maintainers -- start here! ***

In the case that one CPU maps a page and then sets a flag to tell another CPU:

	CPU 0
	-----

	MOV	X0, <valid pte>
	STR	X0, [Xptep]	// Store new PTE to page table
	DSB	ISHST
	ISB
	MOV	X1, #1
	STR	X1, [Xflag]	// Set the flag

	CPU 1
	-----

loop: LDAR X0, [Xflag] // Poll flag with Acquire semantics
CBZ X0, loop
LDR X1, [X2] // Translates using the new PTE

then the final load on CPU 1 can raise a translation fault for the same reasons as mentioned at the start of this description. In reality, code such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	*ptr = vmalloc(size);		if (*ptr)
	spin_unlock(&lock);			foo = **ptr;
					spin_unlock(&lock);

will not trigger the fault because there is an address dependency on CPU1 which prevents the speculative translation. However, more exotic code where the virtual address is known ahead of time, such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	set_fixmap(0, paddr, prot);	if (mapped)
	mapped = true;				foo = *fix_to_virt(0);
	spin_unlock(&lock);		spin_unlock(&lock);

could fault. This can be avoided by any of:

* Introducing broadcast TLB maintenance on the map path
* Adding a DSB;ISB sequence after checking a flag which indicates that a virtual address is now mapped
* Handling the spurious fault

Given that we have never observed a problem in the concurrent case under Linux and future revisions of the architecture are being tightened so that translation table walks are effectively ordered in the same way as explicit memory accesses, we no longer treat spurious kernel faults as fatal if the page table indicates that the access was valid.

Anyway, this patch series attempts to implement some of this and I plan to queue it for 5.4.

Will

Cc Catalin Marinas <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Marc Zyngier <[email protected]>
Cc Peter Zijlstra <[email protected]>

--->8

Will Deacon (6):
Revert "arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}"
arm64: tlb: Ensure we execute an ISB following walk cache invalidation
arm64: mm: Add ISB instruction to set_pgd()
arm64: sysreg: Add some field definitions for PAR_EL1
arm64: mm: Ignore spurious translation faults taken from the kernel
arm64: kvm: Replace hardcoded '1' with SYS_PAR_EL1_F

 arch/arm64/include/asm/pgtable.h  | 13 ++++++++++---
 arch/arm64/include/asm/sysreg.h   |  3 +++
 arch/arm64/include/asm/tlbflush.h |  1 +
 arch/arm64/kvm/hyp/switch.c       |  2 +-
 arch/arm64/mm/fault.c             | 33 +++++++++++++++++++++++++++++++++
 5 files changed, 48 insertions(+), 4 deletions(-)
Signature · 1 line
2.11.0
#9 lore raw
In reply to Will Deacon

Will Deacon's on August 27, 2019 11:18 pm:

Quoted 5 lines

can actually raise a translation fault on the load instruction because the translation can be performed speculatively before the page table update and then marked as "faulting" by the CPU. For user PTEs, this is ok because we can handle the spurious fault, but for kernel PTEs and intermediate table entries this results in a panic().

powerpc sounds like it has the same coherency issue with stores vs loads from the MMU's page table walker, and a barrier called ptesync to order them.

Quoted 7 lines

We can fix this by reverting 24fe1b0efad4fcdd, but the fun doesn't stop there. If we consider the unmap case, then a similar constraint applies to ordering subsequent memory accesses after the completion of the TLB invalidation, so we also need to add an ISB instruction to __flush_tlb_kernel_pgtable(). For user addresses, the exception return provides the necessary context synchronisation.

Quoted 10 lines

This then raises an interesting question: if an ISB is required after a TLBI instruction to prevent speculative translation of subsequent instructions, how is this speculation prevented on concurrent CPUs that receive the broadcast TLB invalidation message? Sending and completing a broadcast TLB invalidation message does not imply execution of an ISB on the remote CPU, however it /does/ require that the remote CPU will no longer make use of any old translations because otherwise we wouldn't be able to guarantee that an unmapped page could no longer be modified. In this regard, receiving a TLB invalidation is in some ways stronger than sending one (where you need the ISB).

Similar with powerpc's tlbie, sender requires extra barriers!

Quoted 20 lines

So far, so good, but the final piece of the puzzle isn't quite so rosy.

*** Other architecture maintainers -- start here! ***

In the case that one CPU maps a page and then sets a flag to tell another CPU:

	CPU 0
	-----

	MOV	X0, <valid pte>
	STR	X0, [Xptep]	// Store new PTE to page table
	DSB	ISHST
	ISB
	MOV	X1, #1
	STR	X1, [Xflag]	// Set the flag

	CPU 1
	-----
Quoted 6 lines

loop: LDAR X0, [Xflag] // Poll flag with Acquire semantics
CBZ X0, loop
LDR X1, [X2] // Translates using the new PTE

then the final load on CPU 1 can raise a translation fault for the same reasons as mentioned at the start of this description.

powerpc's ptesync instruction is defined to order MMU memory accesses on all other CPUs. ptesync does not go out to the fabric though. How does it work then?

Because the MMU coherency problem (at least we have) is not that the load will begin to "partially" execute ahead of the store, enough to kick off a table walk that goes ahead of the store, but not so much that it violates the regular CPU barriers. It's just that the loads from the MMU don't participate in the LSU pipeline, they don't snoop the store queues aren't inserted into load queues so the regular memory barrier instructions won't see stores from other threads cuasing ordering violations.

In your first example, if powerpc just has a normal memory barrier there instead of a ptesync, it could all execute completely non-speculatively and in-order but still cause a fault, because the table walker's loads didn't see the store in the store queue.

From the other side of the fabric you have no such problem. The table walker is cache coherent apart from the local stores, so we don't need a special barrier on the other side. That's why ptesync doesn't broadcast.

I would be surprised if ARM's issue is different, but interested to hear if it is.

Quoted 14 lines

In reality, code
such as:

	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	*ptr = vmalloc(size);		if (*ptr)
	spin_unlock(&lock);			foo = **ptr;
					spin_unlock(&lock);

will not trigger the fault because there is an address dependency on CPU1 which prevents the speculative translation. However, more exotic code where the virtual address is known ahead of time, such as:

Quoted 8 lines
	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	set_fixmap(0, paddr, prot);	if (mapped)
	mapped = true;				foo = *fix_to_virt(0);
	spin_unlock(&lock);		spin_unlock(&lock);

could fault.

This is kind of a different issue, or part of a wider one at least. Consider speculative execution more generally, any branch mispredict can send us off to crazy town executing instructions using nonsense register values. CPU0 does not have to be in the picture, or any kernel page table modification at all, CPU1 alone will be doing speculative loads wildly all over the kernel address space and trying to access pages with no pte.

Yet we don't have to flush TLB when creating a new kernel mapping, and we don't get spurious kernel faults. The page table walker won't install negative entries, at least not "architectural" i.e., that cause faults and require flushing. My guess is ARM is similar, or you would have seen bigger problems by now?

If you have CPU0 doing a ro->rw upgrade on a kernel PTE, then it may be possible another CPU1 would speculatively install a ro TLB and then spurious fault on it when attempting to store to it. But no amount of barriers would help because CPU1 could have picked up that TLB any time in the past.

Thanks,
Nick

maintainer
#2 lore raw

Re: [PATCH 1/6] Revert "arm64: Remove unnecessary ISBs from set_{pte, pmd, pud}"

In reply to Will Deacon

Hi,

[This is an automated email]

This commit has been processed because it contains a "Fixes:" tag, fixing commit: 24fe1b0efad4 arm64: Remove unnecessary ISBs from set_{pte,pmd,pud}.

The bot has tested the following trees: v5.2.10, v4.19.68.

v5.2.10: Build OK!
v4.19.68: Failed to apply! Possible dependencies:

    0795edaf3f1f ("arm64: pgtable: Implement p[mu]d_valid() and check in set_p[mu]d()")

NOTE: The patch will not be queued to stable trees until it is upstream.

How should we proceed with this patch?

--
Thanks,
Sasha

author maintainer
#10 lore raw
In reply to Nicholas Piggin

Hi Nick,

Quoted 41 lines · Nicholas Piggin

Will Deacon's on August 27, 2019 11:18 pm:

Quoted 20 lines

So far, so good, but the final piece of the puzzle isn't quite so rosy.

*** Other architecture maintainers -- start here! ***

In the case that one CPU maps a page and then sets a flag to tell another CPU:

	CPU 0
	-----

	MOV	X0, <valid pte>
	STR	X0, [Xptep]	// Store new PTE to page table
	DSB	ISHST
	ISB
	MOV	X1, #1
	STR	X1, [Xflag]	// Set the flag

	CPU 1
	-----
Quoted 6 lines

loop: LDAR X0, [Xflag] // Poll flag with Acquire semantics
CBZ X0, loop
LDR X1, [X2] // Translates using the new PTE

then the final load on CPU 1 can raise a translation fault for the same reasons as mentioned at the start of this description.

powerpc's ptesync instruction is defined to order MMU memory accesses on all other CPUs. ptesync does not go out to the fabric though. How does it work then?

Because the MMU coherency problem (at least we have) is not that the load will begin to "partially" execute ahead of the store, enough to kick off a table walk that goes ahead of the store, but not so much that it violates the regular CPU barriers. It's just that the loads from the MMU don't participate in the LSU pipeline, they don't snoop the store queues aren't inserted into load queues so the regular memory barrier instructions won't see stores from other threads cuasing ordering violations.

Quoted 4 lines

In your first example, if powerpc just has a normal memory barrier there instead of a ptesync, it could all execute completely non-speculatively and in-order but still cause a fault, because the table walker's loads didn't see the store in the store queue.

Ah, so I think this is where our DSB comes in, as opposed to our usual DMB. DMB provides ordering guarantees and is generally the only barrier instruction you need for shared memory communication. DSB, on the other hand, has additional properties such as making page-table updates visible to the table walker and completing pending TLB invalidation.

So in practice, DSB is likely to drain the store buffer to ensure that pending page-table writes are visible at L1, which is coherent with all CPUs (and their page-table walkers).

Quoted 3 lines

From the other side of the fabric you have no such problem. The table walker is cache coherent apart from the local stores, so we don't need a special barrier on the other side. That's why ptesync doesn't broadcast.

Curious: but do you need to do anything extra to take into account instruction fetch on remote CPUs if you're mapping an executable page? We added an IPI to flush_icache_range() in 3b8c9f1cdfc5 to handle this, because our broadcast I-cache maintenance doesn't force a pipeline flush for remote CPUs (and may even execute as a NOP on recent cores).

Quoted 2 lines

I would be surprised if ARM's issue is different, but interested to hear if it is.

If you take the four scenarios of Map/Unmap for the UP/SMP cases:

Map+UP // Some sort of fence instruction (DSB;ISB/ptesync)
Map+SMP // Same as above
Unmap+UP // Local TLB invalidation
Unmap+SMP // Broadcast TLB invalidation

then the most interesting case is Map+SMP, where one CPU transitions a PTE from invalid to valid and executes its DSB;ISB/PTESYNC sequence without affecting the remote CPU. That's what I'm trying to get at in my example below:

Quoted 17 lines
Quoted 8 lines
	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	set_fixmap(0, paddr, prot);	if (mapped)
	mapped = true;				foo = *fix_to_virt(0);
	spin_unlock(&lock);		spin_unlock(&lock);

could fault.

This is kind of a different issue, or part of a wider one at least. Consider speculative execution more generally, any branch mispredict can send us off to crazy town executing instructions using nonsense register values. CPU0 does not have to be in the picture, or any kernel page table modification at all, CPU1 alone will be doing speculative loads wildly all over the kernel address space and trying to access pages with no pte.

Quoted 5 lines

Yet we don't have to flush TLB when creating a new kernel mapping, and we don't get spurious kernel faults. The page table walker won't install negative entries, at least not "architectural" i.e., that cause faults and require flushing. My guess is ARM is similar, or you would have seen bigger problems by now?

Right, we don't allow negative (invalid) entries to be cached in the TLB, but CPUs can effectively cache the result of a translation for a load/store instruction whilst that instruction flows down the pipe after its virtual address is known. /That/ caching is not restricted to valid translations. For example, if we just take a simple message passing example where we have two global variables: a 'mapped' flag (initialised to zero) and a pointer (initialised to point at a page that is not yet mapped):

[please excuse the mess I've made of your assembly language]

	CPU 0

	// Set PTE which maps the page pointed to by pointer
	stw	r5, 0(r4)
	ptesync
	lwsync

	// Set 'mapped' flag to 1
	li	r9, 1
	stb	r9, 0(r3)


	CPU 1

	// Poll for 'mapped' flag to be set
loop:	lbz	r9, 0(r3)
	lwsync
	cmpdi	cr7, r0, 0
	beq	cr7, loop

	// Dereference pointer
	lwz	r4, 0(r5)

So in this example, I think it's surprising that CPU 1's dereference of 'pointer' can fault. The way this happens on arm64 is that CPU 1 can translate the 'pointer' dereference before the load of the 'mapped' flag has returned its data. The walker may come back with a fault, but then if the flag data later comes back as 1, the fault will be taken when the lwz commits. In other words, translation table walks can occur out-of-order with respect to the accesses they are translating for, even in the presence of memory barriers.

In practice, I think this kind of code sequence is unusual and triggering the problem relies on CPU 1 knowing the virtual address it's going to dereference before it's actually mapped. However, this could conceivably happen with the fixmap and possibly also if the page-table itself was being written concurrently using cmpxchg(), in which case you might use the actual pte value in the same way as the 'mapped' flag above.

But yes, adding the spurious fault handler is belt and braces, which is why I've kept a WARN_RATELIMIT() in there if it ever triggers.

Quoted 5 lines

If you have CPU0 doing a ro->rw upgrade on a kernel PTE, then it may be possible another CPU1 would speculatively install a ro TLB and then spurious fault on it when attempting to store to it. But no amount of barriers would help because CPU1 could have picked up that TLB any time in the past.

Transitioning from ro->rw requires TLB invalidation, and so that falls into the Unmap+SMP combination which I'm not worried about because it doesn't pose a problem for us.

Will

#11 lore raw

Will Deacon's on August 29, 2019 2:12 am:

Quoted 63 lines

Hi Nick,

Quoted 41 lines · Nicholas Piggin

Will Deacon's on August 27, 2019 11:18 pm:

Quoted 20 lines

So far, so good, but the final piece of the puzzle isn't quite so rosy.

*** Other architecture maintainers -- start here! ***

In the case that one CPU maps a page and then sets a flag to tell another CPU:

	CPU 0
	-----

	MOV	X0, <valid pte>
	STR	X0, [Xptep]	// Store new PTE to page table
	DSB	ISHST
	ISB
	MOV	X1, #1
	STR	X1, [Xflag]	// Set the flag

	CPU 1
	-----
Quoted 6 lines

loop: LDAR X0, [Xflag] // Poll flag with Acquire semantics
CBZ X0, loop
LDR X1, [X2] // Translates using the new PTE

then the final load on CPU 1 can raise a translation fault for the same reasons as mentioned at the start of this description.

powerpc's ptesync instruction is defined to order MMU memory accesses on all other CPUs. ptesync does not go out to the fabric though. How does it work then?

Because the MMU coherency problem (at least we have) is not that the load will begin to "partially" execute ahead of the store, enough to kick off a table walk that goes ahead of the store, but not so much that it violates the regular CPU barriers. It's just that the loads from the MMU don't participate in the LSU pipeline, they don't snoop the store queues aren't inserted into load queues so the regular memory barrier instructions won't see stores from other threads cuasing ordering violations.

Quoted 4 lines

In your first example, if powerpc just has a normal memory barrier there instead of a ptesync, it could all execute completely non-speculatively and in-order but still cause a fault, because the table walker's loads didn't see the store in the store queue.

Ah, so I think this is where our DSB comes in, as opposed to our usual DMB. DMB provides ordering guarantees and is generally the only barrier instruction you need for shared memory communication. DSB, on the other hand, has additional properties such as making page-table updates visible to the table walker and completing pending TLB invalidation.

So in practice, DSB is likely to drain the store buffer to ensure that pending page-table writes are visible at L1, which is coherent with all CPUs (and their page-table walkers).

Quoted 3 lines

From the other side of the fabric you have no such problem. The table walker is cache coherent apart from the local stores, so we don't need a special barrier on the other side. That's why ptesync doesn't broadcast.

Quoted 5 lines

Curious: but do you need to do anything extra to take into account instruction fetch on remote CPUs if you're mapping an executable page? We added an IPI to flush_icache_range() in 3b8c9f1cdfc5 to handle this, because our broadcast I-cache maintenance doesn't force a pipeline flush for remote CPUs (and may even execute as a NOP on recent cores).

Ah, I think the tlbie does not force re-fetch indeed. We may need something like that as well.

What do you do on the user side? Require threads to ISB themselves?

Quoted 38 lines
Quoted 2 lines

I would be surprised if ARM's issue is different, but interested to hear if it is.

If you take the four scenarios of Map/Unmap for the UP/SMP cases:

Map+UP // Some sort of fence instruction (DSB;ISB/ptesync)
Map+SMP // Same as above
Unmap+UP // Local TLB invalidation
Unmap+SMP // Broadcast TLB invalidation

then the most interesting case is Map+SMP, where one CPU transitions a PTE from invalid to valid and executes its DSB;ISB/PTESYNC sequence without affecting the remote CPU. That's what I'm trying to get at in my example below:

Quoted 17 lines
Quoted 8 lines
	CPU 0				CPU 1
	-----				-----
	spin_lock(&lock);		spin_lock(&lock);
	set_fixmap(0, paddr, prot);	if (mapped)
	mapped = true;				foo = *fix_to_virt(0);
	spin_unlock(&lock);		spin_unlock(&lock);

could fault.

This is kind of a different issue, or part of a wider one at least. Consider speculative execution more generally, any branch mispredict can send us off to crazy town executing instructions using nonsense register values. CPU0 does not have to be in the picture, or any kernel page table modification at all, CPU1 alone will be doing speculative loads wildly all over the kernel address space and trying to access pages with no pte.

Quoted 5 lines

Yet we don't have to flush TLB when creating a new kernel mapping, and we don't get spurious kernel faults. The page table walker won't install negative entries, at least not "architectural" i.e., that cause faults and require flushing. My guess is ARM is similar, or you would have seen bigger problems by now?

Quoted 4 lines

Right, we don't allow negative (invalid) entries to be cached in the TLB, but CPUs can effectively cache the result of a translation for a load/store instruction whilst that instruction flows down the pipe after its virtual address is known. /That/ caching is not restricted to valid translations.

Ah, I misunderstood you sorry. Yeah that is interesting, I don't think that is explicitly prohibited in the power ISA either. I believe CPU1 would have to do a ptesync to avoid the fault there.

Quoted 46 lines

For example, if we just take a simple message passing example where we have two global variables: a 'mapped' flag (initialised to zero) and a pointer (initialised to point at a page that is not yet mapped):

[please excuse the mess I've made of your assembly language]

	CPU 0

	// Set PTE which maps the page pointed to by pointer
	stw	r5, 0(r4)
	ptesync
	lwsync

	// Set 'mapped' flag to 1
	li	r9, 1
	stb	r9, 0(r3)


	CPU 1

	// Poll for 'mapped' flag to be set
loop:	lbz	r9, 0(r3)
	lwsync
	cmpdi	cr7, r0, 0
	beq	cr7, loop

	// Dereference pointer
	lwz	r4, 0(r5)

So in this example, I think it's surprising that CPU 1's dereference of 'pointer' can fault. The way this happens on arm64 is that CPU 1 can translate the 'pointer' dereference before the load of the 'mapped' flag has returned its data. The walker may come back with a fault, but then if the flag data later comes back as 1, the fault will be taken when the lwz commits. In other words, translation table walks can occur out-of-order with respect to the accesses they are translating for, even in the presence of memory barriers.

In practice, I think this kind of code sequence is unusual and triggering the problem relies on CPU 1 knowing the virtual address it's going to dereference before it's actually mapped. However, this could conceivably happen with the fixmap and possibly also if the page-table itself was being written concurrently using cmpxchg(), in which case you might use the actual pte value in the same way as the 'mapped' flag above.

Quoted 2 lines

But yes, adding the spurious fault handler is belt and braces, which is why I've kept a WARN_RATELIMIT() in there if it ever triggers.

This could be a theoretical problem for powerpc too, I think. Maybe. I'll ask around, I might not be understanding the architecture or Linux code properly.

Thanks,
Nick

author maintainer
#12 lore raw
Quoted 12 lines · Nicholas Piggin

Will Deacon's on August 29, 2019 2:12 am:

Quoted 10 lines · Nicholas Piggin
Quoted 3 lines

From the other side of the fabric you have no such problem. The table walker is cache coherent apart from the local stores, so we don't need a special barrier on the other side. That's why ptesync doesn't broadcast.

Curious: but do you need to do anything extra to take into account instruction fetch on remote CPUs if you're mapping an executable page? We added an IPI to flush_icache_range() in 3b8c9f1cdfc5 to handle this, because our broadcast I-cache maintenance doesn't force a pipeline flush for remote CPUs (and may even execute as a NOP on recent cores).

Quoted 4 lines

Ah, I think the tlbie does not force re-fetch indeed. We may need something like that as well.

What do you do on the user side? Require threads to ISB themselves?

I think they'd probably have to use sys_membarrier() with MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE, yes.

Will