arm64: mm: install KPTI nG mappings with MMU enabled

In cases where we unmap the kernel while running in user space, we rely on ASIDs to distinguish the minimal trampoline from the full kernel mapping, and this means we must use non-global attributes for those mappings, to ensure they are scoped by ASID and will not hit in the TLB inadvertently.

We only do this when needed, as this is generally more costly in terms of TLB pressure, and so we boot without these non-global attributes, and apply them to all existing kernel mappings once all CPUs are up and we know whether or not the non-global attributes are needed. At this point, we cannot simply unmap and remap the entire address space, so we have to update all existing block and page descriptors in place.

Currently, we go through a lot of trouble to perform these updates with the MMU and caches off, to avoid violating break before make (BBM) rules imposed by the architecture. Since we make changes to page tables that are not covered by the ID map, we gain access to those descriptors by disabling translations altogether. This means that the stores to memory are issued with device attributes, and require extra care in terms of coherency, which is costly. We also rely on the ID map to access a shared flag, which requires the ID map to be executable and writable at the same time, which is another thing we'd prefer to avoid.

So let's switch to an approach where we replace the kernel mapping with a minimal mapping of a few pages that can be used for the shared flag, as well as a minimal, ad-hoc fixmap that we can use to map each page table in turn as we traverse the hierarchy. This requires one PTE per level, and an associated page worth of VA space in the temporary mapping.

Note that table descriptors do not have a non-global attribute, so avoid setting this bit unnecessarily as well. Also, given that the bulk of time is spent updating the dense level 3 entries that cover the linear map, let's just rewrite all block and page entries, and OR the valid bit into the nG bit position.

On a KVM guest with 64G of memory booting with rodata=full under a ThunderX2 host (*), this reduces the boot delay from 25 seconds to 50 milliseconds.

(*) ThunderX2 does not need KPTI to mitigate Meltdown so the build was rigged to enable it nonetheless when taking this measurement as otherwise, this routine never executes.

---
v2: - use break before make when remapping the page table pages
- switch via reserved_pg_dir when going to the temporary page tables and back
- improve comments describing what is going on
- leave it to the last CPU that leaves to free the temporary pages

 arch/arm64/include/asm/mmu.h   |   4 +
 arch/arm64/kernel/cpufeature.c |  68 ++++++++++-
 arch/arm64/mm/mmu.c            |   8 +-
 arch/arm64/mm/proc.S           | 119 +++++++++++---------
 4 files changed, 138 insertions(+), 61 deletions(-)

index 48f8466a4be9..b896f0ac4985 100644
@@ -68,6 +68,10 @@ extern void init_mem_pgprot(void);
 extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys,
 			       unsigned long virt, phys_addr_t size,
 			       pgprot_t prot, bool page_mappings_only);
+extern void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,
+				 unsigned long virt, phys_addr_t size,
+				 pgprot_t prot,
+				 phys_addr_t (*pgtable_alloc)(int), int flags);
 extern void *fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot);
 extern void mark_linear_text_alias_ro(void);
 extern bool kaslr_requires_kpti(void);

index 626ec32873c6..1c7299dfaa84 100644
@@ -360,11 +360,9 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end,
 		mutex_unlock(&fixmap_lock);
 }
 
-static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,
-				 unsigned long virt, phys_addr_t size,
-				 pgprot_t prot,
-				 phys_addr_t (*pgtable_alloc)(int),
-				 int flags)
+void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt,
+			  phys_addr_t size, pgprot_t prot,
+			  phys_addr_t (*pgtable_alloc)(int), int flags)
 {
 	unsigned long addr, end, next;
 	pgd_t *pgdp = pgd_offset_pgd(pgdir, virt);

2.30.2


_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel