arm64: apply G-to-nG conversion for KPTI with MMU enabled
v4 cover arm64: apply G-to-nG conversion for KPTI with MMU enabled
Rationale for the series is in patch #2.
Changes since v3 [0]:
- fix issue pointed out by Mark in patch #1, and add his Rb/Tb;
- move shared flag back into the ID map, as this allows us to revert to the existing synchronization scheme, and drastically simplify the patch - we can move the flag into a separate page later, if needed;
- use symbol alias to access __create_pgd_mapping() so we are not forced to expose it in general.
Changes since v2 [1]:
- reinstate optimization that avoids descending into the same page tables repeatedly when KASAN is enabled;
- use broadcast TLB maintenance when doing BBM for the temporary mappings, to prevent potential TLB conflicts on the secondary cores that are running from the same set of temporary page tables;
- remove a wait loop in the C code, by using 'num_cpus + 1' as the signal value rather then 0x0;
- declutter and simplify the asm helper code, to make it more maintainable.
[0] https://lore.kernel.org/linux-arm-kernel/[email protected]/
[1] https://lore.kernel.org/linux-arm-kernel/[email protected]/
Ard Biesheuvel (2):
arm64: kpti-ng: simplify page table traversal logic
arm64: mm: install KPTI nG mappings with MMU enabled
arch/arm64/kernel/cpufeature.c | 54 +++++- arch/arm64/mm/mmu.c | 7 + arch/arm64/mm/proc.S | 173 ++++++++++---------- 3 files changed, 145 insertions(+), 89 deletions(-)
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 1/2 arm64: kpti-ng: simplify page table traversal logic
Simplify the KPTI G-to-nG asm helper code by:
- pulling the 'table bit' test into the get/put macros so we can combine them and incorporate the entire loop;
- moving the 'table bit' test after the update of bit #11 so we no longer need separate next_xxx and skip_xxx labels;
- redefining the pmd/pud register aliases and the next_pmd/next_pud labels instead of branching to them if the number of configured page table levels is less than 3 or 4, respectively;
- folding the descriptor pointer increment into the LDR instructions.
No functional change intended, except for the fact that we now descend into a next level table after setting bit #11 on its descriptor but this should make no difference in practice.
While at it, switch to .L prefixed local labels so they don't clutter up the symbol tables, kallsyms, etc, and clean up the indentation for legibility.
--- arch/arm64/mm/proc.S | 100 +++++++------------- 1 file changed, 36 insertions(+), 64 deletions(-)
arch/arm64/mm/proc.S +36 -64
@@ -202,19 +202,25 @@ SYM_FUNC_END(idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 .pushsection ".idmap.text", "awx" - .macro __idmap_kpti_get_pgtable_ent, type - dc cvac, cur_\()\type\()p // Ensure any existing dirty + .macro kpti_mk_tbl_ng, type, num_entries + add end_\type\()p, cur_\type\()p, #\num_entries * 8 +.Ldo_\type: + dc cvac, cur_\type\()p // Ensure any existing dirty dmb sy // lines are written back before - ldr \type, [cur_\()\type\()p] // loading the entry - tbz \type, #0, skip_\()\type // Skip invalid and - tbnz \type, #11, skip_\()\type // non-global entries - .endm - - .macro __idmap_kpti_put_pgtable_ent_ng, type + ldr \type, [cur_\type\()p] // loading the entry + tbz \type, #0, .Lnext_\type // Skip invalid and + tbnz \type, #11, .Lnext_\type // non-global entries orr \type, \type, #PTE_NG // Same bit for blocks and pages - str \type, [cur_\()\type\()p] // Update the entry and ensure + str \type, [cur_\type\()p] // Update the entry and ensure dmb sy // that it is visible to all dc civac, cur_\()\type\()p // CPUs. + .ifnc \type, pte + tbnz \type, #1, .Lderef_\type + .endif +.Lnext_\type: + add cur_\type\()p, cur_\type\()p, #8 + cmp cur_\type\()p, end_\type\()p + b.ne .Ldo_\type .endm /*
@@ -235,10 +241,8 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) pgd .req x7 cur_pudp .req x8 end_pudp .req x9 - pud .req x10 cur_pmdp .req x11 end_pmdp .req x12 - pmd .req x13 cur_ptep .req x14 end_ptep .req x15 pte .req x16
@@ -265,16 +269,8 @@ SYM_FUNC_START(idmap_kpti_install_ng_mappings) /* Everybody is enjoying the idmap, so we can rewrite swapper. */ /* PGD */ - mov cur_pgdp, swapper_pa - add end_pgdp, cur_pgdp, #(PTRS_PER_PGD * 8) -do_pgd: __idmap_kpti_get_pgtable_ent pgd - tbnz pgd, #1, walk_puds -next_pgd: - __idmap_kpti_put_pgtable_ent_ng pgd -skip_pgd: - add cur_pgdp, cur_pgdp, #8 - cmp cur_pgdp, end_pgdp - b.ne do_pgd + mov cur_pgdp, swapper_pa + kpti_mk_tbl_ng pgd, PTRS_PER_PGD /* Publish the updated tables and nuke all the TLBs */ dsb sy
@@ -291,59 +287,35 @@ skip_pgd: str wzr, [flag_ptr] ret +.Lderef_pgd: /* PUD */ -walk_puds: - .if CONFIG_PGTABLE_LEVELS > 3 + .if CONFIG_PGTABLE_LEVELS > 3 + pud .req x10 pte_to_phys cur_pudp, pgd - add end_pudp, cur_pudp, #(PTRS_PER_PUD * 8) -do_pud: __idmap_kpti_get_pgtable_ent pud - tbnz pud, #1, walk_pmds -next_pud: - __idmap_kpti_put_pgtable_ent_ng pud -skip_pud: - add cur_pudp, cur_pudp, 8 - cmp cur_pudp, end_pudp - b.ne do_pud - b next_pgd - .else /* CONFIG_PGTABLE_LEVELS <= 3 */ - mov pud, pgd - b walk_pmds -next_pud: - b next_pgd + kpti_mk_tbl_ng pud, PTRS_PER_PUD + b .Lnext_pgd + .else /* CONFIG_PGTABLE_LEVELS <= 3 */ + pud .req pgd + .set .Lnext_pud, .Lnext_pgd .endif +.Lderef_pud: /* PMD */ -walk_pmds: - .if CONFIG_PGTABLE_LEVELS > 2 + .if CONFIG_PGTABLE_LEVELS > 2 + pmd .req x13 pte_to_phys cur_pmdp, pud - add end_pmdp, cur_pmdp, #(PTRS_PER_PMD * 8) -do_pmd: __idmap_kpti_get_pgtable_ent pmd - tbnz pmd, #1, walk_ptes -next_pmd: - __idmap_kpti_put_pgtable_ent_ng pmd -skip_pmd: - add cur_pmdp, cur_pmdp, #8 - cmp cur_pmdp, end_pmdp - b.ne do_pmd - b next_pud - .else /* CONFIG_PGTABLE_LEVELS <= 2 */ - mov pmd, pud - b walk_ptes -next_pmd: - b next_pud + kpti_mk_tbl_ng pmd, PTRS_PER_PMD + b .Lnext_pud + .else /* CONFIG_PGTABLE_LEVELS <= 2 */ + pmd .req pgd + .set .Lnext_pmd, .Lnext_pgd .endif +.Lderef_pmd: /* PTE */ -walk_ptes: pte_to_phys cur_ptep, pmd - add end_ptep, cur_ptep, #(PTRS_PER_PTE * 8) -do_pte: __idmap_kpti_get_pgtable_ent pte - __idmap_kpti_put_pgtable_ent_ng pte -skip_pte: - add cur_ptep, cur_ptep, #8 - cmp cur_ptep, end_ptep - b.ne do_pte - b next_pmd + kpti_mk_tbl_ng pte, PTRS_PER_PTE + b .Lnext_pmd .unreq cpu .unreq num_cpus
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v4 2/2 arm64: mm: install KPTI nG mappings with MMU enabled
In reply to Ard BiesheuvelIn cases where we unmap the kernel while running in user space, we rely on ASIDs to distinguish the minimal trampoline from the full kernel mapping, and this means we must use non-global attributes for those mappings, to ensure they are scoped by ASID and will not hit in the TLB inadvertently.
We only do this when needed, as this is generally more costly in terms of TLB pressure, and so we boot without these non-global attributes, and apply them to all existing kernel mappings once all CPUs are up and we know whether or not the non-global attributes are needed. At this point, we cannot simply unmap and remap the entire address space, so we have to update all existing block and page descriptors in place.
Currently, we go through a lot of trouble to perform these updates with the MMU and caches off, to avoid violating break before make (BBM) rules imposed by the architecture. Since we make changes to page tables that are not covered by the ID map, we gain access to those descriptors by disabling translations altogether. This means that the stores to memory are issued with device attributes, and require extra care in terms of coherency, which is costly. We also rely on the ID map to access a shared flag, which requires the ID map to be executable and writable at the same time, which is another thing we'd prefer to avoid.
So let's switch to an approach where we replace the kernel mapping with a minimal mapping of a few pages that can be used for a minimal, ad-hoc fixmap that we can use to map each page table in turn as we traverse the hierarchy.
--- arch/arm64/kernel/cpufeature.c | 54 ++++++++++++- arch/arm64/mm/mmu.c | 7 ++ arch/arm64/mm/proc.S | 81 +++++++++++++------- 3 files changed, 113 insertions(+), 29 deletions(-)
arch/arm64/kernel/cpufeature.c +51 -3
@@ -1645,14 +1645,34 @@ static bool unmap_kernel_at_el0(const struct arm64_cpu_capabilities *entry, }#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +#define KPTI_NG_TEMP_VA (-(1UL << PMD_SHIFT)) + +extern +void create_kpti_ng_temp_pgd (pgd_t *pgdir,phys_addr_t phys,unsigned long virt, +phys_addr_t size,pgprot_t prot, +phys_addr_t (*pgtable_alloc)(int ),int flags); + +static phys_addr_t kpti_ng_temp_alloc; + +static phys_addr_t kpti_ng_pgd_alloc (int shift) +{ + kpti_ng_temp_alloc -=PAGE_SIZE ; +return kpti_ng_temp_alloc; +} + static void __nocfikpti_install_ng_mappings (const struct arm64_cpu_capabilities *__unused) { -typedef void (kpti_remap_fn )(int ,int ,phys_addr_t ); +typedef void (kpti_remap_fn )(int ,int ,phys_addr_t ,unsigned long );extern kpti_remap_fn idmap_kpti_install_ng_mappings;kpti_remap_fn *remap_fn;int cpu =smp_processor_id (); +int levels =CONFIG_PGTABLE_LEVELS ; +int order =order_base_2 (levels); +u64 kpti_ng_temp_pgd_pa =0 ; +pgd_t *kpti_ng_temp_pgd; +u64 alloc =0 ;if (__this_cpu_read (this_cpu_vector) == vectors) {const char *v =arm64_get_bp_hardening_vector (EL1_VECTOR_KPTI );
@@ -1670,12 +1690,40 @@ kpti_install_ng_mappings(const struct arm64_cpu_capabilities *__unused) remap_fn = (void *)__pa_symbol (function_nocfi (idmap_kpti_install_ng_mappings)); +if (!cpu) { + alloc =__get_free_pages (GFP_ATOMIC | __GFP_ZERO, order); + kpti_ng_temp_pgd = (pgd_t *)(alloc + levels *PAGE_SIZE ); + kpti_ng_temp_alloc = kpti_ng_temp_pgd_pa =__pa (kpti_ng_temp_pgd); + +// +// Create a minimal page table hierarchy that permits us to map +// the swapper page tables temporarily as we traverse them. +// +// The physical pages are laid out as follows: +// +// +--------+-/-------+-/------ +-\\--------+ +// : PTE[] : | PMD[] : | PUD[] : || PGD[] : +// +--------+-\-------+-\------ +-//--------+ +// ^ +// The first page is mapped into this hierarchy at a PMD_SHIFT +// aligned virtual address, so that we can manipulate the PTE +// level entries while the mapping is active. The first entry +// covers the PTE[] page itself, the remaining entries are free +// to be used as a ad-hoc fixmap. +// +create_kpti_ng_temp_pgd (kpti_ng_temp_pgd,__pa (alloc), +KPTI_NG_TEMP_VA ,PAGE_SIZE ,PAGE_KERNEL , + kpti_ng_pgd_alloc,0 ); + } +cpu_install_idmap (); -remap_fn (cpu,num_online_cpus (),__pa_symbol (swapper_pg_dir)); +remap_fn (cpu,num_online_cpus (), kpti_ng_temp_pgd_pa,KPTI_NG_TEMP_VA );cpu_uninstall_idmap (); -if (!cpu) +if (!cpu) { +free_pages (alloc, order); arm64_use_ng_mappings = true; + } }#else static void
arch/arm64/mm/mmu.c +7 -0
@@ -388,6 +388,13 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, }while (pgdp++, addr = next, addr != end); } +#ifdef CONFIG_UNMAP_KERNEL_AT_EL0 +extern __alias (__create_pgd_mapping) +void create_kpti_ng_temp_pgd(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, + phys_addr_t size, pgprot_t prot, + phys_addr_t (*pgtable_alloc)(int ),int flags); +#endif +static phys_addr_t __pgd_pgtable_alloc (int shift) {void *ptr = (void *)__get_free_page (GFP_PGTABLE_KERNEL );
arch/arm64/mm/proc.S +55 -26
171 lines not on this page · raw message
Signature · 7 lines
2.30.2 _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v4 2/2] arm64: mm: install KPTI nG mappings with MMU enabled
Quoted 24 lines · Ard Biesheuvel
In cases where we unmap the kernel while running in user space, we rely on ASIDs to distinguish the minimal trampoline from the full kernel mapping, and this means we must use non-global attributes for those mappings, to ensure they are scoped by ASID and will not hit in the TLB inadvertently.
We only do this when needed, as this is generally more costly in terms of TLB pressure, and so we boot without these non-global attributes, and apply them to all existing kernel mappings once all CPUs are up and we know whether or not the non-global attributes are needed. At this point, we cannot simply unmap and remap the entire address space, so we have to update all existing block and page descriptors in place.
Currently, we go through a lot of trouble to perform these updates with the MMU and caches off, to avoid violating break before make (BBM) rules imposed by the architecture. Since we make changes to page tables that are not covered by the ID map, we gain access to those descriptors by disabling translations altogether. This means that the stores to memory are issued with device attributes, and require extra care in terms of coherency, which is costly. We also rely on the ID map to access a shared flag, which requires the ID map to be executable and writable at the same time, which is another thing we'd prefer to avoid.
Quoted 6 lines
So let's switch to an approach where we replace the kernel mapping with a minimal mapping of a few pages that can be used for a minimal, ad-hoc fixmap that we can use to map each page table in turn as we traverse the hierarchy.
Signed-off-by Ard Biesheuvel <[email protected]>
Ugh this patch is broken - please disregard.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel