arm64: Reorganize kernel VA space for LPA2
v6 cover arm64: Reorganize kernel VA space for LPA2
From: Ard Biesheuvel <[email protected]>
At the request of Catalin, this series was split off from my LPA2 series
[0] in order to make the changes a bit more manageable.
This series reorganizes the kernel VA space, and refactors/replaces the early mapping code so that:
- everything is done only once, in the appropriate order;
- everything is done with the MMU and caches enabled (*)
- everything is done from C code (notably, 100s of lines of incomprehensible asm code are removed from head.S).
(*) the initial ID map will be populated with the MMU and caches disabled if that is how we entered from the bootloader.
This is important for LPA2, but also for other future extensions to the page table format, as managing this entirely in early asm code as we do today would become intractable. This applies also to things such as copying the KAsan shadow or the fixmap from the early page tables into the permanent ones - this is all being removed by this series. This approach also ensures that we never execute from writable memory, or parse the DT (which is external input) while the text/rodata segments are mapped writable; this is an important general hardening principle, but also a prerequisite for adding WXN support (which is implemented in the second half of the series that has been omitted from this drop)
Another notable difference implemented by this series is the fact that the permanent ID map always covers 48 bits of VA space, and is no longer tied to the size of the kernel VA space. This removes awkward logic to add a translation level above PGD level, and will be beneficial for other reasons too (it permits future changes in the EFI logic to get rid of SetVirtualAddressMap() entirely)
Changes since v5 [1]:
- add helpers to deal with CPU feature overrides, rather than applying the value and mask directly - this is necessary because an override may be invalid for the field that we care about or for another field that shares the same CPUID system register
- add missing patch to make __cpu_replace_ttbr1() out of line and insert it in the correct place in the series to ensure bisectability
- incorporate maz's strict type changes (and more) into the prel64 handling in the early idreg override code
Changes since v4:
- merge a couple of followup tweaks for issues that were reported while the v4 was briefly queued up and pulled into -next
- rebase onto v6.7-rc1
- omit LVA/LPA2 and WXN related changes
[0] https://lore.kernel.org/all/[email protected]/
[1] https://lore.kernel.org/all/[email protected]/
Ard Biesheuvel (41):
arm64: kernel: Disable latent_entropy GCC plugin in early C runtime
arm64: mm: Take potential load offset into account when KASLR is off
arm64: mm: get rid of kimage_vaddr global variable
arm64: mm: Move PCI I/O emulation region above the vmemmap region
arm64: mm: Move fixmap region above vmemmap region
arm64: ptdump: Allow all region boundaries to be defined at boot time
arm64: ptdump: Discover start of vmemmap region at runtime
arm64: vmemmap: Avoid base2 order of struct page size to dimension
region
arm64: mm: Reclaim unused vmemmap region for vmalloc use
arm64: kaslr: Adjust randomization range dynamically
arm64: kernel: Manage absolute relocations in code built under pi/
arm64: kernel: Don't rely on objcopy to make code under pi/ __init
arm64: head: move relocation handling to C code
arm64: idreg-override: Omit non-NULL checks for override pointer
arm64: idreg-override: Prepare for place relative reloc patching
arm64: idreg-override: Avoid parameq() and parameqn()
arm64: idreg-override: avoid strlen() to check for empty strings
arm64: idreg-override: Avoid sprintf() for simple string concatenation
arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit arm64/kernel: Move 'nokaslr' parsing out of early idreg code
arm64: idreg-override: Move to early mini C runtime
arm64: kernel: Remove early fdt remap code
arm64: head: Clear BSS and the kernel page tables in one go
arm64: Move feature overrides into the BSS section
arm64: head: Run feature override detection before mapping the kernel
arm64: head: move dynamic shadow call stack patching into early C
runtime
arm64: cpufeature: Add helper to test for CPU feature overrides
arm64: kaslr: Use feature override instead of parsing the cmdline
again
arm64: idreg-override: Create a pseudo feature for rodata=off
arm64: Add helpers to probe local CPU for PAC and BTI support
arm64: head: allocate more pages for the kernel mapping
arm64: head: move memstart_offset_seed handling to C code
arm64: mm: Make kaslr_requires_kpti() a static inline
arm64: mmu: Make __cpu_replace_ttbr1() out of line
arm64: head: Move early kernel mapping routines into C code
arm64: mm: Use 48-bit virtual addressing for the permanent ID map
arm64: pgtable: Decouple PGDIR size macros from PGD/PUD/PMD levels
arm64: kernel: Create initial ID map from C code
arm64: mm: avoid fixmap for early swapper_pg_dir updates
arm64: mm: omit redundant remap of kernel image
arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()"
arch/arm64/include/asm/archrandom.h | 2 -
arch/arm64/include/asm/assembler.h | 14 -
arch/arm64/include/asm/cpufeature.h | 77 ++++
arch/arm64/include/asm/fixmap.h | 1 -
arch/arm64/include/asm/kasan.h | 2 -
arch/arm64/include/asm/kernel-pgtable.h | 128 +++---
arch/arm64/include/asm/memory.h | 20 +-
arch/arm64/include/asm/mmu.h | 40 +-
arch/arm64/include/asm/mmu_context.h | 53 +--
arch/arm64/include/asm/pgtable.h | 10 +-
arch/arm64/include/asm/scs.h | 36 +-
arch/arm64/include/asm/setup.h | 3 -
arch/arm64/kernel/Makefile | 7 +-
arch/arm64/kernel/cpufeature.c | 65 +--
arch/arm64/kernel/head.S | 428 ++------------------
arch/arm64/kernel/image-vars.h | 33 ++
arch/arm64/kernel/kaslr.c | 11 +-
arch/arm64/kernel/module.c | 2 +-
arch/arm64/kernel/pi/Makefile | 28 +-
arch/arm64/kernel/{ => pi}/idreg-override.c | 182 +++++----
arch/arm64/kernel/pi/kaslr_early.c | 78 +---
arch/arm64/kernel/pi/map_kernel.c | 186 +++++++++
arch/arm64/kernel/pi/map_range.c | 100 +++++
arch/arm64/kernel/{ => pi}/patch-scs.c | 36 +-
arch/arm64/kernel/pi/pi.h | 36 ++
arch/arm64/kernel/pi/relacheck.c | 130 ++++++
arch/arm64/kernel/pi/relocate.c | 64 +++
arch/arm64/kernel/setup.c | 22 -
arch/arm64/kernel/vmlinux.lds.S | 17 +-
arch/arm64/kvm/mmu.c | 15 +-
arch/arm64/mm/fixmap.c | 34 --
arch/arm64/mm/kasan_init.c | 19 +-
arch/arm64/mm/mmu.c | 167 ++++----
arch/arm64/mm/proc.S | 13 +-
arch/arm64/mm/ptdump.c | 56 ++-
35 files changed, 1070 insertions(+), 1045 deletions(-)
rename arch/arm64/kernel/{ => pi}/idreg-override.c (59%)
create mode 100644 arch/arm64/kernel/pi/map_kernel.c
create mode 100644 arch/arm64/kernel/pi/map_range.c
rename arch/arm64/kernel/{ => pi}/patch-scs.c (89%)
create mode 100644 arch/arm64/kernel/pi/pi.h
create mode 100644 arch/arm64/kernel/pi/relacheck.c
create mode 100644 arch/arm64/kernel/pi/relocate.cSignature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 4/41 arm64: mm: Move PCI I/O emulation region above the vmemmap region
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
arch/arm64/include/asm/memory.h +2 -2
@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)#if VA_BITS >48
arch/arm64/mm/ptdump.c +2 -2
@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 1/41 arm64: kernel: Disable latent_entropy GCC plugin in early C runtime
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
In subsequent patches, mark portions of the early C code will be marked as __init. Unfortunarely, __init implies __latent_entropy, and this would result in the early C code being instrumented in an unsafe manner.
Disable the latent entropy plugin for the early C code.
--- arch/arm64/kernel/pi/Makefile | 1 + 1 file changed, 1 insertion(+)
arch/arm64/kernel/pi/Makefile +1 -0
@@ -3,6 +3,7 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -Os -DDISABLE_BRANCH_PROFILING $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_LATENT_ENTROPY_PLUGIN) \ $(call cc-option,-mbranch-protection=none) \ -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ -include $(srctree)/include/linux/hidden.h \
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 2/41 arm64: mm: Take potential load offset into account when KASLR is off
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
We enable CONFIG_RELOCATABLE even when CONFIG_RANDOMIZE_BASE is disabled, and this permits the loader (i.e., EFI) to place the kernel anywhere in physical memory as long as the base address is 64k aligned.
This means that the 'KASLR' case described in the header that defines the size of the statically allocated page tables could take effect even when CONFIG_RANDMIZE_BASE=n. So check for CONFIG_RELOCATABLE instead.
--- arch/arm64/include/asm/kernel-pgtable.h | 27 +++++--------------- 1 file changed, 6 insertions(+), 21 deletions(-)
arch/arm64/include/asm/kernel-pgtable.h +6 -21
@@ -37,27 +37,12 @@/* -* If KASLR is enabled, then an offset K is added to the kernel address -* space. The bottom 21 bits of this offset are zero to guarantee 2MB -* alignment for PA and VA. -* -* For each pagetable level of the swapper, we know that the shift will -* be larger than 21 (for the 4KB granule case we use section maps thus -* the smallest shift is actually 30) thus there is the possibility that -* KASLR can increase the number of pagetable entries by 1, so we make -* room for this extra entry. -* -* Note KASLR cannot increase the number of required entries for a level -* by more than one because it increments both the virtual start and end -* addresses equally (the extra entry comes from the case where the end -* address is just pushed over a boundary and the start address isn't). +* A relocatable kernel may execute from an address that differs from the one at +* which it was linked. In the worst case, its runtime placement may intersect +* with two adjacent PGDIR entries, which means that an additional page table +* may be needed at each subordinate level. */ - -#ifdef CONFIG_RANDOMIZE_BASE -#define EARLY_KASLR (1) -#else -#define EARLY_KASLR (0) -#endif +#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE)#define SPAN_NR_ENTRIES (vstart, vend, shift) \ ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1)
@@ -83,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR)) +#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE))/* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS <48
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 5/41 arm64: mm: Move fixmap region above vmemmap region
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Move the fixmap region above the vmemmap region, so that the start of the vmemmap delineates the end of the region available for vmalloc and vmap allocations and the randomized placement of the kernel and modules.
In a subsequent patch, we will take advantage of this to reclaim most of the vmemmap area when running a 52-bit VA capable build with 52-bit virtual addressing disabled at runtime.
Note that the existing guard region of 256 MiB covers the fixmap and PCI I/O regions as well, so we can reduce it 8 MiB, which is what we use in other places too.
--- arch/arm64/include/asm/memory.h | 2 +- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/mm/ptdump.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)
arch/arm64/include/asm/memory.h +1 -1
@@ -51,7 +51,7 @@#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE)#define PCI_IO_START (VMEMMAP_END + SZ_8M)#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) -#define FIXADDR_TOP (VMEMMAP_START - SZ_32M) +#define FIXADDR_TOP (-UL(SZ_8M))#if VA_BITS >48 #define VA_BITS_MIN (48)
arch/arm64/include/asm/pgtable.h +1 -1
@@ -22,7 +22,7 @@* and fixed mappings */ #define VMALLOC_START (MODULES_END) -#define VMALLOC_END (VMEMMAP_START - SZ_256M) +#define VMALLOC_END (VMEMMAP_START - SZ_8M)#define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT))
arch/arm64/mm/ptdump.c +2 -2
@@ -45,12 +45,12 @@ static struct addr_marker address_markers[] = { { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, { -1, NULL }, };
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 3/41 arm64: mm: get rid of kimage_vaddr global variable
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
We store the address of _text in kimage_vaddr, but since commit 09e3c22a86f6889d ("arm64: Use a variable to store non-global mappings decision"), we no longer reference this variable from modules so we no longer need to export it.
In fact, we don't need it at all so let's just get rid of it.
--- arch/arm64/include/asm/memory.h | 6 ++---- arch/arm64/kernel/head.S | 2 +- arch/arm64/mm/mmu.c | 3 --- 3 files changed, 3 insertions(+), 8 deletions(-)
arch/arm64/include/asm/memory.h +2 -4
@@ -182,6 +182,7 @@#include <linux/types.h> #include <asm/boot.h> #include <asm/bug.h> +#include <asm/sections.h> #if VA_BITS >48 extern u64 vabits_actual;
@@ -193,15 +194,12 @@ extern s64 memstart_addr;/* PHYS_OFFSET - the physical address of the start of memory. */ #define PHYS_OFFSET ({ VM_BUG_ON(memstart_addr & 1); memstart_addr; }) -/* the virtual base of the kernel image */ -extern u64 kimage_vaddr; -/* the offset between the kernel virtual and physical mappings */ extern u64 kimage_voffset;static inline unsigned long kaslr_offset (void ) { -return kimage_vaddr -KIMAGE_VADDR ; +return (u64 )&_text -KIMAGE_VADDR ; }#ifdef CONFIG_RANDOMIZE_BASE
arch/arm64/kernel/head.S +1 -1
@@ -482,7 +482,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) str_l x21, __fdt_pointer, x5 // Save FDT pointer - ldr_l x4, kimage_vaddr // Save the offset between + adrp x4, _text // Save the offset between sub x4, x4, x0 // the kernel virtual and str_l x4, kimage_voffset, x5 // physical mappings
arch/arm64/mm/mmu.c +0 -3
@@ -52,9 +52,6 @@ u64 vabits_actual __ro_after_init = VA_BITS_MIN;EXPORT_SYMBOL (vabits_actual);#endif -u64 kimage_vaddr__ro_after_init = (u64)&_text; -EXPORT_SYMBOL (kimage_vaddr); -u64 kimage_voffset__ro_after_init ;EXPORT_SYMBOL (kimage_voffset);
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 6/41 arm64: ptdump: Allow all region boundaries to be defined at boot time
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Rework the way the address_markers array is populated so that we can tolerate values that are not compile time constants generally, rather than keeping track manually of the array indexes in question, and poking new values into them manually. This will be needed for VMALLOC_END, which will cease to be a compile time constant after a subsequent patch.
--- arch/arm64/mm/ptdump.c | 54 ++++++++------------ 1 file changed, 22 insertions(+), 32 deletions(-)
arch/arm64/mm/ptdump.c +22 -32
@@ -26,34 +26,6 @@#include <asm/ptdump.h> -enum address_markers_idx { -PAGE_OFFSET_NR =0 , -PAGE_END_NR , -#if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) -KASAN_START_NR , -#endif -}; - -static struct addr_marker address_markers[] = { - {PAGE_OFFSET ,"Linear Mapping start" }, - {0 /* PAGE_END */ ,"Linear Mapping end" }, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - {0 /* KASAN_SHADOW_START */ ,"Kasan shadow start" }, - {KASAN_SHADOW_END ,"Kasan shadow end" }, -#endif - {MODULES_VADDR ,"Modules start" }, - {MODULES_END ,"Modules end" }, - {VMALLOC_START ,"vmalloc() area" }, - {VMALLOC_END ,"vmalloc() end" }, - {VMEMMAP_START ,"vmemmap start" }, - {VMEMMAP_START +VMEMMAP_SIZE ,"vmemmap end" }, - {PCI_IO_START ,"PCI I/O start" }, - {PCI_IO_END ,"PCI I/O end" }, - {FIXADDR_TOT_START ,"Fixmap start" }, - {FIXADDR_TOP ,"Fixmap end" }, - {-1 ,NULL }, -}; - #define pt_dump_seq_printf(m, fmt, args...) \ ({ \ if (m) \
@@ -339,9 +311,8 @@ static void __init ptdump_initialize(void) pg_level[i].mask |= pg_level[i].bits[j].mask; } -static struct ptdump_info kernel_ptdump_info = { +static struct ptdump_info kernel_ptdump_info__ro_after_init = { .mm = &init_mm, - .markers = address_markers, .base_addr =PAGE_OFFSET , };
@@ -375,10 +346,29 @@ void ptdump_check_wx(void)static int __init ptdump_init (void ) { - address_markers[PAGE_END_NR ].start_address =PAGE_END ; +struct addr_marker m[] = { + {PAGE_OFFSET ,"Linear Mapping start" }, + {PAGE_END ,"Linear Mapping end" }, #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - address_markers[KASAN_START_NR ].start_address =KASAN_SHADOW_START ; + {KASAN_SHADOW_START ,"Kasan shadow start" }, + {KASAN_SHADOW_END ,"Kasan shadow end" }, #endif + {MODULES_VADDR ,"Modules start" }, + {MODULES_END ,"Modules end" }, + {VMALLOC_START ,"vmalloc() area" }, + {VMALLOC_END ,"vmalloc() end" }, + {VMEMMAP_START ,"vmemmap start" }, + {VMEMMAP_START +VMEMMAP_SIZE ,"vmemmap end" }, + {PCI_IO_START ,"PCI I/O start" }, + {PCI_IO_END ,"PCI I/O end" }, + {FIXADDR_TOT_START ,"Fixmap start" }, + {FIXADDR_TOP ,"Fixmap end" }, + {-1 ,NULL }, + }; +static struct addr_marker address_markers[ARRAY_SIZE (m)]__ro_after_init ; + + kernel_ptdump_info.markers =memcpy (address_markers, m,sizeof (m)); +ptdump_initialize ();ptdump_debugfs_register (&kernel_ptdump_info,"kernel_page_tables" );return 0 ;
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 7/41 arm64: ptdump: Discover start of vmemmap region at runtime
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
We will soon reclaim the part of the vmemmap region that covers VA space that is not addressable by the hardware. To avoid confusion, ensure that the 'vmemmap start' marker points at the start of the region that is actually being used for the struct page array, rather than the start of the region we set aside for it at build time.
--- arch/arm64/mm/ptdump.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
arch/arm64/mm/ptdump.c +3 -1
@@ -346,6 +346,8 @@ void ptdump_check_wx(void)static int __init ptdump_init (void ) { +u64 page_offset =_PAGE_OFFSET (vabits_actual); +u64 vmemmap_start = (u64 )virt_to_page ((void *)page_offset);struct addr_marker m[] = { {PAGE_OFFSET ,"Linear Mapping start" }, {PAGE_END ,"Linear Mapping end" },
@@ -357,7 +359,7 @@ static int __init ptdump_init(void) { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { VMEMMAP_START, "vmemmap start" }, + { vmemmap_start, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" },
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 17/41 arm64: idreg-override: avoid strlen() to check for empty strings
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
strlen() is a costly way to decide whether a string is empty, as in that case, the first character will be NUL so we can check for that directly.
--- arch/arm64/kernel/idreg-override.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/arm64/kernel/idreg-override.c +1 -1
@@ -232,7 +232,7 @@ static void __init match_options(const char *cmdline) override =prel64_pointer (reg->override); -for (f =0 ;strlen (reg->fields[f].name); f++) { +for (f =0 ; reg->fields[f].name[0 ] !='\0' ; f++) {u64 shift = reg->fields[f].shift;u64 width = reg->fields[f].width ?:4 ;u64 mask =GENMASK_ULL (shift + width -1 , shift);
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 16/41 arm64: idreg-override: Avoid parameq() and parameqn()
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The only way parameq() and parameqn() deviate from the ordinary string and memory routines is that they ignore the difference between dashes and underscores.
Since we copy each command line argument into a buffer before passing it to parameq() and parameqn() numerous times, let's just convert all dashes to underscores just once, and update the alias array accordingly.
This also helps reduce the dependency on kernel APIs that are no longer available once we move this code into the early mini C runtime.
--- arch/arm64/kernel/idreg-override.c | 28 ++++++++++++-------- 1 file changed, 17 insertions(+), 11 deletions(-)
arch/arm64/kernel/idreg-override.c +17 -11
@@ -185,8 +185,8 @@ static const struct {char alias[FTR_ALIAS_NAME_LEN ];char feature[FTR_ALIAS_OPTION_LEN ]; } aliases[] __initconst = { - { "kvm-arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, - { "kvm-arm.mode=protected", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=nvhe", "id_aa64mmfr1.vh=0" }, + { "kvm_arm.mode=protected", "id_aa64mmfr1.vh=0" }, { "arm64.nosve", "id_aa64pfr0.sve=0" }, { "arm64.nosme", "id_aa64pfr1.sme=0" }, { "arm64.nobti", "id_aa64pfr1.bt=0" },
@@ -215,7 +215,7 @@ static int __init find_field(const char *cmdline, len =snprintf (opt,ARRAY_SIZE (opt),"%s.%s=" , reg->name, reg->fields[f].name); -if (!parameqn (cmdline, opt, len)) +if (memcmp (cmdline, opt, len))return -1 ;return kstrtou64 (cmdline + len,0 , v);
@@ -272,23 +272,29 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) cmdline =skip_spaces (cmdline); -for (len =0 ; cmdline[len] && !isspace (cmdline[len]); len++); -if (!len) +/* terminate on "--" appearing on the command line by itself */ +if (cmdline[0 ] =='-' && cmdline[1 ] =='-' &&isspace (cmdline[2 ]))return ; - len =min (len,ARRAY_SIZE (buf) -1 ); -memcpy (buf, cmdline, len); - buf[len] ='\0' ; - -if (strcmp (buf,"--" ) ==0 ) +for (len =0 ; cmdline[len] && !isspace (cmdline[len]); len++) { +if (len >=sizeof (buf) -1 ) +break ; +if (cmdline[len] =='-' ) + buf[len] ='_' ; +else + buf[len] = cmdline[len]; + } +if (!len)return ; + buf[len] =0 ; + cmdline += len;match_options (buf);for (i =0 ; parse_aliases && i <ARRAY_SIZE (aliases); i++) -if (parameq (buf, aliases[i].alias)) +if (!memcmp (buf, aliases[i].alias, len +1 ))__parse_cmdline (aliases[i].feature, false); }while (1 ); }
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 20/41 arm64/kernel: Move 'nokaslr' parsing out of early idreg code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Parsing and ignoring 'nokaslr' can be done from anywhere, except from the code that runs very early and is therefore built with limitations on the kind of relocations it is permitted to use.
So move it to a source file that is part of the ordinary kernel build.
--- arch/arm64/kernel/idreg-override.c | 7 ------- arch/arm64/kernel/kaslr.c | 7 +++++++ 2 files changed, 7 insertions(+), 7 deletions(-)
arch/arm64/kernel/idreg-override.c +0 -7
@@ -199,13 +199,6 @@ static const struct { { "nokaslr", "arm64_sw.nokaslr=1" }, }; -static int __init parse_nokaslr (char *unused) -{ -/* nokaslr param handling is done by early cpufeature code */ -return 0 ; -} -early_param ("nokaslr" , parse_nokaslr); -static int __init parse_hexdigit (const char *p,u64 *v) {// skip "0x" if it comes next
arch/arm64/kernel/kaslr.c +7 -0
@@ -36,3 +36,10 @@ void __init kaslr_init(void)pr_info ("KASLR enabled\n" ); __kaslr_is_enabled = true; } + +static int __init parse_nokaslr (char *unused) +{ +/* nokaslr param handling is done by early cpufeature code */ +return 0 ; +} +early_param ("nokaslr" , parse_nokaslr);
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 19/41 arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
All ID register value overrides are =0 with the exception of the nokaslr pseudo feature which uses =1. In order to remove the dependency on kstrtou64(), which is part of the core kernel and no longer usable once we move idreg-override into the early mini C runtime, let's just parse a single hex digit (with optional leading 0x) and set the output value accordingly.
--- arch/arm64/kernel/idreg-override.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
arch/arm64/kernel/idreg-override.c +15 -1
@@ -206,6 +206,20 @@ static int __init parse_nokaslr(char *unused) }early_param ("nokaslr" , parse_nokaslr); +static int __init parse_hexdigit (const char *p,u64 *v) +{ +// skip "0x" if it comes next +if (p[0 ] =='0' &&tolower (p[1 ]) =='x' ) + p +=2 ; + +// check whether the RHS is a single hex digit +if (!isxdigit (p[0 ]) || (p[1 ] && !isspace (p[1 ]))) +return -EINVAL ; + + *v =tolower (*p) - (isdigit (*p) ?'0' :'a' -10 ); +return 0 ; +} +static int __init find_field (const char *cmdline,char *opt,int len,const struct ftr_set_desc *reg,int f,u64 *v) {
@@ -219,7 +233,7 @@ static int __init find_field(const char *cmdline, char *opt, int len,if (memcmp (cmdline, opt, len))return -1 ; -return kstrtou64 (cmdline + len,0 , v); +return parse_hexdigit (cmdline + len, v); }static void __init match_options (const char *cmdline)
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 26/41 arm64: head: move dynamic shadow call stack patching into early C runtime
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Once we update the early kernel mapping code to only map the kernel once with the right permissions, we can no longer perform code patching via this mapping.
So move this code to an earlier stage of the boot, right after applying the relocations.
---
arch/arm64/include/asm/scs.h | 4 +--
arch/arm64/kernel/Makefile | 2 --
arch/arm64/kernel/head.S | 8 +++---
arch/arm64/kernel/module.c | 2 +-
arch/arm64/kernel/pi/Makefile | 10 +++++---
arch/arm64/kernel/{ => pi}/patch-scs.c | 26 ++++++++++----------
6 files changed, 27 insertions(+), 25 deletions(-)arch/arm64/include/asm/scs.h +2 -2
@@ -72,8 +72,8 @@ static inline void dynamic_scs_init(void)static inline void dynamic_scs_init (void ) {}#endif -int scs_patch (const u8 eh_frame[],int size); -asmlinkage void scs_patch_vmlinux(void); +int __pi_scs_patch (const u8 eh_frame[],int size); +asmlinkage void __pi_scs_patch_vmlinux(void);#endif /* __ASSEMBLY __ */
arch/arm64/kernel/Makefile +0 -2
@@ -71,8 +71,6 @@ obj-$(CONFIG_ARM64_PTR_AUTH) += pointer_auth.o obj-$(CONFIG_ARM64_MTE) += mte.o obj-y += vdso-wrap.o obj-$(CONFIG_COMPAT_VDSO) += vdso32-wrap.o -obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.o -CFLAGS_patch-scs.o += -mbranch-protection=none # Force dependency (vdso*-wrap.S includes vdso.so through incbin) $(obj)/vdso-wrap.o: $(obj)/vdso/vdso.so
arch/arm64/kernel/head.S +5 -3
@@ -490,9 +490,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init -#endif -#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS - bl scs_patch_vmlinux #endif mov x0, x20 bl finalise_el2 // Prefer VHE if possible
@@ -794,6 +791,11 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RELOCATABLE mov x0, x23 bl __pi_relocate_kernel +#endif +#ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS + ldr x0, =__eh_frame_start + ldr x1, =__eh_frame_end + bl __pi_scs_patch_vmlinux #endif ldr x8, =__primary_switched adrp x0, KERNEL_START // __pa(KERNEL_START)
arch/arm64/kernel/module.c +1 -1
@@ -595,7 +595,7 @@ int module_finalize(const Elf_Ehdr *hdr,if (scs_is_dynamic ()) { s =find_section (hdr, sechdrs,".init.eh_frame" );if (s) -scs_patch ((void *)s->sh_addr, s->sh_size); +__pi_scs_patch ((void *)s->sh_addr, s->sh_size); }return module_init_ftrace_plt (hdr, sechdrs, me);
arch/arm64/kernel/pi/Makefile +6 -4
@@ -38,7 +38,9 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -obj-$(CONFIG_RELOCATABLE) += relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := idreg-override.pi.o \ + lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o +obj-$(CONFIG_UNWIND_PATCH_PAC_INTO_SCS) += patch-scs.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y))
arch/arm64/kernel/pi/patch-scs.c renamed from arch/arm64/kernel/patch-scs.c +13 -13
@@ -4,14 +4,11 @@* Author: Ard Biesheuvel <[email protected]> */ -#include <linux/bug.h> #include <linux/errno.h> #include <linux/init.h> #include <linux/linkage.h> -#include <linux/printk.h> #include <linux/types.h> -#include <asm/cacheflush.h> #include <asm/scs.h> //
@@ -81,7 +78,11 @@ static void __always_inline scs_patch_loc(u64 loc)*/ return ; } -dcache_clean_pou (loc, loc +sizeof (u32)); +if (IS_ENABLED (CONFIG_ARM64_WORKAROUND_CLEAN_CACHE )) + asm("dc civac, %0" ::"r" (loc)); +else + asm(ALTERNATIVE("dc cvau, %0", "nop", ARM64_HAS_CACHE_IDC) + :: "r"(loc)); }/*
@@ -128,10 +129,10 @@ struct eh_frame { }; }; -static int noinstr scs_handle_fde_frame(const struct eh_frame *frame, -bool fde_has_augmentation_data, -int code_alignment_factor, -bool dry_run) +static int scs_handle_fde_frame (const struct eh_frame *frame, +bool fde_has_augmentation_data, +int code_alignment_factor, +bool dry_run) {int size = frame->size - offsetof(struct eh_frame , opcodes) +4 ;u64 loc = (u64 )offset_to_ptr (&frame->initial_loc);
@@ -198,14 +199,13 @@ static int noinstr scs_handle_fde_frame(const struct eh_frame *frame,break ;default : -pr_err ("unhandled opcode: %02x in FDE frame %lx\n" , opcode[-1 ], (uintptr_t )frame);return -ENOEXEC ; } }return 0 ; } -int noinstr scs_patch(const u8 eh_frame[], int size) +int scs_patch (const u8 eh_frame[],int size) {const u8 *p = eh_frame;
@@ -251,12 +251,12 @@ int noinstr scs_patch(const u8 eh_frame[], int size)return 0 ; } -asmlinkage void __init scs_patch_vmlinux(void) +asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) {if (!should_patch_pac_into_scs ())return ; -WARN_ON (scs_patch (__eh_frame_start, __eh_frame_end - __eh_frame_start)); -icache_inval_all_pou (); +scs_patch (start, end - start); + asm("ic ialluis" );isb (); }
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 33/41 arm64: mm: Make kaslr_requires_kpti() a static inline
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
In preparation for moving the first assignment of arm64_use_ng_mappings to an earlier stage in the boot, ensure that kaslr_requires_kpti() is accessible without relying on the core kernel's view on whether or not KASLR is enabled. So make it a static inline, and move the kaslr_enabled() check out of it and into the callers, one of which will disappear in a subsequent patch.
Once/when support for the obsolete ThunderX 1 platform is dropped, this check reduces to a E0PD feature check on the local CPU.
--- arch/arm64/include/asm/mmu.h | 38 +++++++++++++++++- arch/arm64/kernel/cpufeature.c | 42 +------------------- arch/arm64/kernel/setup.c | 2 +- 3 files changed, 39 insertions(+), 43 deletions(-)
arch/arm64/include/asm/mmu.h +37 -1
@@ -71,7 +71,43 @@ extern void create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, pgprot_t prot, bool page_mappings_only);extern void *fixmap_remap_fdt (phys_addr_t dt_phys,int *size,pgprot_t prot);extern void mark_linear_text_alias_ro (void ); -extern bool kaslr_requires_kpti (void ); + +/* +* This check is triggered during the early boot before the cpufeature +* is initialised. Checking the status on the local CPU allows the boot +* CPU to detect the need for non-global mappings and thus avoiding a +* pagetable re-write after all the CPUs are booted. This check will be +* anyway run on individual CPUs, allowing us to get the consistent +* state once the SMP CPUs are up and thus make the switch to non-global +* mappings if required. +*/ +static inline bool kaslr_requires_kpti (void ) +{ +/* +* E0PD does a similar job to KPTI so can be used instead +* where available. +*/ +if (IS_ENABLED (CONFIG_ARM64_E0PD )) { +u64 mmfr2 =read_sysreg_s (SYS_ID_AA64MMFR2_EL1 ); +if (cpuid_feature_extract_unsigned_field (mmfr2, +ID_AA64MMFR2_EL1_E0PD_SHIFT )) +return false; + } + +/* +* Systems affected by Cavium erratum 24756 are incompatible +* with KPTI. +*/ +if (IS_ENABLED (CONFIG_CAVIUM_ERRATUM_27456 )) { +extern const struct midr_range cavium_erratum_27456_cpus[]; + +if (is_midr_in_range_list (read_cpuid_id (), + cavium_erratum_27456_cpus)) +return false; + } + +return true; +}#define INIT_MM_CONTEXT (name) \ .pgd = init_pg_dir,
arch/arm64/kernel/cpufeature.c +1 -41
54 lines not on this page · raw message
arch/arm64/kernel/setup.c +1 -1
@@ -288,7 +288,7 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)* mappings from the start, avoiding the cost of rewriting * everything later. */ - arm64_use_ng_mappings =kaslr_requires_kpti (); + arm64_use_ng_mappings =kaslr_enabled () &&kaslr_requires_kpti ();early_fixmap_init ();early_ioremap_init ();
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 32/41 arm64: head: move memstart_offset_seed handling to C code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that we can set BSS variables from the early code running from the ID map, we can set memstart_offset_seed directly from the C code that derives the value instead of passing it back and forth between C and asm code.
--- arch/arm64/kernel/head.S | 7 ------- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/kaslr_early.c | 4 ++++ 3 files changed, 5 insertions(+), 7 deletions(-)
arch/arm64/kernel/head.S +0 -7
@@ -82,7 +82,6 @@ * x21 primary_entry() .. start_kernel() FDT pointer passed at boot in x0 * x22 create_idmap() .. start_kernel() ID map VA of the DT blob * x23 __primary_switch() physical misalignment/KASLR offset - * x24 __primary_switch() linear map KASLR seed * x25 primary_entry() .. start_kernel() supported VA size * x28 create_idmap() callee preserved temp register */
@@ -483,11 +482,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) str x25, [x8] // ... observes the correct value dc civac, x8 // Make visible to booting secondaries #endif - -#ifdef CONFIG_RANDOMIZE_BASE - adrp x5, memstart_offset_seed // Save KASLR linear map seed - strh w24, [x5, :lo12:memstart_offset_seed] -#endif #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif
@@ -779,7 +773,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init - and x24, x0, #SZ_2M - 1 // capture memstart offset seed bic x0, x0, #SZ_2M - 1 orr x23, x23, x0 // record kernel offset #endif
arch/arm64/kernel/image-vars.h +1 -0
@@ -46,6 +46,7 @@ PROVIDE(__pi_id_aa64smfr0_override = id_aa64smfr0_override);PROVIDE (__pi_id_aa64zfr0_override = id_aa64zfr0_override);PROVIDE (__pi_arm64_sw_feature_override = arm64_sw_feature_override);PROVIDE (__pi__ctype = _ctype); +PROVIDE (__pi_memstart_offset_seed = memstart_offset_seed);#ifdef CONFIG_KVM
arch/arm64/kernel/pi/kaslr_early.c +4 -0
@@ -16,6 +16,8 @@#include <asm/memory.h> #include <asm/pgtable.h> +extern u16 memstart_offset_seed; +static u64 __init get_kaslr_seed (void *fdt) {static char const chosen_str[]__initconst ="chosen" ;
@@ -51,6 +53,8 @@ asmlinkage u64 __init kaslr_early_init(void *fdt)return 0 ; } + memstart_offset_seed = seed &U16_MAX ; +/* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 27/41 arm64: cpufeature: Add helper to test for CPU feature overrides
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Add some helpers to extract and apply feature overrides to the bare idreg values. This involves inspecting the value and mask of the specific field that we are interested in, given that an override value/mask pair might be invalid for one field but valid for another.
Then, wire up the new helper for the hVHE test - note that we can drop the sysreg test here, as the override will be invalid when trying to enable hVHE on non-VHE capable hardware.
--- arch/arm64/include/asm/cpufeature.h | 39 ++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 9 +---- 2 files changed, 40 insertions(+), 8 deletions(-)
arch/arm64/include/asm/cpufeature.h +39 -0
@@ -909,6 +909,45 @@ extern struct arm64_ftr_override id_aa64isar2_override;extern struct arm64_ftr_override arm64_sw_feature_override; +static inline +u64 arm64_apply_feature_override (u64 val,int feat,int width, +const struct arm64_ftr_override *override) +{ +u64 oval = override->val; + +/* +* When it encounters an invalid override (e.g., an override that +* cannot be honoured due to a missing CPU feature), the early idreg +* override code will set the mask to 0x0 and the value to non-zero for +* the field in question. In order to determine whether the override is +* valid or not for the field we are interested in, we first need to +* disregard bits belonging to other fields. +*/ + oval &=GENMASK_ULL (feat + width -1 , feat); + +/* +* The override is valid if all value bits are accounted for in the +* mask. If so, replace the masked bits with the override value. +*/ +if (oval == (oval & override->mask)) { + val &= ~override->mask; + val |= oval; + } + +/* Extract the field from the updated value */ +return cpuid_feature_extract_unsigned_field (val, feat); +} + +static inline bool arm64_test_sw_feature_override (int feat) +{ +/* +* Software features are pseudo CPU features that have no underlying +* CPUID system register value to apply the override to. +*/ +return arm64_apply_feature_override (0 , feat,4 , + &arm64_sw_feature_override); +} +u32 get_kvm_ipa_limit (void );void dump_cpu_features (void );
arch/arm64/kernel/cpufeature.c +1 -8
@@ -2034,14 +2034,7 @@ static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap,static bool hvhe_possible (const struct arm64_cpu_capabilities *entry,int __unused) { -u64 val; - - val =read_sysreg (id_aa64mmfr1_el1); -if (!cpuid_feature_extract_unsigned_field (val,ID_AA64MMFR1_EL1_VH_SHIFT )) -return false; - - val = arm64_sw_feature_override.val & arm64_sw_feature_override.mask; -return cpuid_feature_extract_unsigned_field (val,ARM64_SW_FEATURE_OVERRIDE_HVHE ); +return arm64_test_sw_feature_override (ARM64_SW_FEATURE_OVERRIDE_HVHE ); }#ifdef CONFIG_ARM64_PAN
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 25/41 arm64: head: Run feature override detection before mapping the kernel
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
To permit the feature overrides to be taken into account before the KASLR init code runs and the kernel mapping is created, move the detection code to an earlier stage in the boot.
In a subsequent patch, this will be taken advantage of by merging the preliminary and permanent mappings of the kernel text and data into a single one that gets created and relocated before start_kernel() is called.
--- arch/arm64/kernel/head.S | 17 +++++++++-------- arch/arm64/kernel/vmlinux.lds.S | 4 +--- 2 files changed, 10 insertions(+), 11 deletions(-)
arch/arm64/kernel/head.S +9 -8
@@ -375,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap BSS and the kernel page tables r/w in the ID map */ + /* Remap [.init].data, BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, __bss_start + adrp x2, __initdata_begin adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS
@@ -491,9 +491,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x20 // pass the full boot status - mov x1, x22 // pass the low FDT mapping - bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif
@@ -772,12 +769,16 @@ SYM_FUNC_START_LOCAL(__primary_switch) bl __pi_memset dsb ishst // Make zero page visible to PTW -#ifdef CONFIG_RELOCATABLE - adrp x23, KERNEL_START - and x23, x23, MIN_KIMG_ALIGN - 1 adrp x1, early_init_stack mov sp, x1 mov x29, xzr + mov x0, x20 // pass the full boot status + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides + +#ifdef CONFIG_RELOCATABLE + adrp x23, KERNEL_START + and x23, x23, MIN_KIMG_ALIGN - 1 #ifdef CONFIG_RANDOMIZE_BASE mov x0, x22 bl __pi_kaslr_early_init
arch/arm64/kernel/vmlinux.lds.S +1 -3
@@ -320,10 +320,8 @@ SECTIONS init_pg_end = .; /* end of zero-init region */ -#ifdef CONFIG_RELOCATABLE - . += SZ_4K; /* stack for the early relocation code */ + . += SZ_4K; /* stack for the early C runtime */ early_init_stack = .; -#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin);
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 34/41 arm64: mmu: Make __cpu_replace_ttbr1() out of line
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
__cpu_replace_ttbr1() is a static inline, and so it gets instantiated wherever it is used. This is not really necessary, as it is never called on a hot path. It also has the unfortunate side effect that the symbol idmap_cpu_replace_ttbr1 may never be referenced from kCFI enabled C code, and this means the type id symbol may not exist either. This will result in a build error once we start referring to this symbol from asm code as well. (Note that this problem only occurs when CnP, KAsan and suspend/resume are all disabled in the Kconfig but that is a valid config, if unusual).
So let's just move it out of line so all callers will share the same implementation, which will reference idmap_cpu_replace_ttbr1 unconditionally.
--- arch/arm64/include/asm/mmu_context.h | 32 +------------------- arch/arm64/mm/mmu.c | 32 ++++++++++++++++++++ 2 files changed, 33 insertions(+), 31 deletions(-)
arch/arm64/include/asm/mmu_context.h +1 -31
@@ -148,37 +148,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz)isb (); } -/* -* Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, -* avoiding the possibility of conflicting TLB entries being allocated. -*/ -static inline void __cpu_replace_ttbr1 (pgd_t *pgdp,pgd_t *idmap,bool cnp) -{ -typedef void (ttbr_replace_func )(phys_addr_t ); -extern ttbr_replace_func idmap_cpu_replace_ttbr1; -ttbr_replace_func *replace_phys; -unsigned long daif; - -/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ -phys_addr_t ttbr1 =phys_to_ttbr (virt_to_phys (pgdp)); - -if (cnp) - ttbr1 |=TTBR_CNP_BIT ; - - replace_phys = (void *)__pa_symbol (idmap_cpu_replace_ttbr1); - -__cpu_install_idmap (idmap); - -/* -* We really don't want to take *any* exceptions while TTBR1 is -* in the process of being replaced so mask everything. -*/ - daif =local_daif_save (); -replace_phys (ttbr1); -local_daif_restore (daif); - -cpu_uninstall_idmap (); -} +void __cpu_replace_ttbr1 (pgd_t *pgdp,pgd_t *idmap,bool cnp);static inline void cpu_enable_swapper_cnp (void ) {
arch/arm64/mm/mmu.c +32 -0
@@ -1483,3 +1483,35 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte {set_pte_at (vma->vm_mm, addr, ptep, pte); } + +/* +* Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, +* avoiding the possibility of conflicting TLB entries being allocated. +*/ +void __cpu_replace_ttbr1 (pgd_t *pgdp,pgd_t *idmap,bool cnp) +{ +typedef void (ttbr_replace_func )(phys_addr_t ); +extern ttbr_replace_func idmap_cpu_replace_ttbr1; +ttbr_replace_func *replace_phys; +unsigned long daif; + +/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */ +phys_addr_t ttbr1 =phys_to_ttbr (virt_to_phys (pgdp)); + +if (cnp) + ttbr1 |=TTBR_CNP_BIT ; + + replace_phys = (void *)__pa_symbol (idmap_cpu_replace_ttbr1); + +__cpu_install_idmap (idmap); + +/* +* We really don't want to take *any* exceptions while TTBR1 is +* in the process of being replaced so mask everything. +*/ + daif =local_daif_save (); +replace_phys (ttbr1); +local_daif_restore (daif); + +cpu_uninstall_idmap (); +}
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 28/41 arm64: kaslr: Use feature override instead of parsing the cmdline again
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The early kaslr code open codes the detection of 'nokaslr' on the kernel command line, and this is no longer necessary now that the feature detection code, which also looks for the same string, executes before this code.
--- arch/arm64/include/asm/cpufeature.h | 5 ++ arch/arm64/kernel/kaslr.c | 4 +- arch/arm64/kernel/pi/kaslr_early.c | 53 +------------------- 3 files changed, 7 insertions(+), 55 deletions(-)
arch/arm64/include/asm/cpufeature.h +5 -0
@@ -948,6 +948,11 @@ static inline bool arm64_test_sw_feature_override(int feat) &arm64_sw_feature_override); } +static inline bool kaslr_disabled_cmdline (void ) +{ +return arm64_test_sw_feature_override (ARM64_SW_FEATURE_OVERRIDE_NOKASLR ); +} +u32 get_kvm_ipa_limit (void );void dump_cpu_features (void );
arch/arm64/kernel/kaslr.c +1 -3
@@ -16,9 +16,7 @@ bool __ro_after_init __kaslr_is_enabled = false;void __init kaslr_init (void ) { -if (cpuid_feature_extract_unsigned_field (arm64_sw_feature_override.val & - arm64_sw_feature_override.mask, -ARM64_SW_FEATURE_OVERRIDE_NOKASLR )) { +if (kaslr_disabled_cmdline ()) {pr_info ("KASLR disabled on command line\n" );return ; }
arch/arm64/kernel/pi/kaslr_early.c +1 -52
65 lines not on this page · raw message
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 39/41 arm64: mm: avoid fixmap for early swapper_pg_dir updates
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Early in the boot, when .rodata is still writable, we can poke swapper_pg_dir entries directly, and there is no need to go through the fixmap. After a future patch, we will enter the kernel with swapper_pg_dir already active, and early swapper_pg_dir updates for creating the fixmap page table hierarchy itself cannot go through the fixmap for obvious reaons. So let's keep track of whether rodata is writable, and update the descriptor directly in that case.
As the same reasoning applies to early KASAN init, make the function noinstr as well.
--- arch/arm64/mm/mmu.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-)
arch/arm64/mm/mmu.c +15 -1
@@ -55,6 +55,8 @@ EXPORT_SYMBOL(kimage_voffset);u32 __boot_cpu_mode[] = {BOOT_CPU_MODE_EL2 ,BOOT_CPU_MODE_EL1 }; +static bool rodata_is_rw__ro_after_init = true; +/* * The booting CPU updates the failed status @__early_cpu_boot_status, * with MMU turned off.
@@ -71,10 +73,21 @@ EXPORT_SYMBOL(empty_zero_page);static DEFINE_SPINLOCK (swapper_pgdir_lock);static DEFINE_MUTEX (fixmap_lock); -void set_swapper_pgd (pgd_t *pgdp,pgd_t pgd) +void noinstr set_swapper_pgd(pgd_t *pgdp, pgd_t pgd) {pgd_t *fixmap_pgdp; +/* +* Don't bother with the fixmap if swapper_pg_dir is still mapped +* writable in the kernel mapping. +*/ +if (rodata_is_rw) { +WRITE_ONCE (*pgdp, pgd); +dsb (ishst); +isb (); +return ; + } +spin_lock (&swapper_pgdir_lock); fixmap_pgdp =pgd_set_fixmap (__pa_symbol (pgdp));WRITE_ONCE (*fixmap_pgdp, pgd);
@@ -628,6 +641,7 @@ void mark_rodata_ro(void)* to cover NOTES and EXCEPTION_TABLE. */ section_size = (unsigned long )__init_begin - (unsigned long )__start_rodata; +WRITE_ONCE (rodata_is_rw, false);update_mapping_prot (__pa_symbol (__start_rodata), (unsigned long )__start_rodata, section_size,PAGE_KERNEL_RO );
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 41/41 arm64: Revert "mm: provide idmap pointer to cpu_replace_ttbr1()"
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
This reverts commit 1682c45b920643c, which is no longer needed now that we create the permanent kernel mapping directly during early boot.
This is a RINO (revert in name only) given that some of the code has moved around, but the changes are straight-forward.
--- arch/arm64/include/asm/mmu_context.h | 17 ++++++----------- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 4 ++-- 3 files changed, 10 insertions(+), 15 deletions(-)
arch/arm64/include/asm/mmu_context.h +6 -11
@@ -108,18 +108,13 @@ static inline void cpu_uninstall_idmap(void)cpu_switch_mm (mm->pgd, mm); } -static inline void __cpu_install_idmap (pgd_t *idmap) +static inline void cpu_install_idmap (void ) {cpu_set_reserved_ttbr0 ();local_flush_tlb_all ();cpu_set_idmap_tcr_t0sz (); -cpu_switch_mm (lm_alias (idmap), &init_mm); -} - -static inline void cpu_install_idmap (void ) -{ -__cpu_install_idmap (idmap_pg_dir); +cpu_switch_mm (lm_alias (idmap_pg_dir), &init_mm); }/*
@@ -146,21 +141,21 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz)isb (); } -void __cpu_replace_ttbr1 (pgd_t *pgdp,pgd_t *idmap,bool cnp); +void __cpu_replace_ttbr1 (pgd_t *pgdp,bool cnp);static inline void cpu_enable_swapper_cnp (void ) { -__cpu_replace_ttbr1 (lm_alias (swapper_pg_dir), idmap_pg_dir, true); +__cpu_replace_ttbr1 (lm_alias (swapper_pg_dir), true); } -static inline void cpu_replace_ttbr1 (pgd_t *pgdp,pgd_t *idmap) +static inline void cpu_replace_ttbr1 (pgd_t *pgdp) {/* * Only for early TTBR1 replacement before cpucaps are finalized and * before we've decided whether to use CNP. */ WARN_ON (system_capabilities_finalized ()); -__cpu_replace_ttbr1 (pgdp, idmap, false); +__cpu_replace_ttbr1 (pgdp, false); }/*
arch/arm64/mm/kasan_init.c +2 -2
@@ -220,7 +220,7 @@ static void __init kasan_init_shadow(void)*/ memcpy (tmp_pg_dir, swapper_pg_dir,sizeof (tmp_pg_dir));dsb (ishst); -cpu_replace_ttbr1 (lm_alias (tmp_pg_dir), idmap_pg_dir); +cpu_replace_ttbr1 (lm_alias (tmp_pg_dir));clear_pgds (KASAN_SHADOW_START ,KASAN_SHADOW_END );
@@ -256,7 +256,7 @@ static void __init kasan_init_shadow(void) PAGE_KERNEL_RO));memset (kasan_early_shadow_page,KASAN_SHADOW_INIT ,PAGE_SIZE ); -cpu_replace_ttbr1 (lm_alias (swapper_pg_dir), idmap_pg_dir); +cpu_replace_ttbr1 (lm_alias (swapper_pg_dir)); }static void __init kasan_init_depth (void )
arch/arm64/mm/mmu.c +2 -2
@@ -1442,7 +1442,7 @@ void ptep_modify_prot_commit(struct vm_area_struct *vma, unsigned long addr, pte* Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD, * avoiding the possibility of conflicting TLB entries being allocated. */ -void __cpu_replace_ttbr1 (pgd_t *pgdp,pgd_t *idmap,bool cnp) +void __cpu_replace_ttbr1 (pgd_t *pgdp,bool cnp) {typedef void (ttbr_replace_func )(phys_addr_t );extern ttbr_replace_func idmap_cpu_replace_ttbr1;
@@ -1457,7 +1457,7 @@ void __cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap, bool cnp) replace_phys = (void *)__pa_symbol (idmap_cpu_replace_ttbr1); -__cpu_install_idmap (idmap); +cpu_install_idmap ();/* * We really don't want to take *any* exceptions while TTBR1 is
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 36/41 arm64: mm: Use 48-bit virtual addressing for the permanent ID map
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Even though we support loading kernels anywhere in 48-bit addressable physical memory, we create the ID maps based on the number of levels that we happened to configure for the kernel VA and user VA spaces.
The reason for this is that the PGD/PUD/PMD based classification of translation levels, along with the associated folding when the number of levels is less than 5, does not permit creating a page table hierarchy of a set number of levels. This means that, for instance, on 39-bit VA kernels we need to configure an additional level above PGD level on the fly, and 36-bit VA kernels still only support 47-bit virtual addressing with this trick applied.
Now that we have a separate helper to populate page table hierarchies that does not define the levels in terms of PUDS/PMDS/etc at all, let's reuse it to create the permanent ID map with a fixed VA size of 48 bits.
--- arch/arm64/include/asm/kernel-pgtable.h | 3 ++ arch/arm64/kernel/head.S | 5 +++ arch/arm64/kvm/mmu.c | 15 +++------ arch/arm64/mm/mmu.c | 32 +++++++++++--------- arch/arm64/mm/proc.S | 9 ++---- 5 files changed, 32 insertions(+), 32 deletions(-)
arch/arm64/include/asm/kernel-pgtable.h +3 -0
@@ -35,6 +35,9 @@#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS)#endif +#define IDMAP_VA_BITS 48 +#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS) +#define IDMAP_ROOT_LEVEL (4 - IDMAP_LEVELS)/* * A relocatable kernel may execute from an address that differs from the one at
arch/arm64/kernel/head.S +5 -0
@@ -729,6 +729,11 @@ SYM_FUNC_START_LOCAL(__no_granule_support) SYM_FUNC_END(__no_granule_support) SYM_FUNC_START_LOCAL(__primary_switch) + mrs x1, tcr_el1 + mov x2, #64 - VA_BITS + tcr_set_t0sz x1, x2 + msr tcr_el1, x1 + adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu
arch/arm64/kvm/mmu.c +4 -11
@@ -1873,16 +1873,9 @@ int __init kvm_mmu_init(u32 *hyp_va_bits)BUG_ON ((hyp_idmap_start ^ (hyp_idmap_end -1 )) &PAGE_MASK );/* -* The ID map may be configured to use an extended virtual address -* range. This is only the case if system RAM is out of range for the -* currently configured page size and VA_BITS_MIN, in which case we will -* also need the extended virtual range for the HYP ID map, or we won't -* be able to enable the EL2 MMU. -* -* However, in some cases the ID map may be configured for fewer than -* the number of VA bits used by the regular kernel stage 1. This -* happens when VA_BITS=52 and the kernel image is placed in PA space -* below 48 bits. +* The ID map is always configured for 48 bits of translation, which +* may be fewer than the number of VA bits used by the regular kernel +* stage 1, when VA_BITS=52. * * At EL2, there is only one TTBR register, and we can't switch between * translation tables *and* update TCR_EL2.T0SZ at the same time. Bottom
@@ -1893,7 +1886,7 @@ int __init kvm_mmu_init(u32 *hyp_va_bits)* 1 VA bits to assure that the hypervisor can both ID map its code page * and map any kernel memory. */ - idmap_bits =64 - ((idmap_t0sz &TCR_T0SZ_MASK ) >>TCR_T0SZ_OFFSET ); + idmap_bits =IDMAP_VA_BITS ; kernel_bits = vabits_actual; *hyp_va_bits =max (idmap_bits, kernel_bits);
arch/arm64/mm/mmu.c +17 -15
@@ -754,22 +754,21 @@ static void __init map_kernel(pgd_t *pgdp)kasan_copy_shadow (pgdp); } +void __pi_map_range (u64 *pgd,u64 start,u64 end,u64 pa,pgprot_t prot, +int level,pte_t *tbl,bool may_use_cont,u64 va_offset); + +static u8 idmap_ptes[IDMAP_LEVELS -1 ][PAGE_SIZE ]__aligned(PAGE_SIZE) __ro_after_init , + kpti_ptes[IDMAP_LEVELS -1 ][PAGE_SIZE ]__aligned(PAGE_SIZE) __ro_after_init ; +static void __init create_idmap (void ) {u64 start =__pa_symbol (__idmap_text_start); -u64 size =__pa_symbol (__idmap_text_end) - start; -pgd_t *pgd = idmap_pg_dir; -u64 pgd_phys; - -/* check if we need an additional level of translation */ -if (VA_BITS <48 && idmap_t0sz < (64 -VA_BITS_MIN )) { - pgd_phys =early_pgtable_alloc (PAGE_SHIFT ); -set_pgd (&idmap_pg_dir[start >>VA_BITS ], -__pgd (pgd_phys |P4D_TYPE_TABLE )); - pgd =__va (pgd_phys); - } -__create_pgd_mapping (pgd, start, start, size,PAGE_KERNEL_ROX , - early_pgtable_alloc,0 ); +u64 end =__pa_symbol (__idmap_text_end); +u64 ptep =__pa_symbol (idmap_ptes); + +__pi_map_range (&ptep, start, end, start,PAGE_KERNEL_ROX , +IDMAP_ROOT_LEVEL , (pte_t *)idmap_pg_dir, false, +__phys_to_virt (ptep) - ptep);if (IS_ENABLED (CONFIG_UNMAP_KERNEL_AT_EL0 )) {extern u32 __idmap_kpti_flag;
@@ -779,8 +778,10 @@ static void __init create_idmap(void)* The KPTI G-to-nG conversion code needs a read-write mapping * of its synchronization flag in the ID map. */ -__create_pgd_mapping (pgd, pa, pa,sizeof (u32),PAGE_KERNEL , - early_pgtable_alloc,0 ); + ptep =__pa_symbol (kpti_ptes); +__pi_map_range (&ptep, pa, pa +sizeof (u32), pa,PAGE_KERNEL , +IDMAP_ROOT_LEVEL , (pte_t *)idmap_pg_dir, false, +__phys_to_virt (ptep) - ptep); } }
@@ -805,6 +806,7 @@ void __init paging_init(void)memblock_allow_resize ();create_idmap (); + idmap_t0sz =TCR_T0SZ (IDMAP_VA_BITS ); }#ifdef CONFIG_MEMORY_HOTPLUG
arch/arm64/mm/proc.S +3 -6
@@ -421,9 +421,9 @@ SYM_FUNC_START(__cpu_setup) mair .req x17 tcr .req x16 mov_q mair, MAIR_EL1_SET - mov_q tcr, TCR_TxSZ(VA_BITS) | TCR_CACHE_FLAGS | TCR_SMP_FLAGS | \ - TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ - TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS + mov_q tcr, TCR_T0SZ(IDMAP_VA_BITS) | TCR_T1SZ(VA_BITS) | TCR_CACHE_FLAGS | \ + TCR_SMP_FLAGS | TCR_TG_FLAGS | TCR_KASLR_FLAGS | TCR_ASID16 | \ + TCR_TBI0 | TCR_A1 | TCR_KASAN_SW_FLAGS | TCR_MTE_FLAGS tcr_clear_errata_bits tcr, x9, x5
@@ -431,10 +431,7 @@ SYM_FUNC_START(__cpu_setup) sub x9, xzr, x0 add x9, x9, #64 tcr_set_t1sz tcr, x9 -#else - idmap_get_t0sz x9 #endif - tcr_set_t0sz tcr, x9 /* * Set the IPS bits in TCR_EL1.
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 40/41 arm64: mm: omit redundant remap of kernel image
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that the early kernel mapping is created with all the right attributes and segment boundaries, there is no longer a need to recreate it and switch to it. This also means we no longer have to copy the kasan shadow or some parts of the fixmap from one set of page tables to the other.
--- arch/arm64/include/asm/fixmap.h | 1 - arch/arm64/include/asm/kasan.h | 2 - arch/arm64/include/asm/mmu.h | 2 +- arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/map_kernel.c | 6 +- arch/arm64/mm/fixmap.c | 34 -------- arch/arm64/mm/kasan_init.c | 15 ---- arch/arm64/mm/mmu.c | 85 ++++---------------- 8 files changed, 21 insertions(+), 125 deletions(-)
arch/arm64/include/asm/fixmap.h +0 -1
@@ -100,7 +100,6 @@ enum fixed_addresses {#define FIXMAP_PAGE_IO __pgprot(PROT_DEVICE_nGnRE)void __init early_fixmap_init (void ); -void __init fixmap_copy (pgd_t *pgdir);#define __early_set_fixmap __set_fixmap
arch/arm64/include/asm/kasan.h +0 -2
@@ -36,12 +36,10 @@ void kasan_init(void);#define _KASAN_SHADOW_START (va) (KASAN_SHADOW_END - (1UL << ((va) - KASAN_SHADOW_SCALE_SHIFT)))#define KASAN_SHADOW_START _KASAN_SHADOW_START(vabits_actual) -void kasan_copy_shadow (pgd_t *pgdir); asmlinkage void kasan_early_init(void);#else static inline void kasan_init (void ) { } -static inline void kasan_copy_shadow (pgd_t *pgdir) { }#endif #endif
arch/arm64/include/asm/mmu.h +1 -1
@@ -110,7 +110,7 @@ static inline bool kaslr_requires_kpti(void) }#define INIT_MM_CONTEXT (name) \ - .pgd = init_pg_dir, + .pgd = swapper_pg_dir,#endif /* !__ASSEMBLY__ */ #endif
arch/arm64/kernel/image-vars.h +1 -0
@@ -55,6 +55,7 @@ PROVIDE(__pi_memstart_offset_seed = memstart_offset_seed);PROVIDE (__pi_init_idmap_pg_dir = init_idmap_pg_dir);PROVIDE (__pi_init_pg_dir = init_pg_dir);PROVIDE (__pi_init_pg_end = init_pg_end); +PROVIDE (__pi_swapper_pg_dir = swapper_pg_dir);PROVIDE (__pi__text = _text);PROVIDE (__pi__stext = _stext);
arch/arm64/kernel/pi/map_kernel.c +5 -1
@@ -124,8 +124,12 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) text_prot, true, root_level);map_segment (init_pg_dir,NULL , va_offset, __inittext_begin, __inittext_end, text_prot, false, root_level); -dsb (ishst); } + +/* Copy the root page table to its final location */ +memcpy ((void *)swapper_pg_dir + va_offset, init_pg_dir,PGD_SIZE ); +dsb (ishst); +idmap_cpu_replace_ttbr1 (swapper_pg_dir); }static void __init map_fdt (u64 fdt)
arch/arm64/mm/fixmap.c +0 -34
@@ -167,37 +167,3 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot)return dt_virt; } - -/* -* Copy the fixmap region into a new pgdir. -*/ -void __init fixmap_copy (pgd_t *pgdir) -{ -if (!READ_ONCE (pgd_val (*pgd_offset_pgd (pgdir,FIXADDR_TOT_START )))) { -/* -* The fixmap falls in a separate pgd to the kernel, and doesn't -* live in the carveout for the swapper_pg_dir. We can simply -* re-use the existing dir for the fixmap. -*/ -set_pgd (pgd_offset_pgd (pgdir,FIXADDR_TOT_START ), -READ_ONCE (*pgd_offset_k (FIXADDR_TOT_START ))); - }else if (CONFIG_PGTABLE_LEVELS >3 ) { -pgd_t *bm_pgdp; -p4d_t *bm_p4dp; -pud_t *bm_pudp; -/* -* The fixmap shares its top level pgd entry with the kernel -* mapping. This can really only occur when we are running -* with 16k/4 levels, so we can simply reuse the pud level -* entry instead. -*/ -BUG_ON (!IS_ENABLED (CONFIG_ARM64_16K_PAGES )); - bm_pgdp =pgd_offset_pgd (pgdir,FIXADDR_TOT_START ); - bm_p4dp =p4d_offset (bm_pgdp,FIXADDR_TOT_START ); - bm_pudp =pud_set_fixmap_offset (bm_p4dp,FIXADDR_TOT_START ); -pud_populate (&init_mm, bm_pudp,lm_alias (bm_pmd)); -pud_clear_fixmap (); - }else { -BUG (); - } -}
arch/arm64/mm/kasan_init.c +0 -15
@@ -184,21 +184,6 @@ static void __init kasan_map_populate(unsigned long start, unsigned long end,kasan_pgd_populate (start &PAGE_MASK ,PAGE_ALIGN (end), node, false); } -/* -* Copy the current shadow region into a new pgdir. -*/ -void __init kasan_copy_shadow (pgd_t *pgdir) -{ -pgd_t *pgdp, *pgdp_new, *pgdp_end; - - pgdp =pgd_offset_k (KASAN_SHADOW_START ); - pgdp_end =pgd_offset_k (KASAN_SHADOW_END ); - pgdp_new =pgd_offset_pgd (pgdir,KASAN_SHADOW_START ); -do { -set_pgd (pgdp_new,READ_ONCE (*pgdp)); - }while (pgdp++, pgdp_new++, pgdp != pgdp_end); -} -static void __init clear_pgds (unsigned long start,unsigned long end) {
arch/arm64/mm/mmu.c +14 -71
127 lines not on this page · raw message
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Marc Zyngier
maintainerRe: [PATCH v6 11/41] arm64: kernel: Manage absolute relocations in code built under pi/
In reply to Ard BiesheuvelQuoted 67 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables.
Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type
R_AARCH64_ABS64exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that haveSHF_ALLOCset andSHF_EXECINSTRcleared.To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 18 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 155 insertions(+), 2 deletions(-)arch/arm64/kernel/pi/Makefile +7
-2@@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c)arch/arm64/kernel/pi/pi.h new file +8
-0@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel <[email protected]> + +#define __prel64_initconst __section(".init.rodata.prel64") + +#define PREL64 (type, name) union { type *name; prel64_t name ## _prel; } +
arch/arm64/kernel/pi/pi.h line 18
Quoted 10 lines
arch/arm64/kernel/pi/pi.h new file +10
-0+#define prel64_pointer (__d) (typeof(__d))prel64_to_pointer(&__d##_prel) + +typedef volatile signed long prel64_t ; + +static inline void *prel64_to_pointer (const prel64_t *offset) +{ +if (!*offset) +return NULL ; +return (void *)offset + *offset; +}
Is there any use for these definitions before the override code gets moved to pi/ in patch 21? It is otherwise a bit odd to see two sets of definitions between patches 15 and 20.
Thanks,
M.
Signature · 6 lines
Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 8/41 arm64: vmemmap: Avoid base2 order of struct page size to dimension region
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The placement and size of the vmemmap region in the kernel virtual address space is currently derived from the base2 order of the size of a struct page. This makes for nicely aligned constants with lots of leading 0xf and trailing 0x0 digits, but given that the actual struct pages are indexed as an ordinary array, this resulting region is severely overdimensioned when the size of a struct page is just over a power of 2.
This doesn't matter today, but once we enable 52-bit virtual addressing for 4k pages configurations, the vmemmap region may take up almost half of the upper VA region with the current struct page upper bound at 64 bytes. And once we enable KMSAN or other features that push the size of a struct page over 64 bytes, we will run out of VMALLOC space entirely.
So instead, let's derive the region size from the actual size of a struct page, and place the entire region 1 GB from the top of the VA space, where it still doesn't share any lower level translation table entries with the fixmap.
--- arch/arm64/include/asm/memory.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
arch/arm64/include/asm/memory.h +4 -4
@@ -30,8 +30,8 @@* keep a constant PAGE_OFFSET and "fallback" to using the higher end * of the VMEMMAP where 52-bit support is not available in hardware. */ -#define VMEMMAP_SHIFT (PAGE_SHIFT - STRUCT_PAGE_MAX_SHIFT) -#define VMEMMAP_SIZE ((_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) >> VMEMMAP_SHIFT) +#define VMEMMAP_RANGE (_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) +#define VMEMMAP_SIZE ((VMEMMAP_RANGE >> PAGE_SHIFT) * sizeof(struct page))/* * PAGE_OFFSET - the virtual address of the start of the linear map, at the
@@ -47,8 +47,8 @@#define MODULES_END (MODULES_VADDR + MODULES_VSIZE)#define MODULES_VADDR (_PAGE_END(VA_BITS_MIN))#define MODULES_VSIZE (SZ_2G) -#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) -#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) +#define VMEMMAP_START (VMEMMAP_END - VMEMMAP_SIZE) +#define VMEMMAP_END (-UL(SZ_1G))#define PCI_IO_START (VMEMMAP_END + SZ_8M)#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (-UL(SZ_8M))
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 14/41 arm64: idreg-override: Omit non-NULL checks for override pointer
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that override pointers are always set, we can drop the various non-NULL checks that we have in the code.
--- arch/arm64/kernel/idreg-override.c | 16 +++++----------- 1 file changed, 5 insertions(+), 11 deletions(-)
arch/arm64/kernel/idreg-override.c +5 -11
@@ -216,9 +216,6 @@ static void __init match_options(const char *cmdline)for (i =0 ; i <ARRAY_SIZE (regs); i++) {int f; -if (!regs[i]->override) -continue ; -for (f =0 ;strlen (regs[i]->fields[f].name); f++) {u64 shift = regs[i]->fields[f].shift;u64 width = regs[i]->fields[f].width ?:4 ;
@@ -319,10 +316,8 @@ asmlinkage void __init init_feature_override(u64 boot_status)int i;for (i =0 ; i <ARRAY_SIZE (regs); i++) { -if (regs[i]->override) { - regs[i]->override->val =0 ; - regs[i]->override->mask =0 ; - } + regs[i]->override->val =0 ; + regs[i]->override->mask =0 ; } __boot_status = boot_status;
@@ -330,9 +325,8 @@ asmlinkage void __init init_feature_override(u64 boot_status)parse_cmdline ();for (i =0 ; i <ARRAY_SIZE (regs); i++) { -if (regs[i]->override) -dcache_clean_inval_poc ((unsigned long )regs[i]->override, - (unsigned long )regs[i]->override + -sizeof (*regs[i]->override)); +dcache_clean_inval_poc ((unsigned long )regs[i]->override, + (unsigned long )regs[i]->override + +sizeof (*regs[i]->override)); } }
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 9/41 arm64: mm: Reclaim unused vmemmap region for vmalloc use
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The vmemmap array is statically sized based on the maximum supported size of the virtual address space, but it is located inside the upper VA region, which is statically sized based on the *minimum* supported size of the VA space. This doesn't matter much when using 64k pages, which is the only configuration that currently supports 52-bit virtual addressing.
However, upcoming LPA2 support will change this picture somewhat, as in that case, the vmemmap array will take up more than 25% of the upper VA region when using 4k pages. Given that most of this space is never used when running on a system that does not support 52-bit virtual addressing, let's reclaim the unused vmemmap area in that case.
--- arch/arm64/include/asm/pgtable.h | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
arch/arm64/include/asm/pgtable.h +6 -2
@@ -18,11 +18,15 @@* VMALLOC range. * * VMALLOC_START: beginning of the kernel vmalloc space -* VMALLOC_END: extends to the available space below vmemmap, PCI I/O space -* and fixed mappings +* VMALLOC_END: extends to the available space below vmemmap */ #define VMALLOC_START (MODULES_END) +#if VA_BITS ==VA_BITS_MIN #define VMALLOC_END (VMEMMAP_START - SZ_8M) +#else +#define VMEMMAP_UNUSED_NPAGES ((_PAGE_OFFSET(vabits_actual) - PAGE_OFFSET) >> PAGE_SHIFT) +#define VMALLOC_END (VMEMMAP_START + VMEMMAP_UNUSED_NPAGES * sizeof(struct page) - SZ_8M) +#endif #define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT))
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 11/41 arm64: kernel: Manage absolute relocations in code built under pi/
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables.
Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type R_AARCH64_ABS64 exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that have SHF_ALLOC set and SHF_EXECINSTR cleared.
To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use.
--- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 18 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 155 insertions(+), 2 deletions(-)
arch/arm64/kernel/pi/Makefile +7 -2
@@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c)
arch/arm64/kernel/pi/pi.h new file +18 -0
@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel <[email protected]> + +#define __prel64_initconst __section(".init.rodata.prel64") + +#define PREL64 (type, name) union { type *name; prel64_t name ## _prel; } + +#define prel64_pointer (__d) (typeof(__d))prel64_to_pointer(&__d##_prel) + +typedef volatile signed long prel64_t ; + +static inline void *prel64_to_pointer (const prel64_t *offset) +{ +if (!*offset) +return NULL ; +return (void *)offset + *offset; +}
arch/arm64/kernel/pi/relacheck.c new file +130 -0
130 lines not on this page · raw message
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 12/41 arm64: kernel: Don't rely on objcopy to make code under pi/ __init
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
We will add some code under pi/ that contains global variables that should not end up in __initdata, as they will not be writable via the initial ID map. So only rely on objcopy for making the libfdt code __init, and use explicit annotations for the rest.
--- arch/arm64/kernel/pi/Makefile | 6 ++++-- arch/arm64/kernel/pi/kaslr_early.c | 16 +++++++++------- 2 files changed, 13 insertions(+), 9 deletions(-)
arch/arm64/kernel/pi/Makefile +4 -2
@@ -28,11 +28,13 @@ quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ - --remove-section=.note.gnu.property \ - --prefix-alloc-sections=.init + --remove-section=.note.gnu.property $(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE $(call if_changed,piobjcopy) +# ensure that all the lib- code ends up as __init code and data +$(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init + $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c)
arch/arm64/kernel/pi/kaslr_early.c +9 -7
@@ -17,7 +17,7 @@#include <asm/pgtable.h> /* taken from lib/string.c */ -static char *__strstr (const char *s1,const char *s2) +static char *__init __strstr (const char *s1,const char *s2) {size_t l1, l2;
@@ -33,7 +33,7 @@ static char *__strstr(const char *s1, const char *s2) }return NULL ; } -static bool cmdline_contains_nokaslr (const u8 *cmdline) +static bool __init cmdline_contains_nokaslr (const u8 *cmdline) {const u8 *str;
@@ -41,7 +41,7 @@ static bool cmdline_contains_nokaslr(const u8 *cmdline)return str == cmdline || (str > cmdline && *(str -1 ) ==' ' ); } -static bool is_kaslr_disabled_cmdline (void *fdt) +static bool __init is_kaslr_disabled_cmdline (void *fdt) {if (!IS_ENABLED (CONFIG_CMDLINE_FORCE )) {int node;
@@ -67,17 +67,19 @@ static bool is_kaslr_disabled_cmdline(void *fdt)return cmdline_contains_nokaslr (CONFIG_CMDLINE ); } -static u64 get_kaslr_seed (void *fdt) +static u64 __init get_kaslr_seed (void *fdt) { +static char const chosen_str[]__initconst ="chosen" ; +static char const seed_str[]__initconst ="kaslr-seed" ;int node, len;fdt64_t *prop;u64 ret; - node =fdt_path_offset (fdt,"/chosen" ); + node =fdt_path_offset (fdt, chosen_str);if (node <0 )return 0 ; - prop =fdt_getprop_w (fdt, node,"kaslr-seed" , &len); + prop =fdt_getprop_w (fdt, node, seed_str, &len);if (!prop || len !=sizeof (u64))return 0 ;
@@ -86,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt)return ret; } -asmlinkage u64 kaslr_early_init(void *fdt) +asmlinkage u64 __init kaslr_early_init(void *fdt) {u64 seed, range;
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 13/41 arm64: head: move relocation handling to C code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Now that we have a mini C runtime before the kernel mapping is up, we can move the non-trivial relocation processing code out of head.S and reimplement it in C.
--- arch/arm64/kernel/Makefile | 3 +- arch/arm64/kernel/head.S | 104 ++------------------ arch/arm64/kernel/pi/Makefile | 5 +- arch/arm64/kernel/pi/relocate.c | 62 ++++++++++++ arch/arm64/kernel/vmlinux.lds.S | 12 ++- 5 files changed, 82 insertions(+), 104 deletions(-)
arch/arm64/kernel/Makefile +2 -1
@@ -57,7 +57,8 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o pi/ +obj-$(CONFIG_RELOCATABLE) += pi/ +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o \
arch/arm64/kernel/head.S +7 -97
136 lines not on this page · raw message
arch/arm64/kernel/pi/Makefile +3 -2
@@ -38,5 +38,6 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o -extra-y := $(patsubst %.pi.o,%.o,$(obj-y)) +obj-y := relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +extra-y := $(patsubst %.pi.o,%.o,$(obj-y))
arch/arm64/kernel/pi/relocate.c new file +62 -0
62 lines not on this page · raw message
arch/arm64/kernel/vmlinux.lds.S +8 -4
@@ -270,15 +270,15 @@ SECTIONS HYPERVISOR_RELOC_SECTION .rela.dyn : ALIGN(8) { - __rela_start = .; + __pi_rela_start = .; *(.rela .rela*) - __rela_end = .; + __pi_rela_end = .; } .relr.dyn : ALIGN(8) { - __relr_start = .; + __pi_relr_start = .; *(.relr.dyn) - __relr_end = .; + __pi_relr_end = .; } . = ALIGN(SEGMENT_ALIGN);
@@ -317,6 +317,10 @@ SECTIONS init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; +#ifdef CONFIG_RELOCATABLE + . += SZ_4K; /* stack for the early relocation code */ + early_init_stack = .; +#endif . = ALIGN(SEGMENT_ALIGN); __pecoff_data_size = ABSOLUTE(. - __initdata_begin);
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 18/41 arm64: idreg-override: Avoid sprintf() for simple string concatenation
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Instead of using sprintf() with the "%s.%s=" format, where the first string argument is always the same in the inner loop of match_options(), use simple memcpy() for string concatenation, and move the first copy to the outer loop. This removes the dependency on sprintf(), which will be difficult to fulfil when we move this code into the early mini C runtime.
--- arch/arm64/kernel/idreg-override.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-)
arch/arm64/kernel/idreg-override.c +13 -6
@@ -206,14 +206,15 @@ static int __init parse_nokaslr(char *unused) }early_param ("nokaslr" , parse_nokaslr); -static int __init find_field (const char *cmdline, +static int __init find_field (const char *cmdline,char *opt,int len,const struct ftr_set_desc *reg,int f,u64 *v) { -char opt[FTR_DESC_NAME_LEN +FTR_DESC_FIELD_LEN +2 ]; -int len; +int flen =strlen (reg->fields[f].name); - len =snprintf (opt,ARRAY_SIZE (opt),"%s.%s=" , - reg->name, reg->fields[f].name); +// append '<fieldname>=' to obtain '<name>.<fieldname>=' +memcpy (opt + len, reg->fields[f].name, flen); + len += flen; + opt[len++] ='=' ;if (memcmp (cmdline, opt, len))return -1 ;
@@ -223,15 +224,21 @@ static int __init find_field(const char *cmdline,static void __init match_options (const char *cmdline) { +char opt[FTR_DESC_NAME_LEN +FTR_DESC_FIELD_LEN +2 ];int i;for (i =0 ; i <ARRAY_SIZE (regs); i++) {const struct ftr_set_desc *reg =prel64_pointer (regs[i].reg);struct arm64_ftr_override *override; +int len =strlen (reg->name);int f; override =prel64_pointer (reg->override); +// set opt[] to '<name>.' +memcpy (opt, reg->name, len); + opt[len++] ='.' ; +for (f =0 ; reg->fields[f].name[0 ] !='\0' ; f++) {u64 shift = reg->fields[f].shift;u64 width = reg->fields[f].width ?:4 ;
@@ -239,7 +246,7 @@ static void __init match_options(const char *cmdline)bool (*filter)(u64 val);u64 v; -if (find_field (cmdline, reg, f, &v)) +if (find_field (cmdline, opt, len, reg, f, &v))continue ;/*
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 10/41 arm64: kaslr: Adjust randomization range dynamically
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Currently, we base the KASLR randomization range on a rough estimate of the available space in the upper VA region: the lower 1/4th has the module region and the upper 1/4th has the fixmap, vmemmap and PCI I/O ranges, and so we pick a random location in the remaining space in the middle.
Once we enable support for 5-level paging with 4k pages, this no longer
works: the vmemmap region, being dimensioned to cover a 52-bit linear region, takes up so much space in the upper VA region (the size of which is based on a 48-bit VA space for compatibility with non-LVA hardware) that the region above the vmalloc region takes up more than a quarter of the available space.
So instead of a heuristic, let's derive the randomization range from the actual boundaries of the vmalloc region.
--- arch/arm64/kernel/image-vars.h | 2 ++ arch/arm64/kernel/pi/kaslr_early.c | 11 ++++++----- 2 files changed, 8 insertions(+), 5 deletions(-)
arch/arm64/kernel/image-vars.h +2 -0
@@ -36,6 +36,8 @@ PROVIDE(__pi___memcpy = __pi_memcpy);PROVIDE (__pi___memmove = __pi_memmove);PROVIDE (__pi___memset = __pi_memset); +PROVIDE (__pi_vabits_actual = vabits_actual); +#ifdef CONFIG_KVM /*
arch/arm64/kernel/pi/kaslr_early.c +6 -5
@@ -14,6 +14,7 @@#include <asm/archrandom.h> #include <asm/memory.h> +#include <asm/pgtable.h> /* taken from lib/string.c */ static char *__strstr (const char *s1,const char *s2)
@@ -87,7 +88,7 @@ static u64 get_kaslr_seed(void *fdt) asmlinkage u64 kaslr_early_init(void *fdt) { -u64 seed; +u64 seed, range;if (is_kaslr_disabled_cmdline (fdt))return 0 ;
@@ -102,9 +103,9 @@ asmlinkage u64 kaslr_early_init(void *fdt)/* * OK, so we are proceeding with KASLR enabled. Calculate a suitable * kernel image offset from the seed. Let's place the kernel in the -* middle half of the VMALLOC area (VA_BITS_MIN - 2), and stay clear of -* the lower and upper quarters to avoid colliding with other -* allocations. +* 'middle' half of the VMALLOC area, and stay clear of the lower and +* upper quarters to avoid colliding with other allocations. */ -return BIT (VA_BITS_MIN -3 ) + (seed &GENMASK (VA_BITS_MIN -3 ,0 )); + range = (VMALLOC_END -KIMAGE_VADDR ) /2 ; +return range /2 + (((__uint128_t )range * seed) >>64 ); }
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 21/41 arm64: idreg-override: Move to early mini C runtime
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
We will want to parse the ID register overrides even earlier, so that we can take them into account before creating the kernel mapping. So migrate the code and make it work in the context of the early C runtime. We will move the invocation to an earlier stage in a subsequent patch.
---
arch/arm64/kernel/Makefile | 4 +--
arch/arm64/kernel/head.S | 5 ++-
arch/arm64/kernel/image-vars.h | 9 +++++
arch/arm64/kernel/pi/Makefile | 5 +--
arch/arm64/kernel/{ => pi}/idreg-override.c | 35 +++++++++-----------
5 files changed, 30 insertions(+), 28 deletions(-)arch/arm64/kernel/Makefile +1 -3
@@ -33,8 +33,7 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ return_address.o cpuinfo.o cpu_errata.o \ cpufeature.o alternative.o cacheinfo.o \ smp.o smp_spin_table.o topology.o smccc-call.o \ - syscall.o proton-pack.o idreg-override.o idle.o \ - patching.o + syscall.o proton-pack.o idle.o patching.o pi/ obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ sys_compat.o
@@ -57,7 +56,6 @@ obj-$(CONFIG_ACPI) += acpi.o obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o obj-$(CONFIG_ARM64_ACPI_PARKING_PROTOCOL) += acpi_parking_protocol.o obj-$(CONFIG_PARAVIRT) += paravirt.o -obj-$(CONFIG_RELOCATABLE) += pi/ obj-$(CONFIG_RANDOMIZE_BASE) += kaslr.o obj-$(CONFIG_HIBERNATION) += hibernate.o hibernate-asm.o obj-$(CONFIG_ELF_CORE) += elfcore.o
arch/arm64/kernel/head.S +2 -3
@@ -510,10 +510,9 @@ SYM_FUNC_START_LOCAL(__primary_switched) #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) bl kasan_early_init #endif - mov x0, x21 // pass FDT address in x0 - bl early_fdt_map // Try mapping the FDT early mov x0, x20 // pass the full boot status - bl init_feature_override // Parse cpu feature overrides + mov x1, x22 // pass the low FDT mapping + bl __pi_init_feature_override // Parse cpu feature overrides #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS bl scs_patch_vmlinux #endif
arch/arm64/kernel/image-vars.h +9 -0
@@ -37,6 +37,15 @@ PROVIDE(__pi___memmove = __pi_memmove);PROVIDE (__pi___memset = __pi_memset);PROVIDE (__pi_vabits_actual = vabits_actual); +PROVIDE (__pi_id_aa64isar1_override = id_aa64isar1_override); +PROVIDE (__pi_id_aa64isar2_override = id_aa64isar2_override); +PROVIDE (__pi_id_aa64mmfr1_override = id_aa64mmfr1_override); +PROVIDE (__pi_id_aa64pfr0_override = id_aa64pfr0_override); +PROVIDE (__pi_id_aa64pfr1_override = id_aa64pfr1_override); +PROVIDE (__pi_id_aa64smfr0_override = id_aa64smfr0_override); +PROVIDE (__pi_id_aa64zfr0_override = id_aa64zfr0_override); +PROVIDE (__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE (__pi__ctype = _ctype);#ifdef CONFIG_KVM
arch/arm64/kernel/pi/Makefile +3 -2
@@ -38,6 +38,7 @@ $(obj)/lib-%.pi.o: OBJCOPYFLAGS += --prefix-alloc-sections=.init $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) -obj-y := relocate.pi.o -obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-y := idreg-override.pi.o lib-fdt.pi.o lib-fdt_ro.pi.o +obj-$(CONFIG_RELOCATABLE) += relocate.pi.o +obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o extra-y := $(patsubst %.pi.o,%.o,$(obj-y))
arch/arm64/kernel/pi/idreg-override.c renamed from arch/arm64/kernel/idreg-override.c +15 -20
@@ -14,6 +14,8 @@#include <asm/cpufeature.h> #include <asm/setup.h> +#include "pi.h" +#define FTR_DESC_NAME_LEN 20#define FTR_DESC_FIELD_LEN 10#define FTR_ALIAS_NAME_LEN 30
@@ -21,15 +23,6 @@static u64 __boot_status__initdata ; -// temporary __prel64 related definitions -// to be removed when this code is moved under pi/ - -#define __prel64_initconst __initconst - -#define PREL64 (type, name) union { type *name; } - -#define prel64_pointer (__d) (__d) -typedef bool filter_t (u64 val);struct ftr_set_desc {
@@ -313,16 +306,11 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) }while (1 ); } -static __init const u8 *get_bootargs_cmdline (void ) +static __init const u8 *get_bootargs_cmdline (const void *fdt) {const u8 *prop; -void *fdt;int node; - fdt =get_early_fdt_ptr (); -if (!fdt) -return NULL ; - node =fdt_path_offset (fdt,"/chosen" );if (node <0 )return NULL ;
@@ -334,9 +322,9 @@ static __init const u8 *get_bootargs_cmdline(void)return strlen (prop) ? prop :NULL ; } -static __init void parse_cmdline (void ) +static __init void parse_cmdline (const void *fdt) { -const u8 *prop =get_bootargs_cmdline (); +const u8 *prop =get_bootargs_cmdline (fdt);if (IS_ENABLED (CONFIG_CMDLINE_FORCE ) || !prop)__parse_cmdline (CONFIG_CMDLINE , true);
@@ -346,9 +334,9 @@ static __init void parse_cmdline(void) }/* Keep checkers quiet */ -void init_feature_override (u64 boot_status); +void init_feature_override (u64 boot_status,const void *fdt); -asmlinkage void __init init_feature_override(u64 boot_status) +asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) {struct arm64_ftr_override *override;const struct ftr_set_desc *reg;
@@ -364,7 +352,7 @@ asmlinkage void __init init_feature_override(u64 boot_status) __boot_status = boot_status; -parse_cmdline (); +parse_cmdline (fdt);for (i =0 ; i <ARRAY_SIZE (regs); i++) { reg =prel64_pointer (regs[i].reg);
@@ -373,3 +361,10 @@ asmlinkage void __init init_feature_override(u64 boot_status) (unsigned long)(override + 1)); } } + +char *__init skip_spaces (const char *str) +{ +while (isspace (*str)) + ++str; +return (char *)str; +}
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 22/41 arm64: kernel: Remove early fdt remap code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The early FDT remap code is no longer used so let's drop it.
--- arch/arm64/include/asm/setup.h | 3 --- arch/arm64/kernel/setup.c | 15 --------------- 2 files changed, 18 deletions(-)
arch/arm64/include/asm/setup.h +0 -3
@@ -7,9 +7,6 @@#include <uapi/asm/setup.h> -void *get_early_fdt_ptr (void ); -void early_fdt_map (u64 dt_phys); -/* * These two variables are used in the head.S file. */
arch/arm64/kernel/setup.c +0 -15
@@ -166,21 +166,6 @@ static void __init smp_build_mpidr_hash(void)pr_warn ("Large number of MPIDR hash buckets detected\n" ); } -static void *early_fdt_ptr__initdata ; - -void __init *get_early_fdt_ptr (void ) -{ -return early_fdt_ptr; -} - -asmlinkage void __init early_fdt_map(u64 dt_phys) -{ -int fdt_size; - -early_fixmap_init (); - early_fdt_ptr =fixmap_remap_fdt (dt_phys, &fdt_size,PAGE_KERNEL ); -} -static void __init setup_machine_fdt (phys_addr_t dt_phys) {int size;
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 23/41 arm64: head: Clear BSS and the kernel page tables in one go
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
We will move the CPU feature overrides into BSS in a subsequent patch, and this requires that BSS is zeroed before the feature override detection code runs. So let's map BSS read-write in the ID map, and zero it via this mapping.
Since the kernel page tables are right next to it, and also zeroed via the ID map, let's drop the separate clear_page_tables() function, and just zero everything in one go.
--- arch/arm64/kernel/head.S | 33 +++++++------------- arch/arm64/kernel/vmlinux.lds.S | 3 ++ 2 files changed, 14 insertions(+), 22 deletions(-)
arch/arm64/kernel/head.S +11 -22
@@ -177,17 +177,6 @@ SYM_CODE_START_LOCAL(preserve_boot_args) ret SYM_CODE_END(preserve_boot_args) -SYM_FUNC_START_LOCAL(clear_page_tables) - /* - * Clear the init page tables. - */ - adrp x0, init_pg_dir - adrp x1, init_pg_end - sub x2, x1, x0 - mov x1, xzr - b __pi_memset // tail call -SYM_FUNC_END(clear_page_tables) - /* * Macro to populate page table entries, these entries can be pointers to the next level * or last level entries pointing to physical memory.
@@ -386,9 +375,9 @@ SYM_FUNC_START_LOCAL(create_idmap) map_memory x0, x1, x3, x6, x7, x3, IDMAP_PGD_ORDER, x10, x11, x12, x13, x14, EXTRA_SHIFT - /* Remap the kernel page tables r/w in the ID map */ + /* Remap BSS and the kernel page tables r/w in the ID map */ adrp x1, _text - adrp x2, init_pg_dir + adrp x2, __bss_start adrp x3, _end bic x4, x2, #SWAPPER_BLOCK_SIZE - 1 mov_q x5, SWAPPER_RW_MMUFLAGS
@@ -489,14 +478,6 @@ SYM_FUNC_START_LOCAL(__primary_switched) mov x0, x20 bl set_cpu_boot_mode_flag - // Clear BSS - adr_l x0, __bss_start - mov x1, xzr - adr_l x2, __bss_stop - sub x2, x2, x0 - bl __pi_memset - dsb ishst // Make zero page visible to PTW - #if VA_BITS > 48 adr_l x8, vabits_actual // Set this early so KASAN early init str x25, [x8] // ... observes the correct value
@@ -782,6 +763,15 @@ SYM_FUNC_START_LOCAL(__primary_switch) adrp x1, reserved_pg_dir adrp x2, init_idmap_pg_dir bl __enable_mmu + + // Clear BSS + adrp x0, __bss_start + mov x1, xzr + adrp x2, init_pg_end + sub x2, x2, x0 + bl __pi_memset + dsb ishst // Make zero page visible to PTW + #ifdef CONFIG_RELOCATABLE adrp x23, KERNEL_START and x23, x23, MIN_KIMG_ALIGN - 1
@@ -796,7 +786,6 @@ SYM_FUNC_START_LOCAL(__primary_switch) orr x23, x23, x0 // record kernel offset #endif #endif - bl clear_page_tables bl create_kernel_mapping adrp x1, init_pg_dir
arch/arm64/kernel/vmlinux.lds.S +3 -0
@@ -311,12 +311,15 @@ SECTIONS __pecoff_data_rawsize = ABSOLUTE(. - __initdata_begin); _edata = .; + /* start of zero-init region */ BSS_SECTION(SBSS_ALIGN, 0, 0) . = ALIGN(PAGE_SIZE); init_pg_dir = .; . += INIT_DIR_SIZE; init_pg_end = .; + /* end of zero-init region */ + #ifdef CONFIG_RELOCATABLE . += SZ_4K; /* stack for the early relocation code */ early_init_stack = .;
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 24/41 arm64: Move feature overrides into the BSS section
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
In order to allow the CPU feature override detection code to run even earlier, move the feature override global variables into BSS, which is the only part of the static kernel image that is mapped read-write in the initial ID map.
--- arch/arm64/kernel/cpufeature.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
arch/arm64/kernel/cpufeature.c +7 -7
@@ -655,13 +655,13 @@ static const struct arm64_ftr_bits ftr_raz[] = {#define ARM64_FTR_REG (id, table) \ __ARM64_FTR_REG_OVERRIDE(#id, id, table, &no_override) -struct arm64_ftr_override __ro_after_init id_aa64mmfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64pfr1_override; -struct arm64_ftr_override __ro_after_init id_aa64zfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64smfr0_override; -struct arm64_ftr_override __ro_after_init id_aa64isar1_override; -struct arm64_ftr_override __ro_after_init id_aa64isar2_override; +struct arm64_ftr_override id_aa64mmfr1_override; +struct arm64_ftr_override id_aa64pfr0_override; +struct arm64_ftr_override id_aa64pfr1_override; +struct arm64_ftr_override id_aa64zfr0_override; +struct arm64_ftr_override id_aa64smfr0_override; +struct arm64_ftr_override id_aa64isar1_override; +struct arm64_ftr_override id_aa64isar2_override;struct arm64_ftr_override arm64_sw_feature_override;
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 37/41 arm64: pgtable: Decouple PGDIR size macros from PGD/PUD/PMD levels
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The mapping from PGD/PUD/PMD to levels and shifts is very confusing, given that, due to folding, the shifts may be equal for different levels, if the macros are even #define'd to begin with.
In a subsequent patch, we will modify the ID mapping code to decouple the number of levels from the kernel's view of how these types are folded, so prepare for this by reformulating the macros without the use of these types.
Instead, use SWAPPER_BLOCK_SHIFT as the base quantity, and derive it from either PAGE_SHIFT or PMD_SHIFT, which -if defined at all- are defined unambiguously for a given page size, regardless of the number of configured levels.
--- arch/arm64/include/asm/kernel-pgtable.h | 65 ++++++-------------- 1 file changed, 19 insertions(+), 46 deletions(-)
arch/arm64/include/asm/kernel-pgtable.h +19 -46
@@ -13,27 +13,22 @@#include <asm/sparsemem.h> /* -* The linear mapping and the start of memory are both 2M aligned (per -* the arm64 booting.txt requirements). Hence we can use section mapping -* with 4K (section size = 2M) but not with 16K (section size = 32M) or -* 64K (section size = 512M). +* The physical and virtual addresses of the start of the kernel image are +* equal modulo 2 MiB (per the arm64 booting.txt requirements). Hence we can +* use section mapping with 4K (section size = 2M) but not with 16K (section +* size = 32M) or 64K (section size = 512M). */ - -/* -* The idmap and swapper page tables need some space reserved in the kernel -* image. Both require pgd, pud (4 levels only) and pmd tables to (section) -* map the kernel. With the 64K page configuration, swapper and idmap need to -* map to pte level. The swapper also maps the FDT (see __create_page_tables -* for more information). Note that the number of ID map translation levels -* could be increased on the fly if system RAM is out of reach for the default -* VA range, so pages required to map highest possible PA are reserved in all -* cases. -*/ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - 1) +#if defined(PMD_SIZE ) &&PMD_SIZE <=MIN_KIMG_ALIGN +#define SWAPPER_BLOCK_SHIFT PMD_SHIFT +#define SWAPPER_SKIP_LEVEL 1#else -#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS) +#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT +#define SWAPPER_SKIP_LEVEL 0#endif +#define SWAPPER_BLOCK_SIZE (UL(1) << SWAPPER_BLOCK_SHIFT) +#define SWAPPER_TABLE_SHIFT (SWAPPER_BLOCK_SHIFT + PAGE_SHIFT - 3) + +#define SWAPPER_PGTABLE_LEVELS (CONFIG_PGTABLE_LEVELS - SWAPPER_SKIP_LEVEL)#define IDMAP_VA_BITS 48#define IDMAP_LEVELS ARM64_HW_PGTABLE_LEVELS(IDMAP_VA_BITS)
@@ -53,24 +48,13 @@#define EARLY_ENTRIES (vstart, vend, shift, add) \ (SPAN_NR_ENTRIES(vstart, vend, shift) + (add)) -#define EARLY_PGDS (vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PGDIR_SHIFT, add)) - -#if SWAPPER_PGTABLE_LEVELS >3 -#define EARLY_PUDS (vstart, vend, add) (EARLY_ENTRIES(vstart, vend, PUD_SHIFT, add)) -#else -#define EARLY_PUDS (vstart, vend, add) (0) -#endif +#define EARLY_LEVEL (lvl, vstart, vend, add) \ + (SWAPPER_PGTABLE_LEVELS > lvl ? EARLY_ENTRIES(vstart, vend, SWAPPER_BLOCK_SHIFT + lvl * (PAGE_SHIFT - 3), add) : 0) -#if SWAPPER_PGTABLE_LEVELS >2 -#define EARLY_PMDS (vstart, vend, add) (EARLY_ENTRIES(vstart, vend, SWAPPER_TABLE_SHIFT, add)) -#else -#define EARLY_PMDS (vstart, vend, add) (0) -#endif - -#define EARLY_PAGES (vstart, vend, add) ( 1/* PGDIR page */ \ - +EARLY_PGDS ((vstart), (vend), add)/* each PGDIR needs a next level page table */ \ - +EARLY_PUDS ((vstart), (vend), add)/* each PUD needs a next level page table */ \ - + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ +#define EARLY_PAGES (vstart, vend, add) (1/* PGDIR page */ \ + +EARLY_LEVEL (3 , (vstart), (vend), add)/* each entry needs a next level page table */ \ + +EARLY_LEVEL (2 , (vstart), (vend), add)/* each entry needs a next level page table */ \ + + EARLY_LEVEL(1, (vstart), (vend), add))/* each entry needs a next level page table */ #define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES))/* the initial ID map may need two extra pages if it needs to be extended */
@@ -81,17 +65,6 @@#endif #define INIT_IDMAP_DIR_PAGES EARLY_PAGES(KIMAGE_VADDR, _end + MAX_FDT_SIZE + SWAPPER_BLOCK_SIZE, 1) -/* Initial memory map size */ -#ifdef CONFIG_ARM64_4K_PAGES -#define SWAPPER_BLOCK_SHIFT PMD_SHIFT -#define SWAPPER_BLOCK_SIZE PMD_SIZE -#define SWAPPER_TABLE_SHIFT PUD_SHIFT -#else -#define SWAPPER_BLOCK_SHIFT PAGE_SHIFT -#define SWAPPER_BLOCK_SIZE PAGE_SIZE -#define SWAPPER_TABLE_SHIFT PMD_SHIFT -#endif -/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ #define KERNEL_SEGMENT_COUNT 5
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 15/41 arm64: idreg-override: Prepare for place relative reloc patching
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The ID reg override handling code uses a rather elaborate data structure that relies on statically initialized absolute address values in pointer fields. This means that this code cannot run until relocation fixups have been applied, and this is unfortunate, because it means we cannot discover overrides for KASLR or LVA/LPA without creating the kernel mapping and performing the relocations first.
This can be solved by switching to place-relative relocations, which can be applied by the linker at build time. This means some additional arithmetic is required when dereferencing these pointers, as we can no longer dereference the pointer members directly.
So let's implement this for idreg-override.c in a preliminary way, i.e., convert all the references in code to use a special accessor that produces the correct absolute value at runtime.
--- arch/arm64/kernel/idreg-override.c | 89 ++++++++++++-------- 1 file changed, 56 insertions(+), 33 deletions(-)
arch/arm64/kernel/idreg-override.c +56 -33
@@ -21,14 +21,25 @@static u64 __boot_status__initdata ; +// temporary __prel64 related definitions +// to be removed when this code is moved under pi/ + +#define __prel64_initconst __initconst + +#define PREL64 (type, name) union { type *name; } + +#define prel64_pointer (__d) (__d) + +typedef bool filter_t (u64 val); +struct ftr_set_desc {char name[FTR_DESC_NAME_LEN ]; -struct arm64_ftr_override *override; + PREL64(struct arm64_ftr_override, override);struct {char name[FTR_DESC_FIELD_LEN ];u8 shift;u8 width; -bool (*filter)(u64 val); + PREL64(filter_t, filter); } fields[]; };
@@ -46,7 +57,7 @@ static bool __init mmfr1_vh_filter(u64 val) val == 0); } -static const struct ftr_set_desc mmfr1__initconst = { +static const struct ftr_set_desc mmfr1 __prel64_initconst = { .name ="id_aa64mmfr1" , .override = &id_aa64mmfr1_override, .fields = {
@@ -70,7 +81,7 @@ static bool __init pfr0_sve_filter(u64 val)return true; } -static const struct ftr_set_desc pfr0__initconst = { +static const struct ftr_set_desc pfr0 __prel64_initconst = { .name ="id_aa64pfr0" , .override = &id_aa64pfr0_override, .fields = {
@@ -94,7 +105,7 @@ static bool __init pfr1_sme_filter(u64 val)return true; } -static const struct ftr_set_desc pfr1__initconst = { +static const struct ftr_set_desc pfr1 __prel64_initconst = { .name ="id_aa64pfr1" , .override = &id_aa64pfr1_override, .fields = {
@@ -105,7 +116,7 @@ static const struct ftr_set_desc pfr1 __initconst = { }, }; -static const struct ftr_set_desc isar1__initconst = { +static const struct ftr_set_desc isar1 __prel64_initconst = { .name ="id_aa64isar1" , .override = &id_aa64isar1_override, .fields = {
@@ -117,7 +128,7 @@ static const struct ftr_set_desc isar1 __initconst = { }, }; -static const struct ftr_set_desc isar2__initconst = { +static const struct ftr_set_desc isar2 __prel64_initconst = { .name ="id_aa64isar2" , .override = &id_aa64isar2_override, .fields = {
@@ -128,7 +139,7 @@ static const struct ftr_set_desc isar2 __initconst = { }, }; -static const struct ftr_set_desc smfr0__initconst = { +static const struct ftr_set_desc smfr0 __prel64_initconst = { .name ="id_aa64smfr0" , .override = &id_aa64smfr0_override, .fields = {
@@ -149,7 +160,7 @@ static bool __init hvhe_filter(u64 val) ID_AA64MMFR1_EL1_VH_SHIFT)); } -static const struct ftr_set_desc sw_features__initconst = { +static const struct ftr_set_desc sw_features __prel64_initconst = { .name ="arm64_sw" , .override = &arm64_sw_feature_override, .fields = {
@@ -159,14 +170,15 @@ static const struct ftr_set_desc sw_features __initconst = { }, }; -static const struct ftr_set_desc *const regs[]__initconst = { - &mmfr1, - &pfr0, - &pfr1, - &isar1, - &isar2, - &smfr0, - &sw_features, +static const +PREL64(const struct ftr_set_desc, reg) regs[] __prel64_initconst = { + { &mmfr1 }, + { &pfr0 }, + { &pfr1 }, + { &isar1 }, + { &isar2 }, + { &smfr0 }, + { &sw_features }, }; static const struct {
@@ -214,15 +226,20 @@ static void __init match_options(const char *cmdline)int i;for (i =0 ; i <ARRAY_SIZE (regs); i++) { +const struct ftr_set_desc *reg =prel64_pointer (regs[i].reg); +struct arm64_ftr_override *override;int f; -for (f =0 ;strlen (regs[i]->fields[f].name); f++) { -u64 shift = regs[i]->fields[f].shift; -u64 width = regs[i]->fields[f].width ?:4 ; + override =prel64_pointer (reg->override); + +for (f =0 ;strlen (reg->fields[f].name); f++) { +u64 shift = reg->fields[f].shift; +u64 width = reg->fields[f].width ?:4 ;u64 mask =GENMASK_ULL (shift + width -1 , shift); +bool (*filter)(u64 val);u64 v; -if (find_field (cmdline, regs[i], f, &v)) +if (find_field (cmdline, reg, f, &v))continue ;/*
@@ -230,16 +247,16 @@ static void __init match_options(const char *cmdline)* it by setting the value to the all-ones while * clearing the mask... Yes, this is fragile. */ -if (regs[i]->fields[f].filter && - !regs[i]->fields[f].filter (v)) { - regs[i]->override->val |= mask; - regs[i]->override->mask &= ~mask; + filter =prel64_pointer (reg->fields[f].filter); +if (filter && !filter (v)) { + override->val |= mask; + override->mask &= ~mask;continue ; } - regs[i]->override->val &= ~mask; - regs[i]->override->val |= (v << shift) & mask; - regs[i]->override->mask |= mask; + override->val &= ~mask; + override->val |= (v << shift) & mask; + override->mask |= mask;return ; }
@@ -313,11 +330,16 @@ void init_feature_override(u64 boot_status); asmlinkage void __init init_feature_override(u64 boot_status) { +struct arm64_ftr_override *override; +const struct ftr_set_desc *reg;int i;for (i =0 ; i <ARRAY_SIZE (regs); i++) { - regs[i]->override->val =0 ; - regs[i]->override->mask =0 ; + reg =prel64_pointer (regs[i].reg); + override =prel64_pointer (reg->override); + + override->val =0 ; + override->mask =0 ; } __boot_status = boot_status;
@@ -325,8 +347,9 @@ asmlinkage void __init init_feature_override(u64 boot_status)parse_cmdline ();for (i =0 ; i <ARRAY_SIZE (regs); i++) { -dcache_clean_inval_poc ((unsigned long )regs[i]->override, - (unsigned long )regs[i]->override + -sizeof (*regs[i]->override)); + reg =prel64_pointer (regs[i].reg); + override =prel64_pointer (reg->override); +dcache_clean_inval_poc ((unsigned long )override, + (unsigned long )(override +1 )); } }
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 29/41 arm64: idreg-override: Create a pseudo feature for rodata=off
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Add rodata=off to the set of kernel command line options that is parsed early using the CPU feature override detection code, so we can easily refer to it when creating the kernel mapping.
--- arch/arm64/include/asm/cpufeature.h | 1 + arch/arm64/kernel/pi/idreg-override.c | 2 ++ 2 files changed, 3 insertions(+)
arch/arm64/include/asm/cpufeature.h +1 -0
@@ -17,6 +17,7 @@#define ARM64_SW_FEATURE_OVERRIDE_NOKASLR 0#define ARM64_SW_FEATURE_OVERRIDE_HVHE 4 +#define ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF 8#ifndef __ASSEMBLY__
arch/arm64/kernel/pi/idreg-override.c +2 -0
@@ -159,6 +159,7 @@ static const struct ftr_set_desc sw_features __prel64_initconst = { .fields = {FIELD ("nokaslr" ,ARM64_SW_FEATURE_OVERRIDE_NOKASLR ,NULL ),FIELD ("hvhe" ,ARM64_SW_FEATURE_OVERRIDE_HVHE , hvhe_filter), + FIELD("rodataoff", ARM64_SW_FEATURE_OVERRIDE_RODATA_OFF, NULL), {} }, };
@@ -190,6 +191,7 @@ static const struct { { "arm64.nomops", "id_aa64isar2.mops=0" }, { "arm64.nomte", "id_aa64pfr1.mte=0" }, { "nokaslr", "arm64_sw.nokaslr=1" }, + { "rodata=off", "arm64_sw.rodataoff=1" }, };static int __init parse_hexdigit (const char *p,u64 *v)
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 30/41 arm64: Add helpers to probe local CPU for PAC and BTI support
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
Add some helpers that will be used by the early kernel mapping code to check feature support on the local CPU. This permits the early kernel mapping to be created with the right attributes, removing the need for tearing it down and recreating it.
--- arch/arm64/include/asm/cpufeature.h | 32 ++++++++++++++++++++ 1 file changed, 32 insertions(+)
arch/arm64/include/asm/cpufeature.h +32 -0
@@ -957,6 +957,38 @@ static inline bool kaslr_disabled_cmdline(void)u32 get_kvm_ipa_limit (void );void dump_cpu_features (void ); +static inline bool cpu_has_bti (void ) +{ +if (!IS_ENABLED (CONFIG_ARM64_BTI )) +return false; + +return arm64_apply_feature_override (read_cpuid (ID_AA64PFR1_EL1 ), +ID_AA64PFR1_EL1_BT_SHIFT ,4 , + &id_aa64pfr1_override); +} + +static inline bool cpu_has_pac (void ) +{ +u64 isar1, isar2; + +if (!IS_ENABLED (CONFIG_ARM64_PTR_AUTH )) +return false; + + isar1 =read_cpuid (ID_AA64ISAR1_EL1 ); + isar2 =read_cpuid (ID_AA64ISAR2_EL1 ); + +if (arm64_apply_feature_override (isar1,ID_AA64ISAR1_EL1_APA_SHIFT ,4 , + &id_aa64isar1_override)) +return true; + +if (arm64_apply_feature_override (isar1,ID_AA64ISAR1_EL1_API_SHIFT ,4 , + &id_aa64isar1_override)) +return true; + +return arm64_apply_feature_override (isar2,ID_AA64ISAR2_EL1_APA3_SHIFT ,4 , + &id_aa64isar2_override); +} +#endif /* __ASSEMBLY__ */ #endif
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 31/41 arm64: head: allocate more pages for the kernel mapping
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
In preparation for switching to an early kernel mapping routine that maps each segment according to its precise boundaries, and with the correct attributes, let's allocate some extra pages for page tables for the 4k page size configuration. This is necessary because the start and end of each segment may not be aligned to the block size, and so we'll need an extra page table at each segment boundary.
--- arch/arm64/include/asm/kernel-pgtable.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
arch/arm64/include/asm/kernel-pgtable.h +10 -1
@@ -68,7 +68,7 @@ +EARLY_PGDS ((vstart), (vend), add)/* each PGDIR needs a next level page table */ \ +EARLY_PUDS ((vstart), (vend), add)/* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE)) +#define INIT_DIR_SIZE (PAGE_SIZE * (EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE) + EARLY_SEGMENT_EXTRA_PAGES)) /* the initial ID map may need two extra pages if it needs to be extended */#if VA_BITS <48
@@ -89,6 +89,15 @@#define SWAPPER_TABLE_SHIFT PMD_SHIFT#endif +/* The number of segments in the kernel image (text, rodata, inittext, initdata, data+bss) */ +#define KERNEL_SEGMENT_COUNT 5 + +#if SWAPPER_BLOCK_SIZE >SEGMENT_ALIGN +#define EARLY_SEGMENT_EXTRA_PAGES (KERNEL_SEGMENT_COUNT + 1) +#else +#define EARLY_SEGMENT_EXTRA_PAGES 0 +#endif +/* * Initial memory map attributes. */
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 35/41 arm64: head: Move early kernel mapping routines into C code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The asm version of the kernel mapping code works fine for creating a coarse grained identity map, but for mapping the kernel down to its exact boundaries with the right attributes, it is not suitable. This is why we create a preliminary RWX kernel mapping first, and then rebuild it from scratch later on.
So let's reimplement this in C, in a way that will make it unnecessary to create the kernel page tables yet another time in paging_init().
--- arch/arm64/include/asm/archrandom.h | 2 - arch/arm64/include/asm/scs.h | 32 +--- arch/arm64/kernel/head.S | 52 +------ arch/arm64/kernel/image-vars.h | 19 +++ arch/arm64/kernel/pi/Makefile | 1 + arch/arm64/kernel/pi/idreg-override.c | 22 ++- arch/arm64/kernel/pi/kaslr_early.c | 12 +- arch/arm64/kernel/pi/map_kernel.c | 164 ++++++++++++++++++++ arch/arm64/kernel/pi/map_range.c | 88 +++++++++++ arch/arm64/kernel/pi/patch-scs.c | 16 +- arch/arm64/kernel/pi/pi.h | 14 ++ arch/arm64/kernel/pi/relocate.c | 2 + arch/arm64/kernel/setup.c | 7 - arch/arm64/kernel/vmlinux.lds.S | 4 +- arch/arm64/mm/proc.S | 1 + 15 files changed, 315 insertions(+), 121 deletions(-)
arch/arm64/include/asm/archrandom.h +0 -2
@@ -129,6 +129,4 @@ static inline bool __init __early_cpu_has_rndr(void) return (ftr >> ID_AA64ISAR0_EL1_RNDR_SHIFT) & 0xf; } -u64 kaslr_early_init (void *fdt); -#endif /* _ASM_ARCHRANDOM_H */
arch/arm64/include/asm/scs.h +3 -29
@@ -33,37 +33,11 @@#include <asm/cpufeature.h> #ifdef CONFIG_UNWIND_PATCH_PAC_INTO_SCS -static inline bool should_patch_pac_into_scs (void ) -{ -u64 reg; - -/* -* We only enable the shadow call stack dynamically if we are running -* on a system that does not implement PAC or BTI. PAC and SCS provide -* roughly the same level of protection, and BTI relies on the PACIASP -* instructions serving as landing pads, preventing us from patching -* those instructions into something else. -*/ - reg =read_sysreg_s (SYS_ID_AA64ISAR1_EL1 ); -if (SYS_FIELD_GET (ID_AA64ISAR1_EL1 ,APA , reg) | -SYS_FIELD_GET (ID_AA64ISAR1_EL1 ,API , reg)) -return false; - - reg =read_sysreg_s (SYS_ID_AA64ISAR2_EL1 ); -if (SYS_FIELD_GET (ID_AA64ISAR2_EL1 ,APA3 , reg)) -return false; - -if (IS_ENABLED (CONFIG_ARM64_BTI_KERNEL )) { - reg =read_sysreg_s (SYS_ID_AA64PFR1_EL1 ); -if (reg & (0xf <<ID_AA64PFR1_EL1_BT_SHIFT )) -return false; - } -return true; -} -static inline void dynamic_scs_init (void ) { -if (should_patch_pac_into_scs ()) { +extern bool __pi_dynamic_scs_is_enabled; + +if (__pi_dynamic_scs_is_enabled) {pr_info ("Enabling dynamic shadow call stack\n" );static_branch_enable (&dynamic_scs_enabled); }
arch/arm64/kernel/head.S +1 -51
76 lines not on this page · raw message
arch/arm64/kernel/image-vars.h +19 -0
@@ -45,9 +45,28 @@ PROVIDE(__pi_id_aa64pfr1_override = id_aa64pfr1_override);PROVIDE (__pi_id_aa64smfr0_override = id_aa64smfr0_override);PROVIDE (__pi_id_aa64zfr0_override = id_aa64zfr0_override);PROVIDE (__pi_arm64_sw_feature_override = arm64_sw_feature_override); +PROVIDE (__pi_arm64_use_ng_mappings = arm64_use_ng_mappings); +#ifdef CONFIG_CAVIUM_ERRATUM_27456 +PROVIDE (__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus); +#endif PROVIDE (__pi__ctype = _ctype);PROVIDE (__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE (__pi_init_pg_dir = init_pg_dir); +PROVIDE (__pi_init_pg_end = init_pg_end); + +PROVIDE (__pi__text = _text); +PROVIDE (__pi__stext = _stext); +PROVIDE (__pi__etext = _etext); +PROVIDE (__pi___start_rodata = __start_rodata); +PROVIDE (__pi___inittext_begin = __inittext_begin); +PROVIDE (__pi___inittext_end = __inittext_end); +PROVIDE (__pi___initdata_begin = __initdata_begin); +PROVIDE (__pi___initdata_end = __initdata_end); +PROVIDE (__pi__data = _data); +PROVIDE (__pi___bss_start = __bss_start); +PROVIDE (__pi__end = _end); +#ifdef CONFIG_KVM /*
arch/arm64/kernel/pi/Makefile +1 -0
@@ -39,6 +39,7 @@ $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c) obj-y := idreg-override.pi.o \ + map_kernel.pi.o map_range.pi.o \ lib-fdt.pi.o lib-fdt_ro.pi.o obj-$(CONFIG_RELOCATABLE) += relocate.pi.o obj-$(CONFIG_RANDOMIZE_BASE) += kaslr_early.pi.o
arch/arm64/kernel/pi/idreg-override.c +10 -12
@@ -308,37 +308,35 @@ static __init void __parse_cmdline(const char *cmdline, bool parse_aliases) }while (1 ); } -static __init const u8 *get_bootargs_cmdline (const void *fdt) +static __init const u8 *get_bootargs_cmdline (const void *fdt,int node) { +static char const bootargs[]__initconst ="bootargs" ;const u8 *prop; -int node; - node =fdt_path_offset (fdt,"/chosen" );if (node <0 )return NULL ; - prop =fdt_getprop (fdt, node,"bootargs" ,NULL ); + prop =fdt_getprop (fdt, node, bootargs,NULL );if (!prop)return NULL ;return strlen (prop) ? prop :NULL ; } -static __init void parse_cmdline (const void *fdt) +static __init void parse_cmdline (const void *fdt,int chosen) { -const u8 *prop =get_bootargs_cmdline (fdt); +static char const cmdline[]__initconst =CONFIG_CMDLINE ; +const u8 *prop =get_bootargs_cmdline (fdt, chosen);if (IS_ENABLED (CONFIG_CMDLINE_FORCE ) || !prop) -__parse_cmdline (CONFIG_CMDLINE , true); +__parse_cmdline (cmdline, true);if (!IS_ENABLED (CONFIG_CMDLINE_FORCE ) && prop)__parse_cmdline (prop, true); } -/* Keep checkers quiet */ -void init_feature_override (u64 boot_status,const void *fdt); - -asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) +void __init init_feature_override (u64 boot_status,const void *fdt, +int chosen) {struct arm64_ftr_override *override;const struct ftr_set_desc *reg;
@@ -354,7 +352,7 @@ asmlinkage void __init init_feature_override(u64 boot_status, const void *fdt) __boot_status = boot_status; -parse_cmdline (fdt); +parse_cmdline (fdt, chosen);for (i =0 ; i <ARRAY_SIZE (regs); i++) { reg =prel64_pointer (regs[i].reg);
arch/arm64/kernel/pi/kaslr_early.c +6 -6
@@ -16,17 +16,17 @@#include <asm/memory.h> #include <asm/pgtable.h> +#include "pi.h" +extern u16 memstart_offset_seed; -static u64 __init get_kaslr_seed (void *fdt) +static u64 __init get_kaslr_seed (void *fdt,int node) { -static char const chosen_str[]__initconst ="chosen" ;static char const seed_str[]__initconst ="kaslr-seed" ; -int node, len;fdt64_t *prop;u64 ret; +int len; - node =fdt_path_offset (fdt, chosen_str);if (node <0 )return 0 ;
@@ -39,14 +39,14 @@ static u64 __init get_kaslr_seed(void *fdt)return ret; } -asmlinkage u64 __init kaslr_early_init(void *fdt) +u64 __init kaslr_early_init (void *fdt,int chosen) {u64 seed, range;if (kaslr_disabled_cmdline ())return 0 ; - seed =get_kaslr_seed (fdt); + seed =get_kaslr_seed (fdt, chosen);if (!seed) {if (!__early_cpu_has_rndr () || !__arm64_rndr ((unsigned long *)&seed))
arch/arm64/kernel/pi/map_kernel.c new file +164 -0
164 lines not on this page · raw message
arch/arm64/kernel/pi/map_range.c new file +88 -0
88 lines not on this page · raw message
arch/arm64/kernel/pi/patch-scs.c +4 -12
@@ -11,6 +11,10 @@#include <asm/scs.h> +#include "pi.h" + +bool dynamic_scs_is_enabled; +// // This minimal DWARF CFI parser is partially based on the code in // arch/arc/kernel/unwind.c, and on the document below:
@@ -46,8 +50,6 @@#define DW_CFA_GNU_negative_offset_extended 0x2f#define DW_CFA_hi_user 0x3f -extern const u8 __eh_frame_start[], __eh_frame_end[]; -enum {PACIASP =0xd503233f ,AUTIASP =0xd50323bf ,
@@ -250,13 +252,3 @@ int scs_patch(const u8 eh_frame[], int size) }return 0 ; } - -asmlinkage void __init scs_patch_vmlinux(const u8 start[], const u8 end[]) -{ -if (!should_patch_pac_into_scs ()) -return ; - -scs_patch (start, end - start); - asm("ic ialluis" ); -isb (); -}
arch/arm64/kernel/pi/pi.h +14 -0
@@ -2,6 +2,8 @@// Copyright 2023 Google LLC // Author: Ard Biesheuvel <[email protected]> +#include <linux/types.h> +#define __prel64_initconst __section(".init.rodata.prel64")#define PREL64 (type, name) union { type *name; prel64_t name ## _prel; }
@@ -16,3 +18,15 @@ static inline void *prel64_to_pointer(const prel64_t *offset)return NULL ;return (void *)offset + *offset; } + +extern bool dynamic_scs_is_enabled; + +void init_feature_override (u64 boot_status,const void *fdt,int chosen); +u64 kaslr_early_init (void *fdt,int chosen); +void relocate_kernel (u64 offset); +int scs_patch (const u8 eh_frame[],int size); + +void map_range (u64 *pgd,u64 start,u64 end,u64 pa,pgprot_t prot, +int level,pte_t *tbl,bool may_use_cont,u64 va_offset); + +asmlinkage void early_map_kernel(u64 boot_status, void *fdt);
arch/arm64/kernel/pi/relocate.c +2 -0
@@ -7,6 +7,8 @@#include <linux/init.h> #include <linux/types.h> +#include "pi.h" +extern const Elf64_Rela rela_start[], rela_end[];extern const u64 relr_start[], relr_end[];
arch/arm64/kernel/setup.c +0 -7
@@ -283,13 +283,6 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p)kaslr_init (); -/* -* If know now we are going to need KPTI then use non-global -* mappings from the start, avoiding the cost of rewriting -* everything later. -*/ - arm64_use_ng_mappings =kaslr_enabled () &&kaslr_requires_kpti (); -early_fixmap_init ();early_ioremap_init ();
arch/arm64/kernel/vmlinux.lds.S +2 -2
@@ -126,9 +126,9 @@ jiffies = jiffies_64; #ifdef CONFIG_UNWIND_TABLES #define UNWIND_DATA_SECTIONS \ .eh_frame : { \ - __eh_frame_start = .; \ + __pi___eh_frame_start = .; \ *(.eh_frame) \ - __eh_frame_end = .; \ + __pi___eh_frame_end = .; \ } #else #define UNWIND_DATA_SECTIONS
arch/arm64/mm/proc.S +1 -0
@@ -195,6 +195,7 @@ SYM_TYPED_FUNC_START(idmap_cpu_replace_ttbr1) ret SYM_FUNC_END(idmap_cpu_replace_ttbr1) +SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) .popsection #ifdef CONFIG_UNMAP_KERNEL_AT_EL0
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH v6 38/41 arm64: kernel: Create initial ID map from C code
In reply to Ard BiesheuvelFrom: Ard Biesheuvel <[email protected]>
The asm code that creates the initial ID map is rather intricate and hard to follow. This is problematic because it makes adding support for things like LPA2 or WXN more difficult than necessary. Also, it is parameterized like the rest of the MM code to run with a configurable number of levels, which is rather pointless, given that all AArch64 CPUs implement support for 48-bit virtual addressing, and that many systems exist with DRAM located outside of the 39-bit addressable range, which is the only smaller VA size that is widely used, and we need additional tricks to make things work in that combination.
So let's bite the bullet, and rip out all the asm macros, and fiddly code, and replace it with a C implementation based on the newly added routines for creating the early kernel VA mappings. And while at it, create the initial ID map based on 48-bit virtual addressing as well, regardless of the number of configured levels for the kernel proper.
Note that this code may execute with the MMU and caches disabled, and is therefore not permitted to make unaligned accesses. This shouldn't generally happen in any case for the algorithm as implemented, but to be sure, let's pass -mstrict-align to the compiler just in case.
--- arch/arm64/include/asm/assembler.h | 14 - arch/arm64/include/asm/kernel-pgtable.h | 50 ++-- arch/arm64/include/asm/mmu_context.h | 6 +- arch/arm64/kernel/head.S | 267 ++------------------ arch/arm64/kernel/image-vars.h | 1 + arch/arm64/kernel/pi/Makefile | 3 + arch/arm64/kernel/pi/map_kernel.c | 18 ++ arch/arm64/kernel/pi/map_range.c | 12 + arch/arm64/kernel/pi/pi.h | 4 + arch/arm64/mm/mmu.c | 5 - arch/arm64/mm/proc.S | 3 +- 11 files changed, 88 insertions(+), 295 deletions(-)
arch/arm64/include/asm/assembler.h +0 -14
@@ -345,20 +345,6 @@ alternative_cb_end bfi \valreg, \t1sz, #TCR_T1SZ_OFFSET, #TCR_TxSZ_WIDTH .endm -/* -* idmap_get_t0sz - get the T0SZ value needed to cover the ID map -* -* Calculate the maximum allowed value for TCR_EL1.T0SZ so that the -* entire ID map region can be mapped. As T0SZ == (64 - #bits used), -* this number conveniently equals the number of leading zeroes in -* the physical address of _end. -*/ - .macro idmap_get_t0sz, reg - adrp \reg, _end - orr \reg, \reg, #(1 << VA_BITS_MIN) - 1 - clz \reg, \reg - .endm -/* * tcr_compute_pa_size - set TCR.(I)PS to the highest supported * ID_AA64MMFR0_EL1.PARange value
arch/arm64/include/asm/kernel-pgtable.h +23 -27
73 lines not on this page · raw message
arch/arm64/include/asm/mmu_context.h +2 -4
@@ -61,11 +61,9 @@ static inline void cpu_switch_mm(pgd_t *pgd, struct mm_struct *mm) }/* -* TCR.T0SZ value to use when the ID map is active. Usually equals -* TCR_T0SZ(VA_BITS), unless system RAM is positioned very high in -* physical memory, in which case it will be smaller. +* TCR.T0SZ value to use when the ID map is active. */ -extern int idmap_t0sz; +#define idmap_t0sz TCR_T0SZ(IDMAP_VA_BITS)/* * Ensure TCR.T0SZ is set to the provided value.
arch/arm64/kernel/head.S +23 -244
305 lines not on this page · raw message
arch/arm64/kernel/image-vars.h +1 -0
@@ -52,6 +52,7 @@ PROVIDE(__pi_cavium_erratum_27456_cpus = cavium_erratum_27456_cpus);PROVIDE (__pi__ctype = _ctype);PROVIDE (__pi_memstart_offset_seed = memstart_offset_seed); +PROVIDE (__pi_init_idmap_pg_dir = init_idmap_pg_dir);PROVIDE (__pi_init_pg_dir = init_pg_dir);PROVIDE (__pi_init_pg_end = init_pg_end);
arch/arm64/kernel/pi/Makefile +3 -0
@@ -11,6 +11,9 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -fno-asynchronous-unwind-tables -fno-unwind-tables \ $(call cc-option,-fno-addrsig) +# this code may run with the MMU off so disable unaligned accesses +CFLAGS_map_range.o += -mstrict-align + # remove SCS flags from all objects in this directory KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # disable LTO
arch/arm64/kernel/pi/map_kernel.c +18 -0
@@ -128,6 +128,22 @@ static void __init map_kernel(u64 kaslr_offset, u64 va_offset, int root_level) } } +static void __init map_fdt (u64 fdt) +{ +static u8 ptes[INIT_IDMAP_FDT_SIZE ]__initdata __aligned(PAGE_SIZE) ; +u64 efdt = fdt +MAX_FDT_SIZE ; +u64 ptep = (u64 )ptes; + +/* +* Map up to MAX_FDT_SIZE bytes, but avoid overlap with +* the kernel image. +*/ +map_range (&ptep, fdt, (u64 )_text > fdt ?min ((u64 )_text, efdt) : efdt, + fdt,PAGE_KERNEL ,IDMAP_ROOT_LEVEL , + (pte_t *)init_idmap_pg_dir, false,0 ); +dsb (ishst); +} + asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt) {static char const chosen_str[]__initconst ="/chosen" ;
@@ -136,6 +152,8 @@ asmlinkage void __init early_map_kernel(u64 boot_status, void *fdt)int root_level =4 -CONFIG_PGTABLE_LEVELS ;int chosen; +map_fdt ((u64 )fdt); +/* Clear BSS and the initial page tables */ memset (__bss_start,0 , (u64 )init_pg_end - (u64 )__bss_start);
arch/arm64/kernel/pi/map_range.c +12 -0
@@ -86,3 +86,15 @@ void __init map_range(u64 *pte, u64 start, u64 end, u64 pa, pgprot_t prot, tbl++; } } + +asmlinkage u64 __init create_init_idmap(pgd_t *pg_dir) +{ +u64 ptep = (u64 )pg_dir +PAGE_SIZE ; + +map_range (&ptep, (u64 )_stext, (u64 )__initdata_begin, (u64 )_stext, +PAGE_KERNEL_ROX ,IDMAP_ROOT_LEVEL , (pte_t *)pg_dir, false,0 ); +map_range (&ptep, (u64 )__initdata_begin, (u64 )_end, (u64 )__initdata_begin, +PAGE_KERNEL ,IDMAP_ROOT_LEVEL , (pte_t *)pg_dir, false,0 ); + +return ptep; +}
arch/arm64/kernel/pi/pi.h +4 -0
@@ -21,6 +21,8 @@ static inline void *prel64_to_pointer(const prel64_t *offset)extern bool dynamic_scs_is_enabled; +extern pgd_t init_idmap_pg_dir[]; +void init_feature_override (u64 boot_status,const void *fdt,int chosen);u64 kaslr_early_init (void *fdt,int chosen);void relocate_kernel (u64 offset);
@@ -30,3 +32,5 @@ void map_range(u64 *pgd, u64 start, u64 end, u64 pa, pgprot_t prot, int level, pte_t *tbl, bool may_use_cont, u64 va_offset); asmlinkage void early_map_kernel(u64 boot_status, void *fdt); + +asmlinkage u64 create_init_idmap(pgd_t *pgd);
arch/arm64/mm/mmu.c +0 -5
@@ -45,8 +45,6 @@#define NO_CONT_MAPPINGS BIT(1)#define NO_EXEC_MAPPINGS BIT(2)/* assumes FEAT_HPDS is not used */ -int idmap_t0sz__ro_after_init ; -#if VA_BITS >48 u64 vabits_actual__ro_after_init =VA_BITS_MIN ;EXPORT_SYMBOL (vabits_actual);
@@ -790,8 +788,6 @@ void __init paging_init(void)pgd_t *pgdp =pgd_set_fixmap (__pa_symbol (swapper_pg_dir));extern pgd_t init_idmap_pg_dir[]; - idmap_t0sz =63UL -__fls (__pa_symbol (_end) |GENMASK (VA_BITS_MIN -1 ,0 )); -map_kernel (pgdp);map_mem (pgdp);
@@ -806,7 +802,6 @@ void __init paging_init(void)memblock_allow_resize ();create_idmap (); - idmap_t0sz =TCR_T0SZ (IDMAP_VA_BITS ); }#ifdef CONFIG_MEMORY_HOTPLUG
arch/arm64/mm/proc.S +2 -1
@@ -200,7 +200,8 @@ SYM_FUNC_ALIAS(__pi_idmap_cpu_replace_ttbr1, idmap_cpu_replace_ttbr1) #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 -#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | SWAPPER_PTE_FLAGS | PTE_WRITE) +#define KPTI_NG_PTE_FLAGS (PTE_ATTRINDX(MT_NORMAL) | PTE_TYPE_PAGE | \ + PTE_AF | PTE_SHARED | PTE_UXN | PTE_WRITE) .pushsection ".idmap.text", "a"
Signature · 7 lines
2.43.0.rc1.413.gea7ed67945-goog _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 11/41] arm64: kernel: Manage absolute relocations in code built under pi/
In reply to Marc ZyngierQuoted 76 lines · Marc Zyngier
Quoted 73 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The mini C runtime runs before relocations are processed, and so it cannot rely on statically initialized pointer variables.
Add a check to ensure that such code does not get introduced by accident, by going over the relocations in each object, identifying the ones that operate on data sections that are part of the executable image, and raising an error if any relocations of type
R_AARCH64_ABS64exist. Note that such relocations are permitted in other places (e.g., debug sections) and will never occur in compiler generated code sections when using the small code model, so only check sections that haveSHF_ALLOCset andSHF_EXECINSTRcleared.To accommodate cases where statically initialized symbol references are unavoidable, introduce a special case for ELF input data sections that have ".rodata.prel64" in their names, and in these cases, instead of rejecting any encountered ABS64 relocations, convert them into PREL64 relocations, which don't require any runtime fixups. Note that the code in question must still be modified to deal with this, as it needs to convert the 64-bit signed offsets into absolute addresses before use.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/kernel/pi/Makefile | 9 +- arch/arm64/kernel/pi/pi.h | 18 +++ arch/arm64/kernel/pi/relacheck.c | 130 ++++++++++++++++++++ 3 files changed, 155 insertions(+), 2 deletions(-)arch/arm64/kernel/pi/Makefile +7
-2@@ -22,11 +22,16 @@ KCSAN_SANITIZE := n UBSAN_SANITIZE := n KCOV_INSTRUMENT := n +hostprogs := relacheck + +quiet_cmd_piobjcopy = $(quiet_cmd_objcopy) + cmd_piobjcopy = $(cmd_objcopy) && $(obj)/relacheck $(@) $(<) + $(obj)/%.pi.o: OBJCOPYFLAGS := --prefix-symbols=__pi_ \ --remove-section=.note.gnu.property \ --prefix-alloc-sections=.init -$(obj)/%.pi.o: $(obj)/%.o FORCE - $(call if_changed,objcopy) +$(obj)/%.pi.o: $(obj)/%.o $(obj)/relacheck FORCE + $(call if_changed,piobjcopy) $(obj)/lib-%.o: $(srctree)/lib/%.c FORCE $(call if_changed_rule,cc_o_c)arch/arm64/kernel/pi/pi.h new file +14
-0@@ -0,0 +1,18 @@ +// SPDX-License-Identifier: GPL-2.0-only +// Copyright 2023 Google LLC +// Author: Ard Biesheuvel <[email protected]> + +#define __prel64_initconst __section(".init.rodata.prel64") + +#define PREL64 (type, name) union { type *name; prel64_t name ## _prel; } + +#define prel64_pointer (__d) (typeof(__d))prel64_to_pointer(&__d##_prel) + +typedef volatile signed long prel64_t ; + +static inline void *prel64_to_pointer (const prel64_t *offset) +{
Quoted 9 lines
arch/arm64/kernel/pi/pi.h line 18
Quoted 4 lines
arch/arm64/kernel/pi/pi.h new file +4
-0+if (!*offset) +return NULL ; +return (void *)offset + *offset; +}Is there any use for these definitions before the override code gets moved to pi/ in patch 21? It is otherwise a bit odd to see two sets of definitions between patches 15 and 20.
No they are unused at this point, but they are tied closely to the host tool that this patch introduces.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 01/41] arm64: kernel: Disable latent_entropy GCC plugin in early C runtime
In reply to Ard BiesheuvelQuoted 3 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
In subsequent patches, mark portions of the early C code will be marked
Small nit - ^^^ redundant usage for mark ^^^^^^
Quoted 1 line
as
__init. Unfortunarely,__initimplies__latent_entropy, and this
Typo here - s/Unfortunarely/Unfortunately/
Quoted 3 lines
would result in the early C code being instrumented in an unsafe manner.
Disable the latent entropy plugin for the early C code.
I am just wondering if this '__latent_entropy' could be problematic for all early __init code in each supported platform. Although this plugin does not get disabled in that many places inside the kernel tree.
$git grep DISABLE_LATENT_ENTROPY_PLUGINarch/arm64/kernel/pi/Makefile: $(DISABLE_LATENT_ENTROPY_PLUGIN) \
arch/powerpc/kernel/Makefile:CFLAGS_early_32.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/kernel/Makefile:CFLAGS_cputable.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/kernel/Makefile:CFLAGS_prom_init.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/kernel/Makefile:CFLAGS_btext.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/kernel/Makefile:CFLAGS_prom.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/kernel/vdso/Makefile: CFLAGS_vgettimeofday-32.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/kernel/vdso/Makefile: CFLAGS_vgettimeofday-64.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/lib/Makefile:CFLAGS_code-patching.o += $(DISABLE_LATENT_ENTROPY_PLUGIN) arch/powerpc/lib/Makefile:CFLAGS_feature-fixups.o += $(DISABLE_LATENT_ENTROPY_PLUGIN)scripts/Makefile.gcc-plugins: DISABLE_LATENT_ENTROPY_PLUGIN += -fplugin-arg-latent_entropy_plugin-disable -ULATENT_ENTROPY_PLUGIN
scripts/Makefile.gcc-plugins:export DISABLE_LATENT_ENTROPY_PLUGIN
Quoted 12 lines
Acked-by Mark Rutland <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/kernel/pi/Makefile | 1 + 1 file changed, 1 insertion(+)arch/arm64/kernel/pi/Makefile +0
-0
arch/arm64/kernel/pi/Makefile line 9
Quoted 7 lines
arch/arm64/kernel/pi/Makefile +1
-0@@ -3,6 +3,7 @@ KBUILD_CFLAGS := $(subst $(CC_FLAGS_FTRACE),,$(KBUILD_CFLAGS)) -fpie \ -Os -DDISABLE_BRANCH_PROFILING $(DISABLE_STACKLEAK_PLUGIN) \ + $(DISABLE_LATENT_ENTROPY_PLUGIN) \ $(call cc-option,-mbranch-protection=none) \ -I$(srctree)/scripts/dtc/libfdt -fno-stack-protector \ -include $(srctree)/include/linux/hidden.h \
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 02/41] arm64: mm: Take potential load offset into account when KASLR is off
In reply to Ard BiesheuvelQuoted 4 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
We enable
CONFIG_RELOCATABLEeven whenCONFIG_RANDOMIZE_BASEis disabled, and this permits the loader (i.e., EFI) to place the kernel
Indeed, could validate this out via defconfig based build and boot.
Quoted 5 lines
anywhere in physical memory as long as the base address is 64k aligned.
This means that the 'KASLR' case described in the header that defines the size of the statically allocated page tables could take effect even when
CONFIG_RANDMIZE_BASE=n. So check forCONFIG_RELOCATABLEinstead.
Makes sense.
Quoted 23 lines
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/kernel-pgtable.h | 27 +++++--------------- 1 file changed, 6 insertions(+), 21 deletions(-)arch/arm64/include/asm/kernel-pgtable.h +0
-9@@ -37,27 +37,12 @@/* -* If KASLR is enabled, then an offset K is added to the kernel address -* space. The bottom 21 bits of this offset are zero to guarantee 2MB -* alignment for PA and VA. -* -* For each pagetable level of the swapper, we know that the shift will -* be larger than 21 (for the 4KB granule case we use section maps thus -* the smallest shift is actually 30) thus there is the possibility that -* KASLR can increase the number of pagetable entries by 1, so we make -* room for this extra entry.
arch/arm64/include/asm/kernel-pgtable.h line 44
Quoted 10 lines
arch/arm64/include/asm/kernel-pgtable.h +4
-5-* -* Note KASLR cannot increase the number of required entries for a level -* by more than one because it increments both the virtual start and end -* addresses equally (the extra entry comes from the case where the end -* address is just pushed over a boundary and the start address isn't). +* A relocatable kernel may execute from an address that differs from the one at +* which it was linked. In the worst case, its runtime placement may intersect +* with two adjacent PGDIR entries, which means that an additional page table +* may be needed at each subordinate level. */
This is a better explanation.
arch/arm64/include/asm/kernel-pgtable.h line 45
Quoted 7 lines
arch/arm64/include/asm/kernel-pgtable.h +1
-6- -#ifdef CONFIG_RANDOMIZE_BASE -#define EARLY_KASLR (1) -#else -#define EARLY_KASLR (0) -#endif +#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE)
EXTRA_INIT_DIR_PAGE instead ? Just to have some more context.
arch/arm64/include/asm/kernel-pgtable.h line 74
Quoted 12 lines
arch/arm64/include/asm/kernel-pgtable.h +1
-1#define SPAN_NR_ENTRIES (vstart, vend, shift) \ ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1)@@ -83,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR)) +#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE))/* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS <48
Regardless
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 03/41] arm64: mm: get rid of kimage_vaddr global variable
In reply to Ard BiesheuvelQuoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
We store the address of
_textinkimage_vaddr, but since commit09e3c22a86f6889d("arm64: Use a variable to store non-global mappings decision"), we no longer reference this variable from modules so we no longer need to export it.In fact, we don't need it at all so let's just get rid of it.
Acked-by Mark Rutland <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>
There is a checkpatch.pl error for this patch.
--------
ERROR: Please use git commit description style 'commit <12+ chars of sha1> ("<title line>")' -
ie: 'commit 09e3c22a86f6 ("arm64: Use a variable to store non-global mappings decision")' #6:
We store the address of _text in kimage_vaddr, but since commit
--------
Otherwise LGTM.
Quoted 53 lines
--- arch/arm64/include/asm/memory.h | 6 ++---- arch/arm64/kernel/head.S | 2 +- arch/arm64/mm/mmu.c | 3 --- 3 files changed, 3 insertions(+), 8 deletions(-)arch/arm64/include/asm/memory.h +2
-4@@ -182,6 +182,7 @@#include <linux/types.h> #include <asm/boot.h> #include <asm/bug.h> +#include <asm/sections.h> #if VA_BITS >48 extern u64 vabits_actual;@@ -193,15 +194,12 @@ extern s64 memstart_addr;/* PHYS_OFFSET - the physical address of the start of memory. */ #define PHYS_OFFSET ({ VM_BUG_ON(memstart_addr & 1); memstart_addr; }) -/* the virtual base of the kernel image */ -extern u64 kimage_vaddr; -/* the offset between the kernel virtual and physical mappings */ extern u64 kimage_voffset;static inline unsigned long kaslr_offset (void ) { -return kimage_vaddr -KIMAGE_VADDR ; +return (u64 )&_text -KIMAGE_VADDR ; }#ifdef CONFIG_RANDOMIZE_BASE arch/arm64/kernel/head.S +1
-1@@ -482,7 +482,7 @@ SYM_FUNC_START_LOCAL(__primary_switched) str_l x21, __fdt_pointer, x5 // Save FDT pointer - ldr_l x4, kimage_vaddr // Save the offset between + adrp x4, _text // Save the offset between sub x4, x4, x0 // the kernel virtual and str_l x4, kimage_voffset, x5 // physical mappingsarch/arm64/mm/mmu.c +0
-0
arch/arm64/mm/mmu.c u64 vabits_actual __ro_after_init = VA_BITS_MIN; · line 57
Quoted 9 lines
arch/arm64/mm/mmu.c +0
-3@@ -52,9 +52,6 @@ u64 vabits_actual __ro_after_init = VA_BITS_MIN;EXPORT_SYMBOL (vabits_actual);#endif -u64 kimage_vaddr__ro_after_init = (u64)&_text; -EXPORT_SYMBOL (kimage_vaddr); -u64 kimage_voffset__ro_after_init ;EXPORT_SYMBOL (kimage_voffset);
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
In reply to Ard BiesheuvelQuoted 35 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)#if VA_BITS >48 arch/arm64/mm/ptdump.c +0
-0@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" },
arch/arm64/mm/ptdump.c static struct addr_marker address_markers[] = { · line 56
Quoted 10 lines
arch/arm64/mm/ptdump.c +2
-2{ FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };
I have not debugged this any further, but the PCI region displays no entries via ptdump just after this patch being applied.
With the patch:
---[ Fixmap end ]---
0xfffffbfffe000000-0xfffffc0000000000 32M PMD
---[ vmemmap start ]---
0xfffffc0000000000-0xfffffc0002000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0002000000-0xfffffc0020000000 480M PMD
0xfffffc0020000000-0xfffffc0022000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0022000000-0xfffffc0040000000 480M PMD
0xfffffc0040000000-0xfffffc8000000000 511G PUD
0xfffffc8000000000-0xfffffe0000000000 1536G PGD
---[ vmemmap end ]---
0xfffffe0000000000-0xfffffe8000000000 512G PGD
---[ PCI I/O start ]---
---[ PCI I/O end ]--- -----> No entries
0xfffffe8000000000-0x0000000000000000 1536G PGD
Without the patch:
--[ Fixmap start ]---
0xfffffbfffdc31000-0xfffffbfffddf6000 1812K PTE
0xfffffbfffddf6000-0xfffffbfffddf9000 12K PTE ro x SHD AF UXN MEM/NORMAL 0xfffffbfffddf9000-0xfffffbfffddfa000 4K PTE ro NX SHD AF NG UXN MEM/NORMAL 0xfffffbfffddfa000-0xfffffbfffddfd000 12K PTE 0xfffffbfffddfd000-0xfffffbfffddfe000 4K PTE RW NX SHD AF NG UXN DEVICE/nGnRE 0xfffffbfffddfe000-0xfffffbfffde00000 8K PTE ro NX SHD AF NG UXN MEM/NORMAL 0xfffffbfffde00000-0xfffffbfffe000000 2M PTE
---[ Fixmap end ]---
0xfffffbfffe000000-0xfffffbfffe800000 8M PMD
---[ PCI I/O start ]---
0xfffffbfffe800000-0xfffffbffff800000 16M PMD
---[ PCI I/O end ]---
0xfffffbffff800000-0xfffffc0000000000 8M PMD
---[ vmemmap start ]---
0xfffffc0000000000-0xfffffc0002000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0002000000-0xfffffc0020000000 480M PMD
0xfffffc0020000000-0xfffffc0022000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0022000000-0xfffffc0040000000 480M PMD
0xfffffc0040000000-0xfffffc8000000000 511G PUD
0xfffffc8000000000-0xfffffe0000000000 1536G PGD
---[ vmemmap end ]---
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
Quoted 41 lines · Anshuman Khandual
Quoted 37 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)#if VA_BITS >48 arch/arm64/mm/ptdump.c +0
-1@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" },
Quoted 8 lines
arch/arm64/mm/ptdump.c static struct addr_marker address_markers[] = { · line 56
Quoted 8 lines
arch/arm64/mm/ptdump.c +2
-1- { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };
Hi Anshuman,
Quoted 4 lines
I have not debugged this any further, but the PCI region displays no entries via ptdump just after this patch being applied.
It also does not display any mappings without the patch. What exactly were you expecting to see here?
Quoted 27 lines
With the patch:
---[ Fixmap end ]---
0xfffffbfffe000000-0xfffffc0000000000 32M PMD
---[ vmemmap start ]---
0xfffffc0000000000-0xfffffc0002000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0002000000-0xfffffc0020000000 480M PMD
0xfffffc0020000000-0xfffffc0022000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0022000000-0xfffffc0040000000 480M PMD
0xfffffc0040000000-0xfffffc8000000000 511G PUD
0xfffffc8000000000-0xfffffe0000000000 1536G PGD
---[ vmemmap end ]---
0xfffffe0000000000-0xfffffe8000000000 512G PGD
---[ PCI I/O start ]---
---[ PCI I/O end ]--- -----> No entries
0xfffffe8000000000-0x0000000000000000 1536G PGDWithout the patch:
--[ Fixmap start ]---
0xfffffbfffdc31000-0xfffffbfffddf6000 1812K PTE0xfffffbfffddf6000-0xfffffbfffddf9000 12K PTE ro x SHD AF UXN MEM/NORMAL 0xfffffbfffddf9000-0xfffffbfffddfa000 4K PTE ro NX SHD AF NG UXN MEM/NORMAL 0xfffffbfffddfa000-0xfffffbfffddfd000 12K PTE 0xfffffbfffddfd000-0xfffffbfffddfe000 4K PTE RW NX SHD AF NG UXN DEVICE/nGnRE 0xfffffbfffddfe000-0xfffffbfffde00000 8K PTE ro NX SHD AF NG UXN MEM/NORMAL 0xfffffbfffde00000-0xfffffbfffe000000 2M PTE
Quoted 14 lines
---[ Fixmap end ]---
0xfffffbfffe000000-0xfffffbfffe800000 8M PMD
---[ PCI I/O start ]---
0xfffffbfffe800000-0xfffffbffff800000 16M PMD
---[ PCI I/O end ]---
0xfffffbffff800000-0xfffffc0000000000 8M PMD
---[ vmemmap start ]---
0xfffffc0000000000-0xfffffc0002000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0002000000-0xfffffc0020000000 480M PMD
0xfffffc0020000000-0xfffffc0022000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0022000000-0xfffffc0040000000 480M PMD
0xfffffc0040000000-0xfffffc8000000000 511G PUD
0xfffffc8000000000-0xfffffe0000000000 1536G PGD
---[ vmemmap end ]---
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
Quoted 55 lines · Ard Biesheuvel
Quoted 41 lines · Anshuman Khandual
Quoted 37 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)#if VA_BITS >48 arch/arm64/mm/ptdump.c +0
-1@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" },Quoted 8 lines
arch/arm64/mm/ptdump.c static struct addr_marker address_markers[] = { · line 56
Quoted 8 lines
arch/arm64/mm/ptdump.c +2
-1- { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };Hi Anshuman,
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
I have not debugged this any further, but the PCI region displays no entries via ptdump just after this patch being applied.
It also does not display any mappings without the patch. What exactly were you expecting to see here?
Is not there a single entry inside - without this patch section. OR does it signify the gap-width of the PCI mapping area ?
---[ PCI I/O start ]---
0xfffffbfffe800000-0xfffffbffff800000 16M PMD
---[ PCI I/O end ]---
Quoted 34 lines
Quoted 34 lines
With the patch:
---[ Fixmap end ]---
0xfffffbfffe000000-0xfffffc0000000000 32M PMD
---[ vmemmap start ]---
0xfffffc0000000000-0xfffffc0002000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0002000000-0xfffffc0020000000 480M PMD
0xfffffc0020000000-0xfffffc0022000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0022000000-0xfffffc0040000000 480M PMD
0xfffffc0040000000-0xfffffc8000000000 511G PUD
0xfffffc8000000000-0xfffffe0000000000 1536G PGD
---[ vmemmap end ]---
0xfffffe0000000000-0xfffffe8000000000 512G PGD
---[ PCI I/O start ]---
---[ PCI I/O end ]--- -----> No entries
0xfffffe8000000000-0x0000000000000000 1536G PGDWithout the patch:
--[ Fixmap start ]---
0xfffffbfffdc31000-0xfffffbfffddf6000 1812K PTE0xfffffbfffddf6000-0xfffffbfffddf9000 12K PTE ro x SHD AF UXN MEM/NORMAL 0xfffffbfffddf9000-0xfffffbfffddfa000 4K PTE ro NX SHD AF NG UXN MEM/NORMAL 0xfffffbfffddfa000-0xfffffbfffddfd000 12K PTE 0xfffffbfffddfd000-0xfffffbfffddfe000 4K PTE RW NX SHD AF NG UXN DEVICE/nGnRE 0xfffffbfffddfe000-0xfffffbfffde00000 8K PTE ro NX SHD AF NG UXN MEM/NORMAL 0xfffffbfffde00000-0xfffffbfffe000000 2M PTE---[ Fixmap end ]---
0xfffffbfffe000000-0xfffffbfffe800000 8M PMD
---[ PCI I/O start ]---
0xfffffbfffe800000-0xfffffbffff800000 16M PMD
---[ PCI I/O end ]---
0xfffffbffff800000-0xfffffc0000000000 8M PMD
Quoted 8 lines
Quoted 8 lines
---[ vmemmap start ]---
0xfffffc0000000000-0xfffffc0002000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0002000000-0xfffffc0020000000 480M PMD
0xfffffc0020000000-0xfffffc0022000000 32M PMD RW NX SHD AF NG BLK UXN MEM/NORMAL
0xfffffc0022000000-0xfffffc0040000000 480M PMD
0xfffffc0040000000-0xfffffc8000000000 511G PUD
0xfffffc8000000000-0xfffffe0000000000 1536G PGD
---[ vmemmap end ]---
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
Quoted 66 lines · Anshuman Khandual
Quoted 55 lines · Ard Biesheuvel
Quoted 41 lines · Anshuman Khandual
Quoted 37 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)#if VA_BITS >48 arch/arm64/mm/ptdump.c +0
-1@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" },Quoted 8 lines
arch/arm64/mm/ptdump.c static struct addr_marker address_markers[] = { · line 56
Quoted 8 lines
arch/arm64/mm/ptdump.c +2
-1- { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };Hi Anshuman,
Quoted 1 line
Quoted 6 lines
Quoted 3 lines
I have not debugged this any further, but the PCI region displays no entries via ptdump just after this patch being applied.
It also does not display any mappings without the patch. What exactly were you expecting to see here?
Quoted 7 lines
Is not there a single entry inside - without this patch section. OR does it signify the gap-width of the PCI mapping area ?
---[ PCI I/O start ]---
0xfffffbfffe800000-0xfffffbffff800000 16M PMD
---[ PCI I/O end ]---
This means that the PCI I/O space is covered by 8 invalid level 2 entries.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
0xfffffe0000000000-0xfffffe8000000000 512G PGD
---[ PCI I/O start ]---
---[ PCI I/O end ]--- -----> No entries
0xfffffe8000000000-0x0000000000000000 1536G PGD
This means that the PCI I/O space is covered by 1 invalid level 0 entry.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v6 02/41] arm64: mm: Take potential load offset into account when KASLR is off
In reply to Ard BiesheuvelQuoted 5 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
We enable
CONFIG_RELOCATABLEeven whenCONFIG_RANDOMIZE_BASEis disabled, and this permits the loader (i.e., EFI) to place the kernel anywhere in physical memory as long as the base address is 64k aligned.
I don't think that case is something we actually intend to permit today:
(a) When CONFIG_RANDOMIZE_BASE=n, the EFI stub will load the kernel at SZ_2M alignment. We initialize efi_nokaslr to !IS_ENABLED(CONFIG_RANDOMIZE_BASE), and so arm64's efi_get_kimg_min_align() will return SZ_2M.
... unless I'm missing something there?
(b) We don't expose anything in the Image header such that an external bootloader (i.e. not the EFI stub) can decide that 64K alignment is sufficient. It would be unsound for a bootloader to load the kernel at less than 2M alignment.
(c) We never documented 64K alignment as being permitted. In booting.txt we say "The Image must be placed text_offset bytes from a 2MB aligned base address anywhere in usable system RAM and called there.", with no mention of a relaxation down to 64K.
... so I don't think this patch is necessary, unless it's going to make something else simpler later in the series?
Mark.
Quoted 58 lines
This means that the 'KASLR' case described in the header that defines the size of the statically allocated page tables could take effect even when
CONFIG_RANDMIZE_BASE=n. So check forCONFIG_RELOCATABLEinstead.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/kernel-pgtable.h | 27 +++++--------------- 1 file changed, 6 insertions(+), 21 deletions(-)arch/arm64/include/asm/kernel-pgtable.h +6
-21@@ -37,27 +37,12 @@/* -* If KASLR is enabled, then an offset K is added to the kernel address -* space. The bottom 21 bits of this offset are zero to guarantee 2MB -* alignment for PA and VA. -* -* For each pagetable level of the swapper, we know that the shift will -* be larger than 21 (for the 4KB granule case we use section maps thus -* the smallest shift is actually 30) thus there is the possibility that -* KASLR can increase the number of pagetable entries by 1, so we make -* room for this extra entry. -* -* Note KASLR cannot increase the number of required entries for a level -* by more than one because it increments both the virtual start and end -* addresses equally (the extra entry comes from the case where the end -* address is just pushed over a boundary and the start address isn't). +* A relocatable kernel may execute from an address that differs from the one at +* which it was linked. In the worst case, its runtime placement may intersect +* with two adjacent PGDIR entries, which means that an additional page table +* may be needed at each subordinate level. */ - -#ifdef CONFIG_RANDOMIZE_BASE -#define EARLY_KASLR (1) -#else -#define EARLY_KASLR (0) -#endif +#define EXTRA_PAGE __is_defined(CONFIG_RELOCATABLE)#define SPAN_NR_ENTRIES (vstart, vend, shift) \ ((((vend) - 1) >> (shift)) - ((vstart) >> (shift)) + 1)@@ -83,7 +68,7 @@ + EARLY_PGDS((vstart), (vend), add) /* each PGDIR needs a next level page table */ \ + EARLY_PUDS((vstart), (vend), add) /* each PUD needs a next level page table */ \ + EARLY_PMDS((vstart), (vend), add)) /* each PMD needs a next level page table */ -#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EARLY_KASLR)) +#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end, EXTRA_PAGE))/* the initial ID map may need two extra pages if it needs to be extended */ #if VA_BITS <48 Signature · 1 line
2.43.0.rc1.413.gea7ed67945-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v6 03/41] arm64: mm: get rid of kimage_vaddr global variable
In reply to Anshuman KhandualQuoted 15 lines · Anshuman Khandual
Quoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
We store the address of
_textinkimage_vaddr, but since commit09e3c22a86f6889d("arm64: Use a variable to store non-global mappings decision"), we no longer reference this variable from modules so we no longer need to export it.In fact, we don't need it at all so let's just get rid of it.
Acked-by Mark Rutland <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>
Quoted 8 lines
There is a checkpatch.pl error for this patch.
--------
ERROR: Please use git commit description style 'commit <12+ chars of sha1> ("<title line>")' -
ie: 'commit09e3c22a86f6("arm64: Use a variable to store non-global mappings decision")' #6:
We store the address of_textinkimage_vaddr, but since commit
--------
That looks like a spurious warning. Ard wrote:
[...] commit 09e3c22a86f6889d ("arm64: Use a variable to store non-global mappings decision"), [...]
... which is 'commit <16 chars of sha1> ("<title line>")', just as the checkpatch message says.
Clearly, this checkpatch warning isn't helpful.
Regardless of whether the warning is spurious, please use judgement for checkpatch warnings. They're not always a hard rule to be followed.
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 02/41] arm64: mm: Take potential load offset into account when KASLR is off
In reply to Mark RutlandQuoted 21 lines · Mark Rutland
Quoted 5 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
We enable
CONFIG_RELOCATABLEeven whenCONFIG_RANDOMIZE_BASEis disabled, and this permits the loader (i.e., EFI) to place the kernel anywhere in physical memory as long as the base address is 64k aligned.I don't think that case is something we actually intend to permit today:
(a) When
CONFIG_RANDOMIZE_BASE=n, the EFI stub will load the kernel atSZ_2Malignment. We initializeefi_nokaslrto !IS_ENABLED(CONFIG_RANDOMIZE_BASE), and so arm64'sefi_get_kimg_min_align()will returnSZ_2M.... unless I'm missing something there?
(b) We don't expose anything in the Image header such that an external bootloader (i.e. not the EFI stub) can decide that 64K alignment is sufficient. It would be unsound for a bootloader to load the kernel at less than 2M alignment.
Quoted 8 lines
(c) We never documented 64K alignment as being permitted. In booting.txt we say "The Image must be placed
text_offsetbytes from a 2MB aligned base address anywhere in usable system RAM and called there.", with no mention of a relaxation down to 64K.... so I don't think this patch is necessary, unless it's going to make something else simpler later in the series?
Your analysis is correct afaict, and I got things confused as to whether/how EFI will relocate the image.
However, we do have a quirk that permits loading with a hard coded TEXT_OFFSET of 512k even if the TEXT_OFFSET in the header is 0x0. This is the reason why we enabled CONFIG_RELOCATABLE without CONFIG_RANDOMIZE_BASE by default.
I think that ultimately, the fact that CONFIG_RELOCATABLE can be enabled without CONFIG_RANDOMIZE_BASE is sufficient grounds to add this extra space.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 03/41] arm64: mm: get rid of kimage_vaddr global variable
In reply to Mark RutlandQuoted 25 lines · Mark Rutland
Quoted 15 lines · Anshuman Khandual
Quoted 11 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
We store the address of
_textinkimage_vaddr, but since commit09e3c22a86f6889d("arm64: Use a variable to store non-global mappings decision"), we no longer reference this variable from modules so we no longer need to export it.In fact, we don't need it at all so let's just get rid of it.
Acked-by Mark Rutland <[email protected]>Signed-off-by Ard Biesheuvel <[email protected]>Quoted 8 lines
There is a checkpatch.pl error for this patch.
--------
ERROR: Please use git commit description style 'commit <12+ chars of sha1> ("<title line>")' -
ie: 'commit09e3c22a86f6("arm64: Use a variable to store non-global mappings decision")' #6:
We store the address of_textinkimage_vaddr, but since commit
--------
Quoted 7 lines
That looks like a spurious warning. Ard wrote:
[...] commit
09e3c22a86f6889d("arm64: Use a variable to store non-global mappings decision"), [...]... which is 'commit <16 chars of sha1> ("<title line>")', just as the checkpatch message says.
Although it does not make any difference TBH, sometimes moving 'commit' before the SHA string on the same line just removes the warning. That line break might be the trigger.
Quoted 7 lines
Clearly, this checkpatch warning isn't helpful.
Regardless of whether the warning is spurious, please use judgement for checkpatch warnings. They're not always a hard rule to be followed.
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
In reply to Ard BiesheuvelHi Ard,
Quoted 18 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +0
-0
arch/arm64/include/asm/memory.h line 54
Quoted 8 lines
arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)
This looks simple, but it does have a subtle dependency on VMEMMAP_SIZE being sufficiently smaller than the negative offset we choose for VMEMMAP_START, and it took mw a while to convince myself that we always ensure that today. I could well imagine that we try to reduce that in future (given for most configurations we're reserving around double the space we actually need), and it would be nice if we could avoid that going wrong.
We need to make sure that PCI_IO_START and PCI_IO_END fall between VMEMMAP_END and the last page of the VA space (due to ERR_PTR). Could we add some comment and/or assertions to that effect?
Mark.
Quoted 22 lines
arch/arm64/include/asm/memory.h +0
-0#if VA_BITS >48 arch/arm64/mm/ptdump.c +2
-2@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };Signature · 1 line
2.43.0.rc1.413.gea7ed67945-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
Quoted 29 lines · Mark Rutland
Hi Ard,
Quoted 25 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)
Quoted 9 lines
arch/arm64/include/asm/memory.h line 54
Quoted 1 line
arch/arm64/include/asm/memory.h +0
-0#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)This looks simple, but it does have a subtle dependency on
VMEMMAP_SIZEbeing sufficiently smaller than the negative offset we choose forVMEMMAP_START, and it took mw a while to convince myself that we always ensure that today. I could well imagine that we try to reduce that in future (given for most configurations we're reserving around double the space we actually need), and it would be nice if we could avoid that going wrong.
Yes, we are relying on the fact that the upper region is not covered by struct pages, so there is guaranteed to be an unused region at the very top.
One of the constants in the equation that describes the size of this unused region is the size of a struct page, and so I agree this needs a couple of static asserts to ensure we don't get inadvertent overlaps.
Quoted 25 lines
We need to make sure that
PCI_IO_STARTandPCI_IO_ENDfall betweenVMEMMAP_ENDand the last page of the VA space (due toERR_PTR). Could we add some comment and/or assertions to that effect?Mark.
Quoted 19 lines
arch/arm64/include/asm/memory.h +0
-0#if VA_BITS >48 arch/arm64/mm/ptdump.c +2
-2@@ -47,10 +47,10 @@ static struct addr_marker address_markers[] = { { VMALLOC_END, "vmalloc() end" }, { FIXADDR_TOT_START, "Fixmap start" }, { FIXADDR_TOP, "Fixmap end" }, - { PCI_IO_START, "PCI I/O start" }, - { PCI_IO_END, "PCI I/O end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, { -1, NULL }, };
Quoted 3 lines
Quoted 3 lines
--
2.43.0.rc1.413.gea7ed67945-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v6 04/41] arm64: mm: Move PCI I/O emulation region above the vmemmap region
Quoted 40 lines · Ard Biesheuvel
Quoted 29 lines · Mark Rutland
Hi Ard,
Quoted 25 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the PCI I/O region above the vmemmap region in the kernel's VA space. This will permit us to reclaim the lower part of the vmemmap region for vmalloc/vmap allocations when running a 52-bit VA capable build on a 48-bit VA capable system.
Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/include/asm/memory.h | 4 ++-- arch/arm64/mm/ptdump.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +2
-2@@ -49,8 +49,8 @@#define MODULES_VSIZE (SZ_2G)#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) -#define PCI_IO_END (VMEMMAP_START - SZ_8M) -#define PCI_IO_START (PCI_IO_END - PCI_IO_SIZE) +#define PCI_IO_START (VMEMMAP_END + SZ_8M) +#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)Quoted 9 lines
arch/arm64/include/asm/memory.h line 54
Quoted 1 line
arch/arm64/include/asm/memory.h +0
-0#define FIXADDR_TOP (VMEMMAP_START - SZ_32M)This looks simple, but it does have a subtle dependency on
VMEMMAP_SIZEbeing sufficiently smaller than the negative offset we choose forVMEMMAP_START, and it took mw a while to convince myself that we always ensure that today. I could well imagine that we try to reduce that in future (given for most configurations we're reserving around double the space we actually need), and it would be nice if we could avoid that going wrong.
Quoted 3 lines
Yes, we are relying on the fact that the upper region is not covered by struct pages, so there is guaranteed to be an unused region at the very top.
Yep; exactly. My fear is that in future someone realises that:
(-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))
... is actually reserving ~2x the space it needs for *most* configurations (since for most configurations we only need the VMEMMAP to account for (VA_BITS-1) bits of linear map). If they try to shrink the VMEMMAP region in for those cases, it's liable to overlap subsequent regions and/or to wrap those past the end of the VA space.
Quoted 4 lines
One of the constants in the equation that describes the size of this unused region is the size of a struct page, and so I agree this needs a couple of static asserts to ensure we don't get inadvertent overlaps.
That'd be great, thanks!
Mark.
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
reviewerRe: [PATCH v6 05/41] arm64: mm: Move fixmap region above vmemmap region
In reply to Ard BiesheuvelQuoted 10 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Move the fixmap region above the vmemmap region, so that the start of the vmemmap delineates the end of the region available for vmalloc and vmap allocations and the randomized placement of the kernel and modules.
In a subsequent patch, we will take advantage of this to reclaim most of the vmemmap area when running a 52-bit VA capable build with 52-bit virtual addressing disabled at runtime.
Quoted 5 lines
Note that the existing guard region of 256 MiB covers the fixmap and PCI I/O regions as well, so we can reduce it 8 MiB, which is what we use in other places too.
Signed-off-by Ard Biesheuvel <[email protected]>
As with the prior patch, this looks simple enough, but it'd be good if we could do something to assert that we don't overlap this region with others.
Mark.
Quoted 54 lines
--- arch/arm64/include/asm/memory.h | 2 +- arch/arm64/include/asm/pgtable.h | 2 +- arch/arm64/mm/ptdump.c | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +1
-1@@ -51,7 +51,7 @@#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE)#define PCI_IO_START (VMEMMAP_END + SZ_8M)#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) -#define FIXADDR_TOP (VMEMMAP_START - SZ_32M) +#define FIXADDR_TOP (-UL(SZ_8M))#if VA_BITS >48 #define VA_BITS_MIN (48)arch/arm64/include/asm/pgtable.h +1
-1@@ -22,7 +22,7 @@* and fixed mappings */ #define VMALLOC_START (MODULES_END) -#define VMALLOC_END (VMEMMAP_START - SZ_256M) +#define VMALLOC_END (VMEMMAP_START - SZ_8M)#define vmemmap ((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT))arch/arm64/mm/ptdump.c +2
-2@@ -45,12 +45,12 @@ static struct addr_marker address_markers[] = { { MODULES_END, "Modules end" }, { VMALLOC_START, "vmalloc() area" }, { VMALLOC_END, "vmalloc() end" }, - { FIXADDR_TOT_START, "Fixmap start" }, - { FIXADDR_TOP, "Fixmap end" }, { VMEMMAP_START, "vmemmap start" }, { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, { PCI_IO_START, "PCI I/O start" }, { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, { -1, NULL }, };Signature · 1 line
2.43.0.rc1.413.gea7ed67945-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 08/41] arm64: vmemmap: Avoid base2 order of struct page size to dimension region
In reply to Ard BiesheuvelQuoted 16 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The placement and size of the vmemmap region in the kernel virtual address space is currently derived from the base2 order of the size of a struct page. This makes for nicely aligned constants with lots of leading
0xfand trailing0x0digits, but given that the actual struct pages are indexed as an ordinary array, this resulting region is severely overdimensioned when the size of a struct page is just over a power of 2.This doesn't matter today, but once we enable 52-bit virtual addressing for 4k pages configurations, the vmemmap region may take up almost half of the upper VA region with the current struct page upper bound at 64 bytes. And once we enable KMSAN or other features that push the size of a struct page over 64 bytes, we will run out of VMALLOC space entirely.
Quoted 6 lines
So instead, let's derive the region size from the actual size of a struct page, and place the entire region 1 GB from the top of the VA space, where it still doesn't share any lower level translation table entries with the fixmap.
Signed-off-by Ard Biesheuvel <[email protected]>
This is nice, and largely addresses the fear I mentioned on the earlier patches move the PCI_IO_* and FIXMAP_* regions. I'd still like to ensure that we have assertions that those don't overlap with the VMEMMAP region, but assuming those get added to the earlier patches, this looks good as-is:
Mark.
Quoted 33 lines
--- arch/arm64/include/asm/memory.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)arch/arm64/include/asm/memory.h +4
-4@@ -30,8 +30,8 @@* keep a constant PAGE_OFFSET and "fallback" to using the higher end * of the VMEMMAP where 52-bit support is not available in hardware. */ -#define VMEMMAP_SHIFT (PAGE_SHIFT - STRUCT_PAGE_MAX_SHIFT) -#define VMEMMAP_SIZE ((_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) >> VMEMMAP_SHIFT) +#define VMEMMAP_RANGE (_PAGE_END(VA_BITS_MIN) - PAGE_OFFSET) +#define VMEMMAP_SIZE ((VMEMMAP_RANGE >> PAGE_SHIFT) * sizeof(struct page))/* * PAGE_OFFSET - the virtual address of the start of the linear map, at the @@ -47,8 +47,8 @@#define MODULES_END (MODULES_VADDR + MODULES_VSIZE)#define MODULES_VADDR (_PAGE_END(VA_BITS_MIN))#define MODULES_VSIZE (SZ_2G) -#define VMEMMAP_START (-(UL(1) << (VA_BITS - VMEMMAP_SHIFT))) -#define VMEMMAP_END (VMEMMAP_START + VMEMMAP_SIZE) +#define VMEMMAP_START (VMEMMAP_END - VMEMMAP_SIZE) +#define VMEMMAP_END (-UL(SZ_1G))#define PCI_IO_START (VMEMMAP_END + SZ_8M)#define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE)#define FIXADDR_TOP (-UL(SZ_8M))Signature · 1 line
2.43.0.rc1.413.gea7ed67945-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 06/41] arm64: ptdump: Allow all region boundaries to be defined at boot time
In reply to Ard BiesheuvelQuoted 81 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
Rework the way the
address_markersarray is populated so that we can tolerate values that are not compile time constants generally, rather than keeping track manually of the array indexes in question, and poking new values into them manually. This will be needed forVMALLOC_END, which will cease to be a compile time constant after a subsequent patch.Signed-off-by Ard Biesheuvel <[email protected]>--- arch/arm64/mm/ptdump.c | 54 ++++++++------------ 1 file changed, 22 insertions(+), 32 deletions(-)arch/arm64/mm/ptdump.c +11
-32@@ -26,34 +26,6 @@#include <asm/ptdump.h> -enum address_markers_idx { -PAGE_OFFSET_NR =0 , -PAGE_END_NR , -#if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) -KASAN_START_NR , -#endif -}; - -static struct addr_marker address_markers[] = { - {PAGE_OFFSET ,"Linear Mapping start" }, - {0 /* PAGE_END */ ,"Linear Mapping end" }, -#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - {0 /* KASAN_SHADOW_START */ ,"Kasan shadow start" }, - {KASAN_SHADOW_END ,"Kasan shadow end" }, -#endif - {MODULES_VADDR ,"Modules start" }, - {MODULES_END ,"Modules end" }, - {VMALLOC_START ,"vmalloc() area" }, - {VMALLOC_END ,"vmalloc() end" }, - {VMEMMAP_START ,"vmemmap start" }, - {VMEMMAP_START +VMEMMAP_SIZE ,"vmemmap end" }, - {PCI_IO_START ,"PCI I/O start" }, - {PCI_IO_END ,"PCI I/O end" }, - {FIXADDR_TOT_START ,"Fixmap start" }, - {FIXADDR_TOP ,"Fixmap end" }, - {-1 ,NULL }, -}; - #define pt_dump_seq_printf(m, fmt, args...) \ ({ \ if (m) \@@ -339,9 +311,8 @@ static void __init ptdump_initialize(void) pg_level[i].mask |= pg_level[i].bits[j].mask; } -static struct ptdump_info kernel_ptdump_info = { +static struct ptdump_info kernel_ptdump_info__ro_after_init = { .mm = &init_mm, - .markers = address_markers, .base_addr =PAGE_OFFSET , };@@ -375,10 +346,29 @@ void ptdump_check_wx(void)static int __init ptdump_init (void ) { - address_markers[PAGE_END_NR ].start_address =PAGE_END ; +struct addr_marker m[] = { + {PAGE_OFFSET ,"Linear Mapping start" }, + {PAGE_END ,"Linear Mapping end" }, #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) - address_markers[KASAN_START_NR ].start_address =KASAN_SHADOW_START ; + {KASAN_SHADOW_START ,"Kasan shadow start" }, + {KASAN_SHADOW_END ,"Kasan shadow end" }, #endif + {MODULES_VADDR ,"Modules start" }, + {MODULES_END ,"Modules end" }, + {VMALLOC_START ,"vmalloc() area" }, + {VMALLOC_END ,"vmalloc() end" }, + {VMEMMAP_START ,"vmemmap start" },
arch/arm64/mm/ptdump.c void ptdump_check_wx(void) · line 370
Quoted 10 lines
arch/arm64/mm/ptdump.c +10
-0+ { VMEMMAP_START + VMEMMAP_SIZE, "vmemmap end" }, + { PCI_IO_START, "PCI I/O start" }, + { PCI_IO_END, "PCI I/O end" }, + { FIXADDR_TOT_START, "Fixmap start" }, + { FIXADDR_TOP, "Fixmap end" }, + { -1, NULL }, + }; +static struct addr_marker address_markers[ARRAY_SIZE (m)]__ro_after_init ; + + kernel_ptdump_info.markers =memcpy (address_markers, m,sizeof (m));
That is a neat trick.
Mark.
Quoted 7 lines
arch/arm64/mm/ptdump.c +1
-0+ptdump_initialize ();ptdump_debugfs_register (&kernel_ptdump_info,"kernel_page_tables" );return 0 ;Signature · 1 line
2.43.0.rc1.413.gea7ed67945-goog
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Will Deacon
maintainerHi Ard,
I applied some of this:
Quoted 5 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
At the request of Catalin, this series was split off from my LPA2 series
[0] in order to make the changes a bit more manageable.
Quoted 8 lines
This series reorganizes the kernel VA space, and refactors/replaces the early mapping code so that:
- everything is done only once, in the appropriate order;
- everything is done with the MMU and caches enabled (*)
- everything is done from C code (notably, 100s of lines of incomprehensible asm code are removed fromhead.S).[...]
Applied the first three to arm64 (for-next/lpa2-prep):
[01/41] arm64: kernel: Disable latent_entropy GCC plugin in early C runtime
https://git.kernel.org/arm64/c/3dfdc2750c6c [02/41] arm64: mm: Take potential load offset into account when KASLR is off https://git.kernel.org/arm64/c/a22fc8e102dc [03/41] arm64: mm: get rid of kimage_vaddr global variable https://git.kernel.org/arm64/c/376f5a3bd7e2
Applied the idreg-override part to arm64 (for-next/early-idreg-overrides):
[14/41] arm64: idreg-override: Omit non-NULL checks for override pointer
https://git.kernel.org/arm64/c/cbc59c9a4e57 [15/41] arm64: idreg-override: Prepare for place relative reloc patching https://git.kernel.org/arm64/c/01fd29092a35 [16/41] arm64: idreg-override: Avoid parameq() and parameqn() https://git.kernel.org/arm64/c/dc3f5aae0638 [17/41] arm64: idreg-override: avoid strlen() to check for empty strings https://git.kernel.org/arm64/c/bcf1eed3f8a0 [18/41] arm64: idreg-override: Avoid sprintf() for simple string concatenation https://git.kernel.org/arm64/c/060260a6be47 [19/41] arm64: idreg-override: Avoid kstrtou64() to parse a single hex digit https://git.kernel.org/arm64/c/ea48626f8f0e [20/41] arm64/kernel: Move 'nokaslr' parsing out of early idreg code https://git.kernel.org/arm64/c/50f176175e96
Cheers,
Signature · 10 lines
Will https://fixes.arm64.dev https://next.arm64.dev https://will.arm64.dev _______________________________________________ linux-arm-kernel mailing list [email protected] http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Re: [PATCH v6 08/41] arm64: vmemmap: Avoid base2 order of struct page size to dimension region
In reply to Mark RutlandQuoted 23 lines · Mark Rutland
Quoted 21 lines · Ard Biesheuvel
From: Ard Biesheuvel <[email protected]>
The placement and size of the vmemmap region in the kernel virtual address space is currently derived from the base2 order of the size of a struct page. This makes for nicely aligned constants with lots of leading
0xfand trailing0x0digits, but given that the actual struct pages are indexed as an ordinary array, this resulting region is severely overdimensioned when the size of a struct page is just over a power of 2.This doesn't matter today, but once we enable 52-bit virtual addressing for 4k pages configurations, the vmemmap region may take up almost half of the upper VA region with the current struct page upper bound at 64 bytes. And once we enable KMSAN or other features that push the size of a struct page over 64 bytes, we will run out of VMALLOC space entirely.
So instead, let's derive the region size from the actual size of a struct page, and place the entire region 1 GB from the top of the VA space, where it still doesn't share any lower level translation table entries with the fixmap.
Quoted 9 lines
Quoted 1 line
Signed-off-by Ard Biesheuvel <[email protected]>This is nice, and largely addresses the fear I mentioned on the earlier patches move the PCI_IO_* and FIXMAP_* regions. I'd still like to ensure that we have assertions that those don't overlap with the VMEMMAP region, but assuming those get added to the earlier patches, this looks good as-is:
Acked-by Mark Rutland <[email protected]>
Thanks.
But actually, the only meaningful assert here is that the fixmap does not grow down into the PCI I/O region. Everything else is hard coded to unambiguous numeric constants, i.e.
#define VMEMMAP_END (-UL(SZ_1G)) #define PCI_IO_START (VMEMMAP_END + SZ_8M) #define PCI_IO_END (PCI_IO_START + PCI_IO_SIZE) #define FIXADDR_TOP (-UL(SZ_8M))
So I will add the hunk below to
arm64: mm: Move fixmap region above vmemmap region
but adding a static assert that a region that starts at VMEMMAP_END+8M does not overlap with the VMEMMAP region seems rather futile to me.
arch/arm64/mm/fixmap.c +3 -0
@@ -16,6 +16,9 @@#include <asm/pgalloc.h> #include <asm/tlbflush.h> +/* ensure that the fixmap region does not grow down into the PCI I/O region */ +static_assert (FIXADDR_TOT_START >PCI_IO_END ); +#define NR_BM_PTE_TABLES \ SPAN_NR_ENTRIES(FIXADDR_TOT_START, FIXADDR_TOP, PMD_SHIFT)#define NR_BM_PMD_TABLES \
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel