Skip to contentSkip to conversation

arm64 kernel text replication

PATCH New Russell King (Oracle) 3 participants 18 messages Jul 27, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover arm64 kernel text replication

This is the second posting of arm64 kernel text replication, rebased on v6.5-rc1.

There are a number of issues outstanding from the previous posting:
- Making it work with different numbers of page tables / page sizes etc
- Using 2-stage page tables to replicate (which needs to be done in EDK2/UEFI rather than the kernel.)

The changes since the last version have been:
- rox page protection consolidation patch has been merged
- Mark's VA-range sanity check patch is now also merged which allows my changes in that area to be dropped.
- reordered the first three patches

I am not expecting this to be merged, but it would be nice to get a few more eyes and comments on it - and maybe people trying it out.

 Documentation/admin-guide/kernel-parameters.txt |   5 +
 arch/arm64/Kconfig                              |  10 +-
 arch/arm64/include/asm/cacheflush.h             |   2 +
 arch/arm64/include/asm/ktext.h                  |  45 ++++++
 arch/arm64/include/asm/memory.h                 |  28 +++-
 arch/arm64/include/asm/mmu_context.h            |  12 +-
 arch/arm64/include/asm/pgtable.h                |  35 ++++-
 arch/arm64/include/asm/smp.h                    |   1 +
 arch/arm64/kernel/alternative.c                 |   4 +-
 arch/arm64/kernel/asm-offsets.c                 |   1 +
 arch/arm64/kernel/cpufeature.c                  |   2 +-
 arch/arm64/kernel/head.S                        |   3 +-
 arch/arm64/kernel/hibernate.c                   |   2 +-
 arch/arm64/kernel/kaslr.c                       |   1 +
 arch/arm64/kernel/patching.c                    |   7 +-
 arch/arm64/kernel/smp.c                         |   3 +
 arch/arm64/kernel/suspend.c                     |   3 +-
 arch/arm64/kernel/vmlinux.lds.S                 |   3 +
 arch/arm64/mm/Makefile                          |   2 +
 arch/arm64/mm/init.c                            |   3 +
 arch/arm64/mm/ktext.c                           | 198 ++++++++++++++++++++++++
 arch/arm64/mm/mmu.c                             |  73 +++++++--
 22 files changed, 416 insertions(+), 27 deletions(-)
 create mode 100644 arch/arm64/include/asm/ktext.h
 create mode 100644 arch/arm64/mm/ktext.c

Original message below.

Quoted 165 lines · Russell King (Oracle)

Problem
-------

NUMA systems have greater latency when accessing data and instructions across nodes, which can lead to a reduction in performance on CPU cores that mainly perform accesses beyond their local node.

Normally when an ARM64 system boots, the kernel will end up placed in memory, and each CPU core will have to fetch instructions and data from which ever NUMA node the kernel has been placed. This means that while executing kernel code, CPUs local to that node will run faster than CPUs in remote nodes.

The higher the latency to access remote NUMA node memory, the more the kernel performance suffers on those nodes.

If there is a local copy of the kernel text in each node's RAM, and each node runs the kernel using its local copy of the kernel text, then it stands to reason that the kernel will run faster due to fewer stalls while instructions are fetched from remote memory.

The question then arises how to achieve this.

Background
----------

An important issue to contend with is what happens when a thread migrates between nodes. Essentially, the thread's state (including instruction pointer) is saved to memory, and the scheduler on that CPU loads some other thread's state and that CPU resumes executing that new thread.

The CPU gaining the migrating thread loads the saved state, again including the instruction pointer, and the gaining CPU resumes fetching instructions at the virtual address where the original CPU left off.

The key point is that the virtual address is what matters here, and this gives us a way to implement kernel text replication fairly easily. At a practical level, all we need to do is to ensure that the virtual addresses which contain the kernel text point to a local copy of the that text.

This is exactly how this proposal of kernel text replication achieves the replication. We can go a little bit further and include most of the read-only data in this replication, as that will never be written to by the kernel (and thus remains constant.)

Solution
--------

So, what we need to achieve is:

1. multiple identical copies of the kernel text (and read-only data)
2. point the virtual mappings to the appropriate copy of kernel text for the NUMA node.

(1) is fairly easy to achieve - we just need to allocate some memory in the appropriate node and copy the parts of the kernel we want to replicate. However, we also need to deal with ARM64's kernel patching. There are two functions that patch the kernel text, __apply_alternatives() and aarch64_insn_patch_text_nosync(). Both of these need to to be modified to update all copies of the kernel text.

(2) is slightly harder.

Firstly, the aarch64 architecture has a very useful feature here - the kernel page tables are entirely separate from the user page tables. The hardware contains two page table pointers, one is used for user mappings, the other is used for kernel mappings.

Therefore, we only have one page table to be concerned with: the table which maps kernel space. We do not need to be concerned with each user processes page table.

The approach taken here is to ensure that the kernel is located in an area of kernel virtual address space covered by a level-0 page table entry which is not shared with any other user. We can then maintain separate per-node level-0 page tables for kernel space where the only difference between them is this level-0 page table entry.

This gives a couple of benefits. Firstly, when updates to the level-0 page table happen (e.g. when establishing new mappings) these updates can simply be copied to the other level-0 page tables provided it isn't for the kernel image. Secondly, we don't need complexity at lower levels of the page table code to figure out whether a level-1 or lower update needs to be propagated to other nodes.

The level-0 page table entry for the kernel can then be used to point at a node-unique set of level 1..N page tables to make the appropriate copy of the kernel text (and read-only data) into kernel space, while keeping the kernel read-write data shared between nodes.

Performance Analysis
--------------------

Needless to say, the performance results from kernel text replication are workload specific, but appear to show a gain of between 6% and 17% for database-centric like workloads. When combined with userspace awareness of NUMA, this can result in a gain of over 50%.

Problems
--------

There are a few areas that are a problem for kernel text replication:
1) As this series changes the kernel space virtual address space layout, it breaks KASAN - and I've zero knowledge of KASAN so I have no idea how to fix it. I would be grateful for input from KASAN folk for suggestions how to fix this.

2) KASLR can not be used with kernel text replication, since we need to place the kernel in its own L0 page table entry, not in vmalloc space. KASLR is disabled when support for kernel text replication is enabled.

3) Changing the kernel virtual address space layout also means that kaslr_offset() and kaslr_enabled() need to become macros rather than inline functions due to the use of PGDIR_SIZE in the calculation of KIMAGE_VADDR. Since asm/pgtable.h defines this constant, but asm/memory.h is included by asm/pgtable.h, having this symbol available would produce a circular include dependency, so I don't think there is any choice here.

4) read-only protection for replicated kernel images is not yet implemented.

Patch overview:

Patch 1 cleans up the rox page protection logic.
Patch 2 reoganises the kernel virtual address space layout (causing
problems (1 and 3).
Patch 3 provides a version of cpu_replace_ttbr1 that takes physical
addresses.
Patch 4 makes a needed cache flushing function visible. Patch 5 through 16 are the guts of kernel text replication. Patch 17 adds the Kconfig entry for it.

Further patches not included in this set add a Kconfig for the default state, a test module, and add code to verify the replicated kernel text matches the node 0 text after the kernel has completed most of its boot.

 Documentation/admin-guide/kernel-parameters.txt |   5 +
 arch/arm64/Kconfig                              |  10 +-
 arch/arm64/include/asm/cacheflush.h             |   2 +
 arch/arm64/include/asm/ktext.h                  |  45 ++++++
 arch/arm64/include/asm/memory.h                 |  26 ++--
 arch/arm64/include/asm/mmu_context.h            |  12 +-
 arch/arm64/include/asm/pgtable.h                |  35 ++++-
 arch/arm64/include/asm/smp.h                    |   1 +
 arch/arm64/kernel/alternative.c                 |   4 +-
 arch/arm64/kernel/asm-offsets.c                 |   1 +
 arch/arm64/kernel/cpufeature.c                  |   2 +-
 arch/arm64/kernel/head.S                        |   3 +-
 arch/arm64/kernel/hibernate.c                   |   2 +-
 arch/arm64/kernel/patching.c                    |   7 +-
 arch/arm64/kernel/smp.c                         |   3 +
 arch/arm64/kernel/suspend.c                     |   3 +-
 arch/arm64/kernel/vmlinux.lds.S                 |   3 +
 arch/arm64/mm/Makefile                          |   2 +
 arch/arm64/mm/init.c                            |   3 +
 arch/arm64/mm/ktext.c                           | 198 ++++++++++++++++++++++++
 arch/arm64/mm/mmu.c                             |  85 ++++++++--
 21 files changed, 413 insertions(+), 39 deletions(-)
 create mode 100644 arch/arm64/include/asm/ktext.h
 create mode 100644 arch/arm64/mm/ktext.c
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 80Mbps down 10Mbps up. Decent connectivity at last!
#1 lore raw

PATCH v2 1/16 arm64: provide cpu_replace_ttbr1_phys()

Provide a version of cpu_replace_ttbr1_phys() which operates using a physical address rather than the virtual address of the page tables.

---
 arch/arm64/include/asm/mmu_context.h | 12 +++++++++---
 1 file changed, 9 insertions(+), 3 deletions(-)
arch/arm64/include/asm/mmu_context.h +9 -3
index a6fb325424e7..42085b2b1d92 100644
@@ -152,7 +152,7 @@ static inline void cpu_install_ttbr0(phys_addr_t ttbr0, unsigned long t0sz)
  * Atomically replaces the active TTBR1_EL1 PGD with a new VA-compatible PGD,
  * avoiding the possibility of conflicting TLB entries being allocated.
  */
-static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap)
+static inline void cpu_replace_ttbr1_phys(phys_addr_t pgd_phys, pgd_t *idmap)
 {
 	typedef void (ttbr_replace_func)(phys_addr_t);
 	extern ttbr_replace_func idmap_cpu_replace_ttbr1;
@@ -160,9 +160,10 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap)
 	unsigned long daif;
 
 	/* phys_to_ttbr() zeros lower 2 bits of ttbr with 52-bit PA */
-	phys_addr_t ttbr1 = phys_to_ttbr(virt_to_phys(pgdp));
+	phys_addr_t ttbr1 = phys_to_ttbr(pgd_phys);
 
-	if (system_supports_cnp() && !WARN_ON(pgdp != lm_alias(swapper_pg_dir))) {
+	if (system_supports_cnp() &&
+	    !WARN_ON(pgd_phys != virt_to_phys(lm_alias(swapper_pg_dir)))) {
 		/*
 		 * cpu_replace_ttbr1() is used when there's a boot CPU
 		 * up (i.e. cpufeature framework is not up yet) and
@@ -189,6 +190,11 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap)
 	cpu_uninstall_idmap();
 }
 
+static inline void __nocfi cpu_replace_ttbr1(pgd_t *pgdp, pgd_t *idmap)
+{
+	cpu_replace_ttbr1_phys(virt_to_phys(pgdp), idmap);
+}
+
 /*
  * It would be nice to return ASIDs back to the allocator, but unfortunately
  * that introduces a race with a generation rollover where we could erroneously
Signature · 1 line
2.30.2
#2 lore raw

PATCH v2 2/16 arm64: make clean_dcache_range_nopatch() visible

In reply to Russell King (Oracle)

When we hook into the kernel text patching code, we will need to call clean_dcache_range_nopatch() to ensure that the patching of the replicated kernel text is properly visible to other CPUs. Make this function available to the replication code.

---
 arch/arm64/include/asm/cacheflush.h | 2 ++
 arch/arm64/kernel/alternative.c     | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
arch/arm64/include/asm/cacheflush.h +2 -0
index 37185e978aeb..ac9ad56d5212 100644
@@ -104,6 +104,8 @@ static inline void flush_icache_range(unsigned long start, unsigned long end)
 }
 #define flush_icache_range flush_icache_range
 
+void clean_dcache_range_nopatch(u64 start, u64 end);
+
 /*
  * Copy user data from/to a page which is mapped into a different
  * processes address space.  Really, we want to allow our "user
arch/arm64/kernel/alternative.c +1 -1
index 8ff6610af496..ea3f4104771d 100644
@@ -121,7 +121,7 @@ static noinstr void patch_alternative(struct alt_instr *alt,
  * accidentally call into the cache.S code, which is patched by us at
  * runtime.
  */
-static noinstr void clean_dcache_range_nopatch(u64 start, u64 end)
+noinstr void clean_dcache_range_nopatch(u64 start, u64 end)
 {
 	u64 cur, d_size, ctr_el0;
 
Signature · 1 line
2.30.2
#3 lore raw

PATCH v2 3/16 arm64: place kernel in its own L0 page table entry

In reply to Russell King (Oracle)

Kernel text replication needs to maintain separate per-node page tables for the kernel text. In order to do this without affecting other kernel memory mappings, placing the kernel such that it does not share a L0 page table entry with any other mapping is desirable.

Prior to this commit, the layout without KASLR was:

+----------+
|  vmalloc |
+----------+
|  Kernel  |
+----------+ MODULES_END, VMALLOC_START, KIMAGE_VADDR =
|  Modules |                 MODULES_VADDR + MODULES_VSIZE
+----------+ MODULES_VADDR = _PAGE_END(VA_BITS_MIN)
| VA space |
+----------+ 0

This becomes:

+----------+
|  vmalloc |
+----------+ VMALLOC_START = MODULES_END + PGDIR_SIZE
|  Kernel  |
+----------+ MODULES_END, KIMAGE_VADDR = _PAGE_END(VA_BITS_MIN) +
|  Modules |    max(PGDIR_SIZE, MODULES_VSIZE)
+----------+ MODULES_VADDR = MODULES_END - MODULES_VSIZE
| VA space |
+----------+ 0

This assumes MODULES_VSIZE (128M) <= PGDIR_SIZE.

One side effect of this change is that KIMAGE_VADDR's definition now includes PGDIR_SIZE (to leave room for the modules) but this is not defined when asm/memory.h is included. This means KIMAGE_VADDR can not be used in inline functions within this file, so we convert kaslr_offset() and kaslr_enabled() to be macros instead.

---
 arch/arm64/include/asm/memory.h  | 28 +++++++++++++++++++++-------
 arch/arm64/include/asm/pgtable.h |  2 +-
 arch/arm64/kernel/kaslr.c        |  1 +
 3 files changed, 23 insertions(+), 8 deletions(-)
arch/arm64/include/asm/memory.h +21 -7
index fde4186cc387..9410ec4e4207 100644
@@ -43,9 +43,26 @@
 #define VA_BITS			(CONFIG_ARM64_VA_BITS)
 #define _PAGE_OFFSET(va)	(-(UL(1) << (va)))
 #define PAGE_OFFSET		(_PAGE_OFFSET(VA_BITS))
-#define KIMAGE_VADDR		(MODULES_END)
-#define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)
-#define MODULES_VADDR		(_PAGE_END(VA_BITS_MIN))
+
+/*
+ * Setting KIMAGE_VADDR has got a lot harder, ideally we'd like to use
+ * min(PGDIR_SIZE, MODULES_VSIZE) but this can't work because this is used
+ * both in assembly as C, where it causes problems. min_t() solves the
+ * C problems but can't be used in assembly.
+ * CONFIG_ARM64_4K_PAGES, PGDIR_SIZE is 2M, 1G, 512G
+ * CONFIG_ARM64_16K_PAGES, PGDIR_SIZE is 32M, 64G or 128T
+ * CONFIG_ARM64_64K_PAGES, PGDIR_SIZE is 512M or 4T
+ */
+#if (CONFIG_ARM64_4K_PAGES && CONFIG_PGTABLE_LEVELS < 4) || \
+    (CONFIG_ARM64_16K_PAGES && CONFIG_PGTABLE_LEVELS < 3) || \
+    (CONFIG_ARM64_64K_PAGES && CONFIG_PGTABLE_LEVELS < 2)
+#define KIMAGE_OFFSET		MODULES_VSIZE
+#else
+#define KIMAGE_OFFSET		PGDIR_SIZE
+#endif
+#define KIMAGE_VADDR		(_PAGE_END(VA_BITS_MIN) + KIMAGE_OFFSET)
+#define MODULES_END		(KIMAGE_VADDR)
+#define MODULES_VADDR		(MODULES_END - MODULES_VSIZE)
 #define MODULES_VSIZE		(SZ_2G)
 #define VMEMMAP_START		(-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))
 #define VMEMMAP_END		(VMEMMAP_START + VMEMMAP_SIZE)
@@ -199,10 +216,7 @@ extern u64			kimage_vaddr;
 /* the offset between the kernel virtual and physical mappings */
 extern u64			kimage_voffset;
 
-static inline unsigned long kaslr_offset(void)
-{
-	return kimage_vaddr - KIMAGE_VADDR;
-}
+#define kaslr_offset()	((unsigned long)(kimage_vaddr - KIMAGE_VADDR))
 
 #ifdef CONFIG_RANDOMIZE_BASE
 void kaslr_init(void);
arch/arm64/include/asm/pgtable.h +1 -1
index 0bd18de9fd97..cb526e69299d 100644
@@ -21,7 +21,7 @@
  * VMALLOC_END: extends to the available space below vmemmap, PCI I/O space
  *	and fixed mappings
  */
-#define VMALLOC_START		(MODULES_END)
+#define VMALLOC_START		(MODULES_END + PGDIR_SIZE)
 #define VMALLOC_END		(VMEMMAP_START - SZ_256M)
 
 #define vmemmap			((struct page *)VMEMMAP_START - (memstart_addr >> PAGE_SHIFT))
arch/arm64/kernel/kaslr.c +1 -0
index 94a269cd1f07..6ffea2ce1a11 100644
@@ -9,6 +9,7 @@
 
 #include <asm/cpufeature.h>
 #include <asm/memory.h>
+#include <asm/pgtable.h>
 
 u16 __initdata memstart_offset_seed;
 
Signature · 1 line
2.30.2
#4 lore raw

PATCH v2 4/16 arm64: text replication: add init function

In reply to Russell King (Oracle)

A simple patch that adds an empty function for kernel text replication initialisation and hooks it into the initialisation path.

---
 arch/arm64/include/asm/ktext.h | 20 ++++++++++++++++++++
 arch/arm64/mm/Makefile         |  2 ++
 arch/arm64/mm/init.c           |  3 +++
 arch/arm64/mm/ktext.c          |  8 ++++++++
 4 files changed, 33 insertions(+)
 create mode 100644 arch/arm64/include/asm/ktext.h
 create mode 100644 arch/arm64/mm/ktext.c
arch/arm64/include/asm/ktext.h new file +20 -0
new file mode 100644
index 000000000000..1a5f7452a3bf
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Copyright (C) 2022, Oracle and/or its affiliates.
+ */
+#ifndef ASM_KTEXT_H
+#define ASM_KTEXT_H
+
+#ifdef CONFIG_REPLICATE_KTEXT
+
+void ktext_replication_init(void);
+
+#else
+
+static inline void ktext_replication_init(void)
+{
+}
+
+#endif
+
+#endif
arch/arm64/mm/Makefile +2 -0
index dbd1bc95967d..41e705027c57 100644
@@ -14,3 +14,5 @@ KASAN_SANITIZE_physaddr.o	+= n
 
 obj-$(CONFIG_KASAN)		+= kasan_init.o
 KASAN_SANITIZE_kasan_init.o	:= n
+
+obj-$(CONFIG_REPLICATE_KTEXT)	+= ktext.o
arch/arm64/mm/init.c +3 -0
index d31c3a9290c5..df90738fd6bb 100644
@@ -36,6 +36,7 @@
 #include <asm/fixmap.h>
 #include <asm/kasan.h>
 #include <asm/kernel-pgtable.h>
+#include <asm/ktext.h>
 #include <asm/kvm_host.h>
 #include <asm/memory.h>
 #include <asm/numa.h>
@@ -425,6 +426,8 @@ void __init bootmem_init(void)
 
 	arch_numa_init();
 
+	ktext_replication_init();
+
 	/*
 	 * must be done after arch_numa_init() which calls numa_init() to
 	 * initialize node_online_map that gets used in hugetlb_cma_reserve()
arch/arm64/mm/ktext.c new file +8 -0
new file mode 100644
index 000000000000..3a8d37c9abc4
@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2022, Oracle and/or its affiliates.
+ */
+
+void __init ktext_replication_init(void)
+{
+}
Signature · 1 line
2.30.2
#5 lore raw

PATCH v2 5/16 arm64: text replication: add sanity checks

In reply to Russell King (Oracle)

The kernel text and modules must be in separate L0 page table entries.

---
 arch/arm64/mm/ktext.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
arch/arm64/mm/ktext.c +21 -0
index 3a8d37c9abc4..901f159c65e6 100644
@@ -3,6 +3,27 @@
  * Copyright (C) 2022, Oracle and/or its affiliates.
  */
 
+#include <linux/kernel.h>
+#include <linux/pgtable.h>
+
+#include <asm/ktext.h>
+#include <asm/memory.h>
+
 void __init ktext_replication_init(void)
 {
+	int kidx = pgd_index((phys_addr_t)KERNEL_START);
+
+	/*
+	 * If we've messed up and the kernel shares a L0 entry with the
+	 * module or vmalloc area, then don't even attempt to use text
+	 * replication.
+	 */
+	if (pgd_index(MODULES_VADDR) == kidx) {
+		pr_warn("Kernel is located in the same L0 index as modules - text replication disabled\n");
+		return;
+	}
+	if (pgd_index(VMALLOC_START) == kidx) {
+		pr_warn("Kernel is located in the same L0 index as vmalloc - text replication disabled\n");
+		return;
+	}
 }
Signature · 1 line
2.30.2
#6 lore raw

PATCH v2 6/16 arm64: text replication: copy initial kernel text

In reply to Russell King (Oracle)

Allocate memory on the appropriate node for the per-node copies of the kernel text, and copy the kernel text to that memory. Clean and invalidate the caches to the point of unification so that the copied text is correctly visible to the target node.

---
 arch/arm64/mm/ktext.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)
arch/arm64/mm/ktext.c +21 -0
index 901f159c65e6..4c803b89fcfe 100644
@@ -4,14 +4,23 @@
  */
 
 #include <linux/kernel.h>
+#include <linux/memblock.h>
+#include <linux/numa.h>
 #include <linux/pgtable.h>
+#include <linux/string.h>
 
+#include <asm/cacheflush.h>
 #include <asm/ktext.h>
 #include <asm/memory.h>
 
+static void *kernel_texts[MAX_NUMNODES];
+
+/* Allocate memory for the replicated kernel texts. */
 void __init ktext_replication_init(void)
 {
+	size_t size = _etext - _stext;
 	int kidx = pgd_index((phys_addr_t)KERNEL_START);
+	int nid;
 
 	/*
 	 * If we've messed up and the kernel shares a L0 entry with the
@@ -26,4 +35,16 @@ void __init ktext_replication_init(void)
 		pr_warn("Kernel is located in the same L0 index as vmalloc - text replication disabled\n");
 		return;
 	}
+
+	for_each_node(nid) {
+		/* Nothing to do for node 0 */
+		if (!nid)
+			continue;
+
+		/* Allocate and copy initial kernel text for this node */
+		kernel_texts[nid] = memblock_alloc_node(size, PAGE_SIZE, nid);
+		memcpy(kernel_texts[nid], _stext, size);
+		caches_clean_inval_pou((u64)kernel_texts[nid],
+				       (u64)kernel_texts[nid] + size);
+	}
 }
Signature · 1 line
2.30.2
#7 lore raw

PATCH v2 7/16 arm64: text replication: add node text patching

In reply to Russell King (Oracle)

Add support for text patching on our replicated texts.

---
 arch/arm64/include/asm/ktext.h  | 12 +++++++
 arch/arm64/kernel/alternative.c |  2 ++
 arch/arm64/kernel/patching.c    |  7 +++-
 arch/arm64/mm/ktext.c           | 58 +++++++++++++++++++++++++++++++++
 4 files changed, 78 insertions(+), 1 deletion(-)
arch/arm64/include/asm/ktext.h +12 -0
index 1a5f7452a3bf..289e11289c06 100644
@@ -5,9 +5,13 @@
 #ifndef ASM_KTEXT_H
 #define ASM_KTEXT_H
 
+#include <linux/kprobes.h>
+
 #ifdef CONFIG_REPLICATE_KTEXT
 
 void ktext_replication_init(void);
+void __kprobes ktext_replication_patch(u32 *tp,  __le32 insn);
+void ktext_replication_patch_alternative(__le32 *src, int nr_inst);
 
 #else
 
@@ -15,6 +19,14 @@ static inline void ktext_replication_init(void)
 {
 }
 
+static inline void __kprobes ktext_replication_patch(u32 *tp,  __le32 insn)
+{
+}
+
+static inline void ktext_replication_patch_alternative(__le32 *src, int nr_inst)
+{
+}
+
 #endif
 
 #endif
arch/arm64/kernel/alternative.c +2 -0
index ea3f4104771d..6f17e2b4e1c3 100644
@@ -15,6 +15,7 @@
 #include <asm/alternative.h>
 #include <asm/cpufeature.h>
 #include <asm/insn.h>
+#include <asm/ktext.h>
 #include <asm/module.h>
 #include <asm/sections.h>
 #include <asm/vdso.h>
@@ -174,6 +175,7 @@ static void __apply_alternatives(const struct alt_region *region,
 		alt_cb(alt, origptr, updptr, nr_inst);
 
 		if (!is_module) {
+			ktext_replication_patch_alternative(updptr, nr_inst);
 			clean_dcache_range_nopatch((u64)origptr,
 						   (u64)(origptr + nr_inst));
 		}
arch/arm64/kernel/patching.c +6 -1
index b4835f6d594b..627fff6ddda2 100644
@@ -10,6 +10,7 @@
 #include <asm/fixmap.h>
 #include <asm/insn.h>
 #include <asm/kprobes.h>
+#include <asm/ktext.h>
 #include <asm/patching.h>
 #include <asm/sections.h>
 
@@ -115,9 +116,13 @@ int __kprobes aarch64_insn_patch_text_nosync(void *addr, u32 insn)
 		return -EINVAL;
 
 	ret = aarch64_insn_write(tp, insn);
-	if (ret == 0)
+	if (ret == 0) {
+		/* Also patch the other nodes */
+		ktext_replication_patch(tp, cpu_to_le32(insn));
+
 		caches_clean_inval_pou((uintptr_t)tp,
 				     (uintptr_t)tp + AARCH64_INSN_SIZE);
+	}
 
 	return ret;
 }
arch/arm64/mm/ktext.c +58 -0

72 lines not on this page · raw message

Signature · 1 line
2.30.2
#8 lore raw

PATCH v2 8/16 arm64: text replication: add node 0 page table definitions

In reply to Russell King (Oracle)

Add a struct definition for the level zero page table group (the optional trampoline page tables, reserved page tables, and swapper page tables).

Add a symbol and extern declaration for the node 0 page table group.

Add an array of pointers to per-node page tables, which will default to using the node 0 page table group.

---
 arch/arm64/include/asm/pgtable.h | 14 ++++++++++++++
 arch/arm64/kernel/vmlinux.lds.S  |  3 +++
 arch/arm64/mm/ktext.c            |  4 ++++
 3 files changed, 21 insertions(+)
arch/arm64/include/asm/pgtable.h +14 -0
index cb526e69299d..1e72067d1e9e 100644
@@ -615,6 +615,20 @@ extern pgd_t idmap_pg_dir[PTRS_PER_PGD];
 extern pgd_t tramp_pg_dir[PTRS_PER_PGD];
 extern pgd_t reserved_pg_dir[PTRS_PER_PGD];
 
+struct pgtables {
+#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
+	pgd_t tramp_pg_dir[PTRS_PER_PGD];
+#endif
+	pgd_t reserved_pg_dir[PTRS_PER_PGD];
+	pgd_t swapper_pg_dir[PTRS_PER_PGD];
+};
+
+extern struct pgtables pgtable_node0;
+
+#ifdef CONFIG_REPLICATE_KTEXT
+extern struct pgtables *pgtables[MAX_NUMNODES];
+#endif
+
 extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd);
 
 static inline bool in_swapper_pgdir(void *addr)
arch/arm64/kernel/vmlinux.lds.S +3 -0
index 3cd7e76cc562..d3c7ed76adbf 100644
@@ -212,6 +212,9 @@ SECTIONS
 	idmap_pg_dir = .;
 	. += PAGE_SIZE;
 
+	/* pgtable struct - covers the tramp, reserved and swapper pgdirs */
+	pgtable_node0 = .;
+
 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0
 	tramp_pg_dir = .;
 	. += PAGE_SIZE;
arch/arm64/mm/ktext.c +4 -0
index 04b5ceddae4e..48d7943d6907 100644
@@ -15,6 +15,10 @@
 #include <asm/ktext.h>
 #include <asm/memory.h>
 
+struct pgtables *pgtables[MAX_NUMNODES] = {
+	[0 ... MAX_NUMNODES - 1] = &pgtable_node0,
+};
+
 static void *kernel_texts[MAX_NUMNODES];
 
 void __kprobes ktext_replication_patch(u32 *tp, __le32 insn)
Signature · 1 line
2.30.2
#9 lore raw

PATCH v2 9/16 arm64: text replication: add swapper page directory helpers

In reply to Russell King (Oracle)

Add a series of helpers for the swapper page directories - a set which return those for the calling CPU, and those which take the NUMA node number.

---
 arch/arm64/include/asm/pgtable.h | 19 +++++++++++++++++++
 arch/arm64/kernel/hibernate.c    |  2 +-
 arch/arm64/mm/ktext.c            | 20 ++++++++++++++++++++
 3 files changed, 40 insertions(+), 1 deletion(-)
arch/arm64/include/asm/pgtable.h +19 -0
index 1e72067d1e9e..5cfff64e4944 100644
@@ -627,6 +627,25 @@ extern struct pgtables pgtable_node0;
 
 #ifdef CONFIG_REPLICATE_KTEXT
 extern struct pgtables *pgtables[MAX_NUMNODES];
+
+pgd_t *swapper_pg_dir_node(void);
+phys_addr_t __swapper_pg_dir_node_phys(int nid);
+phys_addr_t swapper_pg_dir_node_phys(void);
+#else
+static inline pgd_t *swapper_pg_dir_node(void)
+{
+	return swapper_pg_dir;
+}
+
+static inline phys_addr_t __swapper_pg_dir_node_phys(int nid)
+{
+	return __pa_symbol(swapper_pg_dir);
+}
+
+static inline phys_addr_t swapper_pg_dir_node_phys(void)
+{
+	return __pa_symbol(swapper_pg_dir);
+}
 #endif
 
 extern void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd);
arch/arm64/kernel/hibernate.c +1 -1
index 02870beb271e..be69515da802 100644
@@ -113,7 +113,7 @@ int arch_hibernation_header_save(void *addr, unsigned int max_size)
 		return -EOVERFLOW;
 
 	arch_hdr_invariants(&hdr->invariants);
-	hdr->ttbr1_el1		= __pa_symbol(swapper_pg_dir);
+	hdr->ttbr1_el1		= swapper_pg_dir_node_phys();
 	hdr->reenter_kernel	= _cpu_resume;
 
 	/* We can't use __hyp_get_vectors() because kvm may still be loaded */
arch/arm64/mm/ktext.c +20 -0
index 48d7943d6907..7b9a1f1b12a1 100644
@@ -21,6 +21,26 @@ struct pgtables *pgtables[MAX_NUMNODES] = {
 
 static void *kernel_texts[MAX_NUMNODES];
 
+static pgd_t *__swapper_pg_dir_node(int nid)
+{
+	return pgtables[nid]->swapper_pg_dir;
+}
+
+pgd_t *swapper_pg_dir_node(void)
+{
+	return __swapper_pg_dir_node(numa_node_id());
+}
+
+phys_addr_t __swapper_pg_dir_node_phys(int nid)
+{
+	return __pa(__swapper_pg_dir_node(nid));
+}
+
+phys_addr_t swapper_pg_dir_node_phys(void)
+{
+	return __swapper_pg_dir_node_phys(numa_node_id());
+}
+
 void __kprobes ktext_replication_patch(u32 *tp, __le32 insn)
 {
 	unsigned long offset;
Signature · 1 line
2.30.2
#10 lore raw

PATCH v2 10/16 arm64: text replication: create per-node kernel page tables

In reply to Russell King (Oracle)

Allocate the level 0 page tables for the per-node kernel text replication, but copy all level 0 table entries from the NUMA node 0 table. Therefore, for the time being, each node's level 0 page tables will contain identical entries, and thus other nodes will continue to use the node 0 kernel text.

Since the level 0 page tables can be updated at runtime to add entries for vmalloc and module space, propagate these updates to the other swapper page tables. The exception is if we see an update for the level 0 entry which points to the kernel mapping.

We also need to setup a copy of the trampoline page tables as well, as the assembly code relies on the two page tables being a fixed offset apart.

---
 arch/arm64/include/asm/ktext.h | 12 ++++++++++
 arch/arm64/mm/ktext.c          | 42 +++++++++++++++++++++++++++++++++-
 arch/arm64/mm/mmu.c            |  5 ++++
 3 files changed, 58 insertions(+), 1 deletion(-)
arch/arm64/include/asm/ktext.h +12 -0
index 289e11289c06..386f9812d3c1 100644
@@ -7,11 +7,15 @@
 
 #include <linux/kprobes.h>
 
+#include <asm/pgtable-types.h>
+
 #ifdef CONFIG_REPLICATE_KTEXT
 
 void ktext_replication_init(void);
 void __kprobes ktext_replication_patch(u32 *tp,  __le32 insn);
 void ktext_replication_patch_alternative(__le32 *src, int nr_inst);
+void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd);
+void ktext_replication_init_tramp(void);
 
 #else
 
@@ -27,6 +31,14 @@ static inline void ktext_replication_patch_alternative(__le32 *src, int nr_inst)
 {
 }
 
+static inline void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd)
+{
+}
+
+static inline void ktext_replication_init_tramp(void)
+{
+}
+
 #endif
 
 #endif
arch/arm64/mm/ktext.c +41 -1

59 lines not on this page · raw message

arch/arm64/mm/mmu.c +5 -0
index 95d360805f8a..4ffa2650afd6 100644
@@ -31,6 +31,7 @@
 #include <asm/fixmap.h>
 #include <asm/kasan.h>
 #include <asm/kernel-pgtable.h>
+#include <asm/ktext.h>
 #include <asm/sections.h>
 #include <asm/setup.h>
 #include <linux/sizes.h>
@@ -81,6 +82,7 @@ void set_swapper_pgd(pgd_t *pgdp, pgd_t pgd)
 	pgd_t *fixmap_pgdp;
 
 	spin_lock(&swapper_pgdir_lock);
+	ktext_replication_set_swapper_pgd(pgdp, pgd);
 	fixmap_pgdp = pgd_set_fixmap(__pa_symbol(pgdp));
 	WRITE_ONCE(*fixmap_pgdp, pgd);
 	/*
@@ -694,6 +696,9 @@ static int __init map_entry_trampoline(void)
 		__set_fixmap(FIX_ENTRY_TRAMP_TEXT1 - i,
 			     pa_start + i * PAGE_SIZE, PAGE_KERNEL_RO);
 
+	/* Copy trampoline page tables to other numa nodes */
+	ktext_replication_init_tramp();
+
 	return 0;
 }
 core_initcall(map_entry_trampoline);
Signature · 1 line
2.30.2
#11 lore raw

PATCH v2 11/16 arm64: text replication: boot secondary CPUs with appropriate TTBR1

In reply to Russell King (Oracle)

Arrange for secondary CPUs to boot with TTBR1 pointing at the appropriate per-node copy of the kernel page tables for the CPUs NUMA node.

---
 arch/arm64/include/asm/smp.h    | 1 +
 arch/arm64/kernel/asm-offsets.c | 1 +
 arch/arm64/kernel/head.S        | 3 ++-
 arch/arm64/kernel/smp.c         | 3 +++
 4 files changed, 7 insertions(+), 1 deletion(-)
arch/arm64/include/asm/smp.h +1 -0
index 9b31e6d0da17..9635e473969b 100644
@@ -79,6 +79,7 @@ asmlinkage void secondary_start_kernel(void);
 struct secondary_data {
 	struct task_struct *task;
 	long status;
+	phys_addr_t ttbr1;
 };
 
 extern struct secondary_data secondary_data;
arch/arm64/kernel/asm-offsets.c +1 -0
index 5ff1942b04fc..ce9d265bc099 100644
@@ -121,6 +121,7 @@ int main(void)
   DEFINE(IRQ_CPUSTAT_SOFTIRQ_PENDING, offsetof(irq_cpustat_t, __softirq_pending));
   BLANK();
   DEFINE(CPU_BOOT_TASK,		offsetof(struct secondary_data, task));
+  DEFINE(CPU_BOOT_TTBR1,	offsetof(struct secondary_data, ttbr1));
   BLANK();
   DEFINE(FTR_OVR_VAL_OFFSET,	offsetof(struct arm64_ftr_override, val));
   DEFINE(FTR_OVR_MASK_OFFSET,	offsetof(struct arm64_ftr_override, mask));
arch/arm64/kernel/head.S +2 -1
index 757a0de07f91..52643080a2c2 100644
@@ -648,7 +648,8 @@ SYM_FUNC_START_LOCAL(secondary_startup)
 	ldr_l	x0, vabits_actual
 #endif
 	bl	__cpu_setup			// initialise processor
-	adrp	x1, swapper_pg_dir
+	adr_l	x1, secondary_data
+	ldr	x1, [x1, #CPU_BOOT_TTBR1]
 	adrp	x2, idmap_pg_dir
 	bl	__enable_mmu
 	ldr	x8, =__secondary_switched
arch/arm64/kernel/smp.c +3 -0
index edd63894d61e..95b649eb5999 100644
@@ -119,6 +119,9 @@ int __cpu_up(unsigned int cpu, struct task_struct *idle)
 	 * page tables.
 	 */
 	secondary_data.task = idle;
+	secondary_data.ttbr1 = __swapper_pg_dir_node_phys(cpu_to_node(cpu));
+	dcache_clean_poc((uintptr_t)&secondary_data,
+			 (uintptr_t)&secondary_data + sizeof(secondary_data));
 	update_cpu_boot_status(CPU_MMU_OFF);
 
 	/* Now bring the CPU into our world */
Signature · 1 line
2.30.2
#12 lore raw

PATCH v2 12/16 arm64: text replication: update cnp support

In reply to Russell King (Oracle)

Add changes for CNP (Common Not Private) support of kernel text replication. Although text replication has only been tested on dual-socket Ampere A1 systems, provided the different NUMA nodes are not part of the same inner shareable domain, CNP should not be a problem.

---
 arch/arm64/include/asm/mmu_context.h | 2 +-
 arch/arm64/kernel/cpufeature.c       | 2 +-
 arch/arm64/kernel/suspend.c          | 3 ++-
 3 files changed, 4 insertions(+), 3 deletions(-)
arch/arm64/include/asm/mmu_context.h +1 -1
index 42085b2b1d92..36f6b1b65ae5 100644
@@ -163,7 +163,7 @@ static inline void cpu_replace_ttbr1_phys(phys_addr_t pgd_phys, pgd_t *idmap)
 	phys_addr_t ttbr1 = phys_to_ttbr(pgd_phys);
 
 	if (system_supports_cnp() &&
-	    !WARN_ON(pgd_phys != virt_to_phys(lm_alias(swapper_pg_dir)))) {
+	    !WARN_ON(pgd_phys != swapper_pg_dir_node_phys())) {
 		/*
 		 * cpu_replace_ttbr1() is used when there's a boot CPU
 		 * up (i.e. cpufeature framework is not up yet) and
arch/arm64/kernel/cpufeature.c +1 -1
index f9d456fe132d..e7dc7ec849ab 100644
@@ -3417,7 +3417,7 @@ subsys_initcall_sync(init_32bit_el0_mask);
 
 static void __maybe_unused cpu_enable_cnp(struct arm64_cpu_capabilities const *cap)
 {
-	cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir);
+	cpu_replace_ttbr1_phys(swapper_pg_dir_node_phys(), idmap_pg_dir);
 }
 
 /*
arch/arm64/kernel/suspend.c +2 -1
index 0fbdf5fe64d8..49fa80bafd6d 100644
@@ -55,7 +55,8 @@ void notrace __cpu_suspend_exit(void)
 
 	/* Restore CnP bit in TTBR1_EL1 */
 	if (system_supports_cnp())
-		cpu_replace_ttbr1(lm_alias(swapper_pg_dir), idmap_pg_dir);
+		cpu_replace_ttbr1_phys(swapper_pg_dir_node_phys(),
+				       idmap_pg_dir);
 
 	/*
 	 * PSTATE was not saved over suspend/resume, re-enable any detected
Signature · 1 line
2.30.2
#13 lore raw

PATCH v2 13/16 arm64: text replication: setup page tables for copied kernel

In reply to Russell King (Oracle)

Setup page table entries in each non-boot NUMA node page table to point at each node's own copy of the kernel text. This switches each node to use its own unique copy of the kernel text.

---
 arch/arm64/include/asm/ktext.h |  1 +
 arch/arm64/mm/ktext.c          |  8 +++++
 arch/arm64/mm/mmu.c            | 53 ++++++++++++++++++++++++++++------
 3 files changed, 53 insertions(+), 9 deletions(-)
arch/arm64/include/asm/ktext.h +1 -0
index 386f9812d3c1..6ece59ca90a2 100644
@@ -16,6 +16,7 @@ void __kprobes ktext_replication_patch(u32 *tp,  __le32 insn);
 void ktext_replication_patch_alternative(__le32 *src, int nr_inst);
 void ktext_replication_set_swapper_pgd(pgd_t *pgdp, pgd_t pgd);
 void ktext_replication_init_tramp(void);
+void create_kernel_nid_map(pgd_t *pgdp, void *ktext);
 
 #else
 
arch/arm64/mm/ktext.c +8 -0
index 9efd21eb3308..6692759e78a8 100644
@@ -136,6 +136,14 @@ void __init ktext_replication_init(void)
 
 		/* Copy initial swapper page directory */
 		memcpy(pgtables[nid]->swapper_pg_dir, swapper_pg_dir, PGD_SIZE);
+
+		/* Clear the kernel mapping */
+		memset(&pgtables[nid]->swapper_pg_dir[kidx], 0,
+		       sizeof(pgtables[nid]->swapper_pg_dir[kidx]));
+
+		/* Create kernel mapping pointing at our local copy */
+		create_kernel_nid_map(pgtables[nid]->swapper_pg_dir,
+				      kernel_texts[nid]);
 	}
 }
 
arch/arm64/mm/mmu.c +44 -9

78 lines not on this page · raw message

Signature · 1 line
2.30.2
#14 lore raw

PATCH v2 14/16 arm64: text replication: include most of read-only data as well

In reply to Russell King (Oracle)

Include as much of the read-only data in the replication as we can without needing to move away from the generic RO_DATA() macro in the linker script.

Unfortunately, the read-only data section is immedaitely followed by the read-only after init data with no page alignment, which means we can't have separate mappings for the read-only data section and everything else. Changing that would mean replacing the generic RO_DATA() macro which increases the maintenance burden.

however, this is likely not worth the effort as the majority of read-only data will be covered.

---
 arch/arm64/mm/ktext.c |  2 +-
 arch/arm64/mm/mmu.c   | 21 ++++++++++++++++++---
 2 files changed, 19 insertions(+), 4 deletions(-)
arch/arm64/mm/ktext.c +1 -1
index 6692759e78a8..6265a2db449b 100644
@@ -101,7 +101,7 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst)
 /* Allocate page tables and memory for the replicated kernel texts. */
 void __init ktext_replication_init(void)
 {
-	size_t size = _etext - _stext;
+	size_t size = __end_rodata - _stext;
 	int kidx = pgd_index((phys_addr_t)KERNEL_START);
 	int nid;
 
arch/arm64/mm/mmu.c +18 -3
index fb9c476605d1..bf674bdaf336 100644
@@ -752,11 +752,26 @@ static pgprot_t __init kernel_text_pgprot(void)
 #ifdef CONFIG_REPLICATE_KTEXT
 void __init create_kernel_nid_map(pgd_t *pgdp, void *ktext)
 {
+	phys_addr_t pa_ktext;
+	size_t ro_offset;
+	void *ro_end;
 	pgprot_t text_prot = kernel_text_pgprot();
 
-	create_kernel_mapping(pgdp, __pa(ktext), _stext, _etext, text_prot, 0);
-	create_kernel_mapping(pgdp, __pa_symbol(__start_rodata),
-			      __start_rodata, __inittext_begin,
+	pa_ktext = __pa(ktext);
+	ro_offset = __pa_symbol(__start_rodata) - __pa_symbol(_stext);
+	/*
+	 * We must not cover the read-only data after init, since this
+	 * is written to during boot, and thus must be shared between
+	 * the NUMA nodes.
+	 */
+	ro_end = PTR_ALIGN_DOWN((void *)__start_ro_after_init, PAGE_SIZE);
+
+	create_kernel_mapping(pgdp, pa_ktext, _stext, _etext, text_prot, 0);
+	create_kernel_mapping(pgdp, pa_ktext + ro_offset,
+			      __start_rodata, ro_end,
+			      PAGE_KERNEL, NO_CONT_MAPPINGS);
+	create_kernel_mapping(pgdp, __pa_symbol(ro_end),
+			      ro_end, __inittext_begin,
 			      PAGE_KERNEL, NO_CONT_MAPPINGS);
 	create_kernel_mapping(pgdp, __pa_symbol(__inittext_begin),
 			      __inittext_begin, __inittext_end,
Signature · 1 line
2.30.2
#15 lore raw

PATCH v2 15/16 arm64: text replication: early kernel option to enable replication

In reply to Russell King (Oracle)

Provide an early kernel option "ktext=" which allows the kernel text replication to be enabled. This takes a boolean argument.

The way this has been implemented means that we take all the same paths through the kernel at runtime whether kernel text replication has been enabled or not; this allows the performance effects of the code changes to be evaluated separately from the act of running with replicating the kernel text.

---
 .../admin-guide/kernel-parameters.txt          |  5 +++++
 arch/arm64/mm/ktext.c                          | 18 ++++++++++++++++++
 2 files changed, 23 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +5 -0
index a1457995fd41..ceb351a6ff2a 100644
@@ -2494,6 +2494,11 @@
 			0: force disabled
 			1: force enabled
 
+	ktext=		[ARM64] Control kernel text replication on NUMA
+			machines. Default: disabled.
+			0: disable kernel text replication
+			1: enable kernel text replication
+
 	kunit.enable=	[KUNIT] Enable executing KUnit tests. Requires
 			CONFIG_KUNIT to be set to be fully enabled. The
 			default value can be overridden via
arch/arm64/mm/ktext.c +18 -0
index 6265a2db449b..f5580ae14ca4 100644
@@ -98,6 +98,21 @@ void ktext_replication_patch_alternative(__le32 *src, int nr_inst)
 	}
 }
 
+static bool ktext_enabled;
+
+static int __init parse_ktext(char *str)
+{
+	bool enabled;
+	int ret = strtobool(str, &enabled);
+
+	if (ret)
+		return ret;
+
+	ktext_enabled = enabled;
+	return 0;
+}
+early_param("ktext", parse_ktext);
+
 /* Allocate page tables and memory for the replicated kernel texts. */
 void __init ktext_replication_init(void)
 {
@@ -119,6 +134,9 @@ void __init ktext_replication_init(void)
 		return;
 	}
 
+	if (!ktext_enabled)
+		return;
+
 	for_each_node(nid) {
 		/* Nothing to do for node 0 */
 		if (!nid)
Signature · 1 line
2.30.2
#16 lore raw

PATCH v2 16/16 arm64: text replication: add Kconfig

In reply to Russell King (Oracle)

Add the Kconfig symbol for kernel text replication. This unfortunately requires KASAN and kernel text randomisation options to be disabled at the moment.

---
 arch/arm64/Kconfig | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
arch/arm64/Kconfig +9 -1
index 7856c3a3e35a..53143324c392 100644
@@ -160,7 +160,7 @@ config ARM64
 	select HAVE_ARCH_HUGE_VMAP
 	select HAVE_ARCH_JUMP_LABEL
 	select HAVE_ARCH_JUMP_LABEL_RELATIVE
-	select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48)
+	select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48 && !REPLICATE_KTEXT)
 	select HAVE_ARCH_KASAN_VMALLOC if HAVE_ARCH_KASAN
 	select HAVE_ARCH_KASAN_SW_TAGS if HAVE_ARCH_KASAN
 	select HAVE_ARCH_KASAN_HW_TAGS if (HAVE_ARCH_KASAN && ARM64_MTE)
@@ -1426,6 +1426,13 @@ config NODES_SHIFT
 	  Specify the maximum number of NUMA Nodes available on the target
 	  system.  Increases memory reserved to accommodate various tables.
 
+config REPLICATE_KTEXT
+	bool "Replicate kernel text across numa nodes"
+	depends on NUMA
+	help
+	  Say Y here to enable replicating the kernel text across multiple
+	  nodes in a NUMA cluster.  This trades memory for speed.
+
 source "kernel/Kconfig.hz"
 
 config ARCH_SPARSEMEM_ENABLE
@@ -2173,6 +2180,7 @@ config RELOCATABLE
 
 config RANDOMIZE_BASE
 	bool "Randomize the address of the kernel image"
+	depends on !REPLICATE_KTEXT
 	select RELOCATABLE
 	help
 	  Randomizes the virtual address at which the kernel image is
Signature · 1 line
2.30.2
reviewer
#17 lore raw

Re: [PATCH v2 16/16] arm64: text replication: add Kconfig

Hi Russell,

arch/arm64/Kconfig config NODES_SHIFT · line 1430

Quoted 2 lines · Russell King (Oracle)
arch/arm64/Kconfig +2 -0
+config REPLICATE_KTEXT
+	bool "Replicate kernel text across numa nodes"

Please use NUMA
for consistency

arch/arm64/Kconfig config NODES_SHIFT · line 1434

Quoted 4 lines
arch/arm64/Kconfig +4 -0
+	depends on NUMA
+	help
+	  Say Y here to enable replicating the kernel text across multiple
+	  nodes in a NUMA cluster.  This trades memory for speed.

with ^^^^.

Signature · 1 line
~Randy