Skip to contentSkip to conversation

riscv: Memory Hot(Un)Plug support

PATCH Applied Björn Töpel 3 participants 19 messages Jun 5, 2024 – Jun 27, 2024

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

v4 cover riscv: Memory Hot(Un)Plug support

From: Björn Töpel <[email protected]>

================================================================
Memory Hot(Un)Plug support (and ZONE_DEVICE) for the RISC-V port
================================================================

(For the restless folks: change log in the bottom!)

Introduction
============

To quote "Documentation/admin-guide/mm/memory-hotplug.rst": "Memory hot(un)plug allows for increasing and decreasing the size of physical memory available to a machine at runtime."

This series adds memory hot(un)plugging, and ZONE_DEVICE support for the RISC-V Linux port.

MM configuration
================

RISC-V MM has the following configuration:

* Memory blocks are 128M, analogous to x86-64. It uses PMD ("hugepage") vmemmaps. From that follows that 2M (PMD) worth of vmemmap spans 32768 pages á 4K which gets us 128M.

* The pageblock size is the minimum minimum virtio_mem size, and on RISC-V it's 2M (2^9 * 4K).

Implementation
==============

The PGD table on RISC-V is shared/copied between for all processes. To avoid doing page table synchronization, the first patch (patch 1) pre-allocated the PGD entries for vmemmap/direct map. By doing that the init_mm PGD will be fixed at kernel init, and synchronization can be avoided all together.

The following two patches (patch 2-3) does some preparations, followed by the actual MHP implementation (patch 4-5). Then, MHP and virtio-mem are enabled (patch 6-7), and finally ZONE_DEVICE support is added (patch 8).

MHP and locking
===============

TL;DR: The MHP does not step on any toes, except for ptdump. Additional locking is required for ptdump.

Long version: For v2 I spent some time digging into init_mm synchronization/update. Here are my findings, and I'd love them to be corrected if incorrect.

It's been a gnarly path...

The init_mm structure is a special mm (perhaps not a "real" one). It's a "lazy context" that tracks kernel page table resources, e.g., the kernel page table (swapper_pg_dir), a kernel page_table_lock (more about the usage below), mmap_lock, and such.

init_mm does not track/contain any VMAs. Having the init_mm is convenient, so that the regular kernel page table walk/modify functions can be used.

Now, init_mm being special means that the locking for kernel page tables are special as well.

On RISC-V the PGD (top-level page table structure), similar to x86, is shared (copied) with user processes. If the kernel PGD is modified, it has to be synched to user-mode processes PGDs. This is avoided by pre-populating the PGD, so it'll be fixed from boot.

The in-kernel pgd regions are documented in
Documentation/arch/riscv/vm-layout.rst.

The distinct regions are:
* vmemmap
* vmalloc/ioremap space
* direct mapping of all physical memory
* kasan
* modules, BPF
* kernel

Memory hotplug is the process of adding/removing memory to/from the kernel.

Adding is done in two phases:
1. Add the memory to the kernel
2. Online memory, making it available to the page allocator.

Step 1 is partially architecture dependent, and updates the init_mm page table:
* Update the direct map page tables. The direct map is a linear map, representing all physical memory: virt = phys + PAGE_OFFSET
* Add a struct page for each added page of memory. Update the vmemmap (virtual mapping to the struct page, so we can easily transform a kernel virtual address to a struct page * address.

From an MHP perspective, there are two regions of the PGD that are updated:
* vmemmap
* direct mapping of all physical memory

The struct mm_struct has a couple of locks in play:
* spinlock_t page_table_lock protects the page table, and some counters
* struct rw_semaphore mmap_lock protect an mm's VMAs

Note again that init_mm does not contain any VMAs, but still uses the mmap_lock in some places.

The page_table_lock was originally used to to protect all pages tables, but more recently a split page table lock has been introduced. The split lock has a per-table lock for the PTE and PMD tables. If split lock is disabled, all tables are guarded by mm->page_table_lock (for user processes). Split page table locks are not used for init_mm.

MHP operations is typically synchronized using
DEFINE_STATIC_PERCPU_RWSEM(mem_hotplug_lock).

Actors
------

The following non-MHP actors in the kernel traverses (read), and/or modifies the kernel PGD.

* ptdump

Walks the entire init_mm, via ptdump_walk_pgd() with the mmap_write_lock(init_mm) taken.

Observation: ptdump can race with MHP, and needs additional locking to avoid crashes/races.

* set_direct_* / arch/riscv/mm/pageattr.c

The set_direct_* functionality is used to "synchronize" the direct map to other kernel mappings, e.g. modules/kernel text. The direct map is using "as large huge table mappings as possible", which means that the set_direct_* might need to split the direct map.

The set_direct_* functions operates with the
mmap_write_lock(init_mm) taken.

Observation: set_direct_* uses the direct map, but will never modify the same entry as MHP. If there is a mapping, that entry will never race with MHP. Further, MHP acts when memory is offline.

* HVO / mm/hugetlb_vmemmap

HVO optimizes the backing struct page for hugetlb pages, which means changing the "vmemmap" region. HVO can split (merge?) a vmemmap pmd. However, it will never race with MHP, since HVO only operates at online memory. HVO cannot touch memory being MHP added or removed.

* apply_to_page_range

Walks a range, creates pages and applies a callback (setting permissions) for the page.

When creating a table, it might use int __pte_alloc_kernel(pmd_t *pmd) which takes the init_mm.page_table_lock to synchronize pmd populate.

Used by: mm/vmalloc.c and mm/kasan/shadow.c. The KASAN callback takes the init_mm.page_table_lock to synchronize pte creation.

Observations: apply_to_page_range applies to the "vmalloc/ioremap space" region, and "kasan" region. *Not* affected by MHP.

* apply_to_existing_page_range

Walks a range, applies a callback (setting permissions) for the page (no page creation).

Used by: kernel/bpf/arena.c and mm/kasan/shadow.c. The KASAN callback takes the init_mm.page_table_lock to synchronize pte creation. *Not* affected by MHP regions.

* apply_to_existing_page_range applies to the "vmalloc/ioremap space" region, and "kasan" region. *Not* affected by MHP regions.

* ioremap_page_range and vmap_page_range

Uses the same internal function, and might create table entries at the "vmalloc/ioremap space" region. Can call __pte_alloc_kernel() which takes the init_mm.page_table_lock synchronizing pmd populate in the region. *Not* affected by MHP regions.

Summary:
* MHP add will never modify the same page table entries, as any of the other actors.
* MHP remove is done when memory is offlined, and will not clash with any of the actors.
* Functions that walk the entire kernel page table need synchronization

* It's sufficient to add the MHP lock ptdump.

Testing
=======

This series adds basic DT supported hotplugging. There is a QEMU series enabling MHP for the RISC-V "virt" machine here: [1]

ACPI/MSI support is still in the making for RISC-V, and prior proper (ACPI) PCI MSI support lands [2] and NUMA SRAT support [3], it hard to try it out.

I've prepared a QEMU branch with proper ACPI GED/PC-DIMM support [4], and a this series with the required prerequisites [5] (AIA, ACPI AIA MADT, ACPI NUMA SRAT).

To test with virtio-mem, e.g.:

  | qemu-system-riscv64 \
  |     -machine virt,aia=aplic-imsic \
  |     -cpu rv64,v=true,vlen=256,elen=64,h=true,zbkb=on,zbkc=on,zbkx=on,zkr=on,zkt=on,svinval=on,svnapot=on,svpbmt=on \
  |     -nodefaults \
  |     -nographic -smp 8 -kernel rv64-u-boot.bin \
  |     -drive file=rootfs.img,format=raw,if=virtio \
  |     -device virtio-rng-pci \
  |     -m 16G,slots=3,maxmem=32G \
  |     -object memory-backend-ram,id=mem0,size=16G \
  |     -numa node,nodeid=0,memdev=mem0 \
  |     -serial chardev:char0 \
  |     -mon chardev=char0,mode=readline \
  |     -chardev stdio,mux=on,id=char0 \
  |     -device pci-serial,id=serial0,chardev=char0 \
  |     -object memory-backend-ram,id=vmem0,size=2G \
  |     -device virtio-mem-pci,id=vm0,memdev=vmem0,node=0

where "rv64-u-boot.bin" is U-boot with EFI/ACPI-support (use [6] if you're lazy).

In the QEMU monitor:
| (qemu) info memory-devices
| (qemu) qom-set vm0 requested-size 1G

...to test DAX/KMEM, use the follow QEMU parameters:

  |  -object memory-backend-file,id=mem1,share=on,mem-path=virtio_pmem.img,size=4G \
  |  -device virtio-pmem-pci,memdev=mem1,id=nv1

and the regular ndctl/daxctl dance.

If you're brave to try the ACPI branch, add "acpi=on" to "-machine virt", and test PC-DIMM MHP (in addition to virtio-{p},mem):

In the QEMU monitor:
| (qemu) object_add memory-backend-ram,id=mem1,size=1G | (qemu) device_add pc-dimm,id=dimm1,memdev=mem1

You can also try hot-remove with some QEMU options, say:

  | -object memory-backend-file,id=mem-1,size=256M,mem-path=/pagesize-2MB
  | -device pc-dimm,id=mem1,memdev=mem-1
  | -object memory-backend-file,id=mem-2,size=1G,mem-path=/pagesize-1GB
  | -device pc-dimm,id=mem2,memdev=mem-2
  | -object memory-backend-file,id=mem-3,size=256M,mem-path=/pagesize-2MB
  | -device pc-dimm,id=mem3,memdev=mem-3

Remove "acpi=on" to run with DT.

Thanks to Alex, Andrew, David, and Oscar for all
comments/tests/fixups.

Changelog
=========

v3->v4:
Collect tags 0002-riscv-mm-Pre-allocate-vmemmap-direct-map-kasan-PG
* Pre-allocate KASAN PGD entries as well (Alex)
0005-riscv-mm-Add-pfn_to_kaddr-implementation
* New patch, add missing function for KASAN
0006-riscv-mm-Add-memory-hotplugging-support
* Use pXXX_get() instead of READ_ONCE (Alex)
* Separated vmemmap storage and pgtable allocation. Use matching alloc/dealloc functions (e.g. ptdesc alloc/delloc) with also avoids WARNs ;-)
* Cleaned up the reserved page freeing (Alex)
0008-riscv-Enable-memory-hotplugging-for-RISC-V
* Simplified the dependencies (Alex)
* Added ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE
0011-riscv-Enable-DAX-VMEMMAP-optimization
* New patch, added ARCH_WANT_OPTIMIZE_DAX_VMEMMAP

v2->v3:
Collect tags 0001-riscv-mm-Properly-forward-vmemmap_populate-altmap
* New patch, split from attribute change (Alex, Oscar)
0005-riscv-mm-Add-memory-hotplugging-support
* Fix page table free to take PG_reserved into consideration (David, Oscar)
* Depend on SPARSE_VMEMMAP (Alex, David, Oscar)
* Proper TLB flushing (Alex)
0009-riscv-mm-Add-support-for-ZONE_DEVICE
* Fixed 32b build

v1->v2:
Reviewed a lot of MHP locking scenarios
Various config build issues (e.g. build !CONFIG_MEMORY_HOTPLUG) (Andrew) Added arch_get_mappable_range() implementation
Acquire MHP lock for ptdump, analogous to arm64
ACPI MHP tests
Add ZONE_DEVICE patch

References
==========

[1] https://lore.kernel.org/qemu-devel/[email protected]/
[2] https://lore.kernel.org/linux-riscv/[email protected]/
[3] https://lore.kernel.org/linux-riscv/[email protected]/
[4] https://github.com/bjoto/qemu/commits/virtio-mem-pc-dimm-mhp-acpi-v2/
[5] https://github.com/bjoto/linux/commits/mhp-v4-acpi
[6] https://github.com/bjoto/riscv-rootfs-utils/tree/acpi

Thanks,
Björn

Björn Töpel (11):
riscv: mm: Properly forward vmemmap_populate() altmap parameter
riscv: mm: Pre-allocate vmemmap/direct map/kasan PGD entries
riscv: mm: Change attribute from __init to __meminit for page
functions
riscv: mm: Refactor create_linear_mapping_range() for memory hot add
riscv: mm: Add pfn_to_kaddr() implementation
riscv: mm: Add memory hotplugging support
riscv: mm: Take memory hotplug read-lock during kernel page table dump
riscv: Enable memory hotplugging for RISC-V
virtio-mem: Enable virtio-mem for RISC-V
riscv: mm: Add support for ZONE_DEVICE
riscv: Enable DAX VMEMMAP optimization

 arch/riscv/Kconfig                    |   5 +
 arch/riscv/include/asm/kasan.h        |   4 +-
 arch/riscv/include/asm/mmu.h          |   4 +-
 arch/riscv/include/asm/page.h         |   5 +
 arch/riscv/include/asm/pgtable-64.h   |  20 ++
 arch/riscv/include/asm/pgtable-bits.h |   1 +
 arch/riscv/include/asm/pgtable.h      |  19 +-
 arch/riscv/mm/init.c                  | 347 +++++++++++++++++++++++---
 arch/riscv/mm/ptdump.c                |   3 +
 drivers/virtio/Kconfig                |   2 +-
 10 files changed, 364 insertions(+), 46 deletions(-)

base-commit: e2c79b4c5c4d83520abb570ca633ded09621c0a6

Signature · 1 line
2.43.0
#1 lore raw

PATCH v4 1/11 riscv: mm: Properly forward vmemmap_populate() altmap parameter

From: Björn Töpel <[email protected]>

Make sure that the altmap parameter is properly passed on to vmemmap_populate_hugepages().

---
 arch/riscv/mm/init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/riscv/mm/init.c +1 -1
index e3405e4b99af..fe5072f66c8c 100644
@@ -1439,7 +1439,7 @@ int __meminit vmemmap_populate(unsigned long start, unsigned long end, int node,
 	 * memory hotplug, we are not able to update all the page tables with
 	 * the new PMDs.
 	 */
-	return vmemmap_populate_hugepages(start, end, node, NULL);
+	return vmemmap_populate_hugepages(start, end, node, altmap);
 }
 #endif
 
Signature · 1 line
2.43.0
#2 lore raw

PATCH v4 2/11 riscv: mm: Pre-allocate vmemmap/direct map/kasan PGD entries

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

The RISC-V port copies the PGD table from init_mm/swapper_pg_dir to all userland page tables, which means that if the PGD level table is changed, other page tables has to be updated as well.

Instead of having the PGD changes ripple out to all tables, the synchronization can be avoided by pre-allocating the PGD entries/pages at boot, avoiding the synchronization all together.

This is currently done for the bpf/modules, and vmalloc PGD regions. Extend this scheme for the PGD regions touched by memory hotplugging.

Prepare the RISC-V port for memory hotplug by pre-allocate vmemmap/direct map/kasan entries at the PGD level. This will roughly waste ~128 (plus 32 if KASAN is enabled) worth of 4K pages when memory hotplugging is enabled in the kernel configuration.

---
 arch/riscv/include/asm/kasan.h | 4 ++--
 arch/riscv/mm/init.c           | 9 +++++++++
 2 files changed, 11 insertions(+), 2 deletions(-)
arch/riscv/include/asm/kasan.h +2 -2
index 0b85e363e778..e6a0071bdb56 100644
@@ -6,8 +6,6 @@
 
 #ifndef __ASSEMBLY__
 
-#ifdef CONFIG_KASAN
-
 /*
  * The following comment was copied from arm64:
  * KASAN_SHADOW_START: beginning of the kernel virtual addresses.
@@ -34,6 +32,8 @@
  */
 #define KASAN_SHADOW_START	((KASAN_SHADOW_END - KASAN_SHADOW_SIZE) & PGDIR_MASK)
 #define KASAN_SHADOW_END	MODULES_LOWEST_VADDR
+
+#ifdef CONFIG_KASAN
 #define KASAN_SHADOW_OFFSET	_AC(CONFIG_KASAN_SHADOW_OFFSET, UL)
 
 void kasan_init(void);
arch/riscv/mm/init.c +9 -0
index fe5072f66c8c..a5b3bc1f3b88 100644
@@ -28,6 +28,7 @@
 
 #include <asm/fixmap.h>
 #include <asm/io.h>
+#include <asm/kasan.h>
 #include <asm/numa.h>
 #include <asm/pgtable.h>
 #include <asm/sections.h>
@@ -1493,11 +1494,19 @@ static void __init preallocate_pgd_pages_range(unsigned long start, unsigned lon
 	panic("Failed to pre-allocate %s pages for %s area\n", lvl, area);
 }
 
+#define PAGE_END KASAN_SHADOW_START
+
 void __init pgtable_cache_init(void)
 {
 	preallocate_pgd_pages_range(VMALLOC_START, VMALLOC_END, "vmalloc");
 	if (IS_ENABLED(CONFIG_MODULES))
 		preallocate_pgd_pages_range(MODULES_VADDR, MODULES_END, "bpf/modules");
+	if (IS_ENABLED(CONFIG_MEMORY_HOTPLUG)) {
+		preallocate_pgd_pages_range(VMEMMAP_START, VMEMMAP_END, "vmemmap");
+		preallocate_pgd_pages_range(PAGE_OFFSET, PAGE_END, "direct map");
+		if (IS_ENABLED(CONFIG_KASAN))
+			preallocate_pgd_pages_range(KASAN_SHADOW_START, KASAN_SHADOW_END, "kasan");
+	}
 }
 #endif
 
Signature · 1 line
2.43.0
#3 lore raw

PATCH v4 3/11 riscv: mm: Change attribute from __init to __meminit for page functions

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

Prepare for memory hotplugging support by changing from __init to __meminit for the page table functions that are used by the upcoming architecture specific callbacks.

Changing the __init attribute to __meminit, avoids that the functions are removed after init. The __meminit attribute makes sure the functions are kept in the kernel text post init, but only if memory hotplugging is enabled for the build.

---
 arch/riscv/include/asm/mmu.h     |  4 +--
 arch/riscv/include/asm/pgtable.h |  2 +-
 arch/riscv/mm/init.c             | 56 ++++++++++++++------------------
 3 files changed, 28 insertions(+), 34 deletions(-)
arch/riscv/include/asm/mmu.h +2 -2
index 947fd60f9051..c9e03e9da3dc 100644
@@ -31,8 +31,8 @@ typedef struct {
 #define cntx2asid(cntx)		((cntx) & SATP_ASID_MASK)
 #define cntx2version(cntx)	((cntx) & ~SATP_ASID_MASK)
 
-void __init create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa,
-			       phys_addr_t sz, pgprot_t prot);
+void __meminit create_pgd_mapping(pgd_t *pgdp, uintptr_t va, phys_addr_t pa, phys_addr_t sz,
+				  pgprot_t prot);
 #endif /* __ASSEMBLY__ */
 
 #endif /* _ASM_RISCV_MMU_H */
arch/riscv/include/asm/pgtable.h +1 -1
index aad8b8ca51f1..41f1b2c6f949 100644
@@ -165,7 +165,7 @@ struct pt_alloc_ops {
 #endif
 };
 
-extern struct pt_alloc_ops pt_ops __initdata;
+extern struct pt_alloc_ops pt_ops __meminitdata;
 
 #ifdef CONFIG_MMU
 /* Number of PGD entries that a user-mode program can use */
arch/riscv/mm/init.c +25 -31

164 lines not on this page · raw message

Signature · 1 line
2.43.0
#4 lore raw

PATCH v4 4/11 riscv: mm: Refactor create_linear_mapping_range() for memory hot add

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

Add a parameter to the direct map setup function, so it can be used in arch_add_memory() later.

---
 arch/riscv/mm/init.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
arch/riscv/mm/init.c +6 -9
index 0dd04cedc0d2..1f7e7c223bec 100644
@@ -1232,7 +1232,7 @@ asmlinkage void __init setup_vm(uintptr_t dtb_pa)
 }
 
 static void __meminit create_linear_mapping_range(phys_addr_t start, phys_addr_t end,
-						  uintptr_t fixed_map_size)
+						  uintptr_t fixed_map_size, const pgprot_t *pgprot)
 {
 	phys_addr_t pa;
 	uintptr_t va, map_size;
@@ -1243,7 +1243,7 @@ static void __meminit create_linear_mapping_range(phys_addr_t start, phys_addr_t
 					    best_map_size(pa, va, end - pa);
 
 		create_pgd_mapping(swapper_pg_dir, va, pa, map_size,
-				   pgprot_from_va(va));
+				   pgprot ? *pgprot : pgprot_from_va(va));
 	}
 }
 
@@ -1287,22 +1287,19 @@ static void __init create_linear_mapping_page_table(void)
 		if (end >= __pa(PAGE_OFFSET) + memory_limit)
 			end = __pa(PAGE_OFFSET) + memory_limit;
 
-		create_linear_mapping_range(start, end, 0);
+		create_linear_mapping_range(start, end, 0, NULL);
 	}
 
 #ifdef CONFIG_STRICT_KERNEL_RWX
-	create_linear_mapping_range(ktext_start, ktext_start + ktext_size, 0);
-	create_linear_mapping_range(krodata_start,
-				    krodata_start + krodata_size, 0);
+	create_linear_mapping_range(ktext_start, ktext_start + ktext_size, 0, NULL);
+	create_linear_mapping_range(krodata_start, krodata_start + krodata_size, 0, NULL);
 
 	memblock_clear_nomap(ktext_start,  ktext_size);
 	memblock_clear_nomap(krodata_start, krodata_size);
 #endif
 
 #ifdef CONFIG_KFENCE
-	create_linear_mapping_range(kfence_pool,
-				    kfence_pool + KFENCE_POOL_SIZE,
-				    PAGE_SIZE);
+	create_linear_mapping_range(kfence_pool, kfence_pool + KFENCE_POOL_SIZE, PAGE_SIZE, NULL);
 
 	memblock_clear_nomap(kfence_pool, KFENCE_POOL_SIZE);
 #endif
Signature · 1 line
2.43.0
#5 lore raw

PATCH v4 5/11 riscv: mm: Add pfn_to_kaddr() implementation

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

The pfn_to_kaddr() function is used by KASAN's memory hotplugging path. Add the missing function to the RISC-V port, so that it can be built with MHP and CONFIG_KASAN.

---
 arch/riscv/include/asm/page.h | 5 +++++
 1 file changed, 5 insertions(+)
arch/riscv/include/asm/page.h +5 -0
index 115ac98b8d72..235fd45d998d 100644
@@ -188,6 +188,11 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);
 
 unsigned long kaslr_offset(void);
 
+static __always_inline void *pfn_to_kaddr(unsigned long pfn)
+{
+	return __va(pfn << PAGE_SHIFT);
+}
+
 #endif /* __ASSEMBLY__ */
 
 #define virt_addr_valid(vaddr)	({						\
Signature · 1 line
2.43.0
#8 lore raw

PATCH v4 6/11 riscv: mm: Add memory hotplugging support

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

For an architecture to support memory hotplugging, a couple of callbacks needs to be implemented:

arch_add_memory()
This callback is responsible for adding the physical memory into the direct map, and call into the memory hotplugging generic code via __add_pages() that adds the corresponding struct page entries, and updates the vmemmap mapping.

arch_remove_memory()
This is the inverse of the callback above.

vmemmap_free()
This function tears down the vmemmap mappings (if CONFIG_SPARSEMEM_VMEMMAP is enabled), and also deallocates the backing vmemmap pages. Note that for persistent memory, an alternative allocator for the backing pages can be used; The vmem_altmap. This means that when the backing pages are cleared, extra care is needed so that the correct deallocation method is used.

arch_get_mappable_range()
This functions returns the PA range that the direct map can map. Used by the MHP internals for sanity checks.

The page table unmap/teardown functions are heavily based on code from the x86 tree. The same remove_pgd_mapping() function is used in both vmemmap_free() and arch_remove_memory(), but in the latter function the backing pages are not removed.

---
 arch/riscv/mm/init.c | 267 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 267 insertions(+)
arch/riscv/mm/init.c +267 -0
index 1f7e7c223bec..bfa2dea95354 100644
@@ -1534,3 +1534,270 @@ struct execmem_info __init *execmem_arch_setup(void)
 }
 #endif /* CONFIG_MMU */
 #endif /* CONFIG_EXECMEM */
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
+{
+	struct page *page = pmd_page(*pmd);
+	struct ptdesc *ptdesc = page_ptdesc(page);
+	pte_t *pte;
+	int i;
+
+	for (i = 0; i < PTRS_PER_PTE; i++) {
+		pte = pte_start + i;
+		if (!pte_none(*pte))
+			return;
+	}
+
+	pagetable_pte_dtor(ptdesc);
+	if (PageReserved(page))
+		free_reserved_page(page);
+	else
+		pagetable_free(ptdesc);
+	pmd_clear(pmd);
+}
+
+static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud)
+{
+	struct page *page = pud_page(*pud);
+	struct ptdesc *ptdesc = page_ptdesc(page);
+	pmd_t *pmd;
+	int i;
+
+	for (i = 0; i < PTRS_PER_PMD; i++) {
+		pmd = pmd_start + i;
+		if (!pmd_none(*pmd))
+			return;
+	}
+
+	pagetable_pmd_dtor(ptdesc);
+	if (PageReserved(page))
+		free_reserved_page(page);
+	else
+		pagetable_free(ptdesc);
+	pud_clear(pud);
+}
+
+static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d)
+{
+	struct page *page = p4d_page(*p4d);
+	pud_t *pud;
+	int i;
+
+	for (i = 0; i < PTRS_PER_PUD; i++) {
+		pud = pud_start + i;
+		if (!pud_none(*pud))
+			return;
+	}
+
+	if (PageReserved(page))
+		free_reserved_page(page);
+	else
+		free_pages((unsigned long)page_address(page), 0);
+	p4d_clear(p4d);
+}
+
+static void __meminit free_vmemmap_storage(struct page *page, size_t size,
+					   struct vmem_altmap *altmap)
+{
+	int order = get_order(size);
+
+	if (altmap) {
+		vmem_altmap_free(altmap, size >> PAGE_SHIFT);
+		return;
+	}
+
+	if (PageReserved(page)) {
+		unsigned int nr_pages = 1 << order;
+
+		while (nr_pages--)
+			free_reserved_page(page++);
+		return;
+	}
+
+	free_pages((unsigned long)page_address(page), order);
+}
+
+static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, unsigned long end,
+					 bool is_vmemmap, struct vmem_altmap *altmap)
+{
+	unsigned long next;
+	pte_t *ptep, pte;
+
+	for (; addr < end; addr = next) {
+		next = (addr + PAGE_SIZE) & PAGE_MASK;
+		if (next > end)
+			next = end;
+
+		ptep = pte_base + pte_index(addr);
+		pte = ptep_get(ptep);
+		if (!pte_present(*ptep))
+			continue;
+
+		pte_clear(&init_mm, addr, ptep);
+		if (is_vmemmap)
+			free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
+	}
+}
+
+static void __meminit remove_pmd_mapping(pmd_t *pmd_base, unsigned long addr, unsigned long end,
+					 bool is_vmemmap, struct vmem_altmap *altmap)
+{
+	unsigned long next;
+	pte_t *pte_base;
+	pmd_t *pmdp, pmd;
+
+	for (; addr < end; addr = next) {
+		next = pmd_addr_end(addr, end);
+		pmdp = pmd_base + pmd_index(addr);
+		pmd = pmdp_get(pmdp);
+		if (!pmd_present(pmd))
+			continue;
+
+		if (pmd_leaf(pmd)) {
+			pmd_clear(pmdp);
+			if (is_vmemmap)
+				free_vmemmap_storage(pmd_page(pmd), PMD_SIZE, altmap);
+			continue;
+		}
+
+		pte_base = (pte_t *)pmd_page_vaddr(*pmdp);
+		remove_pte_mapping(pte_base, addr, next, is_vmemmap, altmap);
+		free_pte_table(pte_base, pmdp);
+	}
+}
+
+static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long addr, unsigned long end,
+					 bool is_vmemmap, struct vmem_altmap *altmap)
+{
+	unsigned long next;
+	pud_t *pudp, pud;
+	pmd_t *pmd_base;
+
+	for (; addr < end; addr = next) {
+		next = pud_addr_end(addr, end);
+		pudp = pud_base + pud_index(addr);
+		pud = pudp_get(pudp);
+		if (!pud_present(pud))
+			continue;
+
+		if (pud_leaf(pud)) {
+			if (pgtable_l4_enabled) {
+				pud_clear(pudp);
+				if (is_vmemmap)
+					free_vmemmap_storage(pud_page(pud), PUD_SIZE, altmap);
+			}
+			continue;
+		}
+
+		pmd_base = pmd_offset(pudp, 0);
+		remove_pmd_mapping(pmd_base, addr, next, is_vmemmap, altmap);
+
+		if (pgtable_l4_enabled)
+			free_pmd_table(pmd_base, pudp);
+	}
+}
+
+static void __meminit remove_p4d_mapping(p4d_t *p4d_base, unsigned long addr, unsigned long end,
+					 bool is_vmemmap, struct vmem_altmap *altmap)
+{
+	unsigned long next;
+	p4d_t *p4dp, p4d;
+	pud_t *pud_base;
+
+	for (; addr < end; addr = next) {
+		next = p4d_addr_end(addr, end);
+		p4dp = p4d_base + p4d_index(addr);
+		p4d = p4dp_get(p4dp);
+		if (!p4d_present(p4d))
+			continue;
+
+		if (p4d_leaf(p4d)) {
+			if (pgtable_l5_enabled) {
+				p4d_clear(p4dp);
+				if (is_vmemmap)
+					free_vmemmap_storage(p4d_page(p4d), P4D_SIZE, altmap);
+			}
+			continue;
+		}
+
+		pud_base = pud_offset(p4dp, 0);
+		remove_pud_mapping(pud_base, addr, next, is_vmemmap, altmap);
+
+		if (pgtable_l5_enabled)
+			free_pud_table(pud_base, p4dp);
+	}
+}
+
+static void __meminit remove_pgd_mapping(unsigned long va, unsigned long end, bool is_vmemmap,
+					 struct vmem_altmap *altmap)
+{
+	unsigned long addr, next;
+	p4d_t *p4d_base;
+	pgd_t *pgd;
+
+	for (addr = va; addr < end; addr = next) {
+		next = pgd_addr_end(addr, end);
+		pgd = pgd_offset_k(addr);
+
+		if (!pgd_present(*pgd))
+			continue;
+
+		if (pgd_leaf(*pgd))
+			continue;
+
+		p4d_base = p4d_offset(pgd, 0);
+		remove_p4d_mapping(p4d_base, addr, next, is_vmemmap, altmap);
+	}
+
+	flush_tlb_all();
+}
+
+static void __meminit remove_linear_mapping(phys_addr_t start, u64 size)
+{
+	unsigned long va = (unsigned long)__va(start);
+	unsigned long end = (unsigned long)__va(start + size);
+
+	remove_pgd_mapping(va, end, false, NULL);
+}
+
+struct range arch_get_mappable_range(void)
+{
+	struct range mhp_range;
+
+	mhp_range.start = __pa(PAGE_OFFSET);
+	mhp_range.end = __pa(PAGE_END - 1);
+	return mhp_range;
+}
+
+int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params)
+{
+	int ret = 0;
+
+	create_linear_mapping_range(start, start + size, 0, &params->pgprot);
+	ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, params);
+	if (ret) {
+		remove_linear_mapping(start, size);
+		goto out;
+	}
+
+	max_pfn = PFN_UP(start + size);
+	max_low_pfn = max_pfn;
+
+ out:
+	flush_tlb_all();
+	return ret;
+}
+
+void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
+{
+	__remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);
+	remove_linear_mapping(start, size);
+	flush_tlb_all();
+}
+
+void __ref vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap)
+{
+	remove_pgd_mapping(start, end, true, altmap);
+}
+#endif /* CONFIG_MEMORY_HOTPLUG */
Signature · 1 line
2.43.0
#11 lore raw

PATCH v4 7/11 riscv: mm: Take memory hotplug read-lock during kernel page table dump

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

During memory hot remove, the ptdump functionality can end up touching stale data. Avoid any potential crashes (or worse), by holding the memory hotplug read-lock while traversing the page table.

This change is analogous to arm64's commit bf2b59f60ee1 ("arm64/mm:
Hold memory hotplug lock while walking for kernel page table dump").

---
 arch/riscv/mm/ptdump.c | 3 +++
 1 file changed, 3 insertions(+)
arch/riscv/mm/ptdump.c +3 -0
index 1289cc6d3700..9d5f657a251b 100644
@@ -6,6 +6,7 @@
 #include <linux/efi.h>
 #include <linux/init.h>
 #include <linux/debugfs.h>
+#include <linux/memory_hotplug.h>
 #include <linux/seq_file.h>
 #include <linux/ptdump.h>
 
@@ -370,7 +371,9 @@ bool ptdump_check_wx(void)
 
 static int ptdump_show(struct seq_file *m, void *v)
 {
+	get_online_mems();
 	ptdump_walk(m, m->private);
+	put_online_mems();
 
 	return 0;
 }
Signature · 1 line
2.43.0
#12 lore raw

PATCH v4 8/11 riscv: Enable memory hotplugging for RISC-V

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

Enable ARCH_ENABLE_MEMORY_HOTPLUG and ARCH_ENABLE_MEMORY_HOTREMOVE for RISC-V.

---
 arch/riscv/Kconfig | 3 +++
 1 file changed, 3 insertions(+)
arch/riscv/Kconfig +3 -0
index 0525ee2d63c7..8d860ad3b171 100644
@@ -16,6 +16,8 @@ config RISCV
 	select ACPI_REDUCED_HARDWARE_ONLY if ACPI
 	select ARCH_DMA_DEFAULT_COHERENT
 	select ARCH_ENABLE_HUGEPAGE_MIGRATION if HUGETLB_PAGE && MIGRATION
+	select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP
+	select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
 	select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
 	select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
 	select ARCH_HAS_BINFMT_FLAT
@@ -46,6 +48,7 @@ config RISCV
 	select ARCH_HAS_UBSAN
 	select ARCH_HAS_VDSO_DATA
 	select ARCH_KEEP_MEMBLOCK if ACPI
+	select ARCH_MHP_MEMMAP_ON_MEMORY_ENABLE	if 64BIT && MMU
 	select ARCH_OPTIONAL_KERNEL_RWX if ARCH_HAS_STRICT_KERNEL_RWX
 	select ARCH_OPTIONAL_KERNEL_RWX_DEFAULT
 	select ARCH_STACKWALK
Signature · 1 line
2.43.0
#13 lore raw

PATCH v4 9/11 virtio-mem: Enable virtio-mem for RISC-V

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

Now that RISC-V has memory hotplugging support, virtio-mem can be used on the platform.

---
 drivers/virtio/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/virtio/Kconfig +1 -1
index 6284538a8184..42a48ac763ee 100644
@@ -122,7 +122,7 @@ config VIRTIO_BALLOON
 
 config VIRTIO_MEM
 	tristate "Virtio mem driver"
-	depends on X86_64 || ARM64
+	depends on X86_64 || ARM64 || RISCV
 	depends on VIRTIO
 	depends on MEMORY_HOTPLUG
 	depends on MEMORY_HOTREMOVE
Signature · 1 line
2.43.0
#14 lore raw

PATCH v4 10/11 riscv: mm: Add support for ZONE_DEVICE

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

ZONE_DEVICE pages need DEVMAP PTEs support to function (ARCH_HAS_PTE_DEVMAP). Claim another RSW (reserved for software) bit in the PTE for DEVMAP mark, add the corresponding helpers, and enable ARCH_HAS_PTE_DEVMAP for riscv64.

---
 arch/riscv/Kconfig                    |  1 +
 arch/riscv/include/asm/pgtable-64.h   | 20 ++++++++++++++++++++
 arch/riscv/include/asm/pgtable-bits.h |  1 +
 arch/riscv/include/asm/pgtable.h      | 17 +++++++++++++++++
 4 files changed, 39 insertions(+)
arch/riscv/Kconfig +1 -0
index 8d860ad3b171..8a49b5f4c017 100644
@@ -37,6 +37,7 @@ config RISCV
 	select ARCH_HAS_NON_OVERLAPPING_ADDRESS_SPACE
 	select ARCH_HAS_PMEM_API
 	select ARCH_HAS_PREPARE_SYNC_CORE_CMD
+	select ARCH_HAS_PTE_DEVMAP if 64BIT && MMU
 	select ARCH_HAS_PTE_SPECIAL
 	select ARCH_HAS_SET_DIRECT_MAP if MMU
 	select ARCH_HAS_SET_MEMORY if MMU
arch/riscv/include/asm/pgtable-64.h +20 -0
index 8c36a8818432..0897dd99ab8d 100644
@@ -398,4 +398,24 @@ static inline struct page *pgd_page(pgd_t pgd)
 #define p4d_offset p4d_offset
 p4d_t *p4d_offset(pgd_t *pgd, unsigned long address);
 
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+static inline int pte_devmap(pte_t pte);
+static inline pte_t pmd_pte(pmd_t pmd);
+
+static inline int pmd_devmap(pmd_t pmd)
+{
+	return pte_devmap(pmd_pte(pmd));
+}
+
+static inline int pud_devmap(pud_t pud)
+{
+	return 0;
+}
+
+static inline int pgd_devmap(pgd_t pgd)
+{
+	return 0;
+}
+#endif
+
 #endif /* _ASM_RISCV_PGTABLE_64_H */
arch/riscv/include/asm/pgtable-bits.h +1 -0
index 179bd4afece4..a8f5205cea54 100644
@@ -19,6 +19,7 @@
 #define _PAGE_SOFT      (3 << 8)    /* Reserved for software */
 
 #define _PAGE_SPECIAL   (1 << 8)    /* RSW: 0x1 */
+#define _PAGE_DEVMAP    (1 << 9)    /* RSW, devmap */
 #define _PAGE_TABLE     _PAGE_PRESENT
 
 /*
arch/riscv/include/asm/pgtable.h +17 -0
index 41f1b2c6f949..dd4d52940106 100644
@@ -390,6 +390,13 @@ static inline int pte_special(pte_t pte)
 	return pte_val(pte) & _PAGE_SPECIAL;
 }
 
+#ifdef CONFIG_ARCH_HAS_PTE_DEVMAP
+static inline int pte_devmap(pte_t pte)
+{
+	return pte_val(pte) & _PAGE_DEVMAP;
+}
+#endif
+
 /* static inline pte_t pte_rdprotect(pte_t pte) */
 
 static inline pte_t pte_wrprotect(pte_t pte)
@@ -431,6 +438,11 @@ static inline pte_t pte_mkspecial(pte_t pte)
 	return __pte(pte_val(pte) | _PAGE_SPECIAL);
 }
 
+static inline pte_t pte_mkdevmap(pte_t pte)
+{
+	return __pte(pte_val(pte) | _PAGE_DEVMAP);
+}
+
 static inline pte_t pte_mkhuge(pte_t pte)
 {
 	return pte;
@@ -721,6 +733,11 @@ static inline pmd_t pmd_mkdirty(pmd_t pmd)
 	return pte_pmd(pte_mkdirty(pmd_pte(pmd)));
 }
 
+static inline pmd_t pmd_mkdevmap(pmd_t pmd)
+{
+	return pte_pmd(pte_mkdevmap(pmd_pte(pmd)));
+}
+
 static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr,
 				pmd_t *pmdp, pmd_t pmd)
 {
Signature · 1 line
2.43.0
#15 lore raw

PATCH v4 11/11 riscv: Enable DAX VMEMMAP optimization

In reply to Björn Töpel

From: Björn Töpel <[email protected]>

Now that DAX is usable, enable the DAX VMEMMAP optimization as well.

---
 arch/riscv/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/Kconfig +1 -0
index 8a49b5f4c017..1631bf568158 100644
@@ -73,6 +73,7 @@ config RISCV
 	select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT
 	select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
 	select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL
+	select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP
 	select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
 	select ARCH_WANTS_NO_INSTR
 	select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
Signature · 1 line
2.43.0
Reviewed-by on v4 5/11
#6 lore raw

Re: [PATCH v4 05/11] riscv: mm: Add pfn_to_kaddr() implementation

In reply to Björn Töpel

Hi Björn,

Quoted 29 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The pfn_to_kaddr() function is used by KASAN's memory hotplugging path. Add the missing function to the RISC-V port, so that it can be built with MHP and CONFIG_KASAN.

---
 arch/riscv/include/asm/page.h | 5 +++++
 1 file changed, 5 insertions(+)
arch/riscv/include/asm/page.h +5 -0
index 115ac98b8d72..235fd45d998d 100644
@@ -188,6 +188,11 @@ extern phys_addr_t __phys_addr_symbol(unsigned long
 x);
 
 unsigned long kaslr_offset(void);
 
+static __always_inline void *pfn_to_kaddr(unsigned long pfn)
+{
+       return __va(pfn << PAGE_SHIFT);
+}
+
 #endif /* __ASSEMBLY__ */
 
 #define virt_addr_valid(vaddr) ({
      \
Quoted 4 lines

--
2.43.0

Thanks,

Alex

Reviewed-by on v4 6/11
#9 lore raw

Re: [PATCH v4 06/11] riscv: mm: Add memory hotplugging support

In reply to Björn Töpel
Quoted 336 lines · Björn Töpel

From: Björn Töpel <[email protected]>

For an architecture to support memory hotplugging, a couple of callbacks needs to be implemented:

arch_add_memory()
This callback is responsible for adding the physical memory into the direct map, and call into the memory hotplugging generic code via __add_pages() that adds the corresponding struct page entries, and updates the vmemmap mapping.

arch_remove_memory()
This is the inverse of the callback above.

vmemmap_free()
This function tears down the vmemmap mappings (if CONFIG_SPARSEMEM_VMEMMAP is enabled), and also deallocates the backing vmemmap pages. Note that for persistent memory, an alternative allocator for the backing pages can be used; The vmem_altmap. This means that when the backing pages are cleared, extra care is needed so that the correct deallocation method is used.

arch_get_mappable_range()
This functions returns the PA range that the direct map can map. Used by the MHP internals for sanity checks.

The page table unmap/teardown functions are heavily based on code from the x86 tree. The same remove_pgd_mapping() function is used in both vmemmap_free() and arch_remove_memory(), but in the latter function the backing pages are not removed.

---
 arch/riscv/mm/init.c | 267 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 267 insertions(+)
arch/riscv/mm/init.c +85 -0
index 1f7e7c223bec..bfa2dea95354 100644
@@ -1534,3 +1534,270 @@ struct execmem_info __init
 *execmem_arch_setup(void)
 }
 #endif /* CONFIG_MMU */
 #endif /* CONFIG_EXECMEM */
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
+{
+       struct page *page = pmd_page(*pmd);
+       struct ptdesc *ptdesc = page_ptdesc(page);
+       pte_t *pte;
+       int i;
+
+       for (i = 0; i < PTRS_PER_PTE; i++) {
+               pte = pte_start + i;
+               if (!pte_none(*pte))
+                       return;
+       }
+
+       pagetable_pte_dtor(ptdesc);
+       if (PageReserved(page))
+               free_reserved_page(page);
+       else
+               pagetable_free(ptdesc);
+       pmd_clear(pmd);
+}
+
+static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud)
+{
+       struct page *page = pud_page(*pud);
+       struct ptdesc *ptdesc = page_ptdesc(page);
+       pmd_t *pmd;
+       int i;
+
+       for (i = 0; i < PTRS_PER_PMD; i++) {
+               pmd = pmd_start + i;
+               if (!pmd_none(*pmd))
+                       return;
+       }
+
+       pagetable_pmd_dtor(ptdesc);
+       if (PageReserved(page))
+               free_reserved_page(page);
+       else
+               pagetable_free(ptdesc);
+       pud_clear(pud);
+}
+
+static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d)
+{
+       struct page *page = p4d_page(*p4d);
+       pud_t *pud;
+       int i;
+
+       for (i = 0; i < PTRS_PER_PUD; i++) {
+               pud = pud_start + i;
+               if (!pud_none(*pud))
+                       return;
+       }
+
+       if (PageReserved(page))
+               free_reserved_page(page);
+       else
+               free_pages((unsigned long)page_address(page), 0);
+       p4d_clear(p4d);
+}
+
+static void __meminit free_vmemmap_storage(struct page *page, size_t size,
+                                          struct vmem_altmap *altmap)
+{
+       int order = get_order(size);
+
+       if (altmap) {
+               vmem_altmap_free(altmap, size >> PAGE_SHIFT);
+               return;
+       }
+
+       if (PageReserved(page)) {
+               unsigned int nr_pages = 1 << order;
+
+               while (nr_pages--)
+                       free_reserved_page(page++);
+               return;
+       }
+
+       free_pages((unsigned long)page_address(page), order);
+}
+
+static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long

addr, unsigned long end,
+ bool is_vmemmap, struct
vmem_altmap *altmap)

arch/riscv/mm/init.c +17 -0
+{
+       unsigned long next;
+       pte_t *ptep, pte;
+
+       for (; addr < end; addr = next) {
+               next = (addr + PAGE_SIZE) & PAGE_MASK;
+               if (next > end)
+                       next = end;
+
+               ptep = pte_base + pte_index(addr);
+               pte = ptep_get(ptep);
+               if (!pte_present(*ptep))
+                       continue;
+
+               pte_clear(&init_mm, addr, ptep);
+               if (is_vmemmap)
+                       free_vmemmap_storage(pte_page(pte), PAGE_SIZE,

altmap);

arch/riscv/mm/init.c +4 -0
+       }
+}
+
+static void __meminit remove_pmd_mapping(pmd_t *pmd_base, unsigned long

addr, unsigned long end,
+ bool is_vmemmap, struct
vmem_altmap *altmap)

arch/riscv/mm/init.c +16 -0
+{
+       unsigned long next;
+       pte_t *pte_base;
+       pmd_t *pmdp, pmd;
+
+       for (; addr < end; addr = next) {
+               next = pmd_addr_end(addr, end);
+               pmdp = pmd_base + pmd_index(addr);
+               pmd = pmdp_get(pmdp);
+               if (!pmd_present(pmd))
+                       continue;
+
+               if (pmd_leaf(pmd)) {
+                       pmd_clear(pmdp);
+                       if (is_vmemmap)
+                               free_vmemmap_storage(pmd_page(pmd),

PMD_SIZE, altmap);

arch/riscv/mm/init.c +5 -0
+                       continue;
+               }
+
+               pte_base = (pte_t *)pmd_page_vaddr(*pmdp);
+               remove_pte_mapping(pte_base, addr, next, is_vmemmap,

altmap);

arch/riscv/mm/init.c +5 -0
+               free_pte_table(pte_base, pmdp);
+       }
+}
+
+static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long

addr, unsigned long end,
+ bool is_vmemmap, struct
vmem_altmap *altmap)

arch/riscv/mm/init.c +23 -0
+{
+       unsigned long next;
+       pud_t *pudp, pud;
+       pmd_t *pmd_base;
+
+       for (; addr < end; addr = next) {
+               next = pud_addr_end(addr, end);
+               pudp = pud_base + pud_index(addr);
+               pud = pudp_get(pudp);
+               if (!pud_present(pud))
+                       continue;
+
+               if (pud_leaf(pud)) {
+                       if (pgtable_l4_enabled) {
+                               pud_clear(pudp);
+                               if (is_vmemmap)
+
 free_vmemmap_storage(pud_page(pud), PUD_SIZE, altmap);
+                       }
+                       continue;
+               }
+
+               pmd_base = pmd_offset(pudp, 0);
+               remove_pmd_mapping(pmd_base, addr, next, is_vmemmap,

altmap);

arch/riscv/mm/init.c +7 -0
+
+               if (pgtable_l4_enabled)
+                       free_pmd_table(pmd_base, pudp);
+       }
+}
+
+static void __meminit remove_p4d_mapping(p4d_t *p4d_base, unsigned long

addr, unsigned long end,
+ bool is_vmemmap, struct
vmem_altmap *altmap)

arch/riscv/mm/init.c +23 -0
+{
+       unsigned long next;
+       p4d_t *p4dp, p4d;
+       pud_t *pud_base;
+
+       for (; addr < end; addr = next) {
+               next = p4d_addr_end(addr, end);
+               p4dp = p4d_base + p4d_index(addr);
+               p4d = p4dp_get(p4dp);
+               if (!p4d_present(p4d))
+                       continue;
+
+               if (p4d_leaf(p4d)) {
+                       if (pgtable_l5_enabled) {
+                               p4d_clear(p4dp);
+                               if (is_vmemmap)
+
 free_vmemmap_storage(p4d_page(p4d), P4D_SIZE, altmap);
+                       }
+                       continue;
+               }
+
+               pud_base = pud_offset(p4dp, 0);
+               remove_pud_mapping(pud_base, addr, next, is_vmemmap,

altmap);

arch/riscv/mm/init.c +7 -0
+
+               if (pgtable_l5_enabled)
+                       free_pud_table(pud_base, p4dp);
+       }
+}
+
+static void __meminit remove_pgd_mapping(unsigned long va, unsigned long

end, bool is_vmemmap,

arch/riscv/mm/init.c +18 -0
+                                        struct vmem_altmap *altmap)
+{
+       unsigned long addr, next;
+       p4d_t *p4d_base;
+       pgd_t *pgd;
+
+       for (addr = va; addr < end; addr = next) {
+               next = pgd_addr_end(addr, end);
+               pgd = pgd_offset_k(addr);
+
+               if (!pgd_present(*pgd))
+                       continue;
+
+               if (pgd_leaf(*pgd))
+                       continue;
+
+               p4d_base = p4d_offset(pgd, 0);
+               remove_p4d_mapping(p4d_base, addr, next, is_vmemmap,

altmap);

arch/riscv/mm/init.c +23 -0
+       }
+
+       flush_tlb_all();
+}
+
+static void __meminit remove_linear_mapping(phys_addr_t start, u64 size)
+{
+       unsigned long va = (unsigned long)__va(start);
+       unsigned long end = (unsigned long)__va(start + size);
+
+       remove_pgd_mapping(va, end, false, NULL);
+}
+
+struct range arch_get_mappable_range(void)
+{
+       struct range mhp_range;
+
+       mhp_range.start = __pa(PAGE_OFFSET);
+       mhp_range.end = __pa(PAGE_END - 1);
+       return mhp_range;
+}
+
+int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params

*params)

arch/riscv/mm/init.c +4 -0
+{
+       int ret = 0;
+
+       create_linear_mapping_range(start, start + size, 0,

&params->pgprot);
+ ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT,
params);

arch/riscv/mm/init.c +14 -0
+       if (ret) {
+               remove_linear_mapping(start, size);
+               goto out;
+       }
+
+       max_pfn = PFN_UP(start + size);
+       max_low_pfn = max_pfn;
+
+ out:
+       flush_tlb_all();
+       return ret;
+}
+
+void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap

*altmap)

arch/riscv/mm/init.c +7 -0
+{
+       __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);
+       remove_linear_mapping(start, size);
+       flush_tlb_all();
+}
+
+void __ref vmemmap_free(unsigned long start, unsigned long end, struct

vmem_altmap *altmap)

arch/riscv/mm/init.c +4 -1
+{
+       remove_pgd_mapping(start, end, true, altmap);
+}
+#endif /* CONFIG_MEMORY_HOTPLUG */
--
Quoted 3 lines

2.43.0

Thanks,

Alex

Reviewed-by on v4 11/11
#16 lore raw

Re: [PATCH v4 11/11] riscv: Enable DAX VMEMMAP optimization

In reply to Björn Töpel
Quoted 22 lines · Björn Töpel

From: Björn Töpel <[email protected]>

Now that DAX is usable, enable the DAX VMEMMAP optimization as well.

---
 arch/riscv/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/Kconfig +1 -1
index 8a49b5f4c017..1631bf568158 100644
@@ -73,6 +73,7 @@ config RISCV
        select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT
        select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
        select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL
+       select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP
        select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
        select ARCH_WANTS_NO_INSTR
        select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
--
Quoted 3 lines

2.43.0

Thanks,

Alex

Reviewed-by on v4 5/11
#7 lore raw

Re: [PATCH v4 05/11] riscv: mm: Add pfn_to_kaddr() implementation

In reply to Björn Töpel

(resending as gmail client turned to html)

Quoted 29 lines · Björn Töpel

From: Björn Töpel <[email protected]>

The pfn_to_kaddr() function is used by KASAN's memory hotplugging path. Add the missing function to the RISC-V port, so that it can be built with MHP and CONFIG_KASAN.

---
 arch/riscv/include/asm/page.h | 5 +++++
 1 file changed, 5 insertions(+)
arch/riscv/include/asm/page.h +5 -1
index 115ac98b8d72..235fd45d998d 100644
@@ -188,6 +188,11 @@ extern phys_addr_t __phys_addr_symbol(unsigned long x);
 
 unsigned long kaslr_offset(void);
 
+static __always_inline void *pfn_to_kaddr(unsigned long pfn)
+{
+       return __va(pfn << PAGE_SHIFT);
+}
+
 #endif /* __ASSEMBLY__ */
 
 #define virt_addr_valid(vaddr) ({                                              \
--
Quoted 2 lines

2.43.0

Thanks,

Alex

Reviewed-by on v4 6/11
#10 lore raw

Re: [PATCH v4 06/11] riscv: mm: Add memory hotplugging support

In reply to Björn Töpel
Quoted 314 lines · Björn Töpel

From: Björn Töpel <[email protected]>

For an architecture to support memory hotplugging, a couple of callbacks needs to be implemented:

arch_add_memory()
This callback is responsible for adding the physical memory into the direct map, and call into the memory hotplugging generic code via __add_pages() that adds the corresponding struct page entries, and updates the vmemmap mapping.

arch_remove_memory()
This is the inverse of the callback above.

vmemmap_free()
This function tears down the vmemmap mappings (if CONFIG_SPARSEMEM_VMEMMAP is enabled), and also deallocates the backing vmemmap pages. Note that for persistent memory, an alternative allocator for the backing pages can be used; The vmem_altmap. This means that when the backing pages are cleared, extra care is needed so that the correct deallocation method is used.

arch_get_mappable_range()
This functions returns the PA range that the direct map can map. Used by the MHP internals for sanity checks.

The page table unmap/teardown functions are heavily based on code from the x86 tree. The same remove_pgd_mapping() function is used in both vmemmap_free() and arch_remove_memory(), but in the latter function the backing pages are not removed.

---
 arch/riscv/mm/init.c | 267 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 267 insertions(+)
arch/riscv/mm/init.c +267 -1
index 1f7e7c223bec..bfa2dea95354 100644
@@ -1534,3 +1534,270 @@ struct execmem_info __init *execmem_arch_setup(void)
 }
 #endif /* CONFIG_MMU */
 #endif /* CONFIG_EXECMEM */
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)
+{
+       struct page *page = pmd_page(*pmd);
+       struct ptdesc *ptdesc = page_ptdesc(page);
+       pte_t *pte;
+       int i;
+
+       for (i = 0; i < PTRS_PER_PTE; i++) {
+               pte = pte_start + i;
+               if (!pte_none(*pte))
+                       return;
+       }
+
+       pagetable_pte_dtor(ptdesc);
+       if (PageReserved(page))
+               free_reserved_page(page);
+       else
+               pagetable_free(ptdesc);
+       pmd_clear(pmd);
+}
+
+static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud)
+{
+       struct page *page = pud_page(*pud);
+       struct ptdesc *ptdesc = page_ptdesc(page);
+       pmd_t *pmd;
+       int i;
+
+       for (i = 0; i < PTRS_PER_PMD; i++) {
+               pmd = pmd_start + i;
+               if (!pmd_none(*pmd))
+                       return;
+       }
+
+       pagetable_pmd_dtor(ptdesc);
+       if (PageReserved(page))
+               free_reserved_page(page);
+       else
+               pagetable_free(ptdesc);
+       pud_clear(pud);
+}
+
+static void __meminit free_pud_table(pud_t *pud_start, p4d_t *p4d)
+{
+       struct page *page = p4d_page(*p4d);
+       pud_t *pud;
+       int i;
+
+       for (i = 0; i < PTRS_PER_PUD; i++) {
+               pud = pud_start + i;
+               if (!pud_none(*pud))
+                       return;
+       }
+
+       if (PageReserved(page))
+               free_reserved_page(page);
+       else
+               free_pages((unsigned long)page_address(page), 0);
+       p4d_clear(p4d);
+}
+
+static void __meminit free_vmemmap_storage(struct page *page, size_t size,
+                                          struct vmem_altmap *altmap)
+{
+       int order = get_order(size);
+
+       if (altmap) {
+               vmem_altmap_free(altmap, size >> PAGE_SHIFT);
+               return;
+       }
+
+       if (PageReserved(page)) {
+               unsigned int nr_pages = 1 << order;
+
+               while (nr_pages--)
+                       free_reserved_page(page++);
+               return;
+       }
+
+       free_pages((unsigned long)page_address(page), order);
+}
+
+static void __meminit remove_pte_mapping(pte_t *pte_base, unsigned long addr, unsigned long end,
+                                        bool is_vmemmap, struct vmem_altmap *altmap)
+{
+       unsigned long next;
+       pte_t *ptep, pte;
+
+       for (; addr < end; addr = next) {
+               next = (addr + PAGE_SIZE) & PAGE_MASK;
+               if (next > end)
+                       next = end;
+
+               ptep = pte_base + pte_index(addr);
+               pte = ptep_get(ptep);
+               if (!pte_present(*ptep))
+                       continue;
+
+               pte_clear(&init_mm, addr, ptep);
+               if (is_vmemmap)
+                       free_vmemmap_storage(pte_page(pte), PAGE_SIZE, altmap);
+       }
+}
+
+static void __meminit remove_pmd_mapping(pmd_t *pmd_base, unsigned long addr, unsigned long end,
+                                        bool is_vmemmap, struct vmem_altmap *altmap)
+{
+       unsigned long next;
+       pte_t *pte_base;
+       pmd_t *pmdp, pmd;
+
+       for (; addr < end; addr = next) {
+               next = pmd_addr_end(addr, end);
+               pmdp = pmd_base + pmd_index(addr);
+               pmd = pmdp_get(pmdp);
+               if (!pmd_present(pmd))
+                       continue;
+
+               if (pmd_leaf(pmd)) {
+                       pmd_clear(pmdp);
+                       if (is_vmemmap)
+                               free_vmemmap_storage(pmd_page(pmd), PMD_SIZE, altmap);
+                       continue;
+               }
+
+               pte_base = (pte_t *)pmd_page_vaddr(*pmdp);
+               remove_pte_mapping(pte_base, addr, next, is_vmemmap, altmap);
+               free_pte_table(pte_base, pmdp);
+       }
+}
+
+static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long addr, unsigned long end,
+                                        bool is_vmemmap, struct vmem_altmap *altmap)
+{
+       unsigned long next;
+       pud_t *pudp, pud;
+       pmd_t *pmd_base;
+
+       for (; addr < end; addr = next) {
+               next = pud_addr_end(addr, end);
+               pudp = pud_base + pud_index(addr);
+               pud = pudp_get(pudp);
+               if (!pud_present(pud))
+                       continue;
+
+               if (pud_leaf(pud)) {
+                       if (pgtable_l4_enabled) {
+                               pud_clear(pudp);
+                               if (is_vmemmap)
+                                       free_vmemmap_storage(pud_page(pud), PUD_SIZE, altmap);
+                       }
+                       continue;
+               }
+
+               pmd_base = pmd_offset(pudp, 0);
+               remove_pmd_mapping(pmd_base, addr, next, is_vmemmap, altmap);
+
+               if (pgtable_l4_enabled)
+                       free_pmd_table(pmd_base, pudp);
+       }
+}
+
+static void __meminit remove_p4d_mapping(p4d_t *p4d_base, unsigned long addr, unsigned long end,
+                                        bool is_vmemmap, struct vmem_altmap *altmap)
+{
+       unsigned long next;
+       p4d_t *p4dp, p4d;
+       pud_t *pud_base;
+
+       for (; addr < end; addr = next) {
+               next = p4d_addr_end(addr, end);
+               p4dp = p4d_base + p4d_index(addr);
+               p4d = p4dp_get(p4dp);
+               if (!p4d_present(p4d))
+                       continue;
+
+               if (p4d_leaf(p4d)) {
+                       if (pgtable_l5_enabled) {
+                               p4d_clear(p4dp);
+                               if (is_vmemmap)
+                                       free_vmemmap_storage(p4d_page(p4d), P4D_SIZE, altmap);
+                       }
+                       continue;
+               }
+
+               pud_base = pud_offset(p4dp, 0);
+               remove_pud_mapping(pud_base, addr, next, is_vmemmap, altmap);
+
+               if (pgtable_l5_enabled)
+                       free_pud_table(pud_base, p4dp);
+       }
+}
+
+static void __meminit remove_pgd_mapping(unsigned long va, unsigned long end, bool is_vmemmap,
+                                        struct vmem_altmap *altmap)
+{
+       unsigned long addr, next;
+       p4d_t *p4d_base;
+       pgd_t *pgd;
+
+       for (addr = va; addr < end; addr = next) {
+               next = pgd_addr_end(addr, end);
+               pgd = pgd_offset_k(addr);
+
+               if (!pgd_present(*pgd))
+                       continue;
+
+               if (pgd_leaf(*pgd))
+                       continue;
+
+               p4d_base = p4d_offset(pgd, 0);
+               remove_p4d_mapping(p4d_base, addr, next, is_vmemmap, altmap);
+       }
+
+       flush_tlb_all();
+}
+
+static void __meminit remove_linear_mapping(phys_addr_t start, u64 size)
+{
+       unsigned long va = (unsigned long)__va(start);
+       unsigned long end = (unsigned long)__va(start + size);
+
+       remove_pgd_mapping(va, end, false, NULL);
+}
+
+struct range arch_get_mappable_range(void)
+{
+       struct range mhp_range;
+
+       mhp_range.start = __pa(PAGE_OFFSET);
+       mhp_range.end = __pa(PAGE_END - 1);
+       return mhp_range;
+}
+
+int __ref arch_add_memory(int nid, u64 start, u64 size, struct mhp_params *params)
+{
+       int ret = 0;
+
+       create_linear_mapping_range(start, start + size, 0, &params->pgprot);
+       ret = __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, params);
+       if (ret) {
+               remove_linear_mapping(start, size);
+               goto out;
+       }
+
+       max_pfn = PFN_UP(start + size);
+       max_low_pfn = max_pfn;
+
+ out:
+       flush_tlb_all();
+       return ret;
+}
+
+void __ref arch_remove_memory(u64 start, u64 size, struct vmem_altmap *altmap)
+{
+       __remove_pages(start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap);
+       remove_linear_mapping(start, size);
+       flush_tlb_all();
+}
+
+void __ref vmemmap_free(unsigned long start, unsigned long end, struct vmem_altmap *altmap)
+{
+       remove_pgd_mapping(start, end, true, altmap);
+}
+#endif /* CONFIG_MEMORY_HOTPLUG */
--
Quoted 2 lines

2.43.0

Thanks,

Alex

Reviewed-by on v4 11/11
#17 lore raw

Re: [PATCH v4 11/11] riscv: Enable DAX VMEMMAP optimization

In reply to Björn Töpel
Quoted 23 lines · Björn Töpel

From: Björn Töpel <[email protected]>

Now that DAX is usable, enable the DAX VMEMMAP optimization as well.

---
 arch/riscv/Kconfig | 1 +
 1 file changed, 1 insertion(+)
arch/riscv/Kconfig +1 -1
index 8a49b5f4c017..1631bf568158 100644
@@ -73,6 +73,7 @@ config RISCV
        select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT
        select ARCH_WANT_HUGE_PMD_SHARE if 64BIT
        select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL
+       select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP
        select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP
        select ARCH_WANTS_NO_INSTR
        select ARCH_WANTS_THP_SWAP if HAVE_ARCH_TRANSPARENT_HUGEPAGE
--
Quoted 2 lines

2.43.0

Thanks,

Alex

#18 lore raw
In reply to Björn Töpel
This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Hello:

This series was applied to riscv/linux.git (for-next) by Palmer Dabbelt <[email protected]>:

Quoted 9 lines · On Wed, 5 Jun 2024 13:40:43 +0200 you wrote:

From: Björn Töpel <[email protected]>

================================================================
Memory Hot(Un)Plug support (and ZONE_DEVICE) for the RISC-V port
================================================================

(For the restless folks: change log in the bottom!)

[...]

Here is the summary with links:
- [v4,01/11] riscv: mm: Properly forward vmemmap_populate() altmap parameter

    https://git.kernel.org/riscv/c/e3ecf2fdc8f3
  - [v4,02/11] riscv: mm: Pre-allocate vmemmap/direct map/kasan PGD entries
    https://git.kernel.org/riscv/c/66673099f734
  - [v4,03/11] riscv: mm: Change attribute from __init to __meminit for page functions
    https://git.kernel.org/riscv/c/fe122b89da67
  - [v4,04/11] riscv: mm: Refactor create_linear_mapping_range() for memory hot add
    https://git.kernel.org/riscv/c/007480fe84a9
  - [v4,05/11] riscv: mm: Add pfn_to_kaddr() implementation
    https://git.kernel.org/riscv/c/6e6c5e21b8cb
  - [v4,06/11] riscv: mm: Add memory hotplugging support
    https://git.kernel.org/riscv/c/c75a74f4ba19
  - [v4,07/11] riscv: mm: Take memory hotplug read-lock during kernel page table dump
    https://git.kernel.org/riscv/c/37992b7f1097
  - [v4,08/11] riscv: Enable memory hotplugging for RISC-V
    https://git.kernel.org/riscv/c/f8c2a240556e
  - [v4,09/11] virtio-mem: Enable virtio-mem for RISC-V
    https://git.kernel.org/riscv/c/0546d7043e55
  - [v4,10/11] riscv: mm: Add support for ZONE_DEVICE
    https://git.kernel.org/riscv/c/216e04bf1e4d
  - [v4,11/11] riscv: Enable DAX VMEMMAP optimization
    https://git.kernel.org/riscv/c/4705c1571ad3

You are awesome, thank you!

Signature · 2 lines
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html