mm/vmemmap/devdax: Fix kernel crash when probing devdax devices
Subsystems (8)
PATCH 1/1 mm/vmemmap/devdax: Fix kernel crash when probing devdax devices
commit c4386bd8ee3a ("mm/memremap: add ZONE_DEVICE support for compound pages") added support for using optimized vmmemap for devdax devices. But how vmemmap mappings are created are architecture specific. For example, powerpc with hash translation doesn't have vmemmap mappings in init_mm page table instead they are bolted table entries in the hardware page table
vmemmap_populate_compound_pages() used by vmemmap optimization code is not aware of these architecture-specific mapping. Hence allow architecture to opt for this feature. I selected architectures supporting HUGETLB_PAGE_OPTIMIZE_VMEMMAP option as also supporting this feature. I added vmemmap_can_optimize() even though page_vmemmap_nr(pgmap) > 1 check should filter architecture not supporting this. IMHO that brings clarity to the code where we are populating vmemmap.
This patch fixes the below crash on ppc64.
BUG: Unable to handle kernel data access on write at 0xc00c000100400038 Faulting instruction address: 0xc000000001269d90 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries Modules linked in: CPU: 7 PID: 1 Comm: swapper/0 Not tainted 6.3.0-rc5-150500.34-default+ #2 5c90a668b6bbd142599890245c2fb5de19d7d28a Hardware name: IBM,9009-42G POWER9 (raw) 0x4e0202 0xf000005 of:IBM,FW950.40 (VL950_099) hv:phyp pSeries NIP: c000000001269d90 LR: c0000000004c57d4 CTR: 0000000000000000 REGS: c000000003632c30 TRAP: 0300 Not tainted (6.3.0-rc5-150500.34-default+) MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24842228 XER: 00000000 CFAR: c0000000004c57d0 DAR: c00c000100400038 DSISR: 42000000 IRQMASK: 0 .... NIP [c000000001269d90] __init_single_page.isra.74+0x14/0x4c LR [c0000000004c57d4] __init_zone_device_page+0x44/0xd0 Call Trace:
[c000000003632ed0] [c000000003632f60] 0xc000000003632f60 (unreliable) [c000000003632f10] [c0000000004c5ca0] memmap_init_zone_device+0x170/0x250 [c000000003632fe0] [c0000000005575f8] memremap_pages+0x2c8/0x7f0 [c0000000036330c0] [c000000000557b5c] devm_memremap_pages+0x3c/0xa0 [c000000003633100] [c000000000d458a8] dev_dax_probe+0x108/0x3e0 [c0000000036331a0] [c000000000d41430] dax_bus_probe+0xb0/0x140 [c0000000036331d0] [c000000000cef27c] really_probe+0x19c/0x520 [c000000003633260] [c000000000cef6b4] __driver_probe_device+0xb4/0x230 [c0000000036332e0] [c000000000cef888] driver_probe_device+0x58/0x120 [c000000003633320] [c000000000cefa6c] __device_attach_driver+0x11c/0x1e0 [c0000000036333a0] [c000000000cebc58] bus_for_each_drv+0xa8/0x130 [c000000003633400] [c000000000ceefcc] __device_attach+0x15c/0x250 [c0000000036334a0] [c000000000ced458] bus_probe_device+0x108/0x110 [c0000000036334f0] [c000000000ce92dc] device_add+0x7fc/0xa10 [c0000000036335b0] [c000000000d447c8] devm_create_dev_dax+0x1d8/0x530 [c000000003633640] [c000000000d46b60] __dax_pmem_probe+0x200/0x270 [c0000000036337b0] [c000000000d46bf0] dax_pmem_probe+0x20/0x70 [c0000000036337d0] [c000000000d2279c] nvdimm_bus_probe+0xac/0x2b0 [c000000003633860] [c000000000cef27c] really_probe+0x19c/0x520 [c0000000036338f0] [c000000000cef6b4] __driver_probe_device+0xb4/0x230 [c000000003633970] [c000000000cef888] driver_probe_device+0x58/0x120 [c0000000036339b0] [c000000000cefd08] __driver_attach+0x1d8/0x240 [c000000003633a30] [c000000000cebb04] bus_for_each_dev+0xb4/0x130 [c000000003633a90] [c000000000cee564] driver_attach+0x34/0x50 [c000000003633ab0] [c000000000ced878] bus_add_driver+0x218/0x300 [c000000003633b40] [c000000000cf1144] driver_register+0xa4/0x1b0 [c000000003633bb0] [c000000000d21a0c] __nd_driver_register+0x5c/0x100 [c000000003633c10] [c00000000206a2e8] dax_pmem_init+0x34/0x48 [c000000003633c30] [c0000000000132d0] do_one_initcall+0x60/0x320 [c000000003633d00] [c0000000020051b0] kernel_init_freeable+0x360/0x400 [c000000003633de0] [c000000000013764] kernel_init+0x34/0x1d0 [c000000003633e50] [c00000000000de14] ret_from_kernel_thread+0x5c/0x64
--- arch/arm64/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + drivers/dax/device.c | 3 ++- include/linux/mm.h | 16 ++++++++++++++++ mm/Kconfig | 3 +++ mm/sparse-vmemmap.c | 3 +-- 8 files changed, 26 insertions(+), 3 deletions(-)
arch/arm64/Kconfig +1 -0
@@ -103,6 +103,7 @@ config ARM64 select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES select ARCH_HAS_UBSAN_SANITIZE_ALL select ARM_AMBA
arch/loongarch/Kconfig +1 -0
@@ -56,6 +56,7 @@ config LOONGARCH select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select COMMON_CLK select CPU_PM
arch/s390/Kconfig +1 -0
@@ -127,6 +127,7 @@ config S390 select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCI
arch/x86/Kconfig +1 -0
@@ -127,6 +127,7 @@ config X86 select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT
drivers/dax/device.c +2 -1
@@ -448,7 +448,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type =MEMORY_DEVICE_GENERIC ; -if (dev_dax->align >PAGE_SIZE ) +if (dev_dax->align >PAGE_SIZE && +IS_ENABLED (CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP )) pgmap->vmemmap_shift =order_base_2 (dev_dax->align >>PAGE_SHIFT ); addr =devm_memremap_pages (dev, pgmap);
include/linux/mm.h +16 -0
@@ -3442,6 +3442,22 @@ void vmemmap_populate_print_last(void);void vmemmap_free (unsigned long start,unsigned long end,struct vmem_altmap *altmap);#endif + +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return is_power_of_2 (sizeof (struct page )) && + pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +} +#else +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return false; +} +#endif +void register_page_bootmem_memmap (unsigned long section_nr,struct page *map,unsigned long nr_pages);
mm/Kconfig +3 -0
@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. +config ARCH_WANT_OPTIMIZE_VMEMMAP + bool + config HAVE_MEMBLOCK_PHYS_MAP bool
mm/sparse-vmemmap.c +1 -2
@@ -458,8 +458,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION)))return NULL ; -if (is_power_of_2 (sizeof (struct page )) && - pgmap &&pgmap_vmemmap_nr (pgmap) >1 && !altmap) +if (vmemmap_can_optimize (altmap, pgmap)) r =vmemmap_populate_compound_pages (pfn, start, end, nid, pgmap);else r =vmemmap_populate (start, end, nid, altmap);
Signature · 1 line
2.39.2
mm/Kconfig config SPARSEMEM_VMEMMAP · line 466
Quoted 11 lines · Aneesh Kumar K.V
mm/Kconfig +3
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. +config ARCH_WANT_OPTIMIZE_VMEMMAP + bool +
Could this devdax specific config switch be added to drivers/dax/Kconfig ? also, how about adding 'DAX' to the config switch name?
BTW, I noticed something minor and unrelated in the original commit (c4386bd8ee3a):
-static unsigned long pfn_next(unsigned long pfn)
+static unsigned long pfn_next(struct dev_pagemap *pgmap, unsigned long pfn)
{
- if (pfn % 1024 == 0)
+ if (pfn % (1024 << pgmap->vmemmap_shift)) <---- this line
cond_resched();
- return pfn + 1;
+ return pfn + pgmap_vmemmap_nr(pgmap);
+}
should be
+ if (pfn % (1024 * pgmap_vmemmap_nr(pgmap))
to be consistent.
thanks,
-jane
Quoted 9 lines · Jane Chu
Quoted 8 lines · Aneesh Kumar K.V
mm/Kconfig +0
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available.
Quoted 6 lines
mm/Kconfig config SPARSEMEM_VMEMMAP · line 466
Quoted 3 lines
mm/Kconfig +3
-0+config ARCH_WANT_OPTIMIZE_VMEMMAP + bool +Could this devdax specific config switch be added to
drivers/dax/Kconfig? also, how about adding 'DAX' to the config switch name?
I would say we want to make it more generic. ie, both hugetlb and devdax can now derive the feature support via ARCH_WANT_OPTIMIZE_VMEMAP
commit aafb4790ea0250c8d2450e9d23a4be80c663d2ec
Author: Aneesh Kumar K.V <[email protected]> Date: Sat Apr 8 15:41:48 2023 +0530
mm/hugetlb: Remove ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
We can depend on the more generic ARCH_WACH_OPTIMIZE_VMEMAP which is now used to enable both hugetlb and devddax vmemmap optimization.
arch/arm64/Kconfig +0 -1
@@ -100,7 +100,6 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANT_OPTIMIZE_VMEMMAP
arch/loongarch/Kconfig +0 -1
@@ -53,7 +53,6 @@ config LOONGARCH select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANT_OPTIMIZE_VMEMMAP
arch/s390/Kconfig +0 -1
@@ -126,7 +126,6 @@ config S390 select ARCH_WANTS_NO_INSTR select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2
arch/x86/Kconfig +0 -1
@@ -125,7 +125,6 @@ config X86 select ARCH_WANTS_NO_INSTR select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64
fs/Kconfig +1 -8
@@ -250,16 +250,9 @@ config HUGETLBFS config HUGETLB_PAGE def_bool HUGETLBFS -# -# Select this config option from the architecture Kconfig, if it is preferred -# to enable the feature of HugeTLB Vmemmap Optimization (HVO). -# -config ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - bool - config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE - depends on ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + depends on ARCH_WANT_OPTIMIZE_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON
mm/Kconfig +4 -1
@@ -460,7 +460,10 @@ config SPARSEMEM_VMEMMAP SPARSEMEM_VMEMMAP uses a virtually mapped memmap to optimise pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. - +# +# Select this config option from the architecture Kconfig, if it is preferred +# to enable the feature of HugeTLB/dev_dax vmemmap optimization. +# config ARCH_WANT_OPTIMIZE_VMEMMAP bool
Quoted 18 lines · Aneesh Kumar K.V
Quoted 9 lines · Jane Chu
Quoted 8 lines · Aneesh Kumar K.V
mm/Kconfig +0
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available.Quoted 6 lines
mm/Kconfig config SPARSEMEM_VMEMMAP · line 466
Quoted 3 lines
mm/Kconfig +3
-0+config ARCH_WANT_OPTIMIZE_VMEMMAP + bool +Could this devdax specific config switch be added to
drivers/dax/Kconfig? also, how about adding 'DAX' to the config switch name?
Quoted 2 lines
I would say we want to make it more generic. ie, both hugetlb and devdax can now derive the feature support via
ARCH_WANT_OPTIMIZE_VMEMAP
The two config switches have different purposes, it's better to keep them separate. For example, recent hugetlb high granularity mapping (HGM) project requires to users to make a choice between HGM and hugetlb vmemmap optimization(at least for now), while one can keep devdax compound page support enabled.
Quoted 89 lines
commit
aafb4790ea0250c8d2450e9d23a4be80c663d2ecAuthor: Aneesh Kumar K.V <[email protected]> Date: Sat Apr 8 15:41:48 2023 +0530mm/hugetlb: Remove
ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAPWe can depend on the more generic
ARCH_WACH_OPTIMIZE_VMEMAPwhich is now used to enable both hugetlb and devddax vmemmap optimization.Signed-off-by Aneesh Kumar K.V <[email protected]>arch/arm64/Kconfig +0
-1@@ -100,7 +100,6 @@ config ARM64 select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANT_OPTIMIZE_VMEMMAParch/loongarch/Kconfig +0
-1@@ -53,7 +53,6 @@ config LOONGARCH select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR select ARCH_WANT_OPTIMIZE_VMEMMAParch/s390/Kconfig +0
-1@@ -126,7 +126,6 @@ config S390 select ARCH_WANTS_NO_INSTR select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2arch/x86/Kconfig +0
-1@@ -125,7 +125,6 @@ config X86 select ARCH_WANTS_NO_INSTR select ARCH_WANT_GENERAL_HUGETLB select ARCH_WANT_HUGE_PMD_SHARE - select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64fs/Kconfig +1
-8@@ -250,16 +250,9 @@ config HUGETLBFS config HUGETLB_PAGE def_bool HUGETLBFS -# -# Select this config option from the architecture Kconfig, if it is preferred -# to enable the feature of HugeTLB Vmemmap Optimization (HVO). -# -config ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - bool - config HUGETLB_PAGE_OPTIMIZE_VMEMMAP def_bool HUGETLB_PAGE - depends on ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + depends on ARCH_WANT_OPTIMIZE_VMEMMAP depends on SPARSEMEM_VMEMMAP config HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ONmm/Kconfig +0
-0@@ -460,7 +460,10 @@ config SPARSEMEM_VMEMMAP SPARSEMEM_VMEMMAP uses a virtually mapped memmap to optimise
mm/Kconfig config SPARSEMEM_VMEMMAP · line 469
Quoted 10 lines
mm/Kconfig +4
-1pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. - +# +# Select this config option from the architecture Kconfig, if it is preferred +# to enable the feature of HugeTLB/dev_dax vmemmap optimization. +# config ARCH_WANT_OPTIMIZE_VMEMMAP bool
thanks,
-jane
Quoted 19 lines · Jane Chu
Quoted 17 lines · Aneesh Kumar K.V
Quoted 14 lines · Jane Chu
Quoted 10 lines · Aneesh Kumar K.V
mm/Kconfig +0
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP
pfn_to_pageandpage_to_pfnoperations. This is the most efficient option when sufficient kernel resources are available. +configARCH_WANT_OPTIMIZE_VMEMMAP
+ bool
+Could this devdax specific config switch be added to
drivers/dax/Kconfig? also, how about adding 'DAX' to the config switch name?
Quoted 9 lines
Quoted 2 lines
I would say we want to make it more generic. ie, both hugetlb and devdax can now derive the feature support via
ARCH_WANT_OPTIMIZE_VMEMAPThe two config switches have different purposes, it's better to keep them separate. For example, recent hugetlb high granularity mapping (HGM) project requires to users to make a choice between HGM and hugetlb vmemmap optimization(at least for now), while one can keep devdax compound page support enabled.
Is it done by kconfig? If it helps:
* there's a static key hugetlb can use to tell if this is enabled or not like how page_fixed_fake_head() uses it:
if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key))
...
* there's a hugetlb page bit for that vmemmap optimized pages with HpageOptimized(page).
* there is a separate hugetlb CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP which is not the same as the ARCH kconfig.
But perhaps there's some relevance in the ARCH_HUGETLB specific to HGMv2 that I am unaware.
Quoted 98 lines
Quoted 92 lines
commit
aafb4790ea0250c8d2450e9d23a4be80c663d2ec
Author: Aneesh Kumar K.V <[email protected]> Date: Sat Apr 8 15:41:48 2023 +0530mm/hugetlb: Remove
ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
We can depend on the more genericARCH_WACH_OPTIMIZE_VMEMAPwhich is now used to enable both hugetlb and devddax vmemmap optimization.Signed-off-by Aneesh Kumar K.V <[email protected]>arch/arm64/Kconfig +0
-0@@ -100,7 +100,6 @@ config ARM64select
ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT
selectARCH_WANT_FRAME_POINTERS
selectARCH_WANT_HUGE_PMD_SHAREifARM64_4K_PAGES|| (ARM64_16K_PAGES&& !ARM64_VA_BITS_36)
- selectARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP
selectARCH_WANT_LD_ORPHAN_WARN
selectARCH_WANTS_NO_INSTR
selectARCH_WANT_OPTIMIZE_VMEMMAParch/loongarch/Kconfig +0
-0@@ -53,7 +53,6 @@ config LOONGARCHselect
ARCH_USE_QUEUED_RWLOCKS
selectARCH_USE_QUEUED_SPINLOCKSselectARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT- selectARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAPselectARCH_WANT_LD_ORPHAN_WARNselectARCH_WANTS_NO_INSTR
selectARCH_WANT_OPTIMIZE_VMEMMAParch/s390/Kconfig +0
-0@@ -126,7 +126,6 @@ config S390select
ARCH_WANTS_NO_INSTR
selectARCH_WANT_DEFAULT_BPF_JITselectARCH_WANT_IPC_PARSE_VERSION
- selectARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAPselectARCH_WANT_OPTIMIZE_VMEMMAPselectBUILDTIME_TABLE_SORT
selectCLONE_BACKWARDS2arch/x86/Kconfig +0
-0@@ -125,7 +125,6 @@ config X86select
ARCH_WANTS_NO_INSTR
selectARCH_WANT_GENERAL_HUGETLB
selectARCH_WANT_HUGE_PMD_SHARE
- select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP ifX86_64selectARCH_WANT_LD_ORPHAN_WARN
select ARCH_WANT_OPTIMIZE_VMEMMAP ifX86_64select ARCH_WANTS_THP_SWAP ifX86_64fs/Kconfig +0
-0@@ -250,16 +250,9 @@ config HUGETLBFSconfig
HUGETLB_PAGE
def_boolHUGETLBFS
-#fs/Kconfig +0
-6-# Select this config option from the architecture Kconfig, if it is preferred -# to enable the feature of HugeTLB Vmemmap Optimization (HVO). -# -config ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP - bool -config
HUGETLB_PAGE_OPTIMIZE_VMEMMAPdef_boolHUGETLB_PAGE
- depends onARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP+ depends onARCH_WANT_OPTIMIZE_VMEMMAPdepends onSPARSEMEM_VMEMMAP
configHUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ONmm/Kconfig +0
-0@@ -460,7 +460,10 @@ config SPARSEMEM_VMEMMAP
SPARSEMEM_VMEMMAPuses a virtually mapped memmap to optimisepfn_to_pageandpage_to_pfnoperations. This is the most efficient option when sufficient kernel resources are available.mm/Kconfig +4
-1- +# +# Select this config option from the architecture Kconfig, if it is preferred +# to enable the feature of HugeTLB/dev_dax vmemmap optimization. +#Quoted 3 lines
config
ARCH_WANT_OPTIMIZE_VMEMMAP
boolthanks,
-jane
Quoted 40 lines · Joao Martins
Quoted 19 lines · Jane Chu
Quoted 17 lines · Aneesh Kumar K.V
Quoted 14 lines · Jane Chu
mm/Kconfig config SPARSEMEM_VMEMMAP · line 465
Quoted 10 lines · Aneesh Kumar K.V
mm/Kconfig +2
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. +config ARCH_WANT_OPTIMIZE_VMEMMAP + bool +Could this devdax specific config switch be added to
drivers/dax/Kconfig? also, how about adding 'DAX' to the config switch name?Quoted 9 lines
Quoted 2 lines
I would say we want to make it more generic. ie, both hugetlb and devdax can now derive the feature support via
ARCH_WANT_OPTIMIZE_VMEMAPThe two config switches have different purposes, it's better to keep them separate. For example, recent hugetlb high granularity mapping (HGM) project requires to users to make a choice between HGM and hugetlb vmemmap optimization(at least for now), while one can keep devdax compound page support enabled.
Is it done by kconfig? If it helps:
* there's a static key hugetlb can use to tell if this is enabled or not like how
page_fixed_fake_head()uses it:if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key))
...
Quoted 9 lines
* there's a hugetlb page bit for that vmemmap optimized pages with HpageOptimized(page).
* there is a separate hugetlb
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAPwhich is not the same as the ARCH kconfig.But perhaps there's some relevance in the
ARCH_HUGETLBspecific to HGMv2 that I am unaware.
Looks like there is no consumer of CONFIG_ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP.
There are three ways to enable HVO via CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON alone, or a combination of CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP and boot param 'vmemmap_optimize_enabled' or sysctl variable /proc/sys/vm/hugetlb_optimize_vmemmap.
It seems that the devdax compound page support does not cross path with HVO which has a lot more moving arms, and so it's better to keep the devdax config switch local to devdax and leave HVO alone.
thanks!
-jane
Quoted 47 lines · Jane Chu
Quoted 36 lines · Joao Martins
Quoted 17 lines · Jane Chu
Quoted 16 lines · Aneesh Kumar K.V
Quoted 14 lines · Jane Chu
Quoted 10 lines · Aneesh Kumar K.V
mm/Kconfig +0
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP
pfn_to_pageandpage_to_pfnoperations. This is the most efficient option when sufficient kernel resources are available. +configARCH_WANT_OPTIMIZE_VMEMMAP
+ bool
+Could this devdax specific config switch be added to
drivers/dax/Kconfig? also, how about adding 'DAX' to the config switch name?Quoted 9 lines
Quoted 2 lines
I would say we want to make it more generic. ie, both hugetlb and devdax can now derive the feature support via
ARCH_WANT_OPTIMIZE_VMEMAPThe two config switches have different purposes, it's better to keep them separate. For example, recent hugetlb high granularity mapping (HGM) project requires to users to make a choice between HGM and hugetlb vmemmap optimization(at least for now), while one can keep devdax compound page support enabled.
Is it done by kconfig? If it helps:
* there's a static key hugetlb can use to tell if this is enabled or not like how
page_fixed_fake_head()uses it:if (!static_branch_unlikely(&hugetlb_optimize_vmemmap_key))
...Quoted 9 lines
* there's a hugetlb page bit for that vmemmap optimized pages with HpageOptimized(page).
* there is a separate hugetlb
CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAPwhich is not the same as the ARCH kconfig.But perhaps there's some relevance in the
ARCH_HUGETLBspecific to HGMv2 that I am unaware.
Quoted 7 lines
Looks like there is no consumer of
CONFIG_ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP. There are three ways to enable HVO viaCONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ONalone, or a combination ofCONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAPand boot param 'vmemmap_optimize_enabled' or sysctl variable/proc/sys/vm/hugetlb_optimize_vmemmap.It seems that the devdax compound page support
Let's call it devdax vmemmap deduplication as compound pages can work regardless of the trick.
Quoted 3 lines
does not cross path with HVO
which has a lot more moving arms, and so it's better to keep the devdax config switch local to devdax and leave HVO alone.
I agree; I would rather not change CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON nor CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP and its knobs. But to be clear the one I was talking about is ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP (which is selected by architecture) ... not the other ones. Which if it was a generic one would translate to 'can the architecture support deduplicated vmemmap' regardless of whether it is active or used by DAX/HVO. HVO would use its hugetlb kconfig knobs and function helpers pointed above to differentiate in the HVO-specific case as it does today. Perhaps we can hear also from Muchun on what he thinks is right for HVO.
Joao
Quoted 64 lines · Aneesh Kumar K.V
commit
c4386bd8ee3a("mm/memremap: addZONE_DEVICEsupport for compound pages") added support for using optimized vmmemap for devdax devices. But how vmemmap mappings are created are architecture specific. For example, powerpc with hash translation doesn't have vmemmap mappings ininit_mmpage table instead they are bolted table entries in the hardware page table
vmemmap_populate_compound_pages()used by vmemmap optimization code is not aware of these architecture-specific mapping. Hence allow architecture to opt for this feature. I selected architectures supportingHUGETLB_PAGE_OPTIMIZE_VMEMMAPoption as also supporting this feature. I addedvmemmap_can_optimize()even thoughpage_vmemmap_nr(pgmap)> 1 check should filter architecture not supporting this. IMHO that brings clarity to the code where we are populating vmemmap.This patch fixes the below crash on ppc64.
BUG: Unable to handle kernel data access on write at 0xc00c000100400038 Faulting instruction address: 0xc000000001269d90 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries Modules linked in: CPU: 7 PID: 1 Comm: swapper/0 Not tainted 6.3.0-rc5-150500.34-default+ #2 5c90a668b6bbd142599890245c2fb5de19d7d28a Hardware name: IBM,9009-42G POWER9 (raw) 0x4e0202 0xf000005 of:IBM,FW950.40 (VL950_099) hv:phyp pSeries NIP: c000000001269d90 LR: c0000000004c57d4 CTR: 0000000000000000 REGS: c000000003632c30 TRAP: 0300 Not tainted (6.3.0-rc5-150500.34-default+) MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24842228 XER: 00000000 CFAR: c0000000004c57d0 DAR: c00c000100400038 DSISR: 42000000 IRQMASK: 0 .... NIP [c000000001269d90] __init_single_page.isra.74+0x14/0x4c LR [c0000000004c57d4] __init_zone_device_page+0x44/0xd0 Call Trace:[
c000000003632ed0] [c000000003632f60]0xc000000003632f60(unreliable) [c000000003632f10] [c0000000004c5ca0]memmap_init_zone_device+0x170/0x250[c000000003632fe0] [c0000000005575f8]memremap_pages+0x2c8/0x7f0[c0000000036330c0] [c000000000557b5c]devm_memremap_pages+0x3c/0xa0[c000000003633100] [c000000000d458a8]dev_dax_probe+0x108/0x3e0[c0000000036331a0] [c000000000d41430]dax_bus_probe+0xb0/0x140[c0000000036331d0] [c000000000cef27c]really_probe+0x19c/0x520[c000000003633260] [c000000000cef6b4]__driver_probe_device+0xb4/0x230[c0000000036332e0] [c000000000cef888]driver_probe_device+0x58/0x120[c000000003633320] [c000000000cefa6c]__device_attach_driver+0x11c/0x1e0[c0000000036333a0] [c000000000cebc58]bus_for_each_drv+0xa8/0x130[c000000003633400] [c000000000ceefcc]__device_attach+0x15c/0x250[c0000000036334a0] [c000000000ced458]bus_probe_device+0x108/0x110[c0000000036334f0] [c000000000ce92dc]device_add+0x7fc/0xa10[c0000000036335b0] [c000000000d447c8]devm_create_dev_dax+0x1d8/0x530[c000000003633640] [c000000000d46b60]__dax_pmem_probe+0x200/0x270[c0000000036337b0] [c000000000d46bf0]dax_pmem_probe+0x20/0x70[c0000000036337d0] [c000000000d2279c]nvdimm_bus_probe+0xac/0x2b0[c000000003633860] [c000000000cef27c]really_probe+0x19c/0x520[c0000000036338f0] [c000000000cef6b4]__driver_probe_device+0xb4/0x230[c000000003633970] [c000000000cef888]driver_probe_device+0x58/0x120[c0000000036339b0] [c000000000cefd08]__driver_attach+0x1d8/0x240[c000000003633a30] [c000000000cebb04]bus_for_each_dev+0xb4/0x130[c000000003633a90] [c000000000cee564]driver_attach+0x34/0x50[c000000003633ab0] [c000000000ced878]bus_add_driver+0x218/0x300[c000000003633b40] [c000000000cf1144]driver_register+0xa4/0x1b0[c000000003633bb0] [c000000000d21a0c]__nd_driver_register+0x5c/0x100[c000000003633c10] [c00000000206a2e8]dax_pmem_init+0x34/0x48[c000000003633c30] [c0000000000132d0]do_one_initcall+0x60/0x320[c000000003633d00] [c0000000020051b0]kernel_init_freeable+0x360/0x400[c000000003633de0] [c000000000013764]kernel_init+0x34/0x1d0[c000000003633e50] [c00000000000de14]ret_from_kernel_thread+0x5c/0x64
Quoted 6 lines
Fixes c4386bd8ee3a ("mm/memremap: add ZONE_DEVICE support for compound pages")Cc Joao Martins <[email protected]>Cc Muchun Song <[email protected]>Cc Dan Williams <[email protected]>Reported-by Tarun Sahu <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>
Looks good to me,
...I guess device-dax is not often used on ppc if this has been lingering since v5.17.
Quoted 2 lines · Aneesh Kumar K.V
commit
c4386bd8ee3a("mm/memremap: addZONE_DEVICEsupport for compound pages") added support for using optimized vmmemap for devdax devices.
Not really. It added *compound pages* for devdax ZONE device which is what the commit describes. It is meant to represent the non base page metadata. Should fsdax support the equivalent, we can have GUP performance on the dax path and remove today's special zone device case of ref-per-base-page GUP-fast path.
One was tied when it was brought in but in theory[*] we didn't require anything from arch-es to do this (contrary to vmemmap_populate()). The commit you should refer to is this instead:
4917f55b4ef9 ("mm/sparse-vmemmap: improve memory savings for compound devmaps")
[*] or so I thought before next paragraph
Quoted 5 lines
But how vmemmap
mappings are created are architecture specific. For example, powerpc with hash translation doesn't have vmemmap mappings ininit_mmpage table instead they are bolted table entries in the hardware page table
Very interesting, I didn't know this (no init_mm) was an option. And my apologies for regressing s390 :/ So, s390 does not support vmemmap_populate_basepages() then and always need to go to arch vmemmap_populate() to create the vmemmap entries.
Quoted 8 lines
vmemmap_populate_compound_pages()used by vmemmap optimization code is not aware of these architecture-specific mapping. Hence allow architecture to opt for this feature. I selected architectures supportingHUGETLB_PAGE_OPTIMIZE_VMEMMAPoption as also supporting this feature. I addedvmemmap_can_optimize()even thoughpage_vmemmap_nr(pgmap)> 1 check should filter architecture not supporting this. > IMHO that brings clarity to the code where we are populating vmemmap.
I am not sure the last two sentences are right. I would remove, see above and below comments at the end on why.
Quoted 50 lines
This patch fixes the below crash on ppc64.
BUG: Unable to handle kernel data access on write at 0xc00c000100400038 Faulting instruction address: 0xc000000001269d90 Oops: Kernel access of bad area, sig: 11 [#1] LE PAGE_SIZE=64K MMU=Hash SMP NR_CPUS=2048 NUMA pSeries Modules linked in: CPU: 7 PID: 1 Comm: swapper/0 Not tainted 6.3.0-rc5-150500.34-default+ #2 5c90a668b6bbd142599890245c2fb5de19d7d28a Hardware name: IBM,9009-42G POWER9 (raw) 0x4e0202 0xf000005 of:IBM,FW950.40 (VL950_099) hv:phyp pSeries NIP: c000000001269d90 LR: c0000000004c57d4 CTR: 0000000000000000 REGS: c000000003632c30 TRAP: 0300 Not tainted (6.3.0-rc5-150500.34-default+) MSR: 8000000000009033 <SF,EE,ME,IR,DR,RI,LE> CR: 24842228 XER: 00000000 CFAR: c0000000004c57d0 DAR: c00c000100400038 DSISR: 42000000 IRQMASK: 0 .... NIP [c000000001269d90] __init_single_page.isra.74+0x14/0x4c LR [c0000000004c57d4] __init_zone_device_page+0x44/0xd0 Call Trace:[
c000000003632ed0] [c000000003632f60]0xc000000003632f60(unreliable) [c000000003632f10] [c0000000004c5ca0]memmap_init_zone_device+0x170/0x250[c000000003632fe0] [c0000000005575f8]memremap_pages+0x2c8/0x7f0[c0000000036330c0] [c000000000557b5c]devm_memremap_pages+0x3c/0xa0[c000000003633100] [c000000000d458a8]dev_dax_probe+0x108/0x3e0[c0000000036331a0] [c000000000d41430]dax_bus_probe+0xb0/0x140[c0000000036331d0] [c000000000cef27c]really_probe+0x19c/0x520[c000000003633260] [c000000000cef6b4]__driver_probe_device+0xb4/0x230[c0000000036332e0] [c000000000cef888]driver_probe_device+0x58/0x120[c000000003633320] [c000000000cefa6c]__device_attach_driver+0x11c/0x1e0[c0000000036333a0] [c000000000cebc58]bus_for_each_drv+0xa8/0x130[c000000003633400] [c000000000ceefcc]__device_attach+0x15c/0x250[c0000000036334a0] [c000000000ced458]bus_probe_device+0x108/0x110[c0000000036334f0] [c000000000ce92dc]device_add+0x7fc/0xa10[c0000000036335b0] [c000000000d447c8]devm_create_dev_dax+0x1d8/0x530[c000000003633640] [c000000000d46b60]__dax_pmem_probe+0x200/0x270[c0000000036337b0] [c000000000d46bf0]dax_pmem_probe+0x20/0x70[c0000000036337d0] [c000000000d2279c]nvdimm_bus_probe+0xac/0x2b0[c000000003633860] [c000000000cef27c]really_probe+0x19c/0x520[c0000000036338f0] [c000000000cef6b4]__driver_probe_device+0xb4/0x230[c000000003633970] [c000000000cef888]driver_probe_device+0x58/0x120[c0000000036339b0] [c000000000cefd08]__driver_attach+0x1d8/0x240[c000000003633a30] [c000000000cebb04]bus_for_each_dev+0xb4/0x130[c000000003633a90] [c000000000cee564]driver_attach+0x34/0x50[c000000003633ab0] [c000000000ced878]bus_add_driver+0x218/0x300[c000000003633b40] [c000000000cf1144]driver_register+0xa4/0x1b0[c000000003633bb0] [c000000000d21a0c]__nd_driver_register+0x5c/0x100[c000000003633c10] [c00000000206a2e8]dax_pmem_init+0x34/0x48[c000000003633c30] [c0000000000132d0]do_one_initcall+0x60/0x320[c000000003633d00] [c0000000020051b0]kernel_init_freeable+0x360/0x400[c000000003633de0] [c000000000013764]kernel_init+0x34/0x1d0[c000000003633e50] [c00000000000de14]ret_from_kernel_thread+0x5c/0x64
Quoted 1 line
Fixes c4386bd8ee3a ("mm/memremap: add ZONE_DEVICE support for compound pages")
It should be:
devmaps")
But if what you included in your patch was what lead to the crash, then your problem is not the vmemmap optimization ... as the latter came after. If it's the hash I included above, it would reduce the affected trees to v5.19+
Quoted 57 lines
Cc Joao Martins <[email protected]>Cc Muchun Song <[email protected]>Cc Dan Williams <[email protected]>Reported-by Tarun Sahu <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/arm64/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + drivers/dax/device.c | 3 ++- include/linux/mm.h | 16 ++++++++++++++++ mm/Kconfig | 3 +++ mm/sparse-vmemmap.c | 3 +-- 8 files changed, 26 insertions(+), 3 deletions(-)arch/arm64/Kconfig +1
-0@@ -103,6 +103,7 @@ config ARM64 select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES select ARCH_HAS_UBSAN_SANITIZE_ALL select ARM_AMBAarch/loongarch/Kconfig +1
-0@@ -56,6 +56,7 @@ config LOONGARCH select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select COMMON_CLK select CPU_PMarch/s390/Kconfig +1
-0@@ -127,6 +127,7 @@ config S390 select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCIarch/x86/Kconfig +0
-0
arch/x86/Kconfig config X86 · line 133
Quoted 7 lines
arch/x86/Kconfig +1
-0@@ -127,6 +127,7 @@ config X86 select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT
I would remove these and instead use the ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, and have your other snip be a second patch to drop the 'HUGETLB_' part
drivers/dax/device.c int dev_dax_probe(struct dev_dax *dev_dax) · line 454
Quoted 13 lines
drivers/dax/device.c +2
-1@@ -448,7 +448,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type =MEMORY_DEVICE_GENERIC ; -if (dev_dax->align >PAGE_SIZE ) +if (dev_dax->align >PAGE_SIZE && +IS_ENABLED (CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP )) pgmap->vmemmap_shift =order_base_2 (dev_dax->align >>PAGE_SHIFT );
vmemmap_shift relates to the compound page size we will use in memmap_init_zone_device(). Otherwise we are back to the base struct page on PMD/PUD case. Instead move the:
IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP)
further below to __populate_section_memmap() ...
Quoted 33 lines
drivers/dax/device.c +0
-0addr =devm_memremap_pages (dev, pgmap);include/linux/mm.h +16
-0@@ -3442,6 +3442,22 @@ void vmemmap_populate_print_last(void);void vmemmap_free (unsigned long start,unsigned long end,struct vmem_altmap *altmap);#endif + +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return is_power_of_2 (sizeof (struct page )) && + pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +} +#else +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return false; +} +#endif +void register_page_bootmem_memmap (unsigned long section_nr,struct page *map,unsigned long nr_pages);mm/Kconfig +0
-0
mm/Kconfig config SPARSEMEM_VMEMMAP · line 469
Quoted 9 lines
mm/Kconfig +3
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. +config ARCH_WANT_OPTIMIZE_VMEMMAP + bool + config HAVE_MEMBLOCK_PHYS_MAP bool
As you mentioned in your other followup email it probably makes sense that you just use ARCH_HUGETLB_WANT_OPTIMIZE_VMEMMAP if we switch this to per-arch. And then have that kconfig name be renamed into the above. It simplifies the fix in case it pickups stable trees (the dax vmemmap deduplication came after hugetlb). The case for the head page reusal case for hugetlb is anyways captured by a hugetlb static key.
Quoted 5 lines
mm/sparse-vmemmap.c +0
-0
mm/sparse-vmemmap.c struct page * __meminit __populate_section_memmap(unsigned long pfn, · line 464
Quoted 9 lines
mm/sparse-vmemmap.c +1
-2@@ -458,8 +458,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION)))return NULL ; -if (is_power_of_2 (sizeof (struct page )) && - pgmap &&pgmap_vmemmap_nr (pgmap) >1 && !altmap) +if (vmemmap_can_optimize (altmap, pgmap)) r =vmemmap_populate_compound_pages (pfn, start, end, nid, pgmap);else r =vmemmap_populate (start, end, nid, altmap);
Same comment(s) as further above. I think it should be:
if (IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP))
r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r = vmemmap_populate(start, end, nid, altmap);
In principle all these arches below can support vmemmap_populate_compound_pages() but now it would require them enable it explicitly, which I guess makes sense considering this bug you are fixing. But the arch-requirement is not really if we support optimized-vmemmap or not but if we support the init-mm at all. OTOH tieing both hugetlb and dax additionally eliminates confusion around an optimization shared by both, even if the dax could encompass more. I am not sure what is the arch requirement for hugetlb vmemmap opt these days.
arch/arm64/mm/mmu.c: return vmemmap_populate_basepages(start, end,
node, altmap);arch/ia64/mm/discontig.c: return vmemmap_populate_basepages(start, end,
node, NULL);arch/loongarch/mm/init.c: return vmemmap_populate_basepages(start, end,
node, NULL);
arch/riscv/mm/init.c: return vmemmap_populate_basepages(start, end, node, NULL); arch/x86/mm/init_64.c: err = vmemmap_populate_basepages(start, end,
node, NULL);arch/x86/mm/init_64.c: err = vmemmap_populate_basepages(start, end,
node, NULL);
Quoted 4 lines · Joao Martins
Quoted 2 lines · Aneesh Kumar K.V
commit
c4386bd8ee3a("mm/memremap: addZONE_DEVICEsupport for compound pages") added support for using optimized vmmemap for devdax devices.
Quoted 8 lines
Not really. It added *compound pages* for devdax ZONE device which is what the commit describes. It is meant to represent the non base page metadata. Should fsdax support the equivalent, we can have GUP performance on the dax path and remove today's special zone device case of ref-per-base-page GUP-fast path.
One was tied when it was brought in but in theory[*] we didn't require anything from arch-es to do this (contrary to
vmemmap_populate()). The commit you should refer to is this instead:
ok the compound page dependency came via
commit 6fd3620b3428 ("mm/page_alloc: reuse tail struct pages for compound devmaps")
I have now reworked it such that we do
static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,
unsigned long nr_pages)
{
#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP /* * this depends on how vmemmap is populated by * vmemmap_populate_compound_pages() */ return is_power_of_2(sizeof(struct page)) && !altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages; #else return nr_pages;
#endif
}
Quoted 24 lines
4917f55b4ef9("mm/sparse-vmemmap: improve memory savings for compound devmaps")[*] or so I thought before next paragraph
Quoted 5 lines
But how vmemmap
mappings are created are architecture specific. For example, powerpc with hash translation doesn't have vmemmap mappings ininit_mmpage table instead they are bolted table entries in the hardware page tableVery interesting, I didn't know this (no
init_mm) was an option. And my apologies for regressing s390 :/ So, s390 does not supportvmemmap_populate_basepages()then and always need to go to archvmemmap_populate()to create the vmemmap entries.Quoted 8 lines
vmemmap_populate_compound_pages()used by vmemmap optimization code is not aware of these architecture-specific mapping. Hence allow architecture to opt for this feature. I selected architectures supportingHUGETLB_PAGE_OPTIMIZE_VMEMMAPoption as also supporting this feature. I addedvmemmap_can_optimize()even thoughpage_vmemmap_nr(pgmap)> 1 check should filter architecture not supporting this. > IMHO that brings clarity to the code where we are populating vmemmap.
Quoted 2 lines
I am not sure the last two sentences are right. I would remove, see above and below comments at the end on why.
That is true because i had vmemmap_shift = 0 if arch didn't support vmemmap optimization. Based on your reply above, I have now moved that out and kept vmemmap_can_optimize()
modified mm/page_alloc.c
@@ -6846,8 +6846,16 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,
static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,
unsigned long nr_pages)
{
+#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP
+ /*
+ * this depends on how vmemmap is populated by
+ * vmemmap_populate_compound_pages()
+ */
return is_power_of_2(sizeof(struct page)) &&
!altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages;
+#else
+ return nr_pages;
+#endif
}
static void __ref memmap_init_compound(struct page *head,
modified mm/sparse-vmemmap.c
@@ -458,8 +458,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn,
!IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION)))
return NULL;
- if (is_power_of_2(sizeof(struct page)) &&
- pgmap && pgmap_vmemmap_nr(pgmap) > 1 && !altmap)
+ if (vmemmap_can_optimize(altmap, pgmap))
r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap);
else
r = vmemmap_populate(start, end, nid, altmap);
....
Quoted 7 lines
Quoted 2 lines
Fixes c4386bd8ee3a ("mm/memremap: add ZONE_DEVICE support for compound pages")It should be:
Fixes 4917f55b4ef9 ("mm/sparse-vmemmap: improve memory savings for compounddevmaps")
updated.
Quoted 132 lines
But if what you included in your patch was what lead to the crash, then your problem is not the vmemmap optimization ... as the latter came after. If it's the hash I included above, it would reduce the affected trees to v5.19+
Quoted 57 lines
Cc Joao Martins <[email protected]>Cc Muchun Song <[email protected]>Cc Dan Williams <[email protected]>Reported-by Tarun Sahu <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/arm64/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + drivers/dax/device.c | 3 ++- include/linux/mm.h | 16 ++++++++++++++++ mm/Kconfig | 3 +++ mm/sparse-vmemmap.c | 3 +-- 8 files changed, 26 insertions(+), 3 deletions(-)arch/arm64/Kconfig +1
-0@@ -103,6 +103,7 @@ config ARM64 select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES select ARCH_HAS_UBSAN_SANITIZE_ALL select ARM_AMBAarch/loongarch/Kconfig +1
-0@@ -56,6 +56,7 @@ config LOONGARCH select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select COMMON_CLK select CPU_PMarch/s390/Kconfig +1
-0@@ -127,6 +127,7 @@ config S390 select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCIarch/x86/Kconfig +0
-0arch/x86/Kconfig config X86 · line 133
Quoted 7 lines
arch/x86/Kconfig +1
-0@@ -127,6 +127,7 @@ config X86 select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORTI would remove these and instead use the
ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, and have your other snip be a second patch to drop the 'HUGETLB_' partdrivers/dax/device.c int dev_dax_probe(struct dev_dax *dev_dax) · line 454
Quoted 13 lines
drivers/dax/device.c +2
-1@@ -448,7 +448,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type =MEMORY_DEVICE_GENERIC ; -if (dev_dax->align >PAGE_SIZE ) +if (dev_dax->align >PAGE_SIZE && +IS_ENABLED (CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP )) pgmap->vmemmap_shift =order_base_2 (dev_dax->align >>PAGE_SHIFT );
vmemmap_shiftrelates to the compound page size we will use inmemmap_init_zone_device(). Otherwise we are back to the base struct page on PMD/PUD case. Instead move the:
IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP)further below to
__populate_section_memmap()...Quoted 36 lines
drivers/dax/device.c +0
-0addr =devm_memremap_pages (dev, pgmap);include/linux/mm.h +16
-0@@ -3442,6 +3442,22 @@ void vmemmap_populate_print_last(void);void vmemmap_free (unsigned long start,unsigned long end,struct vmem_altmap *altmap);#endif + +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return is_power_of_2 (sizeof (struct page )) && + pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +} +#else +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return false; +} +#endif +void register_page_bootmem_memmap (unsigned long section_nr,struct page *map,unsigned long nr_pages);mm/Kconfig +0
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available.
Quoted 12 lines
mm/Kconfig config SPARSEMEM_VMEMMAP · line 469
Quoted 6 lines
mm/Kconfig +3
-0+config ARCH_WANT_OPTIMIZE_VMEMMAP + bool + config HAVE_MEMBLOCK_PHYS_MAP boolAs you mentioned in your other followup email it probably makes sense that you just use
ARCH_HUGETLB_WANT_OPTIMIZE_VMEMMAPif we switch this to per-arch. And then have that kconfig name be renamed into the above. It simplifies the fix in case it pickups stable trees (the dax vmemmap deduplication came after hugetlb). The case for the head page reusal case for hugetlb is anyways captured by a hugetlb static key.
will do
Quoted 19 lines
Quoted 7 lines
mm/sparse-vmemmap.c +0
-0mm/sparse-vmemmap.c struct page * __meminit __populate_section_memmap(unsigned long pfn, · line 464
Quoted 9 lines
mm/sparse-vmemmap.c +1
-2@@ -458,8 +458,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION)))return NULL ; -if (is_power_of_2 (sizeof (struct page )) && - pgmap &&pgmap_vmemmap_nr (pgmap) >1 && !altmap) +if (vmemmap_can_optimize (altmap, pgmap)) r =vmemmap_populate_compound_pages (pfn, start, end, nid, pgmap);else r =vmemmap_populate (start, end, nid, altmap);Same comment(s) as further above. I think it should be:
Quoted 4 lines
if (
IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP))r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r = vmemmap_populate(start, end, nid, altmap);
Even if arch want vmemmap optimization the ability to use compound vmemmap_populate is further restricted by
if (is_power_of_2(sizeof(struct page)) && pgmap && pgmap_vmemmap_nr(pgmap) > 1 && !altmap)
Hence we still want if (vmemmap_can_optimize(..)) right?
Quoted 21 lines
In principle all these arches below can support
vmemmap_populate_compound_pages()but now it would require them enable it explicitly, which I guess makes sense considering this bug you are fixing. But the arch-requirement is not really if we support optimized-vmemmap or not but if we support the init-mm at all. OTOH tieing both hugetlb and dax additionally eliminates confusion around an optimization shared by both, even if the dax could encompass more. I am not sure what is the arch requirement for hugetlb vmemmap opt these days.
arch/arm64/mm/mmu.c: return vmemmap_populate_basepages(start, end,
node, altmap);arch/ia64/mm/discontig.c: return vmemmap_populate_basepages(start, end,
node, NULL);arch/loongarch/mm/init.c: return vmemmap_populate_basepages(start, end,
node, NULL);arch/riscv/mm/init.c: return vmemmap_populate_basepages(start, end, node, NULL); arch/x86/mm/init_64.c: err = vmemmap_populate_basepages(start, end,node, NULL);
arch/x86/mm/init_64.c: err = vmemmap_populate_basepages(start, end,
node, NULL);
Quoted 11 lines · Aneesh Kumar K.V
Quoted 9 lines · Joao Martins
Quoted 2 lines · Aneesh Kumar K.V
commit
c4386bd8ee3a("mm/memremap: addZONE_DEVICEsupport for compound pages") added support for using optimized vmmemap for devdax devices.Not really. It added *compound pages* for devdax ZONE device which is what the commit describes. It is meant to represent the non base page metadata. Should fsdax support the equivalent, we can have GUP performance on the dax path and remove today's special zone device case of ref-per-base-page GUP-fast path.
Quoted 7 lines
Quoted 3 lines
One was tied when it was brought in but in theory[*] we didn't require anything from arch-es to do this (contrary to
vmemmap_populate()). The commit you should refer to is this instead:ok the compound page dependency came via
commit6fd3620b3428("mm/page_alloc: reuse tail struct pages for compound devmaps")
It's an optimization to remove unnecessary work, not exactly a real dependency. But I understand what you mean.
Quoted 14 lines
I have now reworked it such that we do
static inline unsigned long compound_nr_pages(struct
vmem_altmap*altmap,
unsigned longnr_pages)
{#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP /* * this depends on how vmemmap is populated by * vmemmap_populate_compound_pages() */ return is_power_of_2(sizeof(struct page)) && !altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages; #else return nr_pages;
Quoted 3 lines
#endif
}
I would instead just simplify it to:
static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap,
unsigned long nr_pages)
{
return IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP) &&
is_power_of_2(sizeof(struct page)) &&
!altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages;
}
As the ternary false condition already handles the non-optimized-vmemmap case.
With your vmemmap_can_optimize() helper perhaps:
static inline unsigned long compound_nr_pages(struct dev_pagemap *pgmap,
struct vmem_altmap *altmap, unsigned long nr_pages)
{
return vmemmap_can_optimize(pgmap, altmap) ?
2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages; }
Quoted 24 lines
Quoted 24 lines
4917f55b4ef9("mm/sparse-vmemmap: improve memory savings for compound devmaps")[*] or so I thought before next paragraph
Quoted 5 lines
But how vmemmap
mappings are created are architecture specific. For example, powerpc with hash translation doesn't have vmemmap mappings ininit_mmpage table instead they are bolted table entries in the hardware page tableVery interesting, I didn't know this (no
init_mm) was an option. And my apologies for regressing s390 :/ So, s390 does not supportvmemmap_populate_basepages()then and always need to go to archvmemmap_populate()to create the vmemmap entries.Quoted 8 lines
vmemmap_populate_compound_pages()used by vmemmap optimization code is not aware of these architecture-specific mapping. Hence allow architecture to opt for this feature. I selected architectures supportingHUGETLB_PAGE_OPTIMIZE_VMEMMAPoption as also supporting this feature. I addedvmemmap_can_optimize()even thoughpage_vmemmap_nr(pgmap)> 1 check should filter architecture not supporting this. > IMHO that brings clarity to the code where we are populating vmemmap.
Quoted 7 lines
Quoted 2 lines
I am not sure the last two sentences are right. I would remove, see above and below comments at the end on why.
That is true because i had
vmemmap_shift= 0 if arch didn't support vmemmap optimization. Based on your reply above, I have now moved that out and keptvmemmap_can_optimize()
/me nods
Quoted 213 lines
modified
mm/page_alloc.c(quoted diff) +8
-0@@ -6846,8 +6846,16 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn, static inline unsigned long compound_nr_pages(struct vmem_altmap *altmap, unsigned long nr_pages) { +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP + /* + * this depends on how vmemmap is populated by + * vmemmap_populate_compound_pages() + */ return is_power_of_2(sizeof(struct page)) && !altmap ? 2 * (PAGE_SIZE / sizeof(struct page)) : nr_pages; +#else + return nr_pages; +#endif } static void __ref memmap_init_compound(struct page *head,modified
mm/sparse-vmemmap.cmm/sparse-vmemmap.c +1
-2@@ -458,8 +458,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION)))return NULL ; -if (is_power_of_2 (sizeof (struct page )) && - pgmap &&pgmap_vmemmap_nr (pgmap) >1 && !altmap) +if (vmemmap_can_optimize (altmap, pgmap)) r =vmemmap_populate_compound_pages (pfn, start, end, nid, pgmap);else r =vmemmap_populate (start, end, nid, altmap);Quoted 8 lines
Quoted 3 lines
Fixes c4386bd8ee3a ("mm/memremap: add ZONE_DEVICE support for compound pages")It should be:
Fixes 4917f55b4ef9 ("mm/sparse-vmemmap: improve memory savings for compounddevmaps")
updated.
Quoted 132 lines
But if what you included in your patch was what lead to the crash, then your problem is not the vmemmap optimization ... as the latter came after. If it's the hash I included above, it would reduce the affected trees to v5.19+
Quoted 57 lines
Cc Joao Martins <[email protected]>Cc Muchun Song <[email protected]>Cc Dan Williams <[email protected]>Reported-by Tarun Sahu <[email protected]>Signed-off-by Aneesh Kumar K.V <[email protected]>--- arch/arm64/Kconfig | 1 + arch/loongarch/Kconfig | 1 + arch/s390/Kconfig | 1 + arch/x86/Kconfig | 1 + drivers/dax/device.c | 3 ++- include/linux/mm.h | 16 ++++++++++++++++ mm/Kconfig | 3 +++ mm/sparse-vmemmap.c | 3 +-- 8 files changed, 26 insertions(+), 3 deletions(-)arch/arm64/Kconfig +1
-0@@ -103,6 +103,7 @@ config ARM64 select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select ARCH_WANTS_THP_SWAP if ARM64_4K_PAGES select ARCH_HAS_UBSAN_SANITIZE_ALL select ARM_AMBAarch/loongarch/Kconfig +1
-0@@ -56,6 +56,7 @@ config LOONGARCH select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_NO_INSTR + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select COMMON_CLK select CPU_PMarch/s390/Kconfig +1
-0@@ -127,6 +127,7 @@ config S390 select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_IPC_PARSE_VERSION select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP + select ARCH_WANT_OPTIMIZE_VMEMMAP select BUILDTIME_TABLE_SORT select CLONE_BACKWARDS2 select DMA_OPS if PCIarch/x86/Kconfig +0
-0arch/x86/Kconfig config X86 · line 133
Quoted 7 lines
arch/x86/Kconfig +1
-0@@ -127,6 +127,7 @@ config X86 select ARCH_WANT_HUGE_PMD_SHARE select ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANT_LD_ORPHAN_WARN + select ARCH_WANT_OPTIMIZE_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORTI would remove these and instead use the
ARCH_WANT_HUGETLB_PAGE_OPTIMIZE_VMEMMAP, and have your other snip be a second patch to drop the 'HUGETLB_' partdrivers/dax/device.c int dev_dax_probe(struct dev_dax *dev_dax) · line 454
Quoted 13 lines
drivers/dax/device.c +2
-1@@ -448,7 +448,8 @@ int dev_dax_probe(struct dev_dax *dev_dax) } pgmap->type =MEMORY_DEVICE_GENERIC ; -if (dev_dax->align >PAGE_SIZE ) +if (dev_dax->align >PAGE_SIZE && +IS_ENABLED (CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP )) pgmap->vmemmap_shift =order_base_2 (dev_dax->align >>PAGE_SHIFT );
vmemmap_shiftrelates to the compound page size we will use inmemmap_init_zone_device(). Otherwise we are back to the base struct page on PMD/PUD case. Instead move the:
IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP)further below to
__populate_section_memmap()...Quoted 36 lines
drivers/dax/device.c +0
-0addr =devm_memremap_pages (dev, pgmap);include/linux/mm.h +16
-0@@ -3442,6 +3442,22 @@ void vmemmap_populate_print_last(void);void vmemmap_free (unsigned long start,unsigned long end,struct vmem_altmap *altmap);#endif + +#ifdef CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return is_power_of_2 (sizeof (struct page )) && + pgmap && (pgmap_vmemmap_nr (pgmap) >1 ) && !altmap; +} +#else +static inline bool vmemmap_can_optimize (struct vmem_altmap *altmap, +struct dev_pagemap *pgmap) +{ +return false; +} +#endif +void register_page_bootmem_memmap (unsigned long section_nr,struct page *map,unsigned long nr_pages);mm/Kconfig +0
-0@@ -461,6 +461,9 @@ config SPARSEMEM_VMEMMAP pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available.Quoted 12 lines
mm/Kconfig config SPARSEMEM_VMEMMAP · line 469
Quoted 6 lines
mm/Kconfig +3
-0+config ARCH_WANT_OPTIMIZE_VMEMMAP + bool + config HAVE_MEMBLOCK_PHYS_MAP boolAs you mentioned in your other followup email it probably makes sense that you just use
ARCH_HUGETLB_WANT_OPTIMIZE_VMEMMAPif we switch this to per-arch. And then have that kconfig name be renamed into the above. It simplifies the fix in case it pickups stable trees (the dax vmemmap deduplication came after hugetlb). The case for the head page reusal case for hugetlb is anyways captured by a hugetlb static key.will do
Quoted 19 lines
Quoted 7 lines
mm/sparse-vmemmap.c +0
-0mm/sparse-vmemmap.c struct page * __meminit __populate_section_memmap(unsigned long pfn, · line 464
Quoted 9 lines
mm/sparse-vmemmap.c +1
-2@@ -458,8 +458,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, !IS_ALIGNED(nr_pages, PAGES_PER_SUBSECTION)))return NULL ; -if (is_power_of_2 (sizeof (struct page )) && - pgmap &&pgmap_vmemmap_nr (pgmap) >1 && !altmap) +if (vmemmap_can_optimize (altmap, pgmap)) r =vmemmap_populate_compound_pages (pfn, start, end, nid, pgmap);else r =vmemmap_populate (start, end, nid, altmap);Same comment(s) as further above. I think it should be:
Quoted 4 lines
if (
IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP))r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap); else r = vmemmap_populate(start, end, nid, altmap);
Quoted 8 lines
Even if arch want vmemmap optimization the ability to use compound
vmemmap_populateis further restricted byif (is_power_of_2(sizeof(struct page)) && pgmap && pgmap_vmemmap_nr(pgmap) > 1 && !altmap)Hence we still want if (
vmemmap_can_optimize(..)) right?
Correct. It would become:
if (IS_ENABLED(CONFIG_ARCH_WANT_OPTIMIZE_VMEMMAP) &&
is_power_of_2(sizeof(struct page)) &&
pgmap && pgmap_vmemmap_nr(pgmap) > 1 && !altmap)
r = vmemmap_populate_compound_pages(pfn, start, end, nid, pgmap);I suppose a helper could have this in vmemmap_can_optimize() as you suggest, as the condition is getting more unreadable. Probably worth using it too in compound_nr_pages(), but you would need to add a pgmap argument to that function and change its caller.
vmemmap_can_optimize() is not really the greatest name, but I can't think of anything better that aligns with the rest of naming of this feature.
Quoted 21 lines
Quoted 21 lines
In principle all these arches below can support
vmemmap_populate_compound_pages()but now it would require them enable it explicitly, which I guess makes sense considering this bug you are fixing. But the arch-requirement is not really if we support optimized-vmemmap or not but if we support the init-mm at all. OTOH tieing both hugetlb and dax additionally eliminates confusion around an optimization shared by both, even if the dax could encompass more. I am not sure what is the arch requirement for hugetlb vmemmap opt these days.
arch/arm64/mm/mmu.c: return vmemmap_populate_basepages(start, end,
node, altmap);arch/ia64/mm/discontig.c: return vmemmap_populate_basepages(start, end,
node, NULL);arch/loongarch/mm/init.c: return vmemmap_populate_basepages(start, end,
node, NULL);arch/riscv/mm/init.c: return vmemmap_populate_basepages(start, end, node, NULL); arch/x86/mm/init_64.c: err = vmemmap_populate_basepages(start, end,node, NULL);
arch/x86/mm/init_64.c: err = vmemmap_populate_basepages(start, end,
node, NULL);