Simplify special kernel page table handling
Subsystems (18)
cover Simplify special kernel page table handling
Page table constructors and destructors were originally introduced specifically for user PTE pages. Over time, their use became more widespread, and starting from v6.16 they are called at all levels and for both user and kernel page tables [1]. Or so I thought. Turns out, we still have various special kernel page tables where the ctor/dtor are not called; vmemmap page tables is one such case.
Looking closer, we do not actually need per-level ctors for kernel page tables. In fact we do not need ctors for them at all, because nothing is specific to kernel page tables in the ctor/dtor and nowadays they should be allocated with pagetable_alloc().
Here's the plan: all the generic page table handling moves to pagetable_{alloc,free}(). Whatever is left is specific to user page tables; the ctor/dtor are only called for user page tables at the required levels (PTE/PMD), and renamed to make that clear.
Doing this will allow us to remove a good amount of special casing, but there's quite a bit of churn to get there. Here's what this series does:
1. Introduce the concept of "kernel mm" and replace most comparisons with &init_mm with a flag check (patch 1-12); no functional change so far.
2. Move the page table accounting/type setting to pagetable_{alloc,free} (patch 13-15). Kernel mm's do not need to call the ctor/dtor any more.
3. Mark special mm's (efi_mm, tboot_mm on x86) as kernel mm's (patch 16-19). They now abide by the same rules as init_mm, which means in particular that split page table locks are no longer used.
4. Remove the ctor/dtor calls (now no-ops) for special kernel page tables on arm/arm64/riscv (patch 20-22).
[1] https://lore.kernel.org/all/[email protected]/
---
I stopped there for part 1 to keep it digestible. Part 2 will then ensure that the ctor/dtor are only called for user PTE/PMD pages and rename them accordingly. This is comparatively simpler but also involves a lot of churn.
This series was suggested in reaction to [2]; removing dtor calls entirely for kernel page tables systematically solves such issues.
Worth noting that some comparisons against &init_mm have been left untouched, as they are truly specific to init_mm. Reviewers are encouraged to take a look at the remaining cases:
$ git grep -nE '([=!]= *&init_mm|&init_mm *[=!]=)'
Documentation/mm/active_mm.rst:78: if (current->mm == &init_mm)
Documentation/translations/zh_CN/mm/active_mm.rst:76: if (current->mm == &init_mm)
arch/arm/include/asm/mmu_context.h:144: if (mm != &init_mm)
arch/arm64/include/asm/mmu_context.h:100: if (mm != &init_mm && !system_uses_ttbr0_pan())
arch/arm64/include/asm/mmu_context.h:210: if (mm == &init_mm)
arch/arm64/include/asm/mmu_context.h:242: if (next == &init_mm) {
arch/arm64/mm/fault.c:121: if (mm == &init_mm)
arch/arm64/mm/fault.c:139: if (mm == &init_mm) {
arch/arm64/mm/fault.c:154: mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K,
arch/arm64/mm/mmu.c:997: BUG_ON(mm == &init_mm);
arch/powerpc/xmon/xmon.c:3314: if (mm == &init_mm)
arch/riscv/mm/fault.c:42: mm == &init_mm ? (u64)__pa_symbol(mm->pgd) : virt_to_phys(mm->pgd));
kernel/cpu.c:917: WARN_ON(mm != &init_mm);
kernel/fork.c:725: BUG_ON(mm == &init_mm);
kernel/sched/core.c:8390: if (mm != &init_mm) {
mm/ptdump.c:182: if (mm != &init_mm)
mm/ptdump.c:191: if (mm != &init_mm)Testing:
* Built all modified architectures
* arm64
- mm kselftests (FVP)
- Hot adding/removing memory + memory-hotplug kselftests (QEMU)
- Reading /sys/kernel/debug/{efi,kernel}_page_tables
- Reading from /sys/firmware/efi/efivars/
*riscv
- memory-hotplug kselftests (QEMU)
* x86
- mm kselftests (lkvm)
- tboot page table setup using a fake tboot shared page (QEMU)
- CONFIG_EFI_PGT_DUMP=y (QEMU)
Patches based on mm-unstable.
[2] https://lore.kernel.org/all/[email protected]/
---
To: [email protected]
Cc: Andrew Morton <[email protected]>
Cc: David Hildenbrand <[email protected]>
Cc: Lorenzo Stoakes <[email protected]>
Cc: "Liam R. Howlett" <[email protected]>
Cc: Vlastimil Babka <[email protected]>
Cc: Mike Rapoport <[email protected]>
Cc: Suren Baghdasaryan <[email protected]>
Cc: Michal Hocko <[email protected]>
Cc: Pasha Tatashin <[email protected]>
---
Kevin Brodsky (22):
mm: drop unused __mm_flags_set_mask_bits_word()
mm: move mm_flags helpers to mm_types.h
mm: introduce MMF_KERNEL flag and set it for init_mm
mm: use mm_is_kernel() in generic page table code
arm64: mm: use mm_is_kernel() for kernel mm checks
loongarch: mm: use mm_is_kernel() in switch_mm_irqs_off()
parisc: mm: use mm_is_kernel() for kernel mm checks
powerpc: mm: use mm_is_kernel() for kernel mm checks
s390: mm: use mm_is_kernel() for kernel mm checks
sparc: mm: use mm_is_kernel() for kernel mm checks
um: mm: use mm_is_kernel() in TLB sync
x86/mm: use mm_is_kernel() for kernel mm checks
mm: account page table pages when allocated
mm: set page table page type when allocated
mm: only initialise pt_share_count for user pgtables
efi: mark efi_mm as a kernel mm
mm: pagewalk: drop redundant address check for kernel mm walks
arm64: mm: drop explicit mm_is_efi() check in contpte x86/tboot: mark tboot_mm as a kernel mm
arm64: mm: drop ctor/dtor calls for kernel page tables
arm: mm: drop ctor call for kernel page tables
riscv: mm: drop ctor/dtor calls for kernel page tables
arch/arm/mm/mmu.c | 3 +- arch/arm64/include/asm/pgalloc.h | 10 ++-- arch/arm64/mm/contpte.c | 26 ++++------ arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/mmu.c | 82 +++++++++----------------------- arch/loongarch/include/asm/mmu_context.h | 2 +- arch/parisc/include/asm/mmu_context.h | 2 +- arch/parisc/include/asm/pgalloc.h | 2 +- arch/parisc/include/asm/tlbflush.h | 2 +- arch/powerpc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/mmu_context.c | 2 +- arch/powerpc/mm/pgtable.c | 2 +- arch/riscv/mm/init.c | 63 ++++++------------------ arch/s390/include/asm/mmu_context.h | 2 +- arch/s390/include/asm/pgtable.h | 6 +-- arch/s390/mm/pgalloc.c | 4 +- arch/sparc/include/asm/mmu_context_64.h | 2 +- arch/sparc/include/asm/pgtable_64.h | 6 +-- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/tlb.c | 2 +- arch/um/kernel/tlb.c | 2 +- arch/x86/include/asm/tlbflush.h | 8 ++-- arch/x86/kernel/tboot.c | 4 +- arch/x86/mm/pgtable.c | 2 +- arch/x86/mm/tlb.c | 9 ++-- drivers/firmware/efi/efi.c | 1 + include/asm-generic/pgalloc.h | 16 +++---- include/linux/mm.h | 69 +++++++-------------------- include/linux/mm_types.h | 52 +++++++++++++++----- mm/init-mm.c | 1 + mm/memory.c | 6 +-- mm/page_table_check.c | 14 +++--- mm/pagewalk.c | 7 +-- 34 files changed, 163 insertions(+), 254 deletions(-)
---
base-commit: a531f0cf705af3374388eafe4115f651c625d8db
change-id: 20260611-remove_pgtable_cdtor-3b123938f8cd
PATCH 1/22 mm: drop unused __mm_flags_set_mask_bits_word()
Commit 39f8049cd49f ("mm: update coredump logic to correctly use bitmap mm flags") added coredump-specific helpers for the remaining masked mm-flags update users.
That left __mm_flags_set_mask_bits_word() without any callers. This is a special-purpose helper that is unlikely to be useful for other purposes; best to remove it.
--- include/linux/mm_types.h | 12 ------------ 1 file changed, 12 deletions(-)
include/linux/mm_types.h +0 -12
@@ -1432,18 +1432,6 @@ static inline unsigned long __mm_flags_get_word(const struct mm_struct *mm)return *__mm_flags_get_bitmap (mm); } -/* -* Update the first system word of mm flags ONLY, applying the specified mask to -* it, then setting all flags specified by bits. -*/ -static inline void __mm_flags_set_mask_bits_word (struct mm_struct *mm, -unsigned long mask,unsigned long bits) -{ -unsigned long *bitmap =ACCESS_PRIVATE (&mm->flags, __mm_flags); - -set_mask_bits (bitmap, mask, bits); -} -#define MM_MT_FLAGS (MT_FLAGS_ALLOC_RANGE | MT_FLAGS_LOCK_EXTERN | \ MT_FLAGS_USE_RCU)extern struct mm_struct init_mm;
Signature · 1 line
2.51.2
Re: [PATCH 01/22] mm: drop unused __mm_flags_set_mask_bits_word()
Quoted 11 lines · Kevin Brodsky
Commit
39f8049cd49f("mm: update coredump logic to correctly use bitmap mm flags") added coredump-specific helpers for the remaining masked mm-flags update users.That left
__mm_flags_set_mask_bits_word()without any callers. This is a special-purpose helper that is unlikely to be useful for other purposes; best to remove it.Assisted-by Codex:GPT-5.5Signed-off-by Kevin Brodsky <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH 2/22 mm: move mm_flags helpers to mm_types.h
In reply to Kevin BrodskySome of the mm_flags helpers are defined in <linux/mm_types.h>, while others are defined in <linux/mm.h>. We will soon need to call one of the latter helpers from mm_types.h.
Regroup all the helpers in mm_types.h to keep things simple.
Also remove #include's that were added specifically for these helpers in mm.h.
--- include/linux/mm.h | 32 -------------------------------- include/linux/mm_types.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+), 32 deletions(-)
include/linux/mm.h +0 -32
@@ -34,8 +34,6 @@#include <linux/slab.h> #include <linux/cacheinfo.h> #include <linux/rcuwait.h> -#include <linux/bitmap.h> -#include <linux/bitops.h> #include <linux/iommu-debug-pagealloc.h> #include <linux/kcsan-checks.h>
@@ -948,36 +946,6 @@ static inline void assert_fault_locked(const struct vm_fault *vmf) }#endif /* CONFIG_PER_VMA_LOCK */ -static inline bool mm_flags_test (int flag,const struct mm_struct *mm) -{ -return test_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); -} - -static inline bool mm_flags_test_and_set (int flag,struct mm_struct *mm) -{ -return test_and_set_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); -} - -static inline bool mm_flags_test_and_clear (int flag,struct mm_struct *mm) -{ -return test_and_clear_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); -} - -static inline void mm_flags_set (int flag,struct mm_struct *mm) -{ -set_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); -} - -static inline void mm_flags_clear (int flag,struct mm_struct *mm) -{ -clear_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); -} - -static inline void mm_flags_clear_all (struct mm_struct *mm) -{ -bitmap_zero (ACCESS_PRIVATE (&mm->flags, __mm_flags),NUM_MM_FLAG_BITS ); -} -extern const struct vm_operations_struct vma_dummy_vm_ops;static inline void vma_init (struct vm_area_struct *vma,struct mm_struct *mm)
include/linux/mm_types.h +30 -0
@@ -1414,6 +1414,36 @@ struct mm_struct {char flexible_array[]__aligned(__alignof__(unsigned long)) ; }; +static inline bool mm_flags_test (int flag,const struct mm_struct *mm) +{ +return test_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); +} + +static inline bool mm_flags_test_and_set (int flag,struct mm_struct *mm) +{ +return test_and_set_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); +} + +static inline bool mm_flags_test_and_clear (int flag,struct mm_struct *mm) +{ +return test_and_clear_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); +} + +static inline void mm_flags_set (int flag,struct mm_struct *mm) +{ +set_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); +} + +static inline void mm_flags_clear (int flag,struct mm_struct *mm) +{ +clear_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags)); +} + +static inline void mm_flags_clear_all (struct mm_struct *mm) +{ +bitmap_zero (ACCESS_PRIVATE (&mm->flags, __mm_flags),NUM_MM_FLAG_BITS ); +} +/* Copy value to the first system word of mm flags, non-atomically. */ static inline void __mm_flags_overwrite_word (struct mm_struct *mm,unsigned long value) {
Signature · 1 line
2.51.2
Re: [PATCH 02/22] mm: move mm_flags helpers to mm_types.h
Quoted 11 lines · Kevin Brodsky
Some of the
mm_flagshelpers are defined in <linux/mm_types.h>, while others are defined in <linux/mm.h>. We will soon need to call one of the latter helpers frommm_types.h.Regroup all the helpers in
mm_types.hto keep things simple.Also remove #include's that were added specifically for these helpers in
mm.h.Signed-off-by Kevin Brodsky <[email protected]>---
Signature · 3 lines
Cheers, David
PATCH 3/22 mm: introduce MMF_KERNEL flag and set it for init_mm
In reply to Kevin Brodskymm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with &init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g. efi_mm) may also represent parts of the kernel address space.
Introduce a new mm flag MMF_KERNEL and set it for init_mm. Subsequent patches will use this flag to replace comparisons with &init_mm. No functional change is introduced for now.
--- include/linux/mm_types.h | 10 ++++++++++ mm/init-mm.c | 1 + 2 files changed, 11 insertions(+)
include/linux/mm_types.h +10 -0
@@ -1414,6 +1414,8 @@ struct mm_struct {char flexible_array[]__aligned(__alignof__(unsigned long)) ; }; +#define MM_FLAGS_INIT (flags) { .__mm_flags = { BITMAP_FROM_U64(flags) } } +static inline bool mm_flags_test (int flag,const struct mm_struct *mm) {return test_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags));
@@ -2001,6 +2003,9 @@ enum {#define MMF_TOPDOWN 31/* mm searches top down by default */ #define MMF_TOPDOWN_MASK BIT(MMF_TOPDOWN) +#define MMF_KERNEL 32/* mm belongs to the kernel */ +#define MMF_KERNEL_MASK BIT_ULL(MMF_KERNEL) +#define MMF_INIT_LEGACY_MASK (MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)
@@ -2008,6 +2013,11 @@ enum {/* Legacy flags must fit within 32 bits. */ static_assert ((u64 )MMF_INIT_LEGACY_MASK <= (u64 )UINT_MAX ); +static inline bool mm_is_kernel (const struct mm_struct *mm) +{ +return mm &&mm_flags_test (MMF_KERNEL , mm); +} +/* * Initialise legacy flags according to masks, propagating selected flags on * fork. Further flag manipulation can be performed by the caller.
mm/init-mm.c +1 -0
@@ -34,6 +34,7 @@ struct mm_struct init_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .flags = MM_FLAGS_INIT(MMF_KERNEL_MASK), .write_protect_seq =SEQCNT_ZERO (init_mm.write_protect_seq),MMAP_LOCK_INITIALIZER (init_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock),
Signature · 1 line
2.51.2
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
include/linux/mm_types.h enum { · line 2019
Quoted 4 lines · Kevin Brodsky
include/linux/mm_types.h +4
-0+static inline bool mm_is_kernel (const struct mm_struct *mm) +{ +return mm &&mm_flags_test (MMF_KERNEL , mm); +}
Could we give this some nice comments explaining what a kernel mm is, please? Part of the problem with the init_mm checks is that they're magic and it's not always clear what's special about init_mm.
Maybe start with this list?
1. There's only one of them.
2. All kernel threads share it. tsk->mm is the same for all kernel threads.
3. It holds the reference copy of the kernel page tables
4. Userspace can't be entered when it is the current mm
5. It has different TLB flushing rules than userspace mms
I _think_ those are universal across all architectures.
Lorenzo Stoakes (ARM)
reviewerRe: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 12 lines · Dave Hansen
include/linux/mm_types.h enum { · line 2019
Quoted 4 lines · Kevin Brodsky
include/linux/mm_types.h +4
-0+static inline bool mm_is_kernel (const struct mm_struct *mm) +{ +return mm &&mm_flags_test (MMF_KERNEL , mm); +}Could we give this some nice comments explaining what a kernel mm is, please? Part of the problem with the
init_mmchecks is that they're magic and it's not always clear what's special aboutinit_mm.Maybe start with this list?
Quoted 8 lines
1. There's only one of them.
2. All kernel threads share it.tsk->mmis the same for all kernel threads.
3. It holds the reference copy of the kernel page tables
4. Userspace can't be entered when it is the current mm
5. It has different TLB flushing rules than userspace mmsI _think_ those are universal across all architectures.
Well point 1 isn't true of efimm or tboot_mm so we possibly need a better name :)
"Special" is overloaded too much already. I quite like "eternal" so:
static inline bool mm_is_eternal(const struct mm_struct *mm)
{
return mm && mm_flags_test(MMF_ETERNAL, mm);
}:)
Cheers, Lorenzo
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 9 lines · Lorenzo Stoakes (ARM)
Quoted 8 lines · Dave Hansen
include/linux/mm_types.h enum { · line 2019
Quoted 4 lines · Kevin Brodsky
include/linux/mm_types.h +4
-0+static inline bool mm_is_kernel (const struct mm_struct *mm) +{ +return mm &&mm_flags_test (MMF_KERNEL , mm); +}Could we give this some nice comments explaining what a kernel mm is, please? Part of the problem with the
init_mmchecks is that they're magic and it's not always clear what's special aboutinit_mm.
Agreed, we need to define what this property means exactly, and your comments on patch 14 show that extending it to efi_mm isn't necessarily as benign as it appeared to me at first.
My motivation really is about how page tables are handled (in particular whether ptlocks are used), so as you suggested on patch 19 maybe this flag should be narrower in scope, and the naming should reflect it. Possibly MMF_KERNEL_PGTABLES? That's at least one thing that should be true of all of init_mm, efi_mm and tboot_mm: their page tables use kernel permissions, not user, and should follow the same rules including not using ptlocks.
Quoted 13 lines
Quoted 10 lines
Maybe start with this list?
1. There's only one of them.
2. All kernel threads share it.tsk->mmis the same for all kernel threads.
3. It holds the reference copy of the kernel page tables
4. Userspace can't be entered when it is the current mm
5. It has different TLB flushing rules than userspace mmsI _think_ those are universal across all architectures.
Well point 1 isn't true of efimm or
tboot_mmso we possibly need a better name :)
Quoted 6 lines
"Special" is overloaded too much already. I quite like "eternal" so:
static inline bool mm_is_eternal(const struct mm_struct *mm) { return mm && mm_flags_test(MMF_ETERNAL, mm); }
Cheeky but why not! I do wonder whether this conveys the right idea though. The main point of this flag is that page tables are allocated/initialised/locked differently; in principle you could have a temporary non-user mm.
- Kevin
David Hildenbrand (Arm)
maintainerRe: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 38 lines · Kevin Brodsky
Quoted 4 lines · Lorenzo Stoakes (ARM)
Quoted 3 lines · Dave Hansen
Could we give this some nice comments explaining what a kernel mm is, please? Part of the problem with the
init_mmchecks is that they're magic and it's not always clear what's special aboutinit_mm.Agreed, we need to define what this property means exactly, and your comments on patch 14 show that extending it to
efi_mmisn't necessarily as benign as it appeared to me at first.My motivation really is about how page tables are handled (in particular whether ptlocks are used), so as you suggested on patch 19 maybe this flag should be narrower in scope, and the naming should reflect it. Possibly
MMF_KERNEL_PGTABLES? That's at least one thing that should be true of all ofinit_mm,efi_mmandtboot_mm: their page tables use kernel permissions, not user, and should follow the same rules including not using ptlocks.Quoted 13 lines
Quoted 10 lines
Maybe start with this list?
1. There's only one of them.
2. All kernel threads share it.tsk->mmis the same for all kernel threads.
3. It holds the reference copy of the kernel page tables
4. Userspace can't be entered when it is the current mm
5. It has different TLB flushing rules than userspace mmsI _think_ those are universal across all architectures.
Well point 1 isn't true of efimm or
tboot_mmso we possibly need a better name :)Quoted 6 lines
"Special" is overloaded too much already. I quite like "eternal" so:
static inline bool mm_is_eternal(const struct mm_struct *mm) { return mm && mm_flags_test(MMF_ETERNAL, mm); }
Quoted 4 lines
Cheeky but why not! I do wonder whether this conveys the right idea though. The main point of this flag is that page tables are allocated/initialised/locked differently; in principle you could have a temporary non-user mm.
I mean, we talk about MMs that are not used for actual user space processes?
IOW, the following:
arch/x86/kernel/tboot.c:static struct mm_struct tboot_mm = { drivers/firmware/efi/efi.c:struct mm_struct efi_mm = { drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c:static struct mm_struct sva_mm = { mm/init-mm.c:struct mm_struct init_mm = {
So "user" vs. "kernel" is quite intuitive. Sure, there can be multiple "kernel" ones, but each with a distinct purpose (IOW, we don't have multiple init_mm MMs).
Signature · 3 lines
Cheers, David
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 45 lines · David Hildenbrand (Arm)
Quoted 32 lines · Kevin Brodsky
Quoted 4 lines · Lorenzo Stoakes (ARM)
Quoted 3 lines · Dave Hansen
Could we give this some nice comments explaining what a kernel mm is, please? Part of the problem with the
init_mmchecks is that they're magic and it's not always clear what's special aboutinit_mm.Agreed, we need to define what this property means exactly, and your comments on patch 14 show that extending it to
efi_mmisn't necessarily as benign as it appeared to me at first.My motivation really is about how page tables are handled (in particular whether ptlocks are used), so as you suggested on patch 19 maybe this flag should be narrower in scope, and the naming should reflect it. Possibly
MMF_KERNEL_PGTABLES? That's at least one thing that should be true of all ofinit_mm,efi_mmandtboot_mm: their page tables use kernel permissions, not user, and should follow the same rules including not using ptlocks.Quoted 15 lines
Quoted 10 lines
Maybe start with this list?
1. There's only one of them.
2. All kernel threads share it.tsk->mmis the same for all kernel threads.
3. It holds the reference copy of the kernel page tables
4. Userspace can't be entered when it is the current mm
5. It has different TLB flushing rules than userspace mmsI _think_ those are universal across all architectures.
Well point 1 isn't true of efimm or
tboot_mmso we possibly need a better name :)"Special" is overloaded too much already. I quite like "eternal" so:
Quoted 8 lines
Quoted 4 lines
static inline bool mm_is_eternal(const struct mm_struct *mm) { return mm && mm_flags_test(MMF_ETERNAL, mm); }Cheeky but why not! I do wonder whether this conveys the right idea though. The main point of this flag is that page tables are allocated/initialised/locked differently; in principle you could have a temporary non-user mm.
I mean, we talk about MMs that are not used for actual user space processes?
IOW, the following:
Quoted 8 lines
arch/x86/kernel/tboot.c:static struct
mm_structtboot_mm= { drivers/firmware/efi/efi.c:structmm_structefi_mm= { drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3-test.c:static structmm_structsva_mm= { mm/init-mm.c:structmm_structinit_mm= {So "user" vs. "kernel" is quite intuitive. Sure, there can be multiple "kernel" ones, but each with a distinct purpose (IOW, we don't have multiple
init_mmMMs).
That was my rationale as well - anything that isn't a user mm is a kernel mm, in the sense that it owns kernel-space page tables.
However if we reduce the scope of the flag as per Dave Hansen's feedback, maybe MMF_KERNEL_PGTABLES is more appropriate. I think it's a question of whether we want to keep this flag with a narrow scope (strictly about page table handling), or possibly widen it later.
- Kevin
Christophe Leroy (CS GROUP)
reviewerRe: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
In reply to Kevin BrodskyLe 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.
Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mm is known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048 <assert_pte_locked>:
c0014048: 3d 40 c1 09 lis r10,-16119 c001404c: 39 4a 03 98 addi r10,r10,920 c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...
After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with 0x1. This load might be costly.
c0014048 <assert_pte_locked>:
c0014048: 2c 03 00 00 cmpwi r3,0 c001404c: 7c 85 23 78 mr r5,r4 c0014050: 41 82 00 10 beq c0014060 <assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3) c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelr
...
Christophe
Quoted 56 lines
Assisted-by Codex:GPT-5.5Signed-off-by Kevin Brodsky <[email protected]>--- include/linux/mm_types.h | 10 ++++++++++ mm/init-mm.c | 1 + 2 files changed, 11 insertions(+)include/linux/mm_types.h +10
-0@@ -1414,6 +1414,8 @@ struct mm_struct {char flexible_array[]__aligned(__alignof__(unsigned long)) ; }; +#define MM_FLAGS_INIT (flags) { .__mm_flags = { BITMAP_FROM_U64(flags) } } +static inline bool mm_flags_test (int flag,const struct mm_struct *mm) {return test_bit (flag,ACCESS_PRIVATE (&mm->flags, __mm_flags));@@ -2001,6 +2003,9 @@ enum {#define MMF_TOPDOWN 31/* mm searches top down by default */ #define MMF_TOPDOWN_MASK BIT(MMF_TOPDOWN) +#define MMF_KERNEL 32/* mm belongs to the kernel */ +#define MMF_KERNEL_MASK BIT_ULL(MMF_KERNEL) +#define MMF_INIT_LEGACY_MASK (MMF_DUMP_FILTER_MASK |\ MMF_DISABLE_THP_MASK | MMF_HAS_MDWE_MASK |\ MMF_VM_MERGE_ANY_MASK | MMF_TOPDOWN_MASK)@@ -2008,6 +2013,11 @@ enum {/* Legacy flags must fit within 32 bits. */ static_assert ((u64 )MMF_INIT_LEGACY_MASK <= (u64 )UINT_MAX ); +static inline bool mm_is_kernel (const struct mm_struct *mm) +{ +return mm &&mm_flags_test (MMF_KERNEL , mm); +} +/* * Initialise legacy flags according to masks, propagating selected flags on * fork. Further flag manipulation can be performed by the caller. mm/init-mm.c +1
-0@@ -34,6 +34,7 @@ struct mm_struct init_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .flags = MM_FLAGS_INIT(MMF_KERNEL_MASK), .write_protect_seq =SEQCNT_ZERO (init_mm.write_protect_seq),MMAP_LOCK_INITIALIZER (init_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock),
David Hildenbrand (Arm)
maintainerRe: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 29 lines · Christophe Leroy (CS GROUP)
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:
c0014048: 3d 40 c1 09 lis r10,-16119
c001404c: 39 4a 03 98 addi r10,r10,920
c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.
Quoted 7 lines
c0014048<assert_pte_locked>:
c0014048: 2c 03 00 00 cmpwi r3,0
c001404c: 7c 85 23 78 mr r5,r4
c0014050: 41 82 00 10 beqc0014060<assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3)
c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelr
Is that a real problem, though?
What we could do is to just keep the == &init_mm check internally on configs where we know that there is only a single such MM context.
Just a thought.
Signature · 3 lines
Cheers, David
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 27 lines · David Hildenbrand (Arm)
Quoted 26 lines · Christophe Leroy (CS GROUP)
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:
c0014048: 3d 40 c1 09 lis r10,-16119
c001404c: 39 4a 03 98 addi r10,r10,920
c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.
Quoted 8 lines
Quoted 7 lines
c0014048<assert_pte_locked>:
c0014048: 2c 03 00 00 cmpwi r3,0
c001404c: 7c 85 23 78 mr r5,r4
c0014050: 41 82 00 10 beqc0014060<assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3)
c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelrIs that a real problem, though?
I doubt it, this is a load from a fixed address and therefore highly cacheable/predictable. At worst we're keeping an extra cache line allocated, but I'd expect init_mm to be already in the cache most of the time.
Quoted 2 lines
What we could do is to just keep the ==
&init_mmcheck internally on configs where we know that there is only a single such MM context.
Sure, if this provides a clear performance benefit on powerpc or other architectures/configs that don't have an efi_mm we could do that.
- Kevin
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
In reply to David Hildenbrand (Arm)Quoted 38 lines · On Wed, 5 Aug 2026 14:07:01 +0200 "David Hildenbrand (Arm)" <[email protected]>…
Quoted 29 lines · Christophe Leroy (CS GROUP)
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:
c0014048: 3d 40 c1 09 lis r10,-16119
c001404c: 39 4a 03 98 addi r10,r10,920
c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.Quoted 7 lines
c0014048<assert_pte_locked>:
c0014048: 2c 03 00 00 cmpwi r3,0
c001404c: 7c 85 23 78 mr r5,r4
c0014050: 41 82 00 10 beqc0014060<assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3)
c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelr
Quoted 1 line
Is that a real problem, though?
The cost of the read is likely to matter most if the branch gets mispredicted. I'd also guess the mm isn't usually NULL - so that branch needs to statically predicted correctly as well.
I'd also not assume that the mm is in the cache (unless the surrounding code has already accessed it), the L1 data caches are small.
David
Quoted 6 lines
What we could do is to just keep the ==
&init_mmcheck internally on configs where we know that there is only a single such MM context.Just a thought.
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 38 lines · David Laight
Quoted 35 lines · On Wed, 5 Aug 2026 14:07:01 +0200 "David Hildenbrand (Arm)" <[email protected]>…
Quoted 34 lines · Christophe Leroy (CS GROUP)
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:
c0014048: 3d 40 c1 09 lis r10,-16119
c001404c: 39 4a 03 98 addi r10,r10,920
c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.
c0014048<assert_pte_locked>:
c0014048: 2c 03 00 00 cmpwi r3,0
c001404c: 7c 85 23 78 mr r5,r4
c0014050: 41 82 00 10 beqc0014060<assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3)
c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelr
Quoted 6 lines
Quoted 1 line
Is that a real problem, though?
The cost of the read is likely to matter most if the branch gets mispredicted. I'd also guess the mm isn't usually NULL - so that branch needs to statically predicted correctly as well.
I'd also not assume that the mm is in the cache (unless the surrounding code has already accessed it), the L1 data caches are small.
You're right, I didn't think about this the right way round - in general it may not be easy to predict that mm points to init_mm, and an arbitrary user mm isn't guaranteed to be cached. Repeated calls to mm_is_kernel(mm) in the same function shouldn't cost much though.
Would some (micro)benchmark be particularly relevant to measure the overhead, if any?
- Kevin
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 46 lines · On Tue, 11 Aug 2026 11:00:09 +0200 Kevin Brodsky <[email protected]> wrote:
Quoted 38 lines · David Laight
Quoted 35 lines · On Wed, 5 Aug 2026 14:07:01 +0200 "David Hildenbrand (Arm)" <[email protected]>…
Quoted 34 lines · Christophe Leroy (CS GROUP)
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:
c0014048: 3d 40 c1 09 lis r10,-16119
c001404c: 39 4a 03 98 addi r10,r10,920
c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.
c0014048<assert_pte_locked>:
c0014048: 2c 03 00 00 cmpwi r3,0
c001404c: 7c 85 23 78 mr r5,r4
c0014050: 41 82 00 10 beqc0014060<assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3)
c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelrQuoted 6 lines
Quoted 1 line
Is that a real problem, though?
The cost of the read is likely to matter most if the branch gets mispredicted. I'd also guess the mm isn't usually NULL - so that branch needs to statically predicted correctly as well.
I'd also not assume that the mm is in the cache (unless the surrounding code has already accessed it), the L1 data caches are small.
Quoted 4 lines
You're right, I didn't think about this the right way round - in general it may not be easy to predict that mm points to
init_mm, and an arbitrary user mm isn't guaranteed to be cached. Repeated calls tomm_is_kernel(mm)in the same function shouldn't cost much though.
Depends.
If it is real function call the code will be in the I-cache and is likely to get predicted correctly by the third call.
OTOH if inlined you've all the I-cache loads and that static branch prediction might get them all wrong.
Oh, and saving the result in a local might cause a register spill.
David
Quoted 5 lines
Would some (micro)benchmark be particularly relevant to measure the overhead, if any?
- Kevin
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
In reply to Christophe Leroy (CS GROUP)Quoted 28 lines · On Mon, 3 Aug 2026 15:59:35 +0200 "Christophe Leroy (CS GROUP)"…
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:c0014048: 3d 40 c1 09 lis r10,-16119 c001404c: 39 4a 03 98 addi r10,r10,920 c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.
Quoted 8 lines
c0014048<assert_pte_locked>:c0014048: 2c 03 00 00 cmpwi r3,0 c001404c: 7c 85 23 78 mr r5,r4 c0014050: 41 82 00 10 beq c0014060 <assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3) c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelr
...
Thought...
Could the check be replaced by one that checks for kernel static data? That would require that other mm that represent part of the kernel address space be static (and probably not in modules).
ISTR there is an address range check that can be used - and might be cheaper than the explicit test for init_mm on many 64bit archs.
David
Re: [PATCH 03/22] mm: introduce MMF_KERNEL flag and set it for init_mm
Quoted 38 lines · David Laight
Quoted 27 lines · On Mon, 3 Aug 2026 15:59:35 +0200 "Christophe Leroy (CS GROUP)"…
Le 14/07/2026 à 16:03, Kevin Brodsky a écrit :
Quoted 9 lines
mm code often needs to know whether some mm represents a kernel or user address space. This is currently done by comparing the mm pointer with
&init_mm; besides not being particularly elegant, this ignores the fact that other mm's (e.g.efi_mm) may also represent parts of the kernel address space.Introduce a new mm flag
MMF_KERNELand set it forinit_mm. Subsequent patches will use this flag to replace comparisons with&init_mm. No functional change is introduced for now.Did you consider performance impact ? This test is usually done in quite critical memory handling functions.
init_mmis known at link time. Before your patch 08/22 there is just a comparison of mm (r3) with a constant (loaded in r10):
c0014048<assert_pte_locked>:c0014048: 3d 40 c1 09 lis r10,-16119 c001404c: 39 4a 03 98 addi r10,r10,920 c0014050: 7c 03 50 00 cmpw r3,r10
c0014054: 4d 82 00 20 beqlr
...After patch 08/22 we have, it first checks that mm is not 0, then it loads the word located at mm+528 then AND it with
0x1. This load might be costly.Quoted 8 lines
c0014048<assert_pte_locked>:c0014048: 2c 03 00 00 cmpwi r3,0 c001404c: 7c 85 23 78 mr r5,r4 c0014050: 41 82 00 10 beq c0014060 <assert_pte_locked+0x18> c0014054: 81 23 02 10 lwz r9,528(r3) c0014058: 71 29 00 01 andi. r9,r9,1
c001405c: 4c 82 00 20 bnelr
...
Quoted 6 lines
Thought...
Could the check be replaced by one that checks for kernel static data? That would require that other mm that represent part of the kernel address space be static (and probably not in modules).
ISTR there is an address range check that can be used - and might be cheaper than the explicit test forinit_mmon many 64bit archs.
That is actually an interesting idea. Finding those "kernel mms" boils down to looking for global mm_struct, so checking if mm points to global data would achieve essentially the same thing without a flag (and likely faster). I'm not sure we want to bake in the assumption that "kernel mm" is equivalent to "mm is global" though.
- Kevin
PATCH 4/22 mm: use mm_is_kernel() in generic page table code
In reply to Kevin BrodskyGeneric page-table allocation, constructor and walker code uses init_mm comparisons to select kernel page table handling. Use mm_is_kernel() instead so kernel-owned mm's can share that handling once they are marked with MMF_KERNEL.
No functional change as only init_mm has MMF_KERNEL set for now.
--- include/asm-generic/pgalloc.h | 16 ++++++++-------- include/linux/mm.h | 4 ++-- mm/memory.c | 6 +++--- mm/page_table_check.c | 14 +++++++------- mm/pagewalk.c | 4 ++-- 5 files changed, 22 insertions(+), 22 deletions(-)
include/asm-generic/pgalloc.h +8 -8
@@ -138,7 +138,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long adstruct ptdesc *ptdesc;gfp_t gfp =GFP_PGTABLE_USER ; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp =GFP_PGTABLE_KERNEL ; ptdesc =pagetable_alloc_noprof (gfp,0 );if (!ptdesc)
@@ -148,7 +148,7 @@ static inline pmd_t *pmd_alloc_one_noprof(struct mm_struct *mm, unsigned long adreturn NULL ; } -if (mm == &init_mm) +if (mm_is_kernel (mm))ptdesc_set_kernel (ptdesc);return ptdesc_address (ptdesc);
@@ -175,7 +175,7 @@ static inline pud_t *__pud_alloc_one_noprof(struct mm_struct *mm, unsigned longgfp_t gfp =GFP_PGTABLE_USER ;struct ptdesc *ptdesc; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp =GFP_PGTABLE_KERNEL ; ptdesc =pagetable_alloc_noprof (gfp,0 );
@@ -184,7 +184,7 @@ static inline pud_t *__pud_alloc_one_noprof(struct mm_struct *mm, unsigned longpagetable_pud_ctor (ptdesc); -if (mm == &init_mm) +if (mm_is_kernel (mm))ptdesc_set_kernel (ptdesc);return ptdesc_address (ptdesc);
@@ -232,7 +232,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned longgfp_t gfp =GFP_PGTABLE_USER ;struct ptdesc *ptdesc; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp =GFP_PGTABLE_KERNEL ; ptdesc =pagetable_alloc_noprof (gfp,0 );
@@ -241,7 +241,7 @@ static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned longpagetable_p4d_ctor (ptdesc); -if (mm == &init_mm) +if (mm_is_kernel (mm))ptdesc_set_kernel (ptdesc);return ptdesc_address (ptdesc);
@@ -279,7 +279,7 @@ static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int ordergfp_t gfp =GFP_PGTABLE_USER ;struct ptdesc *ptdesc; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp =GFP_PGTABLE_KERNEL ; ptdesc =pagetable_alloc_noprof (gfp, order);
@@ -288,7 +288,7 @@ static inline pgd_t *__pgd_alloc_noprof(struct mm_struct *mm, unsigned int orderpagetable_pgd_ctor (ptdesc); -if (mm == &init_mm) +if (mm_is_kernel (mm))ptdesc_set_kernel (ptdesc);return ptdesc_address (ptdesc);
include/linux/mm.h +2 -2
@@ -3818,7 +3818,7 @@ static inline void pagetable_dtor_free(struct ptdesc *ptdesc)static inline bool pagetable_pte_ctor (struct mm_struct *mm,struct ptdesc *ptdesc) { -if (mm != &init_mm && !ptlock_init (ptdesc)) +if (!mm_is_kernel (mm) && !ptlock_init (ptdesc))return false;__pagetable_ctor (ptdesc);return true;
@@ -3909,7 +3909,7 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd)static inline bool pagetable_pmd_ctor (struct mm_struct *mm,struct ptdesc *ptdesc) { -if (mm != &init_mm && !pmd_ptlock_init (ptdesc)) +if (!mm_is_kernel (mm) && !pmd_ptlock_init (ptdesc))return false;ptdesc_pmd_pts_init (ptdesc);__pagetable_ctor (ptdesc);
mm/memory.c +3 -3
@@ -3394,13 +3394,13 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,spinlock_t *ptl;if (create) { - mapped_pte = pte = (mm == &init_mm) ? + mapped_pte = pte =mm_is_kernel (mm) ?pte_alloc_kernel_track (pmd, addr, mask) :pte_alloc_map_lock (mm, pmd, addr, &ptl);if (!pte)return -ENOMEM ; }else { - mapped_pte = pte = (mm == &init_mm) ? + mapped_pte = pte =mm_is_kernel (mm) ?pte_offset_kernel (pmd, addr) :pte_offset_map_lock (mm, pmd, addr, &ptl);if (!pte)
@@ -3422,7 +3422,7 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,lazy_mmu_mode_disable (); -if (mm != &init_mm) +if (!mm_is_kernel (mm))pte_unmap_unlock (mapped_pte, ptl);return err; }
mm/page_table_check.c +7 -7
@@ -148,7 +148,7 @@ void __page_table_check_zero(struct page *page, unsigned int order)void __page_table_check_pte_clear (struct mm_struct *mm,unsigned long addr,pte_t pte) { -if (&init_mm == mm) +if (mm_is_kernel (mm))return ;if (pte_user_accessible_page (mm, addr, pte))
@@ -159,7 +159,7 @@ EXPORT_SYMBOL(__page_table_check_pte_clear);void __page_table_check_pmd_clear (struct mm_struct *mm,unsigned long addr,pmd_t pmd) { -if (&init_mm == mm) +if (mm_is_kernel (mm))return ;if (pmd_user_accessible_page (mm, addr, pmd))
@@ -170,7 +170,7 @@ EXPORT_SYMBOL(__page_table_check_pmd_clear);void __page_table_check_pud_clear (struct mm_struct *mm,unsigned long addr,pud_t pud) { -if (&init_mm == mm) +if (mm_is_kernel (mm))return ;if (pud_user_accessible_page (mm, addr, pud))
@@ -201,7 +201,7 @@ void __page_table_check_ptes_set(struct mm_struct *mm, unsigned long addr, {unsigned int i; -if (&init_mm == mm) +if (mm_is_kernel (mm))return ;page_table_check_pte_flags (pte);
@@ -231,7 +231,7 @@ void __page_table_check_pmds_set(struct mm_struct *mm, unsigned long addr,unsigned long stride =PMD_SIZE >>PAGE_SHIFT ;unsigned int i; -if (&init_mm == mm) +if (mm_is_kernel (mm))return ;page_table_check_pmd_flags (pmd);
@@ -249,7 +249,7 @@ void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr,unsigned long stride =PUD_SIZE >>PAGE_SHIFT ;unsigned int i; -if (&init_mm == mm) +if (mm_is_kernel (mm))return ;for (i =0 ; i < nr; i++)
@@ -263,7 +263,7 @@ void __page_table_check_pte_clear_range(struct mm_struct *mm, unsigned long addr, pmd_t pmd) { -if (&init_mm == mm) + if (mm_is_kernel(mm)) return;if (!pmd_bad (pmd) && !pmd_leaf (pmd)) {
mm/pagewalk.c +2 -2
@@ -72,13 +72,13 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,* fit its pmd_bad() check (_PAGE_NX set and _PAGE_RW clear), * and CONFIG_EFI_PGT_DUMP efi_mm goes so far as to walk them. */ -if (walk->mm == &init_mm || addr >=TASK_SIZE ) +if (mm_is_kernel (walk->mm) || addr >=TASK_SIZE ) pte =pte_offset_kernel (pmd, addr);else pte =pte_offset_map (pmd, addr);if (pte) { err =walk_pte_range_inner (pte, addr, end, walk); -if (walk->mm != &init_mm && addr <TASK_SIZE ) +if (!mm_is_kernel (walk->mm) && addr <TASK_SIZE )pte_unmap (pte); } }else {
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH 04/22] mm: use mm_is_kernel() in generic page table code
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] apply_to_pte_range() routes all kernel MMs to pte_alloc_kernel_track(), which explicitly…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] apply_to_pte_range() routes all kernel MMs to pte_alloc_kernel_track(), which explicitly hardcodes &init_mm, leading to incorrect locking and mm attribution. --
--- Patch [4]: [PATCH 04/22] mm: use mm_is_kernel() in generic page table code --- commit 94fec0def7956a4362140e84c87769a3c5a046b2
Author: Kevin Brodsky <[email protected]>
mm: use mm_is_kernel() in generic page table code
Generic page-table allocation, constructor and walker code uses init_mm comparisons to select kernel page table handling. Use mm_is_kernel() instead so kernel-owned mm's can share that handling once they are marked with MMF_KERNEL.
Quoted 4 lines
mm/memory.c +0
-0
[ ... ]
mm/memory.c static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 3398
Quoted 7 lines
mm/memory.c +1
-1@@ -3394,13 +3394,13 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,spinlock_t *ptl;if (create) { - mapped_pte = pte = (mm == &init_mm) ? + mapped_pte = pte =mm_is_kernel (mm) ? pte_alloc_kernel_track(pmd, addr, mask) :
[Severity: Medium]
If a non-init_mm kernel MM (like efi_mm) has MMF_KERNEL set in the future, will this route it to pte_alloc_kernel_track() which hardcodes init_mm?
Looking at __pte_alloc_kernel() in mm/memory.c, which pte_alloc_kernel_track() calls:
int __pte_alloc_kernel(pmd_t *pmd)
{
pte_t *new = pte_alloc_one_kernel(&init_mm);
if (!new)
return -ENOMEM;
spin_lock(&init_mm.page_table_lock);...
Since __pte_alloc_kernel() drops the mm context and explicitly locks init_mm.page_table_lock instead of the target MM's lock, would this lead to data races during concurrent page table modifications for other kernel MMs?
Quoted 3 lines
mm/memory.c +0
-0pte_alloc_map_lock (mm, pmd, addr, &ptl);if (!pte)return -ENOMEM ;
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Re: [PATCH 04/22] mm: use mm_is_kernel() in generic page table code
In reply to Kevin Brodskymm/memory.c static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 3399
Quoted 12 lines · Kevin Brodsky
mm/memory.c +1
-1@@ -3394,13 +3394,13 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,spinlock_t *ptl;if (create) { - mapped_pte = pte = (mm == &init_mm) ? + mapped_pte = pte =mm_is_kernel (mm) ?pte_alloc_kernel_track (pmd, addr, mask) :pte_alloc_map_lock (mm, pmd, addr, &ptl);
As noted by Sashiko, this is a little problematic because if apply_to_page_range() is called on efi_mm, then we will end up with __pte_alloc_kernel() taking the init_mm lock, since that's hardcoded.
That probably means we should pass the mm to __pte_alloc_kernel().
FWIW calling the kernel alloc function for efi_mm is the correct thing to do: this ensures we use pmd_populate_kernel() and therefore kernel page table attributes (just like efi_create_mapping()). This path is currently never hit because EFI regions whose permissions are modified are PTE-mapped, but it is clearly more correct to use the kernel path for such mm's.
- Kevin
David Hildenbrand (Arm)
maintainerRe: [PATCH 04/22] mm: use mm_is_kernel() in generic page table code
Quoted 18 lines · Kevin Brodsky
mm/memory.c static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 3399
Quoted 12 lines · Kevin Brodsky
mm/memory.c +1
-1@@ -3394,13 +3394,13 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,spinlock_t *ptl;if (create) { - mapped_pte = pte = (mm == &init_mm) ? + mapped_pte = pte =mm_is_kernel (mm) ?pte_alloc_kernel_track (pmd, addr, mask) :pte_alloc_map_lock (mm, pmd, addr, &ptl);As noted by Sashiko, this is a little problematic because if
apply_to_page_range()is called onefi_mm, then we will end up with__pte_alloc_kernel()taking theinit_mmlock, since that's hardcoded.
Quoted 1 line
That probably means we should pass the mm to
__pte_alloc_kernel().
Agreed.
I guess we have to audit all existing init_mm usage?
Signature · 3 lines
Cheers, David
Re: [PATCH 04/22] mm: use mm_is_kernel() in generic page table code
Quoted 18 lines · David Hildenbrand (Arm)
Quoted 17 lines · Kevin Brodsky
mm/memory.c static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, · line 3399
Quoted 12 lines · Kevin Brodsky
mm/memory.c +1
-1@@ -3394,13 +3394,13 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,spinlock_t *ptl;if (create) { - mapped_pte = pte = (mm == &init_mm) ? + mapped_pte = pte =mm_is_kernel (mm) ?pte_alloc_kernel_track (pmd, addr, mask) :pte_alloc_map_lock (mm, pmd, addr, &ptl);As noted by Sashiko, this is a little problematic because if
apply_to_page_range()is called onefi_mm, then we will end up with__pte_alloc_kernel()taking theinit_mmlock, since that's hardcoded.
Quoted 4 lines
Quoted 1 line
That probably means we should pass the mm to
__pte_alloc_kernel().Agreed.
I guess we have to audit all existing
init_mmusage?
Indeed, I looked through comparisons with &init_mm but really all references to init_mm should be checked. Will do for v2.
- Kevin
PATCH 5/22 arm64: mm: use mm_is_kernel() for kernel mm checks
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
In some cases, such as show_pte(), the handling is truly specific to init_mm and we leave those comparisons with &init_mm unchanged.
No functional change, as only init_mm has MMF_KERNEL set for now.
mm_is_user() will eventually be replaced with !mm_is_kernel(), once efi_mm is also marked with MMF_KERNEL.
--- arch/arm64/include/asm/pgalloc.h | 10 +++++----- arch/arm64/mm/contpte.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- 3 files changed, 7 insertions(+), 7 deletions(-)
arch/arm64/include/asm/pgalloc.h +5 -5
@@ -30,7 +30,7 @@ static inline void pud_populate(struct mm_struct *mm, pud_t *pudp, pmd_t *pmdp) {pudval_t pudval =PUD_TYPE_TABLE |PUD_TABLE_AF ; - pudval |= (mm == &init_mm) ?PUD_TABLE_UXN :PUD_TABLE_PXN ; + pudval |=mm_is_kernel (mm) ?PUD_TABLE_UXN :PUD_TABLE_PXN ;__pud_populate (pudp,__pa (pmdp), pudval); }#else
@@ -52,7 +52,7 @@ static inline void p4d_populate(struct mm_struct *mm, p4d_t *p4dp, pud_t *pudp) {p4dval_t p4dval =P4D_TYPE_TABLE |P4D_TABLE_AF ; - p4dval |= (mm == &init_mm) ?P4D_TABLE_UXN :P4D_TABLE_PXN ; + p4dval |=mm_is_kernel (mm) ?P4D_TABLE_UXN :P4D_TABLE_PXN ;__p4d_populate (p4dp,__pa (pudp), p4dval); }
@@ -81,7 +81,7 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) {pgdval_t pgdval =PGD_TYPE_TABLE |PGD_TABLE_AF ; - pgdval |= (mm == &init_mm) ?PGD_TABLE_UXN :PGD_TABLE_PXN ; + pgdval |=mm_is_kernel (mm) ?PGD_TABLE_UXN :PGD_TABLE_PXN ;__pgd_populate (pgdp,__pa (p4dp), pgdval); }
@@ -108,7 +108,7 @@ static inline void __pmd_populate(pmd_t *pmdp, phys_addr_t ptep,static inline void pmd_populate_kernel (struct mm_struct *mm,pmd_t *pmdp,pte_t *ptep) { -VM_BUG_ON (mm && mm != &init_mm); +VM_BUG_ON (mm && !mm_is_kernel (mm));__pmd_populate (pmdp,__pa (ptep),PMD_TYPE_TABLE |PMD_TABLE_AF |PMD_TABLE_UXN ); }
@@ -116,7 +116,7 @@ pmd_populate_kernel(struct mm_struct *mm, pmd_t *pmdp, pte_t *ptep)static inline void pmd_populate (struct mm_struct *mm,pmd_t *pmdp,pgtable_t ptep) { -VM_BUG_ON (mm == &init_mm); +VM_BUG_ON (mm_is_kernel (mm));__pmd_populate (pmdp,page_to_phys (ptep),PMD_TYPE_TABLE |PMD_TABLE_AF |PMD_TABLE_PXN ); }
arch/arm64/mm/contpte.c +1 -1
@@ -18,7 +18,7 @@ static inline bool mm_is_user(struct mm_struct *mm)*/ if (unlikely (mm_is_efi (mm)))return false; -return mm != &init_mm; +return !mm_is_kernel (mm); }static inline pte_t *contpte_align_down (pte_t *ptep)
arch/arm64/mm/hugetlbpage.c +1 -1
@@ -212,7 +212,7 @@ static void clear_flush(struct mm_struct *mm,for (i =0 ; i < ncontig; i++, addr += pgsize, ptep++)__ptep_get_and_clear_anysz (mm, addr, ptep, pgsize); -if (mm == &init_mm) +if (mm_is_kernel (mm))flush_tlb_kernel_range (saddr, addr);else __flush_hugetlb_tlb_range (&vma, saddr, addr, pgsize,TLBF_NOWALKCACHE );
Signature · 1 line
2.51.2
PATCH 6/22 loongarch: mm: use mm_is_kernel() in switch_mm_irqs_off()
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
--- arch/loongarch/include/asm/mmu_context.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/loongarch/include/asm/mmu_context.h +1 -1
@@ -95,7 +95,7 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *if (!asid_valid (next, cpu))get_new_mmu_context (next, cpu, &need_flush); -if (next != &init_mm) +if (!mm_is_kernel (next))atomic_update_pgd_asid (cpu_asid (cpu, next), (unsigned long )next->pgd);else atomic_update_pgd_asid (cpu_asid (cpu, next), (unsigned long )invalid_pg_dir);
Signature · 1 line
2.51.2
PATCH 7/22 parisc: mm: use mm_is_kernel() for kernel mm checks
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
--- arch/parisc/include/asm/mmu_context.h | 2 +- arch/parisc/include/asm/pgalloc.h | 2 +- arch/parisc/include/asm/tlbflush.h | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
arch/parisc/include/asm/mmu_context.h +1 -1
@@ -87,7 +87,7 @@ static inline void activate_mm(struct mm_struct *prev, struct mm_struct *next)* already, so we should be OK. */ -BUG_ON (next == &init_mm);/* Should never happen */ +BUG_ON (mm_is_kernel (next));/* Should never happen */ if (next->context.space_id ==0 ) next->context.space_id =alloc_sid ();
arch/parisc/include/asm/pgalloc.h +1 -1
@@ -34,7 +34,7 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address)struct ptdesc *ptdesc;gfp_t gfp =GFP_PGTABLE_USER ; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp =GFP_PGTABLE_KERNEL ; ptdesc =pagetable_alloc (gfp,PMD_TABLE_ORDER );if (!ptdesc)
arch/parisc/include/asm/tlbflush.h +1 -1
@@ -35,7 +35,7 @@ int __flush_tlb_range(unsigned long sid,static inline void flush_tlb_mm (struct mm_struct *mm) { -BUG_ON (mm == &init_mm);/* Should never happen */ +BUG_ON (mm_is_kernel (mm));/* Should never happen */ #if 1 || defined(CONFIG_SMP )/* Except for very small threads, flushing the whole TLB is
Signature · 1 line
2.51.2
PATCH 8/22 powerpc: mm: use mm_is_kernel() for kernel mm checks
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
--- arch/powerpc/include/asm/pgalloc.h | 2 +- arch/powerpc/mm/book3s64/pgtable.c | 2 +- arch/powerpc/mm/mmu_context.c | 2 +- arch/powerpc/mm/pgtable.c | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-)
arch/powerpc/include/asm/pgalloc.h +1 -1
@@ -7,7 +7,7 @@#ifndef MODULE static inline gfp_t pgtable_gfp_flags (struct mm_struct *mm,gfp_t gfp) { -if (unlikely (mm == &init_mm)) +if (unlikely (mm_is_kernel (mm)))return gfp;return gfp | __GFP_ACCOUNT; }
arch/powerpc/mm/book3s64/pgtable.c +1 -1
@@ -395,7 +395,7 @@ static pmd_t *__alloc_for_pmdcache(struct mm_struct *mm)struct ptdesc *ptdesc;gfp_t gfp =GFP_KERNEL_ACCOUNT | __GFP_ZERO; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp &= ~__GFP_ACCOUNT; ptdesc =pagetable_alloc (gfp,0 );if (!ptdesc)
arch/powerpc/mm/mmu_context.c +1 -1
@@ -48,7 +48,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,/* Mark this context has been used on the new CPU */ if (!cpumask_test_cpu (cpu,mm_cpumask (next))) { -VM_WARN_ON_ONCE (next == &init_mm); +VM_WARN_ON_ONCE (mm_is_kernel (next));cpumask_set_cpu (cpu,mm_cpumask (next));inc_mm_active_cpus (next);
arch/powerpc/mm/pgtable.c +1 -1
@@ -393,7 +393,7 @@ void assert_pte_locked(struct mm_struct *mm, unsigned long addr)pte_t *pte;spinlock_t *ptl; -if (mm == &init_mm) +if (mm_is_kernel (mm))return ; pgd = mm->pgd +pgd_index (addr);BUG_ON (pgd_none (*pgd));
Signature · 1 line
2.51.2
PATCH 9/22 s390: mm: use mm_is_kernel() for kernel mm checks
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
--- arch/s390/include/asm/mmu_context.h | 2 +- arch/s390/include/asm/pgtable.h | 6 +++--- arch/s390/mm/pgalloc.c | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-)
arch/s390/include/asm/mmu_context.h +1 -1
@@ -69,7 +69,7 @@ static inline void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct * {int cpu =smp_processor_id (); -if (next == &init_mm) +if (mm_is_kernel (next))get_lowcore ()->user_asce = s390_invalid_asce;else get_lowcore ()->user_asce.val = next->context.asce;
arch/s390/include/asm/pgtable.h +3 -3
@@ -1836,21 +1836,21 @@ static inline int has_transparent_hugepage(void)#ifdef CONFIG_PAGE_TABLE_CHECK static inline bool pte_user_accessible_page (struct mm_struct *mm,unsigned long addr,pte_t pte) { -VM_BUG_ON (mm == &init_mm); +VM_BUG_ON (mm_is_kernel (mm));return pte_present (pte); }static inline bool pmd_user_accessible_page (struct mm_struct *mm,unsigned long addr,pmd_t pmd) { -VM_BUG_ON (mm == &init_mm); +VM_BUG_ON (mm_is_kernel (mm));return pmd_leaf (pmd) && (pmd_val (pmd) & _SEGMENT_ENTRY_READ); }static inline bool pud_user_accessible_page (struct mm_struct *mm,unsigned long addr,pud_t pud) { -VM_BUG_ON (mm == &init_mm); +VM_BUG_ON (mm_is_kernel (mm));return pud_leaf (pud); }
arch/s390/mm/pgalloc.c +2 -2
@@ -20,7 +20,7 @@ unsigned long *crst_table_alloc_noprof(struct mm_struct *mm)struct ptdesc *ptdesc;unsigned long *table; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp &= ~__GFP_ACCOUNT; ptdesc =pagetable_alloc_noprof (gfp,CRST_ALLOC_ORDER );if (!ptdesc)
@@ -120,7 +120,7 @@ unsigned long *page_table_alloc_noprof(struct mm_struct *mm)struct ptdesc *ptdesc;unsigned long *table; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp &= ~__GFP_ACCOUNT; ptdesc =pagetable_alloc_noprof (gfp,0 );if (!ptdesc)
Signature · 1 line
2.51.2
PATCH 10/22 sparc: mm: use mm_is_kernel() for kernel mm checks
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
pgtable_64.h previously forward-declared init_mm to avoid pulling in mm_types.h. It seems that including mm_types.h is not causing troubles any more, so we can also use mm_is_kernel() there and drop the forward declaration.
--- arch/sparc/include/asm/mmu_context_64.h | 2 +- arch/sparc/include/asm/pgtable_64.h | 6 ++---- arch/sparc/mm/init_64.c | 2 +- arch/sparc/mm/tlb.c | 2 +- 4 files changed, 5 insertions(+), 7 deletions(-)
arch/sparc/include/asm/mmu_context_64.h +1 -1
@@ -83,7 +83,7 @@ static inline void switch_mm(struct mm_struct *old_mm, struct mm_struct *mm, strint cpu =smp_processor_id ();per_cpu (per_cpu_secondary_mm, cpu) = mm; -if (unlikely (mm == &init_mm)) +if (unlikely (mm_is_kernel (mm)))return ;spin_lock_irqsave (&mm->context.lock, flags);
arch/sparc/include/asm/pgtable_64.h +2 -4
@@ -86,6 +86,7 @@ extern unsigned long VMALLOC_END;#define vmemmap ((struct page *)VMEMMAP_BASE)#include <linux/sched.h> +#include <linux/mm_types.h> #include <asm/tlbflush.h> bool kern_addr_valid (unsigned long addr);
@@ -885,9 +886,6 @@ static inline unsigned long pud_pfn(pud_t pud)#define p4d_set (p4dp, pudp) \ (p4d_val(*(p4dp)) = (__pa((unsigned long) (pudp)))) -/* We cannot include <linux/mm_types.h> at this point yet: */ -extern struct mm_struct init_mm; -/* Actual page table PTE updates. */ void tlb_batch_add(struct mm_struct *mm, unsigned long vaddr, pte_t *ptep, pte_t orig, int fullmm,
@@ -903,7 +901,7 @@ static void maybe_tlb_batch_add(struct mm_struct *mm, unsigned long vaddr,* SUN4V NOTE: _PAGE_VALID is the same value in both the SUN4U * and SUN4V pte layout, so this inline test is fine. */ -if (likely (mm != &init_mm) &&pte_accessible (mm, orig)) +if (likely (!mm_is_kernel (mm)) &&pte_accessible (mm, orig))tlb_batch_add (mm, vaddr, ptep, orig, fullmm, hugepage_shift); }
arch/sparc/mm/init_64.c +1 -1
@@ -788,7 +788,7 @@ static void mmu_context_wrap(void)*/ mm =per_cpu (per_cpu_secondary_mm, cpu); -if (unlikely (!mm || mm == &init_mm)) +if (unlikely (!mm ||mm_is_kernel (mm)))continue ; old_ctx = mm->context.sparc64_ctx_val;
arch/sparc/mm/tlb.c +1 -1
@@ -179,7 +179,7 @@ static void tlb_batch_pmd_scan(struct mm_struct *mm, unsigned long vaddr,static void __set_pmd_acct (struct mm_struct *mm,unsigned long addr,pmd_t orig,pmd_t pmd) { -if (mm == &init_mm) +if (mm_is_kernel (mm))return ;if ((pmd_val (pmd) ^pmd_val (orig)) & _PAGE_PMD_HUGE) {
Signature · 1 line
2.51.2
PATCH 11/22 um: mm: use mm_is_kernel() in TLB sync
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
--- arch/um/kernel/tlb.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/um/kernel/tlb.c +1 -1
@@ -171,7 +171,7 @@ int um_tlb_sync(struct mm_struct *mm)return 0 ; ops.mm_idp = &mm->context.id; -if (mm == &init_mm) { +if (mm_is_kernel (mm)) { ops.mmap = kern_map; ops.unmap = kern_unmap; }else {
Signature · 1 line
2.51.2
PATCH 12/22 x86/mm: use mm_is_kernel() for kernel mm checks
In reply to Kevin BrodskyThe new MMF_KERNEL flag identifies kernel-owned mm's. Checking the flag with mm_is_kernel() is preferred over comparing directly against &init_mm.
No functional change, as only init_mm has MMF_KERNEL set for now.
--- arch/x86/include/asm/tlbflush.h | 8 ++++---- arch/x86/mm/pgtable.c | 2 +- arch/x86/mm/tlb.c | 9 +++++---- 3 files changed, 10 insertions(+), 9 deletions(-)
arch/x86/include/asm/tlbflush.h +4 -4
@@ -159,10 +159,10 @@ struct tlb_state_shared {* - Actively using an mm. Our CPU's bit will be set in * mm_cpumask(loaded_mm) and is_lazy == false; * -* - Not using a real mm. loaded_mm == &init_mm. Our CPU's bit -* will not be set in mm_cpumask(&init_mm) and is_lazy == false. +* - Not using a real mm. loaded_mm is a kernel mm. Our CPU's bit +* will not be set in mm_cpumask(loaded_mm) and is_lazy == false. * -* - Lazily using a real mm. loaded_mm != &init_mm, our bit +* - Lazily using a real mm. loaded_mm is not a kernel mm, our bit * is set in mm_cpumask(loaded_mm), but is_lazy == true. * We're heuristically guessing that the CR3 load we * skipped more than makes up for the overhead added by
@@ -188,7 +188,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state_shared, cpu_tlbstate_shared);#define enter_lazy_tlb enter_lazy_tlb static __always_inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { -if (this_cpu_read (cpu_tlbstate.loaded_mm) == &init_mm) +if (mm_is_kernel (this_cpu_read (cpu_tlbstate.loaded_mm)))return ;this_cpu_write (cpu_tlbstate_shared.is_lazy, true);
arch/x86/mm/pgtable.c +1 -1
@@ -168,7 +168,7 @@ static int preallocate_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)bool failed = false;gfp_t gfp =GFP_PGTABLE_USER ; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp &= ~__GFP_ACCOUNT; gfp &= ~__GFP_HIGHMEM;
arch/x86/mm/tlb.c +5 -4
@@ -594,7 +594,7 @@ void leave_mm(void)* This needs to happen before any other sanity checks due to * intel_idle's shenanigans. */ -if (loaded_mm == &init_mm) +if (mm_is_kernel (loaded_mm))return ;/* Warn if we're not lazy. */
@@ -855,7 +855,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,* cpu_tlbstate_shared.is_lazy whether or not to send an IPI. */ if (IS_ENABLED (CONFIG_DEBUG_VM ) && -WARN_ON_ONCE (prev != &init_mm && !is_notrack_mm (prev) && +WARN_ON_ONCE (!mm_is_kernel (prev) && !is_notrack_mm (prev) && !cpumask_test_cpu (cpu,mm_cpumask (next))))cpumask_set_cpu (cpu,mm_cpumask (next));
@@ -932,7 +932,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,* This way switch_mm() must see the new tlb_gen or * flush_tlb_mm_range() must see the new loaded_mm, or both. */ -if (next != &init_mm && !cpumask_test_cpu (cpu,mm_cpumask (next))) +if (!mm_is_kernel (next) && + !cpumask_test_cpu (cpu,mm_cpumask (next)))cpumask_set_cpu (cpu,mm_cpumask (next));else smp_mb ();
@@ -1134,7 +1135,7 @@ static void flush_tlb_func(void *info)return ; } -if (unlikely (loaded_mm == &init_mm)) +if (unlikely (mm_is_kernel (loaded_mm)))return ;/* Reload the ASID if transitioning into or out of a global ASID */
Signature · 1 line
2.51.2
Re: [PATCH 12/22] x86/mm: use mm_is_kernel() for kernel mm checks
On 7/14/26 07:04, Kevin Brodsky wrote:
...
arch/x86/include/asm/tlbflush.h DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state_shared, cpu_tlbstate_shared); · line 194
Quoted 9 lines
arch/x86/include/asm/tlbflush.h +1
-1@@ -188,7 +188,7 @@ DECLARE_PER_CPU_SHARED_ALIGNED(struct tlb_state_shared, cpu_tlbstate_shared);#define enter_lazy_tlb enter_lazy_tlb static __always_inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { -if (this_cpu_read (cpu_tlbstate.loaded_mm) == &init_mm) +if (mm_is_kernel (this_cpu_read (cpu_tlbstate.loaded_mm)))return ;this_cpu_write (cpu_tlbstate_shared.is_lazy, true);
This one makes me nervous, especially combined with the efi_mm changes later in the series. init_mm really is special. Before this patch, running on init_mm qualifies as being in lazy tlb mode. After this patch, efi_mm qualifies too.
Is that OK?
arch/x86/mm/pgtable.c static int preallocate_pmds(struct mm_struct *mm, pmd_t *pmds[], int count) · line 173
Quoted 12 lines
arch/x86/mm/pgtable.c +1
-1@@ -168,7 +168,7 @@ static int preallocate_pmds(struct mm_struct *mm, pmd_t *pmds[], int count)bool failed = false;gfp_t gfp =GFP_PGTABLE_USER ; -if (mm == &init_mm) +if (mm_is_kernel (mm)) gfp &= ~__GFP_ACCOUNT; gfp &= ~__GFP_HIGHMEM;
This hunk is probably fine.
arch/x86/mm/tlb.c void leave_mm(void) · line 598
Quoted 11 lines
arch/x86/mm/tlb.c +1
-1@@ -594,7 +594,7 @@ void leave_mm(void)* This needs to happen before any other sanity checks due to * intel_idle's shenanigans. */ -if (loaded_mm == &init_mm) +if (mm_is_kernel (loaded_mm))return ;
This one is almost *certainly* buggy.
The efi_mm stuff uses leave_mm(). In fact, I think it's what's responsible for switching the hardware back over to use the init_mm page tables. If this check trips for init_mm, then the kernel will stay running on efi_mm when it's not intended to.
Remember when I asked about whether there was going to be *a* kernel mm or lots? This is where it starts to matter. It's outside the diff context but the code here is basically:
if (loaded_mm == &init_mm)
return;
switch_mm(NULL, &init_mm, NULL);So either all kernel mm's are equivalent or this code doesn't quite work.
Quoted 11 lines
arch/x86/mm/tlb.c +1
-1/* Warn if we're not lazy. */ @@ -855,7 +855,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,* cpu_tlbstate_shared.is_lazy whether or not to send an IPI. */ if (IS_ENABLED (CONFIG_DEBUG_VM ) && -WARN_ON_ONCE (prev != &init_mm && !is_notrack_mm (prev) && +WARN_ON_ONCE (!mm_is_kernel (prev) && !is_notrack_mm (prev) && !cpumask_test_cpu (cpu,mm_cpumask (next))))cpumask_set_cpu (cpu,mm_cpumask (next));
arch/x86/mm/tlb.c void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, · line 939
Quoted 9 lines
arch/x86/mm/tlb.c +2
-1@@ -932,7 +932,8 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next,* This way switch_mm() must see the new tlb_gen or * flush_tlb_mm_range() must see the new loaded_mm, or both. */ -if (next != &init_mm && !cpumask_test_cpu (cpu,mm_cpumask (next))) +if (!mm_is_kernel (next) && + !cpumask_test_cpu (cpu,mm_cpumask (next)))cpumask_set_cpu (cpu,mm_cpumask (next));else smp_mb ();
This one is interesting. It's basically saying that init_mm doesn't get mm_cpumask() updates. That makes sense since it has special TLB flushing rules. But what does this mean for efi_mm? Does it needs mm_cpumask() updates?
PATCH 13/22 mm: account page table pages when allocated
In reply to Kevin BrodskyAs a step towards removing pagetable ctor/dtor calls for kernel mm's, move the memory accounting from the ctor/dtor to pagetable_{alloc,free}.
The ctor is normally called after pagetable_alloc() (and dtor before pagetable_free()) so this patch should have no functional effect in most cases. Where calls to ctor/dtor are currently missing (e.g. x86's split_large_page()), this patch should improve the accounting.
This patch is based on Matthew's earlier implementation linked below, with the difference that when freeing the accounting is updated in __pagetable_free(), meaning that the update is delayed until the pages are actually freed in the CONFIG_ASYNC_KERNEL_PGTABLE_FREE case. This shouldn't matter in practice.
--- include/linux/mm.h | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)
include/linux/mm.h +7 -2
@@ -3681,6 +3681,11 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde {struct page *page =alloc_pages_noprof (gfp | __GFP_COMP, order); +if (!page) +return NULL ; + +lruvec_stat_add_folio (page_folio (page),NR_PAGETABLE ); +return page_ptdesc (page); }#define pagetable_alloc (...) alloc_hooks(pagetable_alloc_noprof(__VA_ARGS__))
@@ -3689,6 +3694,8 @@ static inline void __pagetable_free(struct ptdesc *pt) {struct page *page =ptdesc_page (pt); +lruvec_stat_sub_folio (page_folio (page),NR_PAGETABLE ); +__free_pages (page,compound_order (page)); }
@@ -3797,7 +3804,6 @@ static inline void __pagetable_ctor(struct ptdesc *ptdesc)struct folio *folio =ptdesc_folio (ptdesc);__folio_set_pgtable (folio); -lruvec_stat_add_folio (folio,NR_PAGETABLE ); }static inline void pagetable_dtor (struct ptdesc *ptdesc)
@@ -3806,7 +3812,6 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc)ptlock_free (ptdesc);__folio_clear_pgtable (folio); -lruvec_stat_sub_folio (folio,NR_PAGETABLE ); }static inline void pagetable_dtor_free (struct ptdesc *ptdesc)
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH 13/22] mm: account page table pages when allocated
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [Medium] NR_PAGETABLE statistic leak on architectures that do not define CONFIG_MMU_GATHER_TABLE_FREE.…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [Medium] NR_PAGETABLE statistic leak on architectures that do not define CONFIG_MMU_GATHER_TABLE_FREE.
- [Medium] Direct calls to pagetable_dtor() followed by generic page freeing functions bypass the NR_PAGETABLE decrement, causing stat leaks on ARM64 memory hot-remove.
- [Medium] SPARC32 SRMMU page tables completely lose NR_PAGETABLE accounting.
- [Medium] Memory hot-remove and vmemmap teardown on x86 will underflow the NR_PAGETABLE counter.
--
--- Patch [13]: [PATCH 13/22] mm: account page table pages when allocated --- commit 5c303720c0cd77f50e7b180d025a9a9869aab6a2
Author: Kevin Brodsky <[email protected]>
mm: account page table pages when allocated
This commit moves memory accounting of pagetable pages from the ctor/dtor functions to pagetable_alloc() and pagetable_free() in order to improve accounting where ctor/dtor calls might be missing.
Quoted 5 lines
include/linux/mm.h +0
-0
include/linux/mm.h static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde · line 3690
Quoted 10 lines
include/linux/mm.h +5
-0@@ -3681,6 +3681,11 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int orde {struct page *page =alloc_pages_noprof (gfp | __GFP_COMP, order); +if (!page) +return NULL ; + +lruvec_stat_add_folio (page_folio (page),NR_PAGETABLE ); +return page_ptdesc (page); }
[Severity: Medium]
For architectures or code paths that allocate page tables without using pagetable_alloc_noprof(), will we now lose the NR_PAGETABLE accounting?
For example, on SPARC32, pte_alloc_one() in arch/sparc/mm/srmmu.c allocates from a pre-allocated pool via srmmu_get_nocache() and previously relied on pagetable_pte_ctor() to increment the stat:
arch/sparc/mm/srmmu.c:pte_alloc_one() {
...
if (page_ref_inc_return(page) == 2 &&
!pagetable_pte_ctor(mm, page_ptdesc(page))) {
...
}Similarly, the freeing path mirrors this bypass, using srmmu_free_nocache() instead of pagetable_free(). Does this change cause SPARC32 to completely lose user PTE accounting?
include/linux/mm.h static inline void __pagetable_free(struct ptdesc *pt) · line 3700
Quoted 8 lines
include/linux/mm.h +2
-0@@ -3689,6 +3694,8 @@ static inline void __pagetable_free(struct ptdesc *pt) {struct page *page =ptdesc_page (pt); +lruvec_stat_sub_folio (page_folio (page),NR_PAGETABLE ); +__free_pages (page,compound_order (page)); }
[Severity: Medium]
Will this cause an underflow on x86 during memory hot-remove?
In arch/x86/mm/init_64.c, free_pagetable() calls pagetable_free() (which now decrements the stat) during hot-remove:
arch/x86/mm/init_64.c:free_pagetable() {
...
if (PageReserved(page))
free_reserved_page(page);
else
pagetable_free(page_ptdesc(page));
}However, these kernel page tables were originally allocated during hot-add with spp_getpage() or vmemmap_alloc_block_buf(). Both of these use base allocators and bypass pagetable_alloc_noprof().
Because the stat is never incremented on allocation but is now decremented on free, does this cause an underflow?
[ ... ]
Quoted 9 lines
include/linux/mm.h +0
-1@@ -3797,7 +3804,6 @@ static inline void __pagetable_ctor(struct ptdesc *ptdesc)struct folio *folio =ptdesc_folio (ptdesc);__folio_set_pgtable (folio); -lruvec_stat_add_folio (folio,NR_PAGETABLE ); }static inline void pagetable_dtor (struct ptdesc *ptdesc)
include/linux/mm.h static inline void pagetable_dtor(struct ptdesc *ptdesc) · line 3815
Quoted 5 lines
include/linux/mm.h +0
-1@@ -3806,7 +3812,6 @@ static inline void pagetable_dtor(struct ptdesc *ptdesc)ptlock_free (ptdesc);__folio_clear_pgtable (folio); -lruvec_stat_sub_folio (folio,NR_PAGETABLE ); }
[Severity: Medium]
Will this leak the NR_PAGETABLE counter on architectures that bypass __pagetable_free() and only call pagetable_dtor()?
On architectures lacking CONFIG_MMU_GATHER_TABLE_FREE (like m68k or xtensa), the fallback tlb_remove_table() implementation explicitly calls pagetable_dtor() and then enqueues the page for generic batch freeing:
include/asm-generic/tlb.h:tlb_remove_table() {
struct ptdesc *ptdesc = (struct ptdesc *)table;
pagetable_dtor(ptdesc);
tlb_remove_page(tlb, ptdesc_page(ptdesc));
}Since the decrement was moved from pagetable_dtor() to __pagetable_free(), which is bypassed here, will this leak the counter when userspace unmaps memory?
A similar bypass appears to occur on ARM64 during memory hot-remove:
arch/arm64/mm/mmu.c:free_hotplug_pgtable_page() {
pagetable_dtor(page_ptdesc(page));
free_hotplug_page_range(page, PAGE_SIZE, NULL);
}Since this directly calls pagetable_dtor() and skips __pagetable_free(), will this also leave the stat un-decremented?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
PATCH 14/22 mm: set page table page type when allocated
In reply to Kevin BrodskyAs a step towards removing pagetable ctor/dtor calls for kernel mm's, move the page type updates from the ctor/dtor to pagetable_{alloc,free}.
__pagetable_ctor() is now empty so remove it; pagetable_{pud,p4d,pgd}_ctor() will be removed at a later stage.
The ctor is normally called after pagetable_alloc() (and dtor before pagetable_free()) so this patch should have no functional effect in most cases. Where calls to ctor/dtor are currently missing (e.g. x86's split_large_page()), this patch will ensure the page type is set correctly.
This patch is based on Matthew's earlier implementation linked below, with the difference that when freeing the page type is cleared in __pagetable_free(), meaning that the update is delayed until the pages are actually freed in the CONFIG_ASYNC_KERNEL_PGTABLE_FREE case. This shouldn't matter in practice.
--- include/linux/mm.h | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-)
include/linux/mm.h +2 -15
@@ -3684,6 +3684,7 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int ordeif (!page)return NULL ; +__folio_set_pgtable (page_folio (page));lruvec_stat_add_folio (page_folio (page),NR_PAGETABLE );return page_ptdesc (page);
@@ -3694,6 +3695,7 @@ static inline void __pagetable_free(struct ptdesc *pt) {struct page *page =ptdesc_page (pt); +__folio_clear_pgtable (page_folio (page));lruvec_stat_sub_folio (page_folio (page),NR_PAGETABLE );__free_pages (page,compound_order (page));
@@ -3799,19 +3801,9 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; }static inline void ptlock_free (struct ptdesc *ptdesc) {}#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void __pagetable_ctor (struct ptdesc *ptdesc) -{ -struct folio *folio =ptdesc_folio (ptdesc); - -__folio_set_pgtable (folio); -} -static inline void pagetable_dtor (struct ptdesc *ptdesc) { -struct folio *folio =ptdesc_folio (ptdesc); -ptlock_free (ptdesc); -__folio_clear_pgtable (folio); }static inline void pagetable_dtor_free (struct ptdesc *ptdesc)
@@ -3825,7 +3817,6 @@ static inline bool pagetable_pte_ctor(struct mm_struct *mm, {if (!mm_is_kernel (mm) && !ptlock_init (ptdesc))return false; -__pagetable_ctor (ptdesc);return true; }
@@ -3917,7 +3908,6 @@ static inline bool pagetable_pmd_ctor(struct mm_struct *mm,if (!mm_is_kernel (mm) && !pmd_ptlock_init (ptdesc))return false;ptdesc_pmd_pts_init (ptdesc); -__pagetable_ctor (ptdesc);return true; }
@@ -3942,17 +3932,14 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud)static inline void pagetable_pud_ctor (struct ptdesc *ptdesc) { -__pagetable_ctor (ptdesc); }static inline void pagetable_p4d_ctor (struct ptdesc *ptdesc) { -__pagetable_ctor (ptdesc); }static inline void pagetable_pgd_ctor (struct ptdesc *ptdesc) { -__pagetable_ctor (ptdesc); }extern void __init pagecache_init (void );
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH 14/22] mm: set page table page type when allocated
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider: - [Medium] Missing PG_table page type initialization for custom allocators that bypass…
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Medium] Missing PG_table page type initialization for custom allocators that bypass pagetable_alloc_noprof().
- [Medium] Bypassing pagetable_free() causes leaked PG_table state and permanent NR_PAGETABLE accounting leaks. --
--- Patch [14]: [PATCH 14/22] mm: set page table page type when allocated --- commit ec9e373da7e406797591704d23e05fdbf402dfe3
Author: Kevin Brodsky <[email protected]>
mm: set page table page type when allocated
This commit moves page table page type updates from the constructors and destructors directly into pagetable_alloc_noprof() and __pagetable_free(). It leaves __pagetable_ctor() empty and removes it.
Quoted 4 lines
include/linux/mm.h +0
-0
[ ... ]
Quoted 8 lines
include/linux/mm.h +0
-4@@ -3799,19 +3801,9 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; }static inline void ptlock_free (struct ptdesc *ptdesc) {}#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void __pagetable_ctor (struct ptdesc *ptdesc) -{ -struct folio *folio =ptdesc_folio (ptdesc); -
include/linux/mm.h static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; } · line 3807
Quoted 10 lines
include/linux/mm.h +0
-6-__folio_set_pgtable (folio); -} -static inline void pagetable_dtor (struct ptdesc *ptdesc) { -struct folio *folio =ptdesc_folio (ptdesc); -ptlock_free (ptdesc); -__folio_clear_pgtable (folio); }
[Severity: Medium]
Since __folio_clear_pgtable() was moved to __pagetable_free(), what happens to architectures that bypass __pagetable_free() entirely?
For example, without CONFIG_MMU_GATHER_TABLE_FREE, tlb_remove_table() calls pagetable_dtor() and frees the page directly via the batcher:
include/asm-generic/tlb.h:tlb_remove_table() {
...
pagetable_dtor(ptdesc);
tlb_remove_page(tlb, ptdesc_page(ptdesc));
}Similarly, arm64 memory hot-remove frees pages directly:
arch/arm64/mm/mmu.c:free_hotplug_pgtable_page() {
pagetable_dtor(page_ptdesc(page));
free_hotplug_page_range(page, PAGE_SIZE, NULL);
}Will this bypass cause a permanent leak in the NR_PAGETABLE counter and leave the PG_table flag set on the page as it enters the buddy allocator?
include/linux/mm.h static inline bool pagetable_pte_ctor(struct mm_struct *mm, · line 3821
Quoted 8 lines
include/linux/mm.h +0
-1static inline void pagetable_dtor_free (struct ptdesc *ptdesc)@@ -3825,7 +3817,6 @@ static inline bool pagetable_pte_ctor(struct mm_struct *mm, {if (!mm_is_kernel (mm) && !ptlock_init (ptdesc))return false; -__pagetable_ctor (ptdesc);return true; }
[Severity: Medium]
Are there custom allocators that rely on pagetable_pte_ctor() to initialize the page type?
For instance, on SPARC32, user PTE tables are allocated via a custom pool using srmmu_get_nocache() which bypasses pagetable_alloc_noprof(). They rely on pagetable_pte_ctor() to set the page type:
arch/sparc/mm/srmmu.c:pte_alloc_one() {
...
if (page_ref_inc_return(page) == 2 &&
!pagetable_pte_ctor(mm, page_ptdesc(page))) {
...
}By removing __pagetable_ctor() here, it seems these pages will no longer have the PG_table flag set. Is this the intended behavior?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
Re: [PATCH 14/22] mm: set page table page type when allocated
In reply to Kevin BrodskyQuoted 3 lines · Kevin Brodsky
As a step towards removing pagetable ctor/dtor calls for kernel mm's, move the page type updates from the ctor/dtor to pagetable_{alloc,free}.
Thanks for doing this patchset.
We should switch over from __folio_set_pgtable() to __SetPagePgtable() in this patch. Ptdescs and folios should be treated as distinctly independent of each other.
Eventually, we want to replace the memcg accounting as well, but there's no existing API for that right now. Lets leave that as is for now; this patchset is big enough already ;)
Quoted 101 lines
__pagetable_ctor()is now empty so remove it; pagetable_{pud,p4d,pgd}_ctor() will be removed at a later stage.The ctor is normally called after
pagetable_alloc()(and dtor beforepagetable_free()) so this patch should have no functional effect in most cases. Where calls to ctor/dtor are currently missing (e.g. x86'ssplit_large_page()), this patch will ensure the page type is set correctly.This patch is based on Matthew's earlier implementation linked below, with the difference that when freeing the page type is cleared in
__pagetable_free(), meaning that the update is delayed until the pages are actually freed in theCONFIG_ASYNC_KERNEL_PGTABLE_FREEcase. This shouldn't matter in practice.Suggested-by "Matthew Wilcox (Oracle)" <[email protected]>Signed-off-by Kevin Brodsky <[email protected]>--- include/linux/mm.h | 17 ++--------------- 1 file changed, 2 insertions(+), 15 deletions(-)include/linux/mm.h +2
-15@@ -3684,6 +3684,7 @@ static inline struct ptdesc *pagetable_alloc_noprof(gfp_t gfp, unsigned int ordeif (!page)return NULL ; +__folio_set_pgtable (page_folio (page));lruvec_stat_add_folio (page_folio (page),NR_PAGETABLE );return page_ptdesc (page);@@ -3694,6 +3695,7 @@ static inline void __pagetable_free(struct ptdesc *pt) {struct page *page =ptdesc_page (pt); +__folio_clear_pgtable (page_folio (page));lruvec_stat_sub_folio (page_folio (page),NR_PAGETABLE );__free_pages (page,compound_order (page));@@ -3799,19 +3801,9 @@ static inline bool ptlock_init(struct ptdesc *ptdesc) { return true; }static inline void ptlock_free (struct ptdesc *ptdesc) {}#endif /* defined(CONFIG_SPLIT_PTE_PTLOCKS) */ -static inline void __pagetable_ctor (struct ptdesc *ptdesc) -{ -struct folio *folio =ptdesc_folio (ptdesc); - -__folio_set_pgtable (folio); -} -static inline void pagetable_dtor (struct ptdesc *ptdesc) { -struct folio *folio =ptdesc_folio (ptdesc); -ptlock_free (ptdesc); -__folio_clear_pgtable (folio); }static inline void pagetable_dtor_free (struct ptdesc *ptdesc)@@ -3825,7 +3817,6 @@ static inline bool pagetable_pte_ctor(struct mm_struct *mm, {if (!mm_is_kernel (mm) && !ptlock_init (ptdesc))return false; -__pagetable_ctor (ptdesc);return true; }@@ -3917,7 +3908,6 @@ static inline bool pagetable_pmd_ctor(struct mm_struct *mm,if (!mm_is_kernel (mm) && !pmd_ptlock_init (ptdesc))return false;ptdesc_pmd_pts_init (ptdesc); -__pagetable_ctor (ptdesc);return true; }@@ -3942,17 +3932,14 @@ static inline spinlock_t *pud_lock(struct mm_struct *mm, pud_t *pud)static inline void pagetable_pud_ctor (struct ptdesc *ptdesc) { -__pagetable_ctor (ptdesc); }static inline void pagetable_p4d_ctor (struct ptdesc *ptdesc) { -__pagetable_ctor (ptdesc); }static inline void pagetable_pgd_ctor (struct ptdesc *ptdesc) { -__pagetable_ctor (ptdesc); }extern void __init pagecache_init (void );Signature · 2 lines
-- 2.51.2
Re: [PATCH 14/22] mm: set page table page type when allocated
Quoted 5 lines · Vishal Moola
Quoted 3 lines · Kevin Brodsky
As a step towards removing pagetable ctor/dtor calls for kernel mm's, move the page type updates from the ctor/dtor to pagetable_{alloc,free}.
Thanks for doing this patchset.
Thanks for suggesting it :) It's turning out to be quite a bit more involved than I expected though! (There are multiple cases where we call the ctor/dtor without using pagetable_{alloc,free}, see my reply to the cover letter.)
Quoted 3 lines
We should switch over from
__folio_set_pgtable()to__SetPagePgtable()in this patch. Ptdescs and folios should be treated as distinctly independent of each other.
I sure remember the previous discussion, but my reading of your comments
[1] is that we cannot really change this as long as pagetable_alloc() can be called with order > 0, right? Multiple archs still rely on that.
Quoted 3 lines
Eventually, we want to replace the memcg accounting as well, but there's no existing API for that right now. Lets leave that as is for now; this patchset is big enough already ;)
Yep, looks like there was an attempt to improve this but it had to be reverted [2].
- Kevin
[1] https://lore.kernel.org/all/aha321ndGSdReBJR@fedora/
[2]
https://lore.kernel.org/all/[email protected]/
Re: [PATCH 14/22] mm: set page table page type when allocated
Quoted 12 lines · Kevin Brodsky
Quoted 5 lines · Vishal Moola
Quoted 3 lines · Kevin Brodsky
As a step towards removing pagetable ctor/dtor calls for kernel mm's, move the page type updates from the ctor/dtor to pagetable_{alloc,free}.
Thanks for doing this patchset.
Thanks for suggesting it :) It's turning out to be quite a bit more involved than I expected though! (There are multiple cases where we call the ctor/dtor without using pagetable_{alloc,free}, see my reply to the cover letter.)
Quoted 7 lines
Quoted 3 lines
We should switch over from
__folio_set_pgtable()to__SetPagePgtable()in this patch. Ptdescs and folios should be treated as distinctly independent of each other.I sure remember the previous discussion, but my reading of your comments
[1] is that we cannot really change this as long aspagetable_alloc()can be called with order > 0, right? Multiple archs still rely on that.
Kinda...
The concern was about callers accessing tail pages directly for that information. Using the folio API ensures that we always look into the head page.
Since we're now doing it at allocation time, we know we always have the head page, so it's fine.
The dtor/free should be more questionable because the underlying api uses virt_to_page() which could look at a tail page... oopsie. I haven't seen any reports tied to it so its probably fine (the setting/clearing has been using the folio API since ptdescs).
We can say thats a happy accident that makes distinguishing ptdescs and folios easier :)
Quoted 13 lines
Quoted 3 lines
Eventually, we want to replace the memcg accounting as well, but there's no existing API for that right now. Lets leave that as is for now; this patchset is big enough already ;)
Yep, looks like there was an attempt to improve this but it had to be reverted [2].
- Kevin
[1] https://lore.kernel.org/all/aha321ndGSdReBJR@fedora/
[2]
https://lore.kernel.org/all/[email protected]/
Re: [PATCH 14/22] mm: set page table page type when allocated
Quoted 7 lines · Vishal Moola
Quoted 6 lines · Kevin Brodsky
Quoted 3 lines
We should switch over from
__folio_set_pgtable()to__SetPagePgtable()in this patch. Ptdescs and folios should be treated as distinctly independent of each other.I sure remember the previous discussion, but my reading of your comments
[1] is that we cannot really change this as long aspagetable_alloc()can be called with order > 0, right? Multiple archs still rely on that.
Quoted 8 lines
Kinda...
The concern was about callers accessing tail pages directly for that information. Using the folio API ensures that we always look into the head page.
Since we're now doing it at allocation time, we know we always have the head page, so it's fine.
Ah got it, that makes sense.
Quoted 5 lines
The dtor/free should be more questionable because the underlying api uses
virt_to_page()which could look at a tail page... oopsie. I haven't seen any reports tied to it so its probably fine (the setting/clearing has been using the folio API since ptdescs).
Right, that should be fine - calling pagetable_free() on anything else than a ptdesc allocated by pagetable_alloc() is a bug, so if we get any crash because of this we should just go and fix the underlying issue. I'd still switch to __SetPagePgtable() in a separate commit to improve bisectability.
FWIW there are definitely cases where pagetable_free() and pagetable_alloc() calls aren't properly matched (see my reply to the cover letter), this needs to be addressed for this series anyway.
Quoted 2 lines
We can say thats a happy accident that makes distinguishing ptdescs and folios easier :)
Indeed!
- Kevin
PATCH 15/22 mm: only initialise pt_share_count for user pgtables
In reply to Kevin Brodskyptdesc_pmd_pts_init() initialises the pt_share_count ptdesc field. That field is only used for hugetlb tracking purposes, so there is no need to initialise it for kernel page tables. Skip the call for kernel page tables, like pmd_ptlock_init().
From now on pagetable_*_ctor() and pagetable_dtor() do nothing for kernel page tables.
--- include/linux/mm.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
include/linux/mm.h +6 -3
@@ -3905,9 +3905,12 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd)static inline bool pagetable_pmd_ctor (struct mm_struct *mm,struct ptdesc *ptdesc) { -if (!mm_is_kernel (mm) && !pmd_ptlock_init (ptdesc)) -return false; -ptdesc_pmd_pts_init (ptdesc); +if (!mm_is_kernel (mm)) { +if (!pmd_ptlock_init (ptdesc)) +return false; +ptdesc_pmd_pts_init (ptdesc); + } +return true; }
Signature · 1 line
2.51.2
Re: [PATCH 15/22] mm: only initialise pt_share_count for user pgtables
Quoted 20 lines · Kevin Brodsky
ptdesc_pmd_pts_init()initialises thept_share_countptdesc field. That field is only used for hugetlb tracking purposes, so there is no need to initialise it for kernel page tables. Skip the call for kernel page tables, likepmd_ptlock_init().From now on pagetable_*_ctor() and
pagetable_dtor()do nothing for kernel page tables.Signed-off-by Kevin Brodsky <[email protected]>--- include/linux/mm.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)include/linux/mm.h +0
-0@@ -3905,9 +3905,12 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd)static inline bool pagetable_pmd_ctor (struct mm_struct *mm,struct ptdesc *ptdesc)
include/linux/mm.h static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd) · line 3913
Quoted 10 lines
include/linux/mm.h +6
-3{ -if (!mm_is_kernel (mm) && !pmd_ptlock_init (ptdesc)) -return false; -ptdesc_pmd_pts_init (ptdesc); +if (!mm_is_kernel (mm)) { +if (!pmd_ptlock_init (ptdesc)) +return false; +ptdesc_pmd_pts_init (ptdesc); + } +
I'd have said "the initialization is cheap, why bother", but we already do have the mm_is_kernel() check in place, so
Signature · 3 lines
Cheers, David
Re: [PATCH 15/22] mm: only initialise pt_share_count for user pgtables
Quoted 25 lines · David Hildenbrand (Arm)
Quoted 24 lines · Kevin Brodsky
ptdesc_pmd_pts_init()initialises thept_share_countptdesc field. That field is only used for hugetlb tracking purposes, so there is no need to initialise it for kernel page tables. Skip the call for kernel page tables, likepmd_ptlock_init().From now on pagetable_*_ctor() and
pagetable_dtor()do nothing for kernel page tables.Signed-off-by Kevin Brodsky <[email protected]>--- include/linux/mm.h | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)include/linux/mm.h +0
-3@@ -3905,9 +3905,12 @@ static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd)static inline bool pagetable_pmd_ctor (struct mm_struct *mm,struct ptdesc *ptdesc) { -if (!mm_is_kernel (mm) && !pmd_ptlock_init (ptdesc)) -return false; -ptdesc_pmd_pts_init (ptdesc);
Quoted 8 lines
include/linux/mm.h static inline spinlock_t *pmd_lock(struct mm_struct *mm, pmd_t *pmd) · line 3913
Quoted 6 lines
include/linux/mm.h +6
-0+if (!mm_is_kernel (mm)) { +if (!pmd_ptlock_init (ptdesc)) +return false; +ptdesc_pmd_pts_init (ptdesc); + } +I'd have said "the initialization is cheap, why bother", but we already do have the
mm_is_kernel()check in place, so
This isn't mainly about the overhead, but rather the fact that from here on the ctor/dtor do nothing for kernel page tables. Patch 20-22 rely on this, and in part 2 I would go further and only ever call the ctor/dtor for user page tables, so that the mm_is_kernel() check can be dropped.
Quoted 1 line
Thanks for the reviews!
- Kevin
PATCH 16/22 efi: mark efi_mm as a kernel mm
In reply to Kevin Brodskyefi_mm is a kernel-owned address space used for EFI runtime mappings, but it is currently treated like a user address space when manipulating its page tables. There does not seem to be a good reason for this, and it creates unnecessary complications when creating its page tables.
Mark it with the new MMF_KERNEL flag so that the page table handling treats it like init_mm. In particular, this means that:
* Split page table locks are no longer initialised or used for efi_mm's page tables.
* pagetable_*_ctor() and pagetable_dtor() no longer need to be called for efi_mm's page tables.
--- drivers/firmware/efi/efi.c | 1 + 1 file changed, 1 insertion(+)
drivers/firmware/efi/efi.c +1 -0
@@ -69,6 +69,7 @@ struct mm_struct efi_mm = { .mm_mt = MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, efi_mm.mmap_lock), .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .flags = MM_FLAGS_INIT(MMF_KERNEL_MASK), .write_protect_seq =SEQCNT_ZERO (efi_mm.write_protect_seq),MMAP_LOCK_INITIALIZER (efi_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock),
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH 16/22] efi: mark efi_mm as a kernel mm
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Medium] Incomplete implementation: split page table locks are still initialized for efi_mm on ARM64 and…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Incomplete implementation: split page table locks are still initialized for efi_mm on ARM64 and RISC-V. --
--- Patch [16]: [PATCH 16/22] efi: mark efi_mm as a kernel mm --- commit ebd104815a1ccc8b9a1160dde0da2bdd7e88ce96
Author: Kevin Brodsky <[email protected]>
efi: mark efi_mm as a kernel mm
This commit marks efi_mm with the MMF_KERNEL flag, treating it as a kernel-owned address space. This avoids unnecessary complications by skipping split page table lock initialization and constructor/destructor calls.
drivers/firmware/efi/efi.c struct mm_struct efi_mm = { · line 72
Quoted 9 lines
drivers/firmware/efi/efi.c +1
-0@@ -69,6 +69,7 @@ struct mm_struct efi_mm = { .mm_mt = MTREE_INIT_EXT(mm_mt, MM_MT_FLAGS, efi_mm.mmap_lock), .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .flags = MM_FLAGS_INIT(MMF_KERNEL_MASK),
[Severity: Medium]
The commit message notes that split page table locks are no longer initialized for efi_mm's page tables. Is this accurate for all architectures?
On ARM64 and RISC-V, the EFI page table allocation paths pass NULL instead of the mm_struct to the generic page table constructors:
arch/arm64/mm/mmu.c:pgd_pgtable_alloc_special_mm() {
...
return __pgd_pgtable_alloc(NULL, GFP_PGTABLE_KERNEL, pgtable_level);
}arch/riscv/mm/init.c:alloc_pte_late() {
...
BUG_ON(!ptdesc || !pagetable_pte_ctor(NULL, ptdesc));
...
}Because NULL is passed instead of &efi_mm, the constructors cannot check the MMF_KERNEL flag, meaning the split page table locks will still be unconditionally initialized.
Does this optimization require updating the architecture-specific EFI allocation paths to pass the mm_struct instead of NULL?
Quoted 3 lines
drivers/firmware/efi/efi.c +0
-0.write_protect_seq = SEQCNT_ZERO(efi_mm.write_protect_seq),MMAP_LOCK_INITIALIZER (efi_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(efi_mm.page_table_lock),
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=16
PATCH 17/22 mm: pagewalk: drop redundant address check for kernel mm walks
In reply to Kevin BrodskyCommit 8b1cb4a2e819 ("mm/pagewalk: fix EFI_PGT_DUMP of espfix area") made no-VMA page table walks use pte_offset_kernel() for any address above TASK_SIZE. This avoided pte_offset_map() validation on EFI page table dumps, where efi_mm could walk x86 espfix PMDs that intentionally look bad to pmd_bad().
efi_mm is now marked with MMF_KERNEL, and no-VMA walks use mm_is_kernel() to select pte_offset_kernel() for kernel mm's. As a result we no longer need to handle the EFI_PGT_DUMP case explicitly.The address check is therefore redundant for EFI_PGT_DUMP and makes the condition harder to reason about.
Remove the address check and associated comment; mm_is_kernel() should be sufficient on its own.
Tested by booting x86 QEMU with CONFIG_EFI_PGT_DUMP=y and reading /sys/kernel/debug/page_tables/efi.
--- mm/pagewalk.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
mm/pagewalk.c +2 -5
@@ -68,17 +68,14 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (walk->no_vma) {/* * pte_offset_map() might apply user-specific validation. -* Indeed, on x86_64 the pmd entries set up by init_espfix_ap() -* fit its pmd_bad() check (_PAGE_NX set and _PAGE_RW clear), -* and CONFIG_EFI_PGT_DUMP efi_mm goes so far as to walk them. */ -if (mm_is_kernel (walk->mm) || addr >=TASK_SIZE ) +if (mm_is_kernel (walk->mm)) pte =pte_offset_kernel (pmd, addr);else pte =pte_offset_map (pmd, addr);if (pte) { err =walk_pte_range_inner (pte, addr, end, walk); -if (!mm_is_kernel (walk->mm) && addr <TASK_SIZE ) +if (!mm_is_kernel (walk->mm))pte_unmap (pte); } }else {
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH 17/22] mm: pagewalk: drop redundant address check for kernel mm walks
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Removing the `addr >= TASK_SIZE` check in `walk_pte_range()` causes `ptdump` to clear valid kernel…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Removing the addr >= TASK_SIZE check in walk_pte_range() causes ptdump to clear valid kernel PMDs, leading to memory corruption and a kernel panic. --
--- Patch [17]: [PATCH 17/22] mm: pagewalk: drop redundant address check for kernel mm walks --- commit 85b512e55cb35ec57a18d00b67fbf62a68320627
Author: Kevin Brodsky <[email protected]>
mm: pagewalk: drop redundant address check for kernel mm walks
This commit removes the addr >= TASK_SIZE check from walk_pte_range() for no-VMA walks. It relies entirely on mm_is_kernel() to determine whether to use pte_offset_kernel() or pte_offset_map().
Quoted 7 lines
mm/pagewalk.c +0
-0@@ -68,17 +68,14 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (walk->no_vma) {/*
mm/pagewalk.c static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, · line 75
Quoted 10 lines
mm/pagewalk.c +1
-4* pte_offset_map() might apply user-specific validation. -* Indeed, on x86_64 the pmd entries set up by init_espfix_ap() -* fit its pmd_bad() check (_PAGE_NX set and _PAGE_RW clear), -* and CONFIG_EFI_PGT_DUMP efi_mm goes so far as to walk them. */ -if (mm_is_kernel (walk->mm) || addr >=TASK_SIZE ) +if (mm_is_kernel (walk->mm)) pte =pte_offset_kernel (pmd, addr);else pte =pte_offset_map (pmd, addr);
[Severity: High]
Does removing the addr >= TASK_SIZE check here cause ptdump to clear valid kernel PMDs?
When tools like ptdump walk the address space (e.g. via /sys/kernel/debug/page_tables/current_kernel), they may walk kernel addresses using current->mm, which is a user mm. In this case, mm_is_kernel() returns false, causing walk_pte_range() to use pte_offset_map() for kernel addresses.
On x86, pte_offset_map() eventually calls __pte_offset_map(), which validates the PMD using pmd_bad(). If a kernel PMD (such as the espfix mapping) fails this check because it intentionally lacks _PAGE_RW and sets _PAGE_NX, __pte_offset_map() unconditionally clears it via pmd_clear_bad().
Since kernel page tables are shared via init_mm, clearing the PMD permanently destroys the global espfix mapping. The next IRET to a 16-bit userspace segment would then encounter a missing page table, triggering a Double Fault kernel panic.
Should we keep the addr >= TASK_SIZE check to ensure kernel addresses always use pte_offset_kernel(), even when walked via a user mm?
Quoted 7 lines
mm/pagewalk.c +1
-1if (pte) { err =walk_pte_range_inner (pte, addr, end, walk); -if (!mm_is_kernel (walk->mm) && addr <TASK_SIZE ) +if (!mm_is_kernel (walk->mm))pte_unmap (pte); } }else {
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=17
Re: [PATCH 17/22] mm: pagewalk: drop redundant address check for kernel mm walks
In reply to Kevin BrodskyQuoted 5 lines · Kevin Brodsky
Commit
8b1cb4a2e819("mm/pagewalk: fixEFI_PGT_DUMPof espfix area") made no-VMA page table walks usepte_offset_kernel()for any address aboveTASK_SIZE. This avoidedpte_offset_map()validation on EFI page table dumps, whereefi_mmcould walk x86 espfix PMDs that intentionally look bad topmd_bad().
Sashiko rightly doesn't like this patch. Turns out 8b1cb4a2e819's commit message and the comment it added are narrower than what the problem really is: on x86, due to things like /sys/kernel/debug/page_tables/current_kernel, we may be walking kernel page tables even when walk->mm is a regular user mm. Just checking the mm is therefore not sufficient.
Will replace this patch with a rewording of the comment to avoid suggesting the issue is specific to efi_mm.
- Kevin
Quoted 43 lines
efi_mmis now marked withMMF_KERNEL, and no-VMA walks usemm_is_kernel()to selectpte_offset_kernel()for kernel mm's. As a result we no longer need to handle theEFI_PGT_DUMPcase explicitly.The address check is therefore redundant forEFI_PGT_DUMPand makes the condition harder to reason about.Remove the address check and associated comment;
mm_is_kernel()should be sufficient on its own.Tested by booting x86 QEMU with
CONFIG_EFI_PGT_DUMP=yand reading/sys/kernel/debug/page_tables/efi.Assisted-by Codex:GPT-5.5Signed-off-by Kevin Brodsky <[email protected]>--- mm/pagewalk.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)mm/pagewalk.c +2
-5@@ -68,17 +68,14 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (walk->no_vma) {/* * pte_offset_map() might apply user-specific validation. -* Indeed, on x86_64 the pmd entries set up by init_espfix_ap() -* fit its pmd_bad() check (_PAGE_NX set and _PAGE_RW clear), -* and CONFIG_EFI_PGT_DUMP efi_mm goes so far as to walk them. */ -if (mm_is_kernel (walk->mm) || addr >=TASK_SIZE ) +if (mm_is_kernel (walk->mm)) pte =pte_offset_kernel (pmd, addr);else pte =pte_offset_map (pmd, addr);if (pte) { err =walk_pte_range_inner (pte, addr, end, walk); -if (!mm_is_kernel (walk->mm) && addr <TASK_SIZE ) +if (!mm_is_kernel (walk->mm))pte_unmap (pte); } }else {
PATCH 18/22 arm64: mm: drop explicit mm_is_efi() check in contpte
In reply to Kevin Brodskyefi_mm is now marked as a kernel mm, like init_mm. mm_is_kernel() therefore gives the right answer for efi_mm and mm_is_user() is now equivalent to !mm_is_kernel().
Remove mm_is_user() and directly use mm_is_kernel() instead.
--- arch/arm64/mm/contpte.c | 26 +++++++++----------------- 1 file changed, 9 insertions(+), 17 deletions(-)
arch/arm64/mm/contpte.c +9 -17
@@ -4,23 +4,9 @@*/ #include <linux/mm.h> -#include <linux/efi.h> #include <linux/export.h> #include <asm/tlbflush.h> -static inline bool mm_is_user (struct mm_struct *mm) -{ -/* -* Don't attempt to apply the contig bit to kernel mappings, because -* dynamically adding/removing the contig bit can cause page faults. -* These racing faults are ok for user space, since they get serialized -* on the PTL. But kernel mappings can't tolerate faults. -*/ -if (unlikely (mm_is_efi (mm))) -return false; -return !mm_is_kernel (mm); -} -static inline pte_t *contpte_align_down (pte_t *ptep) {return PTR_ALIGN_DOWN (ptep,sizeof (*ptep) *CONT_PTES );
@@ -263,7 +249,13 @@ void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,int i; -if (!mm_is_user (mm)) +/* +* Don't attempt to apply the contig bit to kernel mappings, because +* dynamically adding/removing the contig bit can cause page faults. +* These racing faults are ok for user space, since they get serialized +* on the PTL. But kernel mappings can't tolerate faults. +*/ +if (mm_is_kernel (mm))return ; page =pte_page (pte);
@@ -302,7 +294,7 @@ void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr,* We have already checked that the ptes are contiguous in * contpte_try_unfold(), so just check that the mm is user space. */ -if (!mm_is_user (mm)) +if (mm_is_kernel (mm))return ; pte =pte_mknoncont (pte);
@@ -465,7 +457,7 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr,*/ VM_WARN_ON (nr ==1 ); -if (!mm_is_user (mm)) +if (mm_is_kernel (mm))return __set_ptes (mm, addr, ptep, pte, nr); end = addr + (nr <<PAGE_SHIFT );
Signature · 1 line
2.51.2
PATCH 19/22 x86/tboot: mark tboot_mm as a kernel mm
In reply to Kevin Brodskytboot_mm is a kernel-owned address space used to build the identity map for tboot shutdown. Mark it with MMF_KERNEL so page table allocation and initialisation code treats its page tables as kernel page tables.
Having marked tboot_mm as a kernel mm, we also need to convert pte_alloc_map() (user PTE) to pte_alloc_kernel() (kernel PTE), and drop the matching pte_unmap().
--- arch/x86/kernel/tboot.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/kernel/tboot.c +2 -2
@@ -99,6 +99,7 @@ static struct mm_struct tboot_mm = { .pgd = swapper_pg_dir, .mm_users = ATOMIC_INIT(2), .mm_count = ATOMIC_INIT(1), + .flags = MM_FLAGS_INIT(MMF_KERNEL_MASK), .write_protect_seq =SEQCNT_ZERO (tboot_mm.write_protect_seq),MMAP_LOCK_INITIALIZER (init_mm) .page_table_lock = __SPIN_LOCK_UNLOCKED(init_mm.page_table_lock),
@@ -129,11 +130,10 @@ static int map_tboot_page(unsigned long vaddr, unsigned long pfn, pmd =pmd_alloc (&tboot_mm, pud, vaddr);if (!pmd)return -1 ; - pte =pte_alloc_map (&tboot_mm, pmd, vaddr); + pte =pte_alloc_kernel (pmd, vaddr);if (!pte)return -1 ;set_pte_at (&tboot_mm, vaddr, pte,pfn_pte (pfn, prot)); -pte_unmap (pte);/* * PTI poisons low addresses in the kernel page tables in the
Signature · 1 line
2.51.2
Re: [PATCH 19/22] x86/tboot: mark tboot_mm as a kernel mm
Quoted 8 lines · Kevin Brodsky
tboot_mmis a kernel-owned address space used to build the identity map for tboot shutdown. Mark it withMMF_KERNELso page table allocation and initialisation code treats its page tables as kernel page tables.Having marked
tboot_mmas a kernel mm, we also need to convertpte_alloc_map()(user PTE) topte_alloc_kernel()(kernel PTE), and drop the matchingpte_unmap().
This makes me wonder whether MMF_KERNEL is the right way to mark these mms.
I think this series tries to cram too much into that one bit.
Using MMF_KERNEL to drive the page table accounting seems like a good idea. It also seems like a good thing to make sure that those mm's don't ever exit to userspace, for instance.
But I think it goes a step too far to say that all the special 'init_mm' TLB flushing should also be common among all MMF_KERNEL mms.
My suggestion for now would be to just go after the page table accounting alone. Don't mechanically replace mm==&mm_init checks.
PATCH 20/22 arm64: mm: drop ctor/dtor calls for kernel page tables
In reply to Kevin BrodskyThanks to recent changes:
* pagetable_*_ctor() and pagetable_dtor() do nothing for kernel page tables.
* efi_mm is also treated like a kernel mm by the page table logic, including ctors.
We can therefore drop the calls to ctors when allocating special kernel page tables in mmu.c, as well as the dtor call when freeing hotplug page tables.
Since we no longer need the owning mm or page table level when allocating them, we can also remove both the init_mm/special_mm wrappers and the pgtable_level callback argument.
--- arch/arm64/mm/mmu.c | 82 +++++++++++++++-------------------------------------- 1 file changed, 23 insertions(+), 59 deletions(-)
arch/arm64/mm/mmu.c +23 -59
@@ -103,7 +103,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, }EXPORT_SYMBOL (phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc (enum pgtable_level pgtable_level) +static phys_addr_t __init early_pgtable_alloc (void ) {phys_addr_t phys;
@@ -195,7 +195,7 @@ static bool pte_range_has_valid_noncont(pte_t *ptep)static int alloc_init_cont_pte (pmd_t *pmdp,unsigned long addr,unsigned long end,phys_addr_t phys,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {unsigned long next;
@@ -210,7 +210,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,if (flags &NO_EXEC_MAPPINGS ) pmdval |=PMD_TABLE_PXN ;BUG_ON (!pgtable_alloc); - pte_phys =pgtable_alloc (PGTABLE_LEVEL_PTE ); + pte_phys =pgtable_alloc ();if (pte_phys ==INVALID_PHYS_ADDR )return -ENOMEM ; ptep =pte_set_fixmap (pte_phys);
@@ -251,7 +251,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,static int init_pmd (pmd_t *pmdp,unsigned long addr,unsigned long end,phys_addr_t phys,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ),int flags) +phys_addr_t (*pgtable_alloc)(void ),int flags) {unsigned long next;
@@ -303,7 +303,7 @@ static bool pmd_range_has_valid_noncont(pmd_t *pmdp)static int alloc_init_cont_pmd (pud_t *pudp,unsigned long addr,unsigned long end,phys_addr_t phys,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {int ret;
@@ -322,7 +322,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,if (flags &NO_EXEC_MAPPINGS ) pudval |=PUD_TABLE_PXN ;BUG_ON (!pgtable_alloc); - pmd_phys =pgtable_alloc (PGTABLE_LEVEL_PMD ); + pmd_phys =pgtable_alloc ();if (pmd_phys ==INVALID_PHYS_ADDR )return -ENOMEM ; pmdp =pmd_set_fixmap (pmd_phys);
@@ -361,7 +361,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,static int alloc_init_pud (p4d_t *p4dp,unsigned long addr,unsigned long end,phys_addr_t phys,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {int ret =0 ;
@@ -376,7 +376,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,if (flags &NO_EXEC_MAPPINGS ) p4dval |=P4D_TABLE_PXN ;BUG_ON (!pgtable_alloc); - pud_phys =pgtable_alloc (PGTABLE_LEVEL_PUD ); + pud_phys =pgtable_alloc ();if (pud_phys ==INVALID_PHYS_ADDR )return -ENOMEM ; pudp =pud_set_fixmap (pud_phys);
@@ -428,7 +428,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,static int alloc_init_p4d (pgd_t *pgdp,unsigned long addr,unsigned long end,phys_addr_t phys,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {int ret;
@@ -443,7 +443,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,if (flags &NO_EXEC_MAPPINGS ) pgdval |=PGD_TABLE_PXN ;BUG_ON (!pgtable_alloc); - p4d_phys =pgtable_alloc (PGTABLE_LEVEL_P4D ); + p4d_phys =pgtable_alloc ();if (p4d_phys ==INVALID_PHYS_ADDR )return -ENOMEM ; p4dp =p4d_set_fixmap (p4d_phys);
@@ -480,7 +480,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,static int __create_pgd_mapping_locked (pgd_t *pgdir,phys_addr_t phys,unsigned long virt,phys_addr_t size,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {int ret;
@@ -513,7 +513,7 @@ static int __create_pgd_mapping_locked(pgd_t *pgdir, phys_addr_t phys,static int __create_pgd_mapping (pgd_t *pgdir,phys_addr_t phys,unsigned long virt,phys_addr_t size,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {int ret;
@@ -529,7 +529,7 @@ static int __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,static void early_create_pgd_mapping (pgd_t *pgdir,phys_addr_t phys,unsigned long virt,phys_addr_t size,pgprot_t prot, -phys_addr_t (*pgtable_alloc)(enum pgtable_level ), +phys_addr_t (*pgtable_alloc)(void ),int flags) {int ret;
@@ -540,55 +540,20 @@ static void early_create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys,panic ("Failed to create page tables\n" ); } -static phys_addr_t __pgd_pgtable_alloc (struct mm_struct *mm,gfp_t gfp, -enum pgtable_level pgtable_level) +static phys_addr_t pgd_pgtable_alloc_gfp (gfp_t gfp) {/* Page is zeroed by init_clear_pgtable() so don't duplicate effort. */ struct ptdesc *ptdesc =pagetable_alloc (gfp & ~__GFP_ZERO,0 ); -phys_addr_t pa;if (!ptdesc)return INVALID_PHYS_ADDR ; - pa =page_to_phys (ptdesc_page (ptdesc)); - -switch (pgtable_level) { -case PGTABLE_LEVEL_PTE : -BUG_ON (!pagetable_pte_ctor (mm, ptdesc)); -break ; -case PGTABLE_LEVEL_PMD : -BUG_ON (!pagetable_pmd_ctor (mm, ptdesc)); -break ; -case PGTABLE_LEVEL_PUD : -pagetable_pud_ctor (ptdesc); -break ; -case PGTABLE_LEVEL_P4D : -pagetable_p4d_ctor (ptdesc); -break ; -case PGTABLE_LEVEL_PGD : -VM_WARN_ON (1 ); -break ; - } - -return pa; -} - -static phys_addr_t -pgd_pgtable_alloc_init_mm_gfp (enum pgtable_level pgtable_level,gfp_t gfp) -{ -return __pgd_pgtable_alloc (&init_mm, gfp, pgtable_level); -} - -static phys_addr_t __maybe_unused -pgd_pgtable_alloc_init_mm (enum pgtable_level pgtable_level) -{ -return pgd_pgtable_alloc_init_mm_gfp (pgtable_level,GFP_PGTABLE_KERNEL ); +return page_to_phys (ptdesc_page (ptdesc)); } -static phys_addr_t -pgd_pgtable_alloc_special_mm (enum pgtable_level pgtable_level) +static phys_addr_t pgd_pgtable_alloc (void ) { -return __pgd_pgtable_alloc (NULL ,GFP_PGTABLE_KERNEL , pgtable_level); +return pgd_pgtable_alloc_gfp (GFP_PGTABLE_KERNEL ); }static void split_contpte (pte_t *ptep)
@@ -609,7 +574,7 @@ static int split_pmd(pmd_t *pmdp, pmd_t pmd, gfp_t gfp, bool to_cont)pte_t *ptep;int i; - pte_phys =pgd_pgtable_alloc_init_mm_gfp (PGTABLE_LEVEL_PTE , gfp); + pte_phys =pgd_pgtable_alloc_gfp (gfp);if (pte_phys ==INVALID_PHYS_ADDR )return -ENOMEM ; ptep = (pte_t *)phys_to_virt (pte_phys);
@@ -656,7 +621,7 @@ static int split_pud(pud_t *pudp, pud_t pud, gfp_t gfp, bool to_cont)pmd_t *pmdp;int i; - pmd_phys =pgd_pgtable_alloc_init_mm_gfp (PGTABLE_LEVEL_PMD , gfp); + pmd_phys =pgd_pgtable_alloc_gfp (gfp);if (pmd_phys ==INVALID_PHYS_ADDR )return -ENOMEM ; pmdp = (pmd_t *)phys_to_virt (pmd_phys);
@@ -1035,7 +1000,7 @@ void __init create_pgd_mapping(struct mm_struct *mm, phys_addr_t phys, flags =NO_BLOCK_MAPPINGS |NO_CONT_MAPPINGS ;early_create_pgd_mapping (mm->pgd, phys, virt, size, prot, - pgd_pgtable_alloc_special_mm, flags); + pgd_pgtable_alloc, flags); } static void update_mapping_prot(phys_addr_t phys, unsigned long virt,
@@ -1249,7 +1214,7 @@ static void __init declare_vma(struct vm_struct *vma,static phys_addr_t kpti_ng_temp_alloc__initdata ; -static phys_addr_t __init kpti_ng_pgd_alloc (enum pgtable_level pgtable_level) +static phys_addr_t __init kpti_ng_pgd_alloc (void ) { kpti_ng_temp_alloc -=PAGE_SIZE ;return kpti_ng_temp_alloc;
@@ -1357,7 +1322,7 @@ static int __init map_entry_trampoline(void)memset (tramp_pg_dir,0 ,PGD_SIZE );early_create_pgd_mapping (tramp_pg_dir, pa_start,TRAMP_VALIAS ,entry_tramp_text_size (), prot, - pgd_pgtable_alloc_init_mm,NO_BLOCK_MAPPINGS ); + pgd_pgtable_alloc,NO_BLOCK_MAPPINGS );/* Map both the text and data into the kernel page table */ for (i =0 ; i <DIV_ROUND_UP (entry_tramp_text_size (),PAGE_SIZE ); i++)
@@ -1445,7 +1410,6 @@ static void free_hotplug_page_range(struct page *page, size_t size,static void free_hotplug_pgtable_page (struct page *page) { -pagetable_dtor (page_ptdesc (page));free_hotplug_page_range (page,PAGE_SIZE ,NULL ); }
@@ -1958,7 +1922,7 @@ int arch_add_memory(int nid, u64 start, u64 size, flags |=NO_BLOCK_MAPPINGS |NO_CONT_MAPPINGS ; ret =__create_pgd_mapping (swapper_pg_dir, start,__phys_to_virt (start), - size, params->pgprot, pgd_pgtable_alloc_init_mm, + size, params->pgprot, pgd_pgtable_alloc, flags);if (ret) goto err;
Signature · 1 line
2.51.2
[email protected]
AI reviewRe: [PATCH 20/22] arm64: mm: drop ctor/dtor calls for kernel page tables
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] Kernel page tables allocated during memory hotplug leak the NR_PAGETABLE stat and retain the…
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] Kernel page tables allocated during memory hotplug leak the NR_PAGETABLE stat and retain the PG_pgtable flag when freed, causing a buddy allocator BUG on reallocation. --
--- Patch [20]: [PATCH 20/22] arm64: mm: drop ctor/dtor calls for kernel page tables --- commit eb398eca88012b977463df47eaf81e1fcd66213d
Author: Kevin Brodsky <[email protected]>
arm64: mm: drop ctor/dtor calls for kernel page tables
This patch removes pagetable constructor and destructor calls for kernel page tables, under the assumption that they no longer perform necessary work. It also removes the pgtable level callback arguments from the early allocation wrappers.
Quoted 4 lines
arch/arm64/mm/mmu.c +0
-0
[ ... ]
arch/arm64/mm/mmu.c static void free_hotplug_page_range(struct page *page, size_t size, · line 1414
Quoted 7 lines
arch/arm64/mm/mmu.c +0
-1@@ -1445,7 +1410,6 @@ static void free_hotplug_page_range(struct page *page, size_t size,static void free_hotplug_pgtable_page (struct page *page) { -pagetable_dtor (page_ptdesc (page));free_hotplug_page_range (page,PAGE_SIZE ,NULL ); }
[Severity: High]
Does this cause the buddy allocator to reject the physical page and leak the NR_PAGETABLE statistic during memory hot-remove?
By dropping the destructor call here, the PG_pgtable flag is no longer cleared. Since free_hotplug_page_range() eventually calls __free_pages() directly rather than using pagetable_free(), the required cleanup is entirely skipped.
If the buddy allocator later reallocates this page, check_new_page_bad() will detect the leftover PG_pgtable flag, trigger a bad_page() warning, and permanently leak the page.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=20
PATCH 21/22 arm: mm: drop ctor call for kernel page tables
In reply to Kevin BrodskyThanks to recent changes:
* pagetable_pte_ctor() does nothing for kernel page tables.
* efi_mm is also treated like a kernel mm by the page table logic, including ctors.
We can therefore drop the ctor call when allocating special kernel page tables in mmu.c.
--- arch/arm/mm/mmu.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
arch/arm/mm/mmu.c +1 -2
@@ -728,8 +728,7 @@ static void *__init late_alloc(unsigned long sz)void *ptdesc =pagetable_alloc (GFP_PGTABLE_KERNEL & ~__GFP_HIGHMEM,get_order (sz)); -if (!ptdesc || !pagetable_pte_ctor (NULL , ptdesc)) -BUG (); +BUG_ON (!ptdesc);return ptdesc_address (ptdesc); }
Signature · 1 line
2.51.2
PATCH 22/22 riscv: mm: drop ctor/dtor calls for kernel page tables
In reply to Kevin BrodskyThanks to recent changes:
* pagetable_*_ctor() and pagetable_dtor() do nothing for kernel page tables.
* efi_mm is also treated like a kernel mm by the page table logic, including ctors.
We can therefore drop the calls to ctors when allocating special kernel page tables in mmu.c, as well as the dtor call when freeing hotplug page tables.
The alloc_*_late() helpers are all identical after removing the ctor calls, so replace them with alloc_pgtable_late().
--- arch/riscv/mm/init.c | 63 ++++++++++++---------------------------------------- 1 file changed, 14 insertions(+), 49 deletions(-)
arch/riscv/mm/init.c +14 -49
@@ -382,6 +382,14 @@ void __set_fixmap(enum fixed_addresses idx, phys_addr_t phys, pgprot_t prot)local_flush_tlb_page (addr); } +static phys_addr_t __meminit alloc_pgtable_late (uintptr_t va) +{ +struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); + +BUG_ON (!ptdesc); +return __pa (ptdesc_address (ptdesc)); +} +static inline pte_t *__init get_pte_virt_early (phys_addr_t pa) {return (pte_t *)((uintptr_t )pa);
@@ -412,19 +420,6 @@ static inline phys_addr_t __init alloc_pte_fixmap(uintptr_t va)return memblock_phys_alloc (PAGE_SIZE ,PAGE_SIZE ); } -static phys_addr_t __meminit alloc_pte_late (uintptr_t va) -{ -struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); - -/* -* We do not know which mm the PTE page is associated to at this point. -* Passing NULL to the ctor is the safe option, though it may result -* in unnecessary work (e.g. initialising the ptlock for init_mm). -*/ -BUG_ON (!ptdesc || !pagetable_pte_ctor (NULL , ptdesc)); -return __pa ((pte_t *)ptdesc_address (ptdesc)); -} -static void __meminit create_pte_mapping (pte_t *ptep,uintptr_t va,phys_addr_t pa,phys_addr_t sz,pgprot_t prot) {
@@ -479,15 +474,6 @@ static phys_addr_t __init alloc_pmd_fixmap(uintptr_t va)return memblock_phys_alloc (PAGE_SIZE ,PAGE_SIZE ); } -static phys_addr_t __meminit alloc_pmd_late (uintptr_t va) -{ -struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); - -/* See comment in alloc_pte_late() regarding NULL passed the ctor */ -BUG_ON (!ptdesc || !pagetable_pmd_ctor (NULL , ptdesc)); -return __pa ((pmd_t *)ptdesc_address (ptdesc)); -} -static void __meminit create_pmd_mapping (pmd_t *pmdp,uintptr_t va,phys_addr_t pa,phys_addr_t sz,pgprot_t prot)
@@ -544,15 +530,6 @@ static phys_addr_t __init alloc_pud_fixmap(uintptr_t va)return memblock_phys_alloc (PAGE_SIZE ,PAGE_SIZE ); } -static phys_addr_t __meminit alloc_pud_late (uintptr_t va) -{ -struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); - -BUG_ON (!ptdesc); -pagetable_pud_ctor (ptdesc); -return __pa ((pud_t *)ptdesc_address (ptdesc)); -} -static p4d_t *__init get_p4d_virt_early (phys_addr_t pa) {return (p4d_t *)((uintptr_t )pa);
@@ -582,15 +559,6 @@ static phys_addr_t __init alloc_p4d_fixmap(uintptr_t va)return memblock_phys_alloc (PAGE_SIZE ,PAGE_SIZE ); } -static phys_addr_t __meminit alloc_p4d_late (uintptr_t va) -{ -struct ptdesc *ptdesc =pagetable_alloc (GFP_KERNEL ,0 ); - -BUG_ON (!ptdesc); -pagetable_p4d_ctor (ptdesc); -return __pa ((p4d_t *)ptdesc_address (ptdesc)); -} -static void __meminit create_pud_mapping (pud_t *pudp,uintptr_t va,phys_addr_t pa,phys_addr_t sz,pgprot_t prot) {
@@ -1003,14 +971,14 @@ static void __init pt_ops_set_fixmap(void)*/ static void __init pt_ops_set_late (void ) { - pt_ops.alloc_pte = alloc_pte_late; + pt_ops.alloc_pte = alloc_pgtable_late; pt_ops.get_pte_virt = get_pte_virt_late;#ifndef __PAGETABLE_PMD_FOLDED - pt_ops.alloc_pmd = alloc_pmd_late; + pt_ops.alloc_pmd = alloc_pgtable_late; pt_ops.get_pmd_virt = get_pmd_virt_late; - pt_ops.alloc_pud = alloc_pud_late; + pt_ops.alloc_pud = alloc_pgtable_late; pt_ops.get_pud_virt = get_pud_virt_late; - pt_ops.alloc_p4d = alloc_p4d_late; + pt_ops.alloc_p4d = alloc_pgtable_late; pt_ops.get_p4d_virt = get_p4d_virt_late;#endif }
@@ -1488,7 +1456,6 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)return ; } -pagetable_dtor (ptdesc);if (PageReserved (page))free_reserved_page (page);else
@@ -1496,7 +1463,7 @@ static void __meminit free_pte_table(pte_t *pte_start, pmd_t *pmd)pmd_clear (pmd); } -static void __meminit free_pmd_table (pmd_t *pmd_start,pud_t *pud,bool is_vmemmap) +static void __meminit free_pmd_table (pmd_t *pmd_start,pud_t *pud) {struct page *page =pud_page (*pud);struct ptdesc *ptdesc =page_ptdesc (page);
@@ -1509,8 +1476,6 @@ static void __meminit free_pmd_table(pmd_t *pmd_start, pud_t *pud, bool is_vmemmreturn ; } -if (!is_vmemmap) -pagetable_dtor (ptdesc);if (PageReserved (page))free_reserved_page (page);else
@@ -1634,7 +1599,7 @@ static void __meminit remove_pud_mapping(pud_t *pud_base, unsigned long addr, unremove_pmd_mapping (pmd_base, addr, next, is_vmemmap, altmap);if (pgtable_l4_enabled) -free_pmd_table (pmd_base, pudp, is_vmemmap); +free_pmd_table (pmd_base, pudp); } }
Signature · 1 line
2.51.2
Quoted 21 lines · Kevin Brodsky
Page table constructors and destructors were originally introduced specifically for user PTE pages. Over time, their use became more widespread, and starting from v6.16 they are called at all levels and for both user and kernel page tables [1]. Or so I thought. Turns out, we still have various special kernel page tables where the ctor/dtor are not called; vmemmap page tables is one such case.
Looking closer, we do not actually need per-level ctors for kernel page tables. In fact we do not need ctors for them at all, because nothing is specific to kernel page tables in the ctor/dtor and nowadays they should be allocated with
pagetable_alloc().Here's the plan: all the generic page table handling moves to pagetable_{alloc,free}(). Whatever is left is specific to user page tables; the ctor/dtor are only called for user page tables at the required levels (PTE/PMD), and renamed to make that clear.
Doing this will allow us to remove a good amount of special casing, but there's quite a bit of churn to get there. Here's what this series does:
Quoted 7 lines
1. Introduce the concept of "kernel mm" and replace most comparisons with
&init_mmwith a flag check (patch 1-12); no functional change so far.2. Move the page table accounting/type setting to pagetable_{alloc,free} (patch 13-15). Kernel mm's do not need to call the ctor/dtor any more.
The underlying assumption here is that pagetable_{alloc,free} are used wherever the ctor/dtor are called, and as Sashiko pointed out there are counterexamples:
1. sparc32 calls the ctor/dtor for user PTE pages, but uses a custom allocator.
2. archs that do not select MMU_GATHER_TABLE_FREE do not use pagetable_free() in tlb_remove_table().
3. arch_remove_memory() and vmemmap_free() are a bit of a mess - some archs call the dtor and/or use pagetable_free() for freeing the direct map page tables, some don't.
How I'm thinking to address these:
1. sparc32 does not use split ptlocks, so maybe doing the memory accounting explicitly? Otherwise, we may need to keep the page type handling + memory accounting in a separate helper that sparc32 can explicitly call.
2. Always enable MMU_GATHER_TABLE_FREE. pagetable_free() should always be used for pages allocated with pagetable_alloc().
3. Go through all the implementations of memory hotplug and vmemmap, and ensure that pagetable_free() is used for those pages allocated with pagetable_free() (and only those). That may well call for a separate series.
- Kevin