s390/mm: Batch PTE updates in lazy MMU mode
Subsystems (36)
Alexander Gordeev
maintainerv8 cover s390/mm: Batch PTE updates in lazy MMU mode
Hi All,
This is v8 of the batched PTE updates in lazy MMU mode rework.
Patches 1-9 are a prerequisite series [1] backported from mm-new to master. It is only posted to facilitate the follow-up s390 lazy MMU rework review.
Patch 10 is a cleanup that could be picked independently and is a prerequisite to patch 11
Patch 11 is the s390 variant of the ARM64 series [2] and provides hw_pte_t type enablement
Patches 12,13 are the lazy MMU mode implementation and the main objective of this rework.
Patches 14,15 are an additional KASAN-based lazy-PTE guard mechanism.
1. https://lore.kernel.org/linux-mm/[email protected]/
2. https://lore.kernel.org/all/[email protected]/
Changes since v7:
- CPUHP_BP_PREPARE_DYN hotplug event is used to (de-)allocate per-cpu data
- lazy_mmu_enabled static key is used to indicate the lazy mmu mode availability
- is_lazy_mmu_active() is implemented using CLIY alternative
- lazy_mmu_count lowcore variable is turned 1 byte to allow CLIY alternative
Changes since v6:
- bottom-halves are disabled on entering and leaving the lazy mmu mode
Changes since v5:
- IPTE optimization is not applied to secure guests [4]
- __kasan_(un)poison_pte() are marked as EXPORT_SYMBOL_GPL() [5]
- PTE table poisoning is not applied to architectures with PTE entry size= s
unaligned on KASAN_GRANULE_SIZE [5]
4. https://lore.kernel.org/linux-s390/cover.1783945507.git.agordeev@linux=
.ibm.com/T/#md98724bd3b66d0a0711deb6089fa541f420566d8
5. https://lore.kernel.org/linux-s390/cover.1783945507.git.agordeev@linux=
.ibm.com/T/#m9c8b31b4863416732d0cbbc5f5f63290db4522c1
Changes since v4:
- verified that a presumable sashiko performance regression finding [2] although appears valid does not really degrade performance
- applied sashiko suggestion [3] and added "direct-pte-access" kasan bug = type
2. https://lore.kernel.org/linux-s390/20260623062703.269982B28-agordeev@l=
inux.ibm.com/#r
3. https://lore.kernel.org/linux-s390/20260623061321.269982A64-agordeev@l=
inux.ibm.com/
Changes since v3:
- all prerequisite patches are landed in -next and removed from the serie= s
Changes since v2:
- lazy_mmu_mode_enable_for_pte_range() renamed to lazy_mmu_mode_enable_wi= th_ptes()
(David Hildenbrand)
- patch "mm/pgtable: Fix bogus comment to clear_not_present_full_ptes()" is dropped (David Hildenbrand)
- direct PTE dereferencing KASAN sanitizer added (Heiko Carstens)
- CONFIG_IPTE_BATCH option is dropped (Heiko Carstens)
- PTE_POISON changed from zero to 0x800 (Heiko Carstens)
- allocate per-cpu caches on CPU hot-plug (Heiko Carstens)
- introduced a lowcore field for fast lazy mode checking (Heiko Carstens)
- few minor code changes (Heiko Carstens)
Changes since v1:
- lazy_mmu_mode_enable_pte() renamed to lazy_mmu_mode_enable_for_pte_rang= e()
- lazy_mmu_mode_enable_for_pte_range() semantics clarified
- some sashiko comments addressed [1] including one bug fix [1]
- patches 2-4 added
1. https://sashiko.dev/#/patchset/cover.1774420056.git.agordeev%40linux.i=
bm.com
This series addresses an s390-specific aspect of how page table entries are modified. In many cases, changing a valid PTE (for example, setting or clearing a hardware bit) requires issuing an Invalidate Page Table Entry (IPTE) instruction beforehand.
A disadvantage of the IPTE instruction is that it may initiate a machine-wide quiesce state. This state acts as an expensive global hardware lock and should be avoided whenever possible.
Currently, IPTE is invoked for each individual PTE update in most code paths. However, the instruction itself supports invalidating multiple PTEs at once, covering up to 256 entries. Using this capability can significantly reduce the number of quiesce events, with a positive impact on overall system performance. At present, this feature is not utilized.
An effort was therefore made to identify kernel code paths that update large numbers of consecutive PTEs. Such updates can be batched and handled by a single IPTE invocation, leveraging the hardware support described above.
A natural candidate for this optimization is page-table walkers that change attributes of memory ranges and thus modify contiguous ranges of PTEs. Many memory-management system calls enter lazy MMU mode while updating such ranges.
This lazy MMU mode can be leveraged to build on the already existing infrastructure and implement a software-level lazy MMU mechanism, allowing expensive PTE invalidations on s390 to be batched.
Thanks!
Alexander Gordeev (6):
s390/mm: Cleanup pXXp_flush_lazy() routines
s390: Distinguish hardware and software PTEs
mm: Make lazy MMU mode context-aware
s390/mm: Batch PTE updates in lazy MMU mode mm/kasan: Introduce helpers for lazy MMU mode sanitizer s390/mm: Lazy MMU mode sanitizer
Muhammad Usama Anjum (9):
mm: introduce hw_pte_t for PTE table storage
mm: rename pointers to software PTE values as ptentp
mm: use hw_pte_t for generic PTE table storage
mm: convert PTE table entries in ptep_get()
mm: convert PTE table entry to pte
mm: add hw_pte_val for HW PTE storage mm/kasan: use hw_pte_t for the early shadow PTE table drm/i915: use hw_pte_t for PTE range callbacks
xen: use hw_pte_t for PTE range callbacks
MAINTAINERS | 1 + arch/s390/Kconfig | 2 + arch/s390/boot/startup.c | 2 +- arch/s390/boot/vmem.c | 17 +- arch/s390/include/asm/gmap_helpers.h | 2 +- arch/s390/include/asm/hugetlb.h | 18 +- arch/s390/include/asm/lowcore.h | 3 +- arch/s390/include/asm/maccess.h | 2 +- arch/s390/include/asm/page.h | 3 +- arch/s390/include/asm/pgalloc.h | 6 +- arch/s390/include/asm/pgtable.h | 215 +++++++-- arch/s390/kernel/uv.c | 4 +- arch/s390/kvm/s390/pv.c | 2 +- arch/s390/mm/Makefile | 2 +- arch/s390/mm/gmap_helpers.c | 18 +- arch/s390/mm/hugetlbpage.c | 20 +- arch/s390/mm/lazy_mmu.c | 449 ++++++++++++++++++ arch/s390/mm/maccess.c | 2 +- arch/s390/mm/pageattr.c | 10 +- arch/s390/mm/pgtable.c | 34 +- arch/s390/mm/vmem.c | 26 +- .../drm/i915/gem/selftests/i915_gem_mman.c | 4 +- drivers/gpu/drm/i915/i915_mm.c | 4 +- drivers/xen/gntdev.c | 2 +- drivers/xen/privcmd.c | 2 +- drivers/xen/xenbus/xenbus_client.c | 2 +- drivers/xen/xlate_mmu.c | 4 +- fs/hugetlbfs/inode.c | 3 +- fs/proc/task_mmu.c | 35 +- include/asm-generic/hugetlb.h | 15 +- include/asm-generic/pgalloc.h | 6 +- include/asm-generic/tlb.h | 5 +- include/linux/hugetlb.h | 53 ++- include/linux/kasan.h | 21 +- include/linux/mm.h | 26 +- include/linux/page_table_check.h | 10 +- include/linux/pagewalk.h | 10 +- include/linux/pgtable.h | 127 +++-- include/linux/pgtable_types.h | 23 + include/linux/rmap.h | 2 +- include/linux/swapops.h | 6 +- include/linux/vmalloc.h | 4 +- include/trace/events/xen.h | 10 +- kernel/bpf/arena.c | 9 +- kernel/events/core.c | 3 +- mm/Kconfig | 3 + mm/damon/ops-common.c | 2 +- mm/damon/ops-common.h | 2 +- mm/damon/vaddr.c | 20 +- mm/debug_vm_pgtable.c | 2 +- mm/filemap.c | 4 +- mm/gup.c | 9 +- mm/highmem.c | 15 +- mm/hmm.c | 6 +- mm/huge_memory.c | 4 +- mm/hugetlb.c | 60 +-- mm/hugetlb_vmemmap.c | 13 +- mm/internal.h | 16 +- mm/kasan/common.c | 14 + mm/kasan/init.c | 14 +- mm/kasan/kasan.h | 2 + mm/kasan/report_generic.c | 3 + mm/kasan/shadow.c | 6 +- mm/khugepaged.c | 50 +- mm/ksm.c | 11 +- mm/madvise.c | 26 +- mm/mapping_dirty_helpers.c | 4 +- mm/memory-failure.c | 6 +- mm/memory.c | 86 ++-- mm/mempolicy.c | 4 +- mm/migrate.c | 4 +- mm/migrate_device.c | 4 +- mm/mincore.c | 4 +- mm/mlock.c | 4 +- mm/mprotect.c | 21 +- mm/mremap.c | 6 +- mm/page_table_check.c | 4 +- mm/pagewalk.c | 9 +- mm/percpu.c | 2 +- mm/pgtable-generic.c | 20 +- mm/ptdump.c | 4 +- mm/rmap.c | 6 +- mm/sparse-vmemmap.c | 22 +- mm/swap_state.c | 3 +- mm/swapfile.c | 5 +- mm/userfaultfd.c | 32 +- mm/util.c | 2 +- mm/vmalloc.c | 17 +- mm/vmscan.c | 6 +- 89 files changed, 1269 insertions(+), 512 deletions(-) create mode 100644 arch/s390/mm/lazy_mmu.c create mode 100644 include/linux/pgtable_types.h
Signature · 1 line
2.53.0
Alexander Gordeev
maintainerPATCH v8 1/15 mm: introduce hw_pte_t for PTE table storage
From: Muhammad Usama Anjum <[email protected]>
pte_t is used both for software PTE values and for entries stored in a PTE table, so pte_t * does not distinguish a pointer to a software PTE value from a pointer to table storage.
Introduce hw_pte_t as the generic name for a PTE table element. Define it as a macro alias of pte_t by default. When an architecture selects ARCH_HAS_HW_PTE_T, define it as a structure containing a pte_t instead. This preserves the representation while allowing converted architectures to enforce the distinction at compile time.
Name the generic wrapper structure __hw_pte_t so architectures can forward-declare it when pgtable_t must be defined before the generic hw_pte_t typedef is visible. This avoids header-order dependencies.
Keep the C type definitions behind an __ASSEMBLY__ check because architecture assembly sources can include this header indirectly. Include asm/page.h so consumers such as linux/vmalloc.h retain the page definitions they previously obtained from that header.
--- MAINTAINERS | 1 + include/linux/pgtable_types.h | 17 +++++++++++++++++ mm/Kconfig | 3 +++ 3 files changed, 21 insertions(+) create mode 100644 include/linux/pgtable_types.h
MAINTAINERS +1 -0
@@ -17142,6 +17142,7 @@ F: include/linux/mmu_notifier.h F: include/linux/pagewalk.h F: include/linux/pgalloc.h F: include/linux/pgtable.h +F: include/linux/pgtable_types.h F: include/linux/ptdump.h F: include/linux/vmpressure.h F: include/linux/vmstat.h
include/linux/pgtable_types.h new file +17 -0
@@ -0,0 +1,17 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_PGTABLE_TYPES_H +#define _LINUX_PGTABLE_TYPES_H + +#include <asm/page.h> + +#ifndef __ASSEMBLY__ + +#ifdef CONFIG_ARCH_HAS_HW_PTE_T +typedef struct __hw_pte_t {pte_t __pte; }hw_pte_t ; +#else +#define hw_pte_t pte_t +#endif + +#endif /* !__ASSEMBLY__ */ + +#endif /* _LINUX_PGTABLE_TYPES_H */
mm/Kconfig +3 -0
@@ -1317,6 +1317,9 @@ comment "GUP_TEST needs to have DEBUG_FS enabled" config GUP_GET_PXX_LOW_HIGH bool +config ARCH_HAS_HW_PTE_T + bool + config DMAPOOL_TEST tristate "Enable a module to run time tests on dma_pool" depends on HAS_DMA
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 01/15] mm: introduce hw_pte_t for PTE table storage
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 16 lines
From: Muhammad Usama Anjum <[email protected]>
pte_tis used both for software PTE values and for entries stored in a PTE table, sopte_t* does not distinguish a pointer to a software PTE value from a pointer to table storage.Introduce
hw_pte_tas the generic name for a PTE table element. Define it as a macro alias ofpte_tby default. When an architecture selectsARCH_HAS_HW_PTE_T, define it as a structure containing apte_tinstead. This preserves the representation while allowing converted architectures to enforce the distinction at compile time.Name the generic wrapper structure
__hw_pte_tso architectures can forward-declare it whenpgtable_tmust be defined before the generichw_pte_ttypedef is visible. This avoids header-order dependencies.
Quoted 7 lines
Keep the C type definitions behind an
__ASSEMBLY__check because architecture assembly sources can include this header indirectly. Includeasm/page.hso consumers such aslinux/vmalloc.hretain the page definitions they previously obtained from that header.Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Alexander Gordeev
maintainerPATCH v8 2/15 mm: rename pointers to software PTE values as ptentp
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
Some interfaces use pte_t * for a software PTE value rather than for an entry stored in a PTE table. These pointers must remain pte_t * when pointers to PTE table storage are converted to hw_pte_t *.
Rename these parameters in the install_pte callback, write_protect_page(), and guard_install_set_pte() to ptentp. The later Coccinelle conversion skips pointers named ptentp, allowing it to convert the remaining PTE table pointers without changing these interfaces.
Some functions already use ptentp for such pointers, including:
- madvise_folio_pte_batch()
- folio_pte_batch_flags()
No need to rename them.
This patch only renames parameters and makes no functional change.
--- include/linux/pagewalk.h | 2 +- mm/ksm.c | 4 ++-- mm/madvise.c | 4 ++-- 3 files changed, 5 insertions(+), 5 deletions(-)
include/linux/pagewalk.h +1 -1
@@ -89,7 +89,7 @@ struct mm_walk_ops { struct mm_walk *walk);void (*post_vma)(struct mm_walk *walk);int (*install_pte)(unsigned long addr,unsigned long next, -pte_t *ptep,struct mm_walk *walk); +pte_t *ptentp,struct mm_walk *walk);enum page_walk_lock walk_lock; };
mm/ksm.c +2 -2
@@ -1291,7 +1291,7 @@ static u32 calc_checksum(struct page *page) }static int write_protect_page (struct vm_area_struct *vma,struct folio *folio, -pte_t *orig_pte) +pte_t *ptentp) {struct mm_struct *mm = vma->vm_mm;DEFINE_FOLIO_VMA_WALK (pvmw, folio, vma,0 ,0 );
@@ -1370,7 +1370,7 @@ static int write_protect_page(struct vm_area_struct *vma, struct folio *folio,set_pte_at (mm, pvmw.address, pvmw.pte, entry); } - *orig_pte = entry; + *ptentp = entry; err =0 ; out_unlock:
mm/madvise.c +2 -2
@@ -1110,12 +1110,12 @@ static int guard_install_pte_entry(pte_t *pte, unsigned long addr, }static int guard_install_set_pte (unsigned long addr,unsigned long next, -pte_t *ptep,struct mm_walk *walk) +pte_t *ptentp,struct mm_walk *walk) {unsigned long *nr_pages = (unsigned long *)walk->private;/* Simply install a PTE marker, this causes segfault on access. */ - *ptep =make_pte_marker (PTE_MARKER_GUARD ); + *ptentp =make_pte_marker (PTE_MARKER_GUARD ); (*nr_pages)++;return 0 ;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 02/15] mm: rename pointers to software PTE values as ptentp
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] mm: missed software PTE pointer rename in damon_hugetlb_ptep_mkold --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] mm: missed software PTE pointer rename in damon_hugetlb_ptep_mkold --
commit a7aa8dc368dae477c79fe8a52a9e505b4cbf8802
Author: Muhammad Usama Anjum <[email protected]>
mm: rename pointers to software PTE values as ptentp
This commit renames pointers to software PTE values to ptentp in certain functions. This ensures they are excluded from a subsequent Coccinelle conversion of PTE table pointers to hw_pte_t *.
Quoted 4 lines
Rename these parameters in the
install_ptecallback,write_protect_page(), andguard_install_set_pte()to ptentp. The later Coccinelle conversion skips pointers named ptentp, allowing it to convert the remaining PTE table pointers without changing these interfaces.
[Severity: Low]
Was damon_hugetlb_ptep_mkold() in mm/damon/vaddr.c missed during this renaming sweep?
It appears the entry parameter in damon_hugetlb_ptep_mkold() is a pointer to a software PTE value:
mm/damon/vaddr.c:
static bool damon_hugetlb_ptep_mkold(pte_t *pte, struct mm_struct *mm,
struct vm_area_struct *vma, unsigned long addr, pte_t *entry)
Because this wasn't renamed to ptentp, a subsequent Coccinelle conversion to hw_pte_t * misses this parameter. Meanwhile, the caller damon_hugetlb_mkold() is updated to use hw_pte_t *.
On architectures like s390 where hw_pte_t is a distinct struct type rather than a typedef, this mismatch between the caller's hw_pte_t * and the callee's pte_t * (as well as passing a pte_t * to huge_ptep_get_and_clear which expects hw_pte_t *) triggers an incompatible pointer type build failure when CONFIG_HUGETLB_PAGE is enabled.
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
Alexander Gordeev
maintainerPATCH v8 3/15 mm: use hw_pte_t for generic PTE table storage
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
Generic page-table interfaces use pte_t * for both pointers to PTE table storage and pointers to software PTE values. Convert the generic declarations and their MM, fs, and kernel users together so parameters, return types, callbacks, and local pointers that designate table storage use hw_pte_t *.
Include linux/pgtable_types.h from headers that expose the converted interfaces. It continues to provide pgprot_t to vmalloc.h through asm/page.h.
Keep software PTE values as pte_t and retain pte_t * for value interfaces.
No architecture selects ARCH_HAS_HW_PTE_T at this point, so hw_pte_t remains an alias of pte_t and this changes the interface vocabulary without changing representation or behavior.
Most of this mechanical conversion was generated with the Coccinelle script included in the cover letter. The script deliberately ignores pte_t * pointers named ptentp because they designate software PTE values. The result was then audited, and sites the script could not convert were updated by hand.
--- fs/hugetlbfs/inode.c | 3 +- fs/proc/task_mmu.c | 33 ++++++------- include/asm-generic/hugetlb.h | 15 +++--- include/asm-generic/pgalloc.h | 6 +-- include/asm-generic/tlb.h | 5 +- include/linux/hugetlb.h | 50 +++++++++++--------- include/linux/mm.h | 26 +++++------ include/linux/page_table_check.h | 10 ++-- include/linux/pagewalk.h | 8 ++-- include/linux/pgtable.h | 79 +++++++++++++++++--------------- include/linux/rmap.h | 2 +- include/linux/swapops.h | 6 ++- include/linux/vmalloc.h | 4 +- include/trace/events/xen.h | 10 ++-- kernel/bpf/arena.c | 9 ++-- kernel/events/core.c | 3 +- mm/damon/ops-common.c | 2 +- mm/damon/ops-common.h | 2 +- mm/damon/vaddr.c | 20 ++++---- mm/debug_vm_pgtable.c | 2 +- mm/filemap.c | 4 +- mm/gup.c | 9 ++-- mm/highmem.c | 15 +++--- mm/hmm.c | 6 +-- mm/huge_memory.c | 4 +- mm/hugetlb.c | 60 ++++++++++++------------ mm/hugetlb_vmemmap.c | 13 +++--- mm/internal.h | 16 +++---- mm/kasan/init.c | 12 ++--- mm/kasan/shadow.c | 6 +-- mm/khugepaged.c | 50 ++++++++++++-------- mm/ksm.c | 7 +-- mm/madvise.c | 14 +++--- mm/mapping_dirty_helpers.c | 4 +- mm/memory-failure.c | 6 +-- mm/memory.c | 78 ++++++++++++++++--------------- mm/mempolicy.c | 4 +- mm/migrate.c | 4 +- mm/migrate_device.c | 4 +- mm/mincore.c | 4 +- mm/mlock.c | 4 +- mm/mprotect.c | 19 ++++---- mm/mremap.c | 4 +- mm/page_table_check.c | 4 +- mm/pagewalk.c | 9 ++-- mm/percpu.c | 2 +- mm/pgtable-generic.c | 20 ++++---- mm/ptdump.c | 2 +- mm/rmap.c | 6 +-- mm/sparse-vmemmap.c | 22 ++++----- mm/swap_state.c | 3 +- mm/swapfile.c | 5 +- mm/userfaultfd.c | 32 +++++++------ mm/util.c | 2 +- mm/vmalloc.c | 11 +++-- mm/vmscan.c | 6 +-- 56 files changed, 410 insertions(+), 356 deletions(-)
fs/hugetlbfs/inode.c +2 -1
@@ -328,7 +328,8 @@ static void hugetlb_delete_from_page_cache(struct folio *folio)static bool hugetlb_vma_maps_pfn (struct vm_area_struct *vma,unsigned long addr,unsigned long pfn) { -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte; ptep =hugetlb_walk (vma, addr,huge_page_size (hstate_vma (vma)));if (!ptep)
fs/proc/task_mmu.c +17 -16
@@ -1046,7 +1046,7 @@ static void smaps_pte_hole_lookup(unsigned long addr, struct mm_walk *walk)#endif } -static void smaps_pte_entry (pte_t *pte,unsigned long addr, +static void smaps_pte_entry (hw_pte_t *pte,unsigned long addr,struct mm_walk *walk) {struct mem_size_stats *mss = walk->private;
@@ -1140,7 +1140,7 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) {struct vm_area_struct *vma = walk->vma; -pte_t *pte; +hw_pte_t *pte;spinlock_t *ptl; ptl =pmd_trans_huge_lock (pmd, vma);
@@ -1263,7 +1263,7 @@ static void show_smap_vma_flags(struct seq_file *m, struct vm_area_struct *vma) }#ifdef CONFIG_HUGETLB_PAGE -static int smaps_hugetlb_range (pte_t *pte,unsigned long hmask, +static int smaps_hugetlb_range (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -1704,7 +1704,7 @@ static inline bool pte_is_pinned(struct vm_area_struct *vma, unsigned long addr, }static inline void clear_soft_dirty (struct vm_area_struct *vma, -unsigned long addr,pte_t *pte) +unsigned long addr,hw_pte_t *pte) {if (!pgtable_supports_soft_dirty ())return ;
@@ -1772,7 +1772,8 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, {struct clear_refs_private *cp = walk->private;struct vm_area_struct *vma = walk->vma; -pte_t *pte, ptent; +hw_pte_t *pte; +pte_t ptent;spinlock_t *ptl;struct folio *folio;
@@ -2172,7 +2173,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,struct vm_area_struct *vma = walk->vma;struct pagemapread *pm = walk->private;spinlock_t *ptl; -pte_t *pte, *orig_pte; +hw_pte_t *pte, *orig_pte;int err =0 ;#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -2210,7 +2211,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end,#ifdef CONFIG_HUGETLB_PAGE /* This function walks within one hugetlb entry in the single call */ -static int pagemap_hugetlb_range (pte_t *ptep,unsigned long hmask, +static int pagemap_hugetlb_range (hw_pte_t *ptep,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -2501,7 +2502,7 @@ static unsigned long pagemap_page_category(struct pagemap_scan_private *p, }static void make_uffd_wp_pte (struct vm_area_struct *vma, -unsigned long addr,pte_t *pte,pte_t ptent) +unsigned long addr,hw_pte_t *pte,pte_t ptent) {if (pte_present (ptent)) {pte_t old_pte;
@@ -2634,7 +2635,7 @@ static unsigned long pagemap_hugetlb_category(struct vm_area_struct *vma, }static void make_uffd_wp_huge_pte (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep, +unsigned long addr,hw_pte_t *ptep,pte_t ptent) {const unsigned long psize =huge_page_size (hstate_vma (vma));
@@ -2868,7 +2869,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,struct pagemap_scan_private *p = walk->private;struct vm_area_struct *vma = walk->vma;unsigned long addr, flush_end =0 ; -pte_t *pte, *start_pte; +hw_pte_t *pte, *start_pte;spinlock_t *ptl;int ret;
@@ -2962,7 +2963,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, }#ifdef CONFIG_HUGETLB_PAGE -static int pagemap_scan_hugetlb_entry (pte_t *ptep,unsigned long hmask, +static int pagemap_scan_hugetlb_entry (hw_pte_t *ptep,unsigned long hmask,unsigned long start,unsigned long end,struct mm_walk *walk) {
@@ -3043,7 +3044,7 @@ static int pagemap_scan_hugetlb_hole_wp(struct vm_area_struct *vma,unsigned long psize =huge_page_size (h);struct mm_struct *mm = vma->vm_mm;spinlock_t *ptl; -pte_t *ptep; +hw_pte_t *ptep;pte_t pte;for (addr =ALIGN_DOWN (addr, psize); addr < end; addr += psize) {
@@ -3427,8 +3428,8 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr,struct numa_maps *md = walk->private;struct vm_area_struct *vma = walk->vma;spinlock_t *ptl; -pte_t *orig_pte; -pte_t *pte; +hw_pte_t *orig_pte; +hw_pte_t *pte;#ifdef CONFIG_TRANSPARENT_HUGEPAGE ptl =pmd_trans_huge_lock (pmd, vma);
@@ -3461,7 +3462,7 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr,return 0 ; }#ifdef CONFIG_HUGETLB_PAGE -static int gather_hugetlb_stats (pte_t *pte,unsigned long hmask, +static int gather_hugetlb_stats (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {pte_t huge_pte;
@@ -3484,7 +3485,7 @@ static int gather_hugetlb_stats(pte_t *pte, unsigned long hmask, }#else -static int gather_hugetlb_stats (pte_t *pte,unsigned long hmask, +static int gather_hugetlb_stats (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {return 0 ;
include/asm-generic/hugetlb.h +8 -7
@@ -60,7 +60,7 @@ static inline int huge_pte_uffd(pte_t pte)#ifndef __HAVE_ARCH_HUGE_PTE_CLEARstatic inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,unsigned long sz) +hw_pte_t *ptep,unsigned long sz) {pte_clear (mm, addr, ptep); }
@@ -68,7 +68,7 @@ static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr,#ifndef __HAVE_ARCH_HUGE_SET_HUGE_PTE_ATstatic inline void set_huge_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte,unsigned long sz) +hw_pte_t *ptep,pte_t pte,unsigned long sz) {set_pte_at (mm, addr, ptep, pte); }
@@ -76,7 +76,7 @@ static inline void set_huge_pte_at(struct mm_struct *mm, unsigned long addr,#ifndef __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEARstatic inline pte_t huge_ptep_get_and_clear (struct mm_struct *mm, -unsigned long addr,pte_t *ptep,unsigned long sz) +unsigned long addr,hw_pte_t *ptep,unsigned long sz) {return ptep_get_and_clear (mm, addr, ptep); }
@@ -84,7 +84,7 @@ static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,#ifndef __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSHstatic inline pte_t huge_ptep_clear_flush (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep) +unsigned long addr,hw_pte_t *ptep) {return ptep_clear_flush (vma, addr, ptep); }
@@ -99,7 +99,7 @@ static inline int huge_pte_none(pte_t pte)#ifndef __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECTstatic inline void huge_ptep_set_wrprotect (struct mm_struct *mm, -unsigned long addr,pte_t *ptep) +unsigned long addr,hw_pte_t *ptep) {ptep_set_wrprotect (mm, addr, ptep); }
@@ -107,7 +107,7 @@ static inline void huge_ptep_set_wrprotect(struct mm_struct *mm,#ifndef __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGSstatic inline int huge_ptep_set_access_flags (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep, +unsigned long addr,hw_pte_t *ptep,pte_t pte,int dirty) {return ptep_set_access_flags (vma, addr, ptep, pte, dirty);
@@ -115,7 +115,8 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma,#endif #ifndef __HAVE_ARCH_HUGE_PTEP_GET -static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +static inline pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr, +hw_pte_t *ptep) {return ptep_get (ptep); }
include/asm-generic/pgalloc.h +3 -3
@@ -16,7 +16,7 @@* * Return: pointer to the allocated memory or %NULL on error */ -static inline pte_t *__pte_alloc_one_kernel_noprof (struct mm_struct *mm) +static inline hw_pte_t *__pte_alloc_one_kernel_noprof (struct mm_struct *mm) {struct ptdesc *ptdesc =pagetable_alloc_noprof (GFP_PGTABLE_KERNEL ,0 );
@@ -40,7 +40,7 @@ static inline pte_t *__pte_alloc_one_kernel_noprof(struct mm_struct *mm)* * Return: pointer to the allocated memory or %NULL on error */ -static inline pte_t *pte_alloc_one_kernel_noprof (struct mm_struct *mm) +static inline hw_pte_t *pte_alloc_one_kernel_noprof (struct mm_struct *mm) {return __pte_alloc_one_kernel_noprof (mm); }
@@ -52,7 +52,7 @@ static inline pte_t *pte_alloc_one_kernel_noprof(struct mm_struct *mm)* @mm: the mm_struct of the current context * @pte: pointer to the memory containing the page table */ -static inline void pte_free_kernel (struct mm_struct *mm,pte_t *pte) +static inline void pte_free_kernel (struct mm_struct *mm,hw_pte_t *pte) {pagetable_dtor_free (virt_to_ptdesc (pte)); }
include/asm-generic/tlb.h +3 -2
@@ -644,7 +644,8 @@ static inline void tlb_flush_p4d_range(struct mmu_gather *tlb, }#ifndef __tlb_remove_tlb_entry -static inline void __tlb_remove_tlb_entry (struct mmu_gather *tlb,pte_t *ptep,unsigned long address) +static inline void __tlb_remove_tlb_entry (struct mmu_gather *tlb, +hw_pte_t *ptep,unsigned long address) { }#endif
@@ -670,7 +671,7 @@ static inline void __tlb_remove_tlb_entry(struct mmu_gather *tlb, pte_t *ptep, u* consecutive ptes instead of only a single one. */ static inline void tlb_remove_tlb_entries (struct mmu_gather *tlb, -pte_t *ptep,unsigned int nr,unsigned long address) +hw_pte_t *ptep,unsigned int nr,unsigned long address) {tlb_flush_pte_range (tlb, address,PAGE_SIZE * nr);for (;;) {
include/linux/hugetlb.h +28 -22
168 lines not on this page · raw message
include/linux/mm.h +13 -13
@@ -779,7 +779,7 @@ struct vm_fault {* VM_FAULT_ERROR). */ /* These three entries are valid only while holding ptl lock */ -pte_t *pte;/* Pointer to pte entry matching +hw_pte_t *pte;/* Pointer to pte entry matching * the 'address'. NULL if the page * table hasn't been allocated. */
@@ -3249,7 +3249,7 @@ struct follow_pfnmap_args {* The caller shouldn't touch any of these. */ spinlock_t *lock; -pte_t *ptep; +hw_pte_t *ptep;/** * Outputs: *
@@ -3586,7 +3586,7 @@ static inline pud_t pud_mkspecial(pud_t pud) }#endif /* CONFIG_ARCH_SUPPORTS_PUD_PFNMAP */ -extern pte_t *get_locked_pte (struct mm_struct *mm,unsigned long addr, +extern hw_pte_t *get_locked_pte (struct mm_struct *mm,unsigned long addr,spinlock_t **ptl);#ifdef __PAGETABLE_P4D_FOLDED
@@ -3864,7 +3864,7 @@ static inline spinlock_t *pte_lockptr(struct mm_struct *mm, pmd_t *pmd)return ptlock_ptr (page_ptdesc (pmd_page (*pmd))); } -static inline spinlock_t *ptep_lockptr (struct mm_struct *mm,pte_t *pte) +static inline spinlock_t *ptep_lockptr (struct mm_struct *mm,hw_pte_t *pte) {BUILD_BUG_ON (IS_ENABLED (CONFIG_HIGHPTE ));BUILD_BUG_ON (MAX_PTRS_PER_PTE *sizeof (pte_t) >PAGE_SIZE );
@@ -3895,7 +3895,7 @@ static inline spinlock_t *pte_lockptr(struct mm_struct *mm, pmd_t *pmd) {return &mm->page_table_lock; } -static inline spinlock_t *ptep_lockptr (struct mm_struct *mm,pte_t *pte) +static inline spinlock_t *ptep_lockptr (struct mm_struct *mm,hw_pte_t *pte) {return &mm->page_table_lock; }
@@ -3936,19 +3936,19 @@ static inline bool pagetable_pte_ctor(struct mm_struct *mm,return true; } -pte_t *__pte_offset_map (pmd_t *pmd,unsigned long addr,pmd_t *pmdvalp); +hw_pte_t *__pte_offset_map (pmd_t *pmd,unsigned long addr,pmd_t *pmdvalp); -static inline pte_t *pte_offset_map (pmd_t *pmd,unsigned long addr) +static inline hw_pte_t *pte_offset_map (pmd_t *pmd,unsigned long addr) {return __pte_offset_map (pmd, addr,NULL ); } -pte_t *pte_offset_map_lock (struct mm_struct *mm,pmd_t *pmd, +hw_pte_t *pte_offset_map_lock (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,spinlock_t **ptlp); -pte_t *pte_offset_map_ro_nolock (struct mm_struct *mm,pmd_t *pmd, +hw_pte_t *pte_offset_map_ro_nolock (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,spinlock_t **ptlp); -pte_t *pte_offset_map_rw_nolock (struct mm_struct *mm,pmd_t *pmd, +hw_pte_t *pte_offset_map_rw_nolock (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,pmd_t *pmdvalp,spinlock_t **ptlp);
@@ -4898,7 +4898,7 @@ static inline bool gup_can_follow_protnone(const struct vm_area_struct *vma,return !vma_is_accessible (vma); } -typedef int (*pte_fn_t )(pte_t *pte,unsigned long addr,void *data); +typedef int (*pte_fn_t )(hw_pte_t *pte,unsigned long addr,void *data);extern int apply_to_page_range (struct mm_struct *mm,unsigned long address,unsigned long size,pte_fn_t fn,void *data); extern int apply_to_existing_page_range(struct mm_struct *mm,
@@ -5148,7 +5148,7 @@ void *vmemmap_alloc_block(unsigned long size, int node);struct vmem_altmap ;void *vmemmap_alloc_block_buf (unsigned long size,int node,struct vmem_altmap *altmap); -void vmemmap_verify (pte_t *,int ,unsigned long ,unsigned long ); +void vmemmap_verify (hw_pte_t *,int ,unsigned long ,unsigned long );void vmemmap_set_pmd (pmd_t *pmd,void *p,int node,unsigned long addr,unsigned long next); int vmemmap_check_pmd(pmd_t *pmd, int node,
@@ -5516,7 +5516,7 @@ static inline bool snapshot_page_is_faithful(const struct page_snapshot *ps)void snapshot_page (struct page_snapshot *ps,const struct page *page); -void map_anon_folio_pte_nopf (struct folio *folio,pte_t *pte, +void map_anon_folio_pte_nopf (struct folio *folio,hw_pte_t *pte,struct vm_area_struct *vma,unsigned long addr,bool uffd_wp);
include/linux/page_table_check.h +7 -3
@@ -7,6 +7,8 @@#ifndef __LINUX_PAGE_TABLE_CHECK_H#define __LINUX_PAGE_TABLE_CHECK_H +#include <linux/pgtable_types.h> +#ifdef CONFIG_PAGE_TABLE_CHECK #include <linux/jump_label.h>
@@ -21,7 +23,7 @@ void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr,void __page_table_check_pud_clear (struct mm_struct *mm,unsigned long addr,pud_t pud);void __page_table_check_ptes_set (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte,unsigned int nr); +hw_pte_t *ptep,pte_t pte,unsigned int nr);void __page_table_check_pmds_set (struct mm_struct *mm,unsigned long addr,pmd_t *pmdp,pmd_t pmd,unsigned int nr); void __page_table_check_puds_set(struct mm_struct *mm, unsigned long addr,
@@ -74,7 +76,8 @@ static inline void page_table_check_pud_clear(struct mm_struct *mm, }static inline void page_table_check_ptes_set (struct mm_struct *mm, -unsigned long addr,pte_t *ptep, +unsigned long addr, +hw_pte_t *ptep,pte_t pte,unsigned int nr) {if (static_branch_likely (&page_table_check_disabled))
@@ -137,7 +140,8 @@ static inline void page_table_check_pud_clear(struct mm_struct *mm, }static inline void page_table_check_ptes_set (struct mm_struct *mm, -unsigned long addr,pte_t *ptep, +unsigned long addr, +hw_pte_t *ptep,pte_t pte,unsigned int nr) { }
include/linux/pagewalk.h +4 -4
@@ -38,7 +38,7 @@ enum page_walk_lock {* not trigger for any populated ranges. * @hugetlb_entry: if set, called for each hugetlb entry. This hook * function is called with the vma lock held, in order to -* protect against a concurrent freeing of the pte_t* or +* protect against a concurrent freeing of the hw_pte_t* or * the ptl. In some cases, the hook function needs to drop * and retake the vma lock in order to avoid deadlocks * while calling other functions. In such cases the hook
@@ -76,11 +76,11 @@ struct mm_walk_ops { unsigned long next, struct mm_walk *walk);int (*pmd_entry)(pmd_t *pmd,unsigned long addr,unsigned long next,struct mm_walk *walk); -int (*pte_entry)(pte_t *pte,unsigned long addr, +int (*pte_entry)(hw_pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk);int (*pte_hole)(unsigned long addr,unsigned long next,int depth,struct mm_walk *walk); -int (*hugetlb_entry)(pte_t *pte,unsigned long hmask, +int (*hugetlb_entry)(hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long next,struct mm_walk *walk); int (*test_walk)(unsigned long addr, unsigned long next,
@@ -173,7 +173,7 @@ struct folio_walk {struct page *page;enum folio_walk_level level;union { -pte_t *ptep; +hw_pte_t *ptep;pud_t *pudp;pmd_t *pmdp; };
include/linux/pgtable.h +42 -37
297 lines not on this page · raw message
include/linux/rmap.h +1 -1
@@ -868,7 +868,7 @@ struct page_vma_mapped_walk {struct vm_area_struct *vma;unsigned long address;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte;spinlock_t *ptl;unsigned int flags; bool pgoff_is_anon : 1;
include/linux/swapops.h +4 -2
@@ -216,7 +216,8 @@ static inline swp_entry_t make_migration_entry_dirty(swp_entry_t entry)extern void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address); -extern void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,pte_t *pte); +extern void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr, +hw_pte_t *pte);#else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry (pgoff_t offset) {
@@ -236,7 +237,8 @@ static inline swp_entry_t make_writable_migration_entry(pgoff_t offset)static inline void migration_entry_wait (struct mm_struct *mm,pmd_t *pmd,unsigned long address) { }static inline void migration_entry_wait_huge (struct vm_area_struct *vma, -unsigned long addr,pte_t *pte) { } +unsigned long addr, +hw_pte_t *pte) { }static inline swp_entry_t make_migration_entry_young (swp_entry_t entry) {
include/linux/vmalloc.h +2 -2
@@ -8,7 +8,7 @@#include <linux/init.h> #include <linux/list.h> #include <linux/llist.h> -#include <asm/page.h> /* pgprot_t */ +#include <linux/pgtable_types.h> /* pgprot_t, hw_pte_t */ #include <linux/rbtree.h> #include <linux/overflow.h>
@@ -120,7 +120,7 @@ static inline unsigned long arch_vmap_pte_range_map_size(unsigned long addr, uns#ifndef arch_vmap_pte_range_unmap_sizestatic inline unsigned long arch_vmap_pte_range_unmap_size (unsigned long addr, -pte_t *ptep) +hw_pte_t *ptep) {return PAGE_SIZE ; }
include/trace/events/xen.h +5 -5
@@ -138,10 +138,10 @@ TRACE_EVENT(xen_mc_extend_args,TRACE_DEFINE_SIZEOF (pteval_t); TRACE_EVENT(xen_mmu_set_pte, - TP_PROTO(pte_t *ptep, pte_t pteval), + TP_PROTO(hw_pte_t *ptep, pte_t pteval), TP_ARGS(ptep, pteval), TP_STRUCT__entry( - __field(pte_t *, ptep) + __field(hw_pte_t *, ptep) __field(pteval_t, pteval) ), TP_fast_assign(__entry->ptep = ptep;
@@ -207,12 +207,12 @@ TRACE_EVENT(xen_mmu_set_p4d, DECLARE_EVENT_CLASS(xen_mmu_ptep_modify_prot, TP_PROTO(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval), + hw_pte_t *ptep, pte_t pteval), TP_ARGS(mm, addr, ptep, pteval), TP_STRUCT__entry( __field(struct mm_struct *, mm) __field(unsigned long, addr) - __field(pte_t *, ptep) + __field(hw_pte_t *, ptep) __field(pteval_t, pteval) ), TP_fast_assign(__entry->mm = mm;
@@ -227,7 +227,7 @@ DECLARE_EVENT_CLASS(xen_mmu_ptep_modify_prot,#define DEFINE_XEN_MMU_PTEP_MODIFY_PROT (name) \ DEFINE_EVENT(xen_mmu_ptep_modify_prot, name, \ TP_PROTO(struct mm_struct *mm, unsigned long addr, \ - pte_t *ptep, pte_t pteval), \ + hw_pte_t *ptep, pte_t pteval), \ TP_ARGS(mm, addr, ptep, pteval))DEFINE_XEN_MMU_PTEP_MODIFY_PROT (xen_mmu_ptep_modify_prot_start);
kernel/bpf/arena.c +5 -4
@@ -155,7 +155,7 @@ struct clear_range_data {struct llist_head *free_pages; }; -static int apply_range_set_cb (pte_t *pte,unsigned long addr,void *data) +static int apply_range_set_cb (hw_pte_t *pte,unsigned long addr,void *data) {struct apply_range_data *d = data;struct page *page;
@@ -207,7 +207,7 @@ static void flush_vmap_cache(unsigned long start, unsigned long size)flush_cache_vmap (start, start + size); } -static int apply_range_clear_cb (pte_t *pte,unsigned long addr,void *data) +static int apply_range_clear_cb (hw_pte_t *pte,unsigned long addr,void *data) {struct clear_range_data *d = data;pte_t old_pte;
@@ -238,7 +238,8 @@ static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data)return 0 ; } -static int apply_range_set_scratch_cb (pte_t *pte,unsigned long addr,void *data) +static int apply_range_set_scratch_cb (hw_pte_t *pte,unsigned long addr, +void *data) {struct page *scratch_page = data;
@@ -340,7 +341,7 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr)return ERR_PTR (err); } -static int existing_page_cb (pte_t *ptep,unsigned long addr,void *data) +static int existing_page_cb (hw_pte_t *ptep,unsigned long addr,void *data) {struct bpf_arena *arena = data;struct page *page;
kernel/events/core.c +2 -1
@@ -8534,7 +8534,8 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr)p4d_t *p4dp, p4d;pud_t *pudp, pud;pmd_t *pmdp, pmd; -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte; pgdp =pgd_offset (mm, addr); pgd =pgdp_get (pgdp);
mm/damon/ops-common.c +1 -1
@@ -39,7 +39,7 @@ struct folio *damon_get_folio(unsigned long pfn)return folio; } -void damon_ptep_mkold (pte_t *pte,struct vm_area_struct *vma,unsigned long addr) +void damon_ptep_mkold (hw_pte_t *pte,struct vm_area_struct *vma,unsigned long addr) {pte_t pteval =ptep_get (pte);struct folio *folio;
mm/damon/ops-common.h +1 -1
@@ -7,7 +7,7 @@struct folio *damon_get_folio (unsigned long pfn); -void damon_ptep_mkold (pte_t *pte,struct vm_area_struct *vma,unsigned long addr); +void damon_ptep_mkold (hw_pte_t *pte,struct vm_area_struct *vma,unsigned long addr);void damon_pmdp_mkold (pmd_t *pmd,struct vm_area_struct *vma,unsigned long addr);void damon_folio_mkold (struct folio *folio);bool damon_folio_young (struct folio *folio);
mm/damon/vaddr.c +11 -9
@@ -268,7 +268,7 @@ static void damon_va_walk_page_range(struct mm_struct *mm, unsigned long start,static int damon_mkold_pmd_entry (pmd_t *pmd,unsigned long addr,unsigned long next,struct mm_walk *walk) { -pte_t *pte; +hw_pte_t *pte;spinlock_t *ptl; ptl =pmd_trans_huge_lock (pmd, walk->vma);
@@ -306,7 +306,7 @@ static bool damon_hugetlb_ptep_mkold(pte_t *pte, struct mm_struct *mm,return true; } -static void damon_hugetlb_mkold (pte_t *pte,struct mm_struct *mm, +static void damon_hugetlb_mkold (hw_pte_t *pte,struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr) {bool referenced = false;
@@ -327,7 +327,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm,folio_put (folio); } -static int damon_mkold_hugetlb_entry (pte_t *pte,unsigned long hmask, +static int damon_mkold_hugetlb_entry (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -396,7 +396,7 @@ struct damon_young_walk_private {static int damon_young_pmd_entry (pmd_t *pmd,unsigned long addr,unsigned long next,struct mm_walk *walk) { -pte_t *pte; +hw_pte_t *pte;pte_t ptent;spinlock_t *ptl;struct folio *folio;
@@ -440,7 +440,7 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, }#ifdef CONFIG_HUGETLB_PAGE -static int damon_young_hugetlb_entry (pte_t *pte,unsigned long hmask, +static int damon_young_hugetlb_entry (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
@@ -529,7 +529,7 @@ static unsigned int damon_va_check_accesses(struct damon_ctx *ctx)static bool damos_va_filter_young_match (struct damos_filter *filter,struct folio *folio,struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,pmd_t *pmdp) +unsigned long addr,hw_pte_t *ptep,pmd_t *pmdp) {bool young = false;
@@ -551,7 +551,7 @@ static bool damos_va_filter_young_match(struct damos_filter *filter,static bool damos_va_filter_out (struct damos *scheme,struct folio *folio,struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep,pmd_t *pmdp) +hw_pte_t *ptep,pmd_t *pmdp) {struct damos_filter *filter;bool matched;
@@ -648,7 +648,8 @@ static int damos_va_migrate_pmd_entry(pmd_t *pmd, unsigned long addr,struct damos_migrate_dests *dests = &s->migrate_dests;struct folio *folio;spinlock_t *ptl; -pte_t *start_pte, *pte, ptent; +hw_pte_t *start_pte, *pte; +pte_t ptent;int nr;#ifdef CONFIG_TRANSPARENT_HUGEPAGE
@@ -808,7 +809,8 @@ static int damos_va_stat_pmd_entry(pmd_t *pmd, unsigned long addr,struct vm_area_struct *vma = walk->vma;struct folio *folio;spinlock_t *ptl; -pte_t *start_pte, *pte, ptent; +hw_pte_t *start_pte, *pte; +pte_t ptent;int nr;#ifdef CONFIG_TRANSPARENT_HUGEPAGE
mm/debug_vm_pgtable.c +1 -1
@@ -50,7 +50,7 @@ struct pgtable_debug_args {p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp; -pte_t *ptep; +hw_pte_t *ptep;p4d_t *start_p4dp;pud_t *start_pudp;
mm/filemap.c +2 -2
@@ -3490,7 +3490,7 @@ static vm_fault_t filemap_fault_recheck_pte_none(struct vm_fault *vmf) {struct vm_area_struct *vma = vmf->vma;vm_fault_t ret =0 ; -pte_t *ptep; +hw_pte_t *ptep;/* * We might have COW'ed a pagecache folio and might now have an mlocked
@@ -3796,7 +3796,7 @@ static vm_fault_t filemap_map_folio_range(struct vm_fault *vmf,vm_fault_t ret =0 ;struct page *page =folio_page (folio, start);unsigned int count =0 ; -pte_t *old_ptep = vmf->pte; +hw_pte_t *old_ptep = vmf->pte;unsigned long addr0;/*
mm/gup.c +5 -4
@@ -761,7 +761,7 @@ static struct page *follow_huge_pmd(struct vm_area_struct *vma,#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ static int follow_pfn_pte (struct vm_area_struct *vma,unsigned long address, -pte_t *pte,unsigned int flags) +hw_pte_t *pte,unsigned int flags) {if (flags &FOLL_TOUCH ) {pte_t orig_entry =ptep_get (pte);
@@ -806,7 +806,8 @@ static struct page *follow_page_pte(struct vm_area_struct *vma,struct folio *folio;struct page *page;spinlock_t *ptl; -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte;int ret; ptep =pte_offset_map_lock (mm, pmd, address, &ptl);
@@ -1035,7 +1036,7 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address,p4d_t *p4d;pud_t *pud;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte;pte_t entry;int ret = -EFAULT ;
@@ -2831,7 +2832,7 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, int *nr) {int ret =0 ; -pte_t *ptep, *ptem; +hw_pte_t *ptep, *ptem; ptem = ptep =pte_offset_map (&pmd, addr);if (!ptep)
mm/highmem.c +8 -7
@@ -141,7 +141,7 @@ EXPORT_SYMBOL(__totalhigh_pages);static int pkmap_count[LAST_PKMAP ];static __cacheline_aligned_in_smp DEFINE_SPINLOCK (kmap_lock ); -pte_t *pkmap_page_table; +hw_pte_t *pkmap_page_table;/* * Most architectures have no use for kmap_high_get(), so let's abstract
@@ -532,9 +532,9 @@ static inline bool kmap_high_unmap_local(unsigned long vaddr)return false; } -static pte_t *__kmap_pte; +static hw_pte_t *__kmap_pte; -static pte_t *kmap_get_pte (unsigned long vaddr,int idx) +static hw_pte_t *kmap_get_pte (unsigned long vaddr,int idx) {if (IS_ENABLED (CONFIG_KMAP_LOCAL_NON_LINEAR_PTE_ARRAY ))/*
@@ -549,8 +549,9 @@ static pte_t *kmap_get_pte(unsigned long vaddr, int idx)void *__kmap_local_pfn_prot (unsigned long pfn,pgprot_t prot) { -pte_t pteval, *kmap_pte;unsigned long vaddr; +hw_pte_t *kmap_pte; +pte_t pteval;int idx;/*
@@ -597,7 +598,7 @@ EXPORT_SYMBOL(__kmap_local_page_prot);void kunmap_local_indexed (const void *vaddr) {unsigned long addr = (unsigned long ) vaddr &PAGE_MASK ; -pte_t *kmap_pte; +hw_pte_t *kmap_pte;int idx; if (addr < __fix_to_virt(FIX_KMAP_END) ||
@@ -646,7 +647,7 @@ EXPORT_SYMBOL(kunmap_local_indexed);void __kmap_local_sched_out (void ) {struct task_struct *tsk = current; -pte_t *kmap_pte; +hw_pte_t *kmap_pte;int i;/* Clear kmaps */
@@ -683,7 +684,7 @@ void __kmap_local_sched_out(void)void __kmap_local_sched_in (void ) {struct task_struct *tsk = current; -pte_t *kmap_pte; +hw_pte_t *kmap_pte;int i;/* Restore kmaps */
mm/hmm.c +3 -3
@@ -240,7 +240,7 @@ static inline unsigned long pte_to_hmm_pfn_flags(struct hmm_range *range, }static int hmm_vma_handle_pte (struct mm_walk *walk,unsigned long addr, -unsigned long end,pmd_t *pmdp,pte_t *ptep, +unsigned long end,pmd_t *pmdp,hw_pte_t *ptep,unsigned long *hmm_pfn) {struct hmm_vma_walk *hmm_vma_walk = walk->private;
@@ -411,7 +411,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, &range->hmm_pfns[(start - range->start) >>PAGE_SHIFT ];unsigned long npages = (end - start) >>PAGE_SHIFT ;unsigned long addr = start; -pte_t *ptep; +hw_pte_t *ptep;pmd_t pmd; again:
@@ -547,7 +547,7 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end,#endif #ifdef CONFIG_HUGETLB_PAGE -static int hmm_vma_walk_hugetlb_entry (pte_t *pte,unsigned long hmask, +static int hmm_vma_walk_hugetlb_entry (hw_pte_t *pte,unsigned long hmask,unsigned long start,unsigned long end,struct mm_walk *walk) {
mm/huge_memory.c +2 -2
@@ -3142,7 +3142,7 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma,pgtable_t pgtable;pmd_t _pmd, old_pmd;unsigned long addr; -pte_t *pte; +hw_pte_t *pte;int i;/*
@@ -3192,7 +3192,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,bool soft_dirty, uffd_wp = false, young = false, write = false;bool anon_exclusive = false, dirty = false;unsigned long addr; -pte_t *pte; +hw_pte_t *pte;int i;VM_BUG_ON (haddr & ~HPAGE_PMD_MASK );
mm/hugetlb.c +31 -29
225 lines not on this page · raw message
mm/hugetlb_vmemmap.c +7 -6
@@ -33,7 +33,7 @@* operations. */ struct vmemmap_remap_walk { -void (*remap_pte)(pte_t *pte,unsigned long addr, +void (*remap_pte)(hw_pte_t *pte,unsigned long addr,struct vmemmap_remap_walk *walk);unsigned long nr_walked;
@@ -55,7 +55,7 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,pmd_t __pmd;int i;unsigned long addr = start; -pte_t *pgtable; +hw_pte_t *pgtable; pgtable =pte_alloc_one_kernel (&init_mm);if (!pgtable)
@@ -64,7 +64,8 @@ static int vmemmap_split_pmd(pmd_t *pmd, struct page *head, unsigned long start,pmd_populate_kernel (&init_mm, &__pmd, pgtable);for (i =0 ; i <PTRS_PER_PTE ; i++, addr +=PAGE_SIZE ) { -pte_t entry, *pte; +pte_t entry; +hw_pte_t *pte;pgprot_t pgprot =PAGE_KERNEL ; entry =mk_pte (head + i, pgprot);
@@ -136,7 +137,7 @@ static int vmemmap_pmd_entry(pmd_t *pmd, unsigned long addr,return vmemmap_split_pmd (pmd, head, addr &PMD_MASK , vmemmap_walk); } -static int vmemmap_pte_entry (pte_t *pte,unsigned long addr, +static int vmemmap_pte_entry (hw_pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk) {struct vmemmap_remap_walk *vmemmap_walk = walk->private;
@@ -198,7 +199,7 @@ static void free_vmemmap_page_list(struct list_head *list)free_vmemmap_page (page); } -static void vmemmap_remap_pte (pte_t *pte,unsigned long addr, +static void vmemmap_remap_pte (hw_pte_t *pte,unsigned long addr,struct vmemmap_remap_walk *walk) {struct page *page =pte_page (ptep_get (pte));
@@ -232,7 +233,7 @@ static void vmemmap_remap_pte(pte_t *pte, unsigned long addr,set_pte_at (&init_mm, addr, pte, entry); } -static void vmemmap_restore_pte (pte_t *pte,unsigned long addr, +static void vmemmap_restore_pte (hw_pte_t *pte,unsigned long addr,struct vmemmap_remap_walk *walk) {struct page *src =pte_page (ptep_get (pte)), *dst;
mm/internal.h +8 -8
@@ -274,7 +274,7 @@ void unmap_vmas(struct mmu_gather *tlb, struct unmap_desc *unmap);#ifdef CONFIG_MMU bool cond_install_uffd_wp_ptes (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,pte_t pte, +unsigned long addr,hw_pte_t *ptep,pte_t pte,unsigned long nr_ptes);static inline void get_anon_vma (struct anon_vma *anon_vma)
@@ -410,7 +410,7 @@ static inline pte_t __pte_batch_clear_ignored(pte_t pte, fpb_t flags)* Return: the number of table entries in the batch. */ static inline unsigned int folio_pte_batch_flags (struct folio *folio, -struct vm_area_struct *vma,pte_t *ptep,pte_t *ptentp, +struct vm_area_struct *vma,hw_pte_t *ptep,pte_t *ptentp,unsigned int max_nr,fpb_t flags) {bool any_writable = false, any_young = false, any_dirty = false;
@@ -464,7 +464,7 @@ static inline unsigned int folio_pte_batch_flags(struct folio *folio,return min (nr, max_nr); } -unsigned int folio_pte_batch (struct folio *folio,pte_t *ptep,pte_t pte, +unsigned int folio_pte_batch (struct folio *folio,hw_pte_t *ptep,pte_t pte,unsigned int max_nr);/**
@@ -521,11 +521,11 @@ static inline pte_t pte_next_swp_offset(pte_t pte)* * Return: the number of table entries in the batch. */ -static inline int swap_pte_batch (pte_t *start_ptep,int max_nr,pte_t pte) +static inline int swap_pte_batch (hw_pte_t *start_ptep,int max_nr,pte_t pte) {pte_t expected_pte =pte_next_swp_offset (pte); -const pte_t *end_ptep = start_ptep + max_nr; -pte_t *ptep = start_ptep +1 ; +const hw_pte_t *end_ptep = start_ptep + max_nr; +hw_pte_t *ptep = start_ptep +1 ;VM_WARN_ON (max_nr <1 );VM_WARN_ON (!softleaf_is_swap (softleaf_from_pte (pte)));
@@ -1563,7 +1563,7 @@ static inline void maybe_rmap_unlock_action(struct vm_area_struct *vma,#ifdef CONFIG_MMU_NOTIFIER static inline bool clear_flush_young_ptes_notify (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,unsigned int nr) +unsigned long addr,hw_pte_t *ptep,unsigned int nr) {bool young;
@@ -1584,7 +1584,7 @@ static inline bool pmdp_clear_flush_young_notify(struct vm_area_struct *vma, }static inline bool test_and_clear_young_ptes_notify (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,unsigned int nr) +unsigned long addr,hw_pte_t *ptep,unsigned int nr) {bool young;
mm/kasan/init.c +6 -6
@@ -92,7 +92,7 @@ static __init void *early_alloc(size_t size, int node)static void __ref zero_pte_populate (pmd_t *pmd,unsigned long addr,unsigned long end) { -pte_t *pte =pte_offset_kernel (pmd, addr); +hw_pte_t *pte =pte_offset_kernel (pmd, addr);pte_t zero_pte; zero_pte = pfn_pte(PFN_DOWN(__pa_symbol(kasan_early_shadow_page)),
@@ -122,7 +122,7 @@ static int __ref zero_pmd_populate(pud_t *pud, unsigned long addr, }if (pmd_none (*pmd)) { -pte_t *p; +hw_pte_t *p;if (slab_is_available ()) p =pte_alloc_one_kernel (&init_mm);
@@ -281,9 +281,9 @@ int __ref kasan_populate_early_shadow(const void *shadow_start,return 0 ; } -static void kasan_free_pte (pte_t *pte_start,pmd_t *pmd) +static void kasan_free_pte (hw_pte_t *pte_start,pmd_t *pmd) { -pte_t *pte; +hw_pte_t *pte;int i;for (i =0 ; i <PTRS_PER_PTE ; i++) {
@@ -341,7 +341,7 @@ static void kasan_free_p4d(p4d_t *p4d_start, pgd_t *pgd)pgd_clear (pgd); } -static void kasan_remove_pte_table (pte_t *pte,unsigned long addr, +static void kasan_remove_pte_table (hw_pte_t *pte,unsigned long addr,unsigned long end) {unsigned long next;
@@ -369,7 +369,7 @@ static void kasan_remove_pmd_table(pmd_t *pmd, unsigned long addr,unsigned long next;for (; addr < end; addr = next, pmd++) { -pte_t *pte; +hw_pte_t *pte; next =pmd_addr_end (addr, end);
mm/kasan/shadow.c +3 -3
@@ -189,7 +189,7 @@ static bool shadow_mapped(unsigned long addr)p4d_t *p4d;pud_t *pud;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte;if (pgd_none (*pgd))return false;
@@ -297,7 +297,7 @@ struct vmalloc_populate_data {struct page **pages; }; -static int kasan_populate_vmalloc_pte (pte_t *ptep,unsigned long addr, +static int kasan_populate_vmalloc_pte (hw_pte_t *ptep,unsigned long addr,void *_data) {struct vmalloc_populate_data *data = _data;
@@ -465,7 +465,7 @@ int __kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mreturn 0 ; } -static int kasan_depopulate_vmalloc_pte (pte_t *ptep,unsigned long addr, +static int kasan_depopulate_vmalloc_pte (hw_pte_t *ptep,unsigned long addr,void *unused) {pte_t pte;
mm/khugepaged.c +30 -20
115 lines not on this page · raw message
mm/ksm.c +4 -3
@@ -618,7 +618,7 @@ static int break_ksm_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned long en {unsigned long *found_addr = (unsigned long *) walk->private;struct mm_struct *mm = walk->mm; -pte_t *start_ptep, *ptep; +hw_pte_t *start_ptep, *ptep;spinlock_t *ptl;int found =0 ;
@@ -1398,7 +1398,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page,struct folio *folio =page_folio (page);pmd_t *pmd;pmd_t pmde; -pte_t *ptep; +hw_pte_t *ptep;pte_t newpte;spinlock_t *ptl;unsigned long addr;
@@ -2529,7 +2529,8 @@ static int ksm_next_page_pmd_entry(pmd_t *pmdp, unsigned long addr, unsigned lon {struct ksm_next_page_arg *private = walk->private;struct vm_area_struct *vma = walk->vma; -pte_t *start_ptep =NULL , *ptep, pte; +hw_pte_t *start_ptep =NULL , *ptep; +pte_t pte;struct mm_struct *mm = walk->mm;struct folio *folio;struct page *page;
mm/madvise.c +8 -6
@@ -198,7 +198,7 @@ static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, {struct vm_area_struct *vma = walk->private;struct swap_io_ctx ctx = {}; -pte_t *ptep =NULL ; +hw_pte_t *ptep =NULL ;spinlock_t *ptl;unsigned long addr;
@@ -350,7 +350,7 @@ static inline bool can_do_file_pageout(struct vm_area_struct *vma) }static inline int madvise_folio_pte_batch (unsigned long addr,unsigned long end, -struct folio *folio,pte_t *ptep, +struct folio *folio,hw_pte_t *ptep,pte_t *ptentp) {int max_nr = (end - addr) /PAGE_SIZE ;
@@ -368,7 +368,8 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,bool pageout = private->pageout;struct mm_struct *mm = tlb->mm;struct vm_area_struct *vma = walk->vma; -pte_t *start_pte, *pte, ptent; +hw_pte_t *start_pte, *pte; +pte_t ptent;spinlock_t *ptl;struct folio *folio =NULL ;LIST_HEAD (folio_list);
@@ -667,7 +668,8 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,struct mm_struct *mm = tlb->mm;struct vm_area_struct *vma = walk->vma;spinlock_t *ptl; -pte_t *start_pte, *pte, ptent; +hw_pte_t *start_pte, *pte; +pte_t ptent;struct folio *folio;int nr_swap =0 ;unsigned long next;
@@ -1092,7 +1094,7 @@ static int guard_install_pmd_entry(pmd_t *pmd, unsigned long addr,return pmd_trans_huge (pmdval); } -static int guard_install_pte_entry (pte_t *pte,unsigned long addr, +static int guard_install_pte_entry (hw_pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk) {pte_t pteval =ptep_get (pte);
@@ -1235,7 +1237,7 @@ static int guard_remove_pmd_entry(pmd_t *pmd, unsigned long addr,return 0 ; } -static int guard_remove_pte_entry (pte_t *pte,unsigned long addr, +static int guard_remove_pte_entry (hw_pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk) {pte_t ptent =ptep_get (pte);
mm/mapping_dirty_helpers.c +2 -2
@@ -31,7 +31,7 @@ struct wp_walk {* The function write-protects a pte and records the range in * virtual address space of touched ptes for efficient range TLB flushes. */ -static int wp_pte (pte_t *pte,unsigned long addr,unsigned long end, +static int wp_pte (hw_pte_t *pte,unsigned long addr,unsigned long end,struct mm_walk *walk) {struct wp_walk *wpwalk = walk->private;
@@ -86,7 +86,7 @@ struct clean_walk {* in the address_space, as well as the first and last of the bits * touched. */ -static int clean_record_pte (pte_t *pte,unsigned long addr, +static int clean_record_pte (hw_pte_t *pte,unsigned long addr,unsigned long end,struct mm_walk *walk) {struct wp_walk *wpwalk = walk->private;
mm/memory-failure.c +3 -3
@@ -342,7 +342,7 @@ static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma,p4d_t *p4d;pud_t *pud;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte;pte_t ptent;VM_BUG_ON_VMA (address == -EFAULT , vma);
@@ -742,7 +742,7 @@ static int hwpoison_pte_range(pmd_t *pmdp, unsigned long addr, {struct hwpoison_walk *hwp = walk->private;int ret =0 ; -pte_t *ptep, *mapped_pte; +hw_pte_t *ptep, *mapped_pte;spinlock_t *ptl; ptl =pmd_trans_huge_lock (pmdp, walk->vma);
@@ -770,7 +770,7 @@ static int hwpoison_pte_range(pmd_t *pmdp, unsigned long addr, }#ifdef CONFIG_HUGETLB_PAGE -static int hwpoison_hugetlb_range (pte_t *ptep,unsigned long hmask, +static int hwpoison_hugetlb_range (hw_pte_t *ptep,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
mm/memory.c +40 -38
294 lines not on this page · raw message
mm/mempolicy.c +2 -2
@@ -691,7 +691,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,struct folio *folio;struct queue_pages *qp = walk->private;unsigned long flags = qp->flags; -pte_t *pte, *mapped_pte; +hw_pte_t *pte, *mapped_pte;pte_t ptent;spinlock_t *ptl;int max_nr, nr;
@@ -771,7 +771,7 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr,return 0 ; } -static int queue_folios_hugetlb (pte_t *pte,unsigned long hmask, +static int queue_folios_hugetlb (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {
mm/migrate.c +2 -2
@@ -494,7 +494,7 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address) {spinlock_t *ptl; -pte_t *ptep; +hw_pte_t *ptep;pte_t pte;softleaf_t entry;
@@ -525,7 +525,7 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd,* * This function will release the vma lock before returning. */ -void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,pte_t *ptep) +void migration_entry_wait_huge (struct vm_area_struct *vma,unsigned long addr,hw_pte_t *ptep) {spinlock_t *ptl =huge_pte_lockptr (hstate_vma (vma), vma->vm_mm, ptep);softleaf_t entry;
mm/migrate_device.c +2 -2
@@ -254,7 +254,7 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp,spinlock_t *ptl;struct folio *fault_folio = migrate->fault_page ?page_folio (migrate->fault_page) :NULL ; -pte_t *ptep; +hw_pte_t *ptep; again:if (pmd_trans_huge (*pmdp) || !pmd_present (*pmdp)) {
@@ -989,7 +989,7 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate,p4d_t *p4dp;pud_t *pudp;pmd_t *pmdp; -pte_t *ptep; +hw_pte_t *ptep;pte_t orig_pte;/* Only allow populating anonymous memory */
mm/mincore.c +2 -2
@@ -24,7 +24,7 @@#include "swap.h" #include "internal.h" -static int mincore_hugetlb (pte_t *pte,unsigned long hmask,unsigned long addr, +static int mincore_hugetlb (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long end,struct mm_walk *walk) {#ifdef CONFIG_HUGETLB_PAGE
@@ -164,7 +164,7 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, {spinlock_t *ptl;struct vm_area_struct *vma = walk->vma; -pte_t *ptep; +hw_pte_t *ptep;unsigned char *vec = walk->private;int nr = (end - addr) >>PAGE_SHIFT ;int step, i;
mm/mlock.c +2 -2
@@ -305,7 +305,7 @@ void munlock_folio(struct folio *folio) }static inline unsigned int folio_mlock_step (struct folio *folio, -pte_t *pte,unsigned long addr,unsigned long end) +hw_pte_t *pte,unsigned long addr,unsigned long end) {unsigned int count = (end - addr) >>PAGE_SHIFT ;pte_t ptent =ptep_get (pte);
@@ -353,7 +353,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, {struct vm_area_struct *vma = walk->vma;spinlock_t *ptl; -pte_t *start_pte, *pte; +hw_pte_t *start_pte, *pte;pte_t ptent;struct folio *folio;unsigned int step =1 ;
mm/mprotect.c +10 -9
@@ -103,7 +103,7 @@ bool can_change_pte_writable(struct vm_area_struct *vma, unsigned long addr,return can_change_shared_pte_writable (vma, pte); } -static int mprotect_folio_pte_batch (struct folio *folio,pte_t *ptep, +static int mprotect_folio_pte_batch (struct folio *folio,hw_pte_t *ptep,pte_t pte,int max_nr_ptes,fpb_t flags) {/* No underlying folio, so cannot batch */
@@ -118,7 +118,7 @@ static int mprotect_folio_pte_batch(struct folio *folio, pte_t *ptep,/* Set nr_ptes number of ptes, starting from idx */ static __always_inline void prot_commit_flush_ptes(struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep,pte_t oldpte,pte_t ptent, +unsigned long addr,hw_pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,int idx,bool set_write,struct mmu_gather *tlb) {/*
@@ -170,7 +170,7 @@ static __always_inline int page_anon_exclusive_batch(int start_idx, int max_len,* retrieve sub-batches. */ static __always_inline void commit_anon_folio_batch(struct vm_area_struct *vma, -struct folio *folio,struct page *first_page,unsigned long addr,pte_t *ptep, +struct folio *folio,struct page *first_page,unsigned long addr,hw_pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) {bool expected_anon_exclusive;
@@ -189,7 +189,7 @@ static __always_inline void commit_anon_folio_batch(struct vm_area_struct *vma, } static __always_inline void set_write_prot_commit_flush_ptes(struct vm_area_struct *vma, -struct folio *folio,struct page *page,unsigned long addr,pte_t *ptep, +struct folio *folio,struct page *page,unsigned long addr,hw_pte_t *ptep,pte_t oldpte,pte_t ptent,int nr_ptes,struct mmu_gather *tlb) {bool set_write;
@@ -212,7 +212,7 @@ static __always_inline void set_write_prot_commit_flush_ptes(struct vm_area_stru }static long change_softleaf_pte (struct vm_area_struct *vma, -unsigned long addr,pte_t *pte,pte_t oldpte,unsigned long cp_flags) +unsigned long addr,hw_pte_t *pte,pte_t oldpte,unsigned long cp_flags) {const bool uffd_prot = cp_flags & (MM_CP_UFFD_WP |MM_CP_UFFD_RWP ); const bool uffd_prot_resolve = cp_flags &
@@ -279,7 +279,7 @@ static long change_softleaf_pte(struct vm_area_struct *vma, } static __always_inline void change_present_ptes(struct mmu_gather *tlb, -struct vm_area_struct *vma,unsigned long addr,pte_t *ptep, +struct vm_area_struct *vma,unsigned long addr,hw_pte_t *ptep,int nr_ptes,unsigned long end,pgprot_t newprot,struct folio *folio,struct page *page,unsigned long cp_flags) {
@@ -332,7 +332,8 @@ static long change_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { - pte_t *pte, oldpte; + hw_pte_t *pte; +pte_t oldpte;spinlock_t *ptl;long pages =0 ;bool is_private_single_threaded;
@@ -727,7 +728,7 @@ long change_protection(struct mmu_gather *tlb,return pages; } -static int prot_none_pte_entry (pte_t *pte,unsigned long addr, +static int prot_none_pte_entry (hw_pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk) { return pfn_modify_allowed(pte_pfn(ptep_get(pte)),
@@ -736,7 +737,7 @@ static int prot_none_pte_entry(pte_t *pte, unsigned long addr, }#ifdef CONFIG_HUGETLB_PAGE -static int prot_none_hugetlb_entry (pte_t *pte,unsigned long hmask, +static int prot_none_hugetlb_entry (hw_pte_t *pte,unsigned long hmask,unsigned long addr,unsigned long next,struct mm_walk *walk) {
mm/mremap.c +2 -2
@@ -176,7 +176,7 @@ static pte_t move_soft_dirty_pte(pte_t pte) }static int mremap_folio_pte_batch (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep,pte_t pte,int max_nr) +hw_pte_t *ptep,pte_t pte,int max_nr) {struct folio *folio;
@@ -200,7 +200,7 @@ static int move_ptes(struct pagetable_move_control *pmc,struct vm_area_struct *vma = pmc->old;bool need_clear_uffd_wp =vma_has_uffd_without_event_remap (vma);struct mm_struct *mm = vma->vm_mm; -pte_t *old_ptep, *new_ptep; +hw_pte_t *old_ptep, *new_ptep;pte_t old_pte, pte;pmd_t dummy_pmdval;spinlock_t *old_ptl, *new_ptl;
mm/page_table_check.c +2 -2
@@ -208,7 +208,7 @@ static void page_table_check_pte_flags(pte_t pte) }void __page_table_check_ptes_set (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte,unsigned int nr) +hw_pte_t *ptep,pte_t pte,unsigned int nr) {unsigned int i;
@@ -279,7 +279,7 @@ void __page_table_check_pte_clear_range(struct mm_struct *mm,return ;if (!pmd_bad (pmd) && !pmd_leaf (pmd)) { -pte_t *ptep =pte_offset_map (&pmd, addr); +hw_pte_t *ptep =pte_offset_map (&pmd, addr);unsigned long i;if (WARN_ON (!ptep))
mm/pagewalk.c +5 -4
@@ -26,7 +26,7 @@ static int real_depth(int depth)return depth; } -static int walk_pte_range_inner (pte_t *pte,unsigned long addr, +static int walk_pte_range_inner (hw_pte_t *pte,unsigned long addr,unsigned long end,struct mm_walk *walk) {const struct mm_walk_ops *ops = walk->ops;
@@ -61,7 +61,7 @@ static int walk_pte_range_inner(pte_t *pte, unsigned long addr,static int walk_pte_range (pmd_t *pmd,unsigned long addr,unsigned long end,struct mm_walk *walk) { -pte_t *pte; +hw_pte_t *pte;int err =0 ;spinlock_t *ptl;
@@ -341,7 +341,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end,unsigned long next;unsigned long hmask =huge_page_mask (h);unsigned long sz =huge_page_size (h); -pte_t *pte; +hw_pte_t *pte;const struct mm_walk_ops *ops = walk->ops;int err =0 ;
@@ -907,7 +907,8 @@ struct folio *folio_walk_start(struct folio_walk *fw,struct page *page;pud_t *pudp, pud;pmd_t *pmdp, pmd; -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte;spinlock_t *ptl;pgd_t *pgdp;p4d_t *p4dp;
mm/percpu.c +1 -1
@@ -3176,7 +3176,7 @@ void __init __weak pcpu_populate_pte(unsigned long addr) pmd =pmd_offset (pud, addr);if (!pmd_present (*pmd)) { -pte_t *new; +hw_pte_t *new; new =memblock_alloc_or_panic (PTE_TABLE_SIZE ,PTE_TABLE_SIZE );pmd_populate_kernel (&init_mm, pmd, new);
mm/pgtable-generic.c +10 -10
@@ -68,7 +68,7 @@ void pmd_clear_bad(pmd_t *pmd)* force that call on sun4c so we changed this macro slightly */ int ptep_set_access_flags (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep, +unsigned long address,hw_pte_t *ptep,pte_t entry,int dirty) {int changed = !pte_same (ptep_get (ptep), entry);
@@ -82,7 +82,7 @@ int ptep_set_access_flags(struct vm_area_struct *vma,#ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSHbool ptep_clear_flush_young (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) +unsigned long address,hw_pte_t *ptep) {bool young;
@@ -95,7 +95,7 @@ bool ptep_clear_flush_young(struct vm_area_struct *vma,#ifndef __HAVE_ARCH_PTEP_CLEAR_FLUSHpte_t ptep_clear_flush (struct vm_area_struct *vma,unsigned long address, -pte_t *ptep) +hw_pte_t *ptep) {struct mm_struct *mm = (vma)->vm_mm;pte_t pte;
@@ -282,7 +282,7 @@ static unsigned long pmdp_get_lockless_start(void) { return 0; }static void pmdp_get_lockless_end (unsigned long irqflags) { }#endif -pte_t *__pte_offset_map (pmd_t *pmd,unsigned long addr,pmd_t *pmdvalp) +hw_pte_t *__pte_offset_map (pmd_t *pmd,unsigned long addr,pmd_t *pmdvalp) {unsigned long irqflags;pmd_t pmdval;
@@ -308,11 +308,11 @@ pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp)return NULL ; } -pte_t *pte_offset_map_ro_nolock (struct mm_struct *mm,pmd_t *pmd, +hw_pte_t *pte_offset_map_ro_nolock (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,spinlock_t **ptlp) {pmd_t pmdval; -pte_t *pte; +hw_pte_t *pte; pte =__pte_offset_map (pmd, addr, &pmdval);if (likely (pte))
@@ -320,11 +320,11 @@ pte_t *pte_offset_map_ro_nolock(struct mm_struct *mm, pmd_t *pmd,return pte; } -pte_t *pte_offset_map_rw_nolock (struct mm_struct *mm,pmd_t *pmd, +hw_pte_t *pte_offset_map_rw_nolock (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,pmd_t *pmdvalp,spinlock_t **ptlp) { -pte_t *pte; +hw_pte_t *pte;VM_WARN_ON_ONCE (!pmdvalp); pte =__pte_offset_map (pmd, addr, pmdvalp);
@@ -390,12 +390,12 @@ pte_t *pte_offset_map_rw_nolock(struct mm_struct *mm, pmd_t *pmd,* table, and may not use RCU at all: "outsiders" like khugepaged should avoid * pte_offset_map() and co once the vma is detached from mm or mm_users is zero. */ -pte_t *pte_offset_map_lock (struct mm_struct *mm,pmd_t *pmd, +hw_pte_t *pte_offset_map_lock (struct mm_struct *mm,pmd_t *pmd,unsigned long addr,spinlock_t **ptlp) {spinlock_t *ptl;pmd_t pmdval; -pte_t *pte; +hw_pte_t *pte; again: pte =__pte_offset_map (pmd, addr, &pmdval);if (unlikely (!pte))
mm/ptdump.c +1 -1
@@ -117,7 +117,7 @@ static int ptdump_pmd_entry(pmd_t *pmd, unsigned long addr,return 0 ; } -static int ptdump_pte_entry (pte_t *pte,unsigned long addr, +static int ptdump_pte_entry (hw_pte_t *pte,unsigned long addr,unsigned long next,struct mm_walk *walk) {struct ptdump_state *st = walk->private;
mm/rmap.c +3 -3
@@ -1128,7 +1128,7 @@ static int page_vma_mkclean_one(struct page_vma_mapped_walk *pvmw) address = pvmw->address;if (pvmw->pte) { -pte_t *pte = pvmw->pte; +hw_pte_t *pte = pvmw->pte;pte_t entry =ptep_get (pte);/*
@@ -2145,7 +2145,7 @@ static pte_t swp_pte_prepare(swp_entry_t entry, pte_t old_pte,static bool ttu_anon_swapbacked_folio (struct vm_area_struct *vma,struct folio *folio,struct page *page,unsigned long address, -pte_t *ptep,pte_t pteval) +hw_pte_t *ptep,pte_t pteval) {const bool anon_exclusive =folio_test_anon (folio) &&PageAnonExclusive (page);
@@ -2180,7 +2180,7 @@ static bool ttu_anon_swapbacked_folio(struct vm_area_struct *vma, }static bool ttu_anon_folio (struct vm_area_struct *vma,struct folio *folio, -struct page *page,unsigned long address,pte_t *ptep, +struct page *page,unsigned long address,hw_pte_t *ptep,pte_t pteval,unsigned long nr_pages) {/*
mm/sparse-vmemmap.c +11 -11
@@ -137,7 +137,7 @@ static void * __meminit altmap_alloc_block_buf(unsigned long size,return __va (__pfn_to_phys (pfn)); } -void __meminit vmemmap_verify (pte_t *pte,int node, +void __meminit vmemmap_verify (hw_pte_t *pte,int node,unsigned long start,unsigned long end) {unsigned long pfn =pte_pfn (ptep_get (pte));
@@ -148,11 +148,11 @@ void __meminit vmemmap_verify(pte_t *pte, int node, start, end - 1); } -static pte_t *__meminit vmemmap_pte_populate (pmd_t *pmd,unsigned long addr,int node, +static hw_pte_t *__meminit vmemmap_pte_populate (pmd_t *pmd,unsigned long addr,int node,struct vmem_altmap *altmap,unsigned long ptpfn,unsigned long flags) { -pte_t *pte =pte_offset_kernel (pmd, addr); +hw_pte_t *pte =pte_offset_kernel (pmd, addr);if (pte_none (ptep_get (pte))) {pte_t entry;void *p;
@@ -243,7 +243,7 @@ static pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node)return pgd; } -static pte_t *__meminit vmemmap_populate_address (unsigned long addr,int node, +static hw_pte_t *__meminit vmemmap_populate_address (unsigned long addr,int node,struct vmem_altmap *altmap,unsigned long ptpfn,unsigned long flags)
@@ -252,7 +252,7 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node,p4d_t *p4d;pud_t *pud;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte; pgd =vmemmap_pgd_populate (addr, node);if (!pgd)
@@ -281,7 +281,7 @@ static int __meminit vmemmap_populate_range(unsigned long start, unsigned long flags) {unsigned long addr = start; -pte_t *pte; +hw_pte_t *pte;for (; addr < end; addr +=PAGE_SIZE ) { pte = vmemmap_populate_address(addr, node, altmap,
@@ -314,7 +314,7 @@ void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, int node, unsigned long headsize) {unsigned long maddr; -pte_t *pte; +hw_pte_t *pte;for (maddr = addr + headsize; maddr < end; maddr +=PAGE_SIZE ) { pte =virt_to_kpte (maddr);
@@ -396,7 +396,7 @@ int __weak __meminit vmemmap_check_pmd(pmd_t *pmd, int node, {if (!pmd_leaf (pmdp_get (pmd)))return 0 ; -vmemmap_verify ((pte_t *)pmd, node, addr, next); +vmemmap_verify ((hw_pte_t *)pmd, node, addr, next);return 1 ; }
@@ -474,9 +474,9 @@ static bool __meminit reuse_compound_section(unsigned long start_pfn,return !IS_ALIGNED (offset, nr_pages) && nr_pages >PAGES_PER_SUBSECTION ; } -static pte_t *__meminit compound_section_tail_page (unsigned long addr) +static hw_pte_t *__meminit compound_section_tail_page (unsigned long addr) { -pte_t *pte; +hw_pte_t *pte; addr -=PAGE_SIZE ;
@@ -497,7 +497,7 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, struct dev_pagemap *pgmap) {unsigned long size, addr; -pte_t *pte; +hw_pte_t *pte;int rc;if (reuse_compound_section (start_pfn, pgmap)) {
mm/swap_state.c +2 -1
@@ -918,7 +918,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,struct swap_io_ctx ctx = {};struct blk_plug plug;struct folio *folio; -pte_t *pte =NULL , pentry; +hw_pte_t *pte =NULL ; +pte_t pentry;int win;unsigned long start, end, addr;pgoff_t ilx = targ_ilx;
mm/swapfile.c +3 -2
@@ -2422,7 +2422,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd,struct page *page;struct folio *swapcache;spinlock_t *ptl; -pte_t *pte, new_pte, old_pte; +hw_pte_t *pte; +pte_t new_pte, old_pte;bool hwpoisoned = false;int ret =1 ;
@@ -2533,7 +2534,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, unsigned int type) { - pte_t *pte = NULL; + hw_pte_t *pte = NULL; do { struct folio *folio;
mm/userfaultfd.c +18 -14
@@ -361,13 +361,13 @@ static int mfill_atomic_install_pte(pmd_t *dst_pmd, {int ret;struct mm_struct *dst_mm = dst_vma->vm_mm; -pte_t _dst_pte, *dst_pte; +hw_pte_t *dst_pte;bool writable = dst_vma->vm_flags &VM_WRITE ;bool vm_shared = dst_vma->vm_flags &VM_SHARED ;spinlock_t *ptl;struct folio *folio =page_folio (page);bool page_in_cache =folio_mapping (folio); -pte_t dst_ptep; +pte_t _dst_pte, dst_ptep; _dst_pte =mk_pte (page, dst_vma->vm_page_prot); _dst_pte =pte_mkdirty (_dst_pte);
@@ -656,7 +656,8 @@ static int mfill_atomic_pte_zeropage(struct mfill_state *state)struct vm_area_struct *dst_vma = state->vma;unsigned long dst_addr = state->dst_addr;pmd_t *dst_pmd = state->pmd; -pte_t _dst_pte, *dst_pte; +hw_pte_t *dst_pte; +pte_t _dst_pte;spinlock_t *ptl;int ret;
@@ -737,7 +738,8 @@ static int mfill_atomic_pte_poison(struct mfill_state *state)struct mm_struct *dst_mm = dst_vma->vm_mm;unsigned long dst_addr = state->dst_addr;pmd_t *dst_pmd = state->pmd; -pte_t _dst_pte, *dst_pte; +hw_pte_t *dst_pte; +pte_t _dst_pte;spinlock_t *ptl;int ret;
@@ -784,7 +786,7 @@ static __always_inline ssize_t mfill_atomic_hugetlb( {struct mm_struct *dst_mm = dst_vma->vm_mm;ssize_t err; -pte_t *dst_pte; +hw_pte_t *dst_pte;unsigned long src_addr, dst_addr;long copied;struct folio *folio;
@@ -1261,7 +1263,7 @@ void double_pt_unlock(spinlock_t *ptl1,__release (ptl2); } -static inline bool is_pte_pages_stable (pte_t *dst_pte,pte_t *src_pte, +static inline bool is_pte_pages_stable (hw_pte_t *dst_pte,hw_pte_t *src_pte,pte_t orig_dst_pte,pte_t orig_src_pte,pmd_t *dst_pmd,pmd_t dst_pmdval) {
@@ -1279,7 +1281,8 @@ static inline bool is_pte_pages_stable(pte_t *dst_pte, pte_t *src_pte,*/ static struct folio *check_ptes_for_batched_move (struct vm_area_struct *src_vma,unsigned long src_addr, -pte_t *src_pte,pte_t *dst_pte) +hw_pte_t *src_pte, +hw_pte_t *dst_pte) {pte_t orig_dst_pte, orig_src_pte;struct folio *folio;
@@ -1310,7 +1313,7 @@ static long move_present_ptes(struct mm_struct *mm,struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, unsigned long dst_addr, unsigned long src_addr, - pte_t *dst_pte, pte_t *src_pte, + hw_pte_t *dst_pte, hw_pte_t *src_pte, pte_t orig_dst_pte, pte_t orig_src_pte, pmd_t *dst_pmd, pmd_t dst_pmdval, spinlock_t *dst_ptl, spinlock_t *src_ptl,
@@ -1397,7 +1400,7 @@ static long move_present_ptes(struct mm_struct *mm, static int move_swap_pte(struct mm_struct *mm, struct vm_area_struct *dst_vma, unsigned long dst_addr, unsigned long src_addr, - pte_t *dst_pte, pte_t *src_pte, + hw_pte_t *dst_pte, hw_pte_t *src_pte, pte_t orig_dst_pte, pte_t orig_src_pte, pmd_t *dst_pmd, pmd_t dst_pmdval, spinlock_t *dst_ptl, spinlock_t *src_ptl,
@@ -1462,7 +1465,7 @@ static int move_zeropage_pte(struct mm_struct *mm,struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, unsigned long dst_addr, unsigned long src_addr, - pte_t *dst_pte, pte_t *src_pte, + hw_pte_t *dst_pte, hw_pte_t *src_pte, pte_t orig_dst_pte, pte_t orig_src_pte, pmd_t *dst_pmd, pmd_t dst_pmdval, spinlock_t *dst_ptl, spinlock_t *src_ptl)
@@ -1508,8 +1511,8 @@ static long move_pages_ptes(struct mm_struct *mm, pmd_t *dst_pmd, pmd_t *src_pmdpte_t orig_src_pte, orig_dst_pte;pte_t src_folio_pte;spinlock_t *src_ptl, *dst_ptl; -pte_t *src_pte =NULL ; -pte_t *dst_pte =NULL ; +hw_pte_t *src_pte =NULL ; +hw_pte_t *dst_pte =NULL ;pmd_t dummy_pmdval;pmd_t dst_pmdval;struct folio *src_folio =NULL ;
@@ -2650,7 +2653,8 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, unsigned long reason) {struct vm_area_struct *vma = vmf->vma; -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte;assert_fault_locked (vmf);
@@ -2723,7 +2727,7 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx,p4d_t *p4d;pud_t *pud;pmd_t *pmd, _pmd; -pte_t *pte; +hw_pte_t *pte;pte_t ptent;bool ret;
mm/util.c +1 -1
@@ -1564,7 +1564,7 @@ EXPORT_SYMBOL(mmap_action_complete);* * Return: the number of table entries in the batch. */ -unsigned int folio_pte_batch (struct folio *folio,pte_t *ptep,pte_t pte, +unsigned int folio_pte_batch (struct folio *folio,hw_pte_t *ptep,pte_t pte,unsigned int max_nr) {return folio_pte_batch_flags (folio,NULL , ptep, &pte, max_nr,0 );
mm/vmalloc.c +6 -5
@@ -97,7 +97,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift, pgtbl_mod_mask *mask) { - pte_t *pte; + hw_pte_t *pte;u64 pfn;struct page *page;unsigned long size =PAGE_SIZE ;
@@ -389,7 +389,7 @@ int ioremap_page_range(unsigned long addr, unsigned long end,static void vunmap_pte_range (pmd_t *pmd,unsigned long addr,unsigned long end,pgtbl_mod_mask *mask) { -pte_t *pte; +hw_pte_t *pte;pte_t ptent;unsigned long size =PAGE_SIZE ;
@@ -550,7 +550,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, pgtbl_mod_mask *mask) {int err =0 ; -pte_t *pte; +hw_pte_t *pte;/* * nr is a running index into the array which helps higher level
@@ -827,7 +827,8 @@ struct page *vmalloc_to_page(const void *vmalloc_addr)p4d_t *p4d;pud_t *pud;pmd_t *pmd; -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte;/* * XXX we might need to change this if we add VIRTUAL_BUG_ON for
@@ -3605,7 +3606,7 @@ struct vmap_pfn_data {unsigned int idx; }; -static int vmap_pfn_apply (pte_t *pte,unsigned long addr,void *private) +static int vmap_pfn_apply (hw_pte_t *pte,unsigned long addr,void *private) {struct vmap_pfn_data *data = private;unsigned long pfn = data->pfns[data->idx];
mm/vmscan.c +3 -3
@@ -3540,7 +3540,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, {int i;bool dirty; -pte_t *pte; +hw_pte_t *pte;spinlock_t *ptl;unsigned long addr;int total =0 ;
@@ -3573,7 +3573,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end,for (i =pte_index (start), addr = start; addr != end; i += nr, addr += nr *PAGE_SIZE ) {unsigned long pfn;struct folio *folio; -pte_t *cur_pte = pte + i; +hw_pte_t *cur_pte = pte + i;pte_t ptent =ptep_get (cur_pte); nr =1 ;
@@ -4262,7 +4262,7 @@ bool lru_gen_look_around(struct page_vma_mapped_walk *pvmw, unsigned int nr)struct lru_gen_mm_walk *walk;struct folio *last =NULL ;int young = nr; -pte_t *pte = pvmw->pte; +hw_pte_t *pte = pvmw->pte;unsigned long addr = pvmw->address;struct vm_area_struct *vma = pvmw->vma;struct folio *folio =pfn_folio (pvmw->pfn);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 03/15] mm: use hw_pte_t for generic PTE table storage
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 18 lines
From: Muhammad Usama Anjum <[email protected]>
Generic page-table interfaces use
pte_t* for both pointers to PTE table storage and pointers to software PTE values. Convert the generic declarations and their MM, fs, and kernel users together so parameters, return types, callbacks, and local pointers that designate table storage usehw_pte_t*.Include
linux/pgtable_types.hfrom headers that expose the converted interfaces. It continues to providepgprot_ttovmalloc.hthroughasm/page.h.Keep software PTE values as
pte_tand retainpte_t* for value interfaces.No architecture selects
ARCH_HAS_HW_PTE_Tat this point, sohw_pte_tremains an alias ofpte_tand this changes the interface vocabulary without changing representation or behavior.
Quoted 7 lines
Most of this mechanical conversion was generated with the Coccinelle script included in the cover letter. The script deliberately ignores
pte_t* pointers named ptentp because they designate software PTE values. The result was then audited, and sites the script could not convert were updated by hand.Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
Alexander Gordeev
maintainerPATCH v8 4/15 mm: convert PTE table entries in ptep_get()
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
ptep_get() now accepts a pointer to hw_pte_t storage but must continue to return a software PTE value. Add __pte_from_hw for both generic hw_pte_t definitions. Read the hw_pte_t table element atomically before converting it to pte_t.
--- include/linux/pgtable.h | 2 +- include/linux/pgtable_types.h | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-)
include/linux/pgtable.h +1 -1
@@ -493,7 +493,7 @@ static inline int pudp_set_access_flags(struct vm_area_struct *vma,#ifndef ptep_getstatic inline pte_t ptep_get (hw_pte_t *ptep) { -return READ_ONCE (*ptep); +return __pte_from_hw (READ_ONCE (*ptep)); }#endif
include/linux/pgtable_types.h +2 -0
@@ -8,8 +8,10 @@#ifdef CONFIG_ARCH_HAS_HW_PTE_T typedef struct __hw_pte_t {pte_t __pte; }hw_pte_t ; +#define __pte_from_hw (pte) ((pte).__pte)#else #define hw_pte_t pte_t +#define __pte_from_hw (pte) (pte)#endif #endif /* !__ASSEMBLY__ */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 04/15] mm: convert PTE table entries in ptep_get()
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 8 lines
From: Muhammad Usama Anjum <[email protected]>
ptep_get()now accepts a pointer tohw_pte_tstorage but must continue to return a software PTE value. Add__pte_from_hwfor both generichw_pte_tdefinitions. Read thehw_pte_ttable element atomically before converting it topte_t.Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4
Alexander Gordeev
maintainerPATCH v8 5/15 mm: convert PTE table entry to pte
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
The non-MMU stub receives hw_pte_t but returns a software PTE value. It has no attached PTE that requires ptep_get(), so convert only the stored entry through __pte_from_hw().
--- include/linux/hugetlb.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/linux/hugetlb.h +2 -1
@@ -1283,7 +1283,8 @@ static inline pte_t huge_ptep_clear_flush(struct vm_area_struct *vma,#ifdef CONFIG_MMU return ptep_get (ptep);#else -return *ptep; +/* No attached PTE that requires ptep_get(). */ +return __pte_from_hw (*ptep);#endif }
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 05/15] mm: convert PTE table entry to pte
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines
From: Muhammad Usama Anjum <[email protected]>
The non-MMU stub receives
hw_pte_tbut returns a software PTE value. It has no attached PTE that requiresptep_get(), so convert only the stored entry through__pte_from_hw().Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=5
Alexander Gordeev
maintainerPATCH v8 6/15 mm: add hw_pte_val for HW PTE storage
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
Atomic PTE updates need an lvalue for the bits stored in an HW PTE. pte_val() only accepts a SW PTE value, so it cannot operate directly on a distinct hw_pte_t.
Add hw_pte_val() to expose the underlying pte_val() lvalue. Access the wrapper's __pte member when hw_pte_t is distinct, and use pte_val() directly when it remains an alias of pte_t.
--- include/linux/pgtable_types.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/pgtable_types.h +4 -0
@@ -9,9 +9,13 @@#ifdef CONFIG_ARCH_HAS_HW_PTE_T typedef struct __hw_pte_t {pte_t __pte; }hw_pte_t ;#define __pte_from_hw (pte) ((pte).__pte) + +#define hw_pte_val (x) pte_val((x).__pte)#else #define hw_pte_t pte_t#define __pte_from_hw (pte) (pte) + +#define hw_pte_val (x) pte_val(x)#endif #endif /* !__ASSEMBLY__ */
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 06/15] mm: add hw_pte_val for HW PTE storage
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [Low] mm: hw_pte_val expands to rvalue breaking lvalue API contract --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Low] mm: hw_pte_val expands to rvalue breaking lvalue API contract
--
commit 5410dae6e74b38457df98b3df5924d0050eb76a9
Author: Muhammad Usama Anjum <[email protected]>
mm: add hw_pte_val for HW PTE storage
This commit adds a hw_pte_val() macro to expose an lvalue for hardware PTE storage. It accesses the wrapper's __pte member when hw_pte_t is distinct, and uses pte_val() directly otherwise.
include/linux/pgtable_types.h line 13
Quoted 10 lines
include/linux/pgtable_types.h +2
-0@@ -9,9 +9,13 @@#ifdef CONFIG_ARCH_HAS_HW_PTE_T typedef struct __hw_pte_t {pte_t __pte; }hw_pte_t ;#define __pte_from_hw (pte) ((pte).__pte) + +#define hw_pte_val (x) pte_val((x).__pte)
[Severity: Low]
The commit message states that hw_pte_val() is meant to expose an lvalue. However, on architectures where pte_val() is implemented as a static inline function rather than a macro (such as x86), doesn't pte_val() evaluate to an rvalue instead?
Will taking the address of hw_pte_val() in generic code yield an "lvalue required" compiler error on those architectures?
include/linux/pgtable_types.h line 18
Quoted 5 lines
include/linux/pgtable_types.h +2
-0#else #define hw_pte_t pte_t#define __pte_from_hw (pte) (pte) + +#define hw_pte_val (x) pte_val(x)
[Severity: Low]
Similarly, does this also evaluate to an rvalue on architectures where pte_val() is a static inline function, breaking the expected lvalue API contract?
Quoted 3 lines
include/linux/pgtable_types.h +0
-0#endif #endif /* !__ASSEMBLY__ */
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=6
Alexander Gordeev
maintainerPATCH v8 7/15 mm/kasan: use hw_pte_t for the early shadow PTE table
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
kasan_early_shadow_pte is a complete PTE table rather than a software PTE value. Declare and define its elements as hw_pte_t so the object uses the PTE table storage type.
Read the first element through ptep_get(), which returns the software PTE value expected by note_page_pte(), instead of accessing hw_pte_t storage directly. This also uses the accessor selected by the architecture.
No architecture selects ARCH_HAS_HW_PTE_T at this point, so the storage representation remains unchanged.
--- include/linux/kasan.h | 2 +- mm/kasan/init.c | 2 +- mm/ptdump.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
include/linux/kasan.h +1 -1
@@ -51,7 +51,7 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;#endif extern unsigned char kasan_early_shadow_page[PAGE_SIZE ]; -extern pte_t kasan_early_shadow_pte[MAX_PTRS_PER_PTE +PTE_HWTABLE_PTRS ]; +extern hw_pte_t kasan_early_shadow_pte[MAX_PTRS_PER_PTE +PTE_HWTABLE_PTRS ];extern pmd_t kasan_early_shadow_pmd[MAX_PTRS_PER_PMD ];extern pud_t kasan_early_shadow_pud[MAX_PTRS_PER_PUD ];extern p4d_t kasan_early_shadow_p4d[MAX_PTRS_PER_P4D ];
mm/kasan/init.c +1 -1
@@ -64,7 +64,7 @@ static inline bool kasan_pmd_table(pud_t pud)return false; }#endif -pte_t kasan_early_shadow_pte[MAX_PTRS_PER_PTE + PTE_HWTABLE_PTRS] +hw_pte_t kasan_early_shadow_pte[MAX_PTRS_PER_PTE + PTE_HWTABLE_PTRS] __bss_pgtbl;static inline bool kasan_pte_table (pmd_t pmd)
mm/ptdump.c +1 -1
@@ -19,7 +19,7 @@ static inline int note_kasan_page_table(struct mm_walk *walk, {struct ptdump_state *st = walk->private; - st->note_page_pte (st, addr, kasan_early_shadow_pte[0 ]); + st->note_page_pte (st, addr,ptep_get (kasan_early_shadow_pte)); walk->action =ACTION_CONTINUE ;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 07/15] mm/kasan: use hw_pte_t for the early shadow PTE table
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 6 lines
From: Muhammad Usama Anjum <[email protected]>
kasan_early_shadow_pteis a complete PTE table rather than a software PTE value. Declare and define its elements ashw_pte_tso the object uses the PTE table storage type.
Quoted 8 lines
Read the first element through
ptep_get(), which returns the software PTE value expected bynote_page_pte(), instead of accessinghw_pte_tstorage directly. This also uses the accessor selected by the architecture.No architecture selects
ARCH_HAS_HW_PTE_Tat this point, so the storage representation remains unchanged.Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=7
Alexander Gordeev
maintainerPATCH v8 8/15 drm/i915: use hw_pte_t for PTE range callbacks
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
apply_to_page_range() now passes PTE table storage to its callback as hw_pte_t *. Update the i915 remap and selftest callbacks to match the new type.
Continue to use ptep_get() for software PTE values and set_pte_at() for updates.
The type remains an alias of pte_t on x86 until that architecture opts in.
--- drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c | 4 ++-- drivers/gpu/drm/i915/i915_mm.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
drivers/gpu/drm/i915/gem/selftests/i915_gem_mman.c +2 -2
@@ -1690,7 +1690,7 @@ static int igt_mmap_gpu(void *arg)return 0 ; } -static int check_present_pte (pte_t *pte,unsigned long addr,void *data) +static int check_present_pte (hw_pte_t *pte,unsigned long addr,void *data) {pte_t ptent =ptep_get (pte);
@@ -1703,7 +1703,7 @@ static int check_present_pte(pte_t *pte, unsigned long addr, void *data)return 0 ; } -static int check_absent_pte (pte_t *pte,unsigned long addr,void *data) +static int check_absent_pte (hw_pte_t *pte,unsigned long addr,void *data) {pte_t ptent =ptep_get (pte);
drivers/gpu/drm/i915/i915_mm.c +2 -2
@@ -48,7 +48,7 @@ static inline unsigned long sgt_pfn(const struct remap_pfn *r)return r->sgt.pfn + (r->sgt.curr >>PAGE_SHIFT ); } -static int remap_sg (pte_t *pte,unsigned long addr,void *data) +static int remap_sg (hw_pte_t *pte,unsigned long addr,void *data) {struct remap_pfn *r = data;
@@ -70,7 +70,7 @@ static int remap_sg(pte_t *pte, unsigned long addr, void *data)#define EXPECTED_FLAGS (VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP)#if IS_ENABLED (CONFIG_X86 ) -static int remap_pfn (pte_t *pte,unsigned long addr,void *data) +static int remap_pfn (hw_pte_t *pte,unsigned long addr,void *data) {struct remap_pfn *r = data;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 08/15] drm/i915: use hw_pte_t for PTE range callbacks
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
From: Muhammad Usama Anjum <[email protected]>
apply_to_page_range()now passes PTE table storage to its callback ashw_pte_t*. Update the i915 remap and selftest callbacks to match the new type.Continue to use
ptep_get()for software PTE values andset_pte_at()for updates.
The type remains an alias ofpte_ton x86 until that architecture opts in.Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=8
Alexander Gordeev
maintainerPATCH v8 9/15 xen: use hw_pte_t for PTE range callbacks
In reply to Alexander GordeevFrom: Muhammad Usama Anjum <[email protected]>
Generic PTE range and remapping helpers now pass pointers to PTE table storage as hw_pte_t *. Update the Xen callbacks to match those interfaces.
Keep software PTE values as pte_t and continue to access them through the existing PTE helpers. This is required when Xen is built for an architecture that selects the distinct hw_pte_t wrapper.
--- drivers/xen/gntdev.c | 2 +- drivers/xen/privcmd.c | 2 +- drivers/xen/xenbus/xenbus_client.c | 2 +- drivers/xen/xlate_mmu.c | 4 ++-- 4 files changed, 5 insertions(+), 5 deletions(-)
drivers/xen/gntdev.c +1 -1
@@ -301,7 +301,7 @@ void gntdev_put_map(struct gntdev_priv *priv, struct gntdev_grant_map *map)/* ------------------------------------------------------------------ */ -static int find_grant_ptes (pte_t *pte,unsigned long addr,void *data) +static int find_grant_ptes (hw_pte_t *pte,unsigned long addr,void *data) {struct gntdev_grant_map *map = data;unsigned int pgnr = (addr - map->pages_vm_start) >>PAGE_SHIFT ;
drivers/xen/privcmd.c +1 -1
@@ -1658,7 +1658,7 @@ static int privcmd_mmap(struct file *file, struct vm_area_struct *vma)* on a per pfn/pte basis. Mapping calls that fail with ENOENT * can be then retried until success. */ -static int is_mapped_fn (pte_t *pte,unsigned long addr,void *data) +static int is_mapped_fn (hw_pte_t *pte,unsigned long addr,void *data) {return pte_none (ptep_get (pte)) ?0 : -EBUSY ; }
drivers/xen/xenbus/xenbus_client.c +1 -1
@@ -749,7 +749,7 @@ int xenbus_unmap_ring_vfree(struct xenbus_device *dev, void *vaddr)EXPORT_SYMBOL_GPL (xenbus_unmap_ring_vfree);#ifdef CONFIG_XEN_PV -static int map_ring_apply (pte_t *pte,unsigned long addr,void *data) +static int map_ring_apply (hw_pte_t *pte,unsigned long addr,void *data) {struct map_ring_valloc *info = data;
drivers/xen/xlate_mmu.c +2 -2
@@ -93,7 +93,7 @@ static void setup_hparams(unsigned long gfn, void *data) info->fgfn++; } -static int remap_pte_fn (pte_t *ptep,unsigned long addr,void *data) +static int remap_pte_fn (hw_pte_t *ptep,unsigned long addr,void *data) {struct remap_data *info = data;struct page *page = info->pages[info->index++];
@@ -269,7 +269,7 @@ struct remap_pfn {unsigned long i; }; -static int remap_pfn_fn (pte_t *ptep,unsigned long addr,void *data) +static int remap_pfn_fn (hw_pte_t *ptep,unsigned long addr,void *data) {struct remap_pfn *r = data;struct page *page = r->pages[r->i];
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 09/15] xen: use hw_pte_t for PTE range callbacks
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
From: Muhammad Usama Anjum <[email protected]>
Generic PTE range and remapping helpers now pass pointers to PTE table storage as
hw_pte_t*. Update the Xen callbacks to match those interfaces.Keep software PTE values as
pte_tand continue to access them through the existing PTE helpers. This is required when Xen is built for an architecture that selects the distincthw_pte_twrapper.Reviewed-by Juergen Gross <[email protected]>Signed-off-by Muhammad Usama Anjum <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=9
Alexander Gordeev
maintainerPATCH v8 10/15 s390/mm: Cleanup pXXp_flush_lazy() routines
In reply to Alexander GordeevAvoid unnecessary pgtable entry read in ptep_flush_lazy() and pmdp_flush_lazy() routines. Use the already obtained value of a pgtable entry instead of dereferencing it again.
It might appear that the optimized out second dereferencing happens under mm_context_t::flush_count lock and therefore a race condition is possible. However, both routines write invalid entry bits to the pgtable entries and therefore no reader is allowed to interpret the rest of the value afterwards.
--- arch/s390/mm/pgtable.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
arch/s390/mm/pgtable.c +2 -2
@@ -107,7 +107,7 @@ static inline pte_t ptep_flush_lazy(struct mm_struct *mm,atomic_inc (&mm->context.flush_count);if (cpumask_equal (&mm->context.cpu_attach_mask,cpumask_of (smp_processor_id ()))) { -set_pte (ptep,set_pte_bit (*ptep,__pgprot (_PAGE_INVALID))); +set_pte (ptep,set_pte_bit (old,__pgprot (_PAGE_INVALID))); mm->context.flush_mm =1 ; }else ptep_ipte_global (mm, addr, ptep, nodat);
@@ -227,7 +227,7 @@ static inline pmd_t pmdp_flush_lazy(struct mm_struct *mm,atomic_inc (&mm->context.flush_count);if (cpumask_equal (&mm->context.cpu_attach_mask,cpumask_of (smp_processor_id ()))) { -set_pmd (pmdp,set_pmd_bit (*pmdp,__pgprot (_SEGMENT_ENTRY_INVALID))); +set_pmd (pmdp,set_pmd_bit (old,__pgprot (_SEGMENT_ENTRY_INVALID))); mm->context.flush_mm =1 ; }else {pmdp_idte_global (mm, addr, pmdp);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 10/15] s390/mm: Cleanup pXXp_flush_lazy() routines
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Avoid unnecessary pgtable entry read in
ptep_flush_lazy()andpmdp_flush_lazy()routines. Use the already obtained value of a pgtable entry instead of dereferencing it again.It might appear that the optimized out second dereferencing happens under
mm_context_t::flush_countlock and therefore a race condition is possible. However, both routines write invalid entry bits to the pgtable entries and therefore no reader is allowed to interpret the rest of the value afterwards.Signed-off-by Alexander Gordeev <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=10
Re: [PATCH v8 10/15] s390/mm: Cleanup pXXp_flush_lazy() routines
In reply to Alexander GordeevQuoted 10 lines · On Wed, 7 Oct 2026 13:41:53 +0200 Alexander Gordeev <[email protected]>…
Avoid unnecessary pgtable entry read in
ptep_flush_lazy()andpmdp_flush_lazy()routines. Use the already obtained value of a pgtable entry instead of dereferencing it again.It might appear that the optimized out second dereferencing happens under
mm_context_t::flush_countlock and therefore a race condition is possible. However, both routines write invalid entry bits to the pgtable entries and therefore no reader is allowed to interpret the rest of the value afterwards.
Quoted 4 lines
Signed-off-by Alexander Gordeev <[email protected]>--- arch/s390/mm/pgtable.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Looks good to me.
Re: [PATCH v8 10/15] s390/mm: Cleanup pXXp_flush_lazy() routines
In reply to Alexander GordeevQuoted 10 lines · Alexander Gordeev
Avoid unnecessary pgtable entry read in
ptep_flush_lazy()andpmdp_flush_lazy()routines. Use the already obtained value of a pgtable entry instead of dereferencing it again.It might appear that the optimized out second dereferencing happens under
mm_context_t::flush_countlock and therefore a race condition is possible. However, both routines write invalid entry bits to the pgtable entries and therefore no reader is allowed to interpret the rest of the value afterwards.
Quoted 4 lines
Signed-off-by Alexander Gordeev <[email protected]>--- arch/s390/mm/pgtable.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
Applied, thanks!
Alexander Gordeev
maintainerPATCH v8 11/15 s390: Distinguish hardware and software PTEs
In reply to Alexander GordeevThis is s390 version of ARM64 commit XXXXXXXXXXXX ("arm64:
distinguish HW PTE pointers from SW PTE value pointers").
It distinguishes hardware and software PTEs, introduces hw_pte_t type for hardware PTEs, while software PTEs still use pte_t type and enables ARCH_HAS_HW_PTE_T configuration for s390.
--- arch/s390/Kconfig | 1 + arch/s390/boot/startup.c | 2 +- arch/s390/boot/vmem.c | 17 ++++----- arch/s390/include/asm/gmap_helpers.h | 2 +- arch/s390/include/asm/hugetlb.h | 18 +++++----- arch/s390/include/asm/maccess.h | 2 +- arch/s390/include/asm/page.h | 3 +- arch/s390/include/asm/pgalloc.h | 6 ++-- arch/s390/include/asm/pgtable.h | 54 +++++++++++++++------------- arch/s390/kernel/uv.c | 4 +-- arch/s390/kvm/s390/pv.c | 2 +- arch/s390/mm/gmap_helpers.c | 18 +++++----- arch/s390/mm/hugetlbpage.c | 20 +++++------ arch/s390/mm/maccess.c | 2 +- arch/s390/mm/pageattr.c | 10 +++--- arch/s390/mm/pgtable.c | 26 +++++++------- arch/s390/mm/vmem.c | 26 +++++++------- 17 files changed, 112 insertions(+), 101 deletions(-)
arch/s390/Kconfig +1 -0
@@ -100,6 +100,7 @@ config S390 select ARCH_HAS_FORTIFY_SOURCE select ARCH_HAS_GCOV_PROFILE_ALL select ARCH_HAS_GIGANTIC_PAGE + select ARCH_HAS_HW_PTE_T select ARCH_HAS_KCOV select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_MEM_ENCRYPT
arch/s390/boot/startup.c +1 -1
@@ -30,7 +30,7 @@struct vm_layout __bootdata_preserved (vm_layout );unsigned long __bootdata_preserved (__abs_lowcore );unsigned long __bootdata_preserved (__memcpy_real_area ); -pte_t *__bootdata_preserved (memcpy_real_ptep ); +hw_pte_t *__bootdata_preserved (memcpy_real_ptep );unsigned long __bootdata_preserved (VMALLOC_START );unsigned long __bootdata_preserved (VMALLOC_END );struct page *__bootdata_preserved (vmemmap );
arch/s390/boot/vmem.c +9 -8
@@ -75,7 +75,7 @@ static void pgtable_populate(unsigned long addr, unsigned long end, enum populat#ifdef CONFIG_KASAN #define kasan_early_shadow_page vmlinux.kasan_early_shadow_page_off -#define kasan_early_shadow_pte ((pte_t *)vmlinux.kasan_early_shadow_pte_off) +#define kasan_early_shadow_pte ((hw_pte_t *)vmlinux.kasan_early_shadow_pte_off)#define kasan_early_shadow_pmd ((pmd_t *)vmlinux.kasan_early_shadow_pmd_off)#define kasan_early_shadow_pud ((pud_t *)vmlinux.kasan_early_shadow_pud_off)#define kasan_early_shadow_p4d ((p4d_t *)vmlinux.kasan_early_shadow_p4d_off)
@@ -178,7 +178,7 @@ static bool kasan_pmd_populate_zero_shadow(pmd_t *pmd, unsigned long addr,return false; } -static bool kasan_pte_populate_zero_shadow (pte_t *pte,enum populate_mode mode) +static bool kasan_pte_populate_zero_shadow (hw_pte_t *pte,enum populate_mode mode) {if (mode ==POPULATE_KASAN_ZERO_SHADOW ) {set_pte (pte, pte_z);
@@ -216,7 +216,7 @@ static inline bool kasan_pmd_populate_zero_shadow(pmd_t *pmd, unsigned long addrreturn false; } -static bool kasan_pte_populate_zero_shadow (pte_t *pte,enum populate_mode mode) +static bool kasan_pte_populate_zero_shadow (hw_pte_t *pte,enum populate_mode mode) {return false; }
@@ -226,7 +226,7 @@ static bool kasan_pte_populate_zero_shadow(pte_t *pte, enum populate_mode mode)/* * Mimic virt_to_kpte() in lack of init_mm symbol. Skip pmd NULL check though. */ -static inline pte_t *__virt_to_kpte (unsigned long va) +static inline hw_pte_t *__virt_to_kpte (unsigned long va) {return pte_offset_kernel (pmd_offset (pud_offset (p4d_offset (pgd_offset_k (va), va), va), va), va); }
@@ -242,9 +242,9 @@ static void *boot_crst_alloc(unsigned long val)return table; } -static pte_t *boot_pte_alloc (void ) +static hw_pte_t *boot_pte_alloc (void ) { -pte_t *pte; +hw_pte_t *pte; pte = (void *)physmem_alloc_or_die (RR_VMEM ,PAGE_SIZE ,PAGE_SIZE );__arch_set_page_dat (pte,1 );
@@ -334,7 +334,8 @@ static void pgtable_pte_populate(pmd_t *pmd, unsigned long addr, unsigned long e enum populate_mode mode) {unsigned long pages =0 ; -pte_t *pte, entry; +hw_pte_t *pte; +pte_t entry; pte =pte_offset_kernel (pmd, addr);for (; addr < end; addr +=PAGE_SIZE , pte++) {
@@ -356,7 +357,7 @@ static void pgtable_pmd_populate(pud_t *pud, unsigned long addr, unsigned long e {unsigned long pa, next, pages =0 ;pmd_t *pmd, entry, large_entry; -pte_t *pte; +hw_pte_t *pte; pmd =pmd_offset (pud, addr);for (; addr < end; addr = next, pmd++) {
arch/s390/include/asm/gmap_helpers.h +1 -1
@@ -12,6 +12,6 @@ void gmap_helper_zap_one_page(struct mm_struct *mm, unsigned long vmaddr);void gmap_helper_discard (struct mm_struct *mm,unsigned long vmaddr,unsigned long end);int gmap_helper_disable_cow_sharing (void );void gmap_helper_try_set_pte_unused (struct mm_struct *mm,unsigned long vmaddr); -pte_t *try_get_locked_pte (struct mm_struct *mm,unsigned long addr,spinlock_t **ptl); +hw_pte_t *try_get_locked_pte (struct mm_struct *mm,unsigned long addr,spinlock_t **ptl);#endif /* _ASM_S390_GMAP_HELPERS_H */
arch/s390/include/asm/hugetlb.h +9 -9
@@ -19,19 +19,19 @@#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_ATvoid set_huge_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte,unsigned long sz); +hw_pte_t *ptep,pte_t pte,unsigned long sz);void __set_huge_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte); +hw_pte_t *ptep,pte_t pte);#define __HAVE_ARCH_HUGE_PTEP_GET -pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep); +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,hw_pte_t *ptep);pte_t __huge_ptep_get_and_clear (struct mm_struct *mm,unsigned long addr, -pte_t *ptep); +hw_pte_t *ptep);#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEARstatic inline pte_t huge_ptep_get_and_clear (struct mm_struct *mm, -unsigned long addr,pte_t *ptep, +unsigned long addr,hw_pte_t *ptep,unsigned long sz) {return __huge_ptep_get_and_clear (mm, addr, ptep);
@@ -39,7 +39,7 @@ static inline pte_t huge_ptep_get_and_clear(struct mm_struct *mm,#define __HAVE_ARCH_HUGE_PTE_CLEARstatic inline void huge_pte_clear (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,unsigned long sz) +hw_pte_t *ptep,unsigned long sz) {if ((pte_val (ptep_get (ptep)) & _REGION_ENTRY_TYPE_MASK) == _REGION_ENTRY_TYPE_R3)set_pud ((pud_t *)ptep,__pud (_REGION3_ENTRY_EMPTY));
@@ -49,14 +49,14 @@ static inline void huge_pte_clear(struct mm_struct *mm, unsigned long addr,#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSHstatic inline pte_t huge_ptep_clear_flush (struct vm_area_struct *vma, -unsigned long address,pte_t *ptep) +unsigned long address,hw_pte_t *ptep) {return __huge_ptep_get_and_clear (vma->vm_mm, address, ptep); }#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGSstatic inline int huge_ptep_set_access_flags (struct vm_area_struct *vma, -unsigned long addr,pte_t *ptep, +unsigned long addr,hw_pte_t *ptep,pte_t pte,int dirty) {int changed = !pte_same (huge_ptep_get (vma->vm_mm, addr, ptep), pte);
@@ -70,7 +70,7 @@ static inline int huge_ptep_set_access_flags(struct vm_area_struct *vma,#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECTstatic inline void huge_ptep_set_wrprotect (struct mm_struct *mm, -unsigned long addr,pte_t *ptep) +unsigned long addr,hw_pte_t *ptep) {pte_t pte =__huge_ptep_get_and_clear (mm, addr, ptep);
arch/s390/include/asm/maccess.h +1 -1
@@ -10,7 +10,7 @@struct iov_iter ;extern unsigned long __memcpy_real_area; -extern pte_t *memcpy_real_ptep; +extern hw_pte_t *memcpy_real_ptep;size_t memcpy_real_iter (struct iov_iter *iter,unsigned long src,size_t count);int memcpy_real (void *dest,unsigned long src,size_t count);#ifdef CONFIG_CRASH_DUMP
arch/s390/include/asm/page.h +2 -1
@@ -113,7 +113,8 @@ DEFINE_PGVAL_FUNC(pud)DEFINE_PGVAL_FUNC (p4d)DEFINE_PGVAL_FUNC (pgd) -typedef pte_t *pgtable_t; +struct __hw_pte_t; +typedef struct __hw_pte_t *pgtable_t; #define __pgprot(x) ((pgprot_t) { (x) } )#define __pte (x) ((pte_t) { (x) } )
arch/s390/include/asm/pgalloc.h +3 -3
@@ -159,8 +159,8 @@ static inline void pmd_populate(struct mm_struct *mm,/* * page table entry allocation/free routines. */ -#define pte_alloc_one_kernel (mm) ((pte_t *)page_table_alloc(mm)) -#define pte_alloc_one (mm) ((pte_t *)page_table_alloc(mm)) +#define pte_alloc_one_kernel (mm) ((hw_pte_t *)page_table_alloc(mm)) +#define pte_alloc_one (mm) ((hw_pte_t *)page_table_alloc(mm))#define pte_free_kernel (mm, pte) page_table_free(mm, (unsigned long *) pte)#define pte_free (mm, pte) page_table_free(mm, (unsigned long *) pte)
@@ -171,7 +171,7 @@ void pte_free_defer(struct mm_struct *mm, pgtable_t pgtable);void vmem_map_init (void );void *vmem_crst_alloc (unsigned long val); -pte_t *vmem_pte_alloc (void ); +hw_pte_t *vmem_pte_alloc (void );unsigned long base_asce_alloc (unsigned long addr,unsigned long num_pages);void base_asce_free (unsigned long asce);
arch/s390/include/asm/pgtable.h +29 -25
165 lines not on this page · raw message
arch/s390/kernel/uv.c +2 -2
@@ -210,7 +210,7 @@ int uv_convert_from_secure_pte(pte_t pte)return uv_convert_from_secure_folio (pfn_folio (pte_pfn (pte))); } -static int uv_free_range_cb (pte_t *ptep,unsigned long addr,void *data) +static int uv_free_range_cb (hw_pte_t *ptep,unsigned long addr,void *data) {pte_t pte =ptep_get (ptep);
@@ -242,7 +242,7 @@ void uv_free_stor_var(void *stor_var) }EXPORT_SYMBOL_FOR_MODULES (uv_free_stor_var,"kvm" ); -static int uv_alloc_range_cb (pte_t *ptep,unsigned long addr,void *data) +static int uv_alloc_range_cb (hw_pte_t *ptep,unsigned long addr,void *data) {struct page *page;pte_t pte;
arch/s390/kvm/s390/pv.c +1 -1
@@ -106,7 +106,7 @@ static void _kvm_s390_pv_make_secure(struct guest_fault *f)struct pv_make_secure *priv = f->priv;struct folio *folio;spinlock_t *ptl;/* pte lock from try_get_locked_pte() */ -pte_t *ptep; +hw_pte_t *ptep; folio =pfn_folio (f->pfn); priv->rc = -EAGAIN ;
arch/s390/mm/gmap_helpers.c +9 -9
@@ -39,14 +39,14 @@* * the pointer to the pte corresponding to @addr in @mm, if it can be reached * and locked. */ -pte_t *try_get_locked_pte (struct mm_struct *mm,unsigned long vmaddr,spinlock_t **ptl) +hw_pte_t *try_get_locked_pte (struct mm_struct *mm,unsigned long vmaddr,spinlock_t **ptl)__context_unsafe (/* Returns nonnull if lock taken or not taken */ ) {pmd_t *pmdp, pmd, pmdval;pud_t *pudp, pud;p4d_t *p4dp, p4d;pgd_t *pgdp, pgd; -pte_t *ptep; +hw_pte_t *ptep; pgdp =pgd_offset (mm, vmaddr); pgd =pgdp_get (pgdp);
@@ -96,7 +96,7 @@ __context_unsafe(/* pte_unmap_unlock() not instrumented */)struct vm_area_struct *vma;spinlock_t *ptl;/* Lock for the host (userspace) page table */ softleaf_t sl; -pte_t *ptep; +hw_pte_t *ptep;mmap_assert_locked (mm);
@@ -109,8 +109,8 @@ __context_unsafe(/* pte_unmap_unlock() not instrumented */) ptep =try_get_locked_pte (mm, vmaddr, &ptl);if (IS_ERR_OR_NULL (ptep))return ; - sl =softleaf_from_pte (*ptep); -if (pte_swap (*ptep) &&softleaf_is_swap (sl)) { + sl =softleaf_from_pte (ptep_get (ptep)); +if (pte_swap (ptep_get (ptep)) &&softleaf_is_swap (sl)) {dec_mm_counter (mm,MM_SWAPENTS );swap_put_entries_direct (sl,1 );pte_clear (mm, vmaddr, ptep);
@@ -166,7 +166,7 @@ void gmap_helper_try_set_pte_unused(struct mm_struct *mm, unsigned long vmaddr) __context_unsafe(/* pte_unmap_unlock() not instrumented */) {spinlock_t *ptl;/* Lock for the host (userspace) page table */ -pte_t *ptep; +hw_pte_t *ptep;/* * Several paths exists that takes the ptl lock and then call the
@@ -184,19 +184,19 @@ __context_unsafe(/* pte_unmap_unlock() not instrumented */)if (IS_ERR_OR_NULL (ptep))return ; -if (pte_present (*ptep)) +if (pte_present (ptep_get (ptep)))__atomic64_or (_PAGE_UNUSED, (long *)ptep);pte_unmap_unlock (ptep, ptl); }EXPORT_SYMBOL_GPL (gmap_helper_try_set_pte_unused); -static int find_zeropage_pte_entry (pte_t *pte,unsigned long addr, +static int find_zeropage_pte_entry (hw_pte_t *pte,unsigned long addr,unsigned long end,struct mm_walk *walk) {unsigned long *found_addr = walk->private;/* Return 1 of the page is a zeropage. */ -if (is_zero_pfn (pte_pfn (*pte))) { +if (is_zero_pfn (pte_pfn (ptep_get (pte)))) {/* * Shared zeropage in e.g., a FS DAX mapping? We cannot do the * right thing and likely don't care: FAULT_FLAG_UNSHARE
arch/s390/mm/hugetlbpage.c +10 -10
@@ -136,7 +136,7 @@ static inline pte_t __rste_to_pte(unsigned long rste) }void __set_huge_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte) +hw_pte_t *ptep,pte_t pte) {unsigned long rste;
@@ -156,18 +156,18 @@ void __set_huge_pte_at(struct mm_struct *mm, unsigned long addr, }void set_huge_pte_at (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t pte,unsigned long sz) +hw_pte_t *ptep,pte_t pte,unsigned long sz) {__set_huge_pte_at (mm, addr, ptep, pte); } -pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,pte_t *ptep) +pte_t huge_ptep_get (struct mm_struct *mm,unsigned long addr,hw_pte_t *ptep) {return __rste_to_pte (pte_val (ptep_get (ptep))); }pte_t __huge_ptep_get_and_clear (struct mm_struct *mm, -unsigned long addr,pte_t *ptep) +unsigned long addr,hw_pte_t *ptep) {pte_t pte =huge_ptep_get (mm, addr, ptep);pmd_t *pmdp = (pmd_t *) ptep;
@@ -180,7 +180,7 @@ pte_t __huge_ptep_get_and_clear(struct mm_struct *mm,return pte; } -pte_t *huge_pte_alloc (struct mm_struct *mm,struct vm_area_struct *vma, +hw_pte_t *huge_pte_alloc (struct mm_struct *mm,struct vm_area_struct *vma,unsigned long addr,unsigned long sz) {pgd_t *pgdp;
@@ -194,15 +194,15 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, pudp =pud_alloc (mm, p4dp, addr);if (pudp) {if (sz ==PUD_SIZE ) -return (pte_t *) pudp; +return (hw_pte_t *) pudp;else if (sz ==PMD_SIZE ) pmdp =pmd_alloc (mm, pudp, addr); } } -return (pte_t *) pmdp; +return (hw_pte_t *) pmdp; } -pte_t *huge_pte_offset (struct mm_struct *mm, +hw_pte_t *huge_pte_offset (struct mm_struct *mm,unsigned long addr,unsigned long sz) {pgd_t *pgdp;
@@ -216,12 +216,12 @@ pte_t *huge_pte_offset(struct mm_struct *mm,if (p4d_present (p4dp_get (p4dp))) { pudp =pud_offset (p4dp, addr);if (sz ==PUD_SIZE ) -return (pte_t *)pudp; +return (hw_pte_t *)pudp;if (pud_present (pudp_get (pudp))) pmdp =pmd_offset (pudp, addr); } } -return (pte_t *) pmdp; +return (hw_pte_t *) pmdp; }bool __init arch_hugetlb_valid_size (unsigned long size)
arch/s390/mm/maccess.c +1 -1
@@ -22,7 +22,7 @@#include <asm/ctlreg.h> unsigned long __bootdata_preserved (__memcpy_real_area ); -pte_t *__bootdata_preserved (memcpy_real_ptep ); +hw_pte_t *__bootdata_preserved (memcpy_real_ptep );static DEFINE_MUTEX (memcpy_real_mutex);static notrace long s390_kernel_write_odd (void *dst,const void *src,size_t size)
arch/s390/mm/pageattr.c +6 -4
@@ -79,7 +79,8 @@ static void pgt_set(unsigned long *old, unsigned long new, unsigned long addr,static int walk_pte_level (pmd_t *pmdp,unsigned long addr,unsigned long end,unsigned long flags) { -pte_t *ptep, new; +hw_pte_t *ptep; +pte_t new;if (flags ==SET_MEMORY_4K )return 0 ;
@@ -112,7 +113,7 @@ static int walk_pte_level(pmd_t *pmdp, unsigned long addr, unsigned long end,static int split_pmd_page (pmd_t *pmdp,unsigned long addr) {unsigned long pte_addr, prot; -pte_t *pt_dir, *ptep; +hw_pte_t *pt_dir, *ptep;pmd_t new, pmd;int i, ro, nx;
@@ -421,7 +422,7 @@ bool kernel_page_present(struct page *page)#if defined(CONFIG_DEBUG_PAGEALLOC ) || defined(CONFIG_KFENCE ) -static void ipte_range (pte_t *pte,unsigned long address,int nr) +static void ipte_range (hw_pte_t *pte,unsigned long address,int nr) {int i;
@@ -439,7 +440,8 @@ static void ipte_range(pte_t *pte, unsigned long address, int nr)void __kernel_map_pages (struct page *page,int numpages,int enable) {unsigned long address; -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte;int nr, i, j;for (i =0 ; i < numpages;) {
arch/s390/mm/pgtable.c +13 -13
@@ -37,7 +37,7 @@ pgprot_t pgprot_writecombine(pgprot_t prot)EXPORT_SYMBOL_GPL (pgprot_writecombine);static inline void ptep_ipte_local (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,int nodat) +hw_pte_t *ptep,int nodat) {unsigned long opt, asce;
@@ -57,7 +57,7 @@ static inline void ptep_ipte_local(struct mm_struct *mm, unsigned long addr, }static inline void ptep_ipte_global (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,int nodat) +hw_pte_t *ptep,int nodat) {unsigned long opt, asce;
@@ -77,12 +77,12 @@ static inline void ptep_ipte_global(struct mm_struct *mm, unsigned long addr, }static inline pte_t ptep_flush_direct (struct mm_struct *mm, -unsigned long addr,pte_t *ptep, +unsigned long addr,hw_pte_t *ptep,int nodat) {pte_t old; - old = *ptep; + old =ptep_get (ptep);if (unlikely (pte_val (old) & _PAGE_INVALID))return old;atomic_inc (&mm->context.flush_count);
@@ -96,12 +96,12 @@ static inline pte_t ptep_flush_direct(struct mm_struct *mm, }static inline pte_t ptep_flush_lazy (struct mm_struct *mm, -unsigned long addr,pte_t *ptep, +unsigned long addr,hw_pte_t *ptep,int nodat) {pte_t old; - old = *ptep; + old =ptep_get (ptep);if (unlikely (pte_val (old) & _PAGE_INVALID))return old;atomic_inc (&mm->context.flush_count);
@@ -116,7 +116,7 @@ static inline pte_t ptep_flush_lazy(struct mm_struct *mm, }pte_t ptep_xchg_direct (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t new) +hw_pte_t *ptep,pte_t new) {pte_t old;
@@ -132,7 +132,7 @@ EXPORT_SYMBOL(ptep_xchg_direct);* Caller must check that new PTE only differs in _PAGE_PROTECT HW bit, so that * RDP can be used instead of IPTE. See also comments at pte_allow_rdp(). */ -void ptep_reset_dat_prot (struct mm_struct *mm,unsigned long addr,pte_t *ptep, +void ptep_reset_dat_prot (struct mm_struct *mm,unsigned long addr,hw_pte_t *ptep,pte_t new) {preempt_disable ();
@@ -154,7 +154,7 @@ void ptep_reset_dat_prot(struct mm_struct *mm, unsigned long addr, pte_t *ptep,EXPORT_SYMBOL (ptep_reset_dat_prot);pte_t ptep_xchg_lazy (struct mm_struct *mm,unsigned long addr, -pte_t *ptep,pte_t new) +hw_pte_t *ptep,pte_t new) {pte_t old;
@@ -167,13 +167,13 @@ pte_t ptep_xchg_lazy(struct mm_struct *mm, unsigned long addr,EXPORT_SYMBOL (ptep_xchg_lazy);pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep) +hw_pte_t *ptep) {return ptep_flush_lazy (vma->vm_mm, addr, ptep,1 ); }void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr, -pte_t *ptep,pte_t old_pte,pte_t pte) +hw_pte_t *ptep,pte_t old_pte,pte_t pte) {set_pte (ptep, pte); }
@@ -333,7 +333,7 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp) {struct list_head *lh;pgtable_t pgtable; -pte_t *ptep; +hw_pte_t *ptep;assert_spin_locked (pmd_lockptr (mm, pmdp));
@@ -346,7 +346,7 @@ pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp)pmd_huge_pte (mm, pmdp) = (pgtable_t ) lh->next;list_del (lh); } - ptep = (pte_t *) pgtable; + ptep = (hw_pte_t *) pgtable;set_pte (ptep,__pte (_PAGE_INVALID)); ptep++;set_pte (ptep,__pte (_PAGE_INVALID));
arch/s390/mm/vmem.c +14 -12
@@ -66,14 +66,14 @@ void *vmem_crst_alloc(unsigned long val)return table; } -pte_t __ref *vmem_pte_alloc (void ) +hw_pte_t __ref *vmem_pte_alloc (void ) { -pte_t *pte; +hw_pte_t *pte;if (slab_is_available ()) - pte = (pte_t *)page_table_alloc (&init_mm); + pte = (hw_pte_t *)page_table_alloc (&init_mm);else - pte = (pte_t *)memblock_alloc (PAGE_SIZE ,PAGE_SIZE ); + pte = (hw_pte_t *)memblock_alloc (PAGE_SIZE ,PAGE_SIZE );if (!pte)return NULL ;memset64 ((u64 *)pte, _PAGE_INVALID,PTRS_PER_PTE );
@@ -168,7 +168,8 @@ static int __ref modify_pte_table(pmd_t *pmd, unsigned long addr, {unsigned long prot, pages =0 ;int ret = -ENOMEM ; -pte_t *pte, entry; +hw_pte_t *pte; +pte_t entry; prot =pgprot_val (PAGE_KERNEL ); pte =pte_offset_kernel (pmd, addr);
@@ -204,7 +205,7 @@ static int __ref modify_pte_table(pmd_t *pmd, unsigned long addr,static void try_free_pte_table (pmd_t *pmd,unsigned long start) { -pte_t *pte; +hw_pte_t *pte;int i;/* We can safely assume this is fully in 1:1 mapping & vmemmap area */
@@ -226,7 +227,7 @@ static int __ref modify_pmd_table(pud_t *pud, unsigned long addr,int ret = -ENOMEM ;pmd_t entry;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte; prot =pgprot_val (SEGMENT_KERNEL ); pmd =pmd_offset (pud, addr);
@@ -575,16 +576,16 @@ int vmem_add_mapping(unsigned long start, unsigned long size)* while traversing is an error, since the function is expected to be * called against virtual regions reserved for 4KB mappings only. */ -pte_t *vmem_get_alloc_pte (unsigned long addr,bool alloc) +hw_pte_t *vmem_get_alloc_pte (unsigned long addr,bool alloc) { -pte_t *ptep =NULL ; +hw_pte_t *ptep =NULL ;pud_t pud_entry;pmd_t pmd_entry;pgd_t *pgd;p4d_t *p4d;pud_t *pud;pmd_t *pmd; -pte_t *pte; +hw_pte_t *pte; pgd =pgd_offset_k (addr);if (pgd_none (pgdp_get (pgd))) {
@@ -635,7 +636,8 @@ pte_t *vmem_get_alloc_pte(unsigned long addr, bool alloc)int __vmem_map_4k_page (unsigned long addr,unsigned long phys,pgprot_t prot,bool alloc) { -pte_t *ptep, pte; +hw_pte_t *ptep; +pte_t pte;if (!IS_ALIGNED (addr,PAGE_SIZE ))return -EINVAL ;
@@ -660,7 +662,7 @@ int vmem_map_4k_page(unsigned long addr, unsigned long phys, pgprot_t prot)void vmem_unmap_4k_page (unsigned long addr) { -pte_t *ptep; +hw_pte_t *ptep;mutex_lock (&vmem_mutex); ptep =virt_to_kpte (addr);
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 11/15] s390: Distinguish hardware and software PTEs
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] s390/mm: set_pte() discards _PAGE_UNUSED clearance --
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] s390/mm: set_pte() discards _PAGE_UNUSED clearance
--
commit a9d06a7ccc3dae14aa4eb7e5a46302fce251265b
Author: Alexander Gordeev <[email protected]>
s390: Distinguish hardware and software PTEs
This commit introduces the hw_pte_t type for hardware PTEs on s390, distinguishing them from software PTEs which continue to use the pte_t type. It updates various memory management functions and callbacks to enforce this type separation, similar to ARM64. It also enables the ARCH_HAS_HW_PTE_T configuration for the architecture.
Quoted 4 lines
arch/s390/include/asm/pgtable.h +0
-0
[ ... ]
arch/s390/include/asm/pgtable.h static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) · line 987
Quoted 13 lines
arch/s390/include/asm/pgtable.h +4
-2@@ -978,17 +978,19 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)WRITE_ONCE (*pmdp, pmd); } -static inline void set_pte (pte_t *ptep,pte_t pte) +static inline void set_pte (hw_pte_t *ptep,pte_t pte) { +hw_pte_t hwpte = (hw_pte_t ) { (pte) }; +if (pte_present (pte)) pte =clear_pte_bit (pte,__pgprot (_PAGE_UNUSED)); -WRITE_ONCE (*ptep, pte); +WRITE_ONCE (*ptep, hwpte);
[Severity: High]
Does this discard the _PAGE_UNUSED bit clearance?
Because hwpte is initialized before pte is potentially modified, the WRITE_ONCE call writes the original snapshot to the page table, ignoring the update from clear_pte_bit.
If this _PAGE_UNUSED bit incorrectly remains set on an active page, could the hypervisor (z/VM or KVM) reclaim its backing physical memory and cause silent guest data loss or memory corruption?
Quoted 3 lines
arch/s390/include/asm/pgtable.h +0
-0}#define ptep_get ptep_get
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
Alexander Gordeev
maintainerRe: [PATCH v8 11/15] s390: Distinguish hardware and software PTEs
Quoted 25 lines · On Wed, Oct 07, 2026 at 11:57:13AM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] s390/mm:set_pte()discards_PAGE_UNUSEDclearance
--commit
a9d06a7ccc3dae14aa4eb7e5a46302fce251265b
Author: Alexander Gordeev <[email protected]>
s390: Distinguish hardware and software PTEsThis commit introduces the
hw_pte_ttype for hardware PTEs on s390, distinguishing them from software PTEs which continue to use thepte_ttype. It updates various memory management functions and callbacks to enforce this type separation, similar to ARM64. It also enables theARCH_HAS_HW_PTE_Tconfiguration for the architecture.Quoted 4 lines
arch/s390/include/asm/pgtable.h +0
-0[ ... ]
Quoted 4 lines
arch/s390/include/asm/pgtable.h +0
-0@@ -978,17 +978,19 @@ static inline void set_pmd(pmd_t *pmdp, pmd_t pmd)WRITE_ONCE (*pmdp, pmd); }
Quoted 12 lines
arch/s390/include/asm/pgtable.h static inline void set_pmd(pmd_t *pmdp, pmd_t pmd) · line 987
Quoted 9 lines
arch/s390/include/asm/pgtable.h +4
-2-static inline void set_pte (pte_t *ptep,pte_t pte) +static inline void set_pte (hw_pte_t *ptep,pte_t pte) { +hw_pte_t hwpte = (hw_pte_t ) { (pte) }; +if (pte_present (pte)) pte =clear_pte_bit (pte,__pgprot (_PAGE_UNUSED)); -WRITE_ONCE (*ptep, pte); +WRITE_ONCE (*ptep, hwpte);[Severity: High]
Does this discard the_PAGE_UNUSEDbit clearance?
Yes, obviously. Sadly, not to me ;(
Quoted 14 lines
Because hwpte is initialized before pte is potentially modified, the
WRITE_ONCEcall writes the original snapshot to the page table, ignoring the update fromclear_pte_bit.If this
_PAGE_UNUSEDbit incorrectly remains set on an active page, could the hypervisor (z/VM or KVM) reclaim its backing physical memory and cause silent guest data loss or memory corruption?Quoted 3 lines
arch/s390/include/asm/pgtable.h +0
-0}#define ptep_get ptep_getSignature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=11
kernel test robot
build botRe: [PATCH v8 11/15] s390: Distinguish hardware and software PTEs
In reply to Alexander Gordeevkernel test robot noticed the following build errors:
Hi Alexander,
kernel test robot noticed the following build errors:
[auto build test ERROR on linus/master]
[also build test ERROR on v7.3-rc6]
[cannot apply to akpm-mm/mm-everything s390/features kvms390/next next-20261006] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Alexander-Gordeev/mm-introduce-hw_pte_t-for-PTE-table-storage/20261007-134144 base: linus/master patch link: https://lore.kernel.org/r/c22ed0a2d9ff0f4fff69a17332a377c460d993dc.1791365932.git.agordeev%40linux.ibm.com patch subject: [PATCH v8 11/15] s390: Distinguish hardware and software PTEs config: s390-allmodconfig (https://download.01.org/0day-ci/archive/20261008/[email protected]/config) compiler: clang version 24.0.0git (https://github.com/llvm/llvm-project 80cd965b6323b208519f8ab1d5ee86e4ea9862f4) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261008/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
>> mm/damon/vaddr.c:303:45: error: incompatible pointer types passing 'pte_t *' to parameter of type 'hw_pte_t *' (aka 'struct __hw_pte_t *') [-Wincompatible-pointer-types]
303 | *entry = huge_ptep_get_and_clear(mm, addr, pte, psize);
| ^~~
arch/s390/include/asm/hugetlb.h:34:40: note: passing argument to parameter 'ptep' here
34 | unsigned long addr, hw_pte_t *ptep,
| ^
mm/damon/vaddr.c:305:28: error: incompatible pointer types passing 'pte_t *' to parameter of type 'hw_pte_t *' (aka 'struct __hw_pte_t *') [-Wincompatible-pointer-types]
305 | set_huge_pte_at(mm, addr, pte, *entry, psize);
| ^~~
arch/s390/include/asm/hugetlb.h:22:18: note: passing argument to parameter 'ptep' here
22 | hw_pte_t *ptep, pte_t pte, unsigned long sz);
| ^
>> mm/damon/vaddr.c:318:40: error: incompatible pointer types passing 'hw_pte_t *' (aka 'struct __hw_pte_t *') to parameter of type 'pte_t *' [-Wincompatible-pointer-types]
318 | referenced = damon_hugetlb_ptep_mkold(pte, mm, vma, addr, &entry);
| ^~~ mm/damon/vaddr.c:296:45: note: passing argument to parameter 'pte' here
296 | static bool damon_hugetlb_ptep_mkold(pte_t *pte, struct mm_struct *mm,
| ^
3 errors generated.
vim +303 mm/damon/vaddr.c
3f49584b262cf8 SeongJae Park 2021-09-07 294
49f4203aae06ba Baolin Wang 2022-01-14 295 #ifdef CONFIG_HUGETLB_PAGE
39c0ceedd54557 SJ Park 2026-09-07 296 static bool damon_hugetlb_ptep_mkold(pte_t *pte, struct mm_struct *mm,
39c0ceedd54557 SJ Park 2026-09-07 297 struct vm_area_struct *vma, unsigned long addr, pte_t *entry)
39c0ceedd54557 SJ Park 2026-09-07 298 {
39c0ceedd54557 SJ Park 2026-09-07 299 unsigned long psize = huge_page_size(hstate_vma(vma));
39c0ceedd54557 SJ Park 2026-09-07 300
39c0ceedd54557 SJ Park 2026-09-07 301 if (!pte_young(*entry))
39c0ceedd54557 SJ Park 2026-09-07 302 return false;
39c0ceedd54557 SJ Park 2026-09-07 @303 *entry = huge_ptep_get_and_clear(mm, addr, pte, psize);
39c0ceedd54557 SJ Park 2026-09-07 304 *entry = pte_mkold(*entry);
39c0ceedd54557 SJ Park 2026-09-07 305 set_huge_pte_at(mm, addr, pte, *entry, psize);
39c0ceedd54557 SJ Park 2026-09-07 306 return true;
39c0ceedd54557 SJ Park 2026-09-07 307 }
39c0ceedd54557 SJ Park 2026-09-07 308
6ecdfd6e16f5f4 Muhammad Usama Anjum 2026-10-07 309 static void damon_hugetlb_mkold(hw_pte_t *pte, struct mm_struct *mm,
49f4203aae06ba Baolin Wang 2022-01-14 310 struct vm_area_struct *vma, unsigned long addr)
49f4203aae06ba Baolin Wang 2022-01-14 311 {
49f4203aae06ba Baolin Wang 2022-01-14 312 bool referenced = false;
e6c0c03245b14d Christophe Leroy 2024-07-02 313 pte_t entry = huge_ptep_get(mm, addr, pte);
6b7cea90c82e10 Kefeng Wang 2022-12-30 314 struct folio *folio = pfn_folio(pte_pfn(entry));
49f4203aae06ba Baolin Wang 2022-01-14 315
6b7cea90c82e10 Kefeng Wang 2022-12-30 316 folio_get(folio);
49f4203aae06ba Baolin Wang 2022-01-14 317
39c0ceedd54557 SJ Park 2026-09-07 @318 referenced = damon_hugetlb_ptep_mkold(pte, mm, vma, addr, &entry);
49f4203aae06ba Baolin Wang 2022-01-14 319 if (mmu_notifier_clear_young(mm, addr,
49f4203aae06ba Baolin Wang 2022-01-14 320 addr + huge_page_size(hstate_vma(vma))))
49f4203aae06ba Baolin Wang 2022-01-14 321 referenced = true;
49f4203aae06ba Baolin Wang 2022-01-14 322
49f4203aae06ba Baolin Wang 2022-01-14 323 if (referenced)
6b7cea90c82e10 Kefeng Wang 2022-12-30 324 folio_set_young(folio);
49f4203aae06ba Baolin Wang 2022-01-14 325
6b7cea90c82e10 Kefeng Wang 2022-12-30 326 folio_set_idle(folio);
6b7cea90c82e10 Kefeng Wang 2022-12-30 327 folio_put(folio);
49f4203aae06ba Baolin Wang 2022-01-14 328 }
49f4203aae06ba Baolin Wang 2022-01-14 329 --
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Alexander Gordeev
maintainerPATCH v8 12/15 mm: Make lazy MMU mode context-aware
In reply to Alexander GordeevLazy MMU mode is assumed to be context-independent, in the sense that it does not need any additional information while operating. However, the s390 architecture benefits from knowing the exact page table entries being modified.
Introduce lazy_mmu_mode_enable_with_ptes(), which is provided with the process address space and the page table being operated on. This information is required to enable s390-specific optimizations.
The function takes parameters that are typically passed to page- table level walkers, which implies that the span of PTE entries never crosses a page table boundary.
Architectures that do not require such information simply do not need to define the lazy_mmu_mode_enable_with_ptes() callback.
--- fs/proc/task_mmu.c | 2 +- include/linux/pgtable.h | 46 +++++++++++++++++++++++++++++++++++++++++ mm/madvise.c | 8 +++---- mm/memory.c | 8 +++---- mm/mprotect.c | 2 +- mm/mremap.c | 2 +- mm/vmalloc.c | 6 +++--- 7 files changed, 60 insertions(+), 14 deletions(-)
fs/proc/task_mmu.c +1 -1
@@ -2884,7 +2884,7 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start,return 0 ; } -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (vma->vm_mm, start, end, start_pte);if ((p->arg.flags &PM_SCAN_WP_MATCHING ) && !p->vec_out) {/* Fast path for performing exclusive WP */
include/linux/pgtable.h +46 -0
58 lines not on this page · raw message
mm/madvise.c +4 -4
@@ -462,7 +462,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (!start_pte)return 0 ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);for (; addr < end; pte += nr, addr += nr *PAGE_SIZE ) { nr =1 ; ptent =ptep_get (pte);
@@ -517,7 +517,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,if (!start_pte)break ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);if (!err) nr =0 ;continue ;
@@ -685,7 +685,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,if (!start_pte)return 0 ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);for (; addr != end; pte += nr, addr +=PAGE_SIZE * nr) { nr =1 ; ptent =ptep_get (pte);
@@ -745,7 +745,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr,if (!start_pte)break ;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte);if (!err) nr =0 ;continue ;
mm/memory.c +4 -4
@@ -1354,7 +1354,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma,spin_lock_nested (src_ptl,SINGLE_DEPTH_NESTING ); orig_src_pte = src_pte; orig_dst_pte = dst_pte; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (src_mm, addr, end, src_pte); do { nr = 1;
@@ -2053,7 +2053,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,return addr;flush_tlb_batched_pending (mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, start_pte); do { bool any_skipped = false;
@@ -3051,7 +3051,7 @@ static int remap_pte_range(struct mm_struct *mm, pmd_t *pmd, mapped_pte = pte =pte_alloc_map_lock (mm, pmd, addr, &ptl);if (!pte)return -ENOMEM ; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, mapped_pte); do { BUG_ON(!pte_none(ptep_get(pte))); if (!pfn_modify_allowed(pfn, prot)) {
@@ -3463,7 +3463,7 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd,return -EINVAL ; } -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, addr, end, mapped_pte);if (fn) { do {
mm/mprotect.c +1 -1
@@ -351,7 +351,7 @@ static long change_pte_range(struct mmu_gather *tlb, is_private_single_threaded =vma_is_single_threaded_private (vma);flush_tlb_batched_pending (vma->vm_mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (vma->vm_mm, addr, end, pte); do { nr_ptes = 1; oldpte = ptep_get(pte);
mm/mremap.c +1 -1
@@ -260,7 +260,7 @@ static int move_ptes(struct pagetable_move_control *pmc,if (new_ptl != old_ptl)spin_lock_nested (new_ptl,SINGLE_DEPTH_NESTING );flush_tlb_batched_pending (vma->vm_mm); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (mm, old_addr, old_end, old_ptep);for (; old_addr < old_end; old_ptep += nr_ptes, old_addr += nr_ptes *PAGE_SIZE , new_ptep += nr_ptes, new_addr += nr_ptes *PAGE_SIZE ) {
mm/vmalloc.c +3 -3
@@ -110,7 +110,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,if (!pte)return -ENOMEM ; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (&init_mm, addr, end, pte); do { if (unlikely(!pte_none(ptep_get(pte)))) {
@@ -394,7 +394,7 @@ static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end,unsigned long size =PAGE_SIZE ; pte =pte_offset_kernel (pmd, addr); -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (&init_mm, addr, end, pte); do { #ifdef CONFIG_HUGETLB_PAGE
@@ -561,7 +561,7 @@ static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr,if (!pte)return -ENOMEM ; -lazy_mmu_mode_enable (); +lazy_mmu_mode_enable_with_ptes (&init_mm, addr, end, pte); do { struct page *page = pages[*nr];
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 12/15] mm: Make lazy MMU mode context-aware
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 13 lines
Lazy MMU mode is assumed to be context-independent, in the sense that it does not need any additional information while operating. However, the s390 architecture benefits from knowing the exact page table entries being modified.
Introduce
lazy_mmu_mode_enable_with_ptes(), which is provided with the process address space and the page table being operated on. This information is required to enable s390-specific optimizations.The function takes parameters that are typically passed to page- table level walkers, which implies that the span of PTE entries never crosses a page table boundary.
Quoted 6 lines
Architectures that do not require such information simply do not need to define the
lazy_mmu_mode_enable_with_ptes()callback.Reviewed-by Kevin Brodsky <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=12
Alexander Gordeev
maintainerPATCH v8 13/15 s390/mm: Batch PTE updates in lazy MMU mode
In reply to Alexander GordeevMake use of the IPTE instruction's "Additional Entries" field to invalidate multiple PTEs in one go while in lazy MMU mode. This is the mode in which many memory-management system calls (like mremap(), mprotect(), etc.) update memory attributes.
To achieve that, the set_pte() and ptep_get() primitives use a per-CPU cache to store and retrieve PTE values and apply the cached values to the real page table once lazy MMU mode is left.
The same is done for memory-management platform callbacks that would otherwise cause intense per-PTE IPTE traffic, reducing the number of IPTE instructions from up to PTRS_PER_PTE to a single instruction in the best case. The average reduction is of course smaller.
Since all existing page table iterators called in lazy MMU mode handle one table at a time, the per-CPU cache does not need to be larger than PTRS_PER_PTE entries. That also naturally aligns with the IPTE instruction, which must not cross a page table boundary.
Before this change, the system calls did:
lazy_mmu_mode_enable_with_ptes()
...
<update PTEs> // up to PTRS_PER_PTE single-IPTEs
...
lazy_mmu_mode_disable()With this change, the system calls do:
lazy_mmu_mode_enable_with_ptes()
...
<store new PTE values in the per-CPU cache>
...
lazy_mmu_mode_disable() // apply cache with one multi-IPTE
When applied to large memory ranges, some system calls show significant speedups:
mprotect() ~15x
munmap() ~3x
mremap() ~28xThe overall results depend on memory size and access patterns, but the change generally does not degrade performance.
In addition to a process-wide impact, the rework affects the whole Central Electronics Complex (CEC). Each (global) IPTE instruction initiates a quiesce state in a CEC, so reducing the number of IPTE calls relieves CEC-wide quiesce traffic.
In an extreme case of mprotect() contiguously triggering the quiesce state on four LPARs in parallel, measurements show ~25x fewer quiesce events.
If an interrupt arrives in the middle of enter_ipte_range() or leave_ipte_range() the state of the per-cpu struct ipte_range may be inconsistent. That could lead to crashes if the interrupt handler tries to access a PTE. e.g:
handle_softirqs()->...->free_percpu()->pcpu_chunk_addr_search()-> pcpu_addr_to_page()->vmalloc_to_page()->ptep_get()
To avoid that disable bottom halves on the lazy mmu mode entering or leaving.
Though unexpected nothing prevents a top half from accessing a PTE that belongs to an active lazy mmu mode range. In that case do VM_BUG_ON() and consider disabling the interrupts altogether if it ever hits.
--- arch/s390/Kconfig | 1 + arch/s390/include/asm/lowcore.h | 3 +- arch/s390/include/asm/pgtable.h | 163 ++++++++++-- arch/s390/mm/Makefile | 2 +- arch/s390/mm/lazy_mmu.c | 430 ++++++++++++++++++++++++++++++++ arch/s390/mm/pgtable.c | 8 +- 6 files changed, 583 insertions(+), 24 deletions(-) create mode 100644 arch/s390/mm/lazy_mmu.c
arch/s390/Kconfig +1 -0
@@ -102,6 +102,7 @@ config S390 select ARCH_HAS_GIGANTIC_PAGE select ARCH_HAS_HW_PTE_T select ARCH_HAS_KCOV + select ARCH_HAS_LAZY_MMU_MODE select ARCH_HAS_MEMBARRIER_SYNC_CORE select ARCH_HAS_MEM_ENCRYPT select ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
arch/s390/include/asm/lowcore.h +2 -1
@@ -175,7 +175,8 @@ struct lowcore {__u32 return_lpswe;/* 0x0400 */ __u32 return_mcck_lpswe;/* 0x0404 */ -__u8 pad_0x040a[0x0e00 -0x0408 ];/* 0x0408 */ +__u8 lazy_mmu_count;/* 0x0408 */ +__u8 pad_0x0409[0x0e00 -0x0409 ];/* 0x0409 */ /* * 0xe00 contains the address of the IPL Parameter Information
arch/s390/include/asm/pgtable.h +145 -18
228 lines not on this page · raw message
arch/s390/mm/Makefile +1 -1
@@ -5,7 +5,7 @@ CONTEXT_ANALYSIS := y -obj-y := init.o fault.o extmem.o mmap.o vmem.o maccess.o +obj-y := init.o fault.o extmem.o mmap.o vmem.o maccess.o lazy_mmu.o obj-y += page-states.o pageattr.o pgtable.o pgalloc.o extable.o obj-$(CONFIG_CMM) += cmm.o
arch/s390/mm/lazy_mmu.c new file +430 -0
430 lines not on this page · raw message
arch/s390/mm/pgtable.c +4 -4
@@ -166,14 +166,14 @@ pte_t ptep_xchg_lazy(struct mm_struct *mm, unsigned long addr, }EXPORT_SYMBOL (ptep_xchg_lazy); -pte_t ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr, -hw_pte_t *ptep) +pte_t ___ptep_modify_prot_start (struct vm_area_struct *vma,unsigned long addr, +hw_pte_t *ptep) {return ptep_flush_lazy (vma->vm_mm, addr, ptep,1 ); } -void ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr, -hw_pte_t *ptep,pte_t old_pte,pte_t pte) +void ___ptep_modify_prot_commit (struct vm_area_struct *vma,unsigned long addr, +hw_pte_t *ptep,pte_t old_pte,pte_t pte) {set_pte (ptep, pte); }
Signature · 1 line
2.53.0
[email protected]
AI reviewThank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider: - [High] s390/mm: NULL pointer deref on CPU hotplug lazy MMU rollback failure - [High] s390/mm: Hardirq…
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] s390/mm: NULL pointer deref on CPU hotplug lazy MMU rollback failure
- [High] s390/mm: Hardirq torn read of ipte_range bounds causes OOB memory write
- [High] s390/mm: ptep_get_and_clear_full() bypasses fullmm optimization in lazy MMU mode
- [Medium] s390/mm: ptep_get_and_clear_full() bypasses fullmm TLB optimization --
commit 93b412133f9ff3548a7d0d31445ca8ff8b51da29
Author: Alexander Gordeev <[email protected]>
s390/mm: Batch PTE updates in lazy MMU mode
This patch introduces a per-CPU cache to batch PTE updates (via the IPTE instruction) during lazy MMU mode. This reduces CEC-wide quiesce traffic for memory-management system calls like mprotect(), munmap(), and mremap(), yielding significant performance improvements.
Quoted 4 lines
arch/s390/include/asm/pgtable.h +0
-0
[ ... ]
Quoted 7 lines
arch/s390/include/asm/pgtable.h +0
-3@@ -1263,11 +1383,13 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, {pte_t res; -if (full) { - res =ptep_get (ptep); -set_pte (ptep,__pte (_PAGE_INVALID));
arch/s390/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1393
Quoted 10 lines
arch/s390/include/asm/pgtable.h +7
-2- }else { - res =ptep_xchg_lazy (mm, addr, ptep,__pte (_PAGE_INVALID)); +if (!lazy_mmu_ptep_get_and_clear (addr, ptep, &res)) { +if (full) { + res =__ptep_get (ptep); +__set_pte (ptep,__pte (_PAGE_INVALID)); + }else { + res =ptep_xchg_lazy (mm, addr, ptep,__pte (_PAGE_INVALID)); + } }
[Severity: High]
Does intercepting the clear operation via lazy_mmu_ptep_get_and_clear() before checking the full parameter bypass the fullmm optimization?
During process teardown (e.g. exit_mmap()), fullmm is set to 1 to skip individual IPTE (TLB flush) instructions because a full MM TLB flush will be performed at the end. By unconditionally caching the clear operation in the lazy MMU cache here, leave_ipte_range() will later issue batch IPTE instructions for these entries when lazy MMU mode is disabled. This appears to degrade the O(1) TLB teardown into O(N) CEC-wide synchronizing IPTE instructions.
[ ... ]
Quoted 5 lines
arch/s390/mm/lazy_mmu.c new file +0
-0
[ ... ]
arch/s390/mm/lazy_mmu.c line 105
Quoted 9 lines
arch/s390/mm/lazy_mmu.c new file +9
-0+static void enter_ipte_range (struct mm_struct *mm, +unsigned long addr,unsigned long end,hw_pte_t *pte) +{ +struct ipte_range *range; + +if (!static_branch_likely (&lazy_mmu_enabled)) +return ; + + range =get_cpu_var (ipte_range);
[Severity: High]
If lazy_mmu_alloc() fails during CPU hotplug, could this retrieve a NULL ipte_range pointer?
When a lazy_mmu_alloc() failure triggers a CPU hotplug rollback, lazy_mmu_cpu_offline() frees the memory but does not disable the globally active lazy_mmu_enabled static branch. Subsequent lazy MMU entries on that CPU would bypass the static branch check above, retrieve a NULL ipte_range, and dereference it below.
arch/s390/mm/lazy_mmu.c line 116
Quoted 11 lines
arch/s390/mm/lazy_mmu.c new file +11
-0+local_bh_disable (); +get_lowcore ()->lazy_mmu_count++; + +if (mm_is_protected (mm)) { +local_bh_enable (); +return ; + } + + range->mm = mm; + range->base_addr = addr; + range->base_end = end;
[Severity: High]
Can a hardirq observe a torn state here?
The base_addr and base_end fields are updated sequentially. If a hardirq or NMI strikes after base_addr is set but while base_end is still 0, a concurrent execution of this_ipte_range() would observe base_end == 0.
This would cause the calculation:nr_ptes = (range->base_end - range->base_addr) / PAGE_SIZE;
to underflow to a massive value, bypassing the bounds validation and allowing an out-of-bounds memory write to range->cache[idx].
Quoted 4 lines
arch/s390/mm/lazy_mmu.c new file +4
-0+ range->base_pte = pte; + +local_bh_enable (); +}
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Alexander Gordeev
maintainerQuoted 44 lines · On Wed, Oct 07, 2026 at 12:06:03PM +0000, [email protected] wrote:
Thank you for your contribution! Sashiko AI review found 4 potential issue(s) to consider:
- [High] s390/mm: NULL pointer deref on CPU hotplug lazy MMU rollback failure
- [High] s390/mm: Hardirq torn read ofipte_rangebounds causes OOB memory write
- [High] s390/mm:ptep_get_and_clear_full()bypasses fullmm optimization in lazy MMU mode
- [Medium] s390/mm:ptep_get_and_clear_full()bypasses fullmm TLB optimization --commit
93b412133f9ff3548a7d0d31445ca8ff8b51da29
Author: Alexander Gordeev <[email protected]>s390/mm: Batch PTE updates in lazy MMU mode
This patch introduces a per-CPU cache to batch PTE updates (via the IPTE instruction) during lazy MMU mode. This reduces CEC-wide quiesce traffic for memory-management system calls like
mprotect(),munmap(), andmremap(), yielding significant performance improvements.Quoted 4 lines
arch/s390/include/asm/pgtable.h +0
-0[ ... ]
Quoted 7 lines
arch/s390/include/asm/pgtable.h +0
-3@@ -1263,11 +1383,13 @@ static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, {pte_t res; -if (full) { - res =ptep_get (ptep); -set_pte (ptep,__pte (_PAGE_INVALID));arch/s390/include/asm/pgtable.h static inline pte_t ptep_get_and_clear_full(struct mm_struct *mm, · line 1393
Quoted 10 lines
arch/s390/include/asm/pgtable.h +7
-2- }else { - res =ptep_xchg_lazy (mm, addr, ptep,__pte (_PAGE_INVALID)); +if (!lazy_mmu_ptep_get_and_clear (addr, ptep, &res)) { +if (full) { + res =__ptep_get (ptep); +__set_pte (ptep,__pte (_PAGE_INVALID)); + }else { + res =ptep_xchg_lazy (mm, addr, ptep,__pte (_PAGE_INVALID)); + } }[Severity: High]
Does intercepting the clear operation vialazy_mmu_ptep_get_and_clear()before checking the full parameter bypass the fullmm optimization?
Quoted 7 lines
During process teardown (e.g.
exit_mmap()), fullmm is set to 1 to skip individual IPTE (TLB flush) instructions because a full MM TLB flush will be performed at the end. By unconditionally caching the clear operation in the lazy MMU cache here,leave_ipte_range()will later issue batch IPTE instructions for these entries when lazy MMU mode is disabled. This appears to degrade the O(1) TLB teardown into O(N) CEC-wide synchronizing IPTE instructions.
We can not keep the full = 1 optimization in lazy mode, since nothing prevents other primitives from access a ranged PTE in cache before/after ptep_get_and_clear_full(..., full = 1) updated the PTE directly.
The good news in practice exit() would normally get called after threads on other CPUs have already been shut down, so the final IPTEs going to be local to the executing CPU rather than global.
Quoted 21 lines
[ ... ]
Quoted 5 lines
arch/s390/mm/lazy_mmu.c new file +0
-0[ ... ]
arch/s390/mm/lazy_mmu.c line 105
Quoted 9 lines
arch/s390/mm/lazy_mmu.c new file +9
-0+static void enter_ipte_range (struct mm_struct *mm, +unsigned long addr,unsigned long end,hw_pte_t *pte) +{ +struct ipte_range *range; + +if (!static_branch_likely (&lazy_mmu_enabled)) +return ; + + range =get_cpu_var (ipte_range);[Severity: High]
Iflazy_mmu_alloc()fails during CPU hotplug, could this retrieve a NULLipte_rangepointer?
Quoted 5 lines
When a
lazy_mmu_alloc()failure triggers a CPU hotplug rollback,lazy_mmu_cpu_offline()frees the memory but does not disable the globally activelazy_mmu_enabledstatic branch. Subsequent lazy MMU entries on that CPU would bypass the static branch check above, retrieve a NULLipte_range, and dereference it below.
There will be no subsequent lazy MMU mode entries on a CPU that failed lazy_mmu_alloc(), since such CPU will never get onlined.
Quoted 15 lines
arch/s390/mm/lazy_mmu.c line 116
Quoted 11 lines
arch/s390/mm/lazy_mmu.c new file +11
-0+local_bh_disable (); +get_lowcore ()->lazy_mmu_count++; + +if (mm_is_protected (mm)) { +local_bh_enable (); +return ; + } + + range->mm = mm; + range->base_addr = addr; + range->base_end = end;[Severity: High]
Can a hardirq observe a torn state here?
Quoted 3 lines
The
base_addrandbase_endfields are updated sequentially. If a hardirq or NMI strikes afterbase_addris set but whilebase_endis still 0, a concurrent execution ofthis_ipte_range()would observebase_end== 0.
The contract is a hardirq handler is not allowed to access the ranged PTEs while in the lazy MMU mode. That is what this_ipte_range() -> VM_BUG_ON(in_hardirq()) check is for.
Should this contract is ever get changed, enter_ipte_*range() and leave_ipte_range() will have to disable the top halves as well.
Quoted 13 lines
This would cause the calculation:
nr_ptes= (range->base_end-range->base_addr) /PAGE_SIZE;to underflow to a massive value, bypassing the bounds validation and allowing an out-of-bounds memory write to range->cache[idx].
Quoted 4 lines
arch/s390/mm/lazy_mmu.c new file +4
-0+ range->base_pte = pte; + +local_bh_enable (); +}Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=13
Alexander Gordeev
maintainerPATCH v8 14/15 mm/kasan: Introduce helpers for lazy MMU mode sanitizer
In reply to Alexander GordeevProvide helpers that allow architectures implement illegitimate PTE direct accesses while the lazy MMU mode is enabled, such as:
pte_t pte = *ptep; *ptep = pte;
By contrast, these would have to be:
pte_t pte = ptep_get(ptep); set_pte(ptep, pte);
The direct PTE accesses pose a real issue on s390.
--- include/linux/kasan.h | 19 +++++++++++++++++-- mm/kasan/common.c | 14 ++++++++++++++ mm/kasan/kasan.h | 2 ++ mm/kasan/report_generic.c | 3 +++ 4 files changed, 36 insertions(+), 2 deletions(-)
include/linux/kasan.h +17 -2
@@ -6,6 +6,7 @@#include <linux/kasan-enabled.h> #include <linux/kasan-tags.h> #include <linux/kernel.h> +#include <linux/pgtable.h> #include <linux/static_key.h> #include <linux/types.h>
@@ -35,8 +36,6 @@ typedef unsigned int __bitwise kasan_vmalloc_flags_t;#if defined(CONFIG_KASAN_GENERIC ) || defined(CONFIG_KASAN_SW_TAGS ) -#include <linux/pgtable.h> -/* Software KASAN implementations use shadow memory. */ #ifdef CONFIG_KASAN_SW_TAGS
@@ -134,6 +133,20 @@ static __always_inline void kasan_poison_slab(struct slab *slab)__kasan_poison_slab (slab); } +void __kasan_poison_pte (hw_pte_t *pte,int nr); +static __always_inline void kasan_poison_pte(hw_pte_t *pte, int nr) +{ +if (kasan_enabled ()) +__kasan_poison_pte (pte, nr); +} + +void __kasan_unpoison_pte (hw_pte_t *pte,int nr); +static __always_inline void kasan_unpoison_pte(hw_pte_t *pte, int nr) +{ +if (kasan_enabled ()) +__kasan_unpoison_pte (pte, nr); +} +void __kasan_unpoison_new_object (struct kmem_cache *cache,void *object);/** * kasan_unpoison_new_object - Temporarily unpoison a new slab object.
@@ -414,6 +427,8 @@ static inline bool kasan_unpoison_pages(struct page *page, unsigned int order,return false; }static inline void kasan_poison_slab (struct slab *slab) {} +static inline void kasan_poison_pte (hw_pte_t *pte,int nr) {} +static inline void kasan_unpoison_pte (hw_pte_t *pte,int nr) {}static inline void kasan_unpoison_new_object (struct kmem_cache *cache,void *object) {} static inline void kasan_poison_new_object(struct kmem_cache *cache,
mm/kasan/common.c +14 -0
@@ -174,6 +174,20 @@ void __kasan_poison_slab(struct slab *slab) KASAN_SLAB_REDZONE, false); } +void __kasan_poison_pte (hw_pte_t *pte,int nr) +{ +if (IS_ALIGNED (sizeof (*pte),KASAN_GRANULE_SIZE )) +kasan_poison (pte,sizeof (*pte) * nr,KASAN_LAZY_MMU_PTE , false); +} +EXPORT_SYMBOL_GPL (__kasan_poison_pte); + +void __kasan_unpoison_pte (hw_pte_t *pte,int nr) +{ +if (IS_ALIGNED (sizeof (*pte),KASAN_GRANULE_SIZE )) +kasan_unpoison (pte,sizeof (*pte) * nr, false); +} +EXPORT_SYMBOL_GPL (__kasan_unpoison_pte); +void __kasan_unpoison_new_object (struct kmem_cache *cache,void *object) {kasan_unpoison (object, cache->object_size, false);
mm/kasan/kasan.h +2 -0
@@ -144,12 +144,14 @@ static inline bool kasan_requires_meta(void)#define KASAN_PAGE_REDZONE 0xFE/* redzone for kmalloc_large allocation */ #define KASAN_SLAB_REDZONE 0xFC/* redzone for slab object */ #define KASAN_SLAB_FREE 0xFB/* freed slab object */ +#define KASAN_LAZY_MMU_PTE 0xFD/* direct pte access in lazy mmu mode */ #define KASAN_VMALLOC_INVALID 0xF8/* inaccessible space in vmap area */ #else #define KASAN_PAGE_FREE KASAN_TAG_INVALID#define KASAN_PAGE_REDZONE KASAN_TAG_INVALID#define KASAN_SLAB_REDZONE KASAN_TAG_INVALID#define KASAN_SLAB_FREE KASAN_TAG_INVALID +#define KASAN_LAZY_MMU_PTE KASAN_TAG_INVALID#define KASAN_VMALLOC_INVALID KASAN_TAG_INVALID/* only used for SW_TAGS */ #endif
mm/kasan/report_generic.c +3 -0
@@ -113,6 +113,9 @@ static const char *get_shadow_bug_type(struct kasan_report_info *info)case KASAN_SLAB_FREE_META : bug_type ="slab-use-after-free" ;break ; +case KASAN_LAZY_MMU_PTE : + bug_type ="lazy-mmu-pte-access" ; +break ;case KASAN_ALLOCA_LEFT :case KASAN_ALLOCA_RIGHT : bug_type ="alloca-out-of-bounds" ;
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 9 lines
Provide helpers that allow architectures implement illegitimate PTE direct accesses while the lazy MMU mode is enabled, such as:
pte_t pte = *ptep; *ptep = pte;By contrast, these would have to be:
Quoted 7 lines
pte_t pte = ptep_get(ptep); set_pte(ptep, pte);The direct PTE accesses pose a real issue on s390.
Suggested-by Ilya Leoshkevich <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=14
kernel test robot
build botRe: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer
In reply to Alexander Gordeevkernel test robot noticed the following build errors:
Hi Alexander,
kernel test robot noticed the following build errors:
[auto build test ERROR on linus/master]
[also build test ERROR on v7.3-rc6]
[cannot apply to akpm-mm/mm-everything s390/features kvms390/next next-20261006] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Alexander-Gordeev/mm-introduce-hw_pte_t-for-PTE-table-storage/20261007-134144 base: linus/master patch link: https://lore.kernel.org/r/e42d9f02ec1f8478a50fce424b95d0e7ff6a359f.1791365932.git.agordeev%40linux.ibm.com patch subject: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer config: mips-bmips_stb_defconfig (https://download.01.org/0day-ci/archive/20261008/[email protected]/config) compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261008/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from arch/mips/kernel/asm-offsets.c:14:
In file included from include/linux/compat.h:17:
In file included from include/linux/fs.h:45:
In file included from include/linux/slab.h:276:
In file included from include/linux/kasan.h:9:
In file included from include/linux/pgtable.h:6:
In file included from arch/mips/include/asm/pgtable.h:14:
In file included from arch/mips/include/asm/pgtable-32.h:22:
In file included from arch/mips/include/asm/highmem.h:24:
In file included from include/linux/interrupt.h:11:
In file included from include/linux/hardirq.h:11:
In file included from arch/mips/include/asm/hardirq.h:16:
In file included from include/asm-generic/hardirq.h:17:
include/linux/irq.h:1211:2: error: call to undeclared function 'kfree'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
1211 | kfree(gc);
| ^
In file included from arch/mips/kernel/asm-offsets.c:14:
In file included from include/linux/compat.h:17:
In file included from include/linux/fs.h:45:
>> include/linux/slab.h:595:6: error: conflicting types for 'kfree'
595 | void kfree(const void *objp);
| ^
include/linux/irq.h:1211:2: note: previous implicit declaration is here
1211 | kfree(gc);
| ^
In file included from arch/mips/kernel/asm-offsets.c:26:
In file included from include/linux/kvm_host.h:5:
In file included from include/linux/entry-virt.h:7:
In file included from include/linux/syscalls.h:95:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:6:
In file included from include/linux/ring_buffer.h:6:
include/linux/poll.h:134:27: warning: division by zero is undefined [-Wdivision-by-zero]
134 | M(RDNORM) | M(RDBAND) | M(WRNORM) | M(WRBAND) |
| ^~~~~~~~~
include/linux/poll.h:132:32: note: expanded from macro 'M'
132 | #define M(X) (__force __poll_t)__MAP(val, POLL##X, (__force __u16)EPOLL##X)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/poll.h:118:51: note: expanded from macro '__MAP'
118 | (from < to ? (v & from) * (to/from) : (v & from) / (from/to))
| ^ ~~~~~~~~~
include/linux/poll.h:134:39: warning: division by zero is undefined [-Wdivision-by-zero]
134 | M(RDNORM) | M(RDBAND) | M(WRNORM) | M(WRBAND) |
| ^~~~~~~~~
include/linux/poll.h:132:32: note: expanded from macro 'M'
132 | #define M(X) (__force __poll_t)__MAP(val, POLL##X, (__force __u16)EPOLL##X)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ include/linux/poll.h:118:51: note: expanded from macro '__MAP'
118 | (from < to ? (v & from) * (to/from) : (v & from) / (from/to))
| ^ ~~~~~~~~~
2 warnings and 2 errors generated.
make[3]: *** [scripts/Makefile.build:185: arch/mips/kernel/asm-offsets.s] Error 1 make[3]: Target 'prepare' not remade because of errors.
make[2]: *** [Makefile:1425: prepare0] Error 2
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +/kfree +595 include/linux/slab.h
7bd230a26648ac Suren Baghdasaryan 2024-03-21 594
72d67229f522e3 Kees Cook 2021-11-05 @595 void kfree(const void *objp); af92793e52c3a9 Alexei Starovoitov 2025-09-08 596 void kfree_nolock(const void *objp); 72d67229f522e3 Kees Cook 2021-11-05 597 void kfree_sensitive(const void *objp);
05a940656e1eb2 Kees Cook 2022-09-23 598
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer
In reply to Alexander Gordeevkernel test robot noticed the following build errors:
Hi Alexander,
kernel test robot noticed the following build errors:
[auto build test ERROR on linus/master]
[also build test ERROR on v7.3-rc6]
[cannot apply to akpm-mm/mm-everything s390/features kvms390/next next-20261006] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Alexander-Gordeev/mm-introduce-hw_pte_t-for-PTE-table-storage/20261007-134144 base: linus/master patch link: https://lore.kernel.org/r/e42d9f02ec1f8478a50fce424b95d0e7ff6a359f.1791365932.git.agordeev%40linux.ibm.com patch subject: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer config: arm-allnoconfig (https://download.01.org/0day-ci/archive/20261008/[email protected]/config) compiler: clang version 17.0.6 (https://github.com/llvm/llvm-project 6009708b4367171ccdbf4b5905cb6a803753fe18) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261008/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All errors (new ones prefixed by >>):
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:31:
In file included from include/linux/pgtable.h:6:
In file included from arch/arm/include/asm/pgtable.h:16:
In file included from arch/arm/include/asm/pgtable-nommu.h:13:
In file included from include/linux/slab.h:276:
>> include/linux/kasan.h:430:37: error: unknown type name 'hw_pte_t'
430 | static inline void kasan_poison_pte(hw_pte_t *pte, int nr) {}
| ^
include/linux/kasan.h:431:39: error: unknown type name 'hw_pte_t'
431 | static inline void kasan_unpoison_pte(hw_pte_t *pte, int nr) {}
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:98:11: warning: array index 3 is past the end of the array (that has type 'unsigned long[2]') [-Warray-bounds]
98 | return (set->sig[3] | set->sig[2] |
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:98:25: warning: array index 2 is past the end of the array (that has type 'unsigned long[2]') [-Warray-bounds]
98 | return (set->sig[3] | set->sig[2] |
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:114:11: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
114 | return (set1->sig[3] == set2->sig[3]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:114:27: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
114 | return (set1->sig[3] == set2->sig[3]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:115:5: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
115 | (set1->sig[2] == set2->sig[2]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:115:21: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
115 | (set1->sig[2] == set2->sig[2]) &&
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:157:1: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
157 | _SIG_SET_BINOP(sigorsets, _sig_or)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/signal.h:138:8: note: expanded from macro '_SIG_SET_BINOP'
138 | a3 = a->sig[3]; a2 = a->sig[2]; \
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:157:1: warning: array index 2 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
157 | _SIG_SET_BINOP(sigorsets, _sig_or)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/signal.h:138:24: note: expanded from macro '_SIG_SET_BINOP'
138 | a3 = a->sig[3]; a2 = a->sig[2]; \
| ^ ~
arch/arm/include/asm/signal.h:17:2: note: array 'sig' declared here
17 | unsigned long sig[_NSIG_WORDS];
| ^
In file included from arch/arm/kernel/asm-offsets.c:14:
In file included from include/linux/mm.h:36:
In file included from include/linux/rcuwait.h:6:
In file included from include/linux/sched/signal.h:6:
include/linux/signal.h:157:1: warning: array index 3 is past the end of the array (that has type 'const unsigned long[2]') [-Warray-bounds]
157 | _SIG_SET_BINOP(sigorsets, _sig_or)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
include/linux/signal.h:139:8: note: expanded from macro '_SIG_SET_BINOP'
139 | b3 = b->sig[3]; b2 = b->sig[2]; \vim +/hw_pte_t +430 include/linux/kasan.h
420
421 static inline void kasan_unpoison_range(const void *address, size_t size) {}
422 static inline void kasan_poison_pages(struct page *page, unsigned int order,
423 bool init) {}
424 static inline bool kasan_unpoison_pages(struct page *page, unsigned int order,
425 bool init)
426 {
427 return false;
428 }
429 static inline void kasan_poison_slab(struct slab *slab) {}
> 430 static inline void kasan_poison_pte(hw_pte_t *pte, int nr) {} 431 static inline void kasan_unpoison_pte(hw_pte_t *pte, int nr) {}
432 static inline void kasan_unpoison_new_object(struct kmem_cache *cache,
433 void *object) {}
434 static inline void kasan_poison_new_object(struct kmem_cache *cache,
435 void *object) {}
436 static inline void *kasan_init_slab_obj(struct kmem_cache *cache,
437 const void *object)
438 {
439 return (void *)object;
440 }
441
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
kernel test robot
build botRe: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer
In reply to Alexander Gordeevkernel test robot noticed the following build warnings:
Hi Alexander,
kernel test robot noticed the following build warnings:
[auto build test WARNING on linus/master]
[also build test WARNING on v7.3-rc6]
[cannot apply to akpm-mm/mm-everything s390/features kvms390/next next-20261006] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Alexander-Gordeev/mm-introduce-hw_pte_t-for-PTE-table-storage/20261007-134144 base: linus/master patch link: https://lore.kernel.org/r/e42d9f02ec1f8478a50fce424b95d0e7ff6a359f.1791365932.git.agordeev%40linux.ibm.com patch subject: [PATCH v8 14/15] mm/kasan: Introduce helpers for lazy MMU mode sanitizer config: mips-maltaup_xpa_defconfig (https://download.01.org/0day-ci/archive/20261008/[email protected]/config) compiler: mips-linux-gcc (GCC) 16.1.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20261008/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
In file included from include/asm-generic/hardirq.h:17,
from arch/mips/include/asm/hardirq.h:16,
from include/linux/hardirq.h:11,
from include/linux/interrupt.h:11,
from arch/mips/include/asm/highmem.h:24,
from arch/mips/include/asm/pgtable-32.h:22,
from arch/mips/include/asm/pgtable.h:14,
from include/linux/pgtable.h:6,
from include/linux/kasan.h:9,
from include/linux/slab.h:276,
from include/linux/fs.h:45,
from include/linux/compat.h:17,
from arch/mips/kernel/asm-offsets.c:14:
include/linux/irq.h: In function 'irq_free_generic_chip':
include/linux/irq.h:1211:9: error: implicit declaration of function 'kfree' [-Wimplicit-function-declaration]
1211 | kfree(gc);
| ^~~~~
include/linux/slab.h: At top level:
>> include/linux/slab.h:595:6: warning: conflicting types for 'kfree'; have 'void(const void *)'
595 | void kfree(const void *objp);
| ^~~~~
include/linux/irq.h:1211:9: note: previous implicit declaration of 'kfree' with type 'void(const void *)'
1211 | kfree(gc);
| ^~~~~ make[3]: *** [scripts/Makefile.build:185: arch/mips/kernel/asm-offsets.s] Error 1 make[3]: Target 'prepare' not remade because of errors.
make[2]: *** [Makefile:1425: prepare0] Error 2
make[2]: Target 'prepare' not remade because of errors.
make[1]: *** [Makefile:248: __sub-make] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:248: __sub-make] Error 2
make: Target 'prepare' not remade because of errors.
vim +595 include/linux/slab.h
7bd230a26648ac Suren Baghdasaryan 2024-03-21 594
72d67229f522e3 Kees Cook 2021-11-05 @595 void kfree(const void *objp); af92793e52c3a9 Alexei Starovoitov 2025-09-08 596 void kfree_nolock(const void *objp); 72d67229f522e3 Kees Cook 2021-11-05 597 void kfree_sensitive(const void *objp);
05a940656e1eb2 Kees Cook 2022-09-23 598
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
Alexander Gordeev
maintainerPATCH v8 15/15 s390/mm: Lazy MMU mode sanitizer
In reply to Alexander GordeevDetect PTE entries access in lazy MMU mode by means other than set_pte() and ptep_get() primitives, which would be a read hazard.
The access to kasan shadow memory from ptep_get_lockless() mistakenly hits invalid access in case a concurrent lazy MMU access to the same PTE is happening. To avoid that disable instrumentation for ptep_get_lockless() altogether.
--- arch/s390/include/asm/pgtable.h | 6 ++++++ arch/s390/mm/lazy_mmu.c | 27 +++++++++++++++++++++++---- 2 files changed, 29 insertions(+), 4 deletions(-)
arch/s390/include/asm/pgtable.h +6 -0
@@ -1063,6 +1063,12 @@ static inline void set_pte(hw_pte_t *ptep, pte_t pte)__set_pte (ptep, pte); } +#define ptep_get_lockless ptep_get_lockless +static inline __no_sanitize_address pte_t ptep_get_lockless(hw_pte_t *ptep) +{ +return __pte_from_hw (READ_ONCE (*ptep)); +} +static inline pte_t __ptep_get (hw_pte_t *ptep) {return __pte_from_hw (READ_ONCE (*ptep));
arch/s390/mm/lazy_mmu.c +23 -4
@@ -65,10 +65,13 @@ static int invalidate_pte_range(struct mm_struct *mm, unsigned long addr, }static void set_pte_range (struct mm_struct *mm,unsigned long addr, -hw_pte_t *ptep,hw_pte_t *end,pte_t *cache) +hw_pte_t *start,hw_pte_t *end,pte_t *cache) { -int i, nr_ptes; +int nr_ptes, nr_total = end - start; +hw_pte_t *ptep = start; +int i; +kasan_unpoison_pte (start, nr_total);while (ptep < end) { nr_ptes =invalidate_pte_range (mm, addr, ptep, end);
@@ -79,6 +82,7 @@ static void set_pte_range(struct mm_struct *mm, unsigned long addr, addr += nr_ptes *PAGE_SIZE ; } +kasan_poison_pte (start, nr_total); }static void enter_ipte_norange (void )
@@ -98,6 +102,7 @@ static void enter_ipte_range(struct mm_struct *mm, unsigned long addr, unsigned long end, hw_pte_t *pte) { struct ipte_range *range; +unsigned int nr_ptes;if (!static_branch_likely (&lazy_mmu_enabled))return ;
@@ -116,6 +121,9 @@ static void enter_ipte_range(struct mm_struct *mm, range->base_end = end; range->base_pte = pte; + nr_ptes = (range->base_end - range->base_addr) /PAGE_SIZE ; +kasan_poison_pte (range->base_pte, nr_ptes); +local_bh_enable (); }
@@ -125,6 +133,7 @@ static void leave_ipte_range(void)pte_t *start_cache, *cache;hw_pte_t *ptep, *start;struct ipte_range *range; +unsigned int nr_ptes;int start_idx;if (!static_branch_likely (&lazy_mmu_enabled))
@@ -163,6 +172,9 @@ static void leave_ipte_range(void) range->end_pte =NULL ; done: + nr_ptes = (range->base_end - range->base_addr) /PAGE_SIZE ; +kasan_unpoison_pte (range->base_pte, nr_ptes); + range->mm =NULL ; range->base_addr =0 ; range->base_end =0 ;
@@ -244,10 +256,17 @@ static void __ipte_range_set_pte(struct ipte_range *range, hw_pte_t *ptep, pte_tstatic pte_t __ipte_range_ptep_get (struct ipte_range *range,hw_pte_t *ptep) {unsigned int idx = ptep - range->base_pte; +pte_t pte;lockdep_assert_preemption_disabled (); -if (pte_val (range->cache[idx]) ==PTE_POISON ) -return __ptep_get (ptep); +if (pte_val (range->cache[idx]) ==PTE_POISON ) { +kasan_unpoison_pte (ptep,1 ); + pte =__ptep_get (ptep); +kasan_poison_pte (ptep,1 ); + +return pte; + } +return range->cache[idx]; }
Signature · 1 line
2.53.0
[email protected]
AI reviewRe: [PATCH v8 15/15] s390/mm: Lazy MMU mode sanitizer
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 11 lines
Detect PTE entries access in lazy MMU mode by means other than
set_pte()andptep_get()primitives, which would be a read hazard.The access to kasan shadow memory from
ptep_get_lockless()mistakenly hits invalid access in case a concurrent lazy MMU access to the same PTE is happening. To avoid that disable instrumentation forptep_get_lockless()altogether.Suggested-by Ilya Leoshkevich <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=15