arm64/mm: Convert pgtable READ_ONCE() as pxdp_get()

PATCH In review Anshuman Khandual 2 participants 8 messages Nov 5 – Nov 11

Subsystems (4)

Memory management MM core arm64 KASAN

lore

Share & tools

lore

Anshuman Khandual

author

2026-11-05 04:34 UTC #0 lore raw

v4 cover arm64/mm: Convert pgtable READ_ONCE() as pxdp_get()

Route READ_ONCE() accesses on pgtable entries via level specific generic MM pxdp_get() helpers which platforms could then override when required. These accessors on arm64, will help in ensuring page table accesses are performed in an atomic manner while reading 128 bit page table entries later. But for now they just default to READ_ONCE(), thus preserving existing behaviour on current D64 translation a well.

This is similar to generic MM where all atomic accesses have already been converted from READ_ONCE() into pgtable level specific accessors.

These are preparatory patches for D128 enablement, although standing their own for D64 translation as well. These patches are split from earlier D128 V1 series. There are no functional changes here, and besides mm kselftests do not show any regression.

https://lore.kernel.org/linux-mm/[email protected]/

This series applies on v7.4-rc1 and have been hosted below for reference.

[email protected]:linux-arm/linux-anshuman.git (arm64_pxdp_get_v4)

Changes in V4:

- Dropped all pxdp_get() platform overrides per Catalin
- Moved pmdp_test_and_clear_young() and [pud|p4d]_offset[_phys]() helpers inside arch/arm64/mm/mmu.c to avoid build problems
- Updated all commit messages as required

Changes in V3:

https://lore.kernel.org/all/[email protected]/

- Added overrides for pxdp_get() and replaced remaining READ_ONCE()
- Updated all commit messages as required
- Dropped previous tags

Changes in V2:

https://lore.kernel.org/all/[email protected]/

- Rebased on v7.3-rc4 along with all latest dependencies
- Updated commit messages per Ryan
- Added tags from Ryan

Changes in V1:

https://lore.kernel.org/all/[email protected]/

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

Cc Ryan Roberts <[email protected]>

Cc Mark Rutland <[email protected]>

Cc Lorenzo Stoakes <[email protected]>

Cc Andrew Morton <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Mike Rapoport <[email protected]>

Cc Linu Cherian <[email protected]>

Cc [email protected]

Anshuman Khandual (4):
arm64/mm: Use pmdp_get() for PMD accesses
arm64/mm: Use pudp_get() for PUD accesses
arm64/mm: Use p4dp_get() for P4D accesses
arm64/mm: Use pgdp_get() for PGD accesses

 arch/arm64/include/asm/pgtable.h | 38 +++----------
 arch/arm64/mm/fault.c            |  8 +--
 arch/arm64/mm/fixmap.c           |  6 +-
 arch/arm64/mm/hugetlbpage.c      | 10 ++--
 arch/arm64/mm/kasan_init.c       | 18 +++---
 arch/arm64/mm/mmu.c              | 97 ++++++++++++++++++++++----------
 arch/arm64/mm/pageattr.c         |  8 +--
 arch/arm64/mm/trans_pgd.c        | 14 ++---
 8 files changed, 108 insertions(+), 91 deletions(-)

Signature · 1 line

2.43.0

Anshuman Khandual

author

2026-11-05 04:34 UTC #1 lore raw

PATCH v4 1/4 arm64/mm: Use pmdp_get() for PMD accesses

Replace READ_ONCE() with pmdp_get() for PMD accesses in preparation for supporting both D64 and D128 translation table formats.

READ_ONCE() cannot currently be used for 128-bit page table entries on arm64 because it does not provide the required 128-bit single-copy atomicity, causing builds to fail for accesses wider than 64 bits.

Although LDP/STP provide the required atomicity when FEAT_LSE is available (as required by FEAT_D128), extending READ_ONCE() to support 128-bit accesses is undesirable. READ_ONCE() is a general-purpose API, so doing so could encourage other 128-bit users that would either fail to build in configurations without D128 support or, if D128 becomes a runtime option, silently permit tearing on systems without the required hardware support.

Instead, standardize PMD accesses on the existing page-table helpers. These can be overridden on arm64 to provide 128-bit single-copy atomicity when required.

READ_ONCE() instance in pmdp_test_and_clear_young() could not be converted into pmpd_get() because of cyclic dependency between <linux/pgtable.h> and <asm/pgtable.h>. Move the function inside <arch/arm64/mm/mmu.c> instead to workaround the problem.

No functional change intended.

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

Cc Ryan Roberts <[email protected]>

Cc Mark Rutland <[email protected]>

Cc [email protected]

Reviewed-by Ryan Roberts <[email protected]>

Signed-off-by Anshuman Khandual <[email protected]>

---
Changes in V4:

- Dropped pmdp_get() platform override
- Moved pmdp_test_and_clear_young() inside arch/arm64/mm/mmu.c

 arch/arm64/include/asm/pgtable.h | 11 +++--------
 arch/arm64/mm/fault.c            |  2 +-
 arch/arm64/mm/fixmap.c           |  2 +-
 arch/arm64/mm/hugetlbpage.c      |  2 +-
 arch/arm64/mm/kasan_init.c       |  4 ++--
 arch/arm64/mm/mmu.c              | 31 ++++++++++++++++++++-----------
 arch/arm64/mm/pageattr.c         |  2 +-
 arch/arm64/mm/trans_pgd.c        |  2 +-
 8 files changed, 30 insertions(+), 26 deletions(-)

arch/arm64/include/asm/pgtable.h +3 -8

index a474c9fb485a..f619fa74da3c 100644
@@ -811,7 +811,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
 }
 
 /* Find an entry in the third-level page table. */
-#define pte_offset_phys(dir,addr)	(pmd_page_paddr(READ_ONCE(*(dir))) + pte_index(addr) * sizeof(pte_t))
+#define pte_offset_phys(dir, addr)	(pmd_page_paddr(pmdp_get(dir)) + \
+					 pte_index(addr) * sizeof(pte_t))
 
 #define pte_set_fixmap(addr)		((pte_t *)set_fixmap_offset(FIX_PTE, addr))
 #define pte_set_fixmap_offset(pmd, addr)	pte_set_fixmap(pte_offset_phys(pmd, addr))

@@ -1287,13 +1288,7 @@ static inline bool __ptep_clear_flush_young(struct vm_area_struct *vma,
 
 #if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
 #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG
-static inline bool pmdp_test_and_clear_young(struct vm_area_struct *vma,
-		unsigned long address, pmd_t *pmdp)
-{
-	/* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
-	VM_WARN_ON(pmd_table(READ_ONCE(*pmdp)) && !system_supports_haft());
-	return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
-}
+bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp);
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
 
 static inline pte_t __ptep_get_and_clear_anysz(struct mm_struct *mm,

arch/arm64/mm/fault.c +1 -1

index b22736853fcd..33b05a9f13d7 100644
@@ -188,7 +188,7 @@ static void show_pte(unsigned long addr)
 			break;
 
 		pmdp = pmd_offset_lockless(pudp, pud, addr);
-		pmd = READ_ONCE(*pmdp);
+		pmd = pmdp_get(pmdp);
 		ptval_to_str(pxd_str, pmd_val(pmd));
 		pr_cont(", pmd=%s", pxd_str);
 		if (pmd_none(pmd) || pmd_bad(pmd))

arch/arm64/mm/fixmap.c +1 -1

index f66a0016dd02..3cdac8021d4f 100644
@@ -42,7 +42,7 @@ static inline pte_t *fixmap_pte(unsigned long addr)
 
 static void __init early_fixmap_init_pte(pmd_t *pmdp, unsigned long addr)
 {
-	pmd_t pmd = READ_ONCE(*pmdp);
+	pmd_t pmd = pmdp_get(pmdp);
 	pte_t *ptep;
 
 	if (pmd_none(pmd)) {

arch/arm64/mm/hugetlbpage.c +1 -1

index 8e799c1fe0aa..cdaa4500faf9 100644
@@ -304,7 +304,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		addr &= CONT_PMD_MASK;
 
 	pmdp = pmd_offset(pudp, addr);
-	pmd = READ_ONCE(*pmdp);
+	pmd = pmdp_get(pmdp);
 	if (!(sz == PMD_SIZE || sz == CONT_PMD_SIZE) &&
 	    pmd_none(pmd))
 		return NULL;

arch/arm64/mm/kasan_init.c +2 -2

index 45fbdce684c8..7ca833c5de5e 100644
@@ -62,7 +62,7 @@ static phys_addr_t __init kasan_alloc_raw_page(int node)
 static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
 				      bool early)
 {
-	if (pmd_none(READ_ONCE(*pmdp))) {
+	if (pmd_none(pmdp_get(pmdp))) {
 		phys_addr_t pte_phys = early ?
 				__pa_symbol(kasan_early_shadow_pte)
 					: kasan_alloc_zeroed_page(node);

@@ -138,7 +138,7 @@ static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr,
 	do {
 		next = pmd_addr_end(addr, end);
 		kasan_pte_populate(pmdp, addr, next, node, early);
-	} while (pmdp++, addr = next, addr != end && pmd_none(READ_ONCE(*pmdp)));
+	} while (pmdp++, addr = next, addr != end && pmd_none(pmdp_get(pmdp)));
 }
 
 static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,

arch/arm64/mm/mmu.c +20 ~~-11~~

index 140520b351e2..c998310406f9 100644
@@ -200,7 +200,7 @@ static int alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr,
 			       int flags)
 {
 	unsigned long next;
-	pmd_t pmd = READ_ONCE(*pmdp);
+	pmd_t pmd = pmdp_get(pmdp);
 	pte_t *ptep;
 
 	BUG_ON(pmd_leaf(pmd));

@@ -257,7 +257,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 	unsigned long next;
 
 	do {
-		pmd_t old_pmd = READ_ONCE(*pmdp);
+		pmd_t old_pmd = pmdp_get(pmdp);
 
 		next = pmd_addr_end(addr, end);

@@ -272,7 +272,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 			 * only allow updates to the permission attributes.
 			 */
 			BUG_ON(!pgattr_change_is_safe(pmd_val(old_pmd),
-						      READ_ONCE(pmd_val(*pmdp))));
+						      pmd_val(pmdp_get(pmdp))));
 		} else {
 			int ret;

@@ -282,7 +282,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 				return ret;
 
 			VM_WARN_ON_ONCE(pmd_val(old_pmd) != 0 &&
-					pmd_val(old_pmd) != READ_ONCE(pmd_val(*pmdp)));
+					pmd_val(old_pmd) != pmd_val(pmdp_get(pmdp)));
 		}
 		phys += next - addr;
 	} while (pmdp++, addr = next, addr != end);

@@ -293,7 +293,7 @@ static int init_pmd(pmd_t *pmdp, unsigned long addr, unsigned long end,
 static bool pmd_range_has_valid_noncont(pmd_t *pmdp)
 {
 	for (int i = 0; i < CONT_PMDS; i++) {
-		pte_t pte = pmd_pte(READ_ONCE(pmdp[i]));
+		pte_t pte = pmd_pte(pmdp_get(pmdp + i));
 
 		if (pte_valid(pte) && !pte_cont(pte))
 			return true;

@@ -1553,7 +1553,7 @@ static void unmap_hotplug_pmd_range(pud_t *pudp, unsigned long addr,
 	do {
 		next = pmd_addr_end(addr, end);
 		pmdp = pmd_offset(pudp, addr);
-		pmd = READ_ONCE(*pmdp);
+		pmd = pmdp_get(pmdp);
 		if (pmd_none(pmd))
 			continue;

@@ -1708,7 +1708,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
 	do {
 		next = pmd_addr_end(addr, end);
 		pmdp = pmd_offset(pudp, addr);
-		pmd = READ_ONCE(*pmdp);
+		pmd = pmdp_get(pmdp);
 		if (pmd_none(pmd))
 			continue;

@@ -1729,7 +1729,7 @@ static void free_empty_pmd_table(pud_t *pudp, unsigned long addr,
 	 */
 	pmdp = pmd_offset(pudp, 0UL);
 	for (i = 0; i < PTRS_PER_PMD; i++) {
-		if (!pmd_none(READ_ONCE(pmdp[i])))
+		if (!pmd_none(pmdp_get(pmdp + i)))
 			return;
 	}

@@ -1881,7 +1881,7 @@ int pmd_set_huge(pmd_t *pmdp, phys_addr_t phys, pgprot_t prot)
 	pmd_t new_pmd = pfn_pmd(__phys_to_pfn(phys), mk_pmd_sect_prot(prot));
 
 	/* Only allow permission changes for now */
-	if (!pgattr_change_is_safe(READ_ONCE(pmd_val(*pmdp)),
+	if (!pgattr_change_is_safe(pmd_val(pmdp_get(pmdp)),
 				   pmd_val(new_pmd)))
 		return 0;

@@ -1906,7 +1906,7 @@ int pud_clear_huge(pud_t *pudp)
 
 int pmd_clear_huge(pmd_t *pmdp)
 {
-	if (!pmd_leaf(READ_ONCE(*pmdp)))
+	if (!pmd_leaf(pmdp_get(pmdp)))
 		return 0;
 	pmd_clear(pmdp);
 	return 1;

@@ -1917,7 +1917,7 @@ int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr)
 	pte_t *table;
 	pmd_t pmd;
 
-	pmd = READ_ONCE(*pmdp);
+	pmd = pmdp_get(pmdp);
 
 	if (!pmd_table(pmd)) {
 		VM_WARN_ON(1);

@@ -2441,3 +2441,12 @@ void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, pte_t pte)
 		     __func__, pte_str_old, pte_str);
 }
 #endif /* CONFIG_DEBUG_VM */
+
+#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG)
+bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp)
+{
+	/* Operation applies to PMD table entry only if FEAT_HAFT is enabled */
+	VM_WARN_ON(pmd_table(pmdp_get(pmdp)) && !system_supports_haft());
+	return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
+}
+#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */

arch/arm64/mm/pageattr.c +1 -1

index de7bf95003ea..bf1fd1258a75 100644
@@ -362,7 +362,7 @@ bool kernel_page_present(struct page *page)
 		return pud_valid(pud);
 
 	pmdp = pmd_offset(pudp, addr);
-	pmd = READ_ONCE(*pmdp);
+	pmd = pmdp_get(pmdp);
 	if (pmd_none(pmd))
 		return false;
 	if (pmd_leaf(pmd))

arch/arm64/mm/trans_pgd.c +1 -1

index cca9706a875c..b27b2d2c20c3 100644
@@ -74,7 +74,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
 
 	src_pmdp = pmd_offset(src_pudp, start);
 	do {
-		pmd_t pmd = READ_ONCE(*src_pmdp);
+		pmd_t pmd = pmdp_get(src_pmdp);
 
 		next = pmd_addr_end(addr, end);
 		if (pmd_none(pmd))

Signature · 1 line

2.43.0

Anshuman Khandual

author

2026-11-05 04:34 UTC #2 lore raw

PATCH v4 2/4 arm64/mm: Use pudp_get() for PUD accesses

In reply to Anshuman Khandual

Replace READ_ONCE() with pudp_get() for PUD accesses in preparation for supporting both D64 and D128 translation table formats.

Instead, standardize PUD accesses on the existing page-table helpers. These can be overridden on arm64 to provide 128-bit single-copy atomicity when required.

No functional change intended.

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

Cc Ryan Roberts <[email protected]>

Cc Mark Rutland <[email protected]>

Cc [email protected]

Reviewed-by Ryan Roberts <[email protected]>

Signed-off-by Anshuman Khandual <[email protected]>

---
 arch/arm64/include/asm/pgtable.h |  3 ++-
 arch/arm64/mm/fault.c            |  2 +-
 arch/arm64/mm/fixmap.c           |  2 +-
 arch/arm64/mm/hugetlbpage.c      |  4 ++--
 arch/arm64/mm/kasan_init.c       |  4 ++--
 arch/arm64/mm/mmu.c              | 20 ++++++++++----------
 arch/arm64/mm/pageattr.c         |  2 +-
 arch/arm64/mm/trans_pgd.c        |  4 ++--
 8 files changed, 21 insertions(+), 20 deletions(-)

arch/arm64/include/asm/pgtable.h +2 -1

index f619fa74da3c..96cfcc70acaa 100644
@@ -873,7 +873,8 @@ static inline pmd_t *pud_pgtable(pud_t pud)
 }
 
 /* Find an entry in the second-level page table. */
-#define pmd_offset_phys(dir, addr)	(pud_page_paddr(READ_ONCE(*(dir))) + pmd_index(addr) * sizeof(pmd_t))
+#define pmd_offset_phys(dir, addr)	(pud_page_paddr(pudp_get(dir)) + \
+					 pmd_index(addr) * sizeof(pmd_t))
 
 #define pmd_set_fixmap(addr)		((pmd_t *)set_fixmap_offset(FIX_PMD, addr))
 #define pmd_set_fixmap_offset(pud, addr)	pmd_set_fixmap(pmd_offset_phys(pud, addr))

arch/arm64/mm/fault.c +1 -1

index 33b05a9f13d7..ea0cdc730768 100644
@@ -181,7 +181,7 @@ static void show_pte(unsigned long addr)
 			break;
 
 		pudp = pud_offset_lockless(p4dp, p4d, addr);
-		pud = READ_ONCE(*pudp);
+		pud = pudp_get(pudp);
 		ptval_to_str(pxd_str, pud_val(pud));
 		pr_cont(", pud=%s", pxd_str);
 		if (pud_none(pud) || pud_bad(pud))

arch/arm64/mm/fixmap.c +1 -1

index 3cdac8021d4f..d9a870836faf 100644
@@ -56,7 +56,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr,
 					 unsigned long end)
 {
 	unsigned long next;
-	pud_t pud = READ_ONCE(*pudp);
+	pud_t pud = pudp_get(pudp);
 	pmd_t *pmdp;
 
 	if (pud_none(pud))

arch/arm64/mm/hugetlbpage.c +2 -2

index cdaa4500faf9..c9ad5e75b073 100644
@@ -262,7 +262,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma,
 		WARN_ON(addr & (sz - 1));
 		ptep = pte_alloc_huge(mm, pmdp, addr);
 	} else if (sz == PMD_SIZE) {
-		if (want_pmd_share(vma, addr) && pud_none(READ_ONCE(*pudp)))
+		if (want_pmd_share(vma, addr) && pud_none(pudp_get(pudp)))
 			ptep = huge_pmd_share(mm, vma, addr, pudp);
 		else
 			ptep = (pte_t *)pmd_alloc(mm, pudp, addr);

@@ -292,7 +292,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		return NULL;
 
 	pudp = pud_offset(p4dp, addr);
-	pud = READ_ONCE(*pudp);
+	pud = pudp_get(pudp);
 	if (sz != PUD_SIZE && pud_none(pud))
 		return NULL;
 	/* hugepage or swap? */

arch/arm64/mm/kasan_init.c +2 -2

index 7ca833c5de5e..aad29bcc7622 100644
@@ -76,7 +76,7 @@ static pte_t *__init kasan_pte_offset(pmd_t *pmdp, unsigned long addr, int node,
 static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,
 				      bool early)
 {
-	if (pud_none(READ_ONCE(*pudp))) {
+	if (pud_none(pudp_get(pudp))) {
 		phys_addr_t pmd_phys = early ?
 				__pa_symbol(kasan_early_shadow_pmd)
 					: kasan_alloc_zeroed_page(node);

@@ -150,7 +150,7 @@ static void __init kasan_pud_populate(p4d_t *p4dp, unsigned long addr,
 	do {
 		next = pud_addr_end(addr, end);
 		kasan_pmd_populate(pudp, addr, next, node, early);
-	} while (pudp++, addr = next, addr != end && pud_none(READ_ONCE(*pudp)));
+	} while (pudp++, addr = next, addr != end && pud_none(pudp_get(pudp)));
 }
 
 static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,

arch/arm64/mm/mmu.c +10 ~~-10~~

index c998310406f9..7aa8793f13c5 100644
@@ -309,7 +309,7 @@ static int alloc_init_cont_pmd(pud_t *pudp, unsigned long addr,
 {
 	int ret;
 	unsigned long next;
-	pud_t pud = READ_ONCE(*pudp);
+	pud_t pud = pudp_get(pudp);
 	pmd_t *pmdp;
 
 	/*

@@ -390,7 +390,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 	}
 
 	do {
-		pud_t old_pud = READ_ONCE(*pudp);
+		pud_t old_pud = pudp_get(pudp);
 
 		next = pud_addr_end(addr, end);

@@ -408,7 +408,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 			 * only allow updates to the permission attributes.
 			 */
 			BUG_ON(!pgattr_change_is_safe(pud_val(old_pud),
-						      READ_ONCE(pud_val(*pudp))));
+						      pud_val(pudp_get(pudp))));
 		} else {
 			ret = alloc_init_cont_pmd(pudp, addr, next, phys, prot,
 						  pgtable_alloc, flags);

@@ -416,7 +416,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 				goto out;
 
 			VM_WARN_ON_ONCE(pud_val(old_pud) != 0 &&
-					pud_val(old_pud) != READ_ONCE(pud_val(*pudp)));
+					pud_val(old_pud) != pud_val(pudp_get(pudp)));
 		}
 		phys += next - addr;
 	} while (pudp++, addr = next, addr != end);

@@ -1591,7 +1591,7 @@ static void unmap_hotplug_pud_range(p4d_t *p4dp, unsigned long addr,
 	do {
 		next = pud_addr_end(addr, end);
 		pudp = pud_offset(p4dp, addr);
-		pud = READ_ONCE(*pudp);
+		pud = pudp_get(pudp);
 		if (pud_none(pud))
 			continue;

@@ -1748,7 +1748,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
 	do {
 		next = pud_addr_end(addr, end);
 		pudp = pud_offset(p4dp, addr);
-		pud = READ_ONCE(*pudp);
+		pud = pudp_get(pudp);
 		if (pud_none(pud))
 			continue;

@@ -1769,7 +1769,7 @@ static void free_empty_pud_table(p4d_t *p4dp, unsigned long addr,
 	 */
 	pudp = pud_offset(p4dp, 0UL);
 	for (i = 0; i < PTRS_PER_PUD; i++) {
-		if (!pud_none(READ_ONCE(pudp[i])))
+		if (!pud_none(pudp_get(pudp + i)))
 			return;
 	}

@@ -1867,7 +1867,7 @@ int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot)
 	pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot));
 
 	/* Only allow permission changes for now */
-	if (!pgattr_change_is_safe(READ_ONCE(pud_val(*pudp)),
+	if (!pgattr_change_is_safe(pud_val(pudp_get(pudp)),
 				   pud_val(new_pud)))
 		return 0;

@@ -1898,7 +1898,7 @@ void p4d_clear_huge(p4d_t *p4dp)
 
 int pud_clear_huge(pud_t *pudp)
 {
-	if (!pud_leaf(READ_ONCE(*pudp)))
+	if (!pud_leaf(pudp_get(pudp)))
 		return 0;
 	pud_clear(pudp);
 	return 1;

@@ -1938,7 +1938,7 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr)
 	pud_t pud;
 	unsigned long next, end;
 
-	pud = READ_ONCE(*pudp);
+	pud = pudp_get(pudp);
 
 	if (!pud_table(pud)) {
 		VM_WARN_ON(1);

arch/arm64/mm/pageattr.c +1 -1

index bf1fd1258a75..c544a61a5afd 100644
@@ -355,7 +355,7 @@ bool kernel_page_present(struct page *page)
 		return false;
 
 	pudp = pud_offset(p4dp, addr);
-	pud = READ_ONCE(*pudp);
+	pud = pudp_get(pudp);
 	if (pud_none(pud))
 		return false;
 	if (pud_leaf(pud))

arch/arm64/mm/trans_pgd.c +2 -2

index b27b2d2c20c3..d119119455f1 100644
@@ -64,7 +64,7 @@ static int copy_pmd(struct trans_pgd_info *info, pud_t *dst_pudp,
 	unsigned long next;
 	unsigned long addr = start;
 
-	if (pud_none(READ_ONCE(*dst_pudp))) {
+	if (pud_none(pudp_get(dst_pudp))) {
 		dst_pmdp = trans_alloc(info);
 		if (!dst_pmdp)
 			return -ENOMEM;

@@ -109,7 +109,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
 
 	src_pudp = pud_offset(src_p4dp, start);
 	do {
-		pud_t pud = READ_ONCE(*src_pudp);
+		pud_t pud = pudp_get(src_pudp);
 
 		next = pud_addr_end(addr, end);
 		if (pud_none(pud))

Signature · 1 line

2.43.0

Anshuman Khandual

author

2026-11-05 04:34 UTC #3 lore raw

PATCH v4 3/4 arm64/mm: Use p4dp_get() for P4D accesses

In reply to Anshuman Khandual

Replace READ_ONCE() with p4dp_get() for P4D accesses in preparation for supporting both D64 and D128 translation table formats.

Instead, standardize P4D accesses on the existing page-table helpers. These can be overridden on arm64 to provide 128-bit single-copy atomicity when required.

Both READ_ONCE() instances in pud_offset_[phys]() could not be converted into p4pd_get() because of cyclic dependency between <linux/pgtable.h> and <asm/pgtable.h>. Move the function inside <arch/arm64/mm/mmu.c> instead to workaround the problem.

No functional change intended.

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

Cc Ryan Roberts <[email protected]>

Cc Mark Rutland <[email protected]>

Cc [email protected]

Reviewed-by Ryan Roberts <[email protected]>

Signed-off-by Anshuman Khandual <[email protected]>

---
Changes in V4:

- Dropped p4dp_get() platform override
- Moved both pud_offset[_phys]() inside arch/arm64/mm/mmu.c

 arch/arm64/include/asm/pgtable.h | 12 ++----------
 arch/arm64/mm/fault.c            |  2 +-
 arch/arm64/mm/fixmap.c           |  2 +-
 arch/arm64/mm/hugetlbpage.c      |  2 +-
 arch/arm64/mm/kasan_init.c       |  4 ++--
 arch/arm64/mm/mmu.c              | 26 ++++++++++++++++++++------
 arch/arm64/mm/pageattr.c         |  2 +-
 arch/arm64/mm/trans_pgd.c        |  4 ++--
 8 files changed, 30 insertions(+), 24 deletions(-)

arch/arm64/include/asm/pgtable.h +2 ~~-10~~

index 96cfcc70acaa..8dffa7f787ac 100644
@@ -960,12 +960,7 @@ static inline pud_t *p4d_pgtable(p4d_t p4d)
 	return (pud_t *)__va(p4d_page_paddr(p4d));
 }
 
-static inline phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr)
-{
-	VM_WARN_ON_ONCE(!pgtable_l4_enabled());
-
-	return p4d_page_paddr(READ_ONCE(*p4dp)) + pud_index(addr) * sizeof(pud_t);
-}
+phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr);
 
 static inline
 pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr)

@@ -976,10 +971,7 @@ pud_t *pud_offset_lockless(p4d_t *p4dp, p4d_t p4d, unsigned long addr)
 }
 #define pud_offset_lockless pud_offset_lockless
 
-static inline pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
-{
-	return pud_offset_lockless(p4dp, READ_ONCE(*p4dp), addr);
-}
+pud_t *pud_offset(p4d_t *p4dp, unsigned long addr);
 #define pud_offset	pud_offset
 
 static inline pud_t *pud_set_fixmap(unsigned long addr)

arch/arm64/mm/fault.c +1 -1

index ea0cdc730768..934b534c5b89 100644
@@ -174,7 +174,7 @@ static void show_pte(unsigned long addr)
 			break;
 
 		p4dp = p4d_offset_lockless(pgdp, pgd, addr);
-		p4d = READ_ONCE(*p4dp);
+		p4d = p4dp_get(p4dp);
 		ptval_to_str(pxd_str, p4d_val(p4d));
 		pr_cont(", p4d=%s", pxd_str);
 		if (p4d_none(p4d) || p4d_bad(p4d))

arch/arm64/mm/fixmap.c +1 -1

index d9a870836faf..2a0fe0b0f771 100644
@@ -74,7 +74,7 @@ static void __init early_fixmap_init_pmd(pud_t *pudp, unsigned long addr,
 static void __init early_fixmap_init_pud(p4d_t *p4dp, unsigned long addr,
 					 unsigned long end)
 {
-	p4d_t p4d = READ_ONCE(*p4dp);
+	p4d_t p4d = p4dp_get(p4dp);
 	pud_t *pudp;
 
 	if (CONFIG_PGTABLE_LEVELS > 3 && !p4d_none(p4d) &&

arch/arm64/mm/hugetlbpage.c +1 -1

index c9ad5e75b073..5771bf49e1fe 100644
@@ -288,7 +288,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 		return NULL;
 
 	p4dp = p4d_offset(pgdp, addr);
-	if (!p4d_present(READ_ONCE(*p4dp)))
+	if (!p4d_present(p4dp_get(p4dp)))
 		return NULL;
 
 	pudp = pud_offset(p4dp, addr);

arch/arm64/mm/kasan_init.c +2 -2

index aad29bcc7622..4dd8c1186471 100644
@@ -89,7 +89,7 @@ static pmd_t *__init kasan_pmd_offset(pud_t *pudp, unsigned long addr, int node,
 static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
 				      bool early)
 {
-	if (p4d_none(READ_ONCE(*p4dp))) {
+	if (p4d_none(p4dp_get(p4dp))) {
 		phys_addr_t pud_phys = early ?
 				__pa_symbol(kasan_early_shadow_pud)
 					: kasan_alloc_zeroed_page(node);

@@ -162,7 +162,7 @@ static void __init kasan_p4d_populate(pgd_t *pgdp, unsigned long addr,
 	do {
 		next = p4d_addr_end(addr, end);
 		kasan_pud_populate(p4dp, addr, next, node, early);
-	} while (p4dp++, addr = next, addr != end && p4d_none(READ_ONCE(*p4dp)));
+	} while (p4dp++, addr = next, addr != end && p4d_none(p4dp_get(p4dp)));
 }
 
 static void __init kasan_pgd_populate(unsigned long addr, unsigned long end,

arch/arm64/mm/mmu.c +20 -6

index 7aa8793f13c5..711d734a63f7 100644
@@ -367,7 +367,7 @@ static int alloc_init_pud(p4d_t *p4dp, unsigned long addr, unsigned long end,
 {
 	int ret = 0;
 	unsigned long next;
-	p4d_t p4d = READ_ONCE(*p4dp);
+	p4d_t p4d = p4dp_get(p4dp);
 	pud_t *pudp;
 
 	if (p4d_none(p4d)) {

@@ -457,7 +457,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
 	}
 
 	do {
-		p4d_t old_p4d = READ_ONCE(*p4dp);
+		p4d_t old_p4d = p4dp_get(p4dp);
 
 		next = p4d_addr_end(addr, end);

@@ -467,7 +467,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
 			goto out;
 
 		VM_WARN_ON_ONCE(p4d_val(old_p4d) != 0 &&
-				p4d_val(old_p4d) != READ_ONCE(p4d_val(*p4dp)));
+				p4d_val(old_p4d) != (p4d_val(p4dp_get(p4dp))));
 
 		phys += next - addr;
 	} while (p4dp++, addr = next, addr != end);

@@ -1622,7 +1622,7 @@ static void unmap_hotplug_p4d_range(pgd_t *pgdp, unsigned long addr,
 	do {
 		next = p4d_addr_end(addr, end);
 		p4dp = p4d_offset(pgdp, addr);
-		p4d = READ_ONCE(*p4dp);
+		p4d = p4dp_get(p4dp);
 		if (p4d_none(p4d))
 			continue;

@@ -1788,7 +1788,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
 	do {
 		next = p4d_addr_end(addr, end);
 		p4dp = p4d_offset(pgdp, addr);
-		p4d = READ_ONCE(*p4dp);
+		p4d = p4dp_get(p4dp);
 		if (p4d_none(p4d))
 			continue;

@@ -1809,7 +1809,7 @@ static void free_empty_p4d_table(pgd_t *pgdp, unsigned long addr,
 	 */
 	p4dp = p4d_offset(pgdp, 0UL);
 	for (i = 0; i < PTRS_PER_P4D; i++) {
-		if (!p4d_none(READ_ONCE(p4dp[i])))
+		if (!p4d_none(p4dp_get(p4dp + i)))
 			return;
 	}

@@ -2450,3 +2450,17 @@ bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address
 	return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
 }
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
+
+#if CONFIG_PGTABLE_LEVELS > 3
+phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr)
+{
+	VM_WARN_ON_ONCE(!pgtable_l4_enabled());
+
+	return p4d_page_paddr(p4dp_get(p4dp)) + pud_index(addr) * sizeof(pud_t);
+}
+
+pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
+{
+	return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr);
+}
+#endif /* CONFIG_PGTABLE_LEVELS > 3 */

arch/arm64/mm/pageattr.c +1 -1

index c544a61a5afd..172bc516164c 100644
@@ -351,7 +351,7 @@ bool kernel_page_present(struct page *page)
 		return false;
 
 	p4dp = p4d_offset(pgdp, addr);
-	if (p4d_none(READ_ONCE(*p4dp)))
+	if (p4d_none(p4dp_get(p4dp)))
 		return false;
 
 	pudp = pud_offset(p4dp, addr);

arch/arm64/mm/trans_pgd.c +2 -2

index d119119455f1..7afe2beca4ba 100644
@@ -99,7 +99,7 @@ static int copy_pud(struct trans_pgd_info *info, p4d_t *dst_p4dp,
 	unsigned long next;
 	unsigned long addr = start;
 
-	if (p4d_none(READ_ONCE(*dst_p4dp))) {
+	if (p4d_none(p4dp_get(dst_p4dp))) {
 		dst_pudp = trans_alloc(info);
 		if (!dst_pudp)
 			return -ENOMEM;

@@ -145,7 +145,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
 	src_p4dp = p4d_offset(src_pgdp, start);
 	do {
 		next = p4d_addr_end(addr, end);
-		if (p4d_none(READ_ONCE(*src_p4dp)))
+		if (p4d_none(p4dp_get(src_p4dp)))
 			continue;
 		if (copy_pud(info, dst_p4dp, src_p4dp, addr, next))
 			return -ENOMEM;

Signature · 1 line

2.43.0

[email protected]

AI review

2026-11-06 04:34 UTC #4 lore raw

Re: [PATCH V4 3/4] arm64/mm: Use p4dp_get() for P4D accesses

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] arm64/mm: early boot KASAN fault in out-of-line pud_offset_phys() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] arm64/mm: early boot KASAN fault in out-of-line pud_offset_phys() --

commit eaa67607aabb8af35fe6970c5442e1ab262f0286
Author: Anshuman Khandual <[email protected]>

arm64/mm: Use p4dp_get() for P4D accesses

This commit replaces READ_ONCE() with p4dp_get() for P4D accesses across arm64/mm and moves pud_offset_phys() and pud_offset() out of arch/arm64/include/asm/pgtable.h into arch/arm64/mm/mmu.c to avoid header dependency cycles.

Quoted 4 lines

arch/arm64/mm/mmu.c +0 -0
index afa15b1a8a247..5b5a85a0527d6 100644

[ ... ]

arch/arm64/mm/mmu.c bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address · line 2459

Quoted 12 lines

arch/arm64/mm/mmu.c +8 -0

@@ -2449,3 +2449,17 @@ bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address
 	return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp);
 }
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
+
+#if CONFIG_PGTABLE_LEVELS > 3
+phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr)
+{
+	VM_WARN_ON_ONCE(!pgtable_l4_enabled());
+
+	return p4d_page_paddr(p4dp_get(p4dp)) + pud_index(addr) * sizeof(pud_t);
+}

[Severity: High]
Does moving pud_offset_phys() out-of-line into arch/arm64/mm/mmu.c without noinstr or __no_sanitize_address cause an early boot translation fault when CONFIG_KASAN_GENERIC or CONFIG_KASAN_SW_TAGS is enabled with 4-level or 5-level page tables?

During early boot before start_kernel(), __primary_switched calls kasan_early_init() in arch/arm64/mm/kasan_init.c to populate the initial early KASAN shadow page tables:

__primary_switched

    -> kasan_early_init()
      -> kasan_pgd_populate(..., true)
        -> kasan_p4d_populate(..., true)
          -> kasan_pud_populate(..., true)
            -> kasan_pud_offset(p4dp, addr, node, true)
              -> pud_offset_kimg(p4dp, addr)
                -> pud_offset_phys(p4dp, addr)

At that point, no KASAN shadow PTEs have been mapped yet, which is why arch/arm64/mm/Makefile sets:

KASAN_SANITIZE_kasan_init.o := n

Previously, pud_offset_phys() was a static inline helper in arch/arm64/include/asm/pgtable.h, so it was inlined into kasan_init.o without KASAN instrumentation.

With pud_offset_phys() moved into arch/arm64/mm/mmu.c, which is compiled with KASAN sanitization enabled, calling pud_offset_phys() from kasan_early_init() triggers KASAN shadow memory checks on p4dp before the early shadow memory is mapped.

Should pud_offset_phys() be marked noinstr (like set_swapper_pgd() in the same file) or __no_sanitize_address?

Signature · 1 line

Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3

Anshuman Khandual

author

2026-11-05 04:34 UTC #5 lore raw

PATCH v4 4/4 arm64/mm: Use pgdp_get() for PGD accesses

In reply to Anshuman Khandual

Replace READ_ONCE() with pgdp_get() for PGD accesses in preparation for supporting both D64 and D128 translation table formats.

Instead, standardize PGD accesses on the existing page-table helpers. These can be overridden on arm64 to provide 128-bit single-copy atomicity when required.

Both READ_ONCE() instances in p4d_offset_[phys]() could not be converted into pgpd_get() because of cyclic dependency between <linux/pgtable.h> and <asm/pgtable.h>. Move the function inside <arch/arm64/mm/mmu.c> instead to workaround the problem.

No functional change intended.

Cc Catalin Marinas <[email protected]>

Cc Will Deacon <[email protected]>

Cc Ryan Roberts <[email protected]>

Cc Mark Rutland <[email protected]>

Cc [email protected]

Reviewed-by Ryan Roberts <[email protected]>

Signed-off-by Anshuman Khandual <[email protected]>

---
Changes in V4:

- Dropped pgdp_get() platform override
- Moved both p4d_offset[_phys]() inside arch/arm64/mm/mmu.c

 arch/arm64/include/asm/pgtable.h | 12 ++----------
 arch/arm64/mm/fault.c            |  2 +-
 arch/arm64/mm/hugetlbpage.c      |  2 +-
 arch/arm64/mm/kasan_init.c       |  6 +++---
 arch/arm64/mm/mmu.c              | 20 +++++++++++++++++---
 arch/arm64/mm/pageattr.c         |  2 +-
 arch/arm64/mm/trans_pgd.c        |  4 ++--
 7 files changed, 27 insertions(+), 21 deletions(-)

arch/arm64/include/asm/pgtable.h +2 ~~-10~~

index 8dffa7f787ac..6a38de07e599 100644
@@ -1072,12 +1072,7 @@ static inline p4d_t *pgd_to_folded_p4d(pgd_t *pgdp, unsigned long addr)
 	return (p4d_t *)PTR_ALIGN_DOWN(pgdp, PAGE_SIZE) + p4d_index(addr);
 }
 
-static inline phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
-{
-	VM_WARN_ON_ONCE(!pgtable_l5_enabled());
-
-	return pgd_page_paddr(READ_ONCE(*pgdp)) + p4d_index(addr) * sizeof(p4d_t);
-}
+phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr);
 
 static inline
 p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long addr)

@@ -1088,10 +1083,7 @@ p4d_t *p4d_offset_lockless(pgd_t *pgdp, pgd_t pgd, unsigned long addr)
 }
 #define p4d_offset_lockless p4d_offset_lockless
 
-static inline p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr)
-{
-	return p4d_offset_lockless(pgdp, READ_ONCE(*pgdp), addr);
-}
+p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr);
 
 static inline p4d_t *p4d_set_fixmap(unsigned long addr)
 {

arch/arm64/mm/fault.c +1 -1

index 934b534c5b89..0b489fbeeb07 100644
@@ -160,7 +160,7 @@ static void show_pte(unsigned long addr)
 	guard(irqsave)();
 
 	pgdp = pgd_offset(mm, addr);
-	pgd = READ_ONCE(*pgdp);
+	pgd = pgdp_get(pgdp);
 	ptval_to_str(pxd_str, pgd_val(pgd));
 	pr_alert("[%016lx] pgd=%s", addr, pxd_str);

arch/arm64/mm/hugetlbpage.c +1 -1

index 5771bf49e1fe..e05e5b41ae50 100644
@@ -284,7 +284,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm,
 	pmd_t *pmdp, pmd;
 
 	pgdp = pgd_offset(mm, addr);
-	if (!pgd_present(READ_ONCE(*pgdp)))
+	if (!pgd_present(pgdp_get(pgdp)))
 		return NULL;
 
 	p4dp = p4d_offset(pgdp, addr);

arch/arm64/mm/kasan_init.c +3 -3

index 4dd8c1186471..f61f5bc09467 100644
@@ -102,7 +102,7 @@ static pud_t *__init kasan_pud_offset(p4d_t *p4dp, unsigned long addr, int node,
 static p4d_t *__init kasan_p4d_offset(pgd_t *pgdp, unsigned long addr, int node,
 				      bool early)
 {
-	if (pgd_none(READ_ONCE(*pgdp))) {
+	if (pgd_none(pgdp_get(pgdp))) {
 		phys_addr_t p4d_phys = early ?
 				__pa_symbol(kasan_early_shadow_p4d)
 					: kasan_alloc_zeroed_page(node);

@@ -256,7 +256,7 @@ static int __init root_level_idx(u64 addr)
 static void __init clone_next_level(u64 addr, pgd_t *tmp_pg_dir, pud_t *pud)
 {
 	int idx = root_level_idx(addr);
-	pgd_t pgd = READ_ONCE(swapper_pg_dir[idx]);
+	pgd_t pgd = pgdp_get(swapper_pg_dir + idx);
 	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
 
 	memcpy(pud, pudp, PAGE_SIZE);

@@ -280,7 +280,7 @@ static int __init next_level_idx(u64 addr)
  */
 static void __init clear_next_level(int pgd_idx, int start, int end)
 {
-	pgd_t pgd = READ_ONCE(swapper_pg_dir[pgd_idx]);
+	pgd_t pgd = pgdp_get(swapper_pg_dir + pgd_idx);
 	pud_t *pudp = (pud_t *)__phys_to_kimg(__pgd_to_phys(pgd));
 
 	memset(&pudp[start], 0, (end - start) * sizeof(pud_t));

arch/arm64/mm/mmu.c +17 -3

index 711d734a63f7..1f56db9a946c 100644
@@ -434,7 +434,7 @@ static int alloc_init_p4d(pgd_t *pgdp, unsigned long addr, unsigned long end,
 {
 	int ret;
 	unsigned long next;
-	pgd_t pgd = READ_ONCE(*pgdp);
+	pgd_t pgd = pgdp_get(pgdp);
 	p4d_t *p4dp;
 
 	if (pgd_none(pgd)) {

@@ -1649,7 +1649,7 @@ static void unmap_hotplug_range(unsigned long addr, unsigned long end,
 	do {
 		next = pgd_addr_end(addr, end);
 		pgdp = pgd_offset_k(addr);
-		pgd = READ_ONCE(*pgdp);
+		pgd = pgdp_get(pgdp);
 		if (pgd_none(pgd))
 			continue;

@@ -1827,7 +1827,7 @@ static void free_empty_tables(unsigned long addr, unsigned long end,
 	do {
 		next = pgd_addr_end(addr, end);
 		pgdp = pgd_offset_k(addr);
-		pgd = READ_ONCE(*pgdp);
+		pgd = pgdp_get(pgdp);
 		if (pgd_none(pgd))
 			continue;

@@ -2464,3 +2464,17 @@ pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
 	return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr);
 }
 #endif /* CONFIG_PGTABLE_LEVELS > 3 */
+
+#if CONFIG_PGTABLE_LEVELS > 4
+phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
+{
+	VM_WARN_ON_ONCE(!pgtable_l5_enabled());
+
+	return pgd_page_paddr(pgdp_get(pgdp)) + p4d_index(addr) * sizeof(p4d_t);
+}
+
+p4d_t *p4d_offset(pgd_t *pgdp, unsigned long addr)
+{
+	return p4d_offset_lockless(pgdp, pgdp_get(pgdp), addr);
+}
+#endif /* CONFIG_PGTABLE_LEVELS > 4 */

arch/arm64/mm/pageattr.c +1 -1

index 172bc516164c..b3b7c508564f 100644
@@ -347,7 +347,7 @@ bool kernel_page_present(struct page *page)
 	unsigned long addr = (unsigned long)page_address(page);
 
 	pgdp = pgd_offset_k(addr);
-	if (pgd_none(READ_ONCE(*pgdp)))
+	if (pgd_none(pgdp_get(pgdp)))
 		return false;
 
 	p4dp = p4d_offset(pgdp, addr);

arch/arm64/mm/trans_pgd.c +2 -2

index 7afe2beca4ba..06470d690f9f 100644
@@ -134,7 +134,7 @@ static int copy_p4d(struct trans_pgd_info *info, pgd_t *dst_pgdp,
 	unsigned long next;
 	unsigned long addr = start;
 
-	if (pgd_none(READ_ONCE(*dst_pgdp))) {
+	if (pgd_none(pgdp_get(dst_pgdp))) {
 		dst_p4dp = trans_alloc(info);
 		if (!dst_p4dp)
 			return -ENOMEM;

@@ -164,7 +164,7 @@ static int copy_page_tables(struct trans_pgd_info *info, pgd_t *dst_pgdp,
 	dst_pgdp = pgd_offset_pgd(dst_pgdp, start);
 	do {
 		next = pgd_addr_end(addr, end);
-		if (pgd_none(READ_ONCE(*src_pgdp)))
+		if (pgd_none(pgdp_get(src_pgdp)))
 			continue;
 		if (copy_p4d(info, dst_pgdp, src_pgdp, addr, next))
 			return -ENOMEM;

Signature · 1 line

2.43.0

[email protected]

AI review

2026-11-06 04:34 UTC #6 lore raw

Re: [PATCH V4 4/4] arm64/mm: Use pgdp_get() for PGD accesses

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] arm64/mm: KASAN fault in p4d_offset_phys() during kasan_early_init() --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] arm64/mm: KASAN fault in p4d_offset_phys() during kasan_early_init() --

commit e9bf68a7a555b3c58c2a0a8736f337dde8136ab9
Author: Anshuman Khandual <[email protected]>

arm64/mm: Use pgdp_get() for PGD accesses

This commit replaces READ_ONCE() with pgdp_get() for PGD accesses and moves p4d_offset() and p4d_offset_phys() from arch/arm64/include/asm/pgtable.h into arch/arm64/mm/mmu.c to avoid header include dependencies.

Quoted 4 lines

arch/arm64/mm/mmu.c +0 -0
index 5b5a85a0527d6..11015e1deb32f 100644

[ ... ]

arch/arm64/mm/mmu.c pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) · line 2473

Quoted 12 lines

arch/arm64/mm/mmu.c +8 -0

@@ -2463,3 +2463,17 @@ pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
 	return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr);
 }
 #endif /* CONFIG_PGTABLE_LEVELS > 3 */
+
+#if CONFIG_PGTABLE_LEVELS > 4
+phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
+{
+	VM_WARN_ON_ONCE(!pgtable_l5_enabled());
+
+	return pgd_page_paddr(pgdp_get(pgdp)) + p4d_index(addr) * sizeof(p4d_t);
+}

[Severity: High]
Does moving p4d_offset_phys() (and pud_offset_phys() in the preceding patch) out-of-line into arch/arm64/mm/mmu.c cause a translation fault during early boot when KASAN is enabled with 5-level (or 4-level) page tables?

During early boot, __primary_switched calls kasan_early_init() before the KASAN shadow region is mapped. Because any KASAN shadow check will fault until kasan_early_init() finishes populating the early shadow page tables, arch/arm64/mm/Makefile disables KASAN sanitization for kasan_init.o:

KASAN_SANITIZE_kasan_init.o := n

and set_swapper_pgd() in arch/arm64/mm/mmu.c is marked noinstr for the same reason.

When kasan_early_init() populates the shadow page tables, it calls into p4d_offset_phys() and pud_offset_phys():

    kasan_early_init()
      kasan_pgd_populate(..., early=true)
        kasan_p4d_populate(..., early=true)
          kasan_p4d_offset(..., early=true)
            p4d_offset_kimg()
              p4d_offset_phys()
          kasan_pud_populate(..., early=true)
            kasan_pud_offset(..., early=true)
              pud_offset_kimg()
                pud_offset_phys()

Previously, p4d_offset_phys() and pud_offset_phys() were static inline functions in arch/arm64/include/asm/pgtable.h, so calls from kasan_init.c were inlined into kasan_init.o and compiled without KASAN instrumentation.

Now that both functions live in arch/arm64/mm/mmu.c, which is KASAN-sanitized, the compiler emits KASAN shadow checks for their memory accesses (such as pgdp_get(pgdp), p4dp_get(p4dp), and vabits_actual) before the early shadow memory is mapped.

Should p4d_offset_phys() and pud_offset_phys() be marked noinstr or __no_sanitize_address?

Signature · 1 line

Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4

Anshuman Khandual

author

2026-11-11 11:54 UTC #7 lore raw

Re: [PATCH V4 4/4] arm64/mm: Use pgdp_get() for PGD accesses

Quoted 64 lines · On Fri, Nov 06, 2026 at 04:34:47AM +0000, [email protected] wrote:

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] arm64/mm: KASAN fault in p4d_offset_phys() during kasan_early_init() --
commit e9bf68a7a555b3c58c2a0a8736f337dde8136ab9
Author: Anshuman Khandual <[email protected]>
arm64/mm: Use pgdp_get() for PGD accesses
This commit replaces READ_ONCE() with pgdp_get() for PGD accesses and moves p4d_offset() and p4d_offset_phys() from arch/arm64/include/asm/pgtable.h into arch/arm64/mm/mmu.c to avoid header include dependencies.
Quoted 4 lines
arch/arm64/mm/mmu.c +0 -0
index 5b5a85a0527d6..11015e1deb32f 100644
[ ... ]
arch/arm64/mm/mmu.c pud_t *pud_offset(p4d_t *p4dp, unsigned long addr) · line 2473
Quoted 12 lines
arch/arm64/mm/mmu.c +8 -0
@@ -2463,3 +2463,17 @@ pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
 	return pud_offset_lockless(p4dp, p4dp_get(p4dp), addr);
 }
 #endif /* CONFIG_PGTABLE_LEVELS > 3 */
+
+#if CONFIG_PGTABLE_LEVELS > 4
+phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
+{
+	VM_WARN_ON_ONCE(!pgtable_l5_enabled());
+
+	return pgd_page_paddr(pgdp_get(pgdp)) + p4d_index(addr) * sizeof(p4d_t);
+}
[Severity: High]
Does moving p4d_offset_phys() (and pud_offset_phys() in the preceding patch) out-of-line into arch/arm64/mm/mmu.c cause a translation fault during early boot when KASAN is enabled with 5-level (or 4-level) page tables?
During early boot, __primary_switched calls kasan_early_init() before the KASAN shadow region is mapped. Because any KASAN shadow check will fault until kasan_early_init() finishes populating the early shadow page tables, arch/arm64/mm/Makefile disables KASAN sanitization for kasan_init.o:
KASAN_SANITIZE_kasan_init.o := n
and set_swapper_pgd() in arch/arm64/mm/mmu.c is marked noinstr for the same reason.
When kasan_early_init() populates the shadow page tables, it calls into p4d_offset_phys() and pud_offset_phys():
    kasan_early_init()
      kasan_pgd_populate(..., early=true)
        kasan_p4d_populate(..., early=true)
          kasan_p4d_offset(..., early=true)
            p4d_offset_kimg()
              p4d_offset_phys()
          kasan_pud_populate(..., early=true)
            kasan_pud_offset(..., early=true)
              pud_offset_kimg()
                pud_offset_phys()
Previously, p4d_offset_phys() and pud_offset_phys() were static inline functions in arch/arm64/include/asm/pgtable.h, so calls from kasan_init.c were inlined into kasan_init.o and compiled without KASAN instrumentation.

Quoted 11 lines

Now that both functions live in arch/arm64/mm/mmu.c, which is KASAN-sanitized, the compiler emits KASAN shadow checks for their memory accesses (such as pgdp_get(pgdp), p4dp_get(p4dp), and vabits_actual) before the early shadow memory is mapped.
Should p4d_offset_phys() and pud_offset_phys() be marked noinstr or __no_sanitize_address?
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=4

The kernel does fail to boot with CONFIG_KASAN enabled and PGTABLE_LEVELS = 5. An example configuration would be as follows aka 'defconfig' with CONFIG_KASAN

------------------------
CONFIG_ARM64_4K_PAGES=y
CONFIG_ARM64_VA_BITS_52=y
CONFIG_ARM64_VA_BITS=52
CONFIG_ARM64_PA_BITS_52=y
CONFIG_ARM64_PA_BITS=52

CONFIG_KASAN_SHADOW_OFFSET=0xdfff800000000000
CONFIG_HAVE_ARCH_KASAN=y
CONFIG_HAVE_ARCH_KASAN_SW_TAGS=y
CONFIG_HAVE_ARCH_KASAN_HW_TAGS=y
CONFIG_HAVE_ARCH_KASAN_VMALLOC=y
CONFIG_CC_HAS_KASAN_GENERIC=y
CONFIG_CC_HAS_KASAN_SW_TAGS=y
CONFIG_KASAN=y
CONFIG_CC_HAS_KASAN_MEMINTRINSIC_PREFIX=y
CONFIG_KASAN_GENERIC=y
CONFIG_KASAN_INLINE=y
CONFIG_KASAN_STACK=y
CONFIG_KASAN_VMALLOC=y
------------------------

The following change fixes the problem. Although not sure if this is worth the churn. Earlier vesion V3 might be a better trade off where all generic pxdp_get() helpers were just copied over into the platform thus preventing subsequent header to C movement in the first place which caused this KASAN problem.

------------------------

arch/arm64/include/asm/pgtable.h +2 -2

index 6a38de07e599..d0b3292ee18e 100644
@@ -940,7 +940,7 @@ static inline void p4d_clear(p4d_t *p4dp)
                set_p4d(p4dp, __p4d(0));
 }
 
-static inline phys_addr_t p4d_page_paddr(p4d_t p4d)
+static __always_inline phys_addr_t p4d_page_paddr(p4d_t p4d)
 {
        return __p4d_to_phys(p4d);
 }

@@ -1057,7 +1057,7 @@ static inline void pgd_clear(pgd_t *pgdp)
                set_pgd(pgdp, __pgd(0));
 }
 
-static inline phys_addr_t pgd_page_paddr(pgd_t pgd)
+static __always_inline phys_addr_t pgd_page_paddr(pgd_t pgd)
 {
        return __pgd_to_phys(pgd);
 }

arch/arm64/mm/mmu.c +2 -2

index 1f56db9a946c..14e3b0387ed7 100644
@@ -2452,7 +2452,7 @@ bool pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address
 #endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */
 
 #if CONFIG_PGTABLE_LEVELS > 3
-phys_addr_t pud_offset_phys(p4d_t *p4dp, unsigned long addr)
+phys_addr_t noinstr pud_offset_phys(p4d_t *p4dp, unsigned long addr)
 {
        VM_WARN_ON_ONCE(!pgtable_l4_enabled());

@@ -2466,7 +2466,7 @@ pud_t *pud_offset(p4d_t *p4dp, unsigned long addr)
 #endif /* CONFIG_PGTABLE_LEVELS > 3 */
 
 #if CONFIG_PGTABLE_LEVELS > 4
-phys_addr_t p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
+phys_addr_t noinstr p4d_offset_phys(pgd_t *pgdp, unsigned long addr)
 {
        VM_WARN_ON_ONCE(!pgtable_l5_enabled());

include/linux/pgtable.h +2 -2

index 780fe849ff8b..5e88015cdaac 100644
@@ -511,14 +511,14 @@ static inline pud_t pudp_get(const pud_t *pudp)
 #endif
 
 #ifndef p4dp_get
-static inline p4d_t p4dp_get(const p4d_t *p4dp)
+static __always_inline p4d_t p4dp_get(const p4d_t *p4dp)
 {
        return READ_ONCE(*p4dp);
 }
 #endif
 
 #ifndef pgdp_get
-static inline pgd_t pgdp_get(const pgd_t *pgdp)
+static __always_inline pgd_t pgdp_get(const pgd_t *pgdp)
 {
        return READ_ONCE(*pgdp);
 }

------------------------