Skip to contentSkip to conversation

arm64: mm: Fix lockless walks with static and dynamic page-table folding

PATCH Applied Will Deacon 2 participants 3 messages Jul 25, 2024

Subsystems (2)
Stable branch arm64
lore
Share & tools
lore

Will Deacon

author maintainer
#0 lore raw

PATCH 1/1 arm64: mm: Fix lockless walks with static and dynamic page-table folding

Lina reports random oopsen originating from the fast GUP code when 16K pages are used with 4-level page-tables, the fourth level being folded at runtime due to lack of LPA2.

In this configuration, the generic implementation of p4d_offset_lockless() will return a 'p4d_t *' corresponding to the 'pgd_t' allocated on the stack of the caller, gup_fast_pgd_range(). This is normally fine, but when the fourth level of page-table is folded at runtime, pud_offset_lockless() will offset from the address of the 'p4d_t' to calculate the address of the PUD in the same page-table page. This results in a stray stack read when the 'p4d_t' has been allocated on the stack and can send the walker into the weeds.

Fix the problem by providing our own definition of p4d_offset_lockless() when CONFIG_PGTABLE_LEVELS <= 4 which returns the real page-table pointer rather than the address of the local stack variable.

Cc Catalin Marinas <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc <[email protected]>
Link https://lore.kernel.org/r/[email protected] open here
Fixes 0dd4f60a2c76 ("arm64: mm: Add support for folding PUDs at runtime")
Reported-by Asahi Lina <[email protected]>
Signed-off-by Will Deacon <[email protected]>
---
 arch/arm64/include/asm/pgtable.h | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
arch/arm64/include/asm/pgtable.h +22 -0
index f8efbc128446..7a4f5604be3f 100644
@@ -1065,6 +1065,28 @@ static inline bool pgtable_l5_enabled(void) { return false; }
 
 #define p4d_offset_kimg(dir,addr)	((p4d_t *)dir)
 
+static inline
+p4d_t *p4d_offset_lockless_folded(pgd_t *pgdp, pgd_t pgd, unsigned long addr)
+{
+	/*
+	 * With runtime folding of the pud, pud_offset_lockless() passes
+	 * the 'pgd_t *' we return here to p4d_to_folded_pud(), which
+	 * will offset the pointer assuming that it points into
+	 * a page-table page. However, the fast GUP path passes us a
+	 * pgd_t allocated on the stack and so we must use the original
+	 * pointer in 'pgdp' to construct the p4d pointer instead of
+	 * using the generic p4d_offset_lockless() implementation.
+	 *
+	 * Note: reusing the original pointer means that we may
+	 * dereference the same (live) page-table entry multiple times.
+	 * This is safe because it is still only loaded once in the
+	 * context of each level and the CPU guarantees same-address
+	 * read-after-read ordering.
+	 */
+	return p4d_offset(pgdp, addr);
+}
+#define p4d_offset_lockless p4d_offset_lockless_folded
+
 #endif  /* CONFIG_PGTABLE_LEVELS > 4 */
 
 #define pgd_ERROR(e)	\
Signature · 1 line
2.45.2.1089.g2a221341d9-goog
Reviewed-by on v1 1/1
#1 lore raw
Quoted 25 lines · Will Deacon

Lina reports random oopsen originating from the fast GUP code when 16K pages are used with 4-level page-tables, the fourth level being folded at runtime due to lack of LPA2.

In this configuration, the generic implementation of p4d_offset_lockless() will return a 'p4d_t *' corresponding to the 'pgd_t' allocated on the stack of the caller, gup_fast_pgd_range(). This is normally fine, but when the fourth level of page-table is folded at runtime, pud_offset_lockless() will offset from the address of the 'p4d_t' to calculate the address of the PUD in the same page-table page. This results in a stray stack read when the 'p4d_t' has been allocated on the stack and can send the walker into the weeds.

Fix the problem by providing our own definition of p4d_offset_lockless() when CONFIG_PGTABLE_LEVELS <= 4 which returns the real page-table pointer rather than the address of the local stack variable.

Cc Catalin Marinas <[email protected]>
Cc Ard Biesheuvel <[email protected]>
Cc <[email protected]>
Link https://lore.kernel.org/r/[email protected] open here
Fixes 0dd4f60a2c76 ("arm64: mm: Add support for folding PUDs at runtime")
Reported-by Asahi Lina <[email protected]>
Signed-off-by Will Deacon <[email protected]>
Quoted 4 lines
---
 arch/arm64/include/asm/pgtable.h | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
Reviewed-by Ard Biesheuvel <[email protected]>
author maintainer Applied
#2 lore raw
In reply to Will Deacon
Quoted 13 lines · Will Deacon

Lina reports random oopsen originating from the fast GUP code when 16K pages are used with 4-level page-tables, the fourth level being folded at runtime due to lack of LPA2.

In this configuration, the generic implementation of p4d_offset_lockless() will return a 'p4d_t *' corresponding to the 'pgd_t' allocated on the stack of the caller, gup_fast_pgd_range(). This is normally fine, but when the fourth level of page-table is folded at runtime, pud_offset_lockless() will offset from the address of the 'p4d_t' to calculate the address of the PUD in the same page-table page. This results in a stray stack read when the 'p4d_t' has been allocated on the stack and can send the walker into the weeds.

Quoted 1 line

[...]

Applied to arm64 (for-next/core), thanks!

[1/1] arm64: mm: Fix lockless walks with static and dynamic page-table folding

      https://git.kernel.org/arm64/c/36639013b346

Cheers,

Signature · 5 lines
Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev