Skip to contentSkip to conversation

mm/memory: use pmdp_get_lockless() for the pmd read in do_fault_around()

PATCH Superseded Hui Zhu 2 participants 2 messages Nov 2

Subsystems (2)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH v2 1/1 mm/memory: use pmdp_get_lockless() for the pmd read in do_fault_around()

From: Hui Zhu <[email protected]>

KCSAN reports a data-race in do_read_fault / pmd_install; the same code is in mainline:

write (marked) to 0xffff8d52a75fb810 of 8 bytes by task 27487 on cpu 5:

   native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline]
   pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline]
   pmd_install+0xe0/0x120 mm/memory.c:444
   finish_fault+0x5bc/0xb00 mm/memory.c:5204
   do_cow_fault mm/memory.c:5432 [inline]
   do_fault+0x5d0/0x6e0 mm/memory.c:5526
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   __mm_populate+0x144/0x270 mm/gup.c:2207
   __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764

read to 0xffff8d52a75fb810 of 8 bytes by task 27527 on cpu 9:

   do_fault_around mm/memory.c:5341 [inline]
   do_read_fault+0x162/0x670 mm/memory.c:5381
   do_fault+0x31a/0x6e0 mm/memory.c:5524
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   faultin_page_range+0x227/0x530 mm/gup.c:2156
   __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479

value changed: 0x0000000000000000 -> 0x000000001386c067

Both paths only hold mmap_lock for read, so do_fault_around() tests vmf->pmd while pmd_install() populates it under the pmd lock. The race is
benign: a stale pmd_none() only wastes the prealloc_pte, which do_fault() frees again, and a pmd seen as populated only makes ->map_pages() find no PTE and do_read_fault() fall back to __do_fault().

The load still has to be lockless-safe. pmdp_get() is a plain READ_ONCE() and can tear where pmd_t is wider than a word, as on i386-PAE where pmd_install() publishes two words with a single cmpxchg8b. Use pmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what __pte_offset_map() already uses on this very pmd a few instructions later. It is pmdp_get() everywhere else, so the report is silenced the same way.

No functional change on architectures that can load a pmd in one go.

---
Changelog:
Use pmdp_get_lockless() instead of pmdp_get() (David Hildenbrand).

 mm/memory.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
mm/memory.c +5 -1
index 0fb676dd2bd63..2c1e50b70faeb 100644
@@ -6017,7 +6017,11 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {
+	/*
+	 * Lockless: pmd_install() rechecks under the pmd lock, so a stale
+	 * pmd_none() here only wastes the prealloc_pte allocated below.
+	 */
+	if (pmd_none(pmdp_get_lockless(vmf->pmd))) {
 		vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
Signature · 1 line
2.43.0
#1 lore raw
Quoted 40 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data-race in do_read_fault / pmd_install; the same code is in mainline:

write (marked) to 0xffff8d52a75fb810 of 8 bytes by task 27487 on cpu 5:

   native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline]
   pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline]
   pmd_install+0xe0/0x120 mm/memory.c:444
   finish_fault+0x5bc/0xb00 mm/memory.c:5204
   do_cow_fault mm/memory.c:5432 [inline]
   do_fault+0x5d0/0x6e0 mm/memory.c:5526
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   __mm_populate+0x144/0x270 mm/gup.c:2207
   __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764

read to 0xffff8d52a75fb810 of 8 bytes by task 27527 on cpu 9:

   do_fault_around mm/memory.c:5341 [inline]
   do_read_fault+0x162/0x670 mm/memory.c:5381
   do_fault+0x31a/0x6e0 mm/memory.c:5524
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   faultin_page_range+0x227/0x530 mm/gup.c:2156
   __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479

value changed: 0x0000000000000000 -> 0x000000001386c067

Both paths only hold mmap_lock for read, so do_fault_around() tests vmf->pmd while pmd_install() populates it under the pmd lock. The race is
benign: a stale pmd_none() only wastes the prealloc_pte, which do_fault() frees again, and a pmd seen as populated only makes ->map_pages() find no PTE and do_read_fault() fall back to __do_fault().

The load still has to be lockless-safe. pmdp_get() is a plain READ_ONCE() and can tear where pmd_t is wider than a word, as on i386-PAE where pmd_install() publishes two words with a single cmpxchg8b. Use pmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what __pte_offset_map() already uses on this very pmd a few instructions later. It is pmdp_get() everywhere else, so the report is silenced the same way.

Quoted 2 lines

No functional change on architectures that can load a pmd in one go.

Do we have a Fixes: tag?

I'll give this some review time on the list; once review is done I'll queue this for the next merge window.

Quoted 13 lines

---
Changelog:
Use pmdp_get_lockless() instead of pmdp_get() (David Hildenbrand).

 mm/memory.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
mm/memory.c +0 -0
index 0fb676dd2bd63..2c1e50b70faeb 100644

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6023

Quoted 8 lines
mm/memory.c +4 -1
@@ -6017,7 +6017,11 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {
+	/*
+	 * Lockless: pmd_install() rechecks under the pmd lock, so a stale
+	 * pmd_none() here only wastes the prealloc_pte allocated below.
+	 */

I assume this comment can be shortened. Can be changed when applying.

Signature · 3 lines
Cheers,

David