Skip to contentSkip to conversation

mm/memory: use pmdp_get_lockless() for the pmd read in do_fault_around()

PATCH Superseded Hui Zhu 3 participants 8 messages Nov 3 – Nov 4

Subsystems (2)

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Hui Zhu

author
#0 lore raw

PATCH v3 1/1 mm/memory: use pmdp_get_lockless() for the pmd read in do_fault_around()

From: Hui Zhu <[email protected]>

KCSAN reports a data-race in do_read_fault / pmd_install; the same code is in mainline:

write (marked) to 0xffff8d52a75fb810 of 8 bytes by task 27487 on cpu 5:

   native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline]
   pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline]
   pmd_install+0xe0/0x120 mm/memory.c:444
   finish_fault+0x5bc/0xb00 mm/memory.c:5204
   do_cow_fault mm/memory.c:5432 [inline]
   do_fault+0x5d0/0x6e0 mm/memory.c:5526
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   __mm_populate+0x144/0x270 mm/gup.c:2207
   __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764

read to 0xffff8d52a75fb810 of 8 bytes by task 27527 on cpu 9:

   do_fault_around mm/memory.c:5341 [inline]
   do_read_fault+0x162/0x670 mm/memory.c:5381
   do_fault+0x31a/0x6e0 mm/memory.c:5524
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   faultin_page_range+0x227/0x530 mm/gup.c:2156
   __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479

value changed: 0x0000000000000000 -> 0x000000001386c067

Both paths only hold mmap_lock for read, so do_fault_around() tests vmf->pmd while pmd_install() populates it under the pmd lock. The race is
benign: a stale pmd_none() only wastes the prealloc_pte, which do_fault() frees again, and a pmd seen as populated only makes ->map_pages() find no PTE and do_read_fault() fall back to __do_fault().

The load still has to be lockless-safe. pmdp_get() is a plain READ_ONCE() and can tear where pmd_t is wider than a word, as on i386-PAE where pmd_install() publishes two words with a single cmpxchg8b. Use pmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what __pte_offset_map() already uses on this very pmd a few instructions later. It is pmdp_get() everywhere else, so the report is silenced the same way.

No functional change on architectures that can load a pmd in one go.

---
Changelog:
v3:
Shorten the comment to one line (David Hildenbrand). v2:
Use pmdp_get_lockless() instead of pmdp_get() (David Hildenbrand).

 mm/memory.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/memory.c +2 -1
index 0fb676dd2bd63..f25c2ce0ef153 100644
@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {
+	/* pmd_install() rechecks pmd_none() under the pmd lock. */
+	if (pmd_none(pmdp_get_lockless(vmf->pmd))) {
 		vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
Signature · 1 line
2.43.0
#1 lore raw
Quoted 58 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data-race in do_read_fault / pmd_install; the same code is in mainline:

write (marked) to 0xffff8d52a75fb810 of 8 bytes by task 27487 on cpu 5:

   native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline]
   pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline]
   pmd_install+0xe0/0x120 mm/memory.c:444
   finish_fault+0x5bc/0xb00 mm/memory.c:5204
   do_cow_fault mm/memory.c:5432 [inline]
   do_fault+0x5d0/0x6e0 mm/memory.c:5526
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   __mm_populate+0x144/0x270 mm/gup.c:2207
   __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764

read to 0xffff8d52a75fb810 of 8 bytes by task 27527 on cpu 9:

   do_fault_around mm/memory.c:5341 [inline]
   do_read_fault+0x162/0x670 mm/memory.c:5381
   do_fault+0x31a/0x6e0 mm/memory.c:5524
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   faultin_page_range+0x227/0x530 mm/gup.c:2156
   __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479

value changed: 0x0000000000000000 -> 0x000000001386c067

Both paths only hold mmap_lock for read, so do_fault_around() tests vmf->pmd while pmd_install() populates it under the pmd lock. The race is
benign: a stale pmd_none() only wastes the prealloc_pte, which do_fault() frees again, and a pmd seen as populated only makes ->map_pages() find no PTE and do_read_fault() fall back to __do_fault().

The load still has to be lockless-safe. pmdp_get() is a plain READ_ONCE() and can tear where pmd_t is wider than a word, as on i386-PAE where pmd_install() publishes two words with a single cmpxchg8b. Use pmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what __pte_offset_map() already uses on this very pmd a few instructions later. It is pmdp_get() everywhere else, so the report is silenced the same way.

No functional change on architectures that can load a pmd in one go.

---
Changelog:
v3:
Shorten the comment to one line (David Hildenbrand). v2:
Use pmdp_get_lockless() instead of pmdp_get() (David Hildenbrand).

 mm/memory.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/memory.c +0 -0
index 0fb676dd2bd63..f25c2ce0ef153 100644

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6024

Quoted 9 lines
mm/memory.c +2 -1
@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {
+	/* pmd_install() rechecks pmd_none() under the pmd lock. */
+	if (pmd_none(pmdp_get_lockless(vmf->pmd))) {
 		vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;

I'll wait for for review and then pick it up.

Signature · 3 lines
Cheers,

David
#2 lore raw
In reply to Hui Zhu
Quoted 26 lines · Hui Zhu

From: Hui Zhu <[email protected]>

KCSAN reports a data-race in do_read_fault / pmd_install; the same code is in mainline:

write (marked) to 0xffff8d52a75fb810 of 8 bytes by task 27487 on cpu 5:

   native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline]
   pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline]
   pmd_install+0xe0/0x120 mm/memory.c:444
   finish_fault+0x5bc/0xb00 mm/memory.c:5204
   do_cow_fault mm/memory.c:5432 [inline]
   do_fault+0x5d0/0x6e0 mm/memory.c:5526
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   __mm_populate+0x144/0x270 mm/gup.c:2207
   __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764

read to 0xffff8d52a75fb810 of 8 bytes by task 27527 on cpu 9:

   do_fault_around mm/memory.c:5341 [inline]
   do_read_fault+0x162/0x670 mm/memory.c:5381
   do_fault+0x31a/0x6e0 mm/memory.c:5524
   handle_mm_fault+0x211/0x4c0 mm/memory.c:6218
   faultin_page_range+0x227/0x530 mm/gup.c:2156
   __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479

value changed: 0x0000000000000000 -> 0x000000001386c067

Quoted 8 lines

Both paths only hold mmap_lock for read, so do_fault_around() tests vmf->pmd while pmd_install() populates it under the pmd lock. The race is
benign: a stale pmd_none() only wastes the prealloc_pte, which do_fault() frees again, and a pmd seen as populated only makes ->map_pages() find no PTE and do_read_fault() fall back to __do_fault().

The load still has to be lockless-safe. pmdp_get() is a plain READ_ONCE() and can tear where pmd_t is wider than a word, as on i386-PAE where

Isn't i386 no longer supported? Are there any other arches we are worried about?

It seems now after Arnd's removals this has to be manually selected by the user.

Other than that we are worried about mips32 with 64 bit addresses and sh with x2tlb. More museum-piece stuff...

I hate that we're impacted by this.

Anyway you should probably call out the relevant arches here.

Quoted 11 lines

pmd_install() publishes two words with a single cmpxchg8b. Use pmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what __pte_offset_map() already uses on this very pmd a few instructions later. It is pmdp_get() everywhere else, so the report is silenced the same way.

No functional change on architectures that can load a pmd in one go.

Quoted 6 lines

---
Changelog:
v3:
Shorten the comment to one line (David Hildenbrand). v2:
Use pmdp_get_lockless() instead of pmdp_get() (David Hildenbrand).

Please add links to previous discussion, it's difficult to track stuff otherwise.

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6020

Quoted 13 lines
 mm/memory.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
mm/memory.c +0 -1
index 0fb676dd2bd63..f25c2ce0ef153 100644
@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {

This pattern is definitely bad I think, we have it in many other places so should probably convert those to pmdp_get() at least?

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6021

Quoted 2 lines
mm/memory.c +2 -0
+	/* pmd_install() rechecks pmd_none() under the pmd lock. */
+	if (pmd_none(pmdp_get_lockless(vmf->pmd))) {

It definitely seems right to use at least pmdp_get() here, but I wonder about the need for pmdp_get_lockless() here.

pmd_none() only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:

From x86 pmd_none():

	/* Only check low word on 32-bit platforms, since it might be
	   out of sync with upper half. */

And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.

And we have other instances of this pattern all over the place, when you read the PMD you don't hold the PMD lock at all so 'lockless' is kinda questionable here.

I mean yes it's lockless but no more than other cases like this.

So either I think use pmdp_get() or we need to go convert everywhere else and be strict on pmdp_get() only being used with the PMD lock held.

Quoted 6 lines
mm/memory.c +0 -1
 		vmf->prealloc_pte = pte_alloc_one(vmf->vma->vm_mm);
 		if (!vmf->prealloc_pte)
 			return VM_FAULT_OOM;
--

2.43.0

--
Cheers, Lorenzo

#3 lore raw
Quoted 10 lines

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6020

Quoted 7 lines

+++ b/mm/memory.c

mm/memory.c +0 -1
@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {

This pattern is definitely bad I think, we have it in many other places so should probably convert those to pmdp_get() at least?

Usama is working on this as part of separating HW and SW PMDs (he only sent out the PTE part so far).

Quoted 6 lines

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6021

Quoted 3 lines
mm/memory.c +2 -0
+	/* pmd_install() rechecks pmd_none() under the pmd lock. */
+	if (pmd_none(pmdp_get_lockless(vmf->pmd))) {

It definitely seems right to use at least pmdp_get() here, but I wonder about the need for pmdp_get_lockless() here.

Whenever we're not holding the page table lock we should be using pmdp_get_lockless().

Even if they are semantically equivalent for most cases, it serves as a good reminder that there are no guarantees.

And that you have to look again under PT lock.

Quoted 4 lines

pmd_none() only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:

Quoted 8 lines

From x86 pmd_none():

	/* Only check low word on 32-bit platforms, since it might be
	   out of sync with upper half. */

And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.

I don't understand the discussion really.

No PT lock -> use the lockless variant.

Until we can remove the lockless variant.

Are we consistent with that? Likely not.

But that's all independent of this patch here.

Signature · 3 lines
Cheers,

David
reviewer Acked-by
#4 lore raw
Quoted 44 lines · David Hildenbrand (Arm)
Quoted 10 lines

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6020

Quoted 7 lines

+++ b/mm/memory.c

mm/memory.c +0 -1
@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf)
 	to_pte = min3(from_pte + nr_pages, (pgoff_t)PTRS_PER_PTE,
 		      pte_off + vma_pages(vmf->vma) - vma_off) - 1;
 
-	if (pmd_none(*vmf->pmd)) {

This pattern is definitely bad I think, we have it in many other places so should probably convert those to pmdp_get() at least?

Usama is working on this as part of separating HW and SW PMDs (he only sent out the PTE part so far).

Quoted 6 lines

mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6021

Quoted 3 lines
mm/memory.c +2 -0
+	/* pmd_install() rechecks pmd_none() under the pmd lock. */
+	if (pmd_none(pmdp_get_lockless(vmf->pmd))) {

It definitely seems right to use at least pmdp_get() here, but I wonder about the need for pmdp_get_lockless() here.

Whenever we're not holding the page table lock we should be using pmdp_get_lockless().

Even if they are semantically equivalent for most cases, it serves as a good reminder that there are no guarantees.

And that you have to look again under PT lock.

Quoted 4 lines

pmd_none() only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:

Quoted 8 lines

From x86 pmd_none():

	/* Only check low word on 32-bit platforms, since it might be
	   out of sync with upper half. */

And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.

I don't understand the discussion really.

Quoted 7 lines

No PT lock -> use the lockless variant.

Until we can remove the lockless variant.

Are we consistent with that? Likely not.

But that's all independent of this patch here.

It turns out all of this is a complicated mess, I think I've nerd-sniped myself into sending a series at some point about this...

It turns out we're as wildly inconsistent on pmdp_get_lockless() as we are on pmdp_get(), so TL;DR my main concern is moot so:

But for the non-TL;DR... (optional reading, will probably send something):

* Agreed that the semantics matter - and the 'use pXXp_get_lockless() if no locks, otherwise pXXp_get()' is preferable.

* But it turns out it's all a giant mess (of course)... for CONFIG_GUP_GET_PXX_LOW_HIGH arches you really actually need to also do:

pmdp_get_lockless_start();
pmd = pmdp_get_lockless(...);
pmdp_get_lockless_end();

And that's because, as Hugh points out in commit 146b42e07494 ("mm/pgtable: add PAE safety to __pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.

The explanation is quite good:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146b42e07494e45f7c7bcf2cbf7afd1424afd78e

And then of course there's also pmdp_get_lockless_sync() used by THP collapse which is some kind of 'true' lockless usage that needs to synchronise with TLB invalidation against GUP-fast and ugh.

* So I think what matters is whether you actually NEED the value un-torn or whether it's an optimistic check that you can live with (rechecked under lock as needed).

* So I wonder if we shouldn't just have e.g. pmdp_get_lockless_optimistic() or something like that which is just a pmdp_get() in all cases with correct kdoc description etc.

* Then we can convert ALL lockless users to using a lockless form without having to do the IRQ stuff. Including a ton of the existing lockless users.

* I am not sure why I did this in bullets. :)

--
Cheers, Lorenzo

#5 lore raw
Quoted 43 lines · Lorenzo Stoakes (ARM)
Quoted 42 lines · David Hildenbrand (Arm)
Quoted 4 lines

This pattern is definitely bad I think, we have it in many other places so should probably convert those to pmdp_get() at least?

Usama is working on this as part of separating HW and SW PMDs (he only sent out the PTE part so far).

Quoted 4 lines

It definitely seems right to use at least pmdp_get() here, but I wonder about the need for pmdp_get_lockless() here.

Whenever we're not holding the page table lock we should be using pmdp_get_lockless().

Even if they are semantically equivalent for most cases, it serves as a good reminder that there are no guarantees.

And that you have to look again under PT lock.

Quoted 4 lines

pmd_none() only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:

Quoted 8 lines

From x86 pmd_none():

	/* Only check low word on 32-bit platforms, since it might be
	   out of sync with upper half. */

And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.

I don't understand the discussion really.

No PT lock -> use the lockless variant.

Until we can remove the lockless variant.

Are we consistent with that? Likely not.

Quoted 7 lines
Quoted 1 line

But that's all independent of this patch here.

It turns out all of this is a complicated mess, I think I've nerd-sniped myself into sending a series at some point about this...

It turns out we're as wildly inconsistent on pmdp_get_lockless() as we are on pmdp_get(), so TL;DR my main concern is moot so:

Yeah, I know that we use it in some core places already properly, like in GUP-fast, __handle_mm_fault() or our complicated __pte_offset_map().

Quoted 8 lines

But for the non-TL;DR... (optional reading, will probably send something):

* Agreed that the semantics matter - and the 'use pXXp_get_lockless() if no locks, otherwise pXXp_get()' is preferable.

Ack.

Quoted 10 lines

* But it turns out it's all a giant mess (of course)... for CONFIG_GUP_GET_PXX_LOW_HIGH arches you really actually need to also do:

pmdp_get_lockless_start();
pmd = pmdp_get_lockless(...);
pmdp_get_lockless_end();

And that's because, as Hugh points out in commit 146b42e07494 ("mm/pgtable: add PAE safety to __pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.

Yes, there is this pmdp_get_lockless_start() magic ....

Quoted 12 lines

The explanation is quite good:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146b42e07494e45f7c7bcf2cbf7afd1424afd78e

And then of course there's also pmdp_get_lockless_sync() used by THP collapse which is some kind of 'true' lockless usage that needs to synchronise with TLB invalidation against GUP-fast and ugh.

* So I think what matters is whether you actually NEED the value un-torn or whether it's an optimistic check that you can live with (rechecked under lock as needed).

Quoted 8 lines

* So I wonder if we shouldn't just have e.g. pmdp_get_lockless_optimistic() or something like that which is just a pmdp_get() in all cases with correct kdoc description etc.

* Then we can convert ALL lockless users to using a lockless form without having to do the IRQ stuff. Including a ton of the existing lockless users.

* I am not sure why I did this in bullets. :)

It's really a mess. I wonder if we can remove that somehow completely ...

arch/mips/Kconfig: select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 &&
PHYS_ADDR_T_64BIT

arch/sh/Kconfig:        select GUP_GET_PXX_LOW_HIGH if X2TLB
arch/x86/Kconfig:       select GUP_GET_PXX_LOW_HIGH             if X86_PAE
Signature · 3 lines
Cheers,

David
#6 lore raw
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 2 lines · Lorenzo Stoakes (ARM)

It turns out we're as wildly inconsistent on pmdp_get_lockless() as we are on pmdp_get(), so TL;DR my main concern is moot so:

Yeah, I know that we use it in some core places already properly, like in GUP-fast, __handle_mm_fault() or our complicated __pte_offset_map().

Yup, but also inconsistently, basically mostly interchangeably with pmdp_get(). Rarely is this value actually used.

Quoted 7 lines
Quoted 7 lines

* But it turns out it's all a giant mess (of course)... for CONFIG_GUP_GET_PXX_LOW_HIGH arches you really actually need to also do:

pmdp_get_lockless_start();
pmd = pmdp_get_lockless(...);
pmdp_get_lockless_end();

Quoted 5 lines
Quoted 3 lines

And that's because, as Hugh points out in commit 146b42e07494 ("mm/pgtable: add PAE safety to __pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.

Yes, there is this pmdp_get_lockless_start() magic ....

One of those things that, the more you look into it, the more stuff you find...

Quoted 6 lines

It's really a mess. I wonder if we can remove that somehow completely ...

arch/mips/Kconfig: select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 &&
PHYS_ADDR_T_64BIT

arch/sh/Kconfig:        select GUP_GET_PXX_LOW_HIGH if X2TLB
arch/x86/Kconfig:       select GUP_GET_PXX_LOW_HIGH             if X86_PAE

Yeah that'd be ideal.

But actually could we not just simplify and allow these arches to be a little bit slower (who cares for museum pieces)?

The only user of the start/end magic is __pte_offset_map(). And it does:

	irqflags = pmdp_get_lockless_start();
	pmdval = pmdp_get_lockless(pmd);
	pmdp_get_lockless_end(irqflags);

So that could just get folded into pmdp_get_lockless() for these arches.

That would mean the places that don't require IRQs to be disabled (optimstic checks) would now be needlessly enabling/disabling, but hey - use a system that doesn't live in a museum :)

That way we could safely convert every non-PMD lock holding caller to pmdp_get_lockless() without needing another pmdp_get*() variant + fix confusion about whether the start/end magic is required.

And this would work even in places where the IRQs are already disabled as it's fine with nesting.

So you'd just update thusly:

static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)

{
	pmd_t pmd;

+ unsigned long flags;

+	local_irq_save(flags);
	do {
		pmd.pmd_low = pmdp->pmd_low;
		smp_rmb();
		pmd.pmd_high = pmdp->pmd_high;
		smp_rmb();
	} while (unlikely(pmd.pmd_low != pmdp->pmd_low));
-
+	local_irq_restore(flags);
	return pmd;
}

...

pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) {

-	unsigned long irqflags;
	pmd_t pmdval;

	rcu_read_lock();
-	irqflags = pmdp_get_lockless_start();
	pmdval = pmdp_get_lockless(pmd);

- pmdp_get_lockless_end(irqflags);

	...
}

What do you think?

--
Cheers, Lorenzo

#7 lore raw
Quoted 44 lines · Lorenzo Stoakes (ARM)
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 2 lines · Lorenzo Stoakes (ARM)

It turns out we're as wildly inconsistent on pmdp_get_lockless() as we are on pmdp_get(), so TL;DR my main concern is moot so:

Yeah, I know that we use it in some core places already properly, like in GUP-fast, __handle_mm_fault() or our complicated __pte_offset_map().

Yup, but also inconsistently, basically mostly interchangeably with pmdp_get(). Rarely is this value actually used.

Quoted 7 lines
Quoted 7 lines

* But it turns out it's all a giant mess (of course)... for CONFIG_GUP_GET_PXX_LOW_HIGH arches you really actually need to also do:

pmdp_get_lockless_start();
pmd = pmdp_get_lockless(...);
pmdp_get_lockless_end();

Quoted 5 lines
Quoted 3 lines

And that's because, as Hugh points out in commit 146b42e07494 ("mm/pgtable: add PAE safety to __pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.

Yes, there is this pmdp_get_lockless_start() magic ....

One of those things that, the more you look into it, the more stuff you find...

Quoted 6 lines

It's really a mess. I wonder if we can remove that somehow completely ...

arch/mips/Kconfig: select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 &&
PHYS_ADDR_T_64BIT

arch/sh/Kconfig:        select GUP_GET_PXX_LOW_HIGH if X2TLB
arch/x86/Kconfig:       select GUP_GET_PXX_LOW_HIGH             if X86_PAE

Yeah that'd be ideal.

But actually could we not just simplify and allow these arches to be a little bit slower (who cares for museum pieces)?

The only user of the start/end magic is __pte_offset_map(). And it does:

	irqflags = pmdp_get_lockless_start();
	pmdval = pmdp_get_lockless(pmd);
	pmdp_get_lockless_end(irqflags);
Quoted 5 lines

So that could just get folded into pmdp_get_lockless() for these arches.

That would mean the places that don't require IRQs to be disabled (optimstic checks) would now be needlessly enabling/disabling, but hey - use a system that doesn't live in a museum :)

It would be interesting to know the performance impact. IOW, whether this is just a slight slowdown or whether it makes these installations unusable.

It looks like an interesting thing.

Quoted 42 lines

That way we could safely convert every non-PMD lock holding caller to pmdp_get_lockless() without needing another pmdp_get*() variant + fix confusion about whether the start/end magic is required.

And this would work even in places where the IRQs are already disabled as it's fine with nesting.

So you'd just update thusly:

static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)

{
	pmd_t pmd;
(quoted diff) +2 -0
+	unsigned long flags;
 
+	local_irq_save(flags);
	do {
		pmd.pmd_low = pmdp->pmd_low;
		smp_rmb();
		pmd.pmd_high = pmdp->pmd_high;
		smp_rmb();
	} while (unlikely(pmd.pmd_low != pmdp->pmd_low));
(quoted diff) +1 -1
-
+	local_irq_restore(flags);
	return pmd;
}

...

pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) {

(quoted diff) +0 -1
-	unsigned long irqflags;
	pmd_t pmdval;

	rcu_read_lock();
(quoted diff) +0 -1
-	irqflags = pmdp_get_lockless_start();
	pmdval = pmdp_get_lockless(pmd);
(quoted diff) +0 -1
-	pmdp_get_lockless_end(irqflags);
 
...

}

Quoted 1 line

What do you think?

Sounds reasonable. I guess we should be able to disable IRQs on all callsites.

I am quite surprised that x86 shouldn't have an instruction to read/write PTEs atomically.

I think if we require MMX we could just do atomic read+write, and let x86 implement pmdp_get_lockless() / pmdp_get() itself. But MMX seems to be a bit younger than PAE. Apparently there is a hacky way to read through CMPXCHG8B.

Doesn't sound like a good use of our time ... but removing that handling entirely would be very nice.

Simplifying is nice as well.

Signature · 3 lines
Cheers,

David