mm/memory: use pmdp_get_lockless() for the pmd read in do_fault_around()
Subsystems (2)
PATCH v3 1/1 mm/memory: use pmdp_get_lockless() for the pmd read in do_fault_around()
From: Hui Zhu <[email protected]>
KCSAN reports a data-race in do_read_fault / pmd_install; the same code is in mainline:
write (marked) to 0xffff8d52a75fb810 of 8 bytes by task 27487 on cpu 5:
native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline] pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline] pmd_install+0xe0/0x120 mm/memory.c:444 finish_fault+0x5bc/0xb00 mm/memory.c:5204 do_cow_fault mm/memory.c:5432 [inline] do_fault+0x5d0/0x6e0 mm/memory.c:5526 handle_mm_fault+0x211/0x4c0 mm/memory.c:6218 __mm_populate+0x144/0x270 mm/gup.c:2207 __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764
read to 0xffff8d52a75fb810 of 8 bytes by task 27527 on cpu 9:
do_fault_around mm/memory.c:5341 [inline] do_read_fault+0x162/0x670 mm/memory.c:5381 do_fault+0x31a/0x6e0 mm/memory.c:5524 handle_mm_fault+0x211/0x4c0 mm/memory.c:6218 faultin_page_range+0x227/0x530 mm/gup.c:2156 __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479
value changed: 0x0000000000000000 -> 0x000000001386c067
Both paths only hold mmap_lock for read, so do_fault_around() tests vmf->pmd while pmd_install() populates it under the pmd lock. The race is
benign: a stale pmd_none() only wastes the prealloc_pte, which do_fault() frees again, and a pmd seen as populated only makes ->map_pages() find no PTE and do_read_fault() fall back to __do_fault().
The load still has to be lockless-safe. pmdp_get() is a plain READ_ONCE() and can tear where pmd_t is wider than a word, as on i386-PAE where pmd_install() publishes two words with a single cmpxchg8b. Use pmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what __pte_offset_map() already uses on this very pmd a few instructions later. It is pmdp_get() everywhere else, so the report is silenced the same way.
No functional change on architectures that can load a pmd in one go.
---
Changelog:
v3:
Shorten the comment to one line (David Hildenbrand). v2:
Use pmdp_get_lockless() instead of pmdp_get() (David Hildenbrand).
mm/memory.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
mm/memory.c +2 -1
@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) to_pte =min3 (from_pte + nr_pages, (pgoff_t )PTRS_PER_PTE , pte_off +vma_pages (vmf->vma) - vma_off) -1 ; -if (pmd_none (*vmf->pmd)) { +/* pmd_install() rechecks pmd_none() under the pmd lock. */ +if (pmd_none (pmdp_get_lockless (vmf->pmd))) { vmf->prealloc_pte =pte_alloc_one (vmf->vma->vm_mm);if (!vmf->prealloc_pte)return VM_FAULT_OOM ;
Signature · 1 line
2.43.0
David Hildenbrand (Arm)
maintainerQuoted 58 lines · Hui Zhu
From: Hui Zhu <[email protected]>
KCSAN reports a data-race in
do_read_fault/pmd_install; the same code is in mainline:write (marked) to
0xffff8d52a75fb810of 8 bytes by task 27487 on cpu 5:native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline] pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline] pmd_install+0xe0/0x120 mm/memory.c:444 finish_fault+0x5bc/0xb00 mm/memory.c:5204 do_cow_fault mm/memory.c:5432 [inline] do_fault+0x5d0/0x6e0 mm/memory.c:5526 handle_mm_fault+0x211/0x4c0 mm/memory.c:6218 __mm_populate+0x144/0x270 mm/gup.c:2207 __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764read to
0xffff8d52a75fb810of 8 bytes by task 27527 on cpu 9:do_fault_around mm/memory.c:5341 [inline] do_read_fault+0x162/0x670 mm/memory.c:5381 do_fault+0x31a/0x6e0 mm/memory.c:5524 handle_mm_fault+0x211/0x4c0 mm/memory.c:6218 faultin_page_range+0x227/0x530 mm/gup.c:2156 __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479value changed:
0x0000000000000000->0x000000001386c067Both paths only hold
mmap_lockfor read, sodo_fault_around()testsvmf->pmdwhilepmd_install()populates it under the pmd lock. The race is
benign: a stalepmd_none()only wastes theprealloc_pte, whichdo_fault()frees again, and a pmd seen as populated only makes ->map_pages() find no PTE anddo_read_fault()fall back to__do_fault().The load still has to be lockless-safe.
pmdp_get()is a plainREAD_ONCE()and can tear wherepmd_tis wider than a word, as on i386-PAE wherepmd_install()publishes two words with a single cmpxchg8b. Usepmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what__pte_offset_map()already uses on this very pmd a few instructions later. It ispmdp_get()everywhere else, so the report is silenced the same way.No functional change on architectures that can load a pmd in one go.
Signed-off-by Hui Zhu <[email protected]>---
Changelog:
v3:
Shorten the comment to one line (David Hildenbrand). v2:
Usepmdp_get_lockless()instead ofpmdp_get()(David Hildenbrand).mm/memory.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/memory.c +0
-0
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6024
Quoted 9 lines
mm/memory.c +2
-1@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) to_pte =min3 (from_pte + nr_pages, (pgoff_t )PTRS_PER_PTE , pte_off +vma_pages (vmf->vma) - vma_off) -1 ; -if (pmd_none (*vmf->pmd)) { +/* pmd_install() rechecks pmd_none() under the pmd lock. */ +if (pmd_none (pmdp_get_lockless (vmf->pmd))) { vmf->prealloc_pte =pte_alloc_one (vmf->vma->vm_mm);if (!vmf->prealloc_pte)return VM_FAULT_OOM ;
I'll wait for for review and then pick it up.
Signature · 3 lines
Cheers, David
Lorenzo Stoakes (ARM)
reviewerQuoted 26 lines · Hui Zhu
From: Hui Zhu <[email protected]>
KCSAN reports a data-race in
do_read_fault/pmd_install; the same code is in mainline:write (marked) to
0xffff8d52a75fb810of 8 bytes by task 27487 on cpu 5:native_set_pmd arch/x86/include/asm/pgtable_64.h:83 [inline] pmd_populate arch/x86/include/asm/pgalloc.h:84 [inline] pmd_install+0xe0/0x120 mm/memory.c:444 finish_fault+0x5bc/0xb00 mm/memory.c:5204 do_cow_fault mm/memory.c:5432 [inline] do_fault+0x5d0/0x6e0 mm/memory.c:5526 handle_mm_fault+0x211/0x4c0 mm/memory.c:6218 __mm_populate+0x144/0x270 mm/gup.c:2207 __do_sys_mlockall+0x2cd/0x380 mm/mlock.c:764read to
0xffff8d52a75fb810of 8 bytes by task 27527 on cpu 9:do_fault_around mm/memory.c:5341 [inline] do_read_fault+0x162/0x670 mm/memory.c:5381 do_fault+0x31a/0x6e0 mm/memory.c:5524 handle_mm_fault+0x211/0x4c0 mm/memory.c:6218 faultin_page_range+0x227/0x530 mm/gup.c:2156 __x64_sys_madvise+0x67/0x80 mm/madvise.c:1479value changed:
0x0000000000000000->0x000000001386c067
Quoted 8 lines
Both paths only hold
mmap_lockfor read, sodo_fault_around()testsvmf->pmdwhilepmd_install()populates it under the pmd lock. The race is
benign: a stalepmd_none()only wastes theprealloc_pte, whichdo_fault()frees again, and a pmd seen as populated only makes ->map_pages() find no PTE anddo_read_fault()fall back to__do_fault().The load still has to be lockless-safe.
pmdp_get()is a plainREAD_ONCE()and can tear wherepmd_tis wider than a word, as on i386-PAE where
Isn't i386 no longer supported? Are there any other arches we are worried about?
It seems now after Arnd's removals this has to be manually selected by the user.
Other than that we are worried about mips32 with 64 bit addresses and sh with x2tlb. More museum-piece stuff...
I hate that we're impacted by this.
Anyway you should probably call out the relevant arches here.
Quoted 11 lines
pmd_install()publishes two words with a single cmpxchg8b. Usepmdp_get_lockless(), the accessor for pmd reads without the pmd lock and what__pte_offset_map()already uses on this very pmd a few instructions later. It ispmdp_get()everywhere else, so the report is silenced the same way.No functional change on architectures that can load a pmd in one go.
Signed-off-by Hui Zhu <[email protected]>
Quoted 6 lines
---
Changelog:
v3:
Shorten the comment to one line (David Hildenbrand). v2:
Usepmdp_get_lockless()instead ofpmdp_get()(David Hildenbrand).
Please add links to previous discussion, it's difficult to track stuff otherwise.
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6020
Quoted 13 lines
mm/memory.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)mm/memory.c +0
-1@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) to_pte =min3 (from_pte + nr_pages, (pgoff_t )PTRS_PER_PTE , pte_off +vma_pages (vmf->vma) - vma_off) -1 ; -if (pmd_none (*vmf->pmd)) {
This pattern is definitely bad I think, we have it in many other places so should probably convert those to pmdp_get() at least?
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6021
Quoted 2 lines
mm/memory.c +2
-0+/* pmd_install() rechecks pmd_none() under the pmd lock. */ +if (pmd_none (pmdp_get_lockless (vmf->pmd))) {
It definitely seems right to use at least pmdp_get() here, but I wonder about the need for pmdp_get_lockless() here.
pmd_none() only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:
From x86 pmd_none():
/* Only check low word on 32-bit platforms, since it might be out of sync with upper half. */
And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.
And we have other instances of this pattern all over the place, when you read the PMD you don't hold the PMD lock at all so 'lockless' is kinda questionable here.
I mean yes it's lockless but no more than other cases like this.
So either I think use pmdp_get() or we need to go convert everywhere else and be strict on pmdp_get() only being used with the PMD lock held.
Quoted 6 lines
mm/memory.c +0
-1vmf->prealloc_pte =pte_alloc_one (vmf->vma->vm_mm);if (!vmf->prealloc_pte)return VM_FAULT_OOM ; --2.43.0
--
Cheers, Lorenzo
David Hildenbrand (Arm)
maintainerQuoted 10 lines
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6020
Quoted 7 lines
+++
b/mm/memory.cmm/memory.c +0
-1@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) to_pte =min3 (from_pte + nr_pages, (pgoff_t )PTRS_PER_PTE , pte_off +vma_pages (vmf->vma) - vma_off) -1 ; -if (pmd_none (*vmf->pmd)) {This pattern is definitely bad I think, we have it in many other places so should probably convert those to
pmdp_get()at least?
Usama is working on this as part of separating HW and SW PMDs (he only sent out the PTE part so far).
Quoted 6 lines
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6021
Quoted 3 lines
mm/memory.c +2
-0+/* pmd_install() rechecks pmd_none() under the pmd lock. */ +if (pmd_none (pmdp_get_lockless (vmf->pmd))) {It definitely seems right to use at least
pmdp_get()here, but I wonder about the need forpmdp_get_lockless()here.
Whenever we're not holding the page table lock we should be using pmdp_get_lockless().
Even if they are semantically equivalent for most cases, it serves as a good reminder that there are no guarantees.
And that you have to look again under PT lock.
Quoted 4 lines
pmd_none()only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:
Quoted 8 lines
From x86
pmd_none():/* Only check low word on 32-bit platforms, since it might be out of sync with upper half. */And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.
I don't understand the discussion really.
No PT lock -> use the lockless variant.
Until we can remove the lockless variant.
Are we consistent with that? Likely not.
But that's all independent of this patch here.
Signature · 3 lines
Cheers, David
Quoted 44 lines · David Hildenbrand (Arm)
Quoted 10 lines
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6020
Quoted 7 lines
+++
b/mm/memory.cmm/memory.c +0
-1@@ -6017,7 +6017,8 @@ static vm_fault_t do_fault_around(struct vm_fault *vmf) to_pte =min3 (from_pte + nr_pages, (pgoff_t )PTRS_PER_PTE , pte_off +vma_pages (vmf->vma) - vma_off) -1 ; -if (pmd_none (*vmf->pmd)) {This pattern is definitely bad I think, we have it in many other places so should probably convert those to
pmdp_get()at least?Usama is working on this as part of separating HW and SW PMDs (he only sent out the PTE part so far).
Quoted 6 lines
mm/memory.c static vm_fault_t do_fault_around(struct vm_fault *vmf) · line 6021
Quoted 3 lines
mm/memory.c +2
-0+/* pmd_install() rechecks pmd_none() under the pmd lock. */ +if (pmd_none (pmdp_get_lockless (vmf->pmd))) {It definitely seems right to use at least
pmdp_get()here, but I wonder about the need forpmdp_get_lockless()here.Whenever we're not holding the page table lock we should be using
pmdp_get_lockless().Even if they are semantically equivalent for most cases, it serves as a good reminder that there are no guarantees.
And that you have to look again under PT lock.
Quoted 4 lines
pmd_none()only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:Quoted 8 lines
From x86
pmd_none():/* Only check low word on 32-bit platforms, since it might be out of sync with upper half. */And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.
I don't understand the discussion really.
Quoted 7 lines
No PT lock -> use the lockless variant.
Until we can remove the lockless variant.
Are we consistent with that? Likely not.
But that's all independent of this patch here.
It turns out all of this is a complicated mess, I think I've nerd-sniped myself into sending a series at some point about this...
It turns out we're as wildly inconsistent on pmdp_get_lockless() as we are on pmdp_get(), so TL;DR my main concern is moot so:
But for the non-TL;DR... (optional reading, will probably send something):
* Agreed that the semantics matter - and the 'use pXXp_get_lockless() if no locks, otherwise pXXp_get()' is preferable.
* But it turns out it's all a giant mess (of course)... for CONFIG_GUP_GET_PXX_LOW_HIGH arches you really actually need to also do:
pmdp_get_lockless_start();
pmd = pmdp_get_lockless(...);pmdp_get_lockless_end();
And that's because, as Hugh points out in commit 146b42e07494 ("mm/pgtable: add PAE safety to __pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.
The explanation is quite good:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146b42e07494e45f7c7bcf2cbf7afd1424afd78e
And then of course there's also pmdp_get_lockless_sync() used by THP collapse which is some kind of 'true' lockless usage that needs to synchronise with TLB invalidation against GUP-fast and ugh.
* So I think what matters is whether you actually NEED the value un-torn or whether it's an optimistic check that you can live with (rechecked under lock as needed).
* So I wonder if we shouldn't just have e.g. pmdp_get_lockless_optimistic() or something like that which is just a pmdp_get() in all cases with correct kdoc description etc.
* Then we can convert ALL lockless users to using a lockless form without having to do the IRQ stuff. Including a ton of the existing lockless users.
* I am not sure why I did this in bullets. :)
--
Cheers, Lorenzo
David Hildenbrand (Arm)
maintainerQuoted 43 lines · Lorenzo Stoakes (ARM)
Quoted 42 lines · David Hildenbrand (Arm)
Quoted 4 lines
This pattern is definitely bad I think, we have it in many other places so should probably convert those to
pmdp_get()at least?Usama is working on this as part of separating HW and SW PMDs (he only sent out the PTE part so far).
Quoted 4 lines
It definitely seems right to use at least
pmdp_get()here, but I wonder about the need forpmdp_get_lockless()here.Whenever we're not holding the page table lock we should be using
pmdp_get_lockless().Even if they are semantically equivalent for most cases, it serves as a good reminder that there are no guarantees.
And that you have to look again under PT lock.
Quoted 4 lines
pmd_none()only checks the lower word, so a torn 64-bit word on 32-bit (i.e. the PAE style cases we are worried about) makes no difference:Quoted 8 lines
From x86
pmd_none():/* Only check low word on 32-bit platforms, since it might be out of sync with upper half. */And anyway I guess there is no valid PMD in such systems that would have the lower word cleared and upper word not cleared.
I don't understand the discussion really.
No PT lock -> use the lockless variant.
Until we can remove the lockless variant.
Are we consistent with that? Likely not.
Quoted 7 lines
Quoted 1 line
But that's all independent of this patch here.
It turns out all of this is a complicated mess, I think I've nerd-sniped myself into sending a series at some point about this...
It turns out we're as wildly inconsistent on
pmdp_get_lockless()as we are onpmdp_get(), so TL;DR my main concern is moot so:
Yeah, I know that we use it in some core places already properly, like in GUP-fast, __handle_mm_fault() or our complicated __pte_offset_map().
Quoted 8 lines
But for the non-TL;DR... (optional reading, will probably send something):
* Agreed that the semantics matter - and the 'use
pXXp_get_lockless()if no locks, otherwisepXXp_get()' is preferable.
Ack.
Quoted 10 lines
* But it turns out it's all a giant mess (of course)... for
CONFIG_GUP_GET_PXX_LOW_HIGHarches you really actually need to also do:
pmdp_get_lockless_start();
pmd =pmdp_get_lockless(...);pmdp_get_lockless_end();And that's because, as Hugh points out in commit
146b42e07494("mm/pgtable: add PAE safety to__pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.
Yes, there is this pmdp_get_lockless_start() magic ....
Quoted 12 lines
The explanation is quite good:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=146b42e07494e45f7c7bcf2cbf7afd1424afd78eAnd then of course there's also
pmdp_get_lockless_sync()used by THP collapse which is some kind of 'true' lockless usage that needs to synchronise with TLB invalidation against GUP-fast and ugh.* So I think what matters is whether you actually NEED the value un-torn or whether it's an optimistic check that you can live with (rechecked under lock as needed).
Quoted 8 lines
* So I wonder if we shouldn't just have e.g.
pmdp_get_lockless_optimistic()or something like that which is just apmdp_get()in all cases with correct kdoc description etc.* Then we can convert ALL lockless users to using a lockless form without having to do the IRQ stuff. Including a ton of the existing lockless users.
* I am not sure why I did this in bullets. :)
It's really a mess. I wonder if we can remove that somehow completely ...
arch/mips/Kconfig: select GUP_GET_PXX_LOW_HIGH if CPU_MIPS32 &&PHYS_ADDR_T_64BIT
arch/sh/Kconfig: select GUP_GET_PXX_LOW_HIGH if X2TLB arch/x86/Kconfig: select GUP_GET_PXX_LOW_HIGH if X86_PAE
Signature · 3 lines
Cheers, David
Lorenzo Stoakes (ARM)
reviewerQuoted 6 lines · David Hildenbrand (Arm)
Quoted 2 lines · Lorenzo Stoakes (ARM)
It turns out we're as wildly inconsistent on
pmdp_get_lockless()as we are onpmdp_get(), so TL;DR my main concern is moot so:Yeah, I know that we use it in some core places already properly, like in GUP-fast,
__handle_mm_fault()or our complicated__pte_offset_map().
Yup, but also inconsistently, basically mostly interchangeably with pmdp_get(). Rarely is this value actually used.
Quoted 7 lines
Quoted 7 lines
* But it turns out it's all a giant mess (of course)... for
CONFIG_GUP_GET_PXX_LOW_HIGHarches you really actually need to also do:
pmdp_get_lockless_start();
pmd =pmdp_get_lockless(...);pmdp_get_lockless_end();
Quoted 5 lines
Quoted 3 lines
And that's because, as Hugh points out in commit
146b42e07494("mm/pgtable: add PAE safety to__pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.Yes, there is this
pmdp_get_lockless_start()magic ....
One of those things that, the more you look into it, the more stuff you find...
Quoted 6 lines
It's really a mess. I wonder if we can remove that somehow completely ...
arch/mips/Kconfig: selectGUP_GET_PXX_LOW_HIGHifCPU_MIPS32&&PHYS_ADDR_T_64BITarch/sh/Kconfig: select GUP_GET_PXX_LOW_HIGH if X2TLB arch/x86/Kconfig: select GUP_GET_PXX_LOW_HIGH if X86_PAE
Yeah that'd be ideal.
But actually could we not just simplify and allow these arches to be a little bit slower (who cares for museum pieces)?
The only user of the start/end magic is __pte_offset_map(). And it does:
irqflags = pmdp_get_lockless_start(); pmdval = pmdp_get_lockless(pmd); pmdp_get_lockless_end(irqflags);
So that could just get folded into pmdp_get_lockless() for these arches.
That would mean the places that don't require IRQs to be disabled (optimstic checks) would now be needlessly enabling/disabling, but hey - use a system that doesn't live in a museum :)
That way we could safely convert every non-PMD lock holding caller to pmdp_get_lockless() without needing another pmdp_get*() variant + fix confusion about whether the start/end magic is required.
And this would work even in places where the IRQs are already disabled as it's fine with nesting.
So you'd just update thusly:
static inline pmd_t pmdp_get_lockless(pmd_t *pmdp)
{
pmd_t pmd;+ unsigned long flags;
+ local_irq_save(flags);
do {
pmd.pmd_low = pmdp->pmd_low;
smp_rmb();
pmd.pmd_high = pmdp->pmd_high;
smp_rmb();
} while (unlikely(pmd.pmd_low != pmdp->pmd_low));
-
+ local_irq_restore(flags);
return pmd;
}...
pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) {
- unsigned long irqflags; pmd_t pmdval; rcu_read_lock(); - irqflags = pmdp_get_lockless_start(); pmdval = pmdp_get_lockless(pmd);
- pmdp_get_lockless_end(irqflags);
... }
What do you think?
--
Cheers, Lorenzo
David Hildenbrand (Arm)
maintainerQuoted 44 lines · Lorenzo Stoakes (ARM)
Quoted 6 lines · David Hildenbrand (Arm)
Quoted 2 lines · Lorenzo Stoakes (ARM)
It turns out we're as wildly inconsistent on
pmdp_get_lockless()as we are onpmdp_get(), so TL;DR my main concern is moot so:Yeah, I know that we use it in some core places already properly, like in GUP-fast,
__handle_mm_fault()or our complicated__pte_offset_map().Yup, but also inconsistently, basically mostly interchangeably with
pmdp_get(). Rarely is this value actually used.Quoted 7 lines
Quoted 7 lines
* But it turns out it's all a giant mess (of course)... for
CONFIG_GUP_GET_PXX_LOW_HIGHarches you really actually need to also do:
pmdp_get_lockless_start();
pmd =pmdp_get_lockless(...);pmdp_get_lockless_end();Quoted 5 lines
Quoted 3 lines
And that's because, as Hugh points out in commit
146b42e07494("mm/pgtable: add PAE safety to__pte_offset_map()")), you need IRQs disabled for these arches for it to be safe.Yes, there is this
pmdp_get_lockless_start()magic ....One of those things that, the more you look into it, the more stuff you find...
Quoted 6 lines
It's really a mess. I wonder if we can remove that somehow completely ...
arch/mips/Kconfig: selectGUP_GET_PXX_LOW_HIGHifCPU_MIPS32&&PHYS_ADDR_T_64BITarch/sh/Kconfig: select GUP_GET_PXX_LOW_HIGH if X2TLB arch/x86/Kconfig: select GUP_GET_PXX_LOW_HIGH if X86_PAEYeah that'd be ideal.
But actually could we not just simplify and allow these arches to be a little bit slower (who cares for museum pieces)?
The only user of the start/end magic is
__pte_offset_map(). And it does:irqflags = pmdp_get_lockless_start(); pmdval = pmdp_get_lockless(pmd); pmdp_get_lockless_end(irqflags);
Quoted 5 lines
So that could just get folded into
pmdp_get_lockless()for these arches.That would mean the places that don't require IRQs to be disabled (optimstic checks) would now be needlessly enabling/disabling, but hey - use a system that doesn't live in a museum :)
It would be interesting to know the performance impact. IOW, whether this is just a slight slowdown or whether it makes these installations unusable.
It looks like an interesting thing.
Quoted 42 lines
That way we could safely convert every non-PMD lock holding caller to
pmdp_get_lockless()without needing another pmdp_get*() variant + fix confusion about whether the start/end magic is required.And this would work even in places where the IRQs are already disabled as it's fine with nesting.
So you'd just update thusly:
static inline
pmd_tpmdp_get_lockless(pmd_t *pmdp){ pmd_t pmd;(quoted diff) +2
-0+ unsigned long flags; + local_irq_save(flags);do { pmd.pmd_low = pmdp->pmd_low; smp_rmb(); pmd.pmd_high = pmdp->pmd_high; smp_rmb(); } while (unlikely(pmd.pmd_low != pmdp->pmd_low));(quoted diff) +1
-1- + local_irq_restore(flags);return pmd; }...
pte_t*__pte_offset_map(pmd_t *pmd, unsigned long addr,pmd_t*pmdvalp) {(quoted diff) +0
-1- unsigned long irqflags;pmd_t pmdval; rcu_read_lock();(quoted diff) +0
-1- irqflags = pmdp_get_lockless_start();pmdval = pmdp_get_lockless(pmd);(quoted diff) +0
-1- pmdp_get_lockless_end(irqflags);}
Quoted 1 line
What do you think?
Sounds reasonable. I guess we should be able to disable IRQs on all callsites.
I am quite surprised that x86 shouldn't have an instruction to read/write PTEs atomically.
I think if we require MMX we could just do atomic read+write, and let x86 implement pmdp_get_lockless() / pmdp_get() itself. But MMX seems to be a bit younger than PAE. Apparently there is a hacky way to read through CMPXCHG8B.
Doesn't sound like a good use of our time ... but removing that handling entirely would be very nice.
Simplifying is nice as well.
Signature · 3 lines
Cheers, David