Skip to contentSkip to conversation

mm/gup: cleanup pgtable entry accessors

PATCH Applied Alexander Gordeev 2 participants 4 messages Apr 21 – Apr 29

Subsystems (3)
lore
Share & tools
lore

Alexander Gordeev

author maintainer
#0 lore raw

PATCH v2 1/1 mm/gup: cleanup pgtable entry accessors

PMD and PUD entries revalidation has the same semantics as PTE entry revalidation. Convert the remaining direct entry dereferences to the corresponding accessors.

The PTE validation in gup_fast_pte_range() is inconsistent with the prior value acquisition in the sense that it drops the lockless access semantics.

Use the lockless accessor not only for the PTE, but also for the PMD validation, which is likewise inconsistent with the prior value acquisition in gup_fast_pmd_range().

---
 mm/gup.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/gup.c +4 -4
index 8e7dc2c6ee73..236450feea9a 100644
@@ -2865,8 +2865,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
 		if (!folio)
 			goto pte_unmap;
 
-		if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) ||
-		    unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) {
+		if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
+		    unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
 			gup_put_folio(folio, 1, flags);
 			goto pte_unmap;
 		}
@@ -2942,7 +2942,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr,
 	if (!folio)
 		return 0;
 
-	if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
+	if (unlikely(pmd_val(orig) != pmd_val(pmdp_get_lockless(pmdp)))) {
 		gup_put_folio(folio, refs, flags);
 		return 0;
 	}
@@ -2985,7 +2985,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr,
 	if (!folio)
 		return 0;
 
-	if (unlikely(pud_val(orig) != pud_val(*pudp))) {
+	if (unlikely(pud_val(orig) != pud_val(pudp_get(pudp)))) {
 		gup_put_folio(folio, refs, flags);
 		return 0;
 	}
Signature · 1 line
2.51.0
#1 lore raw
Quoted 42 lines · Alexander Gordeev

PMD and PUD entries revalidation has the same semantics as PTE entry revalidation. Convert the remaining direct entry dereferences to the corresponding accessors.

The PTE validation in gup_fast_pte_range() is inconsistent with the prior value acquisition in the sense that it drops the lockless access semantics.

Use the lockless accessor not only for the PTE, but also for the PMD validation, which is likewise inconsistent with the prior value acquisition in gup_fast_pmd_range().

---
 mm/gup.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/gup.c +3 -3
index 8e7dc2c6ee73..236450feea9a 100644
@@ -2865,8 +2865,8 @@ static int gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr,
 		if (!folio)
 			goto pte_unmap;
 
-		if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) ||
-		    unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) {
+		if (unlikely(pmd_val(pmd) != pmd_val(pmdp_get_lockless(pmdp))) ||
+		    unlikely(pte_val(pte) != pte_val(ptep_get_lockless(ptep)))) {
 			gup_put_folio(folio, 1, flags);
 			goto pte_unmap;
 		}
@@ -2942,7 +2942,7 @@ static int gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp, unsigned long addr,
 	if (!folio)
 		return 0;
 
-	if (unlikely(pmd_val(orig) != pmd_val(*pmdp))) {
+	if (unlikely(pmd_val(orig) != pmd_val(pmdp_get_lockless(pmdp)))) {
 		gup_put_folio(folio, refs, flags);
 		return 0;
 	}

mm/gup.c static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr, · line 2991

Quoted 8 lines
mm/gup.c +1 -1
@@ -2985,7 +2985,7 @@ static int gup_fast_pud_leaf(pud_t orig, pud_t *pudp, unsigned long addr,
 	if (!folio)
 		return 0;
 
-	if (unlikely(pud_val(orig) != pud_val(*pudp))) {
+	if (unlikely(pud_val(orig) != pud_val(pudp_get(pudp)))) {
 		gup_put_folio(folio, refs, flags);
 		return 0;
 	}

In general, LGTM.

I do wonder, though, if we want to switch to pte_same/pmd_same/pud_same while at it?

Signature · 3 lines
Cheers,

David
author maintainer
#2 lore raw
Quoted 3 lines · David Hildenbrand (Arm)

In general, LGTM.

I do wonder, though, if we want to switch to pte_same/pmd_same/pud_same while at it?

I thought about that too, but decided to keep it focused on direct dereferences (as it fixes a real issue with the pte kasan sanitizer).

I can send updated patch or a follow-up patch - as you prefer. But if you like the follow-up I would suggest to un-squash the original series to get it in three logical steps: 1 - pXdp_get() fix, 2 - lockless access fix, 3 - pXd_same() cleanup.

Quoted 4 lines
Signature · 3 lines
Cheers,

David

Thanks!

maintainer Acked-by
#3 lore raw
Quoted 5 lines · Alexander Gordeev
Quoted 3 lines · David Hildenbrand (Arm)

In general, LGTM.

I do wonder, though, if we want to switch to pte_same/pmd_same/pud_same while at it?

Quoted 7 lines

I thought about that too, but decided to keep it focused on direct dereferences (as it fixes a real issue with the pte kasan sanitizer).

I can send updated patch or a follow-up patch - as you prefer. But if you like the follow-up I would suggest to un-squash the original series to get it in three logical steps: 1 - pXdp_get() fix, 2 - lockless access fix, 3 - pXd_same() cleanup.

I'd prefer to keep 1 and 2 squashed. I don't see a good reason to temporarily do the wrong thing :)

A follow-up patch works for me, thanks!

Signature · 3 lines
Cheers,

David