mm/gup: dereference page table entry using helper
Subsystems (2)
PATCH 1/1 mm/gup: dereference page table entry using helper
Commit 0005d20 ("mm/gup: Move page table entry dereference into helper function") wrapped access to page table entries larger than sizeof(long) into a race-aware accessor. One of the two dereferences in gup_fast path was however overlooked.
--- mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
mm/gup.c +1 -1
@@ -2208,7 +2208,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (!head) goto pte_unmap; -if (unlikely (pte_val (pte) !=pte_val (*ptep))) { +if (unlikely (pte_val (pte) !=pte_val (gup_get_pte (ptep)))) {put_compound_head (head,1 , flags); goto pte_unmap; }
Signature · 1 line
1.8.3.1
Quoted 15 lines · Alexander Gordeev
Commit 0005d20 ("mm/gup: Move page table entry dereference into helper function") wrapped access to page table entries larger than sizeof(long) into a race-aware accessor. One of the two dereferences in
gup_fastpath was however overlooked.CC Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/gup.c +1
-0+++ b/mm/gup.c
mm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2211
Quoted 5 lines
mm/gup.c +1
-1@@ -2208,7 +2208,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (!head) goto pte_unmap; -if (unlikely (pte_val (pte) !=pte_val (*ptep))) { +if (unlikely (pte_val (pte) !=pte_val (gup_get_pte (ptep)))) {
It doesn't seem like this needs the special helper as it is just checking that the pte hasn't changed, it doesn't need to be read exactly.
But it probably should technically still be a READ_ONCE. Although I think the atomic inside try_grab_compound_head prevents any real problems.
Jason
Quoted 26 lines · Jason Gunthorpe
Quoted 15 lines · Alexander Gordeev
Commit 0005d20 ("mm/gup: Move page table entry dereference into helper function") wrapped access to page table entries larger than sizeof(long) into a race-aware accessor. One of the two dereferences in
gup_fastpath was however overlooked.CC Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/gup.c +1
-0+++ b/mm/gup.cmm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2211
Quoted 5 lines
mm/gup.c +1
-1@@ -2208,7 +2208,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (!head) goto pte_unmap; -if (unlikely (pte_val (pte) !=pte_val (*ptep))) { +if (unlikely (pte_val (pte) !=pte_val (gup_get_pte (ptep)))) {It doesn't seem like this needs the special helper as it is just checking that the pte hasn't changed, it doesn't need to be read exactly.
Quoted 3 lines
But it probably should technically still be a
READ_ONCE. Although I think the atomic insidetry_grab_compound_headprevents any real problems.
I think we should go for consistency here and use the helper function.
Ira
Quoted 3 lines
Jason
Quoted 27 lines · Ira Weiny
Quoted 26 lines · Jason Gunthorpe
Quoted 15 lines · Alexander Gordeev
Commit 0005d20 ("mm/gup: Move page table entry dereference into helper function") wrapped access to page table entries larger than sizeof(long) into a race-aware accessor. One of the two dereferences in
gup_fastpath was however overlooked.CC Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/gup.c +1
-0+++ b/mm/gup.cmm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2211
Quoted 5 lines
mm/gup.c +1
-1@@ -2208,7 +2208,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (!head) goto pte_unmap; -if (unlikely (pte_val (pte) !=pte_val (*ptep))) { +if (unlikely (pte_val (pte) !=pte_val (gup_get_pte (ptep)))) {It doesn't seem like this needs the special helper as it is just checking that the pte hasn't changed, it doesn't need to be read exactly.
Quoted 5 lines
Quoted 3 lines
But it probably should technically still be a
READ_ONCE. Although I think the atomic insidetry_grab_compound_headprevents any real problems.I think we should go for consistency here and use the helper function.
It seems quite expensive to do two more unncessary barriers..
Jason
Quoted 28 lines · Jason Gunthorpe
Quoted 27 lines · Ira Weiny
Quoted 26 lines · Jason Gunthorpe
Quoted 15 lines · Alexander Gordeev
Commit 0005d20 ("mm/gup: Move page table entry dereference into helper function") wrapped access to page table entries larger than sizeof(long) into a race-aware accessor. One of the two dereferences in
gup_fastpath was however overlooked.CC Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/gup.c +1
-0+++ b/mm/gup.cmm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2211
Quoted 5 lines
mm/gup.c +1
-1@@ -2208,7 +2208,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (!head) goto pte_unmap; -if (unlikely (pte_val (pte) !=pte_val (*ptep))) { +if (unlikely (pte_val (pte) !=pte_val (gup_get_pte (ptep)))) {It doesn't seem like this needs the special helper as it is just checking that the pte hasn't changed, it doesn't need to be read exactly.
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
But it probably should technically still be a
READ_ONCE. Although I think the atomic insidetry_grab_compound_headprevents any real problems.I think we should go for consistency here and use the helper function.
It seems quite expensive to do two more unncessary barriers..
But won't a failure to read the 'real' pte result in falling back to GUP slow?
Not sure which is worse?
And most arch's don't suffer from this...
Ira
Quoted 33 lines · Ira Weiny
Quoted 32 lines · Jason Gunthorpe
Quoted 31 lines · Ira Weiny
Quoted 29 lines · Jason Gunthorpe
Quoted 15 lines · Alexander Gordeev
Commit 0005d20 ("mm/gup: Move page table entry dereference into helper function") wrapped access to page table entries larger than sizeof(long) into a race-aware accessor. One of the two dereferences in
gup_fastpath was however overlooked.CC Kirill A. Shutemov <[email protected]>Signed-off-by Alexander Gordeev <[email protected]>mm/gup.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)mm/gup.c +1
-0+++ b/mm/gup.cmm/gup.c static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end, · line 2211
Quoted 5 lines
mm/gup.c +1
-1@@ -2208,7 +2208,7 @@ static int gup_pte_range(pmd_t pmd, unsigned long addr, unsigned long end,if (!head) goto pte_unmap; -if (unlikely (pte_val (pte) !=pte_val (*ptep))) { +if (unlikely (pte_val (pte) !=pte_val (gup_get_pte (ptep)))) {It doesn't seem like this needs the special helper as it is just checking that the pte hasn't changed, it doesn't need to be read exactly.
But it probably should technically still be a
READ_ONCE. Although I think the atomic insidetry_grab_compound_headprevents any real problems.
Quoted 5 lines
Quoted 3 lines
Quoted 1 line
I think we should go for consistency here and use the helper function.
It seems quite expensive to do two more unncessary barriers..
But won't a failure to read the 'real' pte result in falling back to GUP slow?
If there is no concurrent writer then the direct read will give the same result.
If there is a concurrent writer then it is a random race if fallback to gup slow is required.
Jason