mm: Fix the TLB range flushed when __tlb_remove_page() runs out of slots
PATCH 1/1 mm: Fix the TLB range flushed when __tlb_remove_page() runs out of slots
zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.
Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:
http://www.spinics.net/lists/linux-arch/msg21736.html
--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +6 -3
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);
@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end; + tlb->start = range_start; + tlb->end = addr;#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) { + range_start = addr; goto again; + } }return addr;
Signature · 1 line
1.7.10.4
Quoted 42 lines · Vineet Gupta
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>Cc Andrew Morton <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Rik van Riel <[email protected]>Cc David Rientjes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Max Filippov <[email protected]>--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/memory.c +1
-2@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end;
mm/memory.c again: · line 1227
Quoted 10 lines
mm/memory.c +5
-1+ tlb->start = range_start; + tlb->end = addr;#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) { + range_start = addr; goto again; + } }
Isn't this code only run if force_flush != 0? force_flush is set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP since tlb_fast_mode() is 1. There is no batching on UP with the generic TLB code.
Signature · 7 lines
Catalin -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>Cc Andrew Morton <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Rik van Riel <[email protected]>Cc David Rientjes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Max Filippov <[email protected]>--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/memory.c +4
-3@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end; + tlb->start = range_start; + tlb->end = addr;#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) {
Quoted 8 lines
mm/memory.c again: · line 1227
Quoted 4 lines
mm/memory.c +2
-0+ range_start = addr; goto again; + } }Isn't this code only run if
force_flush!= 0?force_flushis set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP sincetlb_fast_mode()is 1. There is no batching on UP with the generic TLB code.
Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)
However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.
Ignoring all other threads, do we agree that the exiting code - if used in any situations is incorrect semantically ?
-Vineet
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 59 lines · Vineet Gupta
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>Cc Andrew Morton <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Rik van Riel <[email protected]>Cc David Rientjes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Max Filippov <[email protected]>--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/memory.c +4
-3@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end; + tlb->start = range_start; + tlb->end = addr;#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) {Quoted 8 lines
mm/memory.c again: · line 1227
Quoted 4 lines
mm/memory.c +2
-0+ range_start = addr; goto again; + } }Isn't this code only run if
force_flush!= 0?force_flushis set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP sincetlb_fast_mode()is 1. There is no batching on UP with the generic TLB code.
Quoted 5 lines
Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)
However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.
An alternative would be to make sure the above block is always called when tlb_fast_mode():
mm/memory.c +1 -1
@@ -1211,7 +1211,7 @@ again:* the PTE lock to avoid doing the potential expensive TLB invalidate * and page-free while holding it. */ -if (force_flush) { +if (force_flush ||tlb_fast_mode (tlb)) { force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER
Quoted 2 lines
Ignoring all other threads, do we agree that the exiting code - if used in any situations is incorrect semantically ?
It is incorrect unless there are requirements for arch_leave_lazy_mmu_mode() to handle the TLB invalidation (it doesn't look like it's widely implemented though).
Signature · 7 lines
Catalin -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 72 lines · Catalin Marinas
Quoted 58 lines · Vineet Gupta
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>Cc Andrew Morton <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Rik van Riel <[email protected]>Cc David Rientjes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Max Filippov <[email protected]>--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/memory.c +4
-3@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end; + tlb->start = range_start; + tlb->end = addr;#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) {Quoted 8 lines
mm/memory.c again: · line 1227
Quoted 4 lines
mm/memory.c +2
-0+ range_start = addr; goto again; + } }Isn't this code only run if
force_flush!= 0?force_flushis set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP sincetlb_fast_mode()is 1. There is no batching on UP with the generic TLB code.Quoted 5 lines
Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)
However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.
An alternative would be to make sure the above block is always called when
tlb_fast_mode():mm/memory.c +0
-0
mm/memory.c again: · line 1215
Quoted 6 lines
mm/memory.c +1
-1@@ -1211,7 +1211,7 @@ again:* the PTE lock to avoid doing the potential expensive TLB invalidate * and page-free while holding it. */ -if (force_flush) { +if (force_flush ||tlb_fast_mode (tlb)) { force_flush =0 ;
I agree with tlb_fast_mode() addition (to solve Max's issue). The problem however is that when we hit this at the end of loop - @addr is already pointing to @end so range flush gets start = end - not what we really intended.
Quoted 5 lines
Quoted 2 lines
Ignoring all other threads, do we agree that the exiting code - if used in any situations is incorrect semantically ?
It is incorrect unless there are requirements for
arch_leave_lazy_mmu_mode()to handle the TLB invalidation (it doesn't look like it's widely implemented though).
This patch is preparatory - independent of Max's issue. It is fixing just the forced flush case - whoever uses it right now (ofcourse UP + generic TLB doesn't).
Thx,
-Vineet
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Catalin Marinas
Acked-byQuoted 68 lines · Vineet Gupta
Quoted 67 lines · Catalin Marinas
Quoted 58 lines · Vineet Gupta
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>Cc Andrew Morton <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Rik van Riel <[email protected]>Cc David Rientjes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Max Filippov <[email protected]>--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/memory.c +4
-3@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end; + tlb->start = range_start; + tlb->end = addr;#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) {Quoted 8 lines
mm/memory.c again: · line 1227
Quoted 4 lines
mm/memory.c +2
-0+ range_start = addr; goto again; + } }Isn't this code only run if
force_flush!= 0?force_flushis set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP sincetlb_fast_mode()is 1. There is no batching on UP with the generic TLB code.Quoted 5 lines
Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)
However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.
An alternative would be to make sure the above block is always called when
tlb_fast_mode():
Quoted 15 lines
mm/memory.c again: · line 1215
Quoted 11 lines
mm/memory.c +1
-1@@ -1211,7 +1211,7 @@ again:* the PTE lock to avoid doing the potential expensive TLB invalidate * and page-free while holding it. */ -if (force_flush) { +if (force_flush ||tlb_fast_mode (tlb)) { force_flush =0 ;I agree with
tlb_fast_mode()addition (to solve Max's issue). The problem however is that when we hit this at the end of loop - @addr is already pointing to @end so range flush gets start = end - not what we really intended.
OK. So for this part your patch looks fine.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 45 lines · Vineet Gupta
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>Cc Andrew Morton <[email protected]>Cc Mel Gorman <[email protected]>Cc Hugh Dickins <[email protected]>Cc Rik van Riel <[email protected]>Cc David Rientjes <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Catalin Marinas <[email protected]>Cc Max Filippov <[email protected]>--- mm/memory.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)mm/memory.c +3
-2@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,spinlock_t *ptl;pte_t *start_pte;pte_t *pte; +unsigned long range_start = addr; again:init_rss_vec (rss);@@ -1215,12 +1216,14 @@ again: force_flush =0 ;#ifdef HAVE_GENERIC_MMU_GATHER - tlb->start = addr; - tlb->end = end; + tlb->start = range_start; + tlb->end = addr;
mm/memory.c again: · line 1229
Quoted 10 lines
mm/memory.c +3
-1#endif tlb_flush_mmu (tlb); -if (addr != end) +if (addr != end) { + range_start = addr; goto again; + } }return addr;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
From: Vineet Gupta <[email protected]>
Date: Wed, 29 May 2013 18:26:13 +0530
Quoted 10 lines
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.htmlSigned-off-by Vineet Gupta <[email protected]>
As this bug can cause pretty serious memory corruption, I'd like to see this submitted to -stable.
Thanks!
Andrew Morton
maintainerQuoted 12 lines · On Mon, 29 Jul 2013 16:41:06 -0700 (PDT) David Miller <[email protected]>…
From: Vineet Gupta <[email protected]>
Date: Wed, 29 May 2013 18:26:13 +0530Quoted 9 lines
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.html
Quoted 4 lines
Quoted 1 line
Signed-off-by Vineet Gupta <[email protected]>As this bug can cause pretty serious memory corruption, I'd like to see this submitted to -stable.
Greg, e6c495a96ce02574e765d5140039a64c8d4e8c9e from mainline, please.
Quoted 14 lines · Andrew Morton
Quoted 12 lines · On Mon, 29 Jul 2013 16:41:06 -0700 (PDT) David Miller <[email protected]>…
From: Vineet Gupta <[email protected]>
Date: Wed, 29 May 2013 18:26:13 +0530Quoted 9 lines
zap_pte_rangeloops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:http://www.spinics.net/lists/linux-arch/msg21736.html
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Signed-off-by Vineet Gupta <[email protected]>As this bug can cause pretty serious memory corruption, I'd like to see this submitted to -stable.
Greg,
e6c495a96ce02574e765d5140039a64c8d4e8c9efrom mainline, please.
Now applied to 3.10-stable, thanks.
greg k-h
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>