Skip to contentSkip to conversation

mm: Fix the TLB range flushed when __tlb_remove_page() runs out of slots

PATCH In review Vineet Gupta 5 participants 10 messages May 29, 2013 – Jul 30, 2013

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm: Fix the TLB range flushed when __tlb_remove_page() runs out of slots

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +6 -3
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;
+		tlb->start = range_start;
+		tlb->end = addr;
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
+			range_start = addr;
 			goto again;
+		}
 	}
 
 	return addr;
Signature · 1 line
1.7.10.4
#1 lore raw
Quoted 42 lines · Vineet Gupta

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +1 -2
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;

mm/memory.c again: · line 1227

Quoted 10 lines
mm/memory.c +5 -1
+		tlb->start = range_start;
+		tlb->end = addr;
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
+			range_start = addr;
 			goto again;
+		}
 	}

Isn't this code only run if force_flush != 0? force_flush is set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP since tlb_fast_mode() is 1. There is no batching on UP with the generic TLB code.

Signature · 7 lines
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#2 lore raw
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +4 -3
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;
+		tlb->start = range_start;
+		tlb->end = addr;
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
Quoted 8 lines

mm/memory.c again: · line 1227

Quoted 4 lines
mm/memory.c +2 -0
+			range_start = addr;
 			goto again;
+		}
 	}

Isn't this code only run if force_flush != 0? force_flush is set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP since tlb_fast_mode() is 1. There is no batching on UP with the generic TLB code.

Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)

However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.

Ignoring all other threads, do we agree that the exiting code - if used in any situations is incorrect semantically ?

-Vineet

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw
Quoted 59 lines · Vineet Gupta
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +4 -3
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;
+		tlb->start = range_start;
+		tlb->end = addr;
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
Quoted 8 lines

mm/memory.c again: · line 1227

Quoted 4 lines
mm/memory.c +2 -0
+			range_start = addr;
 			goto again;
+		}
 	}

Isn't this code only run if force_flush != 0? force_flush is set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP since tlb_fast_mode() is 1. There is no batching on UP with the generic TLB code.

Quoted 5 lines

Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)

However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.

An alternative would be to make sure the above block is always called when tlb_fast_mode():

mm/memory.c +1 -1
index 6dc1882..f8b1f30 100644
@@ -1211,7 +1211,7 @@ again:
 	 * the PTE lock to avoid doing the potential expensive TLB invalidate
 	 * and page-free while holding it.
 	 */
-	if (force_flush) {
+	if (force_flush || tlb_fast_mode(tlb)) {
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
Quoted 2 lines

Ignoring all other threads, do we agree that the exiting code - if used in any situations is incorrect semantically ?

It is incorrect unless there are requirements for arch_leave_lazy_mmu_mode() to handle the TLB invalidation (it doesn't look like it's widely implemented though).

Signature · 7 lines
Catalin

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#4 lore raw
Quoted 72 lines · Catalin Marinas
Quoted 58 lines · Vineet Gupta
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +4 -3
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;
+		tlb->start = range_start;
+		tlb->end = addr;
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
Quoted 8 lines

mm/memory.c again: · line 1227

Quoted 4 lines
mm/memory.c +2 -0
+			range_start = addr;
 			goto again;
+		}
 	}

Isn't this code only run if force_flush != 0? force_flush is set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP since tlb_fast_mode() is 1. There is no batching on UP with the generic TLB code.

Quoted 5 lines

Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)

However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.

An alternative would be to make sure the above block is always called when tlb_fast_mode():

mm/memory.c +0 -0
index 6dc1882..f8b1f30 100644

mm/memory.c again: · line 1215

Quoted 6 lines
mm/memory.c +1 -1
@@ -1211,7 +1211,7 @@ again:
 	 * the PTE lock to avoid doing the potential expensive TLB invalidate
 	 * and page-free while holding it.
 	 */
-	if (force_flush) {
+	if (force_flush || tlb_fast_mode(tlb)) {
 		force_flush = 0;

I agree with tlb_fast_mode() addition (to solve Max's issue). The problem however is that when we hit this at the end of loop - @addr is already pointing to @end so range flush gets start = end - not what we really intended.

Quoted 5 lines
Quoted 2 lines

Ignoring all other threads, do we agree that the exiting code - if used in any situations is incorrect semantically ?

It is incorrect unless there are requirements for arch_leave_lazy_mmu_mode() to handle the TLB invalidation (it doesn't look like it's widely implemented though).

This patch is preparatory - independent of Max's issue. It is fixing just the forced flush case - whoever uses it right now (ofcourse UP + generic TLB doesn't).

Thx,
-Vineet

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#5 lore raw
Quoted 68 lines · Vineet Gupta
Quoted 67 lines · Catalin Marinas
Quoted 58 lines · Vineet Gupta
Quoted 49 lines · Catalin Marinas
Quoted 48 lines · Vineet Gupta

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +4 -3
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;
+		tlb->start = range_start;
+		tlb->end = addr;
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
Quoted 8 lines

mm/memory.c again: · line 1227

Quoted 4 lines
mm/memory.c +2 -0
+			range_start = addr;
 			goto again;
+		}
 	}

Isn't this code only run if force_flush != 0? force_flush is set to !__tlb_remove_page() and this function always returns 1 on (generic TLB) UP since tlb_fast_mode() is 1. There is no batching on UP with the generic TLB code.

Quoted 5 lines

Correct ! That's why the changelog says I couldn't test it on ARC port itself :-)

However based on the other discussion (Max's TLB/PTE inconsistency), as I started writing code to reuse this block to flush the TLB even for non forced case, I realized that what this is doing is incorrect and won't work for the general flushing.

An alternative would be to make sure the above block is always called when tlb_fast_mode():

Quoted 15 lines

mm/memory.c again: · line 1215

Quoted 11 lines
mm/memory.c +1 -1
index 6dc1882..f8b1f30 100644
@@ -1211,7 +1211,7 @@ again:
 	 * the PTE lock to avoid doing the potential expensive TLB invalidate
 	 * and page-free while holding it.
 	 */
-	if (force_flush) {
+	if (force_flush || tlb_fast_mode(tlb)) {
 		force_flush = 0;

I agree with tlb_fast_mode() addition (to solve Max's issue). The problem however is that when we hit this at the end of loop - @addr is already pointing to @end so range flush gets start = end - not what we really intended.

OK. So for this part your patch looks fine.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
In reply to Vineet Gupta

[[email protected]]

Quoted 45 lines · Vineet Gupta

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Cc Andrew Morton <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Hugh Dickins <[email protected]>
Cc Rik van Riel <[email protected]>
Cc David Rientjes <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Catalin Marinas <[email protected]>
Cc Max Filippov <[email protected]>
---
 mm/memory.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
mm/memory.c +3 -2
index 6dc1882..d9d5fd9 100644
@@ -1110,6 +1110,7 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb,
 	spinlock_t *ptl;
 	pte_t *start_pte;
 	pte_t *pte;
+	unsigned long range_start = addr;
 
 again:
 	init_rss_vec(rss);
@@ -1215,12 +1216,14 @@ again:
 		force_flush = 0;
 
 #ifdef HAVE_GENERIC_MMU_GATHER
-		tlb->start = addr;
-		tlb->end = end;
+		tlb->start = range_start;
+		tlb->end = addr;

mm/memory.c again: · line 1229

Quoted 10 lines
mm/memory.c +3 -1
 #endif
 		tlb_flush_mmu(tlb);
-		if (addr != end)
+		if (addr != end) {
+			range_start = addr;
 			goto again;
+		}
 	}
 
 	return addr;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
In reply to Vineet Gupta

From: Vineet Gupta <[email protected]>
Date: Wed, 29 May 2013 18:26:13 +0530

Quoted 10 lines

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html

As this bug can cause pretty serious memory corruption, I'd like to see this submitted to -stable.

Thanks!

maintainer
#8 lore raw
Quoted 12 lines · On Mon, 29 Jul 2013 16:41:06 -0700 (PDT) David Miller <[email protected]>…

From: Vineet Gupta <[email protected]>
Date: Wed, 29 May 2013 18:26:13 +0530

Quoted 9 lines

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Quoted 4 lines
Quoted 1 line

As this bug can cause pretty serious memory corruption, I'd like to see this submitted to -stable.

Greg, e6c495a96ce02574e765d5140039a64c8d4e8c9e from mainline, please.

#9 lore raw
Quoted 14 lines · Andrew Morton
Quoted 12 lines · On Mon, 29 Jul 2013 16:41:06 -0700 (PDT) David Miller <[email protected]>…

From: Vineet Gupta <[email protected]>
Date: Wed, 29 May 2013 18:26:13 +0530

Quoted 9 lines

zap_pte_range loops from @addr to @end. In the middle, if it runs out of batching slots, TLB entries needs to be flushed for @start to @interim, NOT @interim to @end.

Since ARC port doesn't use page free batching I can't test it myself but this seems like the right thing to do.
Observed this when working on a fix for the issue at thread:

	http://www.spinics.net/lists/linux-arch/msg21736.html
Quoted 6 lines
Quoted 4 lines
Quoted 1 line

As this bug can cause pretty serious memory corruption, I'd like to see this submitted to -stable.

Greg, e6c495a96ce02574e765d5140039a64c8d4e8c9e from mainline, please.

Now applied to 3.10-stable, thanks.

greg k-h

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>