x86/mm/tlb: ignore f->new_tlb_gen when zero

PATCH Applied Nadav Amit 5 participants 7 messages Jul 11, 2022 – Jul 19, 2022

Subsystems (3)

Memory management x86 x86 MM

lore

Share & tools

lore

Nadav Amit

author

2022-07-11 07:03 UTC #0 lore raw

PATCH v2 1/1 x86/mm/tlb: ignore f->new_tlb_gen when zero

From: Nadav Amit <[email protected]>

Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization of skipping the flush if the TLB generation that is flushed (as provided in flush_tlb_info) was already flushed.

However, arch_tlbbatch_flush() does not provide any generation in flush_tlb_info. As a result, try_to_unmap_one() would not perform any TLB flushes.

Fix it by checking whether f->new_tlb_gen is nonzero. Zero value is anyhow is an invalid generation value. To avoid future confusions, introduce TLB_GENERATION_INVALID constant and use it properly. Add some assertions to check no partial flushes are done with TLB_GENERATION_INVALID or when f->mm is NULL, since this does not make any sense.

In addition, add the missing unlikely().

Fixes aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible")

Reported-by Hugh Dickins <[email protected]>

Tested-by Hugh Dickins <[email protected]>

Cc Dave Hansen <[email protected]>

Cc Peter Zijlstra (Intel) <[email protected]>

Cc Andy Lutomirski <[email protected]>

Signed-off-by Nadav Amit <[email protected]>

---

v1 -> v2:
* Introduce TLB_GENERATION_INVALID to clarify intent.
* Leave the early return and do not "goto out".
* Add some assertions to check and document in code the relationship between TLB_GENERATION_INVALID and TLB_FLUSH_ALL.

---
 arch/x86/include/asm/tlbflush.h |  1 +
 arch/x86/mm/tlb.c               | 15 ++++++++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)

arch/x86/include/asm/tlbflush.h +1 -0

index 4af5579c7ef7..cda3118f3b27 100644
@@ -16,6 +16,7 @@
 void __flush_tlb_all(void);
 
 #define TLB_FLUSH_ALL	-1UL
+#define TLB_GENERATION_INVALID	0
 
 void cr4_update_irqsoff(unsigned long set, unsigned long clear);
 unsigned long cr4_read_shadow(void);

arch/x86/mm/tlb.c +12 -3

index d9314cc8b81f..0f346c51dd99 100644
@@ -771,7 +771,8 @@ static void flush_tlb_func(void *info)
 		return;
 	}
 
-	if (f->new_tlb_gen <= local_tlb_gen) {
+	if (unlikely(f->new_tlb_gen != TLB_GENERATION_INVALID &&
+		     f->new_tlb_gen <= local_tlb_gen)) {
 		/*
 		 * The TLB is already up to date in respect to f->new_tlb_gen.
 		 * While the core might be still behind mm_tlb_gen, checking

@@ -843,6 +844,12 @@ static void flush_tlb_func(void *info)
 		/* Partial flush */
 		unsigned long addr = f->start;
 
+		/* Partial flush cannot have invalid generations */
+		VM_BUG_ON(f->new_tlb_gen == TLB_GENERATION_INVALID);
+
+		/* Partial flush must have valid mm */
+		VM_BUG_ON(f->mm == NULL);
+
 		nr_invalidate = (f->end - f->start) >> f->stride_shift;
 
 		while (addr < f->end) {

@@ -1045,7 +1052,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 		struct flush_tlb_info *info;
 
 		preempt_disable();
-		info = get_flush_tlb_info(NULL, start, end, 0, false, 0);
+		info = get_flush_tlb_info(NULL, start, end, 0, false,
+					  TLB_GENERATION_INVALID);
 
 		on_each_cpu(do_kernel_range_flush, info, 1);

@@ -1214,7 +1222,8 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 
 	int cpu = get_cpu();
 
-	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, 0);
+	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false,
+				  TLB_GENERATION_INVALID);
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
 	 * a local TLB flush is needed. Optimize this use-case by calling

Signature · 1 line

2.25.1

Nadav Amit

2022-07-11 17:39 UTC #1 lore raw

Quoted 6 lines · Nadav Amit

From: Nadav Amit <[email protected]>
Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization of skipping the flush if the TLB generation that is flushed (as provided in flush_tlb_info) was already flushed.

Dave,

Can you please review this patch today?

I feel bad (for a good reason) for breaking swap/migration.

Thanks,
Nadav

Nadav Amit

2022-07-13 00:33 UTC #2 lore raw

Quoted 9 lines · Nadav Amit

Quoted 6 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization of skipping the flush if the TLB generation that is flushed (as provided in flush_tlb_info) was already flushed.

Quoted 8 lines

Dave,
Can you please review this patch today?
I feel bad (for a good reason) for breaking swap/migration.
Thanks,
Nadav

Ping?

As you know, this really must go into 5.19 or otherwise aa44284960d5 reverted.

Hugh Dickins

2022-07-13 00:49 UTC #3 lore raw

Quoted 15 lines · Nadav Amit

Quoted 14 lines · Nadav Amit
Quoted 6 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization of skipping the flush if the TLB generation that is flushed (as provided in flush_tlb_info) was already flushed.
Dave,
Can you please review this patch today?
I feel bad (for a good reason) for breaking swap/migration.

Quoted 7 lines

Quoted 2 lines
Thanks,
Nadav
Ping?
As you know, this really must go into 5.19 or otherwise aa44284960d5 reverted.

No, aa44284960d5 is not in 5.19-rc: it's in linux-next heading for 5.20.

Hugh

Nadav Amit

2022-07-13 00:51 UTC #4 lore raw

Quoted 21 lines · Hugh Dickins

⚠ External Email
Quoted 18 lines · Nadav Amit
Quoted 16 lines · Nadav Amit
Quoted 6 lines · Nadav Amit
From: Nadav Amit <[email protected]>
Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization of skipping the flush if the TLB generation that is flushed (as provided in flush_tlb_info) was already flushed.
Dave,
Can you please review this patch today?
I feel bad (for a good reason) for breaking swap/migration.
Thanks,
Nadav

Quoted 6 lines

Quoted 4 lines
Ping?
As you know, this really must go into 5.19 or otherwise aa44284960d5 reverted.
No, aa44284960d5 is not in 5.19-rc: it's in linux-next heading for 5.20.

Oh.. My bad. Thanks for clarifying Hugh.

Dave Hansen

2022-07-19 16:14 UTC #5 lore raw

In reply to Nadav Amit

Quoted 11 lines · Nadav Amit

From: Nadav Amit <[email protected]>
Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization of skipping the flush if the TLB generation that is flushed (as provided in flush_tlb_info) was already flushed.
However, arch_tlbbatch_flush() does not provide any generation in flush_tlb_info. As a result, try_to_unmap_one() would not perform any TLB flushes.

Quoted 8 lines

Fix it by checking whether f->new_tlb_gen is nonzero. Zero value is anyhow is an invalid generation value. To avoid future confusions, introduce TLB_GENERATION_INVALID constant and use it properly. Add some assertions to check no partial flushes are done with TLB_GENERATION_INVALID or when f->mm is NULL, since this does not make any sense.
In addition, add the missing unlikely().

I've applied this:

Quoted 1 line

https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?h=x86/mm&id=8f1d56f64f8d6b80dea2d1978d10071132a695c5

Please double-check that my rewording of the commit message looks good to you. I also replaced the VM_BUG_ON()'s with warnings. Screwing up TLB flushing isn't great, but it's also not worth killing the system.

tip-bot2 for Nadav Amit

merge bot Applied

2022-07-19 16:17 UTC #6 lore raw

[tip: x86/mm] x86/mm/tlb: Ignore f->new_tlb_gen when zero

In reply to Nadav Amit

The following commit has been merged into the x86/mm branch of tip:

Commit-ID:     8f1d56f64f8d6b80dea2d1978d10071132a695c5
Gitweb:        https://git.kernel.org/tip/8f1d56f64f8d6b80dea2d1978d10071132a695c5
Author:        Nadav Amit <[email protected]>
AuthorDate:    Sun, 10 Jul 2022 16:28:37 -07:00
Committer:     Dave Hansen <[email protected]>
CommitterDate: Tue, 19 Jul 2022 09:04:52 -07:00

x86/mm/tlb: Ignore f->new_tlb_gen when zero

Commit aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible") introduced an optimization to skip superfluous TLB flushes based on the generation provided in flush_tlb_info.

However, arch_tlbbatch_flush() does not provide any generation in flush_tlb_info and populates the flush_tlb_info generation with
0. This 0 is causes the flush_tlb_info to be interpreted as a superfluous, old flush. As a result, try_to_unmap_one() would not perform any TLB flushes.

Fix it by checking whether f->new_tlb_gen is nonzero. Zero value is anyhow is an invalid generation value. To avoid future confusion, introduce TLB_GENERATION_INVALID constant and use it properly. Add warnings to ensure no partial flushes are done with TLB_GENERATION_INVALID or when f->mm is NULL, since this does not make any sense.

In addition, add the missing unlikely().

[ dhansen: change VM_BUG_ON() -> VM_WARN_ON(), clarify changelog ]

Fixes aa44284960d5 ("x86/mm/tlb: Avoid reading mm_tlb_gen when possible")

Reported-by Hugh Dickins <[email protected]>

Signed-off-by Nadav Amit <[email protected]>

Signed-off-by Dave Hansen <[email protected]>

Tested-by Hugh Dickins <[email protected]>

Link https://lkml.kernel.org/r/[email protected] open here

---
 arch/x86/include/asm/tlbflush.h |  1 +
 arch/x86/mm/tlb.c               | 15 ++++++++++++---
 2 files changed, 13 insertions(+), 3 deletions(-)

arch/x86/include/asm/tlbflush.h +1 -0

index 98fa0a1..21bf105 100644
@@ -16,6 +16,7 @@
 void __flush_tlb_all(void);
 
 #define TLB_FLUSH_ALL	-1UL
+#define TLB_GENERATION_INVALID	0
 
 void cr4_update_irqsoff(unsigned long set, unsigned long clear);
 unsigned long cr4_read_shadow(void);

arch/x86/mm/tlb.c +12 -3

index d9314cc..c1e31e9 100644
@@ -771,7 +771,8 @@ static void flush_tlb_func(void *info)
 		return;
 	}
 
-	if (f->new_tlb_gen <= local_tlb_gen) {
+	if (unlikely(f->new_tlb_gen != TLB_GENERATION_INVALID &&
+		     f->new_tlb_gen <= local_tlb_gen)) {
 		/*
 		 * The TLB is already up to date in respect to f->new_tlb_gen.
 		 * While the core might be still behind mm_tlb_gen, checking

@@ -843,6 +844,12 @@ static void flush_tlb_func(void *info)
 		/* Partial flush */
 		unsigned long addr = f->start;
 
+		/* Partial flush cannot have invalid generations */
+		VM_WARN_ON(f->new_tlb_gen == TLB_GENERATION_INVALID);
+
+		/* Partial flush must have valid mm */
+		VM_WARN_ON(f->mm == NULL);
+
 		nr_invalidate = (f->end - f->start) >> f->stride_shift;
 
 		while (addr < f->end) {

@@ -1045,7 +1052,8 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end)
 		struct flush_tlb_info *info;
 
 		preempt_disable();
-		info = get_flush_tlb_info(NULL, start, end, 0, false, 0);
+		info = get_flush_tlb_info(NULL, start, end, 0, false,
+					  TLB_GENERATION_INVALID);
 
 		on_each_cpu(do_kernel_range_flush, info, 1);

@@ -1214,7 +1222,8 @@ void arch_tlbbatch_flush(struct arch_tlbflush_unmap_batch *batch)
 
 	int cpu = get_cpu();
 
-	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false, 0);
+	info = get_flush_tlb_info(NULL, 0, TLB_FLUSH_ALL, 0, false,
+				  TLB_GENERATION_INVALID);
 	/*
 	 * flush_tlb_multi() is not optimized for the common case in which only
 	 * a local TLB flush is needed. Optimize this use-case by calling