Skip to contentSkip to conversation

x86/alternative: don't call text_poke() in lazy TLB mode

PATCH Applied Juergen Gross 4 participants 9 messages Oct 9, 2020 – Oct 22, 2020 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/alternative: don't call text_poke() in lazy TLB mode

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")
---
 arch/x86/kernel/alternative.c | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +9 -0
index cdaab30880b9..cd6be6f143e8 100644
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
 	temp_mm_state_t temp_state;
 
 	lockdep_assert_irqs_disabled();
+
+	/*
+	 * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+	 * with a stale address space WITHOUT being in lazy mode after
+	 * restoring the previous mm.
+	 */
+	if (this_cpu_read(cpu_tlbstate.is_lazy))
+		leave_mm(smp_processor_id());
+
 	temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
 	switch_mm_irqs_off(NULL, mm, current);
 
Signature · 1 line
2.26.2
#1 lore raw
Quoted 16 lines · Juergen Gross

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

Quoted 2 lines

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Oh man, that must've been 'fun' :/

Quoted 15 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")
---
 arch/x86/kernel/alternative.c | 9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +1 -0
index cdaab30880b9..cd6be6f143e8 100644
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
 	temp_mm_state_t temp_state;
 
 	lockdep_assert_irqs_disabled();
+

arch/x86/kernel/alternative.c static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 820

Quoted 10 lines
arch/x86/kernel/alternative.c +8 -0
+	/*
+	 * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+	 * with a stale address space WITHOUT being in lazy mode after
+	 * restoring the previous mm.
+	 */
+	if (this_cpu_read(cpu_tlbstate.is_lazy))
+		leave_mm(smp_processor_id());
+
 	temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
 	switch_mm_irqs_off(NULL, mm, current);

Would it make sense to write it like:

	this_state.mm = this_cpu_read(cpu_tlbstate.is_lazy) ?
			&init_mm : this_cpu_read(cpu_tlbstate.loaded_mm);

Possibly with that wrapped in a conveniently named helper function.

#2 lore raw
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Juergen Gross

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

Quoted 4 lines
Quoted 2 lines

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Oh man, that must've been 'fun' :/

Yeah.

Quoted 29 lines
Quoted 16 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")
---
  arch/x86/kernel/alternative.c | 9 +++++++++
  1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +1 -0
index cdaab30880b9..cd6be6f143e8 100644
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
  	temp_mm_state_t temp_state;
 
  	lockdep_assert_irqs_disabled();
+

arch/x86/kernel/alternative.c static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 820

Quoted 10 lines
arch/x86/kernel/alternative.c +8 -0
+	/*
+	 * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+	 * with a stale address space WITHOUT being in lazy mode after
+	 * restoring the previous mm.
+	 */
+	if (this_cpu_read(cpu_tlbstate.is_lazy))
+		leave_mm(smp_processor_id());
+
  	temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
  	switch_mm_irqs_off(NULL, mm, current);

Would it make sense to write it like:

Quoted 4 lines
	this_state.mm = this_cpu_read(cpu_tlbstate.is_lazy) ?
			&init_mm : this_cpu_read(cpu_tlbstate.loaded_mm);

Possibly with that wrapped in a conveniently named helper function.

Fine with me, but I don't think it matters that much.

For each batch of text_poke() it will be hit only once, and I'm not sure it is really a good idea to use the knowledge that leave_mm() is just a switch to init_mm here.

In case it is still the preferred way to do it I can send an update of the patch.

Juergen

#3 lore raw
Quoted 20 lines · Jürgen Groß
Quoted 20 lines
Quoted 4 lines
arch/x86/kernel/alternative.c +1 -0
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
  	temp_mm_state_t temp_state;
  	lockdep_assert_irqs_disabled();
+

arch/x86/kernel/alternative.c static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 819

Quoted 10 lines
arch/x86/kernel/alternative.c +8 -0
+	/*
+	 * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+	 * with a stale address space WITHOUT being in lazy mode after
+	 * restoring the previous mm.
+	 */
+	if (this_cpu_read(cpu_tlbstate.is_lazy))
+		leave_mm(smp_processor_id());
+
  	temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
  	switch_mm_irqs_off(NULL, mm, current);

Would it make sense to write it like:

	this_state.mm = this_cpu_read(cpu_tlbstate.is_lazy) ?
			&init_mm : this_cpu_read(cpu_tlbstate.loaded_mm);
Quoted 7 lines
Quoted 1 line

Possibly with that wrapped in a conveniently named helper function.

Fine with me, but I don't think it matters that much.

For each batch of text_poke() it will be hit only once, and I'm not sure it is really a good idea to use the knowledge that leave_mm() is just a switch to init_mm here.

Yeah, I'm not sure either. But it's something I came up with when looking at all this.

Andy, what's your preference?

#4 lore raw
In reply to Juergen Gross
Quoted 16 lines · Juergen Gross

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

Quoted 5 lines

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")

Can anyone look at this, please? It is fixing a real problem which has been seen several times.

Juergen

Quoted 25 lines
---
  arch/x86/kernel/alternative.c | 9 +++++++++
  1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +9 -0
index cdaab30880b9..cd6be6f143e8 100644
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
  	temp_mm_state_t temp_state;
 
  	lockdep_assert_irqs_disabled();
+
+	/*
+	 * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+	 * with a stale address space WITHOUT being in lazy mode after
+	 * restoring the previous mm.
+	 */
+	if (this_cpu_read(cpu_tlbstate.is_lazy))
+		leave_mm(smp_processor_id());
+
  	temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
  	switch_mm_irqs_off(NULL, mm, current);
 
#5 lore raw
Quoted 17 lines · Jürgen Groß
Quoted 16 lines · Juergen Gross

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

Quoted 8 lines
Quoted 5 lines

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")

Can anyone look at this, please? It is fixing a real problem which has been seen several times.

As it happens I picked it up yesterday, just pushed it out for you.

Thanks!

#6 lore raw
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Jürgen Groß
Quoted 19 lines · Juergen Gross

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")

Can anyone look at this, please? It is fixing a real problem which has been seen several times.

As it happens I picked it up yesterday, just pushed it out for you.

Thank you very much!

Juergen

merge bot Applied
#7 lore raw

[tip: x86/urgent] x86/alternative: Don't call text_poke() in lazy TLB mode

In reply to Juergen Gross
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     abee7c494d8c41bb388839bccc47e06247f0d7de
Gitweb:        https://git.kernel.org/tip/abee7c494d8c41bb388839bccc47e06247f0d7de
Author:        Juergen Gross <[email protected]>
AuthorDate:    Fri, 09 Oct 2020 16:42:25 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 22 Oct 2020 12:37:23 +02:00

x86/alternative: Don't call text_poke() in lazy TLB mode

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")
---
 arch/x86/kernel/alternative.c |  9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +9 -0
index cdaab30..cd6be6f 100644
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
 	temp_mm_state_t temp_state;
 
 	lockdep_assert_irqs_disabled();
+
+	/*
+	 * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+	 * with a stale address space WITHOUT being in lazy mode after
+	 * restoring the previous mm.
+	 */
+	if (this_cpu_read(cpu_tlbstate.is_lazy))
+		leave_mm(smp_processor_id());
+
 	temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
 	switch_mm_irqs_off(NULL, mm, current);
 
#8 lore raw

Re: [tip: x86/urgent] x86/alternative: Don't call text_poke() in lazy TLB mode

Quoted 28 lines · On Thu, Oct 22, 2020 at 3:50 AM tip-bot2 for Juergen Gross…

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     abee7c494d8c41bb388839bccc47e06247f0d7de
Gitweb:        https://git.kernel.org/tip/abee7c494d8c41bb388839bccc47e06247f0d7de
Author:        Juergen Gross <[email protected]>
AuthorDate:    Fri, 09 Oct 2020 16:42:25 +02:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Thu, 22 Oct 2020 12:37:23 +02:00

x86/alternative: Don't call text_poke() in lazy TLB mode

When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.

This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.

As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).

Quoted 2 lines

In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.

Now that I'm actually awake:

although it might be nice to at least have a comment that there's some performance being left on the table.

PeterZ, I like your version except that, if we do that, I also think we should move this whole mess into tlb.c instead of alternative.c.

--Andy

Quoted 29 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")
---
 arch/x86/kernel/alternative.c |  9 +++++++++
 1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +9 -0
index cdaab30..cd6be6f 100644
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)
        temp_mm_state_t temp_state;
 
        lockdep_assert_irqs_disabled();
+
+       /*
+        * Make sure not to be in TLB lazy mode, as otherwise we'll end up
+        * with a stale address space WITHOUT being in lazy mode after
+        * restoring the previous mm.
+        */
+       if (this_cpu_read(cpu_tlbstate.is_lazy))
+               leave_mm(smp_processor_id());
+
        temp_state.mm = this_cpu_read(cpu_tlbstate.loaded_mm);
        switch_mm_irqs_off(NULL, mm, current);
 
Signature · 2 lines
Andy Lutomirski
AMA Capital Management, LLC