x86/alternative: don't call text_poke() in lazy TLB mode
PATCH 1/1 x86/alternative: don't call text_poke() in lazy TLB mode
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.
As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
--- arch/x86/kernel/alternative.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +9 -0
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); + +/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);
Signature · 1 line
2.26.2
Quoted 16 lines · Juergen Gross
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via
text_poke()in a kernel thread, and on another processorexit_mmap()is active for the process which was running on the first cpu before the kernel thread.As
text_poke()is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on whichexit_mmap()is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
Quoted 2 lines
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
Oh man, that must've been 'fun' :/
Quoted 15 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")Signed-off-by Juergen Gross <[email protected]>--- arch/x86/kernel/alternative.c | 9 +++++++++ 1 file changed, 9 insertions(+)arch/x86/kernel/alternative.c +1
-0@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); +
arch/x86/kernel/alternative.c static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 820
Quoted 10 lines
arch/x86/kernel/alternative.c +8
-0+/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);
Would it make sense to write it like:
this_state.mm = this_cpu_read(cpu_tlbstate.is_lazy) ? &init_mm : this_cpu_read(cpu_tlbstate.loaded_mm);
Possibly with that wrapped in a conveniently named helper function.
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Juergen Gross
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via
text_poke()in a kernel thread, and on another processorexit_mmap()is active for the process which was running on the first cpu before the kernel thread.As
text_poke()is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on whichexit_mmap()is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
Quoted 4 lines
Quoted 2 lines
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
Oh man, that must've been 'fun' :/
Yeah.
Quoted 29 lines
Quoted 16 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")Signed-off-by Juergen Gross <[email protected]>--- arch/x86/kernel/alternative.c | 9 +++++++++ 1 file changed, 9 insertions(+)arch/x86/kernel/alternative.c +1
-0@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); +arch/x86/kernel/alternative.c static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 820
Quoted 10 lines
arch/x86/kernel/alternative.c +8
-0+/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);Would it make sense to write it like:
Quoted 4 lines
this_state.mm = this_cpu_read(cpu_tlbstate.is_lazy) ? &init_mm : this_cpu_read(cpu_tlbstate.loaded_mm);Possibly with that wrapped in a conveniently named helper function.
Fine with me, but I don't think it matters that much.
For each batch of text_poke() it will be hit only once, and I'm not sure it is really a good idea to use the knowledge that leave_mm() is just a switch to init_mm here.
In case it is still the preferred way to do it I can send an update of the patch.
Juergen
Quoted 20 lines · Jürgen Groß
Quoted 20 lines
Quoted 4 lines
arch/x86/kernel/alternative.c +1
-0@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); +arch/x86/kernel/alternative.c static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 819
Quoted 10 lines
arch/x86/kernel/alternative.c +8
-0+/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);Would it make sense to write it like:
this_state.mm = this_cpu_read(cpu_tlbstate.is_lazy) ? &init_mm : this_cpu_read(cpu_tlbstate.loaded_mm);
Quoted 7 lines
Quoted 1 line
Possibly with that wrapped in a conveniently named helper function.
Fine with me, but I don't think it matters that much.
For each batch of
text_poke()it will be hit only once, and I'm not sure it is really a good idea to use the knowledge thatleave_mm()is just a switch toinit_mmhere.
Yeah, I'm not sure either. But it's something I came up with when looking at all this.
Andy, what's your preference?
Quoted 16 lines · Juergen Gross
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via
text_poke()in a kernel thread, and on another processorexit_mmap()is active for the process which was running on the first cpu before the kernel thread.As
text_poke()is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on whichexit_mmap()is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
Quoted 5 lines
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")Signed-off-by Juergen Gross <[email protected]>
Can anyone look at this, please? It is fixing a real problem which has been seen several times.
Juergen
Quoted 25 lines
--- arch/x86/kernel/alternative.c | 9 +++++++++ 1 file changed, 9 insertions(+)arch/x86/kernel/alternative.c +9
-0@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); + +/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);
Quoted 17 lines · Jürgen Groß
Quoted 16 lines · Juergen Gross
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via
text_poke()in a kernel thread, and on another processorexit_mmap()is active for the process which was running on the first cpu before the kernel thread.As
text_poke()is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on whichexit_mmap()is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
Quoted 8 lines
Quoted 5 lines
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")Signed-off-by Juergen Gross <[email protected]>Can anyone look at this, please? It is fixing a real problem which has been seen several times.
As it happens I picked it up yesterday, just pushed it out for you.
Thanks!
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Jürgen Groß
Quoted 19 lines · Juergen Gross
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via
text_poke()in a kernel thread, and on another processorexit_mmap()is active for the process which was running on the first cpu before the kernel thread.As
text_poke()is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on whichexit_mmap()is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")Signed-off-by Juergen Gross <[email protected]>Can anyone look at this, please? It is fixing a real problem which has been seen several times.
As it happens I picked it up yesterday, just pushed it out for you.
Thank you very much!
Juergen
[tip: x86/urgent] x86/alternative: Don't call text_poke() in lazy TLB mode
In reply to Juergen GrossThe following commit has been merged into the x86/urgent branch of tip:
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: abee7c494d8c41bb388839bccc47e06247f0d7de Gitweb: https://git.kernel.org/tip/abee7c494d8c41bb388839bccc47e06247f0d7de Author: Juergen Gross <[email protected]> AuthorDate: Fri, 09 Oct 2020 16:42:25 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 22 Oct 2020 12:37:23 +02:00
x86/alternative: Don't call text_poke() in lazy TLB mode
When running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via text_poke() in a kernel thread, and on another processor exit_mmap() is active for the process which was running on the first cpu before the kernel thread.
As text_poke() is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on which exit_mmap() is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
--- arch/x86/kernel/alternative.c | 9 +++++++++ 1 file changed, 9 insertions(+)
arch/x86/kernel/alternative.c +9 -0
@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); + +/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);
Andy Lutomirski
Acked-byRe: [tip: x86/urgent] x86/alternative: Don't call text_poke() in lazy TLB mode
Quoted 28 lines · On Thu, Oct 22, 2020 at 3:50 AM tip-bot2 for Juergen Gross…
The following commit has been merged into the x86/urgent branch of tip:
Commit-ID: abee7c494d8c41bb388839bccc47e06247f0d7de Gitweb: https://git.kernel.org/tip/abee7c494d8c41bb388839bccc47e06247f0d7de Author: Juergen Gross <[email protected]> AuthorDate: Fri, 09 Oct 2020 16:42:25 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 22 Oct 2020 12:37:23 +02:00x86/alternative: Don't call
text_poke()in lazy TLB modeWhen running in lazy TLB mode the currently active page tables might be the ones of a previous process, e.g. when running a kernel thread.
This can be problematic in case kernel code is being modified via
text_poke()in a kernel thread, and on another processorexit_mmap()is active for the process which was running on the first cpu before the kernel thread.As
text_poke()is using a temporary address space and the former address space (obtained via cpu_tlbstate.loaded_mm) is restored afterwards, there is a race possible in case the cpu on whichexit_mmap()is running wants to make sure there are no stale references to that address space on any cpu active (this e.g. is required when running as a Xen PV guest, where this problem has been observed and analyzed).
Quoted 2 lines
In order to avoid that, drop off TLB lazy mode before switching to the temporary address space.
Now that I'm actually awake:
although it might be nice to at least have a comment that there's some performance being left on the table.
PeterZ, I like your version except that, if we do that, I also think we should move this whole mess into tlb.c instead of alternative.c.
--Andy
Quoted 29 lines
Fixes cefa929c034eb5d ("x86/mm: Introduce temporary mm structs")Signed-off-by Juergen Gross <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>--- arch/x86/kernel/alternative.c | 9 +++++++++ 1 file changed, 9 insertions(+)arch/x86/kernel/alternative.c +9
-0@@ -807,6 +807,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)temp_mm_state_t temp_state;lockdep_assert_irqs_disabled (); + +/* +* Make sure not to be in TLB lazy mode, as otherwise we'll end up +* with a stale address space WITHOUT being in lazy mode after +* restoring the previous mm. +*/ +if (this_cpu_read (cpu_tlbstate.is_lazy)) +leave_mm (smp_processor_id ()); + temp_state.mm =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current);
Signature · 2 lines
Andy Lutomirski AMA Capital Management, LLC