Factor out, clean up and use the use_/unuse_temporary_mm() APIs some more
Subsystems (4)
cover Factor out, clean up and use the use_/unuse_temporary_mm() APIs some more
These are a couple of cleanups and micro-optimizations by Andy and Peter around the x86 use_/unuse_temporary_mm() APIs, which were posted back in November, and which I merged on top of the WIP.x86/alternatives tree:
git://git.kernel.org/pub/scm/linux/kernel/git/mingo/tip.git WIP.x86/mm
Thanks,
Ingo
===============>
Andy Lutomirski (5):
x86/events, x86/insn-eval: Remove incorrect current->active_mm references x86/mm: Make use_/unuse_temporary_mm() non-static
x86/mm: Allow temporary MMs when IRQs are on
x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery x86/mm: Opt-in to IRQs-off activate_mm()
Peter Zijlstra (2):
x86/mm: Add 'mm' argument to unuse_temporary_mm() x86/mm: Remove 'mm' argument from unuse_temporary_mm() again
arch/x86/Kconfig | 1 + arch/x86/events/core.c | 9 ++++- arch/x86/include/asm/mmu_context.h | 5 ++- arch/x86/kernel/alternative.c | 64 ----------------------------------- arch/x86/lib/insn-eval.c | 13 +++++-- arch/x86/mm/tlb.c | 69 ++++++++++++++++++++++++++++++++++++++ arch/x86/platform/efi/efi_64.c | 7 ++-- 7 files changed, 94 insertions(+), 74 deletions(-)
Signature · 1 line
2.45.2
PATCH 1/7 x86/mm: Add 'mm' argument to unuse_temporary_mm()
From: Peter Zijlstra <[email protected]>
In commit 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") unuse_temporary_mm() grew the assumption that it gets used on poking_mm exclusively. While this is currently true, lets not hard code this assumption.
--- arch/x86/kernel/alternative.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/alternative.c +3 -3
@@ -2161,14 +2161,14 @@ static inline struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)__ro_after_init struct mm_struct *text_poke_mm;__ro_after_init unsigned long text_poke_mm_addr; -static inline void unuse_temporary_mm (struct mm_struct *prev_mm) +static inline void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm) {lockdep_assert_irqs_disabled ();switch_mm_irqs_off (NULL , prev_mm, current);/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ -cpumask_clear_cpu (raw_smp_processor_id (),mm_cpumask (text_poke_mm)); +cpumask_clear_cpu (raw_smp_processor_id (),mm_cpumask (mm));/* * Restore the breakpoints if they were disabled before the temporary mm
@@ -2275,7 +2275,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l* instruction that already allows the core to see the updated version. * Xen-PV is assumed to serialize execution in a similar manner. */ -unuse_temporary_mm (prev_mm); +unuse_temporary_mm (text_poke_mm, prev_mm);/* * Flushing the TLB might involve IPIs, which would require enabled
Signature · 1 line
2.45.2
PATCH 2/7 x86/events, x86/insn-eval: Remove incorrect current->active_mm references
In reply to Ingo MolnarFrom: Andy Lutomirski <[email protected]>
When decoding an instruction or handling a perf event that references an LDT segment, if we don't have a valid user context, trying to access the LDT by any means other than SLDT is racy. Certainly, using current->active_mm is wrong, as active_mm can point to a real user mm when CR3 and LDTR no longer reference that mm.
Clean up the code. If nmi_uaccess_okay() says we don't have a valid context, just fail. Otherwise use current->mm.
--- arch/x86/events/core.c | 9 ++++++++- arch/x86/lib/insn-eval.c | 13 ++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-)
arch/x86/events/core.c +8 -1
@@ -2803,8 +2803,15 @@ static unsigned long get_segment_base(unsigned int segment)#ifdef CONFIG_MODIFY_LDT_SYSCALL struct ldt_struct *ldt; +/* +* If we're not in a valid context with a real (not just lazy) +* user mm, then don't even try. +*/ +if (!nmi_uaccess_okay ()) +return 0 ; +/* IRQs are off, so this synchronizes with smp_store_release */ - ldt =READ_ONCE (current->active_mm->context.ldt); + ldt =smp_load_acquire (¤t->mm->context.ldt);if (!ldt || idx >= ldt->nr_entries)return 0 ;
arch/x86/lib/insn-eval.c +10 -3
@@ -631,14 +631,21 @@ static bool get_desc(struct desc_struct *out, unsigned short sel)/* Bits [15:3] contain the index of the desired entry. */ sel >>=3 ; -mutex_lock (¤t->active_mm->context.lock); - ldt = current->active_mm->context.ldt; +/* +* If we're not in a valid context with a real (not just lazy) +* user mm, then don't even try. +*/ +if (!nmi_uaccess_okay ()) +return false; + +mutex_lock (¤t->mm->context.lock); + ldt = current->mm->context.ldt;if (ldt && sel < ldt->nr_entries) { *out = ldt->entries[sel]; success = true; } -mutex_unlock (¤t->active_mm->context.lock); +mutex_unlock (¤t->mm->context.lock);return success; }
Signature · 1 line
2.45.2
PATCH 3/7 x86/mm: Make use_/unuse_temporary_mm() non-static
In reply to Ingo MolnarFrom: Andy Lutomirski <[email protected]>
This prepares them for use outside of the alternative machinery. The code is unchanged.
--- arch/x86/include/asm/mmu_context.h | 3 ++ arch/x86/kernel/alternative.c | 64 -------------------------------------- arch/x86/mm/tlb.c | 64 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 67 insertions(+), 64 deletions(-)
arch/x86/include/asm/mmu_context.h +3 -0
@@ -272,4 +272,7 @@ unsigned long __get_current_cr3_fast(void);#include <asm-generic/mmu_context.h> +extern struct mm_struct *use_temporary_mm (struct mm_struct *temp_mm); +extern void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm); +#endif /* _ASM_X86_MMU_CONTEXT_H */
arch/x86/kernel/alternative.c +0 -64
73 lines not on this page · raw message
arch/x86/mm/tlb.c +64 -0
70 lines not on this page · raw message
Signature · 1 line
2.45.2
PATCH 4/7 x86/mm: Remove 'mm' argument from unuse_temporary_mm() again
In reply to Ingo MolnarFrom: Peter Zijlstra <[email protected]>
Now that unuse_temporary_mm() lives in tlb.c it can access cpu_tlbstate.loaded_mm.
[ mingo: Merged it on top of x86/alternatives ]
--- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/alternative.c | 2 +- arch/x86/mm/tlb.c | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/mmu_context.h +1 -1
@@ -273,6 +273,6 @@ unsigned long __get_current_cr3_fast(void);#include <asm-generic/mmu_context.h> extern struct mm_struct *use_temporary_mm (struct mm_struct *temp_mm); -extern void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm); +extern void unuse_temporary_mm (struct mm_struct *prev_mm);#endif /* _ASM_X86_MMU_CONTEXT_H */
arch/x86/kernel/alternative.c +1 -1
@@ -2211,7 +2211,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l* instruction that already allows the core to see the updated version. * Xen-PV is assumed to serialize execution in a similar manner. */ -unuse_temporary_mm (text_poke_mm, prev_mm); +unuse_temporary_mm (prev_mm);/* * Flushing the TLB might involve IPIs, which would require enabled
arch/x86/mm/tlb.c +4 -4
@@ -1019,14 +1019,14 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)return prev_mm; } -void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm) +void unuse_temporary_mm (struct mm_struct *prev_mm) {lockdep_assert_irqs_disabled (); -switch_mm_irqs_off (NULL , prev_mm, current); -/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ -cpumask_clear_cpu (raw_smp_processor_id (),mm_cpumask (mm)); +cpumask_clear_cpu (smp_processor_id (),mm_cpumask (this_cpu_read (cpu_tlbstate.loaded_mm))); + +switch_mm_irqs_off (NULL , prev_mm, current);/* * Restore the breakpoints if they were disabled before the temporary mm
Signature · 1 line
2.45.2
PATCH 5/7 x86/mm: Allow temporary MMs when IRQs are on
In reply to Ingo MolnarFrom: Andy Lutomirski <[email protected]>
EFI runtime services should use temporary MMs, but EFI runtime services want IRQs on. Preemption must still be disabled in a temporary MM context.
At some point, the entirely temporary MM mechanism should be moved out of arch code.
--- arch/x86/mm/tlb.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
arch/x86/mm/tlb.c +12 -7
@@ -978,18 +978,23 @@ void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)* that override the kernel memory protections (e.g., W^X), without exposing the * temporary page-table mappings that are required for these write operations to * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns when the -* mapping is torn down. +* mapping is torn down. Temporary mms can also be used for EFI runtime service +* calls or similar functionality. * -* Context: The temporary mm needs to be used exclusively by a single core. To -* harden security IRQs must be disabled while the temporary mm is -* loaded, thereby preventing interrupt handler bugs from overriding -* the kernel memory protection. +* It is illegal to schedule while using a temporary mm -- the context switch +* code is unaware of the temporary mm and does not know how to context switch. +* Use a real (non-temporary) mm in a kernel thread if you need to sleep. +* +* Note: For sensitive memory writes, the temporary mm needs to be used +* exclusively by a single core, and IRQs should be disabled while the +* temporary mm is loaded, thereby preventing interrupt handler bugs from +* overriding the kernel memory protection. */ struct mm_struct *use_temporary_mm (struct mm_struct *temp_mm) {struct mm_struct *prev_mm; -lockdep_assert_irqs_disabled (); +lockdep_assert_preemption_disabled ();/* * Make sure not to be in TLB lazy mode, as otherwise we'll end up
@@ -1021,7 +1026,7 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)void unuse_temporary_mm (struct mm_struct *prev_mm) { -lockdep_assert_irqs_disabled (); +lockdep_assert_preemption_disabled ();/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ cpumask_clear_cpu (smp_processor_id (),mm_cpumask (this_cpu_read (cpu_tlbstate.loaded_mm)));
Signature · 1 line
2.45.2
PATCH 6/7 x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
In reply to Ingo MolnarFrom: Andy Lutomirski <[email protected]>
This should be considerably more robust. It's also necessary for optimized for_each_possible_lazymm_cpu() on x86 -- without this patch, EFI calls in lazy context would remove the lazy mm from mm_cpumask().
[ mingo: Merged it on top of x86/alternatives ]
--- arch/x86/platform/efi/efi_64.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
arch/x86/platform/efi/efi_64.c +2 -5
@@ -434,15 +434,12 @@ void __init efi_dump_pagetable(void)*/ static void efi_enter_mm (void ) { - efi_prev_mm = current->active_mm; - current->active_mm = &efi_mm; -switch_mm (efi_prev_mm, &efi_mm,NULL ); + efi_prev_mm =use_temporary_mm (&efi_mm); }static void efi_leave_mm (void ) { - current->active_mm = efi_prev_mm; -switch_mm (&efi_mm, efi_prev_mm,NULL ); +unuse_temporary_mm (efi_prev_mm); }void arch_efi_call_virt_setup (void )
Signature · 1 line
2.45.2
PATCH 7/7 x86/mm: Opt-in to IRQs-off activate_mm()
In reply to Ingo MolnarFrom: Andy Lutomirski <[email protected]>
We gain nothing by having the core code enable IRQs right before calling activate_mm() only for us to turn them right back off again in switch_mm().
This will save a few cycles, so execve() should be blazingly fast with this patch applied!
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/mmu_context.h | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -149,6 +149,7 @@ config X86 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH + select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select BUILDTIME_TABLE_SORT select CLKEVT_I8253 select CLOCKSOURCE_WATCHDOG
arch/x86/include/asm/mmu_context.h +1 -1
@@ -190,7 +190,7 @@ extern void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,#define activate_mm (prev, next) \ do { \ paravirt_enter_mmap(next); \ - switch_mm((prev), (next), NULL); \ + switch_mm_irqs_off((prev), (next), NULL); \ } while (0);#ifdef CONFIG_X86_32
Signature · 1 line
2.45.2
[tip: x86/alternatives] x86/mm: Opt-in to IRQs-off activate_mm()
The following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: af8967158f9ad759a93e8e7a933c10e7cbb01ba2 Gitweb: https://git.kernel.org/tip/af8967158f9ad759a93e8e7a933c10e7cbb01ba2 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:40 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:06:08 +02:00
x86/mm: Opt-in to IRQs-off activate_mm()
We gain nothing by having the core code enable IRQs right before calling activate_mm() only for us to turn them right back off again in switch_mm().
This will save a few cycles, so execve() should be blazingly fast with this patch applied!
--- arch/x86/Kconfig | 1 + arch/x86/include/asm/mmu_context.h | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-)
arch/x86/Kconfig +1 -0
@@ -153,6 +153,7 @@ config X86 select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH + select ARCH_WANT_IRQS_OFF_ACTIVATE_MM select BUILDTIME_TABLE_SORT select CLKEVT_I8253 select CLOCKSOURCE_WATCHDOG
arch/x86/include/asm/mmu_context.h +1 -1
@@ -190,7 +190,7 @@ extern void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next,#define activate_mm (prev, next) \ do { \ paravirt_enter_mmap(next); \ - switch_mm((prev), (next), NULL); \ + switch_mm_irqs_off((prev), (next), NULL); \ } while (0);#ifdef CONFIG_X86_32
[tip: x86/alternatives] x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
In reply to Ingo MolnarThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: e7021e2fe0b4335523d3f6e2221000bdfc633b62 Gitweb: https://git.kernel.org/tip/e7021e2fe0b4335523d3f6e2221000bdfc633b62 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:39 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:06:04 +02:00
x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
This should be considerably more robust. It's also necessary for optimized for_each_possible_lazymm_cpu() on x86 -- without this patch, EFI calls in lazy context would remove the lazy mm from mm_cpumask().
[ mingo: Merged it on top of x86/alternatives ]
--- arch/x86/platform/efi/efi_64.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)
arch/x86/platform/efi/efi_64.c +2 -5
@@ -434,15 +434,12 @@ void __init efi_dump_pagetable(void)*/ static void efi_enter_mm (void ) { - efi_prev_mm = current->active_mm; - current->active_mm = &efi_mm; -switch_mm (efi_prev_mm, &efi_mm,NULL ); + efi_prev_mm =use_temporary_mm (&efi_mm); }static void efi_leave_mm (void ) { - current->active_mm = efi_prev_mm; -switch_mm (&efi_mm, efi_prev_mm,NULL ); +unuse_temporary_mm (efi_prev_mm); }void arch_efi_call_virt_setup (void )
[tip: x86/alternatives] x86/mm: Allow temporary MMs when IRQs are on
In reply to Ingo MolnarThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 58f8ffa917669a0c8c027e24d5349f0b488f8181 Gitweb: https://git.kernel.org/tip/58f8ffa917669a0c8c027e24d5349f0b488f8181 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:38 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:06:00 +02:00
x86/mm: Allow temporary MMs when IRQs are on
EFI runtime services should use temporary MMs, but EFI runtime services want IRQs on. Preemption must still be disabled in a temporary MM context.
At some point, the entirely temporary MM mechanism should be moved out of arch code.
--- arch/x86/mm/tlb.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
arch/x86/mm/tlb.c +12 -7
@@ -977,18 +977,23 @@ void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)* that override the kernel memory protections (e.g., W^X), without exposing the * temporary page-table mappings that are required for these write operations to * other CPUs. Using a temporary mm also allows to avoid TLB shootdowns when the -* mapping is torn down. +* mapping is torn down. Temporary mms can also be used for EFI runtime service +* calls or similar functionality. * -* Context: The temporary mm needs to be used exclusively by a single core. To -* harden security IRQs must be disabled while the temporary mm is -* loaded, thereby preventing interrupt handler bugs from overriding -* the kernel memory protection. +* It is illegal to schedule while using a temporary mm -- the context switch +* code is unaware of the temporary mm and does not know how to context switch. +* Use a real (non-temporary) mm in a kernel thread if you need to sleep. +* +* Note: For sensitive memory writes, the temporary mm needs to be used +* exclusively by a single core, and IRQs should be disabled while the +* temporary mm is loaded, thereby preventing interrupt handler bugs from +* overriding the kernel memory protection. */ struct mm_struct *use_temporary_mm (struct mm_struct *temp_mm) {struct mm_struct *prev_mm; -lockdep_assert_irqs_disabled (); +lockdep_assert_preemption_disabled ();/* * Make sure not to be in TLB lazy mode, as otherwise we'll end up
@@ -1020,7 +1025,7 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)void unuse_temporary_mm (struct mm_struct *prev_mm) { -lockdep_assert_irqs_disabled (); +lockdep_assert_preemption_disabled ();/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ cpumask_clear_cpu (smp_processor_id (),mm_cpumask (this_cpu_read (cpu_tlbstate.loaded_mm)));
[tip: x86/alternatives] x86/mm: Remove 'mm' argument from unuse_temporary_mm() again
In reply to Ingo MolnarThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 4873f494bbe4670f353a9b76ce44e6028c811cbb Gitweb: https://git.kernel.org/tip/4873f494bbe4670f353a9b76ce44e6028c811cbb Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:37 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:05:56 +02:00
x86/mm: Remove 'mm' argument from unuse_temporary_mm() again
Now that unuse_temporary_mm() lives in tlb.c it can access cpu_tlbstate.loaded_mm.
[ mingo: Merged it on top of x86/alternatives ]
--- arch/x86/include/asm/mmu_context.h | 2 +- arch/x86/kernel/alternative.c | 2 +- arch/x86/mm/tlb.c | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-)
arch/x86/include/asm/mmu_context.h +1 -1
@@ -273,6 +273,6 @@ unsigned long __get_current_cr3_fast(void);#include <asm-generic/mmu_context.h> extern struct mm_struct *use_temporary_mm (struct mm_struct *temp_mm); -extern void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm); +extern void unuse_temporary_mm (struct mm_struct *prev_mm);#endif /* _ASM_X86_MMU_CONTEXT_H */
arch/x86/kernel/alternative.c +1 -1
@@ -2211,7 +2211,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l* instruction that already allows the core to see the updated version. * Xen-PV is assumed to serialize execution in a similar manner. */ -unuse_temporary_mm (text_poke_mm, prev_mm); +unuse_temporary_mm (prev_mm);/* * Flushing the TLB might involve IPIs, which would require enabled
arch/x86/mm/tlb.c +4 -4
@@ -1018,14 +1018,14 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)return prev_mm; } -void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm) +void unuse_temporary_mm (struct mm_struct *prev_mm) {lockdep_assert_irqs_disabled (); -switch_mm_irqs_off (NULL , prev_mm, current); -/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ -cpumask_clear_cpu (raw_smp_processor_id (),mm_cpumask (mm)); +cpumask_clear_cpu (smp_processor_id (),mm_cpumask (this_cpu_read (cpu_tlbstate.loaded_mm))); + +switch_mm_irqs_off (NULL , prev_mm, current);/* * Restore the breakpoints if they were disabled before the temporary mm
[tip: x86/alternatives] x86/events, x86/insn-eval: Remove incorrect current->active_mm references
In reply to Ingo MolnarThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 81e3cbdef230fd9adfa8569044b07290afd66708 Gitweb: https://git.kernel.org/tip/81e3cbdef230fd9adfa8569044b07290afd66708 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:35 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:05:46 +02:00
x86/events, x86/insn-eval: Remove incorrect current->active_mm references
When decoding an instruction or handling a perf event that references an LDT segment, if we don't have a valid user context, trying to access the LDT by any means other than SLDT is racy. Certainly, using current->active_mm is wrong, as active_mm can point to a real user mm when CR3 and LDTR no longer reference that mm.
Clean up the code. If nmi_uaccess_okay() says we don't have a valid context, just fail. Otherwise use current->mm.
--- arch/x86/events/core.c | 9 ++++++++- arch/x86/lib/insn-eval.c | 13 ++++++++++--- 2 files changed, 18 insertions(+), 4 deletions(-)
arch/x86/events/core.c +8 -1
@@ -2803,8 +2803,15 @@ static unsigned long get_segment_base(unsigned int segment)#ifdef CONFIG_MODIFY_LDT_SYSCALL struct ldt_struct *ldt; +/* +* If we're not in a valid context with a real (not just lazy) +* user mm, then don't even try. +*/ +if (!nmi_uaccess_okay ()) +return 0 ; +/* IRQs are off, so this synchronizes with smp_store_release */ - ldt =READ_ONCE (current->active_mm->context.ldt); + ldt =smp_load_acquire (¤t->mm->context.ldt);if (!ldt || idx >= ldt->nr_entries)return 0 ;
arch/x86/lib/insn-eval.c +10 -3
@@ -631,14 +631,21 @@ static bool get_desc(struct desc_struct *out, unsigned short sel)/* Bits [15:3] contain the index of the desired entry. */ sel >>=3 ; -mutex_lock (¤t->active_mm->context.lock); - ldt = current->active_mm->context.ldt; +/* +* If we're not in a valid context with a real (not just lazy) +* user mm, then don't even try. +*/ +if (!nmi_uaccess_okay ()) +return false; + +mutex_lock (¤t->mm->context.lock); + ldt = current->mm->context.ldt;if (ldt && sel < ldt->nr_entries) { *out = ldt->entries[sel]; success = true; } -mutex_unlock (¤t->active_mm->context.lock); +mutex_unlock (¤t->mm->context.lock);return success; }
[tip: x86/alternatives] x86/mm: Make use_/unuse_temporary_mm() non-static
In reply to Ingo MolnarThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: d376972c9825ac4e8ad74872ee0730a5b4292e44 Gitweb: https://git.kernel.org/tip/d376972c9825ac4e8ad74872ee0730a5b4292e44 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:36 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:05:52 +02:00
x86/mm: Make use_/unuse_temporary_mm() non-static
This prepares them for use outside of the alternative machinery. The code is unchanged.
--- arch/x86/include/asm/mmu_context.h | 3 +- arch/x86/kernel/alternative.c | 64 +----------------------------- arch/x86/mm/tlb.c | 64 +++++++++++++++++++++++++++++- 3 files changed, 67 insertions(+), 64 deletions(-)
arch/x86/include/asm/mmu_context.h +3 -0
@@ -272,4 +272,7 @@ unsigned long __get_current_cr3_fast(void);#include <asm-generic/mmu_context.h> +extern struct mm_struct *use_temporary_mm (struct mm_struct *temp_mm); +extern void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm); +#endif /* _ASM_X86_MMU_CONTEXT_H */
arch/x86/kernel/alternative.c +0 -64
73 lines not on this page · raw message
arch/x86/mm/tlb.c +64 -0
70 lines not on this page · raw message
[tip: x86/alternatives] x86/mm: Add 'mm' argument to unuse_temporary_mm()
In reply to Ingo MolnarThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: 0812e096cff0fd58d88a21a413fba56c0e6c3caa Gitweb: https://git.kernel.org/tip/0812e096cff0fd58d88a21a413fba56c0e6c3caa Author: Peter Zijlstra <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:34 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:05:37 +02:00
x86/mm: Add 'mm' argument to unuse_temporary_mm()
In commit 209954cbc7d0 ("x86/mm/tlb: Update mm_cpumask lazily") unuse_temporary_mm() grew the assumption that it gets used on poking_mm exclusively. While this is currently true, lets not hard code this assumption.
--- arch/x86/kernel/alternative.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-)
arch/x86/kernel/alternative.c +3 -3
@@ -2161,14 +2161,14 @@ static inline struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)__ro_after_init struct mm_struct *text_poke_mm;__ro_after_init unsigned long text_poke_mm_addr; -static inline void unuse_temporary_mm (struct mm_struct *prev_mm) +static inline void unuse_temporary_mm (struct mm_struct *mm,struct mm_struct *prev_mm) {lockdep_assert_irqs_disabled ();switch_mm_irqs_off (NULL , prev_mm, current);/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ -cpumask_clear_cpu (raw_smp_processor_id (),mm_cpumask (text_poke_mm)); +cpumask_clear_cpu (raw_smp_processor_id (),mm_cpumask (mm));/* * Restore the breakpoints if they were disabled before the temporary mm
@@ -2275,7 +2275,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l* instruction that already allows the core to see the updated version. * Xen-PV is assumed to serialize execution in a similar manner. */ -unuse_temporary_mm (prev_mm); +unuse_temporary_mm (text_poke_mm, prev_mm);/* * Flushing the TLB might involve IPIs, which would require enabled
Borislav Petkov
maintainerRe: [tip: x86/alternatives] x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
In reply to tip-bot2 for Andy LutomirskiQuoted 41 lines · On Sat, Apr 12, 2025 at 06:46:48PM -0000, tip-bot2 for Andy Lutomirski wrote:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: e7021e2fe0b4335523d3f6e2221000bdfc633b62 Gitweb: https://git.kernel.org/tip/e7021e2fe0b4335523d3f6e2221000bdfc633b62 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:39 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:06:04 +02:00x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
This should be considerably more robust. It's also necessary for optimized
for_each_possible_lazymm_cpu()on x86 -- without this patch, EFI calls in lazy context would remove the lazy mm frommm_cpumask().[ mingo: Merged it on top of x86/alternatives ]
Signed-off-by Andy Lutomirski <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Cc Rik van Riel <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>--- arch/x86/platform/efi/efi_64.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)arch/x86/platform/efi/efi_64.c +1
-3@@ -434,15 +434,12 @@ void __init efi_dump_pagetable(void)*/ static void efi_enter_mm (void ) { - efi_prev_mm = current->active_mm; - current->active_mm = &efi_mm; -switch_mm (efi_prev_mm, &efi_mm,NULL ); + efi_prev_mm =use_temporary_mm (&efi_mm);
arch/x86/platform/efi/efi_64.c void __init efi_dump_pagetable(void) · line 445
Quoted 10 lines
arch/x86/platform/efi/efi_64.c +1
-2}static void efi_leave_mm (void ) { - current->active_mm = efi_prev_mm; -switch_mm (&efi_mm, efi_prev_mm,NULL ); +unuse_temporary_mm (efi_prev_mm); }void arch_efi_call_virt_setup (void )
mingo thinks this one causes this:
[ 0.119491] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.119498] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format. [ 0.137368] Freeing SMP alternatives memory: 40K [ 0.137381] pid_max: default: 32768 minimum: 301 [ 0.137496] ------------[ cut here ]------------ [ 0.137502] WARNING: CPU: 0 PID: 0 at arch/x86/mm/tlb.c:795 switch_mm_irqs_off+0x3d3/0x460 [ 0.137516] Modules linked in: [ 0.137526] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Not tainted 6.15.0-rc2+ #3 PREEMPT(voluntary) [ 0.137537] Hardware name: HP HP ProBook 635 Aero G7 Notebook PC/8830, BIOS S84 Ver. 01.05.00 05/14/2021 [ 0.137548] RIP: 0010:switch_mm_irqs_off+0x3d3/0x460 [ 0.137556] Code: 28 00 65 ff 0d 3e c9 db 01 0f 85 88 fd ff ff 0f 1f 44 00 00 e9 7e fd ff ff be 00 01 00 00 31 ff e8 02 cb fb ff e9 be fd ff ff <0f> 0b e9 6c fc ff ff 9c 58 f6 c4 02 0f 84 c4 fd ff ff e8 46 3b 59 [ 0.137575] RSP: 0000:ffffffffb6a03e00 EFLAGS: 00010202 [ 0.137583] RAX: 0000000000000246 RBX: ffffffffb6c5fd40 RCX: 0000000100238000 [ 0.137591] RDX: ffffffffb6a149c0 RSI: ffffffffb6c5fd40 RDI: 0000000000000000 [ 0.137599] RBP: ffffffffb6bbcdc0 R08: 00000000b357d000 R09: 0000000000000000 [ 0.137607] R10: 000000010ab06067 R11: 0000000000000000 R12: ffffffffb6a149c0 [ 0.137616] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 [ 0.137624] FS: 0000000000000000(0000) GS:ffff9b6093385000(0000) knlGS:0000000000000000 [ 0.137633] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.137640] CR2: ffff9b6048601000 CR3: 0000000106c26000 CR4: 0000000000350ef0 [ 0.137648] Call Trace: [ 0.137653] <TASK> [ 0.137658] use_temporary_mm+0x55/0x90 [ 0.137666] efi_set_virtual_address_map+0xfd/0x1b0 [ 0.137676] efi_enter_virtual_mode+0x3e3/0x450 [ 0.137685] start_kernel+0x6b7/0x720 [ 0.137693] x86_64_start_reservations+0x24/0x30 [ 0.137700] x86_64_start_kernel+0x7a/0x80 [ 0.137706] common_startup_64+0x13e/0x141 [ 0.137717] </TASK> [ 0.137720] irq event stamp: 128439 [ 0.137725] hardirqs last enabled at (128447): [<ffffffffb579dcd2>] __up_console_sem+0x52/0x60 [ 0.137737] hardirqs last disabled at (128454): [<ffffffffb579dcb7>] __up_console_sem+0x37/0x60 [ 0.137748] softirqs last enabled at (105766): [<ffffffffb56f57b6>] __irq_exit_rcu+0x96/0xc0 [ 0.137759] softirqs last disabled at (105759): [<ffffffffb56f57b6>] __irq_exit_rcu+0x96/0xc0 [ 0.137770] ---[ end trace 0000000000000000 ]--- [ 0.137777] ------------[ cut here ]------------ [ 0.137782] WARNING: CPU: 0 PID: 0 at arch/x86/kernel/cpu/common.c:453 cr4_update_irqsoff+0x45/0x70 [ 0.137794] Modules linked in: [ 0.137800] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G W 6.15.0-rc2+ #3 PREEMPT(voluntary) [ 0.137813] Tainted: [W]=WARN [ 0.137817] Hardware name: HP HP ProBook 635 Aero G7 Notebook PC/8830, BIOS S84 Ver. 01.05.00 05/14/2021 [ 0.137827] RIP: 0010:cr4_update_irqsoff+0x45/0x70 [ 0.137834] Code: 0b 65 8b 0d d5 3d e0 01 85 c9 74 13 48 f7 d7 48 21 d7 48 09 c7 48 39 fa 75 20 e9 f6 8d b8 00 65 8b 0d 9b 3a e0 01 85 c9 74 e2 <0f> 0b 48 f7 d7 48 21 d7 48 09 c7 48 39 fa 74 e0 65 48 89 3d eb 6a [ 0.137853] RSP: 0000:ffffffffb6a03df8 EFLAGS: 00010202 [ 0.137860] RAX: 0000000000000000 RBX: ffffffffb6c5fd40 RCX: 0000000000000001 [ 0.137868] RDX: 0000000000350ef0 RSI: 0000000000000100 RDI: 0000000000000100 [ 0.137876] RBP: ffffffffb6bbcdc0 R08: 00000000b357d000 R09: 0000000000000000 [ 0.137884] R10: 000000010ab06067 R11: 0000000000000000 R12: 000000010022e000 [ 0.137892] R13: 0000000000010000 R14: 0000000000000000 R15: 0000000000000000 [ 0.137900] FS: 0000000000000000(0000) GS:ffff9b6093385000(0000) knlGS:0000000000000000 [ 0.137909] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.137916] CR2: ffff9b6048601000 CR3: 000000010022e000 CR4: 0000000000350ef0 [ 0.137924] Call Trace: [ 0.137928] <TASK> [ 0.137932] switch_mm_irqs_off+0x3ce/0x460 [ 0.137940] use_temporary_mm+0x55/0x90 [ 0.137946] efi_set_virtual_address_map+0xfd/0x1b0 [ 0.137956] efi_enter_virtual_mode+0x3e3/0x450 [ 0.137964] start_kernel+0x6b7/0x720 [ 0.137971] x86_64_start_reservations+0x24/0x30 [ 0.137978] x86_64_start_kernel+0x7a/0x80 [ 0.137984] common_startup_64+0x13e/0x141 [ 0.137994] </TASK> [ 0.137998] irq event stamp: 128723 [ 0.138002] hardirqs last enabled at (128731): [<ffffffffb579dcd2>] __up_console_sem+0x52/0x60 [ 0.138013] hardirqs last disabled at (128738): [<ffffffffb579dcb7>] __up_console_sem+0x37/0x60 [ 0.138024] softirqs last enabled at (105766): [<ffffffffb56f57b6>] __irq_exit_rcu+0x96/0xc0 [ 0.138034] softirqs last disabled at (105759): [<ffffffffb56f57b6>] __irq_exit_rcu+0x96/0xc0 [ 0.138045] ---[ end trace 0000000000000000 ]---
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquettePeter Zijlstra
maintainerRe: [tip: x86/alternatives] x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
Quoted 55 lines · Borislav Petkov
Quoted 41 lines · On Sat, Apr 12, 2025 at 06:46:48PM -0000, tip-bot2 for Andy Lutomirski wrote:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: e7021e2fe0b4335523d3f6e2221000bdfc633b62 Gitweb: https://git.kernel.org/tip/e7021e2fe0b4335523d3f6e2221000bdfc633b62 Author: Andy Lutomirski <[email protected]> AuthorDate: Wed, 02 Apr 2025 11:45:39 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Sat, 12 Apr 2025 10:06:04 +02:00x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
This should be considerably more robust. It's also necessary for optimized
for_each_possible_lazymm_cpu()on x86 -- without this patch, EFI calls in lazy context would remove the lazy mm frommm_cpumask().[ mingo: Merged it on top of x86/alternatives ]
Signed-off-by Andy Lutomirski <[email protected]>Signed-off-by Peter Zijlstra (Intel) <[email protected]>Signed-off-by Ingo Molnar <[email protected]>Cc Rik van Riel <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Linus Torvalds <[email protected]>Cc Andrew Morton <[email protected]>--- arch/x86/platform/efi/efi_64.c | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-)arch/x86/platform/efi/efi_64.c +1
-3@@ -434,15 +434,12 @@ void __init efi_dump_pagetable(void)*/ static void efi_enter_mm (void ) { - efi_prev_mm = current->active_mm; - current->active_mm = &efi_mm; -switch_mm (efi_prev_mm, &efi_mm,NULL ); + efi_prev_mm =use_temporary_mm (&efi_mm);arch/x86/platform/efi/efi_64.c void __init efi_dump_pagetable(void) · line 445
Quoted 10 lines
arch/x86/platform/efi/efi_64.c +1
-2}static void efi_leave_mm (void ) { - current->active_mm = efi_prev_mm; -switch_mm (&efi_mm, efi_prev_mm,NULL ); +unuse_temporary_mm (efi_prev_mm); }void arch_efi_call_virt_setup (void )mingo thinks this one causes this:
Quoted 7 lines
[ 0.119491] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.119498] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'compacted' format. [ 0.137368] Freeing SMP alternatives memory: 40K [ 0.137381] pid_max: default: 32768 minimum: 301 [ 0.137496] ------------[ cut here ]------------ [ 0.137502] WARNING: CPU: 0 PID: 0 at arch/x86/mm/tlb.c:795 switch_mm_irqs_off+0x3d3/0x460 [ 0.137516] Modules linked in:
Ah yes :-( Something like so perhaps..
---
Subject: x86/mm: Fix {,un}use_temporary_mm() IRQ state
As the function switch_mm_irqs_off() implies, it ought to be called with IRQs *off*. Commit 58f8ffa91766 ("x86/mm: Allow temporary MMs when IRQs are on") caused this to not be the case for EFI.
Ensure IRQs are off where it matters.
---
arch/x86/mm/tlb.c +2 -0
@@ -991,6 +991,7 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)struct mm_struct *prev_mm;lockdep_assert_preemption_disabled (); +guard (irqsave)();/* * Make sure not to be in TLB lazy mode, as otherwise we'll end up
@@ -1023,6 +1024,7 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)void unuse_temporary_mm (struct mm_struct *prev_mm) {lockdep_assert_preemption_disabled (); +guard (irqsave)();/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ cpumask_clear_cpu (smp_processor_id (),mm_cpumask (this_cpu_read (cpu_tlbstate.loaded_mm)));
Re: [tip: x86/alternatives] x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
Quoted 1 line · Peter Zijlstra
Ah yes :-( Something like so perhaps..
Thanks, that does it.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteIngo Molnar
AppliedRe: [tip: x86/alternatives] x86/efi: Make efi_enter/leave_mm() use the use_/unuse_temporary_mm() machinery
Quoted 7 lines · Borislav Petkov
Quoted 1 line · Peter Zijlstra
Ah yes :-( Something like so perhaps..
Thanks, that does it.
Reported-by Borislav Petkov (AMD) <[email protected]>
Applied to tip:x86/alternatives, thanks guys!
Ingo
[tip: x86/alternatives] x86/mm: Fix {,un}use_temporary_mm() IRQ state
In reply to Peter ZijlstraThe following commit has been merged into the x86/alternatives branch of tip:
The following commit has been merged into the x86/alternatives branch of tip:
Commit-ID: aef1d0209ddf127a8069aca5fa3a062be4136b76 Gitweb: https://git.kernel.org/tip/aef1d0209ddf127a8069aca5fa3a062be4136b76 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 18 Apr 2025 11:50:34 +02:00 Committer: Ingo Molnar <[email protected]> CommitterDate: Fri, 18 Apr 2025 14:36:18 +02:00
x86/mm: Fix {,un}use_temporary_mm() IRQ state
As the function switch_mm_irqs_off() implies, it ought to be called with IRQs *off*. Commit 58f8ffa91766 ("x86/mm: Allow temporary MMs when IRQs are on") caused this to not be the case for EFI.
Ensure IRQs are off where it matters.
--- arch/x86/mm/tlb.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/mm/tlb.c +2 -0
@@ -986,6 +986,7 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)struct mm_struct *prev_mm;lockdep_assert_preemption_disabled (); +guard (irqsave)();/* * Make sure not to be in TLB lazy mode, as otherwise we'll end up
@@ -1018,6 +1019,7 @@ struct mm_struct *use_temporary_mm(struct mm_struct *temp_mm)void unuse_temporary_mm (struct mm_struct *prev_mm) {lockdep_assert_preemption_disabled (); +guard (irqsave)();/* Clear the cpumask, to indicate no TLB flushing is needed anywhere */ cpumask_clear_cpu (smp_processor_id (),mm_cpumask (this_cpu_read (cpu_tlbstate.loaded_mm)));