Merge text_poke fixes and executable lockdowns
Subsystems (20)
v2 cover Merge text_poke fixes and executable lockdowns
This patchset improves several overlapping issues around stale TLB entries and W^X violations. It is combined from a slightly tweaked "x86/alternative: text_poke() enhancements v7" [1] and a next version of the "Don’t leave executable TLB entries to freed pages v2" [2] patchsets that were conflicting.
The related issues that this fixes:
1. Fixmap PTEs that are used for patching are available for access from other cores and might be exploited. They are not even flushed from the TLB in remote cores, so the risk is even higher. Address this issue by introducing a temporary mm that is only used during patching. Unfortunately, due to init ordering, fixmap is still used during boot-time patching. Future patches can eliminate the need for it.
2. Missing lockdep assertion to ensure text_mutex is taken. It is actually not always taken, so fix the instances that were found not to take the lock (although they should be safe even without taking the lock).
3. Module_alloc returning memory that is RWX until a module is finished loading.
4. Sometimes when memory is freed via the module subsystem, an executable permissioned TLB entry can remain to a freed page. If the page is re-used to back an address that will receive data from userspace, it can result in user data being mapped as executable in the kernel. The root of this behavior is vfree lazily flushing the TLB, but not lazily freeing the underlying pages.
Changes for v2:
- Adding “Reviewed-by tag” [Masami]
- Comment instead of code to warn against module removal while patching [Masami]
- Avoiding open-coded TLB flush [Andy]
- Remove "This patch" [Borislav Petkov]
- Not set global bit during text poking [Andy, hpa]
- Add Ack from [Pavel Machek]
- Split patch 16 "Plug in new special vfree flag" into 4 patches (16-19) to make it easier to review. There were no code changes.
The changes from "Don’t leave executable TLB entries to freed pages v2" to v1:
- Add support for case of hibernate trying to save an unmapped page on the directmap. (Ard Biesheuvel)
- No week arch breakout for vfree-ing special memory (Andy Lutomirski)
- Avoid changing deferred free code by moving modules init free to work queue (Andy Lutomirski)
- Plug in new flag for kprobes and ftrace
- More arch generic names for set_pages functions (Ard Biesheuvel)
- Fix for TLB not always flushing the directmap (Nadav Amit)
Changes from "x86/alternative: text_poke() enhancements v7" to v1
- Fix build failure on CONFIG_RANDOMIZE_BASE=n (Rick)
- Remove text_poke usage from ftrace (Nadav)
[1] https://lkml.org/lkml/2018/12/5/200
[2] https://lkml.org/lkml/2018/12/11/1571
Andy Lutomirski (1):
x86/mm: temporary mm struct
Nadav Amit (12):
Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()" x86/jump_label: Use text_poke_early() during early init
fork: provide a function for copying init_mm x86/alternative: initializing temporary mm for patching x86/alternative: use temporary mm for text poking x86/kgdb: avoid redundant comparison of patched code x86/ftrace: set trampoline pages as executable x86/kprobes: instruction pages initialization enhancements
x86: avoid W^X being broken during modules loading x86/jump-label: remove support for custom poker x86/alternative: Remove the return value of text_poke_*() x86/alternative: comment about module removal races
Rick Edgecombe (7):
Add set_alias_ function and x86 implementation
mm: Make hibernate handle unmapped pages
vmalloc: New flags for safe vfree on special perms
modules: Use vmalloc special flag
bpf: Use vmalloc special flag
x86/ftrace: Use vmalloc special flag x86/kprobes: Use vmalloc special flag
arch/Kconfig | 4 + arch/x86/Kconfig | 1 + arch/x86/include/asm/fixmap.h | 2 - arch/x86/include/asm/mmu_context.h | 32 +++++ arch/x86/include/asm/pgtable.h | 3 + arch/x86/include/asm/set_memory.h | 3 + arch/x86/include/asm/text-patching.h | 7 +- arch/x86/kernel/alternative.c | 199 ++++++++++++++++++++------- arch/x86/kernel/ftrace.c | 14 +- arch/x86/kernel/jump_label.c | 19 ++- arch/x86/kernel/kgdb.c | 25 +--- arch/x86/kernel/kprobes/core.c | 19 ++- arch/x86/kernel/module.c | 2 +- arch/x86/mm/init_64.c | 36 +++++ arch/x86/mm/pageattr.c | 16 ++- arch/x86/xen/mmu_pv.c | 2 - include/linux/filter.h | 18 +-- include/linux/mm.h | 18 +-- include/linux/sched/task.h | 1 + include/linux/set_memory.h | 10 ++ include/linux/vmalloc.h | 13 ++ init/main.c | 3 + kernel/bpf/core.c | 1 - kernel/fork.c | 24 +++- kernel/module.c | 82 ++++++----- mm/page_alloc.c | 7 +- mm/vmalloc.c | 122 +++++++++++++--- 27 files changed, 494 insertions(+), 189 deletions(-)
Signature · 1 line
2.17.1
PATCH v2 1/20 Fix "x86/alternatives: Lockdep-enforce text_mutex in text_poke*()"
From: Nadav Amit <[email protected]>
text_mutex is currently expected to be held before text_poke() is called, but we kgdb does not take the mutex, and instead *supposedly* ensures the lock is not taken and will not be acquired by any other core while text_poke() is running.
The reason for the "supposedly" comment is that it is not entirely clear that this would be the case if kgdb_do_roundup is zero.
Create two wrapper functions, text_poke() and text_poke_kgdb() which do or do not run the lockdep assertion respectively.
While we are at it, change the return code of text_poke() to something meaningful. One day, callers might actually respect it and the existing BUG_ON() when patching fails could be removed. For kgdb, the return value can actually be used.
--- arch/x86/include/asm/text-patching.h | 1 + arch/x86/kernel/alternative.c | 52 ++++++++++++++++++++-------- arch/x86/kernel/kgdb.c | 11 +++--- 3 files changed, 45 insertions(+), 19 deletions(-)
arch/x86/include/asm/text-patching.h +1 -0
@@ -35,6 +35,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);* inconsistent instruction while you patch. */ extern void *text_poke (void *addr,const void *opcode,size_t len); +extern void *text_poke_kgdb (void *addr,const void *opcode,size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem;
arch/x86/kernel/alternative.c +38 -14
70 lines not on this page · raw message
arch/x86/kernel/kgdb.c +6 -5
@@ -758,13 +758,13 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt)if (!err)return err;/* -* It is safe to call text_poke() because normal kernel execution +* It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked (&text_mutex))return -EBUSY ; -text_poke ((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, -BREAK_INSTR_SIZE ); +text_poke_kgdb ((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr, +BREAK_INSTR_SIZE ); err =probe_kernel_read (opc, (char *)bpt->bpt_addr,BREAK_INSTR_SIZE );if (err)return err;
@@ -783,12 +783,13 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt)if (bpt->type !=BP_POKE_BREAKPOINT ) goto knl_write;/* -* It is safe to call text_poke() because normal kernel execution +* It is safe to call text_poke_kgdb() because normal kernel execution * is stopped on all cores, so long as the text_mutex is not locked. */ if (mutex_is_locked (&text_mutex)) goto knl_write; -text_poke ((void *)bpt->bpt_addr, bpt->saved_instr,BREAK_INSTR_SIZE ); +text_poke_kgdb ((void *)bpt->bpt_addr, bpt->saved_instr, +BREAK_INSTR_SIZE ); err =probe_kernel_read (opc, (char *)bpt->bpt_addr,BREAK_INSTR_SIZE );if (err ||memcmp (opc, bpt->saved_instr,BREAK_INSTR_SIZE )) goto knl_write;
Signature · 1 line
2.17.1
PATCH v2 2/20 x86/jump_label: Use text_poke_early() during early init
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
There is no apparent reason not to use text_poke_early() while we are during early-init and we do not patch code that might be on the stack (i.e., we'll return to the middle of the patched code). This appears to be the case of jump-labels, so do so.
This is required for the next patches that would set a temporary mm for patching, which is initialized after some static-keys are enabled/disabled.
--- arch/x86/kernel/jump_label.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
arch/x86/kernel/jump_label.c +6 -1
@@ -50,7 +50,12 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset =jump_entry_target (entry) - (jump_entry_code (entry) +JUMP_LABEL_NOP_SIZE ); -if (early_boot_irqs_disabled) +/* +* As long as we're UP and not yet marked RO, we can use +* text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to +* SYSTEM_SCHEDULING before going either. +*/ +if (system_state ==SYSTEM_BOOTING ) poker = text_poke_early;if (type ==JUMP_LABEL_JMP ) {
Signature · 1 line
2.17.1
PATCH v2 3/20 x86/mm: temporary mm struct
In reply to Rick EdgecombeFrom: Andy Lutomirski <[email protected]>
Sometimes we want to set a temporary page-table entries (PTEs) in one of the cores, without allowing other cores to use - even speculatively - these mappings. There are two benefits for doing so:
(1) Security: if sensitive PTEs are set, temporary mm prevents their use in other cores. This hardens the security as it prevents exploding a dangling pointer to overwrite sensitive data using the sensitive PTE.
(2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables.
To do so a temporary mm_struct can be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled.
The first use-case for temporary PTEs, which will follow, is for poking the kernel text.
[ Commit message was written by Nadav ]
--- arch/x86/include/asm/mmu_context.h | 32 ++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)
arch/x86/include/asm/mmu_context.h +32 -0
@@ -356,4 +356,36 @@ static inline unsigned long __get_current_cr3_fast(void) return cr3; } +typedef struct { +struct mm_struct *prev; +}temporary_mm_state_t ; + +/* +* Using a temporary mm allows to set temporary mappings that are not accessible +* by other cores. Such mappings are needed to perform sensitive memory writes +* that override the kernel memory protections (e.g., W^X), without exposing the +* temporary page-table mappings that are required for these write operations to +* other cores. +* +* Context: The temporary mm needs to be used exclusively by a single core. To +* harden security IRQs must be disabled while the temporary mm is +* loaded, thereby preventing interrupt handler bugs from override the +* kernel memory protection. +*/ +static inline temporary_mm_state_t use_temporary_mm (struct mm_struct *mm) +{ +temporary_mm_state_t state; + +lockdep_assert_irqs_disabled (); + state.prev =this_cpu_read (cpu_tlbstate.loaded_mm); +switch_mm_irqs_off (NULL , mm, current); +return state; +} + +static inline void unuse_temporary_mm (temporary_mm_state_t prev) +{ +lockdep_assert_irqs_disabled (); +switch_mm_irqs_off (NULL , prev.prev, current); +} +#endif /* _ASM_X86_MMU_CONTEXT_H */
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 03/20] x86/mm: temporary mm struct
Quoted 1 line
Subject: Re: [PATCH v2 03/20] x86/mm: temporary mm struct
Subject needs a verb: "Add a temporary... "
Quoted 3 lines · Rick Edgecombe
From: Andy Lutomirski <[email protected]>
Sometimes we want to set a temporary page-table entries (PTEs) in one of
s/a //
Also, drop the "we" and make it impartial and passive:
"Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."
Quoted 5 lines
the cores, without allowing other cores to use - even speculatively - these mappings. There are two benefits for doing so:
(1) Security: if sensitive PTEs are set, temporary mm prevents their use in other cores. This hardens the security as it prevents exploding a
exploding or exploiting? Or exposing? :)
Quoted 4 lines
dangling pointer to overwrite sensitive data using the sensitive PTE.
(2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables.
Those belong in the code comments below, explaining what it is going to be used for.
Quoted 27 lines
To do so a temporary
mm_structcan be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled.The first use-case for temporary PTEs, which will follow, is for poking the kernel text.
[ Commit message was written by Nadav ]
Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Signed-off-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- arch/x86/include/asm/mmu_context.h | 32 ++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+)arch/x86/include/asm/mmu_context.h +0
-0
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 359
Quoted 4 lines
arch/x86/include/asm/mmu_context.h +1
-0@@ -356,4 +356,36 @@ static inline unsigned long __get_current_cr3_fast(void)return cr3; } +typedef struct {
Why does it have to be a typedef?
That prev.prev below looks unnecessary, instead of just using prev.
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 360
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+struct mm_struct *prev;
Why "prev"?
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 361
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+} temporary_mm_state_t;
That's kinda long - it is longer than the function name below. temp_mm_state_t not enough?
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 371
Quoted 10 lines
arch/x86/include/asm/mmu_context.h +10
-0+ +/* +* Using a temporary mm allows to set temporary mappings that are not accessible +* by other cores. Such mappings are needed to perform sensitive memory writes +* that override the kernel memory protections (e.g., W^X), without exposing the +* temporary page-table mappings that are required for these write operations to +* other cores. +* +* Context: The temporary mm needs to be used exclusively by a single core. To +* harden security IRQs must be disabled while the temporary mm is
^ ,
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 372
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+* loaded, thereby preventing interrupt handler bugs from override the
s/override/overriding/
Quoted 22 lines
arch/x86/include/asm/mmu_context.h +18
-0+* kernel memory protection. +*/ +static inline temporary_mm_state_t use_temporary_mm (struct mm_struct *mm) +{ +temporary_mm_state_t state; + +lockdep_assert_irqs_disabled (); + state.prev =this_cpu_read (cpu_tlbstate.loaded_mm); +switch_mm_irqs_off (NULL , mm, current); +return state; +} + +static inline void unuse_temporary_mm (temporary_mm_state_t prev) +{ +lockdep_assert_irqs_disabled (); +switch_mm_irqs_off (NULL , prev.prev, current); +} +#endif /* _ASM_X86_MMU_CONTEXT_H */ Signature · 1 line
2.17.1
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 03/20] x86/mm: temporary mm struct
Quoted 28 lines · Borislav Petkov
Quoted 2 lines
Subject: Re: [PATCH v2 03/20] x86/mm: temporary mm struct
Subject needs a verb: "Add a temporary... "
Quoted 3 lines · Rick Edgecombe
From: Andy Lutomirski <[email protected]>
Sometimes we want to set a temporary page-table entries (PTEs) in one of
s/a //
Also, drop the "we" and make it impartial and passive:
"Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."Quoted 5 lines
the cores, without allowing other cores to use - even speculatively - these mappings. There are two benefits for doing so:
(1) Security: if sensitive PTEs are set, temporary mm prevents their use in other cores. This hardens the security as it prevents exploding a
exploding or exploiting? Or exposing? :)
Quoted 7 lines
Quoted 4 lines
dangling pointer to overwrite sensitive data using the sensitive PTE.
(2) Avoiding TLB shootdowns: the PTEs do not need to be flushed in remote page-tables.
Those belong in the code comments below, explaining what it is going to be used for.
I will add it to the code as well.
Quoted 31 lines
Quoted 31 lines
To do so a temporary
mm_structcan be used. Mappings which are private for this mm can be set in the userspace part of the address-space. During the whole time in which the temporary mm is loaded, interrupts must be disabled.The first use-case for temporary PTEs, which will follow, is for poking the kernel text.
[ Commit message was written by Nadav ]
Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Signed-off-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
arch/x86/include/asm/mmu_context.h| 32 ++++++++++++++++++++++++++++++1 file changed, 32 insertions(+)arch/x86/include/asm/mmu_context.h +0
-0@@ -356,4 +356,36 @@ static inline unsigned long __get_current_cr3_fast(void)return cr3; }
Quoted 3 lines
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 359
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+typedef struct {Why does it have to be a typedef?
Having a different struct can prevent the misuse of using mm_structs in unuse_temporary_mm() that were not “used” using use_temporary_mm. The typedef, I presume, can deter users from starting to play with the internal “private” fields.
Quoted 5 lines
That prev.prev below looks unnecessary, instead of just using prev.
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 360
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+struct mm_struct *prev;Why "prev”?
This is obviously the previous active mm. Feel free to suggest an alternative name.
Quoted 4 lines
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 361
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+} temporary_mm_state_t;That's kinda long - it is longer than the function name below.
temp_mm_state_tnot enough?
I will change it.
Quoted 14 lines
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 371
Quoted 11 lines
arch/x86/include/asm/mmu_context.h +10
-0+ +/* +* Using a temporary mm allows to set temporary mappings that are not accessible +* by other cores. Such mappings are needed to perform sensitive memory writes +* that override the kernel memory protections (e.g., W^X), without exposing the +* temporary page-table mappings that are required for these write operations to +* other cores. +* +* Context: The temporary mm needs to be used exclusively by a single core. To +* harden security IRQs must be disabled while the temporary mm is ^ ,
Quoted 3 lines
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 372
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+* loaded, thereby preventing interrupt handler bugs from override the s/override/overriding/
I will fix all of these typos, comment. Thank you.
Meta-question: could you please review the entire patch-set? This is actually v9 of this particular patch - it was part of a separate patch-set before. I don’t think that the patch has changed since (the real) v1.
These sporadic comments after each version really makes it hard to get this work completed.
Borislav Petkov
maintainerRe: [PATCH v2 03/20] x86/mm: temporary mm struct
Quoted 6 lines · Nadav Amit
Meta-question: could you please review the entire patch-set? This is actually v9 of this particular patch - it was part of a separate patch-set before. I don’t think that the patch has changed since (the real) v1.
These sporadic comments after each version really makes it hard to get this work completed.
Sorry but where I am the day has only 24 hours and this patchset is not the only one in my overflowing mbox. If my sporadic comments are making it hard to finish your work, I better not interfere then.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 03/20] x86/mm: temporary mm struct
Quoted 7 lines · Borislav Petkov
Quoted 4 lines · Nadav Amit
Meta-question: could you please review the entire patch-set? This is actually v9 of this particular patch - it was part of a separate patch-set before. I don’t think that the patch has changed since (the real) v1.
Quoted 6 lines
Quoted 2 lines
These sporadic comments after each version really makes it hard to get this work completed.
Sorry but where I am the day has only 24 hours and this patchset is not the only one in my overflowing mbox. If my sporadic comments are making it hard to finish your work, I better not interfere then.
I certainly did not intend for it to sound this way, and your feedback is obviously valuable.
Just let me know when you are done reviewing the patch-set, so I will not overflow your mailbox with even unnecessary versions of these patches. :)
Borislav Petkov
maintainerRe: [PATCH v2 03/20] x86/mm: temporary mm struct
In reply to Nadav AmitQuoted 4 lines · Nadav Amit
Having a different struct can prevent the misuse of using
mm_structsinunuse_temporary_mm()that were not “used” usinguse_temporary_mm. The typedef, I presume, can deter users from starting to play with the internal “private” fields.
Ok, makes sense.
Quoted 8 lines
Quoted 5 lines
That prev.prev below looks unnecessary, instead of just using prev.
arch/x86/include/asm/mmu_context.h static inline unsigned long __get_current_cr3_fast(void) · line 360
Quoted 1 line
arch/x86/include/asm/mmu_context.h +1
-0+struct mm_struct *prev;Why "prev”?
This is obviously the previous active mm. Feel free to suggest an alternative name.
Well, when I look at the typedef I'm wondering why is it called "prev" but I guess this is to mean that it will be saving the previously used mm, so ack.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.PATCH v2 4/20 fork: provide a function for copying init_mm
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
Provide a function for copying init_mm. This function will be later used for setting a temporary mm.
--- include/linux/sched/task.h | 1 + kernel/fork.c | 24 ++++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-)
include/linux/sched/task.h +1 -0
@@ -76,6 +76,7 @@ extern void exit_itimers(struct signal_struct *);extern long _do_fork (unsigned long ,unsigned long ,unsigned long ,int __user *,int __user *,unsigned long );extern long do_fork (unsigned long ,unsigned long ,unsigned long ,int __user *,int __user *);struct task_struct *fork_idle (int ); +struct mm_struct *copy_init_mm (void );extern pid_t kernel_thread (int (*fn)(void *),void *arg,unsigned long flags);extern long kernel_wait4 (pid_t ,int __user *,int ,struct rusage *);
kernel/fork.c +18 -6
@@ -1299,13 +1299,20 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm)complete_vfork_done (tsk); } -/* -* Allocate a new mm structure and copy contents from the -* mm structure of the passed in task structure. +/** +* dup_mm() - duplicates an existing mm structure +* @tsk: the task_struct with which the new mm will be associated. +* @oldmm: the mm to duplicate. +* +* Allocates a new mm structure and copy contents from the provided +* @oldmm structure. +* +* Return: the duplicated mm or NULL on failure. */ -static struct mm_struct *dup_mm (struct task_struct *tsk) +static struct mm_struct *dup_mm (struct task_struct *tsk, +struct mm_struct *oldmm) { -struct mm_struct *mm, *oldmm = current->mm; +struct mm_struct *mm;int err; mm =allocate_mm ();
@@ -1372,7 +1379,7 @@ static int copy_mm(unsigned long clone_flags, struct task_struct *tsk) } retval = -ENOMEM ; - mm =dup_mm (tsk); + mm =dup_mm (tsk, current->mm);if (!mm) goto fail_nomem;
@@ -2187,6 +2194,11 @@ struct task_struct *fork_idle(int cpu)return task; } +struct mm_struct *copy_init_mm (void ) +{ +return dup_mm (NULL , &init_mm); +} +/* * Ok, this is the main fork-routine. *
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 04/20] fork: provide a function for copying init_mm
Quoted 37 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
Provide a function for copying
init_mm. This function will be later used for setting a temporary mm.Cc Andy Lutomirski <[email protected]>Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- include/linux/sched/task.h | 1 + kernel/fork.c | 24 ++++++++++++++++++------ 2 files changed, 19 insertions(+), 6 deletions(-)include/linux/sched/task.h +1
-0@@ -76,6 +76,7 @@ extern void exit_itimers(struct signal_struct *);extern long _do_fork (unsigned long ,unsigned long ,unsigned long ,int __user *,int __user *,unsigned long );extern long do_fork (unsigned long ,unsigned long ,unsigned long ,int __user *,int __user *);struct task_struct *fork_idle (int ); +struct mm_struct *copy_init_mm (void );extern pid_t kernel_thread (int (*fn)(void *),void *arg,unsigned long flags);extern long kernel_wait4 (pid_t ,int __user *,int ,struct rusage *);kernel/fork.c +0
-0@@ -1299,13 +1299,20 @@ void mm_release(struct task_struct *tsk, struct mm_struct *mm)complete_vfork_done (tsk); }
kernel/fork.c void mm_release(struct task_struct *tsk, struct mm_struct *mm) · line 1307
Quoted 10 lines
kernel/fork.c +6
-3-/* -* Allocate a new mm structure and copy contents from the -* mm structure of the passed in task structure. +/** +* dup_mm() - duplicates an existing mm structure +* @tsk: the task_struct with which the new mm will be associated. +* @oldmm: the mm to duplicate. +* +* Allocates a new mm structure and copy contents from the provided
s/copy/copies/
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Nadav Amit
AppliedRe: [PATCH v2 04/20] fork: provide a function for copying init_mm
Quoted 14 lines · Borislav Petkov
Quoted 10 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
- * Allocate a new mm structure and copy contents from the
- * mm structure of the passed in task structure.
+/**
+ *dup_mm()- duplicates an existing mm structure
+ * @tsk: thetask_structwith which the new mm will be associated.
+ * @oldmm: the mm to duplicate.
+ *
+ * Allocates a new mm structure and copy contents from the provided
Quoted 1 line
s/copy/copies/
Thanks, applied (I revised this sentence a bit).
PATCH v2 5/20 x86/alternative: initializing temporary mm for patching
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init mm.
The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory.
Finally, we randomize the address of the PTEs to harden against exploits that use these PTEs.
--- arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+)
arch/x86/include/asm/pgtable.h +3 -0
@@ -1021,6 +1021,9 @@ static inline void __meminit init_trampoline_default(void)/* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index (__PAGE_OFFSET)]; } + +void __init poking_init (void ); +# ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline (void );# else
arch/x86/include/asm/text-patching.h +2 -0
@@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr;#endif /* _ASM_X86_TEXT_PATCHING_H */
arch/x86/kernel/alternative.c +3 -0
@@ -678,6 +678,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; +static void *__text_poke (void *addr,const void *opcode,size_t len) {unsigned long flags;
arch/x86/mm/init_64.c +36 -0
@@ -53,6 +53,7 @@#include <asm/init.h> #include <asm/uv/uv.h> #include <asm/setup.h> +#include <asm/text-patching.h> #include "mm_internal.h"
@@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void)return memory_block_size_probed; } +/* +* Initialize an mm_struct to be used during poking and a pointer to be used +* during patching. +*/ +void __init poking_init (void ) +{ +spinlock_t *ptl; +pte_t *ptep; + + poking_mm =copy_init_mm (); +BUG_ON (!poking_mm); + +/* +* Randomize the poking address, but make sure that the following page +* will be mapped at the same PMD. We need 2 pages, so find space for 3, +* and adjust the address if the PMD ends after the first one. +*/ + poking_addr =TASK_UNMAPPED_BASE ; +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE )) + poking_addr += (kaslr_get_random_long ("Poking" ) &PAGE_MASK ) % + (TASK_SIZE -TASK_UNMAPPED_BASE -3 *PAGE_SIZE ); + +if (((poking_addr +PAGE_SIZE ) & ~PMD_MASK ) ==0 ) + poking_addr +=PAGE_SIZE ; + +/* +* We need to trigger the allocation of the page-tables that will be +* needed for poking now. Later, poking may be performed in an atomic +* section, which might cause allocation to fail. +*/ + ptep =get_locked_pte (poking_mm, poking_addr, &ptl); +BUG_ON (!ptep); +pte_unmap_unlock (ptep, ptl); +} +#ifdef CONFIG_SPARSEMEM_VMEMMAP /* * Initialise the sparsemem vmemmap using huge-pages at the PMD level.
init/main.c +3 -0
@@ -496,6 +496,8 @@ void __init __weak thread_stack_cache_init(void)void __init __weak mem_encrypt_init (void ) { } +void __init __weak poking_init (void ) { } +bool initcall_debug;core_param (initcall_debug, initcall_debug, bool,0644 );
@@ -730,6 +732,7 @@ asmlinkage __visible void __init start_kernel(void)taskstats_init_early ();delayacct_init (); +poking_init ();check_bugs ();acpi_subsystem_init ();
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Just nitpicks:
Subject: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
s/initailizing/Initialize/
Quoted 4 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init
Please remove the "we" from commit messages and use impartial, passive formulations.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
In reply to Rick EdgecombeQuoted 102 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init mm.
The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory.
Finally, we randomize the address of the PTEs to harden against exploits that use these PTEs.
Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+)arch/x86/include/asm/pgtable.h +3
-0@@ -1021,6 +1021,9 @@ static inline void __meminit init_trampoline_default(void)/* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index (__PAGE_OFFSET)]; } + +void __init poking_init (void ); +# ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline (void );# else arch/x86/include/asm/text-patching.h +2
-0@@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr;#endif /* _ASM_X86_TEXT_PATCHING_H */ arch/x86/kernel/alternative.c +3
-0@@ -678,6 +678,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; +static void *__text_poke (void *addr,const void *opcode,size_t len) {unsigned long flags;arch/x86/mm/init_64.c +15
-0@@ -53,6 +53,7 @@#include <asm/init.h> #include <asm/uv/uv.h> #include <asm/setup.h> +#include <asm/text-patching.h> #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void)return memory_block_size_probed; } +/* +* Initialize an mm_struct to be used during poking and a pointer to be used +* during patching. +*/ +void __init poking_init (void ) +{ +spinlock_t *ptl; +pte_t *ptep; + + poking_mm =copy_init_mm (); +BUG_ON (!poking_mm); + +/* +* Randomize the poking address, but make sure that the following page
arch/x86/mm/init_64.c unsigned long memory_block_size_bytes(void) · line 1410
Quoted 10 lines
arch/x86/mm/init_64.c +10
-0+* will be mapped at the same PMD. We need 2 pages, so find space for 3, +* and adjust the address if the PMD ends after the first one. +*/ + poking_addr =TASK_UNMAPPED_BASE ; +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE )) + poking_addr += (kaslr_get_random_long ("Poking" ) &PAGE_MASK ) % + (TASK_SIZE -TASK_UNMAPPED_BASE -3 *PAGE_SIZE ); + +if (((poking_addr +PAGE_SIZE ) & ~PMD_MASK ) ==0 ) + poking_addr +=PAGE_SIZE ;
Further thinking about it, I think that allocating the virtual address for poking from user address-range is problematic. The user can set watchpoints on different addresses, cause some static-keys to be enabled/disabled, and monitor the signals to derandomize the poking address.
Andy, I think you were pushing this change. Can I go back to use a vmalloc’d address instead, or do you have a better solution? I prefer not to save/restore DR7, of course.
Re: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Quoted 109 lines · Nadav Amit
Quoted 109 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init mm.
The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory.
Finally, we randomize the address of the PTEs to harden against exploits that use these PTEs.
Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+)arch/x86/include/asm/pgtable.h +3
-0@@ -1021,6 +1021,9 @@ static inline void __meminit init_trampoline_default(void)/* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index (__PAGE_OFFSET)]; } + +void __init poking_init (void ); +# ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline (void );# else arch/x86/include/asm/text-patching.h +2
-0@@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr;#endif /* _ASM_X86_TEXT_PATCHING_H */ arch/x86/kernel/alternative.c +3
-0@@ -678,6 +678,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; +static void *__text_poke (void *addr,const void *opcode,size_t len) {unsigned long flags;arch/x86/mm/init_64.c +22
-0@@ -53,6 +53,7 @@#include <asm/init.h> #include <asm/uv/uv.h> #include <asm/setup.h> +#include <asm/text-patching.h> #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void)return memory_block_size_probed; } +/* +* Initialize an mm_struct to be used during poking and a pointer to be used +* during patching. +*/ +void __init poking_init (void ) +{ +spinlock_t *ptl; +pte_t *ptep; + + poking_mm =copy_init_mm (); +BUG_ON (!poking_mm); + +/* +* Randomize the poking address, but make sure that the following page +* will be mapped at the same PMD. We need 2 pages, so find space for 3, +* and adjust the address if the PMD ends after the first one. +*/ + poking_addr =TASK_UNMAPPED_BASE ; +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE )) + poking_addr += (kaslr_get_random_long ("Poking" ) &PAGE_MASK ) % + (TASK_SIZE -TASK_UNMAPPED_BASE -3 *PAGE_SIZE );
Quoted 9 lines
arch/x86/mm/init_64.c unsigned long memory_block_size_bytes(void) · line 1410
Quoted 3 lines
arch/x86/mm/init_64.c +3
-0+ +if (((poking_addr +PAGE_SIZE ) & ~PMD_MASK ) ==0 ) + poking_addr +=PAGE_SIZE ;Further thinking about it, I think that allocating the virtual address for poking from user address-range is problematic. The user can set watchpoints on different addresses, cause some static-keys to be enabled/disabled, and monitor the signals to derandomize the poking address.
Hmm, I hadn’t thought about watchpoints. I’m not sure how much we care about possible derandomization like this, but we certainly don’t want to send signals or otherwise malfunction.
Quoted 2 lines
Andy, I think you were pushing this change. Can I go back to use a vmalloc’d address instead, or do you have a better solution?
Hmm. If we use a vmalloc address, we have to make sure it’s not actually allocated. I suppose we could allocate one once at boot and use that. We also have the problem that the usual APIs for handling “user” addresses might assume they’re actually in the user range, although this seems unlikely to be a problem in practice. More seriously, though, the code that manipulates per-mm paging structures assumes that *all* of the structures up to the top level are per-mm, and, if we use anything less than a private pgd, this isn’t the case.
Quoted 3 lines
I prefer not to
save/restore DR7, of course.
I suspect we may want to use the temporary mm concept for EFI, too, so we may want to just suck it up and save/restore DR7. But only if a watchpoint is in use, of course. I have an old patch I could dust off that tracks DR7 to make things like this efficient.
Re: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Quoted 131 lines · Andy Lutomirski
Quoted 109 lines · Nadav Amit
Quoted 109 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init mm.
The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory.
Finally, we randomize the address of the PTEs to harden against exploits that use these PTEs.
Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+)arch/x86/include/asm/pgtable.h +3
-0@@ -1021,6 +1021,9 @@ static inline void __meminit init_trampoline_default(void)/* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index (__PAGE_OFFSET)]; } + +void __init poking_init (void ); +# ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline (void );# else arch/x86/include/asm/text-patching.h +2
-0@@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr;#endif /* _ASM_X86_TEXT_PATCHING_H */ arch/x86/kernel/alternative.c +3
-0@@ -678,6 +678,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; +static void *__text_poke (void *addr,const void *opcode,size_t len) {unsigned long flags;arch/x86/mm/init_64.c +22
-0@@ -53,6 +53,7 @@#include <asm/init.h> #include <asm/uv/uv.h> #include <asm/setup.h> +#include <asm/text-patching.h> #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void)return memory_block_size_probed; } +/* +* Initialize an mm_struct to be used during poking and a pointer to be used +* during patching. +*/ +void __init poking_init (void ) +{ +spinlock_t *ptl; +pte_t *ptep; + + poking_mm =copy_init_mm (); +BUG_ON (!poking_mm); + +/* +* Randomize the poking address, but make sure that the following page +* will be mapped at the same PMD. We need 2 pages, so find space for 3, +* and adjust the address if the PMD ends after the first one. +*/ + poking_addr =TASK_UNMAPPED_BASE ; +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE )) + poking_addr += (kaslr_get_random_long ("Poking" ) &PAGE_MASK ) % + (TASK_SIZE -TASK_UNMAPPED_BASE -3 *PAGE_SIZE );Quoted 8 lines
arch/x86/mm/init_64.c unsigned long memory_block_size_bytes(void) · line 1410
Quoted 3 lines
arch/x86/mm/init_64.c +3
-0+ +if (((poking_addr +PAGE_SIZE ) & ~PMD_MASK ) ==0 ) + poking_addr +=PAGE_SIZE ;Further thinking about it, I think that allocating the virtual address for poking from user address-range is problematic. The user can set watchpoints on different addresses, cause some static-keys to be enabled/disabled, and monitor the signals to derandomize the poking address.
Hmm, I hadn’t thought about watchpoints. I’m not sure how much we care about possible derandomization like this, but we certainly don’t want to send signals or otherwise malfunction.
Quoted 2 lines
Andy, I think you were pushing this change. Can I go back to use a vmalloc’d address instead, or do you have a better solution?
Quoted 8 lines
Hmm. If we use a vmalloc address, we have to make sure it’s not actually allocated. I suppose we could allocate one once at boot and use that. We also have the problem that the usual APIs for handling “user” addresses might assume they’re actually in the user range, although this seems unlikely to be a problem in practice. More seriously, though, the code that manipulates per-mm paging structures assumes that *all* of the structures up to the top level are per-mm, and, if we use anything less than a private pgd, this isn’t the case.
I forgot that I only had this conversation in my mind ;-)
Well, I did write some code that kept some vmalloc’d area private, and it did require more synchronization between the pgd’s. It is still possible to use another top-level PGD, but … (continued below)
Quoted 8 lines
Quoted 3 lines
I prefer not to
save/restore DR7, of course.I suspect we may want to use the temporary mm concept for EFI, too, so we may want to just suck it up and save/restore DR7. But only if a watchpoint is in use, of course. I have an old patch I could dust off that tracks DR7 to make things like this efficient.
… but, if this is the case, then I will just make (un)use_temporary_mm() to save/restore DR7. I guess you are ok with such a solution. I will incorporate it into Rick’s v3.
Andy Lutomirski
maintainerRe: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Quoted 151 lines · Nadav Amit
Quoted 132 lines · Andy Lutomirski
Quoted 109 lines · Nadav Amit
Quoted 109 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
To prevent improper use of the PTEs that are used for text patching, we want to use a temporary mm struct. We initailize it by copying the init mm.
The address that will be used for patching is taken from the lower area that is usually used for the task memory. Doing so prevents the need to frequently synchronize the temporary-mm (e.g., when BPF programs are installed), since different PGDs are used for the task memory.
Finally, we randomize the address of the PTEs to harden against exploits that use these PTEs.
Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Reviewed-by Masami Hiramatsu <[email protected]>Tested-by Masami Hiramatsu <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>---
arch/x86/include/asm/pgtable.h | 3 +++ arch/x86/include/asm/text-patching.h | 2 ++ arch/x86/kernel/alternative.c | 3 +++ arch/x86/mm/init_64.c | 36 ++++++++++++++++++++++++++++ init/main.c | 3 +++ 5 files changed, 47 insertions(+)arch/x86/include/asm/pgtable.h +3
-0@@ -1021,6 +1021,9 @@ static inline void __meminit init_trampoline_default(void)/* Default trampoline pgd value */ trampoline_pgd_entry = init_top_pgt[pgd_index (__PAGE_OFFSET)]; } + +void __init poking_init (void ); +# ifdef CONFIG_RANDOMIZE_MEMORY void __meminit init_trampoline (void );# else arch/x86/include/asm/text-patching.h +2
-0@@ -39,5 +39,7 @@ extern void *text_poke_kgdb(void *addr, const void *opcode, size_t len);extern int poke_int3_handler (struct pt_regs *regs);extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem; +extern __ro_after_init struct mm_struct *poking_mm; +extern __ro_after_init unsigned long poking_addr;#endif /* _ASM_X86_TEXT_PATCHING_H */ arch/x86/kernel/alternative.c +3
-0@@ -678,6 +678,9 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,return addr; } +__ro_after_init struct mm_struct *poking_mm; +__ro_after_init unsigned long poking_addr; +static void *__text_poke (void *addr,const void *opcode,size_t len) {unsigned long flags;arch/x86/mm/init_64.c +22
-0@@ -53,6 +53,7 @@#include <asm/init.h> #include <asm/uv/uv.h> #include <asm/setup.h> +#include <asm/text-patching.h> #include "mm_internal.h" @@ -1383,6 +1384,41 @@ unsigned long memory_block_size_bytes(void)return memory_block_size_probed; } +/* +* Initialize an mm_struct to be used during poking and a pointer to be used +* during patching. +*/ +void __init poking_init (void ) +{ +spinlock_t *ptl; +pte_t *ptep; + + poking_mm =copy_init_mm (); +BUG_ON (!poking_mm); + +/* +* Randomize the poking address, but make sure that the following page +* will be mapped at the same PMD. We need 2 pages, so find space for 3, +* and adjust the address if the PMD ends after the first one. +*/ + poking_addr =TASK_UNMAPPED_BASE ; +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE )) + poking_addr += (kaslr_get_random_long ("Poking" ) &PAGE_MASK ) % + (TASK_SIZE -TASK_UNMAPPED_BASE -3 *PAGE_SIZE );Quoted 8 lines
arch/x86/mm/init_64.c unsigned long memory_block_size_bytes(void) · line 1410
Quoted 3 lines
arch/x86/mm/init_64.c +3
-0+ +if (((poking_addr +PAGE_SIZE ) & ~PMD_MASK ) ==0 ) + poking_addr +=PAGE_SIZE ;Further thinking about it, I think that allocating the virtual address for poking from user address-range is problematic. The user can set watchpoints on different addresses, cause some static-keys to be enabled/disabled, and monitor the signals to derandomize the poking address.
Hmm, I hadn’t thought about watchpoints. I’m not sure how much we care about possible derandomization like this, but we certainly don’t want to send signals or otherwise malfunction.
Quoted 2 lines
Andy, I think you were pushing this change. Can I go back to use a vmalloc’d address instead, or do you have a better solution?
Quoted 8 lines
Hmm. If we use a vmalloc address, we have to make sure it’s not actually allocated. I suppose we could allocate one once at boot and use that. We also have the problem that the usual APIs for handling “user” addresses might assume they’re actually in the user range, although this seems unlikely to be a problem in practice. More seriously, though, the code that manipulates per-mm paging structures assumes that *all* of the structures up to the top level are per-mm, and, if we use anything less than a private pgd, this isn’t the case.
I forgot that I only had this conversation in my mind ;-)
Well, I did write some code that kept some vmalloc’d area private, and it did require more synchronization between the pgd’s. It is still possible to use another top-level PGD, but … (continued below)
Quoted 4 lines
Quoted 3 lines
I prefer not to
save/restore DR7, of course.
Quoted 9 lines
Quoted 4 lines
I suspect we may want to use the temporary mm concept for EFI, too, so we may want to just suck it up and save/restore DR7. But only if a watchpoint is in use, of course. I have an old patch I could dust off that tracks DR7 to make things like this efficient.
… but, if this is the case, then I will just make (un)use_temporary_mm() to save/restore DR7. I guess you are ok with such a solution. I will incorporate it into Rick’s v3.
I'm certainly amenable to other solutions, but this one does seem the least messy. I looked at my old patch, and it doesn't do what you want. I'd suggest you just add a percpu variable like cpu_dr7 and rig up some accessors so that it stays up to date. Then you can skip the dr7 writes if there are no watchpoints set.
Also, EFI is probably a less interesting example than rare_write. With rare_write, especially the dynamically allocated variants that people keep coming up with, we'll need a swath of address space fully as large as the vmalloc area. and getting *that* right while still using the kernel address range might be more of a mess than we really want to deal with.
--Andy
Re: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Quoted 8 lines · Andy Lutomirski
I'm certainly amenable to other solutions, but this one does seem the least messy. I looked at my old patch, and it doesn't do what you want. I'd suggest you just add a percpu variable like
cpu_dr7and rig up some accessors so that it stays up to date. Then you can skip the dr7 writes if there are no watchpoints set.
Quoted 6 lines
Also, EFI is probably a less interesting example than
rare_write. Withrare_write, especially the dynamically allocated variants that people keep coming up with, we'll need a swath of address space fully as large as the vmalloc area. and getting *that* right while still using the kernel address range might be more of a mess than we really want to deal with.
As long as you feel comfortable with this solution, I’m fine with it.
Here is what I have (untested). I prefer to save/restore all the DRs, because IIRC DR6 indications are updated even if breakpoints are disabled (in DR7). And anyhow, that is the standard interface.
-- >8 --
From: Nadav Amit <[email protected]>
Date: Mon, 11 Feb 2019 03:07:08 -0800
Subject: [PATCH] mm: save DRs when loading temporary mm
--- arch/x86/include/asm/mmu_context.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)
arch/x86/include/asm/mmu_context.h +18 -0
@@ -13,6 +13,7 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/mpx.h> +#include <asm/debugreg.h> extern atomic64_t last_mm_ctx_id;
@@ -358,6 +359,7 @@ static inline unsigned long __get_current_cr3_fast(void)typedef struct {struct mm_struct *prev; +unsigned short bp_enabled :1 ; }temp_mm_state_t ;/*
@@ -380,6 +382,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)lockdep_assert_irqs_disabled (); state.prev =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current); + +/* +* If breakpoints are enabled, disable them while the temporary mm is +* used - they do not belong and might cause wrong signals or crashes. +*/ + state.bp_enabled =hw_breakpoint_active (); +if (state.bp_enabled) +hw_breakpoint_disable (); +return state; }
@@ -387,6 +398,13 @@ static inline void unuse_temporary_mm(temp_mm_state_t prev) {lockdep_assert_irqs_disabled ();switch_mm_irqs_off (NULL , prev.prev, current); + +/* +* Restore the breakpoints if they were disabled before the temporary mm +* was loaded. +*/ +if (prev.bp_enabled) +hw_breakpoint_restore (); }#endif /* _ASM_X86_MMU_CONTEXT_H */
Signature · 1 line
2.17.1
Andy Lutomirski
maintainerRe: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Quoted 16 lines · Nadav Amit
Quoted 9 lines · Andy Lutomirski
I'm certainly amenable to other solutions, but this one does seem the least messy. I looked at my old patch, and it doesn't do what you want. I'd suggest you just add a percpu variable like
cpu_dr7and rig up some accessors so that it stays up to date. Then you can skip the dr7 writes if there are no watchpoints set.Quoted 6 lines
Also, EFI is probably a less interesting example than
rare_write. Withrare_write, especially the dynamically allocated variants that people keep coming up with, we'll need a swath of address space fully as large as the vmalloc area. and getting *that* right while still using the kernel address range might be more of a mess than we really want to deal with.
Quoted 5 lines
As long as you feel comfortable with this solution, I’m fine with it.
Here is what I have (untested). I prefer to save/restore all the DRs, because IIRC DR6 indications are updated even if breakpoints are disabled (in DR7). And anyhow, that is the standard interface.
Seems reasonable, but:
Quoted 34 lines
-- >8 --
From: Nadav Amit <[email protected]>
Date: Mon, 11 Feb 2019 03:07:08 -0800
Subject: [PATCH] mm: save DRs when loading temporary mmSigned-off-by Nadav Amit <[email protected]>--- arch/x86/include/asm/mmu_context.h | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+)arch/x86/include/asm/mmu_context.h +2
-0@@ -13,6 +13,7 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/mpx.h> +#include <asm/debugreg.h> extern atomic64_t last_mm_ctx_id;@@ -358,6 +359,7 @@ static inline unsigned long __get_current_cr3_fast(void)typedef struct {struct mm_struct *prev; +unsigned short bp_enabled :1 ; }temp_mm_state_t ;/*
arch/x86/include/asm/mmu_context.h static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm) · line 389
Quoted 8 lines
arch/x86/include/asm/mmu_context.h +5
-0@@ -380,6 +382,15 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)lockdep_assert_irqs_disabled (); state.prev =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current); + +/* +* If breakpoints are enabled, disable them while the temporary mm is +* used - they do not belong and might cause wrong signals or crashes. +*/
Maybe clarify this? Add some mention that the specific problem is that user code could set a watchpoint on an address that is also used in the temporary mm.
Arguably we should not disable *kernel* breakpoints a la perf, but that seems like quite a minor issue, at least as long as use_temporary_mm() doesn't get wider use. But a comment that this also disables perf breakpoints and that this could be undesirable might be in order as well.
Re: [PATCH v2 05/20] x86/alternative: initializing temporary mm for patching
Quoted 14 lines · Andy Lutomirski
Review
Quoted 6 lines · Nadav Amit
(quoted diff) +5
-0+ + /* + * If breakpoints are enabled, disable them while the temporary mm is + * used - they do not belong and might cause wrong signals or crashes. + */Maybe clarify this? Add some mention that the specific problem is that user code could set a watchpoint on an address that is also used in the temporary mm.
Quoted 5 lines
Arguably we should not disable *kernel* breakpoints a la perf, but that seems like quite a minor issue, at least as long as
use_temporary_mm()doesn't get wider use. But a comment that this also disables perf breakpoints and that this could be undesirable might be in order as well.
I think that in the future there may also be security benefits for disabling breakpoints when you are in a sensitive code-block, for instance when you poke text, to prevent the control flow from being hijacked (by exploiting a bug in the debug exception handler). Some additional steps need to be taken for that to be beneficial so I leave it out of the comment for now.
Anyhow, how about this:
-- >8 --
From: Nadav Amit <[email protected]>
Date: Mon, 11 Feb 2019 03:07:08 -0800
Subject: [PATCH] x86/mm: Save DRs when loading a temporary mm
Prevent user watchpoints from mistakenly firing while the temporary mm is being used. As the addresses that of the temporary mm might overlap those of the user-process, this is necessary to prevent wrong signals or worse things from happening.
--- arch/x86/include/asm/mmu_context.h | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
arch/x86/include/asm/mmu_context.h +25 -0
@@ -13,6 +13,7 @@#include <asm/tlbflush.h> #include <asm/paravirt.h> #include <asm/mpx.h> +#include <asm/debugreg.h> extern atomic64_t last_mm_ctx_id;
@@ -358,6 +359,7 @@ static inline unsigned long __get_current_cr3_fast(void)typedef struct {struct mm_struct *prev; +unsigned short bp_enabled :1 ; }temp_mm_state_t ;/*
@@ -380,6 +382,22 @@ static inline temp_mm_state_t use_temporary_mm(struct mm_struct *mm)lockdep_assert_irqs_disabled (); state.prev =this_cpu_read (cpu_tlbstate.loaded_mm);switch_mm_irqs_off (NULL , mm, current); + +/* +* If breakpoints are enabled, disable them while the temporary mm is +* used. Userspace might set up watchpoints on addresses that are used +* in the temporary mm, which would lead to wrong signals being sent or +* crashes. +* +* Note that breakpoints are not disabled selectively, which also causes +* kernel breakpoints (e.g., perf's) to be disabled. This might be +* undesirable, but still seems reasonable as the code that runs in the +* temporary mm should be short. +*/ + state.bp_enabled =hw_breakpoint_active (); +if (state.bp_enabled) +hw_breakpoint_disable (); +return state; }
@@ -387,6 +405,13 @@ static inline void unuse_temporary_mm(temp_mm_state_t prev) {lockdep_assert_irqs_disabled ();switch_mm_irqs_off (NULL , prev.prev, current); + +/* +* Restore the breakpoints if they were disabled before the temporary mm +* was loaded. +*/ +if (prev.bp_enabled) +hw_breakpoint_restore (); }#endif /* _ASM_X86_MMU_CONTEXT_H */
Signature · 1 line
2.17.1
PATCH v2 6/20 x86/alternative: use temporary mm for text poking
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
text_poke() can potentially compromise the security as it sets temporary PTEs in the fixmap. These PTEs might be used to rewrite the kernel code from other cores accidentally or maliciously, if an attacker gains the ability to write onto kernel memory.
Moreover, since remote TLBs are not flushed after the temporary PTEs are removed, the time-window in which the code is writable is not limited if the fixmap PTEs - maliciously or accidentally - are cached in the TLB. To address these potential security hazards, we use a temporary mm for patching the code.
Finally, text_poke() is also not conservative enough when mapping pages, as it always tries to map 2 pages, even when a single one is sufficient. So try to be more conservative, and do not map more than needed.
--- arch/x86/include/asm/fixmap.h | 2 - arch/x86/kernel/alternative.c | 106 +++++++++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 2 - 3 files changed, 84 insertions(+), 26 deletions(-)
arch/x86/include/asm/fixmap.h +0 -2
@@ -103,8 +103,6 @@ enum fixed_addresses {#ifdef CONFIG_PARAVIRT FIX_PARAVIRT_BOOTMAP,#endif -FIX_TEXT_POKE1 ,/* reserve 2 pages for text_poke() */ -FIX_TEXT_POKE0 ,/* first page is last, because allocation is backward */ #ifdef CONFIG_X86_INTEL_MID FIX_LNW_VRTC,#endif
arch/x86/kernel/alternative.c +84 -22
131 lines not on this page · raw message
arch/x86/xen/mmu_pv.c +0 -2
@@ -2319,8 +2319,6 @@ static void xen_set_fixmap(unsigned idx, phys_addr_t phys, pgprot_t prot)#elif defined(CONFIG_X86_VSYSCALL_EMULATION )case VSYSCALL_PAGE :#endif -case FIX_TEXT_POKE0 : -case FIX_TEXT_POKE1 :/* All local page mappings */ pte =pfn_pte (phys, prot);break ;
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 06/20] x86/alternative: use temporary mm for text poking
Quoted 3 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
text_poke()can potentially compromise the security as it sets temporary
s/the //
Quoted 3 lines
PTEs in the fixmap. These PTEs might be used to rewrite the kernel code from other cores accidentally or maliciously, if an attacker gains the ability to write onto kernel memory.
Eww, sneaky. That would be a really nasty attack.
Quoted 28 lines
Moreover, since remote TLBs are not flushed after the temporary PTEs are removed, the time-window in which the code is writable is not limited if the fixmap PTEs - maliciously or accidentally - are cached in the TLB. To address these potential security hazards, we use a temporary mm for patching the code.
Finally,
text_poke()is also not conservative enough when mapping pages, as it always tries to map 2 pages, even when a single one is sufficient. So try to be more conservative, and do not map more than needed.Cc Andy Lutomirski <[email protected]>Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Cc Masami Hiramatsu <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- arch/x86/include/asm/fixmap.h | 2 - arch/x86/kernel/alternative.c | 106 +++++++++++++++++++++++++++------- arch/x86/xen/mmu_pv.c | 2 - 3 files changed, 84 insertions(+), 26 deletions(-)arch/x86/include/asm/fixmap.h +0
-0
arch/x86/include/asm/fixmap.h enum fixed_addresses { · line 107
Quoted 5 lines
arch/x86/include/asm/fixmap.h +0
-2@@ -103,8 +103,6 @@ enum fixed_addresses {#ifdef CONFIG_PARAVIRT FIX_PARAVIRT_BOOTMAP,#endif -FIX_TEXT_POKE1 ,/* reserve 2 pages for text_poke() */ - FIX_TEXT_POKE0, /* first page is last, because allocation is backward */
Two fixmap slots less - good riddance. :)
Quoted 42 lines
arch/x86/include/asm/fixmap.h +0
-0#ifdef CONFIG_X86_INTEL_MID FIX_LNW_VRTC,#endif arch/x86/kernel/alternative.c +10
-6@@ -11,6 +11,7 @@#include <linux/stop_machine.h> #include <linux/slab.h> #include <linux/kdebug.h> +#include <linux/mmu_context.h> #include <asm/text-patching.h> #include <asm/alternative.h> #include <asm/sections.h> @@ -683,41 +684,102 @@ __ro_after_init unsigned long poking_addr;static void *__text_poke (void *addr,const void *opcode,size_t len) { +bool cross_page_boundary =offset_in_page (addr) + len >PAGE_SIZE ; +temporary_mm_state_t prev; +struct page *pages[2 ] = {NULL };unsigned long flags; -char *vaddr; -struct page *pages[2 ]; -int i; +pte_t pte, *ptep; +spinlock_t *ptl; +pgprot_t prot;/* -* While boot memory allocator is runnig we cannot use struct -* pages as they are not yet initialized. +* While boot memory allocator is running we cannot use struct pages as +* they are not yet initialized. */ BUG_ON (!after_bootmem);if (!core_kernel_text ((unsigned long )addr)) { pages[0 ] =vmalloc_to_page (addr); - pages[1 ] =vmalloc_to_page (addr +PAGE_SIZE ); +if (cross_page_boundary)
arch/x86/kernel/alternative.c __ro_after_init unsigned long poking_addr; · line 711
Quoted 10 lines
arch/x86/kernel/alternative.c +4
-2+ pages[1 ] =vmalloc_to_page (addr +PAGE_SIZE ); }else { pages[0 ] =virt_to_page (addr);WARN_ON (!PageReserved (pages[0 ])); - pages[1 ] =virt_to_page (addr +PAGE_SIZE ); +if (cross_page_boundary) + pages[1 ] =virt_to_page (addr +PAGE_SIZE ); } -BUG_ON (!pages[0 ]); +BUG_ON (!pages[0 ] || (cross_page_boundary && !pages[1 ]));
checkpatch fires a lot for this patchset and I think we should tone down the BUG_ON() use.
WARNING: Avoid crashing the kernel - try using WARN_ON & recovery code rather than BUG() or BUG_ON() #116: FILE: arch/x86/kernel/alternative.c:711:
+ BUG_ON(!pages[0] || (cross_page_boundary && !pages[1]));
While the below BUG_ON makes sense, this here could be a WARN_ON or so.
Which begs the next question: AFAICT, nothing looks at text_poke*()'s retval. So why are we even bothering returning something?
Quoted 24 lines
arch/x86/kernel/alternative.c +9
-14+local_irq_save (flags); -set_fixmap (FIX_TEXT_POKE0 ,page_to_phys (pages[0 ])); -if (pages[1 ]) -set_fixmap (FIX_TEXT_POKE1 ,page_to_phys (pages[1 ])); - vaddr = (char *)fix_to_virt (FIX_TEXT_POKE0 ); -memcpy (&vaddr[(unsigned long )addr & ~PAGE_MASK ], opcode, len); -clear_fixmap (FIX_TEXT_POKE0 ); -if (pages[1 ]) -clear_fixmap (FIX_TEXT_POKE1 ); -local_flush_tlb (); -sync_core (); -/* Could also do a CLFLUSH here to speed up CPU recovery; but -that causes hangs on some VIA CPUs. */ -for (i =0 ; i < len; i++) -BUG_ON (((char *)addr)[i] != ((char *)opcode)[i]); + +/* +* The lock is not really needed, but this allows to avoid open-coding. +*/ + ptep =get_locked_pte (poking_mm, poking_addr, &ptl); + +/* +* This must not fail; preallocated in poking_init().
arch/x86/kernel/alternative.c __ro_after_init unsigned long poking_addr; · line 731
Quoted 10 lines
arch/x86/kernel/alternative.c +10
-0+*/ +VM_BUG_ON (!ptep); + +/* +* flush_tlb_mm_range() would be called when the poking_mm is not +* loaded. When PCID is in use, the flush would be deferred to the time +* the poking_mm is loaded again. Set the PTE as non-global to prevent +* it from being used when we are done. +*/ + prot =__pgprot (pgprot_val (PAGE_KERNEL ) & ~_PAGE_GLOBAL);
So
_KERNPG_TABLE | _PAGE_NX
as this is pagetable page, AFAICT.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Peter Zijlstra
maintainerRe: [PATCH v2 06/20] x86/alternative: use temporary mm for text poking
Quoted 32 lines · Borislav Petkov
Quoted 32 lines
arch/x86/kernel/alternative.c +10
-7@@ -683,41 +684,102 @@ __ro_after_init unsigned long poking_addr;static void *__text_poke (void *addr,const void *opcode,size_t len) { +bool cross_page_boundary =offset_in_page (addr) + len >PAGE_SIZE ; +temporary_mm_state_t prev; +struct page *pages[2 ] = {NULL };unsigned long flags; -char *vaddr; -struct page *pages[2 ]; -int i; +pte_t pte, *ptep; +spinlock_t *ptl; +pgprot_t prot;/* -* While boot memory allocator is runnig we cannot use struct -* pages as they are not yet initialized. +* While boot memory allocator is running we cannot use struct pages as +* they are not yet initialized. */ BUG_ON (!after_bootmem);if (!core_kernel_text ((unsigned long )addr)) { pages[0 ] =vmalloc_to_page (addr); - pages[1 ] =vmalloc_to_page (addr +PAGE_SIZE ); +if (cross_page_boundary) + pages[1 ] =vmalloc_to_page (addr +PAGE_SIZE ); }else { pages[0 ] =virt_to_page (addr);WARN_ON (!PageReserved (pages[0 ])); - pages[1 ] =virt_to_page (addr +PAGE_SIZE );
Quoted 8 lines
arch/x86/kernel/alternative.c __ro_after_init unsigned long poking_addr; · line 711
Quoted 5 lines
arch/x86/kernel/alternative.c +3
-1+if (cross_page_boundary) + pages[1 ] =virt_to_page (addr +PAGE_SIZE ); } -BUG_ON (!pages[0 ]); +BUG_ON (!pages[0 ] || (cross_page_boundary && !pages[1 ]));checkpatch fires a lot for this patchset and I think we should tone down the
BUG_ON()use.
I've been pushing for BUG_ON() in this patch set; sod checkpatch.
Maybe not this BUG_ON in particular, but a number of them introduced here are really situations where we can't do anything sane.
This BUG_ON() in particular is the choice between corrupted text or an instantly dead machine; what would you do?
In general, text_poke() cannot fail:
- suppose changing a single jump label requires poking multiple sites (not uncommon), we fail halfway through and then have to undo the first pokes, but those pokes fail again.
- this then leaves us no way forward and no way back, we've got inconsistent text state -> FAIL.
So even an 'early' fail (like here) doesn't work in the rollback scenario if you combine them.
So while in general I agree with BUG_ON() being undesirable, I think liberal sprinking in text_poke() is fine; you really _REALLY_ want this to work or fail loudly. Text corruption is just painful.
Borislav Petkov
maintainerRe: [PATCH v2 06/20] x86/alternative: use temporary mm for text poking
Quoted 5 lines · Peter Zijlstra
...
So while in general I agree with
BUG_ON()being undesirable, I think liberal sprinking intext_poke()is fine; you really _REALLY_ want this to work or fail loudly. Text corruption is just painful.
Ok. It would be good to have the gist of this sentiment in a comment above it so that it is absolutely clear why we're doing it.
And since text_poke() can't fail, then it doesn't need a retval too. AFAICT, nothing is actually using it.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Peter Zijlstra
maintainerRe: [PATCH v2 06/20] x86/alternative: use temporary mm for text poking
Quoted 7 lines · Borislav Petkov
Quoted 5 lines · Peter Zijlstra
...
So while in general I agree with
BUG_ON()being undesirable, I think liberal sprinking intext_poke()is fine; you really _REALLY_ want this to work or fail loudly. Text corruption is just painful.
Quoted 5 lines
Ok. It would be good to have the gist of this sentiment in a comment above it so that it is absolutely clear why we're doing it.
And since
text_poke()can't fail, then it doesn't need a retval too. AFAICT, nothing is actually using it.
See patch 12, that removes the return value (after fixing the few users that currently 'rely' on it).
Re: [PATCH v2 06/20] x86/alternative: use temporary mm for text poking
In reply to Borislav PetkovQuoted 11 lines · Borislav Petkov
Quoted 5 lines · Peter Zijlstra
...
So while in general I agree with
BUG_ON()being undesirable, I think liberal sprinking intext_poke()is fine; you really _REALLY_ want this to work or fail loudly. Text corruption is just painful.Ok. It would be good to have the gist of this sentiment in a comment above it so that it is absolutely clear why we're doing it.
I added a short comment for v3 above each BUG_ON().
Quoted 2 lines
And since
text_poke()can't fail, then it doesn't need a retval too. AFAICT, nothing is actually using it.
As Peter said, this is addressed in a separate patch (one patch per logical change).
Peter Zijlstra
maintainerRe: [PATCH v2 06/20] x86/alternative: use temporary mm for text poking
In reply to Peter ZijlstraQuoted 8 lines · Peter Zijlstra
In general,
text_poke()cannot fail:- suppose changing a single jump label requires poking multiple sites (not uncommon), we fail halfway through and then have to undo the first pokes, but those pokes fail again.
- this then leaves us no way forward and no way back, we've got inconsistent text state -> FAIL.
Note that this exact fail scenario still exists in the CPU hotplug code. See kernel/cpu.c:cpuhp_thread_fun():
/* * If we fail on a rollback, we're up a creek without no * paddle, no way forward, no way back. We loose, thanks for * playing. */ WARN_ON_ONCE(st->rollback);
PATCH v2 7/20 x86/kgdb: avoid redundant comparison of patched code
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
text_poke() already ensures that the written value is the correct one and fails if that is not the case. There is no need for an additional comparison. Remove it.
--- arch/x86/kernel/kgdb.c | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-)
arch/x86/kernel/kgdb.c +1 -13
@@ -746,7 +746,6 @@ void kgdb_arch_set_pc(struct pt_regs *regs, unsigned long ip)int kgdb_arch_set_breakpoint (struct kgdb_bkpt *bpt) {int err; -char opc[BREAK_INSTR_SIZE ]; bpt->type =BP_BREAKPOINT ; err = probe_kernel_read(bpt->saved_instr, (char *)bpt->bpt_addr,
@@ -765,11 +764,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt)return -EBUSY ;text_poke_kgdb ((void *)bpt->bpt_addr, arch_kgdb_ops.gdb_bpt_instr,BREAK_INSTR_SIZE ); - err =probe_kernel_read (opc, (char *)bpt->bpt_addr,BREAK_INSTR_SIZE ); -if (err) -return err; -if (memcmp (opc, arch_kgdb_ops.gdb_bpt_instr,BREAK_INSTR_SIZE )) -return -EINVAL ; bpt->type =BP_POKE_BREAKPOINT ;return err;
@@ -777,9 +771,6 @@ int kgdb_arch_set_breakpoint(struct kgdb_bkpt *bpt)int kgdb_arch_remove_breakpoint (struct kgdb_bkpt *bpt) { -int err; -char opc[BREAK_INSTR_SIZE ]; -if (bpt->type !=BP_POKE_BREAKPOINT ) goto knl_write;/*
@@ -790,10 +781,7 @@ int kgdb_arch_remove_breakpoint(struct kgdb_bkpt *bpt) goto knl_write;text_poke_kgdb ((void *)bpt->bpt_addr, bpt->saved_instr,BREAK_INSTR_SIZE ); - err =probe_kernel_read (opc, (char *)bpt->bpt_addr,BREAK_INSTR_SIZE ); -if (err ||memcmp (opc, bpt->saved_instr,BREAK_INSTR_SIZE )) - goto knl_write; -return err; +return 0 ; knl_write: return probe_kernel_write((char *)bpt->bpt_addr,
Signature · 1 line
2.17.1
PATCH v2 8/20 x86/ftrace: set trampoline pages as executable
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
Since alloc_module() will not set the pages as executable soon, we need to do so for ftrace trampoline pages after they are allocated.
For the time being, we do not change ftrace to use the text_poke() interface. As a result, ftrace breaks still breaks W^X.
--- arch/x86/kernel/ftrace.c | 8 ++++++++ 1 file changed, 8 insertions(+)
arch/x86/kernel/ftrace.c +8 -0
@@ -742,6 +742,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)unsigned long end_offset;unsigned long op_offset;unsigned long offset; +unsigned long npages;unsigned long size;unsigned long retq;unsigned long *ptr;
@@ -774,6 +775,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)return 0 ; *tramp_size = size +RET_SIZE +sizeof (void *); + npages =DIV_ROUND_UP (*tramp_size,PAGE_SIZE );/* Copy ftrace_caller onto the trampoline memory */ ret =probe_kernel_read (trampoline, (void *)start_offset, size);
@@ -818,6 +820,12 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)/* ALLOC_TRAMP flags lets us know we created it */ ops->flags |=FTRACE_OPS_FL_ALLOC_TRAMP ; +/* +* Module allocation needs to be completed by making the page +* executable. The page is still writable, which is a security hazard, +* but anyhow ftrace breaks W^X completely. +*/ +set_memory_x ((unsigned long )trampoline, npages);return (unsigned long )trampoline; fail:tramp_free (trampoline, *tramp_size);
Signature · 1 line
2.17.1
PATCH v2 9/20 x86/kprobes: instruction pages initialization enhancements
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
Make kprobes instruction pages read-only (and executable) after they are set to prevent them from mistaken or malicious modifications.
This is a preparatory patch for a following patch that makes module allocated pages non-executable and sets the page as executable after allocation.
While at it, do some small cleanup of what appears to be unnecessary masking.
--- arch/x86/kernel/kprobes/core.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
arch/x86/kernel/kprobes/core.c +20 -4
@@ -431,8 +431,20 @@ void *alloc_insn_page(void)void *page; page =module_alloc (PAGE_SIZE ); -if (page) -set_memory_ro ((unsigned long )page &PAGE_MASK ,1 ); +if (page ==NULL ) +return NULL ; + +/* +* First make the page read-only, and then only then make it executable +* to prevent it from being W+X in between. +*/ +set_memory_ro ((unsigned long )page,1 ); + +/* +* TODO: Once additional kernel code protection mechanisms are set, ensure +* that the page was not maliciously altered and it is still zeroed. +*/ +set_memory_x ((unsigned long )page,1 );return page; }
@@ -440,8 +452,12 @@ void *alloc_insn_page(void)/* Recover page to RW mode before releasing it */ void free_insn_page (void *page) { -set_memory_nx ((unsigned long )page &PAGE_MASK ,1 ); -set_memory_rw ((unsigned long )page &PAGE_MASK ,1 ); +/* +* First make the page non-executable, and then only then make it +* writable to prevent it from being W+X in between. +*/ +set_memory_nx ((unsigned long )page,1 ); +set_memory_rw ((unsigned long )page,1 );module_memfree (page); }
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 09/20] x86/kprobes: instruction pages initialization enhancements
Only nitpicks:
Quoted 1 line
Subject: Re: [PATCH v2 09/20] x86/kprobes: instruction pages initialization enhancements
Subject needs a verb.
Quoted 24 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
Make kprobes instruction pages read-only (and executable) after they are set to prevent them from mistaken or malicious modifications.
This is a preparatory patch for a following patch that makes module allocated pages non-executable and sets the page as executable after allocation.
While at it, do some small cleanup of what appears to be unnecessary masking.
Acked-by Masami Hiramatsu <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- arch/x86/kernel/kprobes/core.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)arch/x86/kernel/kprobes/core.c +0
-0
arch/x86/kernel/kprobes/core.c void *alloc_insn_page(void) · line 435
Quoted 7 lines
arch/x86/kernel/kprobes/core.c +2
-2@@ -431,8 +431,20 @@ void *alloc_insn_page(void)void *page; page =module_alloc (PAGE_SIZE ); -if (page) -set_memory_ro ((unsigned long )page &PAGE_MASK ,1 ); +if (page ==NULL ) +return NULL ;
Null tests we generally do like this:
if (! ...
like in the rest of this file.
arch/x86/kernel/kprobes/core.c void *alloc_insn_page(void) · line 438
Quoted 3 lines
arch/x86/kernel/kprobes/core.c +3
-0+ +/* +* First make the page read-only, and then only then make it executable
s/then only then/only then/
ditto below.
Quoted 28 lines
arch/x86/kernel/kprobes/core.c +15
-2+* to prevent it from being W+X in between. +*/ +set_memory_ro ((unsigned long )page,1 ); + +/* +* TODO: Once additional kernel code protection mechanisms are set, ensure +* that the page was not maliciously altered and it is still zeroed. +*/ +set_memory_x ((unsigned long )page,1 );return page; }@@ -440,8 +452,12 @@ void *alloc_insn_page(void)/* Recover page to RW mode before releasing it */ void free_insn_page (void *page) { -set_memory_nx ((unsigned long )page &PAGE_MASK ,1 ); -set_memory_rw ((unsigned long )page &PAGE_MASK ,1 ); +/* +* First make the page non-executable, and then only then make it +* writable to prevent it from being W+X in between. +*/ +set_memory_nx ((unsigned long )page,1 ); +set_memory_rw ((unsigned long )page,1 );module_memfree (page); }
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 09/20] x86/kprobes: instruction pages initialization enhancements
Quoted 3 lines · Borislav Petkov
Only nitpicks:
Thanks for the feedback. Applied.
PATCH v2 10/20 x86: avoid W^X being broken during modules loading
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
When modules and BPF filters are loaded, there is a time window in which some memory is both writable and executable. An attacker that has already found another vulnerability (e.g., a dangling pointer) might be able to exploit this behavior to overwrite kernel code.
Prevent having writable executable PTEs in this stage. In addition, avoiding having W+X mappings can also slightly simplify the patching of modules code on initialization (e.g., by alternatives and static-key), as would be done in the next patch.
To avoid having W+X mappings, set them initially as RW (NX) and after they are set as RO set them as X as well. Setting them as executable is done as a separate step to avoid one core in which the old PTE is cached (hence writable), and another which sees the updated PTE (executable), which would break the W^X protection.
--- arch/x86/kernel/alternative.c | 28 +++++++++++++++++++++------- arch/x86/kernel/module.c | 2 +- include/linux/filter.h | 2 +- kernel/module.c | 5 +++++ 4 files changed, 28 insertions(+), 9 deletions(-)
arch/x86/kernel/alternative.c +21 -7
@@ -667,15 +667,29 @@ void __init alternative_instructions(void)* handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, -size_t len) +size_t len) {unsigned long flags; -local_irq_save (flags); -memcpy (addr, opcode, len); -local_irq_restore (flags); -sync_core (); -/* Could also do a CLFLUSH here to speed up CPU recovery; but -that causes hangs on some VIA CPUs. */ + +if (static_cpu_has (X86_FEATURE_NX ) && +is_module_text_address ((unsigned long )addr)) { +/* +* Modules text is marked initially as non-executable, so the +* code cannot be running and speculative code-fetches are +* prevented. We can just change the code. +*/ +memcpy (addr, opcode, len); + }else { +local_irq_save (flags); +memcpy (addr, opcode, len); +local_irq_restore (flags); +sync_core (); + +/* +* Could also do a CLFLUSH here to speed up CPU recovery; but +* that causes hangs on some VIA CPUs. +*/ + }return addr; }
arch/x86/kernel/module.c +1 -1
@@ -87,7 +87,7 @@ void *module_alloc(unsigned long size) p =__vmalloc_node_range (size,MODULE_ALIGN ,MODULES_VADDR +get_module_load_offset (),MODULES_END ,GFP_KERNEL , -PAGE_KERNEL_EXEC ,0 ,NUMA_NO_NODE , +PAGE_KERNEL ,0 ,NUMA_NO_NODE ,__builtin_return_address (0 ));if (p && (kasan_module_alloc (p, size) <0 )) {vfree (p);
include/linux/filter.h +1 -1
@@ -681,7 +681,6 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default)static inline void bpf_prog_lock_ro (struct bpf_prog *fp) { - fp->undo_set_mem =1 ;set_memory_ro ((unsigned long )fp, fp->pages); }
@@ -694,6 +693,7 @@ static inline void bpf_prog_unlock_ro(struct bpf_prog *fp)static inline void bpf_jit_binary_lock_ro (struct bpf_binary_header *hdr) {set_memory_ro ((unsigned long )hdr, hdr->pages); +set_memory_x ((unsigned long )hdr, hdr->pages); }static inline void bpf_jit_binary_unlock_ro (struct bpf_binary_header *hdr)
kernel/module.c +5 -0
@@ -1950,8 +1950,13 @@ void module_enable_ro(const struct module *mod, bool after_init)return ;frob_text (&mod->core_layout, set_memory_ro); +frob_text (&mod->core_layout, set_memory_x); +frob_rodata (&mod->core_layout, set_memory_ro); +frob_text (&mod->init_layout, set_memory_ro); +frob_text (&mod->init_layout, set_memory_x); +frob_rodata (&mod->init_layout, set_memory_ro);if (after_init)
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 1 line
Subject: Re: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
For your next submission, please fix all your subjects:
The tip tree preferred format for patch subject prefixes is 'subsys/component:', e.g. 'x86/apic:', 'x86/mm/fault:', 'sched/fair:', 'genirq/core:'. Please do not use file names or complete file paths as prefix. 'git log path/to/file' should give you a reasonable hint in most cases.
The condensed patch description in the subject line should start with a uppercase letter and should be written in imperative tone.
Quoted 43 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
When modules and BPF filters are loaded, there is a time window in which some memory is both writable and executable. An attacker that has already found another vulnerability (e.g., a dangling pointer) might be able to exploit this behavior to overwrite kernel code.
Prevent having writable executable PTEs in this stage. In addition, avoiding having W+X mappings can also slightly simplify the patching of modules code on initialization (e.g., by alternatives and static-key), as would be done in the next patch.
To avoid having W+X mappings, set them initially as RW (NX) and after they are set as RO set them as X as well. Setting them as executable is done as a separate step to avoid one core in which the old PTE is cached (hence writable), and another which sees the updated PTE (executable), which would break the W^X protection.
Cc Kees Cook <[email protected]>Cc Peter Zijlstra <[email protected]>Cc Dave Hansen <[email protected]>Cc Masami Hiramatsu <[email protected]>Suggested-by Thomas Gleixner <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- arch/x86/kernel/alternative.c | 28 +++++++++++++++++++++------- arch/x86/kernel/module.c | 2 +- include/linux/filter.h | 2 +- kernel/module.c | 5 +++++ 4 files changed, 28 insertions(+), 9 deletions(-)arch/x86/kernel/alternative.c +1
-1@@ -667,15 +667,29 @@ void __init alternative_instructions(void)* handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, -size_t len) +size_t len)
arch/x86/kernel/alternative.c void __init alternative_instructions(void) · line 674
Quoted 10 lines
arch/x86/kernel/alternative.c +2
-6{unsigned long flags; -local_irq_save (flags); -memcpy (addr, opcode, len); -local_irq_restore (flags); -sync_core (); -/* Could also do a CLFLUSH here to speed up CPU recovery; but -that causes hangs on some VIA CPUs. */ + + if (static_cpu_has(X86_FEATURE_NX) &&
Not a fast path - boot_cpu_has() is fine here.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 16 lines · Borislav Petkov
Quoted 15 lines
arch/x86/kernel/alternative.c +1
-3@@ -667,15 +667,29 @@ void __init alternative_instructions(void)* handlers seeing an inconsistent instruction while you patch. */ void *__init_or_module text_poke_early(void *addr, const void *opcode, -size_t len) +size_t len) {unsigned long flags; -local_irq_save (flags); -memcpy (addr, opcode, len);
Quoted 8 lines
arch/x86/kernel/alternative.c void __init alternative_instructions(void) · line 674
Quoted 6 lines
arch/x86/kernel/alternative.c +2
-4-local_irq_restore (flags); -sync_core (); -/* Could also do a CLFLUSH here to speed up CPU recovery; but -that causes hangs on some VIA CPUs. */ + + if (static_cpu_has(X86_FEATURE_NX) &&Not a fast path -
boot_cpu_has()is fine here.
Are you sure about that? This path is still used when modules are loaded.
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 1 line · Nadav Amit
Are you sure about that? This path is still used when modules are loaded.
Yes, I'm sure. Loading a module does a gazillion things so saving a couple of insns - yes, boot_cpu_has() is usually a RIP-relative MOV and a TEST - doesn't show even as a blip on any radar.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 8 lines · Borislav Petkov
Quoted 1 line · Nadav Amit
Are you sure about that? This path is still used when modules are loaded.
Yes, I'm sure. Loading a module does a gazillion things so saving a couple of insns - yes,
boot_cpu_has()is usually a RIP-relative MOV and a TEST - doesn't show even as a blip on any radar.
I fully agree, if that is the standard.
It is just that I find the use of static_cpu_has()/boot_cpu_has() to be very inconsistent. I doubt that show_cpuinfo_misc(), copy_fpstate_to_sigframe(), or i915_memcpy_init_early() that use static_cpu_has() are any hotter than text_poke_early().
Anyhow, I’ll use boot_cpu_has() as you said.
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 4 lines · Nadav Amit
It is just that I find the use of static_cpu_has()/boot_cpu_has() to be very inconsistent. I doubt that
show_cpuinfo_misc(),copy_fpstate_to_sigframe(), ori915_memcpy_init_early()that usestatic_cpu_has()are any hotter thantext_poke_early().
Would some beefing of the comment over it help?
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 9 lines · Borislav Petkov
Quoted 4 lines · Nadav Amit
It is just that I find the use of static_cpu_has()/boot_cpu_has() to be very inconsistent. I doubt that
show_cpuinfo_misc(),copy_fpstate_to_sigframe(), ori915_memcpy_init_early()that usestatic_cpu_has()are any hotter thantext_poke_early().Would some beefing of the comment over it help?
Is there any comment over static_cpu_has()? ;-)
Anyhow, obviously a comment would be useful.
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 1 line · Nadav Amit
Is there any comment over
static_cpu_has()? ;-)
Almost:
/*
* Static testing of CPU features. Used the same as boot_cpu_has().
* These will statically patch the target code for additional
* performance.
*/
static __always_inline __pure bool _static_cpu_has(u16 bit)
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 5 lines · Borislav Petkov
Quoted 1 line · Nadav Amit
Is there any comment over
static_cpu_has()? ;-)
Quoted 8 lines
Almost:
/*
* Static testing of CPU features. Used the same asboot_cpu_has().
* These will statically patch the target code for additional
* performance.
*/
static__always_inline__purebool_static_cpu_has(u16 bit)
Oh, I missed this comment.
BTW: the “__pure” attribute is useless when “__always_inline” is used. Unless it is intended to be some sort of comment, of course.
Borislav Petkov
maintainerPATCH 1/1 x86/cpufeature: Remove __pure attribute to _static_cpu_has()
Quoted 2 lines · Nadav Amit
BTW: the “__pure” attribute is useless when “__always_inline” is used. Unless it is intended to be some sort of comment, of course.
---
From: Borislav Petkov <[email protected]>
Date: Thu, 7 Mar 2019 15:54:51 +0100
__pure is used to make gcc do Common Subexpression Elimination (CSE) and thus save subsequent invocations of a function which does a complex computation (without side effects). As a simple example:
bool a = _static_cpu_has(x); bool b = _static_cpu_has(x);
gets turned into
bool a = _static_cpu_has(x); bool b = a;
However, gcc doesn't do CSE with asm()s when those get inlined - like it is done with _static_cpu_has() - because, for example, the t_yes/t_no labels are different for each inlined function body and thus cannot be detected as equivalent anymore for the CSE heuristic to hit.
However, this all is beside the point because best it should be avoided to have more than one call to _static_cpu_has(X) in the same function due to the fact that each such call is an alternatives patch site and it is simply pointless.
Therefore, drop the __pure attribute as it is not doing anything.
--- arch/x86/include/asm/cpufeature.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/cpufeature.h +1 -1
@@ -162,7 +162,7 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit);* majority of cases and you should stick to using it as it is generally * only two instructions: a RIP-relative MOV and a TEST. */ -static __always_inline __pure bool _static_cpu_has(u16 bit) +static __always_inline bool _static_cpu_has(u16 bit) {asm_volatile_goto ("1: jmp 6f\n" "2:\n"
Signature · 7 lines
2.21.0
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.[email protected]
reviewerRe: [PATCH] x86/cpufeature: Remove __pure attribute to _static_cpu_has()
Quoted 57 lines · Borislav Petkov
Quoted 1 line · Nadav Amit
BTW: the “__pure” attribute is useless when “__always_inline” is
used.
Quoted 1 line
Unless it is intended to be some sort of comment, of course.
---
From: Borislav Petkov <[email protected]>
Date: Thu, 7 Mar 2019 15:54:51 +0100
__pureis used to make gcc do Common Subexpression Elimination (CSE) and thus save subsequent invocations of a function which does a complex computation (without side effects). As a simple example:bool a =
_static_cpu_has(x);
bool b =_static_cpu_has(x);gets turned into
bool a =
_static_cpu_has(x);
bool b = a;However, gcc doesn't do CSE with asm()s when those get inlined - like it
is done with_static_cpu_has()- because, for example, the t_yes/t_no labels are different for each inlined function body and thus cannot be detected as equivalent anymore for the CSE heuristic to hit.However, this all is beside the point because best it should be avoided to have more than one call to
_static_cpu_has(X)in the same function due to the fact that each such call is an alternatives patch site and it
is simply pointless.Therefore, drop the
__pureattribute as it is not doing anything.Reported-by Nadav Amit <[email protected]>Signed-off-by Borislav Petkov <[email protected]>Cc Peter Zijlstra <[email protected]>---
arch/x86/include/asm/cpufeature.h| 2 +-1 file changed, 1 insertion(+), 1 deletion(-)arch/x86/include/asm/cpufeature.h +1
-1@@ -162,7 +162,7 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c,unsigned int bit); * majority of cases and you should stick to using it as it is generally * only two instructions: a RIP-relative MOV and a TEST. */ -static __always_inline __pure bool _static_cpu_has(u16 bit) +static __always_inline bool _static_cpu_has(u16 bit) { asm_volatile_goto("1: jmp 6f\n"
Quoted 1 line
"2:\n"
Uhm... (a) it is correct, even if the compiler doesn't use it now, it allows the compiler to CSE it in the future; (b) it is documentation; (c) there is an actual bug here: the "volatile" implies a side effect, which in reality is not present, inhibiting CSE.
So the correct fix is to remove "volatile", not remove "__pure".
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Borislav Petkov
maintainerRe: [PATCH] x86/cpufeature: Remove __pure attribute to _static_cpu_has()
Lemme preface this by saying that I've talked to gcc guys before doing this.
Quoted 2 lines · On Thu, Mar 07, 2019 at 08:43:50AM -0800, [email protected] wrote:
Uhm... (a) it is correct, even if the compiler doesn't use it now, it allows the compiler to CSE it in the future;
Well, the compiler won't CSE asm blocks due to the difference in the labels, for example, so the heuristic won't detect them as equivalent blocks.
Also, compiler guys said that they might consider inlining pure functions later, in the IPA stage but that's future stuff.
This is how I understood it, at least.
Quoted 1 line
(b) it is documentation;
That could be a comment instead. Otherwise we will wonder again why this is marked pure.
Quoted 4 lines
(c) there is an actual bug here: the "volatile" implies a side effect, which in reality is not present, inhibiting CSE.
So the correct fix is to remove "volatile", not remove "
__pure".
There's not really a volatile there:
/*
* GCC 'asm goto' miscompiles certain code sequences:
*
* http://gcc.gnu.org/bugzilla/show_bug.cgi?id=58670
*
* Work it around via a compiler barrier quirk suggested by Jakub Jelinek. *
* (asm goto is automatically volatile - the naming reflects this.) */
#define asm_volatile_goto(x...) do { asm goto(x); asm (""); } while (0)
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.tip-bot for Borislav Petkov
merge bot[tip:x86/asm] x86/cpufeature: Remove __pure attribute to _static_cpu_has()
In reply to Borislav Petkovx86/cpufeature: Remove __pure attribute to _static_cpu_has()
Commit-ID: ae37a8cd9b0ad3416d71e54cfaeb3744178189a8 Gitweb: https://git.kernel.org/tip/ae37a8cd9b0ad3416d71e54cfaeb3744178189a8 Author: Borislav Petkov <[email protected]> AuthorDate: Thu, 7 Mar 2019 15:54:51 +0100 Committer: Borislav Petkov <[email protected]> CommitDate: Fri, 29 Mar 2019 19:13:48 +0100
x86/cpufeature: Remove __pure attribute to _static_cpu_has()
__pure is used to make gcc do Common Subexpression Elimination (CSE) and thus save subsequent invocations of a function which does a complex computation (without side effects). As a simple example:
bool a = _static_cpu_has(x); bool b = _static_cpu_has(x);
gets turned into
bool a = _static_cpu_has(x); bool b = a;
However, gcc doesn't do CSE with asm()s when those get inlined - like it is done with _static_cpu_has() - because, for example, the t_yes/t_no labels are different for each inlined function body and thus cannot be detected as equivalent anymore for the CSE heuristic to hit.
However, this all is beside the point because best it should be avoided to have more than one call to _static_cpu_has(X) in the same function due to the fact that each such call is an alternatives patch site and it is simply pointless.
Therefore, drop the __pure attribute as it is not doing anything.
--- arch/x86/include/asm/cpufeature.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/include/asm/cpufeature.h +1 -1
@@ -159,7 +159,7 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit);* These will statically patch the target code for additional * performance. */ -static __always_inline __pure bool _static_cpu_has(u16 bit) +static __always_inline bool _static_cpu_has(u16 bit) {asm_volatile_goto ("1: jmp 6f\n" "2:\n"
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
In reply to Borislav PetkovQuoted 11 lines · Borislav Petkov
Quoted 1 line · Nadav Amit
Is there any comment over
static_cpu_has()? ;-)Almost:
/*
* Static testing of CPU features. Used the same asboot_cpu_has().
* These will statically patch the target code for additional
* performance.
*/
static__always_inline__purebool_static_cpu_has(u16 bit)
Ok, I guess something like that along with converting the obvious slow path callers to boot_cpu_has():
---
arch/x86/include/asm/cpufeature.h +6 -3
@@ -155,9 +155,12 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit);#else /* -* Static testing of CPU features. Used the same as boot_cpu_has(). -* These will statically patch the target code for additional -* performance. +* Static testing of CPU features. Used the same as boot_cpu_has(). It +* statically patches the target code for additional performance. Use +* static_cpu_has() only in fast paths, where every cycle counts. Which +* means that the boot_cpu_has() variant is already fast enough for the +* majority of cases and you should stick to using it as it is generally +* only two instructions: a RIP-relative MOV and a TEST. */ static __always_inline __pure bool _static_cpu_has(u16 bit) {
arch/x86/include/asm/fpu/internal.h +3 -4
@@ -291,7 +291,7 @@ static inline void copy_xregs_to_kernel_booting(struct xregs_state *xstate)WARN_ON (system_state !=SYSTEM_BOOTING ); -if (static_cpu_has (X86_FEATURE_XSAVES )) +if (boot_cpu_has (X86_FEATURE_XSAVES ))XSTATE_OP (XSAVES , xstate, lmask, hmask, err);else XSTATE_OP (XSAVE , xstate, lmask, hmask, err);
@@ -313,7 +313,7 @@ static inline void copy_kernel_to_xregs_booting(struct xregs_state *xstate)WARN_ON (system_state !=SYSTEM_BOOTING ); -if (static_cpu_has (X86_FEATURE_XSAVES )) +if (boot_cpu_has (X86_FEATURE_XSAVES ))XSTATE_OP (XRSTORS , xstate, lmask, hmask, err);else XSTATE_OP (XRSTOR , xstate, lmask, hmask, err);
@@ -528,8 +528,7 @@ static inline void fpregs_activate(struct fpu *fpu)* - switch_fpu_finish() restores the new state as * necessary. */ -static inline void -switch_fpu_prepare (struct fpu *old_fpu,int cpu) +static inline void switch_fpu_prepare (struct fpu *old_fpu,int cpu) {if (static_cpu_has (X86_FEATURE_FPU ) && old_fpu->initialized) {if (!copy_fpregs_to_fpstate (old_fpu))
arch/x86/kernel/apic/apic_numachip.c +1 -1
@@ -175,7 +175,7 @@ static void fixup_cpu_id(struct cpuinfo_x86 *c, int node)this_cpu_write (cpu_llc_id, node);/* Account for nodes per socket in multi-core-module processors */ -if (static_cpu_has (X86_FEATURE_NODEID_MSR )) { +if (boot_cpu_has (X86_FEATURE_NODEID_MSR )) {rdmsrl (MSR_FAM10H_NODE_ID , val); nodes = ((val >>3 ) &7 ) +1 ; }
arch/x86/kernel/cpu/aperfmperf.c +3 -3
@@ -83,7 +83,7 @@ unsigned int aperfmperf_get_khz(int cpu)if (!cpu_khz)return 0 ; -if (!static_cpu_has (X86_FEATURE_APERFMPERF )) +if (!boot_cpu_has (X86_FEATURE_APERFMPERF ))return 0 ;aperfmperf_snapshot_cpu (cpu,ktime_get (), true);
@@ -99,7 +99,7 @@ void arch_freq_prepare_all(void)if (!cpu_khz)return ; -if (!static_cpu_has (X86_FEATURE_APERFMPERF )) +if (!boot_cpu_has (X86_FEATURE_APERFMPERF ))return ;for_each_online_cpu (cpu)
@@ -115,7 +115,7 @@ unsigned int arch_freq_get_on_cpu(int cpu)if (!cpu_khz)return 0 ; -if (!static_cpu_has (X86_FEATURE_APERFMPERF )) +if (!boot_cpu_has (X86_FEATURE_APERFMPERF ))return 0 ;if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true))
arch/x86/kernel/cpu/common.c +1 -1
@@ -1668,7 +1668,7 @@ static void setup_getcpu(int cpu)unsigned long cpudata =vdso_encode_cpunode (cpu,early_cpu_to_node (cpu));struct desc_struct d = { }; -if (static_cpu_has (X86_FEATURE_RDTSCP )) +if (boot_cpu_has (X86_FEATURE_RDTSCP ))write_rdtscp_aux (cpudata);/* Store CPU and node number in limit. */
arch/x86/kernel/cpu/mce/inject.c +1 -1
@@ -528,7 +528,7 @@ static void do_inject(void)* only on the node base core. Refer to D18F3x44[NbMcaToMstCpuEn] for * Fam10h and later BKDGs. */ -if (static_cpu_has (X86_FEATURE_AMD_DCM ) && +if (boot_cpu_has (X86_FEATURE_AMD_DCM ) && b ==4 && boot_cpu_data.x86 <0x17 ) {toggle_nb_mca_mst_cpu (amd_get_nb_id (cpu));
arch/x86/kernel/cpu/proc.c +5 -5
@@ -35,11 +35,11 @@ static void show_cpuinfo_misc(struct seq_file *m, struct cpuinfo_x86 *c)"fpu_exception\t: %s\n" "cpuid level\t: %d\n" "wp\t\t: yes\n" , -static_cpu_has_bug (X86_BUG_FDIV ) ?"yes" :"no" , -static_cpu_has_bug (X86_BUG_F00F ) ?"yes" :"no" , -static_cpu_has_bug (X86_BUG_COMA ) ?"yes" :"no" , -static_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , -static_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , +boot_cpu_has_bug (X86_BUG_FDIV ) ?"yes" :"no" , +boot_cpu_has_bug (X86_BUG_F00F ) ?"yes" :"no" , +boot_cpu_has_bug (X86_BUG_COMA ) ?"yes" :"no" , +boot_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , +boot_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , c->cpuid_level); }#else
arch/x86/kernel/ldt.c +7 -7
@@ -113,7 +113,7 @@ static void do_sanity_check(struct mm_struct *mm,* tables. */ WARN_ON (!had_kernel_mapping); -if (static_cpu_has (X86_FEATURE_PTI )) +if (boot_cpu_has (X86_FEATURE_PTI ))WARN_ON (!had_user_mapping); }else {/*
@@ -121,7 +121,7 @@ static void do_sanity_check(struct mm_struct *mm,* Sync the pgd to the usermode tables. */ WARN_ON (had_kernel_mapping); -if (static_cpu_has (X86_FEATURE_PTI )) +if (boot_cpu_has (X86_FEATURE_PTI ))WARN_ON (had_user_mapping); } }
@@ -156,7 +156,7 @@ static void map_ldt_struct_to_user(struct mm_struct *mm) k_pmd =pgd_to_pmd_walk (k_pgd,LDT_BASE_ADDR ); u_pmd =pgd_to_pmd_walk (u_pgd,LDT_BASE_ADDR ); -if (static_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt) +if (boot_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt)set_pmd (u_pmd, *k_pmd); }
@@ -181,7 +181,7 @@ static void map_ldt_struct_to_user(struct mm_struct *mm) {pgd_t *pgd =pgd_offset (mm,LDT_BASE_ADDR ); -if (static_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt) +if (boot_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt)set_pgd (kernel_to_user_pgdp (pgd), *pgd); }
@@ -208,7 +208,7 @@ map_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt, int slot)spinlock_t *ptl;int i, nr_pages; -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return 0 ;/*
@@ -271,7 +271,7 @@ static void unmap_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt)return ;/* LDT map/unmap is only required for PTI */ -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ; nr_pages =DIV_ROUND_UP (ldt->nr_entries *LDT_ENTRY_SIZE ,PAGE_SIZE );
@@ -311,7 +311,7 @@ static void free_ldt_pgtables(struct mm_struct *mm)unsigned long start =LDT_BASE_ADDR ;unsigned long end =LDT_END_ADDR ; -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ;tlb_gather_mmu (&tlb, mm, start, end);
arch/x86/kernel/paravirt.c +1 -1
@@ -121,7 +121,7 @@ DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (!static_cpu_has (X86_FEATURE_HYPERVISOR )) +if (!boot_cpu_has (X86_FEATURE_HYPERVISOR ))static_branch_disable (&virt_spin_lock_key); }
arch/x86/kernel/process.c +2 -2
@@ -236,7 +236,7 @@ static int get_cpuid_mode(void)static int set_cpuid_mode (struct task_struct *task,unsigned long cpuid_enabled) { -if (!static_cpu_has (X86_FEATURE_CPUID_FAULT )) +if (!boot_cpu_has (X86_FEATURE_CPUID_FAULT ))return -ENODEV ;if (cpuid_enabled)
@@ -666,7 +666,7 @@ static int prefer_mwait_c1_over_halt(const struct cpuinfo_x86 *c)if (c->x86_vendor !=X86_VENDOR_INTEL )return 0 ; -if (!cpu_has (c,X86_FEATURE_MWAIT ) ||static_cpu_has_bug (X86_BUG_MONITOR )) +if (!cpu_has (c,X86_FEATURE_MWAIT ) ||boot_cpu_has_bug (X86_BUG_MONITOR ))return 0 ;return 1 ;
arch/x86/kernel/reboot.c +1 -1
@@ -108,7 +108,7 @@ void __noreturn machine_real_restart(unsigned int type)write_cr3 (real_mode_header->trampoline_pgd);/* Exiting long mode will fail if CR4.PCIDE is set. */ -if (static_cpu_has (X86_FEATURE_PCID )) +if (boot_cpu_has (X86_FEATURE_PCID ))cr4_clear_bits (X86_CR4_PCIDE );#endif
arch/x86/kernel/vm86_32.c +1 -1
@@ -369,7 +369,7 @@ static long do_sys_vm86(struct vm86plus_struct __user *user_vm86, bool plus)preempt_disable (); tsk->thread.sp0 +=16 ; -if (static_cpu_has (X86_FEATURE_SEP )) { +if (boot_cpu_has (X86_FEATURE_SEP )) { tsk->thread.sysenter_cs =0 ;refresh_sysenter_cs (&tsk->thread); }
arch/x86/kvm/svm.c +5 -5
@@ -835,7 +835,7 @@ static void svm_init_erratum_383(void)int err;u64 val; -if (!static_cpu_has_bug (X86_BUG_AMD_TLB_MMATCH )) +if (!boot_cpu_has_bug (X86_BUG_AMD_TLB_MMATCH ))return ;/* Use _safe variants to not break nested virtualization */
@@ -889,7 +889,7 @@ static int has_svm(void)static void svm_hardware_disable (void ) {/* Make sure we clean up behind us */ -if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) +if (boot_cpu_has (X86_FEATURE_TSCRATEMSR ))wrmsrl (MSR_AMD64_TSC_RATIO ,TSC_RATIO_DEFAULT );cpu_svm_disable ();
@@ -931,7 +931,7 @@ static int svm_hardware_enable(void)wrmsrl (MSR_VM_HSAVE_PA ,page_to_pfn (sd->save_area) <<PAGE_SHIFT ); -if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) { +if (boot_cpu_has (X86_FEATURE_TSCRATEMSR )) {wrmsrl (MSR_AMD64_TSC_RATIO ,TSC_RATIO_DEFAULT );__this_cpu_write (current_tsc_ratio,TSC_RATIO_DEFAULT ); }
@@ -2247,7 +2247,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++)rdmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); -if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) { +if (boot_cpu_has (X86_FEATURE_TSCRATEMSR )) {u64 tsc_ratio = vcpu->arch.tsc_scaling_ratio;if (tsc_ratio !=__this_cpu_read (current_tsc_ratio)) {__this_cpu_write (current_tsc_ratio, tsc_ratio);
@@ -2255,7 +2255,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu) } }/* This assumes that the kernel never uses MSR_TSC_AUX */ -if (static_cpu_has (X86_FEATURE_RDTSCP )) +if (boot_cpu_has (X86_FEATURE_RDTSCP ))wrmsrl (MSR_TSC_AUX , svm->tsc_aux);if (sd->current_vmcb != svm->vmcb) {
arch/x86/kvm/vmx/vmx.c +2 -2
@@ -6553,7 +6553,7 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu)if (vcpu->guest_debug &KVM_GUESTDBG_SINGLESTEP )vmx_set_interrupt_shadow (vcpu,0 ); -if (static_cpu_has (X86_FEATURE_PKU ) && +if (boot_cpu_has (X86_FEATURE_PKU ) &&kvm_read_cr4_bits (vcpu,X86_CR4_PKE ) && vcpu->arch.pkru != vmx->host_pkru)__write_pkru (vcpu->arch.pkru);
@@ -6633,7 +6633,7 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu)* back on host, so it is safe to read guest PKRU from current * XSAVE. */ -if (static_cpu_has (X86_FEATURE_PKU ) && +if (boot_cpu_has (X86_FEATURE_PKU ) &&kvm_read_cr4_bits (vcpu,X86_CR4_PKE )) { vcpu->arch.pkru =__read_pkru ();if (vcpu->arch.pkru != vmx->host_pkru)
arch/x86/mm/dump_pagetables.c +2 -2
@@ -579,7 +579,7 @@ void ptdump_walk_pgd_level(struct seq_file *m, pgd_t *pgd)void ptdump_walk_pgd_level_debugfs (struct seq_file *m,pgd_t *pgd,bool user) {#ifdef CONFIG_PAGE_TABLE_ISOLATION -if (user &&static_cpu_has (X86_FEATURE_PTI )) +if (user &&boot_cpu_has (X86_FEATURE_PTI )) pgd =kernel_to_user_pgdp (pgd);#endif ptdump_walk_pgd_level_core (m, pgd, false, false);
@@ -592,7 +592,7 @@ void ptdump_walk_user_pgd_level_checkwx(void)pgd_t *pgd =INIT_PGD ;if (!(__supported_pte_mask & _PAGE_NX) || - !static_cpu_has (X86_FEATURE_PTI )) + !boot_cpu_has (X86_FEATURE_PTI ))return ;pr_info ("x86/mm: Checking user space page tables\n" );
arch/x86/mm/pgtable.c +2 -2
@@ -190,7 +190,7 @@ static void pgd_dtor(pgd_t *pgd)* when PTI is enabled. We need them to map the per-process LDT into the * user-space page-table. */ -#define PREALLOCATED_USER_PMDS (static_cpu_has(X86_FEATURE_PTI) ? \ +#define PREALLOCATED_USER_PMDS (boot_cpu_has(X86_FEATURE_PTI) ? \ KERNEL_PGD_PTRS : 0)#define MAX_PREALLOCATED_USER_PMDS KERNEL_PGD_PTRS
@@ -292,7 +292,7 @@ static void pgd_mop_up_pmds(struct mm_struct *mm, pgd_t *pgdp)#ifdef CONFIG_PAGE_TABLE_ISOLATION -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ; pgdp =kernel_to_user_pgdp (pgdp);
arch/x86/mm/pti.c +1 -1
@@ -626,7 +626,7 @@ void pti_set_kernel_image_nonglobal(void)*/ void __init pti_init (void ) { -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ;pr_info ("enabled\n" );
drivers/cpufreq/amd_freq_sensitivity.c +1 -1
@@ -124,7 +124,7 @@ static int __init amd_freq_sensitivity_init(void) PCI_DEVICE_ID_AMD_KERNCZ_SMBUS, NULL);if (!pcidev) { -if (!static_cpu_has (X86_FEATURE_PROC_FEEDBACK )) +if (!boot_cpu_has (X86_FEATURE_PROC_FEEDBACK ))return -ENODEV ; }
drivers/cpufreq/intel_pstate.c +9 -9
@@ -520,7 +520,7 @@ static s16 intel_pstate_get_epb(struct cpudata *cpu_data)u64 epb;int ret; -if (!static_cpu_has (X86_FEATURE_EPB )) +if (!boot_cpu_has (X86_FEATURE_EPB ))return -ENXIO ; ret =rdmsrl_on_cpu (cpu_data->cpu,MSR_IA32_ENERGY_PERF_BIAS , &epb);
@@ -534,7 +534,7 @@ static s16 intel_pstate_get_epp(struct cpudata *cpu_data, u64 hwp_req_data) {s16 epp; -if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) {/* * When hwp_req_data is 0, means that caller didn't read * MSR_HWP_REQUEST, so need to read and get EPP.
@@ -559,7 +559,7 @@ static int intel_pstate_set_epb(int cpu, s16 pref)u64 epb;int ret; -if (!static_cpu_has (X86_FEATURE_EPB )) +if (!boot_cpu_has (X86_FEATURE_EPB ))return -ENXIO ; ret =rdmsrl_on_cpu (cpu,MSR_IA32_ENERGY_PERF_BIAS , &epb);
@@ -607,7 +607,7 @@ static int intel_pstate_get_energy_pref_index(struct cpudata *cpu_data)if (epp <0 )return epp; -if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) {if (epp ==HWP_EPP_PERFORMANCE )return 1 ;if (epp <=HWP_EPP_BALANCE_PERFORMANCE )
@@ -616,7 +616,7 @@ static int intel_pstate_get_energy_pref_index(struct cpudata *cpu_data)return 3 ; else return 4; - }else if (static_cpu_has (X86_FEATURE_EPB )) { + }else if (boot_cpu_has (X86_FEATURE_EPB )) {/* * Range: * 0x00-0x03 : Performance
@@ -644,7 +644,7 @@ static int intel_pstate_set_energy_pref_index(struct cpudata *cpu_data,mutex_lock (&intel_pstate_limits_lock); -if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) {u64 value; ret =rdmsrl_on_cpu (cpu_data->cpu,MSR_HWP_REQUEST , &value);
@@ -819,7 +819,7 @@ static void intel_pstate_hwp_set(unsigned int cpu) epp = cpu_data->epp_powersave; } update_epp: -if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) { value &= ~GENMASK_ULL (31 ,24 ); value |= (u64 )epp <<24 ; }else {
@@ -844,7 +844,7 @@ static void intel_pstate_hwp_force_min_perf(int cpu) value |=HWP_MIN_PERF (min_perf);/* Set EPP/EPB to min */ -if (static_cpu_has (X86_FEATURE_HWP_EPP )) +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) value |=HWP_ENERGY_PERF_PREFERENCE (HWP_EPP_POWERSAVE );else intel_pstate_set_epb (cpu,HWP_EPP_BALANCE_POWERSAVE );
@@ -1191,7 +1191,7 @@ static void __init intel_pstate_sysfs_expose_params(void)static void intel_pstate_hwp_enable (struct cpudata *cpudata) {/* First disable HWP notification interrupt as we don't process them */ -if (static_cpu_has (X86_FEATURE_HWP_NOTIFY )) +if (boot_cpu_has (X86_FEATURE_HWP_NOTIFY ))wrmsrl_on_cpu (cpudata->cpu,MSR_HWP_INTERRUPT ,0x00 );wrmsrl_on_cpu (cpudata->cpu,MSR_PM_ENABLE ,0x1 );
drivers/cpufreq/powernow-k8.c +1 -1
@@ -1178,7 +1178,7 @@ static int powernowk8_init(void)unsigned int i, supported_cpus =0 ;int ret; -if (static_cpu_has (X86_FEATURE_HW_PSTATE )) { +if (boot_cpu_has (X86_FEATURE_HW_PSTATE )) {__request_acpi_cpufreq ();return -ENODEV ; }
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.[email protected]
reviewerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 561 lines · Borislav Petkov
Quoted 11 lines · Borislav Petkov
Quoted 1 line · Nadav Amit
Is there any comment over
static_cpu_has()? ;-)Almost:
/*
* Static testing of CPU features. Used the same asboot_cpu_has().
* These will statically patch the target code for additional
* performance.
*/
static__always_inline__purebool_static_cpu_has(u16 bit)Ok, I guess something like that along with converting the obvious slow path callers to
boot_cpu_has():---
arch/x86/include/asm/cpufeature.h +3
-3@@ -155,9 +155,12 @@ extern void clear_cpu_cap(struct cpuinfo_x86 *c, unsigned int bit);#else /* -* Static testing of CPU features. Used the same as boot_cpu_has(). -* These will statically patch the target code for additional -* performance. +* Static testing of CPU features. Used the same as boot_cpu_has(). It +* statically patches the target code for additional performance. Use +* static_cpu_has() only in fast paths, where every cycle counts. Which
+ * means that theboot_cpu_has()variant is already fast enough for the
+ * majority of cases and you should stick to using it as it is generally
+ * only two instructions: a RIP-relative MOV and a TEST. */
static__always_inline__purebool_static_cpu_has(u16 bit){arch/x86/include/asm/fpu/internal.h +1
-1@@ -291,7 +291,7 @@ static inline void copy_xregs_to_kernel_booting(struct xregs_state *xstate) WARN_ON(system_state != SYSTEM_BOOTING); -if (static_cpu_has (X86_FEATURE_XSAVES )) +if (boot_cpu_has (X86_FEATURE_XSAVES ))XSTATE_OP (XSAVES , xstate, lmask, hmask, err);else XSTATE_OP(XSAVE, xstate, lmask, hmask, err);arch/x86/include/asm/fpu/internal.h +1
-1@@ -313,7 +313,7 @@ static inline void copy_kernel_to_xregs_booting(struct xregs_state *xstate) WARN_ON(system_state != SYSTEM_BOOTING); -if (static_cpu_has (X86_FEATURE_XSAVES )) +if (boot_cpu_has (X86_FEATURE_XSAVES ))XSTATE_OP (XRSTORS , xstate, lmask, hmask, err);else XSTATE_OP(XRSTOR, xstate, lmask, hmask, err);arch/x86/include/asm/fpu/internal.h +1
-2@@ -528,8 +528,7 @@ static inline void fpregs_activate(struct fpu *fpu)* - switch_fpu_finish() restores the new state as * necessary. */ -static inline void -switch_fpu_prepare (struct fpu *old_fpu,int cpu) +static inline void switch_fpu_prepare (struct fpu *old_fpu,int cpu) {if (static_cpu_has (X86_FEATURE_FPU ) && old_fpu->initialized) {if (!copy_fpregs_to_fpstate (old_fpu))arch/x86/kernel/apic/apic_numachip.c +1
-1@@ -175,7 +175,7 @@ static void fixup_cpu_id(struct cpuinfo_x86 *c, int node)this_cpu_write (cpu_llc_id ,node );/* Account for nodes per socket in multi-core-module processors */ -if (static_cpu_has (X86_FEATURE_NODEID_MSR )) { +if (boot_cpu_has (X86_FEATURE_NODEID_MSR )) {rdmsrl (MSR_FAM10H_NODE_ID , val); nodes = ((val >>3 ) &7 ) +1 ;}arch/x86/kernel/cpu/aperfmperf.c +3
-3@@ -83,7 +83,7 @@ unsigned int aperfmperf_get_khz(int cpu)if (!cpu_khz)return 0 ; -if (!static_cpu_has (X86_FEATURE_APERFMPERF )) +if (!boot_cpu_has (X86_FEATURE_APERFMPERF ))return 0 ;aperfmperf_snapshot_cpu (cpu,ktime_get (), true);@@ -99,7 +99,7 @@ void arch_freq_prepare_all(void)if (!cpu_khz)return ; -if (!static_cpu_has (X86_FEATURE_APERFMPERF )) +if (!boot_cpu_has (X86_FEATURE_APERFMPERF ))return ;for_each_online_cpu (cpu)@@ -115,7 +115,7 @@ unsigned int arch_freq_get_on_cpu(int cpu)if (!cpu_khz)return 0 ; -if (!static_cpu_has (X86_FEATURE_APERFMPERF )) +if (!boot_cpu_has (X86_FEATURE_APERFMPERF ))return 0 ;if (aperfmperf_snapshot_cpu (cpu,ktime_get (), true))arch/x86/kernel/cpu/common.c +1
-1@@ -1668,7 +1668,7 @@ static void setup_getcpu(int cpu)unsigned long cpudata =vdso_encode_cpunode (cpu,early_cpu_to_node (cpu));struct desc_struct d = { }; -if (static_cpu_has (X86_FEATURE_RDTSCP )) +if (boot_cpu_has (X86_FEATURE_RDTSCP ))write_rdtscp_aux (cpudata);/* Store CPU and node number in limit. */arch/x86/kernel/cpu/mce/inject.c +1
-1@@ -528,7 +528,7 @@ static void do_inject(void)* only on the node base core. Refer to D18F3x44[NbMcaToMstCpuEn] for * Fam10h and later BKDGs. */ -if (static_cpu_has (X86_FEATURE_AMD_DCM ) && +if (boot_cpu_has (X86_FEATURE_AMD_DCM ) && b ==4 && boot_cpu_data.x86 <0x17 ) {toggle_nb_mca_mst_cpu (amd_get_nb_id (cpu));arch/x86/kernel/cpu/proc.c +5
-5@@ -35,11 +35,11 @@ static void show_cpuinfo_misc(struct seq_file *m, struct cpuinfo_x86 *c)"fpu_exception\t: %s\n" "cpuid level\t: %d\n" "wp\t\t: yes\n" , -static_cpu_has_bug (X86_BUG_FDIV ) ?"yes" :"no" , -static_cpu_has_bug (X86_BUG_F00F ) ?"yes" :"no" , -static_cpu_has_bug (X86_BUG_COMA ) ?"yes" :"no" , -static_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , -static_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , +boot_cpu_has_bug (X86_BUG_FDIV ) ?"yes" :"no" , +boot_cpu_has_bug (X86_BUG_F00F ) ?"yes" :"no" , +boot_cpu_has_bug (X86_BUG_COMA ) ?"yes" :"no" , +boot_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , +boot_cpu_has (X86_FEATURE_FPU ) ?"yes" :"no" , c->cpuid_level); }#else
arch/x86/kernel/ldt.c +5
-5@@ -113,7 +113,7 @@ static void do_sanity_check(struct mm_struct *mm,* tables. */ WARN_ON (!had_kernel_mapping); -if (static_cpu_has (X86_FEATURE_PTI )) +if (boot_cpu_has (X86_FEATURE_PTI ))WARN_ON (!had_user_mapping); }else {/* @@ -121,7 +121,7 @@ static void do_sanity_check(struct mm_struct *mm,* Sync the pgd to the usermode tables. */ WARN_ON (had_kernel_mapping); -if (static_cpu_has (X86_FEATURE_PTI )) +if (boot_cpu_has (X86_FEATURE_PTI ))WARN_ON (had_user_mapping); } }@@ -156,7 +156,7 @@ static void map_ldt_struct_to_user(struct mm_struct *mm) k_pmd =pgd_to_pmd_walk (k_pgd,LDT_BASE_ADDR ); u_pmd =pgd_to_pmd_walk (u_pgd,LDT_BASE_ADDR ); -if (static_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt) +if (boot_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt)set_pmd (u_pmd, *k_pmd); }@@ -181,7 +181,7 @@ static void map_ldt_struct_to_user(struct mm_struct *mm) {pgd_t *pgd =pgd_offset (mm,LDT_BASE_ADDR ); -if (static_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt) +if (boot_cpu_has (X86_FEATURE_PTI ) && !mm->context.ldt)set_pgd (kernel_to_user_pgdp (pgd), *pgd); }@@ -208,7 +208,7 @@ map_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt, int slot) spinlock_t *ptl;int i, nr_pages; -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return 0 ;/*arch/x86/kernel/ldt.c +1
-1@@ -271,7 +271,7 @@ static void unmap_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt) return;/* LDT map/unmap is only required for PTI */ -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ;nr_pages = DIV_ROUND_UP(ldt->nr_entries * LDT_ENTRY_SIZE, PAGE_SIZE);arch/x86/kernel/ldt.c +1
-1@@ -311,7 +311,7 @@ static void free_ldt_pgtables(struct mm_struct *mm)unsigned long start =LDT_BASE_ADDR ;unsigned long end =LDT_END_ADDR ; -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ;tlb_gather_mmu (&tlb, mm, start, end);arch/x86/kernel/paravirt.c +1
-1@@ -121,7 +121,7 @@ DEFINE_STATIC_KEY_TRUE(virt_spin_lock_key);void __init native_pv_lock_init (void ) { -if (!static_cpu_has (X86_FEATURE_HYPERVISOR )) +if (!boot_cpu_has (X86_FEATURE_HYPERVISOR ))static_branch_disable (&virt_spin_lock_key); }arch/x86/kernel/process.c +0
-0@@ -236,7 +236,7 @@ static int get_cpuid_mode(void)static int
set_cpuid_mode(struct task_struct *task, unsigned long cpuid_enabled)
{arch/x86/kernel/process.c +1
-1-if (!static_cpu_has (X86_FEATURE_CPUID_FAULT )) +if (!boot_cpu_has (X86_FEATURE_CPUID_FAULT ))return -ENODEV; if (cpuid_enabled)arch/x86/kernel/process.c +1
-1@@ -666,7 +666,7 @@ static int prefer_mwait_c1_over_halt(const struct cpuinfo_x86 *c)if (c->x86_vendor !=X86_VENDOR_INTEL )return 0 ; -if (!cpu_has (c,X86_FEATURE_MWAIT ) || static_cpu_has_bug(X86_BUG_MONITOR)) + if (!cpu_has(c, X86_FEATURE_MWAIT) ||boot_cpu_has_bug (X86_BUG_MONITOR ))return 0; return 1;arch/x86/kernel/reboot.c +1
-1@@ -108,7 +108,7 @@ void __noreturn machine_real_restart(unsigned int type) write_cr3(real_mode_header->trampoline_pgd);/* Exiting long mode will fail if CR4.PCIDE is set. */ -if (static_cpu_has (X86_FEATURE_PCID )) +if (boot_cpu_has (X86_FEATURE_PCID ))cr4_clear_bits (X86_CR4_PCIDE );#endif arch/x86/kernel/vm86_32.c +1
-1@@ -369,7 +369,7 @@ static long do_sys_vm86(struct vm86plus_struct __user *user_vm86, bool plus)preempt_disable (); tsk->thread.sp0 +=16 ; -if (static_cpu_has (X86_FEATURE_SEP )) { +if (boot_cpu_has (X86_FEATURE_SEP )) { tsk->thread.sysenter_cs =0 ;refresh_sysenter_cs (&tsk->thread);}arch/x86/kvm/svm.c +4
-4@@ -835,7 +835,7 @@ static void svm_init_erratum_383(void)int err;u64 val; -if (!static_cpu_has_bug (X86_BUG_AMD_TLB_MMATCH )) +if (!boot_cpu_has_bug (X86_BUG_AMD_TLB_MMATCH ))return ;/* Use _safe variants to not break nested virtualization */ @@ -889,7 +889,7 @@ static int has_svm(void)static void svm_hardware_disable (void ) {/* Make sure we clean up behind us */ -if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) +if (boot_cpu_has (X86_FEATURE_TSCRATEMSR ))wrmsrl (MSR_AMD64_TSC_RATIO ,TSC_RATIO_DEFAULT );cpu_svm_disable ();@@ -931,7 +931,7 @@ static int svm_hardware_enable(void)wrmsrl (MSR_VM_HSAVE_PA ,page_to_pfn (sd->save_area) <<PAGE_SHIFT ); -if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) { +if (boot_cpu_has (X86_FEATURE_TSCRATEMSR )) {wrmsrl (MSR_AMD64_TSC_RATIO ,TSC_RATIO_DEFAULT );__this_cpu_write (current_tsc_ratio,TSC_RATIO_DEFAULT ); }@@ -2247,7 +2247,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu)for (i =0 ; i <NR_HOST_SAVE_USER_MSRS ; i++)rdmsrl (host_save_user_msrs[i], svm->host_user_msrs[i]); -if (static_cpu_has (X86_FEATURE_TSCRATEMSR )) { +if (boot_cpu_has (X86_FEATURE_TSCRATEMSR )) {u64 tsc_ratio = vcpu->arch.tsc_scaling_ratio;if (tsc_ratio !=__this_cpu_read (current_tsc_ratio)) {__this_cpu_write(current_tsc_ratio, tsc_ratio);arch/x86/kvm/svm.c +1
-1@@ -2255,7 +2255,7 @@ static void svm_vcpu_load(struct kvm_vcpu *vcpu, int cpu) } }/* This assumes that the kernel never uses MSR_TSC_AUX */ -if (static_cpu_has (X86_FEATURE_RDTSCP )) +if (boot_cpu_has (X86_FEATURE_RDTSCP ))wrmsrl (MSR_TSC_AUX , svm->tsc_aux);if (sd->current_vmcb != svm->vmcb) {arch/x86/kvm/vmx/vmx.c +2
-2@@ -6553,7 +6553,7 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu)if (vcpu->guest_debug &KVM_GUESTDBG_SINGLESTEP )vmx_set_interrupt_shadow (vcpu,0 ); -if (static_cpu_has (X86_FEATURE_PKU ) && +if (boot_cpu_has (X86_FEATURE_PKU ) &&kvm_read_cr4_bits (vcpu,X86_CR4_PKE ) && vcpu->arch.pkru != vmx->host_pkru)__write_pkru (vcpu->arch.pkru);@@ -6633,7 +6633,7 @@ static void vmx_vcpu_run(struct kvm_vcpu *vcpu)* back on host, so it is safe to read guest PKRU from current * XSAVE. */ -if (static_cpu_has (X86_FEATURE_PKU ) && +if (boot_cpu_has (X86_FEATURE_PKU ) &&kvm_read_cr4_bits (vcpu,X86_CR4_PKE )) { vcpu->arch.pkru =__read_pkru ();if (vcpu->arch.pkru != vmx->host_pkru)arch/x86/mm/dump_pagetables.c +0
-0@@ -579,7 +579,7 @@ void ptdump_walk_pgd_level(struct seq_file *m, pgd_t *pgd)void
ptdump_walk_pgd_level_debugfs(struct seq_file *m, pgd_t *pgd, bool user)
{
#ifdefCONFIG_PAGE_TABLE_ISOLATIONarch/x86/mm/dump_pagetables.c +1
-1-if (user &&static_cpu_has (X86_FEATURE_PTI )) +if (user &&boot_cpu_has (X86_FEATURE_PTI ))pgd = kernel_to_user_pgdp(pgd); #endif ptdump_walk_pgd_level_core(m, pgd, false, false);arch/x86/mm/dump_pagetables.c +1
-1@@ -592,7 +592,7 @@ void ptdump_walk_user_pgd_level_checkwx(void)pgd_t *pgd =INIT_PGD ;if (!(__supported_pte_mask & _PAGE_NX) || - !static_cpu_has (X86_FEATURE_PTI )) + !boot_cpu_has (X86_FEATURE_PTI ))return ;pr_info ("x86/mm: Checking user space page tables\n" );arch/x86/mm/pgtable.c +2
-2@@ -190,7 +190,7 @@ static void pgd_dtor(pgd_t *pgd)* when PTI is enabled. We need them to map the per-process LDT into the * user-space page-table. */ -#define PREALLOCATED_USER_PMDS (static_cpu_has(X86_FEATURE_PTI) ? \ +#define PREALLOCATED_USER_PMDS (boot_cpu_has(X86_FEATURE_PTI) ? \ KERNEL_PGD_PTRS : 0)#define MAX_PREALLOCATED_USER_PMDS KERNEL_PGD_PTRS@@ -292,7 +292,7 @@ static void pgd_mop_up_pmds(struct mm_struct *mm, pgd_t *pgdp)#ifdef CONFIG_PAGE_TABLE_ISOLATION -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ;pgdp = kernel_to_user_pgdp(pgdp);arch/x86/mm/pti.c +1
-1@@ -626,7 +626,7 @@ void pti_set_kernel_image_nonglobal(void)*/ void __init pti_init (void ) { -if (!static_cpu_has (X86_FEATURE_PTI )) +if (!boot_cpu_has (X86_FEATURE_PTI ))return ;pr_info ("enabled\n" );drivers/cpufreq/amd_freq_sensitivity.c +1
-1@@ -124,7 +124,7 @@ static int __init amd_freq_sensitivity_init(void) PCI_DEVICE_ID_AMD_KERNCZ_SMBUS, NULL);if (!pcidev) { -if (!static_cpu_has (X86_FEATURE_PROC_FEEDBACK )) +if (!boot_cpu_has (X86_FEATURE_PROC_FEEDBACK ))return -ENODEV ; }drivers/cpufreq/intel_pstate.c +1
-1@@ -520,7 +520,7 @@ static s16 intel_pstate_get_epb(struct cpudata *cpu_data)u64 epb;int ret; -if (!static_cpu_has (X86_FEATURE_EPB )) +if (!boot_cpu_has (X86_FEATURE_EPB ))return -ENXIO ;ret = rdmsrl_on_cpu(cpu_data->cpu, MSR_IA32_ENERGY_PERF_BIAS, &epb);drivers/cpufreq/intel_pstate.c +1
-1@@ -534,7 +534,7 @@ static s16 intel_pstate_get_epp(struct cpudata *cpu_data, u64 hwp_req_data) {s16 epp; -if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) {/* * When hwp_req_data is 0, means that caller didn't read *
MSR_HWP_REQUEST, so need to read and get EPP.drivers/cpufreq/intel_pstate.c +2
-2@@ -559,7 +559,7 @@ static int intel_pstate_set_epb(int cpu, s16 pref)u64 epb;int ret; -if (!static_cpu_has (X86_FEATURE_EPB )) +if (!boot_cpu_has (X86_FEATURE_EPB ))return -ENXIO ; ret =rdmsrl_on_cpu (cpu,MSR_IA32_ENERGY_PERF_BIAS , &epb);@@ -607,7 +607,7 @@ static int intel_pstate_get_energy_pref_index(struct cpudata *cpu_data) if (epp < 0) return epp; - if (static_cpu_has(X86_FEATURE_HWP_EPP)) { + if (boot_cpu_has(X86_FEATURE_HWP_EPP)) {if (epp ==HWP_EPP_PERFORMANCE )return 1 ;if (epp <=
HWP_EPP_BALANCE_PERFORMANCE)drivers/cpufreq/intel_pstate.c +1
-1@@ -616,7 +616,7 @@ static int intel_pstate_get_energy_pref_index(struct cpudata *cpu_data) return 3; else return 4; - }else if (static_cpu_has (X86_FEATURE_EPB )) { + }else if (boot_cpu_has (X86_FEATURE_EPB )) {/* * Range: * 0x00-0x03 : Performancedrivers/cpufreq/intel_pstate.c +1
-1@@ -644,7 +644,7 @@ static int intel_pstate_set_energy_pref_index(struct cpudata *cpu_data, mutex_lock(&intel_pstate_limits_lock); -if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) {u64 value;ret = rdmsrl_on_cpu(cpu_data->cpu, MSR_HWP_REQUEST, &value);drivers/cpufreq/intel_pstate.c +0
-0@@ -819,7 +819,7 @@ static void intel_pstate_hwp_set(unsigned int cpu) epp = cpu_data->epp_powersave; }
update_epp:drivers/cpufreq/intel_pstate.c +1
-1-if (static_cpu_has (X86_FEATURE_HWP_EPP )) { +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) {value &= ~GENMASK_ULL(31, 24); value |= (u64)epp << 24; } else {drivers/cpufreq/intel_pstate.c +1
-1@@ -844,7 +844,7 @@ static void intel_pstate_hwp_force_min_perf(int cpu) value |= HWP_MIN_PERF(min_perf);/* Set EPP/EPB to min */ -if (static_cpu_has (X86_FEATURE_HWP_EPP )) +if (boot_cpu_has (X86_FEATURE_HWP_EPP )) value |=HWP_ENERGY_PERF_PREFERENCE (HWP_EPP_POWERSAVE );else intel_pstate_set_epb(cpu, HWP_EPP_BALANCE_POWERSAVE);drivers/cpufreq/intel_pstate.c +1
-1@@ -1191,7 +1191,7 @@ static void __initintel_pstate_sysfs_expose_params (void ) static void intel_pstate_hwp_enable(struct cpudata *cpudata) {/* First disable HWP notification interrupt as we don't process them */ -if (static_cpu_has (X86_FEATURE_HWP_NOTIFY )) +if (boot_cpu_has (X86_FEATURE_HWP_NOTIFY ))wrmsrl_on_cpu (cpudata->cpu,MSR_HWP_INTERRUPT ,0x00 );wrmsrl_on_cpu(cpudata->cpu, MSR_PM_ENABLE, 0x1);drivers/cpufreq/powernow-k8.c +0
-0
drivers/cpufreq/powernow-k8.c static int powernowk8_init(void) · line 1184
Quoted 8 lines
drivers/cpufreq/powernow-k8.c +1
-1@@ -1178,7 +1178,7 @@ static int powernowk8_init(void)unsigned int i, supported_cpus =0 ;int ret; -if (static_cpu_has (X86_FEATURE_HW_PSTATE )) { +if (boot_cpu_has (X86_FEATURE_HW_PSTATE )) {__request_acpi_cpufreq ();return -ENODEV ; }
I'm confused here, and as I'm on my phone on an aircraft I can't check the back thread, but am I gathering that this tries to unbreak W^X during module loading by removing functions which use alternatives?
The right thing to do is to apply alternatives before the pages are marked +x-w, like we do with the kernel proper during early boot, if this isn't already the case (sorry again, see above.)
If we *do*, what is the issue here? Although boot_cpu_has() isn't slow (it should in general be possible to reduce to one testb instruction followed by a conditional jump) it seems that "avoiding an alternatives slot" *should* be a *very* weak reason, and seems to me to look like papering over some other problem.
Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 5 lines · On Thu, Mar 07, 2019 at 08:53:34AM -0800, [email protected] wrote:
If we *do*, what is the issue here? Although
boot_cpu_has()isn't slow (it should in general be possible to reduce to one testb instruction followed by a conditional jump) it seems that "avoiding an alternatives slot" *should* be a *very* weak reason, and seems to me to look like papering over some other problem.
Forget the current thread: this is simply trying to document when to use static_cpu_has() and when to use boot_cpu_has(). I get asked about it at least once a month.
And then it is replacing clear slow paths using static_cpu_has() with boot_cpu_has() because there's purely no need to patch there. And having a RIP-relative MOV and a JMP is good enough for slow paths.
Makes sense?
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Andy Lutomirski
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 8 lines · Borislav Petkov
Quoted 5 lines · On Thu, Mar 07, 2019 at 08:53:34AM -0800, [email protected] wrote:
If we *do*, what is the issue here? Although
boot_cpu_has()isn't slow (it should in general be possible to reduce to one testb instruction followed by a conditional jump) it seems that "avoiding an alternatives slot" *should* be a *very* weak reason, and seems to me to look like papering over some other problem.
Quoted 8 lines
Forget the current thread: this is simply trying to document when to use
static_cpu_has()and when to useboot_cpu_has(). I get asked about it at least once a month.And then it is replacing clear slow paths using
static_cpu_has()withboot_cpu_has()because there's purely no need to patch there. And having a RIP-relative MOV and a JMP is good enough for slow paths.
Should we maybe rename these functions? static_cpu_has() is at least reasonably obvious. But cpu_feature_enabled() is different for reasons I've never understood, and boot_cpu_has() is IMO terribly named. It's not about the boot cpu -- it's about doing the same thing but with less bloat and less performance.
(And can we maybe collapse cpu_feature_enabled() and static_cpu_has() into the same function?)
--Andy
Borislav Petkov
maintainerRe: [PATCH v2 10/20] x86: avoid W^X being broken during modules loading
Quoted 5 lines · Andy Lutomirski
Should we maybe rename these functions?
static_cpu_has()is at least reasonably obvious. Butcpu_feature_enabled()is different for reasons I've never understood, andboot_cpu_has()is IMO terribly named. It's not about the boot cpu -- it's about doing the same thing but with less bloat and less performance.
Well, it does test bits in boot_cpu_data. I don't care about "boot" in the name though so feel free to suggest something better.
Quoted 2 lines
(And can we maybe collapse
cpu_feature_enabled()andstatic_cpu_has()into the same function?)
I'm not sure it would be always ok to involve the DISABLED_MASK* buildtime stuff in the checks. It probably is but it would need careful auditing to be sure, first.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.PATCH v2 11/20 x86/jump-label: remove support for custom poker
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
There are only two types of poking: early and breakpoint based. The use of a function pointer to perform poking complicates the code and is probably inefficient due to the use of indirect branches.
--- arch/x86/kernel/jump_label.c | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-)
arch/x86/kernel/jump_label.c +8 -16
@@ -37,7 +37,6 @@ static void bug_at(unsigned char *ip, int line)static void __ref __jump_label_transform (struct jump_entry *entry,enum jump_label_type type, -void *(*poker)(void *,const void *,size_t ),int init) {union jump_code_union jmp;
@@ -50,14 +49,6 @@ static void __ref __jump_label_transform(struct jump_entry *entry, jmp.offset =jump_entry_target (entry) - (jump_entry_code (entry) +JUMP_LABEL_NOP_SIZE ); -/* -* As long as we're UP and not yet marked RO, we can use -* text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to -* SYSTEM_SCHEDULING before going either. -*/ -if (system_state ==SYSTEM_BOOTING ) - poker = text_poke_early; -if (type ==JUMP_LABEL_JMP ) {if (init) { expect = default_nop; line = __LINE__;
@@ -80,16 +71,17 @@ static void __ref __jump_label_transform(struct jump_entry *entry,bug_at ((void *)jump_entry_code (entry), line);/* -* Make text_poke_bp() a default fallback poker. +* As long as we're UP and not yet marked RO, we can use +* text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to +* SYSTEM_SCHEDULING before going either. * * At the time the change is being done, just ignore whether we * are doing nop -> jump or jump -> nop transition, and assume * always nop being the 'currently valid' instruction -* */ -if (poker) { - (*poker)((void *)jump_entry_code (entry), code, -JUMP_LABEL_NOP_SIZE ); +if (init || system_state ==SYSTEM_BOOTING ) { +text_poke_early ((void *)jump_entry_code (entry), code, +JUMP_LABEL_NOP_SIZE );return ; }
@@ -101,7 +93,7 @@ void arch_jump_label_transform(struct jump_entry *entry, enum jump_label_type type) {mutex_lock (&text_mutex); -__jump_label_transform (entry, type,NULL ,0 ); +__jump_label_transform (entry, type,0 );mutex_unlock (&text_mutex); }
@@ -131,5 +123,5 @@ __init_or_module void arch_jump_label_transform_static(struct jump_entry *entry, jlstate =JL_STATE_NO_UPDATE ; }if (jlstate ==JL_STATE_UPDATE ) -__jump_label_transform (entry, type, text_poke_early,1 ); +__jump_label_transform (entry, type,1 ); }
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 11/20] x86/jump-label: remove support for custom poker
Quoted 16 lines · Rick Edgecombe
From: Nadav Amit <[email protected]>
There are only two types of poking: early and breakpoint based. The use of a function pointer to perform poking complicates the code and is probably inefficient due to the use of indirect branches.
Cc Andy Lutomirski <[email protected]>Cc Kees Cook <[email protected]>Cc Dave Hansen <[email protected]>Cc Masami Hiramatsu <[email protected]>Signed-off-by Nadav Amit <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- arch/x86/kernel/jump_label.c | 24 ++++++++---------------- 1 file changed, 8 insertions(+), 16 deletions(-)
...
arch/x86/kernel/jump_label.c static void __ref __jump_label_transform(struct jump_entry *entry, · line 76
Quoted 8 lines
arch/x86/kernel/jump_label.c +3
-1@@ -80,16 +71,17 @@ static void __ref __jump_label_transform(struct jump_entry *entry,bug_at ((void *)jump_entry_code (entry), line);/* -* Make text_poke_bp() a default fallback poker. +* As long as we're UP and not yet marked RO, we can use +* text_poke_early; SYSTEM_BOOTING guarantees both, as we switch to +* SYSTEM_SCHEDULING before going either.
s/going/doing/ ?
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.PATCH v2 12/20 x86/alternative: Remove the return value of text_poke_*()
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
The return value of text_poke_early() and text_poke_bp() is useless. Remove it.
--- arch/x86/include/asm/text-patching.h | 4 ++-- arch/x86/kernel/alternative.c | 11 ++++------- 2 files changed, 6 insertions(+), 9 deletions(-)
arch/x86/include/asm/text-patching.h +2 -2
@@ -18,7 +18,7 @@ static inline void apply_paravirt(struct paravirt_patch_site *start,#define __parainstructions_end NULL#endif -extern void *text_poke_early (void *addr,const void *opcode,size_t len); +extern void text_poke_early (void *addr,const void *opcode,size_t len);/* * Clear and restore the kernel write-protection flag on the local CPU.
@@ -37,7 +37,7 @@ extern void *text_poke_early(void *addr, const void *opcode, size_t len);extern void *text_poke (void *addr,const void *opcode,size_t len);extern void *text_poke_kgdb (void *addr,const void *opcode,size_t len);extern int poke_int3_handler (struct pt_regs *regs); -extern void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler); +extern void text_poke_bp (void *addr,const void *opcode,size_t len,void *handler);extern int after_bootmem;extern __ro_after_init struct mm_struct *poking_mm;extern __ro_after_init unsigned long poking_addr;
arch/x86/kernel/alternative.c +4 -7
@@ -264,7 +264,7 @@ static void __init_or_module add_nops(void *insns, unsigned int len)extern struct alt_instr __alt_instructions[], __alt_instructions_end[];extern s32 __smp_locks[], __smp_locks_end[]; -void *text_poke_early (void *addr,const void *opcode,size_t len); +void text_poke_early (void *addr,const void *opcode,size_t len);/* * Are we looking at a near JMP with a 1 or 4-byte displacement.
@@ -666,8 +666,8 @@ void __init alternative_instructions(void)* instructions. And on the local CPU you need to be protected again NMI or MCE * handlers seeing an inconsistent instruction while you patch. */ -void *__init_or_module text_poke_early(void *addr, const void *opcode, -size_t len) +void __init_or_module text_poke_early(void *addr, const void *opcode, +size_t len) {unsigned long flags;
@@ -690,7 +690,6 @@ void *__init_or_module text_poke_early(void *addr, const void *opcode,* that causes hangs on some VIA CPUs. */ } -return addr; }__ro_after_init struct mm_struct *poking_mm;
@@ -890,7 +889,7 @@ int poke_int3_handler(struct pt_regs *regs)* replacing opcode * - sync cores */ -void *text_poke_bp (void *addr,const void *opcode,size_t len,void *handler) +void text_poke_bp (void *addr,const void *opcode,size_t len,void *handler) {unsigned char int3 =0xcc ;
@@ -932,7 +931,5 @@ void *text_poke_bp(void *addr, const void *opcode, size_t len, void *handler)* the writing of the new instruction. */ bp_patching_in_progress = false; - -return addr; }
Signature · 1 line
2.17.1
PATCH v2 13/20 Add set_alias_ function and x86 implementation
In reply to Rick EdgecombeThis adds two new functions set_alias_default_noflush and set_alias_nv_noflush for setting the alias mapping for the page to its default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions to not flush the TLB.
Note, __kernel_map_pages does something similar but flushes the TLB and doesn't reset the permission bits to default on all architectures.
There is also an ARCH config ARCH_HAS_SET_ALIAS for specifying whether these have an actual implementation or a default empty one.
--- arch/Kconfig | 4 ++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/set_memory.h | 3 +++ arch/x86/mm/pageattr.c | 14 +++++++++++--- include/linux/set_memory.h | 10 ++++++++++ 5 files changed, 29 insertions(+), 3 deletions(-)
arch/Kconfig +4 -0
@@ -249,6 +249,10 @@ config ARCH_HAS_FORTIFY_SOURCE config ARCH_HAS_SET_MEMORY bool +# Select if arch has all set_alias_nv/default() functions +config ARCH_HAS_SET_ALIAS + bool + # Select if arch init_task must go in the __init_task_data section config ARCH_TASK_STRUCT_ON_STACK bool
arch/x86/Kconfig +1 -0
@@ -66,6 +66,7 @@ config X86 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_UACCESS_MCSAFE if X86_64 && X86_MCE select ARCH_HAS_SET_MEMORY + select ARCH_HAS_SET_ALIAS select ARCH_HAS_STRICT_KERNEL_RWX select ARCH_HAS_STRICT_MODULE_RWX select ARCH_HAS_SYNC_CORE_BEFORE_USERMODE
arch/x86/include/asm/set_memory.h +3 -0
@@ -85,6 +85,9 @@ int set_pages_nx(struct page *page, int numpages);int set_pages_ro (struct page *page,int numpages);int set_pages_rw (struct page *page,int numpages); +int set_alias_nv_noflush (struct page *page); +int set_alias_default_noflush (struct page *page); +extern int kernel_set_to_readonly;void set_kernel_text_rw (void );void set_kernel_text_ro (void );
arch/x86/mm/pageattr.c +11 -3
@@ -2209,8 +2209,6 @@ int set_pages_rw(struct page *page, int numpages)return set_memory_rw (addr, numpages); } -#ifdef CONFIG_DEBUG_PAGEALLOC -static int __set_pages_p (struct page *page,int numpages) {unsigned long tempaddr = (unsigned long )page_address (page);
@@ -2249,6 +2247,17 @@ static int __set_pages_np(struct page *page, int numpages)return __change_page_attr_set_clr (&cpa,0 ); } +int set_alias_nv_noflush (struct page *page) +{ +return __set_pages_np (page,1 ); +} + +int set_alias_default_noflush (struct page *page) +{ +return __set_pages_p (page,1 ); +} + +#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages (struct page *page,int numpages,int enable) {if (PageHighMem (page))
@@ -2282,7 +2291,6 @@ void __kernel_map_pages(struct page *page, int numpages, int enable) }#ifdef CONFIG_HIBERNATION -bool kernel_page_present (struct page *page) {unsigned int level;
include/linux/set_memory.h +10 -0
@@ -10,6 +10,16 @@#ifdef CONFIG_ARCH_HAS_SET_MEMORY #include <asm/set_memory.h> +#ifndef CONFIG_ARCH_HAS_SET_ALIAS +static inline int set_alias_nv_noflush (struct page *page) +{ +return 0 ; +} +static inline int set_alias_default_noflush (struct page *page) +{ +return 0 ; +} +#endif #else static inline int set_memory_ro (unsigned long addr,int numpages) {return 0 ; }static inline int set_memory_rw (unsigned long addr,int numpages) {return 0 ; }
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 13/20] Add set_alias_ function and x86 implementation
Quoted 1 line · Rick Edgecombe
This adds two new functions
set_alias_default_noflushand
s/This adds/Add/
Quoted 1 line
set_alias_nv_noflushfor setting the alias mapping for the page to its
Please end function names with parentheses, below too.
Quoted 2 lines
default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions to not flush the TLB.
s/to/do/
Also, pls put that description as comments over the functions in the code. Otherwise that "nv" as part of the name doesn't really explain what it does.
Actually, you could just as well call the function
set_alias_invalid_noflush()
All the other words are written in full, no need to have "nv" there.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 13/20] Add set_alias_ function and x86 implementation
Quoted 8 lines · Borislav Petkov
Quoted 1 line · Rick Edgecombe
This adds two new functions
set_alias_default_noflushands/This adds/Add/
Quoted 1 line
set_alias_nv_noflushfor setting the alias mapping for the page to itsPlease end function names with parentheses, below too.
Ok.
Quoted 5 lines
Quoted 2 lines
default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions to not flush the TLB.
s/to/do/
Argh, thanks.
Quoted 11 lines
Also, pls put that description as comments over the functions in the code. Otherwise that "nv" as part of the name doesn't really explain what it does.
Actually, you could just as well call the function
set_alias_invalid_noflush()All the other words are written in full, no need to have "nv" there.
Thx.
Yes, that seems better.
Thanks,
Rick
Andy Lutomirski
maintainerRe: [PATCH v2 13/20] Add set_alias_ function and x86 implementation
In reply to Borislav PetkovQuoted 19 lines · Borislav Petkov
Quoted 1 line · Rick Edgecombe
This adds two new functions
set_alias_default_noflushands/This adds/Add/
Quoted 1 line
set_alias_nv_noflushfor setting the alias mapping for the page to itsPlease end function names with parentheses, below too.
Quoted 2 lines
default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions to not flush the TLB.
s/to/do/
Also, pls put that description as comments over the functions in the code. Otherwise that "nv" as part of the name doesn't really explain what it does.
Quoted 5 lines
Actually, you could just as well call the function
set_alias_invalid_noflush()All the other words are written in full, no need to have "nv" there.
Why are you calling this an "alias"? You're modifying the direct map. Your patches are thinking of the direct map as an alias of the vmap mapping, but that does seem a bit backwards. How about set_direct_map_invalid_noflush(), etc?
--Andy
Re: [PATCH v2 13/20] Add set_alias_ function and x86 implementation
Quoted 22 lines · Andy Lutomirski
Quoted 21 lines · Borislav Petkov
Quoted 1 line · Rick Edgecombe
This adds two new functions
set_alias_default_noflushands/This adds/Add/
Quoted 1 line
set_alias_nv_noflushfor setting the alias mapping for the page to itsPlease end function names with parentheses, below too.
Quoted 2 lines
default valid permissions and to an invalid state that cannot be cached in a TLB, respectively. These functions to not flush the TLB.
s/to/do/
Also, pls put that description as comments over the functions in the code. Otherwise that "nv" as part of the name doesn't really explain what it does.
Actually, you could just as well call the function
Quoted 9 lines
Quoted 3 lines
set_alias_invalid_noflush()All the other words are written in full, no need to have "nv" there.
Why are you calling this an "alias"? You're modifying the direct map. Your patches are thinking of the direct map as an alias of the vmap mapping, but that does seem a bit backwards. How about
set_direct_map_invalid_noflush(), etc?
I picked it up from some of the names in arch/x86/mm/pageattr.c:CPA_NO_CHECK_ALIAS, set_memory_np_noalias(), etc. In that file the directmap address seems to be the "alias". For 32 bit with highmem though, this would also set permissions for a kmap mapping as well (if one existed), since that address will be returned from page_address().
Yea, in vmalloc, vm_unmap_aliases talks about the vmap address "alias". So I guess calling it "alias" is ambiguous. But does set_direct_map_invalid_noflush make sense in the highmem case?
I couldn't think of any names that I loved, which is why I ran the set_alias_*_noflush names by people in an earlier version, although looking back only Ard chimed in on that. "set_direct_map_invalid_noflush" is fine with me if nobody objects.
Thanks,
Rick
PATCH v2 14/20 mm: Make hibernate handle unmapped pages
In reply to Rick EdgecombeFor architectures with CONFIG_ARCH_HAS_SET_ALIAS, pages can be unmapped briefly on the directmap, even when CONFIG_DEBUG_PAGEALLOC is not configured. So this changes kernel_map_pages and kernel_page_present to be defined when CONFIG_ARCH_HAS_SET_ALIAS is defined as well. It also changes places (page_alloc.c) where those functions are assumed to only be implemented when CONFIG_DEBUG_PAGEALLOC is defined.
So now when CONFIG_ARCH_HAS_SET_ALIAS=y, hibernate will handle not present page when saving. Previously this was already done when CONFIG_DEBUG_PAGEALLOC was configured. It does not appear to have a big hibernating performance impact.
Before:
[ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s)
After:
[ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s)
--- arch/x86/mm/pageattr.c | 4 ---- include/linux/mm.h | 18 ++++++------------ mm/page_alloc.c | 7 +++++-- 3 files changed, 11 insertions(+), 18 deletions(-)
arch/x86/mm/pageattr.c +0 -4
@@ -2257,7 +2257,6 @@ int set_alias_default_noflush(struct page *page)return __set_pages_p (page,1 ); } -#ifdef CONFIG_DEBUG_PAGEALLOC void __kernel_map_pages (struct page *page,int numpages,int enable) {if (PageHighMem (page))
@@ -2302,11 +2301,8 @@ bool kernel_page_present(struct page *page) pte =lookup_address ((unsigned long )page_address (page), &level); return (pte_val(*pte) & _PAGE_PRESENT); } -#endif /* CONFIG_HIBERNATION */ -#endif /* CONFIG_DEBUG_PAGEALLOC */ -int __init kernel_map_pages_in_pgd (pgd_t *pgd,u64 pfn,unsigned long address,unsigned numpages,unsigned long page_flags) {
include/linux/mm.h +6 -12
@@ -2642,37 +2642,31 @@ static inline void kernel_poison_pages(struct page *page, int numpages, int enable) { }#endif -#ifdef CONFIG_DEBUG_PAGEALLOC extern bool _debug_pagealloc_enabled; -extern void __kernel_map_pages (struct page *page,int numpages,int enable);static inline bool debug_pagealloc_enabled (void ) { -return _debug_pagealloc_enabled; +return IS_ENABLED (CONFIG_DEBUG_PAGEALLOC ) && _debug_pagealloc_enabled; } +#if defined(CONFIG_DEBUG_PAGEALLOC ) || defined(CONFIG_ARCH_HAS_SET_ALIAS ) +extern void __kernel_map_pages (struct page *page,int numpages,int enable); +static inline void kernel_map_pages (struct page *page,int numpages,int enable) { -if (!debug_pagealloc_enabled ()) -return ; -__kernel_map_pages (page, numpages, enable); }#ifdef CONFIG_HIBERNATION extern bool kernel_page_present (struct page *page);#endif /* CONFIG_HIBERNATION */ -#else /* CONFIG_DEBUG_PAGEALLOC */ +#else /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_ALIAS */ static inline void kernel_map_pages (struct page *page,int numpages,int enable) {}#ifdef CONFIG_HIBERNATION static inline bool kernel_page_present (struct page *page) {return true; }#endif /* CONFIG_HIBERNATION */ -static inline bool debug_pagealloc_enabled (void ) -{ -return false; -} -#endif /* CONFIG_DEBUG_PAGEALLOC */ +#endif /* CONFIG_DEBUG_PAGEALLOC || CONFIG_ARCH_HAS_SET_ALIAS */ #ifdef __HAVE_ARCH_GATE_AREAextern struct vm_area_struct *get_gate_vma (struct mm_struct *mm);
mm/page_alloc.c +5 -2
@@ -1074,7 +1074,9 @@ static __always_inline bool free_pages_prepare(struct page *page, }arch_free_page (page, order);kernel_poison_pages (page,1 << order,0 ); -kernel_map_pages (page,1 << order,0 ); +if (debug_pagealloc_enabled ()) +kernel_map_pages (page,1 << order,0 ); +kasan_free_nondeferred_pages (page, order);return true;
@@ -1944,7 +1946,8 @@ inline void post_alloc_hook(struct page *page, unsigned int order,set_page_refcounted (page);arch_alloc_page (page, order); -kernel_map_pages (page,1 << order,1 ); +if (debug_pagealloc_enabled ()) +kernel_map_pages (page,1 << order,1 );kernel_poison_pages (page,1 << order,1 );kasan_alloc_pages (page, order);set_page_owner (page, order, gfp_flags);
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 14/20] mm: Make hibernate handle unmapped pages
Quoted 3 lines · Rick Edgecombe
For architectures with
CONFIG_ARCH_HAS_SET_ALIAS, pages can be unmapped briefly on the directmap, even whenCONFIG_DEBUG_PAGEALLOCis not configured. So this changeskernel_map_pagesandkernel_page_presentto be
s/this changes/change/
Quoted 1 line
From
Documentation/process/submitting-patches.rst:
"Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."
Also, please end function names with parentheses.
Quoted 3 lines
defined when
CONFIG_ARCH_HAS_SET_ALIASis defined as well. It also changes places (page_alloc.c) where those functions are assumed to only be implemented whenCONFIG_DEBUG_PAGEALLOCis defined.
The commit message doesn't need to say "what" you're doing - that should be obvious from the diff below. It should rather say "why" you're doing it.
Quoted 2 lines
So now when
CONFIG_ARCH_HAS_SET_ALIAS=y, hibernate will handle not present page when saving. Previously this was already done when
pages
Quoted 2 lines
CONFIG_DEBUG_PAGEALLOCwas configured. It does not appear to have a big hibernating performance impact.
Comment over safe_copy_page() needs updating I guess.
Quoted 5 lines
Before:
[ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s)After:
[ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s)
IINM, that's like 1734 pages more. How am I to understand this number?
Code has called set_alias_nv_noflush() on them and safe_copy_page() now maps them one by one to copy them to the hibernation image?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 14/20] mm: Make hibernate handle unmapped pages
Quoted 7 lines · Borislav Petkov
Quoted 3 lines · Rick Edgecombe
For architectures with
CONFIG_ARCH_HAS_SET_ALIAS, pages can be unmapped briefly on the directmap, even whenCONFIG_DEBUG_PAGEALLOCis not configured. So this changeskernel_map_pagesandkernel_page_presentto bes/this changes/change/
Quoted 8 lines
From
Documentation/process/submitting-patches.rst:"Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."Also, please end function names with parentheses.
Yes, gotcha.
Quoted 7 lines
Quoted 3 lines
defined when
CONFIG_ARCH_HAS_SET_ALIASis defined as well. It also changes places (page_alloc.c) where those functions are assumed to only be implemented whenCONFIG_DEBUG_PAGEALLOCis defined.The commit message doesn't need to say "what" you're doing - that should be obvious from the diff below. It should rather say "why" you're doing it.
Ok, sorry. I'll change this to be more concise.
Quoted 9 lines
Quoted 2 lines
So now when
CONFIG_ARCH_HAS_SET_ALIAS=y, hibernate will handle not present page when saving. Previously this was already done whenpages
Quoted 2 lines
CONFIG_DEBUG_PAGEALLOCwas configured. It does not appear to have a big hibernating performance impact.Comment over
safe_copy_page
Oh, yes you are right.
Quoted 6 lines
Quoted 5 lines
Before:
[ 4.670938] PM: Wrote 171996 kbytes in 0.21 seconds (819.02 MB/s)After:
[ 4.504714] PM: Wrote 178932 kbytes in 0.22 seconds (813.32 MB/s)
Quoted 7 lines
IINM, that's like 1734 pages more. How am I to understand this number?
Code has called
set_alias_nv_noflush()on them andsafe_copy_page()now maps them one by one to copy them to the hibernation image?Thx.
These are from logs hibernate generates. The concern was that hibernate could be slightly slower because of the checking of whether the pages are mapped. The bandwidth number can be used to compare, 819.02->813.32 MB/s. Some randomness must have resulted in different amounts of memory used between tests. I can just remove the log lines and include the bandwidth numbers.
Thanks,
Rick
Borislav Petkov
maintainerRe: [PATCH v2 14/20] mm: Make hibernate handle unmapped pages
Quoted 5 lines · Rick P
These are from logs hibernate generates. The concern was that hibernate could be slightly slower because of the checking of whether the pages are mapped. The bandwidth number can be used to compare, 819.02->813.32 MB/s. Some randomness must have resulted in different amounts of memory used between tests. I can just remove the log lines and include the bandwidth numbers.
Nah, I'm just trying to get an idea of the slowdown it would cause. I'm thinking these pages are, as you call them "special" so they should not be a huge chunk of all the system's pages, even if it is a machine with a lot of memory so I guess it ain't that bad. We should keep an eye on it though... :-)
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.PATCH v2 15/20 vmalloc: New flags for safe vfree on special perms
In reply to Rick EdgecombeThis adds a new flags VM_HAS_SPECIAL_PERMS, for enabling vfree operations to immediately clear executable TLB entries to freed pages, and handle freeing memory with special permissions. It also takes care of resetting the direct map permissions for the pages being unmapped. So this flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory.
Although this enables directly vfreeing RO memory now, RO memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list handling was added. Today there is only one case where this happens.
For architectures with set_alias_ implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method using set_memory functions is used to reset the directmap. When arm64 adds set_alias_ functions, this backup can be removed.
When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today vm_unmap_aliases could flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normal vm_unmap_aliases function is unchanged.
--- include/linux/vmalloc.h | 13 +++++ mm/vmalloc.c | 122 +++++++++++++++++++++++++++++++++------- 2 files changed, 116 insertions(+), 19 deletions(-)
include/linux/vmalloc.h +13 -0
@@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */#define VM_UNINITIALIZED 0x00000020/* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040/* don't add guard page */ #define VM_KASAN 0x00000080/* has allocated kasan shadow memory */ +/* +* Memory with VM_HAS_SPECIAL_PERMS cannot be freed in an interrupt or with +* vfree_atomic. +*/ +#define VM_HAS_SPECIAL_PERMS 0x00000200/* Reset directmap and flush TLB on unmap */ /* bits [20..32] reserved for arch specific ioremap internals */ /*
@@ -135,6 +140,14 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size,extern struct vm_struct *remove_vm_area (const void *addr);extern struct vm_struct *find_vm_area (const void *addr); +static inline void set_vm_special (void *addr) +{ +struct vm_struct *vm =find_vm_area (addr); + +if (vm) + vm->flags |=VM_HAS_SPECIAL_PERMS ; +} +extern int map_vm_area (struct vm_struct *area,pgprot_t prot,struct page **pages);#ifdef CONFIG_MMU
mm/vmalloc.c +103 -19
160 lines not on this page · raw message
Signature · 1 line
2.17.1
Borislav Petkov
maintainerRe: [PATCH v2 15/20] vmalloc: New flags for safe vfree on special perms
Quoted 1 line · Rick Edgecombe
This adds a new flags
VM_HAS_SPECIAL_PERMS, for enabling vfree operations
s/This adds/add/ - you get the idea. :)
s/flags/flag/
Quoted 42 lines
to immediately clear executable TLB entries to freed pages, and handle freeing memory with special permissions. It also takes care of resetting the direct map permissions for the pages being unmapped. So this flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory.
Although this enables directly vfreeing RO memory now, RO memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list handling was added. Today there is only one case where this happens.
For architectures with
set_alias_implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method usingset_memoryfunctions is used to reset the directmap. When arm64 addsset_alias_functions, this backup can be removed.When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today
vm_unmap_aliasescould flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normalvm_unmap_aliasesfunction is unchanged.Suggested-by Dave Hansen <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Suggested-by Will Deacon <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- include/linux/vmalloc.h | 13 +++++ mm/vmalloc.c | 122 +++++++++++++++++++++++++++++++++------- 2 files changed, 116 insertions(+), 19 deletions(-)include/linux/vmalloc.h +0
-0
include/linux/vmalloc.h struct notifier_block; /* in notifier.h */ · line 26
Quoted 6 lines
include/linux/vmalloc.h +3
-0@@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */#define VM_UNINITIALIZED 0x00000020/* vm_struct is not fully initialized */ #define VM_NO_GUARD 0x00000040/* don't add guard page */ #define VM_KASAN 0x00000080/* has allocated kasan shadow memory */ +/* +* Memory with VM_HAS_SPECIAL_PERMS cannot be freed in an interrupt or with +* vfree_atomic.
Please end function names with parentheses. You should go over the whole patchset - there are a bunch of places.
include/linux/vmalloc.h struct notifier_block; /* in notifier.h */ · line 28
Quoted 2 lines
include/linux/vmalloc.h +2
-0+*/ +#define VM_HAS_SPECIAL_PERMS 0x00000200/* Reset directmap and flush TLB on unmap */
After 0x00000080 comes 0x00000100. 0x00000010 is free too. What's up?
include/linux/vmalloc.h extern struct vm_struct *__get_vm_area_caller(unsigned long size, · line 143
Quoted 8 lines
include/linux/vmalloc.h +1
-0/* bits [20..32] reserved for arch specific ioremap internals */ /* @@ -135,6 +140,14 @@ extern struct vm_struct *__get_vm_area_caller(unsigned long size,extern struct vm_struct *remove_vm_area (const void *addr);extern struct vm_struct *find_vm_area (const void *addr); +static inline void set_vm_special (void *addr)
You need a different name than "special" for a vm which needs to flush and clear mapping perms on removal. VM_RESET_PERMS or whatever is more to the point than "special", for example, which could mean a lot of things.
Quoted 32 lines
include/linux/vmalloc.h +7
-0+{ +struct vm_struct *vm =find_vm_area (addr); + +if (vm) + vm->flags |=VM_HAS_SPECIAL_PERMS ; +} +extern int map_vm_area (struct vm_struct *area,pgprot_t prot,struct page **pages);#ifdef CONFIG_MMU mm/vmalloc.c +1
-6@@ -18,6 +18,7 @@#include <linux/interrupt.h> #include <linux/proc_fs.h> #include <linux/seq_file.h> +#include <linux/set_memory.h> #include <linux/debugobjects.h> #include <linux/kallsyms.h> #include <linux/list.h> @@ -1055,24 +1056,11 @@ static void vb_free(const void *addr, unsigned long size)spin_unlock (&vb->lock); } -/** -* vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer -* -* The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily -* to amortize TLB flushing overheads. What this means is that any page you -* have now, may, in a former life, have been mapped into kernel virtual
mm/vmalloc.c static void vb_free(const void *addr, unsigned long size) · line 1060
Quoted 10 lines
mm/vmalloc.c +2
-8-* address by the vmap layer and so there might be some CPUs with TLB entries -* still referencing that page (additional to the regular 1:1 kernel mapping). -* -* vm_unmap_aliases flushes all such lazy mappings. After it returns, we can -* be sure that none of the pages we have control over will have any aliases -* from the vmap layer. -*/ -void vm_unmap_aliases (void ) +static void _vm_unmap_aliases (unsigned long start,unsigned long end, +int must_flush)
Align arguments on the opening brace. There's more places below, pls fix them all.
mm/vmalloc.c static void vb_free(const void *addr, unsigned long size) · line 1063
Quoted 5 lines
mm/vmalloc.c +1
-2{ -unsigned long start =ULONG_MAX , end =0 ;int cpu; -int flush =0 ; +int flush = must_flush;
You can't use must_flush directly because...?
gcc produces the same asm here, with or without the local "flush" var.
Quoted 36 lines
mm/vmalloc.c +22
-0if (unlikely (!vmap_initialized))return ;@@ -1109,6 +1097,27 @@ void vm_unmap_aliases(void)flush_tlb_kernel_range (start, end);mutex_unlock (&vmap_purge_lock); } + +/** +* vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer +* +* The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily +* to amortize TLB flushing overheads. What this means is that any page you +* have now, may, in a former life, have been mapped into kernel virtual +* address by the vmap layer and so there might be some CPUs with TLB entries +* still referencing that page (additional to the regular 1:1 kernel mapping). +* +* vm_unmap_aliases flushes all such lazy mappings. After it returns, we can +* be sure that none of the pages we have control over will have any aliases +* from the vmap layer. +*/ +void vm_unmap_aliases (void ) +{ +unsigned long start =ULONG_MAX , end =0 ; +int must_flush =0 ; + +_vm_unmap_aliases (start, end, must_flush); +}EXPORT_SYMBOL_GPL (vm_unmap_aliases);/** @@ -1494,6 +1503,79 @@ struct vm_struct *remove_vm_area(const void *addr)return NULL ; } +static inline void set_area_alias(const struct vm_struct *area,
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1516
Quoted 10 lines
mm/vmalloc.c +10
-0+ int (*set_alias)(struct page *page)) +{ +int i; + +for (i =0 ; i < area->nr_pages; i++) { +unsigned long addr = + (unsigned long )page_address (area->pages[i]); + +if (addr) +set_alias (area->pages[i]);
What's wrong with simply:
for (i = 0; i < area->nr_pages; i++) {
if (page_address(area->pages[i]))
set_alias(area->pages[i]);
}?
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1520
Quoted 4 lines
mm/vmalloc.c +4
-0+ } +} + +/* This handles removing and resetting vm mappings related to the vm_struct. */
s/This handles/Handle/
Quoted 6 lines
mm/vmalloc.c +6
-0+static void vm_remove_mappings (struct vm_struct *area,int deallocate_pages) +{ +unsigned long addr = (unsigned long )area->addr; +unsigned long start =ULONG_MAX , end =0 ; +int special = area->flags &VM_HAS_SPECIAL_PERMS ; +int i;
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1536
Quoted 10 lines
mm/vmalloc.c +10
-0+ +/* +* The below block can be removed when all architectures that have +* direct map permissions also have set_alias_ implementations. This is +* to do resetting on the directmap for any special permissions (today +* only X), without leaving a RW+X window. +*/ +if (special && !IS_ENABLED (CONFIG_ARCH_HAS_SET_ALIAS )) { +set_memory_nx (addr, area->nr_pages); +set_memory_rw (addr, area->nr_pages);
That's two not very cheap calls to the underlying worker function, for example change_memory_common() on ARM64, instead of calling it once with the respective flags. You allude to that in the commit message but you might wanna run it by ARM folks first.
Quoted 17 lines
mm/vmalloc.c +17
-0+ } + +remove_vm_area (area->addr); + +/* If this is not special memory, we can skip the below. */ +if (!special) +return ; + +/* +* If we are not deallocating pages, we can just do the flush of the VM +* area and return. +*/ +if (!deallocate_pages) { +vm_unmap_aliases (); +return ; + } +
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1563
Quoted 10 lines
mm/vmalloc.c +10
-0+/* +* If we are here, we need to flush the vm mapping and reset the direct +* map. +* First find the start and end range of the direct mappings to make +* sure the vm_unmap_aliases flush includes the direct map. +*/ +for (i =0 ; i < area->nr_pages; i++) { +unsigned long addr = + (unsigned long )page_address (area->pages[i]); +if (addr) {
if (page_address(area->pages[i]))
as above.
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1570
Quoted 7 lines
mm/vmalloc.c +7
-0+ start =min (addr, start); + end =max (addr, end); + } + } + +/* +* First we set direct map to something not valid so that it won't be
Above comment says "First" too. In general, all those "we" formulations do not make the comments as easy to read as when you make them impersonal and imperative:
/* * Set the direct map to something invalid...
Just like Documentation/process/submitting-patches.rst says:
"Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."
you simply order your code to do stuff. :-)
Quoted 11 lines
mm/vmalloc.c +8
-0+* cached if there are any accesses after the TLB flush, then we flush +* the TLB, and reset the directmap permissions to the default. +*/ +set_area_alias (area, set_alias_nv_noflush); +_vm_unmap_aliases (start, end,1 ); +set_area_alias (area, set_alias_default_noflush); +} +static void __vunmap (const void *addr,int deallocate_pages) {struct vm_struct *area;
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.Re: [PATCH v2 15/20] vmalloc: New flags for safe vfree on special perms
Thanks Boris.
Ard, Will: An arm question came up below. Any thoughts?
Quoted 4 lines · Borislav Petkov
Quoted 1 line · Rick Edgecombe
This adds a new flags
VM_HAS_SPECIAL_PERMS, for enabling vfree operationss/This adds/add/ - you get the idea. :)
Yes, thanks. Fixed after your comments on other patches.
Quoted 48 lines
s/flags/flag/
Quoted 46 lines
to immediately clear executable TLB entries to freed pages, and handle freeing memory with special permissions. It also takes care of resetting the direct map permissions for the pages being unmapped. So this flag is useful for any kind of memory with elevated permissions, or where there can be related permissions changes on the directmap. Today this is RO+X and RO memory.
Although this enables directly vfreeing RO memory now, RO memory cannot be freed in an interrupt because the allocation itself is used as a node on deferred free list. So when RO memory needs to be freed in an interrupt the code doing the vfree needs to have its own work queue, as was the case before the deferred vfree list handling was added. Today there is only one case where this happens.
For architectures with
set_alias_implementations this whole operation can be done with one TLB flush when centralized like this. For others with directmap permissions, currently only arm64, a backup method usingset_memoryfunctions is used to reset the directmap. When arm64 addsset_alias_functions, this backup can be removed.When the TLB is flushed to both remove TLB entries for the vmalloc range mapping and the direct map permissions, the lazy purge operation could be done to try to save a TLB flush later. However today
vm_unmap_aliasescould flush a TLB range that does not include the directmap. So a helper is added with extra parameters that can allow both the vmalloc address and the direct mapping to be flushed during this operation. The behavior of the normalvm_unmap_aliasesfunction is unchanged.Suggested-by Dave Hansen <[email protected]>Suggested-by Andy Lutomirski <[email protected]>Suggested-by Will Deacon <[email protected]>Signed-off-by Rick Edgecombe <[email protected]>--- include/linux/vmalloc.h | 13 +++++ mm/vmalloc.c | 122 +++++++++++++++++++++++++++++++++------- 2 files changed, 116 insertions(+), 19 deletions(-)include/linux/vmalloc.h +0
-0@@ -21,6 +21,11 @@ struct notifier_block; /* in notifier.h */#define VM_UNINITIALIZED 0x00000020/* vm_struct is not fully initialized */
#defineVM_NO_GUARD0x00000040/* don't add guard page
*/
Quoted 8 lines
Quoted 5 lines
#define
VM_KASAN0x00000080/* has allocated kasan shadow
memory */
+/*
+ * Memory withVM_HAS_SPECIAL_PERMScannot be freed in an interrupt or with
+ *vfree_atomic.Please end function names with parentheses. You should go over the whole patchset - there are a bunch of places.
Ok.
Quoted 5 lines
Quoted 3 lines
+ */
+#defineVM_HAS_SPECIAL_PERMS0x00000200/* Reset directmap and
flush TLB on unmap */After
0x00000080comes0x00000100.0x00000010is free too. What's up?
I was just trying to follow the pattern noticed from the gap at 0x00000010. I'll add it at 0x00000100 in case there is some reason for the other gap.
Quoted 8 lines
Quoted 8 lines
/* bits [20..32] reserved for arch specific ioremap internals */
/*
include/linux/vmalloc.h +0
-0@@ -135,6 +140,14 @@ extern struct vm_struct *__get_vm_area_caller(unsignedlong size, extern struct vm_struct *remove_vm_area(const void *addr);extern struct vm_struct *find_vm_area (const void *addr);
Quoted 6 lines
include/linux/vmalloc.h extern struct vm_struct *__get_vm_area_caller(unsigned long size, · line 143
Quoted 1 line
include/linux/vmalloc.h +1
-0+static inline void set_vm_special (void *addr)You need a different name than "special" for a vm which needs to flush and clear mapping perms on removal.
VM_RESET_PERMSor whatever is more to the point than "special", for example, which could mean a lot of things.
I don't have a strong opinion about the name, but it has gone through some revisions, so I'll summarize the history.
There are two intentions - not leaving a stale TLB entry to a freed page, and then resetting the direct map for architectures that need it. Andy had pointed out that you can do this with only one TLB flush if all the cleanup is centralized here. The original point of this flag was the TLB flush of the vmalloc alias, and it's something that makes sense for all architectures that have something like NX, regardless of what they are doing with their direct map.
At one point there were two flags, one for "immediate flush" and one for "reset direct map", but Andy had also pointed out that you would mostly use them together, so why complicate things.
So to capture both of those intentions, maybe I'll slightly tweak your suggestion to VM_FLUSH_RESET_PERMS?
Quoted 42 lines
Quoted 42 lines
include/linux/vmalloc.h +7
-0+{ +struct vm_struct *vm =find_vm_area (addr); + +if (vm) + vm->flags |=VM_HAS_SPECIAL_PERMS ; +} +extern int map_vm_area (struct vm_struct *area,pgprot_t prot,struct page **pages);#ifdef
CONFIG_MMUmm/vmalloc.c +1
-7@@ -18,6 +18,7 @@#include <linux/interrupt.h> #include <linux/proc_fs.h> #include <linux/seq_file.h> +#include <linux/set_memory.h> #include <linux/debugobjects.h> #include <linux/kallsyms.h> #include <linux/list.h> @@ -1055,24 +1056,11 @@ static void vb_free(const void *addr, unsigned long size) spin_unlock(&vb->lock); } -/** -* vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer -* -* The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily -* to amortize TLB flushing overheads. What this means is that any page you -* have now, may, in a former life, have been mapped into kernel virtual -* address by the vmap layer and so there might be some CPUs with TLB entries
- * still referencing that page (additional to the regular 1:1 kernel mapping).
- *
- *vm_unmap_aliasesflushes all such lazy mappings. After it returns, we can
- * be sure that none of the pages we have control over will have any aliases
Quoted 8 lines
mm/vmalloc.c static void vb_free(const void *addr, unsigned long size) · line 1060
Quoted 5 lines
- * from the vmap layer.
- */mm/vmalloc.c +2
-1-void vm_unmap_aliases (void ) +static void _vm_unmap_aliases (unsigned long start,unsigned long end, +int must_flush)Align arguments on the opening brace. There's more places below, pls fix them all.
Ok.
Quoted 9 lines
mm/vmalloc.c static void vb_free(const void *addr, unsigned long size) · line 1063
Quoted 5 lines
{
mm/vmalloc.c +1
-2-unsigned long start =ULONG_MAX , end =0 ;int cpu; -int flush =0 ; +int flush = must_flush;You can't use
must_flushdirectly because...?gcc produces the same asm here, with or without the local "flush" var.
I had thought it was easier to read. If its not the case, I'll change it as you suggest.
Quoted 53 lines
Quoted 40 lines
mm/vmalloc.c +8
-0if (unlikely (!vmap_initialized))return ;@@ -1109,6 +1097,27 @@ void vm_unmap_aliases(void)flush_tlb_kernel_range (start, end);mutex_unlock (&vmap_purge_lock); } + +/** +* vm_unmap_aliases - unmap outstanding lazy aliases in the vmap layer +* +* The vmap/vmalloc layer lazily flushes kernel virtual mappings primarily +* to amortize TLB flushing overheads. What this means is that any page you +* have now, may, in a former life, have been mapped into kernel virtual +* address by the vmap layer and so there might be some CPUs with TLB entries
+ * still referencing that page (additional to the regular 1:1 kernel mapping).
+ *
+ *vm_unmap_aliasesflushes all such lazy mappings. After it returns, we can
+ * be sure that none of the pages we have control over will have any aliases
+ * from the vmap layer.
+ */mm/vmalloc.c +8
-0+void vm_unmap_aliases (void ) +{ +unsigned long start =ULONG_MAX , end =0 ; +int must_flush =0 ; + +_vm_unmap_aliases (start, end, must_flush); +}EXPORT_SYMBOL_GPL (vm_unmap_aliases);/** @@ -1494,6 +1503,79 @@ struct vm_struct *remove_vm_area(const void *addr)return NULL ; } +static inline void set_area_alias(const struct vm_struct *area,mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1516
Quoted 10 lines
mm/vmalloc.c +10
-0+ int (*set_alias)(struct page *page)) +{ +int i; + +for (i =0 ; i < area->nr_pages; i++) { +unsigned long addr = + (unsigned long )page_address (area->pages[i]); + +if (addr) +set_alias (area->pages[i]);What's wrong with simply:
Quoted 6 lines
for (i = 0; i < area->nr_pages; i++) { if (page_address(area->pages[i])) set_alias(area->pages[i]); }?
Yes, I see that's better.
I think at one point I had played with having the set_alias()'s take an unsigned long address, and so the address was used twice and was worth assigning to something, so this should be fixed. Thanks.
Quoted 22 lines
Quoted 5 lines
mm/vmalloc.c +4
-0+ } +} + +/* This handles removing and resetting vm mappings related to the
vm_struct. */s/This handles/Handle/
Quoted 14 lines
+static void
vm_remove_mappings(struct vm_struct *area, int deallocate_pages)mm/vmalloc.c +12
-0+{ +unsigned long addr = (unsigned long )area->addr; +unsigned long start =ULONG_MAX , end =0 ; +int special = area->flags &VM_HAS_SPECIAL_PERMS ; +int i; + +/* +* The below block can be removed when all architectures that have +* direct map permissions also have set_alias_ implementations. This is +* to do resetting on the directmap for any special permissions (today +* only X), without leaving a RW+X window. +*/
Quoted 8 lines
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1536
Quoted 3 lines
mm/vmalloc.c +3
-0+if (special && !IS_ENABLED (CONFIG_ARCH_HAS_SET_ALIAS )) { +set_memory_nx (addr, area->nr_pages); +set_memory_rw (addr, area->nr_pages);That's two not very cheap calls to the underlying worker function, for example
change_memory_common()on ARM64, instead of calling it once with the respective flags. You allude to that in the commit message but you might wanna run it by ARM folks first.
These calls are basically getting moved to be centralized here from other places. In the later patches they get removed from where they were, so its a net zero (except BPF on ARM I think).
Ard had expressed interest in having the set_alias_() functions for Arm, and the names were chosen to be arch agnostic. He didn't explicitly commit but I was under the impression he might create an implementation for ARM and we could remove this block.
Ard, did I misinterpret that?
Quoted 30 lines
Quoted 17 lines
mm/vmalloc.c +17
-0+ } + +remove_vm_area (area->addr); + +/* If this is not special memory, we can skip the below. */ +if (!special) +return ; + +/* +* If we are not deallocating pages, we can just do the flush of the VM +* area and return. +*/ +if (!deallocate_pages) { +vm_unmap_aliases (); +return ; + } +mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1563
Quoted 10 lines
mm/vmalloc.c +10
-0+/* +* If we are here, we need to flush the vm mapping and reset the direct +* map. +* First find the start and end range of the direct mappings to make +* sure the vm_unmap_aliases flush includes the direct map. +*/ +for (i =0 ; i < area->nr_pages; i++) { +unsigned long addr = + (unsigned long )page_address (area->pages[i]); +if (addr) {if (page_address(area->pages[i]))
Quoted 1 line
as above.
Right. Thanks.
Quoted 15 lines
mm/vmalloc.c struct vm_struct *remove_vm_area(const void *addr) · line 1570
Quoted 7 lines
mm/vmalloc.c +7
-0+ start =min (addr, start); + end =max (addr, end); + } + } + +/* +* First we set direct map to something not valid so that it won't be Above comment says "First" too. In general, all those "we" formulations do not make the comments as easy to read as when you make them impersonal and imperative:
/* * Set the direct map to something invalid...
Quoted 8 lines
Just like
Documentation/process/submitting-patches.rstsays:"Describe your changes in imperative mood, e.g. "make xyzzy do frotz"
instead of "[This patch] makes xyzzy do frotz" or "[I] changed xyzzy to do frotz", as if you are giving orders to the codebase to change its behaviour."you simply order your code to do stuff. :-)
I see, I will make sure to fully apply this grammar rule to commit messages and comments for the next version.
Quoted 13 lines
Quoted 11 lines
mm/vmalloc.c +8
-0+* cached if there are any accesses after the TLB flush, then we flush +* the TLB, and reset the directmap permissions to the default. +*/ +set_area_alias (area, set_alias_nv_noflush); +_vm_unmap_aliases (start, end,1 ); +set_area_alias (area, set_alias_default_noflush); +} +static void __vunmap (const void *addr,int deallocate_pages) {struct vm_struct *area;
Borislav Petkov
maintainerRe: [PATCH v2 15/20] vmalloc: New flags for safe vfree on special perms
Quoted 2 lines · Rick P
So to capture both of those intentions, maybe I'll slightly tweak your suggestion to
VM_FLUSH_RESET_PERMS?
Yeah, sure, better.
VM_HAS_SPECIAL_PERMS doesn't tell me what those special permissions are while flush and reset permissions makes a lot more sense, thx.
Quoted 2 lines
I had thought it was easier to read. If its not the case, I'll change it as you suggest.
My logic is, the less local vars, the easier to scan the code quickly.
Quoted 4 lines
Ard had expressed interest in having the
set_alias_()functions for Arm, and the names were chosen to be arch agnostic. He didn't explicitly commit but I was under the impression he might create an implementation for ARM and we could remove this block.
Yeah, Will has those on his radar too so we should be good here.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.PATCH v2 16/20 modules: Use vmalloc special flag
In reply to Rick EdgecombeUse new flag for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed.
Since vfreeing of VM_HAS_SPECIAL_PERMS memory is not supported in an interrupt by vmalloc, the freeing of init sections is moved to a work queue. Instead of call_rcu it now uses synchronize_rcu() in the work queue.
Lastly, there is now a WARN_ON in module_memfree since it should not be called in an interrupt with special memory as is required for VM_HAS_SPECIAL_PERMS.
--- kernel/module.c | 77 +++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 38 deletions(-)
kernel/module.c +39 -38
@@ -98,6 +98,10 @@ DEFINE_MUTEX(module_mutex);EXPORT_SYMBOL_GPL (module_mutex);static LIST_HEAD (modules); +/* Work queue for freeing init sections in success case */ +static struct work_struct init_free_wq; +static struct llist_head init_free_list; +#ifdef CONFIG_MODULES_TREE_LOOKUP /*
@@ -1949,6 +1953,8 @@ void module_enable_ro(const struct module *mod, bool after_init)if (!rodata_enabled)return ; +set_vm_special (mod->core_layout.base); +set_vm_special (mod->init_layout.base);frob_text (&mod->core_layout, set_memory_ro);frob_text (&mod->core_layout, set_memory_x);
@@ -1972,15 +1978,6 @@ static void module_enable_nx(const struct module *mod)frob_writable_data (&mod->init_layout, set_memory_nx); } -static void module_disable_nx (const struct module *mod) -{ -frob_rodata (&mod->core_layout, set_memory_x); -frob_ro_after_init (&mod->core_layout, set_memory_x); -frob_writable_data (&mod->core_layout, set_memory_x); -frob_rodata (&mod->init_layout, set_memory_x); -frob_writable_data (&mod->init_layout, set_memory_x); -} -/* Iterate through all modules and set each module's text as RW */ void set_all_modules_text_rw (void ) {
@@ -2024,23 +2021,8 @@ void set_all_modules_text_ro(void) }mutex_unlock (&module_mutex); } - -static void disable_ro_nx (const struct module_layout *layout) -{ -if (rodata_enabled) { -frob_text (layout, set_memory_rw); -frob_rodata (layout, set_memory_rw); -frob_ro_after_init (layout, set_memory_rw); - } -frob_rodata (layout, set_memory_x); -frob_ro_after_init (layout, set_memory_x); -frob_writable_data (layout, set_memory_x); -} -#else -static void disable_ro_nx (const struct module_layout *layout) { }static void module_enable_nx (const struct module *mod) { } -static void module_disable_nx (const struct module *mod) { }#endif #ifdef CONFIG_LIVEPATCH
@@ -2120,6 +2102,11 @@ static void free_module_elf(struct module *mod)void __weak module_memfree (void *module_region) { +/* +* This memory may be RO, and freeing RO memory in an interrupt is not +* supported by vmalloc. +*/ +WARN_ON (in_interrupt ());vfree (module_region); }
@@ -2171,7 +2158,6 @@ static void free_module(struct module *mod)mutex_unlock (&module_mutex);/* This may be empty, but that's OK */ -disable_ro_nx (&mod->init_layout);module_arch_freeing_init (mod);module_memfree (mod->init_layout.base);kfree (mod->args);
@@ -2181,7 +2167,6 @@ static void free_module(struct module *mod)lockdep_free_key_range (mod->core_layout.base, mod->core_layout.size);/* Finally, free the core (containing the module structure) */ -disable_ro_nx (&mod->core_layout);module_memfree (mod->core_layout.base); }
@@ -3424,17 +3409,34 @@ static void do_mod_ctors(struct module *mod)/* For freeing module_init on success, in case kallsyms traversing */ struct mod_initfree { -struct rcu_head rcu; +struct llist_node node;void *module_init; }; -static void do_free_init (struct rcu_head *head) +static void do_free_init (struct work_struct *w) { - struct mod_initfree *m = container_of(head, struct mod_initfree, rcu); -module_memfree (m->module_init); -kfree (m); +struct llist_node *pos, *n, *list; +struct mod_initfree *initfree; + + list =llist_del_all (&init_free_list); + +synchronize_rcu (); + +llist_for_each_safe (pos, n, list) { + initfree = container_of(pos, struct mod_initfree, node); +module_memfree (initfree->module_init); +kfree (initfree); + } } +static int __init modules_wq_init (void ) +{ +INIT_WORK (&init_free_wq, do_free_init); +init_llist_head (&init_free_list); +return 0 ; +} +module_init (modules_wq_init); +/* * This is where the real work happens. *
@@ -3511,7 +3513,6 @@ static noinline int do_init_module(struct module *mod)#endif module_enable_ro (mod, true);mod_tree_remove_init (mod); -disable_ro_nx (&mod->init_layout);module_arch_freeing_init (mod); mod->init_layout.base =NULL ; mod->init_layout.size =0 ;
@@ -3522,14 +3523,18 @@ static noinline int do_init_module(struct module *mod)* We want to free module_init, but be aware that kallsyms may be * walking this with preempt disabled. In all the failure paths, we * call synchronize_rcu(), but we don't want to slow down the success -* path, so use actual RCU here. +* path. We can't do module_memfree in an interrupt, so we do the work +* and call synchronize_rcu() in a work queue. +* * Note that module_alloc() on most architectures creates W+X page * mappings which won't be cleaned up until do_free_init() runs. Any * code such as mark_rodata_ro() which depends on those mappings to * be cleaned up needs to sync with the queued work - ie * rcu_barrier() */ -call_rcu (&freeinit->rcu, do_free_init); +if (llist_add (&freeinit->node, &init_free_list)) +schedule_work (&init_free_wq); +mutex_unlock (&module_mutex);wake_up_all (&module_wq);
@@ -3826,10 +3831,6 @@ static int load_module(struct load_info *info, const char __user *uargs,module_bug_cleanup (mod);mutex_unlock (&module_mutex); -/* we can't deallocate the module until we clear memory protection */ -module_disable_ro (mod); -module_disable_nx (mod); - ddebug_cleanup:ftrace_release_mod (mod);dynamic_debug_remove (mod, info->debug);
Signature · 1 line
2.17.1
PATCH v2 17/20 bpf: Use vmalloc special flag
In reply to Rick EdgecombeUse new flag VM_HAS_SPECIAL_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set RW before freeing which is no longer needed. Also we no longer need a bit to track if the memory is RO because it is tracked in vmalloc.
--- include/linux/filter.h | 16 +++------------- kernel/bpf/core.c | 1 - 2 files changed, 3 insertions(+), 14 deletions(-)
include/linux/filter.h +3 -13
@@ -20,6 +20,7 @@#include <linux/set_memory.h> #include <linux/kallsyms.h> #include <linux/if_vlan.h> +#include <linux/vmalloc.h> #include <net/sch_generic.h>
@@ -483,7 +484,6 @@ struct bpf_prog {u16 pages;/* Number of allocated pages */ u16 jited:1, /* Is our filter JIT'ed? */ jit_requested:1,/* archs need to JIT the prog */ - undo_set_mem:1, /* Passed set_memory_ro() checkpoint */ gpl_compatible:1, /* Is filter GPL compatible? */ cb_access:1, /* Is control block accessed? */ dst_needed:1, /* Do we need dst entry? */
@@ -681,26 +681,17 @@ bpf_ctx_narrow_access_ok(u32 off, u32 size, u32 size_default)static inline void bpf_prog_lock_ro (struct bpf_prog *fp) { +set_vm_special (fp);set_memory_ro ((unsigned long )fp, fp->pages); } -static inline void bpf_prog_unlock_ro (struct bpf_prog *fp) -{ -if (fp->undo_set_mem) -set_memory_rw ((unsigned long )fp, fp->pages); -} -static inline void bpf_jit_binary_lock_ro (struct bpf_binary_header *hdr) { +set_vm_special (hdr);set_memory_ro ((unsigned long )hdr, hdr->pages);set_memory_x ((unsigned long )hdr, hdr->pages); } -static inline void bpf_jit_binary_unlock_ro (struct bpf_binary_header *hdr) -{ -set_memory_rw ((unsigned long )hdr, hdr->pages); -} -static inline struct bpf_binary_header *bpf_jit_binary_hdr (const struct bpf_prog *fp) {
@@ -735,7 +726,6 @@ void __bpf_prog_free(struct bpf_prog *fp);static inline void bpf_prog_unlock_free (struct bpf_prog *fp) { -bpf_prog_unlock_ro (fp);__bpf_prog_free (fp); }
kernel/bpf/core.c +0 -1
@@ -804,7 +804,6 @@ void __weak bpf_jit_free(struct bpf_prog *fp)if (fp->jited) {struct bpf_binary_header *hdr =bpf_jit_binary_hdr (fp); -bpf_jit_binary_unlock_ro (hdr);bpf_jit_binary_free (hdr);WARN_ON_ONCE (!bpf_prog_kallsyms_verify_off (fp));
Signature · 1 line
2.17.1
PATCH v2 18/20 x86/ftrace: Use vmalloc special flag
In reply to Rick EdgecombeUse new flag VM_HAS_SPECIAL_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed.
--- arch/x86/kernel/ftrace.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
arch/x86/kernel/ftrace.c +2 -4
@@ -692,10 +692,6 @@ static inline void *alloc_tramp(unsigned long size) }static inline void tramp_free (void *tramp,int size) { -int npages =PAGE_ALIGN (size) >>PAGE_SHIFT ; - -set_memory_nx ((unsigned long )tramp, npages); -set_memory_rw ((unsigned long )tramp, npages);module_memfree (tramp); }#else
@@ -820,6 +816,8 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)/* ALLOC_TRAMP flags lets us know we created it */ ops->flags |=FTRACE_OPS_FL_ALLOC_TRAMP ; +set_vm_special (trampoline); +/* * Module allocation needs to be completed by making the page * executable. The page is still writable, which is a security hazard,
Signature · 1 line
2.17.1
PATCH v2 19/20 x86/kprobes: Use vmalloc special flag
In reply to Rick EdgecombeUse new flag VM_HAS_SPECIAL_PERMS for handling freeing of special permissioned memory in vmalloc and remove places where memory was set NX and RW before freeing which is no longer needed.
--- arch/x86/kernel/kprobes/core.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-)
arch/x86/kernel/kprobes/core.c +1 -6
@@ -434,6 +434,7 @@ void *alloc_insn_page(void)if (page ==NULL )return NULL ; +set_vm_special (page);/* * First make the page read-only, and then only then make it executable * to prevent it from being W+X in between.
@@ -452,12 +453,6 @@ void *alloc_insn_page(void)/* Recover page to RW mode before releasing it */ void free_insn_page (void *page) { -/* -* First make the page non-executable, and then only then make it -* writable to prevent it from being W+X in between. -*/ -set_memory_nx ((unsigned long )page,1 ); -set_memory_rw ((unsigned long )page,1 );module_memfree (page); }
Signature · 1 line
2.17.1
PATCH v2 20/20 x86/alternative: comment about module removal races
In reply to Rick EdgecombeFrom: Nadav Amit <[email protected]>
Add a comment to clarify that users of text_poke() must ensure that no races with module removal take place.
--- arch/x86/kernel/alternative.c | 5 +++++ 1 file changed, 5 insertions(+)
arch/x86/kernel/alternative.c +5 -0
@@ -807,6 +807,11 @@ static void *__text_poke(void *addr, const void *opcode, size_t len)* It means the size must be writable atomically and the address must be aligned * in a way that permits an atomic write. It also makes sure we fit on a single * page. +* +* Note that the caller must ensure that if the modified code is part of a +* module, the module would not be removed during poking. This can be achieved +* by registering a module notifier, and ordering module removal and patching +* trough a mutex. */ void *text_poke (void *addr,const void *opcode,size_t len) {
Signature · 1 line
2.17.1