x86/mce: Recover from poison found while copying from user space
Subsystems (4)
A message that is not in the archive (sent off-list or not yet received): [email protected]
Tony Luck
maintainerPATCH 1/8 x86/mce: Stop mce_reign() from re-computing severity for every CPU
Back in:
commit 20d51a426fe9 ("x86/mce: Reuse one of the u16 padding fields in 'struct mce'")
a field was added to "struct mce" to save the computed error severity.
Make use of this in mce_reign() to avoid re-computing the severity for every CPU.
In the case where we panic, we do still need one call to mce_severity to provide the correct message giving the reason for the panic.
--- arch/x86/kernel/cpu/mce/core.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/mce/core.c +8 -8
@@ -872,7 +872,6 @@ static void mce_reign(void)struct mce *m =NULL ;int global_worst =0 ;char *msg =NULL ; -char *nmsg =NULL ;/* * This CPU is the Monarch and the other CPUs have run
@@ -880,12 +879,10 @@ static void mce_reign(void)* Grade the severity of the errors of all the CPUs. */ for_each_possible_cpu (cpu) { -int severity =mce_severity (&per_cpu (mces_seen, cpu), - mca_cfg.tolerant, - &nmsg, true); -if (severity > global_worst) { - msg = nmsg; - global_worst = severity; +struct mce *mtmp = &per_cpu (mces_seen, cpu); + +if (mtmp->severity > global_worst) { + global_worst = mtmp->severity; m = &per_cpu (mces_seen, cpu); } }
@@ -895,8 +892,11 @@ static void mce_reign(void)* This dumps all the mces in the log buffer and stops the * other CPUs. */ -if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) +if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) { +/* call mce_severity() to get "msg" for panic */ +mce_severity (m, mca_cfg.tolerant, &msg, true);mce_panic ("Fatal machine check" , m, msg); + }/* * For UC somewhere we let the CPU who detects it handle it.
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH 1/8] x86/mce: Stop mce_reign() from re-computing severity for every CPU
Quoted 8 lines · Tony Luck
Back in:
commit
20d51a426fe9("x86/mce: Reuse one of the u16 padding fields in 'struct mce'")a field was added to "struct mce" to save the computed error severity.
Make use of this in
mce_reign()to avoid re-computing the severity for every CPU.
Yah, could go as a standalone optimization...
Quoted 2 lines
In the case where we panic, we do still need one call to
mce_severityto provide the correct message giving the reason for the panic.
... but pls do not use the "we" formulation as this ambiguous. I'll fix that up now.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette[tip: ras/core] x86/mce: Stop mce_reign() from re-computing severity for every CPU
In reply to Tony LuckThe following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: 13c877f4b48b943105ad9e13ba2c7a093fb694e8 Gitweb: https://git.kernel.org/tip/13c877f4b48b943105ad9e13ba2c7a093fb694e8 Author: Tony Luck <[email protected]> AuthorDate: Tue, 08 Sep 2020 10:55:12 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Mon, 14 Sep 2020 19:25:23 +02:00
x86/mce: Stop mce_reign() from re-computing severity for every CPU
Back in commit:
20d51a426fe9 ("x86/mce: Reuse one of the u16 padding fields in 'struct mce'")
a field was added to "struct mce" to save the computed error severity.
Make use of this in mce_reign() to avoid re-computing the severity for every CPU.
In the case where the machine panics, one call to mce_severity() is still needed in order to provide the correct message giving the reason for the panic.
--- arch/x86/kernel/cpu/mce/core.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-)
arch/x86/kernel/cpu/mce/core.c +8 -8
@@ -920,7 +920,6 @@ static void mce_reign(void)struct mce *m =NULL ;int global_worst =0 ;char *msg =NULL ; -char *nmsg =NULL ;/* * This CPU is the Monarch and the other CPUs have run
@@ -928,12 +927,10 @@ static void mce_reign(void)* Grade the severity of the errors of all the CPUs. */ for_each_possible_cpu (cpu) { -int severity =mce_severity (&per_cpu (mces_seen, cpu), - mca_cfg.tolerant, - &nmsg, true); -if (severity > global_worst) { - msg = nmsg; - global_worst = severity; +struct mce *mtmp = &per_cpu (mces_seen, cpu); + +if (mtmp->severity > global_worst) { + global_worst = mtmp->severity; m = &per_cpu (mces_seen, cpu); } }
@@ -943,8 +940,11 @@ static void mce_reign(void)* This dumps all the mces in the log buffer and stops the * other CPUs. */ -if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) +if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) { +/* call mce_severity() to get "msg" for panic */ +mce_severity (m, mca_cfg.tolerant, &msg, true);mce_panic ("Fatal machine check" , m, msg); + }/* * For UC somewhere we let the CPU who detects it handle it.
Tony Luck
maintainerPATCH 4/8 x86/mce: Add _ASM_EXTABLE_CPY for copy user access
In reply to a message not in the archiveFrom: Youquan Song <[email protected]>
_ASM_EXTABLE_UA is a general exception entry to record the exception fixup for all exception spots between kernel and user space access.
To enable recovery from machine checks while coping data from user addresses we need to be able to distinguish the places that are looping copying data from those that copy a single byte/word/etc.
Add a new macro _ASM_EXTABLE_CPY and use it in place of _ASM_EXTABLE_UA in the copy functions.
Record the exception reason number to regs->ax at ex_handler_uaccess which is used to check MCE triggered.
The new fixup routine ex_handler_copy() is almost an exact copy of ex_handler_uaccess() The difference is that it sets regs->ax to the trap number. Following patches use this to avoid trying to copy remaining bytes from the tail of the copy and possibly hitting the poison again.
New mce.kflags bit MCE_IN_KERNEL_COPYIN will be used by mce_severity() calculation to indicate that a machine check is recoverable because the kernel was copying from user space.
--- arch/x86/include/asm/asm.h | 6 +++ arch/x86/include/asm/mce.h | 1 + arch/x86/lib/copy_user_64.S | 96 ++++++++++++++++++------------------- arch/x86/mm/extable.c | 14 +++++- 4 files changed, 68 insertions(+), 49 deletions(-)
arch/x86/include/asm/asm.h +6 -0
@@ -135,6 +135,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
@@ -160,6 +163,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
arch/x86/include/asm/mce.h +1 -0
@@ -137,6 +137,7 @@#define MCE_HANDLED_EDAC BIT_ULL(4)#define MCE_HANDLED_MCELOG BIT_ULL(5)#define MCE_IN_KERNEL_RECOV BIT_ULL(6) +#define MCE_IN_KERNEL_COPYIN BIT_ULL(7)/* * This structure contains all data related to the MCE log. Also
arch/x86/lib/copy_user_64.S +48 -48
131 lines not on this page · raw message
arch/x86/mm/extable.c +13 -1
@@ -80,6 +80,18 @@ __visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_uaccess); +__visible bool ex_handler_copy(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr) +{ +WARN_ONCE (trapnr ==X86_TRAP_GP ,"General protection fault in user access. Non-canonical address?" ); + regs->ip =ex_fixup_addr (fixup); + regs->ax = trapnr; +return true; +} +EXPORT_SYMBOL (ex_handler_copy); + __visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup, struct pt_regs *regs, int trapnr, unsigned long error_code,
@@ -136,7 +148,7 @@ __visible enum handler_type ex_fault_handler_type(unsigned long ip) handler =ex_fixup_handler (e);if (handler == ex_handler_fault)return HANDLER_FAULT ; -else if (handler == ex_handler_uaccess) +else if (handler == ex_handler_uaccess || handler == ex_handler_copy)return HANDLER_UACCESS ;else return HANDLER_OTHER ;
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH 4/8] x86/mce: Add _ASM_EXTABLE_CPY for copy user access
arch/x86/include/asm/mce.h line 140
Quoted 9 lines · Tony Luck
arch/x86/include/asm/mce.h +1
-0@@ -137,6 +137,7 @@#define MCE_HANDLED_EDAC BIT_ULL(4)#define MCE_HANDLED_MCELOG BIT_ULL(5)#define MCE_IN_KERNEL_RECOV BIT_ULL(6) +#define MCE_IN_KERNEL_COPYIN BIT_ULL(7)
That flag needs a comment explaining what it is. And you can do MCE_IN_KERNEL_RECOV, while you're at it, there's a nice comment above its use in do_machine_check which you could use.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerPATCH 5/8 x86/mce: Avoid tail copy when machine check terminated a copy from user
In reply to a message not in the archiveIn the page fault case it is ok to see if a few more unaligned bytes can be copied from the source address. Worst case is that the page fault will be triggered again.
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return as if the copy succeeded.
[Tried returning bytes not copied here, but that puts the kernel
into a loop taking the machine check over and over. I don't know at what level some code is retrying]
--- arch/x86/lib/copy_user_64.S | 7 +++++++ 1 file changed, 7 insertions(+)
arch/x86/lib/copy_user_64.S +7 -0
@@ -15,6 +15,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> .macro ALIGN_DESTINATION /* check for bad alignment of destination */
@@ -221,6 +222,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Try to copy last bytes and clear the rest if needed. * Since protection fault in copy_from/to_user is not a normal situation, * it is not necessary to optimize tail handling. + * Don't try to copy the tail if machine check happened * * Input: * rdi destination
@@ -232,10 +234,15 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx + cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ + je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret +3: xorl %eax,%eax /* pretend we succeeded? */ + ASM_CLAC + ret _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail)
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH 5/8] x86/mce: Avoid tail copy when machine check terminated a copy from user
Quoted 37 lines · Tony Luck
In the page fault case it is ok to see if a few more unaligned bytes can be copied from the source address. Worst case is that the page fault will be triggered again.
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return as if the copy succeeded.
[Tried returning bytes not copied here, but that puts the kernel
into a loop taking the machine check over and over. I don't know at what level some code is retrying]Signed-off-by Tony Luck <[email protected]>--- arch/x86/lib/copy_user_64.S | 7 +++++++ 1 file changed, 7 insertions(+)arch/x86/lib/copy_user_64.S +2
-0@@ -15,6 +15,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> .macro ALIGN_DESTINATION /* check for bad alignment of destination */@@ -221,6 +222,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Try to copy last bytes and clear the rest if needed. * Since protection fault in copy_from/to_user is not a normal situation, * it is not necessary to optimize tail handling. + * Don't try to copy the tail if machine check happened * * Input: * rdi destination
arch/x86/lib/copy_user_64.S EXPORT_SYMBOL(copy_user_enhanced_fast_string) · line 243
Quoted 10 lines
arch/x86/lib/copy_user_64.S +3
-0@@ -232,10 +234,15 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx + cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ + je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret +3: xorl %eax,%eax /* pretend we succeeded? */
Hmm, but copy_*_user returns the uncopied bytes in eax. Users of this need to handle the MC case properly but if you return 0, they would think that they copied everything but there's some trailing stuff they didn't manage to take.
And it's not like they *should* have to retry to copy it because they will walk right into the faulty region and cause more MCEs.
So how is this "I-got-an-MCE-while-copying-from-user" handled on the higher level?
Your 7/8 says:
"Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task."
So how are users of copy_*_user() expected to handle the page disappearing from under them?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH 5/8] x86/mce: Avoid tail copy when machine check terminated a copy from user
Quoted 11 lines · Borislav Petkov
arch/x86/lib/copy_user_64.S EXPORT_SYMBOL(copy_user_enhanced_fast_string) · line 243
Quoted 4 lines · Tony Luck
2: mov %ecx,%eax
arch/x86/lib/copy_user_64.S +1
-0ASM_CLAC ret +3: xorl %eax,%eax /* pretend we succeeded? */Hmm, but copy_*_user returns the uncopied bytes in eax. Users of this need to handle the MC case properly but if you return 0, they would think that they copied everything but there's some trailing stuff they didn't manage to take.
Quoted 5 lines
And it's not like they *should* have to retry to copy it because they will walk right into the faulty region and cause more MCEs.
So how is this "I-got-an-MCE-while-copying-from-user" handled on the higher level?
I got stuck on this part of the series for a long time. This code was pretty subtle before I came to make it even more confusing.
In a normal copy from user operation the expectation is that there will be some page faults. If a #PF happens the kernel either fixes the fault and provides a good mapping, or finds that it can't and doesn't map anything. Either way the extable fixup jumps to the error return of the low level copy function which returns the number of bytes still to be copied.
The subtle part is that the iterator that calls into the low level copy calls fault_in_pages_readable(). I thought this was just some optimization to prefault the pages, but actually it turns out to be critical to how -EFAULT gets returned to a user when they pass a bad address.
When the low level copy stops at a page fault, the next call to fault_in_pages_readable() checks if the fault was fixed and returns -EFAULT if it wasn't.
Interesting back story ... now look at what happens if we take a machine check instead of a page fault.
The machine check handler will invoke the extable fixup to move the RIP for the copy function to the early return path. Machine check handler also set up some task_work for the application to call memory_failure() which will unmap the page. But that work won't happen until we try to return to the user. So the page (and the poison) are still mapped.
So we take a another machine check on the same address when fault_in_pages_readable() does __get_user(). That ought to break us out ... but for some reason I still don't understand didn't. But even if it did ... the second machine check is not at all a good idea.
Returning zero bytes left to say we completed avoids that. The user is guaranteed a SIGBUS when the task_work does fire. So whatever system call was in progress is not going to see the apparent successful return.
Both the code and the commit comment need much more of this description.
Unless you have some better way out of the dilemmma that the real fixup is only scheduled at the point that the extable fixup just arranges for a simple local return from the copy.
Quoted 10 lines
Your 7/8 says:
"Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task."
So how are users of copy_*_user() expected to handle the page disappearing from under them?
When the return to user happens the task_work that was scheduled in the machine check handler takes care of the error return to the user.
-Tony
Borislav Petkov
maintainerRe: [PATCH 5/8] x86/mce: Avoid tail copy when machine check terminated a copy from user
Replying a bit out of order:
Quoted 2 lines
Both the code and the commit comment need much more of this description.
Hell yeah!
Quoted 5 lines · Tony
So we take a another machine check on the same address when
fault_in_pages_readable()does__get_user(). That ought to break us out ... but for some reason I still don't understand didn't. But even if it did ... the second machine check is not at all a good idea.
And I think this is the important point: for MCEs you absolutely don't want to take another MCE and even walk into those fields. So what fault_in_pages_readable() does normally, would be totally wrong. Imagine you're playing minesweeper - you can't just pre-fault blocks without counting the mines. :-P
So actually, I'm thinking:
.LMCE_during_user_access:
mov $-ENODEV, %eax
ASM_CLAC
ret
I have no clue which error code we should put there but it should be an error code which tells you not to retry and to back off immediately.
Quoted 4 lines
Returning zero bytes left to say we completed avoids that. The user is guaranteed a SIGBUS when the
task_workdoes fire. So whatever system call was in progress is not going to see the apparent successful return.
Yes, my only proposition with the error code is in case you're looking at traces, to recognize that the copying encountered an MCE. In addition to the "back off immediately" semantics, if there even is such defined for users of copy_*_user().
Quoted 3 lines
Unless you have some better way out of the dilemmma that the real fixup is only scheduled at the point that the extable fixup just arranges for a simple local return from the copy.
Right, see above: I think it is imperative *not* to walk into that area again and not do any retrying.
Quoted 3 lines
When the return to user happens the
task_workthat was scheduled in the machine check handler takes care of the error return to the user.
Yeah, let's write that whole flow down somewhere - not in a commit message - so that it is clear what happens.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH 5/8] x86/mce: Avoid tail copy when machine check terminated a copy from user
Quoted 9 lines · Borislav Petkov
So actually, I'm thinking:
.LMCE_during_user_access:
mov $-ENODEV, %eax
ASM_CLAC
retI have no clue which error code we should put there but it should be an error code which tells you not to retry and to back off immediately.
That does look a lot easier to understand *at this point* in the code.
But the existing iterator code is not expecting an error code. Just a count of bytes not copied.
So doing this would mean some surgery on the maze of giant #defines that is lib/iov_iter.c
-Tony
Borislav Petkov
maintainerRe: [PATCH 5/8] x86/mce: Avoid tail copy when machine check terminated a copy from user
Quoted 11 lines · Tony
Quoted 9 lines · Borislav Petkov
So actually, I'm thinking:
.LMCE_during_user_access:
mov $-ENODEV, %eax
ASM_CLAC
retI have no clue which error code we should put there but it should be an error code which tells you not to retry and to back off immediately.
Quoted 7 lines
That does look a lot easier to understand *at this point* in the code.
But the existing iterator code is not expecting an error code. Just a count of bytes not copied.
So doing this would mean some surgery on the maze of giant #defines that is
lib/iov_iter.c
Ok, since the user task is guaranteed to get the SIGBUS, let's just resort to documenting this properly for now and we can consider more involved surgery later, if really needed.
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerPATCH 6/8 x86/mce: Change fault_in_kernel_space() from static to global
In reply to a message not in the archiveFrom: Youquan Song <[email protected]>
Machine check code needs to be able to determine if a faulting address is in user of kernel space. There is already a function to do this.
Change from "static int" to "bool" and add declaration to <asm/traps.h>
No functional change.
--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
arch/x86/include/asm/traps.h +2 -0
@@ -35,6 +35,8 @@ extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); +bool fault_in_kernel_space (unsigned long address); +#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/mm/fault.c +1 -1
@@ -1128,7 +1128,7 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)return 0 ; } -static int fault_in_kernel_space (unsigned long address) +bool fault_in_kernel_space (unsigned long address) {/* * On 64-bit systems, the vsyscall page is at an address above
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH 7/8 x86/mce: Recover from poison found while copying from user space
In reply to a message not in the archiveFrom: Youquan Song <[email protected]>
Existing kernel code can only recover from a machine check on code that tagged in the exception table with a fault handling recovery path.
New field in the task structure mce_vaddr is initialized to the user virtual address of the fault. This is so that kill_me_maybe() can provide that information to the user SIGBUS handler.
Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task.
--- arch/x86/kernel/cpu/mce/core.c | 51 ++++++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + 2 files changed, 52 insertions(+)
arch/x86/kernel/cpu/mce/core.c +51 -0
@@ -53,6 +53,8 @@#include <asm/mce.h> #include <asm/msr.h> #include <asm/reboot.h> +#include <asm/insn.h> +#include <asm/insn-eval.h> #include "internal.h"
@@ -1197,6 +1199,32 @@ static void kill_me_maybe(struct callback_head *cb)kill_me_now (cb); } +/* +* Decode a kernel instruction that faulted while reading from a user +* address and return the linear address that was being read. +*/ +static void __user *get_virtual_address (struct pt_regs *regs) +{ +u8 insn_buf[MAX_INSN_SIZE ]; +struct insn insn; + +if (copy_from_kernel_nofault (insn_buf, (void *)regs->ip,MAX_INSN_SIZE )) +return (void __user *)~0ul ; + +kernel_insn_init (&insn, insn_buf,MAX_INSN_SIZE ); +insn_get_length (&insn); +insn_get_modrm (&insn); +insn_get_sib (&insn); + +/* +* For MOVS[BWLQ] the source address is in %rsi +*/ +if (insn.opcode.value ==0xa4 || insn.opcode.value ==0xa5 ) +return (void __user *)regs->si; +else +return insn_get_addr_ref (&insn, regs); +} +/* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18.
@@ -1342,6 +1370,7 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); + current->mce_vaddr =NULL ; current->mce_addr = m.addr; current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); current->mce_whole_page =whole_page (&m);
@@ -1350,6 +1379,13 @@ noinstr void do_machine_check(struct pt_regs *regs) current->mce_kill_me.func = kill_me_now;task_work_add (current, ¤t->mce_kill_me, true); }else { +/* +* Before fixing the exception IP, find the user address +* in the MCE_IN_KERNEL_COPYIN case +*/ +if (m.kflags &MCE_IN_KERNEL_COPYIN ) + current->mce_vaddr =get_virtual_address (regs); +/* * Handle an MCE which has happened in kernel space but from * which the kernel can recover: ex_has_fault_handler() has
@@ -1363,6 +1399,21 @@ noinstr void do_machine_check(struct pt_regs *regs)if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 ))mce_panic ("Failed kernel mode recovery" , &m, msg); } + +/* +* MCE on user data while copying to kernel. Action here is +* very similar to the user hitting the poison themself. +* Poison page will be unmapped and signal sent to process. +*/ +if (m.kflags &MCE_IN_KERNEL_COPYIN ) { + current->mce_addr = m.addr; + current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (&m); + current->mce_kill_me.func = kill_me_maybe; +if (kill_it) + current->mce_kill_me.func = kill_me_now; +task_work_add (current, ¤t->mce_kill_me, true); + } } }EXPORT_SYMBOL_GPL (do_machine_check);
include/linux/sched.h +1 -0
@@ -1308,6 +1308,7 @@ struct task_struct {#endif #ifdef CONFIG_X86_MCE +void __user *mce_vaddr;u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1,
Signature · 1 line
2.21.1
Borislav Petkov
maintainerQuoted 4 lines · Tony Luck
From: Youquan Song <[email protected]>
Existing kernel code can only recover from a machine check on code that tagged in the exception table with a fault handling recovery path.
"is tagged"
Quoted 33 lines
New field in the task structure
mce_vaddris initialized to the user virtual address of the fault. This is so thatkill_me_maybe()can provide that information to the user SIGBUS handler.Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task.
Signed-off-by Youquan Song <[email protected]>Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/mce/core.c | 51 ++++++++++++++++++++++++++++++++++ include/linux/sched.h | 1 + 2 files changed, 52 insertions(+)arch/x86/kernel/cpu/mce/core.c +3
-0@@ -53,6 +53,8 @@#include <asm/mce.h> #include <asm/msr.h> #include <asm/reboot.h> +#include <asm/insn.h> +#include <asm/insn-eval.h> #include "internal.h" @@ -1197,6 +1199,32 @@ static void kill_me_maybe(struct callback_head *cb)kill_me_now (cb); } +/*
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1212
Quoted 10 lines
arch/x86/kernel/cpu/mce/core.c +10
-0+* Decode a kernel instruction that faulted while reading from a user +* address and return the linear address that was being read. +*/ +static void __user *get_virtual_address (struct pt_regs *regs) +{ +u8 insn_buf[MAX_INSN_SIZE ]; +struct insn insn; + +if (copy_from_kernel_nofault (insn_buf, (void *)regs->ip,MAX_INSN_SIZE )) +return (void __user *)~0ul ;
You're initializing ->mce_vaddr to NULL below but you're returning ~0 here. You should return NULL here too. If it is NULL, this check from your next patch will pass:
if (p->mce_vaddr != (void __user *)~0ul) {which would be the wrong thing to do so you need to think about a single invalid vaddr value and stick with it.
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1217
Quoted 4 lines
arch/x86/kernel/cpu/mce/core.c +4
-0+kernel_insn_init (&insn, insn_buf,MAX_INSN_SIZE ); +insn_get_length (&insn); +insn_get_modrm (&insn); +insn_get_sib (&insn);
AFAICT, you need the opcode only so why do all those?
I think you simply need to do:
insn_get_opcode()
and then check opcode->got because otherwise you might be looking at garbage below.
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1220
Quoted 3 lines
arch/x86/kernel/cpu/mce/core.c +3
-0+ +/* +* For MOVS[BWLQ] the source address is in %rsi
Pls end your sentences with a fullstop.
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1223
Quoted 3 lines
arch/x86/kernel/cpu/mce/core.c +3
-0+*/ +if (insn.opcode.value ==0xa4 || insn.opcode.value ==0xa5 ) +return (void __user *)regs->si;
How do you know just by looking at the opcodes, that the source operand in rSI is __user memory?
I see is_copy_from_user() in your next patch so I guess I'll verify that there...
Quoted 16 lines
arch/x86/kernel/cpu/mce/core.c +5
-0+else +return insn_get_addr_ref (&insn, regs); +} +/* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18. @@ -1342,6 +1370,7 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); + current->mce_vaddr =NULL ; current->mce_addr = m.addr; current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); current->mce_whole_page =whole_page (&m);
arch/x86/kernel/cpu/mce/core.c noinstr void do_machine_check(struct pt_regs *regs) · line 1384
Quoted 6 lines
arch/x86/kernel/cpu/mce/core.c +3
-0@@ -1350,6 +1379,13 @@ noinstr void do_machine_check(struct pt_regs *regs) current->mce_kill_me.func = kill_me_now;task_work_add (current, ¤t->mce_kill_me, true); }else { +/* +* Before fixing the exception IP, find the user address +* in the MCE_IN_KERNEL_COPYIN case
^ |-- Fullstop
Quoted 15 lines
arch/x86/kernel/cpu/mce/core.c +8
-0+*/ +if (m.kflags &MCE_IN_KERNEL_COPYIN ) + current->mce_vaddr =get_virtual_address (regs); +/* * Handle an MCE which has happened in kernel space but from * which the kernel can recover: ex_has_fault_handler() has @@ -1363,6 +1399,21 @@ noinstr void do_machine_check(struct pt_regs *regs)if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 ))mce_panic ("Failed kernel mode recovery" , &m, msg); } + +/* +* MCE on user data while copying to kernel. Action here is +* very similar to the user hitting the poison themself.
arch/x86/kernel/cpu/mce/core.c noinstr void do_machine_check(struct pt_regs *regs) · line 1415
Quoted 10 lines
arch/x86/kernel/cpu/mce/core.c +10
-0+* Poison page will be unmapped and signal sent to process. +*/ +if (m.kflags &MCE_IN_KERNEL_COPYIN ) { + current->mce_addr = m.addr; + current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (&m); + current->mce_kill_me.func = kill_me_maybe; +if (kill_it) + current->mce_kill_me.func = kill_me_now; +task_work_add (current, ¤t->mce_kill_me, true);
This hunk is mostly copied from the in-user case above. How about a "goto recover;" label instead of the duplication?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerPATCH 8/8 x86/mce: Decode a kernel instruction to determine if it is copying from user
In reply to a message not in the archiveAll instructions copying data between kernel and user memory are tagged with either _ASM_EXTABLE_UA or _ASM_EXTABLE_CPY entries in the exception table. ex_fault_handler_type() returns HANDLER_UACCESS for both of these.
Recovery is only possible when the machine check was triggered on a read from user memory. In this case the same strategy for recovery applies as if the user had made the access in ring3. If the fault was in kernel memory while copying to user we do not currently have a recovery plan.
Decoding the instruction that was executing when the fault occurred mostly provides enough information to determine direction of the data copy. In the case of "REP MOVS*" instructions it is necessary to also check the value in the %rsi register.
--- arch/x86/kernel/cpu/mce/core.c | 10 +++++++--- arch/x86/kernel/cpu/mce/severity.c | 32 +++++++++++++++++++++++++++++- 2 files changed, 38 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mce/core.c +7 -3
@@ -1190,13 +1190,17 @@ static void kill_me_maybe(struct callback_head *cb)if (!p->mce_ripv) flags |=MF_MUST_KILL ; -if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags)) { +if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags) && !p->mce_vaddr) {set_mce_nospec (p->mce_addr >>PAGE_SHIFT , p->mce_whole_page);return ; } -pr_err ("Memory error not recovered" ); -kill_me_now (cb); +if (p->mce_vaddr != (void __user *)~0ul ) { +force_sig_mceerr (BUS_MCEERR_AR , p->mce_vaddr,PAGE_SHIFT ); + }else { +pr_err ("Memory error not recovered" ); +kill_me_now (cb); + } }/*
arch/x86/kernel/cpu/mce/severity.c +31 -1
@@ -10,6 +10,8 @@#include <linux/init.h> #include <linux/debugfs.h> #include <asm/mce.h> +#include <asm/traps.h> +#include <asm/insn.h> #include <linux/uaccess.h> #include "internal.h"
@@ -198,6 +200,27 @@ static struct severity {#define mc_recoverable (mcg) (((mcg) & (MCG_STATUS_RIPV|MCG_STATUS_EIPV)) == \ (MCG_STATUS_RIPV|MCG_STATUS_EIPV)) +static bool is_copy_from_user (struct pt_regs *regs) +{ +u8 insn_buf[MAX_INSN_SIZE ]; +struct insn insn; + +if (copy_from_kernel_nofault (insn_buf, (void *)regs->ip,MAX_INSN_SIZE )) +return false; +kernel_insn_init (&insn, insn_buf,MAX_INSN_SIZE ); +insn_get_length (&insn); + +switch (insn.opcode.value) { +case 0x8A :case 0x8B :/* MOV */ +case 0xB60F :case 0xB70F :/* MOVZ */ +return true; +case 0xA4 :case 0xA5 :/* MOVS */ +return !fault_in_kernel_space (regs->si); + } + +return false; +} +/* * If mcgstatus indicated that ip/cs on the stack were * no good, then "m->cs" will be zero and we will have
@@ -215,10 +238,17 @@ static int error_context(struct mce *m, struct pt_regs *regs)if ((m->cs &3 ) ==3 )return IN_USER ; +if (!mc_recoverable (m->mcgstatus)) +return IN_KERNEL ; t =ex_fault_handler_type (m->ip); -if (mc_recoverable (m->mcgstatus) && t ==HANDLER_FAULT ) { +if (t ==HANDLER_FAULT ) { + m->kflags |=MCE_IN_KERNEL_RECOV ; +return IN_KERNEL_RECOV ; + } +if (t ==HANDLER_UACCESS && regs &&is_copy_from_user (regs)) { m->kflags |=MCE_IN_KERNEL_RECOV ; + m->kflags |=MCE_IN_KERNEL_COPYIN ;return IN_KERNEL_RECOV ; }
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH 8/8] x86/mce: Decode a kernel instruction to determine if it is copying from user
arch/x86/kernel/cpu/mce/severity.c static struct severity { · line 209
Quoted 7 lines · Tony Luck
arch/x86/kernel/cpu/mce/severity.c +7
-0+static bool is_copy_from_user (struct pt_regs *regs) +{ +u8 insn_buf[MAX_INSN_SIZE ]; +struct insn insn; + +if (copy_from_kernel_nofault (insn_buf, (void *)regs->ip,MAX_INSN_SIZE )) +return false;
<---- newline here.
arch/x86/kernel/cpu/mce/severity.c static struct severity { · line 211
Quoted 2 lines
arch/x86/kernel/cpu/mce/severity.c +2
-0+kernel_insn_init (&insn, insn_buf,MAX_INSN_SIZE ); +insn_get_length (&insn);
insn_get_opcode() I guess.
arch/x86/kernel/cpu/mce/severity.c static struct severity { · line 214
Quoted 3 lines
arch/x86/kernel/cpu/mce/severity.c +3
-0+ +switch (insn.opcode.value) { +case 0x8A :case 0x8B :/* MOV */
No side comments pls - put them ontop.
Also, this comment needs to say that you're looking for MOVs where the source operand can also be a memory operand.
Now lemme stare at an example, let's look at this function:
static __always_inline __must_check unsigned long raw_copy_to_user(void __user *dst, const void *src, unsigned long size)
{
return copy_user_generic((__force void *)dst, src, size);In this case, we copy to user memory, so dst is the user pointer.
Comment over copy_user_generic_unrolled() says rsi is the source so let's look at some of the insns in there:
ffffffff813accc2: 4c 8b 06 mov (%rsi),%r8 ffffffff813accc5: 4c 8b 4e 08 mov 0x8(%rsi),%r9 ffffffff813accc9: 4c 8b 56 10 mov 0x10(%rsi),%r10 ffffffff813acccd: 4c 8b 5e 18 mov 0x18(%rsi),%r11
All those are at labels which are exception-handled with the new _ASM_EXTABLE_CPY().
So according to the above check, this is a copy *from* user. But it ain't. And to confirm that, I added a breakpoint at that insn:
(gdb) break *0xffffffff813accc2
Breakpoint 1 at 0xffffffff813accc2: file arch/x86/lib/copy_user_64.S, line 66.
and the first time it hit, it has this:
Dump of assembler code from 0xffffffff813accc2 to 0xffffffff813accd6:
=> 0xffffffff813accc2 <copy_user_generic_unrolled+50>: 4c 8b 06 mov (%rsi),%r8 0xffffffff813accc5 <copy_user_generic_unrolled+53>: 4c 8b 4e 08 mov 0x8(%rsi),%r9
rsi 0xffffc90000013e10
r8 0x7fff60425120
So this is reading from *kernel* memory and writing to *user* memory. And I don't think you want that, according to the whole intent of those series.
And it makes sense - getting an MCE while writing is probably going to go boom.
arch/x86/kernel/cpu/mce/severity.c static struct severity { · line 215
Quoted 1 line
arch/x86/kernel/cpu/mce/severity.c +1
-0+case 0xB60F :case 0xB70F :/* MOVZ */
Ditto.
Quoted 18 lines
arch/x86/kernel/cpu/mce/severity.c +10
-0+return true; +case 0xA4 :case 0xA5 :/* MOVS */ +return !fault_in_kernel_space (regs->si); + } + +return false; +} +/* * If mcgstatus indicated that ip/cs on the stack were * no good, then "m->cs" will be zero and we will have @@ -215,10 +238,17 @@ static int error_context(struct mce *m, struct pt_regs *regs)if ((m->cs &3 ) ==3 )return IN_USER ; +if (!mc_recoverable (m->mcgstatus)) +return IN_KERNEL ;
arch/x86/kernel/cpu/mce/severity.c static int error_context(struct mce *m, struct pt_regs *regs) · line 252
Quoted 10 lines
arch/x86/kernel/cpu/mce/severity.c +6
-1t =ex_fault_handler_type (m->ip); -if (mc_recoverable (m->mcgstatus) && t ==HANDLER_FAULT ) { +if (t ==HANDLER_FAULT ) { + m->kflags |=MCE_IN_KERNEL_RECOV ; +return IN_KERNEL_RECOV ; + } +if (t ==HANDLER_UACCESS && regs &&is_copy_from_user (regs)) { m->kflags |=MCE_IN_KERNEL_RECOV ; + m->kflags |=MCE_IN_KERNEL_COPYIN ;return IN_KERNEL_RECOV ;
I'm guessing that should be generic enough to do on the other vendors too...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerv2 cover Add machine check recovery when copying from user space
Machine check recovery from uncorrected memory errors currently focusses primarily on errors that are detected while running in user mode. There is a mechanism for recovering from errors in kernel code, but it is currently only used for memcpy_mcsafe().
The existing recover actions for errors found in user mode (unmap the page and send SIGBUS to the task) can also be applied when the error is found while copying data from user space to the kernel.
Roadmap to this series:
Original part 0001 has already been applied to tip ras/core branch:13c877f4b48b ("x86/mce: Stop mce_reign() from re-computing severity for every CPU") so new part 0001 below used to be part 0002 in original series.
0001: First piece of infrastructure update. Severity calculations need
access to the saved registers. So pass pointer down the call chain.
0002: Later we need to know what type of exception handler is present
for a given kernel instruction. Rather than proliferate more functions like ex_has_fault_handler() for each type, replace with a function that looks up the handler and returns an enum describing the type.
Change since v1:
rename enum members s/HANDLER_NONE/EX_HANDLER_NONE/ etc. drop __visible attribute
0003: Need slightly different handling for *copy_user*() faults from
get_user() faults. Create a new exception table tag and apply to the copy functions.
Change since v1:
Add comments for kflags bits MCE_IN_KERNEL_RECOV and MCE_IN_KERNEL_COPYIN
0004: In fixup path of copy functions avoid dealing with the tail
when the copy took a machine check by returning that there are no bytes left to be copied.
Change since v1:
Better comments in both commit and in code to explain the reason for returning 0 == success, and why that is OK (process will be sent a SIGBUS).
0005: For the REP MOVS copy case we need to check if %rsi is a user
or kernel address. There's already a static function to do this. Make it global so x86/mce code can share the goodness.
0006: Changes to do_machine_check() to support the new recovery flow.
Some re-factoring to avoid code duplication (since the flows for "error in user mode" and "error while copying from user mode" are almost identical). Couple of new fields added to the task structure.
Change since v1:
This patch used to have the code to decode the kernel instruction and figure out what address was being accessed. But doing that here was too late ... the severity code needs to do that to distiguish copy to/from user in the unrolled copy functions.
0007: Finally the keystone patch that pulls all the parts together.
An instruction decoder figures out whether an instruction tagged as accessing user space is reading from or writing to user space. The instructions in the switch were found experimentally by looking at what instructions in the base kernel are tagged in the exception table. I didn't add the atomic operations (0x87 = XCHG etc.) that both read and write user addresses. I think they should be safe, but I need a test case where a futex has been poisoned to check. Probably this switch should be expanded with all the instructions that the compiler could possibly generate that read from user space.
Change since v1:
Moved the code to discover user address here in the severity/context flow.
Tony Luck (3):
x86/mce: Provide method to find out the type of exception handle x86/mce: Avoid tail copy when machine check terminated a copy from
user
x86/mce: Decode a kernel instruction to determine if it is copying
from user
Youquan Song (4):
x86/mce: Pass pointer to saved pt_regs to severity calculation
routines
x86/mce: Add _ASM_EXTABLE_CPY for copy user access x86/mce: Change fault_in_kernel_space() from static to global x86/mce: Recover from poison found while copying from user space
arch/x86/include/asm/asm.h | 6 ++ arch/x86/include/asm/extable.h | 9 ++- arch/x86/include/asm/mce.h | 15 ++++ arch/x86/include/asm/traps.h | 2 + arch/x86/kernel/cpu/mce/core.c | 58 +++++++++------ arch/x86/kernel/cpu/mce/internal.h | 3 +- arch/x86/kernel/cpu/mce/severity.c | 68 ++++++++++++++++-- arch/x86/lib/copy_user_64.S | 111 ++++++++++++++++------------- arch/x86/mm/extable.c | 24 +++++-- arch/x86/mm/fault.c | 2 +- include/linux/sched.h | 2 + 11 files changed, 216 insertions(+), 84 deletions(-)
base-commit: dc0592b73715c8e84ad8ebbc50c6057d5e203aac
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH v2 1/7 x86/mce: Pass pointer to saved pt_regs to severity calculation routines
From: Youquan Song <[email protected]>
New recovery features require additional information about processor state when a machine check occurred. Pass pt_regs down to the routines that need it.
No functional change.
--- arch/x86/kernel/cpu/mce/core.c | 14 +++++++------- arch/x86/kernel/cpu/mce/internal.h | 3 ++- arch/x86/kernel/cpu/mce/severity.c | 14 ++++++++------ 3 files changed, 17 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/mce/core.c +7 -7
@@ -793,7 +793,7 @@ bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b) goto clear_it;mce_read_aux (&m, i); - m.severity =mce_severity (&m, mca_cfg.tolerant,NULL , false); + m.severity =mce_severity (&m,NULL , mca_cfg.tolerant,NULL , false);/* * Don't get the IP here because it's unlikely to * have anything to do with the actual error location.
@@ -842,7 +842,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,quirk_no_way_out (i, m, regs); m->bank = i; -if (mce_severity (m, mca_cfg.tolerant, &tmp, true) >=MCE_PANIC_SEVERITY ) { +if (mce_severity (m, regs, mca_cfg.tolerant, &tmp, true) >=MCE_PANIC_SEVERITY ) {mce_read_aux (m, i); *msg = tmp;return 1 ;
@@ -942,7 +942,7 @@ static void mce_reign(void)*/ if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) {/* call mce_severity() to get "msg" for panic */ -mce_severity (m, mca_cfg.tolerant, &msg, true); +mce_severity (m,NULL , mca_cfg.tolerant, &msg, true);mce_panic ("Fatal machine check" , m, msg); }
@@ -1153,7 +1153,7 @@ static noinstr bool mce_check_crashing_cpu(void)return false; } -static void __mc_scan_banks (struct mce *m,struct mce *final, +static void __mc_scan_banks (struct mce *m,struct pt_regs *regs,struct mce *final,unsigned long *toclear,unsigned long *valid_banks,int no_way_out,int *worst) {
@@ -1188,7 +1188,7 @@ static void __mc_scan_banks(struct mce *m, struct mce *final,/* Set taint even when machine check was not enabled. */ add_taint (TAINT_MACHINE_CHECK ,LOCKDEP_NOW_UNRELIABLE ); - severity =mce_severity (m, cfg->tolerant,NULL , true); + severity =mce_severity (m, regs, cfg->tolerant,NULL , true);/* * When machine check was for corrected/deferred handler don't
@@ -1340,7 +1340,7 @@ noinstr void do_machine_check(struct pt_regs *regs) order =mce_start (&no_way_out); } -__mc_scan_banks (&m, final, toclear, valid_banks, no_way_out, &worst); +__mc_scan_banks (&m, regs, final, toclear, valid_banks, no_way_out, &worst);if (!no_way_out)mce_clear_state (toclear);
@@ -1362,7 +1362,7 @@ noinstr void do_machine_check(struct pt_regs *regs)* make sure we have the right "msg". */ if (worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) { -mce_severity (&m, cfg->tolerant, &msg, true); +mce_severity (&m, regs, cfg->tolerant, &msg, true);mce_panic ("Local fatal machine check!" , &m, msg); } }
arch/x86/kernel/cpu/mce/internal.h +2 -1
@@ -38,7 +38,8 @@ int mce_gen_pool_add(struct mce *mce);int mce_gen_pool_init (void );struct llist_node *mce_gen_pool_prepare_records (void ); -extern int (*mce_severity)(struct mce *a,int tolerant,char **msg,bool is_excp); +extern int (*mce_severity)(struct mce *a,struct pt_regs *regs, +int tolerant,char **msg,bool is_excp);struct dentry *mce_get_debugfs_dir (void );extern mce_banks_t mce_banks_ce_disabled;
arch/x86/kernel/cpu/mce/severity.c +8 -6
@@ -209,7 +209,7 @@ static struct severity {* distinguish an exception taken in user from from one * taken in the kernel. */ -static int error_context (struct mce *m) +static int error_context (struct mce *m,struct pt_regs *regs) {if ((m->cs &3 ) ==3 )return IN_USER ;
@@ -253,9 +253,10 @@ static int mce_severity_amd_smca(struct mce *m, enum context err_ctx)* See AMD Error Scope Hierarchy table in a newer BKDG. For example * 49125_15h_Models_30h-3Fh_BKDG.pdf, section "RAS Features" */ -static int mce_severity_amd (struct mce *m,int tolerant,char **msg,bool is_excp) +static int mce_severity_amd (struct mce *m,struct pt_regs *regs,int tolerant, +char **msg,bool is_excp) { -enum context ctx =error_context (m); +enum context ctx =error_context (m, regs);/* Processor Context Corrupt, no need to fumble too much, die! */ if (m->status &MCI_STATUS_PCC )
@@ -305,10 +306,11 @@ static int mce_severity_amd(struct mce *m, int tolerant, char **msg, bool is_excreturn MCE_KEEP_SEVERITY ; } -static int mce_severity_intel (struct mce *m,int tolerant,char **msg,bool is_excp) +static int mce_severity_intel (struct mce *m,struct pt_regs *regs, +int tolerant,char **msg,bool is_excp) {enum exception excp = (is_excp ?EXCP_CONTEXT :NO_EXCP ); -enum context ctx =error_context (m); +enum context ctx =error_context (m, regs);struct severity *s;for (s = severities;; s++) {
@@ -336,7 +338,7 @@ static int mce_severity_intel(struct mce *m, int tolerant, char **msg, bool is_e }/* Default to mce_severity_intel */ -int (*mce_severity)(struct mce *m,int tolerant,char **msg,bool is_excp) = +int (*mce_severity)(struct mce *m,struct pt_regs *regs,int tolerant,char **msg,bool is_excp) = mce_severity_intel;void __init mcheck_vendor_init_severity (void )
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH v2 2/7 x86/mce: Provide method to find out the type of exception handle
In reply to Tony LuckAvoid a proliferation of ex_has_*_handler() functions by having just one function that returns the type of the handler (if any).
Drop the __visible attribute for this function. It is not called from assembler so the attribute is not necessary.
--- arch/x86/include/asm/extable.h | 9 ++++++++- arch/x86/kernel/cpu/mce/severity.c | 5 ++++- arch/x86/mm/extable.c | 12 ++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)
arch/x86/include/asm/extable.h +8 -1
@@ -29,10 +29,17 @@ struct pt_regs; (b)->handler = (tmp).handler - (delta); \ } while (0) +enum handler_type { +EX_HANDLER_NONE , +EX_HANDLER_FAULT , +EX_HANDLER_UACCESS , + EX_HANDLER_OTHER +}; +extern int fixup_exception (struct pt_regs *regs,int trapnr,unsigned long error_code,unsigned long fault_addr);extern int fixup_bug (struct pt_regs *regs,int trapnr); -extern bool ex_has_fault_handler (unsigned long ip); +extern enum handler_type ex_fault_handler_type (unsigned long ip);extern void early_fixup_exception (struct pt_regs *regs,int trapnr);#endif
arch/x86/kernel/cpu/mce/severity.c +4 -1
@@ -211,10 +211,13 @@ static struct severity {*/ static int error_context (struct mce *m,struct pt_regs *regs) { +enum handler_type t; +if ((m->cs &3 ) ==3 )return IN_USER ; -if (mc_recoverable (m->mcgstatus) &&ex_has_fault_handler (m->ip)) { + t =ex_fault_handler_type (m->ip); +if (mc_recoverable (m->mcgstatus) && t ==EX_HANDLER_FAULT ) { m->kflags |=MCE_IN_KERNEL_RECOV ;return IN_KERNEL_RECOV ; }
arch/x86/mm/extable.c +8 -4
@@ -125,17 +125,21 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); -__visible bool ex_has_fault_handler(unsigned long ip) +enum handler_type ex_fault_handler_type (unsigned long ip) {const struct exception_table_entry *e;ex_handler_t handler; e =search_exception_tables (ip);if (!e) -return false; +return EX_HANDLER_NONE ; handler =ex_fixup_handler (e); - -return handler == ex_handler_fault; +if (handler == ex_handler_fault) +return EX_HANDLER_FAULT ; +else if (handler == ex_handler_uaccess) +return EX_HANDLER_UACCESS ; +else +return EX_HANDLER_OTHER ; } int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code,
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH v2 2/7] x86/mce: Provide method to find out the type of exception handle
arch/x86/mm/extable.c __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 128
Quoted 10 lines · Tony Luck
arch/x86/mm/extable.c +1
-1@@ -125,17 +125,21 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); -__visible bool ex_has_fault_handler(unsigned long ip) +enum handler_type ex_fault_handler_type (unsigned long ip)
Function name needs a verb:
s!ex_fault_handler_type!ex_get_fault_handler_type!g
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerPATCH v2 3/7 x86/mce: Add _ASM_EXTABLE_CPY for copy user access
In reply to Tony LuckFrom: Youquan Song <[email protected]>
_ASM_EXTABLE_UA is a general exception entry to record the exception fixup for all exception spots between kernel and user space access.
To enable recovery from machine checks while coping data from user addresses we need to be able to distinguish the places that are looping copying data from those that copy a single byte/word/etc.
Add a new macro _ASM_EXTABLE_CPY and use it in place of _ASM_EXTABLE_UA in the copy functions.
Record the exception reason number to regs->ax at ex_handler_uaccess which is used to check MCE triggered.
The new fixup routine ex_handler_copy() is almost an exact copy of ex_handler_uaccess() The difference is that it sets regs->ax to the trap number. Following patches use this to avoid trying to copy remaining bytes from the tail of the copy and possibly hitting the poison again.
New mce.kflags bit MCE_IN_KERNEL_COPYIN will be used by mce_severity() calculation to indicate that a machine check is recoverable because the kernel was copying from user space.
--- arch/x86/include/asm/asm.h | 6 +++ arch/x86/include/asm/mce.h | 15 ++++++ arch/x86/lib/copy_user_64.S | 96 ++++++++++++++++++------------------- arch/x86/mm/extable.c | 14 +++++- 4 files changed, 82 insertions(+), 49 deletions(-)
arch/x86/include/asm/asm.h +6 -0
@@ -135,6 +135,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
@@ -160,6 +163,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
arch/x86/include/asm/mce.h +15 -0
@@ -136,8 +136,23 @@#define MCE_HANDLED_NFIT BIT_ULL(3)#define MCE_HANDLED_EDAC BIT_ULL(4)#define MCE_HANDLED_MCELOG BIT_ULL(5) + +/* +* Indicates an MCE which has happened in kernel space but from +* which the kernel can recover simply by executing fixup_exception() +* so that an error is returned to the caller of the function that +* hit the machine check. +*/ #define MCE_IN_KERNEL_RECOV BIT_ULL(6) +/* +* Indicates an MCE that happened in kernel space while copying data +* from user. In this case fixup_exception() gets the kernel to the +* error exit for the copy function. Machine check handler can then +* treat it like a fault taken in user mode. +*/ +#define MCE_IN_KERNEL_COPYIN BIT_ULL(7) +/* * This structure contains all data related to the MCE log. Also * carries a signature to make it easier to find from external
arch/x86/lib/copy_user_64.S +48 -48
131 lines not on this page · raw message
arch/x86/mm/extable.c +13 -1
@@ -80,6 +80,18 @@ __visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_uaccess); +__visible bool ex_handler_copy(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr) +{ +WARN_ONCE (trapnr ==X86_TRAP_GP ,"General protection fault in user access. Non-canonical address?" ); + regs->ip =ex_fixup_addr (fixup); + regs->ax = trapnr; +return true; +} +EXPORT_SYMBOL (ex_handler_copy); + __visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup, struct pt_regs *regs, int trapnr, unsigned long error_code,
@@ -136,7 +148,7 @@ enum handler_type ex_fault_handler_type(unsigned long ip) handler =ex_fixup_handler (e);if (handler == ex_handler_fault)return EX_HANDLER_FAULT ; -else if (handler == ex_handler_uaccess) +else if (handler == ex_handler_uaccess || handler == ex_handler_copy)return EX_HANDLER_UACCESS ;else return EX_HANDLER_OTHER ;
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH v2 3/7] x86/mce: Add _ASM_EXTABLE_CPY for copy user access
Quoted 7 lines · Tony Luck
From: Youquan Song <[email protected]>
_ASM_EXTABLE_UAis a general exception entry to record the exception fixup for all exception spots between kernel and user space access.To enable recovery from machine checks while coping data from user addresses we need to be able to distinguish the places that are
Who's "we"?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerPATCH v2 4/7 x86/mce: Avoid tail copy when machine check terminated a copy from user
In reply to Tony LuckIn the page fault case it is ok to see if a few more unaligned bytes can be copied from the source address. Worst case is that the page fault will be triggered again.
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return from the copy code as if the copy succeeded. The machine check handler will use task_work_add() to make sure that the task is sent a SIGBUS.
--- arch/x86/lib/copy_user_64.S | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/x86/lib/copy_user_64.S +15 -0
@@ -15,6 +15,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> .macro ALIGN_DESTINATION /* check for bad alignment of destination */
@@ -221,6 +222,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Try to copy last bytes and clear the rest if needed. * Since protection fault in copy_from/to_user is not a normal situation, * it is not necessary to optimize tail handling. + * Don't try to copy the tail if machine check happened * * Input: * rdi destination
@@ -232,11 +234,24 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx + cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ + je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret + /* + * Return zero to pretend that this copy succeeded. This + * is counter-intuitive, but needed to prevent the code + * in lib/iov_iter.c from retrying and running back into + * the poison cache line again. The machine check handler + * will ensure that a SIGBUS is sent to the task. + */ +3: xorl %eax,%eax + ASM_CLAC + ret + _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail)
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH v2 5/7 x86/mce: Change fault_in_kernel_space() from static to global
In reply to Tony LuckFrom: Youquan Song <[email protected]>
Machine check code needs to be able to determine if a faulting address is in user or kernel space. There is already a function to do this.
Change from "static int" to "bool" and add declaration to <asm/traps.h>
No functional change.
--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
arch/x86/include/asm/traps.h +2 -0
@@ -35,6 +35,8 @@ extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); +bool fault_in_kernel_space (unsigned long address); +#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/mm/fault.c +1 -1
@@ -1081,7 +1081,7 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)return 0 ; } -static int fault_in_kernel_space (unsigned long address) +bool fault_in_kernel_space (unsigned long address) {/* * On 64-bit systems, the vsyscall page is at an address above
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH v2 5/7] x86/mce: Change fault_in_kernel_space() from static to global
Quoted 35 lines · Tony Luck
From: Youquan Song <[email protected]>
Machine check code needs to be able to determine if a faulting address is in user or kernel space. There is already a function to do this.
Change from "static int" to "bool" and add declaration to <
asm/traps.h>No functional change.
Signed-off-by Youquan Song <[email protected]>Signed-off-by <[email protected]>--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/mm/fault.c | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)arch/x86/include/asm/traps.h +2
-0@@ -35,6 +35,8 @@ extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); +bool fault_in_kernel_space (unsigned long address); +#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,arch/x86/mm/fault.c +0
-0
arch/x86/mm/fault.c access_error(unsigned long error_code, struct vm_area_struct *vma) · line 1087
Quoted 8 lines
arch/x86/mm/fault.c +1
-1@@ -1081,7 +1081,7 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)return 0 ; } -static int fault_in_kernel_space (unsigned long address) +bool fault_in_kernel_space (unsigned long address) {/* * On 64-bit systems, the vsyscall page is at an address above
Yeah, merge this one into the last patch where this function is used.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerPATCH v2 6/7 x86/mce: Recover from poison found while copying from user space
In reply to Tony LuckFrom: Youquan Song <[email protected]>
Existing kernel code can only recover from a machine check on code that is tagged in the exception table with a fault handling recovery path.
Two new fields in the task structure to pass information from machine check handler to the "task_work" that is queued to run before the task returns to user mode:
+ mce_vaddr: will be initialized to the user virtual address of the fault in the case where the fault occurred in the kernel copying data from a user address. This is so that kill_me_maybe() can provide that information to the user SIGBUS handler.
+ mce_kflags: copy of the struct mce.kflags needed by kill_me_maybe() to determine if mce_vaddr is applicable to this error.
Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task.
Refactor the recovery code path to share common code between the "fault in user mode" case and the "fault while copying from user" case.
New code paths will be activated by the next patch which sets MCE_IN_KERNEL_COPYIN.
--- arch/x86/kernel/cpu/mce/core.c | 33 +++++++++++++++++++++------------ include/linux/sched.h | 2 ++ 2 files changed, 23 insertions(+), 12 deletions(-)
arch/x86/kernel/cpu/mce/core.c +21 -12
@@ -1386,15 +1386,10 @@ noinstr void do_machine_check(struct pt_regs *regs)if ((m.cs &3 ) ==3 ) {/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); - - current->mce_addr = m.addr; - current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (&m); - current->mce_kill_me.func = kill_me_maybe; -if (kill_it) - current->mce_kill_me.func = kill_me_now; -task_work_add (current, ¤t->mce_kill_me, true); }else { +/* (m.kflags MCE_IN_KERNEL_RECOV must have been set to get here */ +BUG_ON (!(m.kflags &MCE_IN_KERNEL_RECOV )); +/* * Handle an MCE which has happened in kernel space but from * which the kernel can recover: ex_has_fault_handler() has
@@ -1404,11 +1399,25 @@ noinstr void do_machine_check(struct pt_regs *regs)* corresponding exception handler which would do that is the * proper one. */ -if (m.kflags &MCE_IN_KERNEL_RECOV ) { -if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 )) -mce_panic ("Failed kernel mode recovery" , &m, msg); - } +if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 )) +mce_panic ("Failed kernel mode recovery" , &m, msg); + +/* +* If this wasn't a copy from user, then recovery is complete. +*/ +if (!(m.kflags &MCE_IN_KERNEL_COPYIN )) + goto out; } + + current->mce_addr = m.addr; + current->mce_kflags = m.kflags; + current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (&m); + current->mce_kill_me.func = kill_me_maybe; +if (kill_it) + current->mce_kill_me.func = kill_me_now; +task_work_add (current, ¤t->mce_kill_me, true); + out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 ); }
include/linux/sched.h +2 -0
@@ -1308,6 +1308,8 @@ struct task_struct {#endif #ifdef CONFIG_X86_MCE +void __user *mce_vaddr; +__u64 mce_kflags;u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1,
Signature · 1 line
2.21.1
Borislav Petkov
maintainerQuoted 27 lines · Tony Luck
From: Youquan Song <[email protected]>
Existing kernel code can only recover from a machine check on code that is tagged in the exception table with a fault handling recovery path.
Two new fields in the task structure to pass information from machine check handler to the "
task_work" that is queued to run before the task returns to user mode:+
mce_vaddr: will be initialized to the user virtual address of the fault in the case where the fault occurred in the kernel copying data from a user address. This is so thatkill_me_maybe()can provide that information to the user SIGBUS handler.+
mce_kflags: copy of the struct mce.kflags needed bykill_me_maybe()to determine ifmce_vaddris applicable to this error.Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task.
Refactor the recovery code path to share common code between the "fault in user mode" case and the "fault while copying from user" case.
New code paths will be activated by the next patch which sets
MCE_IN_KERNEL_COPYIN.
Quoted 6 lines
Signed-off-by Youquan Song <[email protected]>Signed-off-by Tony Luck <[email protected]>--- arch/x86/kernel/cpu/mce/core.c | 33 +++++++++++++++++++++------------ include/linux/sched.h | 2 ++ 2 files changed, 23 insertions(+), 12 deletions(-)
Isn't that just simpler?
arch/x86/kernel/cpu/mce/core.c +20 -7
@@ -1261,6 +1261,21 @@ static void kill_me_maybe(struct callback_head *cb)kill_me_now (cb); } +static inline void queue_task_work (struct mce *m,int kill_it) +{ + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + +if (kill_it) + current->mce_kill_me.func = kill_me_now; +else + current->mce_kill_me.func = kill_me_maybe; + +task_work_add (current, ¤t->mce_kill_me, true); +} +/* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18.
@@ -1402,13 +1417,8 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); - current->mce_addr = m.addr; - current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (&m); - current->mce_kill_me.func = kill_me_maybe; -if (kill_it) - current->mce_kill_me.func = kill_me_now; -task_work_add (current, ¤t->mce_kill_me, true); +queue_task_work (&m, kill_it); + }else {/* * Handle an MCE which has happened in kernel space but from
@@ -1423,6 +1433,9 @@ noinstr void do_machine_check(struct pt_regs *regs)if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 ))mce_panic ("Failed kernel mode recovery" , &m, msg); } + +if (m.kflags &MCE_IN_KERNEL_COPYIN ) +queue_task_work (&m, kill_it); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +2 -0
@@ -1308,6 +1308,8 @@ struct task_struct {#endif #ifdef CONFIG_X86_MCE +void __user *mce_vaddr; +__u64 mce_kflags;u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1,
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerQuoted 6 lines · Borislav Petkov
Quoted 3 lines · Tony Luck
arch/x86/kernel/cpu/mce/core.c | 33 +++++++++++++++++++++------------ include/linux/sched.h | 2 ++ 2 files changed, 23 insertions(+), 12 deletions(-)Isn't that just simpler?
Yes. A helper function avoids making the code a mess of if/else with subtle fall through cases.
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1264
Quoted 9 lines
arch/x86/kernel/cpu/mce/core.c +1
-0@@ -1261,6 +1261,21 @@ static void kill_me_maybe(struct callback_head *cb)kill_me_now (cb); } +static inline void queue_task_work (struct mce *m,int kill_it)
Does it need to be "inline" though? I hope machine check processing is never in the critical code path for anyone!
Quoted 34 lines
arch/x86/kernel/cpu/mce/core.c +16
-7+{ + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + +if (kill_it) + current->mce_kill_me.func = kill_me_now; +else + current->mce_kill_me.func = kill_me_maybe; + +task_work_add (current, ¤t->mce_kill_me, true); +} +/* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18. @@ -1402,13 +1417,8 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); - current->mce_addr = m.addr; - current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (&m); - current->mce_kill_me.func = kill_me_maybe; -if (kill_it) - current->mce_kill_me.func = kill_me_now; -task_work_add (current, ¤t->mce_kill_me, true); +queue_task_work (&m, kill_it); + }else {/* * Handle an MCE which has happened in kernel space but from
arch/x86/kernel/cpu/mce/core.c noinstr void do_machine_check(struct pt_regs *regs) · line 1441
Quoted 9 lines
arch/x86/kernel/cpu/mce/core.c +3
-0@@ -1423,6 +1433,9 @@ noinstr void do_machine_check(struct pt_regs *regs)if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 ))mce_panic ("Failed kernel mode recovery" , &m, msg); } + +if (m.kflags &MCE_IN_KERNEL_COPYIN ) +queue_task_work (&m, kill_it); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
-Tony
Tony Luck
maintainerPATCH v2 7/7 x86/mce: Decode a kernel instruction to determine if it is copying from user
In reply to Tony LuckAll instructions copying data between kernel and user memory are tagged with either _ASM_EXTABLE_UA or _ASM_EXTABLE_CPY entries in the exception table. ex_fault_handler_type() returns EX_HANDLER_UACCESS for both of these.
Recovery is only possible when the machine check was triggered on a read from user memory. In this case the same strategy for recovery applies as if the user had made the access in ring3. If the fault was in kernel memory while copying to user we do not currently have a recovery plan.
For MOV and MOVZ instructions a full decode of the instruction is done to find the source address. For MOVS instructions the source address is in the %rsi register. The function fault_in_kernel_space() determines whether the source address is kernel or user.
--- arch/x86/kernel/cpu/mce/core.c | 11 +++++-- arch/x86/kernel/cpu/mce/severity.c | 51 +++++++++++++++++++++++++++++- 2 files changed, 58 insertions(+), 4 deletions(-)
arch/x86/kernel/cpu/mce/core.c +8 -3
@@ -1236,14 +1236,19 @@ static void kill_me_maybe(struct callback_head *cb)if (!p->mce_ripv) flags |=MF_MUST_KILL ; -if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags)) { +if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags) && + !(p->mce_kflags &MCE_IN_KERNEL_COPYIN )) {set_mce_nospec (p->mce_addr >>PAGE_SHIFT , p->mce_whole_page);sync_core ();return ; } -pr_err ("Memory error not recovered" ); -kill_me_now (cb); +if (p->mce_vaddr != (void __user *)~0ul ) { +force_sig_mceerr (BUS_MCEERR_AR , p->mce_vaddr,PAGE_SHIFT ); + }else { +pr_err ("Memory error not recovered" ); +kill_me_now (cb); + } }/*
arch/x86/kernel/cpu/mce/severity.c +50 -1
72 lines not on this page · raw message
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH v2 7/7] x86/mce: Decode a kernel instruction to determine if it is copying from user
Quoted 9 lines · Tony Luck
arch/x86/kernel/cpu/mce/core.c +0
-1@@ -1236,14 +1236,19 @@ static void kill_me_maybe(struct callback_head *cb)if (!p->mce_ripv) flags |=MF_MUST_KILL ; -if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags)) {
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1246
Quoted 10 lines
arch/x86/kernel/cpu/mce/core.c +3
-2+if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags) && + !(p->mce_kflags &MCE_IN_KERNEL_COPYIN )) {set_mce_nospec (p->mce_addr >>PAGE_SHIFT , p->mce_whole_page);sync_core ();return ; } -pr_err ("Memory error not recovered" ); -kill_me_now (cb); +if (p->mce_vaddr != (void __user *)~0ul ) {
As previously pointed out, pls test against -1L even if it is the same value so that it is obvious this is the error value coming from insn_get_addr_ref().
Quoted 37 lines
arch/x86/kernel/cpu/mce/core.c +5
-0+force_sig_mceerr (BUS_MCEERR_AR , p->mce_vaddr,PAGE_SHIFT ); + }else { +pr_err ("Memory error not recovered" ); +kill_me_now (cb); + } }/* arch/x86/kernel/cpu/mce/severity.c +14
-0@@ -10,6 +10,9 @@#include <linux/init.h> #include <linux/debugfs.h> #include <asm/mce.h> +#include <asm/traps.h> +#include <asm/insn.h> +#include <asm/insn-eval.h> #include <linux/uaccess.h> #include "internal.h" @@ -198,6 +201,45 @@ static struct severity {#define mc_recoverable (mcg) (((mcg) & (MCG_STATUS_RIPV|MCG_STATUS_EIPV)) == \ (MCG_STATUS_RIPV|MCG_STATUS_EIPV)) +static bool is_copy_from_user (struct pt_regs *regs) +{ +u8 insn_buf[MAX_INSN_SIZE ]; +struct insn insn; +unsigned long addr; + +if (copy_from_kernel_nofault (insn_buf, (void *)regs->ip,MAX_INSN_SIZE )) +return false; + +kernel_insn_init (&insn, insn_buf,MAX_INSN_SIZE ); +insn_get_opcode (&insn);
arch/x86/kernel/cpu/mce/severity.c static struct severity { · line 224
Quoted 10 lines
arch/x86/kernel/cpu/mce/severity.c +10
-0+if (!insn.opcode.got) +return false; + +switch (insn.opcode.value) { +/* MOV mem,reg */ +case 0x8A :case 0x8B : +/* MOVZ mem,reg */ +case 0xB60F :case 0xB70F : +insn_get_modrm (&insn); +insn_get_sib (&insn);
You need to test here:
insn->modrm.got = 1;
and
insn->sib.got = 1;
I know, this is weird - those functions should return an error value instead of being void and I've asked Masami in the past but no reply.
Who knows, one fine day I might convert the crap to do that instead.
Quoted 21 lines
arch/x86/kernel/cpu/mce/severity.c +18
-0+ addr = (unsigned long )insn_get_addr_ref (&insn, regs); +break ; +/* REP MOVS */ +case 0xA4 :case 0xA5 : + addr = regs->si; +break ; +default : +return false; + } + +if (fault_in_kernel_space (addr)) +return false; + + current->mce_vaddr = (void __user *)addr; + +return true; +} +/* * If mcgstatus indicated that ip/cs on the stack were * no good, then "m->cs" will be zero and we will have
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteTony Luck
maintainerv3 cover Add machine check recovery when copying from user space
Machine check recovery from uncorrected memory errors currently focusses primarily on errors that are detected while running in user mode. There is a mechanism for recovering from errors in kernel code, but it is currently only used for memcpy_mcsafe().
The existing recover actions for errors found in user mode (unmap the page and send SIGBUS to the task) can also be applied when the error is found while copying data from user space to the kernel.
Roadmap to this series:
Series is based on top of tip ras/core branch because original part 0001 has already been applied to tip ras/core branch:13c877f4b48b ("x86/mce: Stop mce_reign() from re-computing severity for every CPU") so new part 0001 below used to be part 0002 in v1 series.
In v3 part 0005 has been merged with the final part, so now just 6 parts.
0001: First piece of infrastructure update. Severity calculations need
access to the saved registers. So pass pointer down the call chain.
0002: Need to know what type of exception handler is present
for a given kernel instruction. Rather than proliferate more functions like ex_has_fault_handler() for each type, replace with a function that looks up the handler and returns an enum describing the type.
0003: Need slightly different handling for *copy_user*() faults from
get_user() faults. Create a new exception table tag and apply to the copy functions.
Change since v2: Reword commit message to avoid use of "we".
0004: In fixup path of copy functions avoid dealing with the tail
when the copy took a machine check by returning that there are no bytes left to be copied.
0005: Changes to do_machine_check() to support the new recovery flow.
Some re-factoring to avoid code duplication (since the flows for "error in user mode" and "error while copying from user mode" are almost identical). Couple of new fields added to the task structure.
Change since v2: Boris supplied a helper function to make the re-factor
much simpler. Use it instead of the spaghetti code in v2.
0006: Finally the keystone patch that pulls all the parts together.
An instruction decoder figures out whether an instruction tagged as accessing user space is reading from or writing to user space. The instructions in the switch were found experimentally by looking at what instructions in the base kernel are tagged in the exception table. I didn't add the atomic operations (0x87 = XCHG etc.) that both read and write user addresses. I think they should be safe, but I need a test case where a futex has been poisoned to check. Probably this switch should be expanded with all the instructions that the compiler could possibly generate that read from user space.
Change since v2: Merged old part 0005 into this piece since this is
where function fault_in_kernel_space() is used. Check modrm.got and sib.got fields in "insn" were set before calling insn_get_addr()
Change type of constant from ~0ul to -1l when checking whether address returned by insn_get_addr() is valid.
Tony Luck (4):
x86/mce: Provide method to find out the type of exception handle x86/mce: Avoid tail copy when machine check terminated a copy from
user
x86/mce: Recover from poison found while copying from user space x86/mce: Decode a kernel instruction to determine if it is copying
from user
Youquan Song (2):
x86/mce: Pass pointer to saved pt_regs to severity calculation
routines
x86/mce: Add _ASM_EXTABLE_CPY for copy user access
arch/x86/include/asm/asm.h | 6 ++ arch/x86/include/asm/extable.h | 9 ++- arch/x86/include/asm/mce.h | 15 ++++ arch/x86/include/asm/traps.h | 2 + arch/x86/kernel/cpu/mce/core.c | 52 +++++++++----- arch/x86/kernel/cpu/mce/internal.h | 3 +- arch/x86/kernel/cpu/mce/severity.c | 70 ++++++++++++++++-- arch/x86/lib/copy_user_64.S | 111 ++++++++++++++++------------- arch/x86/mm/extable.c | 24 +++++-- arch/x86/mm/fault.c | 2 +- include/linux/sched.h | 2 + 11 files changed, 217 insertions(+), 79 deletions(-)
base-commit: 5da8e4a658109e3b7e1f45ae672b7c06ac3e7158
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH v3 1/6 x86/mce: Pass pointer to saved pt_regs to severity calculation routines
From: Youquan Song <[email protected]>
New recovery features require additional information about processor state when a machine check occurred. Pass pt_regs down to the routines that need it.
No functional change.
--- arch/x86/kernel/cpu/mce/core.c | 14 +++++++------- arch/x86/kernel/cpu/mce/internal.h | 3 ++- arch/x86/kernel/cpu/mce/severity.c | 14 ++++++++------ 3 files changed, 17 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/mce/core.c +7 -7
@@ -807,7 +807,7 @@ bool machine_check_poll(enum mcp_flags flags, mce_banks_t *b) goto clear_it;mce_read_aux (&m, i); - m.severity =mce_severity (&m, mca_cfg.tolerant,NULL , false); + m.severity =mce_severity (&m,NULL , mca_cfg.tolerant,NULL , false);/* * Don't get the IP here because it's unlikely to * have anything to do with the actual error location.
@@ -856,7 +856,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,quirk_no_way_out (i, m, regs); m->bank = i; -if (mce_severity (m, mca_cfg.tolerant, &tmp, true) >=MCE_PANIC_SEVERITY ) { +if (mce_severity (m, regs, mca_cfg.tolerant, &tmp, true) >=MCE_PANIC_SEVERITY ) {mce_read_aux (m, i); *msg = tmp;return 1 ;
@@ -956,7 +956,7 @@ static void mce_reign(void)*/ if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) {/* call mce_severity() to get "msg" for panic */ -mce_severity (m, mca_cfg.tolerant, &msg, true); +mce_severity (m,NULL , mca_cfg.tolerant, &msg, true);mce_panic ("Fatal machine check" , m, msg); }
@@ -1167,7 +1167,7 @@ static noinstr bool mce_check_crashing_cpu(void)return false; } -static void __mc_scan_banks (struct mce *m,struct mce *final, +static void __mc_scan_banks (struct mce *m,struct pt_regs *regs,struct mce *final,unsigned long *toclear,unsigned long *valid_banks,int no_way_out,int *worst) {
@@ -1202,7 +1202,7 @@ static void __mc_scan_banks(struct mce *m, struct mce *final,/* Set taint even when machine check was not enabled. */ add_taint (TAINT_MACHINE_CHECK ,LOCKDEP_NOW_UNRELIABLE ); - severity =mce_severity (m, cfg->tolerant,NULL , true); + severity =mce_severity (m, regs, cfg->tolerant,NULL , true);/* * When machine check was for corrected/deferred handler don't
@@ -1354,7 +1354,7 @@ noinstr void do_machine_check(struct pt_regs *regs) order =mce_start (&no_way_out); } -__mc_scan_banks (&m, final, toclear, valid_banks, no_way_out, &worst); +__mc_scan_banks (&m, regs, final, toclear, valid_banks, no_way_out, &worst);if (!no_way_out)mce_clear_state (toclear);
@@ -1376,7 +1376,7 @@ noinstr void do_machine_check(struct pt_regs *regs)* make sure we have the right "msg". */ if (worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) { -mce_severity (&m, cfg->tolerant, &msg, true); +mce_severity (&m, regs, cfg->tolerant, &msg, true);mce_panic ("Local fatal machine check!" , &m, msg); } }
arch/x86/kernel/cpu/mce/internal.h +2 -1
@@ -38,7 +38,8 @@ int mce_gen_pool_add(struct mce *mce);int mce_gen_pool_init (void );struct llist_node *mce_gen_pool_prepare_records (void ); -extern int (*mce_severity)(struct mce *a,int tolerant,char **msg,bool is_excp); +extern int (*mce_severity)(struct mce *a,struct pt_regs *regs, +int tolerant,char **msg,bool is_excp);struct dentry *mce_get_debugfs_dir (void );extern mce_banks_t mce_banks_ce_disabled;
arch/x86/kernel/cpu/mce/severity.c +8 -6
@@ -223,7 +223,7 @@ static struct severity {* distinguish an exception taken in user from from one * taken in the kernel. */ -static int error_context (struct mce *m) +static int error_context (struct mce *m,struct pt_regs *regs) {if ((m->cs &3 ) ==3 )return IN_USER ;
@@ -267,9 +267,10 @@ static int mce_severity_amd_smca(struct mce *m, enum context err_ctx)* See AMD Error Scope Hierarchy table in a newer BKDG. For example * 49125_15h_Models_30h-3Fh_BKDG.pdf, section "RAS Features" */ -static int mce_severity_amd (struct mce *m,int tolerant,char **msg,bool is_excp) +static int mce_severity_amd (struct mce *m,struct pt_regs *regs,int tolerant, +char **msg,bool is_excp) { -enum context ctx =error_context (m); +enum context ctx =error_context (m, regs);/* Processor Context Corrupt, no need to fumble too much, die! */ if (m->status &MCI_STATUS_PCC )
@@ -319,10 +320,11 @@ static int mce_severity_amd(struct mce *m, int tolerant, char **msg, bool is_excreturn MCE_KEEP_SEVERITY ; } -static int mce_severity_intel (struct mce *m,int tolerant,char **msg,bool is_excp) +static int mce_severity_intel (struct mce *m,struct pt_regs *regs, +int tolerant,char **msg,bool is_excp) {enum exception excp = (is_excp ?EXCP_CONTEXT :NO_EXCP ); -enum context ctx =error_context (m); +enum context ctx =error_context (m, regs);struct severity *s;for (s = severities;; s++) {
@@ -356,7 +358,7 @@ static int mce_severity_intel(struct mce *m, int tolerant, char **msg, bool is_e }/* Default to mce_severity_intel */ -int (*mce_severity)(struct mce *m,int tolerant,char **msg,bool is_excp) = +int (*mce_severity)(struct mce *m,struct pt_regs *regs,int tolerant,char **msg,bool is_excp) = mce_severity_intel;void __init mcheck_vendor_init_severity (void )
Signature · 1 line
2.21.1
[tip: ras/core] x86/mce: Pass pointer to saved pt_regs to severity calculation routines
The following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: 41ce0564bfe2e129d56730418d8c0a9f9f2d31b5 Gitweb: https://git.kernel.org/tip/41ce0564bfe2e129d56730418d8c0a9f9f2d31b5 Author: Youquan Song <[email protected]> AuthorDate: Tue, 06 Oct 2020 14:09:05 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 07 Oct 2020 10:51:42 +02:00
x86/mce: Pass pointer to saved pt_regs to severity calculation routines
New recovery features require additional information about processor state when a machine check occurred. Pass pt_regs down to the routines that need it.
No functional change.
--- arch/x86/kernel/cpu/mce/core.c | 14 +++++++------- arch/x86/kernel/cpu/mce/internal.h | 3 ++- arch/x86/kernel/cpu/mce/severity.c | 14 ++++++++------ 3 files changed, 17 insertions(+), 14 deletions(-)
arch/x86/kernel/cpu/mce/core.c +7 -7
@@ -807,7 +807,7 @@ log_it: goto clear_it;mce_read_aux (&m, i); - m.severity =mce_severity (&m, mca_cfg.tolerant,NULL , false); + m.severity =mce_severity (&m,NULL , mca_cfg.tolerant,NULL , false);/* * Don't get the IP here because it's unlikely to * have anything to do with the actual error location.
@@ -856,7 +856,7 @@ static int mce_no_way_out(struct mce *m, char **msg, unsigned long *validp,quirk_no_way_out (i, m, regs); m->bank = i; -if (mce_severity (m, mca_cfg.tolerant, &tmp, true) >=MCE_PANIC_SEVERITY ) { +if (mce_severity (m, regs, mca_cfg.tolerant, &tmp, true) >=MCE_PANIC_SEVERITY ) {mce_read_aux (m, i); *msg = tmp;return 1 ;
@@ -956,7 +956,7 @@ static void mce_reign(void)*/ if (m && global_worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) {/* call mce_severity() to get "msg" for panic */ -mce_severity (m, mca_cfg.tolerant, &msg, true); +mce_severity (m,NULL , mca_cfg.tolerant, &msg, true);mce_panic ("Fatal machine check" , m, msg); }
@@ -1167,7 +1167,7 @@ static noinstr bool mce_check_crashing_cpu(void)return false; } -static void __mc_scan_banks (struct mce *m,struct mce *final, +static void __mc_scan_banks (struct mce *m,struct pt_regs *regs,struct mce *final,unsigned long *toclear,unsigned long *valid_banks,int no_way_out,int *worst) {
@@ -1202,7 +1202,7 @@ static void __mc_scan_banks(struct mce *m, struct mce *final,/* Set taint even when machine check was not enabled. */ add_taint (TAINT_MACHINE_CHECK ,LOCKDEP_NOW_UNRELIABLE ); - severity =mce_severity (m, cfg->tolerant,NULL , true); + severity =mce_severity (m, regs, cfg->tolerant,NULL , true);/* * When machine check was for corrected/deferred handler don't
@@ -1354,7 +1354,7 @@ noinstr void do_machine_check(struct pt_regs *regs) order =mce_start (&no_way_out); } -__mc_scan_banks (&m, final, toclear, valid_banks, no_way_out, &worst); +__mc_scan_banks (&m, regs, final, toclear, valid_banks, no_way_out, &worst);if (!no_way_out)mce_clear_state (toclear);
@@ -1376,7 +1376,7 @@ noinstr void do_machine_check(struct pt_regs *regs)* make sure we have the right "msg". */ if (worst >=MCE_PANIC_SEVERITY && mca_cfg.tolerant <3 ) { -mce_severity (&m, cfg->tolerant, &msg, true); +mce_severity (&m, regs, cfg->tolerant, &msg, true);mce_panic ("Local fatal machine check!" , &m, msg); } }
arch/x86/kernel/cpu/mce/internal.h +2 -1
@@ -38,7 +38,8 @@ int mce_gen_pool_add(struct mce *mce);int mce_gen_pool_init (void );struct llist_node *mce_gen_pool_prepare_records (void ); -extern int (*mce_severity)(struct mce *a,int tolerant,char **msg,bool is_excp); +extern int (*mce_severity)(struct mce *a,struct pt_regs *regs, +int tolerant,char **msg,bool is_excp);struct dentry *mce_get_debugfs_dir (void );extern mce_banks_t mce_banks_ce_disabled;
arch/x86/kernel/cpu/mce/severity.c +8 -6
@@ -223,7 +223,7 @@ static struct severity {* distinguish an exception taken in user from from one * taken in the kernel. */ -static int error_context (struct mce *m) +static int error_context (struct mce *m,struct pt_regs *regs) {if ((m->cs &3 ) ==3 )return IN_USER ;
@@ -267,9 +267,10 @@ static int mce_severity_amd_smca(struct mce *m, enum context err_ctx)* See AMD Error Scope Hierarchy table in a newer BKDG. For example * 49125_15h_Models_30h-3Fh_BKDG.pdf, section "RAS Features" */ -static int mce_severity_amd (struct mce *m,int tolerant,char **msg,bool is_excp) +static int mce_severity_amd (struct mce *m,struct pt_regs *regs,int tolerant, +char **msg,bool is_excp) { -enum context ctx =error_context (m); +enum context ctx =error_context (m, regs);/* Processor Context Corrupt, no need to fumble too much, die! */ if (m->status &MCI_STATUS_PCC )
@@ -319,10 +320,11 @@ static int mce_severity_amd(struct mce *m, int tolerant, char **msg, bool is_excreturn MCE_KEEP_SEVERITY ; } -static int mce_severity_intel (struct mce *m,int tolerant,char **msg,bool is_excp) +static int mce_severity_intel (struct mce *m,struct pt_regs *regs, +int tolerant,char **msg,bool is_excp) {enum exception excp = (is_excp ?EXCP_CONTEXT :NO_EXCP ); -enum context ctx =error_context (m); +enum context ctx =error_context (m, regs);struct severity *s;for (s = severities;; s++) {
@@ -356,7 +358,7 @@ static int mce_severity_intel(struct mce *m, int tolerant, char **msg, bool is_e }/* Default to mce_severity_intel */ -int (*mce_severity)(struct mce *m,int tolerant,char **msg,bool is_excp) = +int (*mce_severity)(struct mce *m,struct pt_regs *regs,int tolerant,char **msg,bool is_excp) = mce_severity_intel;void __init mcheck_vendor_init_severity (void )
Tony Luck
maintainerPATCH v3 2/6 x86/mce: Provide method to find out the type of exception handle
In reply to Tony LuckAvoid a proliferation of ex_has_*_handler() functions by having just one function that returns the type of the handler (if any).
Drop the __visible attribute for this function. It is not called from assembler so the attribute is not necessary.
--- arch/x86/include/asm/extable.h | 9 ++++++++- arch/x86/kernel/cpu/mce/severity.c | 5 ++++- arch/x86/mm/extable.c | 12 ++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)
arch/x86/include/asm/extable.h +8 -1
@@ -29,10 +29,17 @@ struct pt_regs; (b)->handler = (tmp).handler - (delta); \ } while (0) +enum handler_type { +EX_HANDLER_NONE , +EX_HANDLER_FAULT , +EX_HANDLER_UACCESS , + EX_HANDLER_OTHER +}; +extern int fixup_exception (struct pt_regs *regs,int trapnr,unsigned long error_code,unsigned long fault_addr);extern int fixup_bug (struct pt_regs *regs,int trapnr); -extern bool ex_has_fault_handler (unsigned long ip); +extern enum handler_type ex_get_fault_handler_type (unsigned long ip);extern void early_fixup_exception (struct pt_regs *regs,int trapnr);#endif
arch/x86/kernel/cpu/mce/severity.c +4 -1
@@ -225,10 +225,13 @@ static struct severity {*/ static int error_context (struct mce *m,struct pt_regs *regs) { +enum handler_type t; +if ((m->cs &3 ) ==3 )return IN_USER ; -if (mc_recoverable (m->mcgstatus) &&ex_has_fault_handler (m->ip)) { + t =ex_get_fault_handler_type (m->ip); +if (mc_recoverable (m->mcgstatus) && t ==EX_HANDLER_FAULT ) { m->kflags |=MCE_IN_KERNEL_RECOV ;return IN_KERNEL_RECOV ; }
arch/x86/mm/extable.c +8 -4
@@ -125,17 +125,21 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); -__visible bool ex_has_fault_handler(unsigned long ip) +enum handler_type ex_get_fault_handler_type (unsigned long ip) {const struct exception_table_entry *e;ex_handler_t handler; e =search_exception_tables (ip);if (!e) -return false; +return EX_HANDLER_NONE ; handler =ex_fixup_handler (e); - -return handler == ex_handler_fault; +if (handler == ex_handler_fault) +return EX_HANDLER_FAULT ; +else if (handler == ex_handler_uaccess) +return EX_HANDLER_UACCESS ; +else +return EX_HANDLER_OTHER ; } int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code,
Signature · 1 line
2.21.1
[tip: ras/core] x86/mce: Provide method to find out the type of an exception handler
The following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: a05d54c41ecfa1a322b229b4e5ce50c157284f74 Gitweb: https://git.kernel.org/tip/a05d54c41ecfa1a322b229b4e5ce50c157284f74 Author: Tony Luck <[email protected]> AuthorDate: Tue, 06 Oct 2020 14:09:06 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 07 Oct 2020 11:08:59 +02:00
x86/mce: Provide method to find out the type of an exception handler
Avoid a proliferation of ex_has_*_handler() functions by having just one function that returns the type of the handler (if any).
Drop the __visible attribute for this function. It is not called from assembler so the attribute is not necessary.
--- arch/x86/include/asm/extable.h | 9 ++++++++- arch/x86/kernel/cpu/mce/severity.c | 5 ++++- arch/x86/mm/extable.c | 12 ++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)
arch/x86/include/asm/extable.h +8 -1
@@ -29,10 +29,17 @@ struct pt_regs; (b)->handler = (tmp).handler - (delta); \ } while (0) +enum handler_type { +EX_HANDLER_NONE , +EX_HANDLER_FAULT , +EX_HANDLER_UACCESS , + EX_HANDLER_OTHER +}; +extern int fixup_exception (struct pt_regs *regs,int trapnr,unsigned long error_code,unsigned long fault_addr);extern int fixup_bug (struct pt_regs *regs,int trapnr); -extern bool ex_has_fault_handler (unsigned long ip); +extern enum handler_type ex_get_fault_handler_type (unsigned long ip);extern void early_fixup_exception (struct pt_regs *regs,int trapnr);#endif
arch/x86/kernel/cpu/mce/severity.c +4 -1
@@ -225,10 +225,13 @@ static struct severity {*/ static int error_context (struct mce *m,struct pt_regs *regs) { +enum handler_type t; +if ((m->cs &3 ) ==3 )return IN_USER ; -if (mc_recoverable (m->mcgstatus) &&ex_has_fault_handler (m->ip)) { + t =ex_get_fault_handler_type (m->ip); +if (mc_recoverable (m->mcgstatus) && t ==EX_HANDLER_FAULT ) { m->kflags |=MCE_IN_KERNEL_RECOV ;return IN_KERNEL_RECOV ; }
arch/x86/mm/extable.c +8 -4
@@ -125,17 +125,21 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); -__visible bool ex_has_fault_handler(unsigned long ip) +enum handler_type ex_get_fault_handler_type (unsigned long ip) {const struct exception_table_entry *e;ex_handler_t handler; e =search_exception_tables (ip);if (!e) -return false; +return EX_HANDLER_NONE ; handler =ex_fixup_handler (e); - -return handler == ex_handler_fault; +if (handler == ex_handler_fault) +return EX_HANDLER_FAULT ; +else if (handler == ex_handler_uaccess) +return EX_HANDLER_UACCESS ; +else +return EX_HANDLER_OTHER ; } int fixup_exception(struct pt_regs *regs, int trapnr, unsigned long error_code,
Tony Luck
maintainerPATCH v3 3/6 x86/mce: Add _ASM_EXTABLE_CPY for copy user access
In reply to Tony LuckFrom: Youquan Song <[email protected]>
_ASM_EXTABLE_UA is a general exception entry to record the exception fixup for all exception spots between kernel and user space access.
To enable recovery from machine checks while coping data from user addresses it is necessary to be able to distinguish the places that are looping copying data from those that copy a single byte/word/etc.
Add a new macro _ASM_EXTABLE_CPY and use it in place of _ASM_EXTABLE_UA in the copy functions.
Record the exception reason number to regs->ax at ex_handler_uaccess which is used to check MCE triggered.
The new fixup routine ex_handler_copy() is almost an exact copy of ex_handler_uaccess() The difference is that it sets regs->ax to the trap number. Following patches use this to avoid trying to copy remaining bytes from the tail of the copy and possibly hitting the poison again.
New mce.kflags bit MCE_IN_KERNEL_COPYIN will be used by mce_severity() calculation to indicate that a machine check is recoverable because the kernel was copying from user space.
--- arch/x86/include/asm/asm.h | 6 +++ arch/x86/include/asm/mce.h | 15 ++++++ arch/x86/lib/copy_user_64.S | 96 ++++++++++++++++++------------------- arch/x86/mm/extable.c | 14 +++++- 4 files changed, 82 insertions(+), 49 deletions(-)
arch/x86/include/asm/asm.h +6 -0
@@ -135,6 +135,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
@@ -160,6 +163,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
arch/x86/include/asm/mce.h +15 -0
@@ -136,8 +136,23 @@#define MCE_HANDLED_NFIT BIT_ULL(3)#define MCE_HANDLED_EDAC BIT_ULL(4)#define MCE_HANDLED_MCELOG BIT_ULL(5) + +/* +* Indicates an MCE which has happened in kernel space but from +* which the kernel can recover simply by executing fixup_exception() +* so that an error is returned to the caller of the function that +* hit the machine check. +*/ #define MCE_IN_KERNEL_RECOV BIT_ULL(6) +/* +* Indicates an MCE that happened in kernel space while copying data +* from user. In this case fixup_exception() gets the kernel to the +* error exit for the copy function. Machine check handler can then +* treat it like a fault taken in user mode. +*/ +#define MCE_IN_KERNEL_COPYIN BIT_ULL(7) +/* * This structure contains all data related to the MCE log. Also * carries a signature to make it easier to find from external
arch/x86/lib/copy_user_64.S +48 -48
131 lines not on this page · raw message
arch/x86/mm/extable.c +13 -1
@@ -80,6 +80,18 @@ __visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_uaccess); +__visible bool ex_handler_copy(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr) +{ +WARN_ONCE (trapnr ==X86_TRAP_GP ,"General protection fault in user access. Non-canonical address?" ); + regs->ip =ex_fixup_addr (fixup); + regs->ax = trapnr; +return true; +} +EXPORT_SYMBOL (ex_handler_copy); + __visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup, struct pt_regs *regs, int trapnr, unsigned long error_code,
@@ -136,7 +148,7 @@ enum handler_type ex_get_fault_handler_type(unsigned long ip) handler =ex_fixup_handler (e);if (handler == ex_handler_fault)return EX_HANDLER_FAULT ; -else if (handler == ex_handler_uaccess) +else if (handler == ex_handler_uaccess || handler == ex_handler_copy)return EX_HANDLER_UACCESS ;else return EX_HANDLER_OTHER ;
Signature · 1 line
2.21.1
[tip: ras/core] x86/mce: Add _ASM_EXTABLE_CPY for copy user access
The following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: 278b917f8cb9b02923c15249f9d1a5769d2c1976 Gitweb: https://git.kernel.org/tip/278b917f8cb9b02923c15249f9d1a5769d2c1976 Author: Youquan Song <[email protected]> AuthorDate: Tue, 06 Oct 2020 14:09:07 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 07 Oct 2020 11:19:11 +02:00
x86/mce: Add _ASM_EXTABLE_CPY for copy user access
_ASM_EXTABLE_UA is a general exception entry to record the exception fixup for all exception spots between kernel and user space access.
To enable recovery from machine checks while coping data from user addresses it is necessary to be able to distinguish the places that are looping copying data from those that copy a single byte/word/etc.
Add a new macro _ASM_EXTABLE_CPY and use it in place of _ASM_EXTABLE_UA in the copy functions.
Record the exception reason number to regs->ax at ex_handler_uaccess which is used to check MCE triggered.
The new fixup routine ex_handler_copy() is almost an exact copy of ex_handler_uaccess() The difference is that it sets regs->ax to the trap number. Following patches use this to avoid trying to copy remaining bytes from the tail of the copy and possibly hitting the poison again.
New mce.kflags bit MCE_IN_KERNEL_COPYIN will be used by mce_severity() calculation to indicate that a machine check is recoverable because the kernel was copying from user space.
--- arch/x86/include/asm/asm.h | 6 ++- arch/x86/include/asm/mce.h | 15 ++++++- arch/x86/lib/copy_user_64.S | 96 ++++++++++++++++++------------------ arch/x86/mm/extable.c | 14 ++++- 4 files changed, 82 insertions(+), 49 deletions(-)
arch/x86/include/asm/asm.h +6 -0
@@ -135,6 +135,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
@@ -160,6 +163,9 @@# define _ASM_EXTABLE_UA (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_uaccess) +# define _ASM_EXTABLE_CPY (from, to) \ + _ASM_EXTABLE_HANDLE(from, to, ex_handler_copy) +# define _ASM_EXTABLE_FAULT (from, to) \ _ASM_EXTABLE_HANDLE(from, to, ex_handler_fault)
arch/x86/include/asm/mce.h +15 -0
@@ -136,9 +136,24 @@#define MCE_HANDLED_NFIT BIT_ULL(3)#define MCE_HANDLED_EDAC BIT_ULL(4)#define MCE_HANDLED_MCELOG BIT_ULL(5) + +/* +* Indicates an MCE which has happened in kernel space but from +* which the kernel can recover simply by executing fixup_exception() +* so that an error is returned to the caller of the function that +* hit the machine check. +*/ #define MCE_IN_KERNEL_RECOV BIT_ULL(6)/* +* Indicates an MCE that happened in kernel space while copying data +* from user. In this case fixup_exception() gets the kernel to the +* error exit for the copy function. Machine check handler can then +* treat it like a fault taken in user mode. +*/ +#define MCE_IN_KERNEL_COPYIN BIT_ULL(7) + +/* * This structure contains all data related to the MCE log. Also * carries a signature to make it easier to find from external * debugging tools. Each entry is only valid when its finished flag
arch/x86/lib/copy_user_64.S +48 -48
131 lines not on this page · raw message
arch/x86/mm/extable.c +13 -1
@@ -80,6 +80,18 @@ __visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_uaccess); +__visible bool ex_handler_copy(const struct exception_table_entry *fixup, +struct pt_regs *regs,int trapnr, +unsigned long error_code, +unsigned long fault_addr) +{ +WARN_ONCE (trapnr ==X86_TRAP_GP ,"General protection fault in user access. Non-canonical address?" ); + regs->ip =ex_fixup_addr (fixup); + regs->ax = trapnr; +return true; +} +EXPORT_SYMBOL (ex_handler_copy); + __visible bool ex_handler_rdmsr_unsafe(const struct exception_table_entry *fixup, struct pt_regs *regs, int trapnr, unsigned long error_code,
@@ -136,7 +148,7 @@ enum handler_type ex_get_fault_handler_type(unsigned long ip) handler =ex_fixup_handler (e);if (handler == ex_handler_fault)return EX_HANDLER_FAULT ; -else if (handler == ex_handler_uaccess) +else if (handler == ex_handler_uaccess || handler == ex_handler_copy)return EX_HANDLER_UACCESS ;else return EX_HANDLER_OTHER ;
Tony Luck
maintainerPATCH v3 4/6 x86/mce: Avoid tail copy when machine check terminated a copy from user
In reply to Tony LuckIn the page fault case it is ok to see if a few more unaligned bytes can be copied from the source address. Worst case is that the page fault will be triggered again.
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return from the copy code as if the copy succeeded. The machine check handler will use task_work_add() to make sure that the task is sent a SIGBUS.
--- arch/x86/lib/copy_user_64.S | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/x86/lib/copy_user_64.S +15 -0
@@ -15,6 +15,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> .macro ALIGN_DESTINATION /* check for bad alignment of destination */
@@ -221,6 +222,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Try to copy last bytes and clear the rest if needed. * Since protection fault in copy_from/to_user is not a normal situation, * it is not necessary to optimize tail handling. + * Don't try to copy the tail if machine check happened * * Input: * rdi destination
@@ -232,11 +234,24 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx + cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ + je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret + /* + * Return zero to pretend that this copy succeeded. This + * is counter-intuitive, but needed to prevent the code + * in lib/iov_iter.c from retrying and running back into + * the poison cache line again. The machine check handler + * will ensure that a SIGBUS is sent to the task. + */ +3: xorl %eax,%eax + ASM_CLAC + ret + _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail)
Signature · 1 line
2.21.1
RE: [PATCH v3 4/6] x86/mce: Avoid tail copy when machine check terminated a copy from user
From: Tony Luck
Quoted 10 lines
Sent: 06 October 2020 22:09
In the page fault case it is ok to see if a few more unaligned bytes can be copied from the source address. Worst case is that the page fault will be triggered again.
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return from the copy code as if the copy succeeded. The machine check handler will use
task_work_add()to make sure that the task is sent a SIGBUS.
Isn't that just plain wrong?
If copy is reported as succeeding the kernel code will use the 'old' data that is in the buffer as if it had been read from userspace. This could end up with kernel stack data being written to a file.
Even zeroing the rest of the kernel buffer is wrong.
IIRC the code to try to maximise the copy has been removed. So the 'slow' retry wont happen any more.
David
Quoted 52 lines
Signed-off-by Tony Luck <[email protected]>--- arch/x86/lib/copy_user_64.S | 15 +++++++++++++++ 1 file changed, 15 insertions(+)arch/x86/lib/copy_user_64.S +15
-1@@ -15,6 +15,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> .macro ALIGN_DESTINATION /* check for bad alignment of destination */@@ -221,6 +222,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Try to copy last bytes and clear the rest if needed. * Since protection fault in copy_from/to_user is not a normal situation, * it is not necessary to optimize tail handling. + * Don't try to copy the tail if machine check happened * * Input: * rdi destination@@ -232,11 +234,24 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx + cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ + je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret + /* + * Return zero to pretend that this copy succeeded. This + * is counter-intuitive, but needed to prevent the code + * in lib/iov_iter.c from retrying and running back into + * the poison cache line again. The machine check handler + * will ensure that a SIGBUS is sent to the task. + */ +3: xorl %eax,%eax + ASM_CLAC + ret + _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail) --
Quoted 1 line
2.21.1
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
Luck, Tony
maintainerRE: [PATCH v3 4/6] x86/mce: Avoid tail copy when machine check terminated a copy from user
Quoted 6 lines
Quoted 4 lines
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return from the copy code as if the copy succeeded. The machine check handler will use
task_work_add()to make sure that the task is sent a SIGBUS.Isn't that just plain wrong?
It isn't pretty. I'm not sure how wrong it is.
Quoted 3 lines
If copy is reported as succeeding the kernel code will use the 'old' data that is in the buffer as if it had been read from userspace. This could end up with kernel stack data being written to a file.
I ran a test with:
write(fd, buf, 512)
With poison injected into buf[256] to force a machine check mid-copy.
The size of the file did get incremented by 512 rather than 256. Which isn't good.
The data in the file up to the 256 byte mark was the user data from buf[0 ... 255].
The data in the file past offset 256 was all zeroes. I suspect that isn't by chance. The kernel has to defend against a user writing a partial page and using mmap(2) on the same file to peek at data past EOF and up to the next PAGE_SIZE boundary. So I think it must zero new pages allocated in page cache as they are allocated to a file.
Quoted 1 line
Even zeroing the rest of the kernel buffer is wrong.
It wouldn't help/change anything.
Quoted 2 lines
IIRC the code to try to maximise the copy has been removed. So the 'slow' retry wont happen any more.
Which code has been removed (and when ... TIP, and my testing, is based on 5.9-rc1)
-Tony
RE: [PATCH v3 4/6] x86/mce: Avoid tail copy when machine check terminated a copy from user
From: Luck, Tony
Quoted 22 lines
Sent: 07 October 2020 19:50
Quoted 6 lines
Quoted 4 lines
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return from the copy code as if the copy succeeded. The machine check handler will use
task_work_add()to make sure that the task is sent a SIGBUS.Isn't that just plain wrong?
It isn't pretty. I'm not sure how wrong it is.
Quoted 3 lines
If copy is reported as succeeding the kernel code will use the 'old' data that is in the buffer as if it had been read from userspace. This could end up with kernel stack data being written to a file.
I ran a test with:
write(fd, buf, 512)With poison injected into buf[256] to force a machine check mid-copy.
The size of the file did get incremented by 512 rather than 256. Which isn't good.
Quoted 7 lines
The data in the file up to the 256 byte mark was the user data from buf[0 ... 255].
The data in the file past offset 256 was all zeroes. I suspect that isn't by chance. The kernel has to defend against a user writing a partial page and using mmap(2) on the same file to peek at data past EOF and up to the next
PAGE_SIZEboundary. So I think it must zero new pages allocated in page cache as they are allocated to a file.
Think about what happens to a device write or an ioctl request. These typically get copied into on-stack buffers.
Quoted 4 lines
Quoted 2 lines
IIRC the code to try to maximise the copy has been removed. So the 'slow' retry wont happen any more.
Which code has been removed (and when ... TIP, and my testing, is based on 5.9-rc1)
The code that retries byte by byte after an initial fault. Might only be removed from 'next' and for some architectures. Basically almost nothing ever relies on partial copies (except mount).
IIRC there is 'magic' in the syscall exit path to convert EFAULT into SIGSEGV.
You probably want to hijack it to generate SIGBUS?
David
-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)
[tip: ras/core] x86/mce: Avoid tail copy when machine check terminated a copy from user
In reply to Tony LuckThe following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: a2f73400e4dfd13f673c6e1b4b98d180fd1e47b3 Gitweb: https://git.kernel.org/tip/a2f73400e4dfd13f673c6e1b4b98d180fd1e47b3 Author: Tony Luck <[email protected]> AuthorDate: Tue, 06 Oct 2020 14:09:08 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 07 Oct 2020 11:26:56 +02:00
x86/mce: Avoid tail copy when machine check terminated a copy from user
In the page fault case it is ok to see if a few more unaligned bytes can be copied from the source address. Worst case is that the page fault will be triggered again.
Machine checks are more serious. Just give up at the point where the main copy loop triggered the #MC and return from the copy code as if the copy succeeded. The machine check handler will use task_work_add() to make sure that the task is sent a SIGBUS.
--- arch/x86/lib/copy_user_64.S | 15 +++++++++++++++ 1 file changed, 15 insertions(+)
arch/x86/lib/copy_user_64.S +15 -0
@@ -15,6 +15,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> .macro ALIGN_DESTINATION /* check for bad alignment of destination */
@@ -221,6 +222,7 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) * Try to copy last bytes and clear the rest if needed. * Since protection fault in copy_from/to_user is not a normal situation, * it is not necessary to optimize tail handling. + * Don't try to copy the tail if machine check happened * * Input: * rdi destination
@@ -232,11 +234,24 @@ EXPORT_SYMBOL(copy_user_enhanced_fast_string) */ SYM_CODE_START_LOCAL(.Lcopy_user_handle_tail) movl %edx,%ecx + cmp $X86_TRAP_MC,%eax /* check if X86_TRAP_MC */ + je 3f 1: rep movsb 2: mov %ecx,%eax ASM_CLAC ret + /* + * Return zero to pretend that this copy succeeded. This + * is counter-intuitive, but needed to prevent the code + * in lib/iov_iter.c from retrying and running back into + * the poison cache line again. The machine check handler + * will ensure that a SIGBUS is sent to the task. + */ +3: xorl %eax,%eax + ASM_CLAC + ret + _ASM_EXTABLE_CPY(1b, 2b) SYM_CODE_END(.Lcopy_user_handle_tail)
Tony Luck
maintainerPATCH v3 5/6 x86/mce: Recover from poison found while copying from user space
In reply to Tony LuckExisting kernel code can only recover from a machine check on code that is tagged in the exception table with a fault handling recovery path.
Two new fields in the task structure to pass information from machine check handler to the "task_work" that is queued to run before the task returns to user mode:
+ mce_vaddr: will be initialized to the user virtual address of the fault in the case where the fault occurred in the kernel copying data from a user address. This is so that kill_me_maybe() can provide that information to the user SIGBUS handler.
+ mce_kflags: copy of the struct mce.kflags needed by kill_me_maybe() to determine if mce_vaddr is applicable to this error.
Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task.
Use a new helper function to share common code between the "fault in user mode" case and the "fault while copying from user" case.
New code paths will be activated by the next patch which sets MCE_IN_KERNEL_COPYIN.
--- arch/x86/kernel/cpu/mce/core.c | 27 ++++++++++++++++++++------- include/linux/sched.h | 2 ++ 2 files changed, 22 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/mce/core.c +20 -7
@@ -1260,6 +1260,21 @@ static void kill_me_maybe(struct callback_head *cb)kill_me_now (cb); } +static void queue_task_work (struct mce *m,int kill_it) +{ + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + +if (kill_it) + current->mce_kill_me.func = kill_me_now; +else + current->mce_kill_me.func = kill_me_maybe; + +task_work_add (current, ¤t->mce_kill_me, true); +} +/* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18.
@@ -1401,13 +1416,8 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); - current->mce_addr = m.addr; - current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (&m); - current->mce_kill_me.func = kill_me_maybe; -if (kill_it) - current->mce_kill_me.func = kill_me_now; -task_work_add (current, ¤t->mce_kill_me, true); +queue_task_work (&m, kill_it); + }else {/* * Handle an MCE which has happened in kernel space but from
@@ -1422,6 +1432,9 @@ noinstr void do_machine_check(struct pt_regs *regs)if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 ))mce_panic ("Failed kernel mode recovery" , &m, msg); } + +if (m.kflags &MCE_IN_KERNEL_COPYIN ) +queue_task_work (&m, kill_it); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +2 -0
@@ -1308,6 +1308,8 @@ struct task_struct {#endif #ifdef CONFIG_X86_MCE +void __user *mce_vaddr; +__u64 mce_kflags;u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1,
Signature · 1 line
2.21.1
The following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: c0ab7ffce275d3f83bd253c70889c28821d4a41d Gitweb: https://git.kernel.org/tip/c0ab7ffce275d3f83bd253c70889c28821d4a41d Author: Tony Luck <[email protected]> AuthorDate: Tue, 06 Oct 2020 14:09:09 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 07 Oct 2020 11:29:41 +02:00
x86/mce: Recover from poison found while copying from user space
Existing kernel code can only recover from a machine check on code that is tagged in the exception table with a fault handling recovery path.
Add two new fields in the task structure to pass information from machine check handler to the "task_work" that is queued to run before the task returns to user mode:
+ mce_vaddr: will be initialized to the user virtual address of the fault in the case where the fault occurred in the kernel copying data from a user address. This is so that kill_me_maybe() can provide that information to the user SIGBUS handler.
+ mce_kflags: copy of the struct mce.kflags needed by kill_me_maybe() to determine if mce_vaddr is applicable to this error.
Add code to recover from a machine check while copying data from user space to the kernel. Action for this case is the same as if the user touched the poison directly; unmap the page and send a SIGBUS to the task.
Use a new helper function to share common code between the "fault in user mode" case and the "fault while copying from user" case.
New code paths will be activated by the next patch which sets MCE_IN_KERNEL_COPYIN.
--- arch/x86/kernel/cpu/mce/core.c | 27 ++++++++++++++++++++------- include/linux/sched.h | 2 ++ 2 files changed, 22 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/mce/core.c +20 -7
@@ -1260,6 +1260,21 @@ static void kill_me_maybe(struct callback_head *cb)kill_me_now (cb); } +static void queue_task_work (struct mce *m,int kill_it) +{ + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + +if (kill_it) + current->mce_kill_me.func = kill_me_now; +else + current->mce_kill_me.func = kill_me_maybe; + +task_work_add (current, ¤t->mce_kill_me, true); +} +/* * The actual machine check handler. This only handles real * exceptions when something got corrupted coming in through int 18.
@@ -1401,13 +1416,8 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); - current->mce_addr = m.addr; - current->mce_ripv = !!(m.mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (&m); - current->mce_kill_me.func = kill_me_maybe; -if (kill_it) - current->mce_kill_me.func = kill_me_now; -task_work_add (current, ¤t->mce_kill_me, true); +queue_task_work (&m, kill_it); + }else {/* * Handle an MCE which has happened in kernel space but from
@@ -1422,6 +1432,9 @@ noinstr void do_machine_check(struct pt_regs *regs)if (!fixup_exception (regs,X86_TRAP_MC ,0 ,0 ))mce_panic ("Failed kernel mode recovery" , &m, msg); } + +if (m.kflags &MCE_IN_KERNEL_COPYIN ) +queue_task_work (&m, kill_it); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +2 -0
@@ -1308,6 +1308,8 @@ struct task_struct {#endif #ifdef CONFIG_X86_MCE +void __user *mce_vaddr; +__u64 mce_kflags;u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1,
Tony Luck
maintainerPATCH v3 6/6 x86/mce: Decode a kernel instruction to determine if it is copying from user
In reply to Tony LuckAll instructions copying data between kernel and user memory are tagged with either _ASM_EXTABLE_UA or _ASM_EXTABLE_CPY entries in the exception table. ex_fault_handler_type() returns EX_HANDLER_UACCESS for both of these.
Recovery is only possible when the machine check was triggered on a read from user memory. In this case the same strategy for recovery applies as if the user had made the access in ring3. If the fault was in kernel memory while copying to user there is no current recovery plan.
For MOV and MOVZ instructions a full decode of the instruction is done to find the source address. For MOVS instructions the source address is in the %rsi register. The function fault_in_kernel_space() determines whether the source address is kernel or user, upgrade it from "static" so it can be used here.
--- arch/x86/include/asm/traps.h | 2 ++ arch/x86/kernel/cpu/mce/core.c | 11 +++++-- arch/x86/kernel/cpu/mce/severity.c | 53 +++++++++++++++++++++++++++++- arch/x86/mm/fault.c | 2 +- 4 files changed, 63 insertions(+), 5 deletions(-)
arch/x86/include/asm/traps.h +2 -0
@@ -35,6 +35,8 @@ extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); +bool fault_in_kernel_space (unsigned long address); +#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/kernel/cpu/mce/core.c +8 -3
@@ -1250,14 +1250,19 @@ static void kill_me_maybe(struct callback_head *cb)if (!p->mce_ripv) flags |=MF_MUST_KILL ; -if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags)) { +if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags) && + !(p->mce_kflags &MCE_IN_KERNEL_COPYIN )) {set_mce_nospec (p->mce_addr >>PAGE_SHIFT , p->mce_whole_page);sync_core ();return ; } -pr_err ("Memory error not recovered" ); -kill_me_now (cb); +if (p->mce_vaddr != (void __user *)-1l ) { +force_sig_mceerr (BUS_MCEERR_AR , p->mce_vaddr,PAGE_SHIFT ); + }else { +pr_err ("Memory error not recovered" ); +kill_me_now (cb); + } }static void queue_task_work (struct mce *m,int kill_it)
arch/x86/kernel/cpu/mce/severity.c +52 -1
74 lines not on this page · raw message
arch/x86/mm/fault.c +1 -1
@@ -1081,7 +1081,7 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)return 0 ; } -static int fault_in_kernel_space (unsigned long address) +bool fault_in_kernel_space (unsigned long address) {/* * On 64-bit systems, the vsyscall page is at an address above
Signature · 1 line
2.21.1
[tip: ras/core] x86/mce: Decode a kernel instruction to determine if it is copying from user
The following commit has been merged into the ras/core branch of tip:
The following commit has been merged into the ras/core branch of tip:
Commit-ID: 300638101329e8f1569115f3d7197ef5ef754a3a Gitweb: https://git.kernel.org/tip/300638101329e8f1569115f3d7197ef5ef754a3a Author: Tony Luck <[email protected]> AuthorDate: Tue, 06 Oct 2020 14:09:10 -07:00 Committer: Borislav Petkov <[email protected]> CommitterDate: Wed, 07 Oct 2020 11:32:40 +02:00
x86/mce: Decode a kernel instruction to determine if it is copying from user
All instructions copying data between kernel and user memory are tagged with either _ASM_EXTABLE_UA or _ASM_EXTABLE_CPY entries in the exception table. ex_fault_handler_type() returns EX_HANDLER_UACCESS for both of these.
Recovery is only possible when the machine check was triggered on a read from user memory. In this case the same strategy for recovery applies as if the user had made the access in ring3. If the fault was in kernel memory while copying to user there is no current recovery plan.
For MOV and MOVZ instructions a full decode of the instruction is done to find the source address. For MOVS instructions the source address is in the %rsi register. The function fault_in_kernel_space() determines whether the source address is kernel or user, upgrade it from "static" so it can be used here.
--- arch/x86/include/asm/traps.h | 2 +- arch/x86/kernel/cpu/mce/core.c | 11 ++++-- arch/x86/kernel/cpu/mce/severity.c | 53 ++++++++++++++++++++++++++++- arch/x86/mm/fault.c | 2 +- 4 files changed, 63 insertions(+), 5 deletions(-)
arch/x86/include/asm/traps.h +2 -0
@@ -35,6 +35,8 @@ extern int panic_on_unrecovered_nmi;void math_emulate (struct math_emu_info *); +bool fault_in_kernel_space (unsigned long address); +#ifdef CONFIG_VMAP_STACK void __noreturn handle_stack_overflow(const char *message, struct pt_regs *regs,
arch/x86/kernel/cpu/mce/core.c +8 -3
@@ -1250,14 +1250,19 @@ static void kill_me_maybe(struct callback_head *cb)if (!p->mce_ripv) flags |=MF_MUST_KILL ; -if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags)) { +if (!memory_failure (p->mce_addr >>PAGE_SHIFT , flags) && + !(p->mce_kflags &MCE_IN_KERNEL_COPYIN )) {set_mce_nospec (p->mce_addr >>PAGE_SHIFT , p->mce_whole_page);sync_core ();return ; } -pr_err ("Memory error not recovered" ); -kill_me_now (cb); +if (p->mce_vaddr != (void __user *)-1l ) { +force_sig_mceerr (BUS_MCEERR_AR , p->mce_vaddr,PAGE_SHIFT ); + }else { +pr_err ("Memory error not recovered" ); +kill_me_now (cb); + } }static void queue_task_work (struct mce *m,int kill_it)
arch/x86/kernel/cpu/mce/severity.c +52 -1
74 lines not on this page · raw message
arch/x86/mm/fault.c +1 -1
@@ -1081,7 +1081,7 @@ access_error(unsigned long error_code, struct vm_area_struct *vma)return 0 ; } -static int fault_in_kernel_space (unsigned long address) +bool fault_in_kernel_space (unsigned long address) {/* * On 64-bit systems, the vsyscall page is at an address above
Tony Luck
maintainercover Add machine check recovery when copying from user space
In reply to a message not in the archiveMachine check recovery from uncorrected memory errors currently focusses primarily on errors that are detected while running in user mode. There is a mechanism for recovering from errors in kernel code, but it is currently only used for memcpy_mcsafe().
The existing recover actions for errors found in user mode (unmap the page and send SIGBUS to the task) can also be applied when the error is found while copying data from user space to the kernel.
Roadmap to this series:
0001: Could be applied as a standalone optimization. But is essential to
this series because mce_reign() does not have access to the saved registers from the CPU that hit the error. So it can't recompute severity correctly for this new class of recoverable error.
0002: First piece of infrastructure update. Severity calculations need
access to the saved registers. So pass pointer down the call chain.
0003: Later we need to know what type of exception handler is present
for a given kernel instruction. Rather than proliferate more functions like ex_has_fault_handler() for each type, replace with a function that looks up the handler and returns an enum describing the type.
0004: Need slightly different handling for *copy_user*() faults from
get_user() faults. Create a new exception table tag and apply to the copy functions.
0005: In fixup path of copy functions avoid dealing with the tail
when the copy took a machine check by returning that there are no bytes left to be copied.
0006: For the REP MOVS copy case we need to check if %rsi is a user
or kernel address. There's already a static function to do this. Make it global so x86/mce code can share the goodness.
0007: New function to get the user virtual address that was used for
a user access that hit a machine check. Add the code fragment into do_machine_check() to take recovery actions. Does nothing at this point because severity code to detect copy from user case and set the kflags MCE_IN_KERNEL_COPYIN bit is in last patch of series.
0008: Finally the keystone patch that pulls all the parts together.
An instruction decoder figures out whether an instruction tagged as accessing user space is reading from or writing to user space. The instructions in the switch were found experimentally by looking at what instructions in the base kernel are tagged in the exception table. I didn't add the atomic operations (0x87 = XCHG etc.) that both read and write user addresses. I think they should be safe, but I need a test case where a futex has been poisoned to check. Probably this switch should be expanded with all the instructions that the compiler could possibly generate that read from user space.
Tony Luck (4):
x86/mce: Stop mce_reign() from re-computing severity for every CPU x86/mce: Provide method to find out the type of exception handle x86/mce: Avoid tail copy when machine check terminated a copy from
user
x86/mce: Decode a kernel instruction to determine if it is copying
from user
Youquan Song (4):
x86/mce: Pass pointer to saved pt_regs to severity calculation
routines
x86/mce: Add _ASM_EXTABLE_CPY for copy user access x86/mce: Change fault_in_kernel_space() from static to global x86/mce: Recover from poison found while copying from user space
arch/x86/include/asm/asm.h | 6 ++ arch/x86/include/asm/extable.h | 9 ++- arch/x86/include/asm/mce.h | 1 + arch/x86/include/asm/traps.h | 2 + arch/x86/kernel/cpu/mce/core.c | 89 ++++++++++++++++++++----- arch/x86/kernel/cpu/mce/internal.h | 3 +- arch/x86/kernel/cpu/mce/severity.c | 49 ++++++++++++-- arch/x86/lib/copy_user_64.S | 103 +++++++++++++++-------------- arch/x86/mm/extable.c | 24 +++++-- arch/x86/mm/fault.c | 2 +- include/linux/sched.h | 1 + 11 files changed, 210 insertions(+), 79 deletions(-)
Signature · 1 line
2.21.1
A message that is not in the archive (sent off-list or not yet received): [email protected]
Borislav Petkov
maintainerRe: [PATCH 3/8] x86/mce: Provide method to find out the type of exception handle
Quoted 15 lines · Tony Luck
Avoid a proliferation of ex_has_*_handler() functions by having just one function that returns the type of the handler (if any).
Signed-off-by Tony Luck <[email protected]>--- arch/x86/include/asm/extable.h | 9 ++++++++- arch/x86/kernel/cpu/mce/severity.c | 5 ++++- arch/x86/mm/extable.c | 12 ++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)arch/x86/include/asm/extable.h +0
-0
arch/x86/include/asm/extable.h struct pt_regs; · line 36
Quoted 8 lines
arch/x86/include/asm/extable.h +5
-0@@ -29,10 +29,17 @@ struct pt_regs; (b)->handler = (tmp).handler - (delta); \ }while (0 ) +enum handler_type { +HANDLER_NONE , +HANDLER_FAULT , +HANDLER_UACCESS , +HANDLER_OTHER
EX_HANDLER_* I guess - HANDLER is too generic.
arch/x86/include/asm/extable.h __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 128
Quoted 6 lines
arch/x86/include/asm/extable.h +1
-1@@ -125,17 +125,21 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); -__visible bool ex_has_fault_handler(unsigned long ip) +__visible enum handler_type ex_fault_handler_type(unsigned long ip)
Why __visible?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH 3/8] x86/mce: Provide method to find out the type of exception handle
Quoted 19 lines · Borislav Petkov
Quoted 18 lines · Tony Luck
Avoid a proliferation of ex_has_*_handler() functions by having just one function that returns the type of the handler (if any).
Signed-off-by Tony Luck <[email protected]>--- arch/x86/include/asm/extable.h | 9 ++++++++- arch/x86/kernel/cpu/mce/severity.c | 5 ++++- arch/x86/mm/extable.c | 12 ++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-)arch/x86/include/asm/extable.h +0
-0@@ -29,10 +29,17 @@ struct pt_regs; (b)->handler = (tmp).handler - (delta); \ }while (0 )
Quoted 7 lines
arch/x86/include/asm/extable.h struct pt_regs; · line 36
Quoted 5 lines
arch/x86/include/asm/extable.h +5
-0+enum handler_type { +HANDLER_NONE , +HANDLER_FAULT , +HANDLER_UACCESS , +HANDLER_OTHER EX_HANDLER_* I guess - HANDLER is too generic.
Yup. Will change.
Quoted 8 lines
arch/x86/include/asm/extable.h __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, · line 128
Quoted 6 lines
arch/x86/include/asm/extable.h +1
-1@@ -125,17 +125,21 @@ __visible bool ex_handler_clear_fs(const struct exception_table_entry *fixup, }EXPORT_SYMBOL (ex_handler_clear_fs); -__visible bool ex_has_fault_handler(unsigned long ip) +__visible enum handler_type ex_fault_handler_type(unsigned long ip)Why
__visible?
I didn't look that hard at this. Just kept the same because the function I was replacing was also __visible.
But it looks like this commit:
80a3e3949b8f ("x86/extable: Mark exception handler functions visible")
shouldn't have touched this function. This one is only called from C code, not from assembler like the others.
Will drop the __visible (and note in commit comment)
Thanks
-Tony