Fix infinite machine check loop in futex_wait_setup()
Subsystems (7)
Tony Luck
maintainercover Fix infinite machine check loop in futex_wait_setup()
Linux can now recover from machine checks where kernel code is doing get_user() to access application memory. But there isn't a way to distinguish whether get_user() failed because of a page fault or a machine check.
Thus there is a problem if any kernel code thinks it can retry an access after doing something that would fix the page fault.
One such example (I'm sure there are more) is in futex_wait_setup() where an attempt to read the futex with page faults disabled. Then a retry (after dropping a lock so page faults are safe):
ret = get_futex_value_locked(&uval, uaddr);
if (ret) {
queue_unlock(*hb);
ret = get_user(uval, uaddr);It would be good to avoid deliberately taking a second machine check (especially as the recovery code does really bad things and ends up in an infinite loop!).
My proposal is to add a new function arch_memory_failure() that can be called after get_user() returns -EFAULT to allow graceful recovery.
Futex reviewers: I just have one new call (that fixes my test case). If you could point out other places this is needed, that would be most helpful.
Patch roadmap:
Part 1: Add code to avoid the infinite loop in the machine check code. Just panic if code runs into the same machine check a second time. This should make it much easier to debug other places where this happens.
Part 2: Add arch_memory_failure() and use it in futex_wait_setup(). [Suggestions gladly accepted for the current best way to handle the #defines etc. to define an arch specific function to be used in generic code]
Tony Luck (2):
x86/mce: Avoid infinite loop for copy from user recovery futex, x86/mce: Avoid double machine checks
arch/x86/include/asm/mmu.h | 7 +++++++ arch/x86/kernel/cpu/mce/core.c | 17 ++++++++++++++++- include/linux/mm.h | 4 ++++ include/linux/sched.h | 3 ++- kernel/futex.c | 3 +++ 5 files changed, 32 insertions(+), 2 deletions(-)
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH 1/2 x86/mce: Avoid infinite loop for copy from user recovery
Recovery action when get_user() triggers a machine check uses the fixup path to make get_user() return -EFAULT. Also queue_task_work() sets up so that kill_me_maybe() will be called on return to user mode to send a SIGBUS to the current process.
But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check queue_task_work() is called again. But since this uses the same callback_head structure that was used in the first call, the net result is an entry on the current->task_works list that points to itself. When task_work_run() is called it loops forever in this code:
do {
next = work->next;
work->func(work);
work = next;
cond_resched();
} while (work);Add a "mce_busy" flag bit to detect this situation and panic when it happens.
--- arch/x86/kernel/cpu/mce/core.c | 7 ++++++- include/linux/sched.h | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mce/core.c +6 -1
@@ -1246,6 +1246,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; + p->mce_busy =0 ;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
@@ -1268,6 +1269,7 @@ static void kill_me_maybe(struct callback_head *cb)static void queue_task_work (struct mce *m,int kill_current_task) { + current->mce_busy =1 ; current->mce_addr = m->addr; current->mce_kflags = m->kflags; current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV );
@@ -1431,8 +1433,11 @@ noinstr void do_machine_check(struct pt_regs *regs)mce_panic ("Failed kernel mode recovery" , &m, msg); } -if (m.kflags &MCE_IN_KERNEL_COPYIN ) +if (m.kflags &MCE_IN_KERNEL_COPYIN ) { +if (current->mce_busy) +mce_panic ("Multiple copyin" , &m, msg);queue_task_work (&m, kill_current_task); + } } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +2 -1
@@ -1360,7 +1360,8 @@ struct task_struct {u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1, - __mce_reserved : 62; + mce_busy : 1, + __mce_reserved : 61;struct callback_head mce_kill_me;#endif
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH 2/2 futex, x86/mce: Avoid double machine checks
In reply to Tony Luckfutex_wait_setup() first tries to read the user value with page faults disabled (because it holds a lock, and so cannot sleep). If that read fails it drops the lock and tries again.
But there are now two reasons why the user space read can fail. Either:
1) legacy case of a page fault, in which case it is reasonable to retry
2) machine check on the user address, bad idea to re-read
Add some infrastructure to differentiate these cases.
--- arch/x86/include/asm/mmu.h | 7 +++++++ arch/x86/kernel/cpu/mce/core.c | 10 ++++++++++ include/linux/mm.h | 4 ++++ kernel/futex.c | 3 +++ 4 files changed, 24 insertions(+)
arch/x86/include/asm/mmu.h +7 -0
@@ -66,4 +66,11 @@ typedef struct {void leave_mm (int cpu);#define leave_mm leave_mm +#if defined(CONFIG_X86_MCE ) && defined(CONFIG_MEMORY_FAILURE ) +#undef arch_memory_failure +#define arch_memory_failure x86_memory_failure +#endif + +bool x86_memory_failure (u32 __user *addr); +#endif /* _ASM_X86_MMU_H */
arch/x86/kernel/cpu/mce/core.c +10 -0
@@ -1236,6 +1236,16 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin *m = *final; } +bool x86_memory_failure (u32 __user *addr) +{ +if (current->mce_busy ==0 ) +return false; + +WARN_ON (current->mce_vaddr != addr); + +return true; +} +static void kill_me_now (struct callback_head *ch) {force_sig (SIGBUS );
include/linux/mm.h +4 -0
@@ -3177,5 +3177,9 @@ unsigned long wp_shared_mapping_range(struct address_space *mapping,extern int sysctl_nr_trim_pages; +#ifndef arch_memory_failure +#define arch_memory_failure (vaddr) (0) +#endif +#endif /* __KERNEL__ */ #endif /* _LINUX_MM_H */
kernel/futex.c +3 -0
@@ -2658,6 +2658,9 @@ static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags,if (ret) {queue_unlock (*hb); +if (arch_memory_failure (uaddr)) +return ret; + ret =get_user (uval, uaddr);if (ret)return ret;
Signature · 1 line
2.21.1
Peter Zijlstra
maintainerRe: [PATCH 2/2] futex, x86/mce: Avoid double machine checks
Quoted 13 lines · Tony Luck
futex_wait_setup()first tries to read the user value with page faults disabled (because it holds a lock, and so cannot sleep). If that read fails it drops the lock and tries again.But there are now two reasons why the user space read can fail. Either:
1) legacy case of a page fault, in which case it is reasonable to retry
2) machine check on the user address, bad idea to re-readAdd some infrastructure to differentiate these cases.
kernel/futex.c static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags, · line 2666
Quoted 9 lines
kernel/futex.c +3
-0@@ -2658,6 +2658,9 @@ static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags,if (ret) {queue_unlock (*hb); +if (arch_memory_failure (uaddr)) +return ret; + ret =get_user (uval, uaddr);if (ret)return ret;
I think this is horrid; why can't we have it return something different then -EFAULT instead?
Luck, Tony
maintainerRE: [PATCH 2/2] futex, x86/mce: Avoid double machine checks
Quoted 2 lines
I think this is horrid; why can't we have it return something different then
-EFAULTinstead?
I did consider this ... but it appears that architectures aren't unified in the return value from get_user()
Here's another function involved in the futex call chain leading to this:
static int get_futex_value_locked(u32 *dest, u32 __user *from)
{
int ret;
pagefault_disable();
ret = __get_user(*dest, from);
pagefault_enable();
return ret ? -EFAULT : 0;
}It seems like the expectation here is just "zero or not" and we don't care what the "not" value is ... just turn it into -EFAULT.
-Tony
Peter Zijlstra
maintainerRe: [PATCH 2/2] futex, x86/mce: Avoid double machine checks
Quoted 5 lines · Tony
Quoted 2 lines
I think this is horrid; why can't we have it return something different then
-EFAULTinstead?I did consider this ... but it appears that architectures aren't unified in the return value from
get_user()
But surely none are currently returning -EMEMERR or whatever name we come up with.
Quoted 13 lines
Here's another function involved in the futex call chain leading to this:
static int
get_futex_value_locked(u32 *dest, u32 __user *from){ int ret; pagefault_disable(); ret = __get_user(*dest, from); pagefault_enable(); return ret ? -EFAULT : 0; }
Quoted 2 lines
It seems like the expectation here is just "zero or not" and we don't care what the "not" value is ... just turn it into
-EFAULT.
Yeah, saw that, but that should be trivially fixable I'm thinking.
Luck, Tony
maintainerRE: [PATCH 2/2] futex, x86/mce: Avoid double machine checks
Quoted 1 line
Yeah, saw that, but that should be trivially fixable I'm thinking.
Trivial, maybe ... but then follows the audit of other get_user() calls:
git grep get_user\( | wc -l
2003
:-(
-Tony
Tony Luck
maintainerv2 cover Fix infinite machine check loop in futex_wait_setup()
In reply to Tony LuckLinux can now recover from machine checks where kernel code is doing get_user() to access application memory. But there isn't a way to distinguish whether get_user() failed because of a page fault or a machine check.
Thus there is a problem if any kernel code thinks it can retry an access after doing something that would fix the page fault.
One such example (I'm sure there are more) is in futex_wait_setup() where an attempt to read the futex with page faults disabled. Then a retry (after dropping a lock so page faults are safe):
ret = get_futex_value_locked(&uval, uaddr);
if (ret) {
queue_unlock(*hb);
ret = get_user(uval, uaddr);It would be good to avoid deliberately taking a second machine check (especially as the recovery code does really bad things and ends up in an infinite loop!).
V2 (thanks to feedback from PeterZ) fixes this by changing get_user() to return -ENXIO ("No such device or address") for the case where a machine check occurred. Peter left it open which error code to use (suggesting "-EMEMERR or whatever name we come up with"). I think the existing ENXIO error code seems appropriate (the address being accessed has effectively gone away). But I don't have a strong attachment if anyone thinks we need a new code.
Callers can check for ENXIO in paths where the access would be retried so they can avoid a second machine check.
Patch roadmap:
Part 1 (unchanged since v1):
Add code to avoid the infinite loop in the machine check code. Just panic if code runs into the same machine check a second time. This should make it much easier to debug other places where this happens.
Part 2: Change recovery path for get_user() to return -ENXIO
Part 3: Fix the one case in futex code that my test case hits (I'm sure there are more).
TBD: There are a few places in arch/x86 code that test "ret == -EFAULT" or have "switch (ret) { case -EFAULT: }" that may benefit from an additional check for -ENXIO. For now those will continue to crash (just like every pre-v5.10 kernel crashed when get_user() touched poison).
Tony Luck (3):
x86/mce: Avoid infinite loop for copy from user recovery x86/mce: Add new return value to get_user() for machine check futex, x86/mce: Avoid double machine checks
arch/x86/kernel/cpu/mce/core.c | 7 ++++++- arch/x86/lib/getuser.S | 8 +++++++- arch/x86/mm/extable.c | 1 + include/linux/sched.h | 3 ++- kernel/futex.c | 5 ++++- 5 files changed, 20 insertions(+), 4 deletions(-)
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH v2 1/3 x86/mce: Avoid infinite loop for copy from user recovery
Recovery action when get_user() triggers a machine check uses the fixup path to make get_user() return -EFAULT. Also queue_task_work() sets up so that kill_me_maybe() will be called on return to user mode to send a SIGBUS to the current process.
But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check queue_task_work() is called again. But since this uses the same callback_head structure that was used in the first call, the net result is an entry on the current->task_works list that points to itself. When task_work_run() is called it loops forever in this code:
do {
next = work->next;
work->func(work);
work = next;
cond_resched();
} while (work);Add a "mce_busy" flag bit to detect this situation and panic when it happens.
--- arch/x86/kernel/cpu/mce/core.c | 7 ++++++- include/linux/sched.h | 3 ++- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/x86/kernel/cpu/mce/core.c +6 -1
@@ -1246,6 +1246,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; + p->mce_busy =0 ;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
@@ -1268,6 +1269,7 @@ static void kill_me_maybe(struct callback_head *cb)static void queue_task_work (struct mce *m,int kill_current_task) { + current->mce_busy =1 ; current->mce_addr = m->addr; current->mce_kflags = m->kflags; current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV );
@@ -1431,8 +1433,11 @@ noinstr void do_machine_check(struct pt_regs *regs)mce_panic ("Failed kernel mode recovery" , &m, msg); } -if (m.kflags &MCE_IN_KERNEL_COPYIN ) +if (m.kflags &MCE_IN_KERNEL_COPYIN ) { +if (current->mce_busy) +mce_panic ("Multiple copyin" , &m, msg);queue_task_work (&m, kill_current_task); + } } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +2 -1
@@ -1360,7 +1360,8 @@ struct task_struct {u64 mce_addr; __u64 mce_ripv : 1, mce_whole_page : 1, - __mce_reserved : 62; + mce_busy : 1, + __mce_reserved : 61;struct callback_head mce_kill_me;#endif
Signature · 1 line
2.21.1
Re: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 13 lines · Tony Luck
Recovery action when
get_user()triggers a machine check uses the fixup path to makeget_user()return-EFAULT. Alsoqueue_task_work()sets up so thatkill_me_maybe()will be called on return to user mode to send a SIGBUS to the current process.But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
Quoted 4 lines
While processing this second machine check
queue_task_work()is called again. But since this uses the samecallback_headstructure that was used in the first call, the net result is an entry on thecurrent->task_workslist that points to itself.
Is this happening in pagefault_disable context or normal sleepable fault context? If the latter, maybe we should reconsider finding a way for the machine check code to do its work inline instead of deferring it.
Yes, I realize this is messy, but maybe it’s not that messy. Conceptually, we just (famous last words) need to arrange for an MCE with IF=1 to switch off the IST stack and run like a normal exception.
Luck, Tony
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 13 lines · Andy Lutomirski
Quoted 13 lines · Tony Luck
Recovery action when
get_user()triggers a machine check uses the fixup path to makeget_user()return-EFAULT. Alsoqueue_task_work()sets up so thatkill_me_maybe()will be called on return to user mode to send a SIGBUS to the current process.But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
Quoted 6 lines
Quoted 4 lines
While processing this second machine check
queue_task_work()is called again. But since this uses the samecallback_headstructure that was used in the first call, the net result is an entry on thecurrent->task_workslist that points to itself.Is this happening in
pagefault_disablecontext or normal sleepable fault context? If the latter, maybe we should reconsider finding a way for the machine check code to do its work inline instead of deferring it.
The first machine check is in pagefault_disable() context.
static int get_futex_value_locked(u32 *dest, u32 __user *from)
{
int ret;
pagefault_disable();
ret = __get_user(*dest, from);
pagefault_enable();
return (ret == -ENXIO) ? ret : ret ? -EFAULT : 0;
}-Tony
Andy Lutomirski
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 23 lines · Tony
Quoted 13 lines · Andy Lutomirski
Quoted 13 lines
Quoted 2 lines · Tony Luck
Recovery action when
get_user()triggers a machine check uses the fixup path to makeget_user()return-EFAULT. Alsoqueue_task_work()sets up so thatkill_me_maybe()will be called on return to user mode to send a SIGBUS to the current process.But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
Quoted 6 lines
Quoted 4 lines
While processing this second machine check
queue_task_work()is called again. But since this uses the samecallback_headstructure that was used in the first call, the net result is an entry on thecurrent->task_workslist that points to itself.Is this happening in
pagefault_disablecontext or normal sleepable fault context? If the latter, maybe we should reconsider finding a way for the machine check code to do its work inline instead of deferring it.
Quoted 8 lines
The first machine check is in
pagefault_disable()context.static int
get_futex_value_locked(u32 *dest, u32 __user *from){ int ret; pagefault_disable(); ret = __get_user(*dest, from);
I have very mixed feelings as to whether we should even try to recover from the first failure like this. If we actually want to try to recover, perhaps we should instead arrange for the second MCE to recover successfully instead of panicking.
--Andy
Quoted 6 lines
pagefault_enable(); return (ret == -ENXIO) ? ret : ret ? -EFAULT : 0; }-Tony
Luck, Tony
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 29 lines · Andy Lutomirski
Quoted 29 lines · Tony
Quoted 13 lines · Andy Lutomirski
Quoted 13 lines
Quoted 2 lines · Tony Luck
Recovery action when
get_user()triggers a machine check uses the fixup path to makeget_user()return-EFAULT. Alsoqueue_task_work()sets up so thatkill_me_maybe()will be called on return to user mode to send a SIGBUS to the current process.But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
Quoted 6 lines
Quoted 4 lines
While processing this second machine check
queue_task_work()is called again. But since this uses the samecallback_headstructure that was used in the first call, the net result is an entry on thecurrent->task_workslist that points to itself.Is this happening in
pagefault_disablecontext or normal sleepable fault context? If the latter, maybe we should reconsider finding a way for the machine check code to do its work inline instead of deferring it.The first machine check is in
pagefault_disable()context.static int
get_futex_value_locked(u32 *dest, u32 __user *from){ int ret;
Quoted 7 lines
Quoted 2 lines
pagefault_disable(); ret = __get_user(*dest, from);I have very mixed feelings as to whether we should even try to recover from the first failure like this. If we actually want to try to recover, perhaps we should instead arrange for the second MCE to recover successfully instead of panicking.
Well we obviously have to "recover" from the first machine check in order to get to the second. Are you saying you don't like the different return value from get_user()?
In my initial playing around with this I just had the second machine check simply skip the task_work_add(). This worked for this case, but only because there wasn't a third, fourth, etc. access to the poisoned data. If the caller keeps peeking, then we'll keep taking more machine checks - possibly forever.
Even if we do recover with just one extra machine check ... that's one more than was necessary.
-Tony
Andy Lutomirski
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 43 lines · Tony
Quoted 29 lines · Andy Lutomirski
Quoted 29 lines · Tony
Quoted 13 lines · Andy Lutomirski
Quoted 13 lines
Quoted 2 lines · Tony Luck
Recovery action when
get_user()triggers a machine check uses the fixup path to makeget_user()return-EFAULT. Alsoqueue_task_work()sets up so thatkill_me_maybe()will be called on return to user mode to send a SIGBUS to the current process.But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
Quoted 6 lines
Quoted 4 lines
While processing this second machine check
queue_task_work()is called again. But since this uses the samecallback_headstructure that was used in the first call, the net result is an entry on thecurrent->task_workslist that points to itself.Is this happening in
pagefault_disablecontext or normal sleepable fault context? If the latter, maybe we should reconsider finding a way for the machine check code to do its work inline instead of deferring it.The first machine check is in
pagefault_disable()context.static int
get_futex_value_locked(u32 *dest, u32 __user *from){ int ret;Quoted 7 lines
Quoted 2 lines
pagefault_disable(); ret = __get_user(*dest, from);I have very mixed feelings as to whether we should even try to recover from the first failure like this. If we actually want to try to recover, perhaps we should instead arrange for the second MCE to recover successfully instead of panicking.
Well we obviously have to "recover" from the first machine check in order to get to the second. Are you saying you don't like the different return value from
get_user()?
Quoted 8 lines
In my initial playing around with this I just had the second machine check simply skip the
task_work_add(). This worked for this case, but only because there wasn't a third, fourth, etc. access to the poisoned data. If the caller keeps peeking, then we'll keep taking more machine checks - possibly forever.Even if we do recover with just one extra machine check ... that's one more than was necessary.
Well, we need to do *something* when the first __get_user() trips the #MC. It would be nice if we could actually fix up the page tables inside the #MC handler, but, if we're in a pagefault_disable() context we might have locks held. Heck, we could have the pagetable lock held, be inside NMI, etc. Skipping the task_work_add() might actually make sense if we get a second one.
We won't actually infinite loop in pagefault_disable() context -- if we would, then we would also infinite loop just from a regular page fault, too.
Luck, Tony
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 10 lines · Andy Lutomirski
Well, we need to do *something* when the first
__get_user()trips the #MC. It would be nice if we could actually fix up the page tables inside the #MC handler, but, if we're in apagefault_disable()context we might have locks held. Heck, we could have the pagetable lock held, be inside NMI, etc. Skipping thetask_work_add()might actually make sense if we get a second one.We won't actually infinite loop in
pagefault_disable()context -- if we would, then we would also infinite loop just from a regular page fault, too.
Fixing the page tables inside the #MC handler to unmap the poison page would indeed be a good solution. But, as you point out, not possible because of locks.
Could we take a more drastic approach? We know that this case the kernel is accessing a user address for the current process. Could the machine check handler just re-write %cr3 to point to a kernel-only page table[1]. I.e. unmap the entire current user process.
Then any subsequent access to user space will page fault. Maybe have a flag in the task structure to help the #PF handler understand what just happened.
The code we execute in the task_work handler can restore %cr3
-Tony
[1] Does such a thing already exist? If not, I'd need some help/pointers to create it.
Andy Lutomirski
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 13 lines · Tony
Quoted 10 lines · Andy Lutomirski
Well, we need to do *something* when the first
__get_user()trips the #MC. It would be nice if we could actually fix up the page tables inside the #MC handler, but, if we're in apagefault_disable()context we might have locks held. Heck, we could have the pagetable lock held, be inside NMI, etc. Skipping thetask_work_add()might actually make sense if we get a second one.We won't actually infinite loop in
pagefault_disable()context -- if we would, then we would also infinite loop just from a regular page fault, too.
Quoted 8 lines
Fixing the page tables inside the #MC handler to unmap the poison page would indeed be a good solution. But, as you point out, not possible because of locks.
Could we take a more drastic approach? We know that this case the kernel is accessing a user address for the current process. Could the machine check handler just re-write %cr3 to point to a kernel-only page table[1]. I.e. unmap the entire current user process.
That seems scary, especially if we're in the middle of a context switch when this happens. We *could* make it work, but I'm not at all convinced it's wise.
Quoted 6 lines
Then any subsequent access to user space will page fault. Maybe have a flag in the task structure to help the #PF handler understand what just happened.
The code we execute in the
task_workhandler can restore %cr3
This would need to be integrated with something much more local IMO. Maybe it could be scoped to pagefault_disable()/pagefault_enable()?
--Andy
Luck, Tony
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 18 lines · Andy Lutomirski
Quoted 17 lines · Tony
Quoted 10 lines · Andy Lutomirski
Well, we need to do *something* when the first
__get_user()trips the #MC. It would be nice if we could actually fix up the page tables inside the #MC handler, but, if we're in apagefault_disable()context we might have locks held. Heck, we could have the pagetable lock held, be inside NMI, etc. Skipping thetask_work_add()might actually make sense if we get a second one.We won't actually infinite loop in
pagefault_disable()context -- if we would, then we would also infinite loop just from a regular page fault, too.Fixing the page tables inside the #MC handler to unmap the poison page would indeed be a good solution. But, as you point out, not possible because of locks.
Quoted 8 lines
Quoted 4 lines
Could we take a more drastic approach? We know that this case the kernel is accessing a user address for the current process. Could the machine check handler just re-write %cr3 to point to a kernel-only page table[1]. I.e. unmap the entire current user process.
That seems scary, especially if we're in the middle of a context switch when this happens. We *could* make it work, but I'm not at all convinced it's wise.
Scary? It's terrifying!
But we know that the fault happend in a get_user() or copy_from_user() call (i.e. an RIP with an extable recovery address). Does context switch access user memory?
-Tony
Re: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 31 lines · Tony
Quoted 18 lines · Andy Lutomirski
Quoted 18 lines · Tony
Quoted 10 lines · Andy Lutomirski
Well, we need to do *something* when the first
__get_user()trips the #MC. It would be nice if we could actually fix up the page tables inside the #MC handler, but, if we're in apagefault_disable()context we might have locks held. Heck, we could have the pagetable lock held, be inside NMI, etc. Skipping thetask_work_add()might actually make sense if we get a second one.We won't actually infinite loop in
pagefault_disable()context -- if we would, then we would also infinite loop just from a regular page fault, too.Fixing the page tables inside the #MC handler to unmap the poison page would indeed be a good solution. But, as you point out, not possible because of locks.
Quoted 8 lines
Quoted 4 lines
Could we take a more drastic approach? We know that this case the kernel is accessing a user address for the current process. Could the machine check handler just re-write %cr3 to point to a kernel-only page table[1]. I.e. unmap the entire current user process.
That seems scary, especially if we're in the middle of a context switch when this happens. We *could* make it work, but I'm not at all convinced it's wise.
Quoted 5 lines
Scary? It's terrifying!
But we know that the fault happend in a
get_user()orcopy_from_user()call (i.e. an RIP with an extable recovery address). Does context switch access user memory?
No, but NMI can.
The case that would be very very hard to deal with is if we get an NMI just before IRET/SYSRET and get #MC inside that NMI.
What we should probably do is have a percpu list of pending memory failure cleanups and just accept that we’re going to sometimes get a second MCE (or third or fourth) before we can get to it.
Can we do the cleanup from an interrupt? IPI-to-self might be a credible approach, if so.
Luck, Tony
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 11 lines · Andy Lutomirski
Quoted 3 lines
But we know that the fault happend in a
get_user()orcopy_from_user()call (i.e. an RIP with an extable recovery address). Does context switch access user memory?No, but NMI can.
The case that would be very very hard to deal with is if we get an NMI just before IRET/SYSRET and get #MC inside that NMI.
What we should probably do is have a percpu list of pending memory failure cleanups and just accept that we’re going to sometimes get a second MCE (or third or fourth) before we can get to it.
Can we do the cleanup from an interrupt? IPI-to-self might be a credible approach, if so.
You seem to be looking for a solution that is entirely contained within the machine check handling code. Willing to allow for repeated machine checks from the same poison address in order to achieve that.
I'm opposed to mutliple machine checks. Willing to make some changes in core code to avoid repeated access to the same poison location.
We need a tie-breaker.
-Tony
Re: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 15 lines · Tony
Quoted 10 lines · Andy Lutomirski
Quoted 3 lines
But we know that the fault happend in a
get_user()orcopy_from_user()call (i.e. an RIP with an extable recovery address). Does context switch access user memory?No, but NMI can.
The case that would be very very hard to deal with is if we get an NMI just before IRET/SYSRET and get #MC inside that NMI.
What we should probably do is have a percpu list of pending memory failure cleanups and just accept that we’re going to sometimes get a second MCE (or third or fourth) before we can get to it.
Quoted 8 lines
Quoted 1 line
Can we do the cleanup from an interrupt? IPI-to-self might be a credible approach, if so.
You seem to be looking for a solution that is entirely contained within the machine check handling code. Willing to allow for repeated machine checks from the same poison address in order to achieve that.
I'm opposed to mutliple machine checks. Willing to make some changes in core code to avoid repeated access to the same poison location.
How about more questions before the showdown?
If we made core code changes to avoid this, what would they be? We really can do user access from NMI and #DB, and those can happen in horrible places. We could have the pagetable lock held, be in the middle of CoWing the very page we tripped over, etc. I think we really can’t count on being able to write to the PTEs from #MC. Similarly, we might have IRQs off, so we can’t broadcast a TLB flush. And we might be in the middle of entry, exit, or CR3 switches, and I don’t see a particularly clean way to write CR3 without risking accidentally returning to user mode with the wrong CR3.
So I’m sort of at a loss as to what we can do. All user memory accessors can handle failure, and they will all avoid infinite looping. If we can tolerate repeated MCE, we can survive. But there’s not a whole lot we can do from these horrible contexts.
Hmm. Maybe if we have SMAP we could play with EFLAGS.AC? I can imagine this having various regrettable side effects, though.
Borislav Petkov
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 1 line · Andy Lutomirski
So I’m sort of at a loss as to what we can do.
I think you guys have veered off into the weeds with this. The current solution - modulo error messages not destined for humans :) - is not soo bad, considering the whole MCA trainwreck.
Or am I missing something completely unacceptable?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRE: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 5 lines
I think you guys have veered off into the weeds with this. The current solution - modulo error messages not destined for humans :) - is not soo bad, considering the whole MCA trainwreck.
Or am I missing something completely unacceptable?
Maybe the other difference in approach between Andy and me is whether to go for a solution that covers all the corner cases, or just make an incremental improvement that allows for recover in some useful subset of remaining fatal cases, but still dies in other cases.
I'm happy to replace error messages with ones that are more descriptive and helpful to humans.
-Tony
Borislav Petkov
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 4 lines · Tony
Maybe the other difference in approach between Andy and me is whether to go for a solution that covers all the corner cases, or just make an incremental improvement that allows for recover in some useful subset of remaining fatal cases, but still dies in other cases.
Does that mean more core code surgery?
Quoted 2 lines
I'm happy to replace error messages with ones that are more descriptive and helpful to humans.
Yap, that: "Multiple copyin" with something more understandable to users...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRE: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 6 lines
Quoted 4 lines
Maybe the other difference in approach between Andy and me is whether to go for a solution that covers all the corner cases, or just make an incremental improvement that allows for recover in some useful subset of remaining fatal cases, but still dies in other cases.
Does that mean more core code surgery?
Yes. I need to look at other user access inside pagefault_disable/enable()
as likely spots where the code may continue after a machine check and
retry the access. So expect some more "if (ret == ENXIO) { do something to give up gracefully }"
Quoted 4 lines
Quoted 2 lines
I'm happy to replace error messages with ones that are more descriptive and helpful to humans.
Yap, that: "Multiple copyin" with something more understandable to users...
I'll work on it. We tend not to have essay length messages as panic() strings. But I can add a comment in the code there so that people who grep whatever panic message we choose can get more details on what happened and what to do.
-Tony
Borislav Petkov
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 3 lines · Tony
I'll work on it. We tend not to have essay length messages as
panic()strings. But I can add a comment in the code there so that people who grep whatever panic message we choose can get more details on what happened and what to do.
I did not mean to write an essay but to simply make it more palatable for "normal" users. Something like:
"PANIC: Second machine check exception while accessing user data."
or along those lines to explain *why* the machine panics.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteBorislav Petkov
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
In reply to Tony Luckarch/x86/kernel/cpu/mce/core.c noinstr void do_machine_check(struct pt_regs *regs) · line 1438
Quoted 8 lines · Tony Luck
arch/x86/kernel/cpu/mce/core.c +3
-1@@ -1431,8 +1433,11 @@ noinstr void do_machine_check(struct pt_regs *regs)mce_panic ("Failed kernel mode recovery" , &m, msg); } -if (m.kflags &MCE_IN_KERNEL_COPYIN ) +if (m.kflags &MCE_IN_KERNEL_COPYIN ) { +if (current->mce_busy) +mce_panic ("Multiple copyin" , &m, msg);
So this: we're currently busy handling the first MCE, why do we must panic?
Can we simply ignore all follow-up MCEs to that page?
I.e., the page will get poisoned eventually and that poisoning is currently executing so all following MCEs are simply nothing new and we can ignore them.
It's not like we're going to corrupt more data - we already are "corrupting" whole 4K.
Am I making sense?
Because if we do this, we won't have to pay attention to any get_user() callers and whatnot - we simply ignore and the solution is simple and you won't have to touch any get_user() callers...
Hmmm?
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH v2 1/3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 13 lines · Borislav Petkov
arch/x86/kernel/cpu/mce/core.c noinstr void do_machine_check(struct pt_regs *regs) · line 1438
Quoted 8 lines · Tony Luck
arch/x86/kernel/cpu/mce/core.c +3
-1@@ -1431,8 +1433,11 @@ noinstr void do_machine_check(struct pt_regs *regs)mce_panic ("Failed kernel mode recovery" , &m, msg); } -if (m.kflags &MCE_IN_KERNEL_COPYIN ) +if (m.kflags &MCE_IN_KERNEL_COPYIN ) { +if (current->mce_busy) +mce_panic ("Multiple copyin" , &m, msg);So this: we're currently busy handling the first MCE, why do we must panic?
Quoted 1 line
Can we simply ignore all follow-up MCEs to that page?
If we s/all/some/ you are saying the same as Andy:
Quoted 7 lines
So I tend to think that the machine check code should arrange to survive some reasonable number of duplicate machine checks.
I.e., the page will get poisoned eventually and that poisoning is currently executing so all following MCEs are simply nothing new and we can ignore them.
Quoted 8 lines
It's not like we're going to corrupt more data - we already are "corrupting" whole 4K.
Am I making sense?
Because if we do this, we won't have to pay attention to any
get_user()callers and whatnot - we simply ignore and the solution is simple and you won't have to touch anyget_user()callers...
Changing get_user() is a can of worms. I don't think its a very big can. Perhaps two or three dozen places where code needs to change to account for the -ENXIO return ... but touching a bunch of different subsystems it is likley to take a while to get everyone in agreement.
I'll try out this new approach, and if it works, I'll post a v3 patch.
Thanks
-Tony
Tony Luck
maintainerPATCH v2 2/3 x86/mce: Add new return value to get_user() for machine check
In reply to Tony LuckWhen an exception occurs during any of the get_user() functions fixup_exception() passes the trap number of the exception in regs->ax to the fixup code.
Check for X86_TRAP_MC and return -ENXIO ("No such device or address") so that callers can take action to avoid repeating an access to an address that has an uncorrectable error.
--- arch/x86/lib/getuser.S | 8 +++++++- arch/x86/mm/extable.c | 1 + 2 files changed, 8 insertions(+), 1 deletion(-)
arch/x86/lib/getuser.S +7 -1
@@ -17,7 +17,7 @@ * * Inputs: %[r|e]ax contains the address. * - * Outputs: %[r|e]ax is error code (0 or -EFAULT) + * Outputs: %[r|e]ax is error code (0 or -EFAULT or -ENXIO) * %[r|e]dx contains zero-extended value * %ecx contains the high half for 32-bit __get_user_8 *
@@ -34,6 +34,7 @@ #include <asm/asm.h> #include <asm/smap.h> #include <asm/export.h> +#include <asm/trapnr.h> #define ASM_BARRIER_NOSPEC ALTERNATIVE "", "lfence", X86_FEATURE_LFENCE_RDTSC
@@ -168,8 +169,13 @@ SYM_CODE_START_LOCAL(.Lbad_get_user_clac) ASM_CLAC bad_get_user: xor %edx,%edx + cmpl $X86_TRAP_MC,%eax + je mce_get_user mov $(-EFAULT),%_ASM_AX ret +mce_get_user: + mov $(-ENXIO),%_ASM_AX + ret SYM_CODE_END(.Lbad_get_user_clac) #ifdef CONFIG_X86_32
arch/x86/mm/extable.c +1 -0
@@ -77,6 +77,7 @@ __visible bool ex_handler_uaccess(const struct exception_table_entry *fixup, {WARN_ONCE (trapnr ==X86_TRAP_GP ,"General protection fault in user access. Non-canonical address?" ); regs->ip =ex_fixup_addr (fixup); + regs->ax = trapnr;return true; }EXPORT_SYMBOL (ex_handler_uaccess);
Signature · 1 line
2.21.1
Tony Luck
maintainerPATCH v2 3/3 futex, x86/mce: Avoid double machine checks
In reply to Tony Luckfutex_wait_setup() first tries to read the user value with page faults disabled (because it holds a lock, and so cannot sleep). If that read fails it drops the lock and tries again.
But there are now two reasons why the user space read can fail. Either:
1) legacy case of a page fault, in which case it is reasonable to retry
2) machine check on the user address, bad idea to re-read
Check for the ENXIO return code from the first get_user() call and immediately return an error without re-reading the futex.
--- kernel/futex.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/futex.c +4 -1
@@ -723,7 +723,7 @@ static int get_futex_value_locked(u32 *dest, u32 __user *from) ret =__get_user (*dest, from);pagefault_enable (); -return ret ? -EFAULT :0 ; +return (ret == -ENXIO ) ? ret : ret ? -EFAULT :0 ; }
@@ -2658,6 +2658,9 @@ static int futex_wait_setup(u32 __user *uaddr, u32 val, unsigned int flags,if (ret) {queue_unlock (*hb); +if (ret == -ENXIO ) +return ret; + ret =get_user (uval, uaddr);if (ret)return ret;
Signature · 1 line
2.21.1
Andy Lutomirski
maintainerQuoted 33 lines · Tony Luck
Linux can now recover from machine checks where kernel code is doing
get_user()to access application memory. But there isn't a way to distinguish whetherget_user()failed because of a page fault or a machine check.Thus there is a problem if any kernel code thinks it can retry an access after doing something that would fix the page fault.
One such example (I'm sure there are more) is in
futex_wait_setup()where an attempt to read the futex with page faults disabled. Then a retry (after dropping a lock so page faults are safe):ret = get_futex_value_locked(&uval, uaddr); if (ret) { queue_unlock(*hb); ret = get_user(uval, uaddr);It would be good to avoid deliberately taking a second machine check (especially as the recovery code does really bad things and ends up in an infinite loop!).
V2 (thanks to feedback from PeterZ) fixes this by changing
get_user()to return-ENXIO("No such device or address") for the case where a machine check occurred. Peter left it open which error code to use (suggesting "-EMEMERRor whatever name we come up with"). I think the existing ENXIO error code seems appropriate (the address being accessed has effectively gone away). But I don't have a strong attachment if anyone thinks we need a new code.
Quoted 3 lines
Callers can check for ENXIO in paths where the access would be retried so they can avoid a second machine check.
I don't love this -- I'm concerned that there will be some code path that expects a failing get_user() to return -EFAULT, not -ENXIO. Also, get_user() *can* return -EFAULT when it hits bad memory even with your patch if the recovery code manages to yank the PTE before get_user().
So I tend to think that the machine check code should arrange to survive some reasonable number of duplicate machine checks.
Tony Luck
maintainerPATCH v3 1/1 x86/mce: Avoid infinite loop for copy from user recovery
In reply to Tony LuckRecovery action when get_user() triggers a machine check uses the fixup path to make get_user() return -EFAULT. Also queue_task_work() sets up so that kill_me_maybe() will be called on return to user mode to send a SIGBUS to the current process.
But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check queue_task_work() is called again. But since this uses the same callback_head structure that was used in the first call, the net result is an entry on the current->task_works list that points to itself. When task_work_run() is called it loops forever in this code:
do {
next = work->next;
work->func(work);
work = next;
cond_resched();
} while (work);Add a "mce_busy" counter so that task_work_add() is only called once per faulty page in this task.
Do not allow too many repeated machine checks, or machine checks to a different page from the first.
---
V3: Thanks to extensive commentary from Andy & Boris
Throws out the changes to get_user() and subsequent changes to core code. Everything is now handled in the machine check code. Downside is that we can (and do) take multiple machine checks from a single poisoned page before generic kernel code finally gets the message that a page is really and truly gone (but all the failed get_user() calls still return the legacy -EFAULT code, so none of that code will ever mistakenly use a value from a bad page). But even on an old machine that does broadcast interrupts for each machine check things survive multiple cycles of my test injection into a futex operation.
I picked "10" as the magic upper limit for how many times the machine check code will allow a fault from the same page before deciding to panic. We can bike shed that value if you like.
arch/x86/kernel/cpu/mce/core.c | 27 ++++++++++++++++++++------- include/linux/sched.h | 1 + 2 files changed, 21 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/mce/core.c +20 -7
@@ -1246,6 +1246,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; + p->mce_count =0 ;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
@@ -1266,12 +1267,24 @@ static void kill_me_maybe(struct callback_head *cb) } } -static void queue_task_work (struct mce *m,int kill_current_task) +static void queue_task_work (struct mce *m,char *msg,int kill_current_task) { - current->mce_addr = m->addr; - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (m); +if (current->mce_count++ ==0 ) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + } + +if (current->mce_count >10 ) +mce_panic ("Too many machine checks while accessing user data" , m, msg); + +if (current->mce_count >1 || (current->mce_addr >>PAGE_SHIFT ) != (m->addr >>PAGE_SHIFT )) +mce_panic ("Machine checks to different user pages" , m, msg); + +/* Do not call task_work_add() more than once */ +if (current->mce_count >1 ) +return ;if (kill_current_task) current->mce_kill_me.func = kill_me_now;
@@ -1414,7 +1427,7 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); }else {/*
@@ -1432,7 +1445,7 @@ noinstr void do_machine_check(struct pt_regs *regs) }if (m.kflags &MCE_IN_KERNEL_COPYIN ) -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +1 -0
@@ -1362,6 +1362,7 @@ struct task_struct { mce_whole_page :1 , __mce_reserved : 62;struct callback_head mce_kill_me; +int mce_count;#endif #ifdef CONFIG_KRETPROBES
Signature · 1 line
2.21.1
Borislav Petkov
maintainerRe: [PATCH v3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 23 lines · Tony Luck
Recovery action when
get_user()triggers a machine check uses the fixup path to makeget_user()return-EFAULT. Alsoqueue_task_work()sets up so thatkill_me_maybe()will be called on return to user mode to send a SIGBUS to the current process.But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check
queue_task_work()is called again. But since this uses the samecallback_headstructure that was used in the first call, the net result is an entry on thecurrent->task_workslist that points to itself. Whentask_work_run()is called it loops forever in this code:do { next = work->next; work->func(work); work = next; cond_resched(); } while (work);
Quoted 2 lines
Add a "
mce_busy" counter so thattask_work_add()is only called once per faulty page in this task.
Yeah, that sentence can be removed now too.
Quoted 8 lines
Do not allow too many repeated machine checks, or machine checks to a different page from the first.
Signed-off-by Tony Luck <[email protected]>---
V3: Thanks to extensive commentary from Andy & Boris
Quoted 9 lines
Throws out the changes to
get_user()and subsequent changes to core code. Everything is now handled in the machine check code. Downside is that we can (and do) take multiple machine checks from a single poisoned page before generic kernel code finally gets the message that a page is really and truly gone (but all the failedget_user()calls still return the legacy-EFAULTcode, so none of that code will ever mistakenly use a value from a bad page). But even on an old machine that does broadcast interrupts for each machine check things survive multiple cycles of my test injection into a futex operation.
Nice.
Quoted 22 lines
I picked "10" as the magic upper limit for how many times the machine check code will allow a fault from the same page before deciding to panic. We can bike shed that value if you like.
arch/x86/kernel/cpu/mce/core.c | 27 ++++++++++++++++++++------- include/linux/sched.h | 1 + 2 files changed, 21 insertions(+), 7 deletions(-)arch/x86/kernel/cpu/mce/core.c +1
-0@@ -1246,6 +1246,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; + p->mce_count =0 ;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1270
Quoted 5 lines
arch/x86/kernel/cpu/mce/core.c +1
-1@@ -1266,12 +1267,24 @@ static void kill_me_maybe(struct callback_head *cb) } } -static void queue_task_work (struct mce *m,int kill_current_task) +static void queue_task_work (struct mce *m,char *msg,int kill_current_task)
So this function gets called in the user mode MCE case too:
if ((m.cs & 3) == 3) {
queue_task_work(&m, msg, kill_current_task);
}Do we want to panic for multiple MCEs to different addresses in user mode?
I don't think so - that should go down the memory failure page offlining path...
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1278
Quoted 11 lines
arch/x86/kernel/cpu/mce/core.c +7
-4- current->mce_addr = m->addr; - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (m); +if (current->mce_count++ ==0 ) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + } +
/* Magic number should be large enough */
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1283
Quoted 5 lines
arch/x86/kernel/cpu/mce/core.c +5
-0+if (current->mce_count >10 ) +mce_panic ("Too many machine checks while accessing user data" , m, msg); + +if (current->mce_count >1 || (current->mce_addr >>PAGE_SHIFT ) != (m->addr >>PAGE_SHIFT )) +mce_panic ("Machine checks to different user pages" , m, msg);
Will this second part of the test expression, after the "||" ever hit?
You do above in the first branch:
if (current->mce_count++ == 0) {
...
current->mce_addr = m->addr;and ->mce_count becomes 1.
In that case that
(current->mce_addr >> PAGE_SHIFT) != (m->addr >> PAGE_SHIFT)
gets tested but that won't ever be true because ->mce_addr = ->addr above.
And then, for other values of mce_count, mce_count > 1 will hit.
In any case, what are you trying to catch with this? Two get_user() to different pages both catching MCEs?
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1287
Quoted 4 lines
arch/x86/kernel/cpu/mce/core.c +4
-0+ +/* Do not call task_work_add() more than once */ +if (current->mce_count >1 ) +return ;
That won't happen either, AFAICT. It'll panic above.
Regardless, I like how this is all confined to the MCE code and there's no need to touch stuff outside...
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH v3] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 5 lines · Borislav Petkov
Quoted 2 lines · Tony Luck
Add a "
mce_busy" counter so thattask_work_add()is only called once per faulty page in this task.Yeah, that sentence can be removed now too.
I will update with new name "mce_count" and some details.
Quoted 5 lines
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1270
Quoted 2 lines
arch/x86/kernel/cpu/mce/core.c +1
-1-static void queue_task_work (struct mce *m,int kill_current_task) +static void queue_task_work (struct mce *m,char *msg,int kill_current_task)So this function gets called in the user mode MCE case too:
Quoted 7 lines
if ((m.cs & 3) == 3) { queue_task_work(&m, msg, kill_current_task); }Do we want to panic for multiple MCEs to different addresses in user mode?
In the user mode case we should only bump mce_count to "1" and before task_work() gets called. It shouldn't hurt to do the same checks. Maybe it will catch something weird - like an NMI handler on return from the machine check doing a get_user() that hits another machine check during the return from this machine check.
AndyL has made me extra paranoid. :-)
Quoted 14 lines
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1278
Quoted 11 lines
arch/x86/kernel/cpu/mce/core.c +7
-4- current->mce_addr = m->addr; - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (m); +if (current->mce_count++ ==0 ) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + } +/* Magic number should be large enough */
Quoted 1 line
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1279
Quoted 1 line
arch/x86/kernel/cpu/mce/core.c +1
-0+if (current->mce_count >10 )
Will add similar comment here ... and to other tests in this function since it may not be obvious to me next year what I was thinking now :-)
Quoted 7 lines
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1283
Quoted 5 lines
arch/x86/kernel/cpu/mce/core.c +5
-0+if (current->mce_count >10 ) +mce_panic ("Too many machine checks while accessing user data" , m, msg); + +if (current->mce_count >1 || (current->mce_addr >>PAGE_SHIFT ) != (m->addr >>PAGE_SHIFT )) +mce_panic ("Machine checks to different user pages" , m, msg);Will this second part of the test expression, after the "||" ever hit?
No :-( This code is wrong. Should be "&&" not "||". Then it makes more sense. Will fix for v4.
Quoted 2 lines
In any case, what are you trying to catch with this? Two
get_user()to different pages both catching MCEs?
Yes. Trying to catch two accesses to different pages. Need to do this because kill_me_maybe() is only going to offline one page.
I'm not expecting that this would ever hit. It means that calling code took a machine check on one page and get_user() said -EFAULT. The the code decided to access a different page *and* that other page also triggered a machine check.
Quoted 5 lines
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1287
Quoted 3 lines
arch/x86/kernel/cpu/mce/core.c +3
-0+/* Do not call task_work_add() more than once */ +if (current->mce_count >1 ) +return ;That won't happen either, AFAICT. It'll panic above.
With the s/||/&&/ above, we can get here.
Quoted 3 lines
Regardless, I like how this is all confined to the MCE code and there's no need to touch stuff outside...
Thanks for the review.
-Tony
Luck, Tony
maintainerPATCH v4 1/1 x86/mce: Avoid infinite loop for copy from user recovery
Recovery action when get_user() triggers a machine check uses the fixup path to make get_user() return -EFAULT. Also queue_task_work() sets up so that kill_me_maybe() will be called on return to user mode to send a SIGBUS to the current process.
But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check queue_task_work() is called again. But since this uses the same callback_head structure that was used in the first call, the net result is an entry on the current->task_works list that points to itself. When task_work_run() is called it loops forever in this code:
do {
next = work->next;
work->func(work);
work = next;
cond_resched();
} while (work);Add a counter (current->mce_count) to keep track of repeated machine checks before task_work() is called. First machine check saves the address information and calls task_work_add(). Subsequent machine checks before that task_work call back is executed check that the address is in the same page as the first machine check (since the callback will offline exactly one page).
Expected worst case is two machine checks before moving on (e.g. one user access with page faults disabled, then a repeat to the same addrsss with page faults enabled). Just in case there is some code that loops forever enforce a limit of 10.
---
V4:
Fix bug with "||" where I meant "&&"
Update stale commit comment referring to mce_busy field in task structure (now called mce_count).
Add some comments to queue_task_work()
arch/x86/kernel/cpu/mce/core.c | 31 ++++++++++++++++++++++++------- include/linux/sched.h | 1 + 2 files changed, 25 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/mce/core.c +24 -7
@@ -1238,6 +1238,7 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *finstatic void kill_me_now (struct callback_head *ch) { + p->mce_count =0 ;force_sig (SIGBUS ); }
@@ -1246,6 +1247,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; + p->mce_count =0 ;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
@@ -1266,12 +1268,27 @@ static void kill_me_maybe(struct callback_head *cb) } } -static void queue_task_work (struct mce *m,int kill_current_task) +static void queue_task_work (struct mce *m,char *msg,int kill_current_task) { - current->mce_addr = m->addr; - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (m); +/* First call, save all the details */ +if (current->mce_count++ ==0 ) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + } + +/* Ten is likley overkill. Don't expect more than two faults before task_work() */ +if (current->mce_count >10 ) +mce_panic ("Too many machine checks while accessing user data" , m, msg); + +/* Second or later call, make sure page address matches the one from first call */ +if (current->mce_count >1 && (current->mce_addr >>PAGE_SHIFT ) != (m->addr >>PAGE_SHIFT )) +mce_panic ("Machine checks to different user pages" , m, msg); + +/* Do not call task_work_add() more than once */ +if (current->mce_count >1 ) +return ;if (kill_current_task) current->mce_kill_me.func = kill_me_now;
@@ -1414,7 +1431,7 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); }else {/*
@@ -1432,7 +1449,7 @@ noinstr void do_machine_check(struct pt_regs *regs) }if (m.kflags &MCE_IN_KERNEL_COPYIN ) -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +1 -0
@@ -1362,6 +1362,7 @@ struct task_struct { mce_whole_page :1 , __mce_reserved : 62;struct callback_head mce_kill_me; +int mce_count;#endif #ifdef CONFIG_KRETPROBES
Signature · 1 line
2.21.1
Luck, Tony
maintainerRe: [PATCH v4] x86/mce: Avoid infinite loop for copy from user recovery
arch/x86/kernel/cpu/mce/core.c static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin · line 1243
Quoted 5 lines · Tony
static void
kill_me_now(struct callback_head *ch){arch/x86/kernel/cpu/mce/core.c +1
-0+ p->mce_count =0 ;force_sig (SIGBUS ); }
Brown paper bag time ... I just pasted that line from kill_me_maybe() and I thought I did a re-compile ... but obviously not since it gives
error: ‘p’ undeclared (first use in this function)
Option a) (just like kill_me_maybe)
struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);
Option b) (simpler ... not sure why PeterZ did the container_of thing
current->mce_count = 0;
-Tony
Borislav Petkov
maintainerRe: [PATCH v4] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 12 lines · Tony
arch/x86/kernel/cpu/mce/core.c static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin · line 1243
Quoted 5 lines · Tony
static void
kill_me_now(struct callback_head *ch){arch/x86/kernel/cpu/mce/core.c +1
-0+ p->mce_count =0 ;force_sig (SIGBUS ); }Brown paper bag time ... I just pasted that line from
kill_me_maybe()and I thought I did a re-compile ... but obviously not since it giveserror: ‘p’ undeclared (first use in this function)
Quoted 7 lines
Option a) (just like
kill_me_maybe)struct
task_struct*p =container_of(cb, struct task_struct, mce_kill_me);Option b) (simpler ... not sure why PeterZ did the
container_ofthingcurrent->mce_count = 0;
Right, he says it is the canonical way to get it out of callback_head. I don't think current will change while the #MC handler runs but we can adhere to the design pattern here and do container_of() ...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH v4] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 17 lines · Borislav Petkov
Quoted 16 lines · Tony
arch/x86/kernel/cpu/mce/core.c static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin · line 1243
Quoted 5 lines · Tony
static void
kill_me_now(struct callback_head *ch){arch/x86/kernel/cpu/mce/core.c +1
-0+ p->mce_count =0 ;force_sig (SIGBUS ); }Brown paper bag time ... I just pasted that line from
kill_me_maybe()and I thought I did a re-compile ... but obviously not since it giveserror: ‘p’ undeclared (first use in this function)
Option a) (just like
kill_me_maybe)struct
task_struct*p =container_of(cb, struct task_struct, mce_kill_me);
Quoted 7 lines
Quoted 3 lines
Option b) (simpler ... not sure why PeterZ did the
container_ofthingcurrent->mce_count = 0;Right, he says it is the canonical way to get it out of
callback_head. I don't think current will change while the #MC handler runs but we can adhere to the design pattern here and docontainer_of()...
Ok ... I'll use the canonical way.
But now I've run into a weird issue. I'd run some basic tests with a dozen machine checks in each of:
1) user access
2) kernel copyin
3) futex (multiple accesses from kernel before task_work())
and it passed my tests before I posted.
But the real validation folks took my patch and found that it has destabilized cases 1 & 2 (and case 3 also chokes if you repeat a few more times). System either hangs or panics. Generally before 100 injection/conumption cycles.
Their tests are still just doing one at a time (i.e. complete recovery of one machine cehck before injecting the next error). So there aren't any complicated race conditions.
So if you see anything obviously broken, let me know. Otherwise I'll be poking around at the patch to figure out what is wrong.
-Tony
Borislav Petkov
maintainerRe: [PATCH v4] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 4 lines · Tony
But the real validation folks took my patch and found that it has destabilized cases 1 & 2 (and case 3 also chokes if you repeat a few more times). System either hangs or panics. Generally before 100 injection/conumption cycles.
Hmm, that mce_count increment and check stuff might turn out to be fragile in the face of a race condition. But I don't see it...
Quoted 2 lines
So if you see anything obviously broken, let me know. Otherwise I'll be poking around at the patch to figure out what is wrong.
Yah, some printk sprinkling might be a good start. But debugging in that atomic context is always nasty. ;-\
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRE: [PATCH v4] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 2 lines
Yah, some printk sprinkling might be a good start. But debugging in that atomic context is always nasty. ;-\
Some very light printk sprinkling (one message in queue_task_work() in atomic context, one each in kill_me_now() and kill_me_maybe() to check when task_work actually called them.
Cases 1 & 2 (user & normal copyin) now work just fine (hundreds of iterations).
Case 3 (my futex test) just hangs with only two characters making it to the serial port "[ "
Deeply strange.
-Tony
Luck, Tony
maintainerRe: [PATCH v4] x86/mce: Avoid infinite loop for copy from user recovery
In reply to Borislav PetkovQuoted 8 lines · Borislav Petkov
Quoted 4 lines · Tony
But the real validation folks took my patch and found that it has destabilized cases 1 & 2 (and case 3 also chokes if you repeat a few more times). System either hangs or panics. Generally before 100 injection/conumption cycles.
Hmm, that
mce_countincrement and check stuff might turn out to be fragile in the face of a race condition. But I don't see it...
I can't see what the race is either. As far as I can see the flow looks like this (for a machine check in user mode)
user-mode
do_machine_check() [in atomic context on IST etc.]
queue_task_work()
increments current->mce_count
first (only) call saves address etc. and calls task_work_add()
return from machine check
Plausibly we might context switch to another process here. We could even resume on a different CPU.
do_task_work() [in process context]
kill_me_maybe()
sets mce_count back to zero, ready for another #MC memory_failure()
send SIGBUS
The kernel get_user() case is similar, but may take extra machine checks before before calling code decides get_user() really is going to keep saying -EFAULT.
But, on a whim, I changed the type of mce_count from "int" to "atomic_t" and fixeed up the increment & clear to use atomic_inc_return() and atomic_set(). See updated patch below.
I can't see why this should have made any difference. But the "int" version crashes in a few dozen machine checks. The "atomic_t" version has run many thousands of machine checks (10213 in the current boot according to /proc/interrupts) with no issues.
-Tony
From d1b003f1de92f87c8dc53dd710fd79ad6e277364 Mon Sep 17 00:00:00 2001
From: Tony Luck <[email protected]>
Date: Wed, 20 Jan 2021 12:40:50 -0800
Subject: [PATCH] x86/mce: Avoid infinite loop for copy from user recovery
Recovery action when get_user() triggers a machine check uses the fixup path to make get_user() return -EFAULT. Also queue_task_work() sets up so that kill_me_maybe() will be called on return to user mode to send a SIGBUS to the current process.
But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check queue_task_work() is called again. But since this uses the same callback_head structure that was used in the first call, the net result is an entry on the current->task_works list that points to itself. When task_work_run() is called it loops forever in this code:
do {
next = work->next;
work->func(work);
work = next;
cond_resched();
} while (work);Add a counter (current->mce_count) to keep track of repeated machine checks before task_work() is called. First machine check saves the address information and calls task_work_add(). Subsequent machine checks before that task_work call back is executed check that the address is in the same page as the first machine check (since the callback will offline exactly one page).
Expected worst case is two machine checks before moving on (e.g. one user access with page faults disabled, then a repeat to the same addrsss with page faults enabled). Just in case there is some code that loops forever enforce a limit of 10.
--- arch/x86/kernel/cpu/mce/core.c | 35 +++++++++++++++++++++++++++------- include/linux/sched.h | 1 + 2 files changed, 29 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/mce/core.c +28 -7
@@ -1238,6 +1238,9 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *finstatic void kill_me_now (struct callback_head *ch) { + struct task_struct *p = container_of(ch, struct task_struct, mce_kill_me); + +atomic_set (&p->mce_count,0 );force_sig (SIGBUS ); }
@@ -1246,6 +1249,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; +atomic_set (&p->mce_count,0 );pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
@@ -1266,12 +1270,29 @@ static void kill_me_maybe(struct callback_head *cb) } } -static void queue_task_work (struct mce *m,int kill_current_task) +static void queue_task_work (struct mce *m,char *msg,int kill_current_task) { - current->mce_addr = m->addr; - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (m); +int count =atomic_inc_return (¤t->mce_count); + +/* First call, save all the details */ +if (count ==1 ) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + } + +/* Ten is likley overkill. Don't expect more than two faults before task_work() */ +if (count >10 ) +mce_panic ("Too many machine checks while accessing user data" , m, msg); + +/* Second or later call, make sure page address matches the one from first call */ +if (count >1 && (current->mce_addr >>PAGE_SHIFT ) != (m->addr >>PAGE_SHIFT )) +mce_panic ("Machine checks to different user pages" , m, msg); + +/* Do not call task_work_add() more than once */ +if (count >1 ) +return ;if (kill_current_task) current->mce_kill_me.func = kill_me_now;
@@ -1414,7 +1435,7 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); }else {/*
@@ -1432,7 +1453,7 @@ noinstr void do_machine_check(struct pt_regs *regs) }if (m.kflags &MCE_IN_KERNEL_COPYIN ) -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +1 -0
@@ -1362,6 +1362,7 @@ struct task_struct { mce_whole_page :1 , __mce_reserved : 62;struct callback_head mce_kill_me; +atomic_t mce_count;#endif #ifdef CONFIG_KRETPROBES
Signature · 1 line
2.21.1
Luck, Tony
maintainerPATCH v5 1/1 x86/mce: Avoid infinite loop for copy from user recovery
Quoted 9 lines · Tony
On Wed, Jan 20, 2021 at 01:18:12PM +0100, Borislav Petkov wrote:
But, on a whim, I changed the type ofmce_countfrom "int" to "atomic_t" and fixeed up the increment & clear to useatomic_inc_return()andatomic_set(). See updated patch below.I can't see why this should have made any difference. But the "int" version crashes in a few dozen machine checks. The "
atomic_t" version has run many thousands of machine checks (10213 in the current boot according to/proc/interrupts) with no issues.
And now I've changed it back to non-atomic (but keeping the slightly cleaner looking code style that I used for the atomic version). This one also works for thousands of injections and recoveries. Maybe take it now before it stops working again :-)
-Tony
From: Tony Luck <[email protected]>
Recovery action when get_user() triggers a machine check uses the fixup path to make get_user() return -EFAULT. Also queue_task_work() sets up so that kill_me_maybe() will be called on return to user mode to send a SIGBUS to the current process.
But there are places in the kernel where the code assumes that this EFAULT return was simply because of a page fault. The code takes some action to fix that, and then retries the access. This results in a second machine check.
While processing this second machine check queue_task_work() is called again. But since this uses the same callback_head structure that was used in the first call, the net result is an entry on the current->task_works list that points to itself. When task_work_run() is called it loops forever in this code:
do {
next = work->next;
work->func(work);
work = next;
cond_resched();
} while (work);Add a counter (current->mce_count) to keep track of repeated machine checks before task_work() is called. First machine check saves the address information and calls task_work_add(). Subsequent machine checks before that task_work call back is executed check that the address is in the same page as the first machine check (since the callback will offline exactly one page).
Expected worst case is two machine checks before moving on (e.g. one user access with page faults disabled, then a repeat to the same addrsss with page faults enabled). Just in case there is some code that loops forever enforce a limit of 10.
--- arch/x86/kernel/cpu/mce/core.c | 35 +++++++++++++++++++++++++++------- include/linux/sched.h | 1 + 2 files changed, 29 insertions(+), 7 deletions(-)
arch/x86/kernel/cpu/mce/core.c +28 -7
@@ -1238,6 +1238,9 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *finstatic void kill_me_now (struct callback_head *ch) { + struct task_struct *p = container_of(ch, struct task_struct, mce_kill_me); + + p->mce_count =0 ;force_sig (SIGBUS ); }
@@ -1246,6 +1249,7 @@ static void kill_me_maybe(struct callback_head *cb) struct task_struct *p = container_of(cb, struct task_struct, mce_kill_me);int flags =MF_ACTION_REQUIRED ; + p->mce_count =0 ;pr_err ("Uncorrected hardware memory error in user-access at %llx" , p->mce_addr);if (!p->mce_ripv)
@@ -1266,12 +1270,29 @@ static void kill_me_maybe(struct callback_head *cb) } } -static void queue_task_work (struct mce *m,int kill_current_task) +static void queue_task_work (struct mce *m,char *msg,int kill_current_task) { - current->mce_addr = m->addr; - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); - current->mce_whole_page =whole_page (m); +int count = ++current->mce_count; + +/* First call, save all the details */ +if (count ==1 ) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus &MCG_STATUS_RIPV ); + current->mce_whole_page =whole_page (m); + } + +/* Ten is likley overkill. Don't expect more than two faults before task_work() */ +if (count >10 ) +mce_panic ("Too many machine checks while accessing user data" , m, msg); + +/* Second or later call, make sure page address matches the one from first call */ +if (count >1 && (current->mce_addr >>PAGE_SHIFT ) != (m->addr >>PAGE_SHIFT )) +mce_panic ("Machine checks to different user pages" , m, msg); + +/* Do not call task_work_add() more than once */ +if (count >1 ) +return ;if (kill_current_task) current->mce_kill_me.func = kill_me_now;
@@ -1414,7 +1435,7 @@ noinstr void do_machine_check(struct pt_regs *regs)/* If this triggers there is no way to recover. Die hard. */ BUG_ON (!on_thread_stack () || !user_mode (regs)); -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); }else {/*
@@ -1432,7 +1453,7 @@ noinstr void do_machine_check(struct pt_regs *regs) }if (m.kflags &MCE_IN_KERNEL_COPYIN ) -queue_task_work (&m, kill_current_task); +queue_task_work (&m, msg, kill_current_task); } out:mce_wrmsrl (MSR_IA32_MCG_STATUS ,0 );
include/linux/sched.h +1 -0
@@ -1362,6 +1362,7 @@ struct task_struct { mce_whole_page :1 , __mce_reserved : 62;struct callback_head mce_kill_me; +int mce_count;#endif #ifdef CONFIG_KRETPROBES
Signature · 1 line
2.29.2
Borislav Petkov
maintainerRe: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 4 lines · Tony
And now I've changed it back to non-atomic (but keeping the slightly cleaner looking code style that I used for the atomic version). This one also works for thousands of injections and recoveries. Maybe take it now before it stops working again :-)
Hmm, so the only differences I see between your v4 and this are:
-@@ -1238,6 +1238,7 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin +@@ -1238,6 +1238,9 @@ static void __mc_scan_banks(struct mce *m, struct pt_regs *regs, struct mce *fin
static void kill_me_now(struct callback_head *ch) {
++ struct task_struct *p = container_of(ch, struct task_struct, mce_kill_me);
++
+ p->mce_count = 0;
force_sig(SIGBUS);
}
Could the container_of() macro have changed something?
Because we don't know yet (right?) why would it fail? Would it read stale ->mce_count data? If so, then a barrier is missing somewhere.
Or what is the failure exactly?
Because if I take it now without us knowing what the issue is, it will start failing somewhere - Murphy's our friend - and then we'll have to deal with breaking people's boxes. Not fun.
The other difference is:
@@ -76,8 +71,10 @@ index 13d3f1cbda17..5460c146edb5 100644
- current->mce_kflags = m->kflags;
- current->mce_ripv = !!(m->mcgstatus & MCG_STATUS_RIPV);
- current->mce_whole_page = whole_page(m);
++ int count = ++current->mce_count;
++
+ /* First call, save all the details */
-+ if (current->mce_count++ == 0) {
++ if (count == 1) {
+ current->mce_addr = m->addr;
+ current->mce_kflags = m->kflags;
+ current->mce_ripv = !!(m->mcgstatus & MCG_STATUS_RIPV);
Hmm, a local variable and a pre-increment. Can that have an effect somehow?
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1285
Quoted 1 line
arch/x86/kernel/cpu/mce/core.c +1
-0+/* Ten is likley overkill. Don't expect more than two faults before task_work() */
Typo: likely.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 19 lines · Borislav Petkov
Quoted 4 lines · Tony
And now I've changed it back to non-atomic (but keeping the slightly cleaner looking code style that I used for the atomic version). This one also works for thousands of injections and recoveries. Maybe take it now before it stops working again :-)
Hmm, so the only differences I see between your v4 and this are:
-@@ -1238,6 +1238,7 @@ static void __mc_scan_banks(struct mce *m, struct
pt_regs*regs, struct mce *fin +@@ -1238,6 +1238,9 @@ static void __mc_scan_banks(struct mce *m, structpt_regs*regs, struct mce *finstatic void
kill_me_now(struct callback_head *ch){
++ structtask_struct*p =container_of(ch, struct task_struct, mce_kill_me);
++
+p->mce_count= 0;
force_sig(SIGBUS);
}
Quoted 1 line
Could the
container_of()macro have changed something?
That change was to fix my brown paper bag moment (does not compile without a variable named "p" in scope to be used on next line.)
Quoted 2 lines
Because we don't know yet (right?) why would it fail? Would it read stale ->mce_count data? If so, then a barrier is missing somewhere.
I don't see how a barrier would make a differece. In the common case all this code is executed on the same logical CPU. Return from the do_machine_check() tries to return to user mode and finds that there is some "task_work" to execute first.
In some cases Linux might context switch to something else. Perhaps this task even gets picked up by another CPU to run the task work queued functions. But I imagine that the context switch should act as a barrier ... shouldn't it?
Quoted 1 line
Or what is the failure exactly?
After a few cycles of the test injection to user mode, I saw an overflow in the machine check bank. As if it hadn't been cleared from the previous iteration ... but all the banks are cleared as soon as we find that the machine check is recoverable. A while before getting to the code I changed.
When the tests were failing, code was on top of v5.11-rc3. Latest experiments moved to -rc5. There's just a tracing fix from PeterZ between rc3 and rc5 to mce/core.c:
737495361d44 ("x86/mce: Remove explicit/superfluous tracing")
which doesn't appear to be a candidate for the problems I saw.
Quoted 3 lines
Because if I take it now without us knowing what the issue is, it will start failing somewhere - Murphy's our friend - and then we'll have to deal with breaking people's boxes. Not fun.
Fair point.
Quoted 15 lines
The other difference is:
(quoted diff) +3
-1@@ -76,8 +71,10 @@ index 13d3f1cbda17..5460c146edb5 100644 - current->mce_kflags = m->kflags; - current->mce_ripv = !!(m->mcgstatus & MCG_STATUS_RIPV); - current->mce_whole_page = whole_page(m); ++ int count = ++current->mce_count; ++ + /* First call, save all the details */ -+ if (current->mce_count++ == 0) { ++ if (count == 1) { + current->mce_addr = m->addr; + current->mce_kflags = m->kflags; + current->mce_ripv = !!(m->mcgstatus & MCG_STATUS_RIPV);
Quoted 1 line
Hmm, a local variable and a pre-increment. Can that have an effect somehow?
This is the bit that changed during my detour using atomic_t mce_count. I added the local variable to capture value from atomic_inc_return(), then used it later, instead of a bunch of atomic_read() calls.
I kept it this way because "if (count == 1)" is marginally easier to read than "if (current->mce_count++ == 0)"
Quoted 3 lines
arch/x86/kernel/cpu/mce/core.c static void kill_me_maybe(struct callback_head *cb) · line 1285
Quoted 1 line
arch/x86/kernel/cpu/mce/core.c +1
-0+/* Ten is likley overkill. Don't expect more than two faults before task_work() */ Typo: likely.
Oops. Fixed.
-Tony
Borislav Petkov
maintainerRe: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 4 lines · Tony
In some cases Linux might context switch to something else. Perhaps this task even gets picked up by another CPU to run the task work queued functions. But I imagine that the context switch should act as a barrier ... shouldn't it?
I'm given to understand that the #MC from user is likely to schedule and a context switch has a barrier character.
Quoted 3 lines
After a few cycles of the test injection to user mode, I saw an overflow in the machine check bank. As if it hadn't been cleared from the previous iteration ...
This sounds weird. As if something else is happening which we haven't thought of yet...
Quoted 7 lines
When the tests were failing, code was on top of v5.11-rc3. Latest experiments moved to -rc5. There's just a tracing fix from PeterZ between rc3 and rc5 to
mce/core.c:
737495361d44("x86/mce: Remove explicit/superfluous tracing")which doesn't appear to be a candidate for the problems I saw.
Doesn't look like it.
Quoted 6 lines
This is the bit that changed during my detour using
atomic_tmce_count. I added the local variable to capture value fromatomic_inc_return(), then used it later, instead of a bunch ofatomic_read()calls.I kept it this way because "if (count == 1)" is marginally easier to read than "if (current->mce_count++ == 0)"
Right.
So still no explanation why it would fail before. ;-\
Crazy idea: if you still can reproduce on -rc3, you could bisect: i.e., if you apply the patch on -rc3 and it explodes and if you apply the same patch on -rc5 and it works, then that could be a start... Yeah, don't have a better idea here. :-\
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRe: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 4 lines · Borislav Petkov
Crazy idea: if you still can reproduce on -rc3, you could bisect: i.e., if you apply the patch on -rc3 and it explodes and if you apply the same patch on -rc5 and it works, then that could be a start... Yeah, don't have a better idea here. :-\
I tried reporoducing (applied the original patch I posted back to -rc3) and the same issue stubbornly refused to show up again.
But I did hit something with the same signature (overflow bit set in bank 1) while running my futex test (which has two processes mapping the poison page). This time I *do* understand what happened. The test failed when the two processes were running on the two hyperhtreads of the same core. Seeing overflow in this case is understandable because bank 1 MSRs on my test machine are shared between the HT threads. When I run the test again using taskset(1) to only allowing running on thread 0 of each core, it keeps going for hunderds of iterations.
I'm not sure I can stitch together how this overflow also happened for my single process test. Maybe a migration from one HT thread to the other at an awkward moment?
-Tony
Borislav Petkov
maintainerRe: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 18 lines · Tony
Quoted 4 lines · Borislav Petkov
Crazy idea: if you still can reproduce on -rc3, you could bisect: i.e., if you apply the patch on -rc3 and it explodes and if you apply the same patch on -rc5 and it works, then that could be a start... Yeah, don't have a better idea here. :-\
I tried reporoducing (applied the original patch I posted back to -rc3) and the same issue stubbornly refused to show up again.
But I did hit something with the same signature (overflow bit set in bank 1) while running my futex test (which has two processes mapping the poison page). This time I *do* understand what happened. The test failed when the two processes were running on the two hyperhtreads of the same core. Seeing overflow in this case is understandable because bank 1 MSRs on my test machine are shared between the HT threads. When I run the test again using taskset(1) to only allowing running on thread 0 of each core, it keeps going for hunderds of iterations.
Quoted 3 lines
I'm not sure I can stitch together how this overflow also happened for my single process test. Maybe a migration from one HT thread to the other at an awkward moment?
Sounds plausible.
And the much more important question is, what is the code supposed to do when that overflow *actually* happens in real life? Because IINM, an overflow condition on the same page would mean killing the task to contain the error and not killing the machine...
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRE: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 4 lines
And the much more important question is, what is the code supposed to do when that overflow *actually* happens in real life? Because IINM, an overflow condition on the same page would mean killing the task to contain the error and not killing the machine...
Correct. The cases I've actually hit, the second machine check is on the same address as the first. But from a recovery perspective Linux is going to take away the whole page anyway ... so not complaining if the second (or subsequent) access is within the same page makes sense (and that's what the patch does).
The code can't handle it if a subsequent #MC is to a different page (because we only have a single spot in the task structure to store the physical page address). But that looks adequate. If the code is wildly accessing different pages *and* getting machine checks from those different pages ... then something is very seriously wrong with the system.
-Tony
Borislav Petkov
maintainerRe: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 11 lines · Tony
Quoted 4 lines
And the much more important question is, what is the code supposed to do when that overflow *actually* happens in real life? Because IINM, an overflow condition on the same page would mean killing the task to contain the error and not killing the machine...
Correct. The cases I've actually hit, the second machine check is on the same address as the first. But from a recovery perspective Linux is going to take away the whole page anyway ... so not complaining if the second (or subsequent) access is within the same page makes sense (and that's what the patch does).
Quoted 5 lines
The code can't handle it if a subsequent #MC is to a different page (because we only have a single spot in the task structure to store the physical page address). But that looks adequate. If the code is wildly accessing different pages *and* getting machine checks from those different pages ... then something is very seriously wrong with the system.
Right, that's clear.
But I meant something else: this thread had somewhere upthread:
"But the real validation folks took my patch and found that it has destabilized cases 1 & 2 (and case 3 also chokes if you repeat a few more times). System either hangs or panics. Generally before 100 injection/conumption cycles."
To which I asked:
"Or what is the failure exactly?"
and you said:
"After a few cycles of the test injection to user mode, I saw an overflow in the machine check bank. As if it hadn't been cleared from the previous iteration ... but all the banks are cleared as soon as we find that the machine check is recoverable. A while before getting to the code I changed."
And you also said:
"I tried reporoducing (applied the original patch I posted back to -rc3) and the same issue stubbornly refused to show up again."
So that "system hang or panic" which the validation folks triggered, that cannot be reproduced anymore? Did they run the latest version of the patch?
And the overflow issue is something else which the patch handles fine, as you say.
So that original mysterious hang is still unsolved.
Or am I missing something and am misrepresenting the situation?
Thx.
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquetteLuck, Tony
maintainerRE: [PATCH v5] x86/mce: Avoid infinite loop for copy from user recovery
Quoted 3 lines
So that "system hang or panic" which the validation folks triggered, that cannot be reproduced anymore? Did they run the latest version of the patch?
I will get the validation folks to run the latest version (and play around with hyperthreading if they see problems).
-Tony
Borislav Petkov
maintainerRe: [PATCH v3] x86/mce: Avoid infinite loop for copy from user recovery
In reply to Luck, TonyQuoted 2 lines · Tony
In the user mode case we should only bump
mce_countto "1" and beforetask_work()gets called.
Ok, right, it should not be possible to trigger a second MCE while queue_task_work() runs when it is a user MCE. The handler itself won't touch the page with the hw error so our assumption is that it'll get poisoned.
If it doesn't, I guess the memory failure code will kill the process yadda yadda...
Quoted 4 lines
It shouldn't hurt to do the same checks. Maybe it will catch something weird - like an NMI handler on return from the machine check doing a
get_user()that hits another machine check during the return from this machine check.
Eww.
Quoted 1 line
AndyL has made me extra paranoid. :-)
Yeah, he comes up with the nuttiest scenarios. :-)
Signature · 4 lines
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette