pr_warn_once() issue in x86 MSR extable code
Hi,
Some changes to the way invalid MSR accesses are reported by the kernel is causing some problems with messages printed on the console.
We have seen several cases of ex_handler_msr() printing invalid MSR accesses once but
the callstack multiple times causing confusion on the console.
The last time the exception MSR code was modified (5.16) by PeterZ was:
d52a7344bdfa x86/msr: Remove .fixup usage:
if (!safe && wrmsr && pr_warn_once("unchecked MSR access error: ..."))
show_stack_regs(regs);Note that this code pattern was also present, though in a different form, before this commit.
The problem here is that another earlier commit (5.13):
a358f40600b3 once: implement DO_ONCE_LITE for non-fast-path "do once" functionality
Modifies all the pr_*_once() calls to always return true claiming that no caller is ever
checking the return value of the functions.
This is why we are seeing the callstack printed without the associated printk() msg.
I believe that having the pr_*_once() functions return true the first time they are called
is useful especially when extra information, such as callstack, must be printed to help
track the origin of the problem.
The exception handling code seems to be the only place where the return value is checked
for pr_warn_once(). A minimal change would be to create another version of that function
that calls DO_ONCE() instead of DO_ONCE_LITE(), e.g., pr_warn_once_return().
I can post a patch to that effect if we all agree on the approach.
Thanks.
Peter Zijlstra
maintainerQuoted 43 lines · Stephane Eranian
Hi,
Some changes to the way invalid MSR accesses are reported by the kernel is causing some problems with messages printed on the console.
We have seen several cases of
ex_handler_msr()printing invalid MSR accesses once but
the callstack multiple times causing confusion on the console.The last time the exception MSR code was modified (5.16) by PeterZ was:
d52a7344bdfax86/msr: Remove .fixup usage:if (!safe && wrmsr && pr_warn_once("unchecked MSR access error: ...")) show_stack_regs(regs);Note that this code pattern was also present, though in a different form, before this commit.
The problem here is that another earlier commit (5.13):
a358f40600b3once: implementDO_ONCE_LITEfor non-fast-path "do once" functionalityModifies all the pr_*_once() calls to always return true claiming that no caller is ever
checking the return value of the functions.This is why we are seeing the callstack printed without the associated
printk()msg.I believe that having the pr_*_once() functions return true the first time they are called
is useful especially when extra information, such as callstack, must be printed to help
track the origin of the problem.The exception handling code seems to be the only place where the return value is checked
forpr_warn_once(). A minimal change would be to create another version of that function
that callsDO_ONCE()instead ofDO_ONCE_LITE(), e.g.,pr_warn_once_return().
Quoted 3 lines
I can post a patch to that effect if we all agree on the approach.
Thanks.
How about something like this?
arch/x86/mm/extable.c +9 -7
@@ -94,16 +94,18 @@ static bool ex_handler_copy(const struct exception_table_entry *fixup,static bool ex_handler_msr (const struct exception_table_entry *fixup,struct pt_regs *regs,bool wrmsr,bool safe,int reg) { -if (!safe && wrmsr && -pr_warn_once ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, (unsigned int )regs->dx, - (unsigned int )regs->ax, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && wrmsr)) { +pr_warn ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, (unsigned int )regs->dx, + (unsigned int )regs->ax, regs->ip, (void *)regs->ip);show_stack_regs (regs); + } -if (!safe && !wrmsr && -pr_warn_once ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && !wrmsr)) { +pr_warn ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, regs->ip, (void *)regs->ip);show_stack_regs (regs); + }if (!wrmsr) {/* Pretend that the read succeeded and returned 0. */
include/linux/once_lite.h +15 -3
@@ -9,15 +9,27 @@*/ #define DO_ONCE_LITE (func, ...) \ DO_ONCE_LITE_IF(true, func, ##__VA_ARGS__) -#define DO_ONCE_LITE_IF (condition, func, ...) \ + +#define __ONCE_LITE_IF (condition) \ ({ \ static bool __section(".data.once") __already_done; \ - bool __ret_do_once = !!(condition); \ + bool __ret_cond = !!(condition); \ + bool __ret_once = false; \ \ if (unlikely(__ret_do_once && !__already_done)) { \ __already_done = true; \ - func(__VA_ARGS__); \ + __ret_once = true; \ } \ + unlikely(__ret_once); \ + }) + +#define DO_ONCE_LITE_IF (condition, func, ...) \ + ({ \ + bool __ret_do_once = !!(condition); \ + \ + if (__ONCE_LITE_IF(__ret_do_once)) \ + func(__VA_ARGS__); \ + \ unlikely(__ret_do_once); \ })
Hi Peter,
Thanks for taking a quick look at this.
I am currently OOO and I cannot test this proposed patch. I am okay with your suggestion.
Thanks.
Quoted 116 lines · Peter Zijlstra
Quoted 43 lines · Stephane Eranian
Hi,
Some changes to the way invalid MSR accesses are reported by the kernel is causing some problems with messages printed on the console.
We have seen several cases of
ex_handler_msr()printing invalid MSR accesses once but
the callstack multiple times causing confusion on the console.The last time the exception MSR code was modified (5.16) by PeterZ was:
d52a7344bdfax86/msr: Remove .fixup usage:if (!safe && wrmsr && pr_warn_once("unchecked MSR access error: ...")) show_stack_regs(regs);Note that this code pattern was also present, though in a different form, before this commit.
The problem here is that another earlier commit (5.13):
a358f40600b3once: implementDO_ONCE_LITEfor non-fast-path "do once" functionalityModifies all the pr_*_once() calls to always return true claiming that no caller is ever
checking the return value of the functions.This is why we are seeing the callstack printed without the associated
printk()msg.I believe that having the pr_*_once() functions return true the first time they are called
is useful especially when extra information, such as callstack, must be printed to help
track the origin of the problem.The exception handling code seems to be the only place where the return value is checked
forpr_warn_once(). A minimal change would be to create another version of that function
that callsDO_ONCE()instead ofDO_ONCE_LITE(), e.g.,pr_warn_once_return().Quoted 3 lines
I can post a patch to that effect if we all agree on the approach.
Thanks.
How about something like this?
arch/x86/mm/extable.c +9
-7@@ -94,16 +94,18 @@ static bool ex_handler_copy(const struct exception_table_entry *fixup,static bool ex_handler_msr (const struct exception_table_entry *fixup,struct pt_regs *regs,bool wrmsr,bool safe,int reg) { -if (!safe && wrmsr && -pr_warn_once ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, (unsigned int )regs->dx, - (unsigned int )regs->ax, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && wrmsr)) { +pr_warn ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, (unsigned int )regs->dx, + (unsigned int )regs->ax, regs->ip, (void *)regs->ip);show_stack_regs (regs); + } -if (!safe && !wrmsr && -pr_warn_once ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && !wrmsr)) { +pr_warn ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, regs->ip, (void *)regs->ip);show_stack_regs (regs); + }if (!wrmsr) {/* Pretend that the read succeeded and returned 0. */ include/linux/once_lite.h +15
-3@@ -9,15 +9,27 @@*/ #define DO_ONCE_LITE (func, ...) \ DO_ONCE_LITE_IF(true, func, ##__VA_ARGS__) -#define DO_ONCE_LITE_IF (condition, func, ...) \ + +#define __ONCE_LITE_IF (condition) \ ({ \ static bool __section(".data.once") __already_done; \ - bool __ret_do_once = !!(condition); \ + bool __ret_cond = !!(condition); \ + bool __ret_once = false; \ \ if (unlikely(__ret_do_once && !__already_done)) { \ __already_done = true; \ - func(__VA_ARGS__); \ + __ret_once = true; \ } \ + unlikely(__ret_once); \ + }) + +#define DO_ONCE_LITE_IF (condition, func, ...) \ + ({ \ + bool __ret_do_once = !!(condition); \ + \ + if (__ONCE_LITE_IF(__ret_do_once)) \ + func(__VA_ARGS__); \ + \ unlikely(__ret_do_once); \ })
Stephane Eranian
Tested-byQuoted 101 lines · Stephane Eranian
Hi Peter,
Thanks for taking a quick look at this.
I am currently OOO and I cannot test this proposed patch. I am okay with your suggestion.Thanks.
Quoted 91 lines · Peter Zijlstra
Quoted 43 lines · Stephane Eranian
Hi,
Some changes to the way invalid MSR accesses are reported by the kernel is causing some problems with messages printed on the console.
We have seen several cases of
ex_handler_msr()printing invalid MSR accesses once but
the callstack multiple times causing confusion on the console.The last time the exception MSR code was modified (5.16) by PeterZ was:
d52a7344bdfax86/msr: Remove .fixup usage:if (!safe && wrmsr && pr_warn_once("unchecked MSR access error: ...")) show_stack_regs(regs);Note that this code pattern was also present, though in a different form, before this commit.
The problem here is that another earlier commit (5.13):
a358f40600b3once: implementDO_ONCE_LITEfor non-fast-path "do once" functionalityModifies all the pr_*_once() calls to always return true claiming that no caller is ever
checking the return value of the functions.This is why we are seeing the callstack printed without the associated
printk()msg.I believe that having the pr_*_once() functions return true the first time they are called
is useful especially when extra information, such as callstack, must be printed to help
track the origin of the problem.The exception handling code seems to be the only place where the return value is checked
forpr_warn_once(). A minimal change would be to create another version of that function
that callsDO_ONCE()instead ofDO_ONCE_LITE(), e.g.,pr_warn_once_return().Quoted 3 lines
I can post a patch to that effect if we all agree on the approach.
Thanks.
How about something like this?
arch/x86/mm/extable.c +9
-7@@ -94,16 +94,18 @@ static bool ex_handler_copy(const struct exception_table_entry *fixup,static bool ex_handler_msr (const struct exception_table_entry *fixup,struct pt_regs *regs,bool wrmsr,bool safe,int reg) { -if (!safe && wrmsr && -pr_warn_once ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, (unsigned int )regs->dx, - (unsigned int )regs->ax, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && wrmsr)) { +pr_warn ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, (unsigned int )regs->dx, + (unsigned int )regs->ax, regs->ip, (void *)regs->ip);show_stack_regs (regs); + } -if (!safe && !wrmsr && -pr_warn_once ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && !wrmsr)) { +pr_warn ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, regs->ip, (void *)regs->ip);show_stack_regs (regs); + }if (!wrmsr) {/* Pretend that the read succeeded and returned 0. */ include/linux/once_lite.h +1
-1@@ -9,15 +9,27 @@*/ #define DO_ONCE_LITE (func, ...) \ DO_ONCE_LITE_IF(true, func, ##__VA_ARGS__) -#define DO_ONCE_LITE_IF (condition, func, ...) \ +
Quoted 8 lines
include/linux/once_lite.h line 19
Quoted 8 lines
include/linux/once_lite.h +3
-1+#define __ONCE_LITE_IF (condition) \ ({ \ static bool __section(".data.once") __already_done; \ - bool __ret_do_once = !!(condition); \ + bool __ret_cond = !!(condition); \ + bool __ret_once = false; \ \ if (unlikely(__ret_do_once && !__already_done)) { \
You need to replace __ret_do_once with __ret_cond above and then it compiles and works.
I have tested with a kernel module that reads and writes to an illegal MSR:
unchecked MSR access error: RDMSR from 0x1234567 at rIP:0xffffffffc00ec138 (rdpmc_intel+0x28/0x21d0 [rdpmc_test])
Call Trace: <TASK> rdpmc_bench_store+0x53/0x80 [rdpmc_test] kobj_attr_store+0xf/0x20 sysfs_kf_write+0x34/0x50 kernfs_fop_write_iter+0xfa/0x180 vfs_write+0x334/0x3d0 ksys_write+0x71/0xe0 __x64_sys_write+0x1b/0x20 do_syscall_64+0x44/0xa0 ? exc_page_fault+0x6e/0x110 entry_SYSCALL_64_after_hwframe+0x63/0xcd
Quoted 18 lines
Quoted 18 lines
include/linux/once_lite.h +11
-1__already_done = true; \ -func (__VA_ARGS__); \ + __ret_once = true; \ } \ +unlikely (__ret_once); \ + }) + +#define DO_ONCE_LITE_IF (condition, func, ...) \ + ({ \ + bool __ret_do_once = !!(condition); \ + \ + if (__ONCE_LITE_IF(__ret_do_once)) \ + func(__VA_ARGS__); \ + \ unlikely(__ret_do_once); \ })
Peter Zijlstra
maintainerPATCH 1/1 x86/extable: Fix ex_handler_msr() print condition
Quoted 1 line · Stephane Eranian
Tested-by Stephane Eranian <[email protected]>
Thanks, I'll queue the below.
---
Subject: x86/extable: Fix ex_handler_msr() print condition
From: Peter Zijlstra <[email protected]>
Date: Fri, 17 Jun 2022 16:52:06 +0200
Quoted 10 lines · Stephane Eranian
Some changes to the way invalid MSR accesses are reported by the kernel is causing some problems with messages printed on the console.
We have seen several cases of
ex_handler_msr()printing invalid MSR accesses once but the callstack multiple times causing confusion on the console.The problem here is that another earlier commit (5.13):
Quoted 7 lines
a358f40600b3("once: implementDO_ONCE_LITEfor non-fast-path "do once" functionality")Modifies all the pr_*_once() calls to always return true claiming that no caller is ever checking the return value of the functions.
This is why we are seeing the callstack printed without the associated
printk()msg.
Extract the ONCE_IF(cond) part into __ONCE_LTE_IF() and use that to implement DO_ONCE_LITE_IF() and fix the extable code.
--- arch/x86/mm/extable.c | 16 +++++++++------- include/linux/once_lite.h | 20 ++++++++++++++++---- 2 files changed, 25 insertions(+), 11 deletions(-)
arch/x86/mm/extable.c +9 -7
@@ -94,16 +94,18 @@ static bool ex_handler_copy(const structstatic bool ex_handler_msr (const struct exception_table_entry *fixup,struct pt_regs *regs,bool wrmsr,bool safe,int reg) { -if (!safe && wrmsr && -pr_warn_once ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, (unsigned int )regs->dx, - (unsigned int )regs->ax, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && wrmsr)) { +pr_warn ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, (unsigned int )regs->dx, + (unsigned int )regs->ax, regs->ip, (void *)regs->ip);show_stack_regs (regs); + } -if (!safe && !wrmsr && -pr_warn_once ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && !wrmsr)) { +pr_warn ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, regs->ip, (void *)regs->ip);show_stack_regs (regs); + }if (!wrmsr) {/* Pretend that the read succeeded and returned 0. */
include/linux/once_lite.h +16 -4
@@ -9,15 +9,27 @@*/ #define DO_ONCE_LITE (func, ...) \ DO_ONCE_LITE_IF(true, func, ##__VA_ARGS__) -#define DO_ONCE_LITE_IF (condition, func, ...) \ + +#define __ONCE_LITE_IF (condition) \ ({ \ static bool __section(".data.once") __already_done; \ - bool __ret_do_once = !!(condition); \ + bool __ret_cond = !!(condition); \ + bool __ret_once = false; \ \ - if (unlikely(__ret_do_once && !__already_done)) { \ + if (unlikely(__ret_cond && !__already_done)) { \ __already_done = true; \ - func(__VA_ARGS__); \ + __ret_once = true; \ } \ + unlikely(__ret_once); \ + }) + +#define DO_ONCE_LITE_IF (condition, func, ...) \ + ({ \ + bool __ret_do_once = !!(condition); \ + \ + if (__ONCE_LITE_IF(__ret_do_once)) \ + func(__VA_ARGS__); \ + \ unlikely(__ret_do_once); \ })
[tip: x86/core] x86/extable: Fix ex_handler_msr() print condition
In reply to Peter ZijlstraThe following commit has been merged into the x86/core branch of tip:
The following commit has been merged into the x86/core branch of tip:
Commit-ID: a1a5482a2c6e38a3ebed32e571625c56a8cc41a6 Gitweb: https://git.kernel.org/tip/a1a5482a2c6e38a3ebed32e571625c56a8cc41a6 Author: Peter Zijlstra <[email protected]> AuthorDate: Fri, 17 Jun 2022 16:52:06 +02:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Thu, 21 Jul 2022 10:39:42 +02:00
x86/extable: Fix ex_handler_msr() print condition
Quoted 10 lines · Stephane Eranian
Some changes to the way invalid MSR accesses are reported by the kernel is causing some problems with messages printed on the console.
We have seen several cases of
ex_handler_msr()printing invalid MSR accesses once but the callstack multiple times causing confusion on the console.The problem here is that another earlier commit (5.13):
Quoted 7 lines
a358f40600b3("once: implementDO_ONCE_LITEfor non-fast-path "do once" functionality")Modifies all the pr_*_once() calls to always return true claiming that no caller is ever checking the return value of the functions.
This is why we are seeing the callstack printed without the associated
printk()msg.
Extract the ONCE_IF(cond) part into __ONCE_LTE_IF() and use that to implement DO_ONCE_LITE_IF() and fix the extable code.
--- arch/x86/mm/extable.c | 16 +++++++++------- include/linux/once_lite.h | 20 ++++++++++++++++---- 2 files changed, 25 insertions(+), 11 deletions(-)
arch/x86/mm/extable.c +9 -7
@@ -94,16 +94,18 @@ static bool ex_handler_copy(const struct exception_table_entry *fixup,static bool ex_handler_msr (const struct exception_table_entry *fixup,struct pt_regs *regs,bool wrmsr,bool safe,int reg) { -if (!safe && wrmsr && -pr_warn_once ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, (unsigned int )regs->dx, - (unsigned int )regs->ax, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && wrmsr)) { +pr_warn ("unchecked MSR access error: WRMSR to 0x%x (tried to write 0x%08x%08x) at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, (unsigned int )regs->dx, + (unsigned int )regs->ax, regs->ip, (void *)regs->ip);show_stack_regs (regs); + } -if (!safe && !wrmsr && -pr_warn_once ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , - (unsigned int )regs->cx, regs->ip, (void *)regs->ip)) +if (__ONCE_LITE_IF (!safe && !wrmsr)) { +pr_warn ("unchecked MSR access error: RDMSR from 0x%x at rIP: 0x%lx (%pS)\n" , + (unsigned int )regs->cx, regs->ip, (void *)regs->ip);show_stack_regs (regs); + }if (!wrmsr) {/* Pretend that the read succeeded and returned 0. */
include/linux/once_lite.h +16 -4
@@ -9,15 +9,27 @@*/ #define DO_ONCE_LITE (func, ...) \ DO_ONCE_LITE_IF(true, func, ##__VA_ARGS__) -#define DO_ONCE_LITE_IF (condition, func, ...) \ + +#define __ONCE_LITE_IF (condition) \ ({ \ static bool __section(".data.once") __already_done; \ - bool __ret_do_once = !!(condition); \ + bool __ret_cond = !!(condition); \ + bool __ret_once = false; \ \ - if (unlikely(__ret_do_once && !__already_done)) { \ + if (unlikely(__ret_cond && !__already_done)) { \ __already_done = true; \ - func(__VA_ARGS__); \ + __ret_once = true; \ } \ + unlikely(__ret_once); \ + }) + +#define DO_ONCE_LITE_IF (condition, func, ...) \ + ({ \ + bool __ret_do_once = !!(condition); \ + \ + if (__ONCE_LITE_IF(__ret_do_once)) \ + func(__VA_ARGS__); \ + \ unlikely(__ret_do_once); \ })