x86/shstk: Provide kernel command line knob to disable
Subsystems (2)
PATCH v2 1/1 x86/shstk: Provide kernel command line knob to disable
Provide a kernel command line option 'shstk=off' to disable CET shadow stacks, much like 'ibt=off' can be used to disable CET IBT.
With both set to off, it avoids setting CR4.CET on capable hardware to allow debugging related issues during early boot which I happened to have done way too many times in the recent past.
Document it along with its sibling option 'ibt' in kernel-parameters.txt to allow others to find it more easily.
---
v2:
- pick up Ack's
- document the new option as well as ibt=
- tweak changelog accordingly
Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++ arch/x86/kernel/shstk.c | 9 +++++++++ 2 files changed, 23 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +14 -0
@@ -2248,6 +2248,16 @@ Kernel parameters syscalls, essentially overriding IA32_EMULATION_DEFAULT_DISABLED at boot time. When false, unconditionally disables IA32 emulation. + ibt= [X86-64] + Format: ibt=warn, ibt=off + Changes the handling of CET IBT violations in the kernel. + + The 'warn' setting makes CET IBT violations emit a + warning only instead of being fatal while the 'off' + setting completely disables CET IBT for the kernel. + + To fully disable CET, use 'ibt=off shstk=off'. + icn= [HW,ISDN] Format: <io>[,<membase>[,<icn_id>[,<icn_id2>]]]
@@ -6924,6 +6934,10 @@ Kernel parameters Specify the MCLK divider for Intel SoundWire buses in case the BIOS does not provide the clock rate properly. + shstk=off [X86-64] Disable CET userspace shadow stack support. + + To fully disable CET, use 'ibt=off shstk=off'. + skew_tick= [KNL,EARLY] Offset the periodic timer tick per cpu to mitigate xtime_lock contention on larger systems, and/or RCU lock contention on all systems with CONFIG_MAXSMP set.
arch/x86/kernel/shstk.c +9 -0
@@ -542,6 +542,15 @@ static int shstk_disable(void)return 0 ; } +static int __init shstk_configure (char *str) +{ +if (!strcmp (str,"off" )) +setup_clear_cpu_cap (X86_FEATURE_SHSTK ); + +return 1 ; +} +__setup ("shstk=" , shstk_configure); + SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags) {bool set_tok = flags &SHADOW_STACK_SET_TOKEN ;
Signature · 1 line
2.47.3
Quoted 10 lines · Mathias Krause
Provide a kernel command line option 'shstk=off' to disable CET shadow stacks, much like 'ibt=off' can be used to disable CET IBT.
With both set to off, it avoids setting CR4.CET on capable hardware to allow debugging related issues during early boot which I happened to have done way too many times in the recent past.
Document it along with its sibling option 'ibt' in kernel-parameters.txt to allow others to find it more easily.
Quoted 9 lines
Signed-off-by Mathias Krause <[email protected]>Acked-by Rick Edgecombe <[email protected]>---
v2:
- pick up Ack's
- document the new option as well as ibt=
- tweak changelog accordingly
Ping!
Anything still to fix with this or is it ready to get merged?
Thanks,
Mathias
Quoted 56 lines
Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++ arch/x86/kernel/shstk.c | 9 +++++++++ 2 files changed, 23 insertions(+)Documentation/admin-guide/kernel-parameters.txt +14
-0@@ -2248,6 +2248,16 @@ Kernel parameters syscalls, essentially overriding IA32_EMULATION_DEFAULT_DISABLED at boot time. When false, unconditionally disables IA32 emulation. + ibt= [X86-64] + Format: ibt=warn, ibt=off + Changes the handling of CET IBT violations in the kernel. + + The 'warn' setting makes CET IBT violations emit a + warning only instead of being fatal while the 'off' + setting completely disables CET IBT for the kernel. + + To fully disable CET, use 'ibt=off shstk=off'. + icn= [HW,ISDN] Format: <io>[,<membase>[,<icn_id>[,<icn_id2>]]]@@ -6924,6 +6934,10 @@ Kernel parameters Specify the MCLK divider for Intel SoundWire buses in case the BIOS does not provide the clock rate properly. + shstk=off [X86-64] Disable CET userspace shadow stack support. + + To fully disable CET, use 'ibt=off shstk=off'. + skew_tick= [KNL,EARLY] Offset the periodic timer tick per cpu to mitigate xtime_lock contention on larger systems, and/or RCU lock contention on all systems with CONFIG_MAXSMP set.arch/x86/kernel/shstk.c +9
-0@@ -542,6 +542,15 @@ static int shstk_disable(void)return 0 ; } +static int __init shstk_configure (char *str) +{ +if (!strcmp (str,"off" )) +setup_clear_cpu_cap (X86_FEATURE_SHSTK ); + +return 1 ; +} +__setup ("shstk=" , shstk_configure); + SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags) {bool set_tok = flags &SHADOW_STACK_SET_TOKEN ;
arch/x86/kernel/shstk.c static int shstk_disable(void) · line 552
Quoted 8 lines · Mathias Krause
arch/x86/kernel/shstk.c +8
-0+static int __init shstk_configure (char *str) +{ +if (!strcmp (str,"off" )) +setup_clear_cpu_cap (X86_FEATURE_SHSTK ); + +return 1 ; +} +__setup ("shstk=" , shstk_configure);
Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.
Adding the documentation for ibt= is definitely a good idea, though.
Quoted 3 lines · Dave Hansen
Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.
Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.
I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.
What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?
Quoted 9 lines · Rick P
Quoted 3 lines · Dave Hansen
Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.
Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.
Unfortunately, neither 'clearcpuid=shstk' nor 'clearcpuid=user_shstk' are of any help.
The former doesn't work because X86_FEATURE_SHSTK has no procfs-visible string attached, therefore no entry in x86_cap_flags[] and therefore can't be found via "shstk" in parse_set_clear_cpuid().
The latter only clears X86_FEATURE_USER_SHSTK which is a synthetic feature bit but setup_cet() only looks for X86_FEATURE_SHSTK.
Quoted 3 lines
I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.
Oh, interesting. That'd be the equivalent of 'clearcpuid=user_shstk', right?
Quoted 5 lines
What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?
Right! Clearing, or, moreover, not setting CR4.CET=1 is what I need for the debugging use cases I have in mind and had to hack around a few times in the past.
Case in point, the last debugging session involved a bug with CPU hotplug where the E-cores did not reset their IA32_S_CET MSR on #INIT but the P-cores did (which wasn't the bug, as that's perfectly fine SDM-documented behaviour (not resetting, that is)).
I had a system with CPU hotplug working on the P-cores but not on the E-cores which took a while to debug because the E-cores died hard very early on. CET was in effect way too early (not only in CR4 but also all features via the (stale) MSR), calling a function that didn't had the required ENDBR, causing an early, unhandled #CP.
So, a way to prevent CR4.CET enabling from happening would be nice to have for future debugging sessions.
Thanks,
Mathias
Quoted 14 lines · Mathias Krause
Quoted 9 lines · Rick P
Quoted 3 lines · Dave Hansen
Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.
Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.
Unfortunately, neither 'clearcpuid=shstk' nor 'clearcpuid=user_shstk' are of any help.
Quoted 6 lines
The former doesn't work because
X86_FEATURE_SHSTKhas no procfs-visible string attached, therefore no entry in x86_cap_flags[] and therefore can't be found via "shstk" inparse_set_clear_cpuid().The latter only clears
X86_FEATURE_USER_SHSTKwhich is a synthetic feature bit butsetup_cet()only looks forX86_FEATURE_SHSTK.
So alternatively we could just do:
(quoted diff) +0 -0
b/tools/arch/x86/include/asm/cpufeatures.h
index c3b53beb13007..270341e786f28 100644
tools/arch/x86/include/asm/cpufeatures.h +0 -0
@@ -392,7 +392,7 @@#define X86_FEATURE_OSPKE (16*32+ 4)/* "ospke" OS Protection Keys
Enable */
#define X86_FEATURE_WAITPKG (16*32+ 5) /* "waitpkg"
UMONITOR/UMWAIT/TPAUSE Instructions */
#define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* "avx512_vbmi2" Additional
AVX512 Vector Bit Manipulation s/
-#define X86_FEATURE_SHSTK (16*32+ 7) /* Shadow stack */ +#define X86_FEATURE_SHSTK (16*32+ 7) /* "shstk" Shadow stack */ #define X86_FEATURE_GFNI (16*32+ 8) /* "gfni" Galois Field New
Instructions */
#define X86_FEATURE_VAES (16*32+ 9) /* "vaes" Vector AES */
#define X86_FEATURE_VPCLMULQDQ (16*32+10) /* "vpclmulqdq" Carry-Less
Multiplication Double Quadword */
Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)
Quoted 6 lines
Quoted 4 lines
I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.
Oh, interesting. That'd be the equivalent of 'clearcpuid=user_shstk', right?
Right, except for no taint.
Quoted 7 lines
Quoted 6 lines
What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?
Quoted 8 lines
Right! Clearing, or, moreover, not setting CR4.CET=1 is what I need for the debugging use cases I have in mind and had to hack around a few times in the past.
Case in point, the last debugging session involved a bug with CPU hotplug where the E-cores did not reset their
IA32_S_CETMSR on #INIT but the P-cores did (which wasn't the bug, as that's perfectly fine SDM-documented behaviour (not resetting, that is)).
But the above works for this case, right? The taint doesn't matter for debugging?
Quoted 39 lines · Rick P
Quoted 14 lines · Mathias Krause
Quoted 9 lines · Rick P
Quoted 3 lines · Dave Hansen
Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.
Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.
Unfortunately, neither 'clearcpuid=shstk' nor 'clearcpuid=user_shstk' are of any help.
Quoted 6 lines
The former doesn't work because
X86_FEATURE_SHSTKhas no procfs-visible string attached, therefore no entry in x86_cap_flags[] and therefore can't be found via "shstk" inparse_set_clear_cpuid().The latter only clears
X86_FEATURE_USER_SHSTKwhich is a synthetic feature bit butsetup_cet()only looks forX86_FEATURE_SHSTK.So alternatively we could just do:
tools/arch/x86/include/asm/cpufeatures.h +0
-0@@ -392,7 +392,7 @@#define X86_FEATURE_OSPKE (16*32+ 4)/* "ospke" OS Protection Keys Enable */
#defineX86_FEATURE_WAITPKG(16*32+ 5) /* "waitpkg"
UMONITOR/UMWAIT/TPAUSE Instructions */
#defineX86_FEATURE_AVX512_VBMI2(16*32+ 6) /* "avx512_vbmi2" Additional
AVX512 Vector Bit Manipulation s/tools/arch/x86/include/asm/cpufeatures.h +1
-1-#define X86_FEATURE_SHSTK (16*32+ 7)/* Shadow stack */ +#define X86_FEATURE_SHSTK (16*32+ 7)/* "shstk" Shadow stack */ #define X86_FEATURE_GFNI (16*32+ 8)/* "gfni" Galois Field New
Quoted 4 lines
Instructions */
#defineX86_FEATURE_VAES(16*32+ 9) /* "vaes" Vector AES */
#defineX86_FEATURE_VPCLMULQDQ(16*32+10) /* "vpclmulqdq" Carry-Less
Multiplication Double Quadword */
I explicitly didn't propose that as I was under the assumption, hiding that feature bit is intentional. But, as it was you who added that bit like that in 701fb66d576e ("x86/cpufeatures: Add CPU feature flags for shadow stacks") and is now proposing otherwise, I won't object either.
Quoted 4 lines
Now that KVM uses this this feature independently of
X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)
But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?
Quoted 19 lines
Quoted 5 lines
Quoted 3 lines
I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.
Oh, interesting. That'd be the equivalent of 'clearcpuid=user_shstk', right?
Right, except for no taint.
Quoted 11 lines
Quoted 6 lines
What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?
Right! Clearing, or, moreover, not setting CR4.CET=1 is what I need for the debugging use cases I have in mind and had to hack around a few times in the past.
Quoted 7 lines
Quoted 4 lines
Case in point, the last debugging session involved a bug with CPU hotplug where the E-cores did not reset their
IA32_S_CETMSR on #INIT but the P-cores did (which wasn't the bug, as that's perfectly fine SDM-documented behaviour (not resetting, that is)).But the above works for this case, right? The taint doesn't matter for debugging?
For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?
Thanks,
Mathias
Quoted 5 lines · Mathias Krause
For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?
Last I checked, end users aren't frequently in the business of editing the kernel command line to turn off CPU features. Maybe once in a blue moon to work around bugs. But those folks can live with taint until they get fixes.
Quoted 11 lines · Dave Hansen
Quoted 5 lines · Mathias Krause
For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?
Last I checked, end users aren't frequently in the business of editing the kernel command line to turn off CPU features. Maybe once in a blue moon to work around bugs. But those folks can live with taint until they get fixes.
Ok, I'll drop this one and go for the 'clearcpuid=shstk' approach.
Thanks,
Mathias
Quoted 5 lines · Mathias Krause
I explicitly didn't propose that as I was under the assumption, hiding that feature bit is intentional. But, as it was you who added that bit like that in
701fb66d576e("x86/cpufeatures: Add CPU feature flags for shadow stacks") and is now proposing otherwise, I won't object either.
It sort of was intentional back then. CET has a weird user/supervisor split, where it is split user/supervisor in some sense but locked together in other senses. There was a suggestion to make up some user and kernel feature bits so Linux could differentiate which it actually supports or is using. Which is where user_shstk came from. However, ibt already has just "ibt" to control the kernel support. And the split idea only works part ways, because user and supervisor are linked together in some ways. Really that one should have been be kernel_ibt or something to be clearer, but it predated the user_shstk discussion.
But from where we are today, I guess adding "shstk" seems ok. I guess your patch has the benefit of not stirring the messy waters. Not stirring the waters is maybe a valid answer to Dave's question. I'm ok either way, maybe lean towards adding "shstk".
Quoted 10 lines
Quoted 5 lines
Now that KVM uses this this feature independently of
X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?
For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.
Quoted 11 lines
Quoted 5 lines
Quoted 2 lines
But the above works for this case, right? The taint doesn't matter for debugging?
For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?
Today I think no? We have nousershstk for shadow stack users, and ibt=off for ibt. So everything is covered from the user's perspective.
Quoted 5 lines · Rick P
Quoted 4 lines · Mathias Krause
Quoted 3 lines
Now that KVM uses this this feature independently of
X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)
Quoted 7 lines
Quoted 4 lines
But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?
For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.
https://git.kernel.org/linus/2d5d3fc593c9b7e41bee86175d7b9e11f470072e
Oh, well....
Quoted 11 lines · Mathias Krause
Quoted 10 lines · Rick P
Quoted 8 lines · Mathias Krause
Quoted 3 lines
Now that KVM uses this this feature independently of
X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?
Quoted 6 lines
Quoted 2 lines
For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.
https://git.kernel.org/linus/2d5d3fc593c9b7e41bee86175d7b9e11f470072e
Oh, well....
Heh, well, the clearcpuid would have helped debugging I guess. And that module param turns off shadow stack for KVM, but not for userspace. So doesn't help your CR4.CET problems I guess.
Are you planning to send another revision of the clearcpuid approach? I'm convinced something like this is a good thing to have, so I'll probably pick it up if you don't.
Quoted 15 lines · Rick P
Quoted 14 lines · Mathias Krause
Quoted 12 lines · Rick P
Quoted 8 lines · Mathias Krause
Quoted 3 lines
Now that KVM uses this this feature independently of
X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?
For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.
Quoted 7 lines
Quoted 3 lines
https://git.kernel.org/linus/2d5d3fc593c9b7e41bee86175d7b9e11f470072e
Oh, well....
Heh, well, the clearcpuid would have helped debugging I guess. And that module param turns off shadow stack for KVM, but not for userspace. So doesn't help your CR4.CET problems I guess.
Yeah, I was more trying to say that if the initial version of my patch would have landed, it could be used to workaround the KVM issue in the meantime as well. Anyhow!
Quoted 4 lines
Are you planning to send another revision of the clearcpuid approach? I'm convinced something like this is a good thing to have, so I'll probably pick it up if you don't.
Sorry, I was busy with holidays and travel. Here we go:
https://lore.kernel.org/lkml/[email protected]/
Thanks,
Mathias