Skip to contentSkip to conversation

x86/shstk: Provide kernel command line knob to disable

PATCH In review Mathias Krause 3 participants 13 messages Apr 2 – May 14

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 x86/shstk: Provide kernel command line knob to disable

Provide a kernel command line option 'shstk=off' to disable CET shadow stacks, much like 'ibt=off' can be used to disable CET IBT.

With both set to off, it avoids setting CR4.CET on capable hardware to allow debugging related issues during early boot which I happened to have done way too many times in the recent past.

Document it along with its sibling option 'ibt' in kernel-parameters.txt to allow others to find it more easily.

---
v2:
- pick up Ack's
- document the new option as well as ibt=
- tweak changelog accordingly

 Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
 arch/x86/kernel/shstk.c                         |  9 +++++++++
 2 files changed, 23 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +14 -0
index 03a550630644..43bdf72f6495 100644
@@ -2248,6 +2248,16 @@ Kernel parameters
 			syscalls, essentially overriding IA32_EMULATION_DEFAULT_DISABLED at
 			boot time. When false, unconditionally disables IA32 emulation.
 
+	ibt=		[X86-64]
+			Format: ibt=warn, ibt=off
+			Changes the handling of CET IBT violations in the kernel.
+
+			The 'warn' setting makes CET IBT violations emit a
+			warning only instead of being fatal while the 'off'
+			setting completely disables CET IBT for the kernel.
+
+			To fully disable CET, use 'ibt=off shstk=off'.
+
 	icn=		[HW,ISDN]
 			Format: <io>[,<membase>[,<icn_id>[,<icn_id2>]]]
 
@@ -6924,6 +6934,10 @@ Kernel parameters
 			Specify the MCLK divider for Intel SoundWire buses in
 			case the BIOS does not provide the clock rate properly.
 
+	shstk=off	[X86-64] Disable CET userspace shadow stack support.
+
+			To fully disable CET, use 'ibt=off shstk=off'.
+
 	skew_tick=	[KNL,EARLY] Offset the periodic timer tick per cpu to mitigate
 			xtime_lock contention on larger systems, and/or RCU lock
 			contention on all systems with CONFIG_MAXSMP set.
arch/x86/kernel/shstk.c +9 -0
index 978232b6d48d..68b46bf1540b 100644
@@ -542,6 +542,15 @@ static int shstk_disable(void)
 	return 0;
 }
 
+static int __init shstk_configure(char *str)
+{
+	if (!strcmp(str, "off"))
+		setup_clear_cpu_cap(X86_FEATURE_SHSTK);
+
+	return 1;
+}
+__setup("shstk=", shstk_configure);
+
 SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags)
 {
 	bool set_tok = flags & SHADOW_STACK_SET_TOKEN;
Signature · 1 line
2.47.3
#1 lore raw
Quoted 10 lines · Mathias Krause

Provide a kernel command line option 'shstk=off' to disable CET shadow stacks, much like 'ibt=off' can be used to disable CET IBT.

With both set to off, it avoids setting CR4.CET on capable hardware to allow debugging related issues during early boot which I happened to have done way too many times in the recent past.

Document it along with its sibling option 'ibt' in kernel-parameters.txt to allow others to find it more easily.

Quoted 9 lines

---
v2:
- pick up Ack's
- document the new option as well as ibt=
- tweak changelog accordingly

Ping!

Anything still to fix with this or is it ready to get merged?

Thanks,
Mathias

Quoted 56 lines
 Documentation/admin-guide/kernel-parameters.txt | 14 ++++++++++++++
 arch/x86/kernel/shstk.c                         |  9 +++++++++
 2 files changed, 23 insertions(+)
Documentation/admin-guide/kernel-parameters.txt +14 -0
index 03a550630644..43bdf72f6495 100644
@@ -2248,6 +2248,16 @@ Kernel parameters
 			syscalls, essentially overriding IA32_EMULATION_DEFAULT_DISABLED at
 			boot time. When false, unconditionally disables IA32 emulation.
 
+	ibt=		[X86-64]
+			Format: ibt=warn, ibt=off
+			Changes the handling of CET IBT violations in the kernel.
+
+			The 'warn' setting makes CET IBT violations emit a
+			warning only instead of being fatal while the 'off'
+			setting completely disables CET IBT for the kernel.
+
+			To fully disable CET, use 'ibt=off shstk=off'.
+
 	icn=		[HW,ISDN]
 			Format: <io>[,<membase>[,<icn_id>[,<icn_id2>]]]
 
@@ -6924,6 +6934,10 @@ Kernel parameters
 			Specify the MCLK divider for Intel SoundWire buses in
 			case the BIOS does not provide the clock rate properly.
 
+	shstk=off	[X86-64] Disable CET userspace shadow stack support.
+
+			To fully disable CET, use 'ibt=off shstk=off'.
+
 	skew_tick=	[KNL,EARLY] Offset the periodic timer tick per cpu to mitigate
 			xtime_lock contention on larger systems, and/or RCU lock
 			contention on all systems with CONFIG_MAXSMP set.
arch/x86/kernel/shstk.c +9 -0
index 978232b6d48d..68b46bf1540b 100644
@@ -542,6 +542,15 @@ static int shstk_disable(void)
 	return 0;
 }
 
+static int __init shstk_configure(char *str)
+{
+	if (!strcmp(str, "off"))
+		setup_clear_cpu_cap(X86_FEATURE_SHSTK);
+
+	return 1;
+}
+__setup("shstk=", shstk_configure);
+
 SYSCALL_DEFINE3(map_shadow_stack, unsigned long, addr, unsigned long, size, unsigned int, flags)
 {
 	bool set_tok = flags & SHADOW_STACK_SET_TOKEN;
#2 lore raw

arch/x86/kernel/shstk.c static int shstk_disable(void) · line 552

Quoted 8 lines · Mathias Krause
arch/x86/kernel/shstk.c +8 -0
+static int __init shstk_configure(char *str)
+{
+	if (!strcmp(str, "off"))
+		setup_clear_cpu_cap(X86_FEATURE_SHSTK);
+
+	return 1;
+}
+__setup("shstk=", shstk_configure);

Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.

Adding the documentation for ibt= is definitely a good idea, though.

#3 lore raw
Quoted 3 lines · Dave Hansen

Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.

Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.

I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.

What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?

#4 lore raw
Quoted 9 lines · Rick P
Quoted 3 lines · Dave Hansen

Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.

Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.

Unfortunately, neither 'clearcpuid=shstk' nor 'clearcpuid=user_shstk' are of any help.

The former doesn't work because X86_FEATURE_SHSTK has no procfs-visible string attached, therefore no entry in x86_cap_flags[] and therefore can't be found via "shstk" in parse_set_clear_cpuid().

The latter only clears X86_FEATURE_USER_SHSTK which is a synthetic feature bit but setup_cet() only looks for X86_FEATURE_SHSTK.

Quoted 3 lines

I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.

Oh, interesting. That'd be the equivalent of 'clearcpuid=user_shstk', right?

Quoted 5 lines

What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?

Right! Clearing, or, moreover, not setting CR4.CET=1 is what I need for the debugging use cases I have in mind and had to hack around a few times in the past.

Case in point, the last debugging session involved a bug with CPU hotplug where the E-cores did not reset their IA32_S_CET MSR on #INIT but the P-cores did (which wasn't the bug, as that's perfectly fine SDM-documented behaviour (not resetting, that is)).

I had a system with CPU hotplug working on the P-cores but not on the E-cores which took a while to debug because the E-cores died hard very early on. CET was in effect way too early (not only in CR4 but also all features via the (stale) MSR), calling a function that didn't had the required ENDBR, causing an early, unhandled #CP.

So, a way to prevent CR4.CET enabling from happening would be nice to have for future debugging sessions.

Thanks,
Mathias

#5 lore raw
Quoted 14 lines · Mathias Krause
Quoted 9 lines · Rick P
Quoted 3 lines · Dave Hansen

Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.

Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.

Unfortunately, neither 'clearcpuid=shstk' nor 'clearcpuid=user_shstk' are of any help.

Quoted 6 lines

The former doesn't work because X86_FEATURE_SHSTK has no procfs-visible string attached, therefore no entry in x86_cap_flags[] and therefore can't be found via "shstk" in parse_set_clear_cpuid().

The latter only clears X86_FEATURE_USER_SHSTK which is a synthetic feature bit but setup_cet() only looks for X86_FEATURE_SHSTK.

So alternatively we could just do:

(quoted diff) +0 -0

b/tools/arch/x86/include/asm/cpufeatures.h
index c3b53beb13007..270341e786f28 100644

tools/arch/x86/include/asm/cpufeatures.h +0 -0
@@ -392,7 +392,7 @@
 #define X86_FEATURE_OSPKE              (16*32+ 4) /* "ospke" OS Protection Keys

Enable */
#define X86_FEATURE_WAITPKG (16*32+ 5) /* "waitpkg"
UMONITOR/UMWAIT/TPAUSE Instructions */
#define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* "avx512_vbmi2" Additional
AVX512 Vector Bit Manipulation s/

-#define X86_FEATURE_SHSTK              (16*32+ 7) /* Shadow stack */
+#define X86_FEATURE_SHSTK              (16*32+ 7) /* "shstk" Shadow stack */
 #define X86_FEATURE_GFNI               (16*32+ 8) /* "gfni" Galois Field New

Instructions */
#define X86_FEATURE_VAES (16*32+ 9) /* "vaes" Vector AES */
#define X86_FEATURE_VPCLMULQDQ (16*32+10) /* "vpclmulqdq" Carry-Less
Multiplication Double Quadword */

Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)

Quoted 6 lines
Quoted 4 lines

I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.

Oh, interesting. That'd be the equivalent of 'clearcpuid=user_shstk', right?

Right, except for no taint.

Quoted 7 lines
Quoted 6 lines

What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?

Quoted 8 lines

Right! Clearing, or, moreover, not setting CR4.CET=1 is what I need for the debugging use cases I have in mind and had to hack around a few times in the past.

Case in point, the last debugging session involved a bug with CPU hotplug where the E-cores did not reset their IA32_S_CET MSR on #INIT but the P-cores did (which wasn't the bug, as that's perfectly fine SDM-documented behaviour (not resetting, that is)).

But the above works for this case, right? The taint doesn't matter for debugging?

#6 lore raw
Quoted 39 lines · Rick P
Quoted 14 lines · Mathias Krause
Quoted 9 lines · Rick P
Quoted 3 lines · Dave Hansen

Is there a reason that clearcpuid=shstk doesn't work in this case? I guess shstk and ibt are peers, but I was kinda hoping we'd stop adding these for every single CPU feature at _some_ point.

Oh yea, for the reason of "debugging related issues during early boot" clearcpuid of shstk and ibt should be fine. It taints the kernel, but should be fine for debugging? If I'm reading this right, the kernel does the clearcpuid processing before setting up CET bits.

Unfortunately, neither 'clearcpuid=shstk' nor 'clearcpuid=user_shstk' are of any help.

Quoted 6 lines

The former doesn't work because X86_FEATURE_SHSTK has no procfs-visible string attached, therefore no entry in x86_cap_flags[] and therefore can't be found via "shstk" in parse_set_clear_cpuid().

The latter only clears X86_FEATURE_USER_SHSTK which is a synthetic feature bit but setup_cet() only looks for X86_FEATURE_SHSTK.

So alternatively we could just do:

tools/arch/x86/include/asm/cpufeatures.h +0 -0
index c3b53beb13007..270341e786f28 100644
@@ -392,7 +392,7 @@
 #define X86_FEATURE_OSPKE              (16*32+ 4) /* "ospke" OS Protection Keys

Enable */
#define X86_FEATURE_WAITPKG (16*32+ 5) /* "waitpkg"
UMONITOR/UMWAIT/TPAUSE Instructions */
#define X86_FEATURE_AVX512_VBMI2 (16*32+ 6) /* "avx512_vbmi2" Additional
AVX512 Vector Bit Manipulation s/

tools/arch/x86/include/asm/cpufeatures.h +1 -1
-#define X86_FEATURE_SHSTK              (16*32+ 7) /* Shadow stack */
+#define X86_FEATURE_SHSTK              (16*32+ 7) /* "shstk" Shadow stack */
 #define X86_FEATURE_GFNI               (16*32+ 8) /* "gfni" Galois Field New
Quoted 4 lines

Instructions */
#define X86_FEATURE_VAES (16*32+ 9) /* "vaes" Vector AES */
#define X86_FEATURE_VPCLMULQDQ (16*32+10) /* "vpclmulqdq" Carry-Less
Multiplication Double Quadword */

I explicitly didn't propose that as I was under the assumption, hiding that feature bit is intentional. But, as it was you who added that bit like that in 701fb66d576e ("x86/cpufeatures: Add CPU feature flags for shadow stacks") and is now proposing otherwise, I won't object either.

Quoted 4 lines

Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)

But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?

Quoted 19 lines
Quoted 5 lines
Quoted 3 lines

I'm remembering we actually already have a "nousershstk" too, which covers the "userspace init cet violations break boot" usage.

Oh, interesting. That'd be the equivalent of 'clearcpuid=user_shstk', right?

Right, except for no taint.

Quoted 11 lines
Quoted 6 lines

What that doesn't do though, is clear CR4.CET. With nousershstk, KVM can still use CET. So that is what is missing. A way to clear CR4.CET without tainting the kernel when HW supports CET. Do we need it?

Right! Clearing, or, moreover, not setting CR4.CET=1 is what I need for the debugging use cases I have in mind and had to hack around a few times in the past.

Quoted 7 lines
Quoted 4 lines

Case in point, the last debugging session involved a bug with CPU hotplug where the E-cores did not reset their IA32_S_CET MSR on #INIT but the P-cores did (which wasn't the bug, as that's perfectly fine SDM-documented behaviour (not resetting, that is)).

But the above works for this case, right? The taint doesn't matter for debugging?

For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?

Thanks,
Mathias

#7 lore raw
Quoted 5 lines · Mathias Krause

For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?

Last I checked, end users aren't frequently in the business of editing the kernel command line to turn off CPU features. Maybe once in a blue moon to work around bugs. But those folks can live with taint until they get fixes.

#8 lore raw
Quoted 11 lines · Dave Hansen
Quoted 5 lines · Mathias Krause

For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?

Last I checked, end users aren't frequently in the business of editing the kernel command line to turn off CPU features. Maybe once in a blue moon to work around bugs. But those folks can live with taint until they get fixes.

Ok, I'll drop this one and go for the 'clearcpuid=shstk' approach.

Thanks,
Mathias

#9 lore raw
In reply to Mathias Krause
Quoted 5 lines · Mathias Krause

I explicitly didn't propose that as I was under the assumption, hiding that feature bit is intentional. But, as it was you who added that bit like that in 701fb66d576e ("x86/cpufeatures: Add CPU feature flags for shadow stacks") and is now proposing otherwise, I won't object either.

It sort of was intentional back then. CET has a weird user/supervisor split, where it is split user/supervisor in some sense but locked together in other senses. There was a suggestion to make up some user and kernel feature bits so Linux could differentiate which it actually supports or is using. Which is where user_shstk came from. However, ibt already has just "ibt" to control the kernel support. And the split idea only works part ways, because user and supervisor are linked together in some ways. Really that one should have been be kernel_ibt or something to be clearer, but it predated the user_shstk discussion.

But from where we are today, I guess adding "shstk" seems ok. I guess your patch has the benefit of not stirring the messy waters. Not stirring the waters is maybe a valid answer to Dave's question. I'm ok either way, maybe lean towards adding "shstk".

Quoted 10 lines
Quoted 5 lines

Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)

But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?

For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.

Quoted 11 lines
Quoted 5 lines
Quoted 2 lines

But the above works for this case, right? The taint doesn't matter for debugging?

For *me*, 'clearcpuid=shstk,ibt' would be sufficient for my debugging needs. It's just a question if there's more demand beside some random kernel hacker needing a knob to disable potential problematic features, i.e. do we expect actual *end users* having a need to fully disable CET shadow stacks too?

Today I think no? We have nousershstk for shadow stack users, and ibt=off for ibt. So everything is covered from the user's perspective.

#10 lore raw
Quoted 5 lines · Rick P
Quoted 4 lines · Mathias Krause
Quoted 3 lines

Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)

Quoted 7 lines
Quoted 4 lines

But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?

For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.

https://git.kernel.org/linus/2d5d3fc593c9b7e41bee86175d7b9e11f470072e

Oh, well....

#11 lore raw
Quoted 11 lines · Mathias Krause
Quoted 10 lines · Rick P
Quoted 8 lines · Mathias Krause
Quoted 3 lines

Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)

But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?

Quoted 6 lines
Quoted 2 lines

For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.

https://git.kernel.org/linus/2d5d3fc593c9b7e41bee86175d7b9e11f470072e

Oh, well....

Heh, well, the clearcpuid would have helped debugging I guess. And that module param turns off shadow stack for KVM, but not for userspace. So doesn't help your CR4.CET problems I guess.

Are you planning to send another revision of the clearcpuid approach? I'm convinced something like this is a good thing to have, so I'll probably pick it up if you don't.

#12 lore raw
Quoted 15 lines · Rick P
Quoted 14 lines · Mathias Krause
Quoted 12 lines · Rick P
Quoted 8 lines · Mathias Krause
Quoted 3 lines

Now that KVM uses this this feature independently of X86_FEATURE_USER_SHSTK, it might be good to have the plain HW shstk feature exposed for just normal runtime user use. (+Chao, for KVM CET)

But that sounds more like having the need for an official chicken bit, like I was proposing, no? Using 'clearcpuid=shstk' as a workaround for whatever KVM bugs, similar in spirit to 'nousershstk', but without the kernel taint?

For users to turn off shadow stack for guests? You can do this via the KVM API in the normal way you customize guests.

Quoted 7 lines
Quoted 3 lines

https://git.kernel.org/linus/2d5d3fc593c9b7e41bee86175d7b9e11f470072e

Oh, well....

Heh, well, the clearcpuid would have helped debugging I guess. And that module param turns off shadow stack for KVM, but not for userspace. So doesn't help your CR4.CET problems I guess.

Yeah, I was more trying to say that if the initial version of my patch would have landed, it could be used to workaround the KVM issue in the meantime as well. Anyhow!

Quoted 4 lines

Are you planning to send another revision of the clearcpuid approach? I'm convinced something like this is a good thing to have, so I'll probably pick it up if you don't.

Sorry, I was busy with holidays and travel. Here we go:
https://lore.kernel.org/lkml/[email protected]/

Thanks,
Mathias