Add support for the ERAPS feature

From: Amit Shah <[email protected]>

AMD CPUs do not fall back to the BTB when the RSB underflows for RET address speculation. AMD CPUs have not needed to stuff the RSB for underflow conditions.

The RSB poisoning case is addressed by RSB filling - clean up the FIXME comment about it.

From: Amit Shah <[email protected]>

AMD CPUs do not fall back to the BTB when the RSB underflows for RET address speculation. AMD CPUs have not needed to stuff the RSB for underflow conditions.

The RSB poisoning case is addressed by RSB filling - clean up the FIXME comment about it.

-----Original Message-----
From: Josh Poimboeuf <[email protected]>
Sent: Monday, November 11, 2024 1:33 PM
To: Amit Shah <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; linux- [email protected]; Shah, Amit <[email protected]>; Lendacky, Thomas <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Das1, Sandipan <[email protected]>; [email protected]; Moger, Babu <[email protected]>; Kaplan, David <[email protected]>; [email protected]; [email protected]
Subject: Re: [RFC PATCH v2 1/3] x86: cpu/bugs: update SpectreRSB comments for AMD

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Quoted 8 lines · Amit Shah

From: Amit Shah <[email protected]>

AMD CPUs do not fall back to the BTB when the RSB underflows for RET address speculation. AMD CPUs have not needed to stuff the RSB for underflow conditions.

The RSB poisoning case is addressed by RSB filling - clean up the FIXME comment about it.

I'm thinking the comments need more clarification in light of BTC and SRSO.

This:

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1832

Quoted 2 lines

arch/x86/kernel/cpu/bugs.c +0 -2

-      *    AMD has it even worse: *all* returns are speculated from the BTB,
-      *    regardless of the state of the RSB.

is still true (mostly: "all" should be "some"), though it doesn't belong in the "RSB underflow" section.

Also the RSB stuffing not only mitigates RET, it mitigates any other instruction which happen to be predicted as a RET. Which is presumably why it's still needed even when SRSO is enabled.

Quoted 7 lines

I'm thinking the comments need more clarification in light of BTC and SRSO.

This:

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1832

Quoted 2 lines

arch/x86/kernel/cpu/bugs.c +0 -2

-      *    AMD has it even worse: *all* returns are speculated from the BTB,
-      *    regardless of the state of the RSB.

Quoted 7 lines

is still true (mostly: "all" should be "some"), though it doesn't belong in the "RSB underflow" section.

Also the RSB stuffing not only mitigates RET, it mitigates any other instruction which happen to be predicted as a RET. Which is presumably why it's still needed even when SRSO is enabled.

While that's partly true, I'm not sure I'm understanding which scenario you're concerned with. As noted in the AMD BTC whitepaper, there are various types of potential mis-speculation depending on what the actual branch is. The 'late redirect' ones are the most concerning since those have enough of a speculation window to be able to do a load-op-load gadget. The only 'late redirect' case involving an instruction being incorrectly predicted as a RET is when the actual instruction is an indirect JMP/CALL. But those instructions are either removed entirely (due to retpoline) or being protected with IBRS. The cases of other instructions (like Jcc) being predicted as a RET are subject to the 'early redirect' behavior which is much more limited (and note that these can also be predicted as indirect branches for which there is no special protection). So I'm not sure why BTC specifically would necessitate needing to stuff the RSB here.

Also, BTC only affects some AMD parts (not Family 19h and later for instance).

This is why it's important to spell out all the different cases in the comments. I was attempting to document the justifications for the existing behavior.

You make some good points, though backing up a bit, I realize my comment was flawed for another reason: the return thunks only protect the kernel, but RSB filling on context switch is meant to protect user space.

So, never mind...

Quoted 9 lines · Josh Poimboeuf

This is why it's important to spell out all the different cases in the comments. I was attempting to document the justifications for the existing behavior.

You make some good points, though backing up a bit, I realize my comment was flawed for another reason: the return thunks only protect the kernel, but RSB filling on context switch is meant to protect user space.

Quoted 1 line

So, never mind...

That said, I still think the comments need an update. I'll try to come up with something later.

Quoted 8 lines · Amit Shah

From: Amit Shah <[email protected]>

AMD CPUs do not fall back to the BTB when the RSB underflows for RET address speculation. AMD CPUs have not needed to stuff the RSB for underflow conditions.

The RSB poisoning case is addressed by RSB filling - clean up the FIXME comment about it.

I'm thinking the comments need more clarification in light of BTC and SRSO.

This:

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1832

Quoted 2 lines

arch/x86/kernel/cpu/bugs.c +0 -2

-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.

is still true (mostly: "all" should be "some"), though it doesn't belong in the "RSB underflow" section.

Also the RSB stuffing not only mitigates RET, it mitigates any other instruction which happen to be predicted as a RET. Which is presumably why it's still needed even when SRSO is enabled.

Something like below?

arch/x86/kernel/cpu/bugs.c +3 -3

index 47a01d4028f6..e95d3aa14259 100644
@@ -1828,9 +1828,6 @@ static void __init spectre_v2_select_mitigation(void)
 	 *    speculated return targets may come from the branch predictor,
 	 *    which could have a user-poisoned BTB or BHB entry.
 	 *
-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.
-	 *
 	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack
 	 *    scenario is mitigated by the IBRS branch prediction isolation
 	 *    properties, so the RSB buffer filling wouldn't be necessary to

@@ -1850,10 +1847,22 @@ static void __init spectre_v2_select_mitigation(void)
 	 *    The "user -> user" scenario, also known as SpectreBHB, requires
 	 *    RSB clearing.
 	 *
+	 *    AMD Branch Type Confusion (aka "AMD retbleed") adds some
+	 *    additional wrinkles:
+	 *

This is my take.  On AMD CPUs, there are two unrelated issues to take into account:

1) SRSO

Affects anything which doesn't enumerate SRSO_NO, which is all parts to date including Zen5.

SRSO ends up overflowing the RAS with arbitrary BTB targets, such that a subsequent genuine RET follows a prediction which never came from a real CALL instruction.

Mitigations for SRSO are either safe-ret, or IBPB-on-entry.  Parts without IBPB_RET using IBPB-on-entry need to manually flush the RAS.

Importantly, SMEP does not protection you against SRSO across the user->kernel boundary, because the bad RAS entries are arbitrary.  New in Zen5 is the SRSO_U/S_NO bit which says this case can't occur any more.  So on Zen5, you can in principle get away without a RAS flush on entry.

Subject: [PATCH] x86/bugs: Update insanely long comment about RSB attacks

Quoted 1 line · Josh Poimboeuf

Subject: [PATCH] x86/bugs: Update insanely long comment about RSB attacks

Why don't you stick this insanely long comment in
Documentation/admin-guide/hw-vuln/ while at it?

Its place is hardly in the code. You can point to it from the code tho...

There are a lot of subtle details to this $#!tstorm, and IMO we probably wouldn't be having these discussions in the first place if the comment lived in the docs, as most people seem to ignore them...

Quoted 3 lines · Josh Poimboeuf

There are a lot of subtle details to this $#!tstorm, and IMO we probably wouldn't be having these discussions in the first place if the comment lived in the docs, as most people seem to ignore them...

That's why I'm saying point to the docs from the code. You can't have a big fat comment in the code about this but everything else in the hw-vuln docs.

This comment relates to the "why" for the code itself (and its poor confused developers), taking all the RSB-related vulnerabilities into account.

Quoted 3 lines · Josh Poimboeuf

This comment relates to the "why" for the code itself (and its poor confused developers), taking all the RSB-related vulnerabilities into account.

So use Documentation/arch/x86/.

This is exactly the reason why we need more "why" documentation - because everytime we have to swap the whole bugs.c horror back in, we're poor confused developers. And we have the "why" spread out across commit messages and other folklore which means everytime we have to change stuff, the git archeology starts. :-\ "err, do you remember why we're doing this?!" And so on converstaions on IRC.

So having an implementation document explaining clearly why we did things is long overdue.

But it's fine - I can move it later when the dust settles here.

I think in-line documentation is better in this case: the primary defense against mistakes and misunderstandings is in the source code itself.

And "it's too long" is an argument *against* moving it out into some obscure place 99% of developers aren't even aware of...

Quoted 20 lines · Andrew Cooper

This is my take.  On AMD CPUs, there are two unrelated issues to take
into account:

1) SRSO

Affects anything which doesn't enumerate SRSO_NO, which is all parts to
date including Zen5.

SRSO ends up overflowing the RAS with arbitrary BTB targets, such that a
subsequent genuine RET follows a prediction which never came from a real
CALL instruction.

Mitigations for SRSO are either safe-ret, or IBPB-on-entry.  Parts without IBPB_RET using IBPB-on-entry need to manually flush the RAS.

Quoted 7 lines

Importantly, SMEP does not protection you against SRSO across the user->kernel boundary, because the bad RAS entries are arbitrary. New
in Zen5 is the SRSO_U/S_NO bit which says this case can't occur any more.  So on Zen5, you can in principle get away without a RAS flush on
entry.

Updated to mention SRSO:

	/*
	 * In general there are two types of RSB attacks:
	 *
	 * 1) RSB underflow ("Intel Retbleed")
	 *
	 *    Some Intel parts have "bottomless RSB".  When the RSB
is empty,
	 *    speculated return targets may come from the branch
predictor,
	 *    which could have a user-poisoned BTB or BHB entry.
	 *
	 *    When IBRS or eIBRS is enabled, the "user -> kernel"

attack is
*    mitigated by the IBRS branch prediction isolation
properties, so
*    the RSB buffer filling wouldn't be necessary to
protect against

	 *    this type of attack.
	 *
	 *    The "user -> user" attack is mitigated by RSB filling
on context
	 *    switch.
	 *
	 *    The "guest -> host" attack is mitigated by IBRS or
eIBRS.
	 *
	 * 2) Poisoned RSB entry
	 *
	 *    If the 'next' in-kernel return stack is shorter than

'prev',
*    'next' could be tricked into speculating with a user-
poisoned RSB
*    entry.  Poisoned RSB entries can also be created by
Branch Type

	 *    Confusion ("AMD retbleed") or SRSO.
	 *
	 *    The "user -> kernel" attack is mitigated by SMEP and
eIBRS.  AMD
	 *    without SRSO_NO also needs the SRSO mitigation.
	 *
	 *    The "user -> user" attack, also known as SpectreBHB,
requires RSB
	 *    clearing.
	 *
	 *    The "guest -> host" attack is mitigated by either

eIBRS (not
*    IBRS!) or RSB clearing on vmexit.  Note that eIBRS *    implementations with X86_BUG_EIBRS_PBRSB still need
"lite" RSB
*    clearing which retires a single CALL before the first
RET.

	 */

arch/x86/kernel/cpu/bugs.c +2 -2

+	 * 1) RSB underflow ("Intel Retbleed")
 	 *
 	 *    Some Intel parts have "bottomless RSB".  When the RSB is empty,
 	 *    speculated return targets may come from the branch predictor,
 	 *    which could have a user-poisoned BTB or BHB entry.
 	 *
-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.
+	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack is

Quoted 9 lines · Josh Poimboeuf

arch/x86/kernel/cpu/bugs.c +2 -2

+	 * 1) RSB underflow ("Intel Retbleed")
 	 *
 	 *    Some Intel parts have "bottomless RSB".  When the RSB is empty,
 	 *    speculated return targets may come from the branch predictor,
 	 *    which could have a user-poisoned BTB or BHB entry.
 	 *
-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.
+	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack is

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1815

Quoted 10 lines

arch/x86/kernel/cpu/bugs.c +5 -4

+	 *    mitigated by the IBRS branch prediction isolation properties, so
+	 *    the RSB buffer filling wouldn't be necessary to protect against
+	 *    this type of attack.
 	 *
-	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack
-	 *    scenario is mitigated by the IBRS branch prediction isolation
-	 *    properties, so the RSB buffer filling wouldn't be necessary to
-	 *    protect against this type of attack.
+	 *    The "user -> user" attack is mitigated by RSB filling on context
+	 *    switch.

user->user SpectreRSB is also mitigated by IBPB, so RSB filling is unnecessary when IBPB is issued. Also, when an appication does not opted-in for IBPB at context switch, spectre-v2 for that app is not mitigated, filling RSB is only a half measure in that case.

Is RSB filling really serving any purpose for userspace?

Quoted 26 lines · Pawan Gupta

Quoted 9 lines · Josh Poimboeuf

arch/x86/kernel/cpu/bugs.c +2 -2

+	 * 1) RSB underflow ("Intel Retbleed")
 	 *
 	 *    Some Intel parts have "bottomless RSB".  When the RSB is empty,
 	 *    speculated return targets may come from the branch predictor,
 	 *    which could have a user-poisoned BTB or BHB entry.
 	 *
-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.
+	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack is

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1815

Quoted 10 lines

arch/x86/kernel/cpu/bugs.c +5 -4

+	 *    mitigated by the IBRS branch prediction isolation properties, so
+	 *    the RSB buffer filling wouldn't be necessary to protect against
+	 *    this type of attack.
 	 *
-	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack
-	 *    scenario is mitigated by the IBRS branch prediction isolation
-	 *    properties, so the RSB buffer filling wouldn't be necessary to
-	 *    protect against this type of attack.
+	 *    The "user -> user" attack is mitigated by RSB filling on context
+	 *    switch.

user->user SpectreRSB is also mitigated by IBPB, so RSB filling is unnecessary when IBPB is issued. Also, when an appication does not opted-in for IBPB at context switch, spectre-v2 for that app is not mitigated, filling RSB is only a half measure in that case.

Quoted 1 line

Is RSB filling really serving any purpose for userspace?

Indeed...

If we don't need to flush RSB for user->user, we'd only need to worry about protecting the kernel. Something like so?

- eIBRS+!PBRSB: no flush
- eIBRS+PBRSB: lite flush

- everything else: full flush

i.e., same logic as spectre_v2_determine_rsb_fill_type_at_vmexit(), but also for context switches.

Quoted 7 lines

Quoted 6 lines

user->user SpectreRSB is also mitigated by IBPB, so RSB filling is unnecessary when IBPB is issued. Also, when an appication does not opted-in for IBPB at context switch, spectre-v2 for that app is not mitigated, filling RSB is only a half measure in that case.

Is RSB filling really serving any purpose for userspace?

Quoted 7 lines

Indeed...

If we don't need to flush RSB for user->user, we'd only need to worry about protecting the kernel. Something like so?

- eIBRS+!PBRSB: no flush
- eIBRS+PBRSB: lite flush

Yes for VMexit, but not at kernel entry. PBRSB requires an unbalanced RET, and it is only a problem until the first retired CALL. At VMexit we do have unbalanced RET but not at kernel entry.

Quoted 4 lines

- everything else: full flush

i.e., same logic as spectre_v2_determine_rsb_fill_type_at_vmexit(), but also for context switches.

Yes, assuming you mean user->kernel switch, and not process context switch.

Quoted 19 lines · Pawan Gupta

Quoted 7 lines

Quoted 6 lines

user->user SpectreRSB is also mitigated by IBPB, so RSB filling is unnecessary when IBPB is issued. Also, when an appication does not opted-in for IBPB at context switch, spectre-v2 for that app is not mitigated, filling RSB is only a half measure in that case.

Is RSB filling really serving any purpose for userspace?

Quoted 7 lines

Indeed...

If we don't need to flush RSB for user->user, we'd only need to worry about protecting the kernel. Something like so?

- eIBRS+!PBRSB: no flush
- eIBRS+PBRSB: lite flush

Yes for VMexit, but not at kernel entry. PBRSB requires an unbalanced RET, and it is only a problem until the first retired CALL. At VMexit we do have unbalanced RET but not at kernel entry.

Quoted 6 lines

Quoted 4 lines

- everything else: full flush

i.e., same logic as spectre_v2_determine_rsb_fill_type_at_vmexit(), but also for context switches.

Yes, assuming you mean user->kernel switch, and not process context switch.

Actually I did mean context switch. AFAIK we don't need to flush RSB at kernel entry.

If user->user RSB is already mitigated by IBPB, then at context switch we only have to worry about user->kernel. e.g., if 'next' has more (in kernel) RETs then 'prev' had (in kernel) CALLs, the user could trigger RSB underflow or corruption inside the kernel after the context switch.

Doesn't eIBRS already protect against that?

For PBRSB, I guess we don't need to worry about that since there would be at least one kernel CALL before context switch.

Quoted 2 lines

For PBRSB, I guess we don't need to worry about that since there would be at least one kernel CALL before context switch.

Right. So the case where we need RSB filling at context switch is retpoline+CDT mitigation.

-----Original Message-----
From: Josh Poimboeuf <[email protected]>
Sent: Thursday, November 14, 2024 11:49 PM
To: Pawan Gupta <[email protected]>
Cc: Andrew Cooper <[email protected]>; Amit Shah <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; Shah, Amit <[email protected]>; Lendacky, Thomas <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; Das1, Sandipan <[email protected]>; [email protected]; Moger, Babu <[email protected]>; Kaplan, David <[email protected]>; [email protected]
Subject: Re: [RFC PATCH v2 1/3] x86: cpu/bugs: update SpectreRSB comments for AMD

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

On Thu, Nov 14, 2024 at 12:01:16AM -0800, Pawan Gupta wrote:

Quoted 5 lines

Quoted 2 lines

For PBRSB, I guess we don't need to worry about that since there would be at least one kernel CALL before context switch.

Right. So the case where we need RSB filling at context switch is retpoline+CDT mitigation.

According to the docs, classic IBRS also needs RSB filling at context switch to protect against corrupt RSB entries (as opposed to RSB underflow).

Quoted 7 lines · Pawan Gupta

Quoted 5 lines

Quoted 2 lines

For PBRSB, I guess we don't need to worry about that since there would be at least one kernel CALL before context switch.

Right. So the case where we need RSB filling at context switch is retpoline+CDT mitigation.

Quoted 2 lines

According to the docs, classic IBRS also needs RSB filling at context switch to protect against corrupt RSB entries (as opposed to RSB underflow).

Which docs are that? Classic IBRS doesn't do anything with returns (at least on AMD). The AMD docs say that if you want to prevent earlier instructions from influencing later RETs, you need to do the 32 CALL sequence. But I'm not sure what corrupt RSB entries mean here, and how it relates to IBRS?

According to the docs, classic IBRS also needs RSB filling at context switch to protect against corrupt RSB entries (as opposed to RSB underflow).

Something like so...

arch/x86/kernel/cpu/bugs.c +34 -52

index 47a01d4028f6..7b9c0a21e478 100644
@@ -1579,27 +1579,44 @@ static void __init spec_ctrl_disable_kernel_rrsba(void)
 	rrsba_disabled = true;
 }
 
-static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_mitigation mode)
+static void __init spectre_v2_mitigate_rsb(enum spectre_v2_mitigation mode)
 {
 	/*
-	 * Similar to context switches, there are two types of RSB attacks
-	 * after VM exit:
+	 * In general there are two types of RSB attacks:
 	 *
-	 * 1) RSB underflow
+	 * 1) RSB underflow ("Intel Retbleed")
+	 *
+	 *    Some Intel parts have "bottomless RSB".  When the RSB is empty,
+	 *    speculated return targets may come from the branch predictor,
+	 *    which could have a user-poisoned BTB or BHB entry.
+	 *
+	 *    user->user attacks are mitigated by IBPB on context switch.
+	 *
+	 *    user->kernel attacks via context switch are mitigated by IBRS,
+	 *    eIBRS, or RSB filling.
+	 *
+	 *    user->kernel attacks via kernel entry are mitigated by IBRS,
+	 *    eIBRS, or call depth tracking.
+	 *
+	 *    On VMEXIT, guest->host attacks are mitigated by IBRS, eIBRS, or
+	 *    RSB filling.
 	 *
 	 * 2) Poisoned RSB entry
 	 *
-	 * When retpoline is enabled, both are mitigated by filling/clearing
-	 * the RSB.
+	 *    On a context switch, the previous task can poison RSB entries
+	 *    used by the next task, controlling its speculative return
+	 *    targets.  Poisoned RSB entries can also be created by "AMD
+	 *    Retbleed" or SRSO.
 	 *
-	 * When IBRS is enabled, while #1 would be mitigated by the IBRS branch
-	 * prediction isolation protections, RSB still needs to be cleared
-	 * because of #2.  Note that SMEP provides no protection here, unlike
-	 * user-space-poisoned RSB entries.
+	 *    user->user attacks are mitigated by IBPB on context switch.
 	 *
-	 * eIBRS should protect against RSB poisoning, but if the EIBRS_PBRSB
-	 * bug is present then a LITE version of RSB protection is required,
-	 * just a single call needs to retire before a RET is executed.
+	 *    user->kernel attacks via context switch are prevented by
+	 *    SMEP+eIBRS+SRSO mitigations, or RSB clearing.
+	 *
+	 *    guest->host attacks are mitigated by eIBRS or RSB clearing on
+	 *    VMEXIT.  eIBRS implementations with X86_BUG_EIBRS_PBRSB still
+	 *    need "lite" RSB filling which retires a CALL before the first
+	 *    RET.
 	 */
 	switch (mode) {
 	case SPECTRE_V2_NONE:

@@ -1608,8 +1625,8 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	case SPECTRE_V2_EIBRS_LFENCE:
 	case SPECTRE_V2_EIBRS:
 		if (boot_cpu_has_bug(X86_BUG_EIBRS_PBRSB)) {
-			setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT_LITE);
 			pr_info("Spectre v2 / PBRSB-eIBRS: Retire a single CALL on VMEXIT\n");
+			setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT_LITE);
 		}
 		return;
 

@@ -1617,12 +1634,13 @@ static void __init spectre_v2_determine_rsb_fill_type_at_vmexit(enum spectre_v2_
 	case SPECTRE_V2_RETPOLINE:
 	case SPECTRE_V2_LFENCE:
 	case SPECTRE_V2_IBRS:
+		pr_info("Spectre v2 / SpectreRSB : Filling RSB on context switch and VMEXIT\n");
+		setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
 		setup_force_cpu_cap(X86_FEATURE_RSB_VMEXIT);
-		pr_info("Spectre v2 / SpectreRSB : Filling RSB on VMEXIT\n");
 		return;
 	}
 
-	pr_warn_once("Unknown Spectre v2 mode, disabling RSB mitigation at VM exit");
+	pr_warn_once("Unknown Spectre v2 mode, disabling RSB mitigation\n");
 	dump_stack();
 }
 

@@ -1817,48 +1835,7 @@ static void __init spectre_v2_select_mitigation(void)
 	spectre_v2_enabled = mode;
 	pr_info("%s\n", spectre_v2_strings[mode]);
 
-	/*
-	 * If Spectre v2 protection has been enabled, fill the RSB during a
-	 * context switch.  In general there are two types of RSB attacks
-	 * across context switches, for which the CALLs/RETs may be unbalanced.
-	 *
-	 * 1) RSB underflow
-	 *
-	 *    Some Intel parts have "bottomless RSB".  When the RSB is empty,
-	 *    speculated return targets may come from the branch predictor,
-	 *    which could have a user-poisoned BTB or BHB entry.
-	 *
-	 *    AMD has it even worse: *all* returns are speculated from the BTB,
-	 *    regardless of the state of the RSB.
-	 *
-	 *    When IBRS or eIBRS is enabled, the "user -> kernel" attack
-	 *    scenario is mitigated by the IBRS branch prediction isolation
-	 *    properties, so the RSB buffer filling wouldn't be necessary to
-	 *    protect against this type of attack.
-	 *
-	 *    The "user -> user" attack scenario is mitigated by RSB filling.
-	 *
-	 * 2) Poisoned RSB entry
-	 *
-	 *    If the 'next' in-kernel return stack is shorter than 'prev',
-	 *    'next' could be tricked into speculating with a user-poisoned RSB
-	 *    entry.
-	 *
-	 *    The "user -> kernel" attack scenario is mitigated by SMEP and
-	 *    eIBRS.
-	 *
-	 *    The "user -> user" scenario, also known as SpectreBHB, requires
-	 *    RSB clearing.
-	 *
-	 * So to mitigate all cases, unconditionally fill RSB on context
-	 * switches.
-	 *

This LGTM.

I think SPECTRE_V2_EIBRS_RETPOLINE is placed in the wrong leg, it doesn't need RSB filling on context switch, and only needs VMEXIT_LITE. Does below change on top of your patch look okay?

---

arch/x86/kernel/cpu/bugs.c +1 -1

index 7b9c0a21e478..d3b9a0d7a2b5 100644
@@ -1622,6 +1622,7 @@ static void __init spectre_v2_mitigate_rsb(enum spectre_v2_mitigation mode)
 	case SPECTRE_V2_NONE:
 		return;
 
+	case SPECTRE_V2_EIBRS_RETPOLINE:
 	case SPECTRE_V2_EIBRS_LFENCE:
 	case SPECTRE_V2_EIBRS:
 		if (boot_cpu_has_bug(X86_BUG_EIBRS_PBRSB)) {

@@ -1630,7 +1631,6 @@ static void __init spectre_v2_mitigate_rsb(enum spectre_v2_mitigation mode)
 		}
 		return;
 
-	case SPECTRE_V2_EIBRS_RETPOLINE:
 	case SPECTRE_V2_RETPOLINE:
 	case SPECTRE_V2_LFENCE:
 	case SPECTRE_V2_IBRS:

Quoted 6 lines · Pawan Gupta

This LGTM.

I think SPECTRE_V2_EIBRS_RETPOLINE is placed in the wrong leg, it doesn't need RSB filling on context switch, and only needs VMEXIT_LITE.
Does below change on top of your patch look okay?

Yeah, I was wondering about that too.  Since it changes existing VMEXIT_LITE behavior I'll make it a separate patch.  And I'll probably
do the comment changes in a separate patch as well.

Remove explicit RET stuffing / filling on VMEXITs and context switches on AMD CPUs with the ERAPS feature (Zen5+).

With the Enhanced Return Address Prediction Security feature, any of the following conditions triggers a flush of the RSB (aka RAP in AMD manuals) in hardware:
* context switch (e.g., move to CR3)

* TLB flush
* some writes to CR4

The feature also explicitly tags host and guest addresses in the RSB - eliminating the need for explicit flushing of the RSB on VMEXIT.

[RFC note: We'll wait for the APM to be updated with the real wording, but assuming it's going to say the ERAPS feature works the way described above, let's continue the discussion re: when the kernel currently calls FILL_RETURN_BUFFER, and what dropping it entirely means.

Dave Hansen pointed out __switch_to_asm fills the RSB each time it's called, so let's address the cases there:

1. user->kernel switch: Switching from userspace to kernelspace, and then using user-stuffed RSB entries in the kernel is not possible thanks to SMEP. We can safely drop the FILL_RETURN_BUFFER call for this case. In fact, this is how the original code was when dwmw2 added it originally in c995efd5a. So while this case currently triggers an RSB flush (and will not after this ERAPS patch), the current flush isn't necessary for AMD systems with SMEP anyway.

2. user->user or kernel->kernel: If a user->user switch does not result in a CR3 change, it's a different thread in the same process context. That's the same case for kernel->kernel switch. In this case, the RSB entries are still valid in that context, just not the correct ones in the new thread's context. It's difficult to imagine this being a security risk. The current code clearing it, and this patch not doing so for AMD-with-ERAPS, isn't a concern as far as I see.
]

arch/x86/include/asm/nospec-branch.h +3 -0

index 96b410b1d4e8..f5ee7fc71db5 100644
@@ -117,6 +117,18 @@
  * We define a CPP macro such that it can be used from both .S files and
  * inline assembly. It's possible to do a .macro and then include that
  * from C via asm(".include <asm/nospec-branch.h>") but let's not go there.
+ *
+ * AMD CPUs with the ERAPS feature may have a larger default RSB.  These CPUs
+ * use the default number of entries on a host, and can optionally (based on

arch/x86/include/asm/nospec-branch.h +0 -0

 #define RETPOLINE_THUNK_SIZE	32

arch/x86/kernel/cpu/bugs.c +0 -0

index 0aa629b5537d..02446815b0de 100644

arch/x86/kernel/cpu/bugs.c +6 -3

+	 * In general there are two types of RSB attacks across context
+	 * switches, for which the CALLs/RETs may be unbalanced.
 	 *
 	 * 1) RSB underflow
 	 *

@@ -1848,12 +1851,21 @@ static void __init spectre_v2_select_mitigation(void)
 	 *    RSB clearing.
 	 *
 	 * So to mitigate all cases, unconditionally fill RSB on context
-	 * switches.
+	 * switches when ERAPS is not present.
 	 */
-	setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
-	pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch\n");
+	if (!boot_cpu_has(X86_FEATURE_ERAPS)) {
+		setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
+		pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch\n");
 

arch/x86/kernel/cpu/bugs.c +2 -1

 	/*
 	 * Retpoline protects the kernel, but doesn't protect firmware.  IBRS

@@ -2866,7 +2878,7 @@ static ssize_t spectre_v2_show_state(char *buf)
 	    spectre_v2_enabled == SPECTRE_V2_EIBRS_LFENCE)
 		return sysfs_emit(buf, "Vulnerable: eIBRS+LFENCE with unprivileged eBPF and SMT\n");
 
-	return sysfs_emit(buf, "%s%s%s%s%s%s%s%s\n",
+	return sysfs_emit(buf, "%s%s%s%s%s%s%s%s%s\n",
 			  spectre_v2_strings[spectre_v2_enabled],
 			  ibpb_state(),
 			  boot_cpu_has(X86_FEATURE_USE_IBRS_FW) ? "; IBRS_FW" : "",

@@ -2874,6 +2886,7 @@ static ssize_t spectre_v2_show_state(char *buf)
 			  boot_cpu_has(X86_FEATURE_RSB_CTXSW) ? "; RSB filling" : "",
 			  pbrsb_eibrs_state(),
 			  spectre_bhi_state(),
+			  boot_cpu_has(X86_FEATURE_ERAPS) ? "; ERAPS hardware RSB flush" : "",
 			  /* this should always be at the end */
 			  spectre_v2_module_string());
 }

Quoted 2 lines · Amit Shah

Remove explicit RET stuffing / filling on VMEXITs and context switches on AMD CPUs with the ERAPS feature (Zen5+).

I'm not a super big fan of how this changelog is constructed.  I personally really like the "background, problem, solution" form. This
starts with the solution before even telling us what it is.

Quoted 6 lines

With the Enhanced Return Address Prediction Security feature,  any of
the following conditions triggers a flush of the RSB (aka RAP in AMD
manuals) in hardware:
* context switch (e.g., move to CR3)

IMNHO, context switching is a broader topic that process-to-process switches.  A user=>kernel thread switch is definitely a context switch,
but doesn't involve a CR3 write.  A user=>user switch in the same mm is
also highly arguable to be a context switch and doesn't involve a CR3 write.  There are userspace threading libraries that do "context switches".

This is further muddled by the very comments you are editing in this patch like: "unconditionally fill RSB on context switches".

Please be very, very precise in the language that's chosen here.

Quoted 2 lines

* TLB flush
* some writes to CR4

... and only one of those is relevant for this patch.  Aren't the others
just noise?

Quoted 16 lines

The feature also explicitly tags host and guest addresses in the RSB -
eliminating the need for explicit flushing of the RSB on VMEXIT.

[RFC note: We'll wait for the APM to be updated with the real wording,
but assuming it's going to say the ERAPS feature works the way described
above, let's continue the discussion re: when the kernel currently calls
FILL_RETURN_BUFFER, and what dropping it entirely means.

Dave Hansen pointed out __switch_to_asm fills the RSB each time it's
called, so let's address the cases there:

Quoted 11 lines

1. user->kernel switch: Switching from userspace to kernelspace, and
   then using user-stuffed RSB entries in the kernel is not possible
   thanks to SMEP.  We can safely drop the FILL_RETURN_BUFFER call for
   this case.  In fact, this is how the original code was when dwmw2
   added it originally in c995efd5a.  So while this case currently triggers an RSB flush (and will not after this ERAPS patch), the current flush isn't necessary for AMD systems with SMEP anyway.

This description makes me uneasy.  It basically boils down to, "SMEP guarantees that the kernel can't consume user-placed RSB entries."

It makes me uneasy because I recall that not holding true on some Intel
CPUs. I believe some CPUs have a partial-width RSB.  Kernel consumption
of a user-placed entry would induce the kernel to speculate to a *kernel* address so SMEP is rather ineffective.

So, instead of just saying, "SMEP magically fixes everything, trust us",
could we please step through a few of the properties of the hardware and
software that make SMEP effective?

First, I think that you're saying that AMD hardware has a full-width, precise RSB.  That ensures that speculation always happens back precisely to the original address, not some weird alias.  Second, ERAPS
guarantees that the the address still has the same stuff mapped there.
Any change to the address space involves either a CR3 write or a TLB flush, both of which would have flushed any user-placed content in the RSB.

Aside: Even the kernel-only text poking mm or EFI mm would
"just
work" in this scenario since they have their own mm_structs, page tables and root CR3 values.

Quoted 7 lines

2. user->user or kernel->kernel: If a user->user switch does not result
   in a CR3 change, it's a different thread in the same process context.
   That's the same case for kernel->kernel switch.  In this case, the
   RSB entries are still valid in that context, just not the

Quoted 8 lines

correct
   ones in the new thread's context.  It's difficult to imagine this
   being a security risk.  The current code clearing it, and this patch
   not doing so for AMD-with-ERAPS, isn't a concern as far as I see.
]

The current rules are dirt simple: if the kernel stack gets switched, the RSB is flushed.  It's kinda hard to have a mismatched stack if it's
never switched in the first place.  That makes the current rules dirt simple.  The problem (stack switching) is dirt simple to correlate 1:1
with the fix (RSB stuffing).

This patch proposes separating the problem (stack switching) and the mitigations (implicit new microcode actions).  That's a hell of a lot more complicated and hardware to audit than the existing rules. Agreed?

So, how are the rules relaxed?

First, it opens up the case where user threads consume RSB entries from
other threads in the same process. Threads are usually at the same privilege level.  Instead of using a speculative attack, an attacker could just read the data directly.  The only place I can imagine this even remotely being a problem would be if threads were relying on protection keys to keep secrets from each other.

The kernel consuming RSB entries from another kernel thread seems less
clear.  I disagree with the idea that a valid entry in a given context
is a _safe_ entry though.  Spectre v2 in _general_ involves nasty speculation to otherwise perfectly safe code. A problematic scenario would be where kswapd wakes up after waiting for I/O and starts speculating back along the return path of the userspace thread that kswapd interrupted. Userspace has some level of control over both stacks
and it doesn't seem super far fetched to think that there could be some
disclosure gadgets in there.

In short: the user-consumes-user-RSB-entry attacks seem fundamentally improbable because user threads are unlikely to have secrets from each
other.

The kernel-consumes-kernel-RSB-entry attacks seem hard rather than fundamentally improbable.  I can't even count how many times our "oh that attack would be too hard to pull off" optimism has gone wrong.

What does that all _mean_?  ERAPS sucks?  Nah.  Maybe we just make sure
that the existing spectre_v2=whatever controls can be used to stop relying on it if asked.

Quoted 25 lines

arch/x86/include/asm/nospec-branch.h +0 -0

index 96b410b1d4e8..f5ee7fc71db5 100644
@@ -117,6 +117,18 @@
   * We define a CPP macro such that it can be used from both .S

files and
  * inline assembly. It's possible to do a .macro and then include that
  * from C via asm(".include <asm/nospec-branch.h>") but let's not go there.
+ *
+ * AMD CPUs with the ERAPS feature may have a larger default RSB. These CPUs
+ * use the default number of entries on a host, and can optionally (based on
+ * hypervisor setup) use 32 (old) or the new default in a guest. The number
+ * of default entries is reflected in CPUID 8000_0021:EBX[23:16].
+ *
+ * With the ERAPS feature, RSB filling is not necessary anymore:
the RSB is
+ * auto-cleared by hardware on context switches, TLB flushes, or some CR4

Quoted 8 lines

+ * writes.  Adapting the value of RSB_CLEAR_LOOPS below for ERAPS would change
+ * it to a runtime variable instead of the current compile-time constant, so
+ * leave it as-is, as this works for both older CPUs, as well as newer ones
+ * with ERAPS.
  */

This comment feels out of place and noisy to me.  Why the heck would we
need to document the RSB size enumeration here and not use it?

Then this goes on to explain why this patch *didn't* bother to change RSB_CLEAR_LOOPS.  That seems much more like changelog material than a code comment to me.

To me, RSB_CLEAR_LOOPS, is for, well, clearing the RSB.  The existing comments say that some CPUs don't need to clear the RSB.  I don't think
we need further explanation of why one specific CPU doesn't need to clear the RSB.  The new CPU isn't special.

Something slightly more useful would be to actually read CPUID 8000_0021:EBX[23:16] and compare it to RSB_CLEAR_LOOPS:

void amd_check_rsb_clearing(void)

{
	/*
	 * Systems with both these features off
	 * do not perform RSB clearing:
	 */
	if (!boot_cpu_has(X86_FEATURE_RSB_CTXSW) &&
	    !boot_cpu_has(X86_FEATURE_RSB_VMEXIT))
		return;

	// with actual helpers and mask generation, not this mess:
	rsb_size = (cpuid_ebx(0x80000021) >> 16) & 0xff;

	WARN_ON_ONCE(rsb_size > RSB_CLEAR_LOOPS);
}

That's almost shorter than the proposed comment.

Quoted 21 lines

 #define RETPOLINE_THUNK_SIZE 32

arch/x86/kernel/cpu/bugs.c +0 -0

index 0aa629b5537d..02446815b0de 100644
@@ -1818,9 +1818,12 @@ static void __init
 spectre_v2_select_mitigation(void)
  	pr_info("%s\n", spectre_v2_strings[mode]);

  /*

arch/x86/kernel/cpu/bugs.c +0 -1

-	 * If Spectre v2 protection has been enabled, fill the RSB

during a

arch/x86/kernel/cpu/bugs.c +0 -1

-	 * context switch.  In general there are two types of RSB

attacks

arch/x86/kernel/cpu/bugs.c +0 -1

-	 * across context switches, for which the CALLs/RETs may

be unbalanced.

arch/x86/kernel/cpu/bugs.c +1 -0

+	 * If Spectre v2 protection has been enabled, the RSB

needs to be

arch/x86/kernel/cpu/bugs.c +1 -0

+	 * cleared during a context switch.  Either do it in

software by

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1823

Quoted 1 line

arch/x86/kernel/cpu/bugs.c +1 -0

+	 * filling the RSB, or in hardware via ERAPS.

I'd move this comment about using ERAPS down to the ERAPS check itself.

Quoted 38 lines

arch/x86/kernel/cpu/bugs.c +1 -0

+	 * In general there are two types of RSB attacks across

context

arch/x86/kernel/cpu/bugs.c +1 -0

+	 * switches, for which the CALLs/RETs may be unbalanced.

  *
  * 1) RSB underflow
  *

arch/x86/kernel/cpu/bugs.c +0 -0

@@ -1848,12 +1851,21 @@ static void __init
 spectre_v2_select_mitigation(void)
  	 *    RSB clearing.
  	 *
  	 * So to mitigate all cases, unconditionally fill RSB on

context

arch/x86/kernel/cpu/bugs.c +1 -1

-	 * switches.
+	 * switches when ERAPS is not present.

  */

arch/x86/kernel/cpu/bugs.c +0 -2

-	setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
-	pr_info("Spectre v2 / SpectreRSB mitigation: Filling RSB

on context switch\n");

arch/x86/kernel/cpu/bugs.c +3 -0

+	if (!boot_cpu_has(X86_FEATURE_ERAPS)) {
+		setup_force_cpu_cap(X86_FEATURE_RSB_CTXSW);
+		pr_info("Spectre v2 / SpectreRSB mitigation:

Filling RSB on context switch\n");

arch/x86/kernel/cpu/bugs.c +2 -1

-	spectre_v2_determine_rsb_fill_type_at_vmexit(mode);
+		/*
+		 * For guest -> host (or vice versa) RSB poisoning

scenarios,

arch/x86/kernel/cpu/bugs.c +1 -0

+		 * determine the mitigation mode here.  With

ERAPS, RSB

arch/x86/kernel/cpu/bugs.c +1 -0

+		 * entries are tagged as host or guest - ensuring

that neither

arch/x86/kernel/cpu/bugs.c +1 -0

+		 * the host nor the guest have to clear or fill

RSB entries to

arch/x86/kernel/cpu/bugs.c +1 -0

+		 * avoid poisoning: skip RSB filling at VMEXIT in

that case.

arch/x86/kernel/cpu/bugs.c +2 -0

+		 */
+		spectre_v2_determine_rsb_fill_type_at_vmexit(mode)

;

arch/x86/kernel/cpu/bugs.c static void __init spectre_v2_select_mitigation(void) · line 1868

Quoted 1 line

arch/x86/kernel/cpu/bugs.c +1 -0

+	}

This seems to be following a pattern of putting excessive amounts of very AMD-specific commenting right in the middle of common code. There's a *HUGE* comment in spectre_v2_determine_rsb_fill_type_at_vmexit().  Wouldn't this be more
at home there?

For shadow paging guests: do not expose this feature to guests; only expose if nested paging is enabled, to ensure context switches within guests trigger TLB flushes on the CPU