x86/alternative: Add ALTERNATIVE_3

From: Borislav Petkov <[email protected]>

Currently, the kernel uses

[LM]FENCE; RDTSC

in the timekeeping code, to guarantee monotonicity of time where the *FENCE is selected based on vendor.

Replace that sequence with RDTSCP which is faster or on-par and gives the same guarantees.

A microbenchmark on Intel shows that the change is on-par.

On AMD, the change is either on-par with the current LFENCE-prefixed RDTSC and some are slightly better with RDTSCP.

The comparison is done with the LFENCE-prefixed RDTSC (and not with the MFENCE-prefixed one, as one would normally expect) because all modern AMD families make LFENCE serializing and thus avoid the heavy MFENCE by effectively enabling X86_FEATURE_LFENCE_RDTSC.

Co-developed-by Thomas Gleixner <[email protected]>

Signed-off-by Thomas Gleixner <[email protected]>

Signed-off-by Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc Andy Lutomirski <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc John Stultz <[email protected]>

Cc [email protected]

Link https://lkml.kernel.org/r/[email protected] open here

---
 arch/x86/include/asm/msr.h | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

arch/x86/include/asm/msr.h +5 -2

index 91e4cf189914..5cc3930cb465 100644
@@ -217,6 +217,8 @@ static __always_inline unsigned long long rdtsc(void)
  */
 static __always_inline unsigned long long rdtsc_ordered(void)
 {
+       DECLARE_ARGS(val, low, high);
+
        /*
         * The RDTSC instruction is not ordered relative to memory
         * access.  The Intel SDM and the AMD APM are both vague on this

@@ -227,9 +229,19 @@ static __always_inline unsigned long long rdtsc_ordered(void)
         * ordering guarantees as reading from a global memory location
         * that some other imaginary CPU is updating continuously with a
         * time stamp.
+        *
+        * Thus, use the preferred barrier on the respective CPU, aiming for
+        * RDTSCP as the default.
         */
-       barrier_nospec();
-       return rdtsc();

Quoted 66 lines · Borislav Petkov

From: Borislav Petkov <[email protected]>

Currently, the kernel uses

[LM]FENCE; RDTSC

in the timekeeping code, to guarantee monotonicity of time where the *FENCE is selected based on vendor.

Replace that sequence with RDTSCP which is faster or on-par and gives the same guarantees.

A microbenchmark on Intel shows that the change is on-par.

On AMD, the change is either on-par with the current LFENCE-prefixed RDTSC and some are slightly better with RDTSCP.

The comparison is done with the LFENCE-prefixed RDTSC (and not with the MFENCE-prefixed one, as one would normally expect) because all modern AMD families make LFENCE serializing and thus avoid the heavy MFENCE by effectively enabling X86_FEATURE_LFENCE_RDTSC.

Co-developed-by Thomas Gleixner <[email protected]>

Signed-off-by Thomas Gleixner <[email protected]>

Signed-off-by Borislav Petkov <[email protected]>

Cc Tom Lendacky <[email protected]>

Cc Andy Lutomirski <[email protected]>

Cc "H. Peter Anvin" <[email protected]>

Cc John Stultz <[email protected]>

Cc [email protected]

Link https://lkml.kernel.org/r/[email protected] open here

---
 arch/x86/include/asm/msr.h | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

arch/x86/include/asm/msr.h +12 -2

index 91e4cf189914..5cc3930cb465 100644
@@ -217,6 +217,8 @@ static __always_inline unsigned long long rdtsc(void)
  */
 static __always_inline unsigned long long rdtsc_ordered(void)
 {
+       DECLARE_ARGS(val, low, high);
+
        /*
         * The RDTSC instruction is not ordered relative to memory
         * access.  The Intel SDM and the AMD APM are both vague on this

@@ -227,9 +229,19 @@ static __always_inline unsigned long long rdtsc_ordered(void)
         * ordering guarantees as reading from a global memory location
         * that some other imaginary CPU is updating continuously with a
         * time stamp.
+        *
+        * Thus, use the preferred barrier on the respective CPU, aiming for
+        * RDTSCP as the default.
         */
-       barrier_nospec();
-       return rdtsc();
+       asm volatile(ALTERNATIVE_3("rdtsc",
+                                  "mfence; rdtsc", X86_FEATURE_MFENCE_RDTSC,
+                                  "lfence; rdtsc", X86_FEATURE_LFENCE_RDTSC,
+                                  "rdtscp", X86_FEATURE_RDTSCP)
+                       : EAX_EDX_RET(val, low, high)
+                       /* RDTSCP clobbers ECX with MSR_TSC_AUX. */
+                       :: "ecx");

arch/x86/include/asm/msr.h static __always_inline unsigned long long rdtsc_ordered(void) · line 245

Quoted 3 lines

arch/x86/include/asm/msr.h +2 -0

+
+       return EAX_EDX_VAL(val, low, high);
 }

This whole series seems reasonable, except that it caused me to look at barrier_nospec(). And I had a bit of a WTF moment, as in "WTF does RDTSC have to do with a speculation protection barrier". Does it actually make sense?

It does seem overloaded in that sense, but the feature means that LFENCE is serializing and so can be used in rdtsc_ordered. In the same sense, barrier_nospec is looking for whether LFENCE is serializing and preferring that over MFENCE since it is lighter weight.

In light of how they're being used now, they could probably stand to be renamed in some way.

Quoted 9 lines · Thomas

It does seem overloaded in that sense, but the feature means that LFENCE is serializing and so can be used in rdtsc_ordered. In the same sense, barrier_nospec is looking for whether LFENCE is serializing and preferring that over MFENCE since it is lighter weight.

In light of how they're being used now, they could probably stand to be renamed in some way.

Actually, come to think of it, what really matters here is whether LFENCE is serializing or not. Because if so, you wanna replace with LFENCE as it is lighter. And in that case a single alternative() - not _2() - should suffice.

BUT(!), that still is not good enough if you do some qemu CPU models like pentium or so which don't even have MFENCE and cause stuff like this:

https://lkml.kernel.org/r/[email protected]

Which means, that you *do* have to alternate between

* no insn at all
* MFENCE
* LFENCE, if it is serializing

so barrier_nospec() does the right thing, AFAICS. And this is why we need an ALTERNATIVE_3() to add RDTSCP into the mix too.

WRT renaming, I guess we can do something like:

* X86_FEATURE_MFENCE_RDTSC -> X86_FEATURE_MFENCE - to mean that CPU has MFENCE support.

and

* X86_FEATURE_LFENCE_RDTSC -> X86_FEATURE_LFENCE_SERIALIZING

Or something to that effect.

Quoted 40 lines · Borislav Petkov

Quoted 9 lines · Thomas

It does seem overloaded in that sense, but the feature means that LFENCE is serializing and so can be used in rdtsc_ordered. In the same sense, barrier_nospec is looking for whether LFENCE is serializing and preferring that over MFENCE since it is lighter weight.

In light of how they're being used now, they could probably stand to be renamed in some way.

Actually, come to think of it, what really matters here is whether LFENCE is serializing or not. Because if so, you wanna replace with LFENCE as it is lighter. And in that case a single alternative() - not _2() - should suffice.

BUT(!), that still is not good enough if you do some qemu CPU models like pentium or so which don't even have MFENCE and cause stuff like this:

https://lkml.kernel.org/r/[email protected]

Which means, that you *do* have to alternate between

* no insn at all
* MFENCE
* LFENCE, if it is serializing

so barrier_nospec() does the right thing, AFAICS. And this is why we need an ALTERNATIVE_3() to add RDTSCP into the mix too.

WRT renaming, I guess we can do something like:

* X86_FEATURE_MFENCE_RDTSC -> X86_FEATURE_MFENCE - to mean that CPU has MFENCE support.

and

* X86_FEATURE_LFENCE_RDTSC -> X86_FEATURE_LFENCE_SERIALIZING

Quoted 1 line

Or something to that effect.

This makes me nervous, since no one knows what “serializing” means. IIRC AMD specifically documents that MFENCE is required before RDTSC to get sensible ordering. So it’s entirely plausible to me that LFENCE is okay for Spectre mitigation but MFENCE is needed for RDTSC on some CPU.

Quoted 40 lines · Andy Lutomirski

Quoted 40 lines · Borislav Petkov

Quoted 9 lines · Thomas

It does seem overloaded in that sense, but the feature means that LFENCE is serializing and so can be used in rdtsc_ordered. In the same sense, barrier_nospec is looking for whether LFENCE is serializing and preferring that over MFENCE since it is lighter weight.

In light of how they're being used now, they could probably stand to be renamed in some way.

Actually, come to think of it, what really matters here is whether LFENCE is serializing or not. Because if so, you wanna replace with LFENCE as it is lighter. And in that case a single alternative() - not _2() - should suffice.

BUT(!), that still is not good enough if you do some qemu CPU models like pentium or so which don't even have MFENCE and cause stuff like this:

https://lkml.kernel.org/r/[email protected]

Which means, that you *do* have to alternate between

* no insn at all
* MFENCE
* LFENCE, if it is serializing

so barrier_nospec() does the right thing, AFAICS. And this is why we need an ALTERNATIVE_3() to add RDTSCP into the mix too.

WRT renaming, I guess we can do something like:

* X86_FEATURE_MFENCE_RDTSC -> X86_FEATURE_MFENCE - to mean that CPU has MFENCE support.

and

* X86_FEATURE_LFENCE_RDTSC -> X86_FEATURE_LFENCE_SERIALIZING

Quoted 7 lines

Quoted 1 line

Or something to that effect.

This makes me nervous, since no one knows what “serializing” means. IIRC AMD specifically documents that MFENCE is required before RDTSC to get sensible ordering. So it’s entirely plausible to me that LFENCE is okay for Spectre mitigation but MFENCE is needed for RDTSC on some CPU.

What we want is IFENCE, an instruction that flushes the complete pipeline. Or alternatively put: holds completion until all prior issued instructions complete.

MFENCE always did that (and a ton more), LFENCE seems to have always done that on Intel, but AMD at some point actually implemented LFENCE as it was specified (only hold completion until all preceding loads are complete) and they (now) have this MSR bit to 'fix' that.

At some point in the past (when all this spectre LFENCE muck was relatively fresh) I suggested we call the thing: instruction_fence() or something like that, maybe we ought to still do that now.

Re RDTSC, waiting for all preceding instructions to complete is 'obviously' sufficient for two RDTSC instructions not to get re-ordered either.

This makes me nervous, since no one knows what “serializing” means.

IIRC AMD specifically documents that MFENCE is required before RDTSC to get sensible ordering. So it’s entirely plausible to me that LFENCE is okay for Spectre mitigation but MFENCE is needed for RDTSC on some CPU.

Quoted 1 line · Andy Lutomirski

This makes me nervous, since no one knows what “serializing” means.

Why no one? If you wanna say that X86_FEATURE_LFENCE_SERIALIZING is not really telling, so is X86_FEATURE_LFENCE_RDTSC, TBH. :)

You're proving my point, I think. CPUID, IRET, MOV to CR, etc are "serializing". LFENCE, on many CPUd and depending on MSRs, is a different kind of serializing. MFENCE is something else. All LOCK instructions are some kind of barrier, but I don't think anyone calls them "serializing".

The uaccess users of barrier_nospec() are presumably looking for a speculation barrier in the sense of "CPU, please don't execute the code after this until you're sure that this code should be executed for real and until all inputs are known, not guessed."

The property I want for RDTSC ordering is much weaker: I want it to be ordered like a load. Imagine that, instead of an on-chip TSC, the TSC is literally a location in main memory that gets incremented by an extra dedicated CPU every nanosecond or so. I want users of RDTSC to work as if they were reading such a location in memory using an ordinary load. I believe this gives the real desired property that it should be impossible to observe the TSC going backwards. This is a much weaker form of serialization.

Quoted 5 lines · Andy Lutomirski

You're proving my point, I think. CPUID, IRET, MOV to CR, etc are "serializing". LFENCE, on many CPUd and depending on MSRs, is a different kind of serializing. MFENCE is something else. All LOCK instructions are some kind of barrier, but I don't think anyone calls them "serializing".

Yeah, peterz and I hashed it out a bit today on IRC about the different meanings of serializing. I see your point now.

Quoted 4 lines

The uaccess users of barrier_nospec() are presumably looking for a speculation barrier in the sense of "CPU, please don't execute the code after this until you're sure that this code should be executed for real and until all inputs are known, not guessed."

Yeah, I believe AMD's paper has this nicely written:

"MITIGATION G-2

Description: Set an MSR in the processor so that LFENCE is a dispatch serializing instruction and then use LFENCE in code streams to serialize dispatch (LFENCE is faster than RDTSCP which is also dispatch serializing). This mode of LFENCE may be enabled by setting MSR C001_1029[1]=1.

Effect: Upon encountering an LFENCE when the MSR bit is set, dispatch will stop until the LFENCE instruction becomes the oldest instruction in the machine."

https://developer.amd.com/wp-content/resources/90343-B_SoftwareTechniquesforManagingSpeculation_WP_7-18Update_FNL.pdf

which is basically what you want for the whole mitigation crap if you want to kill speculation - you simply hold dispatch until the LFENCE retires.

Quoted 8 lines

The property I want for RDTSC ordering is much weaker: I want it to be ordered like a load. Imagine that, instead of an on-chip TSC, the TSC is literally a location in main memory that gets incremented by an extra dedicated CPU every nanosecond or so. I want users of RDTSC to work as if they were reading such a location in memory using an ordinary load. I believe this gives the real desired property that it should be impossible to observe the TSC going backwards. This is a much weaker form of serialization.

Well, in that case you need something new.

Because, the moment you have a RDTSC in flight and a second RDTSC comes in and that second RDTSC must *not* bypass the first one and execute earlier due to OoO, you need to impose some ordering. And that's pretty much uarch-dependent, I'd say.

And I guess on AMD the way to do that is to stop dispatch until the first RDTSC retires.

Can it be done faster? Sure. And I'm pretty sure there's a lot of pesky little hw details we're not even hearing of, which get in the way.

As far as I know, RDTSCP gets the job done, as does LFENCE, RDTSC on Intel.

There was a big discussion a few years ago where we changed it from LFENCE;RDTSC;LFENCE to just LFENCE;RDTSC after everyone was reasonably convinced that the uarch would not dispatch two RDTSCs backwards if the first one was immediately preceeded by LFENCE.

Quoted 2 lines · Andy Lutomirski

As far as I know, RDTSCP gets the job done, as does LFENCE, RDTSC on Intel.

Same on AMD when LFENCE has been made dispatch-serializing.

Quoted 4 lines

There was a big discussion a few years ago where we changed it from LFENCE;RDTSC;LFENCE to just LFENCE;RDTSC after everyone was reasonably convinced that the uarch would not dispatch two RDTSCs backwards if the first one was immediately preceeded by LFENCE.

Yeah, the second one won't pass the LFENCE so you won't see time going backwards, sure.

Also that the uarch doesn't turn:

LFENCE
RDTSC
load some memory

into a load followed a cycle later by RDTSC. If that happened, then some cooperating CPUs could observe time going backwards.

But RDTSC has no dependencies, so, barring odd register renaming effects or similar, this isn't going to be a problem in practice.

Sent: 12 December 2018 18:45

Quoted 8 lines

The property I want for RDTSC ordering is much weaker: I want it to be ordered like a load. Imagine that, instead of an on-chip TSC, the TSC is literally a location in main memory that gets incremented by an extra dedicated CPU every nanosecond or so. I want users of RDTSC to work as if they were reading such a location in memory using an ordinary load. I believe this gives the real desired property that it should be impossible to observe the TSC going backwards. This is a much weaker form of serialization.

Well, in that case you need something new.

Because, the moment you have a RDTSC in flight and a second RDTSC comes in and that second RDTSC must *not* bypass the first one and execute earlier due to OoO, you need to impose some ordering. And that's pretty much uarch-dependent, I'd say.

And I guess on AMD the way to do that is to stop dispatch until the first RDTSC retires.

Can it be done faster? Sure. And I'm pretty sure there's a lot of pesky little hw details we're not even hearing of, which get in the way.

From: Borislav Petkov

Quoted 1 line

Sent: 12 December 2018 18:45

...

Quoted 16 lines

Quoted 8 lines

The property I want for RDTSC ordering is much weaker: I want it to be ordered like a load. Imagine that, instead of an on-chip TSC, the TSC is literally a location in main memory that gets incremented by an extra dedicated CPU every nanosecond or so. I want users of RDTSC to work as if they were reading such a location in memory using an ordinary load. I believe this gives the real desired property that it should be impossible to observe the TSC going backwards. This is a much weaker form of serialization.

Well, in that case you need something new.

Because, the moment you have a RDTSC in flight and a second RDTSC comes in and that second RDTSC must *not* bypass the first one and execute earlier due to OoO, you need to impose some ordering. And that's pretty much uarch-dependent, I'd say.

Quoted 5 lines

And I guess on AMD the way to do that is to stop dispatch until the first RDTSC retires.

Can it be done faster? Sure. And I'm pretty sure there's a lot of pesky little hw details we're not even hearing of, which get in the way.

ISTR one of the problems with RDTSC serialising is that it is used for micro-benchmarks.

Quoted 40 lines · Borislav Petkov

Quoted 9 lines · Thomas

It does seem overloaded in that sense, but the feature means that LFENCE is serializing and so can be used in rdtsc_ordered. In the same sense, barrier_nospec is looking for whether LFENCE is serializing and preferring that over MFENCE since it is lighter weight.

In light of how they're being used now, they could probably stand to be renamed in some way.

Actually, come to think of it, what really matters here is whether LFENCE is serializing or not. Because if so, you wanna replace with LFENCE as it is lighter. And in that case a single alternative() - not _2() - should suffice.

BUT(!), that still is not good enough if you do some qemu CPU models like pentium or so which don't even have MFENCE and cause stuff like this:

https://lkml.kernel.org/r/[email protected]

Which means, that you *do* have to alternate between

* no insn at all
* MFENCE
* LFENCE, if it is serializing

so barrier_nospec() does the right thing, AFAICS. And this is why we need an ALTERNATIVE_3() to add RDTSCP into the mix too.

WRT renaming, I guess we can do something like:

* X86_FEATURE_MFENCE_RDTSC -> X86_FEATURE_MFENCE - to mean that CPU has MFENCE support.

and

* X86_FEATURE_LFENCE_RDTSC -> X86_FEATURE_LFENCE_SERIALIZING

Quoted 1 line

Or something to that effect.

This makes me nervous, since no one knows what “serializing” means. IIRC AMD specifically documents that MFENCE is required before RDTSC to get sensible ordering. So it’s entirely plausible to me that LFENCE is okay for Spectre mitigation but MFENCE is needed for RDTSC on some CPU.

Quoted 40 lines · Andy Lutomirski

Quoted 40 lines · Borislav Petkov

Quoted 9 lines · Thomas

It does seem overloaded in that sense, but the feature means that LFENCE is serializing and so can be used in rdtsc_ordered. In the same sense, barrier_nospec is looking for whether LFENCE is serializing and preferring that over MFENCE since it is lighter weight.

In light of how they're being used now, they could probably stand to be renamed in some way.

Actually, come to think of it, what really matters here is whether LFENCE is serializing or not. Because if so, you wanna replace with LFENCE as it is lighter. And in that case a single alternative() - not _2() - should suffice.

BUT(!), that still is not good enough if you do some qemu CPU models like pentium or so which don't even have MFENCE and cause stuff like this:

https://lkml.kernel.org/r/[email protected]

Which means, that you *do* have to alternate between

* no insn at all
* MFENCE
* LFENCE, if it is serializing

so barrier_nospec() does the right thing, AFAICS. And this is why we need an ALTERNATIVE_3() to add RDTSCP into the mix too.

WRT renaming, I guess we can do something like:

* X86_FEATURE_MFENCE_RDTSC -> X86_FEATURE_MFENCE - to mean that CPU has MFENCE support.

and

* X86_FEATURE_LFENCE_RDTSC -> X86_FEATURE_LFENCE_SERIALIZING

Quoted 7 lines

Quoted 1 line

Or something to that effect.

This makes me nervous, since no one knows what “serializing” means. IIRC AMD specifically documents that MFENCE is required before RDTSC to get sensible ordering. So it’s entirely plausible to me that LFENCE is okay for Spectre mitigation but MFENCE is needed for RDTSC on some CPU.

As long as MSR 0xc0011029[1] is set (MSR_F10H_DECFG_LFENCE_SERIALIZE_BIT), then LFENCE is a proper, lighter weight solution for ordering RDTSC. So we're good there.

Thanks,
Tom

Quoted 1 line