Skip to contentSkip to conversation

x86: nVMX: Add retry loop to advanced RTM debugging subtest

PATCH Applied Jim Mattson 3 participants 7 messages Feb 27 – May 27 KVM

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86: nVMX: Add retry loop to advanced RTM debugging subtest

Linux commit 400816f60c54 ("perf/x86/intel: Implement support for TSX Force Abort") introduced a feature that temporarily disables RTM on Skylake and similar CPUs (06_55H stepping <= 5, 06_4EH, 06_5EH, 06_8EH stepping <= 0BH, and 06_9EH stepping <= 0CH) via the TSX_FORCE_ABORT MSR, so that all four general purpose PMCs can be used by perf. This feature is on by default, but can be disabled by writing 0 to /sys/devices/cpu/allow_tsx_force_abort.

When TSX_FORCE_ABORT.RTM_FORCE_ABORT[bit 0] is set, all RTM transactions will immediately abort, before the xbegin instruction retires.

The test of a single-step #DB delivered in a transactional region, introduced in commit 414bd9d5ebd7 ("x86: nVMX: Basic test of #DB intercept in L1"), does not handle this scenario.

Modify the test to identify an immediate RTM transaction abort and to try up to 30 times before giving up. If the xbegin instruction never retires, report the test as skipped.

Note that when an RTM transaction aborts, the CPU state is rolled back to before the xbegin instruction, but the RIP is modified to point to the fallback code address. Hence, if the transaction aborts before the single-step #DB trap is delivered, the first instruction of the fallback code will retire before the single-step #DB trap is delivered.

Fixes 414bd9d5ebd7 ("x86: nVMX: Basic test of #DB intercept in L1")
---
 lib/x86/msr.h   |  1 +
 x86/vmx_tests.c | 56 ++++++++++++++++++++++++++++++++-----------------
 2 files changed, 38 insertions(+), 19 deletions(-)
lib/x86/msr.h +1 -0
index 7397809c07cd..97f52bb5bb4e 100644
@@ -109,6 +109,7 @@
 #define DEBUGCTLMSR_BTS_OFF_OS		(1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR		(1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI	(1UL << 11)
+#define DEBUGCTLMSR_RTM_DEBUG		(1UL << 15)
 
 #define MSR_LBR_NHM_FROM	0x00000680
 #define MSR_LBR_NHM_TO		0x000006c0
x86/vmx_tests.c +37 -19

90 lines not on this page · raw message

base-commit: 86e53277ac80dabb04f4fa5fa6a6cc7649392bdc
prerequisite-patch-id: 177e49d1b63609e5b421ea64fe7490a4906617a9
prerequisite-patch-id: 7e71aa35841be5d72bf543e4f332554d12e83cc0

Signature · 1 line
2.53.0.473.g4a7958ca14-goog
#1 lore raw
Quoted 62 lines · Jim Mattson

Linux commit 400816f60c54 ("perf/x86/intel: Implement support for TSX Force Abort") introduced a feature that temporarily disables RTM on Skylake and similar CPUs (06_55H stepping <= 5, 06_4EH, 06_5EH, 06_8EH stepping <= 0BH, and 06_9EH stepping <= 0CH) via the TSX_FORCE_ABORT MSR, so that all four general purpose PMCs can be used by perf. This feature is on by default, but can be disabled by writing 0 to /sys/devices/cpu/allow_tsx_force_abort.

When TSX_FORCE_ABORT.RTM_FORCE_ABORT[bit 0] is set, all RTM transactions will immediately abort, before the xbegin instruction retires.

The test of a single-step #DB delivered in a transactional region, introduced in commit 414bd9d5ebd7 ("x86: nVMX: Basic test of #DB intercept in L1"), does not handle this scenario.

Modify the test to identify an immediate RTM transaction abort and to try up to 30 times before giving up. If the xbegin instruction never retires, report the test as skipped.

Note that when an RTM transaction aborts, the CPU state is rolled back to before the xbegin instruction, but the RIP is modified to point to the fallback code address. Hence, if the transaction aborts before the single-step #DB trap is delivered, the first instruction of the fallback code will retire before the single-step #DB trap is delivered.

Fixes 414bd9d5ebd7 ("x86: nVMX: Basic test of #DB intercept in L1")
---
 lib/x86/msr.h   |  1 +
 x86/vmx_tests.c | 56 ++++++++++++++++++++++++++++++++-----------------
 2 files changed, 38 insertions(+), 19 deletions(-)
lib/x86/msr.h +1 -0
index 7397809c07cd..97f52bb5bb4e 100644
@@ -109,6 +109,7 @@
 #define DEBUGCTLMSR_BTS_OFF_OS         (1UL <<  9)
 #define DEBUGCTLMSR_BTS_OFF_USR                (1UL << 10)
 #define DEBUGCTLMSR_FREEZE_LBRS_ON_PMI (1UL << 11)
+#define DEBUGCTLMSR_RTM_DEBUG          (1UL << 15)
 
 #define MSR_LBR_NHM_FROM       0x00000680
 #define MSR_LBR_NHM_TO         0x000006c0
x86/vmx_tests.c +3 -2
index 5ffb80a3d866..2094a0d3ec57 100644
@@ -9217,9 +9217,10 @@ static void vmx_db_test_guest(void)
         * For a hardware generated single-step #DB in a transactional region.
         */
        asm volatile("vmcall;"
-                    ".Lxbegin: xbegin .Lskip_rtm;"
+                    ".Lrtm_begin: xbegin .Lrtm_fallback;"
                     "xend;"
-                    ".Lskip_rtm:");
+                    ".Lrtm_fallback: nop;"
+                    ".Lpost_rtm:");
 }
 
 /*

x86/vmx_tests.c static void single_step_guest(const char *test_name, u64 starting_dr6, · line 9301

Quoted 6 lines
x86/vmx_tests.c +3 -0
@@ -9295,6 +9296,10 @@ static void single_step_guest(const char *test_name, u64 starting_dr6,
  * exception bits are properly accumulated into the exit qualification
  * field.
  */
+
+#define RTM_RETRIES 30
+#define ONE_BILLION 1000000000ul

I think the name would be more descriptive as RTM_DELAY_CYCLES or sth, IIUC this will be in the order of 100s of milliseconds. Do we need to wait that long between retries? If the CPU is in a state where it will always abort RTM, 30 retries will end up taking seconds or 10s of seconds, right?

#2 lore raw
Quoted 6 lines · Yosry Ahmed

x86/vmx_tests.c static void single_step_guest(const char *test_name, u64 starting_dr6, · line 9301

Quoted 2 lines · Jim Mattson
x86/vmx_tests.c +2 -0
+#define RTM_RETRIES 30
+#define ONE_BILLION 1000000000ul

I think the name would be more descriptive as RTM_DELAY_CYCLES or sth,

RTM_RETRY_DELAY?

Quoted 4 lines

IIUC this will be in the order of 100s of milliseconds. Do we need to wait that long between retries? If the CPU is in a state where it will always abort RTM, 30 retries will end up taking seconds or 10s of seconds, right?

I tried reducing the delay by a factor of 10. At 200 retries, I still see a 2% skip rate on a Skylake Xeon E5 @ 2GHz. I'd like to get the skip rate under 1%. But, maybe others don't care as much?

Yes, 30 billion cycles is going to be on the order of 10 seconds.

#3 lore raw
Quoted 9 lines · Jim Mattson
Quoted 6 lines · Yosry Ahmed

x86/vmx_tests.c static void single_step_guest(const char *test_name, u64 starting_dr6, · line 9301

Quoted 2 lines · Jim Mattson
x86/vmx_tests.c +2 -0
+#define RTM_RETRIES 30
+#define ONE_BILLION 1000000000ul

I think the name would be more descriptive as RTM_DELAY_CYCLES or sth,

RTM_RETRY_DELAY?

SG.

Quoted 11 lines
Quoted 5 lines

IIUC this will be in the order of 100s of milliseconds. Do we need to wait that long between retries? If the CPU is in a state where it will always abort RTM, 30 retries will end up taking seconds or 10s of seconds, right?

I tried reducing the delay by a factor of 10. At 200 retries, I still see a 2% skip rate on a Skylake Xeon E5 @ 2GHz. I'd like to get the skip rate under 1%. But, maybe others don't care as much?

Yes, 30 billion cycles is going to be on the order of 10 seconds.

I personally care more about the test time than the fact that it won't test RTM 2% of the time, but my opinion doesn't really matter :P

#4 lore raw
Quoted 6 lines · Yosry Ahmed
Quoted 5 lines · Jim Mattson
Quoted 4 lines

IIUC this will be in the order of 100s of milliseconds. Do we need to wait that long between retries? If the CPU is in a state where it will always abort RTM, 30 retries will end up taking seconds or 10s of seconds, right?

Quoted 8 lines
Quoted 5 lines

I tried reducing the delay by a factor of 10. At 200 retries, I still see a 2% skip rate on a Skylake Xeon E5 @ 2GHz. I'd like to get the skip rate under 1%. But, maybe others don't care as much?

Yes, 30 billion cycles is going to be on the order of 10 seconds.

I personally care more about the test time than the fact that it won't test RTM 2% of the time, but my opinion doesn't really matter :P

I generally care more about runtime too, but isn't 10 seconds only the worst case scenario, and only on these fubar CPUs? E.g. if there's no perf activity in the host, or the CPU isn't one of these oddballs, isn't XBEGIN going to succeed ~100% of the time?

If this were a choice between "eat N seconds every time" and "skip the test", I'd be a-ok with a skip rate of 50% if it meant reducing N. But, assuming this requires perf activity and a Skylake-era CPU, odds are good this will only be hit in CI environments, at which point adding ~10 seconds to the worst case scenario isn't a bad tradeoff (so long as it doesn't push the total runtime close to the timeout).

#5 lore raw
Quoted 22 lines · Sean Christopherson
Quoted 6 lines · Yosry Ahmed
Quoted 5 lines · Jim Mattson
Quoted 4 lines

IIUC this will be in the order of 100s of milliseconds. Do we need to wait that long between retries? If the CPU is in a state where it will always abort RTM, 30 retries will end up taking seconds or 10s of seconds, right?

Quoted 8 lines
Quoted 5 lines

I tried reducing the delay by a factor of 10. At 200 retries, I still see a 2% skip rate on a Skylake Xeon E5 @ 2GHz. I'd like to get the skip rate under 1%. But, maybe others don't care as much?

Yes, 30 billion cycles is going to be on the order of 10 seconds.

I personally care more about the test time than the fact that it won't test RTM 2% of the time, but my opinion doesn't really matter :P

I generally care more about runtime too, but isn't 10 seconds only the worst case scenario, and only on these fubar CPUs? E.g. if there's no perf activity in the host, or the CPU isn't one of these oddballs, isn't XBEGIN going to succeed ~100% of the time?

Quoted 6 lines

If this were a choice between "eat N seconds every time" and "skip the test", I'd be a-ok with a skip rate of 50% if it meant reducing N. But, assuming this requires perf activity and a Skylake-era CPU, odds are good this will only be hit in CI environments, at which point adding ~10 seconds to the worst case scenario isn't a bad tradeoff (so long as it doesn't push the total runtime close to the timeout).

Good point. We can always revisit if we get really unlucky :)

reviewer Applied
#6 lore raw
In reply to Jim Mattson
Quoted 8 lines · Jim Mattson

Linux commit 400816f60c54 ("perf/x86/intel: Implement support for TSX Force Abort") introduced a feature that temporarily disables RTM on Skylake and similar CPUs (06_55H stepping <= 5, 06_4EH, 06_5EH, 06_8EH stepping <= 0BH, and 06_9EH stepping <= 0CH) via the TSX_FORCE_ABORT MSR, so that all four general purpose PMCs can be used by perf. This feature is on by default, but can be disabled by writing 0 to /sys/devices/cpu/allow_tsx_force_abort.

[...]

Applied to kvm-x86 next, thanks!

[1/1] x86: nVMX: Add retry loop to advanced RTM debugging subtest

      https://github.com/kvm-x86/kvm-unit-tests/commit/dbcb11d7f580

--
https://github.com/kvm-x86/kvm-unit-tests/tree/next