Skip to contentSkip to conversation

perf/amd/ibs: Assorted fixes

PATCH In review Ravi Bangoria 3 participants 14 messages Oct 15 – Oct 26

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover perf/amd/ibs: Assorted fixes

#1 Set IBS MSRs in syscore->resume() callback
#2 Fix kernel address leaking through REGS_USER
#3 Fix races by reducing SW states when IBS_CAPS_DIS is present

Ravi Bangoria (3):
perf/amd/ibs: Reset CPU0 IBS state on syscore resume perf/amd/ibs: Fix kernel address leak via REGS_USER perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

 arch/x86/events/amd/ibs.c | 156 +++++++++++++++++++++++++++++++-------
 1 file changed, 129 insertions(+), 27 deletions(-)
Signature · 1 line
2.55.0
#1 lore raw

PATCH 1/3 perf/amd/ibs: Reset CPU0 IBS state on syscore resume

During a suspend/resume cycle, all CPUs except CPU0 go through hotplug callbacks, which sets up IBS state. CPU0 bypasses this path, leaving its IBS state stale after resume. Fix this by explicitly setting CPU0's IBS state in the syscore ->resume() callback.

Fixes efa5700ec0da ("perf/amd/ibs: Support IBS_{FETCH|OP}_CTL2[Dis] to eliminate RMW race")
---
 arch/x86/events/amd/ibs.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/events/amd/ibs.c +1 -1
index 3531f9c23b8c..bfb51f8e184f 100644
@@ -1948,7 +1948,7 @@ static int perf_ibs_suspend(void *data)
 static void perf_ibs_resume(void *data)
 {
 	ibs_eilvt_setup();
-	setup_APIC_ibs();
+	x86_pmu_amd_ibs_starting_cpu(smp_processor_id());
 }
 
 static const struct syscore_ops perf_ibs_syscore_ops = {
Signature · 1 line
2.55.0
#2 lore raw

Re: [PATCH 1/3] perf/amd/ibs: Reset CPU0 IBS state on syscore resume

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

During a suspend/resume cycle, all CPUs except CPU0 go through hotplug callbacks, which sets up IBS state. CPU0 bypasses this path, leaving its IBS state stale after resume. Fix this by explicitly setting CPU0's IBS state in the syscore ->resume() callback.

Fixes efa5700ec0da ("perf/amd/ibs: Support IBS_{FETCH|OP}_CTL2[Dis] to eliminate RMW race")

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
Reviewed-by on v1 1/3
#3 lore raw

Re: [PATCH 1/3] perf/amd/ibs: Reset CPU0 IBS state on syscore resume

In reply to Ravi Bangoria
Quoted 8 lines · Ravi Bangoria

During a suspend/resume cycle, all CPUs except CPU0 go through hotplug callbacks, which sets up IBS state. CPU0 bypasses this path, leaving its IBS state stale after resume. Fix this by explicitly setting CPU0's IBS state in the syscore ->resume() callback.

Fixes efa5700ec0da ("perf/amd/ibs: Support IBS_{FETCH|OP}_CTL2[Dis] to eliminate RMW race")
#4 lore raw

PATCH 2/3 perf/amd/ibs: Fix kernel address leak via REGS_USER

In reply to Ravi Bangoria

IBS hardware captures a sample and raises an NMI. Since the NMI can be delivered with a delay, a privilege change may occur between sample capture and NMI delivery. Because IBS only provides the sample RIP, the kernel records GPRs from the NMI pt_regs, which may reflect kernel mode if a privilege switch occurred in the interim. This can leak kernel addresses to unprivileged users through PERF_SAMPLE_REGS_USER.

Fix this by discarding samples where the NMI arrives in kernel mode and the user has set exclude_kernel.

Fixes d29e744c7167 ("perf/x86: Relax privilege filter restriction on AMD IBS")
---
 arch/x86/events/amd/ibs.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/events/amd/ibs.c +6 -3
index bfb51f8e184f..3cb46c775eaa 100644
@@ -1373,12 +1373,15 @@ static bool perf_ibs_is_kernel_br_target(struct perf_event *event,
 }
 
 static bool perf_ibs_discard_sample(struct perf_ibs *perf_ibs, struct perf_event *event,
-				    struct pt_regs *regs, struct perf_ibs_data *ibs_data,
-				    int br_target_idx)
+				    struct pt_regs *regs, struct pt_regs *iregs,
+				    struct perf_ibs_data *ibs_data, int br_target_idx)
 {
 	if (perf_exclude_event(event, regs))
 		return true;
 
+	if (perf_exclude_event(event, iregs))
+		return true;
+
 	if (perf_ibs != &perf_ibs_op || !event->attr.exclude_kernel)
 		return false;
 
@@ -1529,7 +1532,7 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
 
 	if (((ibs_caps & IBS_CAPS_BIT63_FILTER) ||
 	     (event->attr.config2 & IBS_SW_FILTER_MASK)) &&
-	    perf_ibs_discard_sample(perf_ibs, event, &regs, &ibs_data, br_target_idx)) {
+	    perf_ibs_discard_sample(perf_ibs, event, &regs, iregs, &ibs_data, br_target_idx)) {
 		throttle = perf_event_account_interrupt(event);
 		goto out;
 	}
Signature · 1 line
2.55.0
#5 lore raw

Re: [PATCH 2/3] perf/amd/ibs: Fix kernel address leak via REGS_USER

Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 7 lines

IBS hardware captures a sample and raises an NMI. Since the NMI can be delivered with a delay, a privilege change may occur between sample capture and NMI delivery. Because IBS only provides the sample RIP, the kernel records GPRs from the NMI pt_regs, which may reflect kernel mode if a privilege switch occurred in the interim. This can leak kernel addresses to unprivileged users through PERF_SAMPLE_REGS_USER.

Quoted 6 lines

Fix this by discarding samples where the NMI arrives in kernel mode and the user has set exclude_kernel.

Fixes d29e744c7167 ("perf/x86: Relax privilege filter restriction on AMD IBS")

Sashiko has reviewed this patch and found no issues. It looks great!

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
#6 lore raw

Re: [PATCH 2/3] perf/amd/ibs: Fix kernel address leak via REGS_USER

In reply to Ravi Bangoria
Quoted 10 lines · Ravi Bangoria

IBS hardware captures a sample and raises an NMI. Since the NMI can be delivered with a delay, a privilege change may occur between sample capture and NMI delivery. Because IBS only provides the sample RIP, the kernel records GPRs from the NMI pt_regs, which may reflect kernel mode if a privilege switch occurred in the interim. This can leak kernel addresses to unprivileged users through PERF_SAMPLE_REGS_USER.

Fix this by discarding samples where the NMI arrives in kernel mode and the user has set exclude_kernel.

This will systematically under-sample instructions just before a transition to the kernel. For this special case, perhaps the GPRs could be obtained from task_pt_regs(current) instead of the NMI pt_regs?

Quoted 15 lines
Fixes d29e744c7167 ("perf/x86: Relax privilege filter restriction on AMD IBS")
---
 arch/x86/events/amd/ibs.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/events/amd/ibs.c +0 -0
index bfb51f8e184f..3cb46c775eaa 100644
@@ -1373,12 +1373,15 @@ static bool perf_ibs_is_kernel_br_target(struct perf_event *event,
 }
 
 static bool perf_ibs_discard_sample(struct perf_ibs *perf_ibs, struct perf_event *event,

arch/x86/events/amd/ibs.c static bool perf_ibs_is_kernel_br_target(struct perf_event *event, · line 1383

Quoted 10 lines
arch/x86/events/amd/ibs.c +4 -2
-                                   struct pt_regs *regs, struct perf_ibs_data *ibs_data,
-                                   int br_target_idx)
+                                   struct pt_regs *regs, struct pt_regs *iregs,
+                                   struct perf_ibs_data *ibs_data, int br_target_idx)
 {
        if (perf_exclude_event(event, regs))
                return true;
 
+       if (perf_exclude_event(event, iregs))
+               return true;

This excludes events with an NMI skid past a privilege change in both directions. Do you want to omit a kernel sample taken just before returning to userspace if the NMI arrives in userspace and exclude_user is set?

#7 lore raw

Re: [PATCH 2/3] perf/amd/ibs: Fix kernel address leak via REGS_USER

Jim,

Apologies, it took me some time to get back to this.

Quoted 7 lines
Quoted 7 lines

IBS hardware captures a sample and raises an NMI. Since the NMI can be delivered with a delay, a privilege change may occur between sample capture and NMI delivery. Because IBS only provides the sample RIP, the kernel records GPRs from the NMI pt_regs, which may reflect kernel mode if a privilege switch occurred in the interim. This can leak kernel addresses to unprivileged users through PERF_SAMPLE_REGS_USER.

Quoted 7 lines
Quoted 2 lines

Fix this by discarding samples where the NMI arrives in kernel mode and the user has set exclude_kernel.

This will systematically under-sample instructions just before a transition to the kernel. For this special case, perhaps the GPRs could be obtained from task_pt_regs(current) instead of the NMI pt_regs?

Yes, that's a better approach.

Quoted 23 lines
Quoted 23 lines
Fixes d29e744c7167 ("perf/x86: Relax privilege filter restriction on AMD IBS")
---
 arch/x86/events/amd/ibs.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)
arch/x86/events/amd/ibs.c +2 -2
index bfb51f8e184f..3cb46c775eaa 100644
@@ -1373,12 +1373,15 @@ static bool perf_ibs_is_kernel_br_target(struct perf_event *event,
 }
 
 static bool perf_ibs_discard_sample(struct perf_ibs *perf_ibs, struct perf_event *event,
-                                   struct pt_regs *regs, struct perf_ibs_data *ibs_data,
-                                   int br_target_idx)
+                                   struct pt_regs *regs, struct pt_regs *iregs,
+                                   struct perf_ibs_data *ibs_data, int br_target_idx)
 {
        if (perf_exclude_event(event, regs))
                return true;
 
Quoted 7 lines

arch/x86/events/amd/ibs.c static bool perf_ibs_is_kernel_br_target(struct perf_event *event, · line 1383

Quoted 2 lines
arch/x86/events/amd/ibs.c +2 -0
+       if (perf_exclude_event(event, iregs))
+               return true;

This excludes events with an NMI skid past a privilege change in both directions. Do you want to omit a kernel sample taken just before returning to userspace if the NMI arrives in userspace and exclude_user is set?

Yes, its a valid concern.

Will post a v2 with these addressed.

Thanks,
Ravi

#8 lore raw

PATCH 3/3 perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

In reply to Ravi Bangoria

The IBS_{FETCH|OP}_CTL2[Dis] bit allows IBS sampling to be stopped without a RMW of IBS_{FETCH,OP}_CTL. Systems providing this bit therefore do not require the IBS_STARTED and IBS_STOPPED states used to coordinate control MSR updates with late NMIs.

Add a separate stop path for these systems. Preserve Val while disabling the event so that a delayed NMI can be identified and consumed, and wait briefly for such an NMI before restarting the event.

Leave the existing handling unchanged on systems without IBS_CAPS_DIS.

Fixes efa5700ec0da ("perf/amd/ibs: Support IBS_{FETCH|OP}_CTL2[Dis] to eliminate RMW race")
---
 arch/x86/events/amd/ibs.c | 145 ++++++++++++++++++++++++++++++++------
 1 file changed, 122 insertions(+), 23 deletions(-)
arch/x86/events/amd/ibs.c +122 -23
index 3cb46c775eaa..f49f30f53399 100644
@@ -28,6 +28,7 @@ static u32 ibs_caps;
 #include <linux/hardirq.h>
 
 #include <asm/nmi.h>
+#include <asm/delay.h>
 #include <asm/amd/ibs.h>
 
 /* attr.config2 */
@@ -73,7 +74,12 @@ static u32 ibs_caps;
  *
  * XXX: we could probably be using !atomic bitops for all this.
  */
-
+/*
+ * NOTE: IBS_CAPS_DIS eliminates the RMW race on the IBS_{FETCH|OP}_CTL, making
+ * IBS_STARTED and IBS_STOPPED states unnecessary. We use only IBS_ENABLED and
+ * IBS_STOPPING states when IBS_CAPS_DIS is present. Semantics of IBS_STOPPING
+ * changes a bit as well.
+ */
 enum ibs_states {
 	IBS_ENABLED	= 0,
 	IBS_STARTED	= 1,
@@ -557,7 +563,9 @@ static void perf_ibs_start(struct perf_event *event, int flags)
 	struct hw_perf_event *hwc = &event->hw;
 	struct perf_ibs *perf_ibs = container_of(event->pmu, struct perf_ibs, pmu);
 	struct cpu_perf_ibs *pcpu = this_cpu_ptr(perf_ibs->pcpu);
+	unsigned int delay = 50;
 	u64 period, config = 0;
+	u64 ctl;
 
 	if (WARN_ON_ONCE(!(hwc->state & PERF_HES_STOPPED)))
 		return;
@@ -575,23 +583,87 @@ static void perf_ibs_start(struct perf_event *event, int flags)
 	}
 	config |= period >> 4;
 
-	/*
-	 * Reset the IBS_{FETCH|OP}_CTL MSR before updating pcpu->state.
-	 * Doing so prevents a race condition in which an NMI due to other
-	 * source might accidentally activate the event before we enable
-	 * it ourselves.
-	 */
-	perf_ibs_disable_event(perf_ibs, hwc, 0);
+	if (ibs_caps & IBS_CAPS_DIS) {
+		/*
+		 *      IBS HW                     SW
+		 *
+		 *  o Capture sample       o ->stop()
+		 *  o Set VAL bit            CTL2[Dis] = 1
+		 *  o Raise NMI --.        o ...
+		 *                 \       o ->start()
+		 *                  \        wrmsr(IBS_{FETCH|OP}_CTL, new value);
+		 *                   \       (wrmsr clears the VAL bit)
+		 *                    `--> o Receives delayed NMI
+		 *                           (Unknown NMI since VAL bit is cleared)
+		 *
+		 * Handle this with induced delay.
+		 */
+		rdmsrq(hwc->config_base, ctl);
+		while ((ctl & perf_ibs->valid_mask) && delay--) {
+			udelay(1);
+			rdmsrq(hwc->config_base, ctl);
+		}
+	} else {
+		/*
+		 * Reset the IBS_{FETCH|OP}_CTL MSR before updating pcpu->state.
+		 * Doing so prevents a race condition in which an NMI due to
+		 * other source might accidentally activate the event before we
+		 * enable it ourselves.
+		 */
+		perf_ibs_disable_event(perf_ibs, hwc, 0);
+		/*
+		 * Set STARTED before enabling the hardware, such that a
+		 * subsequent NMI must observe it.
+		 */
+		set_bit(IBS_STARTED,    pcpu->state);
+		clear_bit(IBS_STOPPING, pcpu->state);
+	}
+	perf_ibs_enable_event(perf_ibs, hwc, config);
+
+	perf_event_update_userpage(event);
+}
+
+/* ->stop() for platforms with additional CTL2[Dis] bit. */
+static void perf_ibs_stop_dis(struct perf_event *event, int flags)
+{
+	struct hw_perf_event *hwc = &event->hw;
+	struct perf_ibs *perf_ibs = container_of(event->pmu, struct perf_ibs, pmu);
+	struct cpu_perf_ibs *pcpu = this_cpu_ptr(perf_ibs->pcpu);
+	u64 config;
+	u64 ctl2;
+
+	WARN_ON_ONCE(test_bit(IBS_STOPPING, pcpu->state));
+
+	set_bit(IBS_STOPPING, pcpu->state);
+
+	rdmsrq(hwc->extra_reg.reg, ctl2);
+	if (ctl2 & perf_ibs->disable_mask)
+		goto reset_state;
+
+	wrmsrq(hwc->extra_reg.reg, perf_ibs->disable_mask);
+
+	WARN_ON_ONCE(hwc->state & PERF_HES_STOPPED);
+	hwc->state |= PERF_HES_STOPPED;
+
+	if (hwc->state & PERF_HES_UPTODATE)
+		goto reset_state;
+
+	rdmsrq(hwc->config_base, config);
+
+	config &= ~perf_ibs->valid_mask;
 
 	/*
-	 * Set STARTED before enabling the hardware, such that a subsequent NMI
-	 * must observe it.
+	 * Clear the Val bit otherwise ->start() will unnecessarily
+	 * wait for 50us.
 	 */
-	set_bit(IBS_STARTED,    pcpu->state);
-	clear_bit(IBS_STOPPING, pcpu->state);
-	perf_ibs_enable_event(perf_ibs, hwc, config);
+	if (in_nmi())
+		wrmsrq(hwc->config_base, config);
 
-	perf_event_update_userpage(event);
+	perf_ibs_event_update(perf_ibs, event, &config);
+	hwc->state |= PERF_HES_UPTODATE;
+
+reset_state:
+	clear_bit(IBS_STOPPING, pcpu->state);
 }
 
 static void perf_ibs_stop(struct perf_event *event, int flags)
@@ -602,6 +674,9 @@ static void perf_ibs_stop(struct perf_event *event, int flags)
 	u64 config;
 	int stopping;
 
+	if (ibs_caps & IBS_CAPS_DIS)
+		return perf_ibs_stop_dis(event, flags);
+
 	if (test_and_set_bit(IBS_STOPPING, pcpu->state))
 		return;
 
@@ -1419,19 +1494,43 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)
 	unsigned int msr;
 	u64 *buf, *config, period, new_config = 0;
 	int br_target_idx = -1;
+	u64 ctl2, ctl;
 
-	if (!test_bit(IBS_STARTED, pcpu->state)) {
-fail:
+	if (ibs_caps & IBS_CAPS_DIS) {
 		/*
-		 * Catch spurious interrupts after stopping IBS: After
-		 * disabling IBS there could be still incoming NMIs
-		 * with samples that even have the valid bit cleared.
-		 * Mark all this NMIs as handled.
+		 * ->stop() {
+		 *     set_bit(IBS_STOPPING);
+		 *                               <-- NMI (race 1)
+		 *     CTL2[Dis]=1;
+		 *     ...                       <-- NMI (race 2)
+		 * }
 		 */
-		if (test_and_clear_bit(IBS_STOPPED, pcpu->state))
-			return 1;
+		rdmsrq(perf_ibs->msr2, ctl2);
+		if (test_bit(IBS_STOPPING, pcpu->state) /* race 1 */ ||
+		    (ctl2 & perf_ibs->disable_mask) /* race 2 */) {
+			rdmsrq(perf_ibs->msr, ctl);
+			if (ctl & perf_ibs->valid_mask) {
+				ctl &= ~(perf_ibs->enable_mask |
+					 perf_ibs->valid_mask);
+				wrmsrq(perf_ibs->msr, ctl);
+				return 1;
+			}
+			return 0;
+		}
+	} else {
+		if (!test_bit(IBS_STARTED, pcpu->state)) {
+fail:
+			/*
+			 * Catch spurious interrupts after stopping IBS: After
+			 * disabling IBS there could be still incoming NMIs
+			 * with samples that even have the valid bit cleared.
+			 * Mark all this NMIs as handled.
+			 */
+			if (test_and_clear_bit(IBS_STOPPED, pcpu->state))
+				return 1;
 
-		return 0;
+			return 0;
+		}
 	}
 
 	if (WARN_ON_ONCE(!event))
Signature · 1 line
2.55.0
#9 lore raw

Re: [PATCH 3/3] perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider: - [High] perf/amd/ibs: Unhandled NMI panic due to 50us timeout --

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] perf/amd/ibs: Unhandled NMI panic due to 50us timeout
--

commit be4cf40ba3750348ffc77933b0cfe40c48e335e7
Author: Ravi Bangoria <[email protected]>
Subject: perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

This commit simplifies event state handling on systems supporting IBS_CAPS_DIS. It introduces a separate stop path that preserves the Val bit while disabling the event to catch delayed NMIs.

Quoted 3 lines
arch/x86/events/amd/ibs.c +0 -0

[ ... ]

Quoted 1 line
arch/x86/events/amd/ibs.c +0 -0
@@ -575,23 +583,87 @@ static void perf_ibs_start(struct perf_event *event, int flags)

[ ... ]

Quoted 10 lines
arch/x86/events/amd/ibs.c +10 -0
+	if (ibs_caps & IBS_CAPS_DIS) {
+		/*
+		 *      IBS HW                     SW
+		 *
+		 *  o Capture sample       o ->stop()
+		 *  o Set VAL bit            CTL2[Dis] = 1
+		 *  o Raise NMI --.        o ...
+		 *                 \       o ->start()
+		 *                  \        wrmsr(IBS_{FETCH|OP}_CTL, new value);
+		 *                   \       (wrmsr clears the VAL bit)

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 605

Quoted 10 lines
arch/x86/events/amd/ibs.c +10 -0
+		 *                    `--> o Receives delayed NMI
+		 *                           (Unknown NMI since VAL bit is cleared)
+		 *
+		 * Handle this with induced delay.
+		 */
+		rdmsrq(hwc->config_base, ctl);
+		while ((ctl & perf_ibs->valid_mask) && delay--) {
+			udelay(1);
+			rdmsrq(hwc->config_base, ctl);
+		}

[Severity: High]
What happens if the delayed NMI takes longer than 50us to arrive, for example due to hypervisor scheduling, nested virtualization, or SMIs?

If this delay loop expires, the code proceeds to call perf_ibs_enable_event() which clears the VAL and CTL2[Dis] bits. When the delayed NMI eventually arrives, it appears it will be treated as an unhandled NMI.

[ ... ]

Quoted 1 line
arch/x86/events/amd/ibs.c +0 -0
@@ -1419,19 +1494,43 @@ static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)

[ ... ]

Quoted 6 lines
arch/x86/events/amd/ibs.c +6 -0
+	} else {
+		if (!test_bit(IBS_STARTED, pcpu->state)) {
+fail:
+			/*
+			 * Catch spurious interrupts after stopping IBS: After
+			 * disabling IBS there could be still incoming NMIs

arch/x86/events/amd/ibs.c static int perf_ibs_handle_irq(struct perf_ibs *perf_ibs, struct pt_regs *iregs)

Quoted 10 lines
arch/x86/events/amd/ibs.c +8 -1
+			 * with samples that even have the valid bit cleared.
+			 * Mark all this NMIs as handled.
+			 */
+			if (test_and_clear_bit(IBS_STOPPED, pcpu->state))
+				return 1;
 
-		return 0;
+			return 0;
+		}
+	}

[Severity: High]
If the NMI is delayed past the 50us timeout in perf_ibs_start(), won't perf_ibs_handle_irq() return 0 here?

Since IBS_STOPPED is intentionally not set for systems with IBS_CAPS_DIS, when the delayed NMI arrives, CTL2[Dis] == 0 and VAL == 0. If the code jumps to the 'fail' label due to the cleared VAL bit, IBS_STOPPED will be clear, causing it to return 0.

Could this result in an unhandled NMI and a subsequent kernel panic?

Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=3
#10 lore raw

Re: [PATCH 3/3] perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

Quoted 21 lines
Quoted 10 lines
arch/x86/events/amd/ibs.c +10 -0
+     if (ibs_caps & IBS_CAPS_DIS) {
+             /*
+              *      IBS HW                     SW
+              *
+              *  o Capture sample       o ->stop()
+              *  o Set VAL bit            CTL2[Dis] = 1
+              *  o Raise NMI --.        o ...
+              *                 \       o ->start()
+              *                  \        wrmsr(IBS_{FETCH|OP}_CTL, new value);
+              *                   \       (wrmsr clears the VAL bit)

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 605

Quoted 10 lines
arch/x86/events/amd/ibs.c +10 -0
+              *                    `--> o Receives delayed NMI
+              *                           (Unknown NMI since VAL bit is cleared)
+              *
+              * Handle this with induced delay.
+              */
+             rdmsrq(hwc->config_base, ctl);
+             while ((ctl & perf_ibs->valid_mask) && delay--) {
+                     udelay(1);
+                     rdmsrq(hwc->config_base, ctl);
+             }
Quoted 8 lines

[Severity: High]
What happens if the delayed NMI takes longer than 50us to arrive, for example due to hypervisor scheduling, nested virtualization, or SMIs?

If this delay loop expires, the code proceeds to call perf_ibs_enable_event() which clears the VAL and CTL2[Dis] bits. When the delayed NMI eventually arrives, it appears it will be treated as an unhandled NMI.

A delay exceeding 50us is highly unlikely in practice, and even if it occurs, the consequence is "Unknown NMI", which is not fatal.

Thanks,
Ravi

#11 lore raw

Re: [PATCH 3/3] perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

In reply to Ravi Bangoria
Quoted 5 lines · Ravi Bangoria

The IBS_{FETCH|OP}_CTL2[Dis] bit allows IBS sampling to be stopped without a RMW of IBS_{FETCH,OP}_CTL. Systems providing this bit therefore do not require the IBS_STARTED and IBS_STOPPED states used to coordinate control MSR updates with late NMIs.

CTL2[Dis] doesn't make late NMIs go away. Hence, this patch needs a 50us poll. With the two-line fix in my original patch, the existing scheme handles late NMIs indefinitely: IBS_STOPPED remains set until a late NMI consumes it, even if the event has been restarted in the meantime. This patch trades that for a 50us window, after which a late NMI is unknown (which *is* fatal with unknown_nmi_panic). Moreover, since we still have to handle hardware without IBS_CAPS_DIS, we now have two state machines instead of one.

Quoted 13 lines

Add a separate stop path for these systems. Preserve Val while disabling the event so that a delayed NMI can be identified and consumed, and wait briefly for such an NMI before restarting the event.

Leave the existing handling unchanged on systems without IBS_CAPS_DIS.

Fixes efa5700ec0da ("perf/amd/ibs: Support IBS_{FETCH|OP}_CTL2[Dis] to eliminate RMW race")
---
 arch/x86/events/amd/ibs.c | 145 ++++++++++++++++++++++++++++++++------
 1 file changed, 122 insertions(+), 23 deletions(-)

[ ... ]

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 637

Quoted 12 lines
arch/x86/events/amd/ibs.c +12 -0
+/* ->stop() for platforms with additional CTL2[Dis] bit. */
+static void perf_ibs_stop_dis(struct perf_event *event, int flags)
+{
+       struct hw_perf_event *hwc = &event->hw;
+       struct perf_ibs *perf_ibs = container_of(event->pmu, struct perf_ibs, pmu);
+       struct cpu_perf_ibs *pcpu = this_cpu_ptr(perf_ibs->pcpu);
+       u64 config;
+       u64 ctl2;
+
+       WARN_ON_ONCE(test_bit(IBS_STOPPING, pcpu->state));
+
+       set_bit(IBS_STOPPING, pcpu->state);

This function needs to be reentrant. If an IBS event is a sibling in a group led by a core PMU event, and the leader is throttled, perf_event_throttle_group() will stop the IBS sibling from the core PMU's NMI handler:

    amd_pmu_v2_handle_irq() ->
      perf_event_overflow() ->
        __perf_event_overflow() ->
          __perf_event_account_interrupt() ->
            perf_event_throttle_group() ->
              perf_event_throttle() ->
                perf_ibs_stop() ->
                  perf_ibs_stop_dis()

[ .. ]

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 660

Quoted 2 lines
arch/x86/events/amd/ibs.c +2 -0
+       if (in_nmi())
+               wrmsrq(hwc->config_base, config);

As we saw above, in_nmi() doesn't mean that we're in perf_ibs_handle_irq() for this sample. If IBS Val=1 and the IBS NMI is still pending, this clears Val. When the IBS NMI arrives, perf_ibs_handle_irq() finds CTL2[Dis]=1 and Val=0, and therefore returns 0, leading to an unknown NMI.

#12 lore raw

Re: [PATCH 3/3] perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

Jim,

Thanks for the detailed review.

Quoted 5 lines
Quoted 4 lines

The IBS_{FETCH|OP}_CTL2[Dis] bit allows IBS sampling to be stopped without a RMW of IBS_{FETCH,OP}_CTL. Systems providing this bit therefore do not require the IBS_STARTED and IBS_STOPPED states used to coordinate control MSR updates with late NMIs.

Quoted 8 lines

CTL2[Dis] doesn't make late NMIs go away. Hence, this patch needs a 50us poll. With the two-line fix in my original patch, the existing scheme handles late NMIs indefinitely: IBS_STOPPED remains set until a late NMI consumes it, even if the event has been restarted in the meantime. This patch trades that for a 50us window, after which a late NMI is unknown (which *is* fatal with unknown_nmi_panic). Moreover, since we still have to handle hardware without IBS_CAPS_DIS, we now have two state machines instead of one.

IBS_STOPPED only records that "a late NMI might exist". It can't tell whether the disable actually completed or whether that late NMI has already arrived. Trying to infer from software state is fundamentally broken, since the WRMSR and a software state transition can't be made atomic.

Given that Zen6 provides a clear hardware state machine, I believe using it, rather than a software state machine, is the right approach.

I ran a fuzzer for several hours with my patch (+Manali's patch [1]) and saw no Unknown NMIs on Zen6.

However, With your patch[2] (+Manali's patch), the same fuzzer continued to produce quite a few Unknown NMIs on Zen6. One race I was able to identify (with your patch applied) is described below.

---
o FETCH and OP both are enabled.

  perf_ibs_stop() /* For OP */
  {
       set IBS_STOPPED
       op state: ENABLED | STARTED |
                 STOPPING | STOPPED

OpVal is 0 in hardware

NMI due to FETCH
{
FETCH: Val=1, sample handled, fetch re-armed

OP: STARTED is set, OpVal is 0 so test_and_clear(IBS_STOPPED) clears the STOPPED state

OP continues to run, it overflows, sets OpVal=1 and a new NMI is latched by the OP hardware

return handled=2
}

op state: ENABLED | STARTED |
STOPPING

WRMSR OpCtl2.Dis=1
leaves the valid sample in OpCtl

clear IBS_STARTED

op state: ENABLED | STOPPING

NMI (late) due to OP {
FETCH: Val=0 -> return 0

OP: STARTED is clear and STOPPED is already clear.

return 0 without reading IbsOpCtl

return handled=0

Unknown NMI
}
---

[1]: https://lore.kernel.org/r/[email protected]
[2]: https://lore.kernel.org/r/[email protected]

[...]

Quoted 18 lines

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 637

Quoted 12 lines
arch/x86/events/amd/ibs.c +12 -0
+/* ->stop() for platforms with additional CTL2[Dis] bit. */
+static void perf_ibs_stop_dis(struct perf_event *event, int flags)
+{
+       struct hw_perf_event *hwc = &event->hw;
+       struct perf_ibs *perf_ibs = container_of(event->pmu, struct perf_ibs, pmu);
+       struct cpu_perf_ibs *pcpu = this_cpu_ptr(perf_ibs->pcpu);
+       u64 config;
+       u64 ctl2;
+
+       WARN_ON_ONCE(test_bit(IBS_STOPPING, pcpu->state));
+
+       set_bit(IBS_STOPPING, pcpu->state);

This function needs to be reentrant. If an IBS event is a sibling in a group led by a core PMU event, and the leader is throttled, perf_event_throttle_group() will stop the IBS sibling from the core PMU's NMI handler:

Quoted 8 lines
    amd_pmu_v2_handle_irq() ->
      perf_event_overflow() ->
        __perf_event_overflow() ->
          __perf_event_account_interrupt() ->
            perf_event_throttle_group() ->
              perf_event_throttle() ->
                perf_ibs_stop() ->
                  perf_ibs_stop_dis()

Core and IBS, both are of hardware PMU type and thus can not be grouped together.

[...]

Quoted 8 lines

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 660

Quoted 2 lines
arch/x86/events/amd/ibs.c +2 -0
+       if (in_nmi())
+               wrmsrq(hwc->config_base, config);

As we saw above, in_nmi() doesn't mean that we're in perf_ibs_handle_irq() for this sample. If IBS Val=1 and the IBS NMI is still pending, this clears Val. When the IBS NMI arrives, perf_ibs_handle_irq() finds CTL2[Dis]=1 and Val=0, and therefore returns 0, leading to an unknown NMI.

The back-to-back NMI handling logic in the x86 generic NMI code should already handle this. No?

Thanks,
Ravi

Reviewed-by
#13 lore raw

Re: [PATCH 3/3] perf/amd/ibs: Simplify event state handling with IBS_CAPS_DIS

Quoted 120 lines · Ravi Bangoria

Jim,

Thanks for the detailed review.

Quoted 5 lines
Quoted 4 lines

The IBS_{FETCH|OP}_CTL2[Dis] bit allows IBS sampling to be stopped without a RMW of IBS_{FETCH,OP}_CTL. Systems providing this bit therefore do not require the IBS_STARTED and IBS_STOPPED states used to coordinate control MSR updates with late NMIs.

Quoted 8 lines

CTL2[Dis] doesn't make late NMIs go away. Hence, this patch needs a 50us poll. With the two-line fix in my original patch, the existing scheme handles late NMIs indefinitely: IBS_STOPPED remains set until a late NMI consumes it, even if the event has been restarted in the meantime. This patch trades that for a 50us window, after which a late NMI is unknown (which *is* fatal with unknown_nmi_panic). Moreover, since we still have to handle hardware without IBS_CAPS_DIS, we now have two state machines instead of one.

IBS_STOPPED only records that "a late NMI might exist". It can't tell whether the disable actually completed or whether that late NMI has already arrived. Trying to infer from software state is fundamentally broken, since the WRMSR and a software state transition can't be made atomic.

Given that Zen6 provides a clear hardware state machine, I believe using it, rather than a software state machine, is the right approach.

I ran a fuzzer for several hours with my patch (+Manali's patch [1]) and saw no Unknown NMIs on Zen6.

However, With your patch[2] (+Manali's patch), the same fuzzer continued to produce quite a few Unknown NMIs on Zen6. One race I was able to identify (with your patch applied) is described below.

---
o FETCH and OP both are enabled.

  perf_ibs_stop() /* For OP */
  {
       set IBS_STOPPED
       op state: ENABLED | STARTED |
                 STOPPING | STOPPED

OpVal is 0 in hardware

NMI due to FETCH
{
FETCH: Val=1, sample handled, fetch re-armed

OP: STARTED is set, OpVal is 0 so test_and_clear(IBS_STOPPED) clears the STOPPED state

OP continues to run, it overflows, sets OpVal=1 and a new NMI is latched by the OP hardware

return handled=2
}

op state: ENABLED | STARTED |
STOPPING

WRMSR OpCtl2.Dis=1
leaves the valid sample in OpCtl

clear IBS_STARTED

op state: ENABLED | STOPPING

NMI (late) due to OP {
FETCH: Val=0 -> return 0

OP: STARTED is clear and STOPPED is already clear.

return 0 without reading IbsOpCtl

return handled=0

Unknown NMI
}
---

[1]: https://lore.kernel.org/r/[email protected]
[2]: https://lore.kernel.org/r/[email protected]

[...]

Quoted 18 lines

arch/x86/events/amd/ibs.c static void perf_ibs_start(struct perf_event *event, int flags) · line 637

Quoted 12 lines
arch/x86/events/amd/ibs.c +12 -0
+/* ->stop() for platforms with additional CTL2[Dis] bit. */
+static void perf_ibs_stop_dis(struct perf_event *event, int flags)
+{
+       struct hw_perf_event *hwc = &event->hw;
+       struct perf_ibs *perf_ibs = container_of(event->pmu, struct perf_ibs, pmu);
+       struct cpu_perf_ibs *pcpu = this_cpu_ptr(perf_ibs->pcpu);
+       u64 config;
+       u64 ctl2;
+
+       WARN_ON_ONCE(test_bit(IBS_STOPPING, pcpu->state));
+
+       set_bit(IBS_STOPPING, pcpu->state);

This function needs to be reentrant. If an IBS event is a sibling in a group led by a core PMU event, and the leader is throttled, perf_event_throttle_group() will stop the IBS sibling from the core PMU's NMI handler:

Quoted 8 lines
    amd_pmu_v2_handle_irq() ->
      perf_event_overflow() ->
        __perf_event_overflow() ->
          __perf_event_account_interrupt() ->
            perf_event_throttle_group() ->
              perf_event_throttle() ->
                perf_ibs_stop() ->
                  perf_ibs_stop_dis()
Quoted 2 lines

Core and IBS, both are of hardware PMU type and thus can not be grouped together.

Ah. In that case, perf_ibs_stop() can only be reached in NMI context from perf_ibs_handle_irq(), so I retract my objection.

Quoted 5 lines

[...]

Quoted 3 lines
Quoted 2 lines
arch/x86/events/amd/ibs.c +2 -0
+       if (in_nmi())
+               wrmsrq(hwc->config_base, config);
Quoted 8 lines
Quoted 5 lines

As we saw above, in_nmi() doesn't mean that we're in perf_ibs_handle_irq() for this sample. If IBS Val=1 and the IBS NMI is still pending, this clears Val. When the IBS NMI arrives, perf_ibs_handle_irq() finds CTL2[Dis]=1 and Val=0, and therefore returns 0, leading to an unknown NMI.

The back-to-back NMI handling logic in the x86 generic NMI code should already handle this. No?

Given the skid that the 50us poll is designed to accommodate, I'm not sure that the back-to-back code always handles it, but since perf_ibs_stop() can only be reached in NMI context from perf_ibs_handle_irq(), I retract this objection, too.

There are still problems for older hardware (e.g. the race you identified above predates my change), but that's a different discussion.