Skip to contentSkip to conversation

perf/x86/rapl: Stop doing cpu_relax in the cmpxchg loop

PATCH Applied Uros Bizjak 2 participants 6 messages Aug 7, 2023 – Oct 3, 2023

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/2 perf/x86/rapl: Stop doing cpu_relax in the cmpxchg loop

According to the commit f5fe24ef17b5 ("lockref: stop doing cpu_relax in the cmpxchg loop"):

"On the x86-64 architecture even a failing cmpxchg grants exclusive access to the cacheline, making it preferable to retry the failed op immediately instead of stalling with the pause instruction."

Based on the above observation, remove cpu_relax from the cmpxchg loop of rapl_event_update.

Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Alexander Shishkin <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Ian Rogers <[email protected]>
Cc: Adrian Hunter <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc. "H. Peter Anvin" <[email protected]>

---
 arch/x86/events/rapl.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
arch/x86/events/rapl.c +1 -3
index 52e6e7ed4f78..e8e26733e17b 100644
@@ -184,10 +184,8 @@ static u64 rapl_event_update(struct perf_event *event)
 	rdmsrl(event->hw.event_base, new_raw_count);
 
 	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
-			    new_raw_count) != prev_raw_count) {
-		cpu_relax();
+			    new_raw_count) != prev_raw_count)
 		goto again;
-	}
 
 	/*
 	 * Now we have the new raw value and have updated the prev
Signature · 1 line
2.41.0
#1 lore raw

PATCH 2/2 perf/x86/rapl: Use local64_try_cmpxchg in rapl_event_update

Use local64_try_cmpxchg instead of local64_cmpxchg (*ptr, old, new) == old. x86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after cmpxchg (and related move instruction in front of cmpxchg).

Also, try_cmpxchg implicitly assigns old *ptr value to "old" when cmpxchg fails. There is no need to re-read the value in the loop.

No functional change intended.

Cc: Peter Zijlstra <[email protected]>
Cc: Ingo Molnar <[email protected]>
Cc: Arnaldo Carvalho de Melo <[email protected]>
Cc: Mark Rutland <[email protected]>
Cc: Alexander Shishkin <[email protected]>
Cc: Jiri Olsa <[email protected]>
Cc: Namhyung Kim <[email protected]>
Cc: Ian Rogers <[email protected]>
Cc: Adrian Hunter <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Borislav Petkov <[email protected]>
Cc: Dave Hansen <[email protected]>
Cc. "H. Peter Anvin" <[email protected]>

---
 arch/x86/events/rapl.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
arch/x86/events/rapl.c +4 -6
index e8e26733e17b..7c2376dae79d 100644
@@ -179,13 +179,11 @@ static u64 rapl_event_update(struct perf_event *event)
 	s64 delta, sdelta;
 	int shift = RAPL_CNTR_WIDTH;
 
-again:
 	prev_raw_count = local64_read(&hwc->prev_count);
-	rdmsrl(event->hw.event_base, new_raw_count);
-
-	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
-			    new_raw_count) != prev_raw_count)
-		goto again;
+	do {
+		rdmsrl(event->hw.event_base, new_raw_count);
+	} while (!local64_try_cmpxchg(&hwc->prev_count,
+				      &prev_raw_count, new_raw_count));
 
 	/*
 	 * Now we have the new raw value and have updated the prev
Signature · 1 line
2.41.0
merge bot Applied
#2 lore raw

[tip: perf/core] perf/x86/rapl: Use local64_try_cmpxchg in rapl_event_update()

The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     df60e18058bb6792c85e4ede43876c1df44f5b9a
Gitweb:        https://git.kernel.org/tip/df60e18058bb6792c85e4ede43876c1df44f5b9a
Author:        Uros Bizjak <[email protected]>
AuthorDate:    Mon, 07 Aug 2023 16:51:15 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 03 Oct 2023 09:36:18 +02:00

perf/x86/rapl: Use local64_try_cmpxchg in rapl_event_update()

Use local64_try_cmpxchg() instead of local64_cmpxchg(*ptr, old, new) == old.

X86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after CMPXCHG (and related move instruction in front of CMPXCHG).

Also, try_cmpxchg() implicitly assigns old *ptr value to "old" when CMPXCHG fails. There is no need to re-read the value in the loop.

No functional change intended.

Cc. "H. Peter Anvin" <[email protected]>

---
 arch/x86/events/rapl.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
arch/x86/events/rapl.c +4 -6
index e8f53b2..6d3e738 100644
@@ -179,13 +179,11 @@ static u64 rapl_event_update(struct perf_event *event)
 	s64 delta, sdelta;
 	int shift = RAPL_CNTR_WIDTH;
 
-again:
 	prev_raw_count = local64_read(&hwc->prev_count);
-	rdmsrl(event->hw.event_base, new_raw_count);
-
-	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
-			    new_raw_count) != prev_raw_count)
-		goto again;
+	do {
+		rdmsrl(event->hw.event_base, new_raw_count);
+	} while (!local64_try_cmpxchg(&hwc->prev_count,
+				      &prev_raw_count, new_raw_count));
 
 	/*
 	 * Now we have the new raw value and have updated the prev
merge bot Applied
#3 lore raw

[tip: perf/core] perf/x86/rapl: Use local64_try_cmpxchg in rapl_event_update()

In reply to Uros Bizjak
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     bcc6ec3d954bbcc8bec34a21c05ea536a2e96d6f
Gitweb:        https://git.kernel.org/tip/bcc6ec3d954bbcc8bec34a21c05ea536a2e96d6f
Author:        Uros Bizjak <[email protected]>
AuthorDate:    Mon, 07 Aug 2023 16:51:15 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 03 Oct 2023 21:13:45 +02:00

perf/x86/rapl: Use local64_try_cmpxchg in rapl_event_update()

Use local64_try_cmpxchg() instead of local64_cmpxchg(*ptr, old, new) == old.

X86 CMPXCHG instruction returns success in ZF flag, so this change saves a compare after CMPXCHG (and related move instruction in front of CMPXCHG).

Also, try_cmpxchg() implicitly assigns old *ptr value to "old" when CMPXCHG fails. There is no need to re-read the value in the loop.

No functional change intended.

---
 arch/x86/events/rapl.c | 10 ++++------
 1 file changed, 4 insertions(+), 6 deletions(-)
arch/x86/events/rapl.c +4 -6
index e8f53b2..6d3e738 100644
@@ -179,13 +179,11 @@ static u64 rapl_event_update(struct perf_event *event)
 	s64 delta, sdelta;
 	int shift = RAPL_CNTR_WIDTH;
 
-again:
 	prev_raw_count = local64_read(&hwc->prev_count);
-	rdmsrl(event->hw.event_base, new_raw_count);
-
-	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
-			    new_raw_count) != prev_raw_count)
-		goto again;
+	do {
+		rdmsrl(event->hw.event_base, new_raw_count);
+	} while (!local64_try_cmpxchg(&hwc->prev_count,
+				      &prev_raw_count, new_raw_count));
 
 	/*
 	 * Now we have the new raw value and have updated the prev
merge bot Applied
#4 lore raw

[tip: perf/core] perf/x86/rapl: Stop doing cpu_relax() in the local64_cmpxchg() loop in rapl_event_update()

In reply to Uros Bizjak
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     df22fb4bcdd6f67c4f568e6321c9b0050819d213
Gitweb:        https://git.kernel.org/tip/df22fb4bcdd6f67c4f568e6321c9b0050819d213
Author:        Uros Bizjak <[email protected]>
AuthorDate:    Mon, 07 Aug 2023 16:51:14 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 03 Oct 2023 09:36:17 +02:00

perf/x86/rapl: Stop doing cpu_relax() in the local64_cmpxchg() loop in rapl_event_update()

According to the following commit:

f5fe24ef17b5 ("lockref: stop doing cpu_relax in the cmpxchg loop")

"On the x86-64 architecture even a failing cmpxchg grants exclusive
access to the cacheline, making it preferable to retry the failed op immediately instead of stalling with the pause instruction."

Based on the above observation, remove cpu_relax() from the local64_cmpxchg() loop of rapl_event_update().

Cc. "H. Peter Anvin" <[email protected]>

---
 arch/x86/events/rapl.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
arch/x86/events/rapl.c +1 -3
index 1579429..e8f53b2 100644
@@ -184,10 +184,8 @@ again:
 	rdmsrl(event->hw.event_base, new_raw_count);
 
 	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
-			    new_raw_count) != prev_raw_count) {
-		cpu_relax();
+			    new_raw_count) != prev_raw_count)
 		goto again;
-	}
 
 	/*
 	 * Now we have the new raw value and have updated the prev
merge bot Applied
#5 lore raw

[tip: perf/core] perf/x86/rapl: Stop doing cpu_relax() in the local64_cmpxchg() loop in rapl_event_update()

In reply to Uros Bizjak
The following commit has been merged into the perf/core branch of tip:

The following commit has been merged into the perf/core branch of tip:

Commit-ID:     1ce19bf90bd55bf54f9ed75d594029db63d395b0
Gitweb:        https://git.kernel.org/tip/1ce19bf90bd55bf54f9ed75d594029db63d395b0
Author:        Uros Bizjak <[email protected]>
AuthorDate:    Mon, 07 Aug 2023 16:51:14 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Tue, 03 Oct 2023 21:13:23 +02:00

perf/x86/rapl: Stop doing cpu_relax() in the local64_cmpxchg() loop in rapl_event_update()

According to the following commit:

f5fe24ef17b5 ("lockref: stop doing cpu_relax in the cmpxchg loop")

"On the x86-64 architecture even a failing cmpxchg grants exclusive
access to the cacheline, making it preferable to retry the failed op immediately instead of stalling with the pause instruction."

Based on the above observation, remove cpu_relax() from the local64_cmpxchg() loop of rapl_event_update().

---
 arch/x86/events/rapl.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)
arch/x86/events/rapl.c +1 -3
index 1579429..e8f53b2 100644
@@ -184,10 +184,8 @@ again:
 	rdmsrl(event->hw.event_base, new_raw_count);
 
 	if (local64_cmpxchg(&hwc->prev_count, prev_raw_count,
-			    new_raw_count) != prev_raw_count) {
-		cpu_relax();
+			    new_raw_count) != prev_raw_count)
 		goto again;
-	}
 
 	/*
 	 * Now we have the new raw value and have updated the prev