perf/x86/amd/uncore: Drop redundant counter slot searches
Subsystems (2)
cover perf/x86/amd/uncore: Drop redundant counter slot searches
Both amd_uncore_add() and amd_uncore_del() search ctx->events[] for an event whose counter slot is already recorded in event->hw.idx.
Perf serializes ->add() and ->del() for an event, and the driver never moves an installed event between slots, so an installed event can only be at the recorded index. Both searches are redundant.
Neither callback is normally hot. That changes once more events target a PMU than it has counters. Perf then multiplexes them: every mux tick deschedules the resident set and schedules the next one, so both callbacks run for each rotated event on every tick.
Meta's fleet-wide profiles attribute 1.1% of kernel CPU time, excluding do_idle(), to amd_uncore_add() and amd_uncore_del() combined. On the host that exposed this, a workload holding 15,782 perf event file descriptors drove 56,575 calls per second into each callback from mux rotation, against a 16-counter DF PMU.
Patch 1 removes the scan in amd_uncore_add(). It walks every counter before the free slot search, which is the common case once multiplexing has scheduled the event out.
Patch 2 makes amd_uncore_del() free the recorded slot directly instead of rescanning from slot zero. Deleting all events from a full PMU with N counters drops from N * (N + 1) / 2 compare-exchanges to N.
No functional change intended.
Usama Arif (2):
perf/x86/amd/uncore: Remove redundant event slot scan perf/x86/amd/uncore: Free counter slot by index
arch/x86/events/amd/uncore.c | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-)
Signature · 1 line
2.53.0-Meta
PATCH 1/2 perf/x86/amd/uncore: Remove redundant event slot scan
amd_uncore_add() first checks the slot recorded in event->hw.idx. If that misses, it scans ctx->events[] for the event before looking for a free slot.
Perf serializes ->add() and ->del() for an event. Initialization sets idx to -1. A successful ->add() claims a slot and records its index before returning, while ->del() clears the slot before resetting idx. CPU context migration follows the same delete/add sequence.
Thus an installed event can only reside at the recorded index. If the direct check misses, the event is not present in ctx->events[].
Remove the redundant scan. This avoids walking all counters before each new slot search after multiplexing has scheduled an event out.
On a host running a production workload in the Meta fleet, amd_uncore_add() was called 56,575 times per second from mux rotation. Its 16-counter DF PMU makes each failed search scan 16 pointers across two cache lines.
--- arch/x86/events/amd/uncore.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
arch/x86/events/amd/uncore.c +6 -8
@@ -206,17 +206,15 @@ static int amd_uncore_add(struct perf_event *event, int flags)struct amd_uncore_ctx *ctx = *per_cpu_ptr (pmu->ctx, event->cpu);struct hw_perf_event *hwc = &event->hw; -/* are we already assigned? */ +/* +* Perf serializes ->add() and ->del() for an event. A successful +* ->add() records the claimed slot in hwc->idx before returning, and +* ->del() clears that slot before resetting hwc->idx. Therefore, an +* existing assignment must be at hwc->idx. +*/ if (hwc->idx !=-1 && ctx->events[hwc->idx] == event) goto out; -for (i =0 ; i < pmu->num_counters; i++) { -if (ctx->events[i] == event) { - hwc->idx = i; - goto out; - } - } -/* if not, take the first available counter */ hwc->idx =-1 ;for (i =0 ; i < pmu->num_counters; i++) {
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH 1/2] perf/x86/amd/uncore: Remove redundant event slot scan
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 15 lines
amd_uncore_add()first checks the slot recorded inevent->hw.idx. If that misses, it scans ctx->events[] for the event before looking for a free slot.Perf serializes ->add() and ->del() for an event. Initialization sets idx to -1. A successful ->add() claims a slot and records its index before returning, while ->del() clears the slot before resetting idx. CPU context migration follows the same delete/add sequence.
Thus an installed event can only reside at the recorded index. If the direct check misses, the event is not present in ctx->events[].
Remove the redundant scan. This avoids walking all counters before each new slot search after multiplexing has scheduled an event out.
Quoted 6 lines
On a host running a production workload in the Meta fleet,
amd_uncore_add()was called 56,575 times per second from mux rotation. Its 16-counter DF PMU makes each failed search scan 16 pointers across two cache lines.Cc Sandipan Das <[email protected]>Signed-off-by Usama Arif <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=1
[tip: perf/core] perf/x86/amd/uncore: Remove redundant event slot scan
In reply to Usama ArifThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 4a8557a4e5d2d5f96a989a428e38569451e74405 Gitweb: https://git.kernel.org/tip/4a8557a4e5d2d5f96a989a428e38569451e74405 Author: Usama Arif <[email protected]> AuthorDate: Mon, 21 Sep 2026 08:30:14 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 25 Sep 2026 12:45:54 +02:00
perf/x86/amd/uncore: Remove redundant event slot scan
amd_uncore_add() first checks the slot recorded in event->hw.idx. If that misses, it scans ctx->events[] for the event before looking for a free slot.
Perf serializes ->add() and ->del() for an event. Initialization sets idx to -1. A successful ->add() claims a slot and records its index before returning, while ->del() clears the slot before resetting idx. CPU context migration follows the same delete/add sequence.
Thus an installed event can only reside at the recorded index. If the direct check misses, the event is not present in ctx->events[].
Remove the redundant scan. This avoids walking all counters before each new slot search after multiplexing has scheduled an event out.
On a host running a production workload in the Meta fleet, amd_uncore_add() was called 56,575 times per second from mux rotation. Its 16-counter DF PMU makes each failed search scan 16 pointers across two cache lines.
--- arch/x86/events/amd/uncore.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
arch/x86/events/amd/uncore.c +6 -8
@@ -206,17 +206,15 @@ static int amd_uncore_add(struct perf_event *event, int flags)struct amd_uncore_ctx *ctx = *per_cpu_ptr (pmu->ctx, event->cpu);struct hw_perf_event *hwc = &event->hw; -/* are we already assigned? */ +/* +* Perf serializes ->add() and ->del() for an event. A successful +* ->add() records the claimed slot in hwc->idx before returning, and +* ->del() clears that slot before resetting hwc->idx. Therefore, an +* existing assignment must be at hwc->idx. +*/ if (hwc->idx !=-1 && ctx->events[hwc->idx] == event) goto out; -for (i =0 ; i < pmu->num_counters; i++) { -if (ctx->events[i] == event) { - hwc->idx = i; - goto out; - } - } -/* if not, take the first available counter */ hwc->idx =-1 ;for (i =0 ; i < pmu->num_counters; i++) {
PATCH 2/2 perf/x86/amd/uncore: Free counter slot by index
In reply to Usama Arifamd_uncore_del() scans ctx->events[] from slot zero and attempts a compare-exchange until it finds the event.
amd_uncore_add() records the slot it claimed in event->hw.idx. Perf calls ->del() only after a successful ->add(), and the driver never moves an installed event between slots. The index therefore remains valid until deletion.
Use the recorded index directly and retain the compare-exchange ownership check. Warn if it fails, as that means the driver lost track of the slot.
Deleting all events from a full PMU with N counters now requires N compare-exchanges instead of N * (N + 1) / 2.
On a host running a production workload in the Meta fleet, amd_uncore_del() was called 56,575 times per second from mux rotation.
--- arch/x86/events/amd/uncore.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
arch/x86/events/amd/uncore.c +3 -7
@@ -246,19 +246,15 @@ static int amd_uncore_add(struct perf_event *event, int flags)static void amd_uncore_del (struct perf_event *event,int flags) { -int i;struct amd_uncore_pmu *pmu =event_to_amd_uncore_pmu (event);struct amd_uncore_ctx *ctx = *per_cpu_ptr (pmu->ctx, event->cpu);struct hw_perf_event *hwc = &event->hw; +struct perf_event *old = event; event->pmu->stop (event,PERF_EF_UPDATE ); -for (i =0 ; i < pmu->num_counters; i++) { -struct perf_event *tmp = event; - -if (try_cmpxchg (&ctx->events[i], &tmp,NULL )) -break ; - } +/* ->del() follows a successful ->add(), so hwc->idx owns this slot. */ +WARN_ON_ONCE (!try_cmpxchg (&ctx->events[hwc->idx], &old,NULL )); hwc->idx =-1 ; }
Signature · 1 line
2.53.0-Meta
[email protected]
AI reviewRe: [PATCH 2/2] perf/x86/amd/uncore: Free counter slot by index
Sashiko has reviewed this patch and found no issues. It looks great!
Quoted 12 lines
amd_uncore_del()scans ctx->events[] from slot zero and attempts a compare-exchange until it finds the event.
amd_uncore_add()records the slot it claimed inevent->hw.idx. Perf calls ->del() only after a successful ->add(), and the driver never moves an installed event between slots. The index therefore remains valid until deletion.Use the recorded index directly and retain the compare-exchange ownership check. Warn if it fails, as that means the driver lost track of the slot.
Quoted 8 lines
Deleting all events from a full PMU with N counters now requires N compare-exchanges instead of N * (N + 1) / 2.
On a host running a production workload in the Meta fleet,
amd_uncore_del()was called 56,575 times per second from mux rotation.Cc Sandipan Das <[email protected]>Signed-off-by Usama Arif <[email protected]>
Sashiko has reviewed this patch and found no issues. It looks great!
Signature · 1 line
Sashiko AI review · https://sashiko.dev/#/patchset/[email protected]?part=2
[tip: perf/core] perf/x86/amd/uncore: Free counter slot by index
In reply to Usama ArifThe following commit has been merged into the perf/core branch of tip:
The following commit has been merged into the perf/core branch of tip:
Commit-ID: 6350de8671b94afb7691d110f63bcda42f658a69 Gitweb: https://git.kernel.org/tip/6350de8671b94afb7691d110f63bcda42f658a69 Author: Usama Arif <[email protected]> AuthorDate: Mon, 21 Sep 2026 08:30:15 -07:00 Committer: Peter Zijlstra <[email protected]> CommitterDate: Fri, 25 Sep 2026 12:45:55 +02:00
perf/x86/amd/uncore: Free counter slot by index
amd_uncore_del() scans ctx->events[] from slot zero and attempts a compare-exchange until it finds the event.
amd_uncore_add() records the slot it claimed in event->hw.idx. Perf calls ->del() only after a successful ->add(), and the driver never moves an installed event between slots. The index therefore remains valid until deletion.
Use the recorded index directly and retain the compare-exchange ownership check. Warn if it fails, as that means the driver lost track of the slot.
Deleting all events from a full PMU with N counters now requires N compare-exchanges instead of N * (N + 1) / 2.
On a host running a production workload in the Meta fleet, amd_uncore_del() was called 56,575 times per second from mux rotation.
--- arch/x86/events/amd/uncore.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
arch/x86/events/amd/uncore.c +3 -7
@@ -246,19 +246,15 @@ out:static void amd_uncore_del (struct perf_event *event,int flags) { -int i;struct amd_uncore_pmu *pmu =event_to_amd_uncore_pmu (event);struct amd_uncore_ctx *ctx = *per_cpu_ptr (pmu->ctx, event->cpu);struct hw_perf_event *hwc = &event->hw; +struct perf_event *old = event; event->pmu->stop (event,PERF_EF_UPDATE ); -for (i =0 ; i < pmu->num_counters; i++) { -struct perf_event *tmp = event; - -if (try_cmpxchg (&ctx->events[i], &tmp,NULL )) -break ; - } +/* ->del() follows a successful ->add(), so hwc->idx owns this slot. */ +WARN_ON_ONCE (!try_cmpxchg (&ctx->events[hwc->idx], &old,NULL )); hwc->idx =-1 ; }
Quoted 26 lines · Usama Arif
Both
amd_uncore_add()andamd_uncore_del()search ctx->events[] for an event whose counter slot is already recorded inevent->hw.idx.Perf serializes ->add() and ->del() for an event, and the driver never moves an installed event between slots, so an installed event can only be at the recorded index. Both searches are redundant.
Neither callback is normally hot. That changes once more events target a PMU than it has counters. Perf then multiplexes them: every mux tick deschedules the resident set and schedules the next one, so both callbacks run for each rotated event on every tick.
Meta's fleet-wide profiles attribute 1.1% of kernel CPU time, excluding
do_idle(), toamd_uncore_add()andamd_uncore_del()combined. On the host that exposed this, a workload holding 15,782 perf event file descriptors drove 56,575 calls per second into each callback from mux rotation, against a 16-counter DF PMU.Patch 1 removes the scan in
amd_uncore_add(). It walks every counter before the free slot search, which is the common case once multiplexing has scheduled the event out.Patch 2 makes
amd_uncore_del()free the recorded slot directly instead of rescanning from slot zero. Deleting all events from a full PMU with N counters drops from N * (N + 1) / 2 compare-exchanges to N.
Quoted 9 lines
No functional change intended.
Usama Arif (2):
perf/x86/amd/uncore: Remove redundant event slot scan perf/x86/amd/uncore: Free counter slot by indexarch/x86/events/amd/uncore.c | 24 +++++++++--------------- 1 file changed, 9 insertions(+), 15 deletions(-)
For the series