perf: Fix the throttle error of some clock events
PATCH v2 1/1 perf: Fix the throttle error of some clock events
From: Kan Liang <[email protected]>
The Arm CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2
The cpu-clock and task_clock are two special SW events, which rely on the hrtimer. Instead of invoking the stop(), the HRTIMER_NORESTART is returned to stop the timer. Because the hrtimer interrupt handler cannot cancel itself, which causes infinite loop.
There may be two ways to fix it.
- Add a check of MAX_INTERRUPTS in the event_stop. Return immediately if the stop is invoked by the throttle.
- Introduce a PMU flag to track the case. Avoid the event_stop in perf_event_throttle() if the flag is detected.
The latter looks more generic. It may be used if there are more other cases that want to avoid the stop later. The latter is implemented.
---
Changes since V1:
- Rebase on top of the latest tip.git
- Add Tested-by from Leo
include/linux/perf_event.h | 1 + kernel/events/core.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)
include/linux/perf_event.h +1 -0
@@ -305,6 +305,7 @@ struct perf_event_pmu_context;#define PERF_PMU_CAP_EXTENDED_HW_TYPE 0x0100#define PERF_PMU_CAP_AUX_PAUSE 0x0200#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400 +#define PERF_PMU_CAP_NO_THROTTLE_STOP 0x0800/** * pmu::scope
kernel/events/core.c +20 -3
@@ -2656,7 +2656,22 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)static void perf_event_throttle (struct perf_event *event) { - event->pmu->stop (event,0 ); +/* +* Some PMUs, e.g., cpu-clock and task_clock, may rely on +* a special mechanism (hrtimer) to manipulate counters. +* The regular stop doesn't work, since the hrtimer interrupt +* handler cannot cancel itself. +* +* The stop should be avoided for such cases. Let the +* driver-specific code handle it. +* +* The counters will eventually be disabled in the driver-specific +* code. In unthrottle, they still need to be re-enabled. +* There is no handling for PERF_PMU_CAP_NO_THROTTLE_STOP in +* the perf_event_unthrottle(). +*/ +if (!(event->pmu->capabilities &PERF_PMU_CAP_NO_THROTTLE_STOP )) + event->pmu->stop (event,0 ); event->hw.interrupts =MAX_INTERRUPTS ;if (event == event->group_leader)perf_log_throttle (event,0 );
@@ -11848,7 +11863,8 @@ static int cpu_clock_event_init(struct perf_event *event)static struct pmu perf_cpu_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = cpu_clock_event_init,
@@ -11930,7 +11946,8 @@ static int task_clock_event_init(struct perf_event *event)static struct pmu perf_task_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = task_clock_event_init,
Signature · 1 line
2.38.1
Quoted 5 lines · On 28/05/25 11:28 pm, [email protected] wrote:
From: Kan Liang <[email protected]>
The Arm CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2
This issue is reported on IBM CI as well. But the issue is hit, while running perf test (specifically while running 76: perf list tests)
Quoted 6 lines
The cpu-clock and
task_clockare two special SW events, which rely on the hrtimer. Instead of invoking thestop(), theHRTIMER_NORESTARTis returned to stop the timer. Because the hrtimer interrupt handler cannot cancel itself, which causes infinite loop.
Quoted 8 lines
There may be two ways to fix it.
- Add a check ofMAX_INTERRUPTSin theevent_stop. Return immediately if the stop is invoked by the throttle.
- Introduce a PMU flag to track the case. Avoid theevent_stopinperf_event_throttle()if the flag is detected.The latter looks more generic. It may be used if there are more other cases that want to avoid the stop later. The latter is implemented.
Tested this patch by applying on the linux-mainline kernel on IBM Power9 system, and it fixes the issue. Hence,
Regards,
Venkat.
Quoted 76 lines
Reported-by Leo Yan <[email protected]>Reported-by Aishwarya TCV <[email protected]>Tested-by Leo Yan <[email protected]>Signed-off-by Kan Liang <[email protected]>---
Changes since V1:
- Rebase on top of the latest tip.git
- Add Tested-by from Leoinclude/linux/perf_event.h | 1 + kernel/events/core.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)include/linux/perf_event.h +1
-0@@ -305,6 +305,7 @@ struct perf_event_pmu_context;#define PERF_PMU_CAP_EXTENDED_HW_TYPE 0x0100#define PERF_PMU_CAP_AUX_PAUSE 0x0200#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400 +#define PERF_PMU_CAP_NO_THROTTLE_STOP 0x0800/** * pmu::scope kernel/events/core.c +20
-3@@ -2656,7 +2656,22 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)static void perf_event_throttle (struct perf_event *event) { - event->pmu->stop (event,0 ); +/* +* Some PMUs, e.g., cpu-clock and task_clock, may rely on +* a special mechanism (hrtimer) to manipulate counters. +* The regular stop doesn't work, since the hrtimer interrupt +* handler cannot cancel itself. +* +* The stop should be avoided for such cases. Let the +* driver-specific code handle it. +* +* The counters will eventually be disabled in the driver-specific +* code. In unthrottle, they still need to be re-enabled. +* There is no handling for PERF_PMU_CAP_NO_THROTTLE_STOP in +* the perf_event_unthrottle(). +*/ +if (!(event->pmu->capabilities &PERF_PMU_CAP_NO_THROTTLE_STOP )) + event->pmu->stop (event,0 ); event->hw.interrupts =MAX_INTERRUPTS ;if (event == event->group_leader)perf_log_throttle (event,0 );@@ -11848,7 +11863,8 @@ static int cpu_clock_event_init(struct perf_event *event)static struct pmu perf_cpu_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = cpu_clock_event_init,@@ -11930,7 +11946,8 @@ static int task_clock_event_init(struct perf_event *event)static struct pmu perf_task_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = task_clock_event_init,
Ian Rogers
reviewerQuoted 11 lines · On Wed, May 28, 2025 at 10:59 AM <[email protected]> wrote:
From: Kan Liang <[email protected]>
The Arm CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2The cpu-clock and
task_clockare two special SW events, which rely on the hrtimer. Instead of invoking thestop(), theHRTIMER_NORESTARTis returned to stop the timer. Because the hrtimer interrupt handler cannot cancel itself, which causes infinite loop.
I'm having a hard time understanding this. Currently in the code we have two PMUs for cpu_clock and task_clock:
https://web.git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/kernel/events/core.c#n10729
``` static struct pmu perf_cpu_clock; /* fwd declaration */ static struct pmu perf_task_clock; ```
This is weird as both of them are programmed using type PERF_TYPE_SOFTWARE (1) and there is a corresponding sysfs PMU at /sys/bus/event_source/devices/software/type so wouldn't it make sense for there to be 1 PMU? Anyway, the stop functions are similar:
https://web.git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/kernel/events/core.c#n11804
```
static void cpu_clock_event_stop(struct perf_event *event, int flags)
{
perf_swevent_cancel_hrtimer(event);
cpu_clock_event_update(event);
}
...
static void task_clock_event_stop(struct perf_event *event, int flags)
{
perf_swevent_cancel_hrtimer(event);
task_clock_event_update(event, event->ctx->time);
}
```
and I think you are talking about perf_swevent_cancel_hrtimer:
```
static void perf_swevent_cancel_hrtimer(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
if (is_sampling_event(event)) {
ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
local64_set(&hwc->period_left, ktime_to_ns(remaining));
hrtimer_cancel(&hwc->hrtimer);
}
}
```but I'm not seeing where HRTIMER_NORESTART comes into this.
Quoted 5 lines
There may be two ways to fix it.
- Add a check ofMAX_INTERRUPTSin theevent_stop. Return immediately if the stop is invoked by the throttle.
- Introduce a PMU flag to track the case. Avoid theevent_stopinperf_event_throttle()if the flag is detected.
I'm not sure what the issue is with the hrtimer_cancel from the explanation up to here. If we can't cancel the timer can we have the timer callback perf_swevent_hrtimer drop the event?
I can see how the patch is working around the problem but from an API point-of-view having PERF_PMU_CAP_NO_THROTTLE_STOP seems to be adding more possible complexity to PMUs. Possibly if the software event stop worked better it would help other future users too.
Sorry for my ignorance in this and I'm likely missing the reason this is the best solution. I think improving the commit message would help other future people and not just me.
Thanks,
Ian
Quoted 81 lines
The latter looks more generic. It may be used if there are more other cases that want to avoid the stop later. The latter is implemented.
Reported-by Leo Yan <[email protected]>Reported-by Aishwarya TCV <[email protected]>Tested-by Leo Yan <[email protected]>Signed-off-by Kan Liang <[email protected]>---
Changes since V1:
- Rebase on top of the latest tip.git
- Add Tested-by from Leoinclude/linux/perf_event.h | 1 + kernel/events/core.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)include/linux/perf_event.h +1
-0@@ -305,6 +305,7 @@ struct perf_event_pmu_context;#define PERF_PMU_CAP_EXTENDED_HW_TYPE 0x0100#define PERF_PMU_CAP_AUX_PAUSE 0x0200#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400 +#define PERF_PMU_CAP_NO_THROTTLE_STOP 0x0800/** * pmu::scope kernel/events/core.c +20
-4@@ -2656,7 +2656,22 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)static void perf_event_throttle (struct perf_event *event) { - event->pmu->stop (event,0 ); +/* +* Some PMUs, e.g., cpu-clock and task_clock, may rely on +* a special mechanism (hrtimer) to manipulate counters. +* The regular stop doesn't work, since the hrtimer interrupt +* handler cannot cancel itself. +* +* The stop should be avoided for such cases. Let the +* driver-specific code handle it. +* +* The counters will eventually be disabled in the driver-specific +* code. In unthrottle, they still need to be re-enabled. +* There is no handling for PERF_PMU_CAP_NO_THROTTLE_STOP in +* the perf_event_unthrottle(). +*/ +if (!(event->pmu->capabilities &PERF_PMU_CAP_NO_THROTTLE_STOP )) + event->pmu->stop (event,0 ); event->hw.interrupts =MAX_INTERRUPTS ;if (event == event->group_leader)perf_log_throttle (event,0 );@@ -11848,7 +11863,8 @@ static int cpu_clock_event_init(struct perf_event *event)static struct pmu perf_cpu_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = cpu_clock_event_init,@@ -11930,7 +11946,8 @@ static int task_clock_event_init(struct perf_event *event)static struct pmu perf_task_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = task_clock_event_init, --2.38.1
Quoted 16 lines · Ian Rogers
Quoted 11 lines · On Wed, May 28, 2025 at 10:59 AM <[email protected]> wrote:
From: Kan Liang <[email protected]>
The Arm CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2The cpu-clock and
task_clockare two special SW events, which rely on the hrtimer. Instead of invoking thestop(), theHRTIMER_NORESTARTis returned to stop the timer. Because the hrtimer interrupt handler cannot cancel itself, which causes infinite loop.I'm having a hard time understanding this. Currently in the code we have two PMUs for
cpu_clockandtask_clock:
https://web.git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/kernel/events/core.c#n10729
Quoted 8 lines
``` static struct pmu perf_cpu_clock; /* fwd declaration */ static struct pmu perf_task_clock; ```This is weird as both of them are programmed using type
PERF_TYPE_SOFTWARE(1) and there is a corresponding sysfs PMU at/sys/bus/event_source/devices/software/typeso wouldn't it make sense for there to be 1 PMU?
No, they don't use the type PERF_TYPE_SOFTWARE.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/events/core.c#n14729
They need different operators from the regular software events. I guess that's why they have a dedicated PMU.
Quoted 29 lines
Anyway, the stop functions are similar:
https://web.git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/tree/kernel/events/core.c#n11804``` static void cpu_clock_event_stop(struct perf_event *event, int flags) { perf_swevent_cancel_hrtimer(event); cpu_clock_event_update(event); } ... static void task_clock_event_stop(struct perf_event *event, int flags) { perf_swevent_cancel_hrtimer(event); task_clock_event_update(event, event->ctx->time); } ``` and I think you are talking about perf_swevent_cancel_hrtimer: ``` static void perf_swevent_cancel_hrtimer(struct perf_event *event) { struct hw_perf_event *hwc = &event->hw; if (is_sampling_event(event)) { ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer); local64_set(&hwc->period_left, ktime_to_ns(remaining)); hrtimer_cancel(&hwc->hrtimer); } } ```
Quoted 1 line
but I'm not seeing where
HRTIMER_NORESTARTcomes into this.
That's an issue introduced by the generic throttle patch set, which unconditionally invoke the event_stop() when throttle is triggered.
It's wrong to call the hrtimer_cancel() to stop the timer in the timer handler. Because the hrtimer_cancel() waits for the handler to finish, which is a deadlock.
The correct way (or original way) is to set a HRTIMER_NORESTART.
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/kernel/events/core.c#n11717
So we need a method to specially handle the cpu_clock and task_clock.
Quoted 10 lines
Quoted 6 lines
There may be two ways to fix it.
- Add a check ofMAX_INTERRUPTSin theevent_stop. Return immediately if the stop is invoked by the throttle.
- Introduce a PMU flag to track the case. Avoid theevent_stopinperf_event_throttle()if the flag is detected.I'm not sure what the issue is with the
hrtimer_cancelfrom the explanation up to here. If we can't cancel the timer can we have the timer callbackperf_swevent_hrtimerdrop the event?
A callback is not required. Just ignore the perf_swevent_cancel_hrtimer() when it's invoked from perf_swevent_hrtimer(). It's similar to the first method (way) above.
Quoted 6 lines
I can see how the patch is working around the problem but from an API point-of-view having
PERF_PMU_CAP_NO_THROTTLE_STOPseems to be adding more possible complexity to PMUs. Possibly if the software event stop worked better it would help other future users too.
The PERF_PMU_CAP_NO_THROTTLE_STOP should only be useful if there are some future special PMUs which doesn't want to stop by the generic code. To be honest, I have no idea if there will be such PMUs.
If no such PMUs, we should not need the flag.
Quoted 3 lines
Sorry for my ignorance in this and I'm likely missing the reason this
is the best solution. I think improving the commit message would help
other future people and not just me.I will send a V3 to drop the flag and only apply the fix for the two
clock events. We can have a comparison then.
Thanks,
Kan>
Quoted 85 lines
Thanks,
IanQuoted 81 lines
The latter looks more generic. It may be used if there are more other cases that want to avoid the stop later. The latter is implemented.
Reported-by Leo Yan <[email protected]>Reported-by Aishwarya TCV <[email protected]>Tested-by Leo Yan <[email protected]>Signed-off-by Kan Liang <[email protected]>---
Changes since V1:
- Rebase on top of the latest tip.git
- Add Tested-by from Leoinclude/linux/perf_event.h | 1 + kernel/events/core.c | 23 ++++++++++++++++++++--- 2 files changed, 21 insertions(+), 3 deletions(-)include/linux/perf_event.h +1
-0@@ -305,6 +305,7 @@ struct perf_event_pmu_context;#define PERF_PMU_CAP_EXTENDED_HW_TYPE 0x0100#define PERF_PMU_CAP_AUX_PAUSE 0x0200#define PERF_PMU_CAP_AUX_PREFER_LARGE 0x0400 +#define PERF_PMU_CAP_NO_THROTTLE_STOP 0x0800/** * pmu::scope kernel/events/core.c +20
-4@@ -2656,7 +2656,22 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)static void perf_event_throttle (struct perf_event *event) { - event->pmu->stop (event,0 ); +/* +* Some PMUs, e.g., cpu-clock and task_clock, may rely on +* a special mechanism (hrtimer) to manipulate counters. +* The regular stop doesn't work, since the hrtimer interrupt +* handler cannot cancel itself. +* +* The stop should be avoided for such cases. Let the +* driver-specific code handle it. +* +* The counters will eventually be disabled in the driver-specific +* code. In unthrottle, they still need to be re-enabled. +* There is no handling for PERF_PMU_CAP_NO_THROTTLE_STOP in +* the perf_event_unthrottle(). +*/ +if (!(event->pmu->capabilities &PERF_PMU_CAP_NO_THROTTLE_STOP )) + event->pmu->stop (event,0 ); event->hw.interrupts =MAX_INTERRUPTS ;if (event == event->group_leader)perf_log_throttle (event,0 );@@ -11848,7 +11863,8 @@ static int cpu_clock_event_init(struct perf_event *event)static struct pmu perf_cpu_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = cpu_clock_event_init,@@ -11930,7 +11946,8 @@ static int task_clock_event_init(struct perf_event *event)static struct pmu perf_task_clock = { .task_ctx_nr = perf_sw_context, - .capabilities =PERF_PMU_CAP_NO_NMI , + .capabilities =PERF_PMU_CAP_NO_NMI | +PERF_PMU_CAP_NO_THROTTLE_STOP , .dev =PMU_NULL_DEV , .event_init = task_clock_event_init, --2.38.1