Skip to contentSkip to conversation

perf: Fix the throttle error of some clock events

PATCH Applied [email protected] 4 participants 8 messages Jun 6, 2025 – Jun 11, 2025 perf events

lore
Share & tools
lore
#0 lore raw

PATCH v4 1/1 perf: Fix the throttle error of some clock events

From: Kan Liang <[email protected]>

Both ARM and IBM CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2

The issue is introduced by the generic throttle patch set, which unconditionally invoke the event_stop() when throttle is triggered.

The cpu-clock and task-clock are two special SW events, which rely on the hrtimer. The throttle is invoked in the hrtimer handler. The event_stop()->hrtimer_cancel() waits for the handler to finish, which is a deadlock. Instead of invoking the stop(), the HRTIMER_NORESTART should be used to stop the timer.

There may be two ways to fix it.
- Introduce a PMU flag to track the case. Avoid the event_stop in perf_event_throttle() if the flag is detected.
It has been implemented in the
https://lore.kernel.org/lkml/[email protected]/
The new flag was thought to be an overkill for the issue.
- Add a check in the event_stop. Return immediately if the throttle is invoked in the hrtimer handler. Rely on the existing HRTIMER_NORESTART method to stop the timer.

The latter is implemented here.

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0). But the cpu_clock_event_stop() doesn't handle the flag. In logic, it's wrong. But it didn't bring any problems with the old code, because the stop() was not invoked when handling the throttle. Checking the flag before updating the event.

---

The patch is to fix the issue introduced by

9734e25fbf5a perf: Fix the throttle logic for a group

It is still in the tip.git, I'm not sure if the commit ID is valid. So the Fixes tag is not added.

There are some discussions regarding to a soft lockup issue. That is an existing issue, which are not introduced by the above change. It should be fixed separately.
https://lore.kernel.org/lkml/CAADnVQ+Lx0HWEM8xdLC80wco3BTUPAD_2dQ-3oZFiECZMcw2aQ@mail.gmail.com/

Changes since V3:
- Check before update in event_stop()
- Add Reviewed-by from Ian

 kernel/events/core.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/events/core.c +11 -4
index f34c99f8ce8f..cc77f127e11a 100644
@@ -2656,8 +2656,8 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)
 
 static void perf_event_throttle(struct perf_event *event)
 {
-	event->pmu->stop(event, 0);
 	event->hw.interrupts = MAX_INTERRUPTS;
+	event->pmu->stop(event, 0);
 	if (event == event->group_leader)
 		perf_log_throttle(event, 0);
 }
@@ -11749,7 +11749,12 @@ static void perf_swevent_cancel_hrtimer(struct perf_event *event)
 {
 	struct hw_perf_event *hwc = &event->hw;
 
-	if (is_sampling_event(event)) {
+	/*
+	 * The throttle can be triggered in the hrtimer handler.
+	 * The HRTIMER_NORESTART should be used to stop the timer,
+	 * rather than hrtimer_cancel(). See perf_swevent_hrtimer()
+	 */
+	if (is_sampling_event(event) && (hwc->interrupts != MAX_INTERRUPTS)) {
 		ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
 		local64_set(&hwc->period_left, ktime_to_ns(remaining));
 
@@ -11804,7 +11809,8 @@ static void cpu_clock_event_start(struct perf_event *event, int flags)
 static void cpu_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	cpu_clock_event_update(event);
+	if (flags & PERF_EF_UPDATE)
+		cpu_clock_event_update(event);
 }
 
 static int cpu_clock_event_add(struct perf_event *event, int flags)
@@ -11882,7 +11888,8 @@ static void task_clock_event_start(struct perf_event *event, int flags)
 static void task_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	task_clock_event_update(event, event->ctx->time);
+	if (flags & PERF_EF_UPDATE)
+		task_clock_event_update(event, event->ctx->time);
 }
 
 static int task_clock_event_add(struct perf_event *event, int flags)
Signature · 1 line
2.38.1
Tested-by on v4 1/1
#1 lore raw
Quoted 49 lines · On 07/06/25 12:55 am, [email protected] wrote:

From: Kan Liang <[email protected]>

Both ARM and IBM CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2

The issue is introduced by the generic throttle patch set, which unconditionally invoke the event_stop() when throttle is triggered.

The cpu-clock and task-clock are two special SW events, which rely on the hrtimer. The throttle is invoked in the hrtimer handler. The event_stop()->hrtimer_cancel() waits for the handler to finish, which is a deadlock. Instead of invoking the stop(), the HRTIMER_NORESTART should be used to stop the timer.

There may be two ways to fix it.
- Introduce a PMU flag to track the case. Avoid the event_stop in perf_event_throttle() if the flag is detected.
It has been implemented in the
https://lore.kernel.org/lkml/[email protected]/
The new flag was thought to be an overkill for the issue.
- Add a check in the event_stop. Return immediately if the throttle is invoked in the hrtimer handler. Rely on the existing HRTIMER_NORESTART method to stop the timer.

The latter is implemented here.

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0). But the cpu_clock_event_stop() doesn't handle the flag. In logic, it's wrong. But it didn't bring any problems with the old code, because the stop() was not invoked when handling the throttle. Checking the flag before updating the event.

Quoted 1 line

---

Tested this patch by applying on top of 6.16.0-rc1, and it fixes the reported issue. Hence,

Regards,

Venkat.

Quoted 68 lines

The patch is to fix the issue introduced by

9734e25fbf5a perf: Fix the throttle logic for a group

It is still in the tip.git, I'm not sure if the commit ID is valid. So the Fixes tag is not added.

There are some discussions regarding to a soft lockup issue. That is an existing issue, which are not introduced by the above change. It should be fixed separately.
https://lore.kernel.org/lkml/CAADnVQ+Lx0HWEM8xdLC80wco3BTUPAD_2dQ-3oZFiECZMcw2aQ@mail.gmail.com/

Changes since V3:
- Check before update in event_stop()
- Add Reviewed-by from Ian

  kernel/events/core.c | 15 +++++++++++----
  1 file changed, 11 insertions(+), 4 deletions(-)
kernel/events/core.c +11 -4
index f34c99f8ce8f..cc77f127e11a 100644
@@ -2656,8 +2656,8 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)
 
  static void perf_event_throttle(struct perf_event *event)
  {
-	event->pmu->stop(event, 0);
  	event->hw.interrupts = MAX_INTERRUPTS;
+	event->pmu->stop(event, 0);
  	if (event == event->group_leader)
  		perf_log_throttle(event, 0);
  }
@@ -11749,7 +11749,12 @@ static void perf_swevent_cancel_hrtimer(struct perf_event *event)
  {
  	struct hw_perf_event *hwc = &event->hw;
 
-	if (is_sampling_event(event)) {
+	/*
+	 * The throttle can be triggered in the hrtimer handler.
+	 * The HRTIMER_NORESTART should be used to stop the timer,
+	 * rather than hrtimer_cancel(). See perf_swevent_hrtimer()
+	 */
+	if (is_sampling_event(event) && (hwc->interrupts != MAX_INTERRUPTS)) {
  		ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
  		local64_set(&hwc->period_left, ktime_to_ns(remaining));
 
@@ -11804,7 +11809,8 @@ static void cpu_clock_event_start(struct perf_event *event, int flags)
  static void cpu_clock_event_stop(struct perf_event *event, int flags)
  {
  	perf_swevent_cancel_hrtimer(event);
-	cpu_clock_event_update(event);
+	if (flags & PERF_EF_UPDATE)
+		cpu_clock_event_update(event);
  }
 
  static int cpu_clock_event_add(struct perf_event *event, int flags)
@@ -11882,7 +11888,8 @@ static void task_clock_event_start(struct perf_event *event, int flags)
  static void task_clock_event_stop(struct perf_event *event, int flags)
  {
  	perf_swevent_cancel_hrtimer(event);
-	task_clock_event_update(event, event->ctx->time);
+	if (flags & PERF_EF_UPDATE)
+		task_clock_event_update(event, event->ctx->time);
  }
 
  static int task_clock_event_add(struct perf_event *event, int flags)
#2 lore raw
In reply to [email protected]

Hi Kan,

On Fri, Jun 06, 2025 at 12:25:46PM -0700, [email protected] wrote:

[...]

Quoted 12 lines

There may be two ways to fix it.
- Introduce a PMU flag to track the case. Avoid the event_stop in perf_event_throttle() if the flag is detected.
It has been implemented in the
https://lore.kernel.org/lkml/[email protected]/
The new flag was thought to be an overkill for the issue.
- Add a check in the event_stop. Return immediately if the throttle is invoked in the hrtimer handler. Rely on the existing HRTIMER_NORESTART method to stop the timer.

The latter is implemented here.

Quoted 7 lines

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0).

I am confused for this conclusion. When a CPU or task clock event is stopped by throttling, should it also be updated? Otherwise, we will lose accouting for the period prior to the throttling.

I saw you exchanged with Alexei for a soft lockup issue, the reply [1] shows that skipping event update on throttling does not help to resolve the lockup issue.

Could you elaberate why we don't need to update clock events when throttling?

Thanks,
Leo

[1] https://lore.kernel.org/linux-perf-users/CAADnVQKRJKsG08KkEriuBQop0LgDr+c9rkNE6MUh_n3rzZoXVQ@mail.gmail.com/

Quoted 88 lines

But the cpu_clock_event_stop() doesn't handle the flag. In logic, it's wrong. But it didn't bring any problems with the old code, because the stop() was not invoked when handling the throttle. Checking the flag before updating the event.

---

The patch is to fix the issue introduced by

9734e25fbf5a perf: Fix the throttle logic for a group

It is still in the tip.git, I'm not sure if the commit ID is valid. So the Fixes tag is not added.

There are some discussions regarding to a soft lockup issue. That is an existing issue, which are not introduced by the above change. It should be fixed separately.
https://lore.kernel.org/lkml/CAADnVQ+Lx0HWEM8xdLC80wco3BTUPAD_2dQ-3oZFiECZMcw2aQ@mail.gmail.com/

Changes since V3:
- Check before update in event_stop()
- Add Reviewed-by from Ian

 kernel/events/core.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/events/core.c +11 -4
index f34c99f8ce8f..cc77f127e11a 100644
@@ -2656,8 +2656,8 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)
 
 static void perf_event_throttle(struct perf_event *event)
 {
-	event->pmu->stop(event, 0);
 	event->hw.interrupts = MAX_INTERRUPTS;
+	event->pmu->stop(event, 0);
 	if (event == event->group_leader)
 		perf_log_throttle(event, 0);
 }
@@ -11749,7 +11749,12 @@ static void perf_swevent_cancel_hrtimer(struct perf_event *event)
 {
 	struct hw_perf_event *hwc = &event->hw;
 
-	if (is_sampling_event(event)) {
+	/*
+	 * The throttle can be triggered in the hrtimer handler.
+	 * The HRTIMER_NORESTART should be used to stop the timer,
+	 * rather than hrtimer_cancel(). See perf_swevent_hrtimer()
+	 */
+	if (is_sampling_event(event) && (hwc->interrupts != MAX_INTERRUPTS)) {
 		ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
 		local64_set(&hwc->period_left, ktime_to_ns(remaining));
 
@@ -11804,7 +11809,8 @@ static void cpu_clock_event_start(struct perf_event *event, int flags)
 static void cpu_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	cpu_clock_event_update(event);
+	if (flags & PERF_EF_UPDATE)
+		cpu_clock_event_update(event);
 }
 
 static int cpu_clock_event_add(struct perf_event *event, int flags)
@@ -11882,7 +11888,8 @@ static void task_clock_event_start(struct perf_event *event, int flags)
 static void task_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	task_clock_event_update(event, event->ctx->time);
+	if (flags & PERF_EF_UPDATE)
+		task_clock_event_update(event, event->ctx->time);
 }
 
 static int task_clock_event_add(struct perf_event *event, int flags)
Signature · 1 line
2.38.1
author
#3 lore raw
Quoted 30 lines · Leo Yan

Hi Kan,

On Fri, Jun 06, 2025 at 12:25:46PM -0700, [email protected] wrote:

[...]

Quoted 12 lines

There may be two ways to fix it.
- Introduce a PMU flag to track the case. Avoid the event_stop in perf_event_throttle() if the flag is detected.
It has been implemented in the
https://lore.kernel.org/lkml/[email protected]/
The new flag was thought to be an overkill for the issue.
- Add a check in the event_stop. Return immediately if the throttle is invoked in the hrtimer handler. Rely on the existing HRTIMER_NORESTART method to stop the timer.

The latter is implemented here.

Quoted 7 lines

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0).

I am confused for this conclusion. When a CPU or task clock event is stopped by throttling, should it also be updated? Otherwise, we will lose accouting for the period prior to the throttling.

Quoted 7 lines

I saw you exchanged with Alexei for a soft lockup issue, the reply [1] shows that skipping event update on throttling does not help to resolve the lockup issue.

Could you elaberate why we don't need to update clock events when throttling?

This is to follow the existing behavior before the throttling fix*. When throttling is triggered, the stop(event, 0); will be invoked. As my understanding, it's because the period is not changed with throttling. So we don't need to update the period.

But if the period is changed, the update is required. You may find an example in the perf_adjust_freq_unthr_events(). In the freq mode, stop(event, PERF_EF_UPDATE) is actually invoked for the triggered event.

For the clock event, the existing behavior before the throttling fix* is not to invoke the stop() in throttling. It relies on the HRTIMER_NORESTART instead. My previous throttling fix changes the behavior. It invokes both stop() and HRTIMER_NORESTART. Now, this patch change the behavior back.

Regarding the soft lockup issue, it's a different issue in virtualization. It should be an old issue which is not introduced by my throttling fix.

Thanks,
Kan

* The throttling fix is 9734e25fbf5a ("perf: Fix the throttle logic for a group")

Quoted 94 lines

Thanks,
Leo

[1] https://lore.kernel.org/linux-perf-users/CAADnVQKRJKsG08KkEriuBQop0LgDr+c9rkNE6MUh_n3rzZoXVQ@mail.gmail.com/

Quoted 88 lines

But the cpu_clock_event_stop() doesn't handle the flag. In logic, it's wrong. But it didn't bring any problems with the old code, because the stop() was not invoked when handling the throttle. Checking the flag before updating the event.

---

The patch is to fix the issue introduced by

9734e25fbf5a perf: Fix the throttle logic for a group

It is still in the tip.git, I'm not sure if the commit ID is valid. So the Fixes tag is not added.

There are some discussions regarding to a soft lockup issue. That is an existing issue, which are not introduced by the above change. It should be fixed separately.
https://lore.kernel.org/lkml/CAADnVQ+Lx0HWEM8xdLC80wco3BTUPAD_2dQ-3oZFiECZMcw2aQ@mail.gmail.com/

Changes since V3:
- Check before update in event_stop()
- Add Reviewed-by from Ian

 kernel/events/core.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/events/core.c +11 -4
index f34c99f8ce8f..cc77f127e11a 100644
@@ -2656,8 +2656,8 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)
 
 static void perf_event_throttle(struct perf_event *event)
 {
-	event->pmu->stop(event, 0);
 	event->hw.interrupts = MAX_INTERRUPTS;
+	event->pmu->stop(event, 0);
 	if (event == event->group_leader)
 		perf_log_throttle(event, 0);
 }
@@ -11749,7 +11749,12 @@ static void perf_swevent_cancel_hrtimer(struct perf_event *event)
 {
 	struct hw_perf_event *hwc = &event->hw;
 
-	if (is_sampling_event(event)) {
+	/*
+	 * The throttle can be triggered in the hrtimer handler.
+	 * The HRTIMER_NORESTART should be used to stop the timer,
+	 * rather than hrtimer_cancel(). See perf_swevent_hrtimer()
+	 */
+	if (is_sampling_event(event) && (hwc->interrupts != MAX_INTERRUPTS)) {
 		ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
 		local64_set(&hwc->period_left, ktime_to_ns(remaining));
 
@@ -11804,7 +11809,8 @@ static void cpu_clock_event_start(struct perf_event *event, int flags)
 static void cpu_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	cpu_clock_event_update(event);
+	if (flags & PERF_EF_UPDATE)
+		cpu_clock_event_update(event);
 }
 
 static int cpu_clock_event_add(struct perf_event *event, int flags)
@@ -11882,7 +11888,8 @@ static void task_clock_event_start(struct perf_event *event, int flags)
 static void task_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	task_clock_event_update(event, event->ctx->time);
+	if (flags & PERF_EF_UPDATE)
+		task_clock_event_update(event, event->ctx->time);
 }
 
 static int task_clock_event_add(struct perf_event *event, int flags)
Signature · 1 line
2.38.1
#4 lore raw

On Mon, Jun 09, 2025 at 09:48:12AM -0400, Liang, Kan wrote:

[...]

Quoted 30 lines
Quoted 12 lines
Quoted 7 lines

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0).

I am confused for this conclusion. When a CPU or task clock event is stopped by throttling, should it also be updated? Otherwise, we will lose accouting for the period prior to the throttling.

Quoted 7 lines

I saw you exchanged with Alexei for a soft lockup issue, the reply [1] shows that skipping event update on throttling does not help to resolve the lockup issue.

Could you elaberate why we don't need to update clock events when throttling?

This is to follow the existing behavior before the throttling fix*.

When throttling is triggered, the stop(event, 0); will be invoked. As my understanding, it's because the period is not changed with throttling. So we don't need to update the period.

But if the period is changed, the update is required. You may find an example in the perf_adjust_freq_unthr_events(). In the freq mode, stop(event, PERF_EF_UPDATE) is actually invoked for the triggered event.

Quoted 5 lines

For the clock event, the existing behavior before the throttling fix* is not to invoke the stop() in throttling. It relies on the HRTIMER_NORESTART instead. My previous throttling fix changes the behavior. It invokes both stop() and HRTIMER_NORESTART. Now, this patch change the behavior back.

Actually, the "event->count" has been updated in perf_swevent_hrtimer(), this is why this patch does not cause big deviation if skip updating count in the ->stop() callback:

perf_swevent_hrtimer()
` event->pmu->read(event); => Update count
` __perf_event_overflow()
` perf_event_throttle()
` event->pmu->stop(event, 0) / cpu_clock_event_stop()
` perf_swevent_cancel_hrtimer() => Skip to cancel timer
` task_clock_event_update() => Skip to update count
` return HRTIMER_NORESTART; => Stop timer

It is a bit urgly that we check the throttling separately in two
places: one is in perf_swevent_cancel_hrtime() for skipping cancel timer, and then we skip updating event count in cpu_clock_event_stop().

One solution is it would be fine to update count in ->stop() callback for the throttling. This should not cause any issue (though it is a bit redundant that the count is updated twice).

Or even more clear, we can define a flag PERF_EF_THROTTLING:

    #define PERF_EF_THROTTLING  0x20

    event->pmu->stop(event, PERF_EF_THROTTLING);

    cpu_clock_event_stop(struct perf_event *event, int flags)
    {
        if (flags == PERF_EF_THROTTLING)
            return;

        ....
    }

This might need to do a wider checking to ensure this new flags will not cause any issues.

Thanks,
Leo

author
#5 lore raw
Quoted 52 lines · Leo Yan

On Mon, Jun 09, 2025 at 09:48:12AM -0400, Liang, Kan wrote:

[...]

Quoted 30 lines
Quoted 12 lines
Quoted 7 lines

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0).

I am confused for this conclusion. When a CPU or task clock event is stopped by throttling, should it also be updated? Otherwise, we will lose accouting for the period prior to the throttling.

Quoted 7 lines

I saw you exchanged with Alexei for a soft lockup issue, the reply [1] shows that skipping event update on throttling does not help to resolve the lockup issue.

Could you elaberate why we don't need to update clock events when throttling?

This is to follow the existing behavior before the throttling fix*.

When throttling is triggered, the stop(event, 0); will be invoked. As my understanding, it's because the period is not changed with throttling. So we don't need to update the period.

But if the period is changed, the update is required. You may find an example in the perf_adjust_freq_unthr_events(). In the freq mode, stop(event, PERF_EF_UPDATE) is actually invoked for the triggered event.

Quoted 5 lines

For the clock event, the existing behavior before the throttling fix* is not to invoke the stop() in throttling. It relies on the HRTIMER_NORESTART instead. My previous throttling fix changes the behavior. It invokes both stop() and HRTIMER_NORESTART. Now, this patch change the behavior back.

Actually, the "event->count" has been updated in perf_swevent_hrtimer(), this is why this patch does not cause big deviation if skip updating count in the ->stop() callback:

Quoted 1 line

perf_swevent_hrtimer()

` event->pmu->read(event); => Update count
` __perf_event_overflow()
` perf_event_throttle()
` event->pmu->stop(event, 0) / cpu_clock_event_stop()
` perf_swevent_cancel_hrtimer() => Skip to cancel timer
` task_clock_event_update() => Skip to update count
` return HRTIMER_NORESTART; => Stop timer

Quoted 4 lines

It is a bit urgly that we check the throttling separately in two
places: one is in perf_swevent_cancel_hrtime() for skipping cancel timer, and then we skip updating event count in cpu_clock_event_stop().

The second check before cpu_clock_event_stop() is not a throttling check. It's to implement the missed flag check.
Usually, the stop() should check PERF_EF_UPDATE before updating an event. I think most of the ARCHs do so.
Some cases may ignore the flag. For the clock event, I think it's because the stop(event, 0) is never invoked. So it doesn't matter if the flag is checked. But now, there is a case which the flag matters. So I think we should add the flag check.

Quoted 4 lines

One solution is it would be fine to update count in ->stop() callback for the throttling. This should not cause any issue (though it is a bit redundant that the count is updated twice).

The clock event relies on local_clock(), which never stops. So it still counts between read() and stop().
It's not just redundant. The behavior is changed if the event is updated in the stop() again.

Quoted 7 lines

Or even more clear, we can define a flag PERF_EF_THROTTLING:

    #define PERF_EF_THROTTLING  0x20

    event->pmu->stop(event, PERF_EF_THROTTLING);

The if (hwc->interrupts != MAX_INTERRUPTS) should be good enough to check the throttling case. I don't think we need a new flag here.

Quoted 10 lines
    cpu_clock_event_stop(struct perf_event *event, int flags)
    {
        if (flags == PERF_EF_THROTTLING)
            return;

        ....
    }

This might need to do a wider checking to ensure this new flags will not cause any issues.

Right, it may brings more troubles.

I think we should properly utilize the existing flag rather than introducing a new one.

Thanks,
Kan

#6 lore raw

On Mon, Jun 09, 2025 at 03:59:41PM -0400, Liang, Kan wrote:

[...]

Quoted 41 lines
Quoted 26 lines
Quoted 8 lines

When throttling is triggered, the stop(event, 0); will be invoked. As my understanding, it's because the period is not changed with throttling. So we don't need to update the period.

But if the period is changed, the update is required. You may find an example in the perf_adjust_freq_unthr_events(). In the freq mode, stop(event, PERF_EF_UPDATE) is actually invoked for the triggered event.

Quoted 5 lines

For the clock event, the existing behavior before the throttling fix* is not to invoke the stop() in throttling. It relies on the HRTIMER_NORESTART instead. My previous throttling fix changes the behavior. It invokes both stop() and HRTIMER_NORESTART. Now, this patch change the behavior back.

Actually, the "event->count" has been updated in perf_swevent_hrtimer(), this is why this patch does not cause big deviation if skip updating count in the ->stop() callback:

Quoted 1 line

perf_swevent_hrtimer()

` event->pmu->read(event); => Update count
` __perf_event_overflow()
` perf_event_throttle()
` event->pmu->stop(event, 0) / cpu_clock_event_stop()
` perf_swevent_cancel_hrtimer() => Skip to cancel timer
` task_clock_event_update() => Skip to update count
` return HRTIMER_NORESTART; => Stop timer

Quoted 4 lines

It is a bit urgly that we check the throttling separately in two
places: one is in perf_swevent_cancel_hrtime() for skipping cancel timer, and then we skip updating event count in cpu_clock_event_stop().

The second check before cpu_clock_event_stop() is not a throttling check. It's to implement the missed flag check.
Usually, the stop() should check PERF_EF_UPDATE before updating an event. I think most of the ARCHs do so.
Some cases may ignore the flag. For the clock event, I think it's because the stop(event, 0) is never invoked. So it doesn't matter if the flag is checked. But now, there is a case which the flag matters. So I think we should add the flag check.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

One solution is it would be fine to update count in ->stop() callback for the throttling. This should not cause any issue (though it is a bit redundant that the count is updated twice).

The clock event relies on local_clock(), which never stops.

Ah, good point!

Quoted 6 lines

So it still counts between read() and stop().
It's not just redundant. The behavior is changed if the event is updated in the stop() again.

Quoted 2 lines

Or even more clear, we can define a flag PERF_EF_THROTTLING:

Quoted 7 lines
Quoted 4 lines
    #define PERF_EF_THROTTLING  0x20

    event->pmu->stop(event, PERF_EF_THROTTLING);

The if (hwc->interrupts != MAX_INTERRUPTS) should be good enough to check the throttling case. I don't think we need a new flag here.

Makes sense to me.

Thanks,
Leo

Quoted 19 lines
Quoted 10 lines
    cpu_clock_event_stop(struct perf_event *event, int flags)
    {
        if (flags == PERF_EF_THROTTLING)
            return;

        ....
    }

This might need to do a wider checking to ensure this new flags will not cause any issues.

Right, it may brings more troubles.

I think we should properly utilize the existing flag rather than introducing a new one.

Thanks,
Kan

merge bot Applied
#7 lore raw
In reply to [email protected]
The following commit has been merged into the perf/urgent branch of tip:

The following commit has been merged into the perf/urgent branch of tip:

Commit-ID:     bc4394e5e79cdda1b0997e0be1d65e242f523f02
Gitweb:        https://git.kernel.org/tip/bc4394e5e79cdda1b0997e0be1d65e242f523f02
Author:        Kan Liang <[email protected]>
AuthorDate:    Fri, 06 Jun 2025 12:25:46 -07:00
Committer:     Peter Zijlstra <[email protected]>
CommitterDate: Wed, 11 Jun 2025 14:05:08 +02:00

perf: Fix the throttle error of some clock events

Both ARM and IBM CI reports RCU stall, which can be reproduced by the below perf command.
perf record -a -e cpu-clock -- sleep 2

The issue is introduced by the generic throttle patch set, which unconditionally invoke the event_stop() when throttle is triggered.

The cpu-clock and task-clock are two special SW events, which rely on the hrtimer. The throttle is invoked in the hrtimer handler. The event_stop()->hrtimer_cancel() waits for the handler to finish, which is a deadlock. Instead of invoking the stop(), the HRTIMER_NORESTART should be used to stop the timer.

There may be two ways to fix it:
- Introduce a PMU flag to track the case. Avoid the event_stop in perf_event_throttle() if the flag is detected.
It has been implemented in the
https://lore.kernel.org/lkml/[email protected]/
The new flag was thought to be an overkill for the issue.
- Add a check in the event_stop. Return immediately if the throttle is invoked in the hrtimer handler. Rely on the existing HRTIMER_NORESTART method to stop the timer.

The latter is implemented here.

Move event->hw.interrupts = MAX_INTERRUPTS before the stop(). It makes the order the same as perf_event_unthrottle(). Except the patch, no one checks the hw.interrupts in the stop(). There is no impact from the order change.

When stops in the throttle, the event should not be updated, stop(event, 0). But the cpu_clock_event_stop() doesn't handle the flag. In logic, it's wrong. But it didn't bring any problems with the old code, because the stop() was not invoked when handling the throttle. Checking the flag before updating the event.

---
 kernel/events/core.c | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)
kernel/events/core.c +11 -4
index d7cf008..1f74646 100644
@@ -2674,8 +2674,8 @@ static void perf_event_unthrottle(struct perf_event *event, bool start)
 
 static void perf_event_throttle(struct perf_event *event)
 {
-	event->pmu->stop(event, 0);
 	event->hw.interrupts = MAX_INTERRUPTS;
+	event->pmu->stop(event, 0);
 	if (event == event->group_leader)
 		perf_log_throttle(event, 0);
 }
@@ -11774,7 +11774,12 @@ static void perf_swevent_cancel_hrtimer(struct perf_event *event)
 {
 	struct hw_perf_event *hwc = &event->hw;
 
-	if (is_sampling_event(event)) {
+	/*
+	 * The throttle can be triggered in the hrtimer handler.
+	 * The HRTIMER_NORESTART should be used to stop the timer,
+	 * rather than hrtimer_cancel(). See perf_swevent_hrtimer()
+	 */
+	if (is_sampling_event(event) && (hwc->interrupts != MAX_INTERRUPTS)) {
 		ktime_t remaining = hrtimer_get_remaining(&hwc->hrtimer);
 		local64_set(&hwc->period_left, ktime_to_ns(remaining));
 
@@ -11829,7 +11834,8 @@ static void cpu_clock_event_start(struct perf_event *event, int flags)
 static void cpu_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	cpu_clock_event_update(event);
+	if (flags & PERF_EF_UPDATE)
+		cpu_clock_event_update(event);
 }
 
 static int cpu_clock_event_add(struct perf_event *event, int flags)
@@ -11907,7 +11913,8 @@ static void task_clock_event_start(struct perf_event *event, int flags)
 static void task_clock_event_stop(struct perf_event *event, int flags)
 {
 	perf_swevent_cancel_hrtimer(event);
-	task_clock_event_update(event, event->ctx->time);
+	if (flags & PERF_EF_UPDATE)
+		task_clock_event_update(event, event->ctx->time);
 }
 
 static int task_clock_event_add(struct perf_event *event, int flags)