perf/core: Fix perf_event_disable_inatomic() race
Peter Zijlstra
maintainerQuoted 42 lines · tip-bot for Peter Zijlstra
Commit-ID: 86071b11317550d994b55ce5e31aa06bcad783b5 Gitweb: https://git.kernel.org/tip/86071b11317550d994b55ce5e31aa06bcad783b5 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 4 Apr 2019 15:03:00 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 10 Apr 2019 13:47:09 +0200perf/core: Fix
perf_event_disable_inatomic()raceThomas-Mich Richter reported he triggered a WARN()ing from
event_function_local()on his s390. The problem boils down to:CPU-A CPU-B perf_event_overflow() perf_event_disable_inatomic() @pending_disable = 1 irq_work_queue(); sched-out event_sched_out() @pending_disable = 0 sched-in perf_event_overflow() perf_event_disable_inatomic() @pending_disable = 1; irq_work_queue(); // FAILS irq_work_run() perf_pending_event() if (@pending_disable) perf_event_disable_local(); // WHOOPSThe problem exists in generic, but s390 is particularly sensitive because it doesn't implement
arch_irq_work_raise(), nor does it callirq_work_run()from it's PMU interrupt handler (nor would that be sufficient in this case, because s390 also generatesperf_event_overflow()frompmu::stop). Add to that the fact that s390 is a virtual architecture and (virtual) CPU-A can stall long enough for the above race to happen, even if it would self-IPI.
Quoted 7 lines
Adding a
irq_work_sync()toevent_sched_in()would work for all hardare PMUs that properly useirq_work_run()but fails for software PMUs.Instead encode the CPU number in @pending_disable, such that we can tell which CPU requested the disable. This then allows us to detect the above scenario and even redirect the IPI to make up for the failed queue.
Ingo, could you please fold in the below delta? It turns out I overlooked two insteances :-(
kernel/events/ring_buffer.c +2 -2
@@ -392,7 +392,7 @@ void *perf_aux_output_begin(struct perf_* store that will be enabled on successful return */ if (!handle->size) {/* A, matches D */ - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();perf_output_wakeup (handle);local_set (&rb->aux_nest,0 ); goto err_put;
@@ -480,7 +480,7 @@ void perf_aux_output_end(struct perf_outif (wakeup) {if (handle->aux_flags &PERF_AUX_FLAG_TRUNCATED ) - handle->event->pending_disable =1 ; + handle->event->pending_disable =smp_processor_id ();perf_output_wakeup (handle); }
Quoted 66 lines · Peter Zijlstra
Quoted 42 lines · tip-bot for Peter Zijlstra
Commit-ID: 86071b11317550d994b55ce5e31aa06bcad783b5 Gitweb: https://git.kernel.org/tip/86071b11317550d994b55ce5e31aa06bcad783b5 Author: Peter Zijlstra <[email protected]> AuthorDate: Thu, 4 Apr 2019 15:03:00 +0200 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 10 Apr 2019 13:47:09 +0200perf/core: Fix
perf_event_disable_inatomic()raceThomas-Mich Richter reported he triggered a WARN()ing from
event_function_local()on his s390. The problem boils down to:CPU-A CPU-B perf_event_overflow() perf_event_disable_inatomic() @pending_disable = 1 irq_work_queue(); sched-out event_sched_out() @pending_disable = 0 sched-in perf_event_overflow() perf_event_disable_inatomic() @pending_disable = 1; irq_work_queue(); // FAILS irq_work_run() perf_pending_event() if (@pending_disable) perf_event_disable_local(); // WHOOPSThe problem exists in generic, but s390 is particularly sensitive because it doesn't implement
arch_irq_work_raise(), nor does it callirq_work_run()from it's PMU interrupt handler (nor would that be sufficient in this case, because s390 also generatesperf_event_overflow()frompmu::stop). Add to that the fact that s390 is a virtual architecture and (virtual) CPU-A can stall long enough for the above race to happen, even if it would self-IPI.Quoted 7 lines
Adding a
irq_work_sync()toevent_sched_in()would work for all hardare PMUs that properly useirq_work_run()but fails for software PMUs.Instead encode the CPU number in @pending_disable, such that we can tell which CPU requested the disable. This then allows us to detect the above scenario and even redirect the IPI to make up for the failed queue.
Ingo, could you please fold in the below delta? It turns out I overlooked two insteances :-(
kernel/events/ring_buffer.c +1
-1@@ -392,7 +392,7 @@ void *perf_aux_output_begin(struct perf_* store that will be enabled on successful return */ if (!handle->size) {/* A, matches D */ - event->pending_disable =1 ; + event->pending_disable =smp_processor_id ();perf_output_wakeup (handle);local_set (&rb->aux_nest,0 ); goto err_put;
kernel/events/ring_buffer.c void perf_aux_output_end(struct perf_out · line 485
Quoted 7 lines
kernel/events/ring_buffer.c +1
-1@@ -480,7 +480,7 @@ void perf_aux_output_end(struct perf_outif (wakeup) {if (handle->aux_flags &PERF_AUX_FLAG_TRUNCATED ) - handle->event->pending_disable =1 ; + handle->event->pending_disable =smp_processor_id ();perf_output_wakeup (handle); }
Sure, done!
Thanks,
Ingo