arm64: perf: Add support for Perf NMI interrupts
Subsystems (4)
cover arm64: perf: Add support for Perf NMI interrupts
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiq
Lecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64: Kconfig: Add support for the Perf NMI
arch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)
Signature · 1 line
2.25.1
PATCH 1/3 arm_pmu: Add support for perf NMI interrupts registration
Register perf interrupts by request_nmi()/percpu_nmi() when both ARM64_PSEUDO_NMI and ARM64_PSEUDO_NMI_PERF are enabled and nmi cpufreature is active.
--- drivers/perf/arm_pmu.c | 51 +++++++++++++++++++++++++++++++----- include/linux/perf/arm_pmu.h | 6 +++++ 2 files changed, 51 insertions(+), 6 deletions(-)
drivers/perf/arm_pmu.c +43 -0
@@ -559,6 +559,48 @@ void armpmu_free_irq(int irq, int cpu)per_cpu (cpu_irq, cpu) =0 ; } +static void armpmu_prepare_percpu_nmi_other (void *info) +{ +/* +* We don't need to disable preemption since smp_call_function() +* did this for us. +*/ +prepare_percpu_nmi ((uintptr_t ) info); +} + +static int _armpmu_request_irq (unsigned int irq,irq_handler_t handler, +unsigned long flags,int cpu) +{ +if (armpmu_support_nmi ()) +return request_nmi (irq, handler, flags,"arm-pmu" , +per_cpu_ptr (&cpu_armpmu, cpu)); +return request_irq (irq, handler, flags,"arm-pmu" , +per_cpu_ptr (&cpu_armpmu, cpu)); +} + +static int _armpmu_request_percpu_irq (unsigned int irq,irq_handler_t handler) +{ +if (armpmu_support_nmi ()) { +int err; + + err =request_percpu_nmi (irq, handler,"arm-pmu" , + &cpu_armpmu); +if (err) +return err; + +preempt_disable (); + err =prepare_percpu_nmi (irq); +if (err) { +return err; +preempt_enable (); + } +smp_call_function (armpmu_prepare_percpu_nmi_other, + (void *)(uintptr_t ) irq, true); +preempt_enable (); + } +return request_percpu_irq (irq, handler,"arm-pmu" , + &cpu_armpmu); +} +int armpmu_request_irq (int irq,int cpu) {
int err = 0;
@@ -582,12 +624,9 @@ int armpmu_request_irq(int irq, int cpu)
IRQF_NO_THREAD;
irq_set_status_flags(irq, IRQ_NOAUTOEN);
- err = request_irq(irq, handler, irq_flags, "arm-pmu",
- per_cpu_ptr(&cpu_armpmu, cpu));
- } else if (armpmu_count_irq_users(irq) == 0) {
- err = request_percpu_irq(irq, handler, "arm-pmu",
- &cpu_armpmu);
- }
+ err = _armpmu_request_irq(irq, handler, irq_flags, cpu);
+ } else if (armpmu_count_irq_users(irq) == 0)
+ err = _armpmu_request_percpu_irq(irq, handler);
if (err)
goto err_out;
include/linux/perf/arm_pmu.h +6 -0
@@ -160,6 +160,12 @@ int arm_pmu_acpi_probe(armpmu_init_fn init_fn);static inline int arm_pmu_acpi_probe (armpmu_init_fn init_fn) {return 0 ; }#endif +static inline bool armpmu_support_nmi (void ) +{ +return IS_ENABLED (CONFIG_ARM64_PSEUDO_NMI_PERF ) && +system_uses_irq_prio_masking (); +} +/* Internal functions only for core arm_pmu code */ struct arm_pmu *armpmu_alloc (void );struct arm_pmu *armpmu_alloc_atomic (void );
Signature · 1 line
2.25.1
PATCH 2/3 arm64: perf: Support NMI context for perf event ISR
In reply to Lecopzer ChenPerf ISR doesn't support for NMI context, thus add some necessary condition-if to handle NMI context:
- We should not hold pmu_lock since it may have already been acquired before NMI triggered.
- irq_work should not run at NMI context.
--- arch/arm64/kernel/perf_event.c | 36 +++++++++++++++++++++++++--------- 1 file changed, 27 insertions(+), 9 deletions(-)
arch/arm64/kernel/perf_event.c +27 -9
@@ -313,6 +313,23 @@ static inline bool armv8pmu_event_is_chained(struct perf_event *event) (idx !=ARMV8_IDX_CYCLE_COUNTER ); } +/* +* NMI Perf interrupts may be triggered during kernel holding +* same lock. +* Avoid acquiring lock again in NMI context. +*/ +#define armv8pmu_lock (lock, flags) \ + do { \ + if (!in_nmi()) \ + raw_spin_lock_irqsave(lock, flags); \ + } while (0) + +#define armv8pmu_unlock (lock, flags) \ + do { \ + if (!in_nmi()) \ + raw_spin_unlock_irqrestore(lock, flags);\ + } while (0) +/* * ARMv8 low level PMU access */
@@ -589,7 +606,7 @@ static void armv8pmu_enable_event(struct perf_event *event)* Enable counter and interrupt, and set the counter to count * the event that we're interested in. */ -raw_spin_lock_irqsave (&events->pmu_lock, flags); +armv8pmu_lock (&events->pmu_lock, flags);/* * Disable counter
@@ -611,7 +628,7 @@ static void armv8pmu_enable_event(struct perf_event *event)*/ armv8pmu_enable_event_counter (event); -raw_spin_unlock_irqrestore (&events->pmu_lock, flags); +armv8pmu_unlock (&events->pmu_lock, flags); }static void armv8pmu_disable_event (struct perf_event *event)
@@ -623,7 +640,7 @@ static void armv8pmu_disable_event(struct perf_event *event)/* * Disable counter and interrupt */ -raw_spin_lock_irqsave (&events->pmu_lock, flags); +armv8pmu_lock (&events->pmu_lock, flags);/* * Disable counter
@@ -635,7 +652,7 @@ static void armv8pmu_disable_event(struct perf_event *event)*/ armv8pmu_disable_event_irq (event); -raw_spin_unlock_irqrestore (&events->pmu_lock, flags); +armv8pmu_unlock (&events->pmu_lock, flags); }static void armv8pmu_start (struct arm_pmu *cpu_pmu)
@@ -643,10 +660,10 @@ static void armv8pmu_start(struct arm_pmu *cpu_pmu)unsigned long flags;struct pmu_hw_events *events =this_cpu_ptr (cpu_pmu->hw_events); -raw_spin_lock_irqsave (&events->pmu_lock, flags); +armv8pmu_lock (&events->pmu_lock, flags);/* Enable all counters */ armv8pmu_pmcr_write (armv8pmu_pmcr_read () |ARMV8_PMU_PMCR_E ); -raw_spin_unlock_irqrestore (&events->pmu_lock, flags); +armv8pmu_unlock (&events->pmu_lock, flags); }static void armv8pmu_stop (struct arm_pmu *cpu_pmu)
@@ -654,10 +671,10 @@ static void armv8pmu_stop(struct arm_pmu *cpu_pmu)unsigned long flags;struct pmu_hw_events *events =this_cpu_ptr (cpu_pmu->hw_events); -raw_spin_lock_irqsave (&events->pmu_lock, flags); +armv8pmu_lock (&events->pmu_lock, flags);/* Disable all counters */ armv8pmu_pmcr_write (armv8pmu_pmcr_read () & ~ARMV8_PMU_PMCR_E ); -raw_spin_unlock_irqrestore (&events->pmu_lock, flags); +armv8pmu_unlock (&events->pmu_lock, flags); }static irqreturn_t armv8pmu_handle_irq (struct arm_pmu *cpu_pmu)
@@ -722,7 +739,8 @@ static irqreturn_t armv8pmu_handle_irq(struct arm_pmu *cpu_pmu)* platforms that can have the PMU interrupts raised as an NMI, this * will not work. */ -irq_work_run (); +if (!armpmu_support_nmi ()) +irq_work_run ();return IRQ_HANDLED ; }
Signature · 1 line
2.25.1
PATCH 3/3 arm64: Kconfig: Add support for the Perf NMI
In reply to Lecopzer ChenThis is an extending function for Pseudo NMI that registering Perf events interrupts as NMI.
It's helpful for sampling irq-off context when using perf.
--- arch/arm64/Kconfig | 10 ++++++++++ 1 file changed, 10 insertions(+)
arch/arm64/Kconfig +12 -0
@@ -1679,6 +1679,16 @@ config ARM64_PSEUDO_NMI If unsure, say N if ARM64_PSEUDO_NMI +config ARM64_PSEUDO_NMI_PERF + bool "Register Perf interrupts as Pseudo NMI" + depends on HW_PERF_EVENTS + depends on ARM_PMU + select HAVE_PERF_EVENTS_NMI + help + This registers Perf interrupts to NMI when Pseudo NMI is active. + This option is helpful when you need to debug any context disabled + irq and get more inforamtion. + + If unsure, say N + config ARM64_DEBUG_PRIORITY_MASKING
bool "Debug interrupt priority masking"
help
Signature · 1 line
2.25.1
Re: [PATCH 1/3] arm_pmu: Add support for perf NMI interrupts registration
In reply to Lecopzer ChenThere was some mistakes when merging this patch.
The free nmi part is not present :(
The following part will be added in V2 next weekend.
drivers/perf/arm_pmu.c +34 -2
@@ -544,6 +544,38 @@ static int armpmu_count_irq_users(const int irq)return count; } +static void armpmu_teardown_percpu_nmi_other (void * info) +{ +/* +* We don't need to disable preemption since smp_call_function() +* did this for us. +*/ +teardown_percpu_nmi ((uintptr_t ) info); +} + +static void _armpmu_free_irq (unsigned int irq,void *dev_id) +{ +if (armpmu_support_nmi ()) +free_nmi (irq, dev_id); +else +free_irq (irq, dev_id); +} + +static void _armpmu_free_percpu_irq (unsigned int irq,void __percpu *dev_id) +{ +if (armpmu_support_nmi ()) { +preempt_disable (); +teardown_percpu_nmi (irq); +smp_call_function (armpmu_teardown_percpu_nmi_other, + (void *)(uintptr_t ) irq, true); +preempt_enable (); + +free_percpu_nmi (irq, dev_id); + } +else +free_percpu_irq (irq, dev_id); +} +void armpmu_free_irq (int irq,int cpu) {if (per_cpu (cpu_irq, cpu) ==0 )
@@ -552,9 +584,9 @@ void armpmu_free_irq(int irq, int cpu)return ;if (!irq_is_percpu_devid (irq)) -free_irq (irq,per_cpu_ptr (&cpu_armpmu, cpu)); +_armpmu_free_irq (irq,per_cpu_ptr (&cpu_armpmu, cpu));else if (armpmu_count_irq_users (irq) ==1 ) -free_percpu_irq (irq, &cpu_armpmu); +_armpmu_free_percpu_irq (irq, &cpu_armpmu);per_cpu (cpu_irq, cpu) =0 ; }
Thanks,
Lecopzer
Lecopzer Chen <[email protected]> 於 2020年5月16日 週六 下午8:50寫道:
Quoted 101 lines
Register perf interrupts by request_nmi()/percpu_nmi() when both
ARM64_PSEUDO_NMIandARM64_PSEUDO_NMI_PERFare enabled and nmi cpufreature is active.Signed-off-by Lecopzer Chen <[email protected]>--- drivers/perf/arm_pmu.c | 51 +++++++++++++++++++++++++++++++----- include/linux/perf/arm_pmu.h | 6 +++++ 2 files changed, 51 insertions(+), 6 deletions(-)drivers/perf/arm_pmu.c +46
-6@@ -559,6 +559,48 @@ void armpmu_free_irq(int irq, int cpu)per_cpu (cpu_irq, cpu) =0 ; } +static void armpmu_prepare_percpu_nmi_other (void *info) +{ +/* +* We don't need to disable preemption since smp_call_function() +* did this for us. +*/ +prepare_percpu_nmi ((uintptr_t ) info); +} + +static int _armpmu_request_irq (unsigned int irq,irq_handler_t handler, +unsigned long flags,int cpu) +{ +if (armpmu_support_nmi ()) +return request_nmi (irq, handler, flags,"arm-pmu" , +per_cpu_ptr (&cpu_armpmu, cpu)); +return request_irq (irq, handler, flags,"arm-pmu" , +per_cpu_ptr (&cpu_armpmu, cpu)); +} + +static int _armpmu_request_percpu_irq (unsigned int irq,irq_handler_t handler) +{ +if (armpmu_support_nmi ()) { +int err; + + err =request_percpu_nmi (irq, handler,"arm-pmu" , + &cpu_armpmu); +if (err) +return err; + +preempt_disable (); + err =prepare_percpu_nmi (irq); +if (err) { +return err; +preempt_enable (); + } +smp_call_function (armpmu_prepare_percpu_nmi_other, + (void *)(uintptr_t ) irq, true); +preempt_enable (); + } +return request_percpu_irq (irq, handler,"arm-pmu" , + &cpu_armpmu); +} +int armpmu_request_irq (int irq,int cpu) {int err =0 ;@@ -582,12 +624,9 @@ int armpmu_request_irq(int irq, int cpu)IRQF_NO_THREAD ;irq_set_status_flags (irq,IRQ_NOAUTOEN ); - err =request_irq (irq, handler, irq_flags,"arm-pmu" , -per_cpu_ptr (&cpu_armpmu, cpu)); - }else if (armpmu_count_irq_users (irq) ==0 ) { - err =request_percpu_irq (irq, handler,"arm-pmu" , - &cpu_armpmu); - } + err =_armpmu_request_irq (irq, handler, irq_flags, cpu); + }else if (armpmu_count_irq_users (irq) ==0 ) + err =_armpmu_request_percpu_irq (irq, handler);if (err) goto err_out;include/linux/perf/arm_pmu.h +6
-1@@ -160,6 +160,12 @@ int arm_pmu_acpi_probe(armpmu_init_fn init_fn);static inline int arm_pmu_acpi_probe (armpmu_init_fn init_fn) {return 0 ; }#endif +static inline bool armpmu_support_nmi (void ) +{ +return IS_ENABLED (CONFIG_ARM64_PSEUDO_NMI_PERF ) && +system_uses_irq_prio_masking (); +} +/* Internal functions only for core arm_pmu code */ struct arm_pmu *armpmu_alloc (void );struct arm_pmu *armpmu_alloc_atomic (void ); --2.25.1
+ Julien
Hi Lecopzer,
Quoted 11 lines · Lecopzer Chen
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
-Sumit
Quoted 26 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqLecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)
Hi Julien,
Does any Arm maintainers can proceed this action?
This is really useful in debugging.
Thank you!!
[1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午1:46寫道:
Quoted 62 lines
+ Julien
Hi Lecopzer,
Quoted 11 lines · Lecopzer Chen
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html-Sumit
Quoted 26 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqLecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
maintainerHi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)
Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
Thanks,
Mark.
Quoted 73 lines
This is really useful in debugging.
Thank you!![1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午1:46寫道:
Quoted 62 lines
+ Julien
Hi Lecopzer,
Quoted 11 lines · Lecopzer Chen
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html-Sumit
Quoted 26 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqLecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi,
Quoted 19 lines · Mark Rutland
Hi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
Quoted 7 lines
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is this will take a few weeks.
Thanks,
Alex
Quoted 74 lines
Thanks,
Mark.Quoted 70 lines
This is really useful in debugging.
Thank you!![1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午1:46寫道:
Quoted 59 lines
+ Julien
Hi Lecopzer,
Quoted 10 lines · Lecopzer Chen
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html-Sumit
Quoted 26 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqLecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 28 lines · Alexandru Elisei
Hi,
Quoted 24 lines · Mark Rutland
Hi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 9 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is this will take a few weeks.
Thanks Mark, Alex for the status updates on perf NMI feature.
Alex,
As the hard-lockup detection patch [1] has a dependency on perf NMI patch-set, I will rebase and test hard-lockup detector when you have got a working tree. But due to the dependency, I think patch [1] should be accepted along with perf NMI patch-set. So would you be open to include this patch as part of your series?
[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
-Sumit
Quoted 76 lines
Thanks,
AlexQuoted 74 lines
Thanks,
Mark.Quoted 70 lines
This is really useful in debugging.
Thank you!![1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午1:46寫道:
Quoted 59 lines
+ Julien
Hi Lecopzer,
Quoted 10 lines · Lecopzer Chen
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html-Sumit
Quoted 26 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqLecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Mark Rutland
maintainerQuoted 38 lines · Sumit Garg
Quoted 23 lines · Alexandru Elisei
Quoted 22 lines · Mark Rutland
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 9 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is this will take a few weeks.
Thanks Mark, Alex for the status updates on perf NMI feature.
Alex,
Quoted 7 lines
As the hard-lockup detection patch [1] has a dependency on perf NMI patch-set, I will rebase and test hard-lockup detector when you have got a working tree. But due to the dependency, I think patch [1] should be accepted along with perf NMI patch-set. So would you be open to include this patch as part of your series?
[1] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html
While it depends on the perf NMI bits, I don't think it makes sense to tie that into the series given it's trying to achieve something very different.
I think that should be reposted separately once the perf NMI bits are in shape.
Thanks,
Mark.
Quoted 46 lines · Mark Rutland
Quoted 44 lines · Sumit Garg
Quoted 23 lines · Alexandru Elisei
Quoted 22 lines · Mark Rutland
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 9 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is this will take a few weeks.
Thanks Mark, Alex for the status updates on perf NMI feature.
Alex,
As the hard-lockup detection patch [1] has a dependency on perf NMI patch-set, I will rebase and test hard-lockup detector when you have got a working tree. But due to the dependency, I think patch [1] should be accepted along with perf NMI patch-set. So would you be open to include this patch as part of your series?
Quoted 8 lines
While it depends on the perf NMI bits, I don't think it makes sense to tie that into the series given it's trying to achieve something very different.
I think that should be reposted separately once the perf NMI bits are in shape.
Okay, fair enough. Will keep it as a separate patch then.
-Sumit
Quoted 3 lines
Thanks,
Mark.
Quoted 26 lines · Mark Rutland
Quoted 24 lines
Hi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 8 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is this will take a few weeks.
+1
I would be awesome if perf NMI patches could be re-activated. Right now, it seems it is hard to do "perf annotate" on a kernel function with local_irq disabled.
func()
{local_irq_save();
.....
....local_irq_restore();
return;
}
Perf will report all cycles are used by the last moment of the func().
Thanks,
Barry
Quoted 77 lines
Thanks,
AlexQuoted 15 lines
Thanks,
Mark.Quoted 11 lines
This is really useful in debugging.
Thank you!![1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午
1:46寫道:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
+ Julien
Hi Lecopzer,
On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like
interruts.
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 6 lines
That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2]
http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/73222
7.html-Sumit
Quoted 7 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Lecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51++++++++++++++++++++++++++++++----
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1
Hi,
Quoted 27 lines · Alexandru Elisei
Hi,
Quoted 24 lines · Mark Rutland
Hi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 7 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is this will take a few weeks.
Pushed a WIP branch on linux-arm.org [1]:
git clone -b WIP-pmu-nmi git://linux-arm.org/linux-ae
Practically untested, I only did perf record on a defconfig kernel running on the model.
[1] http://www.linux-arm.org/git?p=linux-ae.git;a=shortlog;h=refs/heads/WIP-pmu-nmi
Thanks,
Alex
Quoted 80 lines
Thanks,
AlexQuoted 73 lines
Thanks,
Mark.Quoted 70 lines
This is really useful in debugging.
Thank you!![1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午1:46寫道:
Quoted 59 lines
+ Julien
Hi Lecopzer,
Quoted 10 lines · Lecopzer Chen
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like interruts. That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside Linux.
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2] http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/732227.html-Sumit
Quoted 26 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqLecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51 ++++++++++++++++++++++++++++++---- include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Quoted 47 lines
-----Original Message-----
From: linux-arm-kernel [mailto:[email protected]] On Behalf Of Alexandru Elisei
Sent: Wednesday, May 20, 2020 10:31 PM>
Hi,Quoted 27 lines · Alexandru Elisei
Hi,
Quoted 24 lines · Mark Rutland
Hi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 7 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is
this will take a few weeks.
Pushed a WIP branch on linux-arm.org [1]:
Quoted 8 lines
git clone -b WIP-pmu-nmi
git://linux-arm.org/linux-aePractically untested, I only did perf record on a defconfig kernel running on the model.
[1]
http://www.linux-arm.org/git?p=linux-ae.git;a=shortlog;h=refs/heads/WIP-pm
u-nmi
Fortunately, it does work. I used this tree to perf annotate arm_smmu_cmdq_issue_cmdlist() which is completely disabling IRQ. Luckily, it reports correct data. Before that, it reported all time was spent by the code which enabled IRQ .
Barry
Quoted 80 lines
Thanks,
AlexQuoted 17 lines
Thanks,
AlexQuoted 14 lines
Thanks,
Mark.Quoted 11 lines
This is really useful in debugging.
Thank you!![1] https://lkml.org/lkml/2020/4/24/328
Lecopzer
Sumit Garg <[email protected]> 於 2020年5月18日 週一 下午
1:46寫道:
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
+ Julien
Hi Lecopzer,
On Sat, 16 May 2020 at 18:20, Lecopzer Chen <[email protected]>
Quoted 4 lines · wrote:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
These series implement Perf NMI funxtionality and depends on Pseudo NMI [1] which has been upstreamed.
In arm64 with GICv3, Pseudo NMI was implemented for NMI-like
interruts.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
That can be extended to Perf NMI which is the prerequisite for hard-lockup detector which had already a standard interface inside
Linux.
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 31 lines
Quoted 4 lines
Thus the first step we need to implement perf NMI interface and make sure it works fine.
This is something that is already implemented via Julien's patch-set [1]. Its v4 has been floating since July, 2019 and I couldn't find any major blocking comments but not sure why things haven't progressed further.
Maybe Julien or Arm maintainers can provide updates on existing patch-set [1] and how we should proceed further with this interesting feature.
And regarding hard-lockup detection, I have been able to enable it based on perf NMI events using Julien's perf patch-set [1]. Have a look at the patch here [2].
[1] https://patchwork.kernel.org/cover/11047407/
[2]
http://lists.infradead.org/pipermail/linux-arm-kernel/2020-May/7322
27.html-Sumit
Quoted 7 lines
Perf NMI has been test by dd if=/dev/urandom of=/dev/null like the link [2] did.
[1] https://lkml.org/lkml/2019/1/31/535
[2] https://www.linaro.org/blog/debugging-arm-kernels-using-nmifiqQuoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Lecopzer Chen (3):
arm_pmu: Add support for perf NMI interrupts registration
arm64: perf: Support NMI context for perf event ISR
arm64:Kconfig: Add support for the Perf NMIarch/arm64/Kconfig | 10 +++++++ arch/arm64/kernel/perf_event.c | 36 ++++++++++++++++++------ drivers/perf/arm_pmu.c | 51++++++++++++++++++++++++++++++----
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
include/linux/perf/arm_pmu.h | 6 ++++ 4 files changed, 88 insertions(+), 15 deletions(-)--
2.25.1
Hi,
Quoted 51 lines · Song Bao Hua (Barry Song)
Quoted 51 lines
-----Original Message-----
From: linux-arm-kernel [mailto:[email protected]] On Behalf Of Alexandru Elisei
Sent: Wednesday, May 20, 2020 10:31 PM>
Hi,Quoted 27 lines · Alexandru Elisei
Hi,
Quoted 24 lines · Mark Rutland
Hi all,
Quoted 10 lines · Lecopzer Chen
HI Sumit,
Thanks for your information.
I've already implemented IPI (same as you did [1], little difference in detail), hardlockup detector and perf in last year(2019) for debuggability.
And now we tend to upstream to reduce kernel maintaining effort. I'm glad if someone in ARM can do this work :)Quoted 3 lines
Hi Julien,
Does any Arm maintainers can proceed this action?
Alexandru (Cc'd) has been rebasing and reworking Julien's patches, which is my preferred approach.
I understand that's not quite ready for posting since he's investigating some of the nastier subtleties (e.g. mutual exclusion with the NMI), but maybe we can put the work-in-progress patches somewhere in the mean time.
Quoted 7 lines
Quoted 2 lines
Alexandru, do you have an idea of what needs to be done, and/or when you expect you could post that?
I'm currently working on rebasing the patches on top of 5.7-rc5, when I have something usable I'll post a link (should be a couple of days). After that I will address the review comments, and I plan to do a thorough testing because I'm not 100% confident that some of the assumptions around the locks that were removed are correct. My guess is
this will take a few weeks.
Pushed a WIP branch on linux-arm.org [1]:
git clone -b WIP-pmu-nmi
git://linux-arm.org/linux-aePractically untested, I only did perf record on a defconfig kernel running on the model.
Quoted 6 lines
Quoted 3 lines
[1]
http://www.linux-arm.org/git?p=linux-ae.git;a=shortlog;h=refs/heads/WIP-pm
u-nmiFortunately, it does work. I used this tree to perf annotate
arm_smmu_cmdq_issue_cmdlist()which is completely disabling IRQ. Luckily, it reports correct data. Before that, it reported all time was spent by the code which enabled IRQ .
That's good news that it works for you, thanks for letting me know.
Alex