lockup detector fixes
cover lockup detector fixes
A bunch of fixes for the lockup detector.
Tested on intel/amd on top of ingo's tip branch.
Akinobu Mita (2):
lockup_detector: convert cpu notifier to return encapsulate errno
value
lockup_detector: remove unnecessary panic_notifier
Don Zickus (1):
[lockup detector] sync touch_*_watchdog back to old semantics
kernel/watchdog.c | 53 +++++++++++++++++++++++------------------------------ 1 files changed, 23 insertions(+), 30 deletions(-)
Signature · 1 line
1.7.2.2
PATCH 1/3 sync touch_*_watchdog back to old semantics
During my rewrite, the semantics of touch_nmi_watchdog and touch_softlockup_watchdog changed enough to break some drivers (mostly over preemptable regions).
This change brings those touch_*_watchdog functions back in line to how they used to work.
--- kernel/watchdog.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/watchdog.c +12 -5
@@ -122,7 +122,7 @@ static void __touch_watchdog(void)void touch_softlockup_watchdog (void ) { -__get_cpu_var (watchdog_touch_ts) =0 ; +__raw_get_cpu_var (watchdog_touch_ts) =0 ; }EXPORT_SYMBOL (touch_softlockup_watchdog);
@@ -142,7 +142,14 @@ void touch_all_softlockup_watchdogs(void)#ifdef CONFIG_HARDLOCKUP_DETECTOR void touch_nmi_watchdog (void ) { -__get_cpu_var (watchdog_nmi_touch) = true; +if (watchdog_enabled) { +unsigned cpu; + +for_each_present_cpu (cpu) { +if (per_cpu (watchdog_nmi_touch, cpu) != true) +per_cpu (watchdog_nmi_touch, cpu) = true; + } + }touch_softlockup_watchdog (); }EXPORT_SYMBOL (touch_nmi_watchdog);
@@ -433,6 +440,9 @@ static int watchdog_enable(int cpu)wake_up_process (p); } +/* if any cpu succeeds, watchdog is considered enabled for the system */ + watchdog_enabled =1 ; +return 0 ; }
@@ -455,9 +465,6 @@ static void watchdog_disable(int cpu)per_cpu (softlockup_watchdog, cpu) =NULL ;kthread_stop (p); } - -/* if any cpu succeeds, watchdog is considered enabled for the system */ - watchdog_enabled =1 ; }static void watchdog_enable_all_cpus (void )
Signature · 1 line
1.7.2.2
Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics
kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151
Quoted 10 lines · Don Zickus
void
touch_nmi_watchdog(void)
{kernel/watchdog.c +7
-1-__get_cpu_var (watchdog_nmi_touch) = true; +if (watchdog_enabled) { +unsigned cpu; + +for_each_present_cpu (cpu) { +if (per_cpu (watchdog_nmi_touch, cpu) != true) +per_cpu (watchdog_nmi_touch, cpu) = true; + }
Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a nr_cpus loop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.
Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.
Thanks,
Ingo
Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics
Quoted 18 lines · Ingo Molnar
kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151
Quoted 10 lines · Don Zickus
void
touch_nmi_watchdog(void)
{kernel/watchdog.c +7
-1-__get_cpu_var (watchdog_nmi_touch) = true; +if (watchdog_enabled) { +unsigned cpu; + +for_each_present_cpu (cpu) { +if (per_cpu (watchdog_nmi_touch, cpu) != true) +per_cpu (watchdog_nmi_touch, cpu) = true; + }Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a
nr_cpusloop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.
Quoted 8 lines
Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.
Thanks,
Ingo
I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.
Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics
Quoted 23 lines · Cyrill Gorcunov
Quoted 22 lines · Ingo Molnar
kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151
Quoted 10 lines · Don Zickus
void
touch_nmi_watchdog(void)
{kernel/watchdog.c +7
-1-__get_cpu_var (watchdog_nmi_touch) = true; +if (watchdog_enabled) { +unsigned cpu; + +for_each_present_cpu (cpu) { +if (per_cpu (watchdog_nmi_touch, cpu) != true) +per_cpu (watchdog_nmi_touch, cpu) = true; + }Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a
nr_cpusloop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.
Quoted 8 lines
Quoted 4 lines
Thanks,
Ingo
I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.
Well, we can't drop it or make per-cpu specific, for example we need it in case of panic with watchdog enabled and panic timeout set, or boot delay set and etc. Seems same applies to printk_delay. Hmm...
Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics
Quoted 28 lines · Cyrill Gorcunov
Quoted 27 lines · Cyrill Gorcunov
Quoted 26 lines · Ingo Molnar
kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151
Quoted 10 lines · Don Zickus
void
touch_nmi_watchdog(void)
{kernel/watchdog.c +7
-1-__get_cpu_var (watchdog_nmi_touch) = true; +if (watchdog_enabled) { +unsigned cpu; + +for_each_present_cpu (cpu) { +if (per_cpu (watchdog_nmi_touch, cpu) != true) +per_cpu (watchdog_nmi_touch, cpu) = true; + }Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a
nr_cpusloop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.
Thanks,
Ingo
Quoted 7 lines
Quoted 3 lines
I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.
Well, we can't drop it or make per-cpu specific, for example we need it in case of panic with watchdog enabled and panic timeout set, or boot delay set and etc. Seems same applies to
printk_delay. Hmm...
Ok - can you cite the old watchdog code, did it really do a nr_cpus loop?
Thanks,
Ingo
Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics
Quoted 39 lines · Ingo Molnar
Quoted 28 lines · Cyrill Gorcunov
Quoted 27 lines · Cyrill Gorcunov
Quoted 26 lines · Ingo Molnar
Quoted 10 lines · Don Zickus
void
touch_nmi_watchdog(void)
{
-__get_cpu_var(watchdog_nmi_touch)= true;
+ if (watchdog_enabled) {
+ unsigned cpu;
+
+for_each_present_cpu(cpu){
+ if (per_cpu(watchdog_nmi_touch, cpu)!= true)
+per_cpu(watchdog_nmi_touch, cpu)= true;
+ }Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a
nr_cpusloop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.
Thanks,
Ingo
Quoted 7 lines
Quoted 3 lines
I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.
Well, we can't drop it or make per-cpu specific, for example we need it in case of panic with watchdog enabled and panic timeout set, or boot delay set and etc. Seems same applies to
printk_delay. Hmm...
Quoted 7 lines
Ok - can you cite the old watchdog code, did it really do a
nr_cpusloop?Thanks,
Ingo
Yes, previous touch_nmi_watchdog really did a loop as for_each_present_cpu and touching per-cpu variable
(bad format)
void touch_nmi_watchdog(void)
{
if (nmi_watchdog_active()) {
unsigned cpu;
for_each_present_cpu(cpu) {
if (per_cpu(nmi_touch,cpu) != 1)
per_cpu(nmi_touch, cpu) = 1;
} }touch_softlockup_watchdog();
}
[tip:perf/urgent] lockup_detector: Sync touch_*_watchdog back to old semantics
In reply to Don ZickusCommit-ID: 68d3f1d810500e8b975bdf0b20dd83d060076b4b Gitweb: http://git.kernel.org/tip/68d3f1d810500e8b975bdf0b20dd83d060076b4b Author: Don Zickus <[email protected]> AuthorDate: Tue, 31 Aug 2010 23:00:07 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 1 Sep 2010 10:02:28 +0200
lockup_detector: Sync touch_*_watchdog back to old semantics
During my rewrite, the semantics of touch_nmi_watchdog and touch_softlockup_watchdog changed enough to break some drivers (mostly over preemptable regions).
These are cases where long delays on one CPU (due to print_delay for example) can cause long delays on other CPUs - so we must 'touch' the nmi_watchdog flag of those other CPUs as well.
This change brings those touch_*_watchdog() functions back in line with to how they used to work.
LKML-Reference: <[email protected]>
--- kernel/watchdog.c | 17 ++++++++++++----- 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/watchdog.c +12 -5
@@ -122,7 +122,7 @@ static void __touch_watchdog(void)void touch_softlockup_watchdog (void ) { -__get_cpu_var (watchdog_touch_ts) =0 ; +__raw_get_cpu_var (watchdog_touch_ts) =0 ; }EXPORT_SYMBOL (touch_softlockup_watchdog);
@@ -142,7 +142,14 @@ void touch_all_softlockup_watchdogs(void)#ifdef CONFIG_HARDLOCKUP_DETECTOR void touch_nmi_watchdog (void ) { -__get_cpu_var (watchdog_nmi_touch) = true; +if (watchdog_enabled) { +unsigned cpu; + +for_each_present_cpu (cpu) { +if (per_cpu (watchdog_nmi_touch, cpu) != true) +per_cpu (watchdog_nmi_touch, cpu) = true; + } + }touch_softlockup_watchdog (); }EXPORT_SYMBOL (touch_nmi_watchdog);
@@ -433,6 +440,9 @@ static int watchdog_enable(int cpu)wake_up_process (p); } +/* if any cpu succeeds, watchdog is considered enabled for the system */ + watchdog_enabled =1 ; +return 0 ; }
@@ -455,9 +465,6 @@ static void watchdog_disable(int cpu)per_cpu (softlockup_watchdog, cpu) =NULL ;kthread_stop (p); } - -/* if any cpu succeeds, watchdog is considered enabled for the system */ - watchdog_enabled =1 ; }static void watchdog_enable_all_cpus (void )
PATCH 2/3 lockup_detector: convert cpu notifier to return encapsulate errno value
In reply to Don ZickusFrom: Akinobu Mita <[email protected]>
By the commit e6bde73b07edeb703d4c89c1daabc09c303de11f ("cpu-hotplug: return better errno on cpu hotplug failure"), the cpu notifier can return encapsulate errno value.
This converts the cpu notifier to return encapsulate errno value for lockup_detector.
--- kernel/watchdog.c | 21 +++++++++++---------- 1 files changed, 11 insertions(+), 10 deletions(-)
kernel/watchdog.c +11 -10
@@ -378,7 +378,7 @@ static int watchdog_nmi_enable(int cpu) }printk (KERN_ERR "NMI watchdog failed to create perf event on cpu%i: %p\n" , cpu, event); -return -1 ; +return PTR_ERR (event);/* success path */ out_save:
@@ -422,17 +422,19 @@ static int watchdog_prepare_cpu(int cpu)static int watchdog_enable (int cpu) {struct task_struct *p =per_cpu (softlockup_watchdog, cpu); +int err;/* enable the perf event */ -if (watchdog_nmi_enable (cpu) !=0 ) -return -1 ; + err =watchdog_nmi_enable (cpu); +if (err) +return err;/* create the watchdog thread */ if (!p) { p =kthread_create (watchdog, (void *)(unsigned long )cpu,"watchdog/%d" , cpu);if (IS_ERR (p)) {printk (KERN_ERR "softlockup watchdog for %i failed\n" , cpu); -return -1 ; +return PTR_ERR (p); }kthread_bind (p, cpu);per_cpu (watchdog_touch_ts, cpu) =0 ;
@@ -526,17 +528,16 @@ static int __cpuinit cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) { int hotcpu = (unsigned long)hcpu; +int err =0 ;switch (action) {case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : -if (watchdog_prepare_cpu (hotcpu)) -return NOTIFY_BAD ; + err =watchdog_prepare_cpu (hotcpu);break ;case CPU_ONLINE :case CPU_ONLINE_FROZEN : -if (watchdog_enable (hotcpu)) -return NOTIFY_BAD ; + err =watchdog_enable (hotcpu);break ;#ifdef CONFIG_HOTPLUG_CPU case CPU_UP_CANCELED :
@@ -549,7 +550,7 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)break ;#endif /* CONFIG_HOTPLUG_CPU */ } -return NOTIFY_OK ; +return notifier_from_errno (err); }static struct notifier_block __cpuinitdata cpu_nfb = {
@@ -565,7 +566,7 @@ static int __init spawn_watchdog_task(void)return 0 ; err =cpu_callback (&cpu_nfb,CPU_UP_PREPARE , cpu); -WARN_ON (err ==NOTIFY_BAD ); +WARN_ON (notifier_to_errno (err));cpu_callback (&cpu_nfb,CPU_ONLINE , cpu);register_cpu_notifier (&cpu_nfb);
Signature · 1 line
1.7.2.2
[tip:perf/core] lockup_detector: Convert cpu notifier to return encapsulate errno value
Commit-ID: eac243355a99d6b9d41bbeba4fc83e7f735485f9 Gitweb: http://git.kernel.org/tip/eac243355a99d6b9d41bbeba4fc83e7f735485f9 Author: Akinobu Mita <[email protected]> AuthorDate: Tue, 31 Aug 2010 23:00:08 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 1 Sep 2010 07:33:34 +0200
lockup_detector: Convert cpu notifier to return encapsulate errno value
By the commit e6bde73b07edeb703d4c89c1daabc09c303de11f ("cpu-hotplug: return better errno on cpu hotplug failure"), the cpu notifier can return encapsulate errno value, resulting in more meaningful error codes for CPU hotplug failures.
This converts the cpu notifier to return encapsulate errno value for the lockup_detector as well.
LKML-Reference: <[email protected]>
--- kernel/watchdog.c | 21 +++++++++++---------- 1 files changed, 11 insertions(+), 10 deletions(-)
kernel/watchdog.c +11 -10
@@ -371,7 +371,7 @@ static int watchdog_nmi_enable(int cpu) }printk (KERN_ERR "NMI watchdog failed to create perf event on cpu%i: %p\n" , cpu, event); -return -1 ; +return PTR_ERR (event);/* success path */ out_save:
@@ -415,17 +415,19 @@ static int watchdog_prepare_cpu(int cpu)static int watchdog_enable (int cpu) {struct task_struct *p =per_cpu (softlockup_watchdog, cpu); +int err;/* enable the perf event */ -if (watchdog_nmi_enable (cpu) !=0 ) -return -1 ; + err =watchdog_nmi_enable (cpu); +if (err) +return err;/* create the watchdog thread */ if (!p) { p =kthread_create (watchdog, (void *)(unsigned long )cpu,"watchdog/%d" , cpu);if (IS_ERR (p)) {printk (KERN_ERR "softlockup watchdog for %i failed\n" , cpu); -return -1 ; +return PTR_ERR (p); }kthread_bind (p, cpu);per_cpu (watchdog_touch_ts, cpu) =0 ;
@@ -519,17 +521,16 @@ static int __cpuinit cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu) { int hotcpu = (unsigned long)hcpu; +int err =0 ;switch (action) {case CPU_UP_PREPARE :case CPU_UP_PREPARE_FROZEN : -if (watchdog_prepare_cpu (hotcpu)) -return NOTIFY_BAD ; + err =watchdog_prepare_cpu (hotcpu);break ;case CPU_ONLINE :case CPU_ONLINE_FROZEN : -if (watchdog_enable (hotcpu)) -return NOTIFY_BAD ; + err =watchdog_enable (hotcpu);break ;#ifdef CONFIG_HOTPLUG_CPU case CPU_UP_CANCELED :
@@ -542,7 +543,7 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)break ;#endif /* CONFIG_HOTPLUG_CPU */ } -return NOTIFY_OK ; +return notifier_from_errno (err); }static struct notifier_block __cpuinitdata cpu_nfb = {
@@ -558,7 +559,7 @@ static int __init spawn_watchdog_task(void)return 0 ; err =cpu_callback (&cpu_nfb,CPU_UP_PREPARE , cpu); -WARN_ON (err ==NOTIFY_BAD ); +WARN_ON (notifier_to_errno (err));cpu_callback (&cpu_nfb,CPU_ONLINE , cpu);register_cpu_notifier (&cpu_nfb);
PATCH 3/3 lockup_detector: remove unnecessary panic_notifier
In reply to Don ZickusFrom: Akinobu Mita <[email protected]>
The panic notifer in lockup_detector just set did_panic to 1. But did_panic is not used anywhere so we can just remove it.
--- kernel/watchdog.c | 15 --------------- 1 files changed, 0 insertions(+), 15 deletions(-)
kernel/watchdog.c +0 -15
@@ -43,7 +43,6 @@ static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts_saved); static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);#endif -static int __read_mostly did_panic;static int __initdata no_watchdog;
@@ -187,18 +186,6 @@ static int is_softlockup(unsigned long touch_ts)return 0 ; } -static int -watchdog_panic (struct notifier_block *this,unsigned long event,void *ptr) -{ - did_panic =1 ; - -return NOTIFY_DONE ; -} - -static struct notifier_block panic_block = { - .notifier_call = watchdog_panic, -}; -#ifdef CONFIG_HARDLOCKUP_DETECTOR static struct perf_event_attr wd_hw_attr = { .type = PERF_TYPE_HARDWARE,
@@ -571,8 +558,6 @@ static int __init spawn_watchdog_task(void)cpu_callback (&cpu_nfb,CPU_ONLINE , cpu);register_cpu_notifier (&cpu_nfb); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); -return 0 ; }early_initcall (spawn_watchdog_task);
Signature · 1 line
1.7.2.2
[tip:perf/core] lockup_detector: Remove unused panic_notifier
Commit-ID: 14416c35b6b9975c9593d7ecc8382d1ecaa0b598 Gitweb: http://git.kernel.org/tip/14416c35b6b9975c9593d7ecc8382d1ecaa0b598 Author: Akinobu Mita <[email protected]> AuthorDate: Tue, 31 Aug 2010 23:00:09 -0400 Committer: Ingo Molnar <[email protected]> CommitDate: Wed, 1 Sep 2010 07:33:34 +0200
lockup_detector: Remove unused panic_notifier
The panic notifer in lockup_detector just set did_panic to 1. But did_panic is not used anywhere so we can just remove it.
LKML-Reference: <[email protected]>
--- kernel/watchdog.c | 15 --------------- 1 files changed, 0 insertions(+), 15 deletions(-)
kernel/watchdog.c +0 -15
@@ -43,7 +43,6 @@ static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts_saved); static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);#endif -static int __read_mostly did_panic;static int __initdata no_watchdog;
@@ -180,18 +179,6 @@ static int is_softlockup(unsigned long touch_ts)return 0 ; } -static int -watchdog_panic (struct notifier_block *this,unsigned long event,void *ptr) -{ - did_panic =1 ; - -return NOTIFY_DONE ; -} - -static struct notifier_block panic_block = { - .notifier_call = watchdog_panic, -}; -#ifdef CONFIG_HARDLOCKUP_DETECTOR static struct perf_event_attr wd_hw_attr = { .type = PERF_TYPE_HARDWARE,
@@ -564,8 +551,6 @@ static int __init spawn_watchdog_task(void)cpu_callback (&cpu_nfb,CPU_ONLINE , cpu);register_cpu_notifier (&cpu_nfb); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); -return 0 ; }early_initcall (spawn_watchdog_task);