Skip to contentSkip to conversation

lockup detector fixes

PATCH In review Don Zickus 4 participants 12 messages Sep 1, 2010

lore
Share & tools
lore

Don Zickus

author
#0 lore raw

cover lockup detector fixes

A bunch of fixes for the lockup detector.

Tested on intel/amd on top of ingo's tip branch.

Akinobu Mita (2):
lockup_detector: convert cpu notifier to return encapsulate errno
value
lockup_detector: remove unnecessary panic_notifier

Don Zickus (1):
[lockup detector] sync touch_*_watchdog back to old semantics

 kernel/watchdog.c |   53 +++++++++++++++++++++++------------------------------
 1 files changed, 23 insertions(+), 30 deletions(-)
Signature · 1 line
1.7.2.2
author
#1 lore raw

PATCH 1/3 sync touch_*_watchdog back to old semantics

During my rewrite, the semantics of touch_nmi_watchdog and touch_softlockup_watchdog changed enough to break some drivers (mostly over preemptable regions).

This change brings those touch_*_watchdog functions back in line to how they used to work.

---
 kernel/watchdog.c |   17 ++++++++++++-----
 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/watchdog.c +12 -5
index 0d53c8e..7f9c3c5 100644
@@ -122,7 +122,7 @@ static void __touch_watchdog(void)
 
 void touch_softlockup_watchdog(void)
 {
-	__get_cpu_var(watchdog_touch_ts) = 0;
+	__raw_get_cpu_var(watchdog_touch_ts) = 0;
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
@@ -142,7 +142,14 @@ void touch_all_softlockup_watchdogs(void)
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 void touch_nmi_watchdog(void)
 {
-	__get_cpu_var(watchdog_nmi_touch) = true;
+	if (watchdog_enabled) {
+		unsigned cpu;
+
+		for_each_present_cpu(cpu) {
+			if (per_cpu(watchdog_nmi_touch, cpu) != true)
+				per_cpu(watchdog_nmi_touch, cpu) = true;
+		}
+	}
 	touch_softlockup_watchdog();
 }
 EXPORT_SYMBOL(touch_nmi_watchdog);
@@ -433,6 +440,9 @@ static int watchdog_enable(int cpu)
 		wake_up_process(p);
 	}
 
+	/* if any cpu succeeds, watchdog is considered enabled for the system */
+	watchdog_enabled = 1;
+
 	return 0;
 }
 
@@ -455,9 +465,6 @@ static void watchdog_disable(int cpu)
 		per_cpu(softlockup_watchdog, cpu) = NULL;
 		kthread_stop(p);
 	}
-
-	/* if any cpu succeeds, watchdog is considered enabled for the system */
-	watchdog_enabled = 1;
 }
 
 static void watchdog_enable_all_cpus(void)
Signature · 1 line
1.7.2.2
#2 lore raw

Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics

kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151

Quoted 10 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +7 -1
-	__get_cpu_var(watchdog_nmi_touch) = true;
+	if (watchdog_enabled) {
+		unsigned cpu;
+
+		for_each_present_cpu(cpu) {
+			if (per_cpu(watchdog_nmi_touch, cpu) != true)
+				per_cpu(watchdog_nmi_touch, cpu) = true;
+		}

Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a nr_cpus loop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.

Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.

Thanks,

Ingo

#3 lore raw

Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics

Quoted 18 lines · Ingo Molnar

kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151

Quoted 10 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +7 -1
-	__get_cpu_var(watchdog_nmi_touch) = true;
+	if (watchdog_enabled) {
+		unsigned cpu;
+
+		for_each_present_cpu(cpu) {
+			if (per_cpu(watchdog_nmi_touch, cpu) != true)
+				per_cpu(watchdog_nmi_touch, cpu) = true;
+		}

Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a nr_cpus loop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.

Quoted 8 lines

Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.

Thanks,

Ingo

I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.

#4 lore raw

Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics

Quoted 23 lines · Cyrill Gorcunov
Quoted 22 lines · Ingo Molnar

kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151

Quoted 10 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +7 -1
-	__get_cpu_var(watchdog_nmi_touch) = true;
+	if (watchdog_enabled) {
+		unsigned cpu;
+
+		for_each_present_cpu(cpu) {
+			if (per_cpu(watchdog_nmi_touch, cpu) != true)
+				per_cpu(watchdog_nmi_touch, cpu) = true;
+		}

Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a nr_cpus loop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.

Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.

Quoted 8 lines
Quoted 4 lines

Thanks,

Ingo

I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.

Well, we can't drop it or make per-cpu specific, for example we need it in case of panic with watchdog enabled and panic timeout set, or boot delay set and etc. Seems same applies to printk_delay. Hmm...

#5 lore raw

Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics

Quoted 28 lines · Cyrill Gorcunov
Quoted 27 lines · Cyrill Gorcunov
Quoted 26 lines · Ingo Molnar

kernel/watchdog.c void touch_all_softlockup_watchdogs(void) · line 151

Quoted 10 lines · Don Zickus

void touch_nmi_watchdog(void)
{

kernel/watchdog.c +7 -1
-	__get_cpu_var(watchdog_nmi_touch) = true;
+	if (watchdog_enabled) {
+		unsigned cpu;
+
+		for_each_present_cpu(cpu) {
+			if (per_cpu(watchdog_nmi_touch, cpu) != true)
+				per_cpu(watchdog_nmi_touch, cpu) = true;
+		}

Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a nr_cpus loop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.

Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.

Thanks,

Ingo

Quoted 7 lines
Quoted 3 lines

I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.

Well, we can't drop it or make per-cpu specific, for example we need it in case of panic with watchdog enabled and panic timeout set, or boot delay set and etc. Seems same applies to printk_delay. Hmm...

Ok - can you cite the old watchdog code, did it really do a nr_cpus loop?

Thanks,

Ingo

#6 lore raw

Re: [PATCH 1/3] [lockup detector] sync touch_*_watchdog back to old semantics

Quoted 39 lines · Ingo Molnar
Quoted 28 lines · Cyrill Gorcunov
Quoted 27 lines · Cyrill Gorcunov
Quoted 26 lines · Ingo Molnar
Quoted 10 lines · Don Zickus

 void touch_nmi_watchdog(void)
 {
- __get_cpu_var(watchdog_nmi_touch) = true;
+ if (watchdog_enabled) {
+         unsigned cpu;
+
+         for_each_present_cpu(cpu) {
+                 if (per_cpu(watchdog_nmi_touch, cpu) != true)
+                         per_cpu(watchdog_nmi_touch, cpu) = true;
+         }

Hm, this is going to be a scalability nightmare with lots of CPUs. Not only do we have a nr_cpus loop, but we touch per-cpu areas of _other_ CPUs - a big scalability nono.

Why do we need to do this? We never needed to touch other CPU's NMI lockup accounting data areas - why has this changed? The changelog does not explain this.

Thanks,

   Ingo

Quoted 7 lines
Quoted 3 lines

I believe this came from old nmi watchdog code where it might be useful when nmi watchdog activated via io-apic. I'm trying to figure out if we really need it still.

Well, we can't drop it or make per-cpu specific, for example we need it in case of panic with watchdog enabled and panic timeout set, or boot delay set and etc. Seems same applies to printk_delay. Hmm...

Quoted 7 lines

Ok - can you cite the old watchdog code, did it really do a nr_cpus loop?

Thanks,

       Ingo

Yes, previous touch_nmi_watchdog really did a loop as for_each_present_cpu and touching per-cpu variable

(bad format)

void touch_nmi_watchdog(void)
{
if (nmi_watchdog_active()) {
unsigned cpu;

  for_each_present_cpu(cpu) {
    if (per_cpu(nmi_touch,cpu) != 1)
       per_cpu(nmi_touch, cpu) = 1;
    }

}
touch_softlockup_watchdog();
}

#7 lore raw

[tip:perf/urgent] lockup_detector: Sync touch_*_watchdog back to old semantics

In reply to Don Zickus
Commit-ID:  68d3f1d810500e8b975bdf0b20dd83d060076b4b
Gitweb:     http://git.kernel.org/tip/68d3f1d810500e8b975bdf0b20dd83d060076b4b
Author:     Don Zickus <[email protected]>
AuthorDate: Tue, 31 Aug 2010 23:00:07 -0400
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 1 Sep 2010 10:02:28 +0200

lockup_detector: Sync touch_*_watchdog back to old semantics

During my rewrite, the semantics of touch_nmi_watchdog and touch_softlockup_watchdog changed enough to break some drivers (mostly over preemptable regions).

These are cases where long delays on one CPU (due to print_delay for example) can cause long delays on other CPUs - so we must 'touch' the nmi_watchdog flag of those other CPUs as well.

This change brings those touch_*_watchdog() functions back in line with to how they used to work.

LKML-Reference: <[email protected]>

---
 kernel/watchdog.c |   17 ++++++++++++-----
 1 files changed, 12 insertions(+), 5 deletions(-)
kernel/watchdog.c +12 -5
index 0d53c8e..7f9c3c5 100644
@@ -122,7 +122,7 @@ static void __touch_watchdog(void)
 
 void touch_softlockup_watchdog(void)
 {
-	__get_cpu_var(watchdog_touch_ts) = 0;
+	__raw_get_cpu_var(watchdog_touch_ts) = 0;
 }
 EXPORT_SYMBOL(touch_softlockup_watchdog);
 
@@ -142,7 +142,14 @@ void touch_all_softlockup_watchdogs(void)
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 void touch_nmi_watchdog(void)
 {
-	__get_cpu_var(watchdog_nmi_touch) = true;
+	if (watchdog_enabled) {
+		unsigned cpu;
+
+		for_each_present_cpu(cpu) {
+			if (per_cpu(watchdog_nmi_touch, cpu) != true)
+				per_cpu(watchdog_nmi_touch, cpu) = true;
+		}
+	}
 	touch_softlockup_watchdog();
 }
 EXPORT_SYMBOL(touch_nmi_watchdog);
@@ -433,6 +440,9 @@ static int watchdog_enable(int cpu)
 		wake_up_process(p);
 	}
 
+	/* if any cpu succeeds, watchdog is considered enabled for the system */
+	watchdog_enabled = 1;
+
 	return 0;
 }
 
@@ -455,9 +465,6 @@ static void watchdog_disable(int cpu)
 		per_cpu(softlockup_watchdog, cpu) = NULL;
 		kthread_stop(p);
 	}
-
-	/* if any cpu succeeds, watchdog is considered enabled for the system */
-	watchdog_enabled = 1;
 }
 
 static void watchdog_enable_all_cpus(void)
author
#8 lore raw

PATCH 2/3 lockup_detector: convert cpu notifier to return encapsulate errno value

In reply to Don Zickus

From: Akinobu Mita <[email protected]>

By the commit e6bde73b07edeb703d4c89c1daabc09c303de11f ("cpu-hotplug: return better errno on cpu hotplug failure"), the cpu notifier can return encapsulate errno value.

This converts the cpu notifier to return encapsulate errno value for lockup_detector.

---
 kernel/watchdog.c |   21 +++++++++++----------
 1 files changed, 11 insertions(+), 10 deletions(-)
kernel/watchdog.c +11 -10
index 7f9c3c5..db70647 100644
@@ -378,7 +378,7 @@ static int watchdog_nmi_enable(int cpu)
 	}
 
 	printk(KERN_ERR "NMI watchdog failed to create perf event on cpu%i: %p\n", cpu, event);
-	return -1;
+	return PTR_ERR(event);
 
 	/* success path */
 out_save:
@@ -422,17 +422,19 @@ static int watchdog_prepare_cpu(int cpu)
 static int watchdog_enable(int cpu)
 {
 	struct task_struct *p = per_cpu(softlockup_watchdog, cpu);
+	int err;
 
 	/* enable the perf event */
-	if (watchdog_nmi_enable(cpu) != 0)
-		return -1;
+	err = watchdog_nmi_enable(cpu);
+	if (err)
+		return err;
 
 	/* create the watchdog thread */
 	if (!p) {
 		p = kthread_create(watchdog, (void *)(unsigned long)cpu, "watchdog/%d", cpu);
 		if (IS_ERR(p)) {
 			printk(KERN_ERR "softlockup watchdog for %i failed\n", cpu);
-			return -1;
+			return PTR_ERR(p);
 		}
 		kthread_bind(p, cpu);
 		per_cpu(watchdog_touch_ts, cpu) = 0;
@@ -526,17 +528,16 @@ static int __cpuinit
 cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 {
 	int hotcpu = (unsigned long)hcpu;
+	int err = 0;
 
 	switch (action) {
 	case CPU_UP_PREPARE:
 	case CPU_UP_PREPARE_FROZEN:
-		if (watchdog_prepare_cpu(hotcpu))
-			return NOTIFY_BAD;
+		err = watchdog_prepare_cpu(hotcpu);
 		break;
 	case CPU_ONLINE:
 	case CPU_ONLINE_FROZEN:
-		if (watchdog_enable(hotcpu))
-			return NOTIFY_BAD;
+		err = watchdog_enable(hotcpu);
 		break;
 #ifdef CONFIG_HOTPLUG_CPU
 	case CPU_UP_CANCELED:
@@ -549,7 +550,7 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 		break;
 #endif /* CONFIG_HOTPLUG_CPU */
 	}
-	return NOTIFY_OK;
+	return notifier_from_errno(err);
 }
 
 static struct notifier_block __cpuinitdata cpu_nfb = {
@@ -565,7 +566,7 @@ static int __init spawn_watchdog_task(void)
 		return 0;
 
 	err = cpu_callback(&cpu_nfb, CPU_UP_PREPARE, cpu);
-	WARN_ON(err == NOTIFY_BAD);
+	WARN_ON(notifier_to_errno(err));
 
 	cpu_callback(&cpu_nfb, CPU_ONLINE, cpu);
 	register_cpu_notifier(&cpu_nfb);
Signature · 1 line
1.7.2.2
#9 lore raw

[tip:perf/core] lockup_detector: Convert cpu notifier to return encapsulate errno value

Commit-ID:  eac243355a99d6b9d41bbeba4fc83e7f735485f9
Gitweb:     http://git.kernel.org/tip/eac243355a99d6b9d41bbeba4fc83e7f735485f9
Author:     Akinobu Mita <[email protected]>
AuthorDate: Tue, 31 Aug 2010 23:00:08 -0400
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 1 Sep 2010 07:33:34 +0200

lockup_detector: Convert cpu notifier to return encapsulate errno value

By the commit e6bde73b07edeb703d4c89c1daabc09c303de11f ("cpu-hotplug: return better errno on cpu hotplug failure"), the cpu notifier can return encapsulate errno value, resulting in more meaningful error codes for CPU hotplug failures.

This converts the cpu notifier to return encapsulate errno value for the lockup_detector as well.

LKML-Reference: <[email protected]>

---
 kernel/watchdog.c |   21 +++++++++++----------
 1 files changed, 11 insertions(+), 10 deletions(-)
kernel/watchdog.c +11 -10
index 0d53c8e..501cb6e 100644
@@ -371,7 +371,7 @@ static int watchdog_nmi_enable(int cpu)
 	}
 
 	printk(KERN_ERR "NMI watchdog failed to create perf event on cpu%i: %p\n", cpu, event);
-	return -1;
+	return PTR_ERR(event);
 
 	/* success path */
 out_save:
@@ -415,17 +415,19 @@ static int watchdog_prepare_cpu(int cpu)
 static int watchdog_enable(int cpu)
 {
 	struct task_struct *p = per_cpu(softlockup_watchdog, cpu);
+	int err;
 
 	/* enable the perf event */
-	if (watchdog_nmi_enable(cpu) != 0)
-		return -1;
+	err = watchdog_nmi_enable(cpu);
+	if (err)
+		return err;
 
 	/* create the watchdog thread */
 	if (!p) {
 		p = kthread_create(watchdog, (void *)(unsigned long)cpu, "watchdog/%d", cpu);
 		if (IS_ERR(p)) {
 			printk(KERN_ERR "softlockup watchdog for %i failed\n", cpu);
-			return -1;
+			return PTR_ERR(p);
 		}
 		kthread_bind(p, cpu);
 		per_cpu(watchdog_touch_ts, cpu) = 0;
@@ -519,17 +521,16 @@ static int __cpuinit
 cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 {
 	int hotcpu = (unsigned long)hcpu;
+	int err = 0;
 
 	switch (action) {
 	case CPU_UP_PREPARE:
 	case CPU_UP_PREPARE_FROZEN:
-		if (watchdog_prepare_cpu(hotcpu))
-			return NOTIFY_BAD;
+		err = watchdog_prepare_cpu(hotcpu);
 		break;
 	case CPU_ONLINE:
 	case CPU_ONLINE_FROZEN:
-		if (watchdog_enable(hotcpu))
-			return NOTIFY_BAD;
+		err = watchdog_enable(hotcpu);
 		break;
 #ifdef CONFIG_HOTPLUG_CPU
 	case CPU_UP_CANCELED:
@@ -542,7 +543,7 @@ cpu_callback(struct notifier_block *nfb, unsigned long action, void *hcpu)
 		break;
 #endif /* CONFIG_HOTPLUG_CPU */
 	}
-	return NOTIFY_OK;
+	return notifier_from_errno(err);
 }
 
 static struct notifier_block __cpuinitdata cpu_nfb = {
@@ -558,7 +559,7 @@ static int __init spawn_watchdog_task(void)
 		return 0;
 
 	err = cpu_callback(&cpu_nfb, CPU_UP_PREPARE, cpu);
-	WARN_ON(err == NOTIFY_BAD);
+	WARN_ON(notifier_to_errno(err));
 
 	cpu_callback(&cpu_nfb, CPU_ONLINE, cpu);
 	register_cpu_notifier(&cpu_nfb);
author
#10 lore raw

PATCH 3/3 lockup_detector: remove unnecessary panic_notifier

In reply to Don Zickus

From: Akinobu Mita <[email protected]>

The panic notifer in lockup_detector just set did_panic to 1. But did_panic is not used anywhere so we can just remove it.

---
 kernel/watchdog.c |   15 ---------------
 1 files changed, 0 insertions(+), 15 deletions(-)
kernel/watchdog.c +0 -15
index db70647..fa71aeb 100644
@@ -43,7 +43,6 @@ static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts_saved);
 static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
 #endif
 
-static int __read_mostly did_panic;
 static int __initdata no_watchdog;
 
 
@@ -187,18 +186,6 @@ static int is_softlockup(unsigned long touch_ts)
 	return 0;
 }
 
-static int
-watchdog_panic(struct notifier_block *this, unsigned long event, void *ptr)
-{
-	did_panic = 1;
-
-	return NOTIFY_DONE;
-}
-
-static struct notifier_block panic_block = {
-	.notifier_call = watchdog_panic,
-};
-
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 static struct perf_event_attr wd_hw_attr = {
 	.type		= PERF_TYPE_HARDWARE,
@@ -571,8 +558,6 @@ static int __init spawn_watchdog_task(void)
 	cpu_callback(&cpu_nfb, CPU_ONLINE, cpu);
 	register_cpu_notifier(&cpu_nfb);
 
-	atomic_notifier_chain_register(&panic_notifier_list, &panic_block);
-
 	return 0;
 }
 early_initcall(spawn_watchdog_task);
Signature · 1 line
1.7.2.2
#11 lore raw

[tip:perf/core] lockup_detector: Remove unused panic_notifier

Commit-ID:  14416c35b6b9975c9593d7ecc8382d1ecaa0b598
Gitweb:     http://git.kernel.org/tip/14416c35b6b9975c9593d7ecc8382d1ecaa0b598
Author:     Akinobu Mita <[email protected]>
AuthorDate: Tue, 31 Aug 2010 23:00:09 -0400
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 1 Sep 2010 07:33:34 +0200

lockup_detector: Remove unused panic_notifier

The panic notifer in lockup_detector just set did_panic to 1. But did_panic is not used anywhere so we can just remove it.

LKML-Reference: <[email protected]>

---
 kernel/watchdog.c |   15 ---------------
 1 files changed, 0 insertions(+), 15 deletions(-)
kernel/watchdog.c +0 -15
index 501cb6e..5b1ee4f4 100644
@@ -43,7 +43,6 @@ static DEFINE_PER_CPU(unsigned long, hrtimer_interrupts_saved);
 static DEFINE_PER_CPU(struct perf_event *, watchdog_ev);
 #endif
 
-static int __read_mostly did_panic;
 static int __initdata no_watchdog;
 
 
@@ -180,18 +179,6 @@ static int is_softlockup(unsigned long touch_ts)
 	return 0;
 }
 
-static int
-watchdog_panic(struct notifier_block *this, unsigned long event, void *ptr)
-{
-	did_panic = 1;
-
-	return NOTIFY_DONE;
-}
-
-static struct notifier_block panic_block = {
-	.notifier_call = watchdog_panic,
-};
-
 #ifdef CONFIG_HARDLOCKUP_DETECTOR
 static struct perf_event_attr wd_hw_attr = {
 	.type		= PERF_TYPE_HARDWARE,
@@ -564,8 +551,6 @@ static int __init spawn_watchdog_task(void)
 	cpu_callback(&cpu_nfb, CPU_ONLINE, cpu);
 	register_cpu_notifier(&cpu_nfb);
 
-	atomic_notifier_chain_register(&panic_notifier_list, &panic_block);
-
 	return 0;
 }
 early_initcall(spawn_watchdog_task);