lockup_detector: Cure hotplug deadlocks and replace duct tape

The lockup detector is broken is several ways:

- It's deadlock prone vs. CPU hotplug in various ways. Some of these are due to recursive cpus_read_lock() others are due to cpus_read_lock() from CPU hotplug callbacks which immediately lock the machine because cpus are write locked.

- The handling of the cpu hotplug threads happens sideways to the smpboot thread infrastructure, which is racy and pointless

- The handling of the user space sysctl interface is a complete trainwreck as it fiddles directly with variables which can be modified or evaluated by the running watchdogs.

- The perf event initialization is a trainwreck as it tries to create perf events over and over even if perf is not functional (no hardware, ....). To avoid excessive dmesg spam it contains magic printk ratelimiting along with either wrong or useless messages.

- The code structure is horrible as ifdef sections are scattered all over the place which makes it unreadable

- There is more wreckage, but see the changelogs for the ugly details.

The following series sanitizes the facility and addresses the problems.

Changes since V1:

- Wrapped the perf specific calls into the weak watchdog_nmi_* functions

- Fixed the compile error pointed out by Don

- Fixed the reconfiguration parameter inconsistency which broke powerpc

- Picked up the updated version of patch 11/29

Delta patch below.

Thanks,

tglx
---
Diffstat for the series:

 arch/parisc/kernel/process.c   |    2 
 arch/powerpc/kernel/watchdog.c |   22 -
 arch/x86/events/intel/core.c   |   11 
 include/linux/nmi.h            |  121 +++----
 include/linux/smpboot.h        |    4 
 kernel/cpu.c                   |    6 
 kernel/smpboot.c               |   22 -
 kernel/sysctl.c                |   22 -
 kernel/watchdog.c              |  633 ++++++++++++++---------------------------
 kernel/watchdog_hld.c          |  193 ++++++------
 10 files changed, 434 insertions(+), 602 deletions(-)

Delta patch vs. V1
8<------------------------

@@ -355,12 +355,12 @@ static void watchdog_calc_timeouts(void)
 	wd_timer_period_ms = watchdog_thresh * 1000 * 2 / 5;
 }
 
-void watchdog_nmi_reconfigure(bool stop)
+void watchdog_nmi_reconfigure(bool run)
 {
 	int cpu;
 
 	cpus_read_lock();
-	if (stop) {
+	if (!run) {
 		for_each_cpu(cpu, &wd_cpus_enabled)
 			stop_wd_on_cpu(cpu);
 	} else {

@@ -102,7 +102,7 @@ static inline void hardlockup_detector_p
 static inline void hardlockup_detector_perf_enable(void) { }
 static inline void hardlockup_detector_perf_cleanup(void) { }
 # if !defined(CONFIG_HAVE_NMI_WATCHDOG)
-static int hardlockup_detector_perf_init(void) { return -ENODEV; }
+static inline int hardlockup_detector_perf_init(void) { return -ENODEV; }
 static inline void arch_touch_nmi_watchdog(void) {}
 # else
 static int hardlockup_detector_perf_init(void) { return 0; }