Skip to contentSkip to conversation

nmi watchdog: x86 32/64 cleanup

RFC PATCH In review dzickus 4 participants 18 messages May 9, 2006 – May 22, 2006 Proc sysctl

lore
Share & tools
lore

dzickus

author
#0 lore raw

cover nmi watchdog: x86 32/64 cleanup

A major cleanup of the nmi watchdog on the i386 and x86_64 archs.

This little project started out as a way to get unknown_nmi_panic working on an x86_64 SMP machine. But after Andi rejected my dirty little hack ;), it grew into a spring cleaning of the nmi watchdog.

The fix Andi wanted led to a conflict with how oprofile worked. This led to a reservation layer around the performance counters and a bunch of changes in oprofile startup. All the reservation layer consists of is reserving a bit that corresponds to a performance counter. The intent here is to prevent subsystems like oprofile and nmi watchdog from stomping on each other when trying to manipulate these counters. Currently, oprofile must stop the nmi watchdog before using the counters. This approach does not encourage any sharing. :)

During the testing of this reservation layer, I ran into more and more bugs from old hacks (notably in the SMP cases). As a result I found myself rewriting a lot of this code. I think I straightened everything out, except for enabling/disabling the nmi watchdog on an ioapic. The enable_irq/disable_irq macros don't seem to work for me and I don't know enough about the subsystem to figure it out either.

Of course the oprofile code is shared with the i386 arch, so all the changes I did in x86_64 had to be duplicated over there too.

I tested these changes on a Xeon (UP/SMP), P4 (32/64, UP/SMP) and under various boot-up defaults (nmi-watchdog=0,1,2). I also worked on some of these changes with Andi Kleen and the oprofile folks, so these ideas shouldn't be too surprising, just perhaps the implementation. :o

Any thoughts/feedback/criticism to clean this up more and probably fix some of my coding style would be appreciated.

Cheers,
Don

--

author
#1 lore raw

PATCH 1/8 nmi watchdog header cleanup

Misc header cleanup for nmi watchdog.

arch/i386/kernel/apic.c +1 -0
Index: linux-don/arch/i386/kernel/apic.c
===================================================================
@@ -36,6 +36,7 @@
 #include <asm/arch_hooks.h>
 #include <asm/hpet.h>
 #include <asm/i8253.h>
+#include <asm/nmi.h>
 
 #include <mach_apic.h>
 #include <mach_apicdef.h>
arch/i386/kernel/io_apic.c +1 -0
Index: linux-don/arch/i386/kernel/io_apic.c
===================================================================
@@ -38,6 +38,7 @@
 #include <asm/desc.h>
 #include <asm/timer.h>
 #include <asm/i8259.h>
+#include <asm/nmi.h>
 
 #include <mach_apic.h>
 
arch/i386/kernel/nmi.c +1 -6
Index: linux-don/arch/i386/kernel/nmi.c
===================================================================
@@ -14,20 +14,15 @@
  */
 
 #include <linux/config.h>
-#include <linux/mm.h>
 #include <linux/delay.h>
-#include <linux/bootmem.h>
-#include <linux/smp_lock.h>
 #include <linux/interrupt.h>
-#include <linux/mc146818rtc.h>
-#include <linux/kernel_stat.h>
 #include <linux/module.h>
 #include <linux/nmi.h>
 #include <linux/sysdev.h>
 #include <linux/sysctl.h>
+#include <linux/percpu.h>
 
 #include <asm/smp.h>
-#include <asm/div64.h>
 #include <asm/nmi.h>
 
 #include "mach_traps.h"
arch/i386/kernel/smpboot.c +1 -0
Index: linux-don/arch/i386/kernel/smpboot.c
===================================================================
@@ -52,6 +52,7 @@
 #include <asm/tlbflush.h>
 #include <asm/desc.h>
 #include <asm/arch_hooks.h>
+#include <asm/nmi.h>
 
 #include <mach_apic.h>
 #include <mach_wakecpu.h>
arch/i386/oprofile/op_model_athlon.c +1 -0
Index: linux-don/arch/i386/oprofile/op_model_athlon.c
===================================================================
@@ -13,6 +13,7 @@
 #include <linux/oprofile.h>
 #include <asm/ptrace.h>
 #include <asm/msr.h>
+#include <asm/nmi.h>
  
 #include "op_x86_model.h"
 #include "op_counter.h"
arch/i386/oprofile/op_model_p4.c +1 -0
Index: linux-don/arch/i386/oprofile/op_model_p4.c
===================================================================
@@ -14,6 +14,7 @@
 #include <asm/ptrace.h>
 #include <asm/fixmap.h>
 #include <asm/apic.h>
+#include <asm/nmi.h>
 
 #include "op_x86_model.h"
 #include "op_counter.h"
arch/i386/oprofile/op_model_ppro.c +1 -0
Index: linux-don/arch/i386/oprofile/op_model_ppro.c
===================================================================
@@ -14,6 +14,7 @@
 #include <asm/ptrace.h>
 #include <asm/msr.h>
 #include <asm/apic.h>
+#include <asm/nmi.h>
  
 #include "op_x86_model.h"
 #include "op_counter.h"
arch/x86_64/kernel/io_apic.c +1 -0
Index: linux-don/arch/x86_64/kernel/io_apic.c
===================================================================
@@ -41,6 +41,7 @@
 #include <asm/mach_apic.h>
 #include <asm/acpi.h>
 #include <asm/dma.h>
+#include <asm/nmi.h>
 
 #define __apicdebuginit  __init
 
include/asm-i386/apic.h +0 -12
Index: linux-don/include/asm-i386/apic.h
===================================================================
@@ -112,24 +112,12 @@ extern void init_apic_mappings (void);
 extern void smp_local_timer_interrupt (struct pt_regs * regs);
 extern void setup_boot_APIC_clock (void);
 extern void setup_secondary_APIC_clock (void);
-extern void setup_apic_nmi_watchdog (void);
-extern int reserve_lapic_nmi(void);
-extern void release_lapic_nmi(void);
-extern void disable_timer_nmi_watchdog(void);
-extern void enable_timer_nmi_watchdog(void);
-extern void nmi_watchdog_tick (struct pt_regs * regs);
 extern int APIC_init_uniprocessor (void);
 extern void disable_APIC_timer(void);
 extern void enable_APIC_timer(void);
 
 extern void enable_NMI_through_LVT0 (void * dummy);
 
-extern unsigned int nmi_watchdog;
-#define NMI_NONE	0
-#define NMI_IO_APIC	1
-#define NMI_LOCAL_APIC	2
-#define NMI_INVALID	3
-
 extern int disable_timer_pin_1;
 
 void smp_send_timer_broadcast_ipi(struct pt_regs *regs);
include/asm-i386/nmi.h +21 -7
Index: linux-don/include/asm-i386/nmi.h
===================================================================
@@ -5,24 +5,38 @@
 #define ASM_NMI_H
 
 #include <linux/pm.h>
- 
+
 struct pt_regs;
- 
+
 typedef int (*nmi_callback_t)(struct pt_regs * regs, int cpu);
- 
-/** 
+
+/**
  * set_nmi_callback
  *
  * Set a handler for an NMI. Only one handler may be
  * set. Return 1 if the NMI was handled.
  */
 void set_nmi_callback(nmi_callback_t callback);
- 
-/** 
+
+/**
  * unset_nmi_callback
  *
  * Remove the handler previously set.
  */
 void unset_nmi_callback(void);
- 
+
+extern void setup_apic_nmi_watchdog (void);
+extern int reserve_lapic_nmi(void);
+extern void release_lapic_nmi(void);
+extern void disable_timer_nmi_watchdog(void);
+extern void enable_timer_nmi_watchdog(void);
+extern void nmi_watchdog_tick (struct pt_regs * regs);
+
+extern unsigned int nmi_watchdog;
+#define NMI_DEFAULT     -1
+#define NMI_NONE	0
+#define NMI_IO_APIC	1
+#define NMI_LOCAL_APIC	2
+#define NMI_INVALID	3
+
 #endif /* ASM_NMI_H */
include/asm-x86_64/apic.h +0 -16
Index: linux-don/include/asm-x86_64/apic.h
===================================================================
@@ -80,27 +80,11 @@ extern void init_apic_mappings (void);
 extern void smp_local_timer_interrupt (struct pt_regs * regs);
 extern void setup_boot_APIC_clock (void);
 extern void setup_secondary_APIC_clock (void);
-extern void setup_apic_nmi_watchdog (void);
-extern int reserve_lapic_nmi(void);
-extern void release_lapic_nmi(void);
-extern void disable_timer_nmi_watchdog(void);
-extern void enable_timer_nmi_watchdog(void);
-extern void nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
 extern int APIC_init_uniprocessor (void);
 extern void disable_APIC_timer(void);
 extern void enable_APIC_timer(void);
 extern void clustered_apic_check(void);
 
-extern void nmi_watchdog_default(void);
-extern int setup_nmi_watchdog(char *);
-
-extern unsigned int nmi_watchdog;
-#define NMI_DEFAULT	-1
-#define NMI_NONE	0
-#define NMI_IO_APIC	1
-#define NMI_LOCAL_APIC	2
-#define NMI_INVALID	3
-
 extern int disable_timer_pin_1;
 
 extern void setup_threshold_lvt(unsigned long lvt_off);
include/asm-x86_64/nmi.h +24 -6
Index: linux-don/include/asm-x86_64/nmi.h
===================================================================
@@ -5,26 +5,27 @@
 #define ASM_NMI_H
 
 #include <linux/pm.h>
+#include <asm/io.h>
  
 struct pt_regs;
- 
+
 typedef int (*nmi_callback_t)(struct pt_regs * regs, int cpu);
- 
-/** 
+
+/**
  * set_nmi_callback
  *
  * Set a handler for an NMI. Only one handler may be
  * set. Return 1 if the NMI was handled.
  */
 void set_nmi_callback(nmi_callback_t callback);
- 
-/** 
+
+/**
  * unset_nmi_callback
  *
  * Remove the handler previously set.
  */
 void unset_nmi_callback(void);
- 
+
 #ifdef CONFIG_PM
  
 /** Replace the PM callback routine for NMI. */
@@ -56,4 +57,21 @@ extern int unknown_nmi_panic;
 
 extern int check_nmi_watchdog(void);
  
+extern void setup_apic_nmi_watchdog (void);
+extern int reserve_lapic_nmi(void);
+extern void release_lapic_nmi(void);
+extern void disable_timer_nmi_watchdog(void);
+extern void enable_timer_nmi_watchdog(void);
+extern void nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
+
+extern void nmi_watchdog_default(void);
+extern int setup_nmi_watchdog(char *);
+
+extern unsigned int nmi_watchdog;
+#define NMI_DEFAULT	-1
+#define NMI_NONE	0
+#define NMI_IO_APIC	1
+#define NMI_LOCAL_APIC	2
+#define NMI_INVALID	3
+
 #endif /* ASM_NMI_H */
arch/x86_64/kernel/nmi.c +0 -8
Index: linux-don/arch/x86_64/kernel/nmi.c
===================================================================
@@ -15,11 +15,7 @@
 #include <linux/config.h>
 #include <linux/mm.h>
 #include <linux/delay.h>
-#include <linux/bootmem.h>
-#include <linux/smp_lock.h>
 #include <linux/interrupt.h>
-#include <linux/mc146818rtc.h>
-#include <linux/kernel_stat.h>
 #include <linux/module.h>
 #include <linux/sysdev.h>
 #include <linux/nmi.h>
@@ -27,13 +23,9 @@
 #include <linux/kprobes.h>
 
 #include <asm/smp.h>
-#include <asm/mtrr.h>
-#include <asm/mpspec.h>
 #include <asm/nmi.h>
-#include <asm/msr.h>
 #include <asm/proto.h>
 #include <asm/kdebug.h>
-#include <asm/local.h>
 #include <asm/mce.h>
 
 /*

--

author
#2 lore raw

PATCH 2/8 Add performance counter reservation framework for UP kernels

In reply to dzickus

Adds basic infrastructure to allow subsystems to reserve performance counters on the x86 chips. Only UP kernels are supported in this patch to make reviewing easier. The SMP portion makes a lot more changes.

Think of this as a locking mechanism where each bit represents a different counter. In addition, each subsystem should also reserve an appropriate event selection register that will correspond to the performance counter it will be using (this is mainly neccessary for the Pentium 4 chips as they break the 1:1 relationship to performance counters).

This will help prevent subsystems like oprofile from interfering with the nmi watchdog.

arch/x86_64/kernel/nmi.c +138 -40
Index: linux-don/arch/x86_64/kernel/nmi.c
===================================================================
@@ -28,6 +28,20 @@
 #include <asm/kdebug.h>
 #include <asm/mce.h>
 
+/* perfctr_nmi_owner tracks the ownership of the perfctr registers:
+ * evtsel_nmi_owner tracks the ownership of the event selection
+ * - different performance counters/ event selection may be reserved for
+ *   different subsystems this reservation system just tries to coordinate
+ *   things a little
+ */
+static DEFINE_PER_CPU(unsigned, perfctr_nmi_owner);
+static DEFINE_PER_CPU(unsigned, evntsel_nmi_owner[2]);
+
+/* this number is calculated from Intel's MSR_P4_CRU_ESCR5 register and it's
+ * offset from MSR_P4_BSU_ESCR0.  It will be the max for all platforms (for now)
+ */
+#define NMI_MAX_COUNTER_BITS 66
+
 /*
  * lapic_nmi_owner tracks the ownership of the lapic NMI hardware:
  * - it may be reserved by some other driver, or not
@@ -91,6 +105,95 @@ static unsigned int nmi_p4_cccr_val;
 	(P4_CCCR_OVF_PMI0|P4_CCCR_THRESHOLD(15)|P4_CCCR_COMPLEMENT|	\
 	 P4_CCCR_COMPARE|P4_CCCR_REQUIRED|P4_CCCR_ESCR_SELECT(4)|P4_CCCR_ENABLE)
 
+/* converts an msr to an appropriate reservation bit */
+static inline unsigned int nmi_perfctr_msr_to_bit(unsigned int msr)
+{
+	/* returns the bit offset of the performance counter register */
+	switch (boot_cpu_data.x86_vendor) {
+	case X86_VENDOR_AMD:
+		return (msr - MSR_K7_PERFCTR0);
+	case X86_VENDOR_INTEL:
+		return (msr - MSR_P4_BPU_PERFCTR0);
+	}
+	return 0;
+}
+
+/* converts an msr to an appropriate reservation bit */
+static inline unsigned int nmi_evntsel_msr_to_bit(unsigned int msr)
+{
+	/* returns the bit offset of the event selection register */
+	switch (boot_cpu_data.x86_vendor) {
+	case X86_VENDOR_AMD:
+		return (msr - MSR_K7_EVNTSEL0);
+	case X86_VENDOR_INTEL:
+		return (msr - MSR_P4_BSU_ESCR0);
+	}
+	return 0;
+}
+
+/* checks for a bit availability (hack for oprofile) */
+int avail_to_resrv_perfctr_nmi_bit(unsigned int counter)
+{
+	BUG_ON(counter > NMI_MAX_COUNTER_BITS);
+
+	return (!test_bit(counter, &__get_cpu_var(perfctr_nmi_owner)));
+}
+
+/* checks the an msr for availability */
+int avail_to_resrv_perfctr_nmi(unsigned int msr)
+{
+	unsigned int counter;
+
+	counter = nmi_perfctr_msr_to_bit(msr);
+	BUG_ON(counter > NMI_MAX_COUNTER_BITS);
+
+	return (!test_bit(counter, &__get_cpu_var(perfctr_nmi_owner)));
+}
+
+int reserve_perfctr_nmi(unsigned int msr)
+{
+	unsigned int counter;
+
+	counter = nmi_perfctr_msr_to_bit(msr);
+	BUG_ON(counter > NMI_MAX_COUNTER_BITS);
+
+	if (!test_and_set_bit(counter, &__get_cpu_var(perfctr_nmi_owner)))
+		return 1;
+	return 0;
+}
+
+void release_perfctr_nmi(unsigned int msr)
+{
+	unsigned int counter;
+
+	counter = nmi_perfctr_msr_to_bit(msr);
+	BUG_ON(counter > NMI_MAX_COUNTER_BITS);
+
+	clear_bit(counter, &__get_cpu_var(perfctr_nmi_owner));
+}
+
+int reserve_evntsel_nmi(unsigned int msr)
+{
+	unsigned int counter;
+
+	counter = nmi_evntsel_msr_to_bit(msr);
+	BUG_ON(counter > NMI_MAX_COUNTER_BITS);
+
+	if (!test_and_set_bit(counter, &__get_cpu_var(evntsel_nmi_owner)))
+		return 1;
+	return 0;
+}
+
+void release_evntsel_nmi(unsigned int msr)
+{
+	unsigned int counter;
+
+	counter = nmi_evntsel_msr_to_bit(msr);
+	BUG_ON(counter > NMI_MAX_COUNTER_BITS);
+
+	clear_bit(counter, &__get_cpu_var(evntsel_nmi_owner));
+}
+
 static __cpuinit inline int nmi_known_cpu(void)
 {
 	switch (boot_cpu_data.x86_vendor) {
@@ -326,34 +429,22 @@ late_initcall(init_lapic_nmi_sysfs);
 
 #endif	/* CONFIG_PM */
 
-/*
- * Activate the NMI watchdog via the local APIC.
- * Original code written by Keith Owens.
- */
-
-static void clear_msr_range(unsigned int base, unsigned int n)
+static int setup_k7_watchdog(void)
 {
-	unsigned int i;
-
-	for(i = 0; i < n; ++i)
-		wrmsr(base+i, 0, 0);
-}
-
-static void setup_k7_watchdog(void)
-{
-	int i;
 	unsigned int evntsel;
 
 	nmi_perfctr_msr = MSR_K7_PERFCTR0;
 
-	for(i = 0; i < 4; ++i) {
-		/* Simulator may not support it */
-		if (checking_wrmsrl(MSR_K7_EVNTSEL0+i, 0UL)) {
-			nmi_perfctr_msr = 0;
-			return;
-		}
-		wrmsrl(MSR_K7_PERFCTR0+i, 0UL);
-	}
+	if (!reserve_perfctr_nmi(nmi_perfctr_msr))
+		goto fail;
+
+	if (!reserve_evntsel_nmi(MSR_K7_EVNTSEL0))
+		goto fail1;
+
+	/* Simulator may not support it */
+	if (checking_wrmsrl(MSR_K7_EVNTSEL0, 0UL))
+		goto fail2;
+	wrmsrl(MSR_K7_PERFCTR0, 0UL);
 
 	evntsel = K7_EVNTSEL_INT
 		| K7_EVNTSEL_OS
@@ -365,6 +456,13 @@ static void setup_k7_watchdog(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 	evntsel |= K7_EVNTSEL_ENABLE;
 	wrmsr(MSR_K7_EVNTSEL0, evntsel, 0);
+	return 1;
+fail2:
+	release_evntsel_nmi(MSR_K7_EVNTSEL0);
+fail1:
+	release_perfctr_nmi(nmi_perfctr_msr);
+fail:
+	return 0;
 }
 
 
@@ -383,22 +481,11 @@ static int setup_p4_watchdog(void)
 		nmi_p4_cccr_val |= P4_CCCR_OVF_PMI1;
 #endif
 
-	if (!(misc_enable & MSR_P4_MISC_ENABLE_PEBS_UNAVAIL))
-		clear_msr_range(0x3F1, 2);
-	/* MSR 0x3F0 seems to have a default value of 0xFC00, but current
-	   docs doesn't fully define it, so leave it alone for now. */
-	if (boot_cpu_data.x86_model >= 0x3) {
-		/* MSR_P4_IQ_ESCR0/1 (0x3ba/0x3bb) removed */
-		clear_msr_range(0x3A0, 26);
-		clear_msr_range(0x3BC, 3);
-	} else {
-		clear_msr_range(0x3A0, 31);
-	}
-	clear_msr_range(0x3C0, 6);
-	clear_msr_range(0x3C8, 6);
-	clear_msr_range(0x3E0, 2);
-	clear_msr_range(MSR_P4_CCCR0, 18);
-	clear_msr_range(MSR_P4_PERFCTR0, 18);
+	if (!reserve_perfctr_nmi(nmi_perfctr_msr))
+		goto fail;
+
+	if (!reserve_evntsel_nmi(MSR_P4_CRU_ESCR0))
+		goto fail1;
 
 	wrmsr(MSR_P4_CRU_ESCR0, P4_NMI_CRU_ESCR0, 0);
 	wrmsr(MSR_P4_IQ_CCCR0, P4_NMI_IQ_CCCR0 & ~P4_CCCR_ENABLE, 0);
@@ -407,6 +494,10 @@ static int setup_p4_watchdog(void)
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
 	wrmsr(MSR_P4_IQ_CCCR0, nmi_p4_cccr_val, 0);
 	return 1;
+fail1:
+	release_perfctr_nmi(nmi_perfctr_msr);
+fail:
+	return 0;
 }
 
 void setup_apic_nmi_watchdog(void)
@@ -417,7 +508,8 @@ void setup_apic_nmi_watchdog(void)
 			return;
 		if (strstr(boot_cpu_data.x86_model_id, "Screwdriver"))
 			return;
-		setup_k7_watchdog();
+		if (!setup_k7_watchdog())
+			return;
 		break;
 	case X86_VENDOR_INTEL:
 		if (boot_cpu_data.x86 != 15)
@@ -587,6 +679,12 @@ int proc_unknown_nmi_panic(struct ctl_ta
 
 EXPORT_SYMBOL(nmi_active);
 EXPORT_SYMBOL(nmi_watchdog);
+EXPORT_SYMBOL(avail_to_resrv_perfctr_nmi);
+EXPORT_SYMBOL(avail_to_resrv_perfctr_nmi_bit);
+EXPORT_SYMBOL(reserve_perfctr_nmi);
+EXPORT_SYMBOL(release_perfctr_nmi);
+EXPORT_SYMBOL(reserve_evntsel_nmi);
+EXPORT_SYMBOL(release_evntsel_nmi);
 EXPORT_SYMBOL(reserve_lapic_nmi);
 EXPORT_SYMBOL(release_lapic_nmi);
 EXPORT_SYMBOL(disable_timer_nmi_watchdog);
include/asm-x86_64/nmi.h +7 -1
Index: linux-don/include/asm-x86_64/nmi.h
===================================================================
@@ -56,7 +56,13 @@ extern int panic_on_timeout;
 extern int unknown_nmi_panic;
 
 extern int check_nmi_watchdog(void);
- 
+extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);
+extern int avail_to_resrv_perfctr_nmi(unsigned int);
+extern int reserve_perfctr_nmi(unsigned int);
+extern void release_perfctr_nmi(unsigned int);
+extern int reserve_evntsel_nmi(unsigned int);
+extern void release_evntsel_nmi(unsigned int);
+
 extern void setup_apic_nmi_watchdog (void);
 extern int reserve_lapic_nmi(void);
 extern void release_lapic_nmi(void);
arch/i386/kernel/nmi.c +156 -32

266 lines not on this page · raw message

include/asm-i386/nmi.h +7 -0
Index: linux-don/include/asm-i386/nmi.h
===================================================================
@@ -25,6 +25,13 @@ void set_nmi_callback(nmi_callback_t cal
  */
 void unset_nmi_callback(void);
 
+extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);
+extern int avail_to_resrv_perfctr_nmi(unsigned int);
+extern int reserve_perfctr_nmi(unsigned int);
+extern void release_perfctr_nmi(unsigned int);
+extern int reserve_evntsel_nmi(unsigned int);
+extern void release_evntsel_nmi(unsigned int);
+
 extern void setup_apic_nmi_watchdog (void);
 extern int reserve_lapic_nmi(void);
 extern void release_lapic_nmi(void);

--

author
#3 lore raw

PATCH 3/8 Utilize performance counter reservation framework in oprofile

In reply to dzickus

Incorporates the new performance counter reservation system in oprofile. Also cleans up a lot of the initialization code. The code original zero'd out every register associated with performance counters regardless if those registers were used or not. This causes issues with the nmi watchdog.

arch/i386/oprofile/nmi_int.c +29 -12
Index: linux-don/arch/i386/oprofile/nmi_int.c
===================================================================
@@ -97,15 +97,19 @@ static void nmi_cpu_save_registers(struc
 	unsigned int i;
 
 	for (i = 0; i < nr_ctrs; ++i) {
-		rdmsr(counters[i].addr,
-			counters[i].saved.low,
-			counters[i].saved.high);
+		if (counters[i].addr){
+			rdmsr(counters[i].addr,
+				counters[i].saved.low,
+				counters[i].saved.high);
+		}
 	}
  
 	for (i = 0; i < nr_ctrls; ++i) {
-		rdmsr(controls[i].addr,
-			controls[i].saved.low,
-			controls[i].saved.high);
+		if (controls[i].addr){
+			rdmsr(controls[i].addr,
+				controls[i].saved.low,
+				controls[i].saved.high);
+		}
 	}
 }
 
@@ -204,15 +208,19 @@ static void nmi_restore_registers(struct
 	unsigned int i;
 
 	for (i = 0; i < nr_ctrls; ++i) {
-		wrmsr(controls[i].addr,
-			controls[i].saved.low,
-			controls[i].saved.high);
+		if (controls[i].addr){
+			wrmsr(controls[i].addr,
+				controls[i].saved.low,
+				controls[i].saved.high);
+		}
 	}
  
 	for (i = 0; i < nr_ctrs; ++i) {
-		wrmsr(counters[i].addr,
-			counters[i].saved.low,
-			counters[i].saved.high);
+		if (counters[i].addr){
+			wrmsr(counters[i].addr,
+				counters[i].saved.low,
+				counters[i].saved.high);
+		}
 	}
 }
  
@@ -233,6 +241,7 @@ static void nmi_cpu_shutdown(void * dumm
 	apic_write(APIC_LVTPC, saved_lvtpc[cpu]);
 	apic_write(APIC_LVTERR, v);
 	nmi_restore_registers(msrs);
+	model->shutdown(msrs);
 }
 
  
@@ -283,6 +292,14 @@ static int nmi_create_files(struct super
 		struct dentry * dir;
 		char buf[2];
  
+ 		/* quick little hack to _not_ expose a counter if it is not
+		 * available for use.  This should protect userspace app.
+		 * NOTE:  assumes 1:1 mapping here (that counters are organized
+		 *        sequentially in their struct assignment).
+		 */
+		if (unlikely(!avail_to_resrv_perfctr_nmi_bit(i)))
+			continue;
+
 		snprintf(buf, 2, "%d", i);
 		dir = oprofilefs_mkdir(sb, root, buf);
 		oprofilefs_create_ulong(sb, dir, "enabled", &counter_config[i].enabled); 
arch/i386/oprofile/op_model_athlon.c +42 -12

109 lines not on this page · raw message

arch/i386/oprofile/op_model_p4.c +74 -78

276 lines not on this page · raw message

arch/i386/oprofile/op_model_ppro.c +53 -12

127 lines not on this page · raw message

arch/i386/oprofile/op_x86_model.h +1 -0
Index: linux-don/arch/i386/oprofile/op_x86_model.h
===================================================================
@@ -40,6 +40,7 @@ struct op_x86_model_spec {
 		struct op_msrs const * const msrs);
 	void (*start)(struct op_msrs const * const msrs);
 	void (*stop)(struct op_msrs const * const msrs);
+	void (*shutdown)(struct op_msrs const * const msrs);
 };
 
 extern struct op_x86_model_spec const op_ppro_spec;

--

author
#4 lore raw

PATCH 4/8 Add SMP support on x86_64 to reservation framework

In reply to dzickus

This patch includes the changes to make the nmi watchdog on x86_64 SMP aware. A bunch of code was moved around to make it simpler to read. In addition, it is now possible to determine if a particular NMI was the result of the watchdog or not. This feature allows the kernel to filter out unknown NMIs easier.

arch/x86_64/kernel/apic.c +1 -2
Index: linux-don/arch/x86_64/kernel/apic.c
===================================================================
@@ -480,8 +480,7 @@ void __cpuinit setup_local_APIC (void)
 	}
 
 	nmi_watchdog_default();
-	if (nmi_watchdog == NMI_LOCAL_APIC)
-		setup_apic_nmi_watchdog();
+	setup_apic_nmi_watchdog(NULL);
 	apic_pm_activate();
 }
 
arch/x86_64/kernel/nmi.c +287 -139

596 lines not on this page · raw message

include/asm-x86_64/nmi.h +2 -1
Index: linux-don/include/asm-x86_64/nmi.h
===================================================================
@@ -63,7 +63,7 @@ extern void release_perfctr_nmi(unsigned
 extern int reserve_evntsel_nmi(unsigned int);
 extern void release_evntsel_nmi(unsigned int);
 
-extern void setup_apic_nmi_watchdog (void);
+extern void setup_apic_nmi_watchdog (void *);
 extern int reserve_lapic_nmi(void);
 extern void release_lapic_nmi(void);
 extern void disable_timer_nmi_watchdog(void);
@@ -73,6 +73,7 @@ extern void nmi_watchdog_tick (struct pt
 extern void nmi_watchdog_default(void);
 extern int setup_nmi_watchdog(char *);
 
+extern atomic_t nmi_active;
 extern unsigned int nmi_watchdog;
 #define NMI_DEFAULT	-1
 #define NMI_NONE	0

--

#5 lore raw

Re: [patch 4/8] Add SMP support on x86_64 to reservation framework

Quoted 6 lines · dzickus

This patch includes the changes to make the nmi watchdog on x86_64 SMP aware. A bunch of code was moved around to make it simpler to read. In addition, it is now possible to determine if a particular NMI was the result of the watchdog or not. This feature allows the kernel to filter out unknown NMIs easier.

Looks all good to me. I merged it all up. Thanks.

Regarding the ioapic watchdog not switching with the sysctl - i guess we can live with that. It is deprecated anyways.

Thanks for doing all that work.

-Andi

author
#6 lore raw

PATCH 5/8 Add SMP support on i386 to reservation framework

In reply to dzickus

This patch includes the changes to make the nmi watchdog on i386 SMP aware. A bunch of code was moved around to make it simpler to read. In addition, it is now possible to determine if a particular NMI was the result of the watchdog or not. This feature allows the kernel to filter out unknown NMIs easier.

arch/i386/kernel/apic.c +1 -2
Index: linux-don/arch/i386/kernel/apic.c
===================================================================
@@ -587,8 +587,7 @@ void __devinit setup_local_APIC(void)
 			printk("No ESR for 82489DX.\n");
 	}
 
-	if (nmi_watchdog == NMI_LOCAL_APIC)
-		setup_apic_nmi_watchdog();
+	setup_apic_nmi_watchdog(NULL);
 	apic_pm_activate();
 }
 
arch/i386/kernel/nmi.c +355 -184

746 lines not on this page · raw message

arch/i386/kernel/traps.c +1 -1
Index: linux-don/arch/i386/kernel/traps.c
===================================================================
@@ -691,7 +691,7 @@ static void default_do_nmi(struct pt_reg
 		 * so it must be the NMI watchdog.
 		 */
 		if (nmi_watchdog) {
-			nmi_watchdog_tick(regs);
+			nmi_watchdog_tick(regs, reason);
 			return;
 		}
 #endif
include/asm-i386/nmi.h +3 -2
Index: linux-don/include/asm-i386/nmi.h
===================================================================
@@ -32,13 +32,14 @@ extern void release_perfctr_nmi(unsigned
 extern int reserve_evntsel_nmi(unsigned int);
 extern void release_evntsel_nmi(unsigned int);
 
-extern void setup_apic_nmi_watchdog (void);
+extern void setup_apic_nmi_watchdog (void *);
 extern int reserve_lapic_nmi(void);
 extern void release_lapic_nmi(void);
 extern void disable_timer_nmi_watchdog(void);
 extern void enable_timer_nmi_watchdog(void);
-extern void nmi_watchdog_tick (struct pt_regs * regs);
+extern void nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
 
+extern atomic_t nmi_active;
 extern unsigned int nmi_watchdog;
 #define NMI_DEFAULT     -1
 #define NMI_NONE	0
arch/i386/oprofile/nmi_timer_int.c +1 -3
Index: linux-don/arch/i386/oprofile/nmi_timer_int.c
===================================================================
@@ -42,9 +42,7 @@ static void timer_stop(void)
 
 int __init op_nmi_timer_init(struct oprofile_operations * ops)
 {
-	extern int nmi_active;
-
-	if (nmi_active <= 0)
+	if (atomic_read(&nmi_active) <= 0)
 		return -ENODEV;
 
 	ops->start = timer_start;

--

#7 lore raw

Re: [patch 5/8] Add SMP support on i386 to reservation framework

@@ -457,143 +434,312 @@ late_initcall(init_lapic_nmi_sysfs); [...]
static int setup_p6_watchdog(void)
[...]
apic_write(APIC_LVTPC, APIC_DM_NMI);
- evntsel |= P6_EVNTSEL0_ENABLE;
- wrmsr(MSR_P6_EVNTSEL0, evntsel, 0);
+ evntsel |= K7_EVNTSEL_ENABLE;

Me thinks you want P6_EVNTSEL0_ENABLE here, although the value is the same.

author
#8 lore raw

Re: [patch 5/8] Add SMP support on i386 to reservation framework

Quoted 11 lines · Markus Armbruster
arch/i386/kernel/nmi.c +1 -2
@@ -457,143 +434,312 @@ late_initcall(init_lapic_nmi_sysfs);
[...]
 static int setup_p6_watchdog(void)
[...]
 	apic_write(APIC_LVTPC, APIC_DM_NMI);
-	evntsel |= P6_EVNTSEL0_ENABLE;
-	wrmsr(MSR_P6_EVNTSEL0, evntsel, 0);
+	evntsel |= K7_EVNTSEL_ENABLE;

Me thinks you want P6_EVNTSEL0_ENABLE here, although the value is the same.

Yup, the downside of copying/pasting...

Thanks.

author
#9 lore raw

PATCH 6/8 Cleanup NMI interrupt path

In reply to dzickus

This patch cleans up the NMI interrupt path. Instead of being gated by if the 'nmi callback' is set, the interrupt handler now calls everyone who is registered on the die_chain and additionally checks the nmi watchdog, reseting it if enabled. This allows more subsystems to hook into the NMI if they need to (without being block by set_nmi_callback).

arch/i386/kernel/nmi.c +13 -3
Index: linux-don/arch/i386/kernel/nmi.c
===================================================================
@@ -780,7 +780,7 @@ void touch_nmi_watchdog (void)
 
 extern void die_nmi(struct pt_regs *, const char *msg);
 
-void nmi_watchdog_tick (struct pt_regs * regs, unsigned reason)
+int nmi_watchdog_tick (struct pt_regs * regs, unsigned reason)
 {
 
 	/*
@@ -793,10 +793,12 @@ void nmi_watchdog_tick (struct pt_regs *
 	int cpu = smp_processor_id();
 	struct nmi_watchdog_ctlblk *wd = &__get_cpu_var(nmi_watchdog_ctlblk);
 	u64 dummy;
+	int rc=0;
 
 	/* check for other users first */
 	if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
 			== NOTIFY_STOP) {
+		rc = 1;
 		touched = 1;
 	}
 
@@ -849,10 +851,18 @@ void nmi_watchdog_tick (struct pt_regs *
 			}
 			/* start the cycle over again */
 			write_watchdog_counter(wd->perfctr_msr, NULL);
-		}
+			rc = 1;
+		} else if (nmi_watchdog == NMI_IO_APIC) {
+			/* don't know how to accurately check for this.
+			 * just assume it was a watchdog timer interrupt
+			 * This matches the old behaviour.
+			 */
+			rc = 1;
+		} else
+			printk(KERN_WARNING "Unknown enabled NMI hardware?!\n");
 	}
 done:
-	return;
+	return rc;
 }
 
 #ifdef CONFIG_SYSCTL
arch/i386/kernel/traps.c +11 -13
Index: linux-don/arch/i386/kernel/traps.c
===================================================================
@@ -673,6 +673,13 @@ void die_nmi (struct pt_regs *regs, cons
 	do_exit(SIGSEGV);
 }
 
+static int dummy_nmi_callback(struct pt_regs * regs, int cpu)
+{
+	return 0;
+}
+
+static nmi_callback_t nmi_callback = dummy_nmi_callback;
+
 static void default_do_nmi(struct pt_regs * regs)
 {
 	unsigned char reason = 0;
@@ -690,12 +697,11 @@ static void default_do_nmi(struct pt_reg
 		 * Ok, so this is none of the documented NMI sources,
 		 * so it must be the NMI watchdog.
 		 */
-		if (nmi_watchdog) {
-			nmi_watchdog_tick(regs, reason);
+		if (nmi_watchdog_tick(regs, reason))
 			return;
-		}
 #endif
-		unknown_nmi_error(reason, regs);
+		if (!rcu_dereference(nmi_callback)(regs, smp_processor_id()))
+			unknown_nmi_error(reason, regs);
 		return;
 	}
 	if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == NOTIFY_STOP)
@@ -711,13 +717,6 @@ static void default_do_nmi(struct pt_reg
 	reassert_nmi();
 }
 
-static int dummy_nmi_callback(struct pt_regs * regs, int cpu)
-{
-	return 0;
-}
- 
-static nmi_callback_t nmi_callback = dummy_nmi_callback;
- 
 fastcall void do_nmi(struct pt_regs * regs, long error_code)
 {
 	int cpu;
@@ -728,8 +727,7 @@ fastcall void do_nmi(struct pt_regs * re
 
 	++nmi_count(cpu);
 
-	if (!rcu_dereference(nmi_callback)(regs, cpu))
-		default_do_nmi(regs);
+	default_do_nmi(regs);
 
 	nmi_exit();
 }
arch/x86_64/kernel/nmi.c +19 -7
Index: linux-don/arch/x86_64/kernel/nmi.c
===================================================================
@@ -683,16 +683,18 @@ void touch_nmi_watchdog (void)
  	touch_softlockup_watchdog();
 }
 
-void __kprobes nmi_watchdog_tick(struct pt_regs * regs, unsigned reason)
+int __kprobes nmi_watchdog_tick(struct pt_regs * regs, unsigned reason)
 {
 	int sum;
 	int touched = 0;
 	struct nmi_watchdog_ctlblk *wd = &__get_cpu_var(nmi_watchdog_ctlblk);
 	u64 dummy;
+	int rc=0;
 
 	/* check for other users first */
 	if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT)
 			== NOTIFY_STOP) {
+		rc = 1;
 		touched = 1;
 	}
 
@@ -747,10 +749,18 @@ void __kprobes nmi_watchdog_tick(struct 
 	 		}
 			/* start the cycle over again */
 			wrmsrl(wd->perfctr_msr, -((u64)cpu_khz * 1000 / nmi_hz));
-		}
+			rc = 1;
+		} else 	if (nmi_watchdog == NMI_IO_APIC) {
+			/* don't know how to accurately check for this.
+			 * just assume it was a watchdog timer interrupt
+			 * This matches the old behaviour.
+			 */
+			rc = 1;
+		} else
+			printk(KERN_WARNING "Unknown enabled NMI hardware?!\n");
 	}
 done:
-	return;
+	return rc;
 }
 
 static __kprobes int dummy_nmi_callback(struct pt_regs * regs, int cpu)
@@ -762,15 +772,17 @@ static nmi_callback_t nmi_callback = dum
  
 asmlinkage __kprobes void do_nmi(struct pt_regs * regs, long error_code)
 {
-	int cpu = safe_smp_processor_id();
-
 	nmi_enter();
 	add_pda(__nmi_count,1);
-	if (!rcu_dereference(nmi_callback)(regs, cpu))
-		default_do_nmi(regs);
+	default_do_nmi(regs);
 	nmi_exit();
 }
 
+int do_nmi_callback(struct pt_regs * regs, int cpu)
+{
+	return rcu_dereference(nmi_callback)(regs, cpu);
+}
+
 void set_nmi_callback(nmi_callback_t callback)
 {
 	vmalloc_sync_all();
arch/x86_64/kernel/traps.c +4 -4
Index: linux-don/arch/x86_64/kernel/traps.c
===================================================================
@@ -657,12 +657,12 @@ asmlinkage __kprobes void default_do_nmi
 		 * Ok, so this is none of the documented NMI sources,
 		 * so it must be the NMI watchdog.
 		 */
-		if (nmi_watchdog > 0) {
-			nmi_watchdog_tick(regs,reason);
+		if (nmi_watchdog_tick(regs,reason))
 			return;
-		}
+		if (!do_nmi_callback(regs,cpu))
 #endif
-		unknown_nmi_error(reason, regs);
+			unknown_nmi_error(reason, regs);
+
 		return;
 	}
 	if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == NOTIFY_STOP)
include/asm-i386/nmi.h +1 -1
Index: linux-don/include/asm-i386/nmi.h
===================================================================
@@ -37,7 +37,7 @@ extern int reserve_lapic_nmi(void);
 extern void release_lapic_nmi(void);
 extern void disable_timer_nmi_watchdog(void);
 extern void enable_timer_nmi_watchdog(void);
-extern void nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
+extern int nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
 
 extern atomic_t nmi_active;
 extern unsigned int nmi_watchdog;
include/asm-x86_64/nmi.h +9 -1
Index: linux-don/include/asm-x86_64/nmi.h
===================================================================
@@ -26,6 +26,14 @@ void set_nmi_callback(nmi_callback_t cal
  */
 void unset_nmi_callback(void);
 
+/**
+ * do_nmi_callback
+ *
+ * Check to see if a callback exists and execute it.  Return 1
+ * if the handler exists and was handled successfully.
+ */
+int do_nmi_callback(struct pt_regs *regs, int cpu);
+
 #ifdef CONFIG_PM
  
 /** Replace the PM callback routine for NMI. */
@@ -68,7 +76,7 @@ extern int reserve_lapic_nmi(void);
 extern void release_lapic_nmi(void);
 extern void disable_timer_nmi_watchdog(void);
 extern void enable_timer_nmi_watchdog(void);
-extern void nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
+extern int nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
 
 extern void nmi_watchdog_default(void);
 extern int setup_nmi_watchdog(char *);

--

author
#10 lore raw

PATCH 7/8 Remove un/set_nmi_callback and reserve/release_lapic_nmi functions

In reply to dzickus

Removes the un/set_nmi_callback and reserve/release_lapic_nmi functions as they are no longer needed. The various subsystems are modified to register with the die_notifier instead.

arch/x86_64/kernel/nmi.c +7 -93
Index: linux-don/arch/x86_64/kernel/nmi.c
===================================================================
@@ -42,20 +42,6 @@ static DEFINE_PER_CPU(unsigned, evntsel_
  */
 #define NMI_MAX_COUNTER_BITS 66
 
-/*
- * lapic_nmi_owner tracks the ownership of the lapic NMI hardware:
- * - it may be reserved by some other driver, or not
- * - when not reserved by some other driver, it may be used for
- *   the NMI watchdog, or not
- *
- * This is maintained separately from nmi_active because the NMI
- * watchdog may also be driven from the I/O APIC timer.
- */
-static DEFINE_SPINLOCK(lapic_nmi_owner_lock);
-static unsigned int lapic_nmi_owner;
-#define LAPIC_NMI_WATCHDOG	(1<<0)
-#define LAPIC_NMI_RESERVED	(1<<1)
-
 /* nmi_active:
  * >0: the lapic NMI watchdog is active, but can be disabled
  * <0: the lapic NMI watchdog has not been set up, and cannot
@@ -322,33 +308,6 @@ static void enable_lapic_nmi_watchdog(vo
 	touch_nmi_watchdog();
 }
 
-int reserve_lapic_nmi(void)
-{
-	unsigned int old_owner;
-
-	spin_lock(&lapic_nmi_owner_lock);
-	old_owner = lapic_nmi_owner;
-	lapic_nmi_owner |= LAPIC_NMI_RESERVED;
-	spin_unlock(&lapic_nmi_owner_lock);
-	if (old_owner & LAPIC_NMI_RESERVED)
-		return -EBUSY;
-	if (old_owner & LAPIC_NMI_WATCHDOG)
-		disable_lapic_nmi_watchdog();
-	return 0;
-}
-
-void release_lapic_nmi(void)
-{
-	unsigned int new_owner;
-
-	spin_lock(&lapic_nmi_owner_lock);
-	new_owner = lapic_nmi_owner & ~LAPIC_NMI_RESERVED;
-	lapic_nmi_owner = new_owner;
-	spin_unlock(&lapic_nmi_owner_lock);
-	if (new_owner & LAPIC_NMI_WATCHDOG)
-		enable_lapic_nmi_watchdog();
-}
-
 void disable_timer_nmi_watchdog(void)
 {
 	BUG_ON(nmi_watchdog != NMI_IO_APIC);
@@ -763,13 +722,6 @@ done:
 	return rc;
 }
 
-static __kprobes int dummy_nmi_callback(struct pt_regs * regs, int cpu)
-{
-	return 0;
-}
- 
-static nmi_callback_t nmi_callback = dummy_nmi_callback;
- 
 asmlinkage __kprobes void do_nmi(struct pt_regs * regs, long error_code)
 {
 	nmi_enter();
@@ -780,18 +732,11 @@ asmlinkage __kprobes void do_nmi(struct 
 
 int do_nmi_callback(struct pt_regs * regs, int cpu)
 {
-	return rcu_dereference(nmi_callback)(regs, cpu);
-}
-
-void set_nmi_callback(nmi_callback_t callback)
-{
-	vmalloc_sync_all();
-	rcu_assign_pointer(nmi_callback, callback);
-}
-
-void unset_nmi_callback(void)
-{
-	nmi_callback = dummy_nmi_callback;
+#ifdef CONFIG_SYSCTL
+	if (unknown_nmi_panic)
+		return unknown_nmi_panic_callback(regs, cpu);
+#endif
+	return 0;
 }
 
 #ifdef CONFIG_SYSCTL
@@ -801,37 +746,8 @@ static int unknown_nmi_panic_callback(st
 	unsigned char reason = get_nmi_reason();
 	char buf[64];
 
-	if (!(reason & 0xc0)) {
-		sprintf(buf, "NMI received for unknown reason %02x\n", reason);
-		die_nmi(buf,regs);
-	}
-	return 0;
-}
-
-/*
- * proc handler for /proc/sys/kernel/unknown_nmi_panic
- */
-int proc_unknown_nmi_panic(struct ctl_table *table, int write, struct file *file,
-			void __user *buffer, size_t *length, loff_t *ppos)
-{
-	int old_state;
-
-	old_state = unknown_nmi_panic;
-	proc_dointvec(table, write, file, buffer, length, ppos);
-	if (!!old_state == !!unknown_nmi_panic)
-		return 0;
-
-	if (unknown_nmi_panic) {
-		if (reserve_lapic_nmi() < 0) {
-			unknown_nmi_panic = 0;
-			return -EBUSY;
-		} else {
-			set_nmi_callback(unknown_nmi_panic_callback);
-		}
-	} else {
-		release_lapic_nmi();
-		unset_nmi_callback();
-	}
+	sprintf(buf, "NMI received for unknown reason %02x\n", reason);
+	die_nmi(buf,regs);
 	return 0;
 }
 
@@ -845,8 +761,6 @@ EXPORT_SYMBOL(reserve_perfctr_nmi);
 EXPORT_SYMBOL(release_perfctr_nmi);
 EXPORT_SYMBOL(reserve_evntsel_nmi);
 EXPORT_SYMBOL(release_evntsel_nmi);
-EXPORT_SYMBOL(reserve_lapic_nmi);
-EXPORT_SYMBOL(release_lapic_nmi);
 EXPORT_SYMBOL(disable_timer_nmi_watchdog);
 EXPORT_SYMBOL(enable_timer_nmi_watchdog);
 EXPORT_SYMBOL(touch_nmi_watchdog);
arch/x86_64/kernel/x8664_ksyms.c +0 -3
Index: linux-don/arch/x86_64/kernel/x8664_ksyms.c
===================================================================
@@ -104,9 +104,6 @@ EXPORT_SYMBOL(screen_info);
 
 EXPORT_SYMBOL(rtc_lock);
 
-EXPORT_SYMBOL_GPL(set_nmi_callback);
-EXPORT_SYMBOL_GPL(unset_nmi_callback);
-
 /* Export string functions. We normally rely on gcc builtin for most of these,
    but gcc sometimes decides not to inline them. */    
 #undef memcpy
include/asm-x86_64/nmi.h +0 -21
Index: linux-don/include/asm-x86_64/nmi.h
===================================================================
@@ -7,25 +7,6 @@
 #include <linux/pm.h>
 #include <asm/io.h>
  
-struct pt_regs;
-
-typedef int (*nmi_callback_t)(struct pt_regs * regs, int cpu);
-
-/**
- * set_nmi_callback
- *
- * Set a handler for an NMI. Only one handler may be
- * set. Return 1 if the NMI was handled.
- */
-void set_nmi_callback(nmi_callback_t callback);
-
-/**
- * unset_nmi_callback
- *
- * Remove the handler previously set.
- */
-void unset_nmi_callback(void);
-
 /**
  * do_nmi_callback
  *
@@ -72,8 +53,6 @@ extern int reserve_evntsel_nmi(unsigned 
 extern void release_evntsel_nmi(unsigned int);
 
 extern void setup_apic_nmi_watchdog (void *);
-extern int reserve_lapic_nmi(void);
-extern void release_lapic_nmi(void);
 extern void disable_timer_nmi_watchdog(void);
 extern void enable_timer_nmi_watchdog(void);
 extern int nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
arch/i386/oprofile/nmi_int.c +30 -17

92 lines not on this page · raw message

arch/i386/oprofile/nmi_timer_int.c +25 -8
Index: linux-don/arch/i386/oprofile/nmi_timer_int.c
===================================================================
@@ -17,32 +17,49 @@
 #include <asm/nmi.h>
 #include <asm/apic.h>
 #include <asm/ptrace.h>
+#include <asm/kdebug.h>
  
-static int nmi_timer_callback(struct pt_regs * regs, int cpu)
+int profile_timer_exceptions_notify(struct notifier_block *self,
+				unsigned long val, void *data)
 {
-	oprofile_add_sample(regs, 0);
-	return 1;
+	struct die_args *args = (struct die_args *)data;
+	int ret = NOTIFY_DONE;
+
+	switch(val) {
+	case DIE_NMI:
+		oprofile_add_sample(args->regs, 0);
+		ret = NOTIFY_STOP;
+		break;
+	default:
+		break;
+	}
+	return ret;
 }
 
+static struct notifier_block profile_timer_exceptions_nb = {
+	.notifier_call = profile_timer_exceptions_notify,
+	.next = NULL,
+	.priority = 0
+};
+
 static int timer_start(void)
 {
-	disable_timer_nmi_watchdog();
-	set_nmi_callback(nmi_timer_callback);
+	if (register_die_notifier(&profile_timer_exceptions_nb))
+		return 1;
 	return 0;
 }
 
 
 static void timer_stop(void)
 {
-	enable_timer_nmi_watchdog();
-	unset_nmi_callback();
+	unregister_die_notifier(&profile_timer_exceptions_nb);
 	synchronize_sched();  /* Allow already-started NMIs to complete. */
 }
 
 
 int __init op_nmi_timer_init(struct oprofile_operations * ops)
 {
-	if (atomic_read(&nmi_active) <= 0)
+	if ((nmi_watchdog != NMI_IO_APIC) || (atomic_read(&nmi_active) <= 0))
 		return -ENODEV;
 
 	ops->start = timer_start;
arch/i386/kernel/nmi.c +11 -74

114 lines not on this page · raw message

arch/i386/kernel/traps.c +2 -21
Index: linux-don/arch/i386/kernel/traps.c
===================================================================
@@ -673,13 +673,6 @@ void die_nmi (struct pt_regs *regs, cons
 	do_exit(SIGSEGV);
 }
 
-static int dummy_nmi_callback(struct pt_regs * regs, int cpu)
-{
-	return 0;
-}
-
-static nmi_callback_t nmi_callback = dummy_nmi_callback;
-
 static void default_do_nmi(struct pt_regs * regs)
 {
 	unsigned char reason = 0;
@@ -699,9 +692,10 @@ static void default_do_nmi(struct pt_reg
 		 */
 		if (nmi_watchdog_tick(regs, reason))
 			return;
+		if (!do_nmi_callback(regs, smp_processor_id()))
 #endif
-		if (!rcu_dereference(nmi_callback)(regs, smp_processor_id()))
 			unknown_nmi_error(reason, regs);
+
 		return;
 	}
 	if (notify_die(DIE_NMI, "nmi", regs, reason, 2, SIGINT) == NOTIFY_STOP)
@@ -732,19 +726,6 @@ fastcall void do_nmi(struct pt_regs * re
 	nmi_exit();
 }
 
-void set_nmi_callback(nmi_callback_t callback)
-{
-	vmalloc_sync_all();
-	rcu_assign_pointer(nmi_callback, callback);
-}
-EXPORT_SYMBOL_GPL(set_nmi_callback);
-
-void unset_nmi_callback(void)
-{
-	nmi_callback = dummy_nmi_callback;
-}
-EXPORT_SYMBOL_GPL(unset_nmi_callback);
-
 #ifdef CONFIG_KPROBES
 fastcall void __kprobes do_int3(struct pt_regs *regs, long error_code)
 {
include/asm-i386/nmi.h +4 -17
Index: linux-don/include/asm-i386/nmi.h
===================================================================
@@ -6,24 +6,13 @@
 
 #include <linux/pm.h>
 
-struct pt_regs;
-
-typedef int (*nmi_callback_t)(struct pt_regs * regs, int cpu);
-
-/**
- * set_nmi_callback
- *
- * Set a handler for an NMI. Only one handler may be
- * set. Return 1 if the NMI was handled.
- */
-void set_nmi_callback(nmi_callback_t callback);
-
 /**
- * unset_nmi_callback
+ * do_nmi_callback
  *
- * Remove the handler previously set.
+ * Check to see if a callback exists and execute it.  Return 1
+ * if the handler exists and was handled successfully.
  */
-void unset_nmi_callback(void);
+int do_nmi_callback(struct pt_regs *regs, int cpu);
 
 extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);
 extern int avail_to_resrv_perfctr_nmi(unsigned int);
@@ -33,8 +22,6 @@ extern int reserve_evntsel_nmi(unsigned 
 extern void release_evntsel_nmi(unsigned int);
 
 extern void setup_apic_nmi_watchdog (void *);
-extern int reserve_lapic_nmi(void);
-extern void release_lapic_nmi(void);
 extern void disable_timer_nmi_watchdog(void);
 extern void enable_timer_nmi_watchdog(void);
 extern int nmi_watchdog_tick (struct pt_regs * regs, unsigned reason);
kernel/sysctl.c +1 -3
Index: linux-don/kernel/sysctl.c
===================================================================
@@ -75,8 +75,6 @@ extern int percpu_pagelist_fraction;
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 int unknown_nmi_panic;
-extern int proc_unknown_nmi_panic(ctl_table *, int, struct file *,
-				  void __user *, size_t *, loff_t *);
 #endif
 
 /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */
@@ -630,7 +628,7 @@ static ctl_table kern_table[] = {
 		.data           = &unknown_nmi_panic,
 		.maxlen         = sizeof (int),
 		.mode           = 0644,
-		.proc_handler   = &proc_unknown_nmi_panic,
+		.proc_handler   = &proc_dointvec,
 	},
 #endif
 #if defined(CONFIG_X86)

--

author
#11 lore raw

PATCH 8/8 Add abilty to enable/disable nmi watchdog from sysfs

In reply to dzickus

Adds a new /proc/sys/kernel/nmi call that will enable/disable the nmi watchdog.

arch/i386/kernel/nmi.c +50 -0
Index: linux-don/arch/i386/kernel/nmi.c
===================================================================
@@ -845,6 +845,56 @@ static int unknown_nmi_panic_callback(st
 	return 0;
 }
 
+/*
+ * proc handler for /proc/sys/kernel/nmi
+ */
+int proc_nmi_enabled(struct ctl_table *table, int write, struct file *file,
+			void __user *buffer, size_t *length, loff_t *ppos)
+{
+	int old_state;
+
+	nmi_watchdog_enabled = (atomic_read(&nmi_active) > 0) ? 1 : 0;
+	old_state = nmi_watchdog_enabled;
+	proc_dointvec(table, write, file, buffer, length, ppos);
+	if (!!old_state == !!nmi_watchdog_enabled)
+		return 0;
+
+	if (atomic_read(&nmi_active) < 0) {
+		printk( KERN_WARNING "NMI watchdog is permanently disabled\n");
+		return 0;
+	}
+
+	if (nmi_watchdog == NMI_DEFAULT) {
+		if (nmi_known_cpu() > 0)
+			nmi_watchdog = NMI_LOCAL_APIC;
+		else
+			nmi_watchdog = NMI_IO_APIC;
+	}
+
+	if (nmi_watchdog == NMI_LOCAL_APIC)
+	{
+		if (nmi_watchdog_enabled)
+			enable_lapic_nmi_watchdog();
+		else
+			disable_lapic_nmi_watchdog();
+	} else if (nmi_watchdog == NMI_IO_APIC) {
+		/* FIXME
+		 * for some reason these functions don't work
+		 */
+		printk("Can not enable/disable NMI on IO APIC\n");
+		return 0;
+		/*if (nmi_watchdog_enabled)
+			enable_timer_nmi_watchdog();
+		else
+			disable_timer_nmi_watchdog();
+		*/
+	} else {
+		printk( KERN_WARNING
+			"NMI watchdog doesn't know what hardware to touch\n");
+	}
+	return 0;
+}
+
 #endif
 
 EXPORT_SYMBOL(nmi_active);
arch/x86_64/kernel/nmi.c +46 -0

52 lines not on this page · raw message

include/asm-i386/nmi.h +1 -0
Index: linux-don/include/asm-i386/nmi.h
===================================================================
@@ -14,6 +14,7 @@
  */
 int do_nmi_callback(struct pt_regs *regs, int cpu);
 
+extern int nmi_watchdog_enabled;
 extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);
 extern int avail_to_resrv_perfctr_nmi(unsigned int);
 extern int reserve_perfctr_nmi(unsigned int);
include/asm-x86_64/nmi.h +1 -0
Index: linux-don/include/asm-x86_64/nmi.h
===================================================================
@@ -43,6 +43,7 @@ extern void die_nmi(char *str, struct pt
 
 extern int panic_on_timeout;
 extern int unknown_nmi_panic;
+extern int nmi_watchdog_enabled;
 
 extern int check_nmi_watchdog(void);
 extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);
include/linux/sysctl.h +1 -0
Index: linux-don/include/linux/sysctl.h
===================================================================
@@ -148,6 +148,7 @@ enum
 	KERN_SPIN_RETRY=70,	/* int: number of spinlock retries */
 	KERN_ACPI_VIDEO_FLAGS=71, /* int: flags for setting up video after ACPI sleep */
 	KERN_IA64_UNALIGNED=72, /* int: ia64 unaligned userland trap enable */
+	KERN_NMI_ENABLED=73, /* int: enable/disable nmi watchdog */
 };
 
 
kernel/sysctl.c +11 -0
Index: linux-don/kernel/sysctl.c
===================================================================
@@ -75,6 +75,9 @@ extern int percpu_pagelist_fraction;
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 int unknown_nmi_panic;
+int nmi_watchdog_enabled;
+extern int proc_nmi_enabled(struct ctl_table *, int , struct file *,
+			void __user *, size_t *, loff_t *);
 #endif
 
 /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */
@@ -630,6 +633,14 @@ static ctl_table kern_table[] = {
 		.mode           = 0644,
 		.proc_handler   = &proc_dointvec,
 	},
+	{
+		.ctl_name       = KERN_NMI_ENABLED,
+		.procname       = "nmi",
+		.data           = &nmi_watchdog_enabled,
+		.maxlen         = sizeof (int),
+		.mode           = 0644,
+		.proc_handler   = &proc_nmi_enabled,
+	},
 #endif
 #if defined(CONFIG_X86)
 	{

--

#12 lore raw

Re: [patch 8/8] Add abilty to enable/disable nmi watchdog from sysfs

Quoted 3 lines · dzickus

Adds a new /proc/sys/kernel/nmi call that will enable/disable the nmi watchdog.

The subject is wrong i think - sysctl != sysfs.

And shouldn't that be called nmi_watchdog?

Quoted 19 lines

Index: linux-don/arch/i386/kernel/nmi.c
===================================================================

arch/i386/kernel/nmi.c +8 -0
@@ -845,6 +845,56 @@ static int unknown_nmi_panic_callback(st
 	return 0;
 }
 
+/*
+ * proc handler for /proc/sys/kernel/nmi
+ */
+int proc_nmi_enabled(struct ctl_table *table, int write, struct file *file,
+			void __user *buffer, size_t *length, loff_t *ppos)
+{
+	int old_state;
+

arch/i386/kernel/nmi.c static int unknown_nmi_panic_callback(st · line 865

Quoted 10 lines
arch/i386/kernel/nmi.c +10 -0
+	nmi_watchdog_enabled = (atomic_read(&nmi_active) > 0) ? 1 : 0;
+	old_state = nmi_watchdog_enabled;
+	proc_dointvec(table, write, file, buffer, length, ppos);
+	if (!!old_state == !!nmi_watchdog_enabled)
+		return 0;
+
+	if (atomic_read(&nmi_active) < 0) {
+		printk( KERN_WARNING "NMI watchdog is permanently disabled\n");
+		return 0;
+	}

The returns should be probably -EIO or similar. Same further down.

arch/i386/kernel/nmi.c static int unknown_nmi_panic_callback(st · line 875

Quoted 10 lines
arch/i386/kernel/nmi.c +10 -0
+
+	if (nmi_watchdog == NMI_DEFAULT) {
+		if (nmi_known_cpu() > 0)
+			nmi_watchdog = NMI_LOCAL_APIC;
+		else
+			nmi_watchdog = NMI_IO_APIC;
+	}
+
+	if (nmi_watchdog == NMI_LOCAL_APIC)
+	{

Move the { up

Anyways I fixed these up myself, no need to resubmit.

-Andi

#13 lore raw

Re: [patch 8/8] Add abilty to enable/disable nmi watchdog from sysfs

In reply to dzickus

Don,

Congratulations on the patch. I am glad to see that some of the SMP issues I reported a long time ago are now fixed.

Quoted 4 lines · dzickus

Adds a new /proc/sys/kernel/nmi call that will enable/disable the nmi watchdog.

This means you can at runtime enable/disbale nmi_watchdog, i.e., reserve some performance counters on the fly. This gets complicated because now the perfmon subsystem (and probably oprofile) cannot check register availability when they are first initialized. Basically each time, the /sys entry is modified, they would have to scan the list of available performance counters. I don't know exactly when Oprofile does this checking. For perfmon, this is done only once, when the PMU description table is loaded.

Also something that I did not see in this code is the error detection in case enable_lapic_nmi_watchdog() fails. Oprofile runs on all CPUs or none. Perfmon lets you monitor on subsets on CPUs. In case NMI was disabled and a monitoring session was active on some CPUs. The enable_lapic_nmi_watchdog() will fail on some CPUs. How is that handled?

Thanks.

Quoted 198 lines

Index: linux-don/arch/i386/kernel/nmi.c
===================================================================

arch/i386/kernel/nmi.c +50 -0
@@ -845,6 +845,56 @@ static int unknown_nmi_panic_callback(st
 	return 0;
 }
 
+/*
+ * proc handler for /proc/sys/kernel/nmi
+ */
+int proc_nmi_enabled(struct ctl_table *table, int write, struct file *file,
+			void __user *buffer, size_t *length, loff_t *ppos)
+{
+	int old_state;
+
+	nmi_watchdog_enabled = (atomic_read(&nmi_active) > 0) ? 1 : 0;
+	old_state = nmi_watchdog_enabled;
+	proc_dointvec(table, write, file, buffer, length, ppos);
+	if (!!old_state == !!nmi_watchdog_enabled)
+		return 0;
+
+	if (atomic_read(&nmi_active) < 0) {
+		printk( KERN_WARNING "NMI watchdog is permanently disabled\n");
+		return 0;
+	}
+
+	if (nmi_watchdog == NMI_DEFAULT) {
+		if (nmi_known_cpu() > 0)
+			nmi_watchdog = NMI_LOCAL_APIC;
+		else
+			nmi_watchdog = NMI_IO_APIC;
+	}
+
+	if (nmi_watchdog == NMI_LOCAL_APIC)
+	{
+		if (nmi_watchdog_enabled)
+			enable_lapic_nmi_watchdog();
+		else
+			disable_lapic_nmi_watchdog();
+	} else if (nmi_watchdog == NMI_IO_APIC) {
+		/* FIXME
+		 * for some reason these functions don't work
+		 */
+		printk("Can not enable/disable NMI on IO APIC\n");
+		return 0;
+		/*if (nmi_watchdog_enabled)
+			enable_timer_nmi_watchdog();
+		else
+			disable_timer_nmi_watchdog();
+		*/
+	} else {
+		printk( KERN_WARNING
+			"NMI watchdog doesn't know what hardware to touch\n");
+	}
+	return 0;
+}
+
 #endif
 
 EXPORT_SYMBOL(nmi_active);

Index: linux-don/arch/x86_64/kernel/nmi.c
===================================================================

arch/x86_64/kernel/nmi.c +46 -0
@@ -751,6 +751,52 @@ static int unknown_nmi_panic_callback(st
 	return 0;
 }
 
+/*
+ * proc handler for /proc/sys/kernel/nmi
+ */
+int proc_nmi_enabled(struct ctl_table *table, int write, struct file *file,
+			void __user *buffer, size_t *length, loff_t *ppos)
+{
+	int old_state;
+
+	nmi_watchdog_enabled = (atomic_read(&nmi_active) > 0) ? 1 : 0;
+	old_state = nmi_watchdog_enabled;
+	proc_dointvec(table, write, file, buffer, length, ppos);
+	if (!!old_state == !!nmi_watchdog_enabled)
+		return 0;
+
+	if (atomic_read(&nmi_active) < 0) {
+		printk( KERN_WARNING "NMI watchdog is permanently disabled\n");
+		return 0;
+	}
+
+	/* if nmi_watchdog is not set yet, then set it */
+	nmi_watchdog_default();
+
+	if (nmi_watchdog == NMI_LOCAL_APIC)
+	{
+		if (nmi_watchdog_enabled)
+			enable_lapic_nmi_watchdog();
+		else
+			disable_lapic_nmi_watchdog();
+	} else if (nmi_watchdog == NMI_IO_APIC) {
+		/* FIXME
+		 * for some reason these functions don't work
+		 */
+		printk("Can not enable/disable NMI on IO APIC\n");
+		return 0;
+		/*if (nmi_watchdog_enabled)
+			enable_timer_nmi_watchdog();
+		else
+			disable_timer_nmi_watchdog();
+		*/
+	} else {
+		printk( KERN_WARNING
+			"NMI watchdog doesn't know what hardware to touch\n");
+	}
+	return 0;
+}
+
 #endif
 
 EXPORT_SYMBOL(nmi_active);

Index: linux-don/include/asm-i386/nmi.h
===================================================================

include/asm-i386/nmi.h +1 -0
@@ -14,6 +14,7 @@
  */
 int do_nmi_callback(struct pt_regs *regs, int cpu);
 
+extern int nmi_watchdog_enabled;
 extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);
 extern int avail_to_resrv_perfctr_nmi(unsigned int);
 extern int reserve_perfctr_nmi(unsigned int);

Index: linux-don/include/asm-x86_64/nmi.h
===================================================================

include/asm-x86_64/nmi.h +1 -0
@@ -43,6 +43,7 @@ extern void die_nmi(char *str, struct pt
 
 extern int panic_on_timeout;
 extern int unknown_nmi_panic;
+extern int nmi_watchdog_enabled;
 
 extern int check_nmi_watchdog(void);
 extern int avail_to_resrv_perfctr_nmi_bit(unsigned int);

Index: linux-don/include/linux/sysctl.h
===================================================================

include/linux/sysctl.h +1 -0
@@ -148,6 +148,7 @@ enum
 	KERN_SPIN_RETRY=70,	/* int: number of spinlock retries */
 	KERN_ACPI_VIDEO_FLAGS=71, /* int: flags for setting up video after ACPI sleep */
 	KERN_IA64_UNALIGNED=72, /* int: ia64 unaligned userland trap enable */
+	KERN_NMI_ENABLED=73, /* int: enable/disable nmi watchdog */
 };
 
 

Index: linux-don/kernel/sysctl.c
===================================================================

kernel/sysctl.c +11 -2
@@ -75,6 +75,9 @@ extern int percpu_pagelist_fraction;
 
 #if defined(CONFIG_X86_LOCAL_APIC) && defined(CONFIG_X86)
 int unknown_nmi_panic;
+int nmi_watchdog_enabled;
+extern int proc_nmi_enabled(struct ctl_table *, int , struct file *,
+			void __user *, size_t *, loff_t *);
 #endif
 
 /* this is needed for the proc_dointvec_minmax for [fs_]overflow UID and GID */
@@ -630,6 +633,14 @@ static ctl_table kern_table[] = {
 		.mode           = 0644,
 		.proc_handler   = &proc_dointvec,
 	},
+	{
+		.ctl_name       = KERN_NMI_ENABLED,
+		.procname       = "nmi",
+		.data           = &nmi_watchdog_enabled,
+		.maxlen         = sizeof (int),
+		.mode           = 0644,
+		.proc_handler   = &proc_nmi_enabled,
+	},
 #endif
 #if defined(CONFIG_X86)
 	{
 
--
 
 
-------------------------------------------------------

Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
oprofile-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/oprofile-list

Signature · 1 line
-Stephane
#14 lore raw

Re: [patch 8/8] Add abilty to enable/disable nmi watchdog from sysfs

Quoted 10 lines · dzickus
Quoted 4 lines

Adds a new /proc/sys/kernel/nmi call that will enable/disable the nmi watchdog.

This means you can at runtime enable/disbale nmi_watchdog, i.e., reserve some performance counters on the fly. This gets complicated because now the perfmon subsystem

Right now we don't care about perfmon at all because it's not in (x86) mainline If you want anybody to care you have to submit and pass review

Quoted 5 lines

(and probably oprofile) cannot check register
availability when they are first initialized. Basically each time, the /sys entry is modified, they would have to scan the list of available performance counters. I don't know exactly when Oprofile does this checking. For perfmon, this is done only once, when the PMU description table is loaded.

I think the NMI watchdog will fail if the register is already allocated. oprofile should check and allocate when it fills in the register.

-Andi

author
#15 lore raw

Re: [patch 8/8] Add abilty to enable/disable nmi watchdog from sysfs

In reply to Stephane Eranian
Quoted 4 lines · Stephane Eranian

Don,

Congratulations on the patch. I am glad to see that some of the SMP issues I reported a long time ago are now fixed.

Thanks and glad I could help.

Quoted 11 lines
Quoted 3 lines

Adds a new /proc/sys/kernel/nmi call that will enable/disable the nmi watchdog.

This means you can at runtime enable/disbale nmi_watchdog, i.e., reserve some performance counters on the fly. This gets complicated because now the perfmon subsystem (and probably oprofile) cannot check register availability when they are first initialized. Basically each time, the /sys entry is modified, they would have to scan the list of available performance counters. I don't know exactly when Oprofile does this checking. For perfmon, this is done only once, when the PMU description table is loaded.

How often did you plan on enabling/disabling the nmi_watchdog? My understanding was you disable nmi_watchdog, run oprofile/perfmon, re-enable nmi_watchdog. I guess I don't understand what type of funky scenarios you are dealing with.

Quoted 6 lines

Also something that I did not see in this code is the error detection in case enable_lapic_nmi_watchdog() fails. Oprofile runs on all CPUs or none. Perfmon lets you monitor on subsets on CPUs. In case NMI was disabled and a monitoring session was active on some CPUs. The enable_lapic_nmi_watchdog() will fail on some CPUs. How is that handled?

It's not. In fact I wouldn't know what to do in such situations. Is it really wrong to only have a subset of cpus being monitored by the nmi_watchdog? This seems to be wandering into the area where the user is looking to do something complicated (profiling a subset of cpus) and as such might be expected to make sure the nmi_watchdog is properly enabled on all cpus when they are done.

Cheers,
Don

#16 lore raw

Re: [patch 8/8] Add abilty to enable/disable nmi watchdog from sysfs

Don,

Quoted 9 lines · Don Zickus
Quoted 8 lines

This means you can at runtime enable/disbale nmi_watchdog, i.e., reserve some performance counters on the fly. This gets complicated because now the perfmon subsystem (and probably oprofile) cannot check register availability when they are first initialized. Basically each time, the /sys entry is modified, they would have to scan the list of available performance counters. I don't know exactly when Oprofile does this checking. For perfmon, this is done only once, when the PMU description table is loaded.

Quoted 5 lines

How often did you plan on enabling/disabling the nmi_watchdog? My understanding was you disable nmi_watchdog, run oprofile/perfmon, re-enable nmi_watchdog. I guess I don't understand what type of funky scenarios you are dealing with.

I thought that part of the exercise was to allow oprofile/perfmon and nmi_watchdog to work simultaneously, i.e., don't disable watchdog when you monitor. Leave watchdog on and have oprofile/perfmon run with fewer counters. The alternative would be to do what you say: disbale nmi, monitor, re-enable nmi.

Quoted 7 lines
Quoted 6 lines

Also something that I did not see in this code is the error detection in case enable_lapic_nmi_watchdog() fails. Oprofile runs on all CPUs or none. Perfmon lets you monitor on subsets on CPUs. In case NMI was disabled and a monitoring session was active on some CPUs. The enable_lapic_nmi_watchdog() will fail on some CPUs. How is that handled?

Quoted 7 lines

It's not. In fact I wouldn't know what to do in such situations. Is it really wrong to only have a subset of cpus being monitored by the nmi_watchdog? This seems to be wandering into the area where the user is looking to do something complicated (profiling a subset of cpus) and as such might be expected to make sure the nmi_watchdog is properly enabled on all cpus when they are done.

On larger machines (think NUMA-style), it does not always make sense to monitor all CPUs.

Signature · 1 line
-Stephane
author
#17 lore raw

Re: [patch 8/8] Add abilty to enable/disable nmi watchdog from sysfs

Quoted 17 lines · Stephane Eranian

Don,

Quoted 9 lines · Don Zickus
Quoted 8 lines

This means you can at runtime enable/disbale nmi_watchdog, i.e., reserve some performance counters on the fly. This gets complicated because now the perfmon subsystem (and probably oprofile) cannot check register availability when they are first initialized. Basically each time, the /sys entry is modified, they would have to scan the list of available performance counters. I don't know exactly when Oprofile does this checking. For perfmon, this is done only once, when the PMU description table is loaded.

Quoted 5 lines

How often did you plan on enabling/disabling the nmi_watchdog? My understanding was you disable nmi_watchdog, run oprofile/perfmon, re-enable nmi_watchdog. I guess I don't understand what type of funky scenarios you are dealing with.

Quoted 4 lines

I thought that part of the exercise was to allow oprofile/perfmon and nmi_watchdog to work simultaneously, i.e., don't disable watchdog when you monitor. Leave watchdog on and have oprofile/perfmon run with fewer counters. The alternative would be to do what you say: disbale nmi, monitor, re-enable nmi.

It was and the patch allows one to run the monitor and the nmi_watchdog at the same time. I guess I am confused to what your original concern was.

Quoted 8 lines
Quoted 8 lines
Quoted 7 lines

Also something that I did not see in this code is the error detection in case enable_lapic_nmi_watchdog() fails. Oprofile runs on all CPUs or none. Perfmon lets you monitor on subsets on CPUs. In case NMI was disabled and a monitoring session was active on some CPUs. The enable_lapic_nmi_watchdog() will fail on some CPUs. How is that handled?

Quoted 8 lines
Quoted 7 lines

It's not. In fact I wouldn't know what to do in such situations. Is it really wrong to only have a subset of cpus being monitored by the nmi_watchdog? This seems to be wandering into the area where the user is looking to do something complicated (profiling a subset of cpus) and as such might be expected to make sure the nmi_watchdog is properly enabled on all cpus when they are done.

On larger machines (think NUMA-style), it does not always make sense to monitor all CPUs.

That makes sense. Thinking about it, it probably wouldn't be too hard to add code to try and re-scan the cpus and enable those nmi_watchdogs that were disabled during your monitoring. But you would have to manually do this once you were done testing.

Cheers,
Don