Skip to contentSkip to conversation

x86/kernel: Add generic handler for NMI events

PATCH New Adrien Mahieux 2 participants 2 messages May 27, 2017 – May 28, 2017

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/kernel: Add generic handler for NMI events

The kernel allows drivers to register NMI handlers, or to use sysctls like unknown_nmi_panic to control the behavior of the system when an NMI is received by the kernel.

This patch adds a generic handler where sysadmins can specify the behaviour to adopt for each NMI event code. List of events is provided at module load or on kernel cmdline, so we can also generate kdump upon boot error. We can specify events to:
- silently ignore: let them go through other handler, and don't log them.
- drop, stop their propagation to other handlers and log it.
- panic, so the kernel will panic (and generate a kdump if set).

A common use-case in servers is to make the kernel panic to get a coredump by kdump/kexec facility. With the drop parameter, it avoids a running-kdump to be stopped by receiving another NMI, thus making the vmcore unusable.

Manufacturers don't provide kernel for this and even the HP's hpwdt driver panics on any NMI.
This is an issue when we have other tools or hardware that generates NMI for non critical events (failing Power supply, FPGA, watchdog...).

Against the warnings of checkpatch.pl, I've left the kernel-version macro as it would be easier to be integrated in current (and old) kernels of production servers.
If you think it's the distrib maintainer's job to backport features, please let me know and I'll replace this by a link to a repo with it.

---
 Documentation/nmimgr.txt | 134 +++++++++++++++++
 arch/Kconfig             |  18 +++
 arch/x86/kernel/Makefile |   1 +
 arch/x86/kernel/nmimgr.c | 374 +++++++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 527 insertions(+)
 create mode 100644 Documentation/nmimgr.txt
 create mode 100644 arch/x86/kernel/nmimgr.c
Documentation/nmimgr.txt new file +134 -0
new file mode 100644
index 000000000000..dbda626387c3
@@ -0,0 +1,134 @@
+NMI Generic Handler
+===================
+
+This module allows you to Panic or Ignore specific NMI events.
+
+
+Author: Adrien Mahieux <[email protected]>
+Tools:  https://github.com/saruspete/kdumptools
+
+
+Description
+-----------
+
+Manage NMI events in a more fine-grained manner than unknown_nmi_panic sysctl.
+
+
+When a host is unresponsive, we'd like to take a vmcore for offline analysis.
+If kdump is correctly setup, we need to crash/panic the system for it to start.
+
+This is usually done by sending an NMI to the system (as no userland process
+is responding anymore) through the BMC.
+
+If no handler registers the vendor-specific NMI event to trigger a crash,
+the kernel logs a "Dazed and confused, but trying to continue" message and
+server is still unresponsive.
+
+
+Why not just using "nmi_panic" sysctls ?
+----------------------------------------
+
+There is 3 sysctls that allows administrators to generate a panic:
+- panic_on_io_nmi
+- panic_on_unrecovered_nmi
+- unknown_nmi_panic
+
+These sysctl are overkill as multiple NMI can be generated for non-critical
+events like:
+- Software debugging, like perf on Pentium processors
+- External cards like FPGA to communicate
+- Motherboard alerts of a dying Power-Supply
+
+However, the NMI event changes between
+vendors and/or revisions. Also, you may get NMI for non-critical events, like
+a redundent power-supply unit dead, or a FPGA card being stuck.
+
+If you are fine with the current unknown_nmi_panic settings, this module can
+also be used to ignore other NMIs during the dump process, even those who have
+a kernel module for handling. This avoid the interruption of the dump process,
+thus having a non-usable vmcore.
+
+
+Parameters
+----------
+
+Available events are (in processing order):
+
+- events_ignore=LIST Events to ignore silently, let them pass to other handlers
+- events_drop=LISt   Events to drop, so no other handler can process them
+- events_panic=LIST  Events to make the kernel Panic
+
+LIST is standard kernel lists, can be composed of
+- simple lists:  0,13,16,44,10
+- ranges:        10-100
+- Mix of both:   0,1,2-8,10
+
+
+Like all modules, you can also configure it from boot cmd by prefixing the
+module name and a dot:
+  nmimgr.events_ignore=... nmimgr.events_drop=... nmimgr.events_panic=...
+
+
+
+
+Discover your hardware NMI events
+---------------------------------
+
+When trying new hardware, you'll have to discover the NMI events assigned
+to the external generator.
+
+To ease this task, you can use the setup.sh script located at:
+  https://raw.githubusercontent.com/Saruspete/nmimgr/master/setup.sh
+
+
+If you want to do it manually, or if your hardware is not referenced yet:
+
+- ensure all nmi sysctls are disabled (sysctl -a|grep _nmi)
+- load the module without parameter
+- generate an NMI from your BMC
+- check dmesg for lines containing "nmimgr:", specifically "Handling new NMI".
+- The code you are interested in (the event) is the decimal value between ( )
+
+If you see this log: "Handling new NMI type:1 event:0x10 (16)"
+Then the event code generated is 16.
+To make the system panic:
+  # modprobe nmimgr events_panic=16
+To ignore it and disable messages:
+  # modprobe nmimgr events_ignore=16
+
+
+Configure it permanently
+------------------------
+
+If you want to be able to reconfigure it, and if built as a module:
+    # echo "options nmimgr events_panic=16 events_drop=10" > \
+	/etc/modprobe.d/nmimgr.conf
+
+If it's built-in, change the cmdline boot, adding parameters
+
+2) Reload it:
+    # rmmod nmimgr; modprobe nmimgr
+3) Check the parameters are correctly set (you should see your value)
+    # cat /sys/module/nmimgr/parameters/events_panic
+
+
+
+Generate an NMI
+---------------
+
+- ipmitool chassis power diag
+- vboxmanage debugvm "VMName" injectnmi
+- virsh inject-nmi "VMName"
+
+
+Kernel Revision history
+-----------------------
+
+2.6.32: Using notifier_block structs
+3.2   : Moved NMI descriptions to an enum: LOCAL, UNKNOWN, MAX
+        https://lwn.net/Articles/461215/
+        https://lkml.org/lkml/2012/3/8/386
+3.5   : Moved "register_nmi_handler" to a macro + static struct nmiaction fn##_na
+        This broke the loop logic used between 3.2 and 3.5
+
+
arch/Kconfig +18 -0
index 6c00e5b00f8b..339a5eeb0591 100644
@@ -12,6 +12,24 @@ config KEXEC_CORE
 config HAVE_IMA_KEXEC
 	bool
 
+config NMI_MGR
+	tristate "NMI Generic Handler"
+	depends on HAVE_NMI
+	depends on X86
+	help
+	  NMI Generic handler allows admins to specify NMI Events to
+	  drop, ignore or make the system panic.
+	  This is needed for servers with BMC (HP ILO, Dell IDRAC,
+	  AMI ASMB...) to specify events instead of killall sysctl
+	  "unknown_nmi_panic".
+
+	  It can be used to avoid kdump being interrupted by double NMI
+
+	  See <file:Documentation/nmimgr.txt>
+
+	  If unsure, say N.
+
+
 config OPROFILE
 	tristate "OProfile system profiling"
 	depends on PROFILING
arch/x86/kernel/Makefile +1 -0
index 4b994232cb57..a37fdfe31849 100644
@@ -42,6 +42,7 @@ obj-y			:= process_$(BITS).o signal.o
 obj-$(CONFIG_COMPAT)	+= signal_compat.o
 obj-y			+= traps.o irq.o irq_$(BITS).o dumpstack_$(BITS).o
 obj-y			+= time.o ioport.o dumpstack.o nmi.o
+obj-$(CONFIG_NMI_MGR)	+= nmimgr.o
 obj-$(CONFIG_MODIFY_LDT_SYSCALL)	+= ldt.o
 obj-y			+= setup.o x86_init.o i8259.o irqinit.o jump_label.o
 obj-$(CONFIG_IRQ_WORK)  += irq_work.o
arch/x86/kernel/nmimgr.c new file +374 -0

374 lines not on this page · raw message

Signature · 1 line
2.5.5
#1 lore raw
url: https://github.com/0day-ci/linux/commits/Adrien-Mahieux/x86-kernel-Add-generic-handler-for-NMI-events/20170527-232216 config: x86_64-randconfig-b0-05281932 (attached as .config) compiler…

Hi Adrien,

[auto build test ERROR on tip/auto-latest]
[also build test ERROR on v4.12-rc2 next-20170526]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]

url: https://github.com/0day-ci/linux/commits/Adrien-Mahieux/x86-kernel-Add-generic-handler-for-NMI-events/20170527-232216
config: x86_64-randconfig-b0-05281932 (attached as .config)
compiler: gcc-4.4 (Debian 4.4.7-8) 4.4.7 reproduce:

        # save the attached .config to linux build tree
        make ARCH=x86_64 

All errors (new ones prefixed by >>):

arch/x86/kernel/nmimgr.c: In function 'nmimgr_handle':

>> arch/x86/kernel/nmimgr.c:183: error: 'x86_platform' undeclared (first use in this function)

arch/x86/kernel/nmimgr.c:183: error: (Each undeclared identifier is reported only once arch/x86/kernel/nmimgr.c:183: error: for each function it appears in.)

vim +/x86_platform +183 arch/x86/kernel/nmimgr.c

177
178 /***** Kernel 3.2+ ***********************************************************/
179 #else
180
181 static int nmimgr_handle(unsigned int type, struct pt_regs *regs)
182 {

 > 183		return __nmimgr_handle(type, x86_platform.get_nmi_reason(), regs);

184 }
185
186

---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation