Skip to contentSkip to conversation

notifier/panic: Introduce panic_notifier_filter

PATCH Superseded Guilherme G. Piccoli 3 participants 5 messages Jan 6, 2022 – Jan 7, 2022

Subsystems (3)

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 notifier/panic: Introduce panic_notifier_filter

The kernel notifier infrastructure allows function callbacks to be added in multiple lists, which are then called in the proper time, like in a reboot or panic event. The panic_notifier_list specifically contains the callbacks that are executed during a panic event. As any other notifier list, the panic one has no filtering and all functions previously registered are executed.

The kdump infrastructure, on the other hand, enables users to set a crash kernel that is kexec'ed in a panic event, and vmcore/logs are collected in such crash kernel. When kdump is set, by default the panic notifiers are ignored - the kexec jumps to the crash kernel before the list is checked and callbacks executed.

There are some cases though in which kdump users might want to allow panic notifier callbacks to execute _before_ the kexec to the crash kernel, for a variety of reasons - for example, users may think kexec is very prone to fail and want to give a chance to kmsg dumpers to run (and save logs using pstore), or maybe some panic notifier is required to properly quiesce some hardware that must be used to the crash kernel. For these cases, we have the kernel parameter "crash_kexec_post_notifiers".

But there's a problem: currently it's an "all-or-nothing" situation, the kdump user choice is either to execute all panic notifiers or none of them. Given that panic notifiers may increase the risk of a kdump failure, this is a tough decision and may affect the debug of hard to reproduce bugs, if for some reason the user choice is to enable panic notifiers, but kdump then fails.

So, this patch aims to ease this decision: we hereby introduce a filter for the panic notifier list, in which users may select specifically which callbacks they wish to run, allowing a safer kdump. The allowlist should be provided using the parameter "panic_notifier_filter=a,b,..." where a, b are valid callback names. Invalid symbols are discarded.

Currently up to 16 symbols may be passed in this list, we consider that this numbers allows enough flexibility (and no matter what architecture is used, at most 30 panic callbacks are registered). In an experiment using a qemu x86 virtual machine, by default only six callbacks are registered in the panic notifier list. Once a valid callback name is provided in the list, such function is allowed to be registered/unregistered in the panic_notifier_list; all other panic callbacks are ignored. Notice that this filter is only for the panic notifiers and has no effect in the other notifiers.

---

V2: Sorry, I've made a confusion/typo in the header file, by inverting the types of 2 variables. Fixed now, and below I kept the design comments. Cheers!

Hi folks, thanks in advance for reviews/suggestions! Some design decisions are worth to mention:

(a) I've played first with a list approach instead of a static array, but noticed that..it'd require relying in memblock allocs, since we need to set the filter pretty early in boot process. So, decided to abandon that - a small array is cheap and much easier to implement.

(b) The size of the array: I've counted < 30 panic notifiers for any architecture, and in my experiment only six are registered by default, in a qemu x86 guest. Also, doesn't make sense for the filter users to add a bunch of callbacks to the list, or else why are they using the filter in first place? So 16 seems a good trade-off.

(c) Allowlist vs. denylist: this is something to consider. My approach is that functions listed in the filter are *allowed* to execute in the panic notifier, because this seems to me the most appropriate use case - kdump users might want one or 2 callbacks to execute, and block all the other in a denylist seems bummer. But I'm open to ideas, of course, if the denylist approach seems more reasonable.

(d) I've also tested __setup() instead of early_param(), but seems we don't catch all notifiers by then, so decided to set the filter really early.

(e) Finally, an alternative would be to check the filter during the notifier call chain, but I thought it is simple and likely less invasive to just prevent registering the disallowed functions.

Cheers,

Guilherme

 .../admin-guide/kernel-parameters.txt         | 14 +++++-
 include/linux/panic_notifier.h                | 10 +++++
 kernel/notifier.c                             | 42 +++++++++++++++++-
 kernel/panic.c                                | 44 +++++++++++++++++++
 4 files changed, 107 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +13 -1
index 2fba82431efb..2dc4e98823ae 100644
@@ -3727,13 +3727,25 @@
 	panic_on_warn	panic() instead of WARN().  Useful to cause kdump
 			on a WARN().
 
+	panic_notifier_filter=[function-list]
+			Limit the functions registered by the panic notifier
+			infrastructure. This allowlist is composed by function
+			names, comma separated (invalid symbols are filtered
+			out). Such functionality is useful for kdump users
+			that set "crash_kexec_post_notifiers" in order to
+			execute	panic notifiers, but@the same time wish to
+			have just a subset of notifiers, not all of them. The
+			list of functions is limited to 16 entries currently.
+
 	crash_kexec_post_notifiers
 			Run kdump after running panic-notifiers and dumping
 			kmsg. This only for the users who doubt kdump always
 			succeeds in any situation.
 			Note that this also increases risks of kdump failure,
 			because some panic notifiers can make the crashed
-			kernel more unstable.
+			kernel more unstable. See the "panic_notifier_filter"
+			parameter to have more control of which notifiers to
+			execute.
 
 	parkbd.port=	[HW] Parallel port number the keyboard adapter is
 			connected to, default is 0.
include/linux/panic_notifier.h +10 -0
index 41e32483d7a7..9a96753e96d8 100644
@@ -5,6 +5,16 @@
 #include <linux/notifier.h>
 #include <linux/types.h>
 
+/*
+ * The panic notifier filter infrastructure - each array element holds a
+ * function address, to be checked against panic_notifier register/unregister
+ * operations; these functions are allowed to be registered in the panic
+ * notifier list. This setting is useful for kdump, since users may want
+ * some panic notifiers to execute, but not all of them.
+ */
+extern unsigned long panic_nf_functions[];
+extern int panic_nf_count;
+
 extern struct atomic_notifier_head panic_notifier_list;
 
 extern bool crash_kexec_post_notifiers;
kernel/notifier.c +40 -2

82 lines not on this page · raw message

kernel/panic.c +44 -0

62 lines not on this page · raw message

Signature · 1 line
2.34.1
#1 lore raw
Quoted 52 lines · Guilherme G. Piccoli

The kernel notifier infrastructure allows function callbacks to be added in multiple lists, which are then called in the proper time, like in a reboot or panic event. The panic_notifier_list specifically contains the callbacks that are executed during a panic event. As any other notifier list, the panic one has no filtering and all functions previously registered are executed.

The kdump infrastructure, on the other hand, enables users to set a crash kernel that is kexec'ed in a panic event, and vmcore/logs are collected in such crash kernel. When kdump is set, by default the panic notifiers are ignored - the kexec jumps to the crash kernel before the list is checked and callbacks executed.

There are some cases though in which kdump users might want to allow panic notifier callbacks to execute _before_ the kexec to the crash kernel, for a variety of reasons - for example, users may think kexec is very prone to fail and want to give a chance to kmsg dumpers to run (and save logs using pstore), or maybe some panic notifier is required to properly quiesce some hardware that must be used to the crash kernel. For these cases, we have the kernel parameter "crash_kexec_post_notifiers".

But there's a problem: currently it's an "all-or-nothing" situation, the kdump user choice is either to execute all panic notifiers or none of them. Given that panic notifiers may increase the risk of a kdump failure, this is a tough decision and may affect the debug of hard to reproduce bugs, if for some reason the user choice is to enable panic notifiers, but kdump then fails.

So, this patch aims to ease this decision: we hereby introduce a filter for the panic notifier list, in which users may select specifically which callbacks they wish to run, allowing a safer kdump. The allowlist should be provided using the parameter "panic_notifier_filter=a,b,..." where a, b are valid callback names. Invalid symbols are discarded.

Currently up to 16 symbols may be passed in this list, we consider that this numbers allows enough flexibility (and no matter what architecture is used, at most 30 panic callbacks are registered). In an experiment using a qemu x86 virtual machine, by default only six callbacks are registered in the panic notifier list. Once a valid callback name is provided in the list, such function is allowed to be registered/unregistered in the panic_notifier_list; all other panic callbacks are ignored. Notice that this filter is only for the panic notifiers and has no effect in the other notifiers.

kernel/notifier.c +0 -0
index b8251dc0bc0f..04cb9e956058 100644

kernel/notifier.c int atomic_notifier_chain_register(struct atomic_notifier_head *nh, · line 171

Quoted 10 lines
kernel/notifier.c +4 -1
@@ -140,10 +163,16 @@ int atomic_notifier_chain_register(struct atomic_notifier_head *nh,
 		struct notifier_block *n)
 {
 	unsigned long flags;
-	int ret;
+	int ret = 0;
 
 	spin_lock_irqsave(&nh->lock, flags);
+	if (unlikely(panic_nf_count) && nh == &panic_notifier_list)
+		if (!is_panic_notifier_filtered(n))
+			goto panic_filtered_out;

Forget the unlikely(); this is not a hot path.

kernel/notifier.c int atomic_notifier_chain_register(struct atomic_notifier_head *nh, · line 178

Quoted 7 lines
kernel/notifier.c +3 -0
+
 	ret = notifier_chain_register(&nh->head, n);
+
+panic_filtered_out:
 	spin_unlock_irqrestore(&nh->lock, flags);
 	return ret;
 }

It would be simpler to do:

	if (!(nh == &panic_notifier_list && panic_nf_count > 0 &&
			is_panic_notifier_filtered(n)))
		ret = notifier_chain_register(&nh->head, n);

If there were special-purpose functions just for registering and unregistering callbacks on the panic_notifier_list, the design would be cleaner (no need to modify core notifier code). But making that change would mean altering a lot of call sites. :-(

Quoted 5 lines
kernel/notifier.c +0 -1
@@ -162,10 +194,16 @@ int atomic_notifier_chain_unregister(struct atomic_notifier_head *nh,
 		struct notifier_block *n)
 {
 	unsigned long flags;
-	int ret;

kernel/notifier.c int atomic_notifier_chain_unregister(struct atomic_notifier_head *nh, · line 206

Quoted 10 lines
kernel/notifier.c +7 -0
+	int ret = 0;
 
 	spin_lock_irqsave(&nh->lock, flags);
+	if (unlikely(panic_nf_count) && nh == &panic_notifier_list)
+		if (!is_panic_notifier_filtered(n))
+			goto panic_filtered_out;
+
 	ret = notifier_chain_unregister(&nh->head, n);
+
+panic_filtered_out:

Same idea here.

Alan Stern

#2 lore raw
Quoted 7 lines · Alan Stern

[...]

kernel/notifier.c int atomic_notifier_chain_register(struct atomic_notifier_head *nh, · line 171

Quoted 3 lines
kernel/notifier.c +3 -0
+	if (unlikely(panic_nf_count) && nh == &panic_notifier_list)
+		if (!is_panic_notifier_filtered(n))
+			goto panic_filtered_out;

Forget the unlikely(); this is not a hot path.

Thanks for the review Alan! Sure, I can remove that in the V3 - good point.

Quoted 10 lines

kernel/notifier.c int atomic_notifier_chain_register(struct atomic_notifier_head *nh, · line 178

Quoted 7 lines
kernel/notifier.c +3 -0
+
 	ret = notifier_chain_register(&nh->head, n);
+
+panic_filtered_out:
 	spin_unlock_irqrestore(&nh->lock, flags);
 	return ret;
 }

It would be simpler to do:

Quoted 9 lines
	if (!(nh == &panic_notifier_list && panic_nf_count > 0 &&
			is_panic_notifier_filtered(n)))
		ret = notifier_chain_register(&nh->head, n);

If there were special-purpose functions just for registering and unregistering callbacks on the panic_notifier_list, the design would be cleaner (no need to modify core notifier code). But making that change would mean altering a lot of call sites. :-(

Good suggestion, I'll change the code with your suggestion in the V3.

And I agree, I considered something like that when writing this "hijack" into core code just for the sake of panic notifier filtering... I'm not sure though it worth the effort and code addition, to change all the call sites and add a special register mechanism to the panic notifier only. Let's see what the others think...

Cheers,

Guilherme

Quoted 20 lines
Quoted 5 lines
kernel/notifier.c +0 -1
@@ -162,10 +194,16 @@ int atomic_notifier_chain_unregister(struct atomic_notifier_head *nh,
 		struct notifier_block *n)
 {
 	unsigned long flags;
-	int ret;

kernel/notifier.c int atomic_notifier_chain_unregister(struct atomic_notifier_head *nh, · line 206

Quoted 10 lines
kernel/notifier.c +7 -0
+	int ret = 0;
 
 	spin_lock_irqsave(&nh->lock, flags);
+	if (unlikely(panic_nf_count) && nh == &panic_notifier_list)
+		if (!is_panic_notifier_filtered(n))
+			goto panic_filtered_out;
+
 	ret = notifier_chain_unregister(&nh->head, n);
+
+panic_filtered_out:

Same idea here.

Alan Stern

#3 lore raw
In reply to Guilherme G. Piccoli
Quoted 44 lines · Guilherme G. Piccoli

The kernel notifier infrastructure allows function callbacks to be added in multiple lists, which are then called in the proper time, like in a reboot or panic event. The panic_notifier_list specifically contains the callbacks that are executed during a panic event. As any other notifier list, the panic one has no filtering and all functions previously registered are executed.

The kdump infrastructure, on the other hand, enables users to set a crash kernel that is kexec'ed in a panic event, and vmcore/logs are collected in such crash kernel. When kdump is set, by default the panic notifiers are ignored - the kexec jumps to the crash kernel before the list is checked and callbacks executed.

There are some cases though in which kdump users might want to allow panic notifier callbacks to execute _before_ the kexec to the crash kernel, for a variety of reasons - for example, users may think kexec is very prone to fail and want to give a chance to kmsg dumpers to run (and save logs using pstore), or maybe some panic notifier is required to properly quiesce some hardware that must be used to the crash kernel. For these cases, we have the kernel parameter "crash_kexec_post_notifiers".

But there's a problem: currently it's an "all-or-nothing" situation, the kdump user choice is either to execute all panic notifiers or none of them. Given that panic notifiers may increase the risk of a kdump failure, this is a tough decision and may affect the debug of hard to reproduce bugs, if for some reason the user choice is to enable panic notifiers, but kdump then fails.

So, this patch aims to ease this decision: we hereby introduce a filter for the panic notifier list, in which users may select specifically which callbacks they wish to run, allowing a safer kdump. The allowlist should be provided using the parameter "panic_notifier_filter=a,b,..." where a, b are valid callback names. Invalid symbols are discarded.

Currently up to 16 symbols may be passed in this list, we consider that this numbers allows enough flexibility (and no matter what architecture is used, at most 30 panic callbacks are registered). In an experiment using a qemu x86 virtual machine, by default only six callbacks are registered in the panic notifier list. Once a valid callback name is provided in the list, such function is allowed to be registered/unregistered in the panic_notifier_list; all other panic callbacks are ignored. Notice that this filter is only for the panic notifiers and has no effect in the other notifiers.

...

kernel/panic.c void nmi_panic(struct pt_regs *regs, const char *msg) · line 169

Quoted 10 lines
kernel/panic.c +10 -0
+static int __init panic_notifier_filter_setup(char *buf)
+{
+	char *func;
+	unsigned long addr;
+
+	if (!buf)
+		return -EINVAL;
+
+	while (buf) {
+		func = strsep(&buf, ",");

Don't we have a parser of this format already?
Anyway, you may reduce code by

	unsigned long addr;
	char *func = buf;

	while ((func = strsep(&func, ",")) {
Quoted 22 lines
kernel/panic.c +22 -0
+		addr = kallsyms_lookup_name(func);
+
+		if (!addr) {
+			pr_info("panic_notifier_filter: invalid symbol %s\n", func);
+			continue;
+		}
+
+		if (panic_nf_count < PANIC_NF_MAX) {
+			panic_nf_functions[panic_nf_count] = addr;
+			panic_nf_count++;
+			pr_debug("panic_notifier_filter: added symbol %s\n", func);
+		} else {
+			pr_warn("panic_notifier_filter: exceeded maximum notifiers (%d), aborting\n",
+				PANIC_NF_MAX);
+			panic_nf_count = 0;
+			break;
+		}
+	}
+
+	return 0;
+}
+early_param("panic_notifier_filter", panic_notifier_filter_setup);
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#4 lore raw
Quoted 7 lines · Andy Shevchenko

[...]

kernel/panic.c void nmi_panic(struct pt_regs *regs, const char *msg) · line 169

Quoted 2 lines
kernel/panic.c +2 -0
+	while (buf) {
+		func = strsep(&buf, ",");

Don't we have a parser of this format already?
Anyway, you may reduce code by

Quoted 5 lines
	unsigned long addr;
	char *func = buf;

	while ((func = strsep(&func, ",")) {

Hi Andy, thanks for the review! Looking the kernel code for some similar parameters, I couldn't see any better alternative (like a helper) to do this parsing - checked for example the ftrace parameters (borrowed my code from there, in fact heh), initcall_blacklist, module_blacklist, usbcore.quirks, etc.

I'll try to follow your suggestion, to reduce a bit the code...though I confess the original code reads more clearly for me.

Cheers,

Guilherme