The panic notifiers refactor
Subsystems (42)
cover The panic notifiers refactor
Hey folks, this is an attempt to improve/refactor the dated panic notifiers infrastructure. This is strongly based in a suggestion made by Pter Mladek [0] some time ago, and it's finally ready. Below I'll detail the patch ordering, testing made, etc.
First, a bit about the reason behind this.
The panic notifiers list is an infrastructure that allows callbacks to execute during panic time. Happens that anybody can add functions there, no ordering is enforced (by default) and the decision to execute or not such notifiers before kdump may lead to high risk of failure in crash scenarios - default is not to execute any of them. There is a parameter acting as a switch for that. But some architectures require some notifiers, so..it's messy.
The suggestion from Petr came after a patch submission to add a notifiers filter, allowing the notifiers selection by function name, which was welcomed by some people, but not by Petr, which claimed the code should indeed have a refactor - and it made a lot of sense, his suggestion makes code more clear and reliable.
So, this series might be split in 3 portions:
Part 1: the first 18 patches are mostly fixes (one or two might be considered improvements), mostly replacing spinlocks/mutexes with safer alternatives for atomic contexts, like spin_trylock, etc. We also focused on commenting everything that is possible and clean-up code.
Part 2, the core: patches 19-25 are the main refactor, which splits the panic notifiers list in three, introduce the concept of panic notifier level and clean-up and highly comment the code, effectively leading to a more reliable and clear, yet highly customizable panic path.
Part 3: The remaining 5 patches are fixes that _require the main refactor_ patches, they don't make sense without the core changes - but again, these are small fixes and not part of the main goal of refactoring the panic code.
I've tried my best to make the patches the more "bisectable" as possible, so they tend to be self-contained and easy to backport (specially patches from part 1). Notice that the series is *based on 5.18-rc4* - usually a refactor like this would be based on linux-next, but since we have many fixes in the series, I kept it based on mainline tree. Of course I could change that in a subsequent iteration, if desired.
Since this touches multiple architectures and drivers, it's very difficult to test it really (by executing all touched code). So, my tests split in two
approaches: build tests and real tests, that involves panic triggering with and without kdump, changing panic notifiers level, etc.
Build tests (using cross-compilers): alpha, arm, arm64, mips (sgi 22 and 32), parisc, s390, sparc, um, x86_64 (couldn't get a functional xtensa cross compiler).
Real/full tests: x86_64 (Hyper-V and QEMU guests) + PowerPC (pseries guest).
Here is the link with the .config files used: https://people.igalia.com/gpiccoli/panic_notifiers_configs/
(tried my best to build all the affected code).
Finally, a bit about my CCing strategy: I've included everybody present in the original thread [0] plus some maintainers and other interested parties as CC in the full series. But the patches have individual CC lists, for people that are definitely related to them but might not care much for the whole series; nevertheless, _everybody_ mentioned at least once in some patch is CCed in this cover-letter. Hopefully I didn't forget to include anybody - all the mailing lists were CCed in the whole series. Apologies in advance if (a) you received emails you didn't want to or, (b) I forgot to include you but it was something considered interesting by you.
Thanks in advance for reviews / comments / suggestions! Cheers,
Guilherme
[0] https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/
Guilherme G. Piccoli (30):
x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart
ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
notifier: Add panic notifiers info and purge trailing whitespaces
firmware: google: Convert regular spinlock into trylock on panic path misc/pvpanic: Convert regular spinlock into trylock on panic path
soc: bcm: brcmstb: Document panic notifier action and remove useless header
mips: ip22: Reword PANICED to PANICKED and remove useless header powerpc/setup: Refactor/untangle panic notifiers
coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
alpha: Clean-up the panic notifier code
um: Improve panic notifiers consistency and ordering
parisc: Replace regular spinlock with spin_trylock on panic path s390/consoles: Improve panic notifiers reliability
panic: Properly identify the panic event to the notifiers' callbacks
bus: brcmstb_gisb: Clean-up panic/die notifiers
drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers
tracing: Improve panic/die notifiers
notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
panic: Add the panic hypervisor notifier list
panic: Add the panic informational notifier list
panic: Introduce the panic pre-reboot notifier list
panic: Introduce the panic post-reboot notifier list
printk: kmsg_dump: Introduce helper to inform number of dumpers
panic: Refactor the panic path
panic, printk: Add console flush parameter and convert panic_print to a notifier
Drivers: hv: Do not force all panic notifiers to execute before kdump
powerpc: Do not force all panic notifiers to execute before kdump
panic: Unexport crash_kexec_post_notifiers
powerpc: ps3, pseries: Avoid duplicate call to kmsg_dump() on panic
um: Avoid duplicate call to kmsg_dump()
.../admin-guide/kernel-parameters.txt | 54 ++- Documentation/admin-guide/sysctl/kernel.rst | 5 +- arch/alpha/kernel/setup.c | 40 +-- arch/arm/kernel/machine_kexec.c | 3 + arch/arm64/kernel/setup.c | 2 +- arch/mips/kernel/relocate.c | 2 +- arch/mips/sgi-ip22/ip22-reset.c | 13 +- arch/mips/sgi-ip32/ip32-reset.c | 3 +- arch/parisc/include/asm/pdc.h | 1 + arch/parisc/kernel/firmware.c | 27 +- arch/parisc/kernel/pdc_chassis.c | 3 +- arch/powerpc/include/asm/bug.h | 2 +- arch/powerpc/kernel/fadump.c | 8 - arch/powerpc/kernel/setup-common.c | 76 ++-- arch/powerpc/kernel/traps.c | 6 +- arch/powerpc/platforms/powernv/opal.c | 2 +- arch/powerpc/platforms/ps3/setup.c | 2 +- arch/powerpc/platforms/pseries/setup.c | 2 +- arch/s390/kernel/ipl.c | 4 +- arch/s390/kernel/setup.c | 19 +- arch/sparc/kernel/setup_32.c | 27 +- arch/sparc/kernel/setup_64.c | 29 +- arch/sparc/kernel/sstate.c | 3 +- arch/um/drivers/mconsole_kern.c | 10 +- arch/um/kernel/um_arch.c | 11 +- arch/x86/include/asm/cpu.h | 1 + arch/x86/kernel/crash.c | 8 +- arch/x86/kernel/reboot.c | 14 +- arch/x86/kernel/setup.c | 2 +- arch/x86/xen/enlighten.c | 2 +- arch/xtensa/platforms/iss/setup.c | 4 +- drivers/bus/brcmstb_gisb.c | 28 +- drivers/char/ipmi/ipmi_msghandler.c | 12 +- drivers/edac/altera_edac.c | 3 +- drivers/firmware/google/gsmi.c | 10 +- drivers/hv/hv_common.c | 12 - drivers/hv/vmbus_drv.c | 113 +++--- .../hwtracing/coresight/coresight-cpu-debug.c | 11 +- drivers/leds/trigger/ledtrig-activity.c | 4 +- drivers/leds/trigger/ledtrig-heartbeat.c | 4 +- drivers/leds/trigger/ledtrig-panic.c | 3 +- drivers/misc/bcm-vk/bcm_vk_dev.c | 6 +- drivers/misc/ibmasm/heartbeat.c | 16 +- drivers/misc/pvpanic/pvpanic.c | 14 +- drivers/net/ipa/ipa_smp2p.c | 5 +- drivers/parisc/power.c | 21 +- drivers/power/reset/ltc2952-poweroff.c | 4 +- drivers/remoteproc/remoteproc_core.c | 6 +- drivers/s390/char/con3215.c | 38 +- drivers/s390/char/con3270.c | 36 +- drivers/s390/char/raw3270.c | 18 + drivers/s390/char/raw3270.h | 1 + drivers/s390/char/sclp_con.c | 30 +- drivers/s390/char/sclp_vt220.c | 44 +-- drivers/s390/char/zcore.c | 5 +- drivers/soc/bcm/brcmstb/pm/pm-arm.c | 18 +- drivers/soc/tegra/ari-tegra186.c | 3 +- drivers/staging/olpc_dcon/olpc_dcon.c | 6 +- drivers/video/fbdev/hyperv_fb.c | 12 +- include/linux/console.h | 2 + include/linux/kmsg_dump.h | 7 + include/linux/notifier.h | 8 +- include/linux/panic.h | 3 - include/linux/panic_notifier.h | 12 +- include/linux/printk.h | 1 + kernel/hung_task.c | 3 +- kernel/kexec_core.c | 8 +- kernel/notifier.c | 48 ++- kernel/panic.c | 335 +++++++++++------- kernel/printk/printk.c | 76 ++++ kernel/rcu/tree.c | 1 - kernel/rcu/tree_stall.h | 3 +- kernel/trace/trace.c | 59 +-- .../selftests/pstore/pstore_crash_test | 5 +- 74 files changed, 953 insertions(+), 486 deletions(-)
Signature · 1 line
2.36.0
PATCH 1/30 x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart
In the panic path we have a list of functions to be called, the panic notifiers - such callbacks perform various actions in the machine's last breath, and sometimes users want them to run before kdump. We have the parameter "crash_kexec_post_notifiers" for that. When such parameter is used, the function "crash_smp_send_stop()" is executed to poweroff all secondary CPUs through the NMI-shootdown mechanism; part of this process involves disabling virtualization features in all CPUs (except the main one).
Now, in the emergency restart procedure we have also a way of disabling VMX in all CPUs, using the same NMI-shootdown mechanism; what happens though is that in case we already NMI-disabled all CPUs, the emergency restart fails due to a second addition of the same items in the NMI list, as per the following log output:
sysrq: Trigger a crash
Kernel panic - not syncing: sysrq triggered crash [...] Rebooting in 2 seconds.. list_add double add: new=<addr1>, prev=<addr2>, next=<addr1>. ------------[ cut here ]------------ kernel BUG at lib/list_debug.c:29!
invalid opcode: 0000 [#1] PREEMPT SMP PTI
In order to reproduce the problem, users just need to set the kernel parameter "crash_kexec_post_notifiers" *without* kdump set in any system with the VMX feature present.
Since there is no benefit in re-disabling VMX in all CPUs in case it was already done, this patch prevents that by guarding the restart routine against doubly issuing NMIs unnecessarily. Notice we still need to disable VMX locally in the emergency restart.
--- arch/x86/include/asm/cpu.h | 1 + arch/x86/kernel/crash.c | 8 ++++---- arch/x86/kernel/reboot.c | 14 ++++++++++++-- 3 files changed, 17 insertions(+), 6 deletions(-)
arch/x86/include/asm/cpu.h +1 -0
@@ -36,6 +36,7 @@ extern int _debug_hotplug_cpu(int cpu, int action);#endif #endif +extern bool crash_cpus_stopped;int mwait_usable (const struct cpuinfo_x86 *);unsigned int x86_family (unsigned int sig);
arch/x86/kernel/crash.c +4 -4
@@ -42,6 +42,8 @@#include <asm/crash.h> #include <asm/cmdline.h> +bool crash_cpus_stopped; +/* Used while preparing memory map entries for second kernel */ struct crash_memmap_data {struct boot_params *params;
@@ -108,9 +110,7 @@ void kdump_nmi_shootdown_cpus(void)/* Override the weak function in kernel/panic.c */ void crash_smp_send_stop (void ) { -static int cpus_stopped; - -if (cpus_stopped) +if (crash_cpus_stopped)return ;if (smp_ops.crash_stop_other_cpus)
@@ -118,7 +118,7 @@ void crash_smp_send_stop(void) elsesmp_send_stop (); - cpus_stopped = 1; + crash_cpus_stopped = true; }#else
arch/x86/kernel/reboot.c +12 -2
@@ -589,8 +589,18 @@ static void native_machine_emergency_restart(void)int orig_reboot_type = reboot_type;unsigned short mode; -if (reboot_emergency) -emergency_vmx_disable_all (); +/* +* We can reach this point in the end of panic path, having +* NMI-disabled all secondary CPUs. This process involves +* disabling the CPU virtualization technologies, so if that +* is the case, we only miss disabling the local CPU VMX... +*/ +if (reboot_emergency) { +if (!crash_cpus_stopped) +emergency_vmx_disable_all (); +else +cpu_emergency_vmxoff (); + }tboot_shutdown (TB_SHUTDOWN_REBOOT );
Signature · 1 line
2.36.0
Re: [PATCH 01/30] x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart
Quoted 40 lines · Guilherme G. Piccoli
In the panic path we have a list of functions to be called, the panic notifiers - such callbacks perform various actions in the machine's last breath, and sometimes users want them to run before kdump. We have the parameter "
crash_kexec_post_notifiers" for that. When such parameter is used, the function "crash_smp_send_stop()" is executed to poweroff all secondary CPUs through the NMI-shootdown mechanism; part of this process involves disabling virtualization features in all CPUs (except the main one).Now, in the emergency restart procedure we have also a way of disabling VMX in all CPUs, using the same NMI-shootdown mechanism; what happens though is that in case we already NMI-disabled all CPUs, the emergency restart fails due to a second addition of the same items in the NMI list, as per the following log output:
sysrq: Trigger a crash
Kernel panic - not syncing: sysrq triggered crash [...] Rebooting in 2 seconds.. list_add double add: new=<addr1>, prev=<addr2>, next=<addr1>. ------------[ cut here ]------------ kernel BUG at lib/list_debug.c:29!invalid opcode: 0000 [#1] PREEMPT SMP PTI
In order to reproduce the problem, users just need to set the kernel parameter "
crash_kexec_post_notifiers" *without* kdump set in any system with the VMX feature present.Since there is no benefit in re-disabling VMX in all CPUs in case it was already done, this patch prevents that by guarding the restart routine against doubly issuing NMIs unnecessarily. Notice we still need to disable VMX locally in the emergency restart.
Fixes ed72736183c4 ("x86/reboot: Force all cpus to exit VMX root if VMX is supported)Fixes 0ee59413c967 ("x86/panic: replace smp_send_stop() with kdump friendly version in panic path")Cc David P. Reed <[email protected]>Cc Hidehiro Kawai <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Sean Christopherson <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Quoted 6 lines
--- arch/x86/include/asm/cpu.h | 1 + arch/x86/kernel/crash.c | 8 ++++---- arch/x86/kernel/reboot.c | 14 ++++++++++++-- 3 files changed, 17 insertions(+), 6 deletions(-)
Hi Paolo / Sean / Vitaly, sorry for the ping.
But do you think this fix is OK from the VMX point-of-view?
I'd like to send a V2 of this set soon, so any review here is highly appreciated!
Cheers,
Guilherme
Re: [PATCH 01/30] x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart
In reply to Guilherme G. PiccoliI find the shortlog to be very confusing, the bug has nothing to do with disabling VMX and I distinctly remember wrapping VMXOFF with exception fixup to prevent doom if VMX is already disabled :-). The issue is really that nmi_shootdown_cpus() doesn't play nice with being called twice.
Quoted 15 lines · Guilherme G. Piccoli
In the panic path we have a list of functions to be called, the panic notifiers - such callbacks perform various actions in the machine's last breath, and sometimes users want them to run before kdump. We have the parameter "
crash_kexec_post_notifiers" for that. When such parameter is used, the function "crash_smp_send_stop()" is executed to poweroff all secondary CPUs through the NMI-shootdown mechanism; part of this process involves disabling virtualization features in all CPUs (except the main one).Now, in the emergency restart procedure we have also a way of disabling VMX in all CPUs, using the same NMI-shootdown mechanism; what happens though is that in case we already NMI-disabled all CPUs, the emergency restart fails due to a second addition of the same items in the NMI list, as per the following log output:
Quoted 8 lines
sysrq: Trigger a crash
Kernel panic - not syncing: sysrq triggered crash [...] Rebooting in 2 seconds.. list_add double add: new=<addr1>, prev=<addr2>, next=<addr1>. ------------[ cut here ]------------ kernel BUG at lib/list_debug.c:29!invalid opcode: 0000 [#1] PREEMPT SMP PTI
Call stacks for the two callers would be very, very helpful.
Quoted 59 lines
In order to reproduce the problem, users just need to set the kernel parameter "
crash_kexec_post_notifiers" *without* kdump set in any system with the VMX feature present.Since there is no benefit in re-disabling VMX in all CPUs in case it was already done, this patch prevents that by guarding the restart routine against doubly issuing NMIs unnecessarily. Notice we still need to disable VMX locally in the emergency restart.
Fixes ed72736183c4 ("x86/reboot: Force all cpus to exit VMX root if VMX is supported)Fixes 0ee59413c967 ("x86/panic: replace smp_send_stop() with kdump friendly version in panic path")Cc David P. Reed <[email protected]>Cc Hidehiro Kawai <[email protected]>Cc Paolo Bonzini <[email protected]>Cc Sean Christopherson <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/x86/include/asm/cpu.h | 1 + arch/x86/kernel/crash.c | 8 ++++---- arch/x86/kernel/reboot.c | 14 ++++++++++++-- 3 files changed, 17 insertions(+), 6 deletions(-)arch/x86/include/asm/cpu.h +1
-0@@ -36,6 +36,7 @@ extern int _debug_hotplug_cpu(int cpu, int action);#endif #endif +extern bool crash_cpus_stopped;int mwait_usable (const struct cpuinfo_x86 *);unsigned int x86_family (unsigned int sig);arch/x86/kernel/crash.c +3
-3@@ -42,6 +42,8 @@#include <asm/crash.h> #include <asm/cmdline.h> +bool crash_cpus_stopped; +/* Used while preparing memory map entries for second kernel */ struct crash_memmap_data {struct boot_params *params;@@ -108,9 +110,7 @@ void kdump_nmi_shootdown_cpus(void)/* Override the weak function in kernel/panic.c */ void crash_smp_send_stop (void ) { -static int cpus_stopped; - -if (cpus_stopped) +if (crash_cpus_stopped)return ;if (smp_ops.crash_stop_other_cpus)
arch/x86/kernel/crash.c void crash_smp_send_stop(void) · line 121
Quoted 5 lines
arch/x86/kernel/crash.c +1
-1@@ -118,7 +118,7 @@ void crash_smp_send_stop(void)else smp_send_stop (); - cpus_stopped =1 ; + crash_cpus_stopped = true;
This feels like were just adding more duct tape to the mess. nmi_shootdown() is still unsafe for more than one caller, and it takes a _lot_ of staring and searching to understand that crash_smp_send_stop() is invoked iff CONFIG_KEXEC_CORE=y, i.e. that it will call smp_ops.crash_stop_other_cpus() and not just smp_send_stop().
Rather than shared a flag between two relatively unrelated functions, what if we instead disabling virtualization in crash_nmi_callback() and then turn the reboot call into a nop if an NMI shootdown has already occurred? That will also add a bit of documentation about multiple shootdowns not working.
And I believe there's also a lurking bug in native_machine_emergency_restart() that can be fixed with cleanup. SVM can also block INIT and so should be disabled during an emergency reboot.
The attached patches are compile tested only. If they seem sane, I'll post an official mini series.
Quoted 31 lines
arch/x86/kernel/crash.c +0
-0}#else arch/x86/kernel/reboot.c +12
-2@@ -589,8 +589,18 @@ static void native_machine_emergency_restart(void)int orig_reboot_type = reboot_type;unsigned short mode; -if (reboot_emergency) -emergency_vmx_disable_all (); +/* +* We can reach this point in the end of panic path, having +* NMI-disabled all secondary CPUs. This process involves +* disabling the CPU virtualization technologies, so if that +* is the case, we only miss disabling the local CPU VMX... +*/ +if (reboot_emergency) { +if (!crash_cpus_stopped) +emergency_vmx_disable_all (); +else +cpu_emergency_vmxoff (); + }tboot_shutdown (TB_SHUTDOWN_REBOOT );Signature · 1 line
2.36.0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-x86-crash-Disable-virt-in-core-NMI-crash-handler-to-.patch
Type: text/x-diff
Size: 6626 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20220509/5e99544c/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-x86-reboot-Disable-virtualization-in-an-emergency-if.patch
Type: text/x-diff
Size: 2721 bytes
Desc: not available
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20220509/5e99544c/attachment-0003.bin>
Re: [PATCH 01/30] x86/crash,reboot: Avoid re-disabling VMX in all CPUs on crash/restart
Quoted 5 lines · Sean Christopherson
I find the shortlog to be very confusing, the bug has nothing to do with disabling VMX and I distinctly remember wrapping VMXOFF with exception fixup to prevent doom if VMX is already disabled :-). The issue is really that
nmi_shootdown_cpus()doesn't play nice with being called twice.
Hey Sean, OK - I agree with you, the issue is really about the double list addition.
Quoted 15 lines
[...]
Call stacks for the two callers would be very, very helpful. [...]
This feels like were just adding more duct tape to the mess.
nmi_shootdown()is still unsafe for more than one caller, and it takes a _lot_ of staring and searching to understand thatcrash_smp_send_stop()is invoked iffCONFIG_KEXEC_CORE=y, i.e. that it will callsmp_ops.crash_stop_other_cpus()and not justsmp_send_stop().Rather than shared a flag between two relatively unrelated functions, what if we instead disabling virtualization in
crash_nmi_callback()and then turn the reboot call into a nop if an NMI shootdown has already occurred? That will also add a bit of documentation about multiple shootdowns not working.
Quoted 6 lines
And I believe there's also a lurking bug in
native_machine_emergency_restart()that can be fixed with cleanup. SVM can also block INIT and so should be disabled during an emergency reboot.The attached patches are compile tested only. If they seem sane, I'll post an official mini series.
Thanks Sean, it makes sense - my patch is more a "band-aid" whereas yours fixes it in a more generic way. Confess I found the logic of your patch complex, but as you said, it requires a *lot* of code analysis to understand these multiple shutdown patches, the problem is complicated by nature heh
I've tested your patch 0001 and it works well for all cases [0], so go ahead and submit the miniseries, feel free to add:
I've read patch 0002 and it makes sense to me as well, a good proactive bug fix =)
With that said, I'll of course drop this one from V2 of this series. Cheers,
Guilherme
[0]
A summary of my tests and the code paths that the panic shutdown take depending on some conditions:
New function that disables VMX/SVM: cpu_crash_disable_virtualization() [should be executed in every online CPU on shutdown)
The panic path triggers the following call stacks depending on kdump and post_notifiers:
(1) kexec/kdump + !crash_kexec_post_notifiers ->machine_crash_shutdown()
----.crash_shutdown() <custom handler> ------native_machine_crash_shutdown() [all custom handlers except Xen PV call the native generic function] --------crash_smp_send_stop() ----------kdump_nmi_shootdown_cpus() ------------nmi_shootdown_cpus(kdump_nmi_callback) --------------crash_nmi_callback() ----------------kdump_nmi_callback() ------------------cpu_crash_disable_virtualization()
(2) kexec/kdump + crash_kexec_post_notifiers
->crash_smp_send_stop()
----kdump_nmi_shootdown_cpus()
------nmi_shootdown_cpus(kdump_nmi_callback)
--------crash_nmi_callback()
----------kdump_nmi_callback()
------------cpu_crash_disable_virtualization()
After this path, will execute machine_crash_shutdown() but crash_smp_send_stop()
is guarded against double execution. Also, emergency restart calls emergency_vmx_disable_all() .
(3) !kexec/kdump + crash_kexec_post_notifiers
Same as (2)
(4) !kexec/kdump + !crash_kexec_post_notifiers
-> smp_send_stop()
----native_stop_other_cpus() ------apic_send_IPI_allbutself(REBOOT_VECTOR) --------sysvec_reboot
----------cpu_emergency_vmxoff() <if the IPI approach succeeded, CPU stopped here>
If not:
------register_stop_handler()
--------apic_send_IPI_allbutself(NMI_VECTOR)
----------smp_stop_nmi_callback()
------------cpu_emergency_vmxoff()
After that, emergency_vmx_disable_all() gets called in the emergency restart path as well.
PATCH 2/30 ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
In reply to Guilherme G. PiccoliCurrently the regular CPU shutdown path for ARM disables IRQs/FIQs in the secondary CPUs - smp_send_stop() calls ipi_cpu_stop(), which is responsible for that. This makes sense, since we're turning off such CPUs, putting them in an endless busy-wait loop.
Problem is that there is an alternative path for disabling CPUs, in the form of function crash_smp_send_stop(), used for kexec/panic paths. This functions relies in a SMP call that also triggers a busy-wait loop [at machine_crash_nonpanic_core()], but *without* disabling interrupts. This might lead to odd scenarios, like early interrupts in the boot of kexec'd kernel or even interrupts in other CPUs while the main one still works in the panic path and assumes all secondary CPUs are (really!) off.
This patch mimics the ipi_cpu_stop() interrupt disable mechanism in the crash CPU shutdown path, hence disabling IRQs/FIQs in all secondary CPUs in the kexec/panic path as well.
--- arch/arm/kernel/machine_kexec.c | 3 +++ 1 file changed, 3 insertions(+)
arch/arm/kernel/machine_kexec.c +3 -0
@@ -86,6 +86,9 @@ void machine_crash_nonpanic_core(void *unused)set_cpu_online (smp_processor_id (), false);atomic_dec (&waiting_for_crash_ipi); +local_fiq_disable (); +local_irq_disable (); +while (1 ) {cpu_relax ();wfe ();
Signature · 1 line
2.36.0
RE: [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
Quoted 9 lines
Currently the regular CPU shutdown path for ARM disables IRQs/FIQs in the secondary CPUs -
smp_send_stop()callsipi_cpu_stop(), which is responsible for that. This makes sense, since we're turning off such CPUs, putting them in an endless busy-wait loop.Problem is that there is an alternative path for disabling CPUs, in the form of function
crash_smp_send_stop(), used for kexec/panic paths. This functions relies in a SMP call that also triggers a
s/functions relies in/function relies on/
Quoted 33 lines
busy-wait loop [at
machine_crash_nonpanic_core()], but *without* disabling interrupts. This might lead to odd scenarios, like early interrupts in the boot of kexec'd kernel or even interrupts in other CPUs while the main one still works in the panic path and assumes all secondary CPUs are (really!) off.This patch mimics the
ipi_cpu_stop()interrupt disable mechanism in the crash CPU shutdown path, hence disabling IRQs/FIQs in all secondary CPUs in the kexec/panic path as well.Cc Marc Zyngier <[email protected]>Cc Russell King <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/arm/kernel/machine_kexec.c | 3 +++ 1 file changed, 3 insertions(+)arch/arm/kernel/machine_kexec.c +3
-1@@ -86,6 +86,9 @@ void machine_crash_nonpanic_core(void *unused)set_cpu_online (smp_processor_id (), false);atomic_dec (&waiting_for_crash_ipi); +local_fiq_disable (); +local_irq_disable (); +while (1 ) {cpu_relax ();wfe (); --2.36.0
Re: [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
In reply to Guilherme G. PiccoliQuoted 31 lines · Guilherme G. Piccoli
Currently the regular CPU shutdown path for ARM disables IRQs/FIQs in the secondary CPUs -
smp_send_stop()callsipi_cpu_stop(), which is responsible for that. This makes sense, since we're turning off such CPUs, putting them in an endless busy-wait loop.Problem is that there is an alternative path for disabling CPUs, in the form of function
crash_smp_send_stop(), used for kexec/panic paths. This functions relies in a SMP call that also triggers a busy-wait loop [atmachine_crash_nonpanic_core()], but *without* disabling interrupts. This might lead to odd scenarios, like early interrupts in the boot of kexec'd kernel or even interrupts in other CPUs while the main one still works in the panic path and assumes all secondary CPUs are (really!) off.This patch mimics the
ipi_cpu_stop()interrupt disable mechanism in the crash CPU shutdown path, hence disabling IRQs/FIQs in all secondary CPUs in the kexec/panic path as well.Cc Marc Zyngier <[email protected]>Cc Russell King <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/arm/kernel/machine_kexec.c | 3 +++ 1 file changed, 3 insertions(+)arch/arm/kernel/machine_kexec.c +0
-0
arch/arm/kernel/machine_kexec.c void machine_crash_nonpanic_core(void *unused) · line 91
Quoted 6 lines
arch/arm/kernel/machine_kexec.c +3
-0@@ -86,6 +86,9 @@ void machine_crash_nonpanic_core(void *unused)set_cpu_online (smp_processor_id (), false);atomic_dec (&waiting_for_crash_ipi); +local_fiq_disable (); +local_irq_disable (); +
My expectations would be that, since we're getting here using an IPI, interrupts are already masked. So what reenabled them the first place?
Thanks,
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
Re: [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
Thanks Marc and Michael for the review/discussion.
Quoted 9 lines · Marc Zyngier
[...]
My expectations would be that, since we're getting here using an IPI, interrupts are already masked. So what reenabled them the first place?
Thanks,
M.
Marc, I did some investigation in the code (and tried/failed in the ARM documentation as well heh), but this is still not 100% clear for me.
You're saying IPI calls disable IRQs/FIQs by default in the the target CPUs? Where does it happen? I'm a bit confused if this a processor mechanism, or it's in code.
Looking the smp_send_stop() in arch/arm/, it does IPI the CPUs, with the flag IPI_CPU_STOP, eventually calling ipi_cpu_stop(), and the latter does disable IRQ/FIQ in code - that's where I stole my code from.
But crash_smp_send_stop() is different, it seems to IPI the other CPUs with the flag IPI_CALL_FUNC, which leads to calling generic_smp_call_function_interrupt() - does it disable interrupts/FIQs as well? I couldn't find it.
Appreciate your clarifications about that, thanks again. Cheers,
Guilherme
Russell King (Oracle)
maintainerRe: [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
Quoted 13 lines · Guilherme G. Piccoli
Thanks Marc and Michael for the review/discussion.
Quoted 9 lines · Marc Zyngier
[...]
My expectations would be that, since we're getting here using an IPI, interrupts are already masked. So what reenabled them the first place?
Thanks,
M.
Quoted 6 lines
Marc, I did some investigation in the code (and tried/failed in the ARM documentation as well heh), but this is still not 100% clear for me.
You're saying IPI calls disable IRQs/FIQs by default in the the target CPUs? Where does it happen? I'm a bit confused if this a processor mechanism, or it's in code.
When we taken an IRQ, IRQs will be masked, FIQs will not. IPIs are themselves interrupts, so IRQs will be masked while the IPI is being processed. Therefore, there should be no need to re-disable the already disabled interrupts.
Quoted 4 lines
But
crash_smp_send_stop()is different, it seems to IPI the other CPUs with the flagIPI_CALL_FUNC, which leads to callinggeneric_smp_call_function_interrupt()- does it disable interrupts/FIQs as well? I couldn't find it.
It's buried in the architecture behaviour. When the CPU takes an interrupt and jumps to the interrupt vector in the vectors page, it is architecturally defined that interrupts will be disabled. If they weren't architecturally disabled at this point, then as soon as the first instruction is processed (at the interrupt vector, likely a branch) the CPU would immediately take another jump to the interrupt vector, and this process would continue indefinitely, making interrupt handling utterly useless.
So, you won't find an explicit instruction in the code path from the vectors to the IPI handler that disables interrupts - because it's written into the architecture that this is what must happen.
IRQs are a lower priority than FIQs, so FIQs remain unmasked.
Signature · 2 lines
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!
Re: [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
Quoted 27 lines · Russell King (Oracle)
[...]
Quoted 6 lines
Marc, I did some investigation in the code (and tried/failed in the ARM documentation as well heh), but this is still not 100% clear for me.
You're saying IPI calls disable IRQs/FIQs by default in the the target CPUs? Where does it happen? I'm a bit confused if this a processor mechanism, or it's in code.
When we taken an IRQ, IRQs will be masked, FIQs will not. IPIs are themselves interrupts, so IRQs will be masked while the IPI is being processed. Therefore, there should be no need to re-disable the already disabled interrupts.
Quoted 4 lines
But
crash_smp_send_stop()is different, it seems to IPI the other CPUs with the flagIPI_CALL_FUNC, which leads to callinggeneric_smp_call_function_interrupt()- does it disable interrupts/FIQs as well? I couldn't find it.It's buried in the architecture behaviour. When the CPU takes an interrupt and jumps to the interrupt vector in the vectors page, it is architecturally defined that interrupts will be disabled. If they weren't architecturally disabled at this point, then as soon as the first instruction is processed (at the interrupt vector, likely a branch) the CPU would immediately take another jump to the interrupt vector, and this process would continue indefinitely, making interrupt handling utterly useless.
Quoted 6 lines
So, you won't find an explicit instruction in the code path from the vectors to the IPI handler that disables interrupts - because it's written into the architecture that this is what must happen.
IRQs are a lower priority than FIQs, so FIQs remain unmasked.
Thanks a lot for the *great* explanation Russell, much appreciated. So, this leads to the both following questions:
a) Shall we then change the patch to only disable FIQs, since it's panic path and we don't want secondary CPUs getting interrupted, but only spinning quietly "forever"?
b) How about cleaning ipi_cpu_stop() then, by dropping the call to local_irq_disable() there, to avoid the double IRQ disabling?
Thanks,
Guilherme
Re: [PATCH 02/30] ARM: kexec: Disable IRQs/FIQs also on crash CPUs shutdown path
In reply to Russell King (Oracle)Quoted 41 lines · Russell King (Oracle)
Quoted 13 lines · Guilherme G. Piccoli
Thanks Marc and Michael for the review/discussion.
Quoted 9 lines · Marc Zyngier
[...]
My expectations would be that, since we're getting here using an IPI, interrupts are already masked. So what reenabled them the first place?
Thanks,
M.Quoted 6 lines
Marc, I did some investigation in the code (and tried/failed in the ARM documentation as well heh), but this is still not 100% clear for me.
You're saying IPI calls disable IRQs/FIQs by default in the the target CPUs? Where does it happen? I'm a bit confused if this a processor mechanism, or it's in code.
When we taken an IRQ, IRQs will be masked, FIQs will not. IPIs are themselves interrupts, so IRQs will be masked while the IPI is being processed. Therefore, there should be no need to re-disable the already disabled interrupts.
Quoted 4 lines
But
crash_smp_send_stop()is different, it seems to IPI the other CPUs with the flagIPI_CALL_FUNC, which leads to callinggeneric_smp_call_function_interrupt()- does it disable interrupts/FIQs as well? I couldn't find it.It's buried in the architecture behaviour. When the CPU takes an interrupt and jumps to the interrupt vector in the vectors page, it is architecturally defined that interrupts will be disabled. If they weren't architecturally disabled at this point, then as soon as the first instruction is processed (at the interrupt vector, likely a branch) the CPU would immediately take another jump to the interrupt vector, and this process would continue indefinitely, making interrupt handling utterly useless.
Quoted 5 lines
So, you won't find an explicit instruction in the code path from the vectors to the IPI handler that disables interrupts - because it's written into the architecture that this is what must happen.
IRQs are a lower priority than FIQs, so FIQs remain unmasked.
Ah, you're of course right. That's one of the huge differences between AArch32 and AArch64, where the former has per target mode masking rules, and the later masks everything on entry...
M.
Signature · 1 line
Without deviation from the norm, progress is not possible.
PATCH 3/30 notifier: Add panic notifiers info and purge trailing whitespaces
In reply to Guilherme G. PiccoliAlthough many notifiers are mentioned in the comments, the panic notifiers infrastructure is not. Also, the file contains some trailing whitespaces. This commit fix both issues.
--- include/linux/notifier.h | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-)
include/linux/notifier.h +5 -3
@@ -201,12 +201,12 @@ static inline int notifier_to_errno(int ret)/* * Declared notifiers so far. I can imagine quite a few more chains -* over time (eg laptop power reset chains, reboot chain (to clean +* over time (eg laptop power reset chains, reboot chain (to clean * device units up), device [un]mount chain, module load/unload chain, -* low memory chain, screenblank chain (for plug in modular screenblankers) +* low memory chain, screenblank chain (for plug in modular screenblankers) * VC switch chains (for loadable kernel svgalib VC switch helpers) etc... */ - +/* CPU notfiers are defined in include/linux/cpu.h. */ /* netdevice notifiers are defined in include/linux/netdevice.h */
@@ -217,6 +217,8 @@ static inline int notifier_to_errno(int ret)/* Virtual Terminal events are defined in include/linux/vt.h. */ +/* Panic notifiers are defined in include/linux/panic_notifier.h. */ +#define NETLINK_URELEASE 0x0001/* Unicast netlink socket released */ /* Console keyboard events.
Signature · 1 line
2.36.0
PATCH 4/30 firmware: google: Convert regular spinlock into trylock on panic path
In reply to Guilherme G. PiccoliCurrently the gsmi driver registers a panic notifier as well as reboot and die notifiers. The callbacks registered are called in atomic and very limited context - for instance, panic disables preemption, local IRQs and all other CPUs that aren't running the current panic function.
With that said, taking a spinlock in this scenario is a dangerous invitation for a deadlock scenario. So, we fix that in this commit by changing the regular spinlock with a trylock, which is a safer approach.
--- drivers/firmware/google/gsmi.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)
drivers/firmware/google/gsmi.c +5 -1
@@ -629,7 +629,10 @@ static int gsmi_shutdown_reason(int reason)if (saved_reason & (1 << reason))return 0 ; -spin_lock_irqsave (&gsmi_dev.lock, flags); +if (!spin_trylock_irqsave (&gsmi_dev.lock, flags)) { + rc = -EBUSY ; + goto out; + } saved_reason |= (1 << reason);
@@ -646,6 +649,7 @@ static int gsmi_shutdown_reason(int reason)spin_unlock_irqrestore (&gsmi_dev.lock, flags); +out:if (rc <0 )printk (KERN_ERR "gsmi: Log Shutdown Reason failed\n" );else
Signature · 1 line
2.36.0
Re: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Quoted 27 lines · Guilherme G. Piccoli
Currently the gsmi driver registers a panic notifier as well as reboot and die notifiers. The callbacks registered are called in atomic and very limited context - for instance, panic disables preemption, local IRQs and all other CPUs that aren't running the current panic function.
With that said, taking a spinlock in this scenario is a dangerous invitation for a deadlock scenario. So, we fix that in this commit by changing the regular spinlock with a trylock, which is a safer approach.
Fixes 74c5b31c6618 ("driver: Google EFI SMI")Cc Ard Biesheuvel <[email protected]>Cc David Gow <[email protected]>Cc Evan Green <[email protected]>Cc Julius Werner <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- drivers/firmware/google/gsmi.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-)drivers/firmware/google/gsmi.c +0
-0
drivers/firmware/google/gsmi.c static int gsmi_shutdown_reason(int reason) · line 635
Quoted 8 lines
drivers/firmware/google/gsmi.c +4
-1@@ -629,7 +629,10 @@ static int gsmi_shutdown_reason(int reason)if (saved_reason & (1 << reason))return 0 ; -spin_lock_irqsave (&gsmi_dev.lock, flags); +if (!spin_trylock_irqsave (&gsmi_dev.lock, flags)) { + rc = -EBUSY ; + goto out; + }
gsmi_shutdown_reason() is a common function called in other scenarios as well, like reboot and thermal trip, where it may still make sense to wait to acquire a spinlock. Maybe we should add a parameter to gsmi_shutdown_reason() so that you can get your change on panic, but we don't convert other callbacks into try-fail scenarios causing us to miss logs.
Though thinking more about it, is this really a Good Change (TM)? The spinlock itself already disables interrupts, meaning the only case where this change makes a difference is if the panic happens from within the function that grabbed the spinlock (in which case the callback is also likely to panic), or in an NMI that panics within that window. The downside of this change is that if one core was politely working through an event with the lock held, and another core panics, we now might lose the panic log, even though it probably would have gone through fine assuming the other core has a chance to continue.
-Evan
Re: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Quoted 8 lines · Evan Green
[...]
gsmi_shutdown_reason()is a common function called in other scenarios as well, like reboot and thermal trip, where it may still make sense to wait to acquire a spinlock. Maybe we should add a parameter togsmi_shutdown_reason()so that you can get your change on panic, but we don't convert other callbacks into try-fail scenarios causing us to miss logs.
Hi Evan, thanks for your feedback, much appreciated!
What I've done in other cases like this was to have a helper checking the spinlock in the panic notifier - if we can acquire that, go ahead but if not, bail out. For a proper example of an implementation, check patch 13 of the series:
https://lore.kernel.org/lkml/20220427224924.592546-14-gpiccoli at igalia.com/ .
Do you agree with that, or prefer really a parameter in gsmi_shutdown_reason() ? I'll follow your choice =)
Quoted 10 lines
Though thinking more about it, is this really a Good Change (TM)? The spinlock itself already disables interrupts, meaning the only case where this change makes a difference is if the panic happens from within the function that grabbed the spinlock (in which case the callback is also likely to panic), or in an NMI that panics within that window. The downside of this change is that if one core was politely working through an event with the lock held, and another core panics, we now might lose the panic log, even though it probably would have gone through fine assuming the other core has a chance to continue.
My feeling is that this is a good change, indeed - a lot of places are getting changed like this, in this series.
Reasoning: the problem with your example is that, by default, secondary CPUs are disabled in the panic path, through an IPI mechanism. IPIs take precedence and interrupt the work in these CPUs, effectively interrupting the "polite work" with the lock held heh
Then, such CPU is put to sleep and we finally reach the panic notifier hereby discussed, in the main CPU. If the other CPU was shut-off *with the lock held*, it's never finishing such work, so the lock is never to be released. Conclusion: the spinlock can't be acquired, hence we broke the machine (which is already broken, given it's panic) in the path of this notifier.
This should be really rare, but..possible. So I think we should protect against this scenario.
We can grab others' feedback if you prefer, and of course you have the rights to refuse this change in the gsmi code, but from my point-of-view, I don't see any advantage in just assume the risk, specially since the change is very very simple.
Cheers,
Guilherme
Re: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Hi Guilherme,
Quoted 18 lines · Guilherme G. Piccoli
Quoted 8 lines · Evan Green
[...]
gsmi_shutdown_reason()is a common function called in other scenarios as well, like reboot and thermal trip, where it may still make sense to wait to acquire a spinlock. Maybe we should add a parameter togsmi_shutdown_reason()so that you can get your change on panic, but we don't convert other callbacks into try-fail scenarios causing us to miss logs.Hi Evan, thanks for your feedback, much appreciated!
What I've done in other cases like this was to have a helper checking the spinlock in the panic notifier - if we can acquire that, go ahead but if not, bail out. For a proper example of an implementation, check patch 13 of the series:
https://lore.kernel.org/lkml/20220427224924.592546-14-gpiccoli at igalia.com/ .
Quoted 2 lines
Do you agree with that, or prefer really a parameter in
gsmi_shutdown_reason()? I'll follow your choice =)
I'm fine with either, thanks for the link. Mostly I want to make sure other paths to gsmi_shutdown_reason() aren't also converted to a try.
Quoted 13 lines
Quoted 12 lines
Though thinking more about it, is this really a Good Change (TM)? The spinlock itself already disables interrupts, meaning the only case where this change makes a difference is if the panic happens from within the function that grabbed the spinlock (in which case the callback is also likely to panic), or in an NMI that panics within that window. The downside of this change is that if one core was politely working through an event with the lock held, and another core panics, we now might lose the panic log, even though it probably would have gone through fine assuming the other core has a chance to continue.
Quoted 7 lines
My feeling is that this is a good change, indeed - a lot of places are getting changed like this, in this series.
Reasoning: the problem with your example is that, by default, secondary CPUs are disabled in the panic path, through an IPI mechanism. IPIs take precedence and interrupt the work in these CPUs, effectively interrupting the "polite work" with the lock held heh
The IPI can only interrupt a CPU with irqs disabled if the IPI is an NMI. I haven't looked before to see if we use NMI IPIs to corral the other CPUs on panic. On x86, I grepped my way down to native_stop_other_cpus(), which looks like it does a normal IPI, waits 1 second, then does an NMI IPI. So, if a secondary CPU has the lock held, on x86 it has roughly 1s to finish what it's doing and re-enable interrupts before smp_send_stop() brings the NMI hammer down. I think this should be more than enough time for the secondary CPU to get out and release the lock.
So then it makes sense to me that you're fixing cases where we panicked with the lock held, or hung with the lock held. Given the 1 second grace period x86 gives us, I'm on board, as that helps mitigate the risk that we bailed out early with the try and should have spun a bit longer instead. Thanks.
-Evan
Quoted 19 lines
Then, such CPU is put to sleep and we finally reach the panic notifier hereby discussed, in the main CPU. If the other CPU was shut-off *with the lock held*, it's never finishing such work, so the lock is never to be released. Conclusion: the spinlock can't be acquired, hence we broke the machine (which is already broken, given it's panic) in the path of this notifier.
This should be really rare, but..possible. So I think we should protect against this scenario.We can grab others' feedback if you prefer, and of course you have the rights to refuse this change in the gsmi code, but from my point-of-view, I don't see any advantage in just assume the risk, specially since the change is very very simple.
Cheers,
Guilherme
Re: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Quoted 7 lines · Evan Green
Hi Guilherme,
[...]Quoted 2 lines
Do you agree with that, or prefer really a parameter in
gsmi_shutdown_reason()? I'll follow your choice =)I'm fine with either, thanks for the link. Mostly I want to make sure other paths to
gsmi_shutdown_reason()aren't also converted to a try.
Hi Evan, thanks for the prompt response! So, I'll proceed like I did in s390, for consistency.
Quoted 16 lines
[...]
Quoted 4 lines
Reasoning: the problem with your example is that, by default, secondary CPUs are disabled in the panic path, through an IPI mechanism. IPIs take precedence and interrupt the work in these CPUs, effectively interrupting the "polite work" with the lock held heh
The IPI can only interrupt a CPU with irqs disabled if the IPI is an NMI. I haven't looked before to see if we use NMI IPIs to corral the other CPUs on panic. On x86, I grepped my way down to
native_stop_other_cpus(), which looks like it does a normal IPI, waits 1 second, then does an NMI IPI. So, if a secondary CPU has the lock held, on x86 it has roughly 1s to finish what it's doing and re-enable interrupts beforesmp_send_stop()brings the NMI hammer down. I think this should be more than enough time for the secondary CPU to get out and release the lock.
Quoted 7 lines
So then it makes sense to me that you're fixing cases where we panicked with the lock held, or hung with the lock held. Given the 1 second grace period x86 gives us, I'm on board, as that helps mitigate the risk that we bailed out early with the try and should have spun a bit longer instead. Thanks.
-Evan
Well, in the old path without "crash_kexec_post_notifiers", we indeed end-up relying on native_stop_other_cpus() for x86 as you said, and the "1s rule" makes sense. But after this series (or even before, if the kernel parameter "crash_kexec_post_notifiers" was used) the function used to stop CPUs in the panic path is crash_smp_send_stop(), and the call chain is like:
Main CPU:crash_smp_send_stop()
--kdump_nmi_shootdown_cpus()
----nmi_shootdown_cpus()
Then, in each CPU (except the main one, running panic() path), we execute kdump_nmi_callback() in NMI context.
So, we seem to indeed interrupt any context (even with IRQs disabled), increasing the likelihood of the potential lockups due to stopped CPUs holding the locks heheh
Thanks again for the good discussion, let me know if anything I'm saying doesn't make sense - this crash path is a bit convoluted, specially in x86, I might have understood something wrongly =)
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
In reply to Guilherme G. PiccoliQuoted 17 lines · Guilherme G. Piccoli
Quoted 8 lines · Evan Green
[...]
gsmi_shutdown_reason()is a common function called in other scenarios as well, like reboot and thermal trip, where it may still make sense to wait to acquire a spinlock. Maybe we should add a parameter togsmi_shutdown_reason()so that you can get your change on panic, but we don't convert other callbacks into try-fail scenarios causing us to miss logs.Hi Evan, thanks for your feedback, much appreciated!
What I've done in other cases like this was to have a helper checking the spinlock in the panic notifier - if we can acquire that, go ahead but if not, bail out. For a proper example of an implementation, check patch 13 of the series:
https://lore.kernel.org/lkml/20220427224924.592546-14-gpiccoli at igalia.com/ .
Quoted 2 lines
Do you agree with that, or prefer really a parameter in
gsmi_shutdown_reason()? I'll follow your choice =)
I see two more alternative solutions:
1st variant is a trick already used in console write() callbacks. They do trylock() when oops_in_progress is set. They remember the result to prevent double unlock when printing Oops messages and the system will try to continue working. For example:
pl011_console_write(struct console *co, const char *s, unsigned int count) {
[...]
int locked = 1; [...] if (uap->port.sysrq) locked = 0; else if (oops_in_progress) locked = spin_trylock(&uap->port.lock); else spin_lock(&uap->port.lock);
[...]
if (locked) spin_unlock(&uap->port.lock); }
2nd variant is to check panic_cpu variable. It is used in printk.c. We might move the function to panic.h:
static bool panic_in_progress(void)
{
return unlikely(atomic_read(&panic_cpu) != PANIC_CPU_INVALID);
}and then do:
if (panic_in_progress()) {
...Quoted 6 lines
Quoted 6 lines
Though thinking more about it, is this really a Good Change (TM)? The spinlock itself already disables interrupts, meaning the only case where this change makes a difference is if the panic happens from within the function that grabbed the spinlock (in which case the callback is also likely to panic), or in an NMI that panics within that window.
As already mentioned in the other reply, panic() sometimes stops the other CPUs using NMI, for example, see kdump_nmi_shootdown_cpus().
Another situation is when the CPU using the lock ends in some infinite loop because something went wrong. The system is in an unpredictable state during panic().
I am not sure if this is possible with the code under gsmi_dev.lock but such things really happen during panic() in other subsystems. Using trylock in the panic() code path is a good practice.
Best Regards,
Petr
Re: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Quoted 31 lines · Petr Mladek
[...]
I see two more alternative solutions:1st variant is a trick already used in console
write()callbacks. They dotrylock()whenoops_in_progressis set. They remember the result to prevent double unlock when printing Oops messages and the system will try to continue working. For example:
pl011_console_write(struct console *co, const char *s, unsigned int count){
[...]int locked = 1; [...] if (uap->port.sysrq) locked = 0; else if (oops_in_progress) locked = spin_trylock(&uap->port.lock); else spin_lock(&uap->port.lock);[...]
if (locked) spin_unlock(&uap->port.lock); }2nd variant is to check
panic_cpuvariable. It is used inprintk.c. We might move the function topanic.h:static bool
panic_in_progress(void)
Quoted 8 lines
{ return unlikely(atomic_read(&panic_cpu) != PANIC_CPU_INVALID); }and then do:
if (panic_in_progress()) { ...
Thanks for the review Petr! I feel alternative two is way better, it checks for panic - the oops_in_progress isn't really enough, since we can call panic() directly, not necessarily through an oops path, correct?
For me, we could stick with the lock check, but I'll defer to Evan - I didn't work the V2 patch yet, what do you prefer Evan?
Quoted 8 lines
[...]
As already mentioned in the other reply,panic()sometimes stops the other CPUs using NMI, for example, seekdump_nmi_shootdown_cpus().Another situation is when the CPU using the lock ends in some infinite loop because something went wrong. The system is in an unpredictable state during
panic().
Quoted 6 lines
I am not sure if this is possible with the code under gsmi_dev.lock but such things really happen during
panic()in other subsystems. Using trylock in thepanic()code path is a good practice.Best Regards,
Petr
Makes total sense, thanks for confirming!
Cheers,
Guilherme
Steven Rostedt
maintainerRe: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
In reply to Petr MladekQuoted 10 lines · On Tue, 10 May 2022 13:38:39 +0200 Petr Mladek <[email protected]> wrote:
As already mentioned in the other reply,
panic()sometimes stops the other CPUs using NMI, for example, seekdump_nmi_shootdown_cpus().Another situation is when the CPU using the lock ends in some infinite loop because something went wrong. The system is in an unpredictable state during
panic().I am not sure if this is possible with the code under gsmi_dev.lock but such things really happen during
panic()in other subsystems. Using trylock in thepanic()code path is a good practice.
I believe that Peter Zijlstra had a special spin lock for NMIs or early printk, where it would not block if the lock was held on the same CPU. That is, if an NMI happened and paniced while this lock was held on the same CPU, it would not deadlock. But it would block if the lock was held on another CPU.
-- Steve
John Ogness
maintainerRe: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Quoted 11 lines · Steven Rostedt
Quoted 10 lines
As already mentioned in the other reply,
panic()sometimes stops the other CPUs using NMI, for example, seekdump_nmi_shootdown_cpus().Another situation is when the CPU using the lock ends in some infinite loop because something went wrong. The system is in an unpredictable state during
panic().I am not sure if this is possible with the code under gsmi_dev.lock but such things really happen during
panic()in other subsystems. Using trylock in thepanic()code path is a good practice.
Quoted 5 lines
I believe that Peter Zijlstra had a special spin lock for NMIs or early printk, where it would not block if the lock was held on the same CPU. That is, if an NMI happened and paniced while this lock was held on the same CPU, it would not deadlock. But it would block if the lock was held on another CPU.
Yes. And starting with 5.19 it will be carrying the name that _you_ came up with (cpu_sync):
printk_cpu_sync_get_irqsave()printk_cpu_sync_put_irqrestore()
John
Petr Mladek
maintainerRe: [PATCH 04/30] firmware: google: Convert regular spinlock into trylock on panic path
Quoted 18 lines · John Ogness
Quoted 11 lines · Steven Rostedt
Quoted 10 lines
As already mentioned in the other reply,
panic()sometimes stops the other CPUs using NMI, for example, seekdump_nmi_shootdown_cpus().Another situation is when the CPU using the lock ends in some infinite loop because something went wrong. The system is in an unpredictable state during
panic().I am not sure if this is possible with the code under gsmi_dev.lock but such things really happen during
panic()in other subsystems. Using trylock in thepanic()code path is a good practice.Quoted 5 lines
I believe that Peter Zijlstra had a special spin lock for NMIs or early printk, where it would not block if the lock was held on the same CPU. That is, if an NMI happened and paniced while this lock was held on the same CPU, it would not deadlock. But it would block if the lock was held on another CPU.
Quoted 5 lines
Yes. And starting with 5.19 it will be carrying the name that _you_ came up with (
cpu_sync):
printk_cpu_sync_get_irqsave()printk_cpu_sync_put_irqrestore()
There is a risk that this lock might become a big kernel lock.
This special lock would need to be used even during normal system operation. It does not make sense to suddenly start using another lock during panic.
So I think that we should think twice before using it. I would prefer using trylock of the original lock when possible during panic.
It is possible that I miss something.
Best Regards,
Petr
PATCH 5/30 misc/pvpanic: Convert regular spinlock into trylock on panic path
In reply to Guilherme G. PiccoliThe pvpanic driver relies on panic notifiers to execute a callback on panic event. Such function is executed in atomic context - the panic function disables local IRQs, preemption and all other CPUs that aren't running the panic code.
With that said, it's dangerous to use regular spinlocks in such path, as introduced by commit b3c0f8774668 ("misc/pvpanic: probe multiple instances"). This patch fixes that by replacing regular spinlocks with the trylock safer approach.
It also fixes an old comment (about a long gone framebuffer code) and the notifier priority - we should execute hypervisor notifiers early, deferring this way the panic action to the hypervisor, as expected by the users that are setting up pvpanic.
--- drivers/misc/pvpanic/pvpanic.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/misc/pvpanic/pvpanic.c +8 -2
@@ -34,7 +34,9 @@ pvpanic_send_event(unsigned int event) {struct pvpanic_instance *pi_cur; -spin_lock (&pvpanic_lock); +if (!spin_trylock (&pvpanic_lock)) +return ; +list_for_each_entry (pi_cur, &pvpanic_list, list) {if (event & pi_cur->capability & pi_cur->events)iowrite8 (event, pi_cur->base);
@@ -55,9 +57,13 @@ pvpanic_panic_notify(struct notifier_block *nb, unsigned long code, void *unusedreturn NOTIFY_DONE ; } +/* +* Call our notifier very early on panic, deferring the +* action taken to the hypervisor. +*/ static struct notifier_block pvpanic_panic_nb = { .notifier_call = pvpanic_panic_notify, - .priority =1 ,/* let this called before broken drm_fb_helper() */ + .priority =INT_MAX , };static void pvpanic_remove (void *param)
Signature · 1 line
2.36.0
Petr Mladek
maintainerRe: [PATCH 05/30] misc/pvpanic: Convert regular spinlock into trylock on panic path
Quoted 9 lines · Guilherme G. Piccoli
The pvpanic driver relies on panic notifiers to execute a callback on panic event. Such function is executed in atomic context - the panic function disables local IRQs, preemption and all other CPUs that aren't running the panic code.
With that said, it's dangerous to use regular spinlocks in such path, as introduced by commit
b3c0f8774668("misc/pvpanic: probe multiple instances"). This patch fixes that by replacing regular spinlocks with the trylock safer approach.
It seems that the lock is used just to manipulating a list. A super safe solution would be to use the rcu API: rcu_add_rcu() and list_del_rcu() under rcu_read_lock(). The spin lock will not be needed and the list will always be valid.
The advantage would be that it will always call members that were successfully added earlier. That said, I am not familiar with pvpanic and am not sure if it is worth it.
Quoted 4 lines
It also fixes an old comment (about a long gone framebuffer code) and the notifier priority - we should execute hypervisor notifiers early, deferring this way the panic action to the hypervisor, as expected by the users that are setting up pvpanic.
This should be done in a separate patch. It changes the behavior. Also there might be a discussion whether it really should be the maximal priority.
Best Regards,
Petr
Re: [PATCH 05/30] misc/pvpanic: Convert regular spinlock into trylock on panic path
Quoted 20 lines · Petr Mladek
[...]
Quoted 4 lines
With that said, it's dangerous to use regular spinlocks in such path, as introduced by commit
b3c0f8774668("misc/pvpanic: probe multiple instances"). This patch fixes that by replacing regular spinlocks with the trylock safer approach.It seems that the lock is used just to manipulating a list. A super safe solution would be to use the rcu API:
rcu_add_rcu()andlist_del_rcu()underrcu_read_lock(). The spin lock will not be needed and the list will always be valid.The advantage would be that it will always call members that were successfully added earlier. That said, I am not familiar with pvpanic and am not sure if it is worth it.
Quoted 4 lines
It also fixes an old comment (about a long gone framebuffer code) and the notifier priority - we should execute hypervisor notifiers early, deferring this way the panic action to the hypervisor, as expected by the users that are setting up pvpanic.
Quoted 6 lines
This should be done in a separate patch. It changes the behavior. Also there might be a discussion whether it really should be the maximal priority.
Best Regards,
Petr
Thanks for the review Petr. Patch was already merged - my goal was to be concise, i.e., a patch per driver / module, so the patch kinda fixes whatever I think is wrong with the driver with regards panic handling.
Do you think it worth to remove this patch from Greg's branch just to split it in 2? Personally I think it's not worth, but opinions are welcome.
About the RCU part, this one really could be a new patch, a good improvement patch - it makes sense to me, we can think about that after the fixes I guess.
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 05/30] misc/pvpanic: Convert regular spinlock into trylock on panic path
Quoted 28 lines · Guilherme G. Piccoli
Quoted 20 lines · Petr Mladek
[...]
Quoted 4 lines
With that said, it's dangerous to use regular spinlocks in such path, as introduced by commit
b3c0f8774668("misc/pvpanic: probe multiple instances"). This patch fixes that by replacing regular spinlocks with the trylock safer approach.It seems that the lock is used just to manipulating a list. A super safe solution would be to use the rcu API:
rcu_add_rcu()andlist_del_rcu()underrcu_read_lock(). The spin lock will not be needed and the list will always be valid.The advantage would be that it will always call members that were successfully added earlier. That said, I am not familiar with pvpanic and am not sure if it is worth it.
Quoted 4 lines
It also fixes an old comment (about a long gone framebuffer code) and the notifier priority - we should execute hypervisor notifiers early, deferring this way the panic action to the hypervisor, as expected by the users that are setting up pvpanic.
Quoted 6 lines
This should be done in a separate patch. It changes the behavior. Also there might be a discussion whether it really should be the maximal priority.
Best Regards,
Petr
Quoted 6 lines
Thanks for the review Petr. Patch was already merged - my goal was to be concise, i.e., a patch per driver / module, so the patch kinda fixes whatever I think is wrong with the driver with regards panic handling.
Do you think it worth to remove this patch from Greg's branch just to split it in 2? Personally I think it's not worth, but opinions are welcome.
No problem. It is not worth the effort.
Quoted 3 lines
About the RCU part, this one really could be a new patch, a good improvement patch - it makes sense to me, we can think about that after the fixes I guess.
Yup.
Best Regards,
Petr
Re: [PATCH 05/30] misc/pvpanic: Convert regular spinlock into trylock on panic path
Quoted 10 lines · Petr Mladek
[...]
Quoted 6 lines
Thanks for the review Petr. Patch was already merged - my goal was to be concise, i.e., a patch per driver / module, so the patch kinda fixes whatever I think is wrong with the driver with regards panic handling.
Do you think it worth to remove this patch from Greg's branch just to split it in 2? Personally I think it's not worth, but opinions are welcome.
No problem. It is not worth the effort.
OK, perfect!
Cheers,
Guilherme
PATCH 6/30 soc: bcm: brcmstb: Document panic notifier action and remove useless header
In reply to Guilherme G. PiccoliThe panic notifier of this driver is very simple code-wise, just a memory write to a special position with some numeric code. But this is not clear from the semantic point-of-view, and there is no public documentation about that either.
After discussing this in the mailing-lists [0] and having Florian explained it very well, this patch just document that in the code for the future generations asking the same questions. Also, it removes a useless header.
[0] https://lore.kernel.org/lkml/781cafb0-8d06-8b56-907a-5175c2da196a at gmail.com
--- drivers/soc/bcm/brcmstb/pm/pm-arm.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
drivers/soc/bcm/brcmstb/pm/pm-arm.c +14 -2
@@ -25,7 +25,6 @@#include <linux/kernel.h> #include <linux/memblock.h> #include <linux/module.h> -#include <linux/notifier.h> #include <linux/of.h> #include <linux/of_address.h> #include <linux/panic_notifier.h>
@@ -664,7 +663,20 @@ static void __iomem *brcmstb_ioremap_match(const struct of_device_id *matches,return of_io_request_and_map (dn, index, dn->full_name); } - +/* +* The AON is a small domain in the SoC that can retain its state across +* various system wide sleep states and specific reset conditions; the +* AON DATA RAM is a small RAM of a few words (< 1KB) which can store +* persistent information across such events. +* +* The purpose of the below panic notifier is to help with notifying +* the bootloader that a panic occurred and so that it should try its +* best to preserve the DRAM contents holding that buffer for recovery +* by the kernel as opposed to wiping out DRAM clean again. +* +* Reference: comment from Florian Fainelli, at +* https://lore.kernel.org/lkml/[email protected] +*/ static int brcmstb_pm_panic_notify (struct notifier_block *nb,unsigned long action,void *data) {
Signature · 1 line
2.36.0
Re: [PATCH 06/30] soc: bcm: brcmstb: Document panic notifier action and remove useless header
Quoted 11 lines · Guilherme G. Piccoli
The panic notifier of this driver is very simple code-wise, just a memory write to a special position with some numeric code. But this is not clear from the semantic point-of-view, and there is no public documentation about that either.
After discussing this in the mailing-lists [0] and having Florian explained it very well, this patch just document that in the code for the future generations asking the same questions. Also, it removes a useless header.
[0] https://lore.kernel.org/lkml/781cafb0-8d06-8b56-907a-5175c2da196a at gmail.com
Quoted 8 lines
Fixes 0b741b8234c8 ("soc: bcm: brcmstb: Add support for S2/S3/S5 suspend states (ARM)")Cc Brian Norris <[email protected]>Cc Doug Berger <[email protected]>Cc Florian Fainelli <[email protected]>Cc Justin Chen <[email protected]>Cc Lee Jones <[email protected]>Cc Markus Mayer <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Likewise, I am not sure if the Fixes tag is necessary here.
Signature · 1 line
Florian
Re: [PATCH 06/30] soc: bcm: brcmstb: Document panic notifier action and remove useless header
Quoted 4 lines · Florian Fainelli
[...]Acked-by Florian Fainelli <[email protected]>Likewise, I am not sure if the Fixes tag is necessary here.
Perfect Florian, thanks! I'll add your Acked-by tag and remove the fixes for V2 =)
Cheers,
Guilherme
PATCH 7/30 mips: ip22: Reword PANICED to PANICKED and remove useless header
In reply to Guilherme G. PiccoliMany other place in the kernel prefer the latter, so let's keep it consistent in MIPS code as well. Also, removes a useless header.
--- arch/mips/sgi-ip22/ip22-reset.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
arch/mips/sgi-ip22/ip22-reset.c +5 -6
@@ -11,7 +11,6 @@#include <linux/interrupt.h> #include <linux/kernel.h> #include <linux/sched/signal.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/pm.h> #include <linux/timer.h>
@@ -41,7 +40,7 @@static struct timer_list power_timer, blink_timer, debounce_timer;static unsigned long blink_timer_timeout; -#define MACHINE_PANICED 1 +#define MACHINE_PANICKED 1#define MACHINE_SHUTTING_DOWN 2static int machine_state;
@@ -112,7 +111,7 @@ static void debounce(struct timer_list *unused)return ; } -if (machine_state &MACHINE_PANICED ) +if (machine_state &MACHINE_PANICKED ) sgimc->cpuctrl0 |=SGIMC_CCTRL0_SYSINIT ;enable_irq (SGI_PANEL_IRQ );
@@ -120,7 +119,7 @@ static void debounce(struct timer_list *unused)static inline void power_button (void ) { -if (machine_state &MACHINE_PANICED ) +if (machine_state &MACHINE_PANICKED )return ; if ((machine_state & MACHINE_SHUTTING_DOWN) ||
@@ -167,9 +166,9 @@ static irqreturn_t panel_int(int irq, void *dev_id)static int panic_event (struct notifier_block *this,unsigned long event,void *ptr) { -if (machine_state &MACHINE_PANICED ) +if (machine_state &MACHINE_PANICKED )return NOTIFY_DONE ; - machine_state |=MACHINE_PANICED ; + machine_state |=MACHINE_PANICKED ; blink_timer_timeout =PANIC_FREQ ;blink_timeout (&blink_timer);
Signature · 1 line
2.36.0
Re: [PATCH 07/30] mips: ip22: Reword PANICED to PANICKED and remove useless header
Quoted 8 lines · Guilherme G. Piccoli
Many other place in the kernel prefer the latter, so let's keep it consistent in MIPS code as well. Also, removes a useless header.
Cc Thomas Bogendoerfer <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/mips/sgi-ip22/ip22-reset.c | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-)
applied to mips-next.
Thomas.
Signature · 2 lines
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a good idea. [ RFC1925, 2.3 ]
Re: [PATCH 07/30] mips: ip22: Reword PANICED to PANICKED and remove useless header
Quoted 6 lines · Thomas Bogendoerfer
[...]
applied to mips-next.
Thomas.
Thanks a bunch Thomas =)
PATCH 8/30 powerpc/setup: Refactor/untangle panic notifiers
In reply to Guilherme G. PiccoliThe panic notifiers infrastructure is a bit limited in the scope of the callbacks - basically every kind of functionality is dropped in a list that runs in the same point during the kernel panic path. This is not really on par with the complexities and particularities of architecture / hypervisors' needs, and a refactor is ongoing.
As part of this refactor, it was observed that powerpc has 2 notifiers, with mixed goals: one is just a KASLR offset dumper, whereas the other aims to hard-disable IRQs (necessary on panic path), warn firmware of the panic event (fadump) and run low-level platform-specific machinery that might stop kernel execution and never come back.
Clearly, the 2nd notifier has opposed goals: disable IRQs / fadump should run earlier while low-level platform actions should run late since it might not even return. Hence, this patch decouples the notifiers splitting them in three:
- First one is responsible for hard-disable IRQs and fadump, should run early;
- The kernel KASLR offset dumper is really an informative notifier, harmless and may run at any moment in the panic path;
- The last notifier should run last, since it aims to perform low-level actions for specific platforms, and might never return. It is also only registered for 2 platforms, pseries and ps3.
The patch better documents the notifiers and clears the code too, also removing a useless header.
Currently no functionality change should be observed, but after the planned panic refactor we should expect more panic reliability with this patch.
---
We'd like to thanks specially the MiniCloud infrastructure [0] maintainers, that allow us to test PowerPC code in a very complete, functional and FREE environment (there's no need even for adding a credit card, like many "free" clouds require ?? ).
[0] https://openpower.ic.unicamp.br/minicloud
arch/powerpc/kernel/setup-common.c | 74 ++++++++++++++++++++++-------- 1 file changed, 54 insertions(+), 20 deletions(-)
arch/powerpc/kernel/setup-common.c +54 -20
@@ -23,7 +23,6 @@#include <linux/console.h> #include <linux/screen_info.h> #include <linux/root_dev.h> -#include <linux/notifier.h> #include <linux/cpu.h> #include <linux/unistd.h> #include <linux/serial.h>
@@ -680,8 +679,25 @@ int check_legacy_ioport(unsigned long base_port) }EXPORT_SYMBOL (check_legacy_ioport); -static int ppc_panic_event (struct notifier_block *this, -unsigned long event,void *ptr) +/* +* Panic notifiers setup +* +* We have 3 notifiers for powerpc, each one from a different "nature": +* +* - ppc_panic_fadump_handler() is a hypervisor notifier, which hard-disables +* IRQs and deal with the Firmware-Assisted dump, when it is configured; +* should run early in the panic path. +* +* - dump_kernel_offset() is an informative notifier, just showing the KASLR +* offset if we have RANDOMIZE_BASE set. +* +* - ppc_panic_platform_handler() is a low-level handler that's registered +* only if the platform wishes to perform final actions in the panic path, +* hence it should run late and might not even return. Currently, only +* pseries and ps3 platforms register callbacks. +*/ +static int ppc_panic_fadump_handler (struct notifier_block *this, +unsigned long event,void *ptr) {/* * panic does a local_irq_disable, but we really
@@ -691,45 +707,63 @@ static int ppc_panic_event(struct notifier_block *this,/* * If firmware-assisted dump has been registered then trigger -* firmware-assisted dump and let firmware handle everything else. +* its callback and let the firmware handles everything else. */ crash_fadump (NULL , ptr); -if (ppc_md.panic) - ppc_md.panic (ptr);/* May not return */ +return NOTIFY_DONE ; } -static struct notifier_block ppc_panic_block = { - .notifier_call = ppc_panic_event, - .priority =INT_MIN /* may not return; must be done last */ -}; - -/* -* Dump out kernel offset information on panic. -*/ static int dump_kernel_offset (struct notifier_block *self,unsigned long v,void *p) {pr_emerg ("Kernel Offset: 0x%lx from 0x%lx\n" ,kaslr_offset (),KERNELBASE ); -return 0 ; +return NOTIFY_DONE ; } +static int ppc_panic_platform_handler (struct notifier_block *this, +unsigned long event,void *ptr) +{ +/* +* This handler is only registered if we have a panic callback +* on ppc_md, hence NULL check is not needed. +* Also, it may not return, so it runs really late on panic path. +*/ + ppc_md.panic (ptr); + +return NOTIFY_DONE ; +} + +static struct notifier_block ppc_fadump_block = { + .notifier_call = ppc_panic_fadump_handler, + .priority =INT_MAX ,/* run early, to notify the firmware ASAP */ +}; +static struct notifier_block kernel_offset_notifier = { - .notifier_call = dump_kernel_offset + .notifier_call = dump_kernel_offset, +}; + +static struct notifier_block ppc_panic_block = { + .notifier_call = ppc_panic_platform_handler, + .priority =INT_MIN ,/* may not return; must be done last */ };void __init setup_panic (void ) { +/* Hard-disables IRQs + deal with FW-assisted dump (fadump) */ +atomic_notifier_chain_register (&panic_notifier_list, + &ppc_fadump_block); +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE ) &&kaslr_offset () >0 )atomic_notifier_chain_register (&panic_notifier_list, &kernel_offset_notifier); -/* PPC64 always does a hard irq disable in its panic handler */ -if (!IS_ENABLED (CONFIG_PPC64 ) && !ppc_md.panic) -return ; -atomic_notifier_chain_register (&panic_notifier_list, &ppc_panic_block); +/* Low-level platform-specific routines that should run on panic */ +if (ppc_md.panic) +atomic_notifier_chain_register (&panic_notifier_list, + &ppc_panic_block); }#ifdef CONFIG_CHECK_CACHE_COHERENCY
Signature · 1 line
2.36.0
Re: [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers
Quoted 34 lines · Guilherme G. Piccoli
The panic notifiers infrastructure is a bit limited in the scope of the callbacks - basically every kind of functionality is dropped in a list that runs in the same point during the kernel panic path. This is not really on par with the complexities and particularities of architecture / hypervisors' needs, and a refactor is ongoing.
As part of this refactor, it was observed that powerpc has 2 notifiers, with mixed goals: one is just a KASLR offset dumper, whereas the other aims to hard-disable IRQs (necessary on panic path), warn firmware of the panic event (fadump) and run low-level platform-specific machinery that might stop kernel execution and never come back.
Clearly, the 2nd notifier has opposed goals: disable IRQs / fadump should run earlier while low-level platform actions should run late since it might not even return. Hence, this patch decouples the notifiers splitting them in three:
- First one is responsible for hard-disable IRQs and fadump, should run early;
- The kernel KASLR offset dumper is really an informative notifier, harmless and may run at any moment in the panic path;
- The last notifier should run last, since it aims to perform low-level actions for specific platforms, and might never return. It is also only registered for 2 platforms, pseries and ps3.
The patch better documents the notifiers and clears the code too, also removing a useless header.
Currently no functionality change should be observed, but after the planned panic refactor we should expect more panic reliability with this patch.
Quoted 6 lines
Cc Benjamin Herrenschmidt <[email protected]>Cc Hari Bathini <[email protected]>Cc Michael Ellerman <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Paul Mackerras <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
The change looks good. I have tested it on an LPAR (ppc64).
Quoted 133 lines
---
We'd like to thanks specially the MiniCloud infrastructure [0] maintainers, that allow us to test PowerPC code in a very complete, functional and FREE environment (there's no need even for adding a credit card, like many "free" clouds require ?? ).
[0] https://openpower.ic.unicamp.br/minicloud
arch/powerpc/kernel/setup-common.c | 74 ++++++++++++++++++++++-------- 1 file changed, 54 insertions(+), 20 deletions(-)arch/powerpc/kernel/setup-common.c +54
-20@@ -23,7 +23,6 @@#include <linux/console.h> #include <linux/screen_info.h> #include <linux/root_dev.h> -#include <linux/notifier.h> #include <linux/cpu.h> #include <linux/unistd.h> #include <linux/serial.h> @@ -680,8 +679,25 @@ int check_legacy_ioport(unsigned long base_port) }EXPORT_SYMBOL (check_legacy_ioport); -static int ppc_panic_event (struct notifier_block *this, -unsigned long event,void *ptr) +/* +* Panic notifiers setup +* +* We have 3 notifiers for powerpc, each one from a different "nature": +* +* - ppc_panic_fadump_handler() is a hypervisor notifier, which hard-disables +* IRQs and deal with the Firmware-Assisted dump, when it is configured; +* should run early in the panic path. +* +* - dump_kernel_offset() is an informative notifier, just showing the KASLR +* offset if we have RANDOMIZE_BASE set. +* +* - ppc_panic_platform_handler() is a low-level handler that's registered +* only if the platform wishes to perform final actions in the panic path, +* hence it should run late and might not even return. Currently, only +* pseries and ps3 platforms register callbacks. +*/ +static int ppc_panic_fadump_handler (struct notifier_block *this, +unsigned long event,void *ptr) {/* * panic does a local_irq_disable, but we really @@ -691,45 +707,63 @@ static int ppc_panic_event(struct notifier_block *this,/* * If firmware-assisted dump has been registered then trigger -* firmware-assisted dump and let firmware handle everything else. +* its callback and let the firmware handles everything else. */ crash_fadump (NULL , ptr); -if (ppc_md.panic) - ppc_md.panic (ptr);/* May not return */ +return NOTIFY_DONE ; } -static struct notifier_block ppc_panic_block = { - .notifier_call = ppc_panic_event, - .priority =INT_MIN /* may not return; must be done last */ -}; - -/* -* Dump out kernel offset information on panic. -*/ static int dump_kernel_offset (struct notifier_block *self,unsigned long v,void *p) {pr_emerg ("Kernel Offset: 0x%lx from 0x%lx\n" ,kaslr_offset (),KERNELBASE ); -return 0 ; +return NOTIFY_DONE ; } +static int ppc_panic_platform_handler (struct notifier_block *this, +unsigned long event,void *ptr) +{ +/* +* This handler is only registered if we have a panic callback +* on ppc_md, hence NULL check is not needed. +* Also, it may not return, so it runs really late on panic path. +*/ + ppc_md.panic (ptr); + +return NOTIFY_DONE ; +} + +static struct notifier_block ppc_fadump_block = { + .notifier_call = ppc_panic_fadump_handler, + .priority =INT_MAX ,/* run early, to notify the firmware ASAP */ +}; +static struct notifier_block kernel_offset_notifier = { - .notifier_call = dump_kernel_offset + .notifier_call = dump_kernel_offset, +}; + +static struct notifier_block ppc_panic_block = { + .notifier_call = ppc_panic_platform_handler, + .priority =INT_MIN ,/* may not return; must be done last */ };void __init setup_panic (void ) { +/* Hard-disables IRQs + deal with FW-assisted dump (fadump) */ +atomic_notifier_chain_register (&panic_notifier_list, + &ppc_fadump_block); +if (IS_ENABLED (CONFIG_RANDOMIZE_BASE ) &&kaslr_offset () >0 )atomic_notifier_chain_register (&panic_notifier_list, &kernel_offset_notifier); -/* PPC64 always does a hard irq disable in its panic handler */ -if (!IS_ENABLED (CONFIG_PPC64 ) && !ppc_md.panic) -return ; -atomic_notifier_chain_register (&panic_notifier_list, &ppc_panic_block); +/* Low-level platform-specific routines that should run on panic */ +if (ppc_md.panic) +atomic_notifier_chain_register (&panic_notifier_list, + &ppc_panic_block); }#ifdef CONFIG_CHECK_CACHE_COHERENCY
Re: [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers
Quoted 5 lines · Hari Bathini
[...]
The change looks good. I have tested it on an LPAR (ppc64).
Reviewed-by Hari Bathini <[email protected]>
Thanks a bunch Hari, much appreciated!
Re: [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers
In reply to Hari BathiniQuoted 5 lines · Hari Bathini
[...]
The change looks good. I have tested it on an LPAR (ppc64).Reviewed-by Hari Bathini <[email protected]>
Hi Michael. do you think it's possible to add this one to powerpc/next (or something like that), or do you prefer a V2 with his tag? Thanks,
Guilherme
Michael Ellerman
reviewerRe: [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers
Quoted 9 lines · Guilherme G. Piccoli
Quoted 5 lines · Hari Bathini
[...]
The change looks good. I have tested it on an LPAR (ppc64).Reviewed-by Hari Bathini <[email protected]>Hi Michael. do you think it's possible to add this one to powerpc/next (or something like that), or do you prefer a V2 with his tag?
Ah sorry, I assumed it was going in as part of the whole series. I guess I misread the cover letter.
So you want me to take this patch on its own via the powerpc tree?
cheers
Re: [PATCH 08/30] powerpc/setup: Refactor/untangle panic notifiers
Quoted 11 lines · Michael Ellerman
Quoted 9 lines · Guilherme G. Piccoli
Quoted 5 lines · Hari Bathini
[...]
The change looks good. I have tested it on an LPAR (ppc64).Reviewed-by Hari Bathini <[email protected]>Hi Michael. do you think it's possible to add this one to powerpc/next (or something like that), or do you prefer a V2 with his tag?
Quoted 6 lines
Ah sorry, I assumed it was going in as part of the whole series. I guess I misread the cover letter.
So you want me to take this patch on its own via the powerpc tree?
cheers
Hi Michael, thanks for the prompt response!
You didn't misread, that was the plan heh
But some maintainers start to take patches and merge in their trees, and in the end, it seems to make sense - almost half of this series are fixes or clean-ups, that are not really necessary to get merged altogether.
So, if you can take this one, I'd appreciate - it'll make V2 a bit smaller =)
Cheers,
Guilherme
PATCH 9/30 coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
In reply to Guilherme G. PiccoliThe panic notifier infrastructure executes registered callbacks when a panic event happens - such callbacks are executed in atomic context, with interrupts and preemption disabled in the running CPU and all other CPUs disabled. That said, mutexes in such context are not a good idea.
This patch replaces a regular mutex with a mutex_trylock safer approach; given the nature of the mutex used in the driver, it should be pretty uncommon being unable to acquire such mutex in the panic path, hence no functional change should be observed (and if it is, that would be likely a deadlock with the regular mutex).
--- drivers/hwtracing/coresight/coresight-cpu-debug.c | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
drivers/hwtracing/coresight/coresight-cpu-debug.c +4 -3
@@ -380,9 +380,10 @@ static int debug_notifier_call(struct notifier_block *self,int cpu;struct debug_drvdata *drvdata; -mutex_lock (&debug_lock); +/* Bail out if we can't acquire the mutex or the functionality is off */ +if (!mutex_trylock (&debug_lock)) +return NOTIFY_DONE ; -/* Bail out if the functionality is disabled */ if (!debug_enable) goto skip_dump;
@@ -401,7 +402,7 @@ static int debug_notifier_call(struct notifier_block *self, skip_dump:mutex_unlock (&debug_lock); -return 0 ; +return NOTIFY_DONE ; }static struct notifier_block debug_notifier = {
Signature · 1 line
2.36.0
Re: [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
Hi Guilherme,
Quoted 11 lines · Guilherme G. Piccoli
The panic notifier infrastructure executes registered callbacks when a panic event happens - such callbacks are executed in atomic context, with interrupts and preemption disabled in the running CPU and all other CPUs disabled. That said, mutexes in such context are not a good idea.
This patch replaces a regular mutex with a
mutex_trylocksafer approach; given the nature of the mutex used in the driver, it should be pretty uncommon being unable to acquire such mutex in the panic path, hence no functional change should be observed (and if it is, that would be likely a deadlock with the regular mutex).
Quoted 6 lines
Fixes 2227b7c74634 ("coresight: add support for CPU debug module")Cc Leo Yan <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Mike Leach <[email protected]>Cc Suzuki K Poulose <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
How would you like to proceed with queuing this ? I am happy either way. In case you plan to push this as part of this series (I don't see any potential conflicts) :
Re: [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
Quoted 7 lines · Suzuki K Poulose
Hi Guilherme,
[...]
How would you like to proceed with queuing this ? I am happy either way. In case you plan to push this as part of this series (I don't see any potential conflicts) :Reviewed-by Suzuki K Poulose <[email protected]>
Thanks for your review Suzuki, much appreciated!
About your question, I'm not sure yet - in case the core changes would take a while (like if community find them polemic, require many changes, etc) I might split this series in 2 parts, the fixes part vs the improvements per se. Either way, a V2 is going to happen for sure, and in that moment, I'll let you know what I think it's best.
But either way, any choice you prefer is fine by me as well (like if you want to merge it now or postpone to get merged in the future), this is not an urgent fix I think =)
Cheers,
Guilherme
Re: [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
In reply to Suzuki K PouloseQuoted 21 lines · Suzuki K Poulose
Hi Guilherme,
Quoted 11 lines · Guilherme G. Piccoli
The panic notifier infrastructure executes registered callbacks when a panic event happens - such callbacks are executed in atomic context, with interrupts and preemption disabled in the running CPU and all other CPUs disabled. That said, mutexes in such context are not a good idea.
This patch replaces a regular mutex with a
mutex_trylocksafer approach; given the nature of the mutex used in the driver, it should be pretty uncommon being unable to acquire such mutex in the panic path, hence no functional change should be observed (and if it is, that would be likely a deadlock with the regular mutex).Quoted 6 lines
Fixes 2227b7c74634 ("coresight: add support for CPU debug module")Cc Leo Yan <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Mike Leach <[email protected]>Cc Suzuki K Poulose <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Quoted 5 lines
How would you like to proceed with queuing this ? I am happy either way. In case you plan to push this as part of this series (I don't see any potential conflicts) :
Reviewed-by Suzuki K Poulose <[email protected]>
Hi Suzuki, some other maintainers are taking the patches to their next branches for example. I'm working on V2, and I guess in the end would be nice to reduce the size of the series a bit.
So, do you think you could pick this one for your coresight/next branch (or even for rc cycle, your call - this is really a fix)? This way, I won't re-submit this one in V2, since it's gonna be merged already in your branch.
Thanks in advance,
Guilherme
Re: [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
Hi
Quoted 28 lines · Guilherme G. Piccoli
Quoted 21 lines · Suzuki K Poulose
Hi Guilherme,
Quoted 11 lines · Guilherme G. Piccoli
The panic notifier infrastructure executes registered callbacks when a panic event happens - such callbacks are executed in atomic context, with interrupts and preemption disabled in the running CPU and all other CPUs disabled. That said, mutexes in such context are not a good idea.
This patch replaces a regular mutex with a
mutex_trylocksafer approach; given the nature of the mutex used in the driver, it should be pretty uncommon being unable to acquire such mutex in the panic path, hence no functional change should be observed (and if it is, that would be likely a deadlock with the regular mutex).Quoted 6 lines
Fixes 2227b7c74634 ("coresight: add support for CPU debug module")Cc Leo Yan <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Mike Leach <[email protected]>Cc Suzuki K Poulose <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>Quoted 5 lines
How would you like to proceed with queuing this ? I am happy either way. In case you plan to push this as part of this series (I don't see any potential conflicts) :
Reviewed-by Suzuki K Poulose <[email protected]>
Quoted 8 lines
Hi Suzuki, some other maintainers are taking the patches to their next branches for example. I'm working on V2, and I guess in the end would be nice to reduce the size of the series a bit.
So, do you think you could pick this one for your coresight/next branch (or even for rc cycle, your call - this is really a fix)? This way, I won't re-submit this one in V2, since it's gonna be merged already in your branch.
I have queued this to coresight/next.
Thanks
Suzuki
Re: [PATCH 09/30] coresight: cpu-debug: Replace mutex with mutex_trylock on panic notifier
Quoted 5 lines · Suzuki K Poulose
[...]>
I have queued this to coresight/next.Thanks
Suzuki
Thanks a lot Suzuki!
PATCH 10/30 alpha: Clean-up the panic notifier code
In reply to Guilherme G. PiccoliThe alpha panic notifier has some code issues, not following the conventions of other notifiers. Also, it might halt the machine but still it is set to run as early as possible, which doesn't seem to be a good idea.
This patch cleans the code, and set the notifier to run as the latest, following the same approach other architectures are doing. Also, we remove the unnecessary include of a header already included indirectly.
--- arch/alpha/kernel/setup.c | 36 +++++++++++++++--------------------- 1 file changed, 15 insertions(+), 21 deletions(-)
arch/alpha/kernel/setup.c +15 -21
@@ -41,19 +41,11 @@#include <linux/sysrq.h> #include <linux/reboot.h> #endif -#include <linux/notifier.h> #include <asm/setup.h> #include <asm/io.h> #include <linux/log2.h> #include <linux/export.h> -static int alpha_panic_event (struct notifier_block *,unsigned long ,void *); -static struct notifier_block alpha_panic_block = { - alpha_panic_event, -NULL , -INT_MAX /* try to do it first */ -}; -#include <linux/uaccess.h> #include <asm/hwrpb.h> #include <asm/dma.h>
@@ -435,6 +427,21 @@ static const struct sysrq_key_op srm_sysrq_reboot_op = { };#endif +static int alpha_panic_event (struct notifier_block *this, +unsigned long event,void *ptr) +{ +/* If we are using SRM and serial console, just hard halt here. */ +if (alpha_using_srm && srmcons_output) +__halt (); + +return NOTIFY_DONE ; +} + +static struct notifier_block alpha_panic_block = { + .notifier_call = alpha_panic_event, + .priority =INT_MIN ,/* may not return, do it last */ +}; +void __init setup_arch (char **cmdline_p) {
@@ -1427,19 +1434,6 @@ const struct seq_operations cpuinfo_op = { .show = show_cpuinfo, }; - -static int -alpha_panic_event (struct notifier_block *this,unsigned long event,void *ptr) -{ -#if 1 -/* FIXME FIXME FIXME */ -/* If we are using SRM and serial console, just hard halt here. */ -if (alpha_using_srm && srmcons_output) -__halt (); -#endif -return NOTIFY_DONE ; -} -static __init int add_pcspkr (void ) {struct platform_device *pd;
Signature · 1 line
2.36.0
Re: [PATCH 10/30] alpha: Clean-up the panic notifier code
Quoted 74 lines · Guilherme G. Piccoli
The alpha panic notifier has some code issues, not following the conventions of other notifiers. Also, it might halt the machine but still it is set to run as early as possible, which doesn't seem to be a good idea.
This patch cleans the code, and set the notifier to run as the latest, following the same approach other architectures are doing. Also, we remove the unnecessary include of a header already included indirectly.
Cc Ivan Kokshaysky <[email protected]>Cc Matt Turner <[email protected]>Cc Richard Henderson <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/alpha/kernel/setup.c | 36 +++++++++++++++--------------------- 1 file changed, 15 insertions(+), 21 deletions(-)arch/alpha/kernel/setup.c +15
-14@@ -41,19 +41,11 @@#include <linux/sysrq.h> #include <linux/reboot.h> #endif -#include <linux/notifier.h> #include <asm/setup.h> #include <asm/io.h> #include <linux/log2.h> #include <linux/export.h> -static int alpha_panic_event (struct notifier_block *,unsigned long ,void *); -static struct notifier_block alpha_panic_block = { - alpha_panic_event, -NULL , -INT_MAX /* try to do it first */ -}; -#include <linux/uaccess.h> #include <asm/hwrpb.h> #include <asm/dma.h> @@ -435,6 +427,21 @@ static const struct sysrq_key_op srm_sysrq_reboot_op = { };#endif +static int alpha_panic_event (struct notifier_block *this, +unsigned long event,void *ptr) +{ +/* If we are using SRM and serial console, just hard halt here. */ +if (alpha_using_srm && srmcons_output) +__halt (); + +return NOTIFY_DONE ; +} + +static struct notifier_block alpha_panic_block = { + .notifier_call = alpha_panic_event, + .priority =INT_MIN ,/* may not return, do it last */ +}; +void __init setup_arch (char **cmdline_p) {@@ -1427,19 +1434,6 @@ const struct seq_operations cpuinfo_op = { .show = show_cpuinfo, }; - -static int -alpha_panic_event (struct notifier_block *this,unsigned long event,void *ptr) -{ -#if 1 -/* FIXME FIXME FIXME */
arch/alpha/kernel/setup.c const struct seq_operations cpuinfo_op = { · line 1439
Quoted 10 lines
arch/alpha/kernel/setup.c +0
-7-/* If we are using SRM and serial console, just hard halt here. */ -if (alpha_using_srm && srmcons_output) -__halt (); -#endif -return NOTIFY_DONE ; -} -static __init int add_pcspkr (void ) {struct platform_device *pd;
Hi folks, I'm updating Richard's email and re-sending the V1, any reviews are greatly appreciated!
Cheers,
Guilherme
Re: [PATCH 10/30] alpha: Clean-up the panic notifier code
Quoted 5 lines · Guilherme G. Piccoli
Quoted 4 lines · Guilherme G. Piccoli
The alpha panic notifier has some code issues, not following the conventions of other notifiers. Also, it might halt the machine but still it is set to run as early as possible, which doesn't seem to be a good idea.
Yeah, it is pretty strange behavior.
I looked into the history. This notifier was added into the alpha code in 2.4.0-test2pre2. In this historic code, the default panic() code either rebooted after a timeout or ended in a infinite loop. There was not crasdump at that times.
The notifier allowed to change the behavior. There were 3 notifiers:
+ mips and mips64 ended with blinking in panic()
+ alpha did __halt() in this srm case
They both still do this. I guess that it is some historic behavior that people using these architectures are used to.
Anyway, it makes sense to do this as the last notifier after dumping other information.
Best Regards,
Petr
Re: [PATCH 10/30] alpha: Clean-up the panic notifier code
Quoted 16 lines · Petr Mladek
[...]
Yeah, it is pretty strange behavior.I looked into the history. This notifier was added into the alpha code in 2.4.0-test2pre2. In this historic code, the default
panic()code either rebooted after a timeout or ended in a infinite loop. There was not crasdump at that times.The notifier allowed to change the behavior. There were 3 notifiers:
+ mips and mips64 ended with blinking in
panic()
+ alpha did__halt()in this srm caseThey both still do this. I guess that it is some historic behavior that people using these architectures are used to.
Quoted 7 lines
Anyway, it makes sense to do this as the last notifier after dumping other information.
Reviewed-by Petr Mladek <[email protected]>Best Regards,
Petr
Thanks a bunch for the review - added your tag for V2 =)
PATCH 11/30 um: Improve panic notifiers consistency and ordering
In reply to Guilherme G. PiccoliCurrently the panic notifiers from user mode linux don't follow the convention for most of the other notifiers present in the kernel (indentation, priority setting, numeric return). More important, the priorities could be improved, since it's a special case (userspace), hence we could run the notifiers earlier; user mode linux shouldn't care much with other panic notifiers but the ordering among the mconsole and arch notifier is important, given that the arch one effectively triggers a core dump.
This patch fixes that by running the mconsole notifier as the first panic notifier, followed by the architecture one (that coredumps). Also, we remove a useless header inclusion.
--- arch/um/drivers/mconsole_kern.c | 8 +++----- arch/um/kernel/um_arch.c | 8 ++++---- 2 files changed, 7 insertions(+), 9 deletions(-)
arch/um/drivers/mconsole_kern.c +3 -5
@@ -11,7 +11,6 @@#include <linux/list.h> #include <linux/mm.h> #include <linux/module.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> #include <linux/sched/debug.h>
@@ -846,13 +845,12 @@ static int notify_panic(struct notifier_block *self, unsigned long unused1,mconsole_notify (notify_socket,MCONSOLE_PANIC , message,strlen (message) +1 ); -return 0 ; +return NOTIFY_DONE ; }static struct notifier_block panic_exit_notifier = { - .notifier_call = notify_panic, - .next =NULL , - .priority =1 + .notifier_call = notify_panic, + .priority =INT_MAX ,/* run as soon as possible */ };static int add_notifier (void )
arch/um/kernel/um_arch.c +4 -4
@@ -246,13 +246,13 @@ static int panic_exit(struct notifier_block *self, unsigned long unused1,bust_spinlocks (0 ); uml_exitcode =1 ;os_dump_core (); -return 0 ; + +return NOTIFY_DONE ; }static struct notifier_block panic_exit_notifier = { - .notifier_call = panic_exit, - .next =NULL , - .priority =0 + .notifier_call = panic_exit, + .priority =INT_MAX -1 ,/* run as 2nd notifier, won't return */ };void uml_finishsetup (void )
Signature · 1 line
2.36.0
Johannes Berg
maintainerRe: [PATCH 11/30] um: Improve panic notifiers consistency and ordering
[trimming massive CC list]
Quoted 2 lines · Guilherme G. Piccoli
Also, we remove a useless header inclusion.
I wouldn't say it's useless, generally we try not to rely on implicit includes so much? And you at least now use NOTIFY_DONE from it.
Otherwise looks fine to me.
johannes
Re: [PATCH 11/30] um: Improve panic notifiers consistency and ordering
Quoted 4 lines · Johannes Berg
[trimming massive CC list]
Quoted 1 line · Guilherme G. Piccoli
Quoted 8 lines
Quoted 1 line
Also, we remove a useless header inclusion.
I wouldn't say it's useless, generally we try not to rely on implicit includes so much? And you at least now use
NOTIFY_DONEfrom it.Otherwise looks fine to me.
johannes
Hi Johannes, thanks for your prompt response, and for clearing a bit the *huge* CC list, it got really humongous...
I agree with you here - I missed that there's also a reboot notifier in "mconsole_kern.c", so seems it makes sense to keep the header. I'll fix that for V2, ok? If it was only a panic notifier I would consider it useless maybe, but we have non-panic notifier as well heh
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 11/30] um: Improve panic notifiers consistency and ordering
In reply to Guilherme G. PiccoliQuoted 8 lines · Guilherme G. Piccoli
Currently the panic notifiers from user mode linux don't follow the convention for most of the other notifiers present in the kernel (indentation, priority setting, numeric return). More important, the priorities could be improved, since it's a special case (userspace), hence we could run the notifiers earlier; user mode linux shouldn't care much with other panic notifiers but the ordering among the mconsole and arch notifier is important, given that the arch one effectively triggers a core dump.
It is not clear to me why user mode linux should not care about the other notifiers. It might be because I do not know much about the user mode linux.
Is the because they always create core dump or are never running in a hypervisor or ...?
AFAIK, the notifiers do many different things. For example, there is a notifier that disables RCU watchdog, print some extra information. Why none of them make sense here?
Quoted 3 lines
This patch fixes that by running the mconsole notifier as the first panic notifier, followed by the architecture one (that coredumps). Also, we remove a useless header inclusion.
Best Regards,
Petr
Re: [PATCH 11/30] um: Improve panic notifiers consistency and ordering
Quoted 5 lines · Petr Mladek
[...]
It is not clear to me why user mode linux should not care about the other notifiers. It might be because I do not know much about the user mode linux.
Quoted 7 lines
Is the because they always create core dump or are never running in a hypervisor or ...?
AFAIK, the notifiers do many different things. For example, there is a notifier that disables RCU watchdog, print some extra information. Why none of them make sense here?
Hi Petr, my understanding is that UML is a form of running Linux as a regular userspace process for testing purposes. With that said, as soon as we exit in the error path, less "pollution" would happen, so users can use GDB to debug the core dump for example.
In later patches of this series (when we split the panic notifiers in 3 lists) these UML notifiers run in the pre-reboot list, so they run after the informational notifiers for example (in the default level). But without the list split we cannot order properly, so my gut feeling is that makes sense to run them rather earlier than later in the panic process...
Maybe Anton / Johannes / Richard could give their opinions - appreciate that, I'm not attached to the priority here, it's more about users' common usage of UML I can think of...
Cheers,
Guilherme
Johannes Berg
maintainerRe: [PATCH 11/30] um: Improve panic notifiers consistency and ordering
Quoted 9 lines · Guilherme G. Piccoli
Quoted 8 lines · Petr Mladek
[...]
It is not clear to me why user mode linux should not care about the other notifiers. It might be because I do not know much about the user mode linux.Is the because they always create core dump or are never running in a hypervisor or ...?
Quoted 7 lines
Quoted 4 lines
AFAIK, the notifiers do many different things. For example, there is a notifier that disables RCU watchdog, print some extra information. Why none of them make sense here?
Hi Petr, my understanding is that UML is a form of running Linux as a regular userspace process for testing purposes.
Correct.
Quoted 11 lines
With that said, as soon
as we exit in the error path, less "pollution" would happen, so users can use GDB to debug the core dump for example.In later patches of this series (when we split the panic notifiers in 3 lists) these UML notifiers run in the pre-reboot list, so they run after the informational notifiers for example (in the default level). But without the list split we cannot order properly, so my gut feeling is that makes sense to run them rather earlier than later in the panic process...
Quoted 3 lines
Maybe Anton / Johannes / Richard could give their opinions - appreciate that, I'm not attached to the priority here, it's more about users' common usage of UML I can think of...
It's hard to say ... In a sense I'm not sure it matters?
OTOH something like the ftrace dump notifier (kernel/trace/trace.c) might still be useful to run before the mconsole and coredump ones, even if you could probably use gdb to figure out the information.
Personally, I don't have a scenario where I'd care about the trace buffers though, and most of the others I found would seem irrelevant (drivers that aren't even compiled, hung tasks won't really happen since we exit immediately, and similar.)
johannes
Re: [PATCH 11/30] um: Improve panic notifiers consistency and ordering
Quoted 11 lines · Johannes Berg
[...]
Quoted 3 lines
Maybe Anton / Johannes / Richard could give their opinions - appreciate that, I'm not attached to the priority here, it's more about users' common usage of UML I can think of...
It's hard to say ... In a sense I'm not sure it matters?
OTOH something like the ftrace dump notifier (
kernel/trace/trace.c) might still be useful to run before the mconsole and coredump ones, even if you could probably use gdb to figure out the information.
Quoted 6 lines
Personally, I don't have a scenario where I'd care about the trace buffers though, and most of the others I found would seem irrelevant (drivers that aren't even compiled, hung tasks won't really happen since we exit immediately, and similar.)
johannes
Thanks Johannes, I agree with you.
We don't have great ordering now, one thing we need to enforce is the order between the 2 UML notifiers, and this patch is doing that..trying to order against other callbacks like the ftrace dumper is messy in the current code.
OTOH if this patch set is accepted at some point, we'll likely have 3 lists, and with that we can improve ordering a lot - this notifier for instance would run in the pre-reboot list, *after* the ftrace dumper (if a kmsg dumper is set).
So, my intention is to keep this patch as is for V2 (with some changes Johannes suggested before), unless Petr or the other maintainers want something different.
Cheers,
Guilherme
PATCH 12/30 parisc: Replace regular spinlock with spin_trylock on panic path
In reply to Guilherme G. PiccoliThe panic notifiers' callbacks execute in an atomic context, with interrupts/preemption disabled, and all CPUs not running the panic function are off, so it's very dangerous to wait on a regular spinlock, there's a risk of deadlock.
This patch refactors the panic notifier of parisc/power driver to make use of spin_trylock - for that, we've added a second version of the soft-power function. Also, some comments were reorganized and trailing white spaces, useless header inclusion and blank lines were removed.
--- arch/parisc/include/asm/pdc.h | 1 + arch/parisc/kernel/firmware.c | 27 +++++++++++++++++++++++---- drivers/parisc/power.c | 17 ++++++++++------- 3 files changed, 34 insertions(+), 11 deletions(-)
arch/parisc/include/asm/pdc.h +1 -0
@@ -83,6 +83,7 @@ int pdc_do_firm_test_reset(unsigned long ftc_bitmap);int pdc_do_reset (void );int pdc_soft_power_info (unsigned long *power_reg);int pdc_soft_power_button (int sw_control); +int pdc_soft_power_button_panic (int sw_control);void pdc_io_reset (void );void pdc_io_reset_devices (void );int pdc_iodc_getc (void );
arch/parisc/kernel/firmware.c +23 -4
@@ -1232,15 +1232,18 @@ int __init pdc_soft_power_info(unsigned long *power_reg) }/* -* pdc_soft_power_button - Control the soft power button behaviour -* @sw_control: 0 for hardware control, 1 for software control +* pdc_soft_power_button{_panic} - Control the soft power button behaviour +* @sw_control: 0 for hardware control, 1 for software control * * * This PDC function places the soft power button under software or * hardware control. -* Under software control the OS may control to when to allow to shut -* down the system. Under hardware control pressing the power button +* Under software control the OS may control to when to allow to shut +* down the system. Under hardware control pressing the power button * powers off the system immediately. +* +* The _panic version relies in spin_trylock to prevent deadlock +* on panic path. */ int pdc_soft_power_button (int sw_control) {
@@ -1254,6 +1257,22 @@ int pdc_soft_power_button(int sw_control)return retval; } +int pdc_soft_power_button_panic (int sw_control) +{ +int retval; +unsigned long flags; + +if (!spin_trylock_irqsave (&pdc_lock, flags)) { +pr_emerg ("Couldn't enable soft power button\n" ); +return -EBUSY ;/* ignored by the panic notifier */ + } + + retval =mem_pdc_call (PDC_SOFT_POWER ,PDC_SOFT_POWER_ENABLE ,__pa (pdc_result), sw_control); +spin_unlock_irqrestore (&pdc_lock, flags); + +return retval; +} +/* * pdc_io_reset - Hack to avoid overlapping range registers of Bridges devices. * Primarily a problem on T600 (which parisc-linux doesn't support) but
drivers/parisc/power.c +10 -7
@@ -37,7 +37,6 @@#include <linux/module.h> #include <linux/init.h> #include <linux/kernel.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> #include <linux/sched/signal.h>
@@ -175,16 +174,21 @@ static void powerfail_interrupt(int code, void *x) -/* parisc_panic_event() is called by the panic handler. -* As soon as a panic occurs, our tasklets above will not be -* executed any longer. This function then re-enables the -* soft-power switch and allows the user to switch off the system +/* +* parisc_panic_event() is called by the panic handler. +* +* As soon as a panic occurs, our tasklets above will not +* be executed any longer. This function then re-enables +* the soft-power switch and allows the user to switch off +* the system. We rely in pdc_soft_power_button_panic() +* since this version spin_trylocks (instead of regular +* spinlock), preventing deadlocks on panic path. */ static int parisc_panic_event (struct notifier_block *this,unsigned long event,void *ptr) {/* re-enable the soft-power switch */ -pdc_soft_power_button (0 ); +pdc_soft_power_button_panic (0 );return NOTIFY_DONE ; }
@@ -193,7 +197,6 @@ static struct notifier_block parisc_panic_block = { .priority = INT_MAX, }; -static int __init power_init (void ) {unsigned long ret;
Signature · 1 line
2.36.0
Re: [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path
Quoted 11 lines · Guilherme G. Piccoli
The panic notifiers' callbacks execute in an atomic context, with interrupts/preemption disabled, and all CPUs not running the panic function are off, so it's very dangerous to wait on a regular spinlock, there's a risk of deadlock.
This patch refactors the panic notifier of parisc/power driver to make use of
spin_trylock- for that, we've added a second version of the soft-power function. Also, some comments were reorganized and trailing white spaces, useless header inclusion and blank lines were removed.
Quoted 3 lines
Cc Helge Deller <[email protected]>Cc "James E.J. Bottomley" <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
You may add:
Helge
Quoted 115 lines
--- arch/parisc/include/asm/pdc.h | 1 + arch/parisc/kernel/firmware.c | 27 +++++++++++++++++++++++---- drivers/parisc/power.c | 17 ++++++++++------- 3 files changed, 34 insertions(+), 11 deletions(-)arch/parisc/include/asm/pdc.h +1
-0@@ -83,6 +83,7 @@ int pdc_do_firm_test_reset(unsigned long ftc_bitmap);int pdc_do_reset (void );int pdc_soft_power_info (unsigned long *power_reg);int pdc_soft_power_button (int sw_control); +int pdc_soft_power_button_panic (int sw_control);void pdc_io_reset (void );void pdc_io_reset_devices (void );int pdc_iodc_getc (void );arch/parisc/kernel/firmware.c +23
-4@@ -1232,15 +1232,18 @@ int __init pdc_soft_power_info(unsigned long *power_reg) }/* -* pdc_soft_power_button - Control the soft power button behaviour -* @sw_control: 0 for hardware control, 1 for software control +* pdc_soft_power_button{_panic} - Control the soft power button behaviour +* @sw_control: 0 for hardware control, 1 for software control * * * This PDC function places the soft power button under software or * hardware control. -* Under software control the OS may control to when to allow to shut -* down the system. Under hardware control pressing the power button +* Under software control the OS may control to when to allow to shut +* down the system. Under hardware control pressing the power button * powers off the system immediately. +* +* The _panic version relies in spin_trylock to prevent deadlock +* on panic path. */ int pdc_soft_power_button (int sw_control) {@@ -1254,6 +1257,22 @@ int pdc_soft_power_button(int sw_control)return retval; } +int pdc_soft_power_button_panic (int sw_control) +{ +int retval; +unsigned long flags; + +if (!spin_trylock_irqsave (&pdc_lock, flags)) { +pr_emerg ("Couldn't enable soft power button\n" ); +return -EBUSY ;/* ignored by the panic notifier */ + } + + retval =mem_pdc_call (PDC_SOFT_POWER ,PDC_SOFT_POWER_ENABLE ,__pa (pdc_result), sw_control); +spin_unlock_irqrestore (&pdc_lock, flags); + +return retval; +} +/* * pdc_io_reset - Hack to avoid overlapping range registers of Bridges devices. * Primarily a problem on T600 (which parisc-linux doesn't support) but drivers/parisc/power.c +10
-7@@ -37,7 +37,6 @@#include <linux/module.h> #include <linux/init.h> #include <linux/kernel.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> #include <linux/sched/signal.h> @@ -175,16 +174,21 @@ static void powerfail_interrupt(int code, void *x) -/* parisc_panic_event() is called by the panic handler. -* As soon as a panic occurs, our tasklets above will not be -* executed any longer. This function then re-enables the -* soft-power switch and allows the user to switch off the system +/* +* parisc_panic_event() is called by the panic handler. +* +* As soon as a panic occurs, our tasklets above will not +* be executed any longer. This function then re-enables +* the soft-power switch and allows the user to switch off +* the system. We rely in pdc_soft_power_button_panic() +* since this version spin_trylocks (instead of regular +* spinlock), preventing deadlocks on panic path. */ static int parisc_panic_event (struct notifier_block *this,unsigned long event,void *ptr) {/* re-enable the soft-power switch */ -pdc_soft_power_button (0 ); +pdc_soft_power_button_panic (0 );return NOTIFY_DONE ; }@@ -193,7 +197,6 @@ static struct notifier_block parisc_panic_block = { .priority = INT_MAX, }; -static int __init power_init (void ) {unsigned long ret;
Re: [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path
Quoted 5 lines · Helge Deller
[...]
You may add:Helge
Thanks Helge, added!
Cheers,
Guilherme
Quoted 118 lines
Quoted 117 lines
--- arch/parisc/include/asm/pdc.h | 1 + arch/parisc/kernel/firmware.c | 27 +++++++++++++++++++++++---- drivers/parisc/power.c | 17 ++++++++++------- 3 files changed, 34 insertions(+), 11 deletions(-)arch/parisc/include/asm/pdc.h +1
-0@@ -83,6 +83,7 @@ int pdc_do_firm_test_reset(unsigned long ftc_bitmap);int pdc_do_reset (void );int pdc_soft_power_info (unsigned long *power_reg);int pdc_soft_power_button (int sw_control); +int pdc_soft_power_button_panic (int sw_control);void pdc_io_reset (void );void pdc_io_reset_devices (void );int pdc_iodc_getc (void );arch/parisc/kernel/firmware.c +23
-4@@ -1232,15 +1232,18 @@ int __init pdc_soft_power_info(unsigned long *power_reg) }/* -* pdc_soft_power_button - Control the soft power button behaviour -* @sw_control: 0 for hardware control, 1 for software control +* pdc_soft_power_button{_panic} - Control the soft power button behaviour +* @sw_control: 0 for hardware control, 1 for software control * * * This PDC function places the soft power button under software or * hardware control. -* Under software control the OS may control to when to allow to shut -* down the system. Under hardware control pressing the power button +* Under software control the OS may control to when to allow to shut +* down the system. Under hardware control pressing the power button * powers off the system immediately. +* +* The _panic version relies in spin_trylock to prevent deadlock +* on panic path. */ int pdc_soft_power_button (int sw_control) {@@ -1254,6 +1257,22 @@ int pdc_soft_power_button(int sw_control)return retval; } +int pdc_soft_power_button_panic (int sw_control) +{ +int retval; +unsigned long flags; + +if (!spin_trylock_irqsave (&pdc_lock, flags)) { +pr_emerg ("Couldn't enable soft power button\n" ); +return -EBUSY ;/* ignored by the panic notifier */ + } + + retval =mem_pdc_call (PDC_SOFT_POWER ,PDC_SOFT_POWER_ENABLE ,__pa (pdc_result), sw_control); +spin_unlock_irqrestore (&pdc_lock, flags); + +return retval; +} +/* * pdc_io_reset - Hack to avoid overlapping range registers of Bridges devices. * Primarily a problem on T600 (which parisc-linux doesn't support) but drivers/parisc/power.c +10
-7@@ -37,7 +37,6 @@#include <linux/module.h> #include <linux/init.h> #include <linux/kernel.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> #include <linux/sched/signal.h> @@ -175,16 +174,21 @@ static void powerfail_interrupt(int code, void *x) -/* parisc_panic_event() is called by the panic handler. -* As soon as a panic occurs, our tasklets above will not be -* executed any longer. This function then re-enables the -* soft-power switch and allows the user to switch off the system +/* +* parisc_panic_event() is called by the panic handler. +* +* As soon as a panic occurs, our tasklets above will not +* be executed any longer. This function then re-enables +* the soft-power switch and allows the user to switch off +* the system. We rely in pdc_soft_power_button_panic() +* since this version spin_trylocks (instead of regular +* spinlock), preventing deadlocks on panic path. */ static int parisc_panic_event (struct notifier_block *this,unsigned long event,void *ptr) {/* re-enable the soft-power switch */ -pdc_soft_power_button (0 ); +pdc_soft_power_button_panic (0 );return NOTIFY_DONE ; }@@ -193,7 +197,6 @@ static struct notifier_block parisc_panic_block = { .priority = INT_MAX, }; -static int __init power_init (void ) {unsigned long ret;
Re: [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path
In reply to Helge DellerQuoted 5 lines · Helge Deller
[...]
You may add:Helge
Hi Helge, do you think would be possible to still pick this one for v5.19 or do you prefer to hold for the next release?
I'm working on V2, so if it's merged for 5.19 I won't send it again. Thanks,
Guilherme
Helge Deller
maintainerRe: [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path
Hello Guilherme,
Quoted 7 lines · Guilherme G. Piccoli
Quoted 3 lines · Helge Deller
[...]
You may add:Hi Helge, do you think would be possible to still pick this one for v5.19 or do you prefer to hold for the next release?
Actually, I'd prefer if you push this patch together with the whole series upstream at once. The patch itself makes not much sense without your series...
Quoted 1 line
I'm working on V2, so if it's merged for 5.19 I won't send it again.
Helge
Re: [PATCH 12/30] parisc: Replace regular spinlock with spin_trylock on panic path
Quoted 11 lines · Helge Deller
Hello Guilherme,
Quoted 7 lines · Guilherme G. Piccoli
Quoted 3 lines · Helge Deller
[...]
You may add:Hi Helge, do you think would be possible to still pick this one for v5.19 or do you prefer to hold for the next release?
Quoted 7 lines
Actually, I'd prefer if you push this patch together with the whole series upstream at once. The patch itself makes not much sense without your series...
Quoted 1 line
I'm working on V2, so if it's merged for 5.19 I won't send it again.
Helge
Sure Helge, I guess I can do that - will resubmit for V2.
But notice the patch is self-contained, as it fixes a current issue in the code - the risk for a lockup due to spinlock taking on atomic context. It doesn't require the panic refactor to be merged in order to achieve its goal...
I agree that such issue is rare to trigger though, so definitely no hurry is needed =)
Cheers,
Guilherme
PATCH 13/30 s390/consoles: Improve panic notifiers reliability
In reply to Guilherme G. PiccoliCurrently many console drivers for s390 rely on panic/reboot notifiers to invoke callbacks on these events. The panic() function disables local IRQs, secondary CPUs and preemption, so callbacks invoked on panic are effectively running in atomic context.
Happens that most of these console callbacks from s390 doesn't take the proper care with regards to atomic context, like taking spinlocks that might be taken in other function/CPU and hence will cause a lockup situation.
The goal for this patch is to improve the notifiers reliability, acting on 4 console drivers, as detailed below:
(1) con3215: changed a regular spinlock to the trylock alternative.
(2) con3270: also changed a regular spinlock to its trylock counterpart, but here we also have another problem: raw3270_activate_view() takes a different spinlock. So, we worked a helper to validate if this other lock is safe to acquire, and if so, raw3270_activate_view() should be safe.
Notice though that there is a functional change here: it's now possible to continue the notifier code [reaching con3270_wait_write() and con3270_rebuild_update()] without executing raw3270_activate_view().
(3) sclp: a global lock is used heavily in the functions called from the notifier, so we added a check here - if the lock is taken already, we just bail-out, preventing the lockup.
(4) sclp_vt220: same as (3), a lock validation was added to prevent the potential lockup problem.
Besides (1)-(4), we also removed useless void functions, adding the code called from the notifier inside its own body, and changed the priority of such notifiers to execute late, since they are "heavyweight" for the panic environment, so we aim to reduce risks here. Changed return values to NOTIFY_DONE as well, the standard one.
---
As a design choice, the option used here to verify a given spinlock is taken was the function "spin_is_locked()" - but we noticed that it is not often used. An alternative would to take the lock with a spin_trylock() and if it succeeds, just release the spinlock and continue the code. But that seemed weird...
Also, we'd like to ask a good validation of case (2) potential functionality change from the s390 console experts - far from expert here, and in our naive code observation, that seems fine, but that analysis might be missing some corner case.
Thanks in advance!
drivers/s390/char/con3215.c | 36 +++++++++++++++-------------- drivers/s390/char/con3270.c | 34 +++++++++++++++------------ drivers/s390/char/raw3270.c | 18 +++++++++++++++ drivers/s390/char/raw3270.h | 1 + drivers/s390/char/sclp_con.c | 28 +++++++++++++---------- drivers/s390/char/sclp_vt220.c | 42 +++++++++++++++++++--------------- 6 files changed, 96 insertions(+), 63 deletions(-)
drivers/s390/char/con3215.c +19 -17
@@ -771,35 +771,37 @@ static struct tty_driver *con3215_device(struct console *c, int *index) }/* -* panic() calls con3215_flush through a panic_notifier -* before the system enters a disabled, endless loop. +* The below function is called as a panic/reboot notifier before the +* system enters a disabled, endless loop. +* +* Notice we must use the spin_trylock() alternative, to prevent lockups +* in atomic context (panic routine runs with secondary CPUs, local IRQs +* and preemption disabled). */ -static void con3215_flush (void ) -{ -struct raw3215_info *raw; -unsigned long flags; - - raw = raw3215[0 ];/* console 3215 is the first one */ -spin_lock_irqsave (get_ccwdev_lock (raw->cdev), flags); -raw3215_make_room (raw,RAW3215_BUFFER_SIZE ); -spin_unlock_irqrestore (get_ccwdev_lock (raw->cdev), flags); -} -static int con3215_notify (struct notifier_block *self,unsigned long event,void *data) { -con3215_flush (); -return NOTIFY_OK ; +struct raw3215_info *raw; +unsigned long flags; + + raw = raw3215[0 ];/* console 3215 is the first one */ +if (!spin_trylock_irqsave (get_ccwdev_lock (raw->cdev), flags)) +return NOTIFY_DONE ; + +raw3215_make_room (raw,RAW3215_BUFFER_SIZE ); +spin_unlock_irqrestore (get_ccwdev_lock (raw->cdev), flags); + +return NOTIFY_DONE ; }static struct notifier_block on_panic_nb = { .notifier_call = con3215_notify, - .priority =0 , + .priority =INT_MIN +1 ,/* run the callback late */ };static struct notifier_block on_reboot_nb = { .notifier_call = con3215_notify, - .priority =0 , + .priority =INT_MIN +1 ,/* run the callback late */ };/*
drivers/s390/char/con3270.c +19 -15
@@ -535,20 +535,29 @@ con3270_wait_write(struct con3270 *cp) }/* -* panic() calls con3270_flush through a panic_notifier -* before the system enters a disabled, endless loop. +* The below function is called as a panic/reboot notifier before the +* system enters a disabled, endless loop. +* +* Notice we must use the spin_trylock() alternative, to prevent lockups +* in atomic context (panic routine runs with secondary CPUs, local IRQs +* and preemption disabled). */ -static void -con3270_flush (void ) +static int con3270_notify (struct notifier_block *self, +unsigned long event,void *data) {struct con3270 *cp;unsigned long flags; cp = condev;if (!cp->view.dev) -return ; -raw3270_activate_view (&cp->view); -spin_lock_irqsave (&cp->view.lock, flags); +return NOTIFY_DONE ; + +if (!raw3270_view_lock_unavailable (&cp->view)) +raw3270_activate_view (&cp->view); + +if (!spin_trylock_irqsave (&cp->view.lock, flags)) +return NOTIFY_DONE ; +con3270_wait_write (cp); cp->nr_up =0 ;con3270_rebuild_update (cp);
@@ -560,23 +569,18 @@ con3270_flush(void)con3270_wait_write (cp); }spin_unlock_irqrestore (&cp->view.lock, flags); -} -static int con3270_notify (struct notifier_block *self, -unsigned long event,void *data) -{ -con3270_flush (); -return NOTIFY_OK ; +return NOTIFY_DONE ; }static struct notifier_block on_panic_nb = { .notifier_call = con3270_notify, - .priority =0 , + .priority =INT_MIN +1 ,/* run the callback late */ };static struct notifier_block on_reboot_nb = { .notifier_call = con3270_notify, - .priority =0 , + .priority =INT_MIN +1 ,/* run the callback late */ };/*
drivers/s390/char/raw3270.c +18 -0
@@ -830,6 +830,24 @@ raw3270_create_device(struct ccw_device *cdev)return rp; } +/* +* This helper just validates that it is safe to activate a +* view in the panic() context, due to locking restrictions. +*/ +int +raw3270_view_lock_unavailable (struct raw3270_view *view) +{ +struct raw3270 *rp = view->dev; + +if (!rp) +return -ENODEV ; + +if (spin_is_locked (get_ccwdev_lock (rp->cdev))) +return -EBUSY ; + +return 0 ; +} +/* * Activate a view. */
drivers/s390/char/raw3270.h +1 -0
@@ -160,6 +160,7 @@ struct raw3270_view { };int raw3270_add_view (struct raw3270_view *,struct raw3270_fn *,int ,int ); +int raw3270_view_lock_unavailable (struct raw3270_view *view);int raw3270_activate_view (struct raw3270_view *);void raw3270_del_view (struct raw3270_view *);void raw3270_deactivate_view (struct raw3270_view *);
drivers/s390/char/sclp_con.c +16 -12
@@ -220,30 +220,34 @@ sclp_console_device(struct console *c, int *index) }/* -* Make sure that all buffers will be flushed to the SCLP. +* This panic/reboot notifier makes sure that all buffers +* will be flushed to the SCLP. */ -static void -sclp_console_flush (void ) -{ -sclp_conbuf_emit (); -sclp_console_sync_queue (); -} -static int sclp_console_notify (struct notifier_block *self,unsigned long event,void *data) { -sclp_console_flush (); -return NOTIFY_OK ; +/* +* Perform the lock check before effectively getting the +* lock on sclp_conbuf_emit() / sclp_console_sync_queue() +* to prevent potential lockups in atomic context. +*/ +if (spin_is_locked (&sclp_con_lock)) +return NOTIFY_DONE ; + +sclp_conbuf_emit (); +sclp_console_sync_queue (); + +return NOTIFY_DONE ; }static struct notifier_block on_panic_nb = { .notifier_call = sclp_console_notify, - .priority =1 , + .priority =INT_MIN +1 ,/* run the callback late */ };static struct notifier_block on_reboot_nb = { .notifier_call = sclp_console_notify, - .priority =1 , + .priority =INT_MIN +1 ,/* run the callback late */ };/*
drivers/s390/char/sclp_vt220.c +23 -19
66 lines not on this page · raw message
Signature · 1 line
2.36.0
Heiko Carstens
maintainerRe: [PATCH 13/30] s390/consoles: Improve panic notifiers reliability
Quoted 57 lines · Guilherme G. Piccoli
Currently many console drivers for s390 rely on panic/reboot notifiers to invoke callbacks on these events. The
panic()function disables local IRQs, secondary CPUs and preemption, so callbacks invoked on panic are effectively running in atomic context.Happens that most of these console callbacks from s390 doesn't take the proper care with regards to atomic context, like taking spinlocks that might be taken in other function/CPU and hence will cause a lockup situation.
The goal for this patch is to improve the notifiers reliability, acting on 4 console drivers, as detailed below:
(1) con3215: changed a regular spinlock to the trylock alternative.
(2) con3270: also changed a regular spinlock to its trylock counterpart, but here we also have another problem:
raw3270_activate_view()takes a different spinlock. So, we worked a helper to validate if this other lock is safe to acquire, and if so,raw3270_activate_view()should be safe.Notice though that there is a functional change here: it's now possible to continue the notifier code [reaching
con3270_wait_write()andcon3270_rebuild_update()] without executingraw3270_activate_view().(3) sclp: a global lock is used heavily in the functions called from the notifier, so we added a check here - if the lock is taken already, we just bail-out, preventing the lockup.
(4)
sclp_vt220: same as (3), a lock validation was added to prevent the potential lockup problem.Besides (1)-(4), we also removed useless void functions, adding the code called from the notifier inside its own body, and changed the priority of such notifiers to execute late, since they are "heavyweight" for the panic environment, so we aim to reduce risks here. Changed return values to
NOTIFY_DONEas well, the standard one.Cc Alexander Gordeev <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Heiko Carstens <[email protected]>Cc Sven Schnelle <[email protected]>Cc Vasily Gorbik <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>---
As a design choice, the option used here to verify a given spinlock is taken was the function "
spin_is_locked()" - but we noticed that it is not often used. An alternative would to take the lock with aspin_trylock()and if it succeeds, just release the spinlock and continue the code. But that seemed weird...Also, we'd like to ask a good validation of case (2) potential functionality change from the s390 console experts - far from expert here, and in our naive code observation, that seems fine, but that analysis might be missing some corner case.
Thanks in advance!
Quoted 7 lines
drivers/s390/char/con3215.c | 36 +++++++++++++++-------------- drivers/s390/char/con3270.c | 34 +++++++++++++++------------ drivers/s390/char/raw3270.c | 18 +++++++++++++++ drivers/s390/char/raw3270.h | 1 + drivers/s390/char/sclp_con.c | 28 +++++++++++++---------- drivers/s390/char/sclp_vt220.c | 42 +++++++++++++++++++--------------- 6 files changed, 96 insertions(+), 63 deletions(-)
Code looks good, and everything still seems to work. I applied this internally for the time being, and if it passes testing, I'll schedule it for the next merge window.
Thanks!
Re: [PATCH 13/30] s390/consoles: Improve panic notifiers reliability
Quoted 7 lines · Heiko Carstens
[...]
Code looks good, and everything still seems to work. I applied this internally for the time being, and if it passes testing, I'll schedule it for the next merge window.
Thanks!
Perfect Heiko, thanks a bunch for your review and tests! Let me know if anything breaks heh
Cheers,
Guilherme
PATCH 14/30 panic: Properly identify the panic event to the notifiers' callbacks
In reply to Guilherme G. PiccoliThe notifiers infrastructure provides a way to pass an "id" to the callbacks to determine what kind of event happened, i.e., what is the reason behind they getting called.
The panic notifier currently pass 0, but this is soon to be used in a multi-targeted notifier, so let's pass a meaningful "id" over there.
--- include/linux/panic_notifier.h | 5 +++++ kernel/panic.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)
include/linux/panic_notifier.h +5 -0
@@ -9,4 +9,9 @@ extern struct atomic_notifier_head panic_notifier_list;extern bool crash_kexec_post_notifiers; +enum panic_notifier_val { +PANIC_UNUSED , +PANIC_NOTIFIER =0xDEAD , +}; +#endif /* _LINUX_PANIC_NOTIFIERS_H */
kernel/panic.c +1 -1
@@ -287,7 +287,7 @@ void panic(const char *fmt, ...)* Run any panic handlers, including those that might need to * add information to the kmsg dump output. */ -atomic_notifier_call_chain (&panic_notifier_list,0 , buf); +atomic_notifier_call_chain (&panic_notifier_list,PANIC_NOTIFIER , buf);panic_print_sys_info (false);
Signature · 1 line
2.36.0
Petr Mladek
maintainerRe: [PATCH 14/30] panic: Properly identify the panic event to the notifiers' callbacks
Quoted 19 lines · Guilherme G. Piccoli
The notifiers infrastructure provides a way to pass an "id" to the callbacks to determine what kind of event happened, i.e., what is the reason behind they getting called.
The panic notifier currently pass 0, but this is soon to be used in a multi-targeted notifier, so let's pass a meaningful "id" over there.
Signed-off-by Guilherme G. Piccoli <[email protected]>--- include/linux/panic_notifier.h | 5 +++++ kernel/panic.c | 2 +- 2 files changed, 6 insertions(+), 1 deletion(-)include/linux/panic_notifier.h +0
-0
include/linux/panic_notifier.h extern struct atomic_notifier_head panic_notifier_list; · line 15
Quoted 7 lines
include/linux/panic_notifier.h +4
-0@@ -9,4 +9,9 @@ extern struct atomic_notifier_head panic_notifier_list;extern bool crash_kexec_post_notifiers; +enum panic_notifier_val { +PANIC_UNUSED , +PANIC_NOTIFIER =0xDEAD , +};
Hmm, this looks like a hack. PANIC_UNUSED will never be used. All notifiers will be always called with PANIC_NOTIFIER.
The @val parameter is normally used when the same notifier_list is used in different situations.
But you are going to use it when the same notifier is used in more lists. This is normally distinguished by the @nh (atomic_notifier_head) parameter.
IMHO, it is a bad idea. First, it would confuse people because it does not follow the original design of the parameters. Second, the related code must be touched anyway when the notifier is moved into another list so it does not help much.
Or do I miss anything, please?
Best Regards,
Petr
Re: [PATCH 14/30] panic: Properly identify the panic event to the notifiers' callbacks
Quoted 17 lines · Petr Mladek
[...]
Hmm, this looks like a hack.PANIC_UNUSEDwill never be used. All notifiers will be always called withPANIC_NOTIFIER.The @val parameter is normally used when the same
notifier_listis used in different situations.But you are going to use it when the same notifier is used in more lists. This is normally distinguished by the @nh (
atomic_notifier_head) parameter.IMHO, it is a bad idea. First, it would confuse people because it does not follow the original design of the parameters. Second, the related code must be touched anyway when the notifier is moved into another list so it does not help much.
Quoted 4 lines
Or do I miss anything, please?
Best Regards,
Petr
Hi Petr, thanks for the review.
I'm not strong attached to this patch, so we could drop it and refactor the code of next patches to use the @nh as identification - but personally, I feel this parameter could be used to identify the list that called such function, in other words, what is the event that triggered the callback. Some notifiers are even declared with this parameter called "ev", like the event that triggers the notifier.
You mentioned 2 cases:
(a) Same notifier_list used in different situations;
(b) Same *notifier callback* used in different lists;
Mine is case (b), right? Can you show me an example of case (a)? You can see in the following patches (or grep the kernel) that people are using this identification parameter to determine which kind of OOPS trigger the callback to condition the execution of the function to specific cases. IIUIC, this is more or less what I'm doing, but extending the idea for panic notifiers.
Again, as a personal preference, it makes sense to me using id's VS comparing pointers to differentiate events/callers.
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 14/30] panic: Properly identify the panic event to the notifiers' callbacks
Quoted 33 lines · Guilherme G. Piccoli
Quoted 17 lines · Petr Mladek
[...]
Hmm, this looks like a hack.PANIC_UNUSEDwill never be used. All notifiers will be always called withPANIC_NOTIFIER.The @val parameter is normally used when the same
notifier_listis used in different situations.But you are going to use it when the same notifier is used in more lists. This is normally distinguished by the @nh (
atomic_notifier_head) parameter.IMHO, it is a bad idea. First, it would confuse people because it does not follow the original design of the parameters. Second, the related code must be touched anyway when the notifier is moved into another list so it does not help much.
Quoted 4 lines
Or do I miss anything, please?
Best Regards,
PetrHi Petr, thanks for the review.
I'm not strong attached to this patch, so we could drop it and refactor the code of next patches to use the @nh as identification - but personally, I feel this parameter could be used to identify the list that called such function, in other words, what is the event that triggered the callback. Some notifiers are even declared with this parameter called "ev", like the event that triggers the notifier.
Quoted 7 lines
You mentioned 2 cases:
(a) Same
notifier_listused in different situations;(b) Same *notifier callback* used in different lists;
Mine is case (b), right? Can you show me an example of case (a)?
There are many examples of case (a):
+ module_notify_list:
MODULE_STATE_LIVE, /* Normal state. */
MODULE_STATE_COMING, /* Full formed, running module_init. */
MODULE_STATE_GOING, /* Going away. */
MODULE_STATE_UNFORMED, /* Still setting it up. */
+ netdev_chain:
NETDEV_UP = 1, /* For now you can't veto a device up/down */
NETDEV_DOWN,
NETDEV_REBOOT, /* Tell a protocol stack a network interface
detected a hardware crash and restarted
- we can use this eg to kick tcp sessions once done */
NETDEV_CHANGE, /* Notify device state change */
NETDEV_REGISTER,
NETDEV_UNREGISTER,
NETDEV_CHANGEMTU, /* notify after mtu change happened */
NETDEV_CHANGEADDR, /* notify after the address change */
NETDEV_PRE_CHANGEADDR, /* notify before the address change */
NETDEV_GOING_DOWN,
...
+ vt_notifier_list:
#define VT_ALLOCATE 0x0001 /* Console got allocated */
#define VT_DEALLOCATE 0x0002 /* Console will be deallocated */
#define VT_WRITE 0x0003 /* A char got output */
#define VT_UPDATE 0x0004 /* A bigger update occurred */
#define VT_PREWRITE 0x0005 /* A char is about to be written to the console */
+ die_chain:
DIE_OOPS = 1,
DIE_INT3,
DIE_DEBUG,
DIE_PANIC,
DIE_NMI,
DIE_DIE,
DIE_KERNELDEBUG,
...
These all call the same list/chain in different situations. The situation is distinguished by @val.
Quoted 4 lines
You can see in the following patches (or grep the kernel) that people are using this identification parameter to determine which kind of OOPS trigger the callback to condition the execution of the function to specific cases.
Could you please show me some existing code for case (b)? I am not able to find any except in your patches.
Anyway, the solution in 16th patch is bad, definitely. hv_die_panic_notify_crash() uses "val" to disinguish both:
+ "panic_notifier_list" vs "die_chain"
+ die_val when callen via "die_chain"
The API around "die_chain" API is not aware of enum panic_notifier_val and the API using "panic_notifier_list" is not aware of enum die_val. As I said, it is mixing apples and oranges and it is error prone.
Best Regards,
Petr
Re: [PATCH 14/30] panic: Properly identify the panic event to the notifiers' callbacks
Quoted 16 lines · Petr Mladek
[...]
Quoted 7 lines
You mentioned 2 cases:
(a) Same
notifier_listused in different situations;(b) Same *notifier callback* used in different lists;
Mine is case (b), right? Can you show me an example of case (a)?
There are many examples of case (a):
[... snip ...]
These all call the same list/chain in different situations. The situation is distinguished by @val.
Quoted 8 lines
Quoted 4 lines
You can see in the following patches (or grep the kernel) that people are using this identification parameter to determine which kind of OOPS trigger the callback to condition the execution of the function to specific cases.
Could you please show me some existing code for case (b)? I am not able to find any except in your patches.
Hi Petr, thanks for the examples - I agree with you. In the end, seems I'm kind of abusing the API. This id is used to distinguish different situations in which the callback is called, but in the same "realm"/notifier list.
In my case I have different list calling the same callback and (ab-)using the id to make distinction. I can rework the patches using pointer comparison, it's fine =)
So, I'll drop this patch in V2.
Quoted 11 lines
Anyway, the solution in 16th patch is bad, definitely.
hv_die_panic_notify_crash()uses "val" to disinguish both:+ "
panic_notifier_list" vs "die_chain"
+die_valwhen callen via "die_chain"The API around "
die_chain" API is not aware of enumpanic_notifier_valand the API using "panic_notifier_list" is not aware of enumdie_val. As I said, it is mixing apples and oranges and it is error prone.
OK, I'll re-work that patch - there's more there to be changed, that one is complex heheh
Cheers!
PATCH 15/30 bus: brcmstb_gisb: Clean-up panic/die notifiers
In reply to Guilherme G. PiccoliThis patch improves the panic/die notifiers in this driver by making use of a passed "id" instead of comparing pointer address; also, it removes an useless prototype declaration and unnecessary header inclusion.
This is part of a panic notifiers refactor - this notifier in the future will be moved to a new list, that encompass the information notifiers only.
--- drivers/bus/brcmstb_gisb.c | 26 +++++++++++--------------- 1 file changed, 11 insertions(+), 15 deletions(-)
drivers/bus/brcmstb_gisb.c +11 -15
@@ -19,7 +19,6 @@#include <linux/pm.h> #include <linux/kernel.h> #include <linux/kdebug.h> -#include <linux/notifier.h> #ifdef CONFIG_MIPS #include <asm/traps.h>
@@ -347,25 +346,14 @@ static irqreturn_t brcmstb_gisb_bp_handler(int irq, void *dev_id)/* * Dump out gisb errors on die or panic. */ -static int dump_gisb_error (struct notifier_block *self,unsigned long v, -void *p); - -static struct notifier_block gisb_die_notifier = { - .notifier_call = dump_gisb_error, -}; - -static struct notifier_block gisb_panic_notifier = { - .notifier_call = dump_gisb_error, -}; -static int dump_gisb_error (struct notifier_block *self,unsigned long v,void *p) {struct brcmstb_gisb_arb_device *gdev; -const char *reason ="panic" ; +const char *reason ="die" ; -if (self == &gisb_die_notifier) - reason ="die" ; +if (v ==PANIC_NOTIFIER ) + reason ="panic" ;/* iterate over each GISB arb registered handlers */ list_for_each_entry (gdev, &brcmstb_gisb_arb_device_list, next)
@@ -374,6 +362,14 @@ static int dump_gisb_error(struct notifier_block *self, unsigned long v,return NOTIFY_DONE ; } +static struct notifier_block gisb_die_notifier = { + .notifier_call = dump_gisb_error, +}; + +static struct notifier_block gisb_panic_notifier = { + .notifier_call = dump_gisb_error, +}; + static DEVICE_ATTR(gisb_arb_timeout, S_IWUSR | S_IRUGO, gisb_arb_get_timeout, gisb_arb_set_timeout);
Signature · 1 line
2.36.0
Re: [PATCH 15/30] bus: brcmstb_gisb: Clean-up panic/die notifiers
Quoted 5 lines · Guilherme G. Piccoli
This patch improves the panic/die notifiers in this driver by making use of a passed "id" instead of comparing pointer address; also, it removes an useless prototype declaration and unnecessary header inclusion.
Quoted 8 lines
This is part of a panic notifiers refactor - this notifier in the future will be moved to a new list, that encompass the information notifiers only.
Fixes 9eb60880d9a9 ("bus: brcmstb_gisb: add notifier handling")Cc Brian Norris <[email protected]>Cc Florian Fainelli <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Not sure if the Fixes tag is warranted however as this is a clean up, and not really fixing a bug.
Signature · 1 line
Florian
Re: [PATCH 15/30] bus: brcmstb_gisb: Clean-up panic/die notifiers
Quoted 6 lines · Florian Fainelli
[...]
Acked-by Florian Fainelli <[email protected]>Not sure if the Fixes tag is warranted however as this is a clean up, and not really fixing a bug.
Perfect, thanks Florian. I'll add your ACK and remove the fixes tag in V2. Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 15/30] bus: brcmstb_gisb: Clean-up panic/die notifiers
In reply to Guilherme G. PiccoliQuoted 27 lines · Guilherme G. Piccoli
This patch improves the panic/die notifiers in this driver by making use of a passed "id" instead of comparing pointer address; also, it removes an useless prototype declaration and unnecessary header inclusion.
This is part of a panic notifiers refactor - this notifier in the future will be moved to a new list, that encompass the information notifiers only.
drivers/bus/brcmstb_gisb.c +0
-11@@ -347,25 +346,14 @@ static irqreturn_t brcmstb_gisb_bp_handler(int irq, void *dev_id)/* * Dump out gisb errors on die or panic. */ -static int dump_gisb_error (struct notifier_block *self,unsigned long v, -void *p); - -static struct notifier_block gisb_die_notifier = { - .notifier_call = dump_gisb_error, -}; - -static struct notifier_block gisb_panic_notifier = { - .notifier_call = dump_gisb_error, -}; - static int dump_gisb_error(struct notifier_block *self, unsigned long v,
drivers/bus/brcmstb_gisb.c static irqreturn_t brcmstb_gisb_bp_handler(int irq, void *dev_id) · line 356
Quoted 10 lines
drivers/bus/brcmstb_gisb.c +3
-3void *p) {struct brcmstb_gisb_arb_device *gdev; -const char *reason ="panic" ; +const char *reason ="die" ; -if (self == &gisb_die_notifier) - reason ="die" ; +if (v ==PANIC_NOTIFIER ) + reason ="panic" ;
IMHO, the check of the @self parameter was the proper solution.
"gisb_die_notifier" list uses @val from enum die_val. "gisb_panic_notifier" list uses @val from enum panic_notifier_val.
These are unrelated types. It might easily break when someone defines the same constant also in enum die_val.
Best Regards,
Petr
Re: [PATCH 15/30] bus: brcmstb_gisb: Clean-up panic/die notifiers
Quoted 11 lines · Petr Mladek
[...]
IMHO, the check of the @self parameter was the proper solution."
gisb_die_notifier" list uses @val from enumdie_val. "gisb_panic_notifier" list uses @val from enumpanic_notifier_val.These are unrelated types. It might easily break when someone defines the same constant also in enum
die_val.Best Regards,
Petr
OK Petr, I'll drop this idea for V2 - will just remove the useless header / prototype then. (CC Florian)
Cheers,
Guilherme
PATCH 16/30 drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers
In reply to Guilherme G. PiccoliCurrently Hyper-V guests are among the most relevant users of the panic infrastructure, like panic notifiers, kmsg dumpers, etc. The reasons rely both in cleaning-up procedures (closing a hypervisor <-> guest connection, disabling a paravirtualized timer) as well as to data collection (sending panic information to the hypervisor) and framebuffer management.
The thing is: some notifiers are related to others, ordering matters, some functionalities are duplicated and there are lots of conditionals behind sending panic information to the hypervisor. This patch, as part of an effort to clean-up the panic notifiers mechanism and better document things, address some of the issues/complexities of Hyper-V panic handling through the following changes:
(a) We have die and panic notifiers on vmbus_drv.c and both have goals of sending panic information to the hypervisor, though the panic notifier is also responsible for a cleaning-up procedure.
This commit clears the code by splitting the panic notifier in two, one for closing the vmbus connection whereas the other is only for sending panic info to hypervisor. With that, it was possible to merge the die and panic notifiers in a single/well-documented function, and clear some conditional complexities on sending such information to the hypervisor.
(b) The new panic notifier created after (a) is only doing a single thing:
cleaning the vmbus connection. This procedure might cause a delay (due to hypervisor I/O completion), so we postpone that to run late. But more
relevant: this *same* vmbus unloading happens in the crash_shutdown() handler, so if kdump is set, we can safely skip this panic notifier and defer such clean-up to the kexec crash handler.
(c) There is also a Hyper-V framebuffer panic notifier, which relies in doing a vmbus operation that demands a valid connection. So, we must order this notifier with the panic notifier from vmbus_drv.c, in order to guarantee that the framebuffer code executes before the vmbus connection is unloaded.
Also, this commit removes a useless header.
Although there is code rework and re-ordering, we expect that this change has no functional regressions but instead optimize the path and increase panic reliability on Hyper-V. This was tested on Hyper-V with success.
---
Special thanks to Michael Kelley for the good information about the Hyper-V panic path in email threads some months ago, and to Fabio for the testing performed.
Michael and all Microsoft folks: a careful analysis to double-check our changes and assumptions here is really appreciated, this code is complex and intricate, it is possible some corner case might have been overlooked.
Thanks in advance!
drivers/hv/vmbus_drv.c | 109 ++++++++++++++++++++------------ drivers/video/fbdev/hyperv_fb.c | 8 +++ 2 files changed, 76 insertions(+), 41 deletions(-)
drivers/hv/vmbus_drv.c +68 -41
@@ -24,11 +24,11 @@#include <linux/sched/task_stack.h> #include <linux/delay.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/ptrace.h> #include <linux/screen_info.h> #include <linux/kdebug.h> +#include <linux/kexec.h> #include <linux/efi.h> #include <linux/random.h> #include <linux/kernel.h>
@@ -68,51 +68,75 @@ static int hyperv_report_reg(void)return !sysctl_record_panic_msg || !hv_panic_page; } -static int hyperv_panic_event (struct notifier_block *nb,unsigned long val, +/* +* The panic notifier below is responsible solely for unloading the +* vmbus connection, which is necessary in a panic event. But notice +* that this same unloading procedure is executed in the Hyper-V +* crash_shutdown() handler [see hv_crash_handler()], which basically +* means that we can postpone its execution if we have kdump set, +* since it will run the crash_shutdown() handler anyway. Even more +* intrincated is the relation of this notifier with Hyper-V framebuffer +* panic notifier - we need vmbus connection alive there in order to +* succeed, so we need to order both with each other [for reference see +* hvfb_on_panic()] - this is done using notifiers' priorities. +*/ +static int hv_panic_vmbus_unload (struct notifier_block *nb,unsigned long val,void *args) +{ +if (!kexec_crash_loaded ()) +vmbus_initiate_unload (true); + +return NOTIFY_DONE ; +} +static struct notifier_block hyperv_panic_vmbus_unload_block = { + .notifier_call = hv_panic_vmbus_unload, + .priority =INT_MIN +1 ,/* almost the latest one to execute */ +}; + +/* +* The following callback works both as die and panic notifier; its +* goal is to provide panic information to the hypervisor unless the +* kmsg dumper is gonna be used [see hv_kmsg_dump()], which provides +* more information but is not always available. +* +* Notice that both the panic/die report notifiers are registered only +* if we have the capability HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE set. +*/ +static int hv_die_panic_notify_crash (struct notifier_block *nb, +unsigned long val,void *args) {struct pt_regs *regs; +bool is_die; -vmbus_initiate_unload (true); - -/* -* Hyper-V should be notified only once about a panic. If we will be -* doing hv_kmsg_dump() with kmsg data later, don't do the notification -* here. -*/ -if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE - &&hyperv_report_reg ()) { +/* Don't notify Hyper-V unless we have a die oops event or panic. */ +switch (val) { +case DIE_OOPS : + is_die = true; + regs = ((struct die_args *)args)->regs; +break ; +case PANIC_NOTIFIER : + is_die = false; regs =current_pt_regs (); -hyperv_report_panic (regs, val, false); - } -return NOTIFY_DONE ; -} - -static int hyperv_die_event (struct notifier_block *nb,unsigned long val, -void *args) -{ -struct die_args *die = args; -struct pt_regs *regs = die->regs; - -/* Don't notify Hyper-V if the die event is other than oops */ -if (val !=DIE_OOPS ) +break ; +default :return NOTIFY_DONE ; + }/* -* Hyper-V should be notified only once about a panic. If we will be -* doing hv_kmsg_dump() with kmsg data later, don't do the notification -* here. +* Hyper-V should be notified only once about a panic/die. If we will +* be calling hv_kmsg_dump() later with kmsg data, don't do the +* notification here. */ if (hyperv_report_reg ()) -hyperv_report_panic (regs, val, true); +hyperv_report_panic (regs, val, is_die); +return NOTIFY_DONE ; } - -static struct notifier_block hyperv_die_block = { - .notifier_call = hyperv_die_event, +static struct notifier_block hyperv_die_report_block = { + .notifier_call = hv_die_panic_notify_crash, }; -static struct notifier_block hyperv_panic_block = { - .notifier_call = hyperv_panic_event, +static struct notifier_block hyperv_panic_report_block = { + .notifier_call = hv_die_panic_notify_crash, };static const char *fb_mmio_name ="fb_range" ;
@@ -1589,16 +1613,17 @@ static int vmbus_bus_init(void)if (hyperv_crash_ctl &HV_CRASH_CTL_CRASH_NOTIFY_MSG )hv_kmsg_dump_register (); -register_die_notifier (&hyperv_die_block); +register_die_notifier (&hyperv_die_report_block); +atomic_notifier_chain_register (&panic_notifier_list, + &hyperv_panic_report_block); }/* -* Always register the panic notifier because we need to unload -* the VMbus channel connection to prevent any VMbus -* activity after the VM panics. +* Always register the vmbus unload panic notifier because we +* need to shut the VMbus channel connection on panic. */ atomic_notifier_chain_register (&panic_notifier_list, - &hyperv_panic_block); + &hyperv_panic_vmbus_unload_block);vmbus_request_offers ();
@@ -2817,15 +2842,17 @@ static void __exit vmbus_exit(void)if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) {kmsg_dump_unregister (&hv_kmsg_dumper); -unregister_die_notifier (&hyperv_die_block); +unregister_die_notifier (&hyperv_die_report_block); +atomic_notifier_chain_unregister (&panic_notifier_list, + &hyperv_panic_report_block); }/* -* The panic notifier is always registered, hence we should +* The vmbus panic notifier is always registered, hence we should * also unconditionally unregister it here as well. */ atomic_notifier_chain_unregister (&panic_notifier_list, - &hyperv_panic_block); + &hyperv_panic_vmbus_unload_block);free_page ((unsigned long )hv_panic_page);unregister_sysctl_table (hv_ctl_table_hdr);
drivers/video/fbdev/hyperv_fb.c +8 -0
@@ -1244,7 +1244,15 @@ static int hvfb_probe(struct hv_device *hdev, par->fb_ready = true; par->synchronous_fb = false; + +/* +* We need to be sure this panic notifier runs _before_ the +* vmbus disconnect, so order it by priority. It must execute +* before the function hv_panic_vmbus_unload() [drivers/hv/vmbus_drv.c], +* which is almost at the end of list, with priority = INT_MIN + 1. +*/ par->hvfb_panic_nb.notifier_call = hvfb_on_panic; + par->hvfb_panic_nb.priority =INT_MIN +10 ,atomic_notifier_chain_register (&panic_notifier_list, &par->hvfb_panic_nb);
Signature · 1 line
2.36.0
RE: [PATCH 16/30] drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
Quoted 24 lines
Currently Hyper-V guests are among the most relevant users of the panic infrastructure, like panic notifiers, kmsg dumpers, etc. The reasons rely both in cleaning-up procedures (closing a hypervisor <-> guest connection, disabling a paravirtualized timer) as well as to data collection (sending panic information to the hypervisor) and framebuffer management.
The thing is: some notifiers are related to others, ordering matters, some functionalities are duplicated and there are lots of conditionals behind sending panic information to the hypervisor. This patch, as part of an effort to clean-up the panic notifiers mechanism and better document things, address some of the issues/complexities of Hyper-V panic handling through the following changes:
(a) We have die and panic notifiers on
vmbus_drv.cand both have goals of sending panic information to the hypervisor, though the panic notifier is also responsible for a cleaning-up procedure.This commit clears the code by splitting the panic notifier in two, one for closing the vmbus connection whereas the other is only for sending panic info to hypervisor. With that, it was possible to merge the die and panic notifiers in a single/well-documented function, and clear some conditional complexities on sending such information to the hypervisor.
Quoted 6 lines
(b) The new panic notifier created after (a) is only doing a single thing:
cleaning the vmbus connection. This procedure might cause a delay (due to hypervisor I/O completion), so we postpone that to run late. But more
relevant: this *same* vmbus unloading happens in thecrash_shutdown()handler, so if kdump is set, we can safely skip this panic notifier and defer such clean-up to the kexec crash handler.
While the last sentence is true for Hyper-V on x86/x64, it's not true for Hyper-V on ARM64. x86/x64 has the 'machine_ops' data structure with the ability to provide a custom crash_shutdown() function, which Hyper-V does in the form of hv_machine_crash_shutdown(). But ARM64 has no mechanism to provide such a custom function that will eventually do the needed vmbus_initiate_unload() before running kdump.
I'm not immediately sure what the best solution is for ARM64. At this point, I'm just pointing out the problem and will think about the tradeoffs for various possible solutions. Please do the same yourself. :-)
Quoted 6 lines
(c) There is also a Hyper-V framebuffer panic notifier, which relies in doing a vmbus operation that demands a valid connection. So, we must order this notifier with the panic notifier from
vmbus_drv.c, in order to guarantee that the framebuffer code executes before the vmbus connection is unloaded.
Patch 21 of this set puts the Hyper-V FB panic notifier on the pre_reboot notifier list, which means it won't execute before the VMbus connection
unload in the case of kdump. This notifier is making sure that Hyper-V
is notified about the last updates made to the frame buffer before the panic, so maybe it needs to be put on the hypervisor notifier list. It sends a message to Hyper-V over its existing VMbus channel, but it does not wait for a reply. It does, however, obtain a spin lock on the
ring buffer used to communicate with Hyper-V. Unless someone has
a better suggestion, I'm inclined to take the risk of blocking on that spin lock.
Quoted 9 lines
Also, this commit removes a useless header.
Although there is code rework and re-ordering, we expect that this change has no functional regressions but instead optimize the path and increase panic reliability on Hyper-V. This was tested on Hyper-V with success.
Fixes 792f232d57ff ("Drivers: hv: vmbus: Fix potential crash on module unload")Fixes 74347a99e73a ("x86/Hyper-V: Unload vmbus channel in hv panic callback")
The "Fixes:" tags imply that these changes should be backported to older longterm kernel versions, which I don't think is the case. There is a dependency on Patch 14 of your series where PANIC_NOTIFIER is introduced.
Quoted 46 lines
Cc Andrea Parri (Microsoft) <[email protected]>Cc Dexuan Cui <[email protected]>Cc Haiyang Zhang <[email protected]>Cc "K. Y. Srinivasan" <[email protected]>Cc Michael Kelley <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Tianyu Lan <[email protected]>Cc Wei Liu <[email protected]>Tested-by Fabio A M Martins <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>---
Special thanks to Michael Kelley for the good information about the Hyper-V panic path in email threads some months ago, and to Fabio for the testing performed.
Michael and all Microsoft folks: a careful analysis to double-check our changes and assumptions here is really appreciated, this code is complex and intricate, it is possible some corner case might have been overlooked.
Thanks in advance!
drivers/hv/vmbus_drv.c | 109 ++++++++++++++++++++------------ drivers/video/fbdev/hyperv_fb.c | 8 +++ 2 files changed, 76 insertions(+), 41 deletions(-)drivers/hv/vmbus_drv.c +1
-1@@ -24,11 +24,11 @@#include <linux/sched/task_stack.h> #include <linux/delay.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/ptrace.h> #include <linux/screen_info.h> #include <linux/kdebug.h> +#include <linux/kexec.h> #include <linux/efi.h> #include <linux/random.h> #include <linux/kernel.h> @@ -68,51 +68,75 @@ static int hyperv_report_reg(void)return !sysctl_record_panic_msg || !hv_panic_page; }
drivers/hv/vmbus_drv.c static int hyperv_report_reg(void) · line 78
Quoted 10 lines
drivers/hv/vmbus_drv.c +8
-1-static int hyperv_panic_event(struct notifier_block *nb, unsigned long val, +/* +* The panic notifier below is responsible solely for unloading the +* vmbus connection, which is necessary in a panic event. But notice +* that this same unloading procedure is executed in the Hyper-V +* crash_shutdown() handler [see hv_crash_handler()], which basically +* means that we can postpone its execution if we have kdump set, +* since it will run the crash_shutdown() handler anyway. Even more +* intrincated is the relation of this notifier with Hyper-V framebuffer
s/intrincated/intricate/
drivers/hv/vmbus_drv.c static int hyperv_report_reg(void) · line 86
Quoted 8 lines
drivers/hv/vmbus_drv.c +7
-0+* panic notifier - we need vmbus connection alive there in order to +* succeed, so we need to order both with each other [for reference see +* hvfb_on_panic()] - this is done using notifiers' priorities. +*/ +static int hv_panic_vmbus_unload (struct notifier_block *nb,unsigned long val,void *args) +{ +if (!kexec_crash_loaded ())
I'm not clear on the purpose of this condition. I think it means we will skip the vmbus_initiate_unload() if a panic occurs in the kdump kernel. Is there a reason a panic in the kdump kernel should be treated differently? Or am I misunderstanding?
Quoted 153 lines
drivers/hv/vmbus_drv.c +52
-39+vmbus_initiate_unload (true); + +return NOTIFY_DONE ; +} +static struct notifier_block hyperv_panic_vmbus_unload_block = { + .notifier_call = hv_panic_vmbus_unload, + .priority =INT_MIN +1 ,/* almost the latest one to execute */ +}; + +/* +* The following callback works both as die and panic notifier; its +* goal is to provide panic information to the hypervisor unless the +* kmsg dumper is gonna be used [see hv_kmsg_dump()], which provides +* more information but is not always available. +* +* Notice that both the panic/die report notifiers are registered only +* if we have the capability HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE set. +*/ +static int hv_die_panic_notify_crash (struct notifier_block *nb, +unsigned long val,void *args) {struct pt_regs *regs; +bool is_die; -vmbus_initiate_unload (true); - -/* -* Hyper-V should be notified only once about a panic. If we will be -* doing hv_kmsg_dump() with kmsg data later, don't do the notification -* here. -*/ -if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE - &&hyperv_report_reg ()) { +/* Don't notify Hyper-V unless we have a die oops event or panic. */ +switch (val) { +case DIE_OOPS : + is_die = true; + regs = ((struct die_args *)args)->regs; +break ; +case PANIC_NOTIFIER : + is_die = false; regs =current_pt_regs (); -hyperv_report_panic (regs, val, false); - } -return NOTIFY_DONE ; -} - -static int hyperv_die_event (struct notifier_block *nb,unsigned long val, -void *args) -{ -struct die_args *die = args; -struct pt_regs *regs = die->regs; - -/* Don't notify Hyper-V if the die event is other than oops */ -if (val !=DIE_OOPS ) +break ; +default :return NOTIFY_DONE ; + }/* -* Hyper-V should be notified only once about a panic. If we will be -* doing hv_kmsg_dump() with kmsg data later, don't do the notification -* here. +* Hyper-V should be notified only once about a panic/die. If we will +* be calling hv_kmsg_dump() later with kmsg data, don't do the +* notification here. */ if (hyperv_report_reg ()) -hyperv_report_panic (regs, val, true); +hyperv_report_panic (regs, val, is_die); +return NOTIFY_DONE ; } - -static struct notifier_block hyperv_die_block = { - .notifier_call = hyperv_die_event, +static struct notifier_block hyperv_die_report_block = { + .notifier_call = hv_die_panic_notify_crash, }; -static struct notifier_block hyperv_panic_block = { - .notifier_call = hyperv_panic_event, +static struct notifier_block hyperv_panic_report_block = { + .notifier_call = hv_die_panic_notify_crash, };static const char *fb_mmio_name ="fb_range" ;@@ -1589,16 +1613,17 @@ static int vmbus_bus_init(void)if (hyperv_crash_ctl &HV_CRASH_CTL_CRASH_NOTIFY_MSG )hv_kmsg_dump_register (); -register_die_notifier (&hyperv_die_block); +register_die_notifier (&hyperv_die_report_block); +atomic_notifier_chain_register (&panic_notifier_list, + &hyperv_panic_report_block); }/* -* Always register the panic notifier because we need to unload -* the VMbus channel connection to prevent any VMbus -* activity after the VM panics. +* Always register the vmbus unload panic notifier because we +* need to shut the VMbus channel connection on panic. */ atomic_notifier_chain_register (&panic_notifier_list, - &hyperv_panic_block); + &hyperv_panic_vmbus_unload_block);vmbus_request_offers ();@@ -2817,15 +2842,17 @@ static void __exit vmbus_exit(void)if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) {kmsg_dump_unregister (&hv_kmsg_dumper); -unregister_die_notifier (&hyperv_die_block); +unregister_die_notifier (&hyperv_die_report_block); +atomic_notifier_chain_unregister (&panic_notifier_list, + &hyperv_panic_report_block); }/* -* The panic notifier is always registered, hence we should +* The vmbus panic notifier is always registered, hence we should * also unconditionally unregister it here as well. */ atomic_notifier_chain_unregister (&panic_notifier_list, - &hyperv_panic_block); + &hyperv_panic_vmbus_unload_block);free_page ((unsigned long )hv_panic_page);unregister_sysctl_table (hv_ctl_table_hdr);drivers/video/fbdev/hyperv_fb.c +8
-1@@ -1244,7 +1244,15 @@ static int hvfb_probe(struct hv_device *hdev, par->fb_ready = true; par->synchronous_fb = false; + +/* +* We need to be sure this panic notifier runs _before_ the +* vmbus disconnect, so order it by priority. It must execute +* before the function hv_panic_vmbus_unload() [drivers/hv/vmbus_drv.c], +* which is almost at the end of list, with priority = INT_MIN + 1. +*/ par->hvfb_panic_nb.notifier_call = hvfb_on_panic; + par->hvfb_panic_nb.priority =INT_MIN +10 ,atomic_notifier_chain_register (&panic_notifier_list, &par->hvfb_panic_nb); --2.36.0
Re: [PATCH 16/30] drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers
Hi Michael, first of all thanks for the great review, much appreciated. Some comments inline below:
Quoted 13 lines · Michael Kelley (LINUX)
[...]
Quoted 4 lines
hypervisor I/O completion), so we postpone that to run late. But more
relevant: this *same* vmbus unloading happens in thecrash_shutdown()handler, so if kdump is set, we can safely skip this panic notifier and defer such clean-up to the kexec crash handler.While the last sentence is true for Hyper-V on x86/x64, it's not true for Hyper-V on ARM64. x86/x64 has the '
machine_ops' data structure with the ability to provide a customcrash_shutdown()function, which Hyper-V does in the form ofhv_machine_crash_shutdown(). But ARM64 has no mechanism to provide such a custom function that will eventually do the neededvmbus_initiate_unload()before running kdump.
Quoted 4 lines
I'm not immediately sure what the best solution is for ARM64. At this point, I'm just pointing out the problem and will think about the tradeoffs for various possible solutions. Please do the same yourself. :-)
Oh, you're totally right! I just assumed ARM64 would the the same, my bad. Just to propose some alternatives, so you/others can also discuss here and we can reach a consensus about the trade-offs:
(a) We could forget about this change, and always do the clean-up here, not relying in machine_crash_shutdown().
Pro: really simple, behaves the same as it is doing currently.
Con: less elegant/concise, doesn't allow arm64 customization.
(b) Add a way to allow ARM64 customization of shutdown crash handler.
Pro: matches x86, more customizable, improves arm64 arch code.
Con: A tad more complex.
Also, a question that came-up: if ARM64 has no way of calling special crash shutdown handler, how can you execute hv_stimer_cleanup() and hv_synic_disable_regs() there? Or are they not required in ARM64?
Quoted 7 lines
Quoted 6 lines
(c) There is also a Hyper-V framebuffer panic notifier, which relies in doing a vmbus operation that demands a valid connection. So, we must order this notifier with the panic notifier from
vmbus_drv.c, in order to guarantee that the framebuffer code executes before the vmbus connection is unloaded.
Quoted 10 lines
Patch 21 of this set puts the Hyper-V FB panic notifier on the
pre_rebootnotifier list, which means it won't execute before the VMbus connection
unload in the case of kdump. This notifier is making sure that Hyper-V
is notified about the last updates made to the frame buffer before the panic, so maybe it needs to be put on the hypervisor notifier list. It sends a message to Hyper-V over its existing VMbus channel, but it does not wait for a reply. It does, however, obtain a spin lock on the
ring buffer used to communicate with Hyper-V. Unless someone has
a better suggestion, I'm inclined to take the risk of blocking on that spin lock.
The logic behind that was: when kdump is set, we'd skip the vmbus disconnect on notifiers, deferring that to crash_shutdown(), logic this one refuted in the above discussion on ARM64 (one more Pro argument to the idea of refactoring aarch64 code to allow a custom crash shutdown handler heh). But you're right, for the default level 2, we skip the pre_reboot notifiers on kdump, effectively skipping this notifier.
Some ideas of what we can do here:
I) we could change the framebuffer notifier to rely on trylocks, instead of risking a lockup scenario, and with that, we can execute it before the vmbus disconnect in the hypervisor list;
II) we ignore the hypervisor notifier in case of kdump _by default_, and if the users don't want that, they can always set the panic notifier level to 4 and run all notifiers prior to kdump; would that be terrible you think? Kdump users might don't care about the framebuffer...
III) we go with approach (b) above and refactor arm64 code to allow the custom crash handler on kdump time, then [with point (I) above] the logic proposed in this series is still valid - seems more and more the most correct/complete solution.
In any case, I guess we should avoid workarounds if possible and do the things the best way we can, to encompass all (or almost all) the possible scenarios and don't force things on users (like enforcing panic notifier level 4 for Hyper-V or something like this...)
More feedback from you / Hyper-V folks is pretty welcome about this.
Quoted 7 lines
Quoted 2 lines
[...]
The "Fixes:" tags imply that these changes should be backported to older longterm kernel versions, which I don't think is the case. There is a dependency on Patch 14 of your series where
PANIC_NOTIFIERis introduced.
Oh, this was more related with archeology of the kernel. When I'm investigating stuff, I really want to understand why code was added and that usually require some time git blaming stuff, so having that pronto in the commit message is a bonus.
But of course we don't need to use the Fixes tag for that, easy to only mention it in the text. A secondary benefit by using this tag is to indicate this is a _real fix_ to some code, and not an improvement, but as you say, I agree we shouldn't backport it to previous releases having or not the Fixes tag (AFAIK it's not mandatory to backport stuff with Fixes tag).
Quoted 4 lines
Quoted 2 lines
[...]
+ * intrincated is the relation of this notifier with Hyper-V framebuffers/intrincated/intricate/
Thanks, fixed in V2!
Quoted 11 lines
drivers/hv/vmbus_drv.c static int hyperv_report_reg(void) · line 86
Quoted 6 lines
[...]
drivers/hv/vmbus_drv.c +3
-0+static int hv_panic_vmbus_unload (struct notifier_block *nb,unsigned long val,void *args) +{ +if (!kexec_crash_loaded ())I'm not clear on the purpose of this condition. I think it means we will skip the
vmbus_initiate_unload()if a panic occurs in the kdump kernel. Is there a reason a panic in the kdump kernel should be treated differently? Or am I misunderstanding?
This is really related with the point discussed in the top of this response - I assumed both ARM64/x86_64 would behave the same and disconnect the vmbus through the custom crash handler when kdump is set, so worth skipping it here in the notifier. But that's not true for ARM64 as you pointed, so this guard against kexec is really part of the decision/discussion on what to do with ARM64 heh
Cheers!
RE: [PATCH 16/30] drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers
From: Guilherme G. Piccoli <[email protected]> Sent: Friday, April 29, 2022 3:35 PM
Quoted 32 lines
Hi Michael, first of all thanks for the great review, much appreciated. Some comments inline below:
Quoted 13 lines · Michael Kelley (LINUX)
[...]
Quoted 4 lines
hypervisor I/O completion), so we postpone that to run late. But more
relevant: this *same* vmbus unloading happens in thecrash_shutdown()handler, so if kdump is set, we can safely skip this panic notifier and defer such clean-up to the kexec crash handler.While the last sentence is true for Hyper-V on x86/x64, it's not true for Hyper-V on ARM64. x86/x64 has the '
machine_ops' data structure with the ability to provide a customcrash_shutdown()function, which Hyper-V does in the form ofhv_machine_crash_shutdown(). But ARM64 has no mechanism to provide such a custom function that will eventually do the neededvmbus_initiate_unload()before running kdump.Quoted 4 lines
I'm not immediately sure what the best solution is for ARM64. At this point, I'm just pointing out the problem and will think about the tradeoffs for various possible solutions. Please do the same yourself. :-)
Oh, you're totally right! I just assumed ARM64 would the the same, my bad. Just to propose some alternatives, so you/others can also discuss here and we can reach a consensus about the trade-offs:
(a) We could forget about this change, and always do the clean-up here, not relying in
machine_crash_shutdown().
Pro: really simple, behaves the same as it is doing currently.
Con: less elegant/concise, doesn't allow arm64 customization.
Quoted 8 lines
(b) Add a way to allow ARM64 customization of shutdown crash handler.
Pro: matches x86, more customizable, improves arm64 arch code.
Con: A tad more complex.Also, a question that came-up: if ARM64 has no way of calling special crash shutdown handler, how can you execute
hv_stimer_cleanup()andhv_synic_disable_regs()there? Or are they not required in ARM64?
My suggestion is to do (a) for now. I suspect (b) could be a more extended discussion and I wouldn't want your patch set to get held up on that discussion. I don't know what the sense of the ARM64 maintainers would be toward (b). They have tried to avoid picking up code warts like have accumulated on the x86/x64 side over the years, and I agree with that effort. But as more and varied hypervisors become available for ARM64, it seems like a framework for supporting a custom shutdown handler may become necessary. But that could take a little time.
You are right about hv_stimer_cleanup() and hv_synic_disable_regs(). We are not running these when a panic occurs on ARM64, and we
should be, though the risk is small. We will pursue (b) and add
these additional cleanups as part of that. But again, I would suggest doing (a) for now, and we will switch back to your solution once
(b) is in place.
Quoted 26 lines
Quoted 8 lines
Quoted 7 lines
(c) There is also a Hyper-V framebuffer panic notifier, which relies in doing a vmbus operation that demands a valid connection. So, we must order this notifier with the panic notifier from
vmbus_drv.c, in order to guarantee that the framebuffer code executes before the vmbus connection is unloaded.Quoted 10 lines
Patch 21 of this set puts the Hyper-V FB panic notifier on the
pre_rebootnotifier list, which means it won't execute before the VMbus connection
unload in the case of kdump. This notifier is making sure that Hyper-V
is notified about the last updates made to the frame buffer before the panic, so maybe it needs to be put on the hypervisor notifier list. It sends a message to Hyper-V over its existing VMbus channel, but it does not wait for a reply. It does, however, obtain a spin lock on the
ring buffer used to communicate with Hyper-V. Unless someone has
a better suggestion, I'm inclined to take the risk of blocking on that spin lock.The logic behind that was: when kdump is set, we'd skip the vmbus disconnect on notifiers, deferring that to
crash_shutdown(), logic this one refuted in the above discussion on ARM64 (one more Pro argument to the idea of refactoring aarch64 code to allow a custom crash shutdown handler heh). But you're right, for the default level 2, we skip thepre_rebootnotifiers on kdump, effectively skipping this notifier.
Quoted 5 lines
Some ideas of what we can do here:
I) we could change the framebuffer notifier to rely on trylocks, instead of risking a lockup scenario, and with that, we can execute it before the vmbus disconnect in the hypervisor list;
I think we have to do this approach for now.
Quoted 10 lines
II) we ignore the hypervisor notifier in case of kdump
_bydefault_, and if the users don't want that, they can always set the panic notifier level to 4 and run all notifiers prior to kdump; would that be terrible you think? Kdump users might don't care about the framebuffer...III) we go with approach (b) above and refactor arm64 code to allow the custom crash handler on kdump time, then [with point (I) above] the logic proposed in this series is still valid - seems more and more the most correct/complete solution.
But even when/if we get approach (b) implemented, having the framebuffer notifier on the pre_reboot list is still too late with the default of panic_notifier_level = 2. The kdump path will reset the VMbus connection and then the framebuffer notifier won't work.
Quoted 50 lines
In any case, I guess we should avoid workarounds if possible and do the things the best way we can, to encompass all (or almost all) the possible scenarios and don't force things on users (like enforcing panic notifier level 4 for Hyper-V or something like this...)
More feedback from you / Hyper-V folks is pretty welcome about this.
Quoted 7 lines
Quoted 2 lines
[...]
The "Fixes:" tags imply that these changes should be backported to older longterm kernel versions, which I don't think is the case. There is a dependency on Patch 14 of your series where
PANIC_NOTIFIERis introduced.Oh, this was more related with archeology of the kernel. When I'm investigating stuff, I really want to understand why code was added and that usually require some time git blaming stuff, so having that pronto in the commit message is a bonus.
But of course we don't need to use the Fixes tag for that, easy to only mention it in the text. A secondary benefit by using this tag is to indicate this is a
_realfix_to some code, and not an improvement, but as you say, I agree we shouldn't backport it to previous releases having or not the Fixes tag (AFAIK it's not mandatory to backport stuff with Fixes tag).Quoted 4 lines
Quoted 2 lines
[...]
+ * intrincated is the relation of this notifier with Hyper-V framebuffers/intrincated/intricate/
Thanks, fixed in V2!
Quoted 11 lines
drivers/hv/vmbus_drv.c static int hyperv_report_reg(void) · line 86
Quoted 6 lines
[...]
drivers/hv/vmbus_drv.c +3
-0+static int hv_panic_vmbus_unload (struct notifier_block *nb,unsigned long val,void *args) +{ +if (!kexec_crash_loaded ())I'm not clear on the purpose of this condition. I think it means we will skip the
vmbus_initiate_unload()if a panic occurs in the kdump kernel. Is there a reason a panic in the kdump kernel should be treated differently? Or am I misunderstanding?
Quoted 6 lines
This is really related with the point discussed in the top of this response - I assumed both ARM64/x86_64 would behave the same and disconnect the vmbus through the custom crash handler when kdump is set, so worth skipping it here in the notifier. But that's not true for ARM64 as you pointed, so this guard against kexec is really part of the decision/discussion on what to do with ARM64 heh
But note that vmbus_initiate_unload() already has a guard built-in. If the intent of this test is just as a guard against running twice, then it isn't needed.
Quoted 2 lines
Cheers!
Re: [PATCH 16/30] drivers/hv/vmbus, video/hyperv_fb: Untangle and refactor Hyper-V panic notifiers
Quoted 25 lines · Michael Kelley (LINUX)
[...]
Quoted 5 lines
(a) We could forget about this change, and always do the clean-up here, not relying in
machine_crash_shutdown().
Pro: really simple, behaves the same as it is doing currently.
Con: less elegant/concise, doesn't allow arm64 customization.Quoted 8 lines
(b) Add a way to allow ARM64 customization of shutdown crash handler.
Pro: matches x86, more customizable, improves arm64 arch code.
Con: A tad more complex.Also, a question that came-up: if ARM64 has no way of calling special crash shutdown handler, how can you execute
hv_stimer_cleanup()andhv_synic_disable_regs()there? Or are they not required in ARM64?My suggestion is to do (a) for now. I suspect (b) could be a more extended discussion and I wouldn't want your patch set to get held up on that discussion. I don't know what the sense of the ARM64 maintainers would be toward (b). They have tried to avoid picking up code warts like have accumulated on the x86/x64 side over the years, and I agree with that effort. But as more and varied hypervisors become available for ARM64, it seems like a framework for supporting a custom shutdown handler may become necessary. But that could take a little time.
Quoted 7 lines
You are right about
hv_stimer_cleanup()andhv_synic_disable_regs(). We are not running these when a panic occurs on ARM64, and we
should be, though the risk is small. We will pursue (b) and add
these additional cleanups as part of that. But again, I would suggest doing (a) for now, and we will switch back to your solution once
(b) is in place.
Thanks again Michael, I'll stick with (a) for now. I'll check with ARM64 community about that, and I might even try to implement something in parallel (if you are not already working on that - lemme know please), so we don't get stuck here. As you said, I feel that this is more and more relevant as the number of panic/crash/kexec scenarios tend to increase in ARM64.
Quoted 20 lines
Quoted 6 lines
[...]
Some ideas of what we can do here:I) we could change the framebuffer notifier to rely on trylocks, instead of risking a lockup scenario, and with that, we can execute it before the vmbus disconnect in the hypervisor list;
I think we have to do this approach for now.
Quoted 10 lines
II) we ignore the hypervisor notifier in case of kdump
_bydefault_, and if the users don't want that, they can always set the panic notifier level to 4 and run all notifiers prior to kdump; would that be terrible you think? Kdump users might don't care about the framebuffer...III) we go with approach (b) above and refactor arm64 code to allow the custom crash handler on kdump time, then [with point (I) above] the logic proposed in this series is still valid - seems more and more the most correct/complete solution.
Quoted 5 lines
But even when/if we get approach (b) implemented, having the framebuffer notifier on the
pre_rebootlist is still too late with the default ofpanic_notifier_level= 2. The kdump path will reset the VMbus connection and then the framebuffer notifier won't work.
OK, perfect! I'll work something along these lines in V2, allowing the FB notifier to always run in the hypervisor list before the vmbus unload mechanism.
Quoted 18 lines
Quoted 11 lines
[...]
Quoted 9 lines
drivers/hv/vmbus_drv.c static int hyperv_report_reg(void) · line 86
Quoted 4 lines
drivers/hv/vmbus_drv.c +3
-0+static int hv_panic_vmbus_unload (struct notifier_block *nb,unsigned long val,void *args) +{ +if (!kexec_crash_loaded ())I'm not clear on the purpose of this condition. I think it means we will skip the
vmbus_initiate_unload()if a panic occurs in the kdump kernel. Is there a reason a panic in the kdump kernel should be treated differently? Or am I misunderstanding?Quoted 6 lines
This is really related with the point discussed in the top of this response - I assumed both ARM64/x86_64 would behave the same and disconnect the vmbus through the custom crash handler when kdump is set, so worth skipping it here in the notifier. But that's not true for ARM64 as you pointed, so this guard against kexec is really part of the decision/discussion on what to do with ARM64 heh
Quoted 3 lines
But note that
vmbus_initiate_unload()already has a guard built-in. If the intent of this test is just as a guard against running twice, then it isn't needed.
Since we're going to avoid relying in the custom crash_shutdown(), due to the lack of ARM64 support for now, this check will be removed in V2.
Its purpose was to skip the notifier *proactively* in case kexec is set, given that...once kexec happens, the custom crash_shutdown() would run the same function (wrong assumption for ARM64, my bad).
Postponing that slightly would maybe gain us some time while the hypervisor finish its work, so we'd delay less in the vmbus unload path
- that was the rationale behind this check.
Cheers!
PATCH 17/30 tracing: Improve panic/die notifiers
In reply to Guilherme G. PiccoliCurrently the tracing dump_on_oops feature is implemented through separate notifiers, one for die/oops and the other for panic. With the addition of panic notifier "id", this patch makes use of such "id" to unify both functions.
It also comments the function and changes the priority of the notifier blocks, in order they run early compared to other notifiers, to prevent useless trace data (like the callback names for the other notifiers). Finally, we also removed an unnecessary header inclusion.
--- kernel/trace/trace.c | 57 +++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 25 deletions(-)
kernel/trace/trace.c +32 -25
@@ -19,7 +19,6 @@#include <linux/kallsyms.h> #include <linux/security.h> #include <linux/seq_file.h> -#include <linux/notifier.h> #include <linux/irqflags.h> #include <linux/debugfs.h> #include <linux/tracefs.h>
@@ -9767,38 +9766,46 @@ static __init int tracer_init_tracefs(void)fs_initcall (tracer_init_tracefs); -static int trace_panic_handler (struct notifier_block *this, -unsigned long event,void *unused) +/* +* The idea is to execute the following die/panic callback early, in order +* to avoid showing irrelevant information in the trace (like other panic +* notifier functions); we are the 2nd to run, after hung_task/rcu_stall +* warnings get disabled (to prevent potential log flooding). +*/ +static int trace_die_panic_handler (struct notifier_block *self, +unsigned long ev,void *unused) { -if (ftrace_dump_on_oops) +int do_dump; + +if (!ftrace_dump_on_oops) +return NOTIFY_DONE ; + +switch (ev) { +case DIE_OOPS : + do_dump =1 ; +break ; +case PANIC_NOTIFIER : + do_dump =1 ; +break ; +default : + do_dump =0 ; +break ; + } + +if (do_dump)ftrace_dump (ftrace_dump_on_oops); -return NOTIFY_OK ; + +return NOTIFY_DONE ; }static struct notifier_block trace_panic_notifier = { - .notifier_call = trace_panic_handler, - .next =NULL , - .priority =150 /* priority: INT_MAX >= x >= 0 */ + .notifier_call = trace_die_panic_handler, + .priority =INT_MAX -1 , }; -static int trace_die_handler (struct notifier_block *self, -unsigned long val, -void *data) -{ -switch (val) { -case DIE_OOPS : -if (ftrace_dump_on_oops) -ftrace_dump (ftrace_dump_on_oops); -break ; -default : -break ; - } -return NOTIFY_OK ; -} -static struct notifier_block trace_die_notifier = { - .notifier_call = trace_die_handler, - .priority =200 + .notifier_call = trace_die_panic_handler, + .priority =INT_MAX -1 , };/*
Signature · 1 line
2.36.0
Re: [PATCH 17/30] tracing: Improve panic/die notifiers
Hello!
Quoted 20 lines · Guilherme G. Piccoli
Currently the tracing
dump_on_oopsfeature is implemented through separate notifiers, one for die/oops and the other for panic. With the addition of panic notifier "id", this patch makes use of such "id" to unify both functions.It also comments the function and changes the priority of the notifier blocks, in order they run early compared to other notifiers, to prevent useless trace data (like the callback names for the other notifiers). Finally, we also removed an unnecessary header inclusion.
Signed-off-by Guilherme G. Piccoli <[email protected]>--- kernel/trace/trace.c | 57 +++++++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 25 deletions(-)kernel/trace/trace.c +0
-0
[...]
Quoted 7 lines
kernel/trace/trace.c +1
-2@@ -9767,38 +9766,46 @@ static __init int tracer_init_tracefs(void)fs_initcall (tracer_init_tracefs); -static int trace_panic_handler (struct notifier_block *this, -unsigned long event,void *unused) +/*
kernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9778
Quoted 10 lines
kernel/trace/trace.c +8
-1+* The idea is to execute the following die/panic callback early, in order +* to avoid showing irrelevant information in the trace (like other panic +* notifier functions); we are the 2nd to run, after hung_task/rcu_stall +* warnings get disabled (to prevent potential log flooding). +*/ +static int trace_die_panic_handler (struct notifier_block *self, +unsigned long ev,void *unused) { -if (ftrace_dump_on_oops) +int do_dump;
bool?
kernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9789
Quoted 11 lines
kernel/trace/trace.c +11
-0+ +if (!ftrace_dump_on_oops) +return NOTIFY_DONE ; + +switch (ev) { +case DIE_OOPS : + do_dump =1 ; +break ; +case PANIC_NOTIFIER : + do_dump =1 ; +break ;
Why not:
case DIE_OOPS: case PANIC_NOTIFIER: do_dump = 1; break;
kernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9799
Quoted 11 lines
kernel/trace/trace.c +8
-1+default : + do_dump =0 ; +break ; + } + +if (do_dump)ftrace_dump (ftrace_dump_on_oops); -return NOTIFY_OK ; + +return NOTIFY_DONE ; }
[...]
MBR, Sergey
Re: [PATCH 17/30] tracing: Improve panic/die notifiers
Quoted 6 lines · On Fri, 29 Apr 2022 12:22:44 +0300 Sergei Shtylyov <[email protected]>…
Quoted 6 lines
kernel/trace/trace.c +6
-0+switch (ev) { +case DIE_OOPS : + do_dump =1 ; +break ; +case PANIC_NOTIFIER : + do_dump =1 ;
Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ break;Why not: case DIE_OOPS: case PANIC_NOTIFIER: do_dump = 1; break;
Agreed.
Other than that.
-- Steve
Re: [PATCH 17/30] tracing: Improve panic/die notifiers
Quoted 18 lines · Steven Rostedt
Quoted 6 lines · On Fri, 29 Apr 2022 12:22:44 +0300 Sergei Shtylyov <[email protected]>…
Quoted 6 lines
kernel/trace/trace.c +6
-0+switch (ev) { +case DIE_OOPS : + do_dump =1 ; +break ; +case PANIC_NOTIFIER : + do_dump =1 ;Quoted 8 lines
Review
Quoted 1 line
(quoted diff) +1
-0+ break;Why not: case DIE_OOPS: case PANIC_NOTIFIER: do_dump = 1; break;
Quoted 7 lines
Agreed.
Other than that.
-- Steve
Thanks Sergei and Steven, good idea! I thought about the switch change you propose, but I confess I got a bit confused by the "fallthrough" keyword - do I need to use it?
About the s/int/bool, for sure! Not sure why I didn't use bool at first...heheh
Cheers,
Guilherme
Steven Rostedt
maintainerRe: [PATCH 17/30] tracing: Improve panic/die notifiers
Quoted 3 lines · On Fri, 29 Apr 2022 10:46:35 -0300 "Guilherme G. Piccoli"…
Thanks Sergei and Steven, good idea! I thought about the switch change you propose, but I confess I got a bit confused by the "fallthrough" keyword - do I need to use it?
No. The fallthrough keyword is only needed when there's code between case labels. As it is very common to list multiple cases for the same code path. That is:
case DIE_OOPS:
case PANIC_NOTIFIER:
do_dump = 1;
break;
Does not need a fall through label, as there's no code between the DIE_OOPS and the PANIC_NOTIFIER. But if you had:
case DIE_OOPS: x = true;
case PANIC_NOTIFIER:
do_dump = 1;
break;
Then you do.
-- Steve
Re: [PATCH 17/30] tracing: Improve panic/die notifiers
Quoted 19 lines · Steven Rostedt
[...]
No. The fallthrough keyword is only needed when there's code between case labels. As it is very common to list multiple cases for the same code path. That is:case
DIE_OOPS:(quoted diff) +0
-0case PANIC_NOTIFIER: do_dump = 1; break;Does not need a fall through label, as there's no code between the
DIE_OOPSand thePANIC_NOTIFIER. But if you had:case DIE_OOPS: x = true;(quoted diff) +0
-0case PANIC_NOTIFIER: do_dump = 1; break;
Quoted 3 lines
Then you do.
-- Steve
Thanks a bunch for the clarification, changed that for V2 =)
Petr Mladek
maintainerRe: [PATCH 17/30] tracing: Improve panic/die notifiers
In reply to Guilherme G. PiccoliQuoted 31 lines · Guilherme G. Piccoli
Currently the tracing
dump_on_oopsfeature is implemented through separate notifiers, one for die/oops and the other for panic. With the addition of panic notifier "id", this patch makes use of such "id" to unify both functions.It also comments the function and changes the priority of the notifier blocks, in order they run early compared to other notifiers, to prevent useless trace data (like the callback names for the other notifiers). Finally, we also removed an unnecessary header inclusion.
kernel/trace/trace.c +10
-3@@ -9767,38 +9766,46 @@ static __init int tracer_init_tracefs(void)fs_initcall (tracer_init_tracefs); -static int trace_panic_handler (struct notifier_block *this, -unsigned long event,void *unused) +/* +* The idea is to execute the following die/panic callback early, in order +* to avoid showing irrelevant information in the trace (like other panic +* notifier functions); we are the 2nd to run, after hung_task/rcu_stall +* warnings get disabled (to prevent potential log flooding). +*/ +static int trace_die_panic_handler (struct notifier_block *self, +unsigned long ev,void *unused) { -if (ftrace_dump_on_oops) +int do_dump; +
kernel/trace/trace.c static __init int tracer_init_tracefs(void) · line 9789
Quoted 10 lines
kernel/trace/trace.c +10
-0+if (!ftrace_dump_on_oops) +return NOTIFY_DONE ; + +switch (ev) { +case DIE_OOPS : + do_dump =1 ; +break ; +case PANIC_NOTIFIER : + do_dump =1 ; +break ;
DIE_OOPS and PANIC_NOTIFIER are from different enum. It feels like comparing apples with oranges here.
IMHO, the proper way to unify the two notifiers is a check of the @self parameter. Something like:
static int trace_die_panic_handler(struct notifier_block *self,
unsigned long ev, void *unused)
{
if (self == trace_die_notifier && val != DIE_OOPS)
goto out;
ftrace_dump(ftrace_dump_on_oops);
out:
return NOTIFY_DONE;
}Best Regards,
Petr
Re: [PATCH 17/30] tracing: Improve panic/die notifiers
Quoted 18 lines · Petr Mladek
[...]
DIE_OOPSandPANIC_NOTIFIERare from different enum. It feels like comparing apples with oranges here.IMHO, the proper way to unify the two notifiers is a check of the @self parameter. Something like:
static int trace_die_panic_handler(struct
notifier_block*self,
unsigned long ev, void *unused){ if (self == trace_die_notifier && val != DIE_OOPS) goto out; ftrace_dump(ftrace_dump_on_oops); out: return NOTIFY_DONE; }
Quoted 2 lines
Best Regards,
Petr
OK Petr, thanks - will implement your suggestion in V2 (CC Steven)
Cheers!
PATCH 18/30 notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
In reply to Guilherme G. PiccoliCurrently we have a debug infrastructure in the notifiers file, but it's very simple/limited. This patch extends it by:
(a) Showing all registered/unregistered notifiers' callback names;
(b) Adding a dynamic debug tuning to allow showing called notifiers' function names. Notice that this should be guarded as a tunable since it can flood the kernel log buffer.
---
We have some design decisions that worth discussing here:
(a) First of call, using C99 helps a lot to write clear and concise code, but due to commit 4d94f910e79a ("Kbuild: use -Wdeclaration-after-statement") we have a warning if mixing variable declarations with code. For this patch though, doing that makes the code way clear, so decision was to add the debug code inside brackets whenever this warning pops up. We can change that, but that'll cause more ifdefs in the same function.
(b) In the symbol lookup helper function, we modify the parameter passed but even more, we return it as well! This is unusual and seems unnecessary, but was the strategy taken to allow embedding such function in the pr_debug() call.
Not doing that would likely requiring 3 symbol_name variables to avoid concurrency (registering notifier A while calling notifier B) - we rely in local variables as a serialization mechanism.
We're open for suggestions in case this design is not appropriate; thanks in advance!
kernel/notifier.c | 48 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)
kernel/notifier.c +46 -2
@@ -7,6 +7,22 @@#include <linux/vmalloc.h> #include <linux/reboot.h> +#ifdef CONFIG_DEBUG_NOTIFIERS +#include <linux/kallsyms.h> + +/* +* Helper to get symbol names in case DEBUG_NOTIFIERS is set. +* Return the modified parameter is a strategy used to achieve +* the pr_debug() functionality - with this, function is only +* executed if the dynamic debug tuning is effectively set. +*/ +static inline char *notifier_name (struct notifier_block *nb,char *sym_name) +{ +lookup_symbol_name ((unsigned long )(nb->notifier_call), sym_name); +return sym_name; +} +#endif +/* * Notifier list for kernel code which wants to be called * at shutdown. This is used to stop any idling DMA operations
@@ -34,20 +50,41 @@ static int notifier_chain_register(struct notifier_block **nl, } n->next = *nl;rcu_assign_pointer (*nl, n); + +#ifdef CONFIG_DEBUG_NOTIFIERS + { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: registered %s()\n" , +notifier_name (n, sym_name)); + } +#endif return 0 ; }static int notifier_chain_unregister (struct notifier_block **nl,struct notifier_block *n) { +int ret = -ENOENT ; +while ((*nl) !=NULL ) {if ((*nl) == n) {rcu_assign_pointer (*nl, n->next); -return 0 ; + ret =0 ; +break ; } nl = &((*nl)->next); } -return -ENOENT ; + +#ifdef CONFIG_DEBUG_NOTIFIERS +if (!ret) { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: unregistered %s()\n" , +notifier_name (n, sym_name)); + } +#endif +return ret; }/**
@@ -80,6 +117,13 @@ static int notifier_call_chain(struct notifier_block **nl, nb = next_nb; continue; } + + { +char sym_name[KSYM_NAME_LEN ]; + +pr_debug ("notifiers: calling %s()\n" , +notifier_name (nb, sym_name)); + }#endif ret = nb->notifier_call (nb, val, v);
Signature · 1 line
2.36.0
Re: [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
Quoted 69 lines · Guilherme G. Piccoli
Currently we have a debug infrastructure in the notifiers file, but it's very simple/limited. This patch extends it by:
(a) Showing all registered/unregistered notifiers' callback names;
(b) Adding a dynamic debug tuning to allow showing called notifiers' function names. Notice that this should be guarded as a tunable since it can flood the kernel log buffer.
Cc Arjan van de Ven <[email protected]>Cc Cong Wang <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Cc Valentin Schneider <[email protected]>Cc Xiaoming Ni <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>---
We have some design decisions that worth discussing here:
(a) First of call, using C99 helps a lot to write clear and concise code, but due to commit
4d94f910e79a("Kbuild: use -Wdeclaration-after-statement") we have a warning if mixing variable declarations with code. For this patch though, doing that makes the code way clear, so decision was to add the debug code inside brackets whenever this warning pops up. We can change that, but that'll cause more ifdefs in the same function.(b) In the symbol lookup helper function, we modify the parameter passed but even more, we return it as well! This is unusual and seems unnecessary, but was the strategy taken to allow embedding such function in the
pr_debug()call.Not doing that would likely requiring 3
symbol_namevariables to avoid concurrency (registering notifier A while calling notifier B) - we rely in local variables as a serialization mechanism.We're open for suggestions in case this design is not appropriate; thanks in advance!
kernel/notifier.c | 48 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)kernel/notifier.c +16
-0@@ -7,6 +7,22 @@#include <linux/vmalloc.h> #include <linux/reboot.h> +#ifdef CONFIG_DEBUG_NOTIFIERS +#include <linux/kallsyms.h> + +/* +* Helper to get symbol names in case DEBUG_NOTIFIERS is set. +* Return the modified parameter is a strategy used to achieve +* the pr_debug() functionality - with this, function is only +* executed if the dynamic debug tuning is effectively set. +*/ +static inline char *notifier_name (struct notifier_block *nb,char *sym_name) +{ +lookup_symbol_name ((unsigned long )(nb->notifier_call), sym_name); +return sym_name; +} +#endif +/* * Notifier list for kernel code which wants to be called * at shutdown. This is used to stop any idling DMA operations @@ -34,20 +50,41 @@ static int notifier_chain_register(struct notifier_block **nl, }
kernel/notifier.c static int notifier_chain_register(struct notifier_block **nl, · line 60
Quoted 10 lines
kernel/notifier.c +8
-0n->next = *nl;rcu_assign_pointer (*nl, n); + +#ifdef CONFIG_DEBUG_NOTIFIERS + { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: registered %s()\n" , +notifier_name (n, sym_name)); + }
Duplicate Code.
Is it better to use __func__ and %pS?
pr_info("%s: %pS\n", __func__, n->notifier_call);
Quoted 17 lines
kernel/notifier.c +5
-1+#endif return 0 ; }static int notifier_chain_unregister (struct notifier_block **nl,struct notifier_block *n) { +int ret = -ENOENT ; +while ((*nl) !=NULL ) {if ((*nl) == n) {rcu_assign_pointer (*nl, n->next); -return 0 ; + ret =0 ; +break ; } nl = &((*nl)->next);
kernel/notifier.c static int notifier_chain_register(struct notifier_block **nl, · line 85
Quoted 10 lines
kernel/notifier.c +8
-1} -return -ENOENT ; + +#ifdef CONFIG_DEBUG_NOTIFIERS +if (!ret) { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: unregistered %s()\n" , +notifier_name (n, sym_name)); + }
Duplicate Code.
Is it better to use __func__ and %pS?
pr_info("%s: %pS\n", __func__, n->notifier_call);
kernel/notifier.c static int notifier_call_chain(struct notifier_block **nl, · line 120
Quoted 10 lines
kernel/notifier.c +3
-0+#endif +return ret; }/** @@ -80,6 +117,13 @@ static int notifier_call_chain(struct notifier_block **nl, nb = next_nb;continue ; } +
Is the "#ifdef" missing here?
kernel/notifier.c static int notifier_call_chain(struct notifier_block **nl, · line 125
Quoted 5 lines
kernel/notifier.c +5
-0+ { +char sym_name[KSYM_NAME_LEN ]; + +pr_debug ("notifiers: calling %s()\n" , +notifier_name (nb, sym_name));
Duplicate Code.
Is it better to use __func__ and %pS?
pr_info("%s: %pS\n", __func__, n->notifier_call);
kernel/notifier.c static int notifier_call_chain(struct notifier_block **nl, · line 129
Quoted 5 lines
kernel/notifier.c +1
-0+ }#endif ret = nb->notifier_call (nb, val, v);
Thanks
Xiaoming Ni
Re: [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
Quoted 8 lines · Xiaoming Ni
[...]
Duplicate Code.Is it better to use
__func__and %pS?
pr_info("%s: %pS\n", __func__, n->notifier_call);
This is a great suggestion Xiaoming, much appreciated!
I feel like reinventing the wheel here - with your idea, code was super clear and concise, very nice suggestion!!
The only 2 things that diverge from your idea: I'm using '%ps' (not showing offsets) and also, kept the wording "(un)registered/calling", not using __func__ - I feel it's a bit odd in the output. OK for you?
I'm definitely using your idea in V2 heh
Cheers,
Guilherme
Steven Rostedt
maintainerRe: [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
In reply to Xiaoming NiQuoted 10 lines · On Thu, 28 Apr 2022 09:01:13 +0800 Xiaoming Ni <[email protected]> wrote:
kernel/notifier.c static int notifier_chain_register(struct notifier_block **nl, · line 60
Quoted 7 lines
kernel/notifier.c +7
-0+#ifdef CONFIG_DEBUG_NOTIFIERS + { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: registered %s()\n" , +notifier_name (n, sym_name)); + }Duplicate Code.
Quoted 6 lines
Is it better to use
__func__and %pS?
pr_info("%s: %pS\n", __func__, n->notifier_call);Quoted 1 line
+#endif
Also, don't sprinkle #ifdef in C code. Instead:
if (IS_ENABLED(CONFIG_DEBUG_NOTIFIERS))
pr_info("notifers: regsiter %ps()\n",
n->notifer_call);Or define a print macro at the start of the C file that is a nop if it is not defined, and use the macro.
-- Steve
Re: [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
Quoted 10 lines · Steven Rostedt
[...]
Also, don't sprinkle #ifdef in C code. Instead:if (IS_ENABLED(CONFIG_DEBUG_NOTIFIERS)) pr_info("notifers: regsiter %ps()\n", n->notifer_call);Or define a print macro at the start of the C file that is a nop if it is not defined, and use the macro.
Thanks, I'll go with the IS_ENABLED() idea in V2 - appreciate the hint. Cheers,
Guilherme
RE: [PATCH 18/30] notifier: Show function names on notifier routines if DEBUG_NOTIFIERS is set
In reply to Guilherme G. PiccoliFrom: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
Quoted 16 lines
Currently we have a debug infrastructure in the notifiers file, but it's very simple/limited. This patch extends it by:
(a) Showing all registered/unregistered notifiers' callback names;
(b) Adding a dynamic debug tuning to allow showing called notifiers' function names. Notice that this should be guarded as a tunable since it can flood the kernel log buffer.
Cc Arjan van de Ven <[email protected]>Cc Cong Wang <[email protected]>Cc Sebastian Andrzej Siewior <[email protected]>Cc Valentin Schneider <[email protected]>Cc Xiaoming Ni <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Quoted 5 lines
---
We have some design decisions that worth discussing here:
(a) First of call, using C99 helps a lot to write clear and concise code, but
s/call/all/
Quoted 107 lines
due to commit
4d94f910e79a("Kbuild: use -Wdeclaration-after-statement") we have a warning if mixing variable declarations with code. For this patch though, doing that makes the code way clear, so decision was to add the debug code inside brackets whenever this warning pops up. We can change that, but that'll cause more ifdefs in the same function.(b) In the symbol lookup helper function, we modify the parameter passed but even more, we return it as well! This is unusual and seems unnecessary, but was the strategy taken to allow embedding such function in the
pr_debug()call.Not doing that would likely requiring 3
symbol_namevariables to avoid concurrency (registering notifier A while calling notifier B) - we rely in local variables as a serialization mechanism.We're open for suggestions in case this design is not appropriate; thanks in advance!
kernel/notifier.c | 48 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 46 insertions(+), 2 deletions(-)kernel/notifier.c +46
-3@@ -7,6 +7,22 @@#include <linux/vmalloc.h> #include <linux/reboot.h> +#ifdef CONFIG_DEBUG_NOTIFIERS +#include <linux/kallsyms.h> + +/* +* Helper to get symbol names in case DEBUG_NOTIFIERS is set. +* Return the modified parameter is a strategy used to achieve +* the pr_debug() functionality - with this, function is only +* executed if the dynamic debug tuning is effectively set. +*/ +static inline char *notifier_name (struct notifier_block *nb,char *sym_name) +{ +lookup_symbol_name ((unsigned long )(nb->notifier_call), sym_name); +return sym_name; +} +#endif +/* * Notifier list for kernel code which wants to be called * at shutdown. This is used to stop any idling DMA operations @@ -34,20 +50,41 @@ static int notifier_chain_register(struct notifier_block **nl, } n->next = *nl;rcu_assign_pointer (*nl, n); + +#ifdef CONFIG_DEBUG_NOTIFIERS + { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: registered %s()\n" , +notifier_name (n, sym_name)); + } +#endif return 0 ; }static int notifier_chain_unregister (struct notifier_block **nl,struct notifier_block *n) { +int ret = -ENOENT ; +while ((*nl) !=NULL ) {if ((*nl) == n) {rcu_assign_pointer (*nl, n->next); -return 0 ; + ret =0 ; +break ; } nl = &((*nl)->next); } -return -ENOENT ; + +#ifdef CONFIG_DEBUG_NOTIFIERS +if (!ret) { +char sym_name[KSYM_NAME_LEN ]; + +pr_info ("notifiers: unregistered %s()\n" , +notifier_name (n, sym_name)); + } +#endif +return ret; }/** @@ -80,6 +117,13 @@ static int notifier_call_chain(struct notifier_block **nl, nb = next_nb; continue; } + + { +char sym_name[KSYM_NAME_LEN ]; + +pr_debug ("notifiers: calling %s()\n" , +notifier_name (nb, sym_name)); + }#endif ret = nb->notifier_call (nb, val, v); --2.36.0
PATCH 19/30 panic: Add the panic hypervisor notifier list
In reply to Guilherme G. PiccoliThe goal of this new panic notifier is to allow its users to register callbacks to run very early in the panic path. This aims hypervisor/FW notification mechanisms as well as simple LED functions, and any other simple and safe mechanism that should run early in the panic path; more dangerous callbacks should execute later.
For now, the patch is almost a no-op (although it changes a bit the ordering in which some panic notifiers are executed). In a subsequent patch, the panic path will be refactored, then the panic hypervisor notifiers will effectively run very early in the panic path.
We also defer documenting it all properly in the subsequent refactor patch. While at it, we removed some useless header inclusions and fixed some notifiers return too (by using the standard NOTIFY_DONE).
--- arch/mips/sgi-ip22/ip22-reset.c | 2 +- arch/mips/sgi-ip32/ip32-reset.c | 3 +-- arch/powerpc/kernel/setup-common.c | 2 +- arch/sparc/kernel/sstate.c | 3 +-- drivers/firmware/google/gsmi.c | 4 ++-- drivers/hv/vmbus_drv.c | 4 ++-- drivers/leds/trigger/ledtrig-activity.c | 4 ++-- drivers/leds/trigger/ledtrig-heartbeat.c | 4 ++-- drivers/misc/bcm-vk/bcm_vk_dev.c | 6 +++--- drivers/misc/pvpanic/pvpanic.c | 4 ++-- drivers/power/reset/ltc2952-poweroff.c | 4 ++-- drivers/s390/char/zcore.c | 5 +++-- drivers/soc/bcm/brcmstb/pm/pm-arm.c | 2 +- include/linux/panic_notifier.h | 1 + kernel/panic.c | 4 ++++ 15 files changed, 28 insertions(+), 24 deletions(-)
arch/mips/sgi-ip22/ip22-reset.c +1 -1
@@ -195,7 +195,7 @@ static int __init reboot_setup(void) }timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);return 0 ; }
arch/mips/sgi-ip32/ip32-reset.c +1 -2
@@ -15,7 +15,6 @@#include <linux/panic_notifier.h> #include <linux/sched.h> #include <linux/sched/signal.h> -#include <linux/notifier.h> #include <linux/delay.h> #include <linux/rtc/ds1685.h> #include <linux/interrupt.h>
@@ -145,7 +144,7 @@ static __init int ip32_reboot_setup(void) pm_power_off = ip32_machine_halt;timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);return 0 ; }
arch/powerpc/kernel/setup-common.c +1 -1
@@ -753,7 +753,7 @@ static struct notifier_block ppc_panic_block = {void __init setup_panic (void ) {/* Hard-disables IRQs + deal with FW-assisted dump (fadump) */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &ppc_fadump_block);if (IS_ENABLED (CONFIG_RANDOMIZE_BASE ) &&kaslr_offset () >0 )
arch/sparc/kernel/sstate.c +1 -2
@@ -5,7 +5,6 @@*/ #include <linux/kernel.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> #include <linux/init.h>
@@ -106,7 +105,7 @@ static int __init sstate_init(void)do_set_sstate (HV_SOFT_STATE_TRANSITION , booting_msg); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &sstate_panic_block);register_reboot_notifier (&sstate_reboot_notifier);
drivers/firmware/google/gsmi.c +2 -2
@@ -1034,7 +1034,7 @@ static __init int gsmi_init(void)register_reboot_notifier (&gsmi_reboot_notifier);register_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &gsmi_panic_notifier);printk (KERN_INFO "gsmi version " DRIVER_VERSION " loaded\n" );
@@ -1061,7 +1061,7 @@ static void __exit gsmi_exit(void) {unregister_reboot_notifier (&gsmi_reboot_notifier);unregister_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &gsmi_panic_notifier);#ifdef CONFIG_EFI efivars_unregister (&efivars);
drivers/hv/vmbus_drv.c +2 -2
@@ -1614,7 +1614,7 @@ static int vmbus_bus_init(void)hv_kmsg_dump_register ();register_die_notifier (&hyperv_die_report_block); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &hyperv_panic_report_block); }
@@ -2843,7 +2843,7 @@ static void __exit vmbus_exit(void)if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) {kmsg_dump_unregister (&hv_kmsg_dumper);unregister_die_notifier (&hyperv_die_report_block); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &hyperv_panic_report_block); }
drivers/leds/trigger/ledtrig-activity.c +2 -2
@@ -247,7 +247,7 @@ static int __init activity_init(void)int rc =led_trigger_register (&activity_led_trigger);if (!rc) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &activity_panic_nb);register_reboot_notifier (&activity_reboot_nb); }
@@ -257,7 +257,7 @@ static int __init activity_init(void) static void __exit activity_exit(void) {unregister_reboot_notifier (&activity_reboot_nb); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &activity_panic_nb);led_trigger_unregister (&activity_led_trigger); }
drivers/leds/trigger/ledtrig-heartbeat.c +2 -2
@@ -190,7 +190,7 @@ static int __init heartbeat_trig_init(void)int rc =led_trigger_register (&heartbeat_led_trigger);if (!rc) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &heartbeat_panic_nb);register_reboot_notifier (&heartbeat_reboot_nb); }
@@ -200,7 +200,7 @@ static int __init heartbeat_trig_init(void) static void __exit heartbeat_trig_exit(void) {unregister_reboot_notifier (&heartbeat_reboot_nb); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &heartbeat_panic_nb);led_trigger_unregister (&heartbeat_led_trigger); }
drivers/misc/bcm-vk/bcm_vk_dev.c +3 -3
@@ -1446,7 +1446,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)/* register for panic notifier */ vk->panic_nb.notifier_call = bcm_vk_on_panic; - err =atomic_notifier_chain_register (&panic_notifier_list, + err =atomic_notifier_chain_register (&panic_hypervisor_list, &vk->panic_nb);if (err) {dev_err (dev,"Fail to register panic notifier\n" );
@@ -1486,7 +1486,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)bcm_vk_tty_exit (vk); err_unregister_panic_notifier: -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &vk->panic_nb); err_destroy_workqueue:
@@ -1559,7 +1559,7 @@ static void bcm_vk_remove(struct pci_dev *pdev)usleep_range (BCM_VK_UCODE_BOOT_US ,BCM_VK_UCODE_BOOT_MAX_US );/* unregister panic notifier */ -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &vk->panic_nb);bcm_vk_msg_remove (vk);
drivers/misc/pvpanic/pvpanic.c +2 -2
@@ -101,7 +101,7 @@ static int pvpanic_init(void)INIT_LIST_HEAD (&pvpanic_list);spin_lock_init (&pvpanic_lock); -atomic_notifier_chain_register (&panic_notifier_list, &pvpanic_panic_nb); +atomic_notifier_chain_register (&panic_hypervisor_list, &pvpanic_panic_nb);return 0 ; }
@@ -109,7 +109,7 @@ module_init(pvpanic_init);static void pvpanic_exit (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, &pvpanic_panic_nb); +atomic_notifier_chain_unregister (&panic_hypervisor_list, &pvpanic_panic_nb); }module_exit (pvpanic_exit);
drivers/power/reset/ltc2952-poweroff.c +2 -2
@@ -279,7 +279,7 @@ static int ltc2952_poweroff_probe(struct platform_device *pdev) pm_power_off = ltc2952_poweroff_kill; data->panic_notifier.notifier_call = ltc2952_poweroff_notify_panic; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &data->panic_notifier);dev_info (&pdev->dev,"probe successful\n" );
@@ -293,7 +293,7 @@ static int ltc2952_poweroff_remove(struct platform_device *pdev) pm_power_off =NULL ;hrtimer_cancel (&data->timer_trigger);hrtimer_cancel (&data->timer_wde); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &data->panic_notifier);return 0 ; }
drivers/s390/char/zcore.c +3 -2
@@ -246,7 +246,7 @@ static int zcore_reboot_and_on_panic_handler(struct notifier_block *self,if (hsa_available)release_hsa (); -return NOTIFY_OK ; +return NOTIFY_DONE ; }static struct notifier_block zcore_reboot_notifier = {
@@ -322,7 +322,8 @@ static int __init zcore_init(void) NULL, &zcore_hsa_fops);register_reboot_notifier (&zcore_reboot_notifier); -atomic_notifier_chain_register (&panic_notifier_list, &zcore_on_panic_notifier); +atomic_notifier_chain_register (&panic_hypervisor_list, + &zcore_on_panic_notifier);return 0 ; fail:
drivers/soc/bcm/brcmstb/pm/pm-arm.c +1 -1
@@ -814,7 +814,7 @@ static int brcmstb_pm_probe(struct platform_device *pdev) goto out; } -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &brcmstb_pm_panic_nb); pm_power_off = brcmstb_pm_poweroff;
include/linux/panic_notifier.h +1 -0
@@ -6,6 +6,7 @@#include <linux/types.h> extern struct atomic_notifier_head panic_notifier_list; +extern struct atomic_notifier_head panic_hypervisor_list;extern bool crash_kexec_post_notifiers;
kernel/panic.c +4 -0
@@ -73,6 +73,9 @@ ATOMIC_NOTIFIER_HEAD(panic_notifier_list);EXPORT_SYMBOL (panic_notifier_list); +ATOMIC_NOTIFIER_HEAD (panic_hypervisor_list); +EXPORT_SYMBOL (panic_hypervisor_list); +static long no_blink (int state) {return 0 ;
@@ -287,6 +290,7 @@ void panic(const char *fmt, ...)* Run any panic handlers, including those that might need to * add information to the kmsg dump output. */ +atomic_notifier_call_chain (&panic_hypervisor_list,PANIC_NOTIFIER , buf);atomic_notifier_call_chain (&panic_notifier_list,PANIC_NOTIFIER , buf);panic_print_sys_info (false);
Signature · 1 line
2.36.0
RE: [PATCH 19/30] panic: Add the panic hypervisor notifier list
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
Quoted 72 lines
The goal of this new panic notifier is to allow its users to register callbacks to run very early in the panic path. This aims hypervisor/FW notification mechanisms as well as simple LED functions, and any other simple and safe mechanism that should run early in the panic path; more dangerous callbacks should execute later.
For now, the patch is almost a no-op (although it changes a bit the ordering in which some panic notifiers are executed). In a subsequent patch, the panic path will be refactored, then the panic hypervisor notifiers will effectively run very early in the panic path.
We also defer documenting it all properly in the subsequent refactor patch. While at it, we removed some useless header inclusions and fixed some notifiers return too (by using the standard
NOTIFY_DONE).Cc Alexander Gordeev <[email protected]>Cc Andrea Parri (Microsoft) <[email protected]>Cc Ard Biesheuvel <[email protected]>Cc Benjamin Herrenschmidt <[email protected]>Cc Brian Norris <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Christophe JAILLET <[email protected]>Cc David Gow <[email protected]>Cc "David S. Miller" <[email protected]>Cc Dexuan Cui <[email protected]>Cc Doug Berger <[email protected]>Cc Evan Green <[email protected]>Cc Florian Fainelli <[email protected]>Cc Haiyang Zhang <[email protected]>Cc Hari Bathini <[email protected]>Cc Heiko Carstens <[email protected]>Cc Julius Werner <[email protected]>Cc Justin Chen <[email protected]>Cc "K. Y. Srinivasan" <[email protected]>Cc Lee Jones <[email protected]>Cc Markus Mayer <[email protected]>Cc Michael Ellerman <[email protected]>Cc Michael Kelley <[email protected]>Cc Mihai Carabas <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Paul Mackerras <[email protected]>Cc Pavel Machek <[email protected]>Cc Scott Branden <[email protected]>Cc Sebastian Reichel <[email protected]>Cc Shile Zhang <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Sven Schnelle <[email protected]>Cc Thomas Bogendoerfer <[email protected]>Cc Tianyu Lan <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Wang ShaoBo <[email protected]>Cc Wei Liu <[email protected]>Cc zhenwei pi <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/mips/sgi-ip22/ip22-reset.c | 2 +- arch/mips/sgi-ip32/ip32-reset.c | 3 +-- arch/powerpc/kernel/setup-common.c | 2 +- arch/sparc/kernel/sstate.c | 3 +-- drivers/firmware/google/gsmi.c | 4 ++-- drivers/hv/vmbus_drv.c | 4 ++-- drivers/leds/trigger/ledtrig-activity.c | 4 ++-- drivers/leds/trigger/ledtrig-heartbeat.c | 4 ++-- drivers/misc/bcm-vk/bcm_vk_dev.c | 6 +++--- drivers/misc/pvpanic/pvpanic.c | 4 ++-- drivers/power/reset/ltc2952-poweroff.c | 4 ++-- drivers/s390/char/zcore.c | 5 +++-- drivers/soc/bcm/brcmstb/pm/pm-arm.c | 2 +- include/linux/panic_notifier.h | 1 + kernel/panic.c | 4 ++++ 15 files changed, 28 insertions(+), 24 deletions(-)
[ snip]
Quoted 15 lines
drivers/hv/vmbus_drv.c +1
-1@@ -1614,7 +1614,7 @@ static int vmbus_bus_init(void)hv_kmsg_dump_register ();register_die_notifier (&hyperv_die_report_block); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &hyperv_panic_report_block); }
drivers/hv/vmbus_drv.c static void __exit vmbus_exit(void) · line 2849
Quoted 8 lines
drivers/hv/vmbus_drv.c +1
-1@@ -2843,7 +2843,7 @@ static void __exit vmbus_exit(void)if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) {kmsg_dump_unregister (&hv_kmsg_dumper);unregister_die_notifier (&hyperv_die_report_block); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &hyperv_panic_report_block); }
Using the hypervisor_list here produces a bit of a mismatch. In many cases this notifier will do nothing, and will defer to the kmsg_dump() mechanism
to notify the hypervisor about the panic. Running the kmsg_dump()
mechanism is linked to the info_list, so I'm thinking the Hyper-V panic report notifier should be on the info_list as well. That way the reporting behavior is triggered at the same point in the panic path regardless of which reporting mechanism is used.
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 13 lines · Michael Kelley (LINUX)
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
drivers/hv/vmbus_drv.c static void __exit vmbus_exit(void) · line 2849
Quoted 11 lines
[...]
drivers/hv/vmbus_drv.c +1
-1@@ -2843,7 +2843,7 @@ static void __exit vmbus_exit(void)if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) {kmsg_dump_unregister (&hv_kmsg_dumper);unregister_die_notifier (&hyperv_die_report_block); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &hyperv_panic_report_block); }
Quoted 8 lines
Using the
hypervisor_listhere produces a bit of a mismatch. In many cases this notifier will do nothing, and will defer to thekmsg_dump()mechanism
to notify the hypervisor about the panic. Running thekmsg_dump()
mechanism is linked to theinfo_list, so I'm thinking the Hyper-V panic report notifier should be on theinfo_listas well. That way the reporting behavior is triggered at the same point in the panic path regardless of which reporting mechanism is used.
Hi Michael, thanks for your feedback! I agree that your idea could work, but...there is one downside: imagine the kmsg_dump() approach is not set in some Hyper-V guest, then we would rely in the regular notification mechanism [hv_die_panic_notify_crash()], right?
But...you want then to run this notifier in the informational list, which...won't execute *by default* before kdump if no kmsg_dump() is set. So, this logic is convoluted when you mix it with the default level concept + kdump.
May I suggest something? If possible, take a run with this patch set + DEBUG_NOTIFIER=y, in *both* cases (with and without the kmsg_dump() set). I did that and they run almost at the same time...I've checked the notifiers called, it's like almost nothing runs in-between.
I feel the panic notification mechanism does really fit with a hypervisor list, it's a good match with the nature of the list, which aims at informing the panic notification to the hypervisor/FW. Of course we can modify it if you prefer...but please take into account the kdump case and how it complicates the logic.
Let me know your considerations, in case you can experiment with the patch set as-is.
Cheers,
Guilherme
RE: [PATCH 19/30] panic: Add the panic hypervisor notifier list
From: Guilherme G. Piccoli <[email protected]> Sent: Friday, April 29, 2022 11:04 AM
Quoted 25 lines · Michael Kelley (LINUX)
Quoted 1 line
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022
3:49 PM
Quoted 12 lines
drivers/hv/vmbus_drv.c static void __exit vmbus_exit(void) · line 2849
Quoted 11 lines
[...]
drivers/hv/vmbus_drv.c +1
-1@@ -2843,7 +2843,7 @@ static void __exit vmbus_exit(void)if (ms_hyperv.misc_features &HV_FEATURE_GUEST_CRASH_MSR_AVAILABLE ) {kmsg_dump_unregister (&hv_kmsg_dumper);unregister_die_notifier (&hyperv_die_report_block); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_hypervisor_list, &hyperv_panic_report_block); }Quoted 8 lines
Using the
hypervisor_listhere produces a bit of a mismatch. In many cases this notifier will do nothing, and will defer to thekmsg_dump()mechanism
to notify the hypervisor about the panic. Running thekmsg_dump()
mechanism is linked to theinfo_list, so I'm thinking the Hyper-V panic report notifier should be on theinfo_listas well. That way the reporting behavior is triggered at the same point in the panic path regardless of which reporting mechanism is used.
Quoted 8 lines
Hi Michael, thanks for your feedback! I agree that your idea could work, but...there is one downside: imagine the
kmsg_dump()approach is not set in some Hyper-V guest, then we would rely in the regular notification mechanism [hv_die_panic_notify_crash()], right?
But...you want then to run this notifier in the informational list, which...won't execute *by default* before kdump if nokmsg_dump()is set. So, this logic is convoluted when you mix it with the default level concept + kdump.
Yes, you are right. But to me that speaks as much to the linkage between the informational list and kmsg_dump() being the core problem. But as I described in my reply to Patch 24, I can live with the linkage as-is.
FWIW, guests on newer versions of Hyper-V will always register a kmsg dumper. The flags that are tested to decide whether to register provide compatibility with older versions of Hyper-V that don?t support the 4K bytes of notification info.
Quoted 11 lines
May I suggest something? If possible, take a run with this patch set + DEBUG_NOTIFIER=y, in *both* cases (with and without the
kmsg_dump()set). I did that and they run almost at the same time...I've checked the notifiers called, it's like almost nothing runs in-between.I feel the panic notification mechanism does really fit with a hypervisor list, it's a good match with the nature of the list, which aims at informing the panic notification to the hypervisor/FW. Of course we can modify it if you prefer...but please take into account the kdump case and how it complicates the logic.
I agree that the runtime effect of one list vs. the other is nil. The code works and can stay as you written it.
I was trying to align from a conceptual standpoint. It was a bit unexpected that one path would be on the hypervisor list, and the other path effectively on the informational list. When I see conceptual mismatches like that, I tend to want to understand why, and if there is something more fundamental that is out-of-whack.
Quoted 7 lines
Let me know your considerations, in case you can experiment with the patch set as-is.
Cheers,Guilherme
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 11 lines · Michael Kelley (LINUX)
[...]
Quoted 9 lines
Hi Michael, thanks for your feedback! I agree that your idea could work, but...there is one downside: imagine the
kmsg_dump()approach is not set in some Hyper-V guest, then we would rely in the regular notification mechanism [hv_die_panic_notify_crash()], right?
But...you want then to run this notifier in the informational list, which...won't execute *by default* before kdump if nokmsg_dump()is set. So, this logic is convoluted when you mix it with the default level concept + kdump.
Quoted 4 lines
Yes, you are right. But to me that speaks as much to the linkage between the informational list and
kmsg_dump()being the core problem. But as I described in my reply to Patch 24, I can live with the linkage as-is.
Thanks for the feedback Michael!
Quoted 7 lines
[...]
Quoted 5 lines
I feel the panic notification mechanism does really fit with a hypervisor list, it's a good match with the nature of the list, which aims at informing the panic notification to the hypervisor/FW. Of course we can modify it if you prefer...but please take into account the kdump case and how it complicates the logic.
Quoted 9 lines
I agree that the runtime effect of one list vs. the other is nil. The code works and can stay as you written it.
I was trying to align from a conceptual standpoint. It was a bit unexpected that one path would be on the hypervisor list, and the other path effectively on the informational list. When I see conceptual mismatches like that, I tend to want to understand why, and if there is something more fundamental that is out-of-whack.
Totally agree with you here, I am like that as well - try to really understand the details, this is very important specially in this patch set, since it's a refactor and affects every user of the notifiers infrastructure.
Again, just to double-say it: feel free to suggest any change for the Hyper-V portion (might as well for any patch in the series, indeed) - you and the other Hyper-V maintainers own this code and I'd be glad to align with your needs, you are honor citizens in the panic notifiers area, being one the most heavy users for that =)
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
In reply to Guilherme G. PiccoliQuoted 18 lines · Guilherme G. Piccoli
The goal of this new panic notifier is to allow its users to register callbacks to run very early in the panic path. This aims hypervisor/FW notification mechanisms as well as simple LED functions, and any other simple and safe mechanism that should run early in the panic path; more dangerous callbacks should execute later.
For now, the patch is almost a no-op (although it changes a bit the ordering in which some panic notifiers are executed). In a subsequent patch, the panic path will be refactored, then the panic hypervisor notifiers will effectively run very early in the panic path.
We also defer documenting it all properly in the subsequent refactor patch. While at it, we removed some useless header inclusions and fixed some notifiers return too (by using the standard
NOTIFY_DONE).
arch/mips/sgi-ip22/ip22-reset.c static int __init reboot_setup(void) · line 198
Quoted 5 lines
arch/mips/sgi-ip22/ip22-reset.c +1
-1@@ -195,7 +195,7 @@ static int __init reboot_setup(void) }timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);
This notifier enables blinking. It is not much safe. It calls mod_timer() that takes a lock internally.
This kind of functionality should go into the last list called before panic() enters the infinite loop. IMHO, all the blinking stuff should go there.
arch/mips/sgi-ip32/ip32-reset.c static __init int ip32_reboot_setup(void) · line 147
Quoted 13 lines
arch/mips/sgi-ip22/ip22-reset.c +0
-0return 0 ; }arch/mips/sgi-ip32/ip32-reset.c +1
-1@@ -145,7 +144,7 @@ static __init int ip32_reboot_setup(void) pm_power_off = ip32_machine_halt;timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);
Same here. Should be done only before the "loop".
drivers/firmware/google/gsmi.c static __init int gsmi_init(void) · line 1038
Quoted 12 lines
arch/mips/sgi-ip32/ip32-reset.c +0
-0return 0 ; }drivers/firmware/google/gsmi.c +1
-1@@ -1034,7 +1034,7 @@ static __init int gsmi_init(void)register_reboot_notifier (&gsmi_reboot_notifier);register_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &gsmi_panic_notifier);
I am not sure about this one. It looks like some logging or pre_reboot stuff.
drivers/leds/trigger/ledtrig-activity.c static int __init activity_init(void) · line 251
Quoted 11 lines
drivers/firmware/google/gsmi.c +0
-0printk (KERN_INFO "gsmi version " DRIVER_VERSION " loaded\n" );drivers/leds/trigger/ledtrig-activity.c +1
-1@@ -247,7 +247,7 @@ static int __init activity_init(void)int rc =led_trigger_register (&activity_led_trigger);if (!rc) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &activity_panic_nb);
The notifier is trivial. It just sets a variable.
But still, it is about blinking and should be done in the last "loop" list.
drivers/leds/trigger/ledtrig-heartbeat.c static int __init heartbeat_trig_init(void) · line 194
Quoted 11 lines
drivers/leds/trigger/ledtrig-activity.c +0
-0register_reboot_notifier (&activity_reboot_nb); }drivers/leds/trigger/ledtrig-heartbeat.c +1
-1@@ -190,7 +190,7 @@ static int __init heartbeat_trig_init(void)int rc =led_trigger_register (&heartbeat_led_trigger);if (!rc) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &heartbeat_panic_nb);
Same here. Blinking => loop list.
drivers/misc/bcm-vk/bcm_vk_dev.c static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent) · line 1450
Quoted 13 lines
drivers/leds/trigger/ledtrig-heartbeat.c +0
-0register_reboot_notifier (&heartbeat_reboot_nb); }drivers/misc/bcm-vk/bcm_vk_dev.c +1
-1@@ -1446,7 +1446,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)/* register for panic notifier */ vk->panic_nb.notifier_call = bcm_vk_on_panic; - err =atomic_notifier_chain_register (&panic_notifier_list, + err =atomic_notifier_chain_register (&panic_hypervisor_list, &vk->panic_nb);
It seems to reset some hardware or so. IMHO, it should go into the pre-reboot list.
drivers/power/reset/ltc2952-poweroff.c static int ltc2952_poweroff_probe(struct platform_device *pdev) · line 283
Quoted 11 lines
drivers/misc/bcm-vk/bcm_vk_dev.c +0
-0if (err) {dev_err (dev,"Fail to register panic notifier\n" );drivers/power/reset/ltc2952-poweroff.c +1
-1@@ -279,7 +279,7 @@ static int ltc2952_poweroff_probe(struct platform_device *pdev) pm_power_off = ltc2952_poweroff_kill; data->panic_notifier.notifier_call = ltc2952_poweroff_notify_panic; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &data->panic_notifier);
I looks like this somehow triggers the reboot. IMHO, it should go into the pre_reboot list.
drivers/soc/bcm/brcmstb/pm/pm-arm.c static int brcmstb_pm_probe(struct platform_device *pdev) · line 818
Quoted 11 lines
drivers/power/reset/ltc2952-poweroff.c +0
-0dev_info (&pdev->dev,"probe successful\n" );drivers/soc/bcm/brcmstb/pm/pm-arm.c +1
-1@@ -814,7 +814,7 @@ static int brcmstb_pm_probe(struct platform_device *pdev) goto out; } -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &brcmstb_pm_panic_nb);
I am not sure about this one. It instruct some HW to preserve DRAM. IMHO, it better fits into pre_reboot category but I do not have strong opinion.
drivers/soc/bcm/brcmstb/pm/pm-arm.c static int brcmstb_pm_probe(struct platform_device *pdev) · line 820
Quoted 2 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +0
-0pm_power_off = brcmstb_pm_poweroff;
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Thanks for the review!
I agree with the blinking stuff, I can rework and add all LED/blinking stuff into the loop list, it does make sense. I'll comment a bit in the others below...
Quoted 22 lines · Petr Mladek
[...]
arch/mips/sgi-ip22/ip22-reset.c static int __init reboot_setup(void) · line 198
Quoted 8 lines
arch/mips/sgi-ip22/ip22-reset.c +1
-1@@ -195,7 +195,7 @@ static int __init reboot_setup(void) }timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);This notifier enables blinking. It is not much safe. It calls
mod_timer()that takes a lock internally.This kind of functionality should go into the last list called before
panic()enters the infinite loop. IMHO, all the blinking stuff should go there.
[...]Quoted 5 lines
arch/mips/sgi-ip32/ip32-reset.c +0
-0@@ -145,7 +144,7 @@ static __init int ip32_reboot_setup(void) pm_power_off = ip32_machine_halt;
Quoted 6 lines
arch/mips/sgi-ip32/ip32-reset.c static __init int ip32_reboot_setup(void) · line 147
Quoted 3 lines
arch/mips/sgi-ip32/ip32-reset.c +1
-1timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);Same here. Should be done only before the "loop".
[...]
Ack.
Quoted 4 lines
Quoted 4 lines
drivers/firmware/google/gsmi.c +0
-0@@ -1034,7 +1034,7 @@ static __init int gsmi_init(void)
Quoted 9 lines
drivers/firmware/google/gsmi.c static __init int gsmi_init(void) · line 1038
Quoted 5 lines
drivers/firmware/google/gsmi.c +1
-1register_reboot_notifier (&gsmi_reboot_notifier);register_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &gsmi_panic_notifier);I am not sure about this one. It looks like some logging or
pre_rebootstuff.
Disagree here. I'm looping Google maintainers, so they can comment. (CCed Evan, David, Julius)
This notifier is clearly a hypervisor notification mechanism. I've fixed a locking stuff there (in previous patch), I feel it's low-risk but even if it's mid-risk, the class of such callback remains a perfect fit with the hypervisor list IMHO.
Quoted 24 lines
[...]
drivers/leds/trigger/ledtrig-activity.c static int __init activity_init(void) · line 251
Quoted 9 lines
drivers/leds/trigger/ledtrig-activity.c +1
-1@@ -247,7 +247,7 @@ static int __init activity_init(void)int rc =led_trigger_register (&activity_led_trigger);if (!rc) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &activity_panic_nb);The notifier is trivial. It just sets a variable.
But still, it is about blinking and should be done in the last "loop" list.
Quoted 7 lines
drivers/leds/trigger/ledtrig-activity.c +0
-0register_reboot_notifier (&activity_reboot_nb); }drivers/leds/trigger/ledtrig-heartbeat.c +0
-0@@ -190,7 +190,7 @@ static int __init heartbeat_trig_init(void)int rc =led_trigger_register (&heartbeat_led_trigger);
Quoted 6 lines
drivers/leds/trigger/ledtrig-heartbeat.c static int __init heartbeat_trig_init(void) · line 194
Quoted 4 lines
drivers/leds/trigger/ledtrig-heartbeat.c +1
-1if (!rc) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &heartbeat_panic_nb);Same here. Blinking => loop list.
Ack.
Quoted 7 lines
Quoted 7 lines
[...]
drivers/misc/bcm-vk/bcm_vk_dev.c +0
-0@@ -1446,7 +1446,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
Quoted 8 lines
drivers/misc/bcm-vk/bcm_vk_dev.c static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent) · line 1450
Quoted 5 lines
drivers/misc/bcm-vk/bcm_vk_dev.c +1
-1/* register for panic notifier */ vk->panic_nb.notifier_call = bcm_vk_on_panic; - err =atomic_notifier_chain_register (&panic_notifier_list, + err =atomic_notifier_chain_register (&panic_hypervisor_list, &vk->panic_nb);It seems to reset some hardware or so. IMHO, it should go into the pre-reboot list.
Mixed feelings here, I'm looping Broadcom maintainers to comment. (CC Scott and Broadcom list)
I'm afraid it breaks kdump if this device is not reset beforehand - it's a doorbell write, so not high risk I think...
But in case the not-reset device can be probed normally in kdump kernel, then I'm fine in moving this to the reboot list! I don't have the HW to test myself.
Quoted 6 lines
[...]
Quoted 5 lines
drivers/power/reset/ltc2952-poweroff.c +0
-0@@ -279,7 +279,7 @@ static int ltc2952_poweroff_probe(struct platform_device *pdev) pm_power_off = ltc2952_poweroff_kill;
Quoted 7 lines
drivers/power/reset/ltc2952-poweroff.c static int ltc2952_poweroff_probe(struct platform_device *pdev) · line 283
Quoted 4 lines
drivers/power/reset/ltc2952-poweroff.c +1
-1data->panic_notifier.notifier_call = ltc2952_poweroff_notify_panic; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &data->panic_notifier);I looks like this somehow triggers the reboot. IMHO, it should go into the
pre_rebootlist.
Mixed feeling again here - CCing the maintainers for comments (Sebastian / PM folks).
This is setting a variable only, and once it's set (data->kernel_panic is the bool's name), it just bails out the IRQ handler and a timer setting - this timer seems kinda tricky, so bailing out ASAP makes sense IMHO.
But my mixed feeling comes from the fact this notifier really is not a fit to any list - it's just a "watchdog"/device quiesce in some form. Since it's very low-risk (IIUC), I've put it here.
Quoted 7 lines
[...]
Quoted 6 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +0
-0@@ -814,7 +814,7 @@ static int brcmstb_pm_probe(struct platform_device *pdev) goto out; }
Quoted 7 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c static int brcmstb_pm_probe(struct platform_device *pdev) · line 818
Quoted 3 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +1
-1-atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &brcmstb_pm_panic_nb);I am not sure about this one. It instruct some HW to preserve DRAM. IMHO, it better fits into
pre_rebootcategory but I do not have strong opinion.
Disagree here, I'm CCing Florian for information.
This notifier preserves RAM so it's *very interesting* if we have kmsg_dump() for example, but maybe might be also relevant in case kdump kernel is configured to store something in a persistent RAM (then, without this notifier, after kdump reboots the system data would be lost).
Cheers,
Guilherme
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 54 lines · Guilherme G. Piccoli
Thanks for the review!
I agree with the blinking stuff, I can rework and add all LED/blinking stuff into the loop list, it does make sense. I'll comment a bit in the others below...
Quoted 22 lines · Petr Mladek
[...]
arch/mips/sgi-ip22/ip22-reset.c static int __init reboot_setup(void) · line 198
Quoted 8 lines
arch/mips/sgi-ip22/ip22-reset.c +1
-1@@ -195,7 +195,7 @@ static int __init reboot_setup(void) }timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);This notifier enables blinking. It is not much safe. It calls
mod_timer()that takes a lock internally.This kind of functionality should go into the last list called before
panic()enters the infinite loop. IMHO, all the blinking stuff should go there.
[...]Quoted 5 lines
arch/mips/sgi-ip32/ip32-reset.c +0
-0@@ -145,7 +144,7 @@ static __init int ip32_reboot_setup(void) pm_power_off = ip32_machine_halt;Quoted 6 lines
arch/mips/sgi-ip32/ip32-reset.c static __init int ip32_reboot_setup(void) · line 147
Quoted 3 lines
arch/mips/sgi-ip32/ip32-reset.c +1
-1timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);Same here. Should be done only before the "loop".
[...]Ack.
Quoted 4 lines
Quoted 4 lines
drivers/firmware/google/gsmi.c +0
-0@@ -1034,7 +1034,7 @@ static __init int gsmi_init(void)Quoted 9 lines
drivers/firmware/google/gsmi.c static __init int gsmi_init(void) · line 1038
Quoted 5 lines
drivers/firmware/google/gsmi.c +1
-1register_reboot_notifier (&gsmi_reboot_notifier);register_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &gsmi_panic_notifier);I am not sure about this one. It looks like some logging or
pre_rebootstuff.
Quoted 7 lines
Disagree here. I'm looping Google maintainers, so they can comment. (CCed Evan, David, Julius)
This notifier is clearly a hypervisor notification mechanism. I've fixed a locking stuff there (in previous patch), I feel it's low-risk but even if it's mid-risk, the class of such callback remains a perfect fit with the hypervisor list IMHO.
This logs a panic to our "eventlog", a tiny logging area in SPI flash for critical and power-related events. In some cases this ends up being the only clue we get in a Chromebook feedback report that a panic occurred, so from my perspective moving it to the front of the line seems like a good idea.
-Evan
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 64 lines · Evan Green
Quoted 54 lines · Guilherme G. Piccoli
Thanks for the review!
I agree with the blinking stuff, I can rework and add all LED/blinking stuff into the loop list, it does make sense. I'll comment a bit in the others below...
Quoted 22 lines · Petr Mladek
[...]
arch/mips/sgi-ip22/ip22-reset.c static int __init reboot_setup(void) · line 198
Quoted 8 lines
arch/mips/sgi-ip22/ip22-reset.c +1
-1@@ -195,7 +195,7 @@ static int __init reboot_setup(void) }timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);This notifier enables blinking. It is not much safe. It calls
mod_timer()that takes a lock internally.This kind of functionality should go into the last list called before
panic()enters the infinite loop. IMHO, all the blinking stuff should go there.
[...]Quoted 5 lines
arch/mips/sgi-ip32/ip32-reset.c +0
-0@@ -145,7 +144,7 @@ static __init int ip32_reboot_setup(void) pm_power_off = ip32_machine_halt;Quoted 6 lines
arch/mips/sgi-ip32/ip32-reset.c static __init int ip32_reboot_setup(void) · line 147
Quoted 3 lines
arch/mips/sgi-ip32/ip32-reset.c +1
-1timer_setup (&blink_timer, blink_timeout,0 ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_hypervisor_list, &panic_block);Same here. Should be done only before the "loop".
[...]Ack.
Quoted 4 lines
Quoted 4 lines
drivers/firmware/google/gsmi.c +0
-0@@ -1034,7 +1034,7 @@ static __init int gsmi_init(void)Quoted 9 lines
drivers/firmware/google/gsmi.c static __init int gsmi_init(void) · line 1038
Quoted 5 lines
drivers/firmware/google/gsmi.c +1
-1register_reboot_notifier (&gsmi_reboot_notifier);register_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &gsmi_panic_notifier);I am not sure about this one. It looks like some logging or
pre_rebootstuff.Quoted 7 lines
Disagree here. I'm looping Google maintainers, so they can comment. (CCed Evan, David, Julius)
This notifier is clearly a hypervisor notification mechanism. I've fixed a locking stuff there (in previous patch), I feel it's low-risk but even if it's mid-risk, the class of such callback remains a perfect fit with the hypervisor list IMHO.
Quoted 5 lines
This logs a panic to our "eventlog", a tiny logging area in SPI flash for critical and power-related events. In some cases this ends up being the only clue we get in a Chromebook feedback report that a panic occurred, so from my perspective moving it to the front of the line seems like a good idea.
IMHO, this would really better fit into the pre-reboot notifier list:
+ the callback stores the log so it is similar to kmsg_dump() or console_flush_on_panic()
+ the callback should be proceed after "info" notifiers that might add some other useful information.
Honestly, I am not sure what exactly hypervisor callbacks do. But I think that they do not try to extract the kernel log because they would need to handle the internal format.
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 20 lines · Petr Mladek
[...]
Quoted 8 lines
Quoted 7 lines
Disagree here. I'm looping Google maintainers, so they can comment. (CCed Evan, David, Julius)
This notifier is clearly a hypervisor notification mechanism. I've fixed a locking stuff there (in previous patch), I feel it's low-risk but even if it's mid-risk, the class of such callback remains a perfect fit with the hypervisor list IMHO.
Quoted 5 lines
This logs a panic to our "eventlog", a tiny logging area in SPI flash for critical and power-related events. In some cases this ends up being the only clue we get in a Chromebook feedback report that a panic occurred, so from my perspective moving it to the front of the line seems like a good idea.
IMHO, this would really better fit into the pre-reboot notifier list:
+ the callback stores the log so it is similar to
kmsg_dump()orconsole_flush_on_panic()
Quoted 7 lines
+ the callback should be proceed after "info" notifiers that might add some other useful information.
Honestly, I am not sure what exactly hypervisor callbacks do. But I think that they do not try to extract the kernel log because they would need to handle the internal format.
I guess the main point in your response is : "I am not sure what exactly hypervisor callbacks do". We need to be sure about the semantics of such list, and agree on that.
So, my opinion about this first list, that we call "hypervisor list",
is: it contains callbacks that
(1) should run early, preferably before kdump (or even if kdump isn't set, should run ASAP);
(2) these callbacks perform some communication with an abstraction that runs "below" the kernel, like a firmware or hypervisor. Classic example:
pvpanic, that communicates with VMM (usually qemu) and allow such VMM to snapshot the full guest memory, for example.
(3) Should be low-risk. What defines risk is the level of reliability of subsequent operations - if the callback have 50% of chance of "bricking" the system totally and prevent kdump / kmsg_dump() / reboot , this is high risk one for example.
Some good fits IMO: pvpanic, sstate_panic_event() [sparc], fadump in powerpc, etc.
So, this is a good case for the Google notifier as well - it's not collecting data like the dmesg (hence your second bullet seems to not apply here, info notifiers won't add info to be collected by gsmi). It is a firmware/hypervisor/whatever-gsmi-is notification mechanism, that tells such "lower" abstraction a panic occurred. It seems low risk and we want it to run ASAP, if possible.
So, I'd like to keep it here, unless gsmi maintainers disagree or I'm perhaps misunderstanding the meaning of this first list. Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 52 lines · Guilherme G. Piccoli
Quoted 20 lines · Petr Mladek
[...]
Quoted 8 lines
Quoted 7 lines
Disagree here. I'm looping Google maintainers, so they can comment. (CCed Evan, David, Julius)
This notifier is clearly a hypervisor notification mechanism. I've fixed a locking stuff there (in previous patch), I feel it's low-risk but even if it's mid-risk, the class of such callback remains a perfect fit with the hypervisor list IMHO.
Quoted 5 lines
This logs a panic to our "eventlog", a tiny logging area in SPI flash for critical and power-related events. In some cases this ends up being the only clue we get in a Chromebook feedback report that a panic occurred, so from my perspective moving it to the front of the line seems like a good idea.
IMHO, this would really better fit into the pre-reboot notifier list:
+ the callback stores the log so it is similar to
kmsg_dump()orconsole_flush_on_panic()Quoted 7 lines
+ the callback should be proceed after "info" notifiers that might add some other useful information.
Honestly, I am not sure what exactly hypervisor callbacks do. But I think that they do not try to extract the kernel log because they would need to handle the internal format.
I guess the main point in your response is : "I am not sure what exactly hypervisor callbacks do". We need to be sure about the semantics of such list, and agree on that.
So, my opinion about this first list, that we call "hypervisor list",
is: it contains callbacks that(1) should run early, preferably before kdump (or even if kdump isn't set, should run ASAP);
(2) these callbacks perform some communication with an abstraction that runs "below" the kernel, like a firmware or hypervisor. Classic example:
pvpanic, that communicates with VMM (usually qemu) and allow such VMM to snapshot the full guest memory, for example.(3) Should be low-risk. What defines risk is the level of reliability of subsequent operations - if the callback have 50% of chance of "bricking" the system totally and prevent kdump /
kmsg_dump()/ reboot , this is high risk one for example.Some good fits IMO: pvpanic,
sstate_panic_event()[sparc], fadump in powerpc, etc.
Quoted 6 lines
So, this is a good case for the Google notifier as well - it's not collecting data like the dmesg (hence your second bullet seems to not apply here, info notifiers won't add info to be collected by gsmi). It is a
firmware/hypervisor/whatever-gsmi-isnotification mechanism, that tells such "lower" abstraction a panic occurred. It seems low risk and we want it to run ASAP, if possible.
"
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
This logs a panic to our "eventlog", a tiny logging area in SPI flash for critical and power-related events. In some cases this ends up
I see. I somehow assumed that it was about the kernel log because Evans wrote:
"This logs a panic to our "eventlog", a tiny logging area in SPI flash
for critical and power-related events. In some cases this ends up"
Anyway, I would distinguish it the following way.
+ If the notifier is preserving kernel log then it should be ideally treated as kmsg_dump().
+ It the notifier is saving another debugging data then it better fits into the "hypervisor" notifier list.
Regarding the reliability. From my POV, any panic notifier enabled in a generic kernel should be reliable with more than 99,9%. Otherwise, they should not be in the notifier list at all.
An exception would be a platform-specific notifier that is called only on some specific platform and developers maintaining this platform agree on this.
The value "99,9%" is arbitrary. I am not sure if it is realistic even in the other code, for example, console_flush_on_panic() or emergency_restart(). I just want to point out that the border should be rather high. Otherwise we would back in the situation where people would want to disable particular notifiers.
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 10 lines · Petr Mladek
[...]
Anyway, I would distinguish it the following way.+ If the notifier is preserving kernel log then it should be ideally treated as
kmsg_dump().+ It the notifier is saving another debugging data then it better fits into the "hypervisor" notifier list.
Definitely, I agree - it's logical, since we want more info in the logs, and happens some notifiers running in the informational list do that, like ftrace_on_oops for example.
Quoted 8 lines
Regarding the reliability. From my POV, any panic notifier enabled in a generic kernel should be reliable with more than 99,9%. Otherwise, they should not be in the notifier list at all.
An exception would be a platform-specific notifier that is called only on some specific platform and developers maintaining this platform agree on this.
Quoted 6 lines
The value "99,9%" is arbitrary. I am not sure if it is realistic even in the other code, for example,
console_flush_on_panic()oremergency_restart(). I just want to point out that the border should be rather high. Otherwise we would back in the situation where people would want to disable particular notifiers.
Totally agree, these percentages are just an example, 50% is ridiculous low reliability in my example heheh
But some notifiers deep dive in abstraction layers (like regmap or GPIO stuff) and it's hard to determine the probability of a lock issue (take a spinlock already taken inside regmap code and live-lock forever, for example). These are better to run, if possible, later than kdump or even info list.
Thanks again for the good analysis Petr!
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
In reply to Guilherme G. PiccoliQuoted 21 lines · Guilherme G. Piccoli
Thanks for the review!
I agree with the blinking stuff, I can rework and add all LED/blinking stuff into the loop list, it does make sense. I'll comment a bit in the others below...
Quoted 4 lines · Petr Mladek
Quoted 4 lines
drivers/firmware/google/gsmi.c +0
-0@@ -1034,7 +1034,7 @@ static __init int gsmi_init(void)Quoted 9 lines
drivers/firmware/google/gsmi.c static __init int gsmi_init(void) · line 1038
Quoted 5 lines
drivers/firmware/google/gsmi.c +1
-1register_reboot_notifier (&gsmi_reboot_notifier);register_die_notifier (&gsmi_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &gsmi_panic_notifier);I am not sure about this one. It looks like some logging or
pre_rebootstuff.
Quoted 7 lines
Disagree here. I'm looping Google maintainers, so they can comment. (CCed Evan, David, Julius)
This notifier is clearly a hypervisor notification mechanism. I've fixed a locking stuff there (in previous patch), I feel it's low-risk but even if it's mid-risk, the class of such callback remains a perfect fit with the hypervisor list IMHO.
It is similar to drivers/soc/bcm/brcmstb/pm/pm-arm.c. See below for another idea.
Quoted 16 lines
Quoted 4 lines
Quoted 4 lines
drivers/misc/bcm-vk/bcm_vk_dev.c +0
-0@@ -1446,7 +1446,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)Quoted 8 lines
drivers/misc/bcm-vk/bcm_vk_dev.c static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent) · line 1450
Quoted 5 lines
drivers/misc/bcm-vk/bcm_vk_dev.c +1
-1/* register for panic notifier */ vk->panic_nb.notifier_call = bcm_vk_on_panic; - err =atomic_notifier_chain_register (&panic_notifier_list, + err =atomic_notifier_chain_register (&panic_hypervisor_list, &vk->panic_nb);It seems to reset some hardware or so. IMHO, it should go into the pre-reboot list.
Mixed feelings here, I'm looping Broadcom maintainers to comment. (CC Scott and Broadcom list)
Quoted 6 lines
I'm afraid it breaks kdump if this device is not reset beforehand - it's a doorbell write, so not high risk I think...
But in case the not-reset device can be probed normally in kdump kernel, then I'm fine in moving this to the reboot list! I don't have the HW to test myself.
Good question. Well, it if has to be called before kdump then even "hypervisor" list is a wrong place because is not always called before kdump.
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
drivers/power/reset/ltc2952-poweroff.c +0
-0@@ -279,7 +279,7 @@ static int ltc2952_poweroff_probe(struct platform_device *pdev) pm_power_off = ltc2952_poweroff_kill;Quoted 7 lines
drivers/power/reset/ltc2952-poweroff.c static int ltc2952_poweroff_probe(struct platform_device *pdev) · line 283
Quoted 4 lines
drivers/power/reset/ltc2952-poweroff.c +1
-1data->panic_notifier.notifier_call = ltc2952_poweroff_notify_panic; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &data->panic_notifier);I looks like this somehow triggers the reboot. IMHO, it should go into the
pre_rebootlist.
Quoted 7 lines
Mixed feeling again here - CCing the maintainers for comments (Sebastian / PM folks).
This is setting a variable only, and once it's set (
data->kernel_panicis the bool's name), it just bails out the IRQ handler and a timer setting - this timer seems kinda tricky, so bailing out ASAP makes sense IMHO.
IMHO, the timer informs the hardware that the system is still alive in the middle of panic(). If the timer is not working then the hardware (chip) will think that the system frozen in panic() and will power off the system. See the comments in drivers/power/reset/ltc2952-poweroff.c:
* The following GPIOs are used:
* - trigger (input)
* A level change indicates the shut-down trigger. If it's state reverts * within the time-out defined by trigger_delay, the shut down is not * executed. If no pin is assigned to this input, the driver will start the * watchdog toggle immediately. The chip will only power off the system if * it is requested to do so through the kill line.
*
* - watchdog (output)
* Once a shut down is triggered, the driver will toggle this signal,
* with an internal (wde_interval) to stall the hardware shut down.
IMHO, we really have to keep it alive until we reach the reboot stage.
Another question is how it actually works when the interrupts are disabled during panic() and the timer callbacks are not handled.
Quoted 15 lines
Quoted 7 lines
[...]
Quoted 6 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +0
-0@@ -814,7 +814,7 @@ static int brcmstb_pm_probe(struct platform_device *pdev) goto out; }Quoted 7 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c static int brcmstb_pm_probe(struct platform_device *pdev) · line 818
Quoted 3 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +1
-1-atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &brcmstb_pm_panic_nb);I am not sure about this one. It instruct some HW to preserve DRAM. IMHO, it better fits into
pre_rebootcategory but I do not have strong opinion.
Quoted 6 lines
Disagree here, I'm CCing Florian for information.
This notifier preserves RAM so it's *very interesting* if we have
kmsg_dump()for example, but maybe might be also relevant in case kdump kernel is configured to store something in a persistent RAM (then, without this notifier, after kdump reboots the system data would be lost).
I see. It is actually similar problem as withdrivers/firmware/google/gsmi.c.
I does similar things like kmsg_dump() so it should be called in the same location (after info notifier list and before kdump).
A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 14 lines · Petr Mladek
[...]
Quoted 13 lines
Quoted 6 lines
drivers/misc/bcm-vk/bcm_vk_dev.c static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
Quoted 3 lines
drivers/misc/bcm-vk/bcm_vk_dev.c +0
-0@@ -1446,7 +1446,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)[... snip ...]
It seems to reset some hardware or so. IMHO, it should go into the pre-reboot list.Mixed feelings here, I'm looping Broadcom maintainers to comment. (CC Scott and Broadcom list)
I'm afraid it breaks kdump if this device is not reset beforehand - it's a doorbell write, so not high risk I think...
Quoted 7 lines
Quoted 3 lines
But in case the not-reset device can be probed normally in kdump kernel, then I'm fine in moving this to the reboot list! I don't have the HW to test myself.
Good question. Well, it if has to be called before kdump then even "hypervisor" list is a wrong place because is not always called before kdump.
Agreed! I'll defer that to Scott and Broadcom folks to comment. If it's not strictly necessary, I'll happily move it to the reboot list.
If necessary, we could use the machine_crash_kexec() approach, but we'll fall into the case arm64 doesn't support it and I'm not sure if this device is available for arm - again a question for the maintainers.
Quoted 17 lines
[...]
Quoted 7 lines
Quoted 2 lines
drivers/power/reset/ltc2952-poweroff.c
Quoted 2 lines
drivers/power/reset/ltc2952-poweroff.c +0
-0[...]
This is setting a variable only, and once it's set (data->kernel_panicis the bool's name), it just bails out the IRQ handler and a timer setting - this timer seems kinda tricky, so bailing out ASAP makes sense IMHO.IMHO, the timer informs the hardware that the system is still alive in the middle of
panic(). If the timer is not working then the hardware (chip) will think that the system frozen inpanic()and will power off the system. See the comments indrivers/power/reset/ltc2952-poweroff.c:
[.... snip ...]
IMHO, we really have to keep it alive until we reach the reboot stage.
Quoted 2 lines
Another question is how it actually works when the interrupts are disabled during
panic()and the timer callbacks are not handled.
Agreed here! Guess I can move this one the reboot list, fine by me. Unless PM folks think otherwise.
Quoted 8 lines
[...]
Quoted 6 lines
Disagree here, I'm CCing Florian for information.
This notifier preserves RAM so it's *very interesting* if we have
kmsg_dump()for example, but maybe might be also relevant in case kdump kernel is configured to store something in a persistent RAM (then, without this notifier, after kdump reboots the system data would be lost).
Quoted 8 lines
I see. It is actually similar problem as with
drivers/firmware/google/gsmi.c.I does similar things like
kmsg_dump()so it should be called in the same location (after info notifier list and before kdump).A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
Here I still disagree. I've commented in the other response thread (about Google gsmi) about the semantics of the hypervisor list, but
again: this list should contain callbacks that
(a) Should run early, _by default_ before a kdump;
(b) Communicate with the firmware/hypervisor in a "low-risk" way;
Imagine a scenario where users configure kdump kernel to save something in a persistent form in DRAM - it'd be like a late pstore, in the next kernel. This callback enables that, it's meant to inform FW "hey, panic happened, please from now on don't clear the RAM in the next FW-reboot". I don't see a reason to postpone that - let's see if the maintainers have an opinion.
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 24 lines · Guilherme G. Piccoli
Quoted 7 lines · Petr Mladek
Quoted 6 lines
Disagree here, I'm CCing Florian for information.
This notifier preserves RAM so it's *very interesting* if we have
kmsg_dump()for example, but maybe might be also relevant in case kdump kernel is configured to store something in a persistent RAM (then, without this notifier, after kdump reboots the system data would be lost).Quoted 8 lines
I see. It is actually similar problem as with
drivers/firmware/google/gsmi.c.I does similar things like
kmsg_dump()so it should be called in the same location (after info notifier list and before kdump).A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
Here I still disagree. I've commented in the other response thread (about Google gsmi) about the semantics of the hypervisor list, but
again: this list should contain callbacks that(a) Should run early,
_bydefault_before a kdump;
(b) Communicate with thefirmware/hypervisorin a "low-risk" way;
Quoted 6 lines
Imagine a scenario where users configure kdump kernel to save something in a persistent form in DRAM - it'd be like a late pstore, in the next kernel. This callback enables that, it's meant to inform FW "hey, panic happened, please from now on don't clear the RAM in the next FW-reboot". I don't see a reason to postpone that - let's see if the maintainers have an opinion.
I have answered this in more detail in the other reply, see
https://lore.kernel.org/r/YoShZVYNAdvvjb7z at alley
I agree that both notifiers in
drivers/soc/bcm/brcmstb/pm/pm-arm.c
drivers/firmware/google/gsmi.c
better fit into the hypervisor list after all.
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 6 lines · Petr Mladek
[...]
I have answered this in more detail in the other reply, see
https://lore.kernel.org/r/YoShZVYNAdvvjb7z at alleyI agree that both notifiers in
Quoted 7 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c
drivers/firmware/google/gsmi.cbetter fit into the hypervisor list after all.
Best Regards,
Petr
Perfect, thanks - will keep both in such list for V2.
Scott Branden
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
In reply to Guilherme G. PiccoliHi Guilherme,
+Desmond
Quoted 23 lines · Guilherme G. Piccoli
Quoted 14 lines · Petr Mladek
[...]
Quoted 13 lines
Quoted 6 lines
drivers/misc/bcm-vk/bcm_vk_dev.c static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
Quoted 3 lines
drivers/misc/bcm-vk/bcm_vk_dev.c +0
-0@@ -1446,7 +1446,7 @@ static int bcm_vk_probe(struct pci_dev *pdev, const struct pci_device_id *ent)[... snip ...]
It seems to reset some hardware or so. IMHO, it should go into the pre-reboot list.Mixed feelings here, I'm looping Broadcom maintainers to comment. (CC Scott and Broadcom list)
I'm afraid it breaks kdump if this device is not reset beforehand - it's a doorbell write, so not high risk I think...
Quoted 7 lines
Quoted 3 lines
But in case the not-reset device can be probed normally in kdump kernel, then I'm fine in moving this to the reboot list! I don't have the HW to test myself.
Good question. Well, it if has to be called before kdump then even "hypervisor" list is a wrong place because is not always called before kdump.
Quoted 6 lines
Agreed! I'll defer that to Scott and Broadcom folks to comment. If it's not strictly necessary, I'll happily move it to the reboot list.
If necessary, we could use the
machine_crash_kexec()approach, but we'll fall into the case arm64 doesn't support it and I'm not sure if this device is available for arm - again a question for the maintainers.
We register to the panic notifier so that we can kill the VK card ASAP to stop DMAing things over to the host side. If it is not notified then memory may not be frozen when kdump is occurring.
Notifying the card on panic is also needed to allow for any type of reset to occur.
So, the only thing preventing moving the notifier later is the chance that memory is modified while kdump is occurring. Or, if DMA is disabled before kdump already then this wouldn't be an issue and the notification to the card (to allow for clean resets) can be done later.
Quoted 57 lines
Quoted 19 lines
[...]
Quoted 7 lines
Quoted 2 lines
drivers/power/reset/ltc2952-poweroff.c
Quoted 2 lines
drivers/power/reset/ltc2952-poweroff.c +0
-0[...]
This is setting a variable only, and once it's set (data->kernel_panicis the bool's name), it just bails out the IRQ handler and a timer setting - this timer seems kinda tricky, so bailing out ASAP makes sense IMHO.IMHO, the timer informs the hardware that the system is still alive in the middle of
panic(). If the timer is not working then the hardware (chip) will think that the system frozen inpanic()and will power off the system. See the comments indrivers/power/reset/ltc2952-poweroff.c:
[.... snip ...]
IMHO, we really have to keep it alive until we reach the reboot stage.Quoted 2 lines
Another question is how it actually works when the interrupts are disabled during
panic()and the timer callbacks are not handled.Agreed here! Guess I can move this one the reboot list, fine by me. Unless PM folks think otherwise.
Quoted 8 lines
[...]
Quoted 6 lines
Disagree here, I'm CCing Florian for information.
This notifier preserves RAM so it's *very interesting* if we have
kmsg_dump()for example, but maybe might be also relevant in case kdump kernel is configured to store something in a persistent RAM (then, without this notifier, after kdump reboots the system data would be lost).Quoted 8 lines
I see. It is actually similar problem as with
drivers/firmware/google/gsmi.c.I does similar things like
kmsg_dump()so it should be called in the same location (after info notifier list and before kdump).A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
Here I still disagree. I've commented in the other response thread (about Google gsmi) about the semantics of the hypervisor list, but
again: this list should contain callbacks that(a) Should run early,
_bydefault_before a kdump;
(b) Communicate with thefirmware/hypervisorin a "low-risk" way;Imagine a scenario where users configure kdump kernel to save something in a persistent form in DRAM - it'd be like a late pstore, in the next kernel. This callback enables that, it's meant to inform FW "hey, panic happened, please from now on don't clear the RAM in the next FW-reboot". I don't see a reason to postpone that - let's see if the maintainers have an opinion.
Quoted 4 lines
Cheers,
Guilherme
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4212 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20220518/325178ed/attachment.p7s>
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 21 lines · Scott Branden
Hi Guilherme,
+Desmond
[...]Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
I'm afraid it breaks kdump if this device is not reset beforehand - it's a doorbell write, so not high risk I think...
But in case the not-reset device can be probed normally in kdump kernel, then I'm fine in moving this to the reboot list! I don't have the HW to test myself.
Good question. Well, it if has to be called before kdump then even "hypervisor" list is a wrong place because is not always called before kdump.
[...]
We register to the panic notifier so that we can kill the VK card ASAP to stop DMAing things over to the host side. If it is not notified then memory may not be frozen when kdump is occurring.
Notifying the card on panic is also needed to allow for any type of reset to occur.
Quoted 4 lines
So, the only thing preventing moving the notifier later is the chance that memory is modified while kdump is occurring. Or, if DMA is disabled before kdump already then this wouldn't be an issue and the notification to the card (to allow for clean resets) can be done later.
Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases?
I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel.
With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters.
With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what.
Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers,
Guilherme
Scott Branden
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 22 lines · Guilherme G. Piccoli
Quoted 21 lines · Scott Branden
Hi Guilherme,
+Desmond
[...]Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
I'm afraid it breaks kdump if this device is not reset beforehand - it's a doorbell write, so not high risk I think...
But in case the not-reset device can be probed normally in kdump kernel, then I'm fine in moving this to the reboot list! I don't have the HW to test myself.
Good question. Well, it if has to be called before kdump then even "hypervisor" list is a wrong place because is not always called before kdump.
[...]
We register to the panic notifier so that we can kill the VK card ASAP to stop DMAing things over to the host side. If it is not notified then memory may not be frozen when kdump is occurring.
Notifying the card on panic is also needed to allow for any type of reset to occur.
Quoted 7 lines
Quoted 4 lines
So, the only thing preventing moving the notifier later is the chance that memory is modified while kdump is occurring. Or, if DMA is disabled before kdump already then this wouldn't be an issue and the notification to the card (to allow for clean resets) can be done later.
Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases?
The adapter may be used in any PCIe design that supports DMA. So it may be possible to run in arm64 servers.
Quoted 8 lines
I'm not expert on that, but I guess whether DMA is "kept" or not depends a bit if IOMMU is used. IIRC, there was a copy of the DMAR table in kdump (at least for Intel IOMMU). Also, devices are not properly quiesced on kdump IIUC, we don't call shutdown/reset handlers, they're skip due to the crash nature - so there is a risk of devices doing bad things in the new kernel.
Quoted 4 lines
With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters.
Sounds the best to keep system operating as tested today.
Quoted 6 lines
With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom
crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what.
It may be beneficial for some other drivers but for our use we would then need to register for the panic path and the crash_shutdown path. We notify the VK card for 2 purposes: one to stop DMA so memory stop changing during a kdump. And also to get the card into a good state so resets happen cleanly.
Quoted 6 lines
Let me know your thoughts Scott / Desmond / Petr and all interested parties. Cheers,
Guilherme
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4212 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.infradead.org/pipermail/kexec/attachments/20220519/da0525db/attachment-0001.p7s>
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 18 lines · Scott Branden
[...]
Quoted 2 lines
Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases?
The adapter may be used in any PCIe design that supports DMA. So it may be possible to run in arm64 servers.
Quoted 6 lines
[...]
With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters.Sounds the best to keep system operating as tested today.
Quoted 6 lines
With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom
crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what.
Quoted 5 lines
It may be beneficial for some other drivers but for our use we would then need to register for the panic path and the
crash_shutdownpath. We notify the VK card for 2 purposes: one to stop DMA so memory stop changing during a kdump. And also to get the card into a good state so resets happen cleanly.
Thanks Scott! With that, I guess it's really better to keep this notifier in this hypervisor/early list - I'm planning to do that for V2. Unless Petr or somebody has strong feelings against that, of course.
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 25 lines · Guilherme G. Piccoli
Quoted 18 lines · Scott Branden
[...]
Quoted 2 lines
Hi Scott / Desmond, thanks for the detailed answer! Is this adapter designed to run in x86 only or you have other architectures' use cases?
The adapter may be used in any PCIe design that supports DMA. So it may be possible to run in arm64 servers.
Quoted 6 lines
[...]
With that said, and given this is a lightweight notifier that ideally should run ASAP, I'd keep this one in the hypervisor list. We can "adjust" the semantic of this list to include lightweight notifiers that reset adapters.Sounds the best to keep system operating as tested today.
Quoted 6 lines
With that said, Petr has a point - not always such list is going to be called before kdump. So, that makes me think in another idea: what if we have another list, but not on panic path, but instead in the custom
crash_shutdown()? Drivers could add callbacks there that must execute before kexec/kdump, no matter what.Quoted 5 lines
It may be beneficial for some other drivers but for our use we would then need to register for the panic path and the
crash_shutdownpath. We notify the VK card for 2 purposes: one to stop DMA so memory stop changing during a kdump. And also to get the card into a good state so resets happen cleanly.
Quoted 3 lines
Thanks Scott! With that, I guess it's really better to keep this notifier in this hypervisor/early list - I'm planning to do that for V2. Unless Petr or somebody has strong feelings against that, of course.
I am fine with it because we do not have a better solution at the moment.
It might be a good candidate for the 5th notifier list mentioned in the thread https://lore.kernel.org/r/YoyQyHHfhIIXSX0U at alley . But I am not sure if the 5th list is worth the complexity.
Best Regards,
Petr
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
In reply to Petr MladekQuoted 17 lines · Petr Mladek
Quoted 16 lines · Guilherme G. Piccoli
Quoted 6 lines
Quoted 6 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +0
-0@@ -814,7 +814,7 @@ static int brcmstb_pm_probe(struct platform_device *pdev) goto out; }Quoted 7 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c static int brcmstb_pm_probe(struct platform_device *pdev) · line 818
Quoted 3 lines
drivers/soc/bcm/brcmstb/pm/pm-arm.c +1
-1-atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_hypervisor_list, &brcmstb_pm_panic_nb);I am not sure about this one. It instruct some HW to preserve DRAM. IMHO, it better fits into
pre_rebootcategory but I do not have strong opinion.Disagree here, I'm CCing Florian for information.
Quoted 7 lines
Quoted 4 lines
This notifier preserves RAM so it's *very interesting* if we have
kmsg_dump()for example, but maybe might be also relevant in case kdump kernel is configured to store something in a persistent RAM (then, without this notifier, after kdump reboots the system data would be lost).I see. It is actually similar problem as with
drivers/firmware/google/gsmi.c.
As discussed in the other other reply, it seems that both affected notifiers do not store kernel logs and should stay in the "hypervisor".
Quoted 5 lines
I does similar things like
kmsg_dump()so it should be called in the same location (after info notifier list and before kdump).A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
I just want to point out that the above idea has problems. Notifiers storing kernel log need to be treated as kmsg_dump(). In particular, we would need to know if there are any. We do not need to call "info" notifier list before kdump when there is no kernel log dumper registered.
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 4 lines · Petr Mladek
[...]
Quoted 3 lines
I does similar things like
kmsg_dump()so it should be called in the same location (after info notifier list and before kdump).
Quoted 9 lines
Quoted 2 lines
A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
I just want to point out that the above idea has problems. Notifiers storing kernel log need to be treated as
kmsg_dump(). In particular, we would need to know if there are any. We do not need to call "info" notifier list before kdump when there is no kernel log dumper registered.
Notifiers respect the priority concept, which is just a number that orders the list addition (and the list is called in order).
I've used the last position to panic_print() [in patch 25] - one idea here is to "reserve" the last position (represented by INT_MIN) for notifiers that act like kmsg_dump(). I couldn't find any IIRC, but that doesn't prevent us to save this position and comment about that.
Makes sense to you ?
Cheers!
Petr Mladek
maintainerRe: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 15 lines · Guilherme G. Piccoli
Quoted 4 lines · Petr Mladek
[...]
Quoted 3 lines
I does similar things like
kmsg_dump()so it should be called in the same location (after info notifier list and before kdump).Quoted 9 lines
Quoted 2 lines
A solution might be to put it at these notifiers at the very end of the "info" list or make extra "dump" notifier list.
I just want to point out that the above idea has problems. Notifiers storing kernel log need to be treated as
kmsg_dump(). In particular, we would need to know if there are any. We do not need to call "info" notifier list before kdump when there is no kernel log dumper registered.
Quoted 7 lines
Notifiers respect the priority concept, which is just a number that orders the list addition (and the list is called in order).
I've used the last position to
panic_print()[in patch 25] - one idea here is to "reserve" the last position (represented byINT_MIN) for notifiers that act likekmsg_dump(). I couldn't find any IIRC, but that doesn't prevent us to save this position and comment about that.
I would ignore it for now. If anyone would want to safe the log then they would need to read it. They will most likely use the existing kmsg_dump() infastructure. In fact, they should use it to avoid a code duplication.
Best Regards,
Petr
Re: [PATCH 19/30] panic: Add the panic hypervisor notifier list
Quoted 8 lines · Petr Mladek
[...]
I would ignore it for now. If anyone would want to safe the log then they would need to read it. They will most likely use the existingkmsg_dump()infastructure. In fact, they should use it to avoid a code duplication.Best Regards,
Petr
Cool, thanks! I agree, let's expect people use kmsg_dump() as they should =)
PATCH 20/30 panic: Add the panic informational notifier list
In reply to Guilherme G. PiccoliThe goal of this new panic notifier is to allow its users to register callbacks to run earlier in the panic path than they currently do. This aims at informational mechanisms, like dumping kernel offsets and showing device error data (in case it's simple registers reading, for example) as well as mechanisms to disable log flooding (like hung_task detector / RCU warnings) and the tracing dump_on_oops (when enabled).
Any (non-invasive) information that should be provided before kmsg_dump() as well as log flooding preventing code should fit here, as long it offers relatively low risk for kdump.
For now, the patch is almost a no-op, although it changes a bit the ordering in which some panic notifiers are executed - specially affected by this are the notifiers responsible for disabling the hung_task detector / RCU warnings, which now run first. In a subsequent patch, the panic path will be refactored, then the panic informational notifiers will effectively run earlier, before ksmg_dump() (and usually before kdump as well).
We also defer documenting it all properly in the subsequent refactor patch. Finally, while at it, we removed some useless header inclusions too.
--- arch/arm64/kernel/setup.c | 2 +- arch/mips/kernel/relocate.c | 2 +- arch/powerpc/kernel/setup-common.c | 2 +- arch/x86/kernel/setup.c | 2 +- drivers/bus/brcmstb_gisb.c | 2 +- drivers/hwtracing/coresight/coresight-cpu-debug.c | 4 ++-- drivers/soc/tegra/ari-tegra186.c | 3 ++- include/linux/panic_notifier.h | 1 + kernel/hung_task.c | 3 ++- kernel/panic.c | 4 ++++ kernel/rcu/tree.c | 1 - kernel/rcu/tree_stall.h | 3 ++- kernel/trace/trace.c | 2 +- 13 files changed, 19 insertions(+), 12 deletions(-)
arch/arm64/kernel/setup.c +1 -1
@@ -444,7 +444,7 @@ static struct notifier_block arm64_panic_block = {static int __init register_arm64_panic_block (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &arm64_panic_block);return 0 ; }
arch/mips/kernel/relocate.c +1 -1
@@ -459,7 +459,7 @@ static struct notifier_block kernel_location_notifier = {static int __init register_kernel_offset_dumper (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &kernel_location_notifier);return 0 ; }
arch/powerpc/kernel/setup-common.c +1 -1
@@ -757,7 +757,7 @@ void __init setup_panic(void) &ppc_fadump_block);if (IS_ENABLED (CONFIG_RANDOMIZE_BASE ) &&kaslr_offset () >0 ) -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &kernel_offset_notifier);/* Low-level platform-specific routines that should run on panic */
arch/x86/kernel/setup.c +1 -1
@@ -1266,7 +1266,7 @@ static struct notifier_block kernel_offset_notifier = {static int __init register_kernel_offset_dumper (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &kernel_offset_notifier);return 0 ; }
drivers/bus/brcmstb_gisb.c +1 -1
@@ -486,7 +486,7 @@ static int __init brcmstb_gisb_arb_probe(struct platform_device *pdev)if (list_is_singular (&brcmstb_gisb_arb_device_list)) {register_die_notifier (&gisb_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &gisb_panic_notifier); }
drivers/hwtracing/coresight/coresight-cpu-debug.c +2 -2
@@ -535,7 +535,7 @@ static int debug_func_init(void) &debug_func_knob_fops);/* Register function to be called for panic */ - ret =atomic_notifier_chain_register (&panic_notifier_list, + ret =atomic_notifier_chain_register (&panic_info_list, &debug_notifier);if (ret) { pr_err("%s: unable to register notifier: %d\n",
@@ -552,7 +552,7 @@ static int debug_func_init(void)static void debug_func_exit (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_info_list, &debug_notifier);debugfs_remove_recursive (debug_debugfs_dir); }
drivers/soc/tegra/ari-tegra186.c +2 -1
@@ -73,7 +73,8 @@ static struct notifier_block tegra186_ari_panic_nb = {static int __init tegra186_ari_init (void ) {if (of_machine_is_compatible ("nvidia,tegra186" )) -atomic_notifier_chain_register (&panic_notifier_list, &tegra186_ari_panic_nb); +atomic_notifier_chain_register (&panic_info_list, + &tegra186_ari_panic_nb);return 0 ; }
include/linux/panic_notifier.h +1 -0
@@ -7,6 +7,7 @@extern struct atomic_notifier_head panic_notifier_list;extern struct atomic_notifier_head panic_hypervisor_list; +extern struct atomic_notifier_head panic_info_list;extern bool crash_kexec_post_notifiers;
kernel/hung_task.c +2 -1
@@ -85,6 +85,7 @@ hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)static struct notifier_block panic_block = { .notifier_call = hung_task_panic, + .priority =INT_MAX ,/* run early to prevent potential log flood */ };static void check_hung_task (struct task_struct *t,unsigned long timeout)
@@ -378,7 +379,7 @@ static int watchdog(void *dummy)static int __init hung_task_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_info_list, &panic_block);/* Disable hung task detector on suspend */ pm_notifier (hungtask_pm_notify,0 );
kernel/panic.c +4 -0
@@ -76,6 +76,9 @@ EXPORT_SYMBOL(panic_notifier_list);ATOMIC_NOTIFIER_HEAD (panic_hypervisor_list);EXPORT_SYMBOL (panic_hypervisor_list); +ATOMIC_NOTIFIER_HEAD (panic_info_list); +EXPORT_SYMBOL (panic_info_list); +static long no_blink (int state) {return 0 ;
@@ -291,6 +294,7 @@ void panic(const char *fmt, ...)* add information to the kmsg dump output. */ atomic_notifier_call_chain (&panic_hypervisor_list,PANIC_NOTIFIER , buf); +atomic_notifier_call_chain (&panic_info_list,PANIC_NOTIFIER , buf);atomic_notifier_call_chain (&panic_notifier_list,PANIC_NOTIFIER , buf);panic_print_sys_info (false);
kernel/rcu/tree.c +0 -1
@@ -35,7 +35,6 @@#include <linux/panic.h> #include <linux/panic_notifier.h> #include <linux/percpu.h> -#include <linux/notifier.h> #include <linux/cpu.h> #include <linux/mutex.h> #include <linux/time.h>
kernel/rcu/tree_stall.h +2 -1
@@ -97,11 +97,12 @@ static int rcu_panic(struct notifier_block *this, unsigned long ev, void *ptr)static struct notifier_block rcu_panic_block = { .notifier_call = rcu_panic, + .priority =INT_MAX ,/* run early to prevent potential log flood */ };static int __init check_cpu_stall_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &rcu_panic_block); +atomic_notifier_chain_register (&panic_info_list, &rcu_panic_block);return 0 ; }early_initcall (check_cpu_stall_init);
kernel/trace/trace.c +1 -1
@@ -10138,7 +10138,7 @@ __init static int tracer_alloc_buffers(void)/* All seems OK, enable tracing */ tracing_disabled =0 ; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &trace_panic_notifier);register_die_notifier (&trace_die_notifier);
Signature · 1 line
2.36.0
Re: [PATCH 20/30] panic: Add the panic informational notifier list
Quoted 39 lines · Guilherme G. Piccoli
The goal of this new panic notifier is to allow its users to register callbacks to run earlier in the panic path than they currently do. This aims at informational mechanisms, like dumping kernel offsets and showing device error data (in case it's simple registers reading, for example) as well as mechanisms to disable log flooding (like
hung_taskdetector / RCU warnings) and the tracingdump_on_oops(when enabled).Any (non-invasive) information that should be provided before
kmsg_dump()as well as log flooding preventing code should fit here, as long it offers relatively low risk for kdump.For now, the patch is almost a no-op, although it changes a bit the ordering in which some panic notifiers are executed - specially affected by this are the notifiers responsible for disabling the
hung_taskdetector / RCU warnings, which now run first. In a subsequent patch, the panic path will be refactored, then the panic informational notifiers will effectively run earlier, beforeksmg_dump()(and usually before kdump as well).We also defer documenting it all properly in the subsequent refactor patch. Finally, while at it, we removed some useless header inclusions too.
Cc Benjamin Herrenschmidt <[email protected]>Cc Catalin Marinas <[email protected]>Cc Florian Fainelli <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Hari Bathini <[email protected]>Cc Joel Fernandes <[email protected]>Cc Jonathan Hunter <[email protected]>Cc Josh Triplett <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Leo Yan <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Michael Ellerman <[email protected]>Cc Mike Leach <[email protected]>
Quoted 8 lines
Cc Mikko Perttunen <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Paul Mackerras <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Thierry Reding <[email protected]>Cc Thomas Bogendoerfer <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
From an RCU perspective:
Quoted 218 lines
--- arch/arm64/kernel/setup.c | 2 +- arch/mips/kernel/relocate.c | 2 +- arch/powerpc/kernel/setup-common.c | 2 +- arch/x86/kernel/setup.c | 2 +- drivers/bus/brcmstb_gisb.c | 2 +- drivers/hwtracing/coresight/coresight-cpu-debug.c | 4 ++-- drivers/soc/tegra/ari-tegra186.c | 3 ++- include/linux/panic_notifier.h | 1 + kernel/hung_task.c | 3 ++- kernel/panic.c | 4 ++++ kernel/rcu/tree.c | 1 - kernel/rcu/tree_stall.h | 3 ++- kernel/trace/trace.c | 2 +- 13 files changed, 19 insertions(+), 12 deletions(-)arch/arm64/kernel/setup.c +1
-1@@ -444,7 +444,7 @@ static struct notifier_block arm64_panic_block = {static int __init register_arm64_panic_block (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &arm64_panic_block);return 0 ; }arch/mips/kernel/relocate.c +1
-1@@ -459,7 +459,7 @@ static struct notifier_block kernel_location_notifier = {static int __init register_kernel_offset_dumper (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &kernel_location_notifier);return 0 ; }arch/powerpc/kernel/setup-common.c +1
-1@@ -757,7 +757,7 @@ void __init setup_panic(void) &ppc_fadump_block);if (IS_ENABLED (CONFIG_RANDOMIZE_BASE ) &&kaslr_offset () >0 ) -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &kernel_offset_notifier);/* Low-level platform-specific routines that should run on panic */ arch/x86/kernel/setup.c +1
-1@@ -1266,7 +1266,7 @@ static struct notifier_block kernel_offset_notifier = {static int __init register_kernel_offset_dumper (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &kernel_offset_notifier);return 0 ; }drivers/bus/brcmstb_gisb.c +1
-1@@ -486,7 +486,7 @@ static int __init brcmstb_gisb_arb_probe(struct platform_device *pdev)if (list_is_singular (&brcmstb_gisb_arb_device_list)) {register_die_notifier (&gisb_die_notifier); -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &gisb_panic_notifier); }drivers/hwtracing/coresight/coresight-cpu-debug.c +2
-2@@ -535,7 +535,7 @@ static int debug_func_init(void) &debug_func_knob_fops);/* Register function to be called for panic */ - ret =atomic_notifier_chain_register (&panic_notifier_list, + ret =atomic_notifier_chain_register (&panic_info_list, &debug_notifier);if (ret) { pr_err("%s: unable to register notifier: %d\n",@@ -552,7 +552,7 @@ static int debug_func_init(void)static void debug_func_exit (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_info_list, &debug_notifier);debugfs_remove_recursive (debug_debugfs_dir); }drivers/soc/tegra/ari-tegra186.c +2
-1@@ -73,7 +73,8 @@ static struct notifier_block tegra186_ari_panic_nb = {static int __init tegra186_ari_init (void ) {if (of_machine_is_compatible ("nvidia,tegra186" )) -atomic_notifier_chain_register (&panic_notifier_list, &tegra186_ari_panic_nb); +atomic_notifier_chain_register (&panic_info_list, + &tegra186_ari_panic_nb);return 0 ; }include/linux/panic_notifier.h +1
-0@@ -7,6 +7,7 @@extern struct atomic_notifier_head panic_notifier_list;extern struct atomic_notifier_head panic_hypervisor_list; +extern struct atomic_notifier_head panic_info_list;extern bool crash_kexec_post_notifiers;kernel/hung_task.c +2
-1@@ -85,6 +85,7 @@ hung_task_panic(struct notifier_block *this, unsigned long event, void *ptr)static struct notifier_block panic_block = { .notifier_call = hung_task_panic, + .priority =INT_MAX ,/* run early to prevent potential log flood */ };static void check_hung_task (struct task_struct *t,unsigned long timeout)@@ -378,7 +379,7 @@ static int watchdog(void *dummy)static int __init hung_task_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_info_list, &panic_block);/* Disable hung task detector on suspend */ pm_notifier (hungtask_pm_notify,0 );kernel/panic.c +4
-0@@ -76,6 +76,9 @@ EXPORT_SYMBOL(panic_notifier_list);ATOMIC_NOTIFIER_HEAD (panic_hypervisor_list);EXPORT_SYMBOL (panic_hypervisor_list); +ATOMIC_NOTIFIER_HEAD (panic_info_list); +EXPORT_SYMBOL (panic_info_list); +static long no_blink (int state) {return 0 ;@@ -291,6 +294,7 @@ void panic(const char *fmt, ...)* add information to the kmsg dump output. */ atomic_notifier_call_chain (&panic_hypervisor_list,PANIC_NOTIFIER , buf); +atomic_notifier_call_chain (&panic_info_list,PANIC_NOTIFIER , buf);atomic_notifier_call_chain (&panic_notifier_list,PANIC_NOTIFIER , buf);panic_print_sys_info (false);kernel/rcu/tree.c +0
-1@@ -35,7 +35,6 @@#include <linux/panic.h> #include <linux/panic_notifier.h> #include <linux/percpu.h> -#include <linux/notifier.h> #include <linux/cpu.h> #include <linux/mutex.h> #include <linux/time.h> kernel/rcu/tree_stall.h +2
-1@@ -97,11 +97,12 @@ static int rcu_panic(struct notifier_block *this, unsigned long ev, void *ptr)static struct notifier_block rcu_panic_block = { .notifier_call = rcu_panic, + .priority =INT_MAX ,/* run early to prevent potential log flood */ };static int __init check_cpu_stall_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &rcu_panic_block); +atomic_notifier_chain_register (&panic_info_list, &rcu_panic_block);return 0 ; }early_initcall (check_cpu_stall_init);kernel/trace/trace.c +1
-1@@ -10138,7 +10138,7 @@ __init static int tracer_alloc_buffers(void)/* All seems OK, enable tracing */ tracing_disabled =0 ; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_info_list, &trace_panic_notifier);register_die_notifier (&trace_die_notifier);Signature · 1 line
2.36.0
Re: [PATCH 20/30] panic: Add the panic informational notifier list
In reply to Guilherme G. PiccoliQuoted 63 lines · Guilherme G. Piccoli
The goal of this new panic notifier is to allow its users to register callbacks to run earlier in the panic path than they currently do. This aims at informational mechanisms, like dumping kernel offsets and showing device error data (in case it's simple registers reading, for example) as well as mechanisms to disable log flooding (like
hung_taskdetector / RCU warnings) and the tracingdump_on_oops(when enabled).Any (non-invasive) information that should be provided before
kmsg_dump()as well as log flooding preventing code should fit here, as long it offers relatively low risk for kdump.For now, the patch is almost a no-op, although it changes a bit the ordering in which some panic notifiers are executed - specially affected by this are the notifiers responsible for disabling the
hung_taskdetector / RCU warnings, which now run first. In a subsequent patch, the panic path will be refactored, then the panic informational notifiers will effectively run earlier, beforeksmg_dump()(and usually before kdump as well).We also defer documenting it all properly in the subsequent refactor patch. Finally, while at it, we removed some useless header inclusions too.
Cc Benjamin Herrenschmidt <[email protected]>Cc Catalin Marinas <[email protected]>Cc Florian Fainelli <[email protected]>Cc Frederic Weisbecker <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Hari Bathini <[email protected]>Cc Joel Fernandes <[email protected]>Cc Jonathan Hunter <[email protected]>Cc Josh Triplett <[email protected]>Cc Lai Jiangshan <[email protected]>Cc Leo Yan <[email protected]>Cc Mathieu Desnoyers <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Michael Ellerman <[email protected]>Cc Mike Leach <[email protected]>Cc Mikko Perttunen <[email protected]>Cc Neeraj Upadhyay <[email protected]>Cc Nicholas Piggin <[email protected]>Cc Paul Mackerras <[email protected]>Cc Suzuki K Poulose <[email protected]>Cc Thierry Reding <[email protected]>Cc Thomas Bogendoerfer <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>--- arch/arm64/kernel/setup.c | 2 +- arch/mips/kernel/relocate.c | 2 +- arch/powerpc/kernel/setup-common.c | 2 +- arch/x86/kernel/setup.c | 2 +- drivers/bus/brcmstb_gisb.c | 2 +- drivers/hwtracing/coresight/coresight-cpu-debug.c | 4 ++-- drivers/soc/tegra/ari-tegra186.c | 3 ++- include/linux/panic_notifier.h | 1 + kernel/hung_task.c | 3 ++- kernel/panic.c | 4 ++++ kernel/rcu/tree.c | 1 - kernel/rcu/tree_stall.h | 3 ++- kernel/trace/trace.c | 2 +- 13 files changed, 19 insertions(+), 12 deletions(-)
...
Quoted 14 lines
drivers/hwtracing/coresight/coresight-cpu-debug.c +1
-1@@ -535,7 +535,7 @@ static int debug_func_init(void) &debug_func_knob_fops);/* Register function to be called for panic */ - ret =atomic_notifier_chain_register (&panic_notifier_list, + ret =atomic_notifier_chain_register (&panic_info_list, &debug_notifier);if (ret) { pr_err("%s: unable to register notifier: %d\n",
drivers/hwtracing/coresight/coresight-cpu-debug.c static int debug_func_init(void) · line 558
Quoted 8 lines
drivers/hwtracing/coresight/coresight-cpu-debug.c +1
-1@@ -552,7 +552,7 @@ static int debug_func_init(void)static void debug_func_exit (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_info_list, &debug_notifier);debugfs_remove_recursive (debug_debugfs_dir); }
Re: [PATCH 20/30] panic: Add the panic informational notifier list
Thanks Paul and Suzuki for the ACKs.
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 20/30] panic: Add the panic informational notifier list
In reply to Guilherme G. PiccoliQuoted 20 lines · Guilherme G. Piccoli
The goal of this new panic notifier is to allow its users to register callbacks to run earlier in the panic path than they currently do. This aims at informational mechanisms, like dumping kernel offsets and showing device error data (in case it's simple registers reading, for example) as well as mechanisms to disable log flooding (like
hung_taskdetector / RCU warnings) and the tracingdump_on_oops(when enabled).Any (non-invasive) information that should be provided before
kmsg_dump()as well as log flooding preventing code should fit here, as long it offers relatively low risk for kdump.For now, the patch is almost a no-op, although it changes a bit the ordering in which some panic notifiers are executed - specially affected by this are the notifiers responsible for disabling the
hung_taskdetector / RCU warnings, which now run first. In a subsequent patch, the panic path will be refactored, then the panic informational notifiers will effectively run earlier, beforeksmg_dump()(and usually before kdump as well).
Quoted 5 lines
We also defer documenting it all properly in the subsequent refactor patch. Finally, while at it, we removed some useless header inclusions too.
Signed-off-by Guilherme G. Piccoli <[email protected]>
All notifiers moved in this patch seems to fit well the "info" notifier list. The patch looks good from this POV.
I still have to review the rest of the patches to see if it is complete.
Best Regards,
Petr
Re: [PATCH 20/30] panic: Add the panic informational notifier list
Quoted 10 lines · Petr Mladek
[...]
All notifiers moved in this patch seems to fit well the "info" notifier list. The patch looks good from this POV.
I still have to review the rest of the patches to see if it is complete.
Best Regards,
Petr
Thanks a bunch for the review =)
PATCH 21/30 panic: Introduce the panic pre-reboot notifier list
In reply to Guilherme G. PiccoliThis patch renames the panic_notifier_list to panic_pre_reboot_list; the idea is that a subsequent patch will refactor the panic path in order to better split the notifiers, running some of them very early, some of them not so early [but still before kmsg_dump()] and finally, the rest should execute late, after kdump. The latter ones are now in the panic pre-reboot list - the name comes from the idea that these notifiers execute before panic() attempts rebooting the machine (if that option is set).
We also took the opportunity to clean-up useless header inclusions, improve some notifier block declarations (e.g. in ibmasm/heartbeat.c) and more important, change some priorities - we hereby set 2 notifiers to run late in the list [iss_panic_event() and the IPMI panic_event()] due to the risks they offer (may not return, for example). Proper documentation is going to be provided in a subsequent patch, that effectively refactors the panic path.
---
Notice that, with this name change, out-of-tree code that relies in the global exported "panic_notifier_list" will fail to build. We could easily keep the retro-compatibility by making the old symbol to still exist and point to the pre_reboot list (or even, keep the old naming).
But our design choice was to allow the breakage, making users rethink their notifiers, adding them in the list that fits best. If that wasn't a good decision, we're open to change it, of course.
Thanks in advance for the review!
arch/alpha/kernel/setup.c | 4 ++-- arch/parisc/kernel/pdc_chassis.c | 3 +-- arch/powerpc/kernel/setup-common.c | 2 +- arch/s390/kernel/ipl.c | 4 ++-- arch/um/drivers/mconsole_kern.c | 2 +- arch/um/kernel/um_arch.c | 2 +- arch/x86/xen/enlighten.c | 2 +- arch/xtensa/platforms/iss/setup.c | 4 ++-- drivers/char/ipmi/ipmi_msghandler.c | 12 +++++++----- drivers/edac/altera_edac.c | 3 +-- drivers/hv/vmbus_drv.c | 4 ++-- drivers/leds/trigger/ledtrig-panic.c | 3 +-- drivers/misc/ibmasm/heartbeat.c | 16 +++++++++------- drivers/net/ipa/ipa_smp2p.c | 5 ++--- drivers/parisc/power.c | 4 ++-- drivers/remoteproc/remoteproc_core.c | 6 ++++-- drivers/s390/char/con3215.c | 2 +- drivers/s390/char/con3270.c | 2 +- drivers/s390/char/sclp_con.c | 2 +- drivers/s390/char/sclp_vt220.c | 2 +- drivers/staging/olpc_dcon/olpc_dcon.c | 6 ++++-- drivers/video/fbdev/hyperv_fb.c | 4 ++-- include/linux/panic_notifier.h | 2 +- kernel/panic.c | 9 ++++----- 24 files changed, 54 insertions(+), 51 deletions(-)
arch/alpha/kernel/setup.c +2 -2
@@ -472,8 +472,8 @@ setup_arch(char **cmdline_p) }/* Register a call for panic conditions. */ -atomic_notifier_chain_register (&panic_notifier_list, - &alpha_panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &alpha_panic_block);#ifndef alpha_using_srm/* Assume that we've booted from SRM if we haven't booted from MILO.
arch/parisc/kernel/pdc_chassis.c +1 -2
@@ -22,7 +22,6 @@#include <linux/kernel.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> -#include <linux/notifier.h> #include <linux/cache.h> #include <linux/proc_fs.h> #include <linux/seq_file.h>
@@ -135,7 +134,7 @@ void __init parisc_pdc_chassis_init(void) PDC_CHASSIS_VER);/* initialize panic notifier chain */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &pdc_chassis_panic_block);/* initialize reboot notifier chain */
arch/powerpc/kernel/setup-common.c +1 -1
@@ -762,7 +762,7 @@ void __init setup_panic(void)/* Low-level platform-specific routines that should run on panic */ if (ppc_md.panic) -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &ppc_panic_block); }
arch/s390/kernel/ipl.c +2 -2
@@ -2034,7 +2034,7 @@ static int on_panic_notify(struct notifier_block *self, unsigned long event, void *data) {do_panic (); -return NOTIFY_OK ; +return NOTIFY_DONE ; } static struct notifier_block on_panic_nb = {
@@ -2069,7 +2069,7 @@ void __init setup_ipl(void)/* We have no info to copy */ break ; } -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb); }void s390_reset_system (void )
arch/um/drivers/mconsole_kern.c +1 -1
@@ -855,7 +855,7 @@ static struct notifier_block panic_exit_notifier = {static int add_notifier (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &panic_exit_notifier);return 0 ; }
arch/um/kernel/um_arch.c +1 -1
@@ -257,7 +257,7 @@ static struct notifier_block panic_exit_notifier = {void uml_finishsetup (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &panic_exit_notifier);uml_postsetup ();
arch/x86/xen/enlighten.c +1 -1
@@ -290,7 +290,7 @@ static struct notifier_block xen_panic_block = {int xen_panic_handler_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &xen_panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, &xen_panic_block);return 0 ; }
arch/xtensa/platforms/iss/setup.c +2 -2
@@ -13,7 +13,6 @@*/ #include <linux/init.h> #include <linux/kernel.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/printk.h> #include <linux/string.h>
@@ -53,6 +52,7 @@ iss_panic_event(struct notifier_block *this, unsigned long event, void *ptr)static struct notifier_block iss_panic_block = { .notifier_call = iss_panic_event, + .priority =INT_MIN ,/* run as late as possible, may not return */ };void __init platform_setup (char **p_cmdline)
@@ -81,5 +81,5 @@ void __init platform_setup(char **p_cmdline) } } -atomic_notifier_chain_register (&panic_notifier_list, &iss_panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, &iss_panic_block); }
drivers/char/ipmi/ipmi_msghandler.c +7 -5
@@ -25,7 +25,6 @@#include <linux/slab.h> #include <linux/ipmi.h> #include <linux/ipmi_smi.h> -#include <linux/notifier.h> #include <linux/init.h> #include <linux/proc_fs.h> #include <linux/rcupdate.h>
@@ -5375,10 +5374,13 @@ static int ipmi_register_driver(void)return rv; } +/* +* we should execute this panic callback late, since it involves +* a complex call-chain and panic() runs in atomic context. +*/ static struct notifier_block panic_block = { .notifier_call = panic_event, - .next =NULL , - .priority =200 /* priority: INT_MAX >= x >= 0 */ + .priority =INT_MIN +1 , };static int ipmi_init_msghandler (void )
@@ -5406,7 +5408,7 @@ static int ipmi_init_msghandler(void)timer_setup (&ipmi_timer, ipmi_timeout,0 );mod_timer (&ipmi_timer, jiffies +IPMI_TIMEOUT_JIFFIES ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, &panic_block); initialized = true;
@@ -5438,7 +5440,7 @@ static void __exit cleanup_ipmi(void)if (initialized) {destroy_workqueue (remove_work_wq); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &panic_block);/*
drivers/edac/altera_edac.c +1 -2
@@ -16,7 +16,6 @@#include <linux/kernel.h> #include <linux/mfd/altera-sysmgr.h> #include <linux/mfd/syscon.h> -#include <linux/notifier.h> #include <linux/of_address.h> #include <linux/of_irq.h> #include <linux/of_platform.h>
@@ -2163,7 +2162,7 @@ static int altr_edac_a10_probe(struct platform_device *pdev)int dberror, err_addr; edac->panic_notifier.notifier_call = s10_edac_dberr_handler; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &edac->panic_notifier);/* Printout a message if uncorrectable error previously. */
drivers/hv/vmbus_drv.c +2 -2
@@ -1622,7 +1622,7 @@ static int vmbus_bus_init(void)* Always register the vmbus unload panic notifier because we * need to shut the VMbus channel connection on panic. */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &hyperv_panic_vmbus_unload_block);vmbus_request_offers ();
@@ -2851,7 +2851,7 @@ static void __exit vmbus_exit(void)* The vmbus panic notifier is always registered, hence we should * also unconditionally unregister it here as well. */ -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &hyperv_panic_vmbus_unload_block);free_page ((unsigned long )hv_panic_page);
drivers/leds/trigger/ledtrig-panic.c +1 -2
@@ -7,7 +7,6 @@#include <linux/kernel.h> #include <linux/init.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/leds.h> #include "../leds.h"
@@ -64,7 +63,7 @@ static long led_panic_blink(int state)static int __init ledtrig_panic_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &led_trigger_panic_nb);led_trigger_register_simple ("panic" , &trigger);
drivers/misc/ibmasm/heartbeat.c +9 -7
@@ -8,7 +8,6 @@* Author: Max Asb?ck <[email protected]> */ -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include "ibmasm.h" #include "dot_command.h"
@@ -24,7 +23,7 @@ static int suspend_heartbeats = 0;* In the case of a panic the interrupt handler continues to work and thus * continues to respond to heartbeats, making the service processor believe * the OS is still running and thus preventing a reboot. -* To prevent this from happening a callback is added the panic_notifier_list. +* To prevent this from happening a callback is added in a panic notifier list. * Before responding to a heartbeat the driver checks if a panic has happened, * if yes it suspends heartbeat, causing the service processor to reboot as * expected.
@@ -32,20 +31,23 @@ static int suspend_heartbeats = 0;static int panic_happened (struct notifier_block *n,unsigned long val,void *v) { suspend_heartbeats =1 ; -return 0 ; +return NOTIFY_DONE ; } -static struct notifier_block panic_notifier = { panic_happened,NULL ,1 }; +static struct notifier_block panic_notifier = { + .notifier_call = panic_happened, +};void ibmasm_register_panic_notifier (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &panic_notifier); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &panic_notifier); }void ibmasm_unregister_panic_notifier (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, - &panic_notifier); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &panic_notifier); }
drivers/net/ipa/ipa_smp2p.c +2 -3
@@ -7,7 +7,6 @@#include <linux/types.h> #include <linux/device.h> #include <linux/interrupt.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/pm_runtime.h> #include <linux/soc/qcom/smem.h>
@@ -138,13 +137,13 @@ static int ipa_smp2p_panic_notifier_register(struct ipa_smp2p *smp2p) smp2p->panic_notifier.notifier_call = ipa_smp2p_panic_notifier; smp2p->panic_notifier.priority =INT_MAX ;/* Do it early */ -return atomic_notifier_chain_register (&panic_notifier_list, +return atomic_notifier_chain_register (&panic_pre_reboot_list, &smp2p->panic_notifier); }static void ipa_smp2p_panic_notifier_unregister (struct ipa_smp2p *smp2p) { -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &smp2p->panic_notifier); }
drivers/parisc/power.c +2 -2
@@ -233,7 +233,7 @@ static int __init power_init(void) }/* Register a call for panic conditions. */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &parisc_panic_block);return 0 ;
@@ -243,7 +243,7 @@ static void __exit power_exit(void) {kthread_stop (power_task); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &parisc_panic_block);pdc_soft_power_button (0 );
drivers/remoteproc/remoteproc_core.c +4 -2
@@ -2795,12 +2795,14 @@ static int rproc_panic_handler(struct notifier_block *nb, unsigned long event,static void __init rproc_init_panic (void ) { rproc_panic_nb.notifier_call = rproc_panic_handler; -atomic_notifier_chain_register (&panic_notifier_list, &rproc_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &rproc_panic_nb); } static void __exit rproc_exit_panic(void) { -atomic_notifier_chain_unregister (&panic_notifier_list, &rproc_panic_nb); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &rproc_panic_nb); }static int __init remoteproc_init (void )
drivers/s390/char/con3215.c +1 -1
@@ -867,7 +867,7 @@ static int __init con3215_init(void) raw3215[0 ] =NULL ;return -ENODEV ; } -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&con3215);return 0 ;
drivers/s390/char/con3270.c +1 -1
@@ -645,7 +645,7 @@ con3270_init(void) condev->cline->len =0 ;con3270_create_status (condev); condev->input =alloc_string (&condev->freemem,80 ); -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&con3270);return 0 ;
drivers/s390/char/sclp_con.c +1 -1
@@ -288,7 +288,7 @@ sclp_console_init(void)timer_setup (&sclp_con_timer, sclp_console_timeout,0 );/* enable printk-access to this driver */ -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&sclp_console);return 0 ;
drivers/s390/char/sclp_vt220.c +1 -1
@@ -838,7 +838,7 @@ sclp_vt220_con_init(void)if (rc)return rc;/* Attach linux console */ -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&sclp_vt220_console);return 0 ;
drivers/staging/olpc_dcon/olpc_dcon.c +4 -2
@@ -653,7 +653,8 @@ static int dcon_probe(struct i2c_client *client, const struct i2c_device_id *id) }register_reboot_notifier (&dcon->reboot_nb); -atomic_notifier_chain_register (&panic_notifier_list, &dcon_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &dcon_panic_nb);return 0 ;
@@ -676,7 +677,8 @@ static int dcon_remove(struct i2c_client *client)struct dcon_priv *dcon =i2c_get_clientdata (client);unregister_reboot_notifier (&dcon->reboot_nb); -atomic_notifier_chain_unregister (&panic_notifier_list, &dcon_panic_nb); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &dcon_panic_nb);free_irq (DCON_IRQ , dcon);
drivers/video/fbdev/hyperv_fb.c +2 -2
@@ -1253,7 +1253,7 @@ static int hvfb_probe(struct hv_device *hdev,*/ par->hvfb_panic_nb.notifier_call = hvfb_on_panic; par->hvfb_panic_nb.priority =INT_MIN +10 , -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &par->hvfb_panic_nb);return 0 ;
@@ -1276,7 +1276,7 @@ static int hvfb_remove(struct hv_device *hdev)struct fb_info *info =hv_get_drvdata (hdev);struct hvfb_par *par = info->par; -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &par->hvfb_panic_nb); par->update = false;
include/linux/panic_notifier.h +1 -1
@@ -5,9 +5,9 @@#include <linux/notifier.h> #include <linux/types.h> -extern struct atomic_notifier_head panic_notifier_list;extern struct atomic_notifier_head panic_hypervisor_list;extern struct atomic_notifier_head panic_info_list; +extern struct atomic_notifier_head panic_pre_reboot_list;extern bool crash_kexec_post_notifiers;
kernel/panic.c +4 -5
@@ -69,16 +69,15 @@ EXPORT_SYMBOL_GPL(panic_timeout);#define PANIC_PRINT_ALL_CPU_BT 0x00000040unsigned long panic_print; -ATOMIC_NOTIFIER_HEAD (panic_notifier_list); - -EXPORT_SYMBOL (panic_notifier_list); -ATOMIC_NOTIFIER_HEAD (panic_hypervisor_list);EXPORT_SYMBOL (panic_hypervisor_list);ATOMIC_NOTIFIER_HEAD (panic_info_list);EXPORT_SYMBOL (panic_info_list); +ATOMIC_NOTIFIER_HEAD (panic_pre_reboot_list); +EXPORT_SYMBOL (panic_pre_reboot_list); +static long no_blink (int state) {return 0 ;
@@ -295,7 +294,7 @@ void panic(const char *fmt, ...)*/ atomic_notifier_call_chain (&panic_hypervisor_list,PANIC_NOTIFIER , buf);atomic_notifier_call_chain (&panic_info_list,PANIC_NOTIFIER , buf); -atomic_notifier_call_chain (&panic_notifier_list,PANIC_NOTIFIER , buf); +atomic_notifier_call_chain (&panic_pre_reboot_list,PANIC_NOTIFIER , buf);panic_print_sys_info (false);
Signature · 1 line
2.36.0
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 17 lines · Guilherme G. Piccoli
This patch renames the
panic_notifier_listtopanic_pre_reboot_list; the idea is that a subsequent patch will refactor the panic path in order to better split the notifiers, running some of them very early, some of them not so early [but still beforekmsg_dump()] and finally, the rest should execute late, after kdump. The latter ones are now in the panic pre-reboot list - the name comes from the idea that these notifiers execute beforepanic()attempts rebooting the machine (if that option is set).We also took the opportunity to clean-up useless header inclusions, improve some notifier block declarations (e.g. in
ibmasm/heartbeat.c) and more important, change some priorities - we hereby set 2 notifiers to run late in the list [iss_panic_event()and the IPMIpanic_event()] due to the risks they offer (may not return, for example). Proper documentation is going to be provided in a subsequent patch, that effectively refactors the panic path.
Quoted 1 line
Cc Alex Elder <[email protected]>
For "drivers/net/ipa/ipa_smp2p.c":
Quoted 36 lines
Cc Alexander Gordeev <[email protected]>Cc Anton Ivanov <[email protected]>Cc Benjamin Herrenschmidt <[email protected]>Cc Bjorn Andersson <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Chris Zankel <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Corey Minyard <[email protected]>Cc Dexuan Cui <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Haiyang Zhang <[email protected]>Cc Heiko Carstens <[email protected]>Cc Helge Deller <[email protected]>Cc Ivan Kokshaysky <[email protected]>Cc "James E.J. Bottomley" <[email protected]>Cc James Morse <[email protected]>Cc Johannes Berg <[email protected]>Cc Juergen Gross <[email protected]>Cc "K. Y. Srinivasan" <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Matt Turner <[email protected]>Cc Mauro Carvalho Chehab <[email protected]>Cc Max Filippov <[email protected]>Cc Michael Ellerman <[email protected]>Cc Paul Mackerras <[email protected]>Cc Pavel Machek <[email protected]>Cc Richard Henderson <[email protected]>Cc Richard Weinberger <[email protected]>Cc Robert Richter <[email protected]>Cc Stefano Stabellini <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Sven Schnelle <[email protected]>Cc Tony Luck <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Wei Liu <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Quoted 2 lines
---
. . .
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
In reply to Guilherme G. PiccoliQuoted 9 lines · Guilherme G. Piccoli
This patch renames the
panic_notifier_listtopanic_pre_reboot_list; the idea is that a subsequent patch will refactor the panic path in order to better split the notifiers, running some of them very early, some of them not so early [but still beforekmsg_dump()] and finally, the rest should execute late, after kdump. The latter ones are now in the panic pre-reboot list - the name comes from the idea that these notifiers execute beforepanic()attempts rebooting the machine (if that option is set).
Quoted 7 lines
We also took the opportunity to clean-up useless header inclusions, improve some notifier block declarations (e.g. in
ibmasm/heartbeat.c) and more important, change some priorities - we hereby set 2 notifiers to run late in the list [iss_panic_event()and the IPMIpanic_event()] due to the risks they offer (may not return, for example). Proper documentation is going to be provided in a subsequent patch, that effectively refactors the panic path.
For the IPMI portion:
Note that the IPMI panic_event() should always return, but it may take some time, especially if the IPMI controller is no longer functional. So the risk of a long delay is there and it makes sense to move it very late.
-corey
Quoted 593 lines
Cc Alex Elder <[email protected]>Cc Alexander Gordeev <[email protected]>Cc Anton Ivanov <[email protected]>Cc Benjamin Herrenschmidt <[email protected]>Cc Bjorn Andersson <[email protected]>Cc Boris Ostrovsky <[email protected]>Cc Chris Zankel <[email protected]>Cc Christian Borntraeger <[email protected]>Cc Corey Minyard <[email protected]>Cc Dexuan Cui <[email protected]>Cc "H. Peter Anvin" <[email protected]>Cc Haiyang Zhang <[email protected]>Cc Heiko Carstens <[email protected]>Cc Helge Deller <[email protected]>Cc Ivan Kokshaysky <[email protected]>Cc "James E.J. Bottomley" <[email protected]>Cc James Morse <[email protected]>Cc Johannes Berg <[email protected]>Cc Juergen Gross <[email protected]>Cc "K. Y. Srinivasan" <[email protected]>Cc Mathieu Poirier <[email protected]>Cc Matt Turner <[email protected]>Cc Mauro Carvalho Chehab <[email protected]>Cc Max Filippov <[email protected]>Cc Michael Ellerman <[email protected]>Cc Paul Mackerras <[email protected]>Cc Pavel Machek <[email protected]>Cc Richard Henderson <[email protected]>Cc Richard Weinberger <[email protected]>Cc Robert Richter <[email protected]>Cc Stefano Stabellini <[email protected]>Cc Stephen Hemminger <[email protected]>Cc Sven Schnelle <[email protected]>Cc Tony Luck <[email protected]>Cc Vasily Gorbik <[email protected]>Cc Wei Liu <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>---
Notice that, with this name change, out-of-tree code that relies in the global exported "
panic_notifier_list" will fail to build. We could easily keep the retro-compatibility by making the old symbol to still exist and point to thepre_rebootlist (or even, keep the old naming).But our design choice was to allow the breakage, making users rethink their notifiers, adding them in the list that fits best. If that wasn't a good decision, we're open to change it, of course.
Thanks in advance for the review!arch/alpha/kernel/setup.c | 4 ++-- arch/parisc/kernel/pdc_chassis.c | 3 +-- arch/powerpc/kernel/setup-common.c | 2 +- arch/s390/kernel/ipl.c | 4 ++-- arch/um/drivers/mconsole_kern.c | 2 +- arch/um/kernel/um_arch.c | 2 +- arch/x86/xen/enlighten.c | 2 +- arch/xtensa/platforms/iss/setup.c | 4 ++-- drivers/char/ipmi/ipmi_msghandler.c | 12 +++++++----- drivers/edac/altera_edac.c | 3 +-- drivers/hv/vmbus_drv.c | 4 ++-- drivers/leds/trigger/ledtrig-panic.c | 3 +-- drivers/misc/ibmasm/heartbeat.c | 16 +++++++++------- drivers/net/ipa/ipa_smp2p.c | 5 ++--- drivers/parisc/power.c | 4 ++-- drivers/remoteproc/remoteproc_core.c | 6 ++++-- drivers/s390/char/con3215.c | 2 +- drivers/s390/char/con3270.c | 2 +- drivers/s390/char/sclp_con.c | 2 +- drivers/s390/char/sclp_vt220.c | 2 +- drivers/staging/olpc_dcon/olpc_dcon.c | 6 ++++-- drivers/video/fbdev/hyperv_fb.c | 4 ++-- include/linux/panic_notifier.h | 2 +- kernel/panic.c | 9 ++++----- 24 files changed, 54 insertions(+), 51 deletions(-)arch/alpha/kernel/setup.c +2
-2@@ -472,8 +472,8 @@ setup_arch(char **cmdline_p) }/* Register a call for panic conditions. */ -atomic_notifier_chain_register (&panic_notifier_list, - &alpha_panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &alpha_panic_block);#ifndef alpha_using_srm/* Assume that we've booted from SRM if we haven't booted from MILO. arch/parisc/kernel/pdc_chassis.c +1
-2@@ -22,7 +22,6 @@#include <linux/kernel.h> #include <linux/panic_notifier.h> #include <linux/reboot.h> -#include <linux/notifier.h> #include <linux/cache.h> #include <linux/proc_fs.h> #include <linux/seq_file.h> @@ -135,7 +134,7 @@ void __init parisc_pdc_chassis_init(void) PDC_CHASSIS_VER);/* initialize panic notifier chain */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &pdc_chassis_panic_block);/* initialize reboot notifier chain */ arch/powerpc/kernel/setup-common.c +1
-1@@ -762,7 +762,7 @@ void __init setup_panic(void)/* Low-level platform-specific routines that should run on panic */ if (ppc_md.panic) -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &ppc_panic_block); }arch/s390/kernel/ipl.c +2
-2@@ -2034,7 +2034,7 @@ static int on_panic_notify(struct notifier_block *self, unsigned long event, void *data) {do_panic (); -return NOTIFY_OK ; +return NOTIFY_DONE ; } static struct notifier_block on_panic_nb = {@@ -2069,7 +2069,7 @@ void __init setup_ipl(void)/* We have no info to copy */ break ; } -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb); }void s390_reset_system (void )arch/um/drivers/mconsole_kern.c +1
-1@@ -855,7 +855,7 @@ static struct notifier_block panic_exit_notifier = {static int add_notifier (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &panic_exit_notifier);return 0 ; }arch/um/kernel/um_arch.c +1
-1@@ -257,7 +257,7 @@ static struct notifier_block panic_exit_notifier = {void uml_finishsetup (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &panic_exit_notifier);uml_postsetup ();arch/x86/xen/enlighten.c +1
-1@@ -290,7 +290,7 @@ static struct notifier_block xen_panic_block = {int xen_panic_handler_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &xen_panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, &xen_panic_block);return 0 ; }arch/xtensa/platforms/iss/setup.c +2
-2@@ -13,7 +13,6 @@*/ #include <linux/init.h> #include <linux/kernel.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/printk.h> #include <linux/string.h> @@ -53,6 +52,7 @@ iss_panic_event(struct notifier_block *this, unsigned long event, void *ptr)static struct notifier_block iss_panic_block = { .notifier_call = iss_panic_event, + .priority =INT_MIN ,/* run as late as possible, may not return */ };void __init platform_setup (char **p_cmdline)@@ -81,5 +81,5 @@ void __init platform_setup(char **p_cmdline) } } -atomic_notifier_chain_register (&panic_notifier_list, &iss_panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, &iss_panic_block); }drivers/char/ipmi/ipmi_msghandler.c +7
-5@@ -25,7 +25,6 @@#include <linux/slab.h> #include <linux/ipmi.h> #include <linux/ipmi_smi.h> -#include <linux/notifier.h> #include <linux/init.h> #include <linux/proc_fs.h> #include <linux/rcupdate.h> @@ -5375,10 +5374,13 @@ static int ipmi_register_driver(void)return rv; } +/* +* we should execute this panic callback late, since it involves +* a complex call-chain and panic() runs in atomic context. +*/ static struct notifier_block panic_block = { .notifier_call = panic_event, - .next =NULL , - .priority =200 /* priority: INT_MAX >= x >= 0 */ + .priority =INT_MIN +1 , };static int ipmi_init_msghandler (void )@@ -5406,7 +5408,7 @@ static int ipmi_init_msghandler(void)timer_setup (&ipmi_timer, ipmi_timeout,0 );mod_timer (&ipmi_timer, jiffies +IPMI_TIMEOUT_JIFFIES ); -atomic_notifier_chain_register (&panic_notifier_list, &panic_block); +atomic_notifier_chain_register (&panic_pre_reboot_list, &panic_block); initialized = true;@@ -5438,7 +5440,7 @@ static void __exit cleanup_ipmi(void)if (initialized) {destroy_workqueue (remove_work_wq); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &panic_block);/* drivers/edac/altera_edac.c +1
-2@@ -16,7 +16,6 @@#include <linux/kernel.h> #include <linux/mfd/altera-sysmgr.h> #include <linux/mfd/syscon.h> -#include <linux/notifier.h> #include <linux/of_address.h> #include <linux/of_irq.h> #include <linux/of_platform.h> @@ -2163,7 +2162,7 @@ static int altr_edac_a10_probe(struct platform_device *pdev)int dberror, err_addr; edac->panic_notifier.notifier_call = s10_edac_dberr_handler; -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &edac->panic_notifier);/* Printout a message if uncorrectable error previously. */ drivers/hv/vmbus_drv.c +2
-2@@ -1622,7 +1622,7 @@ static int vmbus_bus_init(void)* Always register the vmbus unload panic notifier because we * need to shut the VMbus channel connection on panic. */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &hyperv_panic_vmbus_unload_block);vmbus_request_offers ();@@ -2851,7 +2851,7 @@ static void __exit vmbus_exit(void)* The vmbus panic notifier is always registered, hence we should * also unconditionally unregister it here as well. */ -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &hyperv_panic_vmbus_unload_block);free_page ((unsigned long )hv_panic_page);drivers/leds/trigger/ledtrig-panic.c +1
-2@@ -7,7 +7,6 @@#include <linux/kernel.h> #include <linux/init.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/leds.h> #include "../leds.h" @@ -64,7 +63,7 @@ static long led_panic_blink(int state)static int __init ledtrig_panic_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &led_trigger_panic_nb);led_trigger_register_simple ("panic" , &trigger);drivers/misc/ibmasm/heartbeat.c +9
-7@@ -8,7 +8,6 @@* Author: Max Asb?ck <[email protected]> */ -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include "ibmasm.h" #include "dot_command.h" @@ -24,7 +23,7 @@ static int suspend_heartbeats = 0;* In the case of a panic the interrupt handler continues to work and thus * continues to respond to heartbeats, making the service processor believe * the OS is still running and thus preventing a reboot. -* To prevent this from happening a callback is added the panic_notifier_list. +* To prevent this from happening a callback is added in a panic notifier list. * Before responding to a heartbeat the driver checks if a panic has happened, * if yes it suspends heartbeat, causing the service processor to reboot as * expected. @@ -32,20 +31,23 @@ static int suspend_heartbeats = 0;static int panic_happened (struct notifier_block *n,unsigned long val,void *v) { suspend_heartbeats =1 ; -return 0 ; +return NOTIFY_DONE ; } -static struct notifier_block panic_notifier = { panic_happened,NULL ,1 }; +static struct notifier_block panic_notifier = { + .notifier_call = panic_happened, +};void ibmasm_register_panic_notifier (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &panic_notifier); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &panic_notifier); }void ibmasm_unregister_panic_notifier (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, - &panic_notifier); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &panic_notifier); }drivers/net/ipa/ipa_smp2p.c +2
-3@@ -7,7 +7,6 @@#include <linux/types.h> #include <linux/device.h> #include <linux/interrupt.h> -#include <linux/notifier.h> #include <linux/panic_notifier.h> #include <linux/pm_runtime.h> #include <linux/soc/qcom/smem.h> @@ -138,13 +137,13 @@ static int ipa_smp2p_panic_notifier_register(struct ipa_smp2p *smp2p) smp2p->panic_notifier.notifier_call = ipa_smp2p_panic_notifier; smp2p->panic_notifier.priority =INT_MAX ;/* Do it early */ -return atomic_notifier_chain_register (&panic_notifier_list, +return atomic_notifier_chain_register (&panic_pre_reboot_list, &smp2p->panic_notifier); }static void ipa_smp2p_panic_notifier_unregister (struct ipa_smp2p *smp2p) { -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &smp2p->panic_notifier); }drivers/parisc/power.c +2
-2@@ -233,7 +233,7 @@ static int __init power_init(void) }/* Register a call for panic conditions. */ -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &parisc_panic_block);return 0 ;@@ -243,7 +243,7 @@ static void __exit power_exit(void) {kthread_stop (power_task); -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &parisc_panic_block);pdc_soft_power_button (0 );drivers/remoteproc/remoteproc_core.c +4
-2@@ -2795,12 +2795,14 @@ static int rproc_panic_handler(struct notifier_block *nb, unsigned long event,static void __init rproc_init_panic (void ) { rproc_panic_nb.notifier_call = rproc_panic_handler; -atomic_notifier_chain_register (&panic_notifier_list, &rproc_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &rproc_panic_nb); } static void __exit rproc_exit_panic(void) { -atomic_notifier_chain_unregister (&panic_notifier_list, &rproc_panic_nb); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &rproc_panic_nb); }static int __init remoteproc_init (void )drivers/s390/char/con3215.c +1
-1@@ -867,7 +867,7 @@ static int __init con3215_init(void) raw3215[0 ] =NULL ;return -ENODEV ; } -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&con3215);return 0 ;drivers/s390/char/con3270.c +1
-1@@ -645,7 +645,7 @@ con3270_init(void) condev->cline->len =0 ;con3270_create_status (condev); condev->input =alloc_string (&condev->freemem,80 ); -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&con3270);return 0 ;drivers/s390/char/sclp_con.c +1
-1@@ -288,7 +288,7 @@ sclp_console_init(void)timer_setup (&sclp_con_timer, sclp_console_timeout,0 );/* enable printk-access to this driver */ -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&sclp_console);return 0 ;drivers/s390/char/sclp_vt220.c +1
-1@@ -838,7 +838,7 @@ sclp_vt220_con_init(void)if (rc)return rc;/* Attach linux console */ -atomic_notifier_chain_register (&panic_notifier_list, &on_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, &on_panic_nb);register_reboot_notifier (&on_reboot_nb);register_console (&sclp_vt220_console);return 0 ;drivers/staging/olpc_dcon/olpc_dcon.c +4
-2@@ -653,7 +653,8 @@ static int dcon_probe(struct i2c_client *client, const struct i2c_device_id *id) }register_reboot_notifier (&dcon->reboot_nb); -atomic_notifier_chain_register (&panic_notifier_list, &dcon_panic_nb); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &dcon_panic_nb);return 0 ;@@ -676,7 +677,8 @@ static int dcon_remove(struct i2c_client *client)struct dcon_priv *dcon =i2c_get_clientdata (client);unregister_reboot_notifier (&dcon->reboot_nb); -atomic_notifier_chain_unregister (&panic_notifier_list, &dcon_panic_nb); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &dcon_panic_nb);free_irq (DCON_IRQ , dcon);drivers/video/fbdev/hyperv_fb.c +2
-2@@ -1253,7 +1253,7 @@ static int hvfb_probe(struct hv_device *hdev,*/ par->hvfb_panic_nb.notifier_call = hvfb_on_panic; par->hvfb_panic_nb.priority =INT_MIN +10 , -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &par->hvfb_panic_nb);return 0 ;@@ -1276,7 +1276,7 @@ static int hvfb_remove(struct hv_device *hdev)struct fb_info *info =hv_get_drvdata (hdev);struct hvfb_par *par = info->par; -atomic_notifier_chain_unregister (&panic_notifier_list, +atomic_notifier_chain_unregister (&panic_pre_reboot_list, &par->hvfb_panic_nb); par->update = false;include/linux/panic_notifier.h +1
-1@@ -5,9 +5,9 @@#include <linux/notifier.h> #include <linux/types.h> -extern struct atomic_notifier_head panic_notifier_list;extern struct atomic_notifier_head panic_hypervisor_list;extern struct atomic_notifier_head panic_info_list; +extern struct atomic_notifier_head panic_pre_reboot_list;extern bool crash_kexec_post_notifiers;kernel/panic.c +4
-5@@ -69,16 +69,15 @@ EXPORT_SYMBOL_GPL(panic_timeout);#define PANIC_PRINT_ALL_CPU_BT 0x00000040unsigned long panic_print; -ATOMIC_NOTIFIER_HEAD (panic_notifier_list); - -EXPORT_SYMBOL (panic_notifier_list); -ATOMIC_NOTIFIER_HEAD (panic_hypervisor_list);EXPORT_SYMBOL (panic_hypervisor_list);ATOMIC_NOTIFIER_HEAD (panic_info_list);EXPORT_SYMBOL (panic_info_list); +ATOMIC_NOTIFIER_HEAD (panic_pre_reboot_list); +EXPORT_SYMBOL (panic_pre_reboot_list); +static long no_blink (int state) {return 0 ;@@ -295,7 +294,7 @@ void panic(const char *fmt, ...)*/ atomic_notifier_call_chain (&panic_hypervisor_list,PANIC_NOTIFIER , buf);atomic_notifier_call_chain (&panic_info_list,PANIC_NOTIFIER , buf); -atomic_notifier_call_chain (&panic_notifier_list,PANIC_NOTIFIER , buf); +atomic_notifier_call_chain (&panic_pre_reboot_list,PANIC_NOTIFIER , buf);panic_print_sys_info (false);Signature · 1 line
2.36.0
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 5 lines · Corey Minyard
[...]
For the IPMI portion:
Acked-by Corey Minyard <[email protected]>
Thanks Alex and Corey for the ACKs!
Quoted 6 lines
Note that the IPMI
panic_event()should always return, but it may take some time, especially if the IPMI controller is no longer functional. So the risk of a long delay is there and it makes sense to move it very late.
Thanks, I agree - the patch moves it to the (latest - 1) position, since some arch code might run as the latest and effectively stops the machine. Cheers,
Guilherme
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
In reply to Guilherme G. PiccoliQuoted 17 lines · Guilherme G. Piccoli
This patch renames the
panic_notifier_listtopanic_pre_reboot_list; the idea is that a subsequent patch will refactor the panic path in order to better split the notifiers, running some of them very early, some of them not so early [but still beforekmsg_dump()] and finally, the rest should execute late, after kdump. The latter ones are now in the panic pre-reboot list - the name comes from the idea that these notifiers execute beforepanic()attempts rebooting the machine (if that option is set).We also took the opportunity to clean-up useless header inclusions, improve some notifier block declarations (e.g. in
ibmasm/heartbeat.c) and more important, change some priorities - we hereby set 2 notifiers to run late in the list [iss_panic_event()and the IPMIpanic_event()] due to the risks they offer (may not return, for example). Proper documentation is going to be provided in a subsequent patch, that effectively refactors the panic path.
[...]
Quoted 1 line
arch/xtensa/platforms/iss/setup.c| 4 ++--For xtensa:
For xtensa:
Signature · 2 lines
Thanks. -- Max
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 6 lines · Max Filippov
[...]
Quoted 1 line
arch/xtensa/platforms/iss/setup.c| 4 ++--For xtensa:For xtensa:
Acked-by Max Filippov <[email protected]>
Perfect, thanks Max!
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
In reply to Guilherme G. PiccoliQuoted 20 lines · Guilherme G. Piccoli
This patch renames the
panic_notifier_listtopanic_pre_reboot_list; the idea is that a subsequent patch will refactor the panic path in order to better split the notifiers, running some of them very early, some of them not so early [but still beforekmsg_dump()] and finally, the rest should execute late, after kdump. The latter ones are now in the panic pre-reboot list - the name comes from the idea that these notifiers execute beforepanic()attempts rebooting the machine (if that option is set).We also took the opportunity to clean-up useless header inclusions, improve some notifier block declarations (e.g. in
ibmasm/heartbeat.c) and more important, change some priorities - we hereby set 2 notifiers to run late in the list [iss_panic_event()and the IPMIpanic_event()] due to the risks they offer (may not return, for example). Proper documentation is going to be provided in a subsequent patch, that effectively refactors the panic path.
drivers/edac/altera_edac.c static int altr_edac_a10_probe(struct platform_device *pdev) · line 2165
Quoted 5 lines
drivers/edac/altera_edac.c +1
-1@@ -2163,7 +2162,7 @@ static int altr_edac_a10_probe(struct platform_device *pdev)int dberror, err_addr; edac->panic_notifier.notifier_call = s10_edac_dberr_handler; - atomic_notifier_chain_register(&panic_notifier_list, + atomic_notifier_chain_register(&panic_pre_reboot_list,
My understanding is that this notifier first prints info about ECC errors and then triggers reboot. It might make sense to split it into two notifiers.
drivers/leds/trigger/ledtrig-panic.c static long led_panic_blink(int state) · line 67
Quoted 12 lines
drivers/edac/altera_edac.c +0
-0&edac->panic_notifier);/* Printout a message if uncorrectable error previously. */ drivers/leds/trigger/ledtrig-panic.c +1
-1@@ -64,7 +63,7 @@ static long led_panic_blink(int state)static int __init ledtrig_panic_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &led_trigger_panic_nb);
Blinking => should go to the last "post_reboot/loop" list.
Quoted 12 lines
drivers/leds/trigger/ledtrig-panic.c +0
-0led_trigger_register_simple ("panic" , &trigger);drivers/misc/ibmasm/heartbeat.c +1
-1@@ -32,20 +31,23 @@ static int suspend_heartbeats = 0;static int panic_happened (struct notifier_block *n,unsigned long val,void *v) { suspend_heartbeats =1 ; -return 0 ; +return NOTIFY_DONE ; }
drivers/misc/ibmasm/heartbeat.c static int suspend_heartbeats = 0; · line 44
Quoted 10 lines
drivers/misc/ibmasm/heartbeat.c +5
-2-static struct notifier_block panic_notifier = { panic_happened,NULL ,1 }; +static struct notifier_block panic_notifier = { + .notifier_call = panic_happened, +};void ibmasm_register_panic_notifier (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &panic_notifier); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &panic_notifier);
Same here. Blinking => should go to the last "post_reboot/loop" list.
drivers/misc/ibmasm/heartbeat.c static int suspend_heartbeats = 0; · line 51
Quoted 9 lines
drivers/misc/ibmasm/heartbeat.c +2
-2}void ibmasm_unregister_panic_notifier (void ) { -atomic_notifier_chain_unregister (&panic_notifier_list, - &panic_notifier); +atomic_notifier_chain_unregister (&panic_pre_reboot_list, + &panic_notifier); }
The rest of the moved notifiers seem to fit well this "pre_reboot" list.
Best Regards,
Petr
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Thanks again for the review! Comments inline below:
Quoted 6 lines · Petr Mladek
[...]
Quoted 5 lines
drivers/edac/altera_edac.c +0
-0@@ -2163,7 +2162,7 @@ static int altr_edac_a10_probe(struct platform_device *pdev)int dberror, err_addr;
Quoted 7 lines
drivers/edac/altera_edac.c static int altr_edac_a10_probe(struct platform_device *pdev) · line 2165
Quoted 3 lines
drivers/edac/altera_edac.c +1
-1edac->panic_notifier.notifier_call = s10_edac_dberr_handler; - atomic_notifier_chain_register(&panic_notifier_list, + atomic_notifier_chain_register(&panic_pre_reboot_list,My understanding is that this notifier first prints info about ECC errors and then triggers reboot. It might make sense to split it into two notifiers.
I disagree here - looping the maintainers for comments (CCing Dinh / Tony). BTW, sorry for not having you on CC already Dinh, it was my mistake.
So, my reasoning here is: this notifier should fit the info list, definitely! But...it's very high risk for kdump. It deep dives into the regmap API (there are locks in such code) plus there is an (MM)IO write to the device and an ARM firmware call. So, despite the nature of this notifier _fits the informational list_, the _code is risky_ so we should avoid running it before a kdump.
Now, we indeed have a chicken/egg problem: want to avoid it before kdump, BUT in case kdump is not set, kmsg_dump() (and console flushing, after your suggestion Petr) will run before it and not save collected information from EDAC PoV.
My idea: I could call a second kmsg_dump() or at least a panic console flush for within such notifier. Let me know what you think Petr (also Dinh / Tony and all interested parties).
Quoted 28 lines
[...]
drivers/leds/trigger/ledtrig-panic.c static long led_panic_blink(int state) · line 67
Quoted 9 lines
drivers/leds/trigger/ledtrig-panic.c +1
-1@@ -64,7 +63,7 @@ static long led_panic_blink(int state)static int __init ledtrig_panic_init (void ) { -atomic_notifier_chain_register (&panic_notifier_list, +atomic_notifier_chain_register (&panic_pre_reboot_list, &led_trigger_panic_nb);Blinking => should go to the last "post_reboot/loop" list. [...]
Quoted 15 lines
drivers/misc/ibmasm/heartbeat.c +4
-2@@ -32,20 +31,23 @@ static int suspend_heartbeats = 0;static int panic_happened (struct notifier_block *n,unsigned long val,void *v) { suspend_heartbeats =1 ; -return 0 ; +return NOTIFY_DONE ; } -static struct notifier_block panic_notifier = { panic_happened,NULL ,1 }; +static struct notifier_block panic_notifier = { + .notifier_call = panic_happened, +};
Quoted 7 lines
drivers/misc/ibmasm/heartbeat.c static int suspend_heartbeats = 0; · line 44
Quoted 5 lines
drivers/misc/ibmasm/heartbeat.c +2
-1void ibmasm_register_panic_notifier (void ) { -atomic_notifier_chain_register (&panic_notifier_list, &panic_notifier); +atomic_notifier_chain_register (&panic_pre_reboot_list, + &panic_notifier);Same here. Blinking => should go to the last "post_reboot/loop" list.
Ack on both.
IBMasm is not blinking IIUC, but still fits properly the loop list. This notifier would make a heartbeat mechanism stop, and once it's stopped, service processor is allowed to reboot - that's my understanding.
Cheers,
Guilherme
Luck, Tony
maintainerRE: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 11 lines
So, my reasoning here is: this notifier should fit the info list, definitely! But...it's very high risk for kdump. It deep dives into the regmap API (there are locks in such code) plus there is an (MM)IO write to the device and an ARM firmware call. So, despite the nature of this notifier
_fitsthe informationallist_, the_codeisrisky_so we should avoid running it before a kdump.Now, we indeed have a chicken/egg problem: want to avoid it before kdump, BUT in case kdump is not set,
kmsg_dump()(and console flushing, after your suggestion Petr) will run before it and not save collected information from EDAC PoV.
Would it be possible to have some global "kdump is configured + enabled" flag?
Then notifiers could make an informed choice on whether to deep dive to get all the possible details (when there is no kdump) or just skim the high level stuff (to maximize chance of getting a successful kdump).
-Tony
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 8 lines · Tony
Quoted 1 line
[...]
Would it be possible to have some global "kdump is configured + enabled" flag?
Then notifiers could make an informed choice on whether to deep dive to get all the possible details (when there is no kdump) or just skim the high level stuff (to maximize chance of getting a successful kdump).
-Tony
Good idea Tony! What if I wire a kexec_crash_loaded() in the notifier?
With that, are you/Petr/Dinh OK in moving it for the info list? Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 11 lines · Guilherme G. Piccoli
Quoted 8 lines · Tony
Quoted 1 line
[...]
Would it be possible to have some global "kdump is configured + enabled" flag?
Then notifiers could make an informed choice on whether to deep dive to get all the possible details (when there is no kdump) or just skim the high level stuff (to maximize chance of getting a successful kdump).
-Tony
Good idea Tony! What if I wire a
kexec_crash_loaded()in the notifier?
I like this idea.
One small problem is that kexec_crash_loaded() has valid result only under kexec_mutex. On the other hand, it should stay true once loaded so that the small race window should be innocent.
Quoted 1 line
With that, are you/Petr/Dinh OK in moving it for the info list?
Sounds good to me.
Best Regards,
Petr
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 15 lines · Petr Mladek
[...]
Quoted 7 lines
Quoted 5 lines
Then notifiers could make an informed choice on whether to deep dive to get all the possible details (when there is no kdump) or just skim the high level stuff (to maximize chance of getting a successful kdump).
-Tony
Good idea Tony! What if I wire a
kexec_crash_loaded()in the notifier?I like this idea.
One small problem is that
kexec_crash_loaded()has valid result only underkexec_mutex. On the other hand, it should stay true once loaded so that the small race window should be innocent.
Quoted 6 lines
Quoted 1 line
With that, are you/Petr/Dinh OK in moving it for the info list?
Sounds good to me.
Best Regards,
Petr
Perfect, I'll do that for V2 then =)
Tony / Dinh - can I just *skip* this notifier *if kdump* is set or else we run the code as-is? Does that make sense to you?
I'll postpone it to run almost in the end of info list (last position is for panic_print).
Thanks,
Guilherme
Luck, Tony
maintainerRE: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 2 lines
Tony / Dinh - can I just *skip* this notifier *if kdump* is set or else we run the code as-is? Does that make sense to you?
The "skip" option sounds like it needs some special flag associated with an entry on the notifier chain. But there are other notifier chains ... so that sounds messy to me.
Just all the notifiers in priority order. If any want to take different actions based on kdump status, change the code. That seems more flexible than an "all or nothing" approach by skipping.
-Tony
Re: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 7 lines · Tony
Quoted 2 lines
Tony / Dinh - can I just *skip* this notifier *if kdump* is set or else we run the code as-is? Does that make sense to you?
The "skip" option sounds like it needs some special flag associated with an entry on the notifier chain. But there are other notifier chains ... so that sounds messy to me.
Quoted 5 lines
Just all the notifiers in priority order. If any want to take different actions based on kdump status, change the code. That seems more flexible than an "all or nothing" approach by skipping.
-Tony
I guess I've expressed myself in a poor way - sorry!
What I'm planning to do in the altera_edac notifier is:
if (kdump_is_set)
return;
/* regular code */
In other words: if the kdump is set, this notifier will be effectively a nop (although it's gonna be called).
Lemme know your thoughts Tony, if that makes sense. Thanks,
Guilherme
Luck, Tony
maintainerRE: [PATCH 21/30] panic: Introduce the panic pre-reboot notifier list
Quoted 4 lines
What I'm planning to do in the
altera_edacnotifier is:if (
kdump_is_set)
return;
Yes. That's what I think should happen.
-Tony
PATCH 22/30 panic: Introduce the panic post-reboot notifier list
In reply to Guilherme G. PiccoliCurrently we have 3 notifier lists in the panic path, which will be wired in a way to allow the notifier callbacks to run in different moments at panic time, in a subsequent patch.
But there is also an odd set of architecture calls hardcoded in the end of panic path, after the restart machinery. They're responsible for late time tunings / events, like enabling a stop button (Sparc) or effectively stopping the machine (s390).
This patch introduces yet another notifier list to offer the architectures a way to add callbacks in such late moment on panic path without the need of ifdefs / hardcoded approaches.
--- arch/s390/kernel/setup.c | 19 ++++++++++++++++++- arch/sparc/kernel/setup_32.c | 27 +++++++++++++++++++++++---- arch/sparc/kernel/setup_64.c | 29 ++++++++++++++++++++++++----- include/linux/panic_notifier.h | 1 + kernel/panic.c | 19 +++++++------------ 5 files changed, 73 insertions(+), 22 deletions(-)
arch/s390/kernel/setup.c +18 -1
@@ -39,7 +39,6 @@#include <linux/kernel_stat.h> #include <linux/dma-map-ops.h> #include <linux/device.h> -#include <linux/notifier.h> #include <linux/pfn.h> #include <linux/ctype.h> #include <linux/reboot.h>
@@ -51,6 +50,7 @@#include <linux/start_kernel.h> #include <linux/hugetlb.h> #include <linux/kmemleak.h> +#include <linux/panic_notifier.h> #include <asm/boot_data.h> #include <asm/ipl.h>
@@ -943,6 +943,20 @@ static void __init log_component_list(void) } } +/* +* The following notifier executes as one of the latest things in the panic +* path, only if the restart routines weren't executed (or didn't succeed). +*/ +static int panic_event (struct notifier_block *n,unsigned long ev,void *unused) +{ +disabled_wait (); +return NOTIFY_DONE ; +} + +static struct notifier_block post_reboot_panic_block = { + .notifier_call = panic_event, +}; +/* * Setup function called from init/main.c just after the banner * was printed.
@@ -1058,4 +1072,7 @@ void __init setup_arch(char **cmdline_p)/* Add system specific data to the random pool */ setup_randomness (); + +atomic_notifier_chain_register (&panic_post_reboot_list, + &post_reboot_panic_block); }
arch/sparc/kernel/setup_32.c +23 -4
@@ -34,6 +34,7 @@#include <linux/kdebug.h> #include <linux/export.h> #include <linux/start_kernel.h> +#include <linux/panic_notifier.h> #include <uapi/linux/mount.h> #include <asm/io.h>
@@ -51,6 +52,7 @@#include "kernel.h" +int stop_a_enabled =1 ;struct screen_info screen_info = {0 ,0 ,/* orig-x, orig-y */ 0 ,/* unused */
@@ -293,6 +295,24 @@ void __init sparc32_start_kernel(struct linux_romvec *rp)start_kernel (); } +/* +* The following notifier executes as one of the latest things in the panic +* path, only if the restart routines weren't executed (or didn't succeed). +*/ +static int panic_event (struct notifier_block *n,unsigned long ev,void *unused) +{ +/* Make sure the user can actually press Stop-A (L1-A) */ + stop_a_enabled =1 ; +pr_emerg ("Press Stop-A (L1-A) from sun keyboard or send break\n" +"twice on console to return to the boot prom\n" ); + +return NOTIFY_DONE ; +} + +static struct notifier_block post_reboot_panic_block = { + .notifier_call = panic_event, +}; +void __init setup_arch (char **cmdline_p) {int i;
@@ -368,9 +388,10 @@ void __init setup_arch(char **cmdline_p)paging_init ();smp_setup_cpu_possible_map (); -} -extern int stop_a_enabled; +atomic_notifier_chain_register (&panic_post_reboot_list, + &post_reboot_panic_block); +}void sun_do_break (void ) {
@@ -384,8 +405,6 @@ void sun_do_break(void) }EXPORT_SYMBOL (sun_do_break); -int stop_a_enabled =1 ; -static int __init topology_init (void ) {int i, ncpus, err;
arch/sparc/kernel/setup_64.c +24 -5
@@ -33,6 +33,7 @@#include <linux/module.h> #include <linux/start_kernel.h> #include <linux/memblock.h> +#include <linux/panic_notifier.h> #include <uapi/linux/mount.h> #include <asm/io.h>
@@ -62,6 +63,8 @@#include "entry.h" #include "kernel.h" +int stop_a_enabled =1 ; +/* Used to synchronize accesses to NatSemi SUPER I/O chip configure * operations in asm/ns87303.h */
@@ -632,6 +635,24 @@ void __init alloc_irqstack_bootmem(void) } } +/* +* The following notifier executes as one of the latest things in the panic +* path, only if the restart routines weren't executed (or didn't succeed). +*/ +static int panic_event (struct notifier_block *n,unsigned long ev,void *unused) +{ +/* Make sure the user can actually press Stop-A (L1-A) */ + stop_a_enabled =1 ; +pr_emerg ("Press Stop-A (L1-A) from sun keyboard or send break\n" +"twice on console to return to the boot prom\n" ); + +return NOTIFY_DONE ; +} + +static struct notifier_block post_reboot_panic_block = { + .notifier_call = panic_event, +}; +void __init setup_arch (char **cmdline_p) {/* Initialize PROM console and command line. */
@@ -691,9 +712,10 @@ void __init setup_arch(char **cmdline_p)* allocate the IRQ stacks. */ alloc_irqstack_bootmem (); -} -extern int stop_a_enabled; +atomic_notifier_chain_register (&panic_post_reboot_list, + &post_reboot_panic_block); +}void sun_do_break (void ) {
@@ -706,6 +728,3 @@ void sun_do_break(void)prom_cmdline (); }EXPORT_SYMBOL (sun_do_break); - -int stop_a_enabled =1 ; -EXPORT_SYMBOL (stop_a_enabled);
include/linux/panic_notifier.h +1 -0
@@ -8,6 +8,7 @@extern struct atomic_notifier_head panic_hypervisor_list;extern struct atomic_notifier_head panic_info_list;extern struct atomic_notifier_head panic_pre_reboot_list; +extern struct atomic_notifier_head panic_post_reboot_list;extern bool crash_kexec_post_notifiers;
kernel/panic.c +7 -12
@@ -78,6 +78,9 @@ EXPORT_SYMBOL(panic_info_list);ATOMIC_NOTIFIER_HEAD (panic_pre_reboot_list);EXPORT_SYMBOL (panic_pre_reboot_list); +ATOMIC_NOTIFIER_HEAD (panic_post_reboot_list); +EXPORT_SYMBOL (panic_post_reboot_list); +static long no_blink (int state) {return 0 ;
@@ -359,18 +362,10 @@ void panic(const char *fmt, ...) reboot_mode = panic_reboot_mode;emergency_restart (); } -#ifdef __sparc__ - { -extern int stop_a_enabled; -/* Make sure the user can actually press Stop-A (L1-A) */ - stop_a_enabled =1 ; -pr_emerg ("Press Stop-A (L1-A) from sun keyboard or send break\n" -"twice on console to return to the boot prom\n" ); - } -#endif -#if defined(CONFIG_S390 ) -disabled_wait (); -#endif + +atomic_notifier_call_chain (&panic_post_reboot_list, +PANIC_NOTIFIER , buf); +pr_emerg ("---[ end Kernel panic - not syncing: %s ]---\n" , buf);/* Do not scroll important messages printed above */
Signature · 1 line
2.36.0
Re: [PATCH 22/30] panic: Introduce the panic post-reboot notifier list
Quoted 13 lines · Guilherme G. Piccoli
Currently we have 3 notifier lists in the panic path, which will be wired in a way to allow the notifier callbacks to run in different moments at panic time, in a subsequent patch.
But there is also an odd set of architecture calls hardcoded in the end of panic path, after the restart machinery. They're responsible for late time tunings / events, like enabling a stop button (Sparc) or effectively stopping the machine (s390).
This patch introduces yet another notifier list to offer the architectures a way to add callbacks in such late moment on panic path without the need of ifdefs / hardcoded approaches.
Quoted 7 lines
Cc Alexander Gordeev <[email protected]>Cc Christian Borntraeger <[email protected]>Cc "David S. Miller" <[email protected]>Cc Heiko Carstens <[email protected]>Cc Sven Schnelle <[email protected]>Cc Vasily Gorbik <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Hey S390/SPARC folks, sorry for the ping!
Any reviews on this V1 would be greatly appreciated, I'm working on V2 and seeking feedback in the non-reviewed patches.
Thanks in advance,
Guilherme
Re: [PATCH 22/30] panic: Introduce the panic post-reboot notifier list
Quoted 22 lines · Guilherme G. Piccoli
Quoted 13 lines · Guilherme G. Piccoli
Currently we have 3 notifier lists in the panic path, which will be wired in a way to allow the notifier callbacks to run in different moments at panic time, in a subsequent patch.
But there is also an odd set of architecture calls hardcoded in the end of panic path, after the restart machinery. They're responsible for late time tunings / events, like enabling a stop button (Sparc) or effectively stopping the machine (s390).
This patch introduces yet another notifier list to offer the architectures a way to add callbacks in such late moment on panic path without the need of ifdefs / hardcoded approaches.
Quoted 7 lines
Cc Alexander Gordeev <[email protected]>Cc Christian Borntraeger <[email protected]>Cc "David S. Miller" <[email protected]>Cc Heiko Carstens <[email protected]>Cc Sven Schnelle <[email protected]>Cc Vasily Gorbik <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>
Quoted 4 lines
Hey S390/SPARC folks, sorry for the ping!
Any reviews on this V1 would be greatly appreciated, I'm working on V2 and seeking feedback in the non-reviewed patches.
Sorry, missed that this is quite s390 specific. So, yes, this looks good to me and nice to see that one of the remaining CONFIG_S390 in common code will be removed!
For the s390 bits:
Re: [PATCH 22/30] panic: Introduce the panic post-reboot notifier list
Quoted 7 lines · Heiko Carstens
[...]
Quoted 5 lines
Hey S390/SPARC folks, sorry for the ping!
Any reviews on this V1 would be greatly appreciated, I'm working on V2 and seeking feedback in the non-reviewed patches.
Quoted 6 lines
Sorry, missed that this is quite s390 specific. So, yes, this looks good to me and nice to see that one of the remaining
CONFIG_S390in common code will be removed!For the s390 bits:
Acked-by Heiko Carstens <[email protected]>
No need for apologies, I really appreciate your review! Will add your Ack for V2 =)
Cheers,
Guilherme
Petr Mladek
maintainerRe: [PATCH 22/30] panic: Introduce the panic post-reboot notifier list
In reply to Guilherme G. PiccoliQuoted 4 lines · Guilherme G. Piccoli
Currently we have 3 notifier lists in the panic path, which will be wired in a way to allow the notifier callbacks to run in different moments at panic time, in a subsequent patch.
Quoted 8 lines
But there is also an odd set of architecture calls hardcoded in the end of panic path, after the restart machinery. They're responsible for late time tunings / events, like enabling a stop button (Sparc) or effectively stopping the machine (s390).
This patch introduces yet another notifier list to offer the architectures a way to add callbacks in such late moment on panic path without the need of ifdefs / hardcoded approaches.
The patch looks good to me. I would just suggest two changes.
1. I would rename the list to "panic_loop_list" instead of "panic_post_reboot_list".
It will be more clear that it includes things that are needed before panic() enters the infinite loop.
2. I would move all the notifiers that enable blinking here.
The blinking should be done only during the infinite loop when there is nothing else to do. If we enable earlier then it might disturb/break more important functionality (dumping information, reboot).
Best Regards,
Petr
Re: [PATCH 22/30] panic: Introduce the panic post-reboot notifier list
Quoted 11 lines · Petr Mladek
[...]
The patch looks good to me. I would just suggest two changes.
1. I would rename the list to "
panic_loop_list" instead of "panic_post_reboot_list".It will be more clear that it includes things that are needed before
panic()enters the infinite loop.
Quoted 7 lines
2. I would move all the notifiers that enable blinking here.
The blinking should be done only during the infinite loop when there is nothing else to do. If we enable earlier then it might disturb/break more important functionality (dumping information, reboot).
Perfect, I agree with you. I'll change both points in V2 =)
PATCH 23/30 printk: kmsg_dump: Introduce helper to inform number of dumpers
In reply to Guilherme G. PiccoliCurrently we don't have a way to check if there are dumpers set, except counting the list members maybe. This patch introduces a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. It's going to be used on the panic path in the subsequent patch.
Notice that the spinlock guarding kmsg_dumpers list also guards increment/decrement of the dumper's counter, but there's no need for that when reading the counter in the panic path, since that is an atomic path and there's no other (planned) user.
--- include/linux/kmsg_dump.h | 7 +++++++ kernel/printk/printk.c | 14 ++++++++++++++ 2 files changed, 21 insertions(+)
include/linux/kmsg_dump.h +7 -0
@@ -65,6 +65,8 @@ bool kmsg_dump_get_buffer(struct kmsg_dump_iter *iter, bool syslog,void kmsg_dump_rewind (struct kmsg_dump_iter *iter); +bool kmsg_has_dumpers (void ); +int kmsg_dump_register (struct kmsg_dumper *dumper);int kmsg_dump_unregister (struct kmsg_dumper *dumper);
@@ -91,6 +93,11 @@ static inline void kmsg_dump_rewind(struct kmsg_dump_iter *iter) { } +static inline bool kmsg_has_dumpers (void ) +{ +return false; +} +static inline int kmsg_dump_register (struct kmsg_dumper *dumper) {return -EINVAL ;
kernel/printk/printk.c +14 -0
@@ -3399,6 +3399,18 @@ EXPORT_SYMBOL(printk_timed_ratelimit);static DEFINE_SPINLOCK (dump_list_lock);static LIST_HEAD (dump_list); +static int num_dumpers; + +/** +* kmsg_has_dumpers - inform if there is any kmsg dumper registered. +* +* Returns true if there's at least one registered dumper, or false +* if otherwise. +*/ +bool kmsg_has_dumpers (void ) +{ +return num_dumpers ? true : false; +}/** * kmsg_dump_register - register a kernel log dumper.
@@ -3423,6 +3435,7 @@ int kmsg_dump_register(struct kmsg_dumper *dumper) dumper->registered =1 ;list_add_tail_rcu (&dumper->list, &dump_list); err =0 ; + num_dumpers++; }spin_unlock_irqrestore (&dump_list_lock, flags);
@@ -3447,6 +3460,7 @@ int kmsg_dump_unregister(struct kmsg_dumper *dumper) dumper->registered =0 ;list_del_rcu (&dumper->list); err =0 ; + num_dumpers--; }spin_unlock_irqrestore (&dump_list_lock, flags);synchronize_rcu ();
Signature · 1 line
2.36.0
Steven Rostedt
maintainerRe: [PATCH 23/30] printk: kmsg_dump: Introduce helper to inform number of dumpers
Quoted 5 lines · On Wed, 27 Apr 2022 19:49:17 -0300 "Guilherme G. Piccoli"…
Currently we don't have a way to check if there are dumpers set, except counting the list members maybe. This patch introduces a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. It's going to be used on the panic path in the subsequent patch.
FYI, it is considered "bad form" to reference in the change log "this patch". We know this is a patch. The change log should just talk about what is being done. So can you reword your change logs (you do this is almost every patch). Here's what I would reword the above to be:
Currently we don't have a way to check if there are dumpers set, except perhaps by counting the list members. Introduce a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. This will simplify the refactoring of the panic path.
-- Steve
Re: [PATCH 23/30] printk: kmsg_dump: Introduce helper to inform number of dumpers
Quoted 14 lines · Steven Rostedt
Quoted 5 lines · On Wed, 27 Apr 2022 19:49:17 -0300 "Guilherme G. Piccoli"…
Currently we don't have a way to check if there are dumpers set, except counting the list members maybe. This patch introduces a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. It's going to be used on the panic path in the subsequent patch.
FYI, it is considered "bad form" to reference in the change log "this patch". We know this is a patch. The change log should just talk about what is being done. So can you reword your change logs (you do this is almost every patch). Here's what I would reword the above to be:
Quoted 4 lines
Currently we don't have a way to check if there are dumpers set, except perhaps by counting the list members. Introduce a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. This will simplify the refactoring of the panic path.
Thanks for the hint, you're right - it's almost in all of my patches. I'll reword all of them (except the ones already merged) to remove this "bad form".
Cheers,
Guilherme
Re: [PATCH 23/30] printk: kmsg_dump: Introduce helper to inform number of dumpers
Quoted 15 lines · Guilherme G. Piccoli
Quoted 14 lines · Steven Rostedt
Quoted 5 lines · On Wed, 27 Apr 2022 19:49:17 -0300 "Guilherme G. Piccoli"…
Currently we don't have a way to check if there are dumpers set, except counting the list members maybe. This patch introduces a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. It's going to be used on the panic path in the subsequent patch.
FYI, it is considered "bad form" to reference in the change log "this patch". We know this is a patch. The change log should just talk about what is being done. So can you reword your change logs (you do this is almost every patch). Here's what I would reword the above to be:
Quoted 8 lines
Quoted 4 lines
Currently we don't have a way to check if there are dumpers set, except perhaps by counting the list members. Introduce a very simple helper to provide this information, by just keeping track of registered/unregistered kmsg dumpers. This will simplify the refactoring of the panic path.
Thanks for the hint, you're right - it's almost in all of my patches. I'll reword all of them (except the ones already merged) to remove this "bad form".
Shame on me that I do not care that much about the style of the commit message :-)
Anyway, the code looks good to me. With the better commit message:
Best Regards,
Petr
Re: [PATCH 23/30] printk: kmsg_dump: Introduce helper to inform number of dumpers
Quoted 8 lines · Petr Mladek
[...]
Shame on me that I do not care that much about the style of the commit message :-)Anyway, the code looks good to me. With the better commit message:
Reviewed-by Petr Mladek <[email protected]>
Heheh, cool - I'll add your tag and improve the message in V2. Thanks,
Guilherme
PATCH 24/30 panic: Refactor the panic path
In reply to Guilherme G. PiccoliThe panic() function is somewhat convoluted - a lot of changes were made over the years, adding comments that might be misleading/outdated now, it has a code structure that is a bit complex to follow, with lots of conditionals, for example. The panic notifier list is something else - a single list, with multiple callbacks of different purposes, that run in a non-deterministic order and may affect hardly kdump reliability - see the "crash_kexec_post_notifiers" workaround-ish flag.
This patch proposes a major refactor on the panic path based on Petr's idea [0] - basically we split the notifiers list in three, having a set of different call points in the panic path. Below a list of changes proposed in this patch, culminating in the panic notifiers level concept:
(a) First of all, we improved comments all over the function and removed useless variables / includes. Also, as part of this clean-up we concentrate the console flushing functions in a helper.
(b) As mentioned before, there is a split of the panic notifier list in three, based on the purpose of the callback. The code contains good documentation in form of comments, but a summary of the three lists follows:
- the hypervisor list aims low-risk procedures to inform hypervisors or firmware about the panic event, also includes LED-related functions;
- the informational list contains callbacks that provide more details, like kernel offset or trace dump (if enabled) and also includes the callbacks aimed at reducing log pollution or warns, like the RCU and hung task disable callbacks;
- finally, the pre_reboot list is the old notifier list renamed, containing the more risky callbacks that didn't fit the previous lists. There is also a 4th list (the post_reboot one), but it's not related with the original list - it contains late time architecture callbacks aimed at stopping the machine, for example.
The 3 notifiers lists execute in different moments, hypervisor being the first, followed by informational and finally the pre_reboot list.
(c) But then, there is the ordering problem of the notifiers against the crash_kernel() call - kdump must be as reliable as possible. For that, a simple binary "switch" as "crash_kexec_post_notifiers" is not enough, hence we introduce here concept of panic notifier
levels: there are 5 levels, from 0 (no notifier executes before kdump) until 4 (all notifiers run before kdump); the default level is 2, in which the hypervisor and (iff we have any kmsg dumper) the informational notifiers execute before kdump.
The detailed documentation of the levels is present in code comments and in the kernel-parameters.txt file; as an analogy with the previous panic() implementation, the level 0 is exactly the same as the old behavior of notifiers, running all after kdump, and the level 4 is the same as "crash_kexec_post_notifiers=Y" (we kept this parameter as a deprecated one).
(d) Finally, an important change made here: we now use only the function "crash_smp_send_stop()" to shut all the secondary CPUs in the panic path. Before, there was a case of using the regular "smp_send_stop()", but the better approach is to simplify the code and try to use the function which was created exclusively for the panic path. Experiments showed that it works fine, and code was very simplified with that.
Functional change is expected from this refactor, since now we call some notifiers by default before kdump, but the goal here besides code clean-up is to have a better panic path, more reliable and deterministic, but also very customizable.
[0] https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/
---
Special thanks to Petr and Baoquan for the suggestion and feedback in a previous email thread. There's some important design decisions that worth mentioning and discussing:
* The default panic notifiers level is 2, based on Petr Mladek's suggestion, which makes a lot of sense. Of course, this is customizable through the parameter, but would be something worthwhile to have a KConfig option to set the default level? It would help distros that want the old behavior (no notifiers before kdump) as default.
* The implementation choice was to _avoid_ intricate if conditionals in the panic path, which would _definitely_ be present with the panic notifiers levels idea; so, instead of lots of if conditionals, the set/clear bits approach with functions called in 2 points (but executing only in one of them) is much easier to follow an was used here; the ordering helper function and the comments also help a lot to avoid confusion (hopefully).
* Choice was to *always* use crash_smp_send_stop() instead of sometimes making use of the regular smp_send_stop(); for most architectures they are the same, including Xen (on x86). For the ones that override it, all should work fine, in the powerpc case it's even more correct (see the subsequent patch "powerpc: Do not force all panic notifiers to execute before kdump")
There seems to be 2 cases that requires some plumbing to work 100% right:
- ARM doesn't disable local interrupts / FIQs in the crash version of send_stop(); we patched that early in this series;
- x86 could face an issue if we have VMX and do use crash_smp_send_stop() _without_ kdump, but this is fixed in the first patch of the series (and it's a bug present even before this refactor).
* Notice we didn't add a sysrq for panic notifiers level - should have it? Alejandro proposed recently to add a sysrq for "crash_kexec_post_notifiers", let me know if you feel the need here Alejandro, since the core parameters are present in /sys, I didn't consider much gain in having a sysrq, but of course I'm open to suggestions!
Thanks advance for the review!
.../admin-guide/kernel-parameters.txt | 42 ++- include/linux/panic_notifier.h | 1 + kernel/kexec_core.c | 8 +- kernel/panic.c | 292 +++++++++++++----- .../selftests/pstore/pstore_crash_test | 5 +- 5 files changed, 252 insertions(+), 96 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +34 -8
@@ -829,6 +829,13 @@ It will be ignored when crashkernel=X,high is not used or memory reserved is below 4G. + crash_kexec_post_notifiers + This was DEPRECATED - users should always prefer the + parameter "panic_notifiers_level" - check its entry + in this documentation for details on how it works. + Setting this parameter is exactly the same as setting + "panic_notifiers_level=4". + cryptomgr.notests [KNL] Disable crypto self-tests
@@ -3784,6 +3791,33 @@ timeout < 0: reboot immediately Format: <timeout> + panic_notifiers_level= + [KNL] Set the panic notifiers execution order. + Format: <unsigned int> + We currently have 4 lists of panic notifiers; based + on the functionality and risk (for panic success) the + callbacks are added in a given list. The lists are: + - hypervisor/FW notification list (low risk); + - informational list (low/medium risk); + - pre_reboot list (higher risk); + - post_reboot list (only run late in panic and after + kdump, not configurable for now). + This parameter defines the ordering of the first 3 + lists with regards to kdump; the levels determine + which set of notifiers execute before kdump. The + accepted levels are: + 0: kdump is the first thing to run, NO list is + executed before kdump. + 1: only the hypervisor list is executed before kdump. + 2 (default level): the hypervisor list and (*if* + there's any kmsg_dumper defined) the informational + list are executed before kdump. + 3: both the hypervisor and the informational lists + (always) execute before kdump. + 4: the 3 lists (hypervisor, info and pre_reboot) + execute before kdump - this behavior is analog to the + deprecated parameter "crash_kexec_post_notifiers". + panic_print= Bitmask for printing system info when panic happens. User can chose combination of the following bits: bit 0: print all tasks info
@@ -3814,14 +3848,6 @@ panic_on_warn panic() instead of WARN(). Useful to cause kdump on a WARN(). - crash_kexec_post_notifiers - Run kdump after running panic-notifiers and dumping - kmsg. This only for the users who doubt kdump always - succeeds in any situation. - Note that this also increases risks of kdump failure, - because some panic notifiers can make the crashed - kernel more unstable. - parkbd.port= [HW] Parallel port number the keyboard adapter is connected to, default is 0. Format: <parport#>
include/linux/panic_notifier.h +1 -0
@@ -10,6 +10,7 @@ extern struct atomic_notifier_head panic_info_list;extern struct atomic_notifier_head panic_pre_reboot_list;extern struct atomic_notifier_head panic_post_reboot_list; +bool panic_notifiers_before_kdump (void );extern bool crash_kexec_post_notifiers;enum panic_notifier_val {
kernel/kexec_core.c +4 -4
@@ -74,11 +74,11 @@ struct resource crashk_low_res = {int kexec_should_crash (struct task_struct *p) {/* -* If crash_kexec_post_notifiers is enabled, don't run -* crash_kexec() here yet, which must be run after panic -* notifiers in panic(). +* In case any panic notifiers are set to execute before kexec, +* don't run crash_kexec() here yet, which must run after these +* panic notifiers are executed, in the panic() path. */ -if (crash_kexec_post_notifiers) +if (panic_notifiers_before_kdump ())return 0 ;/* * There are 4 panic() calls in make_task_dead() path, each of which
kernel/panic.c +210 -82
372 lines not on this page · raw message
tools/testing/selftests/pstore/pstore_crash_test +3 -2
@@ -25,6 +25,7 @@ touch $REBOOT_FLAG sync # cause crash -# Note: If you use kdump and want to see kmesg-* files after reboot, you should -# specify 'crash_kexec_post_notifiers' in 1st kernel's cmdline. +# Note: If you use kdump and want to see kmsg-* files after reboot, you should +# be sure that the parameter "panic_notifiers_level" is more than '2' (the +# default value for this parameter is '2') in the first kernel's cmdline. echo c > /proc/sysrq-trigger
Signature · 1 line
2.36.0
Randy Dunlap
reviewerRe: [PATCH 24/30] panic: Refactor the panic path
Documentation/admin-guide/kernel-parameters.txt line 833
Quoted 2 lines · Guilherme G. Piccoli
Documentation/admin-guide/kernel-parameters.txt +2
-0+ crash_kexec_post_notifiers + This was DEPRECATED - users should always prefer the
This is DEPRECATED - users should always prefer the
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ parameter "panic_notifiers_level" - check its entry + in this documentation for details on how it works. + Setting this parameter is exactly the same as setting + "panic_notifiers_level=4".
Signature · 1 line
~Randy
Re: [PATCH 24/30] panic: Refactor the panic path
Quoted 8 lines · Randy Dunlap
Documentation/admin-guide/kernel-parameters.txt line 833
Quoted 2 lines · Guilherme G. Piccoli
Documentation/admin-guide/kernel-parameters.txt +2
-0+ crash_kexec_post_notifiers + This was DEPRECATED - users should always prefer theThis is DEPRECATED - users should always prefer the
Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt line 837
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ parameter "panic_notifiers_level" - check its entry + in this documentation for details on how it works. + Setting this parameter is exactly the same as setting + "panic_notifiers_level=4".
Thanks Randy, for your suggestion - but I confess I couldn't understand it properly. It's related to spaces/tabs, right? What you suggest me to change in this formatting? Just by looking the email I can't parse.
Cheers,
Guilherme
Re: [PATCH 24/30] panic: Refactor the panic path
Quoted 15 lines · Guilherme G. Piccoli
Quoted 8 lines · Randy Dunlap
Documentation/admin-guide/kernel-parameters.txt line 833
Quoted 2 lines · Guilherme G. Piccoli
Documentation/admin-guide/kernel-parameters.txt +2
-0+ crash_kexec_post_notifiers + This was DEPRECATED - users should always prefer theThis is DEPRECATED - users should always prefer the
Quoted 5 lines
Documentation/admin-guide/kernel-parameters.txt line 837
Quoted 4 lines
Documentation/admin-guide/kernel-parameters.txt +4
-0+ parameter "panic_notifiers_level" - check its entry + in this documentation for details on how it works. + Setting this parameter is exactly the same as setting + "panic_notifiers_level=4".
Quoted 8 lines
Thanks Randy, for your suggestion - but I confess I couldn't understand it properly. It's related to spaces/tabs, right? What you suggest me to change in this formatting? Just by looking the email I can't parse.
Cheers,
Guilherme
Complete lack of attention from me, apologies!
The suggestions was s/was/is - already fixed for V2, thanks Randy.
RE: [PATCH 24/30] panic: Refactor the panic path
In reply to Guilherme G. PiccoliFrom: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
Quoted 157 lines
The
panic()function is somewhat convoluted - a lot of changes were made over the years, adding comments that might be misleading/outdated now, it has a code structure that is a bit complex to follow, with lots of conditionals, for example. The panic notifier list is something else - a single list, with multiple callbacks of different purposes, that run in a non-deterministic order and may affect hardly kdump reliability - see the "crash_kexec_post_notifiers" workaround-ish flag.This patch proposes a major refactor on the panic path based on Petr's idea [0] - basically we split the notifiers list in three, having a set of different call points in the panic path. Below a list of changes proposed in this patch, culminating in the panic notifiers level concept:
(a) First of all, we improved comments all over the function and removed useless variables / includes. Also, as part of this clean-up we concentrate the console flushing functions in a helper.
(b) As mentioned before, there is a split of the panic notifier list in three, based on the purpose of the callback. The code contains good documentation in form of comments, but a summary of the three lists follows:
- the hypervisor list aims low-risk procedures to inform hypervisors or firmware about the panic event, also includes LED-related functions;
- the informational list contains callbacks that provide more details, like kernel offset or trace dump (if enabled) and also includes the callbacks aimed at reducing log pollution or warns, like the RCU and hung task disable callbacks;
- finally, the
pre_rebootlist is the old notifier list renamed, containing the more risky callbacks that didn't fit the previous lists. There is also a 4th list (thepost_rebootone), but it's not related with the original list - it contains late time architecture callbacks aimed at stopping the machine, for example.The 3 notifiers lists execute in different moments, hypervisor being the first, followed by informational and finally the
pre_rebootlist.(c) But then, there is the ordering problem of the notifiers against the
crash_kernel()call - kdump must be as reliable as possible. For that, a simple binary "switch" as "crash_kexec_post_notifiers" is not enough, hence we introduce here concept of panic notifier
levels: there are 5 levels, from 0 (no notifier executes before kdump) until 4 (all notifiers run before kdump); the default level is 2, in which the hypervisor and (iff we have any kmsg dumper) the informational notifiers execute before kdump.The detailed documentation of the levels is present in code comments and in the kernel-parameters.txt file; as an analogy with the previous
panic()implementation, the level 0 is exactly the same as the old behavior of notifiers, running all after kdump, and the level 4 is the same as "crash_kexec_post_notifiers=Y" (we kept this parameter as a deprecated one).(d) Finally, an important change made here: we now use only the function "
crash_smp_send_stop()" to shut all the secondary CPUs in the panic path. Before, there was a case of using the regular "smp_send_stop()", but the better approach is to simplify the code and try to use the function which was created exclusively for the panic path. Experiments showed that it works fine, and code was very simplified with that.Functional change is expected from this refactor, since now we call some notifiers by default before kdump, but the goal here besides code clean-up is to have a better panic path, more reliable and deterministic, but also very customizable.
[0] https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/
Suggested-by Petr Mladek <[email protected]>Signed-off-by Guilherme G. Piccoli <[email protected]>---
Special thanks to Petr and Baoquan for the suggestion and feedback in a previous email thread. There's some important design decisions that worth mentioning and discussing:
* The default panic notifiers level is 2, based on Petr Mladek's suggestion, which makes a lot of sense. Of course, this is customizable through the parameter, but would be something worthwhile to have a KConfig option to set the default level? It would help distros that want the old behavior (no notifiers before kdump) as default.
* The implementation choice was to _avoid_ intricate if conditionals in the panic path, which would _definitely_ be present with the panic notifiers levels idea; so, instead of lots of if conditionals, the set/clear bits approach with functions called in 2 points (but executing only in one of them) is much easier to follow an was used here; the ordering helper function and the comments also help a lot to avoid confusion (hopefully).
* Choice was to *always* use
crash_smp_send_stop()instead of sometimes making use of the regularsmp_send_stop(); for most architectures they are the same, including Xen (on x86). For the ones that override it, all should work fine, in the powerpc case it's even more correct (see the subsequent patch "powerpc: Do not force all panic notifiers to execute before kdump")There seems to be 2 cases that requires some plumbing to work 100% right:
- ARM doesn't disable local interrupts / FIQs in the crash version ofsend_stop(); we patched that early in this series;
- x86 could face an issue if we have VMX and do usecrash_smp_send_stop()_without_ kdump, but this is fixed in the first patch of the series (and it's a bug present even before this refactor).* Notice we didn't add a sysrq for panic notifiers level - should have it? Alejandro proposed recently to add a sysrq for "
crash_kexec_post_notifiers", let me know if you feel the need here Alejandro, since the core parameters are present in /sys, I didn't consider much gain in having a sysrq, but of course I'm open to suggestions!Thanks advance for the review!
.../admin-guide/kernel-parameters.txt | 42 ++- include/linux/panic_notifier.h | 1 + kernel/kexec_core.c | 8 +- kernel/panic.c | 292 +++++++++++++----- .../selftests/pstore/pstore_crash_test | 5 +- 5 files changed, 252 insertions(+), 96 deletions(-)Documentation/admin-guide/kernel-parameters.txt +20
-0@@ -829,6 +829,13 @@ It will be ignored when crashkernel=X,high is not used or memory reserved is below 4G. + crash_kexec_post_notifiers + This was DEPRECATED - users should always prefer the + parameter "panic_notifiers_level" - check its entry + in this documentation for details on how it works. + Setting this parameter is exactly the same as setting + "panic_notifiers_level=4". + cryptomgr.notests [KNL] Disable crypto self-tests@@ -3784,6 +3791,33 @@ timeout < 0: reboot immediately Format: <timeout> + panic_notifiers_level= + [KNL] Set the panic notifiers execution order. + Format: <unsigned int> + We currently have 4 lists of panic notifiers; based + on the functionality and risk (for panic success) the + callbacks are added in a given list. The lists are: + - hypervisor/FW notification list (low risk); + - informational list (low/medium risk); + - pre_reboot list (higher risk); + - post_reboot list (only run late in panic and after + kdump, not configurable for now). + This parameter defines the ordering of the first 3 + lists with regards to kdump; the levels determine
Documentation/admin-guide/kernel-parameters.txt line 3816
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ which set of notifiers execute before kdump. The + accepted levels are: + 0: kdump is the first thing to run, NO list is + executed before kdump. + 1: only the hypervisor list is executed before kdump. + 2 (default level): the hypervisor list and (*if* + there's any kmsg_dumper defined) the informational + list are executed before kdump. + 3: both the hypervisor and the informational lists + (always) execute before kdump.
I'm not clear on why level 2 exists. What is the scenario where execution of the info list before kdump should be conditional on the
existence of a kmsg_dumper? Maybe the scenario is described
somewhere in the patch set and I just missed it.
Quoted 183 lines
Documentation/admin-guide/kernel-parameters.txt +4
-8+ 4: the 3 lists (hypervisor, info and pre_reboot) + execute before kdump - this behavior is analog to the + deprecated parameter "crash_kexec_post_notifiers". + panic_print= Bitmask for printing system info when panic happens. User can chose combination of the following bits: bit 0: print all tasks info@@ -3814,14 +3848,6 @@ panic_on_warn panic() instead of WARN(). Useful to cause kdump on a WARN(). - crash_kexec_post_notifiers - Run kdump after running panic-notifiers and dumping - kmsg. This only for the users who doubt kdump always - succeeds in any situation. - Note that this also increases risks of kdump failure, - because some panic notifiers can make the crashed - kernel more unstable. - parkbd.port= [HW] Parallel port number the keyboard adapter is connected to, default is 0. Format: <parport#>include/linux/panic_notifier.h +1
-0@@ -10,6 +10,7 @@ extern struct atomic_notifier_head panic_info_list;extern struct atomic_notifier_head panic_pre_reboot_list;extern struct atomic_notifier_head panic_post_reboot_list; +bool panic_notifiers_before_kdump (void );extern bool crash_kexec_post_notifiers;enum panic_notifier_val {kernel/kexec_core.c +4
-4@@ -74,11 +74,11 @@ struct resource crashk_low_res = {int kexec_should_crash (struct task_struct *p) {/* -* If crash_kexec_post_notifiers is enabled, don't run -* crash_kexec() here yet, which must be run after panic -* notifiers in panic(). +* In case any panic notifiers are set to execute before kexec, +* don't run crash_kexec() here yet, which must run after these +* panic notifiers are executed, in the panic() path. */ -if (crash_kexec_post_notifiers) +if (panic_notifiers_before_kdump ())return 0 ;/* * There are 4 panic() calls in make_task_dead() path, each of which kernel/panic.c +87
-7@@ -15,7 +15,6 @@#include <linux/kgdb.h> #include <linux/kmsg_dump.h> #include <linux/kallsyms.h> -#include <linux/notifier.h> #include <linux/vt_kern.h> #include <linux/module.h> #include <linux/random.h> @@ -52,14 +51,23 @@ static unsigned long tainted_mask =static int pause_on_oops;static int pause_on_oops_flag;static DEFINE_SPINLOCK (pause_on_oops_lock); -bool crash_kexec_post_notifiers; +int panic_on_warn__read_mostly ; +bool panic_on_taint_nousertaint;unsigned long panic_on_taint; -bool panic_on_taint_nousertaint = false;int panic_timeout =CONFIG_PANIC_TIMEOUT ;EXPORT_SYMBOL_GPL (panic_timeout); +/* Initialized with all notifiers set to run before kdump */ +static unsigned long panic_notifiers_bits =15 ; + +/* Default level is 2, see kernel-parameters.txt */ +unsigned int panic_notifiers_level =2 ; + +/* DEPRECATED in favor of panic_notifiers_level */ +bool crash_kexec_post_notifiers; +#define PANIC_PRINT_TASK_INFO 0x00000001#define PANIC_PRINT_MEM_INFO 0x00000002#define PANIC_PRINT_TIMER_INFO 0x00000004@@ -109,10 +117,14 @@ void __weak nmi_panic_self_stop(struct pt_regs *regs) }/* -* Stop other CPUs in panic. Architecture dependent code may override this -* with more suitable version. For example, if the architecture supports -* crash dump, it should save registers of each stopped CPU and disable -* per-CPU features such as virtualization extensions. +* Stop other CPUs in panic context. +* +* Architecture dependent code may override this with more suitable version. +* For example, if the architecture supports crash dump, it should save the +* registers of each stopped CPU and disable per-CPU features such as +* virtualization extensions. When not overridden in arch code (and for +* x86/xen), this is exactly the same as execute smp_send_stop(), but +* guarded against duplicate execution. */ void __weak crash_smp_send_stop (void ) {@@ -183,6 +195,112 @@ static void panic_print_sys_info(bool console_flush)ftrace_dump (DUMP_ALL ); } +/* +* Helper that accumulates all console flushing routines executed on panic. +*/ +static void console_flushing (void ) +{ +#ifdef CONFIG_VT +unblank_screen (); +#endif +console_unblank (); + +/* +* In this point, we may have disabled other CPUs, hence stopping the +* CPU holding the lock while still having some valuable data in the +* console buffer. +* +* Try to acquire the lock then release it regardless of the result. +* The release will also print the buffers out. Locks debug should +* be disabled to avoid reporting bad unlock balance when panic() +* is not being called from OOPS. +*/ +debug_locks_off (); +console_flush_on_panic (CONSOLE_FLUSH_PENDING ); + +panic_print_sys_info (true); +} + +#define PN_HYPERVISOR_BIT 0 +#define PN_INFO_BIT 1 +#define PN_PRE_REBOOT_BIT 2 +#define PN_POST_REBOOT_BIT 3 + +/* +* Determine the order of panic notifiers with regards to kdump. +* +* This function relies in the "panic_notifiers_level" kernel parameter +* to determine how to order the notifiers with regards to kdump. We +* have currently 5 levels. For details, please check the kernel docs for +* "panic_notifiers_level" at Documentation/admin-guide/kernel-parameters.txt. +* +* Default level is 2, which means the panic hypervisor and informational +* (unless we don't have any kmsg_dumper) lists will execute before kdump. +*/ +static void order_panic_notifiers_and_kdump (void ) +{ +/* +* The parameter "crash_kexec_post_notifiers" is deprecated, but +* valid. Users that set it want really all panic notifiers to +* execute before kdump, so it's effectively the same as setting +* the panic notifiers level to 4. +*/ +if (panic_notifiers_level >=4 || crash_kexec_post_notifiers) +return ; + +/* +* Based on the level configured (smaller than 4), we clear the +* proper bits in "panic_notifiers_bits". Notice that this bitfield +* is initialized with all notifiers set. +*/ +switch (panic_notifiers_level) { +case 3 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +break ; +case 2 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); + +if (!kmsg_has_dumpers ()) +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +break ;
kernel/panic.c static void panic_print_sys_info(bool console_flush) · line 275
Quoted 10 lines
kernel/panic.c +10
-0+case 1 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +break ; +case 0 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +clear_bit (PN_HYPERVISOR_BIT , &panic_notifiers_bits); +break ; + }
I think the above switch statement could be done as follows:
if (panic_notifiers_level <= 3)
clear_bit(PN_PRE_REBOOT_BIT, &panic_notifiers_bits); if (panic_notifiers_level <= 2) if (!kmsg_has_dumpers()) clear_bit(PN_INFO_BIT, &panic_notifiers_bits); if (panic_notifiers_level <=1) clear_bit(PN_INFO_BIT, &panic_notifiers_bits); if (panic_notifiers_level == 0) clear_bit(PN_HYPERVISOR_BIT, &panic_notifiers_bits);
That's about half the lines of code. It's somewhat a matter of style, so treat this as just a suggestion to consider. I just end up looking for a better solution when I see the same line of code repeated 3 or 4 times!
Quoted 263 lines
kernel/panic.c +113
-75+} + +/* +* Set of helpers to execute the panic notifiers only once. +* Just the informational notifier cares about the return. +*/ +static inline bool notifier_run_once (struct atomic_notifier_head head, +char *buf,long bit) +{ +if (test_and_change_bit (bit, &panic_notifiers_bits)) { +atomic_notifier_call_chain (&head,PANIC_NOTIFIER , buf); +return true; + } +return false; +} + +#define panic_notifier_hypervisor_once (buf)\ + notifier_run_once(panic_hypervisor_list, buf, PN_HYPERVISOR_BIT) + +#define panic_notifier_info_once (buf)\ + notifier_run_once(panic_info_list, buf, PN_INFO_BIT) + +#define panic_notifier_pre_reboot_once (buf)\ + notifier_run_once(panic_pre_reboot_list, buf, PN_PRE_REBOOT_BIT) + +#define panic_notifier_post_reboot_once (buf)\ + notifier_run_once(panic_post_reboot_list, buf, PN_POST_REBOOT_BIT) +/** * panic - halt the system * @fmt: The text string to print @@ -198,32 +316,29 @@ void panic(const char *fmt, ...)long i, i_next =0 , len;int state =0 ;int old_cpu, this_cpu; -bool _crash_kexec_post_notifiers = crash_kexec_post_notifiers; -if (panic_on_warn) { -/* -* This thread may hit another WARN() in the panic path. -* Resetting this prevents additional WARN() from panicking the -* system on this thread. Other threads are blocked by the -* panic_mutex in panic(). -*/ - panic_on_warn =0 ; - } +/* +* This thread may hit another WARN() in the panic path, so +* resetting this option prevents additional WARN() from +* re-panicking the system here. +*/ + panic_on_warn =0 ;/* * Disable local interrupts. This will prevent panic_smp_self_stop -* from deadlocking the first cpu that invokes the panic, since -* there is nothing to prevent an interrupt handler (that runs -* after setting panic_cpu) from invoking panic() again. +* from deadlocking the first cpu that invokes the panic, since there +* is nothing to prevent an interrupt handler (that runs after setting +* panic_cpu) from invoking panic() again. Also disables preemption +* here - notice it's not safe to rely on interrupt disabling to avoid +* preemption, since any cond_resched() or cond_resched_lock() might +* trigger a reschedule if the preempt count is 0 (for reference, see +* Documentation/locking/preempt-locking.rst). Some functions called +* from here want preempt disabled, so no point enabling it later. */ local_irq_disable ();preempt_disable_notrace ();/* -* It's possible to come here directly from a panic-assertion and -* not have preempt disabled. Some functions called from here want -* preempt to be disabled. No point enabling it later though... -* * Only one CPU is allowed to execute the panic code from here. For * multiple parallel invocations of panic, all other CPUs either * stop themself or will wait until they are stopped by the 1st CPU @@ -266,73 +381,75 @@ void panic(const char *fmt, ...)kgdb_panic (buf);/* -* If we have crashed and we have a crash kernel loaded let it handle -* everything else. -* If we want to run this after calling panic_notifiers, pass -* the "crash_kexec_post_notifiers" option to the kernel. +* Here lies one of the most subtle parts of the panic path, +* the panic notifiers and their order with regards to kdump. +* We currently have 4 sets of notifiers: * -* Bypass the panic_cpu check and call __crash_kexec directly. +* - the hypervisor list is composed by callbacks that are related +* to warn the FW / hypervisor about panic, or non-invasive LED +* controlling functions - (hopefully) low-risk for kdump, should +* run early if possible. +* +* - the informational list is composed by functions dumping data +* like kernel offsets, device error registers or tracing buffer; +* also log flooding prevention callbacks fit in this list. It is +* relatively safe to run before kdump. +* +* - the pre_reboot list basically is everything else, all the +* callbacks that don't fit in the 2 previous lists. It should +* run *after* kdump if possible, as it contains high-risk +* functions that may break kdump. +* +* - we also have a 4th list of notifiers, the post_reboot +* callbacks. This is not strongly related to kdump since it's +* always executed late in the panic path, after the restart +* mechanism (if set); its goal is to provide a way for +* architecture code effectively power-off/disable the system. +* +* The kernel provides the "panic_notifiers_level" parameter +* to adjust the ordering in which these notifiers should run +* with regards to kdump - the default level is 2, so both the +* hypervisor and informational notifiers should execute before +* the __crash_kexec(); the info notifier won't run by default +* unless there's some kmsg_dumper() registered. For details +* about it, check Documentation/admin-guide/kernel-parameters.txt. +* +* Notice that the code relies in bits set/clear operations to +* determine the ordering, functions *_once() execute only one +* time, as their name implies. The goal is to prevent too much +* if conditionals and more confusion. Finally, regarding CPUs +* disabling: unless NO panic notifier executes before kdump, +* we always disable secondary CPUs before __crash_kexec() and +* the notifiers execute. */ -if (!_crash_kexec_post_notifiers) { +order_panic_notifiers_and_kdump (); + +/* If no level, we should kdump ASAP. */ +if (!panic_notifiers_level)__crash_kexec (NULL ); -/* -* Note smp_send_stop is the usual smp shutdown function, which -* unfortunately means it may not be hardened to work in a -* panic situation. -*/ -smp_send_stop (); - }else { -/* -* If we want to do crash dump after notifier calls and -* kmsg_dump, we will need architecture dependent extra -* works in addition to stopping other CPUs. -*/ -crash_smp_send_stop (); - } +crash_smp_send_stop (); +panic_notifier_hypervisor_once (buf); -/* -* Run any panic handlers, including those that might need to -* add information to the kmsg dump output. -*/ -atomic_notifier_call_chain (&panic_hypervisor_list,PANIC_NOTIFIER , buf); -atomic_notifier_call_chain (&panic_info_list,PANIC_NOTIFIER , buf); -atomic_notifier_call_chain (&panic_pre_reboot_list,PANIC_NOTIFIER , buf); +if (panic_notifier_info_once (buf)) { +panic_print_sys_info (false); +kmsg_dump (KMSG_DUMP_PANIC ); + } -panic_print_sys_info (false); +panic_notifier_pre_reboot_once (buf); -kmsg_dump (KMSG_DUMP_PANIC ); +__crash_kexec (NULL ); -/* -* If you doubt kdump always works fine in any situation, -* "crash_kexec_post_notifiers" offers you a chance to run -* panic_notifiers and dumping kmsg before kdump. -* Note: since some panic_notifiers can make crashed kernel -* more unstable, it can increase risks of the kdump failure too. -* -* Bypass the panic_cpu check and call __crash_kexec directly. -*/ -if (_crash_kexec_post_notifiers) -__crash_kexec (NULL ); +panic_notifier_hypervisor_once (buf); -#ifdef CONFIG_VT -unblank_screen (); -#endif -console_unblank (); - -/* -* We may have ended up stopping the CPU holding the lock (in -* smp_send_stop()) while still having some valuable data in the console -* buffer. Try to acquire the lock then release it regardless of the -* result. The release will also print the buffers out. Locks debug -* should be disabled to avoid reporting bad unlock balance when -* panic() is not being callled from OOPS. -*/ -debug_locks_off (); -console_flush_on_panic (CONSOLE_FLUSH_PENDING ); +if (panic_notifier_info_once (buf)) { +panic_print_sys_info (false); +kmsg_dump (KMSG_DUMP_PANIC ); + } -panic_print_sys_info (true); +panic_notifier_pre_reboot_once (buf); +console_flushing ();if (!panic_blink) panic_blink = no_blink;@@ -363,8 +480,7 @@ void panic(const char *fmt, ...)emergency_restart (); } -atomic_notifier_call_chain (&panic_post_reboot_list, -PANIC_NOTIFIER , buf); +panic_notifier_post_reboot_once (buf);pr_emerg ("---[ end Kernel panic - not syncing: %s ]---\n" , buf);@@ -383,6 +499,15 @@ void panic(const char *fmt, ...)EXPORT_SYMBOL (panic); +/* +* Helper used in the kexec code, to validate if any +* panic notifier is set to execute early, before kdump. +*/ +inline bool panic_notifiers_before_kdump (void ) +{ +return panic_notifiers_level || crash_kexec_post_notifiers; +} +/* * TAINT_FORCED_RMMOD could be a per-module flag but the module * is being removed anyway. @@ -692,6 +817,9 @@ core_param(panic, panic_timeout, int, 0644);core_param (panic_print, panic_print, ulong,0644 );core_param (pause_on_oops, pause_on_oops, int,0644 );core_param (panic_on_warn, panic_on_warn, int,0644 ); +core_param (panic_notifiers_level, panic_notifiers_level, uint,0644 ); + +/* DEPRECATED in favor of panic_notifiers_level */ core_param (crash_kexec_post_notifiers, crash_kexec_post_notifiers, bool,0644 );static int __init oops_setup (char *s)tools/testing/selftests/pstore/pstore_crash_test +3
-3@@ -25,6 +25,7 @@ touch $REBOOT_FLAG sync # cause crash -# Note: If you use kdump and want to see kmesg-* files after reboot, you should -# specify 'crash_kexec_post_notifiers' in 1st kernel's cmdline. +# Note: If you use kdump and want to see kmsg-* files after reboot, you should +# be sure that the parameter "panic_notifiers_level" is more than '2' (the +# default value for this parameter is '2') in the first kernel's cmdline. echo c > /proc/sysrq-trigger --2.36.0
Re: [PATCH 24/30] panic: Refactor the panic path
Quoted 22 lines · Michael Kelley (LINUX)
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022 3:49 PM
Quoted 21 lines
[...]
Documentation/admin-guide/kernel-parameters.txt +20
-0+ panic_notifiers_level= + [KNL] Set the panic notifiers execution order. + Format: <unsigned int> + We currently have 4 lists of panic notifiers; based + on the functionality and risk (for panic success) the + callbacks are added in a given list. The lists are: + - hypervisor/FW notification list (low risk); + - informational list (low/medium risk); + - pre_reboot list (higher risk); + - post_reboot list (only run late in panic and after + kdump, not configurable for now). + This parameter defines the ordering of the first 3 + lists with regards to kdump; the levels determine + which set of notifiers execute before kdump. The + accepted levels are: + 0: kdump is the first thing to run, NO list is + executed before kdump. + 1: only the hypervisor list is executed before kdump. + 2 (default level): the hypervisor list and (*if* + there's any kmsg_dumper defined) the informational
Quoted 9 lines
Documentation/admin-guide/kernel-parameters.txt line 3816
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ list are executed before kdump. + 3: both the hypervisor and the informational lists + (always) execute before kdump.I'm not clear on why level 2 exists. What is the scenario where execution of the info list before kdump should be conditional on the
existence of akmsg_dumper? Maybe the scenario is described
somewhere in the patch set and I just missed it.
Hi Michael, thanks for your review/consideration. So, this idea started kind of some time ago. It all started with a need of exposing more information on kernel log *before* kdump and *before* pstore - specifically, we're talking about panic_print. But this cause some reactions, Baoquan was very concerned with that [0]. Soon after, I've proposed a panic notifiers filter (orthogonal) approach, to which Petr suggested instead doing a major refactor [1] - it finally is alive in the form of this series.
The theory behind the level 2 is to allow a scenario of kdump with the minimum amount of notifiers - what is the point in printing more information if the user doesn't care, since it's going to kdump? Now, if there is a kmsg dumper, it means that there is likely some interest in collecting information, and that might as well be required before the potential kdump (which is my case, hence the proposal on [0]).
Instead of forcing one of the two behaviors (level 1 or level 3), we have a middle-term/compromise: if there's interest in collecting such data (in the form of a kmsg dumper), we then execute the informational notifiers before kdump. If not, why to increase (even slightly) the risk for kdump?
I'm OK in removing the level 2 if people prefer, but I don't feel it's a burden, quite opposite - seems a good way to accommodate the somewhat antagonistic ideas (jump to kdump ASAP vs collecting more info in the panicked kernel log).
[0] https://lore.kernel.org/lkml/20220126052246.GC2086 at MiWiFi-R3L-srv/
[1] https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/
Quoted 38 lines
[...]
Quoted 14 lines
kernel/panic.c +14
-0+* Based on the level configured (smaller than 4), we clear the +* proper bits in "panic_notifiers_bits". Notice that this bitfield +* is initialized with all notifiers set. +*/ +switch (panic_notifiers_level) { +case 3 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +break ; +case 2 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); + +if (!kmsg_has_dumpers ()) +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +break ;kernel/panic.c static void panic_print_sys_info(bool console_flush) · line 275
Quoted 10 lines
kernel/panic.c +10
-0+case 1 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +break ; +case 0 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +clear_bit (PN_HYPERVISOR_BIT , &panic_notifiers_bits); +break ; + }I think the above switch statement could be done as follows:
if (
panic_notifiers_level<= 3)clear_bit(PN_PRE_REBOOT_BIT, &panic_notifiers_bits); if (panic_notifiers_level <= 2) if (!kmsg_has_dumpers()) clear_bit(PN_INFO_BIT, &panic_notifiers_bits); if (panic_notifiers_level <=1) clear_bit(PN_INFO_BIT, &panic_notifiers_bits); if (panic_notifiers_level == 0) clear_bit(PN_HYPERVISOR_BIT, &panic_notifiers_bits);
Quoted 5 lines
That's about half the lines of code. It's somewhat a matter of style, so treat this as just a suggestion to consider. I just end up looking for a better solution when I see the same line of code repeated 3 or 4 times!
It's a good idea - I liked your code. The switch seems more natural/explicit for me, even duplicating some lines, but in case more people prefer your way, I can definitely change the code - thanks for the suggestion.
Cheers,
Guilherme
RE: [PATCH 24/30] panic: Refactor the panic path
From: Guilherme G. Piccoli <[email protected]> Sent: Friday, April 29, 2022 1:38 PM
Quoted 57 lines · Michael Kelley (LINUX)
Quoted 1 line
From: Guilherme G. Piccoli <[email protected]> Sent: Wednesday, April 27, 2022
3:49 PM
Quoted 21 lines
Quoted 21 lines
[...]
Documentation/admin-guide/kernel-parameters.txt +20
-0+ panic_notifiers_level= + [KNL] Set the panic notifiers execution order. + Format: <unsigned int> + We currently have 4 lists of panic notifiers; based + on the functionality and risk (for panic success) the + callbacks are added in a given list. The lists are: + - hypervisor/FW notification list (low risk); + - informational list (low/medium risk); + - pre_reboot list (higher risk); + - post_reboot list (only run late in panic and after + kdump, not configurable for now). + This parameter defines the ordering of the first 3 + lists with regards to kdump; the levels determine + which set of notifiers execute before kdump. The + accepted levels are: + 0: kdump is the first thing to run, NO list is + executed before kdump. + 1: only the hypervisor list is executed before kdump. + 2 (default level): the hypervisor list and (*if* + there's any kmsg_dumper defined) the informationalQuoted 9 lines
Documentation/admin-guide/kernel-parameters.txt line 3816
Quoted 3 lines
Documentation/admin-guide/kernel-parameters.txt +3
-0+ list are executed before kdump. + 3: both the hypervisor and the informational lists + (always) execute before kdump.I'm not clear on why level 2 exists. What is the scenario where execution of the info list before kdump should be conditional on the
existence of akmsg_dumper? Maybe the scenario is described
somewhere in the patch set and I just missed it.Hi Michael, thanks for your review/consideration. So, this idea started kind of some time ago. It all started with a need of exposing more information on kernel log *before* kdump and *before* pstore - specifically, we're talking about
panic_print. But this cause some reactions, Baoquan was very concerned with that [0]. Soon after, I've proposed a panic notifiers filter (orthogonal) approach, to which Petr suggested instead doing a major refactor [1] - it finally is alive in the form of this series.The theory behind the level 2 is to allow a scenario of kdump with the minimum amount of notifiers - what is the point in printing more information if the user doesn't care, since it's going to kdump? Now, if there is a kmsg dumper, it means that there is likely some interest in collecting information, and that might as well be required before the potential kdump (which is my case, hence the proposal on [0]).
Instead of forcing one of the two behaviors (level 1 or level 3), we have a middle-term/compromise: if there's interest in collecting such data (in the form of a kmsg dumper), we then execute the informational notifiers before kdump. If not, why to increase (even slightly) the risk for kdump?
Quoted 9 lines
I'm OK in removing the level 2 if people prefer, but I don't feel it's a burden, quite opposite - seems a good way to accommodate the somewhat antagonistic ideas (jump to kdump ASAP vs collecting more info in the panicked kernel log).
[0] https://lore.kernel.org/lkml/20220126052246.GC2086 at MiWiFi-R3L-srv/
[1] https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/
To me, it's a weak correlation between having a kmsg dumper, and wanting or not wanting the info level output to come before kdump. Hyper-V is one of only a few places that register a kmsg dumper, so most Linux instances outside of Hyper-V guest (and PowerPC systems?) will have the info level output after kdump. It seems like anyone who cared strongly about the info level output would set the panic_notifier_level to 1 or to 3 so that the result is more deterministic. But that's just my opinion, and it's probably an opinion that is not as well informed on the topic as some others in the discussion. So keeping things as in your patch set is not a show-stopper for me.
However, I would request a clarification in the documentation. Thepanic_notifier_level affects not only the hypervisor, informational, and pre_reboot lists, but it also affects panic_print_sys_info() and kmsg_dump(). Specifically, at level 1, panic_print_sys_info() and kmsg_dump() will not be run before kdump. At level 3, they will always be run before kdump. Your documentation above mentions "informational lists" (plural), which I take to vaguely include kmsg_dump() and panic_print_sys_info(), but being explicit about the effect would be better.
Michael
Quoted 53 lines
Quoted 39 lines
[...]
Quoted 14 lines
kernel/panic.c +14
-0+* Based on the level configured (smaller than 4), we clear the +* proper bits in "panic_notifiers_bits". Notice that this bitfield +* is initialized with all notifiers set. +*/ +switch (panic_notifiers_level) { +case 3 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +break ; +case 2 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); + +if (!kmsg_has_dumpers ()) +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +break ;kernel/panic.c static void panic_print_sys_info(bool console_flush) · line 275
Quoted 10 lines
kernel/panic.c +10
-0+case 1 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +break ; +case 0 : +clear_bit (PN_PRE_REBOOT_BIT , &panic_notifiers_bits); +clear_bit (PN_INFO_BIT , &panic_notifiers_bits); +clear_bit (PN_HYPERVISOR_BIT , &panic_notifiers_bits); +break ; + }I think the above switch statement could be done as follows:
if (
panic_notifiers_level<= 3)clear_bit(PN_PRE_REBOOT_BIT, &panic_notifiers_bits); if (panic_notifiers_level <= 2) if (!kmsg_has_dumpers()) clear_bit(PN_INFO_BIT, &panic_notifiers_bits); if (panic_notifiers_level <=1) clear_bit(PN_INFO_BIT, &panic_notifiers_bits); if (panic_notifiers_level == 0) clear_bit(PN_HYPERVISOR_BIT, &panic_notifiers_bits);Quoted 5 lines
That's about half the lines of code. It's somewhat a matter of style, so treat this as just a suggestion to consider. I just end up looking for a better solution when I see the same line of code repeated 3 or 4 times!
It's a good idea - I liked your code. The switch seems more natural/explicit for me, even duplicating some lines, but in case more people prefer your way, I can definitely change the code - thanks for the suggestion.
Cheers,Guilherme
Re: [PATCH 24/30] panic: Refactor the panic path
Quoted 23 lines · Michael Kelley (LINUX)
[...]
To me, it's a weak correlation between having a kmsg dumper, and wanting or not wanting the info level output to come before kdump. Hyper-V is one of only a few places that register a kmsg dumper, so most Linux instances outside of Hyper-V guest (and PowerPC systems?) will have the info level output after kdump. It seems like anyone who cared strongly about the info level output would set the
panic_notifier_levelto 1 or to 3 so that the result is more deterministic. But that's just my opinion, and it's probably an opinion that is not as well informed on the topic as some others in the discussion. So keeping things as in your patch set is not a show-stopper for me.However, I would request a clarification in the documentation. The
panic_notifier_levelaffects not only the hypervisor, informational, andpre_rebootlists, but it also affectspanic_print_sys_info()andkmsg_dump(). Specifically, at level 1,panic_print_sys_info()andkmsg_dump()will not be run before kdump. At level 3, they will always be run before kdump. Your documentation above mentions "informational lists" (plural), which I take to vaguely includekmsg_dump()andpanic_print_sys_info(), but being explicit about the effect would be better.
Quoted 1 line
Michael
Thanks again Michael, to express your points and concerns - great idea of documentation improvement here, I'll do that for V2, for sure.
The idea of "defaulting" to skip the info list on kdump (if no kmsg_dump() is set) is again a mechanism that aims at accommodating all users and concerns of antagonistic goals, kdump vs notifier lists.
Before this patch set, by default no notifier executed before kdump. So, the "pendulum" was strongly on kdump side, and clearly this was a sub-optimal decision - proof of that is that both Hyper-V / PowerPC code forcibly set the "crash_kexec_post_notifiers". The goal here is to have a more lightweight list that by default runs before kdump, a secondary list that only runs before kdump if there's usage for that (either user sets that or kmsg_dumper set is considered a valid user), and the remaining notifiers run by default only after kdump, all of that very customizable through the levels idea.
Now, one thing we could do to improve consistency for the hyper-v case:
having a kmsg_dump_once() helper, and *for Hyper-V only*, call it on the hypervisor list, within the info notifier (that would be moved to hypervisor list, ofc).
Let's wait for more feedback on that, just throwing some ideas in order we can have everyone happy with the end-result!
Cheers,
Guilherme
Re: [PATCH 24/30] panic: Refactor the panic path
In reply to Guilherme G. PiccoliSorry for the delayed response. Unfortunately, I had 10 days holidays until yesterday...
Quoted 11 lines
.../admin-guide/kernel-parameters.txt | 42 ++- include/linux/panic_notifier.h | 1 + kernel/kexec_core.c | 8 +- kernel/panic.c | 292 +++++++++++++----- .../selftests/pstore/pstore_crash_test | 5 +- 5 files changed, 252 insertions(+), 96 deletions(-)Documentation/admin-guide/kernel-parameters.txt +0
-0
...snip...
Quoted 13 lines
Documentation/admin-guide/kernel-parameters.txt +9
-0@@ -3784,6 +3791,33 @@ timeout < 0: reboot immediately Format: <timeout> + panic_notifiers_level= + [KNL] Set the panic notifiers execution order. + Format: <unsigned int> + We currently have 4 lists of panic notifiers; based + on the functionality and risk (for panic success) the + callbacks are added in a given list. The lists are: + - hypervisor/FW notification list (low risk); + - informational list (low/medium risk); + - pre_reboot list (higher risk);
Documentation/admin-guide/kernel-parameters.txt line 3812
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ - post_reboot list (only run late in panic and after + kdump, not configurable for now). + This parameter defines the ordering of the first 3 + lists with regards to kdump; the levels determine + which set of notifiers execute before kdump. The + accepted levels are: + 0: kdump is the first thing to run, NO list is + executed before kdump. + 1: only the hypervisor list is executed before kdump. + 2 (default level): the hypervisor list and (*if*
Hmmm, why are you trying to change default setting?
Based on the current design of kdump, it's natural to put what the handlers for these level 1 and level 2 handlers do in machine_crash_shutdown(), as these are necessary by default, right?
Or have you already tried that and figured out it's difficult in some reason and reached the current design? If so, why is that difficult? Could you point to if there is already such discussion online?
kdump is designed to perform as little things as possible before transferring the execution to the 2nd kernel in order to increase reliability. Just detour to panic() increases risks of kdump failure in the sense of increasing the executed codes in the abnormal situation, which is very the note in the explanation of crash_kexec_post_notifiers.
Also, the current implementation of crash_kexec_post_notifiers uses the panic notifier, but this is not from the technical reason. Ideally, it should have been implemented in the context of crash_kexec() independently of panic().
That is, it looks to me that, in addition to changing design of panic notifier, you are trying to integrate shutdown code of the crash kexec and the panic paths. If so, this is a big design change for kdump. I'm concerned about increase of reliability. I'd like you to discuss them carefully.
Thanks.
HATAYAMA, Daisuke
Re: [PATCH 24/30] panic: Refactor the panic path
Hey Hatayma, thanks for your great analysis and no need for apologies!
I'll comment/respond properly inline below, just noticing here that I've CCed Mark and Marc (from the ARM64 perspective), Michael (Hyper-V perspective) and Hari (PowerPC perspective), besides the usual suspects as Petr, Baoquan, etc.
Quoted 42 lines · d.hatayama at fujitsu.com
Sorry for the delayed response. Unfortunately, I had 10 days holidays until yesterday...
[...]Quoted 6 lines
Documentation/admin-guide/kernel-parameters.txt +6
-0+ We currently have 4 lists of panic notifiers; based + on the functionality and risk (for panic success) the + callbacks are added in a given list. The lists are: + - hypervisor/FW notification list (low risk); + - informational list (low/medium risk); + - pre_reboot list (higher risk);Documentation/admin-guide/kernel-parameters.txt line 3812
Quoted 10 lines
Documentation/admin-guide/kernel-parameters.txt +10
-0+ - post_reboot list (only run late in panic and after + kdump, not configurable for now). + This parameter defines the ordering of the first 3 + lists with regards to kdump; the levels determine + which set of notifiers execute before kdump. The + accepted levels are: + 0: kdump is the first thing to run, NO list is + executed before kdump. + 1: only the hypervisor list is executed before kdump. + 2 (default level): the hypervisor list and (*if*Hmmm, why are you trying to change default setting?
Based on the current design of kdump, it's natural to put what the handlers for these level 1 and level 2 handlers do in
machine_crash_shutdown(), as these are necessary by default, right?Or have you already tried that and figured out it's difficult in some reason and reached the current design? If so, why is that difficult? Could you point to if there is already such discussion online?
kdump is designed to perform as little things as possible before transferring the execution to the 2nd kernel in order to increase reliability. Just detour to
panic()increases risks of kdump failure in the sense of increasing the executed codes in the abnormal situation, which is very the note in the explanation ofcrash_kexec_post_notifiers.Also, the current implementation of
crash_kexec_post_notifiersuses the panic notifier, but this is not from the technical reason. Ideally, it should have been implemented in the context ofcrash_kexec()independently ofpanic().
Quoted 5 lines
That is, it looks to me that, in addition to changing design of panic notifier, you are trying to integrate shutdown code of the crash kexec and the panic paths. If so, this is a big design change for kdump. I'm concerned about increase of reliability. I'd like you to discuss them carefully.
From my understanding (specially based on both these threads [0] and [1]), 3 facts are clear and quite complex in nature:
(a) Currently, the panic notifier list is a "no man's land" - it's a mess, all sort of callbacks are added there, some of them are extremely risk for breaking kdump, others are quite safe (like setting a variable). Petr's details in thread [0] are really clear and express in great way how confusing and conflicting the panic notifiers goals are.
(b) In order to "address" problems in the antagonistic goals of notifiers (see point (a) above and thread [0]), we have this quirk/workaround called "crash_kexec_post_notifiers". This is useful, but (almost as for attesting how this is working as band-aid over complex and fundamental issues) see the following commits:
a11589563e96 ("x86/Hyper-V: Report crash register data or kmsg before running crash kernel")
06e629c25daa ("powerpc/fadump: Fix inaccurate CPU state info in vmcore generated with panic")
They hardcode such workaround, because they *need* some notifiers' callbacks. But notice they *don't need all of them*, only some important ones (that usually are good considering the risk, it's a good cost/benefit). Since we currently have an all-or-nothing behavior for the panic notifiers, both PowerPC and Hyper-V end-up bringing all of them to run before kdump due to the lack of flexibility, increasing a lot the risk of failure for kdump.
(c) To add on top of all such complexity, we don't have a custom machine_crash_shutdown() handler for some architectures like ARM64, and the feeling is that's not right to add a bunch of callbacks / complexity in such architecture code, specially since we have the notifiers infrastructure in the kernel. I've recently started a discussion about that with ARM64 community, please take a look in [1].
With that said, we can use (a) + (b) + (c) to justify our argument here:
the panic notifiers should be refactored! We need to try to encompass the antagonistic goals of kdump (wants to be the first thing to run, early as possible) VS. the notifiers that are necessary, even before kdump (like Hyper-V / PowerPC fadump ones, but there are more of these, the FW/hypervisors notifiers).
I guarantee to you we cannot make 100% of people happy - the panic-related areas like kdump, etc are full of trade-offs. We improve something, other stuff breaks. This series attempts to clearly distribute the notifiers in 3 buckets, and introduces a level setting that tunes such buckets to run before or after the kdump in a highly flexible way, trying to make the most users happy and capable of tuning their systems.
I understand that likely setting the notifiers to 0 would make kdump maintainers happy, because we'd keep the same behavior as before. But..then we make PPC / Hyper-V and some other users unsatisfied. Level 2 seems to me a good compromise. I'm willing to add a KConfig option to allow distros to hard set their defaults - that might make sense to legacy distros as older RHEL / CentOS or server-only distros.
Finally, you mention about integrating the crash shutdown and the panic paths - I'm not officially doing that, but I see that they are very related and all cases I've ever seen in the last 3-4 years I've been working with kdump, it was triggered from panic, from settings "panic_on_X".
I understand we have the regular kexec path (!kdump), some notifiers might make sense in both paths, and in this case we could duplicate them (one callback, 2 notifier lists) and control against double execution. I tried doing that in patch 16 of this series (Hyper-V stuff), but due to the lack of ARM64 custom crash shutdown handler, I couldn't. We can think in some alternatives, and improve these (naturally) connected/related paths, but this is not the goal of this specific series. I'm hereby trying to improve/clarify the panic (and kdump) path, not the pure kexec path.
Let me know if I can clarify more specific points you have or if there are flaws in my logic - I appreciate the discussions/reviews. Cheers,
Guilherme
[0] "notifier/panic: Introduce panic_notifier_filter" -
https://lore.kernel.org/lkml/YfPxvzSzDLjO5ldp at alley/
[1] "Should arm64 have a custom crash shutdown handler?" -
https://lore.kernel.org/lkml/427a8277-49f0-4317-d6c3-4a15d7070e55 at igalia.com/