kernel, add bug_on_warn
PATCH 1/1 kernel, add bug_on_warn
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a WARN() in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.
A much easier method would be a switch to change the WARN() over to a BUG(). This makes debugging easier in that I can now test the actual image the WARN() was seen on and I do not have to engage in remote debugging.
This patch adds a bug_on_warn kernel parameter, which calls BUG() in the warn_slowpath_common() path. The function will still print out the location of the warning.
Successfully tested by me.
--- Documentation/kernel-parameters.txt | 2 ++ kernel/panic.c | 16 +++++++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-)
Documentation/kernel-parameters.txt +2 -0
@@ -553,6 +553,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner= + bug_on_warn BUG() instead of WARN() + bulk_remove=off [PPC] This parameter disables the use of the pSeries firmware feature for flushing multiple hpte entries at a time.
kernel/panic.c +15 -1
@@ -33,6 +33,7 @@ static int pause_on_oops;static int pause_on_oops_flag;static DEFINE_SPINLOCK (pause_on_oops_lock);static bool crash_kexec_post_notifiers; +static bool bug_on_warn;int panic_timeout =CONFIG_PANIC_TIMEOUT ;EXPORT_SYMBOL_GPL (panic_timeout);
@@ -420,13 +421,19 @@ static void warn_slowpath_common(const char *file, int line, void *caller, {disable_trace_on_warning (); -pr_warn ("------------[ cut here ]------------\n" ); +if (!bug_on_warn) +pr_warn ("------------[ cut here ]------------\n" );pr_warn ("WARNING: CPU: %d PID: %d at %s:%d %pS()\n" ,raw_smp_processor_id (), current->pid, file, line, caller);if (args)vprintk (args->fmt, args->args); +if (bug_on_warn) { +pr_warn ("bug_on_warn set, calling BUG()...\n" ); +BUG (); + } +print_modules ();dump_stack ();print_oops_end_marker ();
@@ -501,3 +508,10 @@ static int __init oops_setup(char *s)return 0 ; }early_param ("oops" , oops_setup); + +static int __init bug_on_warn_setup (char *s) +{ + bug_on_warn = true; +return 0 ; +} +early_param ("bug_on_warn" , bug_on_warn_setup);
Signature · 1 line
1.7.9.3
Quoted 11 lines · On Mon, 20 Oct 2014 08:00:20 -0400 Prarit Bhargava <[email protected]> wrote:
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a
WARN()in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.A much easier method would be a switch to change the
WARN()over to aBUG(). This makes debugging easier in that I can now test the actual image theWARN()was seen on and I do not have to engage in remote debugging.
Quoted 5 lines
This patch adds a
bug_on_warnkernel parameter, which callsBUG()in thewarn_slowpath_common()path. The function will still print out the location of the warning.Successfully tested by me.
Looks nice and simple and useful. However I suspect you're exclusively focussed on "I want a crash dump" and things haven't been fully thought through.
- Do you have any example WARN->BUG console output at hand? I'd like to check for missing or duplicated info.
- Did you consider permitting this to be tweaked at runtime via /proc? Sometimes we get pesky WARNs at boot time and having runtime alteration would permit the user to prevent those from tripping a BUG.
- Also, perhaps bug_on_warn should be single-shot: clear itself after it has triggered one BUG. Because once the kernel has gone WARN->BUG, it's probably messed up and is likely to trigger more WARNs. Also, the kernel might generate many WARNs for the same issue.
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 556
Quoted 7 lines
Documentation/kernel-parameters.txt +1
-0@@ -553,6 +553,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner= + bug_on_warn BUG() instead of WARN()
There's no mention here that this feature is mainly aimed at generating a crash dump. How do we tell the people who aren't reading this email thread (ie: all of humanity except you and me ;)) that this feature even exists? Is there crash dump documentation that we can update?
Quoted 17 lines · Andrew Morton
Quoted 15 lines · On Mon, 20 Oct 2014 08:00:20 -0400 Prarit Bhargava <[email protected]> wrote:
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a
WARN()in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.A much easier method would be a switch to change the
WARN()over to aBUG(). This makes debugging easier in that I can now test the actual image theWARN()was seen on and I do not have to engage in remote debugging.This patch adds a
bug_on_warnkernel parameter, which callsBUG()in thewarn_slowpath_common()path. The function will still print out the location of the warning.
Quoted 8 lines
Quoted 1 line
Successfully tested by me.
Looks nice and simple and useful. However I suspect you're exclusively focussed on "I want a crash dump" and things haven't been fully thought through.
- Do you have any example
WARN->BUGconsole output at hand? I'd like to check for missing or duplicated info.
Yep, here you go, with some additional annotation notes from me. The first line below is from the WARN_ON() to output the WARN_ON()'s location. After that, we hit the new BUG() call.
WARNING: CPU: 27 PID: 3204 at /home/rhel7/redhat/debug/dummy-module/dummy-module.c:25 init_dummy+0x28/0x30 [dummy_module]() bug_on_warn set, calling BUG()... ------------[ cut here ]------------ kernel BUG at kernel/panic.c:434! invalid opcode: 0000 [#1] SMP Modules linked in: dummy_module(OE+) sg nfsv3 rpcsec_gss_krb5 nfsv4
dns_resolver nfs fscache cfg80211 rfkill x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel igb iTCO_wdt aesni_intel iTCO_vendor_support lrw gf128mul sb_edac ptp edac_core glue_helper lpc_ich ioatdma pcspkr ablk_helper pps_core i2c_i801 mfd_core cryptd dca shpchp ipmi_si wmi ipmi_msghandler acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom sd_mod mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper isci ttm drm libsas ahci libahci scsi_transport_sas libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 27 PID: 3204 Comm: insmod Tainted: G OE 3.17.0+ #19 Hardware name: Intel Corporation S2600CP/S2600CP, BIOS RMLSDP.86I.00.29.D696.1311111329 11/11/2013 task: ffff880034e75160 ti: ffff8807fc5ac000 task.ti: ffff8807fc5ac000 RIP: 0010:[<ffffffff81076b81>] [<ffffffff81076b81>] warn_slowpath_common+0xc1/0xd0 RSP: 0018:ffff8807fc5afc68 EFLAGS: 00010246 RAX: 0000000000000021 RBX: ffff8807fc5afcb0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88081efee5f8 RDI: ffff88081efee5f8 RBP: ffff8807fc5afc98 R08: 0000000000000096 R09: 0000000000000000 R10: 0000000000000711 R11: ffff8807fc5af93e R12: ffffffffa0424070 R13: 0000000000000019 R14: ffffffffa0423068 R15: 0000000000000009 FS: 00007f2d4b034740(0000) GS:ffff88081efe0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2d4a99f3c0 CR3: 00000007fd88b000 CR4: 00000000001407e0 Stack: ffff8807fc5afcb8 ffffffff8199f020 ffff88080e396160 0000000000000000 ffffffffa0423040 ffffffffa0425000 ffff8807fc5afd08 ffffffff81076be5 0000000000000008 ffffffffa0424053 ffff880700000018 ffff8807fc5afd18 Call Trace: [<ffffffffa0423040>] ? dummy_greetings+0x40/0x40 [dummy_module] [<ffffffff81076be5>] warn_slowpath_fmt+0x55/0x70 [<ffffffffa0423068>] init_dummy+0x28/0x30 [dummy_module] [<ffffffff81002144>] do_one_initcall+0xd4/0x210 [<ffffffff811b52c2>] ? __vunmap+0xc2/0x110 [<ffffffff810f8889>] load_module+0x16a9/0x1b30 [<ffffffff810f3d30>] ? store_uevent+0x70/0x70 [<ffffffff810f49b9>] ? copy_module_from_fd.isra.44+0x129/0x180 [<ffffffff810f8ec6>] SyS_finit_module+0xa6/0xd0 [<ffffffff8166ce29>] system_call_fastpath+0x12/0x17 Code: c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 c7 c7 20 42 8a 81 31 c0 e8 fc
80 5e 00 eb 80 48 c7 c7 78 42 8a 81 31 c0 e8 ec 80 5e 00 <0f> 0b 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55
RIP [<ffffffff81076b81>] warn_slowpath_common+0xc1/0xd0
RSP <ffff8807fc5afc68>
---[ end trace 428218934a12088b ]---
Quoted 6 lines
- Did you consider permitting this to be tweaked at runtime via /proc? Sometimes we get pesky WARNs at boot time and having runtime alteration would permit the user to prevent those from tripping a BUG.
I did actually, but I was wondering how people liked the idea before I looked at the /proc implementation. It's pretty much the same as panic_on_oops, so it's not difficult to do.
Quoted 5 lines
- Also, perhaps
bug_on_warnshould be single-shot: clear itself after it has triggered one BUG. Because once the kernel has goneWARN->BUG, it's probably messed up and is likely to trigger more WARNs. Also, the kernel might generate many WARNs for the same issue.
Okay, I'll add that.
Quoted 7 lines
Quoted 7 lines
Documentation/kernel-parameters.txt +0
-0@@ -553,6 +553,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner=
Quoted 7 lines
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 556
Quoted 1 line
Documentation/kernel-parameters.txt +1
-0+ bug_on_warn BUG() instead of WARN()There's no mention here that this feature is mainly aimed at generating a crash dump. How do we tell the people who aren't reading this email thread (ie: all of humanity except you and me ;)) that this feature even exists? Is there crash dump documentation that we can update?
I'll look into this too.
P.
Quoted 1 line
Hi Prarit,
Quoted 96 lines · Prarit Bhargava
Quoted 17 lines · Andrew Morton
Quoted 15 lines · On Mon, 20 Oct 2014 08:00:20 -0400 Prarit Bhargava <[email protected]> wrote:
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a
WARN()in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.A much easier method would be a switch to change the
WARN()over to aBUG(). This makes debugging easier in that I can now test the actual image theWARN()was seen on and I do not have to engage in remote debugging.This patch adds a
bug_on_warnkernel parameter, which callsBUG()in thewarn_slowpath_common()path. The function will still print out the location of the warning.Quoted 8 lines
Quoted 1 line
Successfully tested by me.
Looks nice and simple and useful. However I suspect you're exclusively focussed on "I want a crash dump" and things haven't been fully thought through.
- Do you have any example
WARN->BUGconsole output at hand? I'd like to check for missing or duplicated info.Yep, here you go, with some additional annotation notes from me. The first line below is from the
WARN_ON()to output the WARN_ON()'s location. After that, we hit the newBUG()call.WARNING: CPU: 27 PID: 3204 at /home/rhel7/redhat/debug/dummy-module/dummy-module.c:25 init_dummy+0x28/0x30 [dummy_module]() bug_on_warn set, calling BUG()... ------------[ cut here ]------------ kernel BUG at kernel/panic.c:434! invalid opcode: 0000 [#1] SMP Modules linked in: dummy_module(OE+) sg nfsv3 rpcsec_gss_krb5 nfsv4
dns_resolvernfs fscache cfg80211 rfkillx86_pkg_temp_thermalintel_powerclampcoretempkvm_intelkvmcrct10dif_pclmulcrc32_pclmulcrc32c_intelghash_clmulni_inteligbiTCO_wdtaesni_inteliTCO_vendor_supportlrw gf128mulsb_edacptpedac_coreglue_helperlpc_ichioatdma pcspkrablk_helperpps_corei2c_i801mfd_corecryptd dca shpchpipmi_siwmiipmi_msghandleracpi_cpufreqnfsdauth_rpcgssnfs_acllockd grace sunrpc xfs libcrc32csr_modcdromsd_modmgag200 syscopyarea sysfillrect sysimgblti2c_algo_bitdrm_kms_helperisci ttm drm libsas ahci libahciscsi_transport_saslibatai2c_coredm_mirrordm_region_hashdm_logdm_modCPU: 27 PID: 3204 Comm: insmod Tainted: G OE 3.17.0+ #19 Hardware name: Intel Corporation S2600CP/S2600CP, BIOS RMLSDP.86I.00.29.D696.1311111329 11/11/2013 task: ffff880034e75160 ti: ffff8807fc5ac000 task.ti: ffff8807fc5ac000 RIP: 0010:[<ffffffff81076b81>] [<ffffffff81076b81>] warn_slowpath_common+0xc1/0xd0 RSP: 0018:ffff8807fc5afc68 EFLAGS: 00010246 RAX: 0000000000000021 RBX: ffff8807fc5afcb0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88081efee5f8 RDI: ffff88081efee5f8 RBP: ffff8807fc5afc98 R08: 0000000000000096 R09: 0000000000000000 R10: 0000000000000711 R11: ffff8807fc5af93e R12: ffffffffa0424070 R13: 0000000000000019 R14: ffffffffa0423068 R15: 0000000000000009 FS: 00007f2d4b034740(0000) GS:ffff88081efe0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2d4a99f3c0 CR3: 00000007fd88b000 CR4: 00000000001407e0 Stack: ffff8807fc5afcb8 ffffffff8199f020 ffff88080e396160 0000000000000000 ffffffffa0423040 ffffffffa0425000 ffff8807fc5afd08 ffffffff81076be5 0000000000000008 ffffffffa0424053 ffff880700000018 ffff8807fc5afd18 Call Trace: [<ffffffffa0423040>] ? dummy_greetings+0x40/0x40 [dummy_module] [<ffffffff81076be5>] warn_slowpath_fmt+0x55/0x70 [<ffffffffa0423068>] init_dummy+0x28/0x30 [dummy_module] [<ffffffff81002144>] do_one_initcall+0xd4/0x210 [<ffffffff811b52c2>] ? __vunmap+0xc2/0x110 [<ffffffff810f8889>] load_module+0x16a9/0x1b30 [<ffffffff810f3d30>] ? store_uevent+0x70/0x70 [<ffffffff810f49b9>] ? copy_module_from_fd.isra.44+0x129/0x180 [<ffffffff810f8ec6>] SyS_finit_module+0xa6/0xd0 [<ffffffff8166ce29>] system_call_fastpath+0x12/0x17 Code: c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 c7 c7 20 42 8a 81 31 c0 e8 fc80 5e 00 eb 80 48 c7 c7 78 42 8a 81 31 c0 e8 ec 80 5e 00 <0f> 0b 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55
RIP [<ffffffff81076b81>]warn_slowpath_common+0xc1/0xd0
RSP <ffff8807fc5afc68>---[ end trace 428218934a12088b ]---Quoted 6 lines
- Did you consider permitting this to be tweaked at runtime via /proc? Sometimes we get pesky WARNs at boot time and having runtime alteration would permit the user to prevent those from tripping a BUG.
I did actually, but I was wondering how people liked the idea before I looked at the /proc implementation. It's pretty much the same as
panic_on_oops, so it's not difficult to do.
Quoted 7 lines
Quoted 5 lines
- Also, perhaps
bug_on_warnshould be single-shot: clear itself after it has triggered one BUG. Because once the kernel has goneWARN->BUG, it's probably messed up and is likely to trigger more WARNs. Also, the kernel might generate many WARNs for the same issue.Okay, I'll add that.
When you update it, please CC me.
Your patch works well as follows:
WARNING: CPU: 3 PID: 468 at mm/page_alloc.c:4968 free_area_init_node+0x3fe/0x426() bug_on_warn set, calling BUG()... ------------[ cut here ]------------ kernel BUG at kernel/panic.c:434! invalid opcode: 0000 [#1] SMP <...> Workqueue: kacpi_hotplug acpi_hotplug_work_fn task: ffff880866d8c3d0 ti: ffff88086227c000 task.ti: ffff88086227c000 RIP: 0010:[<ffffffff81074001>] [<ffffffff81074001>] warn_slowpath_common+0xc1/0xd0 RSP: 0018:ffff88086227fa68 EFLAGS: 00010246 RAX: 0000000000000021 RBX: 0000000000000000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88087fa6e5f8 RDI: ffff88087fa6e5f8 RBP: ffff88086227fa98 R08: 0000000000000096 R09: 0000000000000000 R10: 0000000000000b37 R11: ffff88086227f73e R12: ffffffff818a9195 R13: 0000000000001368 R14: ffffffff81651b17 R15: 0000000000000009 FS: 0000000000000000(0000) GS:ffff88087fa60000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8301965000 CR3: 0000000001984000 CR4: 00000000001407e0 Stack: ffff88086f00e5e8 0000000000000002 ffff8a07fffb4000 0000000040000000 0000000000000002 0000000000000002 ffff88086227faa8 ffffffff8107410a ffff88086227fb38 ffffffff81651b17 0000000000000296 ffff88087f402120 Call Trace: [<ffffffff8107410a>] warn_slowpath_null+0x1a/0x20 [<ffffffff81651b17>] free_area_init_node+0x3fe/0x426 [<ffffffff810b9d22>] ? up+0x32/0x50 [<ffffffff8164ddd0>] hotadd_new_pgdat+0x90/0x110 [<ffffffff8164df24>] add_memory+0xd4/0x200 [<ffffffff813a8329>] acpi_memory_device_add+0x1aa/0x289 [<ffffffff8137b486>] acpi_bus_attach+0xfd/0x204 [<ffffffff8140fd1e>] ? device_register+0x1e/0x30 [<ffffffff8137b501>] acpi_bus_attach+0x178/0x204 [<ffffffff8137b5f7>] acpi_bus_scan+0x6a/0x90 [<ffffffff813786d2>] ? acpi_bus_get_status+0x2d/0x5f [<ffffffff8137b7a7>] acpi_device_hotplug+0xe8/0x418 [<ffffffff813750d9>] acpi_hotplug_work_fn+0x1f/0x2b [<ffffffff8108cc5e>] process_one_work+0x14e/0x3f0 [<ffffffff8108d36b>] worker_thread+0x11b/0x510 [<ffffffff8108d250>] ? rescuer_thread+0x350/0x350 [<ffffffff81092af1>] kthread+0xe1/0x100 [<ffffffff81092a10>] ? kthread_create_on_node+0x1b0/0x1b0 [<ffffffff81663efc>] ret_from_fork+0x7c/0xb0 [<ffffffff81092a10>] ? kthread_create_on_node+0x1b0/0x1b0 Code: c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 c7 c7 60 20 89 81 31 c0 e8 5c 26 5e 00 eb 80 48 c7 c7 b8 20 89 81 31 c0 e8 4c 26 5e 00 <0f> 0b 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 RIP [<ffffffff81074001>] warn_slowpath_common+0xc1/0xd0 RSP <ffff88086227fa68>
Thanks,
Yasuaki Ishimatsu
Quoted 26 lines
Quoted 8 lines
Quoted 8 lines
Documentation/kernel-parameters.txt +0
-0@@ -553,6 +553,8 @@ bytes respectively. Such letter suffixes can also be entirely omitted. bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner=Quoted 7 lines
Documentation/kernel-parameters.txt bytes respectively. Such letter suffixes can also be entirely omitted. · line 556
Quoted 1 line
Documentation/kernel-parameters.txt +1
-0+ bug_on_warn BUG() instead of WARN()There's no mention here that this feature is mainly aimed at generating a crash dump. How do we tell the people who aren't reading this email thread (ie: all of humanity except you and me ;)) that this feature even exists? Is there crash dump documentation that we can update?
I'll look into this too.
P.
Quoted 1 line
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/