kernel, add bug_on_warn
Subsystems (6)
PATCH v3 1/1 kernel, add bug_on_warn
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a WARN() in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.
A much easier method would be a switch to change the WARN() over to a BUG(). This makes debugging easier in that I can now test the actual image the WARN() was seen on and I do not have to engage in remote debugging.
This patch adds a bug_on_warn kernel parameter, which calls BUG() in the warn_slowpath_common() path. The function will still print out the location of the warning.
An example of the bug_on_warn output:
The first line below is from the WARN_ON() to output the WARN_ON()'s location. After that the new BUG() call is displayed.
WARNING: CPU: 27 PID: 3204 at /home/rhel7/redhat/debug/dummy-module/dummy-module.c:25 init_dummy+0x28/0x30 [dummy_module]() bug_on_warn set, calling BUG()... ------------[ cut here ]------------ kernel BUG at kernel/panic.c:434! invalid opcode: 0000 [#1] SMP Modules linked in: dummy_module(OE+) sg nfsv3 rpcsec_gss_krb5 nfsv4
dns_resolver nfs fscache cfg80211 rfkill x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel igb iTCO_wdt aesni_intel iTCO_vendor_support lrw gf128mul sb_edac ptp edac_core glue_helper lpc_ich ioatdma pcspkr ablk_helper pps_core i2c_i801 mfd_core cryptd dca shpchp ipmi_si wmi ipmi_msghandler acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom sd_mod mgag200 syscopyarea sysfillrect sysimgblt i2c_algo_bit drm_kms_helper isci ttm drm libsas ahci libahci scsi_transport_sas libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 27 PID: 3204 Comm: insmod Tainted: G OE 3.17.0+ #19 Hardware name: Intel Corporation S2600CP/S2600CP, BIOS RMLSDP.86I.00.29.D696.1311111329 11/11/2013 task: ffff880034e75160 ti: ffff8807fc5ac000 task.ti: ffff8807fc5ac000 RIP: 0010:[<ffffffff81076b81>] [<ffffffff81076b81>] warn_slowpath_common+0xc1/0xd0 RSP: 0018:ffff8807fc5afc68 EFLAGS: 00010246 RAX: 0000000000000021 RBX: ffff8807fc5afcb0 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff88081efee5f8 RDI: ffff88081efee5f8 RBP: ffff8807fc5afc98 R08: 0000000000000096 R09: 0000000000000000 R10: 0000000000000711 R11: ffff8807fc5af93e R12: ffffffffa0424070 R13: 0000000000000019 R14: ffffffffa0423068 R15: 0000000000000009 FS: 00007f2d4b034740(0000) GS:ffff88081efe0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2d4a99f3c0 CR3: 00000007fd88b000 CR4: 00000000001407e0 Stack: ffff8807fc5afcb8 ffffffff8199f020 ffff88080e396160 0000000000000000 ffffffffa0423040 ffffffffa0425000 ffff8807fc5afd08 ffffffff81076be5 0000000000000008 ffffffffa0424053 ffff880700000018 ffff8807fc5afd18 Call Trace: [<ffffffffa0423040>] ? dummy_greetings+0x40/0x40 [dummy_module] [<ffffffff81076be5>] warn_slowpath_fmt+0x55/0x70 [<ffffffffa0423068>] init_dummy+0x28/0x30 [dummy_module] [<ffffffff81002144>] do_one_initcall+0xd4/0x210 [<ffffffff811b52c2>] ? __vunmap+0xc2/0x110 [<ffffffff810f8889>] load_module+0x16a9/0x1b30 [<ffffffff810f3d30>] ? store_uevent+0x70/0x70 [<ffffffff810f49b9>] ? copy_module_from_fd.isra.44+0x129/0x180 [<ffffffff810f8ec6>] SyS_finit_module+0xa6/0xd0 [<ffffffff8166ce29>] system_call_fastpath+0x12/0x17 Code: c4 08 5b 41 5c 41 5d 41 5e 41 5f 5d c3 48 c7 c7 20 42 8a 81 31 c0 e8 fc
80 5e 00 eb 80 48 c7 c7 78 42 8a 81 31 c0 e8 ec 80 5e 00 <0f> 0b 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55
RIP [<ffffffff81076b81>] warn_slowpath_common+0xc1/0xd0
RSP <ffff8807fc5afc68>
---[ end trace 428218934a12088b ]---
Successfully tested by me.
[v2]: add /proc/sys/kernel/bug_on_warn, additional documentation, modify
!slowpath cases
[v3]: use proc_dointvec_minmax() in sysctl handler
--- Documentation/kdump/kdump.txt | 7 +++++++ Documentation/kernel-parameters.txt | 3 +++ Documentation/sysctl/kernel.txt | 12 ++++++++++++ include/asm-generic/bug.h | 12 ++++++++++-- include/linux/kernel.h | 1 + include/uapi/linux/sysctl.h | 1 + kernel/panic.c | 21 ++++++++++++++++++++- kernel/sysctl.c | 9 +++++++++ kernel/sysctl_binary.c | 1 + 9 files changed, 64 insertions(+), 3 deletions(-)
Documentation/kdump/kdump.txt +7 -0
@@ -471,6 +471,13 @@ format. Crash is available on Dave Anderson's site at the following URL: http://people.redhat.com/~anderson/ +Trigger Kdump on WARN() +======================= + +The kernel parameter, bug_on_warn, calls BUG() in all WARN() paths. This +will cause a kdump to occur at the BUG() call. In cases where a user +wants to specify this during runtime, /proc/sys/kernel/bug_on_warn can be +set to 1 to achieve the same behaviour. Contact =======
Documentation/kernel-parameters.txt +3 -0
@@ -553,6 +553,9 @@ bytes respectively. Such letter suffixes can also be entirely omitted. bttv.pll= See Documentation/video4linux/bttv/Insmod-options bttv.tuner= + bug_on_warn BUG() instead of WARN(). Useful to cause kdump + on a WARN(). + bulk_remove=off [PPC] This parameter disables the use of the pSeries firmware feature for flushing multiple hpte entries at a time.
Documentation/sysctl/kernel.txt +12 -0
@@ -23,6 +23,7 @@ show up in /proc/sys/kernel: - auto_msgmni - bootloader_type [ X86 only ] - bootloader_version [ X86 only ] +- bug_on_warn - callhome [ S390 only ] - cap_last_cap - core_pattern
@@ -152,6 +153,17 @@ Documentation/x86/boot.txt for additional information. ============================================================== +bug_on_warn: + +Calls BUG() in the WARN() path when set to 1. This is useful to avoid +a kernel rebuild when attempting to kdump at the location of a WARN(). + +0: only WARN(), default behaviour. + +1: call BUG() after printing out WARN() location. + +============================================================== + callhome: Controls the kernel's callhome behavior in case of a kernel panic.
include/asm-generic/bug.h +10 -2
@@ -75,10 +75,18 @@ extern void warn_slowpath_null(const char *file, const int line); #define __WARN_printf_taint(taint, arg...) \ warn_slowpath_fmt_taint(__FILE__, __LINE__, taint, arg)#else -#define __WARN () __WARN_TAINT(TAINT_WARN) +#define check_bug_on_warn () \ + do { \ + if (bug_on_warn) \ + BUG(); \ + } while (0) + +#define __WARN () \ + do { __WARN_TAINT(TAINT_WARN); check_bug_on_warn(); } while (0) + #define __WARN_printf(arg...) do { printk(arg); __WARN(); } while (0) #define __WARN_printf_taint(taint, arg...) \ - do { printk(arg); __WARN_TAINT(taint); } while (0) + do { printk(arg); __WARN_TAINT(taint); check_bug_on_warn(); } while (0)#endif #ifndef WARN_ON
include/linux/kernel.h +1 -0
@@ -422,6 +422,7 @@ extern int panic_on_oops;extern int panic_on_unrecovered_nmi;extern int panic_on_io_nmi;extern int sysctl_panic_on_stackoverflow; +extern int bug_on_warn;/* * Only to be used by arch init code. If the user over-wrote the default * CONFIG_PANIC_TIMEOUT, honor it.
include/uapi/linux/sysctl.h +1 -0
@@ -153,6 +153,7 @@ enumKERN_MAX_LOCK_DEPTH =74 ,/* int: rtmutex's maximum lock depth */ KERN_NMI_WATCHDOG =75 ,/* int: enable/disable nmi watchdog */ KERN_PANIC_ON_NMI =76 ,/* int: whether we will panic on an unrecovered */ + KERN_BUG_ON_WARN=77, /* int: call BUG() in WARN() functions */ };
kernel/panic.c +20 -1
@@ -33,6 +33,7 @@ static int pause_on_oops;static int pause_on_oops_flag;static DEFINE_SPINLOCK (pause_on_oops_lock);static bool crash_kexec_post_notifiers; +int bug_on_warn;int panic_timeout =CONFIG_PANIC_TIMEOUT ;EXPORT_SYMBOL_GPL (panic_timeout);
@@ -420,13 +421,24 @@ static void warn_slowpath_common(const char *file, int line, void *caller, {disable_trace_on_warning (); -pr_warn ("------------[ cut here ]------------\n" ); +if (!bug_on_warn) +pr_warn ("------------[ cut here ]------------\n" );pr_warn ("WARNING: CPU: %d PID: %d at %s:%d %pS()\n" ,raw_smp_processor_id (), current->pid, file, line, caller);if (args)vprintk (args->fmt, args->args); +if (bug_on_warn) { +pr_warn ("bug_on_warn set, calling BUG()...\n" ); +/* +* A flood of WARN()s may occur. Prevent further WARN()s +* from panicking the system. +*/ + bug_on_warn =0 ; +BUG (); + } +print_modules ();dump_stack ();print_oops_end_marker ();
@@ -501,3 +513,10 @@ static int __init oops_setup(char *s)return 0 ; }early_param ("oops" , oops_setup); + +static int __init bug_on_warn_setup (char *s) +{ + bug_on_warn =1 ; +return 0 ; +} +early_param ("bug_on_warn" , bug_on_warn_setup);
kernel/sysctl.c +9 -0
@@ -1103,6 +1103,15 @@ static struct ctl_table kern_table[] = { .proc_handler = proc_dointvec, },#endif + { + .procname = "bug_on_warn", + .data = &bug_on_warn, + .maxlen = sizeof(int), + .mode = 0644, + .proc_handler = proc_dointvec_minmax, + .extra1 = &zero, + .extra2 = &one, + }, { } };
kernel/sysctl_binary.c +1 -0
@@ -137,6 +137,7 @@ static const struct bin_table bin_kern_table[] = { { CTL_INT, KERN_COMPAT_LOG, "compat-log" }, { CTL_INT, KERN_MAX_LOCK_DEPTH, "max_lock_depth" }, { CTL_INT, KERN_PANIC_ON_NMI, "panic_on_unrecovered_nmi" }, + { CTL_INT, KERN_BUG_ON_WARN, "bug_on_warn" }, {} };
Signature · 1 line
1.7.9.3
Quoted 6 lines · On Thu, 23 Oct 2014 08:53:14 -0400 Prarit Bhargava…
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a
WARN()in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.
Quoted 8 lines
A much easier method would be a switch to change the
WARN()over to aBUG(). This makes debugging easier in that I can now test the actual image theWARN()was seen on and I do not have to engage in remote debugging.This patch adds a
bug_on_warnkernel parameter, which callsBUG()in thewarn_slowpath_common()path. The function will still print out the location of the warning.
The changelog doesn't mention /proc/sys/kernel/bug_on_warn?
Why do we need both the sysctl and the kernel parameter? Only to trigger BUG for warnings which occur prior to initscripts. Is there a legitimate case for this? Is kdump even usable at this time?
include/asm-generic/bug.h extern void warn_slowpath_null(const char *file, const int line); · line 82
Quoted 12 lines
include/asm-generic/bug.h +5
-1@@ -75,10 +75,18 @@ extern void warn_slowpath_null(const char *file, const int line); #define __WARN_printf_taint(taint, arg...) \ warn_slowpath_fmt_taint(__FILE__, __LINE__, taint, arg)#else -#define __WARN () __WARN_TAINT(TAINT_WARN) +#define check_bug_on_warn () \ + do { \ + if (bug_on_warn) \ + BUG(); \ + } while (0)
#define check_bug_on_warn() BUG_ON(bug_on_warn)
would suffice?
include/asm-generic/bug.h extern void warn_slowpath_null(const char *file, const int line); · line 90
Quoted 8 lines
include/asm-generic/bug.h +4
-1+#define __WARN () \ + do { __WARN_TAINT(TAINT_WARN); check_bug_on_warn(); } while (0) + #define __WARN_printf(arg...) do { printk(arg); __WARN(); } while (0) #define __WARN_printf_taint(taint, arg...) \ - do { printk(arg); __WARN_TAINT(taint); } while (0) + do { printk(arg); __WARN_TAINT(taint); check_bug_on_warn(); } while (0)#endif
What's this code here for anyway? The changes towarn_slowpath_common() aren't sufficient?
Quoted 30 lines
include/asm-generic/bug.h +0
-0#ifndef WARN_ON include/linux/kernel.h +1
-0@@ -422,6 +422,7 @@ extern int panic_on_oops;extern int panic_on_unrecovered_nmi;extern int panic_on_io_nmi;extern int sysctl_panic_on_stackoverflow; +extern int bug_on_warn;/* * Only to be used by arch init code. If the user over-wrote the default * CONFIG_PANIC_TIMEOUT, honor it. include/uapi/linux/sysctl.h +1
-0@@ -153,6 +153,7 @@ enumKERN_MAX_LOCK_DEPTH =74 ,/* int: rtmutex's maximum lock depth */ KERN_NMI_WATCHDOG =75 ,/* int: enable/disable nmi watchdog */ KERN_PANIC_ON_NMI =76 ,/* int: whether we will panic on an unrecovered */ + KERN_BUG_ON_WARN=77, /* int: call BUG() in WARN() functions */ };kernel/panic.c +0
-0
kernel/panic.c static int pause_on_oops; · line 36
Quoted 4 lines
kernel/panic.c +1
-0@@ -33,6 +33,7 @@ static int pause_on_oops;static int pause_on_oops_flag;static DEFINE_SPINLOCK (pause_on_oops_lock);static bool crash_kexec_post_notifiers; +int bug_on_warn;
I suppose this should be __read_mostly. Assuming __read_mostly is useful :(
Quoted 3 lines
...
Quoted 13 lines · Andrew Morton
Quoted 11 lines · On Thu, 23 Oct 2014 08:53:14 -0400 Prarit Bhargava…
There have been several times where I have had to rebuild a kernel to cause a panic when hitting a
WARN()in the code in order to get a crash dump from a system. Sometimes this is easy to do, other times (such as in the case of a remote admin) it is not trivial to send new images to the user.A much easier method would be a switch to change the
WARN()over to aBUG(). This makes debugging easier in that I can now test the actual image theWARN()was seen on and I do not have to engage in remote debugging.
Quoted 5 lines
Quoted 3 lines
This patch adds a
bug_on_warnkernel parameter, which callsBUG()in thewarn_slowpath_common()path. The function will still print out the location of the warning.The changelog doesn't mention
/proc/sys/kernel/bug_on_warn?
Oops ... I'll put that into a V4.
Quoted 4 lines
Why do we need both the sysctl and the kernel parameter? Only to trigger BUG for warnings which occur prior to initscripts. Is there a legitimate case for this? Is kdump even usable at this time?
The kernel parameter is more of a catch-all IMO. I'm going to be instructing users to add it as a kernel parameter because I've found that most users are willing to do that than muck with echo'ing into a file.
As Rusty pointed out, however, there may be cases where we're caught by an early WARN() in the boot sequence and we want to avoid BUG()'ing there, so that we catch the "run-time" WARN().
Quoted 16 lines
include/asm-generic/bug.h extern void warn_slowpath_null(const char *file, const int line); · line 82
Quoted 13 lines
include/asm-generic/bug.h +5
-1@@ -75,10 +75,18 @@ extern void warn_slowpath_null(const char *file, const int line); #define __WARN_printf_taint(taint, arg...) \ warn_slowpath_fmt_taint(__FILE__, __LINE__, taint, arg)#else -#define __WARN () __WARN_TAINT(TAINT_WARN) +#define check_bug_on_warn () \ + do { \ + if (bug_on_warn) \ + BUG(); \ + } while (0)#define
check_bug_on_warn()BUG_ON(bug_on_warn)
Quoted 2 lines
would suffice?
I'll fix that.
Quoted 11 lines
include/asm-generic/bug.h extern void warn_slowpath_null(const char *file, const int line); · line 90
Quoted 8 lines
include/asm-generic/bug.h +4
-1+#define __WARN () \ + do { __WARN_TAINT(TAINT_WARN); check_bug_on_warn(); } while (0) + #define __WARN_printf(arg...) do { printk(arg); __WARN(); } while (0) #define __WARN_printf_taint(taint, arg...) \ - do { printk(arg); __WARN_TAINT(taint); } while (0) + do { printk(arg); __WARN_TAINT(taint); check_bug_on_warn(); } while (0)#endif What's this code here for anyway? The changes to
warn_slowpath_common()aren't sufficient?
Oh ... you're right. I got confused by the WANT_WARN_ON_SLOWPATH config option.
I'll clean this up.
Quoted 27 lines
Quoted 27 lines
include/asm-generic/bug.h +0
-0#ifndef WARN_ON include/linux/kernel.h +1
-0@@ -422,6 +422,7 @@ extern int panic_on_oops;extern int panic_on_unrecovered_nmi;extern int panic_on_io_nmi;extern int sysctl_panic_on_stackoverflow; +extern int bug_on_warn;/* * Only to be used by arch init code. If the user over-wrote the default * CONFIG_PANIC_TIMEOUT, honor it. include/uapi/linux/sysctl.h +1
-0@@ -153,6 +153,7 @@ enumKERN_MAX_LOCK_DEPTH =74 ,/* int: rtmutex's maximum lock depth */ KERN_NMI_WATCHDOG =75 ,/* int: enable/disable nmi watchdog */ KERN_PANIC_ON_NMI =76 ,/* int: whether we will panic on an unrecovered */ + KERN_BUG_ON_WARN=77, /* int: call BUG() in WARN() functions */ };
Quoted 12 lines
kernel/panic.c static int pause_on_oops; · line 36
Quoted 9 lines
kernel/panic.c +1
-0@@ -33,6 +33,7 @@ static int pause_on_oops;static int pause_on_oops_flag;static DEFINE_SPINLOCK (pause_on_oops_lock);static bool crash_kexec_post_notifiers; +int bug_on_warn;I suppose this should be
__read_mostly. Assuming__read_mostlyis useful :(
I'll change this too.
P.
Quoted 4 lines
Quoted 4 lines
...