Skip to contentSkip to conversation

kernel: taint when the driver firmware crashes

PATCH NAK Luis Chamberlain 6 participants 21 messages May 26, 2020 – Jun 3, 2020

Subsystems (11)
lore
Share & tools
lore

Luis Chamberlain

author maintainer
#0 lore raw

v3 cover kernel: taint when the driver firmware crashes

To those new on CC -- this is intended to be a simple generic interface to the kernel to annotate when the firwmare has crashed leaving the driver or system in a questionable state, in the worst case requiring full system reboot. This series is first addressing only a few networking patches, however, I already have an idea of where such firmware crashes happen across the tree. The goal with this series then is to first introduce the simple framework, and only if that moves forward will I continue to chug on with the rest of the drivers / subsystems.

This is *not* a networking specific problem only.

This v3 augments the last series by introducing the uevent for panic events, one of them is during tainting. The uvent mechanism is independent from any of this firmware taint mechanism. I've also addressed Jessica Yu's feedback. Given I've extended the patches a bit with other minor cleanup which checkpatch.pl complains over, and since this infrastructure is still being discussed, I've trimmed the patch series size to only cover drivers for which I've received an Acked-by from the respective driver maintainer, or where we have bug reports to support such dire situations on the driver such as ath10k.

During the last v2 it was discussed that we should instead use devlink for this work, however the initial RFC patches produced by Jakub Kicinski [0] shows how devlink is networking specific, and the intent behind this series is to produce simple helpers which can be used by *any* device driver, for any subsystem, not just networking. Subsystem specific infrastructure to help address firwmare crashes may still make sense, however that does not mean we *don't* need something even more generic regardless of the subsystem the issue happens on. Since uevents for taints are exposed, we now expose these through uapi as well, and that was something which eventually had to happen given that the current scheme of relying on sensible character representations for each taint will not scale beyond the alphabet.

This series is avaialble my 20200526-taint-firmware-net-intro branch, based on linux-next tag next-20200526 [1].

[0] https://lkml.kernel.org/r/[email protected]
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git/log/?h=20200526-taint-firmware-net-intro

Luis Chamberlain (7):
kernel.h: move taint and system state flags to uapi
panic: add uevent support
taint: add firmware crash taint support
panic: make taint data type clearer
ath10k: use new taint_firmware_crashed()
liquidio: use new taint_firmware_crashed()
qed: use new taint_firmware_crashed()

Vasundhara Volam (1):
bnxt_en: use new taint_firmware_crashed()

 Documentation/admin-guide/tainted-kernels.rst |   6 +
 MAINTAINERS                                   |   8 +
 .../net/ethernet/broadcom/bnxt/bnxt_devlink.c |   1 +
 .../net/ethernet/cavium/liquidio/lio_main.c   |   1 +
 drivers/net/ethernet/qlogic/qed/qed_mcp.c     |   1 +
 drivers/net/wireless/ath/ath10k/pci.c         |   2 +
 drivers/net/wireless/ath/ath10k/sdio.c        |   2 +
 drivers/net/wireless/ath/ath10k/snoc.c        |   1 +
 include/asm-generic/bug.h                     |   4 +-
 include/linux/kernel.h                        |  40 +--
 include/linux/module.h                        |  13 +
 include/linux/panic_events.h                  |  26 ++
 include/trace/events/module.h                 |   3 +-
 include/uapi/linux/kernel.h                   |  36 +++
 include/uapi/linux/panic_events.h             |  17 ++
 init/main.c                                   |   1 +
 kernel/Makefile                               |   1 +
 kernel/module.c                               |  13 +-
 kernel/panic.c                                |  16 +-
 kernel/panic_events.c                         | 289 ++++++++++++++++++
 lib/Kconfig.debug                             |  13 +
 tools/debugging/kernel-chktaint               |   7 +
 22 files changed, 454 insertions(+), 47 deletions(-)
 create mode 100644 include/linux/panic_events.h
 create mode 100644 include/uapi/linux/panic_events.h
 create mode 100644 kernel/panic_events.c
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH v3 1/8 kernel.h: move taint and system state flags to uapi

The taint and system state flags will be used in a subsequent patch exposing these to userspace, so move them to uapi. We keep the TAINT_FLAGS_COUNT outside of uapi, as this value can change per release.

---
 include/linux/kernel.h      | 34 +---------------------------------
 include/uapi/linux/kernel.h | 35 +++++++++++++++++++++++++++++++++++
 2 files changed, 36 insertions(+), 33 deletions(-)
include/linux/kernel.h +1 -33
index 82d91547d122..337634363d00 100644
@@ -569,40 +569,8 @@ extern unsigned long get_taint(void);
 extern int root_mountflags;
 
 extern bool early_boot_irqs_disabled;
+extern enum system_states  system_state;
 
-/*
- * Values used for system_state. Ordering of the states must not be changed
- * as code checks for <, <=, >, >= STATE.
- */
-extern enum system_states {
-	SYSTEM_BOOTING,
-	SYSTEM_SCHEDULING,
-	SYSTEM_RUNNING,
-	SYSTEM_HALT,
-	SYSTEM_POWER_OFF,
-	SYSTEM_RESTART,
-	SYSTEM_SUSPEND,
-} system_state;
-
-/* This cannot be an enum because some may be used in assembly source. */
-#define TAINT_PROPRIETARY_MODULE	0
-#define TAINT_FORCED_MODULE		1
-#define TAINT_CPU_OUT_OF_SPEC		2
-#define TAINT_FORCED_RMMOD		3
-#define TAINT_MACHINE_CHECK		4
-#define TAINT_BAD_PAGE			5
-#define TAINT_USER			6
-#define TAINT_DIE			7
-#define TAINT_OVERRIDDEN_ACPI_TABLE	8
-#define TAINT_WARN			9
-#define TAINT_CRAP			10
-#define TAINT_FIRMWARE_WORKAROUND	11
-#define TAINT_OOT_MODULE		12
-#define TAINT_UNSIGNED_MODULE		13
-#define TAINT_SOFTLOCKUP		14
-#define TAINT_LIVEPATCH			15
-#define TAINT_AUX			16
-#define TAINT_RANDSTRUCT		17
 #define TAINT_FLAGS_COUNT		18
 #define TAINT_FLAGS_MAX			((1UL << TAINT_FLAGS_COUNT) - 1)
 
include/uapi/linux/kernel.h +35 -0
index 0ff8f7477847..4bbd4093eb64 100644
@@ -12,4 +12,39 @@
 
 #define __KERNEL_DIV_ROUND_UP(n, d) (((n) + (d) - 1) / (d))
 
+/*
+ * Values used for system_state. Ordering of the states must not be changed
+ * as code checks for <, <=, >, >= STATE.
+ */
+enum system_states {
+	SYSTEM_BOOTING,
+	SYSTEM_SCHEDULING,
+	SYSTEM_RUNNING,
+	SYSTEM_HALT,
+	SYSTEM_POWER_OFF,
+	SYSTEM_RESTART,
+	SYSTEM_SUSPEND,
+};
+
+/* This cannot be an enum because some may be used in assembly source. */
+#define TAINT_PROPRIETARY_MODULE	0
+#define TAINT_FORCED_MODULE		1
+#define TAINT_CPU_OUT_OF_SPEC		2
+#define TAINT_FORCED_RMMOD		3
+#define TAINT_MACHINE_CHECK		4
+#define TAINT_BAD_PAGE			5
+#define TAINT_USER			6
+#define TAINT_DIE			7
+#define TAINT_OVERRIDDEN_ACPI_TABLE	8
+#define TAINT_WARN			9
+#define TAINT_CRAP			10
+#define TAINT_FIRMWARE_WORKAROUND	11
+#define TAINT_OOT_MODULE		12
+#define TAINT_UNSIGNED_MODULE		13
+#define TAINT_SOFTLOCKUP		14
+#define TAINT_LIVEPATCH			15
+#define TAINT_AUX			16
+#define TAINT_RANDSTRUCT		17
+/* be sure to update TAINT_FLAGS_COUNT when extending this */
+
 #endif /* _UAPI_LINUX_KERNEL_H */
Signature · 1 line
2.26.2
author maintainer
#2 lore raw

PATCH v3 2/8 panic: add uevent support

In reply to Luis Chamberlain

Add support to let panic events such as taints trigger uevents. If you don't have a journalctl -k -f window going and you're not actively monitoring what is going on in the kernel you may not realize that your kernel is hosed. Only those clueful that something may be off might inspect the kernel logs. Since not everyone is, add support to throw some bones to userspace that something might be fishy.

Let userspace figure out how to inform users, and what to do.

---
 MAINTAINERS                       |   8 +
 include/linux/panic_events.h      |  26 +++
 include/uapi/linux/panic_events.h |  17 ++
 init/main.c                       |   1 +
 kernel/Makefile                   |   1 +
 kernel/module.c                   |   2 +
 kernel/panic.c                    |   7 +-
 kernel/panic_events.c             | 289 ++++++++++++++++++++++++++++++
 lib/Kconfig.debug                 |  13 ++
 9 files changed, 363 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/panic_events.h
 create mode 100644 include/uapi/linux/panic_events.h
 create mode 100644 kernel/panic_events.c
MAINTAINERS +8 -0
index 3a003f310574..5bb467c0b36f 100644
@@ -12876,6 +12876,14 @@ L:	[email protected]
 S:	Maintained
 F:	drivers/platform/x86/panasonic-laptop.c
 
+PANIC EVENTS
+M:	Luis Chamberlain <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	include/linux/panic_events.h
+F:	include/uapi/linux/panic_events.h
+F:	kernel/panic_events.c
+
 PARALLAX PING IIO SENSOR DRIVER
 M:	Andreas Klinger <[email protected]>
 L:	[email protected]
include/linux/panic_events.h new file +26 -0
new file mode 100644
index 000000000000..8e7e331ecaaf
@@ -0,0 +1,26 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+#ifndef _LINUX_PANIC_EVENTS_H
+
+#include <linux/module.h>
+#include <uapi/linux/panic_events.h>
+
+#ifdef CONFIG_PANIC_EVENTS
+
+void panic_uevent(enum panic_uevent event);
+void panic_uevent_taint(unsigned int flag, struct module *mod);
+
+#else
+static inline void panic_events_init(void)
+{
+}
+
+static inline panic_uevent(enum panic_uevent event)
+{
+}
+
+static inline void panic_uevent_taint(unsigned int flag, struct module *mod)
+{
+}
+#endif
+
+#endif /* _LINUX_PANIC_EVENTS_H */
include/uapi/linux/panic_events.h new file +17 -0
new file mode 100644
index 000000000000..4f409597be52
@@ -0,0 +1,17 @@
+/* SPDX-License-Identifier: GPL-2.0+ */
+#ifndef _UAPI_PANIC_EVENTS_H
+#define _UAPI_PANIC_EVENTS_H
+
+/**
+ * enum panic_uevent - panic uevents
+ *
+ * @PANIC_LOCKDEP_DISABLED: lockdep has been disabled
+ * @PANIC_TAINT: lockdep has been disabled
+ */
+enum panic_uevent {
+	PANIC_LOCKDEP_DISABLED,
+	PANIC_TAINT,
+	__PANIC_MAX, /* non-ABI */
+};
+
+#endif /* _UAPI_PANIC_EVENTS_H */
init/main.c +1 -0
index 0ead83e86b5a..2bf33b5ec7b4 100644
@@ -96,6 +96,7 @@
 #include <linux/jump_label.h>
 #include <linux/mem_encrypt.h>
 #include <linux/kcsan.h>
+#include <linux/panic_events.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
kernel/Makefile +1 -0
index 0bd4ed7ca157..0c1794818579 100644
@@ -12,6 +12,7 @@ obj-y     = fork.o exec_domain.o panic.o \
 	    notifier.o ksysfs.o cred.o reboot.o \
 	    async.o range.o smpboot.o ucount.o
 
+obj-$(CONFIG_PANIC_EVENTS) += panic_events.o
 obj-$(CONFIG_MODULES) += kmod.o
 obj-$(CONFIG_MULTIUSER) += groups.o
 
kernel/module.c +2 -0
index 128bfc3e7ada..9b85d58441a2 100644
@@ -56,6 +56,7 @@
 #include <linux/bsearch.h>
 #include <linux/dynamic_debug.h>
 #include <linux/audit.h>
+#include <linux/panic_events.h>
 #include <uapi/linux/module.h>
 #include "module-internal.h"
 
@@ -330,6 +331,7 @@ static inline void add_taint_module(struct module *mod, unsigned flag,
 {
 	add_taint(flag, lockdep_ok);
 	set_bit(flag, &mod->taints);
+	panic_uevent_taint(flag, mod);
 }
 
 /*
kernel/panic.c +6 -1
index e2157ca387c8..48e9e2efa5bb 100644
@@ -30,6 +30,7 @@
 #include <linux/console.h>
 #include <linux/bug.h>
 #include <linux/ratelimit.h>
+#include <linux/panic_events.h>
 #include <linux/debugfs.h>
 #include <asm/sections.h>
 
@@ -440,8 +441,10 @@ unsigned long get_taint(void)
  */
 void add_taint(unsigned flag, enum lockdep_ok lockdep_ok)
 {
-	if (lockdep_ok == LOCKDEP_NOW_UNRELIABLE && __debug_locks_off())
+	if (lockdep_ok == LOCKDEP_NOW_UNRELIABLE && __debug_locks_off()) {
 		pr_warn("Disabling lock debugging due to kernel taint\n");
+		panic_uevent(PANIC_LOCKDEP_DISABLED);
+	}
 
 	set_bit(flag, &tainted_mask);
 
@@ -449,6 +452,8 @@ void add_taint(unsigned flag, enum lockdep_ok lockdep_ok)
 		panic_on_taint = 0;
 		panic("panic_on_taint set ...");
 	}
+
+	panic_uevent_taint(flag, NULL);
 }
 EXPORT_SYMBOL(add_taint);
 
kernel/panic_events.c new file +289 -0

289 lines not on this page · raw message

lib/Kconfig.debug +13 -0
index cf77b3881a21..966e8eaad09e 100644
@@ -835,6 +835,19 @@ config DEBUG_SHIRQ
 
 menu "Debug Oops, Lockups and Hangs"
 
+config PANIC_EVENTS
+	bool "Enable uevents relating to panics or taints"
+	help
+	  Say Y here to enable the kernel to send uevents relating to panics or
+	  when taint events happen. This may be useful if you want to craft some
+	  userspace component to analyze a taint, or inform the user of this
+	  event. These events are useful later in boot, prior to device driver
+	  initialization, so it won't capture events early in boot. The events
+	  are also best effort, if the system is low on memory some uevents
+	  not reach userspace.
+
+	  Say N if unsure.
+
 config PANIC_ON_OOPS
 	bool "Panic on Oops"
 	help
Signature · 1 line
2.26.2
author maintainer
#6 lore raw

PATCH v3 4/8 panic: make taint data type clearer

In reply to Luis Chamberlain

Let us be clearer about the the data type for the taint flag.

---
 include/asm-generic/bug.h | 4 ++--
 include/linux/kernel.h    | 4 ++--
 kernel/panic.c            | 8 ++++----
 3 files changed, 8 insertions(+), 8 deletions(-)
include/asm-generic/bug.h +2 -2
index c94e33ae3e7b..87dbe57301f4 100644
@@ -80,7 +80,7 @@ struct bug_entry {
  */
 #ifndef __WARN_FLAGS
 extern __printf(4, 5)
-void warn_slowpath_fmt(const char *file, const int line, unsigned taint,
+void warn_slowpath_fmt(const char *file, const int line, unsigned int taint,
 		       const char *fmt, ...);
 #define __WARN()		__WARN_printf(TAINT_WARN, NULL)
 #define __WARN_printf(taint, arg...) do {				\
@@ -110,7 +110,7 @@ extern __printf(1, 2) void __warn_printk(const char *fmt, ...);
 struct warn_args;
 struct pt_regs;
 
-void __warn(const char *file, int line, void *caller, unsigned taint,
+void __warn(const char *file, int line, void *caller, unsigned int taint,
 	    struct pt_regs *regs, struct warn_args *args);
 
 #ifndef WARN_ON
include/linux/kernel.h +2 -2
index a1974907c320..d154844eb9cd 100644
@@ -563,8 +563,8 @@ enum lockdep_ok {
 	LOCKDEP_STILL_OK,
 	LOCKDEP_NOW_UNRELIABLE
 };
-extern void add_taint(unsigned flag, enum lockdep_ok);
-extern int test_taint(unsigned flag);
+extern void add_taint(unsigned int flag, enum lockdep_ok);
+extern int test_taint(unsigned int flag);
 extern unsigned long get_taint(void);
 extern int root_mountflags;
 
kernel/panic.c +4 -4
index cb1c5619e983..3cfe84318ecf 100644
@@ -421,7 +421,7 @@ const char *print_tainted(void)
 	return buf;
 }
 
-int test_taint(unsigned flag)
+int test_taint(unsigned int flag)
 {
 	return test_bit(flag, &tainted_mask);
 }
@@ -440,7 +440,7 @@ unsigned long get_taint(void)
  * If something bad has gone wrong, you'll want @lockdebug_ok = false, but for
  * some notewortht-but-not-corrupting cases, it can be set to true.
  */
-void add_taint(unsigned flag, enum lockdep_ok lockdep_ok)
+void add_taint(unsigned int flag, enum lockdep_ok lockdep_ok)
 {
 	if (lockdep_ok == LOCKDEP_NOW_UNRELIABLE && __debug_locks_off()) {
 		pr_warn("Disabling lock debugging due to kernel taint\n");
@@ -579,7 +579,7 @@ struct warn_args {
 	va_list args;
 };
 
-void __warn(const char *file, int line, void *caller, unsigned taint,
+void __warn(const char *file, int line, void *caller, unsigned int taint,
 	    struct pt_regs *regs, struct warn_args *args)
 {
 	disable_trace_on_warning();
@@ -622,7 +622,7 @@ void __warn(const char *file, int line, void *caller, unsigned taint,
 }
 
 #ifndef __WARN_FLAGS
-void warn_slowpath_fmt(const char *file, int line, unsigned taint,
+void warn_slowpath_fmt(const char *file, int line, unsigned int taint,
 		       const char *fmt, ...)
 {
 	struct warn_args args;
Signature · 1 line
2.26.2
author maintainer
#7 lore raw

PATCH v3 5/8 ath10k: use new taint_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new taint_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

I have run into this situation with this driver with the latest firmware as of today, May 21, 2020 using v5.6.0, leaving me at a state at which my only option is to reboot. Driver removal and addition does not fix the situation. This is reported on kernel.org bugzilla korg#207851 [0]. But this isn't the first firmware crash reported, others have been filed before and none of these bugs have yet been addressed [1] [2] [3]. Including my own I see these firmware crash reports:

* korg#207851 [0]
* korg#197013 [1]
* korg#201237 [2]
* korg#195987 [3]

[0] https://bugzilla.kernel.org/show_bug.cgi?id=207851
[1] https://bugzilla.kernel.org/show_bug.cgi?id=197013
[2] https://bugzilla.kernel.org/show_bug.cgi?id=201237
[3] https://bugzilla.kernel.org/show_bug.cgi?id=195987

---
 drivers/net/wireless/ath/ath10k/pci.c  | 2 ++
 drivers/net/wireless/ath/ath10k/sdio.c | 2 ++
 drivers/net/wireless/ath/ath10k/snoc.c | 1 +
 3 files changed, 5 insertions(+)
drivers/net/wireless/ath/ath10k/pci.c +2 -0
index 1d941d53fdc9..818c3acc2468 100644
@@ -1767,6 +1767,7 @@ static void ath10k_pci_fw_dump_work(struct work_struct *work)
 		scnprintf(guid, sizeof(guid), "n/a");
 
 	ath10k_err(ar, "firmware crashed! (guid %s)\n", guid);
+	taint_firmware_crashed();
 	ath10k_print_driver_info(ar);
 	ath10k_pci_dump_registers(ar, crash_data);
 	ath10k_ce_dump_registers(ar, crash_data);
@@ -2837,6 +2838,7 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar,
 	if (ret) {
 		if (ath10k_pci_has_fw_crashed(ar)) {
 			ath10k_warn(ar, "firmware crashed during chip reset\n");
+			taint_firmware_crashed();
 			ath10k_pci_fw_crashed_clear(ar);
 			ath10k_pci_fw_crashed_dump(ar);
 		}
drivers/net/wireless/ath/ath10k/sdio.c +2 -0
index e2aff2254a40..8b2fc0b89be4 100644
@@ -794,6 +794,7 @@ static int ath10k_sdio_mbox_proc_dbg_intr(struct ath10k *ar)
 
 	/* TODO: Add firmware crash handling */
 	ath10k_warn(ar, "firmware crashed\n");
+	taint_firmware_crashed();
 
 	/* read counter to clear the interrupt, the debug error interrupt is
 	 * counter 0.
@@ -915,6 +916,7 @@ static int ath10k_sdio_mbox_proc_cpu_intr(struct ath10k *ar)
 	if (cpu_int_status & MBOX_CPU_STATUS_ENABLE_ASSERT_MASK) {
 		ath10k_err(ar, "firmware crashed!\n");
 		queue_work(ar->workqueue, &ar->restart_work);
+		taint_firmware_crashed();
 	}
 	return ret;
 }
drivers/net/wireless/ath/ath10k/snoc.c +1 -0
index 354d49b1cd45..071ee7607a4c 100644
@@ -1451,6 +1451,7 @@ void ath10k_snoc_fw_crashed_dump(struct ath10k *ar)
 		scnprintf(guid, sizeof(guid), "n/a");
 
 	ath10k_err(ar, "firmware crashed! (guid %s)\n", guid);
+	taint_firmware_crashed();
 	ath10k_print_driver_info(ar);
 	ath10k_msa_dump_memory(ar, crash_data);
 	mutex_unlock(&ar->dump_mutex);
Signature · 7 lines
2.26.2


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author maintainer
#5 lore raw

PATCH v3 3/8 taint: add firmware crash taint support

In reply to Luis Chamberlain

Device drivers firmware can crash, and *sometimes*, this can leave your system in a state which makes the device or subsystem completely useless. In the worst of cases not even removing the module and adding it back again will correct your situation and you are left with no other option but to do a full system reboot. Some drivers have work arounds for these situations, and sometimes they can recover the device / functionality but not all device drivers have these arrangements and in the worst cases requiring a full reboot is completely hidden from the user experience, leaving them dumbfounded with what has happened.

Detecting this by inspecting /proc/sys/kernel/tainted instead of scraping some magical words from the kernel log, which is driver specific, is much easier. So instead provide a helper which lets drivers annotate this.

Once this happens, scrapers can easily look for modules taint flags for a firmware crash. This will taint both the kernel and respective calling module.

The new helper taint_firmware_crashed() uses LOCKDEP_STILL_OK as this fact should in no way shape or form affect lockdep. This taint is device driver specific.

While extending the declaration of add_taint_module(), just make the flag unsigned int clear.

---
 Documentation/admin-guide/tainted-kernels.rst |  6 ++++++
 include/linux/kernel.h                        |  2 +-
 include/linux/module.h                        | 13 +++++++++++++
 include/trace/events/module.h                 |  3 ++-
 include/uapi/linux/kernel.h                   |  1 +
 kernel/module.c                               | 13 +++++++++----
 kernel/panic.c                                |  1 +
 tools/debugging/kernel-chktaint               |  7 +++++++
 8 files changed, 40 insertions(+), 6 deletions(-)
Documentation/admin-guide/tainted-kernels.rst +6 -0
index 71e9184a9079..92530f1d60ae 100644
@@ -100,6 +100,7 @@ Bit  Log  Number  Reason that got the kernel tainted
  15  _/K   32768  kernel has been live patched
  16  _/X   65536  auxiliary taint, defined for and used by distros
  17  _/T  131072  kernel was built with the struct randomization plugin
+ 18  _/Q  262144  driver firmware crash annotation
 ===  ===  ======  ========================================================
 
 Note: The character ``_`` is representing a blank in this table to make reading
@@ -162,3 +163,8 @@ More detailed explanation for tainting
      produce extremely unusual kernel structure layouts (even performance
      pathological ones), which is important to know when debugging. Set at
      build time.
+
+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware
+     has crashed and the device's operation has been severely affected. The
+     device may be left in a crippled state, requiring full driver removal /
+     addition, system reboot, or it is unclear how long recovery will take.
include/linux/kernel.h +1 -1
index 337634363d00..a1974907c320 100644
@@ -571,7 +571,7 @@ extern int root_mountflags;
 extern bool early_boot_irqs_disabled;
 extern enum system_states  system_state;
 
-#define TAINT_FLAGS_COUNT		18
+#define TAINT_FLAGS_COUNT		19
 #define TAINT_FLAGS_MAX			((1UL << TAINT_FLAGS_COUNT) - 1)
 
 struct taint_flag {
include/linux/module.h +13 -0
index 2e6670860d27..b3e143d2993e 100644
@@ -705,6 +705,14 @@ static inline bool is_livepatch_module(struct module *mod)
 bool is_module_sig_enforced(void);
 void set_module_sig_enforced(void);
 
+void add_taint_module(struct module *mod, unsigned int flag,
+		      enum lockdep_ok lockdep_ok);
+
+static inline void taint_firmware_crashed(void)
+{
+	add_taint_module(THIS_MODULE, TAINT_FIRMWARE_CRASH, LOCKDEP_STILL_OK);
+}
+
 #else /* !CONFIG_MODULES... */
 
 static inline struct module *__module_address(unsigned long addr)
@@ -852,6 +860,11 @@ void *dereference_module_function_descriptor(struct module *mod, void *ptr)
 	return ptr;
 }
 
+static inline void taint_firmware_crashed(void)
+{
+	add_taint(TAINT_FIRMWARE_CRASH, LOCKDEP_STILL_OK);
+}
+
 #endif /* CONFIG_MODULES */
 
 #ifdef CONFIG_SYSFS
include/trace/events/module.h +2 -1
index 097485c73c01..b749ea25affd 100644
@@ -26,7 +26,8 @@ struct module;
 	{ (1UL << TAINT_OOT_MODULE),		"O" },		\
 	{ (1UL << TAINT_FORCED_MODULE),		"F" },		\
 	{ (1UL << TAINT_CRAP),			"C" },		\
-	{ (1UL << TAINT_UNSIGNED_MODULE),	"E" })
+	{ (1UL << TAINT_UNSIGNED_MODULE),	"E" },		\
+	{ (1UL << TAINT_FIRMWARE_CRASH),	"Q" })
 
 TRACE_EVENT(module_load,
 
include/uapi/linux/kernel.h +1 -0
index 4bbd4093eb64..1e364659afca 100644
@@ -45,6 +45,7 @@ enum system_states {
 #define TAINT_LIVEPATCH			15
 #define TAINT_AUX			16
 #define TAINT_RANDSTRUCT		17
+#define TAINT_FIRMWARE_CRASH		18
 /* be sure to update TAINT_FLAGS_COUNT when extending this */
 
 #endif /* _UAPI_LINUX_KERNEL_H */
kernel/module.c +9 -4
index 9b85d58441a2..538def226332 100644
@@ -326,13 +326,18 @@ static inline int strong_try_module_get(struct module *mod)
 		return -ENOENT;
 }
 
-static inline void add_taint_module(struct module *mod, unsigned flag,
-				    enum lockdep_ok lockdep_ok)
+void add_taint_module(struct module *mod, unsigned int flag,
+		      enum lockdep_ok lockdep_ok)
 {
 	add_taint(flag, lockdep_ok);
-	set_bit(flag, &mod->taints);
-	panic_uevent_taint(flag, mod);
+
+	/* Skip this if the module is built-in */
+	if (mod) {
+		set_bit(flag, &mod->taints);
+		panic_uevent_taint(flag, mod);
+	}
 }
+EXPORT_SYMBOL_GPL(add_taint_module);
 
 /*
  * A thread that wants to hold a reference to a module only while it
kernel/panic.c +1 -0
index 48e9e2efa5bb..cb1c5619e983 100644
@@ -387,6 +387,7 @@ const struct taint_flag taint_flags[TAINT_FLAGS_COUNT] = {
 	[ TAINT_LIVEPATCH ]		= { 'K', ' ', true },
 	[ TAINT_AUX ]			= { 'X', ' ', true },
 	[ TAINT_RANDSTRUCT ]		= { 'T', ' ', true },
+	[ TAINT_FIRMWARE_CRASH ]	= { 'Q', ' ', true },
 };
 
 /**
tools/debugging/kernel-chktaint +7 -0
index 2240cb56e6e5..c397c6aabea7 100755
@@ -194,6 +194,13 @@ else
 	addout "T"
 	echo " * kernel was built with the struct randomization plugin (#17)"
 fi
+T=`expr $T / 2`
+if [ `expr $T % 2` -eq 0 ]; then
+	addout " "
+else
+	addout "Q"
+	echo " * a device driver's firmware has crashed (#18)"
+fi
 
 echo "For a more detailed explanation of the various taint flags see"
 echo " Documentation/admin-guide/tainted-kernels.rst in the the Linux kernel sources"
Signature · 1 line
2.26.2
author maintainer
#9 lore raw

PATCH v3 6/8 bnxt_en: use new taint_firmware_crashed()

In reply to Luis Chamberlain

From: Vasundhara Volam <[email protected]>

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c +1 -0
index a812beb46325..776a7ae0ef7f 100644
@@ -121,6 +121,7 @@ static int bnxt_fw_fatal_recover(struct devlink_health_reporter *reporter,
 	if (!priv_ctx)
 		return -EOPNOTSUPP;
 
+	taint_firmware_crashed();
 	bp->fw_health->fatal = true;
 	event = fw_reporter_ctx->sp_event;
 	if (event == BNXT_FW_RESET_NOTIFY_SP_EVENT)
Signature · 1 line
2.26.2
author maintainer
#11 lore raw

PATCH v3 7/8 liquidio: use new taint_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new taint_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/cavium/liquidio/lio_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/cavium/liquidio/lio_main.c +1 -0
index 66d31c018c7e..ee1796ea4818 100644
@@ -801,6 +801,7 @@ static int liquidio_watchdog(void *param)
 			continue;
 
 		WRITE_ONCE(oct->cores_crashed, true);
+		taint_firmware_crashed();
 		other_oct = get_other_octeon_device(oct);
 		if (other_oct)
 			WRITE_ONCE(other_oct->cores_crashed, true);
Signature · 1 line
2.26.2
author maintainer
#12 lore raw

PATCH v3 8/8 qed: use new taint_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new taint_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/qlogic/qed/qed_mcp.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/qlogic/qed/qed_mcp.c +1 -0
index 9624616806e7..dd4357b0b5d1 100644
@@ -566,6 +566,7 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		DP_NOTICE(p_hwfn,
 			  "The MFW failed to respond to command 0x%08x [param 0x%08x].\n",
 			  p_mb_params->cmd, p_mb_params->param);
+		taint_firmware_crashed();
 		qed_mcp_print_cpu_info(p_hwfn, p_ptt);
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
Signature · 1 line
2.26.2
maintainer Reviewed-by on v3 6/8
#10 lore raw

Re: [PATCH v3 6/8] bnxt_en: use new taint_firmware_crashed()

In reply to Luis Chamberlain
Quoted 9 lines · Luis Chamberlain

From: Vasundhara Volam <[email protected]>

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Quoted 7 lines

Using a taint flag allows us to annotate when this happens clearly.

maintainer Nacked-by on the v3 series NAK
#13 lore raw
In reply to Luis Chamberlain
Quoted 22 lines · On Tue, 26 May 2020 14:58:07 +0000 Luis Chamberlain wrote:

To those new on CC -- this is intended to be a simple generic interface to the kernel to annotate when the firwmare has crashed leaving the driver or system in a questionable state, in the worst case requiring full system reboot. This series is first addressing only a few networking patches, however, I already have an idea of where such firmware crashes happen across the tree. The goal with this series then is to first introduce the simple framework, and only if that moves forward will I continue to chug on with the rest of the drivers / subsystems.

This is *not* a networking specific problem only.

This v3 augments the last series by introducing the uevent for panic events, one of them is during tainting. The uvent mechanism is independent from any of this firmware taint mechanism. I've also addressed Jessica Yu's feedback. Given I've extended the patches a bit with other minor cleanup which checkpatch.pl complains over, and since this infrastructure is still being discussed, I've trimmed the patch series size to only cover drivers for which I've received an Acked-by from the respective driver maintainer, or where we have bug reports to support such dire situations on the driver such as ath10k.

Quoted 12 lines

During the last v2 it was discussed that we should instead use devlink for this work, however the initial RFC patches produced by Jakub Kicinski [0] shows how devlink is networking specific, and the intent behind this series is to produce simple helpers which can be used by *any* device driver, for any subsystem, not just networking. Subsystem specific infrastructure to help address firwmare crashes may still make sense, however that does not mean we *don't* need something even more generic regardless of the subsystem the issue happens on. Since uevents for taints are exposed, we now expose these through uapi as well, and that was something which eventually had to happen given that the current scheme of relying on sensible character representations for each taint will not scale beyond the alphabet.

author maintainer
#14 lore raw
Quoted 36 lines · Jakub Kicinski
Quoted 22 lines · On Tue, 26 May 2020 14:58:07 +0000 Luis Chamberlain wrote:

To those new on CC -- this is intended to be a simple generic interface to the kernel to annotate when the firwmare has crashed leaving the driver or system in a questionable state, in the worst case requiring full system reboot. This series is first addressing only a few networking patches, however, I already have an idea of where such firmware crashes happen across the tree. The goal with this series then is to first introduce the simple framework, and only if that moves forward will I continue to chug on with the rest of the drivers / subsystems.

This is *not* a networking specific problem only.

This v3 augments the last series by introducing the uevent for panic events, one of them is during tainting. The uvent mechanism is independent from any of this firmware taint mechanism. I've also addressed Jessica Yu's feedback. Given I've extended the patches a bit with other minor cleanup which checkpatch.pl complains over, and since this infrastructure is still being discussed, I've trimmed the patch series size to only cover drivers for which I've received an Acked-by from the respective driver maintainer, or where we have bug reports to support such dire situations on the driver such as ath10k.

Quoted 12 lines

During the last v2 it was discussed that we should instead use devlink for this work, however the initial RFC patches produced by Jakub Kicinski [0] shows how devlink is networking specific, and the intent behind this series is to produce simple helpers which can be used by *any* device driver, for any subsystem, not just networking. Subsystem specific infrastructure to help address firwmare crashes may still make sense, however that does not mean we *don't* need something even more generic regardless of the subsystem the issue happens on. Since uevents for taints are exposed, we now expose these through uapi as well, and that was something which eventually had to happen given that the current scheme of relying on sensible character representations for each taint will not scale beyond the alphabet.

Quoted 1 line

Care to elaborate?

Luis

maintainer
#15 lore raw
Quoted 37 lines · On Tue, 26 May 2020 23:07:48 +0000 Luis Chamberlain wrote:
Quoted 36 lines · Jakub Kicinski
Quoted 34 lines · On Tue, 26 May 2020 14:58:07 +0000 Luis Chamberlain wrote:

To those new on CC -- this is intended to be a simple generic interface to the kernel to annotate when the firwmare has crashed leaving the driver or system in a questionable state, in the worst case requiring full system reboot. This series is first addressing only a few networking patches, however, I already have an idea of where such firmware crashes happen across the tree. The goal with this series then is to first introduce the simple framework, and only if that moves forward will I continue to chug on with the rest of the drivers / subsystems.

This is *not* a networking specific problem only.

This v3 augments the last series by introducing the uevent for panic events, one of them is during tainting. The uvent mechanism is independent from any of this firmware taint mechanism. I've also addressed Jessica Yu's feedback. Given I've extended the patches a bit with other minor cleanup which checkpatch.pl complains over, and since this infrastructure is still being discussed, I've trimmed the patch series size to only cover drivers for which I've received an Acked-by from the respective driver maintainer, or where we have bug reports to support such dire situations on the driver such as ath10k.

During the last v2 it was discussed that we should instead use devlink for this work, however the initial RFC patches produced by Jakub Kicinski [0] shows how devlink is networking specific, and the intent behind this series is to produce simple helpers which can be used by *any* device driver, for any subsystem, not just networking. Subsystem specific infrastructure to help address firwmare crashes may still make sense, however that does not mean we *don't* need something even more generic regardless of the subsystem the issue happens on. Since uevents for taints are exposed, we now expose these through uapi as well, and that was something which eventually had to happen given that the current scheme of relying on sensible character representations for each taint will not scale beyond the alphabet.

Quoted 3 lines
Quoted 1 line

Care to elaborate?

I elaborated in the previous thread and told you I will nack this, but sure let's go over this again.

For the third time saying the devlink is networking specific is not true. It was created as a netlink configuration channel for devices when there is no networking reference that could be used. It can be compiled in or out much like sysfs.

And as I've shown you devlink already has the uAPI for what you're trying to achieve.

Regardless of your opinions about wider interfaces, networking drivers should implement devlink, and not have to sprinkle magic taint calls.

author maintainer
#16 lore raw
Quoted 38 lines · Jakub Kicinski
Quoted 37 lines · On Tue, 26 May 2020 23:07:48 +0000 Luis Chamberlain wrote:
Quoted 36 lines · Jakub Kicinski
Quoted 34 lines · On Tue, 26 May 2020 14:58:07 +0000 Luis Chamberlain wrote:

To those new on CC -- this is intended to be a simple generic interface to the kernel to annotate when the firwmare has crashed leaving the driver or system in a questionable state, in the worst case requiring full system reboot. This series is first addressing only a few networking patches, however, I already have an idea of where such firmware crashes happen across the tree. The goal with this series then is to first introduce the simple framework, and only if that moves forward will I continue to chug on with the rest of the drivers / subsystems.

This is *not* a networking specific problem only.

This v3 augments the last series by introducing the uevent for panic events, one of them is during tainting. The uvent mechanism is independent from any of this firmware taint mechanism. I've also addressed Jessica Yu's feedback. Given I've extended the patches a bit with other minor cleanup which checkpatch.pl complains over, and since this infrastructure is still being discussed, I've trimmed the patch series size to only cover drivers for which I've received an Acked-by from the respective driver maintainer, or where we have bug reports to support such dire situations on the driver such as ath10k.

During the last v2 it was discussed that we should instead use devlink for this work, however the initial RFC patches produced by Jakub Kicinski [0] shows how devlink is networking specific, and the intent behind this series is to produce simple helpers which can be used by *any* device driver, for any subsystem, not just networking. Subsystem specific infrastructure to help address firwmare crashes may still make sense, however that does not mean we *don't* need something even more generic regardless of the subsystem the issue happens on. Since uevents for taints are exposed, we now expose these through uapi as well, and that was something which eventually had to happen given that the current scheme of relying on sensible character representations for each taint will not scale beyond the alphabet.

Quoted 5 lines
Quoted 3 lines
Quoted 1 line

Care to elaborate?

I elaborated in the previous thread

No you didn't.

Quoted 1 line

and told you I will nack this,

That's all you said.

Quoted 6 lines

but sure let's go over this again.

For the third time saying the devlink is networking specific is not true. It was created as a netlink configuration channel for devices when there is no networking reference that could be used. It can be compiled in or out much like sysfs.

Perhaps I didn't get your email but this clarification was in no way shape or form present in your reply on that thread.

Quoted 2 lines

And as I've shown you devlink already has the uAPI for what you're trying to achieve.

I read your patch, and granted, I will accept I was under the incorrect assumption that this can only be used by networking devices, however it the devlink approach achieves getting userspace the ability with iproute2 devlink util to query a device health, on to which we can peg firmware health. But *this* patch series is not about health status and letting users query it, its about a *critical* situation which has come up with firmware requiring me to reboot my system, and the lack of *any* infrastructure in the kernel today to inform userspace about it.

So say we use netlink to report a critical health situation, how are we informing userspace with your patch series about requring a reboot?

Luis

maintainer
#17 lore raw
Quoted 11 lines · On Wed, 27 May 2020 03:19:18 +0000 Luis Chamberlain wrote:

I read your patch, and granted, I will accept I was under the incorrect assumption that this can only be used by networking devices, however it the devlink approach achieves getting userspace the ability with iproute2 devlink util to query a device health, on to which we can peg firmware health. But *this* patch series is not about health status and letting users query it, its about a *critical* situation which has come up with firmware requiring me to reboot my system, and the lack of *any* infrastructure in the kernel today to inform userspace about it.

So say we use netlink to report a critical health situation, how are we informing userspace with your patch series about requring a reboot?

One of main features of netlink is pub/sub model of notifications.

Whatever you imagine listening to your uevent can listen to devlink-health notifications via devlink.

In fact I've shown this off in the RFC patches I sent to you, see the devlink mon health command being used.

author maintainer
#18 lore raw
Quoted 13 lines · Jakub Kicinski
Quoted 11 lines · On Wed, 27 May 2020 03:19:18 +0000 Luis Chamberlain wrote:

I read your patch, and granted, I will accept I was under the incorrect assumption that this can only be used by networking devices, however it the devlink approach achieves getting userspace the ability with iproute2 devlink util to query a device health, on to which we can peg firmware health. But *this* patch series is not about health status and letting users query it, its about a *critical* situation which has come up with firmware requiring me to reboot my system, and the lack of *any* infrastructure in the kernel today to inform userspace about it.

So say we use netlink to report a critical health situation, how are we informing userspace with your patch series about requring a reboot?

Quoted 7 lines

One of main features of netlink is pub/sub model of notifications.

Whatever you imagine listening to your uevent can listen to devlink-health notifications via devlink.

In fact I've shown this off in the RFC patches I sent to you, see the devlink mon health command being used.

Yes but I looked at iputils2 devlink and seems I made an incorrect assumption this can only be used for a network device rather than a struct device.

I'll take a second look.

Luis

#19 lore raw
Quoted 19 lines · Luis Chamberlain
Quoted 18 lines · Jakub Kicinski
Quoted 11 lines · On Wed, 27 May 2020 03:19:18 +0000 Luis Chamberlain wrote:

I read your patch, and granted, I will accept I was under the incorrect assumption that this can only be used by networking devices, however it the devlink approach achieves getting userspace the ability with iproute2 devlink util to query a device health, on to which we can peg firmware health. But *this* patch series is not about health status and letting users query it, its about a *critical* situation which has come up with firmware requiring me to reboot my system, and the lack of *any* infrastructure in the kernel today to inform userspace about it.

So say we use netlink to report a critical health situation, how are we informing userspace with your patch series about requring a reboot?

One of main features of netlink is pub/sub model of notifications.

Whatever you imagine listening to your uevent can listen to devlink-health notifications via devlink.

Quoted 8 lines
Quoted 2 lines

In fact I've shown this off in the RFC patches I sent to you, see the devlink mon health command being used.

Yes but I looked at iputils2 devlink and seems I made an incorrect assumption this can only be used for a network device rather than a struct device.

I'll take a second look.

Hello Jakub,

I'm thinking about something similar to what Luis is proposing, but in my case I'd like to report just when the driver knows the hardware is gone and cannot be recovered, like when this is reported:

[ 2548.851832] WARNING: CPU: 3 PID: 98 at backports-4.19.98-1/net/mac80211/util.c:2040 ieee80211_reconfig+0x98/0xb64 [mac80211]
[ 2548.856020] Hardware became unavailable during restart.

I'd like to be able to tie this into a watch-dog program to allow automatic reboot of the system soon after this event is seen, for instance.

Could you post your devlink RFC patches somewhere public?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
author maintainer
#20 lore raw
Quoted 2 lines · Ben Greear

Could you post your devlink RFC patches somewhere public?

This cover letter provided a URL to these.

Luis

#3 lore raw

Re: [PATCH v3 2/8] panic: add uevent support

In reply to Luis Chamberlain
I love your patch! Yet something to improve:

Hi Luis,

I love your patch! Yet something to improve:

[auto build test ERROR on next-20200526]
[cannot apply to ath6kl/ath-next jeyu/modules-next tip/perf/core sparc-next/master linus/master asm-generic/master v5.7-rc7 v5.7-rc6 v5.7-rc5 v5.7-rc7] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Luis-Chamberlain/kernel-taint-when-the-driver-firmware-crashes/20200526-231556
base:    b0523c7b1c9d0edcd6c0fe6d2cb558a9ad5c60a8
config: i386-randconfig-m021-20200531 (attached as .config)
compiler: gcc-9 (Debian 9.3.0-13) 9.3.0

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> error: include/uapi/linux/panic_events.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier

make[2]: *** [scripts/Makefile.headersinst:63: usr/include/linux/panic_events.h] Error 1 make[2]: Target '__headers' not remade because of errors.
make[1]: *** [Makefile:1251: headers] Error 2
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:185: __sub-make] Error 2

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#8 lore raw

Re: [PATCH v3 5/8] ath10k: use new taint_firmware_crashed()

In reply to Luis Chamberlain
Quoted 6 lines · Luis Chamberlain

This makes use of the new taint_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Just for the record, the underlying problem you seem to be complaining about does not appear to be a firmware crash at all. It does happen to result in a firmware crash report much later on (because when the PCIe endpoint is this hosed, sooner or later the driver thinks the firmware is dead), but it's not likely the root cause. More below.

Quoted 7 lines

Using a taint flag allows us to annotate when this happens clearly.

I have run into this situation with this driver with the latest firmware as of today, May 21, 2020 using v5.6.0, leaving me at a state at which my only option is to reboot. Driver removal and addition does not fix the situation. This is reported on kernel.org bugzilla korg#207851 [0].

I took a look, and replied there:
https://bugzilla.kernel.org/show_bug.cgi?id=207851#c2

Per the above, it seems more likely you have a PCI or power management problem, not an ath10k or ath10k-firmware problem.

Quoted 4 lines

But this isn't the first firmware crash reported,
others have been filed before and none of these bugs have yet been addressed [1] [2] [3]. Including my own I see these firmware crash reports:

Yes, firmware does crash. Sometimes repeatedly. It also happens to be closed source, so it's nearly impossible for the average Linux dev to debug. But FWIW, those 3 all appear to be recoverable -- and then they crash again a few minutes later. So just as claimed on prior iterations of this patchset, ath10k is doing fine at recovery [*] -- it's "only" the firmware that's a problem. (And, if a WiFi firmware doesn't like something in the RF environment...it's totally understandable that the crash will happen more than once. Of course that sucks, but it's not unexpected.) Crucially, rebooting won't really do anything to help these people, AIUI.

Maybe what you really want is to taint the kernel every time a non-free firmware is loaded ;)

I'd also note that those 3 reports are 3 years old. There have been many ath10k-firmware updates since then, so it's not necessarily fair to dig those back up. Also, bugzilla.kernel.org is totally ignored by many linux-wireless@ folks. But I digress...

All in all, I have no interest in this proposal, for many of the reasons already mentioned on previous iterations. It's way too coarse and won't be useful in understanding what's going on in a system, IMO, at least for ath10k. But it's also easy enough to ignore, so if it makes somebody happy to claim a taint, then so be it.

Regards,
Brian

[*] Although, at least one of those doesn't appear to be as "clean" of a recovery attempt as typical. Maybe there are some lurking driver bugs in there too.

Quoted 9 lines

* korg#207851 [0]
* korg#197013 [1]
* korg#201237 [2]
* korg#195987 [3]

[0] https://bugzilla.kernel.org/show_bug.cgi?id=207851
[1] https://bugzilla.kernel.org/show_bug.cgi?id=197013
[2] https://bugzilla.kernel.org/show_bug.cgi?id=201237
[3] https://bugzilla.kernel.org/show_bug.cgi?id=195987

#4 lore raw

Re: [PATCH v3 2/8] panic: add uevent support

In reply to Luis Chamberlain
I love your patch! Yet something to improve:

Hi Luis,

I love your patch! Yet something to improve:

[auto build test ERROR on next-20200526]
[cannot apply to ath6kl/ath-next jeyu/modules-next tip/perf/core sparc-next/master linus/master asm-generic/master v5.7-rc7 v5.7-rc6 v5.7-rc5 v5.7] [if your patch is applied to the wrong git tree, please drop us a note to help
improve the system. BTW, we also suggest to use '--base' option to specify the
base tree in git format-patch, please see https://stackoverflow.com/a/37406982]

url:    https://github.com/0day-ci/linux/commits/Luis-Chamberlain/kernel-taint-when-the-driver-firmware-crashes/20200526-231556
base:    b0523c7b1c9d0edcd6c0fe6d2cb558a9ad5c60a8
config: riscv-randconfig-r016-20200603 (attached as .config)
compiler: clang version 11.0.0 (https://github.com/llvm/llvm-project 16437992cac249f6fe1efd392d20e3469b47e39e)

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# install riscv cross compiling tool for clang build
# apt-get install binutils-riscv64-linux-gnu
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=clang make.cross ARCH=riscv

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>, old ones prefixed by <<):

>> error: include/uapi/linux/panic_events.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier

make[2]: *** [scripts/Makefile.headersinst:63: usr/include/linux/panic_events.h] Error 1
make[2]: Target '__headers' not remade because of errors.
make[1]: *** [Makefile:1251: headers] Error 2
arch/riscv/kernel/asm-offsets.c:14:6: warning: no previous prototype for function 'asm_offsets' [-Wmissing-prototypes] void asm_offsets(void)
^
arch/riscv/kernel/asm-offsets.c:14:1: note: declare 'static' if the function is not intended to be used outside of this translation unit void asm_offsets(void)
^
static
1 warning generated.
make[1]: Target 'prepare' not remade because of errors.
make: *** [Makefile:185: __sub-make] Error 2

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org