Skip to contentSkip to conversation

net: taint when the device driver firmware crashes

PATCH Superseded Luis Chamberlain 13 participants 81 messages May 15, 2020 – Jul 30, 2020

Subsystems (17)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Luis Chamberlain

author maintainer
#0 lore raw

v2 cover net: taint when the device driver firmware crashes

On this v2 I've added documenation over the taint flag, and updated our script which parses existing taint flags to describe what has happened when this taint flag is found. I've also updated the location of the taint flag on the qed driver and updated the reviews.

The changes are based on linux-next tag next-20200515. You can find these changes on my tree:

https://git.kernel.org/pub/scm/linux/kernel/git/mcgrof/linux-next.git/log/?h=20200515-taint-firmware-net

Luis Chamberlain (15):
taint: add module firmware crash taint support ethernet/839: use new module_firmware_crashed()
bnx2x: use new module_firmware_crashed()
bnxt: use new module_firmware_crashed()
bna: use new module_firmware_crashed()
liquidio: use new module_firmware_crashed()
cxgb4: use new module_firmware_crashed()
ehea: use new module_firmware_crashed()
qed: use new module_firmware_crashed()
soc: qcom: ipa: use new module_firmware_crashed() wimax/i2400m: use new module_firmware_crashed()
ath10k: use new module_firmware_crashed()
ath6kl: use new module_firmware_crashed()
brcm80211: use new module_firmware_crashed()
mwl8k: use new module_firmware_crashed()

 Documentation/admin-guide/tainted-kernels.rst       |  6 ++++++
 drivers/net/ethernet/8390/axnet_cs.c                |  4 +++-
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c    |  1 +
 drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c   |  1 +
 drivers/net/ethernet/brocade/bna/bfa_ioc.c          |  1 +
 drivers/net/ethernet/cavium/liquidio/lio_main.c     |  1 +
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c     |  1 +
 drivers/net/ethernet/ibm/ehea/ehea_main.c           |  2 ++
 drivers/net/ethernet/qlogic/qed/qed_mcp.c           |  1 +
 drivers/net/ipa/ipa_modem.c                         |  1 +
 drivers/net/wimax/i2400m/rx.c                       |  1 +
 drivers/net/wireless/ath/ath10k/pci.c               |  2 ++
 drivers/net/wireless/ath/ath10k/sdio.c              |  2 ++
 drivers/net/wireless/ath/ath10k/snoc.c              |  1 +
 drivers/net/wireless/ath/ath6kl/hif.c               |  1 +
 .../net/wireless/broadcom/brcm80211/brcmfmac/core.c |  1 +
 drivers/net/wireless/marvell/mwl8k.c                |  1 +
 include/linux/kernel.h                              |  3 ++-
 include/linux/module.h                              | 13 +++++++++++++
 include/trace/events/module.h                       |  3 ++-
 kernel/module.c                                     |  5 +++--
 kernel/panic.c                                      |  1 +
 tools/debugging/kernel-chktaint                     |  7 +++++++
 23 files changed, 55 insertions(+), 5 deletions(-)
Signature · 1 line
2.26.2
author maintainer
#7 lore raw

PATCH v2 3/15 bnx2x: use new module_firmware_crashed()

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +1 -0
index db5107e7937c..c38b8c9c8af0 100644
@@ -909,6 +909,7 @@ void bnx2x_panic_dump(struct bnx2x *bp, bool disable_int)
 	bp->eth_stats.unrecoverable_error++;
 	DP(BNX2X_MSG_STATS, "stats_state - DISABLED\n");
 
+	module_firmware_crashed();
 	BNX2X_ERR("begin crash dump -----------------\n");
 
 	/* Indices */
Signature · 1 line
2.26.2
author maintainer
#18 lore raw

PATCH v2 8/15 ehea: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/ibm/ehea/ehea_main.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/ethernet/ibm/ehea/ehea_main.c +2 -0
index 0273fb7a9d01..6ae35067003f 100644
@@ -3285,6 +3285,8 @@ static void ehea_crash_handler(void)
 {
 	int i;
 
+	module_firmware_crashed();
+
 	if (ehea_fw_handles.arr)
 		for (i = 0; i < ehea_fw_handles.num_entries; i++)
 			ehea_h_free_resource(ehea_fw_handles.arr[i].adh,
Signature · 1 line
2.26.2
author maintainer
#30 lore raw

PATCH v2 12/15 ath10k: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/wireless/ath/ath10k/pci.c  | 2 ++
 drivers/net/wireless/ath/ath10k/sdio.c | 2 ++
 drivers/net/wireless/ath/ath10k/snoc.c | 1 +
 3 files changed, 5 insertions(+)
drivers/net/wireless/ath/ath10k/pci.c +2 -0
index 1d941d53fdc9..6bd0f3b518b9 100644
@@ -1767,6 +1767,7 @@ static void ath10k_pci_fw_dump_work(struct work_struct *work)
 		scnprintf(guid, sizeof(guid), "n/a");
 
 	ath10k_err(ar, "firmware crashed! (guid %s)\n", guid);
+	module_firmware_crashed();
 	ath10k_print_driver_info(ar);
 	ath10k_pci_dump_registers(ar, crash_data);
 	ath10k_ce_dump_registers(ar, crash_data);
@@ -2837,6 +2838,7 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar,
 	if (ret) {
 		if (ath10k_pci_has_fw_crashed(ar)) {
 			ath10k_warn(ar, "firmware crashed during chip reset\n");
+			module_firmware_crashed();
 			ath10k_pci_fw_crashed_clear(ar);
 			ath10k_pci_fw_crashed_dump(ar);
 		}
drivers/net/wireless/ath/ath10k/sdio.c +2 -0
index e2aff2254a40..d34ad289380f 100644
@@ -794,6 +794,7 @@ static int ath10k_sdio_mbox_proc_dbg_intr(struct ath10k *ar)
 
 	/* TODO: Add firmware crash handling */
 	ath10k_warn(ar, "firmware crashed\n");
+	module_firmware_crashed();
 
 	/* read counter to clear the interrupt, the debug error interrupt is
 	 * counter 0.
@@ -915,6 +916,7 @@ static int ath10k_sdio_mbox_proc_cpu_intr(struct ath10k *ar)
 	if (cpu_int_status & MBOX_CPU_STATUS_ENABLE_ASSERT_MASK) {
 		ath10k_err(ar, "firmware crashed!\n");
 		queue_work(ar->workqueue, &ar->restart_work);
+		module_firmware_crashed();
 	}
 	return ret;
 }
drivers/net/wireless/ath/ath10k/snoc.c +1 -0
index 354d49b1cd45..7cfc123c345c 100644
@@ -1451,6 +1451,7 @@ void ath10k_snoc_fw_crashed_dump(struct ath10k *ar)
 		scnprintf(guid, sizeof(guid), "n/a");
 
 	ath10k_err(ar, "firmware crashed! (guid %s)\n", guid);
+	module_firmware_crashed();
 	ath10k_print_driver_info(ar);
 	ath10k_msa_dump_memory(ar, crash_data);
 	mutex_unlock(&ar->dump_mutex);
Signature · 7 lines
2.26.2


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author maintainer
#75 lore raw

PATCH v2 13/15 ath6kl: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/wireless/ath/ath6kl/hif.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/ath/ath6kl/hif.c +1 -0
index d1942537ea10..cfd838607544 100644
@@ -120,6 +120,7 @@ static int ath6kl_hif_proc_dbg_intr(struct ath6kl_device *dev)
 	int ret;
 
 	ath6kl_warn("firmware crashed\n");
+	module_firmware_crashed();
 
 	/*
 	 * read counter to clear the interrupt, the debug error interrupt is
Signature · 7 lines
2.26.2


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author maintainer
#79 lore raw

PATCH v2 15/15 mwl8k: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc Lennert Buytenhek <[email protected]>
Cc Kalle Valo <[email protected]>
Cc "Gustavo A. R. Silva" <[email protected]>
Cc Johannes Berg <[email protected]>
Cc Ganapathi Bhat <[email protected]>
---
 drivers/net/wireless/marvell/mwl8k.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/marvell/mwl8k.c +1 -0
index 97f23f93f6e7..d609ef1bb879 100644
@@ -1551,6 +1551,7 @@ static int mwl8k_tx_wait_empty(struct ieee80211_hw *hw)
 	 * the firmware has crashed
 	 */
 	if (priv->hw_restart_in_progress) {
+		module_firmware_crashed();
 		if (priv->hw_restart_owner == current)
 			return 0;
 		else
Signature · 1 line
2.26.2
author maintainer
#12 lore raw

PATCH v2 5/15 bna: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/brocade/bna/bfa_ioc.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/brocade/bna/bfa_ioc.c +1 -0
index e17bfc87da90..b3f44a912574 100644
@@ -927,6 +927,7 @@ bfa_iocpf_sm_disabled(struct bfa_iocpf *iocpf, enum iocpf_event event)
 static void
 bfa_iocpf_sm_initfail_sync_entry(struct bfa_iocpf *iocpf)
 {
+	module_firmware_crashed();
 	bfa_nw_ioc_debug_save_ftrc(iocpf->ioc);
 	bfa_ioc_hw_sem_get(iocpf->ioc);
 }
Signature · 1 line
2.26.2
author maintainer
#77 lore raw

PATCH v2 14/15 brcm80211: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc Arend van Spriel <[email protected]>
Cc Franky Lin <[email protected]>
Cc Hante Meuleman <[email protected]>
Cc Chi-Hsien Lin <[email protected]>
Cc Wright Feng <[email protected]>
Cc Kalle Valo <[email protected]>
Cc "Rafał Miłecki" <[email protected]>
Cc Pieter-Paul Giesberts <[email protected]>
---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +1 -0
index c88655acc78c..d623f83568b3 100644
@@ -1393,6 +1393,7 @@ void brcmf_fw_crashed(struct device *dev)
 	struct brcmf_pub *drvr = bus_if->drvr;
 
 	bphy_err(drvr, "Firmware has halted or crashed\n");
+	module_firmware_crashed();
 
 	brcmf_dev_coredump(dev);
 
Signature · 1 line
2.26.2
author maintainer
#14 lore raw

PATCH v2 6/15 liquidio: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc Derek Chickles <[email protected]>
Cc Satanand Burla <[email protected]>
Cc Felix Manlunas <[email protected]>
---
 drivers/net/ethernet/cavium/liquidio/lio_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/cavium/liquidio/lio_main.c +1 -0
index 66d31c018c7e..f18085262982 100644
@@ -801,6 +801,7 @@ static int liquidio_watchdog(void *param)
 			continue;
 
 		WRITE_ONCE(oct->cores_crashed, true);
+		module_firmware_crashed();
 		other_oct = get_other_octeon_device(oct);
 		if (other_oct)
 			WRITE_ONCE(other_oct->cores_crashed, true);
Signature · 1 line
2.26.2
author maintainer
#28 lore raw

PATCH v2 11/15 wimax/i2400m: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/wimax/i2400m/rx.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wimax/i2400m/rx.c +1 -0
index c9fb619a9e01..307a62ca183f 100644
@@ -712,6 +712,7 @@ void __i2400m_roq_queue(struct i2400m *i2400m, struct i2400m_roq *roq,
 	dev_err(dev, "SW BUG? failed to insert packet\n");
 	dev_err(dev, "ERX: roq %p [ws %u] skb %p nsn %d sn %u\n",
 		roq, roq->ws, skb, nsn, roq_data->sn);
+	module_firmware_crashed();
 	skb_queue_walk(&roq->queue, skb_itr) {
 		roq_data_itr = (struct i2400m_roq_data *) &skb_itr->cb;
 		nsn_itr = __i2400m_roq_nsn(roq, roq_data_itr->sn);
Signature · 1 line
2.26.2
author maintainer
#22 lore raw

PATCH v2 10/15 soc: qcom: ipa: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ipa/ipa_modem.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ipa/ipa_modem.c +1 -0
index ed10818dd99f..1790b87446ed 100644
@@ -285,6 +285,7 @@ static void ipa_modem_crashed(struct ipa *ipa)
 	struct device *dev = &ipa->pdev->dev;
 	int ret;
 
+	module_firmware_crashed();
 	ipa_endpoint_modem_pause_all(ipa, true);
 
 	ipa_endpoint_modem_hol_block_clear_all(ipa);
Signature · 1 line
2.26.2
author maintainer
#16 lore raw

PATCH v2 7/15 cxgb4: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +1 -0
index a70018f067aa..c67fc86c0e42 100644
@@ -3646,6 +3646,7 @@ void t4_fatal_err(struct adapter *adap)
 	 * could be exposed to the adapter.  RDMA MWs for example...
 	 */
 	t4_shutdown_adapter(adap);
+	module_firmware_crashed();
 	for_each_port(adap, port) {
 		struct net_device *dev = adap->port[port];
 
Signature · 1 line
2.26.2
author maintainer
#20 lore raw

PATCH v2 9/15 qed: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/qlogic/qed/qed_mcp.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/qlogic/qed/qed_mcp.c +1 -0
index 9624616806e7..aea200d465ef 100644
@@ -566,6 +566,7 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		DP_NOTICE(p_hwfn,
 			  "The MFW failed to respond to command 0x%08x [param 0x%08x].\n",
 			  p_mb_params->cmd, p_mb_params->param);
+		module_firmware_crashed();
 		qed_mcp_print_cpu_info(p_hwfn, p_ptt);
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
Signature · 1 line
2.26.2
author maintainer
#9 lore raw

PATCH v2 4/15 bnxt: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +1 -0
index dd0c3f227009..5ba1bd0734e9 100644
@@ -3503,6 +3503,7 @@ static int bnxt_get_dump_data(struct net_device *dev, struct ethtool_dump *dump,
 
 	dump->flag = bp->dump_flag;
 	if (dump->flag == BNXT_DUMP_CRASH) {
+		module_firmware_crashed();
 #ifdef CONFIG_TEE_BNXT_FW
 		return tee_bnxt_copy_coredump(buf, 0, dump->len);
 #endif
Signature · 1 line
2.26.2
author maintainer
#5 lore raw

PATCH v2 2/15 ethernet/839: use new module_firmware_crashed()

In reply to Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Shannon Nelson <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Heiner Kallweit <[email protected]>
---
 drivers/net/ethernet/8390/axnet_cs.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/8390/axnet_cs.c +3 -1
index aeae7966a082..8ad0200db8e9 100644
@@ -1358,9 +1358,11 @@ static void ei_receive(struct net_device *dev)
 		 */
 		if ((netif_msg_rx_err(ei_local)) &&
 		    this_frame != ei_local->current_page &&
-		    (this_frame != 0x0 || rxing_page != 0xFF))
+		    (this_frame != 0x0 || rxing_page != 0xFF)) {
+			module_firmware_crashed();
 			netdev_err(dev, "mismatched read page pointers %2x vs %2x\n",
 				   this_frame, ei_local->current_page);
+		}
 		
 		if (this_frame == rxing_page)	/* Read all the frames? */
 			break;				/* Done for now */
Signature · 1 line
2.26.2
author maintainer
#1 lore raw

PATCH v2 1/15 taint: add module firmware crash taint support

In reply to Luis Chamberlain

Device driver firmware can crash, and sometimes, this can leave your system in a state which makes the device or subsystem completely useless. Detecting this by inspecting /proc/sys/kernel/tainted instead of scraping some magical words from the kernel log, which is driver specific, is much easier. So instead provide a helper which lets drivers annotate this.

Once this happens, scrapers can easily look for modules taint flags for a firmware crash. This will taint both the kernel and respective calling module.

The new helper module_firmware_crashed() uses LOCKDEP_STILL_OK as this fact should in no way shape or form affect lockdep. This taint is device driver specific.

---
 Documentation/admin-guide/tainted-kernels.rst |  6 ++++++
 include/linux/kernel.h                        |  3 ++-
 include/linux/module.h                        | 13 +++++++++++++
 include/trace/events/module.h                 |  3 ++-
 kernel/module.c                               |  5 +++--
 kernel/panic.c                                |  1 +
 tools/debugging/kernel-chktaint               |  7 +++++++
 7 files changed, 34 insertions(+), 4 deletions(-)
Documentation/admin-guide/tainted-kernels.rst +6 -0
index 71e9184a9079..92530f1d60ae 100644
@@ -100,6 +100,7 @@ Bit  Log  Number  Reason that got the kernel tainted
  15  _/K   32768  kernel has been live patched
  16  _/X   65536  auxiliary taint, defined for and used by distros
  17  _/T  131072  kernel was built with the struct randomization plugin
+ 18  _/Q  262144  driver firmware crash annotation
 ===  ===  ======  ========================================================
 
 Note: The character ``_`` is representing a blank in this table to make reading
@@ -162,3 +163,8 @@ More detailed explanation for tainting
      produce extremely unusual kernel structure layouts (even performance
      pathological ones), which is important to know when debugging. Set at
      build time.
+
+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware
+     has crashed and the device's operation has been severely affected. The
+     device may be left in a crippled state, requiring full driver removal /
+     addition, system reboot, or it is unclear how long recovery will take.
include/linux/kernel.h +2 -1
index 04a5885cec1b..19e1541c82c7 100644
@@ -601,7 +601,8 @@ extern enum system_states {
 #define TAINT_LIVEPATCH			15
 #define TAINT_AUX			16
 #define TAINT_RANDSTRUCT		17
-#define TAINT_FLAGS_COUNT		18
+#define TAINT_FIRMWARE_CRASH		18
+#define TAINT_FLAGS_COUNT		19
 
 struct taint_flag {
 	char c_true;	/* character printed when tainted */
include/linux/module.h +13 -0
index 2c2e988bcf10..221200078180 100644
@@ -697,6 +697,14 @@ static inline bool is_livepatch_module(struct module *mod)
 bool is_module_sig_enforced(void);
 void set_module_sig_enforced(void);
 
+void add_taint_module(struct module *mod, unsigned flag,
+		      enum lockdep_ok lockdep_ok);
+
+static inline void module_firmware_crashed(void)
+{
+	add_taint_module(THIS_MODULE, TAINT_FIRMWARE_CRASH, LOCKDEP_STILL_OK);
+}
+
 #else /* !CONFIG_MODULES... */
 
 static inline struct module *__module_address(unsigned long addr)
@@ -844,6 +852,11 @@ void *dereference_module_function_descriptor(struct module *mod, void *ptr)
 	return ptr;
 }
 
+static inline void module_firmware_crashed(void)
+{
+	add_taint(TAINT_FIRMWARE_CRASH, LOCKDEP_STILL_OK);
+}
+
 #endif /* CONFIG_MODULES */
 
 #ifdef CONFIG_SYSFS
include/trace/events/module.h +2 -1
index 097485c73c01..b749ea25affd 100644
@@ -26,7 +26,8 @@ struct module;
 	{ (1UL << TAINT_OOT_MODULE),		"O" },		\
 	{ (1UL << TAINT_FORCED_MODULE),		"F" },		\
 	{ (1UL << TAINT_CRAP),			"C" },		\
-	{ (1UL << TAINT_UNSIGNED_MODULE),	"E" })
+	{ (1UL << TAINT_UNSIGNED_MODULE),	"E" },		\
+	{ (1UL << TAINT_FIRMWARE_CRASH),	"Q" })
 
 TRACE_EVENT(module_load,
 
kernel/module.c +3 -2
index 80faaf2116dd..f98e8c25c6b4 100644
@@ -325,12 +325,13 @@ static inline int strong_try_module_get(struct module *mod)
 		return -ENOENT;
 }
 
-static inline void add_taint_module(struct module *mod, unsigned flag,
-				    enum lockdep_ok lockdep_ok)
+void add_taint_module(struct module *mod, unsigned flag,
+		      enum lockdep_ok lockdep_ok)
 {
 	add_taint(flag, lockdep_ok);
 	set_bit(flag, &mod->taints);
 }
+EXPORT_SYMBOL_GPL(add_taint_module);
 
 /*
  * A thread that wants to hold a reference to a module only while it
kernel/panic.c +1 -0
index ec6d7d788ce7..504fb926947e 100644
@@ -384,6 +384,7 @@ const struct taint_flag taint_flags[TAINT_FLAGS_COUNT] = {
 	[ TAINT_LIVEPATCH ]		= { 'K', ' ', true },
 	[ TAINT_AUX ]			= { 'X', ' ', true },
 	[ TAINT_RANDSTRUCT ]		= { 'T', ' ', true },
+	[ TAINT_FIRMWARE_CRASH ]	= { 'Q', ' ', true },
 };
 
 /**
tools/debugging/kernel-chktaint +7 -0
index 2240cb56e6e5..c397c6aabea7 100755
@@ -194,6 +194,13 @@ else
 	addout "T"
 	echo " * kernel was built with the struct randomization plugin (#17)"
 fi
+T=`expr $T / 2`
+if [ `expr $T % 2` -eq 0 ]; then
+	addout " "
+else
+	addout "Q"
+	echo " * a device driver's firmware has crashed (#18)"
+fi
 
 echo "For a more detailed explanation of the various taint flags see"
 echo " Documentation/admin-guide/tainted-kernels.rst in the the Linux kernel sources"
Signature · 1 line
2.26.2
Reviewed-by on v2 1/15
#2 lore raw

Re: [PATCH v2 01/15] taint: add module firmware crash taint support

Quoted 16 lines · Luis Chamberlain

Device driver firmware can crash, and sometimes, this can leave your system in a state which makes the device or subsystem completely useless. Detecting this by inspecting /proc/sys/kernel/tainted instead of scraping some magical words from the kernel log, which is driver specific, is much easier. So instead provide a helper which lets drivers annotate this.

Once this happens, scrapers can easily look for modules taint flags for a firmware crash. This will taint both the kernel and respective calling module.

The new helper module_firmware_crashed() uses LOCKDEP_STILL_OK as this fact should in no way shape or form affect lockdep. This taint is device driver specific.

Quoted 10 lines
---
 Documentation/admin-guide/tainted-kernels.rst |  6 ++++++
 include/linux/kernel.h                        |  3 ++-
 include/linux/module.h                        | 13 +++++++++++++
 include/trace/events/module.h                 |  3 ++-
 kernel/module.c                               |  5 +++--
 kernel/panic.c                                |  1 +
 tools/debugging/kernel-chktaint               |  7 +++++++
 7 files changed, 34 insertions(+), 4 deletions(-)
Acked-by on v2 2/15
#6 lore raw

Re: [PATCH v2 02/15] ethernet/839: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 37 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc "Michael S. Tsirkin" <[email protected]>
Cc Shannon Nelson <[email protected]>
Cc Jakub Kicinski <[email protected]>
Cc Heiner Kallweit <[email protected]>
---
 drivers/net/ethernet/8390/axnet_cs.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
drivers/net/ethernet/8390/axnet_cs.c +3 -1
index aeae7966a082..8ad0200db8e9 100644
@@ -1358,9 +1358,11 @@ static void ei_receive(struct net_device *dev)
 		 */
 		if ((netif_msg_rx_err(ei_local)) &&
 		    this_frame != ei_local->current_page &&
-		    (this_frame != 0x0 || rxing_page != 0xFF))
+		    (this_frame != 0x0 || rxing_page != 0xFF)) {
+			module_firmware_crashed();
 			netdev_err(dev, "mismatched read page pointers %2x vs %2x\n",
 				   this_frame, ei_local->current_page);
+		}
 
 		if (this_frame == rxing_page)	/* Read all the frames? */
 			break;				/* Done for now */
Signature · 1 line
2.26.2
Acked-by on v2 3/15
#8 lore raw

Re: [PATCH v2 03/15] bnx2x: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 31 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnx2x/bnx2x_main.c +1 -0
index db5107e7937c..c38b8c9c8af0 100644
@@ -909,6 +909,7 @@ void bnx2x_panic_dump(struct bnx2x *bp, bool disable_int)
 	bp->eth_stats.unrecoverable_error++;
 	DP(BNX2X_MSG_STATS, "stats_state - DISABLED\n");
 
+	module_firmware_crashed();
 	BNX2X_ERR("begin crash dump -----------------\n");
 
 	/* Indices */
Signature · 1 line
2.26.2
Acked-by on v2 4/15
#10 lore raw

Re: [PATCH v2 04/15] bnxt: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 29 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +1 -0
index dd0c3f227009..5ba1bd0734e9 100644
@@ -3503,6 +3503,7 @@ static int bnxt_get_dump_data(struct net_device *dev, struct ethtool_dump *dump,
 
 	dump->flag = bp->dump_flag;
 	if (dump->flag == BNXT_DUMP_CRASH) {
+		module_firmware_crashed();
 #ifdef CONFIG_TEE_BNXT_FW
 		return tee_bnxt_copy_coredump(buf, 0, dump->len);
 #endif
Signature · 1 line
2.26.2
Acked-by on v2 5/15
#13 lore raw

Re: [PATCH v2 05/15] bna: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 31 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/brocade/bna/bfa_ioc.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/brocade/bna/bfa_ioc.c +1 -0
index e17bfc87da90..b3f44a912574 100644
@@ -927,6 +927,7 @@ bfa_iocpf_sm_disabled(struct bfa_iocpf *iocpf, enum iocpf_event event)
 static void
 bfa_iocpf_sm_initfail_sync_entry(struct bfa_iocpf *iocpf)
 {
+	module_firmware_crashed();
 	bfa_nw_ioc_debug_save_ftrc(iocpf->ioc);
 	bfa_ioc_hw_sem_get(iocpf->ioc);
 }
Signature · 1 line
2.26.2
Acked-by on v2 6/15
#15 lore raw

Re: [PATCH v2 06/15] liquidio: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 32 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc Derek Chickles <[email protected]>
Cc Satanand Burla <[email protected]>
Cc Felix Manlunas <[email protected]>
---
 drivers/net/ethernet/cavium/liquidio/lio_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/cavium/liquidio/lio_main.c +1 -0
index 66d31c018c7e..f18085262982 100644
@@ -801,6 +801,7 @@ static int liquidio_watchdog(void *param)
 			continue;
 
 		WRITE_ONCE(oct->cores_crashed, true);
+		module_firmware_crashed();
 		other_oct = get_other_octeon_device(oct);
 		if (other_oct)
 			WRITE_ONCE(other_oct->cores_crashed, true);
Signature · 1 line
2.26.2
Acked-by on v2 7/15
#17 lore raw

Re: [PATCH v2 07/15] cxgb4: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 29 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/chelsio/cxgb4/cxgb4_main.c +1 -0
index a70018f067aa..c67fc86c0e42 100644
@@ -3646,6 +3646,7 @@ void t4_fatal_err(struct adapter *adap)
 	 * could be exposed to the adapter.  RDMA MWs for example...
 	 */
 	t4_shutdown_adapter(adap);
+	module_firmware_crashed();
 	for_each_port(adap, port) {
 		struct net_device *dev = adap->port[port];
 
Signature · 1 line
2.26.2
Acked-by on v2 8/15
#19 lore raw

Re: [PATCH v2 08/15] ehea: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 30 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/ibm/ehea/ehea_main.c | 2 ++
 1 file changed, 2 insertions(+)
drivers/net/ethernet/ibm/ehea/ehea_main.c +2 -0
index 0273fb7a9d01..6ae35067003f 100644
@@ -3285,6 +3285,8 @@ static void ehea_crash_handler(void)
 {
 	int i;
 
+	module_firmware_crashed();
+
 	if (ehea_fw_handles.arr)
 		for (i = 0; i < ehea_fw_handles.num_entries; i++)
 			ehea_h_free_resource(ehea_fw_handles.arr[i].adh,
Signature · 1 line
2.26.2
Acked-by on v2 9/15
#21 lore raw

Re: [PATCH v2 09/15] qed: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 31 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/qlogic/qed/qed_mcp.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/qlogic/qed/qed_mcp.c +1 -0
index 9624616806e7..aea200d465ef 100644
@@ -566,6 +566,7 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
 		DP_NOTICE(p_hwfn,
 			  "The MFW failed to respond to command 0x%08x [param 0x%08x].\n",
 			  p_mb_params->cmd, p_mb_params->param);
+		module_firmware_crashed();
 		qed_mcp_print_cpu_info(p_hwfn, p_ptt);
 
 		spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
Signature · 1 line
2.26.2
Acked-by on v2 10/15
#23 lore raw

Re: [PATCH v2 10/15] soc: qcom: ipa: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 29 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ipa/ipa_modem.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ipa/ipa_modem.c +1 -0
index ed10818dd99f..1790b87446ed 100644
@@ -285,6 +285,7 @@ static void ipa_modem_crashed(struct ipa *ipa)
 	struct device *dev = &ipa->pdev->dev;
 	int ret;
 
+	module_firmware_crashed();
 	ipa_endpoint_modem_pause_all(ipa, true);
 
 	ipa_endpoint_modem_hol_block_clear_all(ipa);
Signature · 1 line
2.26.2
Acked-by on v2 11/15
#29 lore raw

Re: [PATCH v2 11/15] wimax/i2400m: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 30 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/wimax/i2400m/rx.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wimax/i2400m/rx.c +1 -0
index c9fb619a9e01..307a62ca183f 100644
@@ -712,6 +712,7 @@ void __i2400m_roq_queue(struct i2400m *i2400m, struct i2400m_roq *roq,
 	dev_err(dev, "SW BUG? failed to insert packet\n");
 	dev_err(dev, "ERX: roq %p [ws %u] skb %p nsn %d sn %u\n",
 		roq, roq->ws, skb, nsn, roq_data->sn);
+	module_firmware_crashed();
 	skb_queue_walk(&roq->queue, skb_itr) {
 		roq_data_itr = (struct i2400m_roq_data *) &skb_itr->cb;
 		nsn_itr = __i2400m_roq_nsn(roq, roq_data_itr->sn);
Signature · 1 line
2.26.2
Acked-by on v2 12/15
#31 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 73 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/wireless/ath/ath10k/pci.c  | 2 ++
 drivers/net/wireless/ath/ath10k/sdio.c | 2 ++
 drivers/net/wireless/ath/ath10k/snoc.c | 1 +
 3 files changed, 5 insertions(+)
drivers/net/wireless/ath/ath10k/pci.c +2 -0
index 1d941d53fdc9..6bd0f3b518b9 100644
@@ -1767,6 +1767,7 @@ static void ath10k_pci_fw_dump_work(struct work_struct *work)
 		scnprintf(guid, sizeof(guid), "n/a");
 
 	ath10k_err(ar, "firmware crashed! (guid %s)\n", guid);
+	module_firmware_crashed();
 	ath10k_print_driver_info(ar);
 	ath10k_pci_dump_registers(ar, crash_data);
 	ath10k_ce_dump_registers(ar, crash_data);
@@ -2837,6 +2838,7 @@ static int ath10k_pci_hif_power_up(struct ath10k *ar,
 	if (ret) {
 		if (ath10k_pci_has_fw_crashed(ar)) {
 			ath10k_warn(ar, "firmware crashed during chip reset\n");
+			module_firmware_crashed();
 			ath10k_pci_fw_crashed_clear(ar);
 			ath10k_pci_fw_crashed_dump(ar);
 		}
drivers/net/wireless/ath/ath10k/sdio.c +2 -0
index e2aff2254a40..d34ad289380f 100644
@@ -794,6 +794,7 @@ static int ath10k_sdio_mbox_proc_dbg_intr(struct ath10k *ar)
 
 	/* TODO: Add firmware crash handling */
 	ath10k_warn(ar, "firmware crashed\n");
+	module_firmware_crashed();
 
 	/* read counter to clear the interrupt, the debug error interrupt is
 	 * counter 0.
@@ -915,6 +916,7 @@ static int ath10k_sdio_mbox_proc_cpu_intr(struct ath10k *ar)
 	if (cpu_int_status & MBOX_CPU_STATUS_ENABLE_ASSERT_MASK) {
 		ath10k_err(ar, "firmware crashed!\n");
 		queue_work(ar->workqueue, &ar->restart_work);
+		module_firmware_crashed();
 	}
 	return ret;
 }
drivers/net/wireless/ath/ath10k/snoc.c +1 -0
index 354d49b1cd45..7cfc123c345c 100644
@@ -1451,6 +1451,7 @@ void ath10k_snoc_fw_crashed_dump(struct ath10k *ar)
 		scnprintf(guid, sizeof(guid), "n/a");
 
 	ath10k_err(ar, "firmware crashed! (guid %s)\n", guid);
+	module_firmware_crashed();
 	ath10k_print_driver_info(ar);
 	ath10k_msa_dump_memory(ar, crash_data);
 	mutex_unlock(&ar->dump_mutex);
Signature · 1 line
2.26.2
Acked-by on v2 13/15
#76 lore raw

Re: [PATCH v2 13/15] ath6kl: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 31 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/wireless/ath/ath6kl/hif.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/ath/ath6kl/hif.c +1 -0
index d1942537ea10..cfd838607544 100644
@@ -120,6 +120,7 @@ static int ath6kl_hif_proc_dbg_intr(struct ath6kl_device *dev)
 	int ret;
 
 	ath6kl_warn("firmware crashed\n");
+	module_firmware_crashed();
 
 	/*
 	 * read counter to clear the interrupt, the debug error interrupt is
Signature · 1 line
2.26.2
Acked-by on v2 14/15
#78 lore raw

Re: [PATCH v2 14/15] brcm80211: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 39 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc Arend van Spriel <[email protected]>
Cc Franky Lin <[email protected]>
Cc Hante Meuleman <[email protected]>
Cc Chi-Hsien Lin <[email protected]>
Cc Wright Feng <[email protected]>
Cc Kalle Valo <[email protected]>
Cc "Rafał Miłecki" <[email protected]>
Cc Pieter-Paul Giesberts <[email protected]>
---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +1 -0
index c88655acc78c..d623f83568b3 100644
@@ -1393,6 +1393,7 @@ void brcmf_fw_crashed(struct device *dev)
 	struct brcmf_pub *drvr = bus_if->drvr;
 
 	bphy_err(drvr, "Firmware has halted or crashed\n");
+	module_firmware_crashed();
 
 	brcmf_dev_coredump(dev);
 
Signature · 1 line
2.26.2
Acked-by on v2 15/15
#80 lore raw

Re: [PATCH v2 15/15] mwl8k: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 34 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

Cc Lennert Buytenhek <[email protected]>
Cc Kalle Valo <[email protected]>
Cc "Gustavo A. R. Silva" <[email protected]>
Cc Johannes Berg <[email protected]>
Cc Ganapathi Bhat <[email protected]>
---
 drivers/net/wireless/marvell/mwl8k.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/marvell/mwl8k.c +1 -0
index 97f23f93f6e7..d609ef1bb879 100644
@@ -1551,6 +1551,7 @@ static int mwl8k_tx_wait_empty(struct ieee80211_hw *hw)
 	 * the firmware has crashed
 	 */
 	if (priv->hw_restart_in_progress) {
+		module_firmware_crashed();
 		if (priv->hw_restart_owner == current)
 			return 0;
 		else
Signature · 1 line
2.26.2
Nacked-by on v2 4/15 NAK
#11 lore raw

Re: [PATCH v2 04/15] bnxt: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 20 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +0 -0
index dd0c3f227009..5ba1bd0734e9 100644

drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c static int bnxt_get_dump_data(struct net_device *dev, struct ethtool_dump *dump, · line 3506

Quoted 4 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +1 -0
@@ -3503,6 +3503,7 @@ static int bnxt_get_dump_data(struct net_device *dev, struct ethtool_dump *dump,
 
        dump->flag = bp->dump_flag;
        if (dump->flag == BNXT_DUMP_CRASH) {
+               module_firmware_crashed();

This is not the right place to annotate the taint flag.

Here the driver is just copying the dump after error recovery which is collected by firmware to DDR, when firmware detects fatal conditions. Driver and firmware will be healthy when the user calls this command.

Also, users can call this command a thousand times when there is no crash.

I will propose a patch to use this wrapper in the error recovery path, where the driver
may not be able to recover.

Quoted 6 lines
drivers/net/ethernet/broadcom/bnxt/bnxt_ethtool.c +0 -1
 #ifdef CONFIG_TEE_BNXT_FW
                return tee_bnxt_copy_coredump(buf, 0, dump->len);
 #endif
--

2.26.2

#32 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Luis Chamberlain

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

Instead of the kernel taint, IMHO you should provide an annotation in sysfs (or somewhere else) for the *struct device* that had its firmware crash. Or maybe, if it's too complex to walk the entire hierarchy checking for that, have a uevent, or add the ability for the kernel to print out elsewhere in debugfs the list of devices that crashed at some point... All of that is fine, but a kernel taint?

johannes

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#33 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 5 lines · Johannes Berg

Instead of the kernel taint, IMHO you should provide an annotation in sysfs (or somewhere else) for the *struct device* that had its firmware crash. Or maybe, if it's too complex to walk the entire hierarchy checking for that, have a uevent, or add the ability for the kernel to print out elsewhere in debugfs the list of devices that crashed at some

I mean sysfs, oops.

In addition, look what we have in iwl_trans_pcie_removal_wk(). If we detect that the device is really wedged enough that the only way we can still try to recover is by completely unbinding the driver from it, then we give userspace a uevent for that. I don't remember exactly how and where that gets used (ChromeOS) though, but it'd be nice to have that sort of thing as part of the infrastructure, in a sort of two-level notification?

Level 1: firmware crashed, but we're recovering, at least mostly, and it's more informational

Level 2: device is wedged, going to try to recover by some more forceful means (perhaps some devices can be power-cycled? etc.) but (more) state would be lost in these cases?

Still don't think a kernel taint is appropriate for either of these.

johannes

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#43 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Johannes Berg
Quoted 9 lines · Johannes Berg

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

Keyword you used here is "most device". And in the worst case, *who* knows what other odd things may happen afterwards.

Quoted 4 lines

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

Sometimes the device can go into a state which requires driver removal and addition to get things back up.

Quoted 3 lines

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

That is one reason, another is that a taint helps support cases *fast* easily detect if the issue was a firmware crash, instead of scraping logs for driver specific ways to say the firmware has crashed.

Quoted 3 lines

Instead of the kernel taint, IMHO you should provide an annotation in sysfs (or somewhere else) for the *struct device* that had its firmware crash.

It would seem the way some folks are thinking about getting more details would be through devlink.

Quoted 4 lines

Or maybe, if it's too complex to walk the entire hierarchy checking for that, have a uevent, or add the ability for the kernel to print out elsewhere in debugfs the list of devices that crashed at some point... All of that is fine, but a kernel taint?

debugfs is optional, a taint is simple, and device agnostic. From a support perspective it is very easy to see if a possible issue may be device firmware specific.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#34 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Johannes Berg
Quoted 8 lines · Johannes Berg
Quoted 5 lines · Johannes Berg

Instead of the kernel taint, IMHO you should provide an annotation in sysfs (or somewhere else) for the *struct device* that had its firmware crash. Or maybe, if it's too complex to walk the entire hierarchy checking for that, have a uevent, or add the ability for the kernel to print out elsewhere in debugfs the list of devices that crashed at some

Quoted 7 lines

I mean sysfs, oops.

In addition, look what we have in iwl_trans_pcie_removal_wk(). If we detect that the device is really wedged enough that the only way we can still try to recover is by completely unbinding the driver from it, then we give userspace a uevent for that.

Nice! Indeed a uevent is in order for these sorts of things, and I'd argue that it begs the question if we should even uevent for any taint as well. Today these are silent. If the kernel crashes, today we only give userspace a log.

Quoted 11 lines

I don't remember exactly how and
where that gets used (ChromeOS) though, but it'd be nice to have that sort of thing as part of the infrastructure, in a sort of two-level notification?

Level 1: firmware crashed, but we're recovering, at least mostly, and it's more informational

Level 2: device is wedged, going to try to recover by some more forceful means (perhaps some devices can be power-cycled? etc.) but (more) state would be lost in these cases?

I agree that *all* this would be ideal. I don't see this as mutually exclusive with a taint on the kernel and module for the device.

Quoted 1 line

Still don't think a kernel taint is appropriate for either of these.

From a support perspective, I do think it is vital quick information.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#44 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 14 lines · Luis Chamberlain
Quoted 9 lines · Johannes Berg

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

Keyword you used here is "most device". And in the worst case, *who* knows what other odd things may happen afterwards.

Quoted 7 lines
Quoted 4 lines

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

Sometimes the device can go into a state which requires driver removal and addition to get things back up.

It would be lovely to be able to detect this case in the driver/system somehow! I haven't seen any such cases recently, but in case there is some common case you see, maybe we can think of a way to detect it?

Quoted 8 lines
Quoted 4 lines

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

That is one reason, another is that a taint helps support cases *fast* easily detect if the issue was a firmware crash, instead of scraping logs for driver specific ways to say the firmware has crashed.

You can listen for udev events (I think that is the right term), and find crashes that way. You get the actual crash info as well.

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author maintainer
#45 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 22 lines · Ben Greear
Quoted 19 lines · Luis Chamberlain
Quoted 9 lines · Johannes Berg

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

Keyword you used here is "most device". And in the worst case, *who* knows what other odd things may happen afterwards.

Quoted 4 lines

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

Quoted 5 lines
Quoted 2 lines

Sometimes the device can go into a state which requires driver removal and addition to get things back up.

It would be lovely to be able to detect this case in the driver/system somehow! I haven't seen any such cases recently,

I assure you that I have run into it. Once it does again I'll report the crash, but the problem with some of this is that unless you scrape the log you won't know. Eventually, a uevent would indeed tell inform me.

Quoted 2 lines

but in case there is
some common case you see, maybe we can think of a way to detect it?

ath10k is just one case, this patch series addresses a simple way to annotate this tree-wide.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

That is one reason, another is that a taint helps support cases *fast* easily detect if the issue was a firmware crash, instead of scraping logs for driver specific ways to say the firmware has crashed.

You can listen for udev events (I think that is the right term), and find crashes that way. You get the actual crash info as well.

My follow up to this was to add uevent to add_taint() as well, this way these could generically be processed by userspace.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#46 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 48 lines · Luis Chamberlain
Quoted 22 lines · Ben Greear
Quoted 19 lines · Luis Chamberlain
Quoted 9 lines · Johannes Berg

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

Keyword you used here is "most device". And in the worst case, *who* knows what other odd things may happen afterwards.

Quoted 4 lines

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

Quoted 5 lines
Quoted 2 lines

Sometimes the device can go into a state which requires driver removal and addition to get things back up.

It would be lovely to be able to detect this case in the driver/system somehow! I haven't seen any such cases recently,

I assure you that I have run into it. Once it does again I'll report the crash, but the problem with some of this is that unless you scrape the log you won't know. Eventually, a uevent would indeed tell inform me.

Quoted 2 lines

but in case there is
some common case you see, maybe we can think of a way to detect it?

ath10k is just one case, this patch series addresses a simple way to annotate this tree-wide.

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

That is one reason, another is that a taint helps support cases *fast* easily detect if the issue was a firmware crash, instead of scraping logs for driver specific ways to say the firmware has crashed.

Quoted 5 lines
Quoted 2 lines

You can listen for udev events (I think that is the right term), and find crashes that way. You get the actual crash info as well.

My follow up to this was to add uevent to add_taint() as well, this way these could generically be processed by userspace.

I'm not opposed to the taint, though I have not thought much on it.

But, if you can already get the crash info from uevent, and it automatically comes without polling or scraping logs, then what benefit beyond that does the taint give you?

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
author maintainer
#47 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 54 lines · Ben Greear
Quoted 51 lines · Luis Chamberlain
Quoted 22 lines · Ben Greear
Quoted 19 lines · Luis Chamberlain
Quoted 9 lines · Johannes Berg

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

Keyword you used here is "most device". And in the worst case, *who* knows what other odd things may happen afterwards.

Quoted 4 lines

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

Quoted 5 lines
Quoted 2 lines

Sometimes the device can go into a state which requires driver removal and addition to get things back up.

It would be lovely to be able to detect this case in the driver/system somehow! I haven't seen any such cases recently,

I assure you that I have run into it. Once it does again I'll report the crash, but the problem with some of this is that unless you scrape the log you won't know. Eventually, a uevent would indeed tell inform me.

Quoted 2 lines

but in case there is
some common case you see, maybe we can think of a way to detect it?

ath10k is just one case, this patch series addresses a simple way to annotate this tree-wide.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

That is one reason, another is that a taint helps support cases *fast* easily detect if the issue was a firmware crash, instead of scraping logs for driver specific ways to say the firmware has crashed.

You can listen for udev events (I think that is the right term), and find crashes that way. You get the actual crash info as well.

Quoted 8 lines
Quoted 2 lines

My follow up to this was to add uevent to add_taint() as well, this way these could generically be processed by userspace.

I'm not opposed to the taint, though I have not thought much on it.

But, if you can already get the crash info from uevent, and it automatically comes without polling or scraping logs, then what benefit beyond that does the taint give you?

From a support perspective it is a *crystal* clear sign that the device and / or device driver may be in a very bad state, in a generic way.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#48 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 59 lines · Luis Chamberlain
Quoted 57 lines · Ben Greear
Quoted 53 lines · Luis Chamberlain
Quoted 22 lines · Ben Greear
Quoted 19 lines · Luis Chamberlain
Quoted 9 lines · Johannes Berg

On Fri, 2020-05-15 at 21:28 +0000, Luis Chamberlain wrote:> module_firmware_crashed

You didn't CC me or the wireless list on the rest of the patches, so I'm replying to a random one, but ...

What is the point here?

This should in no way affect the integrity of the system/kernel, for most devices anyway.

Keyword you used here is "most device". And in the worst case, *who* knows what other odd things may happen afterwards.

Quoted 4 lines

So what if ath10k's firmware crashes? If there's a driver bug it will not handle it right (and probably crash, WARN_ON, or something else), but if the driver is working right then that will not affect the kernel at all.

Quoted 5 lines
Quoted 2 lines

Sometimes the device can go into a state which requires driver removal and addition to get things back up.

It would be lovely to be able to detect this case in the driver/system somehow! I haven't seen any such cases recently,

I assure you that I have run into it. Once it does again I'll report the crash, but the problem with some of this is that unless you scrape the log you won't know. Eventually, a uevent would indeed tell inform me.

Quoted 2 lines

but in case there is
some common case you see, maybe we can think of a way to detect it?

ath10k is just one case, this patch series addresses a simple way to annotate this tree-wide.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

So maybe I can understand that maybe you want an easy way to discover - per device - that the firmware crashed, but that still doesn't warrant a complete kernel taint.

That is one reason, another is that a taint helps support cases *fast* easily detect if the issue was a firmware crash, instead of scraping logs for driver specific ways to say the firmware has crashed.

You can listen for udev events (I think that is the right term), and find crashes that way. You get the actual crash info as well.

My follow up to this was to add uevent to add_taint() as well, this way these could generically be processed by userspace.

Quoted 9 lines
Quoted 5 lines

I'm not opposed to the taint, though I have not thought much on it.

But, if you can already get the crash info from uevent, and it automatically comes without polling or scraping logs, then what benefit beyond that does the taint give you?

From a support perspective it is a *crystal* clear sign that the device and / or device driver may be in a very bad state, in a generic way.

Unfortunately a "taint" is interpreted by many users as: "your kernel is really F#*D up, you better do something about it right now." Assuming they're paying attention at all in the first place of course.

The fact is, WiFi chip firmware crashes, and in most cases the driver is able to recover seamlessly. At least that is the case with most QCA chipsets I work with. And the users or our ability to do anything about it is minimal to none as we don't have access to firmware source. It's too bad and I wish it weren't the case, but we have embraced reality and most drivers have a recovery mechanism built in for this case. In short, it's a non-event. I fear that elevating this to a kernel taint will significantly increase "support" requests that really are nothing but noise; similar to how the firmware load failure messages (fail to load fw-2.bin, fail to load fw-1.bin, yay loaded fw-0.bin) cause a lot of noise.

Not specifically opposed, but I wonder what it really accomplishes in a world where the firmware crashing is pretty much a normal occurrence.

If it goes in, I think that the drivers shouldn't trigger the taint if they're able to recover normally. Only trigger on failure to come back up. In other words, the ideal place in the ath10k driver isn't where you have proposed as at that point operation is normal and we're doing a routine recovery.

- Steve

Quoted 1 line

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#49 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 8 lines · Steve deRosier
Quoted 3 lines · Luis Chamberlain

From a support perspective it is a *crystal* clear sign that the device and / or device driver may be in a very bad state, in a generic way.

Unfortunately a "taint" is interpreted by many users as: "your kernel is really F#*D up, you better do something about it right now." Assuming they're paying attention at all in the first place of course.

Taint historically has been used and still is today to help rule out whether or not you get support, or how you get support.

For instance, a staging driver is not supported by some upstream developers, but it will be by those who help staging and Greg. TAINT_CRAP cannot be even more clear.

So, no, it is not just about "hey your kernel is messed up", there are clear support boundaries being drawn.

Quoted 3 lines

The fact is, WiFi chip firmware crashes, and in most cases the driver is able to recover seamlessly. At least that is the case with most QCA chipsets I work with.

That has not been my exerience with the same driver, and so how do we know? And this patch set is not about ath10k alone, I want you to think about *all* device drivers with firmware. In my journey to scrape the kernel for these cases I was very surprised by the amount of code which clearly annotates these situations.

Quoted 3 lines

And the users or our ability to do anything
about it is minimal to none as we don't have access to firmware source.

This is not true, we have open firmware in WiFi. Some vendors choose to not open source their firmware, that is their decision.

These days though, I think we all admit, that firmware crashes can use a better generic infrastructure for ensuring that clearly affecting-user experience issues. This patch is about that *when and if these happen*, we annotate it in the kernel for support pursposes.

Quoted 3 lines

It's too bad and I wish it weren't the case, but we have embraced reality and most drivers have a recovery mechanism built in for this case.

The mentality about firmware crashes being the end of the world is certainly what will lead developers to often hide these. Where this is openly clear, and not obfucscated I'd argue that firmware issues get fixed likely more common.

So what you describe is not bad, its just accepting evolution.

Quoted 3 lines

In short, it's a non-event. I fear that elevating this
to a kernel taint will significantly increase "support" requests that really are nothing but noise;

That will depend on where you put this on the driver, and that is why it is important to place it in the right place, if any.

Quoted 3 lines

similar to how the firmware load failure
messages (fail to load fw-2.bin, fail to load fw-1.bin, yay loaded fw-0.bin) cause a lot of noise.

That can be fixed, the developers behind this series gave up on it. It has to do with a range version of supported firmwares, and all being optional, but at least one is required.

Quoted 3 lines

Not specifically opposed, but I wonder what it really accomplishes in a world where the firmware crashing is pretty much a normal occurrence.

Recovery without affecting user experience would be great, the taint is *not* for those cases. The taint definition has:

+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware

+     has crashed and the device's operation has been severely affected. The    
+     device may be left in a crippled state, requiring full driver removal /   
+     addition, system reboot, or it is unclear how long recovery will take.

Let me know if this is not clear.

Quoted 5 lines

If it goes in, I think that the drivers shouldn't trigger the taint if they're able to recover normally. Only trigger on failure to come back up. In other words, the ideal place in the ath10k driver isn't where you have proposed as at that point operation is normal and we're doing a routine recovery.

Sure, happy to remove it if indeed it is the case that the firwmare crash is not happening to cripple the device, but I can vouch for the fact that the exact place where I placed it left my device driver in a state where I had to remove / add again.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#50 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 7 lines · Luis Chamberlain
Quoted 3 lines

Unfortunately a "taint" is interpreted by many users as: "your kernel is really F#*D up, you better do something about it right now." Assuming they're paying attention at all in the first place of course.

Taint historically has been used and still is today to help rule out whether or not you get support, or how you get support.

Quoted 6 lines

For instance, a staging driver is not supported by some upstream developers, but it will be by those who help staging and Greg. TAINT_CRAP cannot be even more clear.

So, no, it is not just about "hey your kernel is messed up", there are clear support boundaries being drawn.

Err, no. Those two are most definitely related. Have you looked at (most or some or whatever) staging drivers recently? Those contain all kinds of garbage that might do whatever with your kernel.

Of course that's not a completely clear boundary, maybe you can find a driver in staging that's perfect code just not written to kernel style? But I find that hard to believe, in most cases.

So no, it's really not about "[a] staging driver is not supported" vs. "your kernel is messed up". The very fact that you loaded one of those things might very well have messed up your kernel entirely.

Quoted 4 lines

These days though, I think we all admit, that firmware crashes can use a better generic infrastructure for ensuring that clearly affecting-user experience issues. This patch is about that *when and if these happen*, we annotate it in the kernel for support pursposes.

That's all fine, I just don't think it's appropriate to pretend that your kernel is now 'tainted' (think about the meaning of that word) when the firmware of some random device crashed. Heck, that could have been a USB device that was since unplugged. Unless the driver is complete garbage (hello staging again?) that really should have no lasting effect on the system itself.

Quoted 9 lines

Recovery without affecting user experience would be great, the taint is *not* for those cases. The taint definition has:

+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware

(quoted diff) +3 -0
+     has crashed and the device's operation has been severely affected. The    
+     device may be left in a crippled state, requiring full driver removal /   
+     addition, system reboot, or it is unclear how long recovery will take.

Let me know if this is not clear.

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on, and secondly it actually hides useful information.

Regardless of the support issue, I think this hiding of information is also problematic.

I really think we'd all be better off if you just made a sysfs file (I mistyped debugfs in some other email, sorry, apparently you didn't see the correction in time) that listed which device(s) crashed and how many times. That would actually be useful. Because honestly, if a random device crashed for some random reason, that's pretty much a non-event. If it keeps happening, then we might even want to know about it.

You can obviously save the contents of this file into your bug reports automatically and act accordingly, but I think you'll find that this is far more useful than saying "TAINT_FIRMWARE_CRASHED" so I'll ignore this report. Yeah, that might be reasonable thing if the bug report is about slow wifi *and* you see that ath10k firmware crashed every 10 seconds, but if it just crashed once a few days earlier it's of no importance to the system anymore ... And certainly a reasonable driver (which I believe ath10k to be) would _not_ randomly start corrupting memory because its firmware crashed. Which really is what tainting the kernel is about.

So no, even with all that, I still really believe you're solving the wrong problem. Having information about firmware crashes, preferably with some kind of frequency information attached, and *clearly* with information about which device attached would be _great_. Munging it all into one bit is actively harmful, IMO.

johannes

author maintainer
#51 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 13 lines · Johannes Berg
Quoted 11 lines · Luis Chamberlain
Quoted 3 lines

Unfortunately a "taint" is interpreted by many users as: "your kernel is really F#*D up, you better do something about it right now." Assuming they're paying attention at all in the first place of course.

Taint historically has been used and still is today to help rule out whether or not you get support, or how you get support.

For instance, a staging driver is not supported by some upstream developers, but it will be by those who help staging and Greg. TAINT_CRAP cannot be even more clear.

Quoted 6 lines
Quoted 2 lines

So, no, it is not just about "hey your kernel is messed up", there are clear support boundaries being drawn.

Err, no. Those two are most definitely related. Have you looked at (most or some or whatever) staging drivers recently? Those contain all kinds of garbage that might do whatever with your kernel.

No, I stay away :)

Quoted 8 lines

Of course that's not a completely clear boundary, maybe you can find a driver in staging that's perfect code just not written to kernel style? But I find that hard to believe, in most cases.

So no, it's really not about "[a] staging driver is not supported" vs. "your kernel is messed up". The very fact that you loaded one of those things might very well have messed up your kernel entirely.

Quoted 8 lines
Quoted 4 lines

These days though, I think we all admit, that firmware crashes can use a better generic infrastructure for ensuring that clearly affecting-user experience issues. This patch is about that *when and if these happen*, we annotate it in the kernel for support pursposes.

That's all fine, I just don't think it's appropriate to pretend that your kernel is now 'tainted' (think about the meaning of that word) when the firmware of some random device crashed.

If the firmware crash *does* require driver remove / addition again, or a reboot, would you think that this is a situation that merits a taint?

Quoted 8 lines
Quoted 8 lines

Recovery without affecting user experience would be great, the taint is *not* for those cases. The taint definition has:

+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware

(quoted diff) +3 -0
+     has crashed and the device's operation has been severely affected. The    
+     device may be left in a crippled state, requiring full driver removal /   
+     addition, system reboot, or it is unclear how long recovery will take.
Quoted 4 lines
Quoted 1 line

Let me know if this is not clear.

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on,

We can remove it, for this driver I can vouch for its location as it did reach a state where I required a reboot. And its not the first time this has happened. This got me thinking about the bigger picture of the lack of proper way to address these cases in the kernel, and how the user is left dumbfounded.

Quoted 1 line

and secondly it actually hides useful information.

What is it hiding?

Quoted 7 lines

Regardless of the support issue, I think this hiding of information is also problematic.

I really think we'd all be better off if you just made a sysfs file (I mistyped debugfs in some other email, sorry, apparently you didn't see the correction in time) that listed which device(s) crashed and how many times.

Ah yes, count. The taint does not address count.

Quoted 3 lines

That would actually be useful. Because honestly, if a random device crashed for some random reason, that's pretty much a non-event. If it keeps happening, then we might even want to know about it.

True.

Quoted 4 lines

You can obviously save the contents of this file into your bug reports automatically and act accordingly, but I think you'll find that this is far more useful than saying "TAINT_FIRMWARE_CRASHED" so I'll ignore this report.

Absolutely.

Quoted 7 lines

Yeah, that might be reasonable thing if the bug report is about slow wifi *and* you see that ath10k firmware crashed every 10 seconds, but if it just crashed once a few days earlier it's of no importance to the system anymore ... And certainly a reasonable driver (which I believe ath10k to be) would _not_ randomly start corrupting memory because its firmware crashed. Which really is what tainting the kernel is about.

I still see it as a support thing too. But discussing this further is pointless as I agree that taint does not cover count and that it is important.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#52 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 5 lines · Luis Chamberlain
Quoted 3 lines

Err, no. Those two are most definitely related. Have you looked at (most or some or whatever) staging drivers recently? Those contain all kinds of garbage that might do whatever with your kernel.

No, I stay away :)

:)

Quoted 6 lines
Quoted 3 lines

That's all fine, I just don't think it's appropriate to pretend that your kernel is now 'tainted' (think about the meaning of that word) when the firmware of some random device crashed.

If the firmware crash *does* require driver remove / addition again, or a reboot, would you think that this is a situation that merits a taint?

Not really. In my experience, that's more likely a hardware issue (card not properly seated, for example) that a bus reset happens to "fix".

Quoted 8 lines
Quoted 2 lines

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on,

We can remove it, for this driver I can vouch for its location as it did reach a state where I required a reboot. And its not the first time this has happened. This got me thinking about the bigger picture of the lack of proper way to address these cases in the kernel, and how the user is left dumbfounded.

Fair, so the driver is still broken wrt. recovery here. I still don't think that's a situation where e.g. the system should say "hey you have a taint here, if your graphics go bad now you should not report that bug" (which is effectively what the single taint bit does).

Quoted 3 lines
Quoted 1 line

and secondly it actually hides useful information.

What is it hiding?

Most importantly, which device crashed. Secondarily I'd say how many times (*).

The information "firmware crashed" is really only useful in relation to the device. If your graphics firmware crashed, yeah, well, you probably won't even see this. If your USB wifi firmware crashed? Not really interesting, you'll anyway just unplug. In fact it's very hard for a USB driver (short of arbitrary memory corruption) to significantly mess up the system.

johannes

(*) though if it crashed only once, was that because it was wedged enough to be unusable afterwards, or because everything was fine?

maintainer
#54 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Johannes Berg
Quoted 7 lines · On Mon, 18 May 2020 21:25:09 +0200 Johannes Berg wrote:

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on, and secondly it actually hides useful information.

Regardless of the support issue, I think this hiding of information is also problematic.

Quoted 6 lines

I really think we'd all be better off if you just made a sysfs file (I mistyped debugfs in some other email, sorry, apparently you didn't see the correction in time) that listed which device(s) crashed and how many times. That would actually be useful. Because honestly, if a random device crashed for some random reason, that's pretty much a non-event. If it keeps happening, then we might even want to know about it.

Johannes - have you seen devlink health? I think we should just use that interface, since it supports all the things you're requesting, rather than duplicate it in sysfs.

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#55 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 15 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 18 May 2020 21:25:09 +0200 Johannes Berg wrote:

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on, and secondly it actually hides useful information.

Regardless of the support issue, I think this hiding of information is also problematic.

Quoted 6 lines

I really think we'd all be better off if you just made a sysfs file (I mistyped debugfs in some other email, sorry, apparently you didn't see the correction in time) that listed which device(s) crashed and how many times. That would actually be useful. Because honestly, if a random device crashed for some random reason, that's pretty much a non-event. If it keeps happening, then we might even want to know about it.

Quoted 3 lines

Johannes - have you seen devlink health? I think we should just use that interface, since it supports all the things you're requesting, rather than duplicate it in sysfs.

I haven't, and I'm glad to hear that's there, sounds good!

I suspect that Luis wants something more generic though, that isn't just applicable to netdevices, unless devlink grew some kind of non-netdev stuff while I wasn't looking? :)

johannes

maintainer
#56 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 20 lines · On Mon, 18 May 2020 22:29:53 +0200 Johannes Berg wrote:
Quoted 15 lines · Jakub Kicinski
Quoted 7 lines · On Mon, 18 May 2020 21:25:09 +0200 Johannes Berg wrote:

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on, and secondly it actually hides useful information.

Regardless of the support issue, I think this hiding of information is also problematic.

Quoted 6 lines

I really think we'd all be better off if you just made a sysfs file (I mistyped debugfs in some other email, sorry, apparently you didn't see the correction in time) that listed which device(s) crashed and how many times. That would actually be useful. Because honestly, if a random device crashed for some random reason, that's pretty much a non-event. If it keeps happening, then we might even want to know about it.

Quoted 3 lines

Johannes - have you seen devlink health? I think we should just use that interface, since it supports all the things you're requesting, rather than duplicate it in sysfs.

Quoted 5 lines

I haven't, and I'm glad to hear that's there, sounds good!

I suspect that Luis wants something more generic though, that isn't just applicable to netdevices, unless devlink grew some kind of non-netdev stuff while I wasn't looking? :)

It's intended to be a generic netlink channel for configuring devices.

All the firmware-related interfaces have no dependencies on netdevs, in fact that's one of the reasons we moved to devlink - we don't want to hold rtnl lock just for talking to device firmware.

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#57 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 6 lines · Jakub Kicinski

It's intended to be a generic netlink channel for configuring devices.

All the firmware-related interfaces have no dependencies on netdevs, in fact that's one of the reasons we moved to devlink - we don't want to hold rtnl lock just for talking to device firmware.

Sounds good :)

So I guess Luis just has to add some way in devlink to hook up devlink health in a simple way to drivers, perhaps? I mean, many drivers won't really want to use devlink for anything else, so I guess it should be as simple as the API that Luis proposed ("firmware crashed for this struct device"), if nothing more interesting is done with devlink?

Dunno. But anyway sounds like it should somehow integrate there rather than the way this patchset proposed?

johannes

maintainer
#58 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 9 lines · On Mon, 18 May 2020 22:41:48 +0200 Johannes Berg wrote:
Quoted 5 lines · Jakub Kicinski

It's intended to be a generic netlink channel for configuring devices.

All the firmware-related interfaces have no dependencies on netdevs, in fact that's one of the reasons we moved to devlink - we don't want to hold rtnl lock just for talking to device firmware.

Sounds good :)

Quoted 8 lines

So I guess Luis just has to add some way in devlink to hook up devlink health in a simple way to drivers, perhaps? I mean, many drivers won't really want to use devlink for anything else, so I guess it should be as simple as the API that Luis proposed ("firmware crashed for this struct device"), if nothing more interesting is done with devlink?

Dunno. But anyway sounds like it should somehow integrate there rather than the way this patchset proposed?

Right, that'd be great. Simple API to register a devlink instance with whatever number of reporters the device would need. I'm happy to help.

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#53 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Johannes Berg
Quoted 29 lines · Johannes Berg
Quoted 5 lines · Luis Chamberlain
Quoted 3 lines

Err, no. Those two are most definitely related. Have you looked at (most or some or whatever) staging drivers recently? Those contain all kinds of garbage that might do whatever with your kernel.

No, I stay away :)

:)

Quoted 6 lines
Quoted 3 lines

That's all fine, I just don't think it's appropriate to pretend that your kernel is now 'tainted' (think about the meaning of that word) when the firmware of some random device crashed.

If the firmware crash *does* require driver remove / addition again, or a reboot, would you think that this is a situation that merits a taint?

Not really. In my experience, that's more likely a hardware issue (card not properly seated, for example) that a bus reset happens to "fix".

Quoted 8 lines
Quoted 2 lines

It's pretty clear, but even then, first of all I doubt this is the case for many of the places that you've sprinkled the annotation on,

We can remove it, for this driver I can vouch for its location as it did reach a state where I required a reboot. And its not the first time this has happened. This got me thinking about the bigger picture of the lack of proper way to address these cases in the kernel, and how the user is left dumbfounded.

Quoted 4 lines

Fair, so the driver is still broken wrt. recovery here. I still don't think that's a situation where e.g. the system should say "hey you have a taint here, if your graphics go bad now you should not report that bug" (which is effectively what the single taint bit does).

But again, let's think about the generic type of issue, and the unexpected type of state that can be reached. The circumstance here *does* lead to a case which is not recoverable. Now, consider how many cases in the kernel where similar situations can happen and leave the device or driver in a non-functional state.

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

and secondly it actually hides useful information.

What is it hiding?

Most importantly, which device crashed. Secondarily I'd say how many times (*).

The device is implied by the module, the taint is applied to both. If you had multiple devices, however, yes, it would not be possible to distinguish from the taint which exact device it happened on.

So the only thing *generic* which would be left out is count.

Quoted 2 lines

The information "firmware crashed" is really only useful in relation to the device.

If you have to reboot to get a functional network again then the device is quite useless for many people, regardless of which device that happened on.

But from a support perspective a sysfs interface which provides a tiny bit more generic information indeed provides more value than a taint.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#59 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Jakub Kicinski
Quoted 16 lines · Jakub Kicinski
Quoted 15 lines · On Mon, 18 May 2020 22:41:48 +0200 Johannes Berg wrote:
Quoted 5 lines · Jakub Kicinski

It's intended to be a generic netlink channel for configuring devices.

All the firmware-related interfaces have no dependencies on netdevs, in fact that's one of the reasons we moved to devlink - we don't want to hold rtnl lock just for talking to device firmware.

Sounds good :)

So I guess Luis just has to add some way in devlink to hook up devlink health in a simple way to drivers, perhaps? I mean, many drivers won't really want to use devlink for anything else, so I guess it should be as simple as the API that Luis proposed ("firmware crashed for this struct device"), if nothing more interesting is done with devlink?

Quoted 5 lines
Quoted 2 lines

Dunno. But anyway sounds like it should somehow integrate there rather than the way this patchset proposed?

Right, that'd be great. Simple API to register a devlink instance with whatever number of reporters the device would need. I'm happy to help.

Indeed my issue with devlink is that it did not seem generic enough for all devices which use firmware and for which firmware can crash. Support should not have to be "add devlink support" + "now use this new hook", but rather a very lighweight devlink_crash(device) call we can sprinkly with only the device as a functional requirement.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

maintainer
#60 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 5 lines · On Mon, 18 May 2020 21:22:02 +0000 Luis Chamberlain wrote:

Indeed my issue with devlink is that it did not seem generic enough for all devices which use firmware and for which firmware can crash. Support should not have to be "add devlink support" + "now use this new hook", but rather a very lighweight devlink_crash(device) call we can sprinkly with only the device as a functional requirement.

We can provide a lightweight devlink_crash(device) which only generates the notification, without the need to register the health reporter or a devlink instance upfront. But then we loose the ability to control the recovery, count errors, etc. So I'd think that's not the direction we want to go in.

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#61 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 7 lines · Jakub Kicinski
Quoted 5 lines · On Mon, 18 May 2020 21:22:02 +0000 Luis Chamberlain wrote:

Indeed my issue with devlink is that it did not seem generic enough for all devices which use firmware and for which firmware can crash. Support should not have to be "add devlink support" + "now use this new hook", but rather a very lighweight devlink_crash(device) call we can sprinkly with only the device as a functional requirement.

Quoted 5 lines

We can provide a lightweight devlink_crash(device) which only generates the notification, without the need to register the health reporter or a devlink instance upfront. But then we loose the ability to control the recovery, count errors, etc. So I'd think that's not the direction we want to go in.

Care to show me what the diff stat for a non devlink driver would look like? Just keep in mind this taint is 1 line addition. Granted, if we can use SmPL grammar to automate addition of an initial framework to a driver that'd be great, but since the device addition is subsystem specific (device_add() and friends), I don't suspect this will be easily automated.

Luis

#35 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Johannes Berg
Quoted 7 lines · Johannes Berg

In addition, look what we have in iwl_trans_pcie_removal_wk(). If we detect that the device is really wedged enough that the only way we can still try to recover is by completely unbinding the driver from it, then we give userspace a uevent for that. I don't remember exactly how and where that gets used (ChromeOS) though, but it'd be nice to have that sort of thing as part of the infrastructure, in a sort of two-level notification?

<slight side track>
We use this on certain devices where we know the underlying hardware has design issues that may lead to device failure -- then when we see this sort of unrecoverable "firmware-death", we remove the device[*]+driver, force-reset the PCI device (SBR), and try to reload/reattach the driver. This all happens by way of a udev rule. We also log this sort of stuff (and metrics around it) for bug reports and health statistics, since we really hope to not see this happen often.

[*] "We" (user space) don't actually do this...it happens via the 'remove_when_gone' module parameter abomination found in iwlwifi. I'd personally rather see the EVENT=INACESSIBLE stuff on its own, and let user space deal with when and how to remove and reset the device. But I digress too much here ;)
</slight side track>

I really came to this thread to say that I also love the idea of a generic mechanism (a la $subject) to report firmware crashes, but I also have no interest in seeing a taint flag for it. For Chrome OS, I would readily (as in, we're already looking at more-hacky / non-generic ways to do this for drivers we care about) process these kinds of stats as they happen, logging metrics for bug reports and/or for automated crash statistics, when we see a firmware crash. A uevent would suit us very well I think, although it would be nice if drivers could also supply some small amount of informative text along with it (e.g., a sort of "reason code", in case we can possibly aggregate certain failure types). We already do this sort of thing for WARN() and friends (not via uevent, but via log parsing; at least it has nice "cut here" markers!).

Perhaps devlink (as proposed down-thread) would also fit the bill. I don't think sysfs alone would fit our needs, as we'd like to process these things as they happen, not only when a user submits a bug report.

Quoted 2 lines

Level 1: firmware crashed, but we're recovering, at least mostly, and it's more informational

Chrome OS would love to track these things too, since we'd like to see these minimized, even if they're usually recoverable ;)

Quoted 3 lines

Level 2: device is wedged, going to try to recover by some more forceful means (perhaps some devices can be power-cycled? etc.) but (more) state would be lost in these cases?

And we'd definitely want to know about these. We already get this for the iwlwifi case described above, in a non-generic way.

In general, it's probably not that easy to tell the difference between 1 and 2, since even as you and Luis have noted, with the same driver (and the same driver location), you find the same crashes may or may not be recoverable. iwlwifi has extracted certain level 2 cases into iwl_trans_pcie_removal_wk(), but even iwlwifi doesn't know all the ways in which level 1 crashes actually lead to severe (non-recoverable) failure.

Regards,
Brian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#36 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 9 lines · Brian Norris
Quoted 7 lines · Johannes Berg

In addition, look what we have in iwl_trans_pcie_removal_wk(). If we detect that the device is really wedged enough that the only way we can still try to recover is by completely unbinding the driver from it, then we give userspace a uevent for that. I don't remember exactly how and where that gets used (ChromeOS) though, but it'd be nice to have that sort of thing as part of the infrastructure, in a sort of two-level notification?

Quoted 3 lines

<slight side track>
We use this on certain devices where we know the underlying hardware has design issues that may lead to device failure

Ah, after reading below I see you meant for iwlwifi.

If userspace can indeed grow to support this, that would be fantastic.

I should note that I don't discourage hiding firmware or hardware issues. Quite the contrary, I suspect that taking pride in being trasnparent about it, and dealing with it fast can help lead the pack. I wrote about this long ago in 2015 [0], and stand by it.

[0] https://www.do-not-panic.com/2015/04/god-complex-why-open-models-will-win.html

Quoted 4 lines

-- then when we see
this sort of unrecoverable "firmware-death", we remove the device[*]+driver, force-reset the PCI device (SBR), and try to reload/reattach the driver. This all happens by way of a udev rule.

So you've sprikled your own udev event here as part of your kernel delta?

Quoted 4 lines

We
also log this sort of stuff (and metrics around it) for bug reports and health statistics, since we really hope to not see this happen often.

Assuming perfection is ideal but silly. So, what infrastructure do you use for this sort of issue?

Quoted 2 lines

[*] "We" (user space) don't actually do this...it happens via the 'remove_when_gone' module parameter abomination found in iwlwifi.

Holy moly.. but hey, at least it may seem a bit more seemless than forcing a reboot / manual driver removal / addition to the user.

BTW is this likely a place on iwlwifi where the firmware likely crashed?

Quoted 5 lines

I'd
personally rather see the EVENT=INACESSIBLE stuff on its own, and let user space deal with when and how to remove and reset the device. But I digress too much here ;)
</slight side track>

This is all useful information. We are just touching the surface of the topic by addressing networking first. Imagine when we address other subsystems.

Quoted 7 lines

I really came to this thread to say that I also love the idea of a generic mechanism (a la $subject) to report firmware crashes, but I also have no interest in seeing a taint flag for it. For Chrome OS, I would readily (as in, we're already looking at more-hacky / non-generic ways to do this for drivers we care about) process these kinds of stats as they happen, logging metrics for bug reports and/or for automated crash statistics, when we see a firmware crash.

Great!

Quoted 3 lines

A uevent
would suit us very well I think, although it would be nice if drivers could also supply some small amount of informative text along with it

A follow up to this series was to add a uevent to add_taint(), however since a *count* is not considered I think it is correct to seek alternatives at this point. The leaner the solution the better though.

Do you have a pointer to what guys use so I can read?

Quoted 4 lines

(e.g., a sort of "reason code", in case we can possibly aggregate certain failure types). We already do this sort of thing for WARN() and friends (not via uevent, but via log parsing; at least it has nice "cut here" markers!).

Indeed, similar things can indeed be argued about WARN*()... this however can be non-device specific. With panic-on-warn becoming a "thing", the more important it becomes to really tally exactly *why* these WARN*()s may trigger.

Quoted 1 line

Perhaps

Note below.

Quoted 4 lines

devlink (as proposed down-thread) would also fit the bill. I don't think sysfs alone would fit our needs, as we'd like to process these things as they happen, not only when a user submits a bug report.

I think we've reached a point where using "*Perhaps*" does not suffice, and if there is already a *user* of similar desired infrastructure I think we should jump on the opportunity to replace what you have with something which could be used by other devices / subsystems which require firmware. And indeed, also even consider in the abstract sense, the possibility to leverage something like this for WARN*()s later too.

Quoted 13 lines
Quoted 2 lines

Level 1: firmware crashed, but we're recovering, at least mostly, and it's more informational

Chrome OS would love to track these things too, since we'd like to see these minimized, even if they're usually recoverable ;)

Quoted 3 lines

Level 2: device is wedged, going to try to recover by some more forceful means (perhaps some devices can be power-cycled? etc.) but (more) state would be lost in these cases?

And we'd definitely want to know about these. We already get this for the iwlwifi case described above, in a non-generic way.

Quoted 7 lines

In general, it's probably not that easy to tell the difference between 1 and 2, since even as you and Luis have noted, with the same driver (and the same driver location), you find the same crashes may or may not be recoverable. iwlwifi has extracted certain level 2 cases into iwl_trans_pcie_removal_wk(), but even iwlwifi doesn't know all the ways in which level 1 crashes actually lead to severe (non-recoverable) failure.

And that is fine, accepting these for what they are will help. However, leaving the user in the *dark*, is what we should *not do*.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#3 lore raw

Re: [PATCH v2 01/15] taint: add module firmware crash taint support

In reply to Luis Chamberlain

+++ Luis Chamberlain [15/05/20 21:28 +0000]:

Quoted 66 lines

Device driver firmware can crash, and sometimes, this can leave your system in a state which makes the device or subsystem completely useless. Detecting this by inspecting /proc/sys/kernel/tainted instead of scraping some magical words from the kernel log, which is driver specific, is much easier. So instead provide a helper which lets drivers annotate this.

Once this happens, scrapers can easily look for modules taint flags for a firmware crash. This will taint both the kernel and respective calling module.

The new helper module_firmware_crashed() uses LOCKDEP_STILL_OK as this fact should in no way shape or form affect lockdep. This taint is device driver specific.

---

Documentation/admin-guide/tainted-kernels.rst |  6 ++++++
include/linux/kernel.h                        |  3 ++-
include/linux/module.h                        | 13 +++++++++++++
include/trace/events/module.h                 |  3 ++-
kernel/module.c                               |  5 +++--
kernel/panic.c                                |  1 +
tools/debugging/kernel-chktaint               |  7 +++++++
7 files changed, 34 insertions(+), 4 deletions(-)
Documentation/admin-guide/tainted-kernels.rst +1 -0
index 71e9184a9079..92530f1d60ae 100644
@@ -100,6 +100,7 @@ Bit  Log  Number  Reason that got the kernel tainted
 15  _/K   32768  kernel has been live patched
 16  _/X   65536  auxiliary taint, defined for and used by distros
 17  _/T  131072  kernel was built with the struct randomization plugin
+ 18  _/Q  262144  driver firmware crash annotation

=== === ====== ========================================================

Note: The character ``_`` is representing a blank in this table to make reading

Documentation/admin-guide/tainted-kernels.rst +5 -0
@@ -162,3 +163,8 @@ More detailed explanation for tainting
     produce extremely unusual kernel structure layouts (even performance
     pathological ones), which is important to know when debugging. Set at
     build time.
+
+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware
+     has crashed and the device's operation has been severely affected. The
+     device may be left in a crippled state, requiring full driver removal /
+     addition, system reboot, or it is unclear how long recovery will take.
include/linux/kernel.h +2 -1
index 04a5885cec1b..19e1541c82c7 100644
@@ -601,7 +601,8 @@ extern enum system_states {
 #define TAINT_LIVEPATCH			15
 #define TAINT_AUX			16
 #define TAINT_RANDSTRUCT		17
-#define TAINT_FLAGS_COUNT		18
+#define TAINT_FIRMWARE_CRASH		18
+#define TAINT_FLAGS_COUNT		19
 
 struct taint_flag {
 	char c_true;	/* character printed when tainted */
include/linux/module.h +0 -0
index 2c2e988bcf10..221200078180 100644

include/linux/module.h static inline bool is_livepatch_module(struct module *mod) · line 706

Quoted 10 lines
include/linux/module.h +7 -0
@@ -697,6 +697,14 @@ static inline bool is_livepatch_module(struct module *mod)
 bool is_module_sig_enforced(void);
 void set_module_sig_enforced(void);
 
+void add_taint_module(struct module *mod, unsigned flag,
+		      enum lockdep_ok lockdep_ok);
+
+static inline void module_firmware_crashed(void)
+{
+	add_taint_module(THIS_MODULE, TAINT_FIRMWARE_CRASH, LOCKDEP_STILL_OK);
+}

Just a nit: I think module_firmware_crashed() is a confusing name - it doesn't really tell me what it's doing, and it's not really related to the rest of the module_* symbols, which mostly have to do with module loader/module specifics. Especially since a driver can be built-in, too. How about taint_firmware_crashed() or something similar?

Also, I think we might crash in add_taint_module() if a driver is built into the kernel, because THIS_MODULE will be null and there is no null pointer check in add_taint_module(). We could unify the CONFIG_MODULES and !CONFIG_MODULES stubs and either add an if (mod) check in add_taint_module() or add an #ifdef MODULE check in the stub itself to call add_taint() or add_taint_module() as appropriate. Hope that makes sense.

Thanks!

Jessica

maintainer
#62 lore raw

RFC 1/2 devlink: add simple fw crash helpers

In reply to Luis Chamberlain

Add infra for creating devlink instances for a device to report fw crashes. This patch expects the devlink instance to be registered at probe time. I belive to be the cleanest. We can also add a devm version of the helpers, so that we don't have to do the clean up. Or we can go even further and register the devlink instance only once error has happened (for the first time, then we can just find out if already registered by traversing the list like we do here).

With the patch applied and a sample driver converted we get:

$ devlink dev
pci/0000:07:00.0

Then monitor for errors:

$ devlink mon health
[health,status] pci/0000:07:00.0:
  reporter fw
    state error error 1 recover 0
[health,status] pci/0000:07:00.0:
  reporter fw
    state error error 2 recover 0

These are the events I triggered on purpose. One can also inspect the health of all devices capable of reporting fw errors:

$ devlink health
pci/0000:07:00.0:
  reporter fw
    state error error 7 recover 0

Obviously drivers may upgrade to the full devlink health API which includes state dump, state dump auto-collect and automatic error recovery control.

---
 include/linux/devlink.h               |  11 +++
 net/core/Makefile                     |   2 +-
 net/core/devlink_simple_fw_reporter.c | 101 ++++++++++++++++++++++++++
 3 files changed, 113 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/devlink.h
 create mode 100644 net/core/devlink_simple_fw_reporter.c
include/linux/devlink.h new file +11 -0
new file mode 100644
index 000000000000..2b73987eefca
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef _LINUX_DEVLINK_H_
+#define _LINUX_DEVLINK_H_
+
+struct device;
+
+void devlink_simple_fw_reporter_prepare(struct device *dev);
+void devlink_simple_fw_reporter_cleanup(struct device *dev);
+void devlink_simple_fw_reporter_report_crash(struct device *dev);
+
+#endif
net/core/Makefile +1 -1
index 3e2c378e5f31..6f1513781c17 100644
@@ -31,7 +31,7 @@ obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o
 obj-$(CONFIG_BPF_STREAM_PARSER) += sock_map.o
 obj-$(CONFIG_DST_CACHE) += dst_cache.o
 obj-$(CONFIG_HWBM) += hwbm.o
-obj-$(CONFIG_NET_DEVLINK) += devlink.o
+obj-$(CONFIG_NET_DEVLINK) += devlink.o devlink_simple_fw_reporter.o
 obj-$(CONFIG_GRO_CELLS) += gro_cells.o
 obj-$(CONFIG_FAILOVER) += failover.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o
net/core/devlink_simple_fw_reporter.c new file +101 -0

101 lines not on this page · raw message

Signature · 7 lines
2.25.4


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
maintainer
#74 lore raw

RFC 2/2 i2400m: use devlink health reporter

In reply to Luis Chamberlain

It builds.

---
 drivers/net/wimax/i2400m/rx.c  | 2 ++
 drivers/net/wimax/i2400m/usb.c | 5 +++++
 2 files changed, 7 insertions(+)
drivers/net/wimax/i2400m/rx.c +2 -0
index c9fb619a9e01..cc7fe78f2df0 100644
@@ -144,6 +144,7 @@
  *       i2400m_msg_size_check
  *       wimax_msg
  */
+#include <linux/devlink.h>
 #include <linux/slab.h>
 #include <linux/kernel.h>
 #include <linux/if_arp.h>
@@ -712,6 +713,7 @@ void __i2400m_roq_queue(struct i2400m *i2400m, struct i2400m_roq *roq,
 	dev_err(dev, "SW BUG? failed to insert packet\n");
 	dev_err(dev, "ERX: roq %p [ws %u] skb %p nsn %d sn %u\n",
 		roq, roq->ws, skb, nsn, roq_data->sn);
+	devlink_simple_fw_reporter_report_crash(dev);
 	skb_queue_walk(&roq->queue, skb_itr) {
 		roq_data_itr = (struct i2400m_roq_data *) &skb_itr->cb;
 		nsn_itr = __i2400m_roq_nsn(roq, roq_data_itr->sn);
drivers/net/wimax/i2400m/usb.c +5 -0
index 9659f9e1aaa6..5c811dccbf1d 100644
@@ -49,6 +49,7 @@
  *   usb_reset_device()
  */
 #include "i2400m-usb.h"
+#include <linux/devlink.h>
 #include <linux/wimax/i2400m.h>
 #include <linux/debugfs.h>
 #include <linux/slab.h>
@@ -423,6 +424,8 @@ int i2400mu_probe(struct usb_interface *iface,
 	if (usb_dev->speed != USB_SPEED_HIGH)
 		dev_err(dev, "device not connected as high speed\n");
 
+	devlink_simple_fw_reporter_prepare(dev);
+
 	/* Allocate instance [calls i2400m_netdev_setup() on it]. */
 	result = -ENOMEM;
 	net_dev = alloc_netdev(sizeof(*i2400mu), "wmx%d", NET_NAME_UNKNOWN,
@@ -506,6 +509,7 @@ int i2400mu_probe(struct usb_interface *iface,
 	usb_put_dev(i2400mu->usb_dev);
 	free_netdev(net_dev);
 error_alloc_netdev:
+	devlink_simple_fw_reporter_cleanup(dev);
 	return result;
 }
 
@@ -532,6 +536,7 @@ void i2400mu_disconnect(struct usb_interface *iface)
 	usb_set_intfdata(iface, NULL);
 	usb_put_dev(i2400mu->usb_dev);
 	free_netdev(net_dev);
+	devlink_simple_fw_reporter_cleanup(dev);
 	d_fnend(3, dev, "(iface %p i2400m %p) = void\n", iface, i2400m);
 }
 
Signature · 7 lines
2.25.4


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#24 lore raw

Re: [PATCH v2 10/15] soc: qcom: ipa: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 7 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

I don't fully understand what this is meant to do, so I can't fully assess whether it's the right thing to do.

But in this particular place in the IPA code, the *modem* has crashed. And the IPA driver is not responsible for modem firmware, remoteproc is.

The IPA driver *can* be responsible for loading some other firmware, but even in that case, it only happens on initial boot, and it's basically assumed to never crash.

So regardless of whether this module_firmware_crashed() call is appropriate in some places, I believe it should not be used here.

-Alex

Quoted 20 lines
---
  drivers/net/ipa/ipa_modem.c | 1 +
  1 file changed, 1 insertion(+)
drivers/net/ipa/ipa_modem.c +1 -0
index ed10818dd99f..1790b87446ed 100644
@@ -285,6 +285,7 @@ static void ipa_modem_crashed(struct ipa *ipa)
  	struct device *dev = &ipa->pdev->dev;
  	int ret;
 
+	module_firmware_crashed();
  	ipa_endpoint_modem_pause_all(ipa, true);
 
  	ipa_endpoint_modem_hol_block_clear_all(ipa);
#37 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Luis Chamberlain

Hi Luis,

Quoted 10 lines · Luis Chamberlain
Quoted 9 lines · Brian Norris
Quoted 7 lines · Johannes Berg

In addition, look what we have in iwl_trans_pcie_removal_wk(). If we detect that the device is really wedged enough that the only way we can still try to recover is by completely unbinding the driver from it, then we give userspace a uevent for that. I don't remember exactly how and where that gets used (ChromeOS) though, but it'd be nice to have that sort of thing as part of the infrastructure, in a sort of two-level notification?

Quoted 5 lines
Quoted 3 lines

<slight side track>
We use this on certain devices where we know the underlying hardware has design issues that may lead to device failure

Ah, after reading below I see you meant for iwlwifi.

Sorry, I was replying to Johannes, who I believe had his "we"="Intel" hat (as iwlwifi maintainer) on, and was pointing at iwl_trans_pcie_removal_wk().

Quoted 1 line

If userspace can indeed grow to support this, that would be fantastic.

Well, Chrome OS tailors its user space a bit more to the hardware (and kernel/drivers in use) than the average distro might. We already do this (for some values of "this") today. Is that "fantastic" to you? :D

Quoted 6 lines
Quoted 4 lines

-- then when we see
this sort of unrecoverable "firmware-death", we remove the device[*]+driver, force-reset the PCI device (SBR), and try to reload/reattach the driver. This all happens by way of a udev rule.

So you've sprikled your own udev event here as part of your kernel delta?

No kernel delta -- the event is there already:
iwl_trans_pcie_removal_wk()
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/wireless/intel/iwlwifi/pcie/trans.c?h=v5.6#n2027

And you can see our udev rules and scripts, in all their ugly details here, if you really care:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/net-wireless/iwlwifi_rescan/files/

Quoted 7 lines
Quoted 4 lines

We
also log this sort of stuff (and metrics around it) for bug reports and health statistics, since we really hope to not see this happen often.

Assuming perfection is ideal but silly. So, what infrastructure do you use for this sort of issue?

We don't yet log firmware crashes generally, but for all our current crash reports (including WARN()), they go through this:
https://chromium.googlesource.com/chromiumos/platform2/+/master/crash-reporter/README.md

For example, look for "cut here" in:
https://chromium.googlesource.com/chromiumos/platform2/+/master/crash-reporter/anomaly_detector.cc

For other specific metrics (like counting "EVENT=INACCESSIBLE"), we use the Chrome UMA system:
https://chromium.googlesource.com/chromiumos/platform2/+/master/metrics/README.md

I don't imagine the "infrastructure" side of any of that would be useful to you, but maybe the client-side gathering can at least show you what we do.

Quoted 4 lines
Quoted 2 lines

[*] "We" (user space) don't actually do this...it happens via the 'remove_when_gone' module parameter abomination found in iwlwifi.

BTW is this likely a place on iwlwifi where the firmware likely crashed?

iwl_trans_pcie_removal_wk() is triggered because HW accesses timed out in a way that is likely due to a dead PCIe endpoint. It's not directly a firmware crash, although there may be firmware crashes reported around the same time.

Intel folks can probably give a more nuanced answer, but their firmware crashes usually land in something like iwl_mvm_nic_error() -> iwl_mvm_dump_nic_error_log() + iwl_mvm_nic_restart(). If you can make your proposal less punishing (e.g., removing the "taint" as Johannes requested), maybe iwlwifi would be another good candidate for $subject. iwlwifi generally expects to recover seamlessly, but it's also good to know if you've seen a hundred of these in a row.

Quoted 9 lines
Quoted 3 lines

A uevent
would suit us very well I think, although it would be nice if drivers could also supply some small amount of informative text along with it

A follow up to this series was to add a uevent to add_taint(), however since a *count* is not considered I think it is correct to seek alternatives at this point. The leaner the solution the better though.

Do you have a pointer to what guys use so I can read?

Hopefully the above pointers are useful to you. We don't yet have firmware-crash parsing implemented, so I don't have pointers for that piece yet.

Quoted 9 lines
Quoted 4 lines

(e.g., a sort of "reason code", in case we can possibly aggregate certain failure types). We already do this sort of thing for WARN() and friends (not via uevent, but via log parsing; at least it has nice "cut here" markers!).

Indeed, similar things can indeed be argued about WARN*()... this however can be non-device specific. With panic-on-warn becoming a "thing", the more important it becomes to really tally exactly *why* these WARN*()s may trigger.

panic-on-warn? Yikes. I guess such folks don't run mac80211... I'm probably not supposed to publicize information related to how many Chromebooks are out there, but we regularly see a scary number of non-fatal warnings (WARN(), WARN_ON(), etc.) logged by Chrome OS users every day, and a scary number of those are in WiFi drivers...

Quoted 3 lines
Quoted 1 line

Perhaps

Note below.

(My use of "perhaps" is only because I'm not familiar with devlink at all.)

Quoted 11 lines
Quoted 4 lines

devlink (as proposed down-thread) would also fit the bill. I don't think sysfs alone would fit our needs, as we'd like to process these things as they happen, not only when a user submits a bug report.

I think we've reached a point where using "*Perhaps*" does not suffice, and if there is already a *user* of similar desired infrastructure I think we should jump on the opportunity to replace what you have with something which could be used by other devices / subsystems which require firmware. And indeed, also even consider in the abstract sense, the possibility to leverage something like this for WARN*()s later too.

To precisely lay out what Chrome OS does today:

* WARN() and similar: implemented, see anomaly_detector.cc above. It's just log parsing, and that handy "cut here" stuff -- I'm nearly certain there are other distros using this already? A uevent would probably be nice, but log parsing is good enough for us today.
* EVENT=INACCESSIBLE: iwlwifi-specific, but reference code is linked above. It's a uevent, and we handle it via udev. Code is linked above.
* Other firmware crashes: not yet implemented here, but we're looking at it. Currently, we plan to do something similar to WARN(), but it will be ugly and non-generic. A uevent would be a lovely replacement, if it has some basic metadata with it. Stats in sysfs might be icing on the cake, but mostly redundant for us, if we can grab it on the fly via uevent.

HTH,
Brian

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#38 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Hi all,

<top post intro>

Since I have been involved quite a bit in the firmware debugging features in iwlwifi, I think I can give a few insights here.

But before this, we need to understand that there are several sources of issues:
1) the firmware may crash but the bus is still alive, you can still use the bus to get the crash data
2) the bus is dead, when that happens, the firmware might even be in a good condition, but since the bus is dead, you stop getting any information about the firmware, and then, at some point, you get to the conclusion that the firmware is dead. You can't get the crash data that resides on the other side of the bus (you may have gathered data in the DRAM directly, but that's a different thing), and you don't have much recovery to do besides re-starting the PCI enumeration.

At Intel, we have seen both unfortunately. The bus issues are the ones that are trickier obviously. Trickier to detect (because you just get garbage from any request you issue on the bus), and trickier to handle. One can argue that the kernel should *not* handle those and let this in userspace hands. I guess it all depends on what component you ship to your customer and what you customer asks from you :).

</top post intro>

Quoted 18 lines

Hi Luis,

Quoted 14 lines · Luis Chamberlain
Quoted 12 lines · Brian Norris
Quoted 7 lines · Johannes Berg

In addition, look what we have in iwl_trans_pcie_removal_wk(). If we detect that the device is really wedged enough that the only way we can still try to recover is by completely unbinding the driver from it, then we give userspace a uevent for that. I don't remember exactly how and where that gets used (ChromeOS) though, but it'd be nice to have that sort of thing as part of the infrastructure, in a sort of two-level notification?

<slight side track>
We use this on certain devices where we know the underlying hardware has design issues that may lead to device failure

Quoted 6 lines
Quoted 1 line

Ah, after reading below I see you meant for iwlwifi.

Sorry, I was replying to Johannes, who I believe had his "we"="Intel" hat (as iwlwifi maintainer) on, and was pointing at iwl_trans_pcie_removal_wk().

This pcie_removal thing is for the bus dead thing. My 2) above.

Quoted 5 lines
Quoted 1 line

If userspace can indeed grow to support this, that would be fantastic.

Well, Chrome OS tailors its user space a bit more to the hardware (and kernel/drivers in use) than the average distro might. We already do this (for some values of "this") today. Is that "fantastic" to you? :D

I guess it can be fantastic if other vendors also suffer from this. Or maybe that could be done as part of the PCI bus driver inside the kernel?

Quoted 42 lines
Quoted 7 lines
Quoted 5 lines

-- then when we see
this sort of unrecoverable "firmware-death", we remove the device[*]+driver, force-reset the PCI device (SBR), and try to reload/reattach the driver. This all happens by way of a udev rule.

So you've sprikled your own udev event here as part of your kernel delta?

No kernel delta -- the event is there already:
iwl_trans_pcie_removal_wk()
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/wireless/intel/iwlwifi/pcie/trans.c?h=v5.6#n2027

And you can see our udev rules and scripts, in all their ugly details here, if you really care:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/master/net-wireless/iwlwifi_rescan/files/

Quoted 7 lines
Quoted 4 lines

We
also log this sort of stuff (and metrics around it) for bug reports and health statistics, since we really hope to not see this happen often.

Assuming perfection is ideal but silly. So, what infrastructure do you use for this sort of issue?

We don't yet log firmware crashes generally, but for all our current crash reports (including WARN()), they go through this:
https://chromium.googlesource.com/chromiumos/platform2/+/master/crash-reporter/README.md

For example, look for "cut here" in:
https://chromium.googlesource.com/chromiumos/platform2/+/master/crash-reporter/anomaly_detector.cc

For other specific metrics (like counting "EVENT=INACCESSIBLE"), we use the Chrome UMA system:
https://chromium.googlesource.com/chromiumos/platform2/+/master/metrics/README.md

I don't imagine the "infrastructure" side of any of that would be useful to you, but maybe the client-side gathering can at least show you what we do.

Quoted 3 lines
Quoted 2 lines

[*] "We" (user space) don't actually do this...it happens via the 'remove_when_gone' module parameter abomination found in iwlwifi.

Quoted 6 lines
Quoted 1 line

BTW is this likely a place on iwlwifi where the firmware likely crashed?

iwl_trans_pcie_removal_wk() is triggered because HW accesses timed out in a way that is likely due to a dead PCIe endpoint. It's not directly a firmware crash, although there may be firmware crashes reported around the same time.

iwl_trans_pcie_removal_wk() is only because of a dead bus, not because of a firmware crash.

Quoted 8 lines

Intel folks can probably give a more nuanced answer, but their firmware crashes usually land in something like iwl_mvm_nic_error() -> iwl_mvm_dump_nic_error_log() + iwl_mvm_nic_restart(). If you can make your proposal less punishing (e.g., removing the "taint" as Johannes requested), maybe iwlwifi would be another good candidate for $subject. iwlwifi generally expects to recover seamlessly, but it's also good to know if you've seen a hundred of these in a row.

Yes, you are right, mvm_nic_error is the place.

So here is the proposal.
I think that a standard interface that can allow a driver to report a firmware crash along with a proprietary unique id that makes sense to the vendor can be useful. Then, yes, distros can listen to this, optionally open bugs (automatically?) when that happens. I even planned to do this long ago and integrated with a specific distro, but it never rolled out. The vendor supplied unique id is very important for the bug de-duplication part. For iwlwifi, we have the SYSASSERT number which is basically how the firmware tells us briefly what happened. Of course, there is always more data that can be useful, but for a first level of bug de-duplication this can be enough. Then, you have a standard way to track the firmware crashes.
We need to remember that the firmware recovery path is expected to work, it is, yet, less tested. I specifically remember a bug where a crash because by a bad handling of a CSA frame caused a firmware crash in a flow that caused the driver not to re-add a station or something like that, and because of that, you get another firmware crash. So sometimes it is interesting to see not only the data on which firmware crash happened and how many times, but if there is a timing relationship between them. I guess that's for the next level though... Not really critical for now.

The next problem here is that when you tell the firmware folks that they have a bug, the first they'll do is to ask for more data. Back then, I enabled our firmware debug infra on Linux, and from there devcoredump infra was born (thanks Johannes). What we do at Intel, is that we have a script that runs when a udev event reports the creation of a devcoredump. It parses the kernel log (dmesg) to determine the unique id I mentioned earlier (the SYSSASSERT number basically) and then it creates a file in /var/log/ whose name contain the SYSSASSERT number. It is then quite easy to match the firmware data with the firmware crash.
So I believe the right way to do this is to create the devcoredump along with the id. Meaning that we basically don't need another interface, we just need to use the same one, but to provide the unique id of the crash. This unique id can then be part of the uevent that is thrown to the userspace and userspace can use it to name the dump file with the right id. This way, it is fairly easy (and standard across vendors) to track the firmware crashes, but the most important is that you already have the firmware data that goes along with it. It would look like this.

driver_code.c:
void my_vendor_error_interrupt()
{
u64 uid = get_the_unique_id_from_your_device();
void *firmware_data = get_the_data_you_need();

dev_coredumpsg(dev_struct, firmware_data, firmware_data_len,

                              GFP_whatver, uid);
}

And then, this would cause a:
/var/log/wifi-crash-$(KBUILD_MODNAME)-,uid.bin to appear our your filesystem.

Quoted 9 lines
Quoted 9 lines
Quoted 4 lines

A uevent
would suit us very well I think, although it would be nice if drivers could also supply some small amount of informative text along with it

A follow up to this series was to add a uevent to add_taint(), however since a *count* is not considered I think it is correct to seek alternatives at this point. The leaner the solution the better though.

Quoted 5 lines
Quoted 1 line

Do you have a pointer to what guys use so I can read?

Hopefully the above pointers are useful to you. We don't yet have firmware-crash parsing implemented, so I don't have pointers for that piece yet.

See above. I don't think it is the device driver's role to count those. We can add this count this in userspace I think. Debatable though.

Quoted 37 lines
Quoted 10 lines
Quoted 5 lines

(e.g., a sort of "reason code", in case we can possibly aggregate certain failure types). We already do this sort of thing for WARN() and friends (not via uevent, but via log parsing; at least it has nice "cut here" markers!).

Indeed, similar things can indeed be argued about WARN*()... this however can be non-device specific. With panic-on-warn becoming a "thing", the more important it becomes to really tally exactly *why* these WARN*()s may trigger.

panic-on-warn? Yikes. I guess such folks don't run mac80211... I'm probably not supposed to publicize information related to how many Chromebooks are out there, but we regularly see a scary number of non-fatal warnings (WARN(), WARN_ON(), etc.) logged by Chrome OS users every day, and a scary number of those are in WiFi drivers...

Quoted 3 lines
Quoted 1 line

Perhaps

Note below.

(My use of "perhaps" is only because I'm not familiar with devlink at all.)

Quoted 11 lines
Quoted 4 lines

devlink (as proposed down-thread) would also fit the bill. I don't think sysfs alone would fit our needs, as we'd like to process these things as they happen, not only when a user submits a bug report.

I think we've reached a point where using "*Perhaps*" does not suffice, and if there is already a *user* of similar desired infrastructure I think we should jump on the opportunity to replace what you have with something which could be used by other devices / subsystems which require firmware. And indeed, also even consider in the abstract sense, the possibility to leverage something like this for WARN*()s later too.

To precisely lay out what Chrome OS does today:

Quoted 12 lines

* WARN() and similar: implemented, see anomaly_detector.cc above. It's just log parsing, and that handy "cut here" stuff -- I'm nearly certain there are other distros using this already? A uevent would probably be nice, but log parsing is good enough for us today.
* EVENT=INACCESSIBLE: iwlwifi-specific, but reference code is linked above. It's a uevent, and we handle it via udev. Code is linked above.
* Other firmware crashes: not yet implemented here, but we're looking at it. Currently, we plan to do something similar to WARN(), but it will be ugly and non-generic. A uevent would be a lovely replacement, if it has some basic metadata with it. Stats in sysfs might be icing on the cake, but mostly redundant for us, if we can grab it on the fly via uevent.

So I believe we already have this uevent, it is the devcoredump. All we need is to add the unique id.
Note that all this is good for firmware crashes, and not for bus dead scenarios, but those two are fundamentally different even if they look alike at the beginning of your error detection flow.

Quoted 3 lines

HTH,
Brian

#39 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 14 lines · Emmanuel Grumbach

Since I have been involved quite a bit in the firmware debugging features in iwlwifi, I think I can give a few insights here.

But before this, we need to understand that there are several sources of issues:
1) the firmware may crash but the bus is still alive, you can still use the bus to get the crash data
2) the bus is dead, when that happens, the firmware might even be in a good condition, but since the bus is dead, you stop getting any information about the firmware, and then, at some point, you get to the conclusion that the firmware is dead. You can't get the crash data that resides on the other side of the bus (you may have gathered data in the DRAM directly, but that's a different thing), and you don't have much recovery to do besides re-starting the PCI enumeration.

Quoted 6 lines

At Intel, we have seen both unfortunately. The bus issues are the ones that are trickier obviously. Trickier to detect (because you just get garbage from any request you issue on the bus), and trickier to handle. One can argue that the kernel should *not* handle those and let this in userspace hands. I guess it all depends on what component you ship to your customer and what you customer asks from you :).

Or the two best approaches:
1) get rid of firmware completely;
2) make it OSS (like SOF).

I think any of these is a right thing to do in long-term perspective.

How many firmwares average computer has? 50? 100? Any of them is a burden and PITA.

Signature · 7 lines
With Best Regards,
Andy Shevchenko

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#40 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Emmanuel Grumbach
Quoted 2 lines · Emmanuel Grumbach

So I believe we already have this uevent, it is the devcoredump. All we need is to add the unique id.

I think there are a few reasons that devcoredump doesn't satisfy what either Luis or I want.

1) it can be disabled entirely [1], for good reasons (e.g., think of non-${CHIP_VENDOR} folks, who can't (and don't want to) do anything with the opaque dumps provided by closed-source firmware)
2) not all drivers necessarily have a useful dump to provide when there's a crash; look at the rest of Luis's series to see the kinds of drivers-with-firmware that are crashing, some of which aren't dumping anything
3) for those that do support devcoredump, it may be used for purposes that are not "crashes" -- e.g., some provide debugfs or other knobs to initiate dumps, for diagnostic or debugging purposes

Brian

[1] devcd_disabled
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/devcoredump.c?h=v5.6#n22

#41 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

Quoted 11 lines · Emmanuel Grumbach
Quoted 2 lines

So I believe we already have this uevent, it is the devcoredump. All we need is to add the unique id.

I think there are a few reasons that devcoredump doesn't satisfy what either Luis or I want.

1) it can be disabled entirely [1], for good reasons (e.g., think of non-${CHIP_VENDOR} folks, who can't (and don't want to) do anything with the opaque dumps provided by closed-source firmware)

Ok, if all you're interested into is the information that this event happen (as opposed to report a bug and providing the data), then I agree. True, not everybody want or can enable devcoredump. I am just a bit concerned that we may end up with two interface that notify the same event basically. The ideal maybe would be to be able to optionally reduce the content of the devoredump to nothing more that is already in the dmesg output. But then, it is not what it is meant to be: namely, a core dump..

Quoted 4 lines

2) not all drivers necessarily have a useful dump to provide when there's a crash; look at the rest of Luis's series to see the kinds of drivers-with-firmware that are crashing, some of which aren't dumping anything

Fair enouh.

Quoted 3 lines

3) for those that do support devcoredump, it may be used for purposes that are not "crashes" -- e.g., some provide debugfs or other knobs to initiate dumps, for diagnostic or debugging purposes

Not sure I really think we need to care about those cases, but you already have 2 good arguments :)

Quoted 5 lines

Brian

[1] devcd_disabled
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/base/devcoredump.c?h=v5.6#n22

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#4 lore raw

Re: [PATCH v2 01/15] taint: add module firmware crash taint support

In reply to Jessica Yu
Quoted 70 lines · Jessica Yu

+++ Luis Chamberlain [15/05/20 21:28 +0000]:

Quoted 69 lines

Device driver firmware can crash, and sometimes, this can leave your system in a state which makes the device or subsystem completely useless. Detecting this by inspecting /proc/sys/kernel/tainted instead of scraping some magical words from the kernel log, which is driver specific, is much easier. So instead provide a helper which lets drivers annotate this.

Once this happens, scrapers can easily look for modules taint flags for a firmware crash. This will taint both the kernel and respective calling module.

The new helper module_firmware_crashed() uses LOCKDEP_STILL_OK as this fact should in no way shape or form affect lockdep. This taint is device driver specific.

---

Documentation/admin-guide/tainted-kernels.rst |  6 ++++++
include/linux/kernel.h                        |  3 ++-
include/linux/module.h                        | 13 +++++++++++++
include/trace/events/module.h                 |  3 ++-
kernel/module.c                               |  5 +++--
kernel/panic.c                                |  1 +
tools/debugging/kernel-chktaint               |  7 +++++++
7 files changed, 34 insertions(+), 4 deletions(-)
Documentation/admin-guide/tainted-kernels.rst +1 -0
index 71e9184a9079..92530f1d60ae 100644
@@ -100,6 +100,7 @@ Bit  Log  Number  Reason that got the kernel tainted
 15  _/K   32768  kernel has been live patched
 16  _/X   65536  auxiliary taint, defined for and used by distros
 17  _/T  131072  kernel was built with the struct randomization plugin
+ 18  _/Q  262144  driver firmware crash annotation

=== === ====== ========================================================

Note: The character ``_`` is representing a blank in this table to make reading

Documentation/admin-guide/tainted-kernels.rst +5 -0
@@ -162,3 +163,8 @@ More detailed explanation for tainting
     produce extremely unusual kernel structure layouts (even performance
     pathological ones), which is important to know when debugging. Set at
     build time.
+
+ 18) ``Q`` used by device drivers to annotate that the device driver's firmware
+     has crashed and the device's operation has been severely affected. The
+     device may be left in a crippled state, requiring full driver removal /
+     addition, system reboot, or it is unclear how long recovery will take.
include/linux/kernel.h +2 -1
index 04a5885cec1b..19e1541c82c7 100644
@@ -601,7 +601,8 @@ extern enum system_states {
 #define TAINT_LIVEPATCH			15
 #define TAINT_AUX			16
 #define TAINT_RANDSTRUCT		17
-#define TAINT_FLAGS_COUNT		18
+#define TAINT_FIRMWARE_CRASH		18
+#define TAINT_FLAGS_COUNT		19
 
 struct taint_flag {
 	char c_true;	/* character printed when tainted */
include/linux/module.h +0 -0
index 2c2e988bcf10..221200078180 100644
@@ -697,6 +697,14 @@ static inline bool is_livepatch_module(struct module *mod)
 bool is_module_sig_enforced(void);
 void set_module_sig_enforced(void);
 
Quoted 13 lines

include/linux/module.h static inline bool is_livepatch_module(struct module *mod) · line 706

Quoted 7 lines
include/linux/module.h +7 -0
+void add_taint_module(struct module *mod, unsigned flag,
+		      enum lockdep_ok lockdep_ok);
+
+static inline void module_firmware_crashed(void)
+{
+	add_taint_module(THIS_MODULE, TAINT_FIRMWARE_CRASH, LOCKDEP_STILL_OK);
+}

Just a nit: I think module_firmware_crashed() is a confusing name - it doesn't really tell me what it's doing, and it's not really related to the rest of the module_* symbols, which mostly have to do with module loader/module specifics. Especially since a driver can be built-in, too. How about taint_firmware_crashed() or something similar?

Sure.

Quoted 7 lines

Also, I think we might crash in add_taint_module() if a driver is built into the kernel, because THIS_MODULE will be null and there is no null pointer check in add_taint_module(). We could unify the CONFIG_MODULES and !CONFIG_MODULES stubs and either add an if (mod) check in add_taint_module() or add an #ifdef MODULE check in the stub itself to call add_taint() or add_taint_module() as appropriate. Hope that makes sense.

I had to do something a bit different but I think you'll agree with it. Will include it in my next iteration.

Luis

author maintainer
#63 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Jakub Kicinski
Quoted 1 line · Jakub Kicinski

Add infra for creating devlink instances for a device to report

Thanks for doing this series as a PoC, counter to the module_firmware_crash() which I proposed to taint the kernel with a firmware crash flag to the kernel and module.

For those not famliar about devlink:

https://lwn.net/Articles/677967/
https://www.kernel.org/doc/html/latest/networking/devlink/index.html

The github page also is now 404 as Jiri merged that stuff into iproute2:

git://git.kernel.org/pub/scm/network/iproute2/iproute2.git

Quoted 66 lines

fw crashes. This patch expects the devlink instance to be registered at probe time. I belive to be the cleanest. We can also add a devm version of the helpers, so that we don't have to do the clean up. Or we can go even further and register the devlink instance only once error has happened (for the first time, then we can just find out if already registered by traversing the list like we do here).

With the patch applied and a sample driver converted we get:

$ devlink dev
pci/0000:07:00.0

Then monitor for errors:

$ devlink mon health
[health,status] pci/0000:07:00.0:
  reporter fw
    state error error 1 recover 0
[health,status] pci/0000:07:00.0:
  reporter fw
    state error error 2 recover 0

These are the events I triggered on purpose. One can also inspect the health of all devices capable of reporting fw errors:

$ devlink health
pci/0000:07:00.0:
  reporter fw
    state error error 7 recover 0

Obviously drivers may upgrade to the full devlink health API which includes state dump, state dump auto-collect and automatic error recovery control.

---
 include/linux/devlink.h               |  11 +++
 net/core/Makefile                     |   2 +-
 net/core/devlink_simple_fw_reporter.c | 101 ++++++++++++++++++++++++++
 3 files changed, 113 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/devlink.h
 create mode 100644 net/core/devlink_simple_fw_reporter.c
include/linux/devlink.h new file +11 -0
new file mode 100644
index 000000000000..2b73987eefca
@@ -0,0 +1,11 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+#ifndef _LINUX_DEVLINK_H_
+#define _LINUX_DEVLINK_H_
+
+struct device;
+
+void devlink_simple_fw_reporter_prepare(struct device *dev);
+void devlink_simple_fw_reporter_cleanup(struct device *dev);
+void devlink_simple_fw_reporter_report_crash(struct device *dev);
+
+#endif
net/core/Makefile +0 -0
index 3e2c378e5f31..6f1513781c17 100644

net/core/Makefile obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o · line 34

Quoted 5 lines
net/core/Makefile +1 -1
@@ -31,7 +31,7 @@ obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o
 obj-$(CONFIG_BPF_STREAM_PARSER) += sock_map.o
 obj-$(CONFIG_DST_CACHE) += dst_cache.o
 obj-$(CONFIG_HWBM) += hwbm.o
-obj-$(CONFIG_NET_DEVLINK) += devlink.o
+obj-$(CONFIG_NET_DEVLINK) += devlink.o devlink_simple_fw_reporter.o

This was looking super sexy up to here. This is networking specific. We want something generic for *anything* that requests firmware.

I'm afraid this won't work for something generic. I don't think its throw-away work though, the idea to provide a generic interface to dump firmware through netlink might be nice for networking, or other things.

But I have a feeling we'll want something still more generic than this.

So networking may want to be aware that a firmware crash happened as part of this network device health thing, but firmware crashing is a generic thing.

I have now extended my patch set to include uvents and I am more set on that we need the taint now more than ever.

Luis

Quoted 113 lines
net/core/Makefile +0 -0
 obj-$(CONFIG_GRO_CELLS) += gro_cells.o
 obj-$(CONFIG_FAILOVER) += failover.o
 obj-$(CONFIG_BPF_SYSCALL) += bpf_sk_storage.o
net/core/devlink_simple_fw_reporter.c new file +101 -0
new file mode 100644
index 000000000000..48dde9123c3c
@@ -0,0 +1,101 @@
+#include <linux/devlink.h>
+#include <linux/list.h>
+#include <linux/mutex.h>
+#include <net/devlink.h>
+
+struct devlink_simple_fw_reporter {
+	struct list_head list;
+	struct devlink_health_reporter *reporter;
+};
+
+
+static LIST_HEAD(devlink_simple_fw_reporters);
+static DEFINE_MUTEX(devlink_simple_fw_reporters_mutex);
+
+static const struct devlink_health_reporter_ops simple_devlink_health = {
+	.name = "fw",
+};
+
+static const struct devlink_ops simple_devlink_ops = {
+};
+
+static struct devlink_simple_fw_reporter *
+devlink_simple_fw_reporter_find_for_dev(struct device *dev)
+{
+	struct devlink_simple_fw_reporter *simple_devlink, *ret = NULL;
+	struct devlink *devlink;
+
+	mutex_lock(&devlink_simple_fw_reporters_mutex);
+	list_for_each_entry(simple_devlink, &devlink_simple_fw_reporters,
+			    list) {
+		devlink = priv_to_devlink(simple_devlink);
+		if (devlink->dev == dev) {
+			ret = simple_devlink;
+			break;
+		}
+	}
+	mutex_unlock(&devlink_simple_fw_reporters_mutex);
+
+	return ret;
+}
+
+void devlink_simple_fw_reporter_report_crash(struct device *dev)
+{
+	struct devlink_simple_fw_reporter *simple_devlink;
+
+	simple_devlink = devlink_simple_fw_reporter_find_for_dev(dev);
+	if (!simple_devlink)
+		return;
+
+	devlink_health_report(simple_devlink->reporter, "firmware crash", NULL);
+}
+EXPORT_SYMBOL_GPL(devlink_simple_fw_reporter_report_crash);
+
+void devlink_simple_fw_reporter_prepare(struct device *dev)
+{
+	struct devlink_simple_fw_reporter *simple_devlink;
+	struct devlink *devlink;
+
+	devlink = devlink_alloc(&simple_devlink_ops,
+				sizeof(struct devlink_simple_fw_reporter));
+	if (!devlink)
+		return;
+
+	if (devlink_register(devlink, dev))
+		goto err_free;
+
+	simple_devlink = devlink_priv(devlink);
+	simple_devlink->reporter =
+		devlink_health_reporter_create(devlink, &simple_devlink_health,
+					       0, NULL);
+	if (IS_ERR(simple_devlink->reporter))
+		goto err_unregister;
+
+	mutex_lock(&devlink_simple_fw_reporters_mutex);
+	list_add_tail(&simple_devlink->list, &devlink_simple_fw_reporters);
+	mutex_unlock(&devlink_simple_fw_reporters_mutex);
+
+	return;
+
+err_unregister:
+	devlink_unregister(devlink);
+err_free:
+	devlink_free(devlink);
+}
+EXPORT_SYMBOL_GPL(devlink_simple_fw_reporter_prepare);
+
+void devlink_simple_fw_reporter_cleanup(struct device *dev)
+{
+	struct devlink_simple_fw_reporter *simple_devlink;
+	struct devlink *devlink;
+
+	simple_devlink = devlink_simple_fw_reporter_find_for_dev(dev);
+	if (!simple_devlink)
+		return;
+
+	devlink = priv_to_devlink(simple_devlink);
+	devlink_health_reporter_destroy(simple_devlink->reporter);
+	devlink_unregister(devlink);
+	devlink_free(devlink);
+}
+EXPORT_SYMBOL_GPL(devlink_simple_fw_reporter_cleanup);
Signature · 1 line
2.25.4
author maintainer
#42 lore raw

Re: [PATCH v2 12/15] ath10k: use new module_firmware_crashed()

In reply to Emmanuel Grumbach
Quoted 8 lines · Emmanuel Grumbach
Quoted 8 lines · Emmanuel Grumbach
Quoted 2 lines

So I believe we already have this uevent, it is the devcoredump. All we need is to add the unique id.

I think there are a few reasons that devcoredump doesn't satisfy what either Luis or I want.

Quoted 7 lines
Quoted 3 lines

1) it can be disabled entirely [1], for good reasons (e.g., think of non-${CHIP_VENDOR} folks, who can't (and don't want to) do anything with the opaque dumps provided by closed-source firmware)

Ok, if all you're interested into is the information that this event happen (as opposed to report a bug and providing the data), then I agree.

I've now hit again a firmware crash with ath10k with the latest firwmare and kernel and the *only* thing that helped recovery was a full reboot, so that is a crystal clear case that this needs to taint the kernel, and yes we do want to inform users too, so I've just added uevent support for a few panic / taint events in the kernel now and rolled into my series. I'll run some final tests and then post this as a follow up.

devlink didn't cut it, its networking specific.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#25 lore raw

Re: [PATCH v2 10/15] soc: qcom: ipa: use new module_firmware_crashed()

In reply to Alex Elder
Quoted 11 lines · Alex Elder
Quoted 7 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

I don't fully understand what this is meant to do, so I can't fully assess whether it's the right thing to do.

It is meant to taint the kernel to ensure it is clear that something critically bad has happened with the device firmware, it crashed, and recovery may or may not happen, we are not 100% certain.

Quoted 4 lines

But in this particular place in the IPA code, the *modem* has crashed. And the IPA driver is not responsible for modem firmware, remoteproc is.

Oi vei. So the device it depends on has crashed.

Quoted 3 lines

The IPA driver *can* be responsible for loading some other firmware, but even in that case, it only happens on initial boot, and it's basically assumed to never crash.

OK is this an issue which we can recover from? If for the slightest bit this can affect users it is something we should inform them over.

This patch set is missing uevents for these issues, but I just added support for this.

Quoted 2 lines

So regardless of whether this module_firmware_crashed() call is appropriate in some places, I believe it should not be used here.

OK thanks. Can the user be affected by this crash? If so how? Can we recover ? Is that always guaranteed?

Luis

maintainer
#64 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Luis Chamberlain
Quoted 5 lines · On Fri, 22 May 2020 05:20:46 +0000 Luis Chamberlain wrote:
Quoted 5 lines
net/core/Makefile +0 -0
index 3e2c378e5f31..6f1513781c17 100644
@@ -31,7 +31,7 @@ obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o
Quoted 8 lines

net/core/Makefile obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o · line 34

Quoted 5 lines
net/core/Makefile +1 -1
 obj-$(CONFIG_BPF_STREAM_PARSER) += sock_map.o
 obj-$(CONFIG_DST_CACHE) += dst_cache.o
 obj-$(CONFIG_HWBM) += hwbm.o
-obj-$(CONFIG_NET_DEVLINK) += devlink.o
+obj-$(CONFIG_NET_DEVLINK) += devlink.o devlink_simple_fw_reporter.o  

This was looking super sexy up to here. This is networking specific. We want something generic for *anything* that requests firmware.

You can't be serious. It's network specific because of how the Kconfig is named?

Working for a company operating large data centers I would strongly prefer if we didn't have ten different ways of reporting firmware problems in the fleet.

Quoted 6 lines

I'm afraid this won't work for something generic. I don't think its throw-away work though, the idea to provide a generic interface to dump firmware through netlink might be nice for networking, or other things.

But I have a feeling we'll want something still more generic than this.

Please be specific. Saying generic a lot is not helpful. The code (as you can see in this patch) is in no way network specific. Or are you saying there are machines out there running without netlink sockets?

Quoted 6 lines

So networking may want to be aware that a firmware crash happened as part of this network device health thing, but firmware crashing is a generic thing.

I have now extended my patch set to include uvents and I am more set on that we need the taint now more than ever.

Please expect my nack if you're trying to add this to networking drivers.

The irony is you have a problem with a networking device and all the devices your initial set touched are networking. Two of the drivers you touched either have or will soon have devlink health reporters implemented.

#65 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

Quoted 12 lines · Jakub Kicinski
Quoted 11 lines

net/core/Makefile obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o · line 34

Quoted 8 lines
net/core/Makefile +1 -1
@@ -31,7 +31,7 @@ obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o
 obj-$(CONFIG_BPF_STREAM_PARSER) += sock_map.o
 obj-$(CONFIG_DST_CACHE) += dst_cache.o
 obj-$(CONFIG_HWBM) += hwbm.o
-obj-$(CONFIG_NET_DEVLINK) += devlink.o
+obj-$(CONFIG_NET_DEVLINK) += devlink.o devlink_simple_fw_reporter.o  

This was looking super sexy up to here. This is networking specific. We want something generic for *anything* that requests firmware.

Quoted 2 lines

You can't be serious. It's network specific because of how the Kconfig is named?

Wait, yeah, what?

Quoted 3 lines

Working for a company operating large data centers I would strongly prefer if we didn't have ten different ways of reporting firmware problems in the fleet.

Agree. I don't actually operate anything, but still ...

Thinking about this - maybe there's a way to still combine devcoredump and devlink somehow?

Or (optionally) make devlink trigger devcoredump while userspace migrates?

Quoted 6 lines
Quoted 6 lines

So networking may want to be aware that a firmware crash happened as part of this network device health thing, but firmware crashing is a generic thing.

I have now extended my patch set to include uvents and I am more set on that we need the taint now more than ever.

FWIW, I still completely disagree on that taint. You (Luis) obviously have been running into a bug in that driver, I doubt the firmware actually managed to wedge the hardware.

But even if it did, that's still not really a kernel taint. The kernel itself isn't in any way affected by this.

Yes, the system is in a weird state now. But that's *not* equivalent to "kernel tainted".

Quoted 4 lines

The irony is you have a problem with a networking device and all the devices your initial set touched are networking. Two of the drivers you touched either have or will soon have devlink health reporters implemented.

Like I said above, do you think it'd be feasible to make a devcoredump out of devlink health reports? And can the report be in a way that we control the file format, or are there limits? I guess I should read the code to find out, but I figure you probably just know. But feel free to tell me to read it :)

The reason I'm asking is that it's starting to sound like we really ought to be implementing devlink, but we've got a bunch of infrastructure that uses the devcoredump, and it'll take time (significantly so) to change all that...

johannes

#26 lore raw

Re: [PATCH v2 10/15] soc: qcom: ipa: use new module_firmware_crashed()

In reply to Luis Chamberlain
Quoted 17 lines · Luis Chamberlain
Quoted 11 lines · Alex Elder
Quoted 7 lines · Luis Chamberlain

This makes use of the new module_firmware_crashed() to help annotate when firmware for device drivers crash. When firmware crashes devices can sometimes become unresponsive, and recovery sometimes requires a driver unload / reload and in the worst cases a reboot.

Using a taint flag allows us to annotate when this happens clearly.

I don't fully understand what this is meant to do, so I can't fully assess whether it's the right thing to do.

It is meant to taint the kernel to ensure it is clear that something critically bad has happened with the device firmware, it crashed, and recovery may or may not happen, we are not 100% certain.

Quoted 1 line
Quoted 5 lines
Quoted 3 lines

But in this particular place in the IPA code, the *modem* has crashed. And the IPA driver is not responsible for modem firmware, remoteproc is.

Oi vei. So the device it depends on has crashed.

Yes, more or less. It could be considered a little more nuanced than even that, but I won't get into it here.

Quoted 6 lines
Quoted 3 lines

The IPA driver *can* be responsible for loading some other firmware, but even in that case, it only happens on initial boot, and it's basically assumed to never crash.

OK is this an issue which we can recover from? If for the slightest bit this can affect users it is something we should inform them over.

If the IPA driver successfully loads this firmware, it should be assumed to never crash. So in that respect, there is no recovery required.

Again, the modem (with which the IPA hardware and driver interacts) can crash, or it can be shut down intentionally. And in either case it can recover, automatically or manually. But all of that (and the modem's firmware loading) is the responsibility of the remoteproc subsystem, not IPA.

Quoted 8 lines

This patch set is missing uevents for these issues, but I just added support for this.

Quoted 2 lines

So regardless of whether this module_firmware_crashed() call is appropriate in some places, I believe it should not be used here.

OK thanks. Can the user be affected by this crash? If so how? Can we recover ? Is that always guaranteed?

We can't guarantee anything about recovering from a crash of an independent entity. But by design, recovery from a modem crash is possible and is supposed to leave Linux in a consistent state. A modem crash is likely to be observable to the user.

I'll repeat that I don't believe the new call you inserted in the IPA driver belongs there.

-Alex

Quoted 3 lines

Luis

author maintainer
#73 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Jakub Kicinski
Quoted 15 lines · Jakub Kicinski
Quoted 5 lines · On Fri, 22 May 2020 05:20:46 +0000 Luis Chamberlain wrote:
Quoted 5 lines
net/core/Makefile +0 -0
index 3e2c378e5f31..6f1513781c17 100644
@@ -31,7 +31,7 @@ obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o
Quoted 8 lines

net/core/Makefile obj-$(CONFIG_LWTUNNEL_BPF) += lwt_bpf.o · line 34

Quoted 5 lines
net/core/Makefile +1 -1
 obj-$(CONFIG_BPF_STREAM_PARSER) += sock_map.o
 obj-$(CONFIG_DST_CACHE) += dst_cache.o
 obj-$(CONFIG_HWBM) += hwbm.o
-obj-$(CONFIG_NET_DEVLINK) += devlink.o
+obj-$(CONFIG_NET_DEVLINK) += devlink.o devlink_simple_fw_reporter.o  

This was looking super sexy up to here. This is networking specific. We want something generic for *anything* that requests firmware.

Quoted 2 lines

You can't be serious. It's network specific because of how the Kconfig is named?

Kconfig? What has that to do with anything? The issue I have is that the solution I am looking for is for it to be agnostic to the subsystem. I have found similar firmware crashes on gpu, media, scsci.

Quoted 3 lines

Working for a company operating large data centers I would strongly prefer if we didn't have ten different ways of reporting firmware problems in the fleet.

Indeed.

Quoted 10 lines
Quoted 6 lines

I'm afraid this won't work for something generic. I don't think its throw-away work though, the idea to provide a generic interface to dump firmware through netlink might be nice for networking, or other things.

But I have a feeling we'll want something still more generic than this.

Please be specific. Saying generic a lot is not helpful. The code (as you can see in this patch) is in no way network specific. Or are you saying there are machines out there running without netlink sockets?

No, I am saying I want something to work with any struct device.

Quoted 9 lines
Quoted 6 lines

So networking may want to be aware that a firmware crash happened as part of this network device health thing, but firmware crashing is a generic thing.

I have now extended my patch set to include uvents and I am more set on that we need the taint now more than ever.

Please expect my nack if you're trying to add this to networking drivers.

The uevent mechanism is not for networking.

The taint however is, and I'd like to undertand how it is you do not see that an undesirable requirement for a reboot is a clear case for a taint.

Quoted 4 lines

The irony is you have a problem with a networking device and all the devices your initial set touched are networking. Two of the drivers you touched either have or will soon have devlink health reporters implemented.

That is all great, and I don't think its a bad idea to add infrastructure / extend it to get more information about a firmware crash dump. However, suggesting that devlink is the only solution we need in the kernel without considering other subsystems is what I am suggesting doesn't suit my needs. Networking was just the first subsystem I am taclking now but I have patches where similar situations happen across the kernel.

Luis

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#66 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Johannes Berg
Quoted 3 lines · Johannes Berg

FWIW, I still completely disagree on that taint. You (Luis) obviously have been running into a bug in that driver, I doubt the firmware actually managed to wedge the hardware.

This hasn't happened just once, its happed many times sporadically now, once a week or two weeks I'd say. And the system isn't being moved around.

Quoted 2 lines

But even if it did, that's still not really a kernel taint. The kernel itself isn't in any way affected by this.

Of course it is, a full reboot is required.

Quoted 2 lines

Yes, the system is in a weird state now. But that's *not* equivalent to "kernel tainted".

Requiring a full reboot is a dire situation to be in, and loosing connectivity to the point this is not recoverable likewise.

You guys are making out a taint to be the end of the world. We have a taint even for a kernel warning, and as others have mentioned mac80211 already produces these.

What exactly is the opposition to a taint to clarify that a device firmware has crashed and your system requires a full reboot?

Luis

author maintainer
#27 lore raw

Re: [PATCH v2 10/15] soc: qcom: ipa: use new module_firmware_crashed()

In reply to Alex Elder
Quoted 9 lines · Alex Elder
Quoted 2 lines · Luis Chamberlain

OK thanks. Can the user be affected by this crash? If so how? Can we recover ? Is that always guaranteed?

We can't guarantee anything about recovering from a crash of an independent entity. But by design, recovery from a modem crash is possible and is supposed to leave Linux in a consistent state. A modem crash is likely to be observable to the user.

Thanks this helps, I'll drop this patch!

Luis

#67 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Luis Chamberlain
Quoted 18 lines · Luis Chamberlain
Quoted 3 lines · Johannes Berg

FWIW, I still completely disagree on that taint. You (Luis) obviously have been running into a bug in that driver, I doubt the firmware actually managed to wedge the hardware.

This hasn't happened just once, its happed many times sporadically now, once a week or two weeks I'd say. And the system isn't being moved around.

Quoted 2 lines

But even if it did, that's still not really a kernel taint. The kernel itself isn't in any way affected by this.

Of course it is, a full reboot is required.

Quoted 2 lines

Yes, the system is in a weird state now. But that's *not* equivalent to "kernel tainted".

Quoted 7 lines

Requiring a full reboot is a dire situation to be in, and loosing connectivity to the point this is not recoverable likewise.

You guys are making out a taint to be the end of the world. We have a taint even for a kernel warning, and as others have mentioned mac80211 already produces these.

I had to go RTFM re: kernel taints because it has been a very long time since I looked at them. It had always seemed to me that most were caused by "kernel-unfriendly" user actions. The most famous of course is loading proprietary modules, out-of-tree modules, forced module loads, etc... Honestly, I had forgotten the large variety of uses of the taint flags. For anyone who hasn't looked at taints recently, I
recommend: https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html

In light of this I don't object to setting a taint on this anymore. I'm a little uneasy, but I've softened on it now, and now I feel it depends on implementation.

Specifically, I don't think we should set a taint flag when a driver easily handles a routine firmware crash and is confident that things have come up just fine again. In other words, triggering the taint in every driver module where it spits out a log comment that it had a firmware crash and had to recover seems too much. Sure, firmware shouldn't crash, sure it should be open source so we can fix it, whatever... those sort of wishful comments simply ignore reality and our ability to affect effective change. A lot of WiFi firmware crashes and for well-known cases the drivers handle them well. And in some cases, not so well and that should be a place the driver should detect and thus raise a red flag. If a WiFi firmware crash can bring down the kernel, there's either a major driver bug or some very funky hardware crap going on. That sort of thing we should be able to detect, mark with a taint (or something), and fix if within our sphere of influence. I guess what it comes down to me is how aggressive we are about setting the flag.

I would like there to be a single solution, or a minimized set depending on what makes sense for the requirements. I haven't had time to look into the alternatives mentioned here so I don't have an informed opinion about the solution. I do think Luis is trying to solve a real problem though. Can we look at this from the point of view of what are the requirements? What is it we're trying to solve?

I _think_ that the goal of Luis's original proposal is to report up to the user, at some future point when the user is interested (because something super drastic just occured, but long after the fw crash), that there was a firmware crash without the user having to grep through all logs on the machine. And then if the user sees that flag and suspects it, then they can bother to find it in the logs or do more drastic debugging steps like finding the fw crash in the log and pulling firmware crash dumps, etc.

I think the various alternate solutions are great but perhaps solving a superset of features (like adding in user-space notifications etc)? Perhaps different people on these related threads are trying to solve different problems?

- Steve

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author maintainer
#68 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

Quoted 8 lines · Steve deRosier

Specifically, I don't think we should set a taint flag when a driver easily handles a routine firmware crash and is confident that things have come up just fine again. In other words, triggering the taint in every driver module where it spits out a log comment that it had a firmware crash and had to recover seems too much. Sure, firmware shouldn't crash, sure it should be open source so we can fix it, whatever... those sort of wishful comments simply ignore reality and our ability to affect effective change. A lot of WiFi firmware crashes

Quoted 8 lines

and for well-known cases the drivers handle them well. And in some cases, not so well and that should be a place the driver should detect and thus raise a red flag. If a WiFi firmware crash can bring down the kernel, there's either a major driver bug or some very funky hardware crap going on. That sort of thing we should be able to detect, mark with a taint (or something), and fix if within our sphere of influence. I guess what it comes down to me is how aggressive we are about setting the flag.

Exactly the crux of the issue.

I hope that by now we should all be in agreement that at least a firmware crash requiring a reboot is something we should record and inform the user of. A taint seems like a reasonable standard practice for these sorts of things.

Quoted 7 lines

I would like there to be a single solution, or a minimized set depending on what makes sense for the requirements. I haven't had time to look into the alternatives mentioned here so I don't have an informed opinion about the solution. I do think Luis is trying to solve a real problem though. Can we look at this from the point of view of what are the requirements? What is it we're trying to solve?

Quoted 8 lines

I _think_ that the goal of Luis's original proposal is to report up to the user, at some future point when the user is interested (because something super drastic just occured, but long after the fw crash), that there was a firmware crash without the user having to grep through all logs on the machine. And then if the user sees that flag and suspects it, then they can bother to find it in the logs or do more drastic debugging steps like finding the fw crash in the log and pulling firmware crash dumps, etc.

Yes, that's exactly it. Not all users are clueful to inspect logs. I now have a generic uevent mechanism drafted which sends a uevent for *any* taint. So that is, it does not even depend on this series. But it accomplishes the goal of informing the user of taints.

Quoted 4 lines

I think the various alternate solutions are great but perhaps solving a superset of features (like adding in user-space notifications etc)? Perhaps different people on these related threads are trying to solve different problems?

The uevent mechanism I implemented (but not yet posted for review) at least sends out a smoke signal. I think that if each subsystem wants to expand on this with dumping facilities that is great too!

Luis

#69 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Steve deRosier
Quoted 14 lines · Steve deRosier

On Fri, May 22, 2020 at 2:51 PM Luis Chamberlain <[email protected]> wrote:

I had to go RTFM re: kernel taints because it has been a very long time since I looked at them. It had always seemed to me that most were caused by "kernel-unfriendly" user actions. The most famous of course is loading proprietary modules, out-of-tree modules, forced module loads, etc... Honestly, I had forgotten the large variety of uses of the taint flags. For anyone who hasn't looked at taints recently, I
recommend: https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html

In light of this I don't object to setting a taint on this anymore. I'm a little uneasy, but I've softened on it now, and now I feel it depends on implementation.

Quoted 7 lines

Specifically, I don't think we should set a taint flag when a driver easily handles a routine firmware crash and is confident that things have come up just fine again. In other words, triggering the taint in every driver module where it spits out a log comment that it had a firmware crash and had to recover seems too much. Sure, firmware shouldn't crash, sure it should be open source so we can fix it, whatever...

While it may sound idealistic the firmware for the end-user, and even for mere kernel developer like me, is a complete blackbox which has more access than root user in the kernel. We have tons of firmwares and each of them potentially dangerous beast. As a user I really care about my data and privacy (hacker can oops a firmware in order to set a specific vector attack). So, tainting kernel is _a least_ we can do there, the strict rules would be to reboot immediately.

Quoted 2 lines

those sort of wishful comments simply ignore reality and our ability to affect effective change.

We can encourage users not to buy cheap crap for the starter.

Quoted 9 lines

A lot of WiFi firmware crashes
and for well-known cases the drivers handle them well. And in some cases, not so well and that should be a place the driver should detect and thus raise a red flag. If a WiFi firmware crash can bring down the kernel, there's either a major driver bug or some very funky hardware crap going on. That sort of thing we should be able to detect, mark with a taint (or something), and fix if within our sphere of influence. I guess what it comes down to me is how aggressive we are about setting the flag.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#70 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

Quoted 30 lines · Andy Shevchenko
Quoted 14 lines · Steve deRosier

On Fri, May 22, 2020 at 2:51 PM Luis Chamberlain <[email protected]> wrote:

I had to go RTFM re: kernel taints because it has been a very long time since I looked at them. It had always seemed to me that most were caused by "kernel-unfriendly" user actions. The most famous of course is loading proprietary modules, out-of-tree modules, forced module loads, etc... Honestly, I had forgotten the large variety of uses of the taint flags. For anyone who hasn't looked at taints recently, I
recommend: https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html

In light of this I don't object to setting a taint on this anymore. I'm a little uneasy, but I've softened on it now, and now I feel it depends on implementation.

Quoted 7 lines

Specifically, I don't think we should set a taint flag when a driver easily handles a routine firmware crash and is confident that things have come up just fine again. In other words, triggering the taint in every driver module where it spits out a log comment that it had a firmware crash and had to recover seems too much. Sure, firmware shouldn't crash, sure it should be open source so we can fix it, whatever...

While it may sound idealistic the firmware for the end-user, and even for mere kernel developer like me, is a complete blackbox which has more access than root user in the kernel. We have tons of firmwares and each of them potentially dangerous beast. As a user I really care about my data and privacy (hacker can oops a firmware in order to set a specific vector attack). So, tainting kernel is _a least_ we can do there, the strict rules would be to reboot immediately.

Quoted 4 lines
Quoted 2 lines

those sort of wishful comments simply ignore reality and our ability to affect effective change.

We can encourage users not to buy cheap crap for the starter.

There is no stable wifi firmware for any price.

There is also no obvious feedback from even name-brand NICs like ath10k or AX200 when you report a crash.

That said, at least in my experience with ath10k-ct, the OS normally recovers fine from firmware crashes. ath10k already reports full crash reports on udev, so easy for user-space to notice and report bug reports upstream if it cares to. Probably other NICs do the same, and if not, they certainly could.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
maintainer
#71 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

In reply to Johannes Berg
Quoted 11 lines · On Fri, 22 May 2020 22:46:07 +0200 Johannes Berg wrote:
Quoted 4 lines

The irony is you have a problem with a networking device and all the devices your initial set touched are networking. Two of the drivers you touched either have or will soon have devlink health reporters implemented.

Like I said above, do you think it'd be feasible to make a devcoredump out of devlink health reports? And can the report be in a way that we control the file format, or are there limits? I guess I should read the code to find out, but I figure you probably just know. But feel free to tell me to read it :)

Quoted 4 lines

The reason I'm asking is that it's starting to sound like we really ought to be implementing devlink, but we've got a bunch of infrastructure that uses the devcoredump, and it'll take time (significantly so) to change all that...

In devlink world pure FW core dumps are exposed by devlink regions. An API allowing reading device memory, registers, etc., but also creating dumps of memory regions when things go wrong. It should be a fairly straightforward migration.

Devlink health is more targeted, the dump is supposed to contain only relevant information, selected and formatted by the driver. When device misbehaves driver reads the relevant registers and FW state and creates a formatted state dump. I believe each element of the dump must fit into a netlink message (but there may be multiple elements, see devlink_fmsg_prepare_skb()).

We should be able to convert dl-regions dumps and dl-health dumps into devcoredumps, but since health reporters are supposed to be more targeted there's usually multiple of them per device.

Conversely devcoredumps can be trivially exposed as dl-region dumps, but I believe dl-health would require driver changes to format the information appropriately.

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#72 lore raw

Re: [RFC 1/2] devlink: add simple fw crash helpers

Hi,

Sorry ... long delay.

Quoted 9 lines
Quoted 4 lines

The reason I'm asking is that it's starting to sound like we really ought to be implementing devlink, but we've got a bunch of infrastructure that uses the devcoredump, and it'll take time (significantly so) to change all that...

In devlink world pure FW core dumps are exposed by devlink regions. An API allowing reading device memory, registers, etc., but also creating dumps of memory regions when things go wrong. It should be a fairly straightforward migration.

Right. We also have regions (various memory pieces, registers, ...).

One issue might be that for devlink we wouldn't want to expose these as a single blob, I guess, but for devcoredump we already have a custom format to glue all the things together. Since it seems unlikely that anyone else would want to use the *iwlwifi* format to glue things together, we'd have to do something there.

But perhaps that could be a matter of providing a "glue things into a devcoredump" function that would have a reasonable default but could be overridden by the driver for those migration cases.

Quoted 6 lines

Devlink health is more targeted, the dump is supposed to contain only relevant information, selected and formatted by the driver. When device misbehaves driver reads the relevant registers and FW state and creates a formatted state dump. I believe each element of the dump must fit into a netlink message (but there may be multiple elements, see devlink_fmsg_prepare_skb()).

That wouldn't help for our big memory dumps, but OK.

Quoted 3 lines

We should be able to convert dl-regions dumps and dl-health dumps into devcoredumps, but since health reporters are supposed to be more targeted there's usually multiple of them per device.

Right.

Quoted 3 lines

Conversely devcoredumps can be trivially exposed as dl-region dumps, but I believe dl-health would require driver changes to format the information appropriately.

Agree.

Anyway, thanks. I'll put it on my list of things to look at ... not too hopeful that will be soon, given how long it even took me to get back to this email :)

johannes