Skip to contentSkip to conversation

bnxt_en: Error recovery improvements.

PATCH Applied Michael Chan 4 participants 19 messages Jan 25, 2021 – Jan 26, 2021

Subsystems (3)
lore
Share & tools
lore

Michael Chan

author maintainer
#0 lore raw

cover bnxt_en: Error recovery improvements.

This series contains a number of improvements in the area of error recovery. Most error recovery scenarios are tightly coordinated with the firmware. A number of patches add retry logic to establish connection with the firmware if there are indications that the firmware is still alive and will likely transition back to the normal state. Some patches speed up the recovery process and make it more reliable. There are some cleanup patches as well.

Edwin Peer (3):
bnxt_en: handle CRASH_NO_MASTER during bnxt_open()
bnxt_en: log firmware debug notifications
bnxt_en: attempt to reinitialize after aborted reset

Michael Chan (9):
bnxt_en: Update firmware interface to 1.10.2.11.
bnxt_en: Define macros for the various health register states.
bnxt_en: Retry sending the first message to firmware if it is under
reset.
bnxt_en: Add bnxt_fw_reset_timeout() helper.
bnxt_en: Add a new BNXT_STATE_NAPI_DISABLED flag to keep track of NAPI
state.
bnxt_en: Modify bnxt_disable_int_sync() to be called more than once.
bnxt_en: Improve firmware fatal error shutdown sequence.
bnxt_en: Consolidate firmware reset event logging.
bnxt_en: Do not process completion entries after fatal condition
detected.

Vasundhara Volam (3):
bnxt_en: Move reading VPD info after successful handshake with fw.
bnxt_en: Add an upper bound for all firmware command timeouts.
bnxt_en: Retry open if firmware is in reset.

 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 228 ++++++++++++----
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  22 ++
 .../net/ethernet/broadcom/bnxt/bnxt_devlink.c |   7 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 249 ++++++++++++++----
 4 files changed, 393 insertions(+), 113 deletions(-)
Signature · 1 line
2.18.1
author maintainer
#1 lore raw

PATCH 1/15 bnxt_en: Update firmware interface to 1.10.2.11.

Updates to backing store APIs, QoS profiles, and push buffer initial index support.

Since the new HWRM_FUNC_BACKING_STORE_CFG message size has increased, we need to add some compat. logic to fall back to the smaller legacy size if firmware cannot accept the larger message size. The new fields added to the structure are not used yet.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     |   5 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |   2 +
 drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 249 ++++++++++++++----
 3 files changed, 203 insertions(+), 53 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +4 -1
index d31a5ad7522a..5daef6801512 100644
@@ -6845,6 +6845,7 @@ static int bnxt_hwrm_func_backing_store_cfg(struct bnxt *bp, u32 enables)
 	struct hwrm_func_backing_store_cfg_input req = {0};
 	struct bnxt_ctx_mem_info *ctx = bp->ctx;
 	struct bnxt_ctx_pg_info *ctx_pg;
+	u32 req_len = sizeof(req);
 	__le32 *num_entries;
 	__le64 *pg_dir;
 	u32 flags = 0;
@@ -6855,6 +6856,8 @@ static int bnxt_hwrm_func_backing_store_cfg(struct bnxt *bp, u32 enables)
 	if (!ctx)
 		return 0;
 
+	if (req_len > bp->hwrm_max_ext_req_len)
+		req_len = BNXT_BACKING_STORE_CFG_LEGACY_LEN;
 	bnxt_hwrm_cmd_hdr_init(bp, &req, HWRM_FUNC_BACKING_STORE_CFG, -1, -1);
 	req.enables = cpu_to_le32(enables);
 
@@ -6938,7 +6941,7 @@ static int bnxt_hwrm_func_backing_store_cfg(struct bnxt *bp, u32 enables)
 		bnxt_hwrm_set_pg_attr(&ctx_pg->ring_mem, pg_attr, pg_dir);
 	}
 	req.flags = cpu_to_le32(flags);
-	return hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+	return hwrm_send_message(bp, &req, req_len, HWRM_CMD_TIMEOUT);
 }
 
 static int bnxt_alloc_ctx_mem_blk(struct bnxt *bp,
drivers/net/ethernet/broadcom/bnxt/bnxt.h +2 -0
index 51996c85547e..d68065367cf2 100644
@@ -1441,6 +1441,8 @@ struct bnxt_ctx_pg_info {
 #define BNXT_MAX_TQM_RINGS		\
 	(BNXT_MAX_TQM_SP_RINGS + BNXT_MAX_TQM_FP_RINGS)
 
+#define BNXT_BACKING_STORE_CFG_LEGACY_LEN	256
+
 struct bnxt_ctx_mem_info {
 	u32	qp_max_entries;
 	u16	qp_min_qp1_entries;
drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h +197 -52

439 lines not on this page · raw message

Signature · 1 line
2.18.1
author maintainer
#2 lore raw

PATCH 2/15 bnxt_en: Define macros for the various health register states.

In reply to Michael Chan

Define macros to check for the various states in the lower 16 bits of the health register. Replace the C code that checks for these values with the newly defined macros.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.h         | 10 ++++++++++
 drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c |  7 +++----
 2 files changed, 13 insertions(+), 4 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.h +10 -0
index d68065367cf2..a1dd80a0fcf6 100644
@@ -1534,9 +1534,19 @@ struct bnxt_fw_reporter_ctx {
 #define BNXT_FW_HEALTH_WIN_OFF(reg)	(BNXT_FW_HEALTH_WIN_BASE +	\
 					 ((reg) & BNXT_GRC_OFFSET_MASK))
 
+#define BNXT_FW_STATUS_HEALTH_MSK	0xffff
 #define BNXT_FW_STATUS_HEALTHY		0x8000
 #define BNXT_FW_STATUS_SHUTDOWN		0x100000
 
+#define BNXT_FW_IS_HEALTHY(sts)		(((sts) & BNXT_FW_STATUS_HEALTH_MSK) ==\
+					 BNXT_FW_STATUS_HEALTHY)
+
+#define BNXT_FW_IS_BOOTING(sts)		(((sts) & BNXT_FW_STATUS_HEALTH_MSK) < \
+					 BNXT_FW_STATUS_HEALTHY)
+
+#define BNXT_FW_IS_ERR(sts)		(((sts) & BNXT_FW_STATUS_HEALTH_MSK) > \
+					 BNXT_FW_STATUS_HEALTHY)
+
 struct bnxt {
 	void __iomem		*bar0;
 	void __iomem		*bar1;
drivers/net/ethernet/broadcom/bnxt/bnxt_devlink.c +3 -4
index 6b7b69ed62db..90a31b4a3020 100644
@@ -44,21 +44,20 @@ static int bnxt_fw_reporter_diagnose(struct devlink_health_reporter *reporter,
 				     struct netlink_ext_ack *extack)
 {
 	struct bnxt *bp = devlink_health_reporter_priv(reporter);
-	u32 val, health_status;
+	u32 val;
 	int rc;
 
 	if (test_bit(BNXT_STATE_IN_FW_RESET, &bp->state))
 		return 0;
 
 	val = bnxt_fw_health_readl(bp, BNXT_FW_HEALTH_REG);
-	health_status = val & 0xffff;
 
-	if (health_status < BNXT_FW_STATUS_HEALTHY) {
+	if (BNXT_FW_IS_BOOTING(val)) {
 		rc = devlink_fmsg_string_pair_put(fmsg, "Description",
 						  "Not yet completed initialization");
 		if (rc)
 			return rc;
-	} else if (health_status > BNXT_FW_STATUS_HEALTHY) {
+	} else if (BNXT_FW_IS_ERR(val)) {
 		rc = devlink_fmsg_string_pair_put(fmsg, "Description",
 						  "Encountered fatal error and cannot recover");
 		if (rc)
Signature · 1 line
2.18.1
author maintainer
#3 lore raw

PATCH 3/15 bnxt_en: handle CRASH_NO_MASTER during bnxt_open()

In reply to Michael Chan

From: Edwin Peer <[email protected]>

Add missing support for handling NO_MASTER crashes while ports are administratively down (ifdown). On some SoC platforms, the driver needs to assist the firmware to recover from a crash via OP-TEE. This is performed in a similar fashion to what is done during driver probe.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 67 +++++++++++++----------
 1 file changed, 39 insertions(+), 28 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +39 -28
index 5daef6801512..c091a1023188 100644
@@ -9337,6 +9337,37 @@ static int bnxt_hwrm_shutdown_link(struct bnxt *bp)
 
 static int bnxt_fw_init_one(struct bnxt *bp);
 
+static int bnxt_fw_reset_via_optee(struct bnxt *bp)
+{
+#ifdef CONFIG_TEE_BNXT_FW
+	int rc = tee_bnxt_fw_load();
+
+	if (rc)
+		netdev_err(bp->dev, "Failed FW reset via OP-TEE, rc=%d\n", rc);
+
+	return rc;
+#else
+	netdev_err(bp->dev, "OP-TEE not supported\n");
+	return -ENODEV;
+#endif
+}
+
+static int bnxt_try_recover_fw(struct bnxt *bp)
+{
+	if (bp->fw_health && bp->fw_health->status_reliable) {
+		u32 sts = bnxt_fw_health_readl(bp, BNXT_FW_HEALTH_REG);
+
+		netdev_err(bp->dev, "Firmware not responding, status: 0x%x\n",
+			   sts);
+		if (sts & FW_STATUS_REG_CRASHED_NO_MASTER) {
+			netdev_warn(bp->dev, "Firmware recover via OP-TEE requested\n");
+			return bnxt_fw_reset_via_optee(bp);
+		}
+	}
+
+	return -ENODEV;
+}
+
 static int bnxt_hwrm_if_change(struct bnxt *bp, bool up)
 {
 	struct hwrm_func_drv_if_change_output *resp = bp->hwrm_cmd_resp_addr;
@@ -9356,6 +9387,10 @@ static int bnxt_hwrm_if_change(struct bnxt *bp, bool up)
 	if (!rc)
 		flags = le32_to_cpu(resp->flags);
 	mutex_unlock(&bp->hwrm_cmd_lock);
+	if (rc && up) {
+		rc = bnxt_try_recover_fw(bp);
+		fw_reset = true;
+	}
 	if (rc)
 		return rc;
 
@@ -11183,21 +11218,6 @@ static void bnxt_init_dflt_coal(struct bnxt *bp)
 	bp->stats_coal_ticks = BNXT_DEF_STATS_COAL_TICKS;
 }
 
-static int bnxt_fw_reset_via_optee(struct bnxt *bp)
-{
-#ifdef CONFIG_TEE_BNXT_FW
-	int rc = tee_bnxt_fw_load();
-
-	if (rc)
-		netdev_err(bp->dev, "Failed FW reset via OP-TEE, rc=%d\n", rc);
-
-	return rc;
-#else
-	netdev_err(bp->dev, "OP-TEE not supported\n");
-	return -ENODEV;
-#endif
-}
-
 static int bnxt_fw_init_one_p1(struct bnxt *bp)
 {
 	int rc;
@@ -11206,19 +11226,10 @@ static int bnxt_fw_init_one_p1(struct bnxt *bp)
 	rc = bnxt_hwrm_ver_get(bp);
 	bnxt_try_map_fw_health_reg(bp);
 	if (rc) {
-		if (bp->fw_health && bp->fw_health->status_reliable) {
-			u32 sts = bnxt_fw_health_readl(bp, BNXT_FW_HEALTH_REG);
-
-			netdev_err(bp->dev,
-				   "Firmware not responding, status: 0x%x\n",
-				   sts);
-			if (sts & FW_STATUS_REG_CRASHED_NO_MASTER) {
-				netdev_warn(bp->dev, "Firmware recover via OP-TEE requested\n");
-				rc = bnxt_fw_reset_via_optee(bp);
-				if (!rc)
-					rc = bnxt_hwrm_ver_get(bp);
-			}
-		}
+		rc = bnxt_try_recover_fw(bp);
+		if (rc)
+			return rc;
+		rc = bnxt_hwrm_ver_get(bp);
 		if (rc)
 			return rc;
 	}
Signature · 1 line
2.18.1
author maintainer
#4 lore raw

PATCH 4/15 bnxt_en: Retry sending the first message to firmware if it is under reset.

In reply to Michael Chan

The first HWRM_VER_GET message to firmware during probe may timeout if firmware is under reset. This can happen during hot-plug for example. On P5 and newer chips, we can check if firmware is in the boot stage by reading a status register. Retry 5 times if the status register shows that firmware is not ready and not in error state.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 42 +++++++++++++++++++----
 drivers/net/ethernet/broadcom/bnxt/bnxt.h |  7 ++++
 2 files changed, 42 insertions(+), 7 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +35 -7
index c091a1023188..c460dd796c1c 100644
@@ -7441,9 +7441,22 @@ static void bnxt_try_map_fw_health_reg(struct bnxt *bp)
 
 	sig = readl(hs + offsetof(struct hcomm_status, sig_ver));
 	if ((sig & HCOMM_STATUS_SIGNATURE_MASK) != HCOMM_STATUS_SIGNATURE_VAL) {
-		if (bp->fw_health)
-			bp->fw_health->status_reliable = false;
-		return;
+		if (!bp->chip_num) {
+			__bnxt_map_fw_health_reg(bp, BNXT_GRC_REG_BASE);
+			bp->chip_num = readl(bp->bar0 +
+					     BNXT_FW_HEALTH_WIN_BASE +
+					     BNXT_GRC_REG_CHIP_NUM);
+		}
+		if (!BNXT_CHIP_P5(bp)) {
+			if (bp->fw_health)
+				bp->fw_health->status_reliable = false;
+			return;
+		}
+		status_loc = BNXT_GRC_REG_STATUS_P5 |
+			     BNXT_FW_HEALTH_REG_TYPE_BAR0;
+	} else {
+		status_loc = readl(hs + offsetof(struct hcomm_status,
+						 fw_status_loc));
 	}
 
 	if (__bnxt_alloc_fw_health(bp)) {
@@ -7451,7 +7464,6 @@ static void bnxt_try_map_fw_health_reg(struct bnxt *bp)
 		return;
 	}
 
-	status_loc = readl(hs + offsetof(struct hcomm_status, fw_status_loc));
 	bp->fw_health->regs[BNXT_FW_HEALTH_REG] = status_loc;
 	reg_type = BNXT_FW_HEALTH_REG_TYPE(status_loc);
 	if (reg_type == BNXT_FW_HEALTH_REG_TYPE_GRC) {
@@ -9355,14 +9367,30 @@ static int bnxt_fw_reset_via_optee(struct bnxt *bp)
 static int bnxt_try_recover_fw(struct bnxt *bp)
 {
 	if (bp->fw_health && bp->fw_health->status_reliable) {
-		u32 sts = bnxt_fw_health_readl(bp, BNXT_FW_HEALTH_REG);
+		int retry = 0, rc;
+		u32 sts;
+
+		mutex_lock(&bp->hwrm_cmd_lock);
+		do {
+			rc = __bnxt_hwrm_ver_get(bp, true);
+			sts = bnxt_fw_health_readl(bp, BNXT_FW_HEALTH_REG);
+			if (!sts || !BNXT_FW_IS_BOOTING(sts))
+				break;
+			retry++;
+		} while (rc == -EBUSY && retry < BNXT_FW_RETRY);
+		mutex_unlock(&bp->hwrm_cmd_lock);
 
-		netdev_err(bp->dev, "Firmware not responding, status: 0x%x\n",
-			   sts);
+		if (!BNXT_FW_IS_HEALTHY(sts)) {
+			netdev_err(bp->dev,
+				   "Firmware not responding, status: 0x%x\n",
+				   sts);
+			rc = -ENODEV;
+		}
 		if (sts & FW_STATUS_REG_CRASHED_NO_MASTER) {
 			netdev_warn(bp->dev, "Firmware recover via OP-TEE requested\n");
 			return bnxt_fw_reset_via_optee(bp);
 		}
+		return rc;
 	}
 
 	return -ENODEV;
drivers/net/ethernet/broadcom/bnxt/bnxt.h +7 -0
index a1dd80a0fcf6..867b1d3a134e 100644
@@ -1345,9 +1345,14 @@ struct bnxt_test_info {
 #define BNXT_CAG_REG_LEGACY_INT_STATUS		0x4014
 #define BNXT_CAG_REG_BASE			0x300000
 
+#define BNXT_GRC_REG_STATUS_P5			0x520
+
 #define BNXT_GRCPF_REG_KONG_COMM		0xA00
 #define BNXT_GRCPF_REG_KONG_COMM_TRIGGER	0xB00
 
+#define BNXT_GRC_REG_CHIP_NUM			0x48
+#define BNXT_GRC_REG_BASE			0x260000
+
 #define BNXT_GRC_BASE_MASK			0xfffff000
 #define BNXT_GRC_OFFSET_MASK			0x00000ffc
 
@@ -1547,6 +1552,8 @@ struct bnxt_fw_reporter_ctx {
 #define BNXT_FW_IS_ERR(sts)		(((sts) & BNXT_FW_STATUS_HEALTH_MSK) > \
 					 BNXT_FW_STATUS_HEALTHY)
 
+#define BNXT_FW_RETRY			5
+
 struct bnxt {
 	void __iomem		*bar0;
 	void __iomem		*bar1;
Signature · 1 line
2.18.1
author maintainer
#5 lore raw

PATCH 5/15 bnxt_en: Move reading VPD info after successful handshake with fw.

In reply to Michael Chan

From: Vasundhara Volam <[email protected]>

If firmware is in reset or in bad state, it won't be able to return VPD data. Move bnxt_vpd_read_info() until after bnxt_fw_init_one_p1() successfully returns. By then we would have established proper communications with the firmware.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -3
index c460dd796c1c..2fb9873e0162 100644
@@ -12584,9 +12584,6 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	dev->ethtool_ops = &bnxt_ethtool_ops;
 	pci_set_drvdata(pdev, dev);
 
-	if (BNXT_PF(bp))
-		bnxt_vpd_read_info(bp);
-
 	rc = bnxt_alloc_hwrm_resources(bp);
 	if (rc)
 		goto init_err_pci_clean;
@@ -12598,6 +12595,9 @@ static int bnxt_init_one(struct pci_dev *pdev, const struct pci_device_id *ent)
 	if (rc)
 		goto init_err_pci_clean;
 
+	if (BNXT_PF(bp))
+		bnxt_vpd_read_info(bp);
+
 	if (BNXT_CHIP_P5(bp)) {
 		bp->flags |= BNXT_FLAG_CHIP_P5;
 		if (BNXT_CHIP_SR2(bp))
Signature · 1 line
2.18.1
author maintainer
#6 lore raw

PATCH 6/15 bnxt_en: Add an upper bound for all firmware command timeouts.

In reply to Michael Chan

From: Vasundhara Volam <[email protected]>

The timeout period for firmware messages is passed to the driver from the firmware in the response of the first command. This timeout period is multiplied by a factor for certain long running commands such as NVRAM commands. In some cases, the timeout period can become really long and it can cause hung task warnings if firmware has crashed or is not responding. To avoid such long delays, cap all firmware commands to a max timeout value of 40 seconds.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 2 ++
 drivers/net/ethernet/broadcom/bnxt/bnxt.h | 1 +
 2 files changed, 3 insertions(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +2 -0
index 2fb9873e0162..c06c5f81f087 100644
@@ -4425,6 +4425,8 @@ static int bnxt_hwrm_do_send_msg(struct bnxt *bp, void *msg, u32 msg_len,
 
 	if (!timeout)
 		timeout = DFLT_HWRM_CMD_TIMEOUT;
+	/* Limit timeout to an upper limit */
+	timeout = min(timeout, HWRM_CMD_MAX_TIMEOUT);
 	/* convert timeout to usec */
 	timeout *= 1000;
 
drivers/net/ethernet/broadcom/bnxt/bnxt.h +1 -0
index 867b1d3a134e..cbb338baab07 100644
@@ -656,6 +656,7 @@ struct nqe_cn {
 #define BNXT_HWRM_MAX_REQ_LEN		(bp->hwrm_max_req_len)
 #define BNXT_HWRM_SHORT_REQ_LEN		sizeof(struct hwrm_short_input)
 #define DFLT_HWRM_CMD_TIMEOUT		500
+#define HWRM_CMD_MAX_TIMEOUT		40000
 #define SHORT_HWRM_CMD_TIMEOUT		20
 #define HWRM_CMD_TIMEOUT		(bp->hwrm_cmd_timeout)
 #define HWRM_RESET_TIMEOUT		((HWRM_CMD_TIMEOUT) * 4)
Signature · 1 line
2.18.1
author maintainer
#7 lore raw

PATCH 7/15 bnxt_en: log firmware debug notifications

In reply to Michael Chan

From: Edwin Peer <[email protected]>

Firmware is capable of generating asynchronous debug notifications. The event data is opaque to the driver and is simply logged. Debug notifications can be enabled by turning on hardware status messages using the ethtool msglvl interface.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +8 -0
index c06c5f81f087..c8c25f7644ae 100644
@@ -255,6 +255,7 @@ static const u16 bnxt_async_events_arr[] = {
 	ASYNC_EVENT_CMPL_EVENT_ID_PORT_PHY_CFG_CHANGE,
 	ASYNC_EVENT_CMPL_EVENT_ID_RESET_NOTIFY,
 	ASYNC_EVENT_CMPL_EVENT_ID_ERROR_RECOVERY,
+	ASYNC_EVENT_CMPL_EVENT_ID_DEBUG_NOTIFICATION,
 	ASYNC_EVENT_CMPL_EVENT_ID_RING_MONITOR_MSG,
 };
 
@@ -2072,6 +2073,13 @@ static int bnxt_async_event_process(struct bnxt *bp,
 			bnxt_fw_health_readl(bp, BNXT_FW_RESET_CNT_REG);
 		goto async_event_process_exit;
 	}
+	case ASYNC_EVENT_CMPL_EVENT_ID_DEBUG_NOTIFICATION:
+		if (netif_msg_hw(bp)) {
+			netdev_notice(bp->dev,
+				      "Received firmware debug notification, data1: 0x%x, data2: 0x%x\n",
+				      data1, data2);
+		}
+		goto async_event_process_exit;
 	case ASYNC_EVENT_CMPL_EVENT_ID_RING_MONITOR_MSG: {
 		struct bnxt_rx_ring_info *rxr;
 		u16 grp_idx;
Signature · 1 line
2.18.1
#8 lore raw

Re: [PATCH net-next 07/15] bnxt_en: log firmware debug notifications

Quoted 6 lines · Michael Chan

From: Edwin Peer <[email protected]>

Firmware is capable of generating asynchronous debug notifications. The event data is opaque to the driver and is simply logged. Debug notifications can be enabled by turning on hardware status messages using the ethtool msglvl interface.

[]

drivers/net/ethernet/broadcom/bnxt/bnxt.c

Quoted 1 line
drivers/net/ethernet/broadcom/bnxt/bnxt.c +0 -0

[]

drivers/net/ethernet/broadcom/bnxt/bnxt.c static int bnxt_async_event_process(struct bnxt *bp, · line 2081

Quoted 10 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +6 -0
@@ -2072,6 +2073,13 @@ static int bnxt_async_event_process(struct bnxt *bp,
  			bnxt_fw_health_readl(bp, BNXT_FW_RESET_CNT_REG);
  		goto async_event_process_exit;
  	}
+	case ASYNC_EVENT_CMPL_EVENT_ID_DEBUG_NOTIFICATION:
+		if (netif_msg_hw(bp)) {
+			netdev_notice(bp->dev,
+				      "Received firmware debug notification, data1: 0x%x, data2: 0x%x\n",
+				      data1, data2);
+		}
		netif_notice(bp, hw, bp->dev,
			     "Received firmware debug notification, data1: 0x%x, data2: 0x%x\n",
			     data1, data2);
Quoted 5 lines
drivers/net/ethernet/broadcom/bnxt/bnxt.c +1 -0
+		goto async_event_process_exit;
 
  	case ASYNC_EVENT_CMPL_EVENT_ID_RING_MONITOR_MSG: {
  		struct bnxt_rx_ring_info *rxr;

  u16 grp_idx;

author maintainer
#9 lore raw

PATCH 8/15 bnxt_en: attempt to reinitialize after aborted reset

In reply to Michael Chan

From: Edwin Peer <[email protected]>

Drawing a hard line on aborted resets prevents a NIC open in some scenarios that may otherwise be recoverable. For example, if a firmware recovery happened while a PF was down and an attempt was made to bring up an associated VF in this state, then it was impossible to ever bring up this VF without a rebind or reload of its driver.

Attempt to reinitialize the firmware when an aborted reset (or failed init after a reset) is discovered during open - it may succeed. Also take care to allow the user to retry opening the NIC even after an aborted reset.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 29 +++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +27 -2
index c8c25f7644ae..7f30a9fee0c8 100644
@@ -9768,6 +9768,25 @@ static void bnxt_preset_reg_win(struct bnxt *bp)
 
 static int bnxt_init_dflt_ring_mode(struct bnxt *bp);
 
+static int bnxt_reinit_after_abort(struct bnxt *bp)
+{
+	int rc;
+
+	if (test_bit(BNXT_STATE_IN_FW_RESET, &bp->state))
+		return -EBUSY;
+
+	rc = bnxt_fw_init_one(bp);
+	if (!rc) {
+		bnxt_clear_int_mode(bp);
+		rc = bnxt_init_int_mode(bp);
+		if (!rc) {
+			clear_bit(BNXT_STATE_ABORT_ERR, &bp->state);
+			set_bit(BNXT_STATE_FW_RESET_DET, &bp->state);
+		}
+	}
+	return rc;
+}
+
 static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
 {
 	int rc = 0;
@@ -9926,8 +9945,14 @@ static int bnxt_open(struct net_device *dev)
 	int rc;
 
 	if (test_bit(BNXT_STATE_ABORT_ERR, &bp->state)) {
-		netdev_err(bp->dev, "A previous firmware reset did not complete, aborting\n");
-		return -ENODEV;
+		rc = bnxt_reinit_after_abort(bp);
+		if (rc) {
+			if (rc == -EBUSY)
+				netdev_err(bp->dev, "A previous firmware reset has not completed, aborting\n");
+			else
+				netdev_err(bp->dev, "Failed to reinitialize after aborted firmware reset\n");
+			return -ENODEV;
+		}
 	}
 
 	rc = bnxt_hwrm_if_change(bp, true);
Signature · 1 line
2.18.1
author maintainer
#10 lore raw

PATCH 9/15 bnxt_en: Retry open if firmware is in reset.

In reply to Michael Chan

From: Vasundhara Volam <[email protected]>

Firmware may be in the middle of reset when the driver tries to do ifup. In that case, firmware will return a special error code and the driver will retry 10 times with 50 msecs delay after each retry.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 15 +++++++++++++--
 drivers/net/ethernet/broadcom/bnxt/bnxt.h |  1 +
 2 files changed, 14 insertions(+), 2 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +13 -2
index 7f30a9fee0c8..c35a5d497c1e 100644
@@ -9411,8 +9411,8 @@ static int bnxt_hwrm_if_change(struct bnxt *bp, bool up)
 	struct hwrm_func_drv_if_change_output *resp = bp->hwrm_cmd_resp_addr;
 	struct hwrm_func_drv_if_change_input req = {0};
 	bool resc_reinit = false, fw_reset = false;
+	int rc, retry = 0;
 	u32 flags = 0;
-	int rc;
 
 	if (!(bp->fw_cap & BNXT_FW_CAP_IF_CHANGE))
 		return 0;
@@ -9421,10 +9421,21 @@ static int bnxt_hwrm_if_change(struct bnxt *bp, bool up)
 	if (up)
 		req.flags = cpu_to_le32(FUNC_DRV_IF_CHANGE_REQ_FLAGS_UP);
 	mutex_lock(&bp->hwrm_cmd_lock);
-	rc = _hwrm_send_message(bp, &req, sizeof(req), HWRM_CMD_TIMEOUT);
+	while (retry < BNXT_FW_IF_RETRY) {
+		rc = _hwrm_send_message(bp, &req, sizeof(req),
+					HWRM_CMD_TIMEOUT);
+		if (rc != -EAGAIN)
+			break;
+
+		msleep(50);
+		retry++;
+	}
 	if (!rc)
 		flags = le32_to_cpu(resp->flags);
 	mutex_unlock(&bp->hwrm_cmd_lock);
+
+	if (rc == -EAGAIN)
+		return rc;
 	if (rc && up) {
 		rc = bnxt_try_recover_fw(bp);
 		fw_reset = true;
drivers/net/ethernet/broadcom/bnxt/bnxt.h +1 -0
index cbb338baab07..bd36f00ef28c 100644
@@ -1554,6 +1554,7 @@ struct bnxt_fw_reporter_ctx {
 					 BNXT_FW_STATUS_HEALTHY)
 
 #define BNXT_FW_RETRY			5
+#define BNXT_FW_IF_RETRY		10
 
 struct bnxt {
 	void __iomem		*bar0;
Signature · 1 line
2.18.1
author maintainer
#11 lore raw

PATCH 10/15 bnxt_en: Add bnxt_fw_reset_timeout() helper.

In reply to Michael Chan

This code to check if we have reached the maximum wait time after firmware reset is used multiple times. Add a helper function to do this.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +9 -6
index c35a5d497c1e..98caac9fbdee 100644
@@ -11503,6 +11503,12 @@ static void bnxt_reset_all(struct bnxt *bp)
 	bp->fw_reset_timestamp = jiffies;
 }
 
+static bool bnxt_fw_reset_timeout(struct bnxt *bp)
+{
+	return time_after(jiffies, bp->fw_reset_timestamp +
+			  (bp->fw_reset_max_dsecs * HZ / 10));
+}
+
 static void bnxt_fw_reset_task(struct work_struct *work)
 {
 	struct bnxt *bp = container_of(work, struct bnxt, fw_reset_task.work);
@@ -11524,8 +11530,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
 				   bp->fw_reset_timestamp));
 			goto fw_reset_abort;
 		} else if (n > 0) {
-			if (time_after(jiffies, bp->fw_reset_timestamp +
-				       (bp->fw_reset_max_dsecs * HZ / 10))) {
+			if (bnxt_fw_reset_timeout(bp)) {
 				clear_bit(BNXT_STATE_IN_FW_RESET, &bp->state);
 				bp->fw_reset_state = 0;
 				netdev_err(bp->dev, "Firmware reset aborted, bnxt_get_registered_vfs() returns %d\n",
@@ -11554,8 +11559,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
 
 		val = bnxt_fw_health_readl(bp, BNXT_FW_HEALTH_REG);
 		if (!(val & BNXT_FW_STATUS_SHUTDOWN) &&
-		    !time_after(jiffies, bp->fw_reset_timestamp +
-		    (bp->fw_reset_max_dsecs * HZ / 10))) {
+		    !bnxt_fw_reset_timeout(bp)) {
 			bnxt_queue_fw_reset_work(bp, HZ / 5);
 			return;
 		}
@@ -11597,8 +11601,7 @@ static void bnxt_fw_reset_task(struct work_struct *work)
 		bp->hwrm_cmd_timeout = SHORT_HWRM_CMD_TIMEOUT;
 		rc = __bnxt_hwrm_ver_get(bp, true);
 		if (rc) {
-			if (time_after(jiffies, bp->fw_reset_timestamp +
-				       (bp->fw_reset_max_dsecs * HZ / 10))) {
+			if (bnxt_fw_reset_timeout(bp)) {
 				netdev_err(bp->dev, "Firmware reset aborted\n");
 				goto fw_reset_abort_status;
 			}
Signature · 1 line
2.18.1
author maintainer
#12 lore raw

PATCH 11/15 bnxt_en: Add a new BNXT_STATE_NAPI_DISABLED flag to keep track of NAPI state.

In reply to Michael Chan

Up until now, we don't need to keep track of this state because NAPI is always enabled once and disabled once during bring up and shutdown. For better error recovery in subsequent patches, we want to quiesce the device earlier during fatal error conditions. The normal shutdown sequence will disable NAPI again and the flag will prevent disabling NAPI twice.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 4 +++-
 drivers/net/ethernet/broadcom/bnxt/bnxt.h | 1 +
 2 files changed, 4 insertions(+), 1 deletion(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -1
index 98caac9fbdee..83846b50042a 100644
@@ -8836,7 +8836,8 @@ static void bnxt_disable_napi(struct bnxt *bp)
 {
 	int i;
 
-	if (!bp->bnapi)
+	if (!bp->bnapi ||
+	    test_and_set_bit(BNXT_STATE_NAPI_DISABLED, &bp->state))
 		return;
 
 	for (i = 0; i < bp->cp_nr_rings; i++) {
@@ -8853,6 +8854,7 @@ static void bnxt_enable_napi(struct bnxt *bp)
 {
 	int i;
 
+	clear_bit(BNXT_STATE_NAPI_DISABLED, &bp->state);
 	for (i = 0; i < bp->cp_nr_rings; i++) {
 		struct bnxt_napi *bnapi = bp->bnapi[i];
 		struct bnxt_cp_ring_info *cpr;
drivers/net/ethernet/broadcom/bnxt/bnxt.h +1 -0
index bd36f00ef28c..4ef6888acdc6 100644
@@ -1809,6 +1809,7 @@ struct bnxt {
 #define BNXT_STATE_FW_FATAL_COND	6
 #define BNXT_STATE_DRV_REGISTERED	7
 #define BNXT_STATE_PCI_CHANNEL_IO_FROZEN	8
+#define BNXT_STATE_NAPI_DISABLED	9
 
 #define BNXT_NO_FW_ACCESS(bp)					\
 	(test_bit(BNXT_STATE_FW_FATAL_COND, &(bp)->state) ||	\
Signature · 1 line
2.18.1
author maintainer
#13 lore raw

PATCH 12/15 bnxt_en: Modify bnxt_disable_int_sync() to be called more than once.

In reply to Michael Chan

In the event of a fatal firmware error, we want to disable IRQ early in the recovery sequence. This change will allow it to be called safely again as part of the normal shutdown sequence.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +3 -0
index 83846b50042a..80dab4e622ab 100644
@@ -4280,6 +4280,9 @@ static void bnxt_disable_int_sync(struct bnxt *bp)
 {
 	int i;
 
+	if (!bp->irq_tbl)
+		return;
+
 	atomic_inc(&bp->intr_sem);
 
 	bnxt_disable_int(bp);
Signature · 1 line
2.18.1
author maintainer
#14 lore raw

PATCH 13/15 bnxt_en: Improve firmware fatal error shutdown sequence.

In reply to Michael Chan

In the event of a fatal firmware error, firmware will notify the host and then it will proceed to do core reset when it sees that all functions have disabled Bus Master. To prevent Master Aborts and other hard errors, we need to quiesce all activities in addition to disabling Bus Master before the chip goes into core reset.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +10 -3
index 80dab4e622ab..e7abb3b7ed68 100644
@@ -10905,11 +10905,18 @@ static void bnxt_rx_ring_reset(struct bnxt *bp)
 static void bnxt_fw_reset_close(struct bnxt *bp)
 {
 	bnxt_ulp_stop(bp);
-	/* When firmware is fatal state, disable PCI device to prevent
-	 * any potential bad DMAs before freeing kernel memory.
+	/* When firmware is in fatal state, quiesce device and disable
+	 * bus master to prevent any potential bad DMAs before freeing
+	 * kernel memory.
 	 */
-	if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state))
+	if (test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state)) {
+		bnxt_tx_disable(bp);
+		bnxt_disable_napi(bp);
+		bnxt_disable_int_sync(bp);
+		bnxt_free_irq(bp);
+		bnxt_clear_int_mode(bp);
 		pci_disable_device(bp->pdev);
+	}
 	__bnxt_close_nic(bp, true, false);
 	bnxt_clear_int_mode(bp);
 	bnxt_hwrm_func_drv_unrgtr(bp);
Signature · 1 line
2.18.1
author maintainer
#15 lore raw

PATCH 14/15 bnxt_en: Consolidate firmware reset event logging.

In reply to Michael Chan

Combine the three netdev_warn() calls into a single call, printed at the NETIF_MSG_HW log level.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +10 -7
index e7abb3b7ed68..221f5437884b 100644
@@ -2022,10 +2022,9 @@ static int bnxt_async_event_process(struct bnxt *bp,
 			goto async_event_process_exit;
 		set_bit(BNXT_RESET_TASK_SILENT_SP_EVENT, &bp->sp_event);
 		break;
-	case ASYNC_EVENT_CMPL_EVENT_ID_RESET_NOTIFY:
-		if (netif_msg_hw(bp))
-			netdev_warn(bp->dev, "Received RESET_NOTIFY event, data1: 0x%x, data2: 0x%x\n",
-				    data1, data2);
+	case ASYNC_EVENT_CMPL_EVENT_ID_RESET_NOTIFY: {
+		char *fatal_str = "non-fatal";
+
 		if (!bp->fw_health)
 			goto async_event_process_exit;
 
@@ -2037,14 +2036,18 @@ static int bnxt_async_event_process(struct bnxt *bp,
 		if (!bp->fw_reset_max_dsecs)
 			bp->fw_reset_max_dsecs = BNXT_DFLT_FW_RST_MAX_DSECS;
 		if (EVENT_DATA1_RESET_NOTIFY_FATAL(data1)) {
-			netdev_warn(bp->dev, "Firmware fatal reset event received\n");
+			fatal_str = "fatal";
 			set_bit(BNXT_STATE_FW_FATAL_COND, &bp->state);
-		} else {
-			netdev_warn(bp->dev, "Firmware non-fatal reset event received, max wait time %d msec\n",
+		}
+		if (netif_msg_hw(bp)) {
+			netdev_warn(bp->dev, "Firmware %s reset event, data1: 0x%x, data2: 0x%x, min wait %u ms, max wait %u ms\n",
+				    fatal_str, data1, data2,
+				    bp->fw_reset_min_dsecs * 100,
 				    bp->fw_reset_max_dsecs * 100);
 		}
 		set_bit(BNXT_FW_RESET_NOTIFY_SP_EVENT, &bp->sp_event);
 		break;
+	}
 	case ASYNC_EVENT_CMPL_EVENT_ID_ERROR_RECOVERY: {
 		struct bnxt_fw_health *fw_health = bp->fw_health;
 
Signature · 1 line
2.18.1
author maintainer
#16 lore raw

PATCH 15/15 bnxt_en: Do not process completion entries after fatal condition detected.

In reply to Michael Chan

Once the firmware fatal condition is detected, we should cease comminication with the firmware and hardware quickly even if there are many completion entries in the completion rings. This will speed up the recovery process and prevent further I/Os that may cause further exceptions.

Do not proceed in the NAPI poll function if fatal condition is detected. Call napi_complete() and return without arming interrupts. Cleanup of all rings and reset are imminent.

---
 drivers/net/ethernet/broadcom/bnxt/bnxt.c | 8 ++++++++
 1 file changed, 8 insertions(+)
drivers/net/ethernet/broadcom/bnxt/bnxt.c +8 -0
index 221f5437884b..dd7d2caa57a2 100644
@@ -2405,6 +2405,10 @@ static int bnxt_poll(struct napi_struct *napi, int budget)
 	struct bnxt_cp_ring_info *cpr = &bnapi->cp_ring;
 	int work_done = 0;
 
+	if (unlikely(test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state))) {
+		napi_complete(napi);
+		return 0;
+	}
 	while (1) {
 		work_done += bnxt_poll_work(bp, cpr, budget - work_done);
 
@@ -2479,6 +2483,10 @@ static int bnxt_poll_p5(struct napi_struct *napi, int budget)
 	int work_done = 0;
 	u32 cons;
 
+	if (unlikely(test_bit(BNXT_STATE_FW_FATAL_COND, &bp->state))) {
+		napi_complete(napi);
+		return 0;
+	}
 	if (cpr->has_more_work) {
 		cpr->has_more_work = 0;
 		work_done = __bnxt_poll_cqs(bp, bnapi, budget);
Signature · 1 line
2.18.1
Acked-by on the v1 series
#17 lore raw
In reply to Michael Chan
Quoted 33 lines · Michael Chan

This series contains a number of improvements in the area of error recovery. Most error recovery scenarios are tightly coordinated with the firmware. A number of patches add retry logic to establish connection with the firmware if there are indications that the firmware is still alive and will likely transition back to the normal state. Some patches speed up the recovery process and make it more reliable. There are some cleanup patches as well.

Edwin Peer (3):
bnxt_en: handle CRASH_NO_MASTER during bnxt_open()
bnxt_en: log firmware debug notifications
bnxt_en: attempt to reinitialize after aborted reset

Michael Chan (9):
bnxt_en: Update firmware interface to 1.10.2.11.
bnxt_en: Define macros for the various health register states.
bnxt_en: Retry sending the first message to firmware if it is under
reset.
bnxt_en: Add bnxt_fw_reset_timeout() helper.
bnxt_en: Add a new BNXT_STATE_NAPI_DISABLED flag to keep track of NAPI
state.
bnxt_en: Modify bnxt_disable_int_sync() to be called more than once.
bnxt_en: Improve firmware fatal error shutdown sequence.
bnxt_en: Consolidate firmware reset event logging.
bnxt_en: Do not process completion entries after fatal condition
detected.

Vasundhara Volam (3):
bnxt_en: Move reading VPD info after successful handshake with fw.
bnxt_en: Add an upper bound for all firmware command timeouts.
bnxt_en: Retry open if firmware is in reset.

Quoted 5 lines
 drivers/net/ethernet/broadcom/bnxt/bnxt.c     | 228 ++++++++++++----
 drivers/net/ethernet/broadcom/bnxt/bnxt.h     |  22 ++
 .../net/ethernet/broadcom/bnxt/bnxt_devlink.c |   7 +-
 drivers/net/ethernet/broadcom/bnxt/bnxt_hsi.h | 249 ++++++++++++++----
 4 files changed, 393 insertions(+), 113 deletions(-)

For netdrv:

maintainer Applied
#18 lore raw
Quoted 10 lines · On Mon, 25 Jan 2021 20:37:52 -0500 Willem de Bruijn wrote:
Quoted 7 lines · Michael Chan

This series contains a number of improvements in the area of error recovery. Most error recovery scenarios are tightly coordinated with the firmware. A number of patches add retry logic to establish connection with the firmware if there are indications that the firmware is still alive and will likely transition back to the normal state. Some patches speed up the recovery process and make it more reliable. There are some cleanup patches as well.

Thanks!

Applied.