Skip to contentSkip to conversation

brcmfmac: recover PCIe devices from firmware crashes

PATCH In review Rafał Miłecki 3 participants 10 messages Feb 26, 2019 – Apr 4, 2019 Broadcom BRCM80211 IEEE802.11 wireless drivers

lore
Share & tools
lore
#0 lore raw

cover brcmfmac: recover PCIe devices from firmware crashes

From: Rafał Miłecki <[email protected]>

So far PCIe firmware halts / crashes were resulting in massive timeouts and running out of resources (e.g. slots). There wasn't even a clear message indicating the problem source.

This patches improves it by:
1) Printing an error
2) Reloading a firmware

After that user can setup interface(s) & use a wireless card again. It should be much more convenient than reloading a module manually thanks to:
1) Automation
2) Not affecting other (working) wireless cards

Rafał Miłecki (4):
brcmfmac: support multiple brcmf_fw_alloc_request() calls
brcmfmac: get RAM info right before downloading PCIe firmware
brcmfmac: add function designated for handling firmware failures
brcmfmac: reset PCIe bus on firmware crash

 .../broadcom/brcm80211/brcmfmac/bus.h         | 12 ++++++
 .../broadcom/brcm80211/brcmfmac/chip.c        |  6 ++-
 .../broadcom/brcm80211/brcmfmac/chip.h        |  1 +
 .../broadcom/brcm80211/brcmfmac/core.c        | 22 ++++++++++
 .../broadcom/brcm80211/brcmfmac/core.h        |  2 +
 .../broadcom/brcm80211/brcmfmac/firmware.c    |  1 +
 .../broadcom/brcm80211/brcmfmac/pcie.c        | 43 ++++++++++++++++++-
 .../broadcom/brcm80211/brcmfmac/sdio.c        |  4 +-
 8 files changed, 86 insertions(+), 5 deletions(-)
Signature · 1 line
2.20.1
#1 lore raw

PATCH 1/4 brcmfmac: support repeated brcmf_fw_alloc_request() calls

From: Rafał Miłecki <[email protected]>

During a normal brcmfmac lifetime brcmf_fw_alloc_request() is called once only during the probe. It's safe to assume provided array is clear.

Further brcmfmac improvements may require calling it multiple times though. This patch allows it by fixing invalid firmware paths like:
brcm/brcmfmac4366c-pcie.binbrcm/brcmfmac4366c-pcie.bin

---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c | 1 +
 1 file changed, 1 insertion(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c +1 -0
index 8209a42dea72..65098a02e1ad 100644
@@ -743,6 +743,7 @@ brcmf_fw_alloc_request(u32 chip, u32 chiprev,
 
 	for (j = 0; j < n_fwnames; j++) {
 		fwreq->items[j].path = fwnames[j].path;
+		fwnames[j].path[0] = '\0';
 		/* check if firmware path is provided by module parameter */
 		if (brcmf_mp_global.firmware_path[0] != '\0') {
 			strlcpy(fwnames[j].path, mp_path,
Signature · 1 line
2.20.1
maintainer Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] brcmfmac: support repeated brcmf_fw_alloc_request() calls

Quoted 8 lines · Rafał Miłecki

From: Rafał Miłecki <[email protected]>

During a normal brcmfmac lifetime brcmf_fw_alloc_request() is called once only during the probe. It's safe to assume provided array is clear.

Further brcmfmac improvements may require calling it multiple times though. This patch allows it by fixing invalid firmware paths like:
brcm/brcmfmac4366c-pcie.binbrcm/brcmfmac4366c-pcie.bin

Quoted 4 lines
---
  drivers/net/wireless/broadcom/brcm80211/brcmfmac/firmware.c | 1 +
  1 file changed, 1 insertion(+)
#3 lore raw

Re: [PATCH 1/4] brcmfmac: support repeated brcmf_fw_alloc_request() calls

In reply to Rafał Miłecki
Quoted 11 lines · Rafał Miłecki

From: Rafał Miłecki <[email protected]>

During a normal brcmfmac lifetime brcmf_fw_alloc_request() is called once only during the probe. It's safe to assume provided array is clear.

Further brcmfmac improvements may require calling it multiple times though. This patch allows it by fixing invalid firmware paths like:
brcm/brcmfmac4366c-pcie.binbrcm/brcmfmac4366c-pcie.bin

3 patches applied to wireless-drivers-next.git, thanks.

c9692820710f brcmfmac: support repeated brcmf_fw_alloc_request() calls a2ec87ddbf16 brcmfmac: add a function designated for handling firmware fails 4684997d9eea brcmfmac: reset PCIe bus on a firmware crash

Signature · 3 lines
https://patchwork.kernel.org/patch/10830169/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches
#4 lore raw

PATCH 2/4 brcmfmac: get RAM info right before downloading PCIe firmware

In reply to Rafał Miłecki

From: Rafał Miłecki <[email protected]>

It's important as brcmf_chip_get_raminfo() also makes sure that memory is properly setup. Without it the firmware could report invalid RAM address like 0x04000001.

During a normal brcmfmac lifetime brcmf_chip_get_raminfo() is called on probe by the brcmf_chip_recognition(). This change allows implementing further improvements like handling errors by resetting a device with the brcmf_pcie_reset_device() and redownloading a firmware afterwards.

---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/chip.c | 6 ++++--
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/chip.h | 1 +
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c | 6 ++++++
 3 files changed, 11 insertions(+), 2 deletions(-)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/chip.c +4 -2
index 22534bf2a90c..fcaf19165891 100644
@@ -707,8 +707,10 @@ static u32 brcmf_chip_tcm_rambase(struct brcmf_chip_priv *ci)
 	return 0;
 }
 
-static int brcmf_chip_get_raminfo(struct brcmf_chip_priv *ci)
+int brcmf_chip_get_raminfo(struct brcmf_chip *pub)
 {
+	struct brcmf_chip_priv *ci = container_of(pub, struct brcmf_chip_priv,
+						  pub);
 	struct brcmf_core_priv *mem_core;
 	struct brcmf_core *mem;
 
@@ -990,7 +992,7 @@ static int brcmf_chip_recognition(struct brcmf_chip_priv *ci)
 		brcmf_chip_set_passive(&ci->pub);
 	}
 
-	return brcmf_chip_get_raminfo(ci);
+	return brcmf_chip_get_raminfo(&ci->pub);
 }
 
 static void brcmf_chip_disable_arm(struct brcmf_chip_priv *chip, u16 id)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/chip.h +1 -0
index 0ae3b33bab62..4794cf38b4d3 100644
@@ -80,6 +80,7 @@ struct brcmf_buscore_ops {
 	void (*activate)(void *ctx, struct brcmf_chip *chip, u32 rstvec);
 };
 
+int brcmf_chip_get_raminfo(struct brcmf_chip *pub);
 struct brcmf_chip *brcmf_chip_attach(void *ctx,
 				     const struct brcmf_buscore_ops *ops);
 void brcmf_chip_detach(struct brcmf_chip *chip);
drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c +6 -0
index 58a6bc379358..39f6421885f6 100644
@@ -1727,6 +1727,12 @@ static void brcmf_pcie_setup(struct device *dev, int ret,
 	nvram_len = fwreq->items[BRCMF_PCIE_FW_NVRAM].nv_data.len;
 	kfree(fwreq);
 
+	ret = brcmf_chip_get_raminfo(devinfo->ci);
+	if (ret) {
+		brcmf_err(bus, "Failed to get RAM info\n");
+		goto fail;
+	}
+
 	/* Some of the firmwares have the size of the memory of the device
 	 * defined inside the firmware. This is because part of the memory in
 	 * the device is shared and the devision is determined by FW. Parse
Signature · 1 line
2.20.1
maintainer
#5 lore raw

Re: [PATCH 2/4] brcmfmac: get RAM info right before downloading PCIe firmware

Quoted 10 lines · Rafał Miłecki

From: Rafał Miłecki <[email protected]>

It's important as brcmf_chip_get_raminfo() also makes sure that memory is properly setup. Without it the firmware could report invalid RAM address like 0x04000001.

During a normal brcmfmac lifetime brcmf_chip_get_raminfo() is called on probe by the brcmf_chip_recognition(). This change allows implementing further improvements like handling errors by resetting a device with the brcmf_pcie_reset_device() and redownloading a firmware afterwards.

Thanks, Rafał

How about avoiding the ramsize adjustment in redownloading scenario. We could add a reinit field in devinfo and check that.

	/* Some of the firmwares have the size of the memory of the device
	 * defined inside the firmware. This is because part of the memory in
	 * the device is shared and the devision is determined by FW. Parse
	 * the firmware and adjust the chip memory size now.
	 */
	if (!devinfo->reinit)
		brcmf_pcie_adjust_ramsize(devinfo, (u8 *)fw->data, fw->size);

Regards,
Arend

Quoted 6 lines
---
  drivers/net/wireless/broadcom/brcm80211/brcmfmac/chip.c | 6 ++++--
  drivers/net/wireless/broadcom/brcm80211/brcmfmac/chip.h | 1 +
  drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c | 6 ++++++
  3 files changed, 11 insertions(+), 2 deletions(-)
#6 lore raw

PATCH 3/4 brcmfmac: add a function designated for handling firmware fails

In reply to Rafał Miłecki

From: Rafał Miłecki <[email protected]>

This improves handling PCIe firmware halts by printing a clear error message and replaces a similar code in the SDIO bus support.

It will also allow further improvements like trying to recover from a firmware crash.

---
 drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h |  2 ++
 .../net/wireless/broadcom/brcm80211/brcmfmac/core.c    | 10 ++++++++++
 .../net/wireless/broadcom/brcm80211/brcmfmac/pcie.c    |  2 +-
 .../net/wireless/broadcom/brcm80211/brcmfmac/sdio.c    |  4 ++--
 4 files changed, 15 insertions(+), 3 deletions(-)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h +2 -0
index 3d441c5c745c..801106583ae7 100644
@@ -262,6 +262,8 @@ void brcmf_detach(struct device *dev);
 void brcmf_dev_reset(struct device *dev);
 /* Request from bus module to initiate a coredump */
 void brcmf_dev_coredump(struct device *dev);
+/* Indication that firmware has halted or crashed */
+void brcmf_fw_crashed(struct device *dev);
 
 /* Configure the "global" bus state used by upper layers */
 void brcmf_bus_change_state(struct brcmf_bus *bus, enum brcmf_bus_state state);
drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +10 -0
index 4fbe8791f674..7f4d9356b79e 100644
@@ -1273,6 +1273,16 @@ void brcmf_dev_coredump(struct device *dev)
 		brcmf_dbg(TRACE, "failed to create coredump\n");
 }
 
+void brcmf_fw_crashed(struct device *dev)
+{
+	struct brcmf_bus *bus_if = dev_get_drvdata(dev);
+	struct brcmf_pub *drvr = bus_if->drvr;
+
+	bphy_err(drvr, "Firmware has halted or crashed\n");
+
+	brcmf_dev_coredump(dev);
+}
+
 void brcmf_detach(struct device *dev)
 {
 	s32 i;
drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c +1 -1
index 39f6421885f6..cfa34672315f 100644
@@ -730,7 +730,7 @@ static void brcmf_pcie_handle_mb_data(struct brcmf_pciedev_info *devinfo)
 	}
 	if (dtoh_mb_data & BRCMF_D2H_DEV_FWHALT) {
 		brcmf_dbg(PCIE, "D2H_MB_DATA: FW HALT\n");
-		brcmf_dev_coredump(&devinfo->pdev->dev);
+		brcmf_fw_crashed(&devinfo->pdev->dev);
 	}
 }
 
drivers/net/wireless/broadcom/brcm80211/brcmfmac/sdio.c +2 -2
index 4d104ab80fd8..a06af0cd4a7f 100644
@@ -1090,8 +1090,8 @@ static u32 brcmf_sdio_hostmail(struct brcmf_sdio *bus)
 
 	/* dongle indicates the firmware has halted/crashed */
 	if (hmb_data & HMB_DATA_FWHALT) {
-		brcmf_err("mailbox indicates firmware halted\n");
-		brcmf_dev_coredump(&sdiod->func1->dev);
+		brcmf_dbg(SDIO, "mailbox indicates firmware halted\n");
+		brcmf_fw_crashed(&sdiod->func1->dev);
 	}
 
 	/* Dongle recomposed rx frames, accept them again */
Signature · 1 line
2.20.1
maintainer Reviewed-by on v1 3/4
#7 lore raw

Re: [PATCH 3/4] brcmfmac: add a function designated for handling firmware fails

Quoted 7 lines · Rafał Miłecki

From: Rafał Miłecki <[email protected]>

This improves handling PCIe firmware halts by printing a clear error message and replaces a similar code in the SDIO bus support.

It will also allow further improvements like trying to recover from a firmware crash.

Quoted 7 lines
---
  drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h |  2 ++
  .../net/wireless/broadcom/brcm80211/brcmfmac/core.c    | 10 ++++++++++
  .../net/wireless/broadcom/brcm80211/brcmfmac/pcie.c    |  2 +-
  .../net/wireless/broadcom/brcm80211/brcmfmac/sdio.c    |  4 ++--
  4 files changed, 15 insertions(+), 3 deletions(-)
#8 lore raw

PATCH 4/4 brcmfmac: reset PCIe bus on a firmware crash

In reply to Rafał Miłecki

From: Rafał Miłecki <[email protected]>

This includes bus reset & reloading a firmware. It should be sufficient for a user space to (setup and) use a wireless device again.

Support for reset on USB & SDIO can be added later.

---
 .../broadcom/brcm80211/brcmfmac/bus.h         | 10 ++++++
 .../broadcom/brcm80211/brcmfmac/core.c        | 12 +++++++
 .../broadcom/brcm80211/brcmfmac/core.h        |  2 ++
 .../broadcom/brcm80211/brcmfmac/pcie.c        | 35 +++++++++++++++++++
 4 files changed, 59 insertions(+)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/bus.h +10 -0
index 801106583ae7..2fe167eae22c 100644
@@ -91,6 +91,7 @@ struct brcmf_bus_ops {
 	int (*get_fwname)(struct device *dev, const char *ext,
 			  unsigned char *fw_name);
 	void (*debugfs_create)(struct device *dev);
+	int (*reset)(struct device *dev);
 };
 
 
@@ -245,6 +246,15 @@ void brcmf_bus_debugfs_create(struct brcmf_bus *bus)
 	return bus->ops->debugfs_create(bus->dev);
 }
 
+static inline
+int brcmf_bus_reset(struct brcmf_bus *bus)
+{
+	if (!bus->ops->reset)
+		return -EOPNOTSUPP;
+
+	return bus->ops->reset(bus->dev);
+}
+
 /*
  * interface functions from common layer
  */
drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +12 -0
index 7f4d9356b79e..5f3548b13639 100644
@@ -1084,6 +1084,14 @@ static int brcmf_revinfo_read(struct seq_file *s, void *data)
 	return 0;
 }
 
+static void brcmf_core_bus_reset(struct work_struct *work)
+{
+	struct brcmf_pub *drvr = container_of(work, struct brcmf_pub,
+					      bus_reset);
+
+	brcmf_bus_reset(drvr->bus_if);
+}
+
 static int brcmf_bus_started(struct brcmf_pub *drvr, struct cfg80211_ops *ops)
 {
 	int ret = -1;
@@ -1155,6 +1163,8 @@ static int brcmf_bus_started(struct brcmf_pub *drvr, struct cfg80211_ops *ops)
 #endif
 #endif /* CONFIG_INET */
 
+	INIT_WORK(&drvr->bus_reset, brcmf_core_bus_reset);
+
 	/* populate debugfs */
 	brcmf_debugfs_add_entry(drvr, "revinfo", brcmf_revinfo_read);
 	brcmf_feat_debugfs_create(drvr);
@@ -1281,6 +1291,8 @@ void brcmf_fw_crashed(struct device *dev)
 	bphy_err(drvr, "Firmware has halted or crashed\n");
 
 	brcmf_dev_coredump(dev);
+
+	schedule_work(&drvr->bus_reset);
 }
 
 void brcmf_detach(struct device *dev)
drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.h +2 -0
index d8085ce579f4..9f09aa31eeda 100644
@@ -143,6 +143,8 @@ struct brcmf_pub {
 	struct notifier_block inet6addr_notifier;
 	struct brcmf_mp_device *settings;
 
+	struct work_struct bus_reset;
+
 	u8 clmver[BRCMF_DCMD_SMLEN];
 };
 
drivers/net/wireless/broadcom/brcm80211/brcmfmac/pcie.c +35 -0
index cfa34672315f..e941039ee1c3 100644
@@ -345,6 +345,10 @@ static const u32 brcmf_ring_itemsize[BRCMF_NROF_COMMON_MSGRINGS] = {
 	BRCMF_D2H_MSGRING_RX_COMPLETE_ITEMSIZE
 };
 
+static void brcmf_pcie_setup(struct device *dev, int ret,
+			     struct brcmf_fw_request *fwreq);
+static struct brcmf_fw_request *
+brcmf_pcie_prepare_fw_request(struct brcmf_pciedev_info *devinfo);
 
 static u32
 brcmf_pcie_read_reg32(struct brcmf_pciedev_info *devinfo, u32 reg_offset)
@@ -1409,6 +1413,36 @@ int brcmf_pcie_get_fwname(struct device *dev, const char *ext, u8 *fw_name)
 	return 0;
 }
 
+static int brcmf_pcie_reset(struct device *dev)
+{
+	struct brcmf_bus *bus_if = dev_get_drvdata(dev);
+	struct brcmf_pciedev *buspub = bus_if->bus_priv.pcie;
+	struct brcmf_pciedev_info *devinfo = buspub->devinfo;
+	struct brcmf_fw_request *fwreq;
+	int err;
+
+	brcmf_detach(dev);
+
+	brcmf_pcie_release_irq(devinfo);
+	brcmf_pcie_release_scratchbuffers(devinfo);
+	brcmf_pcie_release_ringbuffers(devinfo);
+	brcmf_pcie_reset_device(devinfo);
+
+	fwreq = brcmf_pcie_prepare_fw_request(devinfo);
+	if (!fwreq) {
+		dev_err(dev, "Failed to prepare FW request\n");
+		return -ENOMEM;
+	}
+
+	err = brcmf_fw_get_firmwares(dev, fwreq, brcmf_pcie_setup);
+	if (err) {
+		dev_err(dev, "Failed to prepare FW request\n");
+		kfree(fwreq);
+	}
+
+	return err;
+}
+
 static const struct brcmf_bus_ops brcmf_pcie_bus_ops = {
 	.txdata = brcmf_pcie_tx,
 	.stop = brcmf_pcie_down,
@@ -1418,6 +1452,7 @@ static const struct brcmf_bus_ops brcmf_pcie_bus_ops = {
 	.get_ramsize = brcmf_pcie_get_ramsize,
 	.get_memdump = brcmf_pcie_get_memdump,
 	.get_fwname = brcmf_pcie_get_fwname,
+	.reset = brcmf_pcie_reset,
 };
 
 
Signature · 1 line
2.20.1
maintainer Reviewed-by on v1 4/4
#9 lore raw

Re: [PATCH 4/4] brcmfmac: reset PCIe bus on a firmware crash

Quoted 6 lines · Rafał Miłecki

From: Rafał Miłecki <[email protected]>

This includes bus reset & reloading a firmware. It should be sufficient for a user space to (setup and) use a wireless device again.

Support for reset on USB & SDIO can be added later.

It should be noted that you end up with a new wiphy instance though. We could try to avoid that in subsequent change.

Quoted 7 lines
---
  .../broadcom/brcm80211/brcmfmac/bus.h         | 10 ++++++
  .../broadcom/brcm80211/brcmfmac/core.c        | 12 +++++++
  .../broadcom/brcm80211/brcmfmac/core.h        |  2 ++
  .../broadcom/brcm80211/brcmfmac/pcie.c        | 35 +++++++++++++++++++
  4 files changed, 59 insertions(+)