Skip to contentSkip to conversation

ath10k: firmware crash dump

PATCH Superseded Kalle Valo 2 participants 17 messages Aug 8, 2014 – Aug 9, 2014 Qualcomm Atheros ATH10K wireless driver

v5 of 8 Superseded Latest is v8 →

lore
Share & tools
lore

Kalle Valo

author
#0 lore raw

v5 cover ath10k: firmware crash dump

Hi,

here's my reworked Ben's patchset adding firmware crash dump support to ath10k. Unfortunately this crashes when reading the stack dump from the firmware but time run out for me to fix that and I wanted to send this for comments anyway.

I did quite a lot of changes, basically to simplify the code, remove ifdefs and so on. Here's some sort of list what I did:

* dump_data->tv_sec and tv_nsec to 64 bits (because long can be 32 bits on some platforms)

* fix long lines

* renamed ath10k_dbg_save_fw_dbg_buffer() to ath10k_debug_dbglog_add()

* add helpers for ath10k_pci_diag* functions

* refactor and rename ath10k_pci_hif_dump_area()

* latest crash dump is always stored (instead of the oldest unread)

* add ath10k_debug_get_fw_crash_data()

* move fw_r?m_bss_* fields to ar->fw

* struct ath10k_fw_crash_data is allocated with vmalloc()

* atomic allocation in ath10k_pci_dump_bss() is bad, fix that by using vmalloc in module initialisation

* separate FW IE entries for BSS regions

* don't use ath10k_err()

* simplify locking and memory allocation for FW IE handling

* add uuid

* move struct ath10k_dump_file_data and enum ath10k_fw_error_dump_type to debug.c

* function and variable naming, using ath10k_fw_crash_ prefix etc

* change warning and debug messages to follow ath10k style

* add ath10k_debug_get_new_fw_crash_data() to avoid ifdefs in pci.c

And I still have TODO:

* rename crashed_since_read to crashed?

* atomic allocation in ath10k_pci_dump_dbglog() is bad. Should we allocate a big buffer with vmalloc and use that?

* what should ath10k_fw_error_dump_open() do if firmware hasn't crashed? check crashed_since_read and return zero len file? or an error code? -ENOMSG?

* should the crash dump file actually be in little endian? would that be easier/simpler?

* should ath10k_pci_hif_dump_area() hold the lock all the time? That way we would guarantee that changes to ath10k_fw_crash_data are atomic.

---

Ben Greear (5):
ath10k: provide firmware crash info via debugfs
ath10k: save firmware debug log messages
ath10k: save firmware stack upon firmware crash
ath10k: dump exception stack contents on firmware crash
ath10k: save firmware RAM and ROM BSS sections on crash

Kalle Valo (2):
ath10k: add ath10k_pci_diag_* helpers
ath10k: rename ath10k_pci_hif_dump_area() to ath10k_pci_firmware_crashed()

 drivers/net/wireless/ath/ath10k/core.c  |   54 +++++
 drivers/net/wireless/ath/ath10k/core.h  |   43 ++++
 drivers/net/wireless/ath/ath10k/debug.c |  314 +++++++++++++++++++++++++++++++
 drivers/net/wireless/ath/ath10k/debug.h |   22 ++
 drivers/net/wireless/ath/ath10k/hw.h    |   36 ++++
 drivers/net/wireless/ath/ath10k/pci.c   |  250 ++++++++++++++++++++++---
 drivers/net/wireless/ath/ath10k/pci.h   |    3 
 drivers/net/wireless/ath/ath10k/wmi.c   |    6 +
 8 files changed, 701 insertions(+), 27 deletions(-)

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#1 lore raw

PATCH v5 1/7 ath10k: add ath10k_pci_diag_* helpers

ath10k_pci_diag_read32() is for reading u32 from a device and ath10k_pci_diag_read_hi() is a helper for reading data using "host interest" table.

---
 drivers/net/wireless/ath/ath10k/pci.c |   55 +++++++++++++++++++++++----------
 1 file changed, 38 insertions(+), 17 deletions(-)
drivers/net/wireless/ath/ath10k/pci.c +38 -17
index 2d340cc522e3..96ce359349cb 100644
@@ -466,13 +466,46 @@ done:
 	return ret;
 }
 
+static int ath10k_pci_diag_read32(struct ath10k *ar, u32 address, u32 *value)
+{
+	return ath10k_pci_diag_read_mem(ar, address, value, sizeof(u32));
+}
+
+static int __ath10k_pci_diag_read_hi(struct ath10k *ar, void *dest,
+				     u32 src, u32 len)
+{
+	u32 host_addr, addr;
+	int ret;
+
+	host_addr = host_interest_item_address(src);
+
+	ret = ath10k_pci_diag_read32(ar, host_addr, &addr);
+	if (ret != 0) {
+		ath10k_warn("failed to get memcpy hi address for firmware address %d: %d\n",
+			    src, ret);
+		return ret;
+	}
+
+	ret = ath10k_pci_diag_read_mem(ar, addr, dest, len);
+	if (ret != 0) {
+		ath10k_warn("failed to memcpy firmware memory from %d (%d B): %d\n",
+			    addr, len, ret);
+		return ret;
+	}
+
+	return 0;
+}
+
+#define ath10k_pci_diag_read_hi(ar, dest, src, len)		\
+	__ath10k_pci_diag_read_hi(ar, dest, HI_ITEM(src), len);
+
 /* Read 4-byte aligned data from Target memory or register */
 static int ath10k_pci_diag_read_access(struct ath10k *ar, u32 address,
 				       u32 *data)
 {
 	/* Assume range doesn't cross this boundary */
 	if (address >= DRAM_BASE_ADDRESS)
-		return ath10k_pci_diag_read_mem(ar, address, data, sizeof(u32));
+		return ath10k_pci_diag_read32(ar, address, data);
 
 	ath10k_pci_wake(ar);
 	*data = ath10k_pci_read32(ar, address);
@@ -836,9 +869,7 @@ static u16 ath10k_pci_hif_get_free_queue_number(struct ath10k *ar, u8 pipe)
 
 static void ath10k_pci_hif_dump_area(struct ath10k *ar)
 {
-	u32 reg_dump_area = 0;
 	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X] = {};
-	u32 host_addr;
 	int ret;
 	u32 i;
 
@@ -847,21 +878,11 @@ static void ath10k_pci_hif_dump_area(struct ath10k *ar)
 		   ar->hw_params.name, ar->target_version);
 	ath10k_err("firmware version: %s\n", ar->hw->wiphy->fw_version);
 
-	host_addr = host_interest_item_address(HI_ITEM(hi_failure_state));
-	ret = ath10k_pci_diag_read_mem(ar, host_addr,
-				       &reg_dump_area, sizeof(u32));
+	ret = ath10k_pci_diag_read_hi(ar, &reg_dump_values[0],
+				      hi_failure_state,
+				      REG_DUMP_COUNT_QCA988X * sizeof(u32));
 	if (ret) {
-		ath10k_err("failed to read FW dump area address: %d\n", ret);
-		return;
-	}
-
-	ath10k_err("target register Dump Location: 0x%08X\n", reg_dump_area);
-
-	ret = ath10k_pci_diag_read_mem(ar, reg_dump_area,
-				       &reg_dump_values[0],
-				       REG_DUMP_COUNT_QCA988X * sizeof(u32));
-	if (ret != 0) {
-		ath10k_err("failed to read FW dump area: %d\n", ret);
+		ath10k_err("failed to read firmware dump area: %d\n", ret);
 		return;
 	}
 

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#2 lore raw

PATCH v5 2/7 ath10k: provide firmware crash info via debugfs

In reply to Kalle Valo

From: Ben Greear <[email protected]>

Store the firmware crash registers and last 128 or so firmware debug-log ids and present them to user-space via debugfs.

Should help with figuring out why the firmware crashed.

---
 drivers/net/wireless/ath/ath10k/core.h  |   25 +++
 drivers/net/wireless/ath/ath10k/debug.c |  264 +++++++++++++++++++++++++++++++
 drivers/net/wireless/ath/ath10k/debug.h |   22 +++
 drivers/net/wireless/ath/ath10k/hw.h    |   30 ++++
 drivers/net/wireless/ath/ath10k/pci.c   |  133 +++++++++++++++-
 drivers/net/wireless/ath/ath10k/pci.h   |    3 
 6 files changed, 466 insertions(+), 11 deletions(-)
drivers/net/wireless/ath/ath10k/core.h +25 -0
index d5c95d46e841..1243de3eae46 100644
@@ -22,6 +22,7 @@
 #include <linux/if_ether.h>
 #include <linux/types.h>
 #include <linux/pci.h>
+#include <linux/uuid.h>
 
 #include "htt.h"
 #include "htc.h"
@@ -278,6 +279,28 @@ struct ath10k_vif_iter {
 	struct ath10k_vif *arvif;
 };
 
+/* This will store at least the last 128 entries.  Each dbglog message
+ * is a max of 7 32-bit integers in length, but the length can be less
+ * than that as well.
+ */
+#define ATH10K_DBGLOG_DATA_LEN (128 * 7 * sizeof(u32))
+
+struct ath10k_dbglog_entry_storage {
+	/* where to write next chunk of data */
+	u32 next_idx;
+
+	u8 data[ATH10K_DBGLOG_DATA_LEN];
+};
+
+/* used for crash-dump storage, protected by data-lock */
+struct ath10k_fw_crash_data {
+	bool crashed_since_read;
+
+	uuid_le uuid;
+	struct ath10k_dbglog_entry_storage dbglog_entry_data;
+	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
+};
+
 struct ath10k_debug {
 	struct dentry *debugfs_phy;
 
@@ -295,6 +318,8 @@ struct ath10k_debug {
 
 	u8 htt_max_amsdu;
 	u8 htt_max_ampdu;
+
+	struct ath10k_fw_crash_data *fw_crash_data;
 };
 
 enum ath10k_state {
drivers/net/wireless/ath/ath10k/debug.c +264 -0

300 lines not on this page · raw message

drivers/net/wireless/ath/ath10k/debug.h +22 -0
index a5824990bd2a..80ff14e4db9b 100644
@@ -53,6 +53,10 @@ void ath10k_debug_read_service_map(struct ath10k *ar,
 				   size_t map_size);
 void ath10k_debug_read_target_stats(struct ath10k *ar,
 				    struct wmi_stats_event *ev);
+struct ath10k_fw_crash_data *
+ath10k_debug_get_new_fw_crash_data(struct ath10k *ar);
+
+void ath10k_debug_dbglog_add(struct ath10k *ar, u8 *buffer, int len);
 
 #define ATH10K_DFS_STAT_INC(ar, c) (ar->debug.dfs_stats.c++)
 
@@ -86,6 +90,17 @@ static inline void ath10k_debug_read_target_stats(struct ath10k *ar,
 {
 }
 
+static inline void ath10k_debug_dbglog_add(struct ath10k *ar, u8 *buffer,
+					   int len)
+{
+}
+
+static inline struct ath10k_fw_crash_data *
+ath10k_debug_get_new_fw_crash_data(struct ath10k *ar)
+{
+	return NULL;
+}
+
 #define ATH10K_DFS_STAT_INC(ar, c) do { } while (0)
 
 #endif /* CONFIG_ATH10K_DEBUGFS */
@@ -96,6 +111,7 @@ __printf(2, 3) void ath10k_dbg(enum ath10k_debug_mask mask,
 void ath10k_dbg_dump(enum ath10k_debug_mask mask,
 		     const char *msg, const char *prefix,
 		     const void *buf, size_t len);
+
 #else /* CONFIG_ATH10K_DEBUG */
 
 static inline int ath10k_dbg(enum ath10k_debug_mask dbg_mask,
@@ -109,5 +125,11 @@ static inline void ath10k_dbg_dump(enum ath10k_debug_mask mask,
 				   const void *buf, size_t len)
 {
 }
+
+static inline void ath10k_dbg_save_fw_dbg_buffer(struct ath10k *ar,
+						 u8 *buffer, int len)
+{
+}
 #endif /* CONFIG_ATH10K_DEBUG */
+
 #endif /* _DEBUG_H_ */
drivers/net/wireless/ath/ath10k/hw.h +30 -0
index ffd04890407e..c391c88096ee 100644
@@ -39,6 +39,8 @@
 /* includes also the null byte */
 #define ATH10K_FIRMWARE_MAGIC               "QCA-ATH10K"
 
+#define REG_DUMP_COUNT_QCA988X 60
+
 struct ath10k_fw_ie {
 	__le32 id;
 	__le32 len;
@@ -362,4 +364,32 @@ enum ath10k_mcast2ucast_mode {
 
 #define RTC_STATE_V_GET(x) (((x) & RTC_STATE_V_MASK) >> RTC_STATE_V_LSB)
 
+
+/* Target debug log related defines and structs */
+
+/* Target is 32-bit CPU, so we just use u32 for
+ * the pointers.  The memory space is relative to the
+ * target, not the host.
+ */
+struct ath10k_fw_dbglog_buf {
+	/* pointer to dblog_buf_s */
+	u32 next;
+
+	/* pointer to u8 buffer */
+	u32 buffer;
+
+	u32 bufsize;
+	u32 length;
+	u32 count;
+	u32 free;
+} __packed;
+
+struct ath10k_fw_dbglog_hdr {
+	/* pointer to dbglog_buf_s */
+	u32 dbuf;
+
+	u32 dropped;
+} __packed;
+
+
 #endif /* _HW_H_ */
drivers/net/wireless/ath/ath10k/pci.c +125 -8

147 lines not on this page · raw message

drivers/net/wireless/ath/ath10k/pci.h +0 -3
index 940129209990..f72a7cdec4d4 100644
@@ -23,9 +23,6 @@
 #include "hw.h"
 #include "ce.h"
 
-/* FW dump area */
-#define REG_DUMP_COUNT_QCA988X 60
-
 /*
  * maximum number of bytes that can be handled atomically by DiagRead/DiagWrite
  */

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#3 lore raw

Re: [PATCH v5 2/7] ath10k: provide firmware crash info via debugfs

drivers/net/wireless/ath/ath10k/debug.c static const struct file_operations fops_chip_id = { · line 764

Quoted 4 lines · Kalle Valo
drivers/net/wireless/ath/ath10k/debug.c +4 -0
+
+	getnstimeofday(&timestamp);
+	dump_data->tv_sec = timestamp.tv_sec;
+	dump_data->tv_nsec = timestamp.tv_nsec;

I wonder if we should take timestamp at crash time instead of crash-dump-gather time?

drivers/net/wireless/ath/ath10k/debug.c static const struct file_operations fops_dfs_stats = { · line 1175

Quoted 6 lines

int ath10k_debug_create(struct ath10k *ar)
{

drivers/net/wireless/ath/ath10k/debug.c +4 -0
+	ar->debug.fw_crash_data = vzalloc(sizeof(ar->debug.fw_crash_data));
+	if (!ar->debug.fw_crash_data)
+		return -ENOMEM;
+

That sizeof looks quite wrong.

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#4 lore raw

Re: [PATCH v5 2/7] ath10k: provide firmware crash info via debugfs

Quoted 9 lines · Ben Greear

drivers/net/wireless/ath/ath10k/debug.c static const struct file_operations fops_chip_id = { · line 764

Quoted 4 lines · Kalle Valo
drivers/net/wireless/ath/ath10k/debug.c +4 -0
+
+	getnstimeofday(&timestamp);
+	dump_data->tv_sec = timestamp.tv_sec;
+	dump_data->tv_nsec = timestamp.tv_nsec;

I wonder if we should take timestamp at crash time instead of crash-dump-gather time?

Good point, we definitely should. I'll change.

Quoted 8 lines

drivers/net/wireless/ath/ath10k/debug.c static const struct file_operations fops_dfs_stats = { · line 1175

Quoted 6 lines

int ath10k_debug_create(struct ath10k *ar)
{

drivers/net/wireless/ath/ath10k/debug.c +4 -0
+	ar->debug.fw_crash_data = vzalloc(sizeof(ar->debug.fw_crash_data));
+	if (!ar->debug.fw_crash_data)
+		return -ENOMEM;
+

That sizeof looks quite wrong.

Ouch, good catch! Will fix.

Signature · 6 lines
Kalle Valo

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#5 lore raw

PATCH v5 3/7 ath10k: save firmware debug log messages

In reply to Kalle Valo

From: Ben Greear <[email protected]>

They may be dumped through the firmware dump debugfs file.

---
 drivers/net/wireless/ath/ath10k/wmi.c |    6 ++++++
 1 file changed, 6 insertions(+)
drivers/net/wireless/ath/ath10k/wmi.c +6 -0
index fffb15b1b50b..cd79858016f3 100644
@@ -1210,6 +1210,12 @@ static int ath10k_wmi_event_debug_mesg(struct ath10k *ar, struct sk_buff *skb)
 
 	trace_ath10k_wmi_dbglog(skb->data, skb->len);
 
+	/* First 4 bytes are a messages-dropped-due-to-overflow counter,
+	 * and should not be recorded in the dbglog buffer, so we skip
+	 * them.
+	 */
+	ath10k_debug_dbglog_add(ar, skb->data + 4, skb->len - 4);
+
 	return 0;
 }
 

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#6 lore raw

PATCH v5 4/7 ath10k: save firmware stack upon firmware crash

In reply to Kalle Valo

From: Ben Greear <[email protected]>

Should help debug firmware crashes, and give users a way to provide some useful debug reports to firmware developers.

---
 drivers/net/wireless/ath/ath10k/core.h  |    1 +
 drivers/net/wireless/ath/ath10k/debug.c |   11 +++++++++++
 drivers/net/wireless/ath/ath10k/hw.h    |    1 +
 drivers/net/wireless/ath/ath10k/pci.c   |   15 +++++++++++++++
 4 files changed, 28 insertions(+)
drivers/net/wireless/ath/ath10k/core.h +1 -0
index 1243de3eae46..e5f43d6960a9 100644
@@ -299,6 +299,7 @@ struct ath10k_fw_crash_data {
 	uuid_le uuid;
 	struct ath10k_dbglog_entry_storage dbglog_entry_data;
 	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
+	u8 stack_buf[ATH10K_FW_STACK_SIZE];
 };
 
 struct ath10k_debug {
drivers/net/wireless/ath/ath10k/debug.c +11 -0
index 2a443e9557ed..e019850dc969 100644
@@ -31,10 +31,12 @@
  * enum ath10k_fw_error_dump_type - types of data in the dump file
  * @ATH10K_FW_CRASH_DUMP_DBGLOG:  Recent firmware debug log entries
  * @ATH10K_FW_CRASH_DUMP_REGDUMP: Register crash dump in binary format
+ * @ATH10K_FW_CRASH_DUMP_STACK:   Stack memory contents.
  */
 enum ath10k_fw_error_dump_type {
 	ATH10K_FW_CRASH_DUMP_DBGLOG = 0,
 	ATH10K_FW_CRASH_DUMP_REGDUMP = 1,
+	ATH10K_FW_CRASH_DUMP_STACK = 2,
 
 	ATH10K_FW_CRASH_DUMP_MAX,
 };
@@ -712,6 +714,7 @@ static struct ath10k_dump_file_data *ath10k_build_dump_file(struct ath10k *ar)
 	len = hdr_len;
 	len += sizeof(*dump_tlv) + sizeof(crash_data->reg_dump_values);
 	len += sizeof(*dump_tlv) + sizeof(crash_data->dbglog_entry_data);
+	len += sizeof(*dump_tlv) + sizeof(crash_data->stack_buf);
 
 	lockdep_assert_held(&ar->conf_mutex);
 
@@ -781,8 +784,16 @@ static struct ath10k_dump_file_data *ath10k_build_dump_file(struct ath10k *ar)
 	       dump_tlv->tlv_len);
 	sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
 
+	/* Gather firmware stack dump */
+	dump_tlv = (struct ath10k_tlv_dump_data *)(buf + sofar);
+	dump_tlv->type = ATH10K_FW_CRASH_DUMP_STACK;
+	dump_tlv->tlv_len = sizeof(crash_data->stack_buf);
+	memcpy(dump_tlv->tlv_data, crash_data->stack_buf, dump_tlv->tlv_len);
+	sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
+
 	spin_unlock_bh(&ar->data_lock);
 
+	WARN_ON(sofar != len);
 	return dump_data;
 }
 
drivers/net/wireless/ath/ath10k/hw.h +1 -0
index c391c88096ee..9c602b0d7277 100644
@@ -40,6 +40,7 @@
 #define ATH10K_FIRMWARE_MAGIC               "QCA-ATH10K"
 
 #define REG_DUMP_COUNT_QCA988X 60
+#define ATH10K_FW_STACK_SIZE 4096
 
 struct ath10k_fw_ie {
 	__le32 id;
drivers/net/wireless/ath/ath10k/pci.c +15 -0
index 19a556facb1f..016f6641d1f7 100644
@@ -867,6 +867,15 @@ static u16 ath10k_pci_hif_get_free_queue_number(struct ath10k *ar, u8 pipe)
 	return ath10k_ce_num_free_src_entries(ar_pci->pipe_info[pipe].ce_hdl);
 }
 
+/* Save the main firmware stack */
+static void ath10k_pci_dump_stack(struct ath10k *ar,
+				  struct ath10k_fw_crash_data *crash_data)
+{
+	BUILD_BUG_ON(ATH10K_FW_STACK_SIZE % 4);
+	ath10k_pci_diag_read_hi(ar, crash_data->stack_buf,
+				hi_stack, ATH10K_FW_STACK_SIZE);
+}
+
 static void ath10k_pci_dump_dbglog(struct ath10k *ar,
 				   struct ath10k_fw_crash_data *crash_data)
 {
@@ -1010,6 +1019,12 @@ static void ath10k_pci_hif_dump_area(struct ath10k *ar)
 	if (!crash_data)
 		goto exit;
 
+	spin_lock_bh(&ar->data_lock);
+
+	ath10k_pci_dump_stack(ar, crash_data);
+
+	spin_unlock_bh(&ar->data_lock);
+
 	ath10k_pci_dump_registers(ar, crash_data);
 	ath10k_pci_dump_dbglog(ar, crash_data);
 

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#7 lore raw

PATCH v5 5/7 ath10k: dump exception stack contents on firmware crash

In reply to Kalle Valo

From: Ben Greear <[email protected]>

Firmware developers can decode this and maybe figure out why the firmware crashed.

---
 drivers/net/wireless/ath/ath10k/core.h  |    1 +
 drivers/net/wireless/ath/ath10k/debug.c |   11 +++++++++++
 drivers/net/wireless/ath/ath10k/pci.c   |   10 ++++++++++
 3 files changed, 22 insertions(+)
drivers/net/wireless/ath/ath10k/core.h +1 -0
index e5f43d6960a9..dd1a939974c2 100644
@@ -300,6 +300,7 @@ struct ath10k_fw_crash_data {
 	struct ath10k_dbglog_entry_storage dbglog_entry_data;
 	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
 	u8 stack_buf[ATH10K_FW_STACK_SIZE];
+	u8 exc_stack_buf[ATH10K_FW_STACK_SIZE];
 };
 
 struct ath10k_debug {
drivers/net/wireless/ath/ath10k/debug.c +11 -0
index e019850dc969..d605fe54522b 100644
@@ -32,11 +32,13 @@
  * @ATH10K_FW_CRASH_DUMP_DBGLOG:  Recent firmware debug log entries
  * @ATH10K_FW_CRASH_DUMP_REGDUMP: Register crash dump in binary format
  * @ATH10K_FW_CRASH_DUMP_STACK:   Stack memory contents.
+ * @ATH10K_FW_CRASH_DUMP_EXC_STACK:  Exception stack contents
  */
 enum ath10k_fw_error_dump_type {
 	ATH10K_FW_CRASH_DUMP_DBGLOG = 0,
 	ATH10K_FW_CRASH_DUMP_REGDUMP = 1,
 	ATH10K_FW_CRASH_DUMP_STACK = 2,
+	ATH10K_FW_CRASH_DUMP_EXC_STACK = 3,
 
 	ATH10K_FW_CRASH_DUMP_MAX,
 };
@@ -715,6 +717,7 @@ static struct ath10k_dump_file_data *ath10k_build_dump_file(struct ath10k *ar)
 	len += sizeof(*dump_tlv) + sizeof(crash_data->reg_dump_values);
 	len += sizeof(*dump_tlv) + sizeof(crash_data->dbglog_entry_data);
 	len += sizeof(*dump_tlv) + sizeof(crash_data->stack_buf);
+	len += sizeof(*dump_tlv) + sizeof(crash_data->exc_stack_buf);
 
 	lockdep_assert_held(&ar->conf_mutex);
 
@@ -791,6 +794,14 @@ static struct ath10k_dump_file_data *ath10k_build_dump_file(struct ath10k *ar)
 	memcpy(dump_tlv->tlv_data, crash_data->stack_buf, dump_tlv->tlv_len);
 	sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
 
+	/* Gather firmware exception (irq) stack dump */
+	dump_tlv = (struct ath10k_tlv_dump_data *)(buf + sofar);
+	dump_tlv->type = ATH10K_FW_CRASH_DUMP_EXC_STACK;
+	dump_tlv->tlv_len = sizeof(crash_data->exc_stack_buf);
+	memcpy(dump_tlv->tlv_data, &crash_data->exc_stack_buf,
+	       dump_tlv->tlv_len);
+	sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
+
 	spin_unlock_bh(&ar->data_lock);
 
 	WARN_ON(sofar != len);
drivers/net/wireless/ath/ath10k/pci.c +10 -0
index 016f6641d1f7..5e4de6cb6fcf 100644
@@ -876,6 +876,15 @@ static void ath10k_pci_dump_stack(struct ath10k *ar,
 				hi_stack, ATH10K_FW_STACK_SIZE);
 }
 
+/* Save the firmware exception stack */
+static void ath10k_pci_dump_exc_stack(struct ath10k *ar,
+				      struct ath10k_fw_crash_data *crash_data)
+{
+	BUILD_BUG_ON(ATH10K_FW_STACK_SIZE % 4);
+	ath10k_pci_diag_read_hi(ar, crash_data->exc_stack_buf,
+				hi_err_stack, ATH10K_FW_STACK_SIZE);
+}
+
 static void ath10k_pci_dump_dbglog(struct ath10k *ar,
 				   struct ath10k_fw_crash_data *crash_data)
 {
@@ -1022,6 +1031,7 @@ static void ath10k_pci_hif_dump_area(struct ath10k *ar)
 	spin_lock_bh(&ar->data_lock);
 
 	ath10k_pci_dump_stack(ar, crash_data);
+	ath10k_pci_dump_exc_stack(ar, crash_data);
 
 	spin_unlock_bh(&ar->data_lock);
 

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

author
#8 lore raw

PATCH v5 6/7 ath10k: save firmware RAM and ROM BSS sections on crash

In reply to Kalle Valo

From: Ben Greear <[email protected]>

This can be used to get a useful back trace out of a firmware crash that involves an interrupt handler. For instance, a null-pointer-exception would be this kind of trace. A user-space tool can read the debugfs file and decode things as wished.

This requires a packaged firmware with a new IE to describe the BSS section starts and length.

---
 drivers/net/wireless/ath/ath10k/core.c  |   54 +++++++++++++++++++++++++++++++
 drivers/net/wireless/ath/ath10k/core.h  |   16 +++++++++
 drivers/net/wireless/ath/ath10k/debug.c |   28 ++++++++++++++++
 drivers/net/wireless/ath/ath10k/hw.h    |    5 +++
 drivers/net/wireless/ath/ath10k/pci.c   |   39 ++++++++++++++++++++++
 5 files changed, 142 insertions(+)
drivers/net/wireless/ath/ath10k/core.c +54 -0
index 440c3ff03aec..39b7383b395c 100644
@@ -479,6 +479,60 @@ static int ath10k_core_fetch_firmware_api_n(struct ath10k *ar, const char *name)
 			ar->otp_len = ie_len;
 
 			break;
+		case ATH10K_FW_IE_RAM_BSS_ADDR:
+			if (ie_len != sizeof(u32))
+				break;
+
+			ar->fw.ram_bss_addr = le32_to_cpup((__le32 *)data);
+
+			ath10k_dbg(ATH10K_DBG_BOOT,
+				   "found RAM BSS address 0x%x\n",
+				   ar->fw.ram_bss_addr);
+			break;
+		case ATH10K_FW_IE_RAM_BSS_LEN:
+			if (ie_len != sizeof(u32))
+				break;
+
+			ar->fw.ram_bss_len = le32_to_cpup((__le32 *)data);
+
+			ath10k_dbg(ATH10K_DBG_BOOT,
+				   "found RAM BSS length 0x%x\n",
+				   ar->fw.ram_bss_len);
+
+			if (ar->fw.ram_bss_len > ATH10K_RAM_BSS_BUF_LEN) {
+				ath10k_warn("too long firmware RAM BSS length: %d\n",
+					    ar->fw.ram_bss_len);
+				ar->fw.ram_bss_len = 0;
+			}
+
+			break;
+		case ATH10K_FW_IE_ROM_BSS_ADDR:
+			if (ie_len != sizeof(u32))
+				break;
+
+			ar->fw.rom_bss_addr = le32_to_cpup((__le32 *)data);
+
+			ath10k_dbg(ATH10K_DBG_BOOT,
+				   "found ROM BSS address 0x%x\n",
+				   ar->fw.rom_bss_addr);
+			break;
+		case ATH10K_FW_IE_ROM_BSS_LEN:
+			if (ie_len != sizeof(u32))
+				break;
+
+			ar->fw.rom_bss_len = le32_to_cpup((__le32 *)data);
+
+			ath10k_dbg(ATH10K_DBG_BOOT,
+				   "found ROM BSS length 0x%x\n",
+				   ar->fw.rom_bss_len);
+
+			if (ar->fw.ram_bss_len > ATH10K_RAM_BSS_BUF_LEN) {
+				ath10k_warn("too long firmware RAM BSS length: %d\n",
+					    ar->fw.ram_bss_len);
+				ar->fw.rom_bss_len = 0;
+			}
+
+			break;
 		default:
 			ath10k_warn("Unknown FW IE: %u\n",
 				    le32_to_cpu(hdr->id));
drivers/net/wireless/ath/ath10k/core.h +16 -0
index dd1a939974c2..19832ac38685 100644
@@ -292,6 +292,10 @@ struct ath10k_dbglog_entry_storage {
 	u8 data[ATH10K_DBGLOG_DATA_LEN];
 };
 
+/* estimated values, hopefully these are enough */
+#define ATH10K_ROM_BSS_BUF_LEN 10000
+#define ATH10K_RAM_BSS_BUF_LEN 30000
+
 /* used for crash-dump storage, protected by data-lock */
 struct ath10k_fw_crash_data {
 	bool crashed_since_read;
@@ -301,6 +305,9 @@ struct ath10k_fw_crash_data {
 	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
 	u8 stack_buf[ATH10K_FW_STACK_SIZE];
 	u8 exc_stack_buf[ATH10K_FW_STACK_SIZE];
+
+	u8 rom_bss_buf[ATH10K_ROM_BSS_BUF_LEN];
+	u8 ram_bss_buf[ATH10K_RAM_BSS_BUF_LEN];
 };
 
 struct ath10k_debug {
@@ -426,6 +433,15 @@ struct ath10k {
 		} fw;
 	} hw_params;
 
+	/* These are written to only during first firmware boot so no need
+	 * for locking. */
+	struct {
+		unsigned int ram_bss_addr;
+		unsigned int ram_bss_len;
+		unsigned int rom_bss_addr;
+		unsigned int rom_bss_len;
+	} fw;
+
 	const struct firmware *board;
 	const void *board_data;
 	size_t board_len;
drivers/net/wireless/ath/ath10k/debug.c +28 -0
index d605fe54522b..dbdf5f638568 100644
@@ -33,12 +33,16 @@
  * @ATH10K_FW_CRASH_DUMP_REGDUMP: Register crash dump in binary format
  * @ATH10K_FW_CRASH_DUMP_STACK:   Stack memory contents.
  * @ATH10K_FW_CRASH_DUMP_EXC_STACK:  Exception stack contents
+ * @ATH10K_FW_CRASH_DUMP_RAM_BSS:  BSS area for RAM code
+ * @ATH10K_FW_CRASH_DUMP_ROM_BSS:  BSS area for ROM code
  */
 enum ath10k_fw_error_dump_type {
 	ATH10K_FW_CRASH_DUMP_DBGLOG = 0,
 	ATH10K_FW_CRASH_DUMP_REGDUMP = 1,
 	ATH10K_FW_CRASH_DUMP_STACK = 2,
 	ATH10K_FW_CRASH_DUMP_EXC_STACK = 3,
+	ATH10K_FW_CRASH_DUMP_RAM_BSS = 4,
+	ATH10K_FW_CRASH_DUMP_ROM_BSS = 5,
 
 	ATH10K_FW_CRASH_DUMP_MAX,
 };
@@ -719,6 +723,12 @@ static struct ath10k_dump_file_data *ath10k_build_dump_file(struct ath10k *ar)
 	len += sizeof(*dump_tlv) + sizeof(crash_data->stack_buf);
 	len += sizeof(*dump_tlv) + sizeof(crash_data->exc_stack_buf);
 
+	if (ar->fw.ram_bss_addr && ar->fw.ram_bss_len)
+		len += sizeof(*dump_tlv) + ar->fw.ram_bss_len;
+
+	if (ar->fw.rom_bss_addr && ar->fw.rom_bss_len)
+		len += sizeof(*dump_tlv) + ar->fw.rom_bss_len;
+
 	lockdep_assert_held(&ar->conf_mutex);
 
 	sofar += hdr_len;
@@ -802,6 +812,24 @@ static struct ath10k_dump_file_data *ath10k_build_dump_file(struct ath10k *ar)
 	       dump_tlv->tlv_len);
 	sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
 
+	if (ar->fw.ram_bss_addr && ar->fw.ram_bss_len) {
+		dump_tlv = (struct ath10k_tlv_dump_data *)(buf + sofar);
+		dump_tlv->type = ATH10K_FW_CRASH_DUMP_RAM_BSS;
+		dump_tlv->tlv_len = ar->fw.ram_bss_len;
+		memcpy(dump_tlv->tlv_data, crash_data->ram_bss_buf,
+		       dump_tlv->tlv_len);
+		sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
+	}
+
+	if (ar->fw.rom_bss_addr && ar->fw.rom_bss_len) {
+		dump_tlv = (struct ath10k_tlv_dump_data *)(buf + sofar);
+		dump_tlv->type = ATH10K_FW_CRASH_DUMP_ROM_BSS;
+		dump_tlv->tlv_len = ar->fw.rom_bss_len;
+		memcpy(dump_tlv->tlv_data, crash_data->rom_bss_buf,
+		       dump_tlv->tlv_len);
+		sofar += sizeof(*dump_tlv) + dump_tlv->tlv_len;
+	}
+
 	spin_unlock_bh(&ar->data_lock);
 
 	WARN_ON(sofar != len);
drivers/net/wireless/ath/ath10k/hw.h +5 -0
index 9c602b0d7277..6c275638ddaa 100644
@@ -54,6 +54,11 @@ enum ath10k_fw_ie_type {
 	ATH10K_FW_IE_FEATURES = 2,
 	ATH10K_FW_IE_FW_IMAGE = 3,
 	ATH10K_FW_IE_OTP_IMAGE = 4,
+	ATH10K_FW_IE_RAM_BSS_ADDR = 5,
+	ATH10K_FW_IE_RAM_BSS_LEN = 6,
+	ATH10K_FW_IE_ROM_BSS_ADDR = 7,
+	ATH10K_FW_IE_ROM_BSS_LEN = 8,
+
 };
 
 /* Known pecularities:
drivers/net/wireless/ath/ath10k/pci.c +39 -0
index 5e4de6cb6fcf..a65d45c78542 100644
@@ -885,6 +885,43 @@ static void ath10k_pci_dump_exc_stack(struct ath10k *ar,
 				hi_err_stack, ATH10K_FW_STACK_SIZE);
 }
 
+static void ath10k_pci_dump_bss_ram(struct ath10k *ar,
+				    struct ath10k_fw_crash_data *crash_data) {
+	int ret;
+
+	if (!ar->fw.ram_bss_addr)
+		return;
+
+	if (!ar->fw.ram_bss_len)
+		return;
+
+	ret = ath10k_pci_diag_read_mem(ar, ar->fw.ram_bss_addr,
+				       crash_data->ram_bss_buf,
+				       ar->fw.ram_bss_len);
+	if (ret)
+		ath10k_warn("failed to read firmware RAM BSS memory from %d (%d B): %d\n",
+			    ar->fw.ram_bss_addr, ar->fw.ram_bss_len, ret);
+}
+
+static void ath10k_pci_dump_bss_rom(struct ath10k *ar,
+				    struct ath10k_fw_crash_data *crash_data)
+{
+	int ret;
+
+	if (!ar->fw.rom_bss_addr)
+		return;
+
+	if (!ar->fw.rom_bss_len)
+		return;
+
+	ret = ath10k_pci_diag_read_mem(ar, ar->fw.rom_bss_addr,
+				       crash_data->rom_bss_buf,
+				       ar->fw.rom_bss_len);
+	if (ret)
+		ath10k_warn("failed to read firmware ROM BSS memory from %d (%d B): %d\n",
+			    ar->fw.rom_bss_addr, ar->fw.rom_bss_len, ret);
+}
+
 static void ath10k_pci_dump_dbglog(struct ath10k *ar,
 				   struct ath10k_fw_crash_data *crash_data)
 {
@@ -1032,6 +1069,8 @@ static void ath10k_pci_hif_dump_area(struct ath10k *ar)
 
 	ath10k_pci_dump_stack(ar, crash_data);
 	ath10k_pci_dump_exc_stack(ar, crash_data);
+	ath10k_pci_dump_bss_ram(ar, crash_data);
+	ath10k_pci_dump_bss_rom(ar, crash_data);
 
 	spin_unlock_bh(&ar->data_lock);
 

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#9 lore raw

Re: [PATCH v5 6/7] ath10k: save firmware RAM and ROM BSS sections on crash

Quoted 9 lines · Kalle Valo
drivers/net/wireless/ath/ath10k/core.h +4 -0
+/* estimated values, hopefully these are enough */
+#define ATH10K_ROM_BSS_BUF_LEN 10000
+#define ATH10K_RAM_BSS_BUF_LEN 30000
+
 /* used for crash-dump storage, protected by data-lock */
 struct ath10k_fw_crash_data {
 	bool crashed_since_read;

drivers/net/wireless/ath/ath10k/core.h struct ath10k_fw_crash_data { · line 311

Quoted 7 lines
drivers/net/wireless/ath/ath10k/core.h +3 -0
@@ -301,6 +305,9 @@ struct ath10k_fw_crash_data {
 	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
 	u8 stack_buf[ATH10K_FW_STACK_SIZE];
 	u8 exc_stack_buf[ATH10K_FW_STACK_SIZE];
+
+	u8 rom_bss_buf[ATH10K_ROM_BSS_BUF_LEN];
+	u8 ram_bss_buf[ATH10K_RAM_BSS_BUF_LEN];
 };

That (using estimates instead of allocating memory when we know the true value and/or when we need it)
is wasting quite a bit of RAM. Doesn't matter on my systems, but AP manufacturers might be more ticklish about RAM usage...

drivers/net/wireless/ath/ath10k/core.h struct ath10k { · line 437

Quoted 2 lines
drivers/net/wireless/ath/ath10k/core.h +2 -0
+	/* These are written to only during first firmware boot so no need
+	 * for locking. */

It's 'firmware load' instead of boot maybe?

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#10 lore raw

Re: [PATCH v5 6/7] ath10k: save firmware RAM and ROM BSS sections on crash

Quoted 14 lines · Ben Greear
Quoted 13 lines · Kalle Valo
drivers/net/wireless/ath/ath10k/core.h +5 -0
+/* estimated values, hopefully these are enough */
+#define ATH10K_ROM_BSS_BUF_LEN 10000
+#define ATH10K_RAM_BSS_BUF_LEN 30000
+
 /* used for crash-dump storage, protected by data-lock */
 struct ath10k_fw_crash_data {
 	bool crashed_since_read;
@@ -301,6 +305,9 @@ struct ath10k_fw_crash_data {
 	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
 	u8 stack_buf[ATH10K_FW_STACK_SIZE];
 	u8 exc_stack_buf[ATH10K_FW_STACK_SIZE];
+
Quoted 8 lines

drivers/net/wireless/ath/ath10k/core.h struct ath10k_fw_crash_data { · line 311

Quoted 3 lines
drivers/net/wireless/ath/ath10k/core.h +2 -0
+	u8 rom_bss_buf[ATH10K_ROM_BSS_BUF_LEN];
+	u8 ram_bss_buf[ATH10K_RAM_BSS_BUF_LEN];
 };

That (using estimates instead of allocating memory when we know the true value and/or when we need it) is wasting quite a bit of RAM. Doesn't matter on my systems, but AP manufacturers might be more ticklish about RAM usage...

Yeah, that is a problem bu this can be avoided by disabling CONFIG_ATH10K_DEBUGFS altogether. But on the other hand, there might be people who would still prefer to enable debugfs but not waste RAM on firmware crash dumps. For those we could add a module parameter or similar to disable firmware crash dumps, but is that overengineering already?

Another option is that we do firmware crash memory allocation on the fly, when the crash happens. But as we are in atomic context it's pretty expensive. We cannot do huge kmalloc() call due to memory fragmentation and I doubt __vmalloc() is acceptable to call in atomic contexts (at least I didn't find use of that in the kernel).

Third option is that we allocate crash dump buffers after the firmware has been loaded and FW IEs are read. That way we don't allocate any extra memory but the fundamental problem still persists.

Quoted 4 lines

drivers/net/wireless/ath/ath10k/core.h struct ath10k { · line 437

Quoted 2 lines
drivers/net/wireless/ath/ath10k/core.h +2 -0
+	/* These are written to only during first firmware boot so no need
+	 * for locking. */

It's 'firmware load' instead of boot maybe?

Correct, I'll change.

Signature · 6 lines
Kalle Valo

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#11 lore raw

Re: [PATCH v5 6/7] ath10k: save firmware RAM and ROM BSS sections on crash

Quoted 38 lines · Kalle Valo
Quoted 14 lines · Ben Greear
Quoted 13 lines · Kalle Valo
drivers/net/wireless/ath/ath10k/core.h +5 -0
+/* estimated values, hopefully these are enough */
+#define ATH10K_ROM_BSS_BUF_LEN 10000
+#define ATH10K_RAM_BSS_BUF_LEN 30000
+
  /* used for crash-dump storage, protected by data-lock */
  struct ath10k_fw_crash_data {
  	bool crashed_since_read;
@@ -301,6 +305,9 @@ struct ath10k_fw_crash_data {
  	u32 reg_dump_values[REG_DUMP_COUNT_QCA988X];
  	u8 stack_buf[ATH10K_FW_STACK_SIZE];
  	u8 exc_stack_buf[ATH10K_FW_STACK_SIZE];
+
Quoted 8 lines

drivers/net/wireless/ath/ath10k/core.h struct ath10k_fw_crash_data { · line 311

Quoted 3 lines
drivers/net/wireless/ath/ath10k/core.h +2 -0
+	u8 rom_bss_buf[ATH10K_ROM_BSS_BUF_LEN];
+	u8 ram_bss_buf[ATH10K_RAM_BSS_BUF_LEN];
  };

That (using estimates instead of allocating memory when we know the true value and/or when we need it) is wasting quite a bit of RAM. Doesn't matter on my systems, but AP manufacturers might be more ticklish about RAM usage...

Yeah, that is a problem bu this can be avoided by disabling CONFIG_ATH10K_DEBUGFS altogether. But on the other hand, there might be people who would still prefer to enable debugfs but not waste RAM on firmware crash dumps. For those we could add a module parameter or similar to disable firmware crash dumps, but is that overengineering already?

Another option is that we do firmware crash memory allocation on the fly, when the crash happens. But as we are in atomic context it's pretty expensive. We cannot do huge kmalloc() call due to memory fragmentation and I doubt __vmalloc() is acceptable to call in atomic contexts (at least I didn't find use of that in the kernel).

Quoted 3 lines

Third option is that we allocate crash dump buffers after the firmware has been loaded and FW IEs are read. That way we don't allocate any extra memory but the fundamental problem still persists.

I'd prefer this third option..seems more elegant all around.

I don't think you can do vmalloc in atomic context, btw. But, can do it on firmware load.

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#12 lore raw

Re: [PATCH v5 6/7] ath10k: save firmware RAM and ROM BSS sections on crash

Quoted 8 lines · Ben Greear
Quoted 3 lines

Third option is that we allocate crash dump buffers after the firmware has been loaded and FW IEs are read. That way we don't allocate any extra memory but the fundamental problem still persists.

I'd prefer this third option..seems more elegant all around.

I don't think you can do vmalloc in atomic context, btw. But, can do it on firmware load.

Ok, I'll look how to implement this.

Signature · 6 lines
Kalle Valo

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#13 lore raw

PATCH v5 7/7 ath10k: rename ath10k_pci_hif_dump_area() to ath10k_pci_firmware_crashed()

In reply to Kalle Valo

Better to have a clear name for the function.

---
 drivers/net/wireless/ath/ath10k/pci.c |    8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
drivers/net/wireless/ath/ath10k/pci.c +4 -4
index a65d45c78542..fe7159c3c8b1 100644
@@ -1045,7 +1045,7 @@ static void ath10k_pci_dump_registers(struct ath10k *ar,
 }
 
 
-static void ath10k_pci_hif_dump_area(struct ath10k *ar)
+static void ath10k_pci_firmware_crashed(struct ath10k *ar)
 {
 	struct ath10k_fw_crash_data *crash_data;
 	char uuid[20];
@@ -2000,7 +2000,7 @@ static void ath10k_pci_fw_interrupt_handler(struct ath10k *ar)
 				   fw_indicator & ~FW_IND_EVENT_PENDING);
 
 		if (ar_pci->started) {
-			ath10k_pci_hif_dump_area(ar);
+			ath10k_pci_firmware_crashed(ar);
 		} else {
 			/*
 			 * Probable Target failure before we're prepared
@@ -2451,7 +2451,7 @@ static void ath10k_pci_early_irq_tasklet(unsigned long data)
 	if (fw_ind & FW_IND_EVENT_PENDING) {
 		ath10k_pci_write32(ar, FW_INDICATOR_ADDRESS,
 				   fw_ind & ~FW_IND_EVENT_PENDING);
-		ath10k_pci_hif_dump_area(ar);
+		ath10k_pci_firmware_crashed(ar);
 	}
 
 	ath10k_pci_sleep(ar);
@@ -2730,7 +2730,7 @@ static int ath10k_pci_wait_for_target_init(struct ath10k *ar)
 		ath10k_warn("device has crashed during init\n");
 		ath10k_pci_write32(ar, FW_INDICATOR_ADDRESS,
 				   val & ~FW_IND_EVENT_PENDING);
-		ath10k_pci_hif_dump_area(ar);
+		ath10k_pci_firmware_crashed(ar);
 		ret = -ECOMM;
 		goto out;
 	}

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k

#14 lore raw
In reply to Kalle Valo
Quoted 11 lines · Kalle Valo

Hi,

here's my reworked Ben's patchset adding firmware crash dump support to ath10k. Unfortunately this crashes when reading the stack dump from the firmware but time run out for me to fix that and I wanted to send this for comments anyway.

I did quite a lot of changes, basically to simplify the code, remove ifdefs and so on. Here's some sort of list what I did:

* dump_data->tv_sec and tv_nsec to 64 bits (because long can be 32 bits on some platforms)

I did u32 on purpose because I know how to do ntohl, htonl if I need to flip the machine order in user-space. Is there something similar for 64-bit numbers?

Quoted 9 lines

* fix long lines

* renamed ath10k_dbg_save_fw_dbg_buffer() to ath10k_debug_dbglog_add()

* add helpers for ath10k_pci_diag* functions

* refactor and rename ath10k_pci_hif_dump_area()

* latest crash dump is always stored (instead of the oldest unread)

At least with the kernel, the first crash is normally the most useful, so I figured the same would be true of the firmware (a firmware crash is excellent way to find hidden bugs in the driver, so upon crash/reload, it is more likely that the driver will be screwed up and thus mis-configure firmware...)

Quoted 24 lines

* add ath10k_debug_get_fw_crash_data()

* move fw_r?m_bss_* fields to ar->fw

* struct ath10k_fw_crash_data is allocated with vmalloc()

* atomic allocation in ath10k_pci_dump_bss() is bad, fix that by using vmalloc in module initialisation

* separate FW IE entries for BSS regions

* don't use ath10k_err()

* simplify locking and memory allocation for FW IE handling

* add uuid

* move struct ath10k_dump_file_data and enum ath10k_fw_error_dump_type to debug.c

* function and variable naming, using ath10k_fw_crash_ prefix etc

* change warning and debug messages to follow ath10k style

Quoted 8 lines

* add ath10k_debug_get_new_fw_crash_data() to avoid ifdefs in pci.c

And I still have TODO:

* rename crashed_since_read to crashed?

* atomic allocation in ath10k_pci_dump_dbglog() is bad. Should we allocate a big buffer with vmalloc and use that?

dbglog entries are probably never going to be large, they are currently in the 2k range, so vmalloc is likely overkill. If you want it allocated at startup, just choose a 4k buffer size, can put the buffer right in the debug struct or something like that so you don't even have more memory management to deal with. It will waste 4k of RAM for normal use, however.

Quoted 4 lines

* what should ath10k_fw_error_dump_open() do if firmware hasn't crashed? check crashed_since_read and return zero len file? or an error code? -ENOMSG?

Maybe a mostly empty crash log with just the header, should be easy enough to figure out it's not really a crash. If we use udev to gather these, it is likely to be stupid shell scripts doing a lot of the work, so being clever on return codes might not be helpful.

Quoted 2 lines

* should the crash dump file actually be in little endian? would that be easier/simpler?

I think it is about the same amount of work in user-space, so I'd keep the kernel simple and dump in the CPU's endian-ness. I don't care that much either way, however.

Quoted 3 lines

* should ath10k_pci_hif_dump_area() hold the lock all the time? That way we would guarantee that changes to ath10k_fw_crash_data are atomic.

I don't think it's worth trying to do that.

I'll go dig through the patches in a bit..have to figure out why systemd is fcked up on my lab machine first. :P

Thanks,
Ben

Signature · 8 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com


_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
author
#15 lore raw
Quoted 7 lines · Ben Greear
Quoted 6 lines · Kalle Valo

Hi,

here's my reworked Ben's patchset adding firmware crash dump support to ath10k. Unfortunately this crashes when reading the stack dump from the firmware but time run out for me to fix that and I wanted to send this for comments anyway.

Quoted 8 lines
Quoted 5 lines

I did quite a lot of changes, basically to simplify the code, remove ifdefs and so on. Here's some sort of list what I did:

* dump_data->tv_sec and tv_nsec to 64 bits (because long can be 32 bits on some platforms)

I did u32 on purpose because I know how to do ntohl, htonl if I need to flip the machine order in user-space.

On my 32-bit box I got a compiler warning for this and that's why I changed it to u64.

Quoted 1 line

Is there something similar for 64-bit numbers?

At least I haven't heard anything. But it should be easy to implement on your own.

Quoted 7 lines
Quoted 1 line

* latest crash dump is always stored (instead of the oldest unread)

At least with the kernel, the first crash is normally the most useful, so I figured the same would be true of the firmware (a firmware crash is excellent way to find hidden bugs in the driver, so upon crash/reload, it is more likely that the driver will be screwed up and thus mis-configure firmware...)

Think of an AP with 3 month uptime where firmware crashes once in June and second time in August and when you retrieve the crash dump you get the one from June. I find that very unintuitive.

I think in your use case we should have a tool which stores all crash dumps based on events from the kernel, that way you have access to all of them. But for the simple case we should just provide the latest dump.

Quoted 5 lines
Quoted 2 lines

* atomic allocation in ath10k_pci_dump_dbglog() is bad. Should we allocate a big buffer with vmalloc and use that?

dbglog entries are probably never going to be large, they are currently in the 2k range, so vmalloc is likely overkill.

But what would be the downside from using vmalloc? This is not in hotpath or anything like that. With vmalloc we don't need to worry about using precious unfragmented memory.

Quoted 4 lines

If you want it allocated at startup, just choose a 4k buffer size, can put the buffer right in the debug struct or something like that so you don't even have more memory management to deal with. It will waste 4k of RAM for normal use, however.

Ok. But I think we can leave this as is and fix it later.

Quoted 8 lines
Quoted 3 lines

* what should ath10k_fw_error_dump_open() do if firmware hasn't crashed? check crashed_since_read and return zero len file? or an error code? -ENOMSG?

Maybe a mostly empty crash log with just the header, should be easy enough to figure out it's not really a crash. If we use udev to gather these, it is likely to be stupid shell scripts doing a lot of the work, so being clever on return codes might not be helpful.

But with cp in that case you also get an empty file, right? That way it's easy to detect that there's no crash dump available.

What is the benefit from providing an empty header to user space? I don't get that.

Quoted 6 lines
Quoted 2 lines

* should the crash dump file actually be in little endian? would that be easier/simpler?

I think it is about the same amount of work in user-space, so I'd keep the kernel simple and dump in the CPU's endian-ness. I don't care that much either way, however.

I don't care much either. But IIRC Michal suggested something like that.

Quoted 5 lines
Quoted 3 lines

* should ath10k_pci_hif_dump_area() hold the lock all the time? That way we would guarantee that changes to ath10k_fw_crash_data are atomic.

I don't think it's worth trying to do that.

Thought more about this and I think we need it. Otherwise we have a theoretical race that uuid and the dumpt content are from separate crashes. I'll take a look if there's anything preventing do all the dumps in atomic context.

Quoted 2 lines

I'll go dig through the patches in a bit..have to figure out why systemd is fcked up on my lab machine first. :P

But wasn't systemd supposed to fix all of our problems? ;)

Signature · 6 lines
Kalle Valo

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k
#16 lore raw
Quoted 12 lines · Kalle Valo
Quoted 10 lines · Ben Greear
Quoted 9 lines · Kalle Valo

Hi,

here's my reworked Ben's patchset adding firmware crash dump support to ath10k. Unfortunately this crashes when reading the stack dump from the firmware but time run out for me to fix that and I wanted to send this for comments anyway.

I did quite a lot of changes, basically to simplify the code, remove ifdefs and so on. Here's some sort of list what I did:

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

* dump_data->tv_sec and tv_nsec to 64 bits (because long can be 32 bits on some platforms)

I did u32 on purpose because I know how to do ntohl, htonl if I need to flip the machine order in user-space.

On my 32-bit box I got a compiler warning for this and that's why I changed it to u64.

Casting could have fixed that I imagine.

Quoted 4 lines
Quoted 1 line

Is there something similar for 64-bit numbers?

At least I haven't heard anything. But it should be easy to implement on your own.

Sure...it's not a big deal, likely there will be at most two different apps that ever decode this anyway (mine and qca's).

Quoted 11 lines
Quoted 7 lines
Quoted 1 line

* latest crash dump is always stored (instead of the oldest unread)

At least with the kernel, the first crash is normally the most useful, so I figured the same would be true of the firmware (a firmware crash is excellent way to find hidden bugs in the driver, so upon crash/reload, it is more likely that the driver will be screwed up and thus mis-configure firmware...)

Think of an AP with 3 month uptime where firmware crashes once in June and second time in August and when you retrieve the crash dump you get the one from June. I find that very unintuitive.

AP should have gathered the crash the first time, but again, not a big deal, as I would gather the crash logs as soon as they happen anyway.

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

* what should ath10k_fw_error_dump_open() do if firmware hasn't crashed? check crashed_since_read and return zero len file? or an error code? -ENOMSG?

Maybe a mostly empty crash log with just the header, should be easy enough to figure out it's not really a crash. If we use udev to gather these, it is likely to be stupid shell scripts doing a lot of the work, so being clever on return codes might not be helpful.

Quoted 5 lines

But with cp in that case you also get an empty file, right? That way it's easy to detect that there's no crash dump available.

What is the benefit from providing an empty header to user space? I don't get that.

An empty file could be any sort of error or mis-configuration ...but an empty header means the infrastructure is working properly.

I think with all of this, it mostly a matter of opinion, and whatever you choose should be fine with me. If we find any serious issues, we can of course fix it later.

Quoted 4 lines
Quoted 2 lines

I'll go dig through the patches in a bit..have to figure out why systemd is fcked up on my lab machine first. :P

But wasn't systemd supposed to fix all of our problems? ;)

On Fedora 19, the latest update forgot how to enable/disable programs. It must have a zero-regression-test policy as well as it's other issues :P

Thanks,
Ben

Signature · 7 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

_______________________________________________
ath10k mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/ath10k