Skip to contentSkip to conversation

CXL: Process event logs

RFC PATCH Superseded [email protected] 5 participants 36 messages Oct 10, 2022 – Oct 21, 2022

Subsystems (2)

v2 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

v2 cover CXL: Process event logs

From: Ira Weiny <[email protected]>

Changes from RFC v1
Add event irqs
General simplification of the code. Resolve field alignment questions Update to rev 3.0 for comments and structures Add reserved fields and output them

Event records inform the OS of various device events. Events are not needed for any kernel operation but various user level software will want to track events.

Add event reporting through the trace event mechanism. On driver load read and clear all device events.

Enable all event logs for interrupts and process each log on interrupt.

Testing of this was performed with additions to QEMU posted here:

	https://lore.kernel.org/all/[email protected]/

Ira Weiny (11):
cxl/mbox: Add debug of hardware error code
cxl/mem: Implement Get Event Records command
cxl/mem: Implement Clear Event Records command
cxl/mem: Clear events on driver load
cxl/mem: Trace General Media Event Record
cxl/mem: Trace DRAM Event Record
cxl/mem: Trace Memory Module Event Record
cxl/test: Add generic mock events
cxl/test: Add specific events
cxl/test: Simulate event log overflow
cxl/mem: Wire up event interrupts

 MAINTAINERS                     |   1 +
 drivers/cxl/core/mbox.c         | 186 ++++++++++++-
 drivers/cxl/cxlmem.h            | 193 +++++++++++++
 drivers/cxl/pci.c               | 154 ++++++++++
 include/trace/events/cxl.h      | 478 ++++++++++++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h    |   4 +
 tools/testing/cxl/test/Kbuild   |   2 +-
 tools/testing/cxl/test/events.c | 329 ++++++++++++++++++++++
 tools/testing/cxl/test/events.h |   9 +
 tools/testing/cxl/test/mem.c    |  34 +++
 10 files changed, 1388 insertions(+), 2 deletions(-)
 create mode 100644 include/trace/events/cxl.h
 create mode 100644 tools/testing/cxl/test/events.c
 create mode 100644 tools/testing/cxl/test/events.h

base-commit: e2302539dd4f1c62d96651c07ddb05aa2461d29c

Signature · 1 line
2.37.2
#1 lore raw

RFC PATCH v2 1/11 cxl/mbox: Add debug of hardware error code

From: Ira Weiny <[email protected]>

If a mailbox command fails the driver always reports ENXIO. But this may not be enough information to understand why the hardware, or in my case Qemu, was failing.

Add a debug print of the error code returned from the hardware.

---
 drivers/cxl/core/mbox.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/core/mbox.c +4 -1
index 16176b9278b4..6c4d024ad0e8 100644
@@ -181,8 +181,11 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,
 	if (rc)
 		return rc;
 
-	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS)
+	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS) {
+		dev_dbg(cxlds->dev, "MB error : %s\n",
+			cxl_mbox_cmd_rc2str(&mbox_cmd));
 		return cxl_mbox_cmd_rc2errno(&mbox_cmd);
+	}
 
 	/*
 	 * Variable sized commands can't be validated and so it's up to the
Signature · 1 line
2.37.2
Reviewed-by on v2 1/11
#2 lore raw

Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code

Quoted 9 lines · On Mon, 10 Oct 2022 15:41:21 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

If a mailbox command fails the driver always reports ENXIO. But this may not be enough information to understand why the hardware, or in my case Qemu, was failing.

Add a debug print of the error code returned from the hardware.

Seems very sensible to me.

Quoted 21 lines
---
 drivers/cxl/core/mbox.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/core/mbox.c +4 -1
index 16176b9278b4..6c4d024ad0e8 100644
@@ -181,8 +181,11 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,
 	if (rc)
 		return rc;
 
-	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS)
+	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS) {
+		dev_dbg(cxlds->dev, "MB error : %s\n",
+			cxl_mbox_cmd_rc2str(&mbox_cmd));
 		return cxl_mbox_cmd_rc2errno(&mbox_cmd);
+	}
 
 	/*
 	 * Variable sized commands can't be validated and so it's up to the
maintainer Reviewed-by on v2 1/11
#3 lore raw

Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code

In reply to [email protected]
Quoted 7 lines · On Mon, 10 Oct 2022, [email protected] wrote:

From: Ira Weiny <[email protected]>

If a mailbox command fails the driver always reports ENXIO. But this may not be enough information to understand why the hardware, or in my case Qemu, was failing.

Add a debug print of the error code returned from the hardware.

with a nit below.

Quoted 11 lines

---
drivers/cxl/core/mbox.c | 5 ++++-

1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/core/mbox.c +0 -0
index 16176b9278b4..6c4d024ad0e8 100644

drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 185

Quoted 6 lines
drivers/cxl/core/mbox.c +2 -1
@@ -181,8 +181,11 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,
 	if (rc)
 		return rc;
 
-	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS)
+	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS) {
+		dev_dbg(cxlds->dev, "MB error : %s\n",

Maybe s/MB/mbox?

Quoted 9 lines
drivers/cxl/core/mbox.c +2 -1
+			cxl_mbox_cmd_rc2str(&mbox_cmd));
 		return cxl_mbox_cmd_rc2errno(&mbox_cmd);
+	}
 
 	/*
 	 * Variable sized commands can't be validated and so it's up to the
--

2.37.2

maintainer
#4 lore raw

Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code

Quoted 5 lines · Davidlohr Bueso

drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 185

Quoted 3 lines
drivers/cxl/core/mbox.c +2 -1
-	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS)
+	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS) {
+		dev_dbg(cxlds->dev, "MB error : %s\n",

Maybe s/MB/mbox?

Actually 'Mailbox' seems to be the standard:

core/regs.c: dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset);
core/regs.c: dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset);
pci.c: dev_dbg(dev, "Mailbox operation had an error\n");
pci.c: dev_err(cxlds->dev, "Mailbox is too small (%zub)",
pci.c: dev_dbg(cxlds->dev, "Mailbox payload sized %zu",

author
#5 lore raw

Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code

Quoted 8 lines · Davidlohr Bueso
Quoted 5 lines · Davidlohr Bueso

drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 185

Quoted 3 lines
drivers/cxl/core/mbox.c +2 -1
-	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS)
+	if (mbox_cmd.return_code != CXL_MBOX_CMD_RC_SUCCESS) {
+		dev_dbg(cxlds->dev, "MB error : %s\n",

Maybe s/MB/mbox?

Actually 'Mailbox' seems to be the standard:

Good point! Changed.

Thanks!
Ira

Quoted 6 lines

core/regs.c: dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset);
core/regs.c: dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset);
pci.c: dev_dbg(dev, "Mailbox operation had an error\n");
pci.c: dev_err(cxlds->dev, "Mailbox is too small (%zub)",
pci.c: dev_dbg(cxlds->dev, "Mailbox payload sized %zu",

#6 lore raw

RFC PATCH v2 2/11 cxl/mem: Implement Get Event Records command

In reply to [email protected]

From: Ira Weiny <[email protected]>

Event records are defined for CXL devices. Each record is reported in one event log. Devices are required to support the storage of at least one event record in each event log type.

Devices track event log overflow by incrementing a counter and tracking the time of the first and last overflow event seen.

Software queries events via the Get Event Record mailbox command; CXL rev 3.0 section 8.2.9.2.2.

Issue the Get Event Record mailbox command on driver load. Trace each record found, as well as any overflow conditions. Only 1 event is requested for each query. Optimization of multiple record queries is deferred.

This patch traces a raw event record only and leaves the specific event record types to subsequent patches.

Macros are created to use for the common CXL Event header fields.

Cc Steven Rostedt <[email protected]>

---
Change from RFC:
Remove redundant error message in get event records loop s/EVENT_RECORD_DATA_LENGTH/CXL_EVENT_RECORD_DATA_LENGTH Use hdr_uuid for the header UUID field
Use cxl_event_log_type_str() for the trace events
Create macros for the header fields and common entries of each event Add reserved buffer output dump
Report error if event query fails
Remove unused record_cnt variable
Steven - reorder overflow record
Remove NOTE about checkpatch
Jonathan
check for exactly 1 record
s/v3.0/rev 3.0
Use 3 byte fields for 24bit fields
Add 3.0 Maintenance Operation Class
Add Dynamic Capacity log type
Fix spelling
Dave Jiang/Dan/Alison
s/cxl-event/cxl
trace/events/cxl-events => trace/events/cxl.h s/cxl_event_overflow/overflow s/cxl_event/generic_event

---
 MAINTAINERS                  |   1 +
 drivers/cxl/core/mbox.c      |  53 ++++++++++++++
 drivers/cxl/cxlmem.h         |  75 ++++++++++++++++++++
 include/trace/events/cxl.h   | 130 +++++++++++++++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h |   1 +
 5 files changed, 260 insertions(+)
 create mode 100644 include/trace/events/cxl.h
MAINTAINERS +1 -0
index 7547ffce5032..49bec9cc8bda 100644
@@ -5219,6 +5219,7 @@ M:	Dan Williams <[email protected]>
 L:	[email protected]
 S:	Maintained
 F:	drivers/cxl/
+F:	include/trace/events/cxl.h
 F:	include/uapi/linux/cxl_mem.h
 
 CONEXANT ACCESSRUNNER USB DRIVER
drivers/cxl/core/mbox.c +53 -0

71 lines not on this page · raw message

drivers/cxl/cxlmem.h +75 -0

99 lines not on this page · raw message

include/trace/events/cxl.h new file +130 -0

130 lines not on this page · raw message

include/uapi/linux/cxl_mem.h +1 -0
index c71021a2a9ed..70459be5bdd4 100644
@@ -24,6 +24,7 @@
 	___C(IDENTIFY, "Identify Command"),                               \
 	___C(RAW, "Raw device command"),                                  \
 	___C(GET_SUPPORTED_LOGS, "Get Supported Logs"),                   \
+	___C(GET_EVENT_RECORD, "Get Event Record"),                       \
 	___C(GET_FW_INFO, "Get FW Info"),                                 \
 	___C(GET_PARTITION_INFO, "Get Partition Information"),            \
 	___C(GET_LSA, "Get Label Storage Area"),                          \
Signature · 1 line
2.37.2
#7 lore raw

Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command

Quoted 17 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

Event records are defined for CXL devices. Each record is reported in one event log. Devices are required to support the storage of at least one event record in each event log type.

Devices track event log overflow by incrementing a counter and tracking the time of the first and last overflow event seen.

Software queries events via the Get Event Record mailbox command; CXL rev 3.0 section 8.2.9.2.2.

Issue the Get Event Record mailbox command on driver load. Trace each record found, as well as any overflow conditions. Only 1 event is requested for each query. Optimization of multiple record queries is deferred.

Quoted 7 lines

This patch traces a raw event record only and leaves the specific event record types to subsequent patches.

Macros are created to use for the common CXL Event header fields.

Cc Steven Rostedt <[email protected]>

Hi Ira,

This looks basically fine, but I'm not convinced that it is a good or sustainable idea to report reserved fields from the underlying events in the trace events.
If they are defined to have meaning later, we can't remove them from the 'reserved' field with out breaking backwards compatibility. So we end up with a weird mixture of some fields in both reserved and new entries in the TP and some not.

I've cc'd Mauro as he's probably more experienced in how to handle this sort of stuff than anyone else I know.

Jonathan

Quoted 26 lines

---
Change from RFC:
Remove redundant error message in get event records loop s/EVENT_RECORD_DATA_LENGTH/CXL_EVENT_RECORD_DATA_LENGTH Use hdr_uuid for the header UUID field
Use cxl_event_log_type_str() for the trace events
Create macros for the header fields and common entries of each event Add reserved buffer output dump
Report error if event query fails
Remove unused record_cnt variable
Steven - reorder overflow record
Remove NOTE about checkpatch
Jonathan
check for exactly 1 record
s/v3.0/rev 3.0
Use 3 byte fields for 24bit fields
Add 3.0 Maintenance Operation Class
Add Dynamic Capacity log type
Fix spelling
Dave Jiang/Dan/Alison
s/cxl-event/cxl
trace/events/cxl-events => trace/events/cxl.h s/cxl_event_overflow/overflow s/cxl_event/generic_event
---

...

Quoted 6 lines
drivers/cxl/core/mbox.c +6 -0
+/**
+ * cxl_mem_get_event_records - Get Event Records from the device
+ * @cxlds: The device data for the operation
+ *
+ * Retrieve all event records available on the device and report them as trace
+ * events.

drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 757

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+ *
+ * See CXL rev 3.0 @8.2.9.2.2 Get Event Records
+ */
+void cxl_mem_get_event_records(struct cxl_dev_state *cxlds)
+{
+	enum cxl_event_log_type log_type;
+
+	dev_dbg(cxlds->dev, "Reading event logs\n");
+
+	for (log_type = CXL_EVENT_TYPE_INFO;

I'd start at 0. To my mind that's clearer than this which basically says there is a contiguous range that may or may not be 0 based.

Quoted 40 lines
drivers/cxl/core/mbox.c +5 -0
+	     log_type < CXL_EVENT_TYPE_MAX; log_type++)
+		cxl_mem_get_records_log(cxlds, log_type);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, CXL);
+
 /**
  * cxl_mem_get_partition_info - Get partition info
  * @cxlds: The device data for the operation
drivers/cxl/cxlmem.h +10 -0
index 88e3a8e54b6a..fa8d248fb299 100644
@@ -4,6 +4,7 @@
 #define __CXL_MEM_H__
 #include <uapi/linux/cxl_mem.h>
 #include <linux/cdev.h>
+#include <linux/uuid.h>
 #include "cxl.h"
 
 /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
@@ -253,6 +254,7 @@ struct cxl_dev_state {
 enum cxl_opcode {
 	CXL_MBOX_OP_INVALID		= 0x0000,
 	CXL_MBOX_OP_RAW			= CXL_MBOX_OP_INVALID,
+	CXL_MBOX_OP_GET_EVENT_RECORD	= 0x0100,
 	CXL_MBOX_OP_GET_FW_INFO		= 0x0200,
 	CXL_MBOX_OP_ACTIVATE_FW		= 0x0202,
 	CXL_MBOX_OP_GET_SUPPORTED_LOGS	= 0x0400,
@@ -322,6 +324,78 @@ struct cxl_mbox_identify {
 	u8 qos_telemetry_caps;
 } __packed;
 
+/*
+ * Common Event Record Format
+ * CXL rev 3.0 section 8.2.9.2.1; Table 8-42
+ */
+#define CXL_EVENT_REC_HDR_RES_LEN 0xf
+struct cxl_event_record_hdr {
+	uuid_t id;
+	u8 length;

drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 344

Quoted 10 lines
drivers/cxl/cxlmem.h +10 -0
+	u8 flags[3];
+	__le16 handle;
+	__le16 related_handle;
+	__le64 timestamp;
+	u8 maint_op_class;
+	u8 reserved[CXL_EVENT_REC_HDR_RES_LEN];
+} __packed;
+
+#define CXL_EVENT_RECORD_DATA_LENGTH 0x50
+struct cxl_event_record_raw {

I'd like some comments here on 'why' this makes sense. Is expectation that it's here for future CXL spec definitions or is it intended to allow some reporting of vendor defined records? Actually maybe a comment down at the TP would make more sense than here. Either way comment somewhere :)

Quoted 9 lines
drivers/cxl/cxlmem.h +7 -0
+	struct cxl_event_record_hdr hdr;
+	u8 data[CXL_EVENT_RECORD_DATA_LENGTH];
+} __packed;
+
 
 
+static inline const char *cxl_event_log_type_str(enum cxl_event_log_type type)
+{
+	switch (type) {

drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 392

Quoted 10 lines
drivers/cxl/cxlmem.h +10 -0
+	case CXL_EVENT_TYPE_INFO:
+		return "Informational";
+	case CXL_EVENT_TYPE_WARN:
+		return "Warning";
+	case CXL_EVENT_TYPE_FAIL:
+		return "Failure";
+	case CXL_EVENT_TYPE_FATAL:
+		return "Fatal";
+	case CXL_EVENT_TYPE_DYNAMIC_CAP:
+		return "Dynamic Capacity";

Array of const char * that you pick from (with limit check) maybe rather than a switch? Doesn't matter that much though.

Quoted 34 lines
drivers/cxl/cxlmem.h +7 -0
+	default:
+		break;
+	}
+	return "<unknown>";
+}
+
 struct cxl_mbox_get_partition_info {
 	__le64 active_volatile_cap;
 	__le64 active_persistent_cap;
@@ -381,6 +455,7 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
 struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
 void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
 void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
+void cxl_mem_get_event_records(struct cxl_dev_state *cxlds);
 #ifdef CONFIG_CXL_SUSPEND
 void cxl_mem_active_inc(void);
 void cxl_mem_active_dec(void);
include/trace/events/cxl.h new file +11 -0
new file mode 100644
index 000000000000..318ba5fe046e
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM cxl
+
+#if !defined(_CXL_TRACE_EVENTS_H) ||  defined(TRACE_HEADER_MULTI_READ)
+#define _CXL_TRACE_EVENTS_H
+
+#include <asm-generic/unaligned.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(overflow,

include/trace/events/cxl.h line 21

Quoted 10 lines
include/trace/events/cxl.h new file +10 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_get_event_payload *payload),
+
+	TP_ARGS(dev_name, log, payload),
+
+	TP_STRUCT__entry(
+		__string(dev_name, dev_name)
+		__field(int, log)
+		__field(u64, first)

first_ts maybe? first alone is a bit vague.

Quoted 4 lines
include/trace/events/cxl.h new file +4 -0
+		__field(u64, last)
+		__field(u16, count)
+	),
+

include/trace/events/cxl.h line 35

Quoted 10 lines
include/trace/events/cxl.h new file +10 -0
+	TP_fast_assign(
+		__assign_str(dev_name, dev_name);
+		__entry->log = log;
+		__entry->count = le16_to_cpu(payload->overflow_err_count);
+		__entry->first = le64_to_cpu(payload->first_overflow_timestamp);
+		__entry->last = le64_to_cpu(payload->last_overflow_timestamp);
+	),
+
+	TP_printk("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu",
+		__get_str(dev_name), cxl_event_log_type_str(__entry->log),

So we have a potential header include ordering issue given this header doesn't include the definition of cxl_event_log_type_str()?

Quoted 45 lines
include/trace/events/cxl.h new file +45 -0
+		__entry->count, __entry->first, __entry->last)
+
+);
+
+/*
+ * Common Event Record Format
+ * CXL 3.0 section 8.2.9.2.1; Table 8-42
+ */
+#define CXL_EVENT_RECORD_FLAG_PERMANENT		BIT(2)
+#define CXL_EVENT_RECORD_FLAG_MAINT_NEEDED	BIT(3)
+#define CXL_EVENT_RECORD_FLAG_PERF_DEGRADED	BIT(4)
+#define CXL_EVENT_RECORD_FLAG_HW_REPLACE	BIT(5)
+#define show_hdr_flags(flags)	__print_flags(flags, " | ",			   \
+	{ CXL_EVENT_RECORD_FLAG_PERMANENT,	"Permanent Condition"		}, \
+	{ CXL_EVENT_RECORD_FLAG_MAINT_NEEDED,	"Maintenance Needed"		}, \
+	{ CXL_EVENT_RECORD_FLAG_PERF_DEGRADED,	"Performance Degraded"		}, \
+	{ CXL_EVENT_RECORD_FLAG_HW_REPLACE,	"Hardware Replacement Needed"	}  \
+)
+
+/*
+ * Define macros for the common header of each CXL event.
+ *
+ * Tracepoints using these macros must do 3 things:
+ *
+ *	1) Add CXL_EVT_TP_entry to TP_STRUCT__entry
+ *	2) Use CXL_EVT_TP_fast_assign within TP_fast_assign;
+ *	   pass the dev_name, log, and CXL event header
+ *	3) Use CXL_EVT_TP_printk() instead of TP_printk()
+ *
+ * See the generic_event tracepoint as an example.
+ */
+#define CXL_EVT_TP_entry					\
+	__string(dev_name, dev_name)				\
+	__field(int, log)					\
+	__array(u8, hdr_uuid, UUID_SIZE)			\
+	__field(u32, hdr_flags)					\
+	__field(u16, hdr_handle)				\
+	__field(u16, hdr_related_handle)			\
+	__field(u64, hdr_timestamp)				\
+	__array(u8, hdr_res, CXL_EVENT_REC_HDR_RES_LEN)		\
+	__field(u8, hdr_length)					\
+	__field(u8, hdr_maint_op_class)
+
+#define CXL_EVT_TP_fast_assign(dname, l, hdr)					\
+	__assign_str(dev_name, (dname));					\

include/trace/events/cxl.h line 90

Quoted 10 lines
include/trace/events/cxl.h new file +10 -0
+	__entry->log = (l);							\
+	memcpy(__entry->hdr_uuid, &(hdr).id, UUID_SIZE);			\
+	__entry->hdr_length = (hdr).length;					\
+	__entry->hdr_flags = get_unaligned_le24((hdr).flags);			\
+	__entry->hdr_handle = le16_to_cpu((hdr).handle);			\
+	__entry->hdr_related_handle = le16_to_cpu((hdr).related_handle);	\
+	__entry->hdr_timestamp = le64_to_cpu((hdr).timestamp);			\
+	__entry->hdr_maint_op_class = (hdr).maint_op_class;			\
+	memcpy(__entry->hdr_res, &(hdr).reserved,				\
+		CXL_EVENT_REC_HDR_RES_LEN)

What's the logic behind having the reserved fields here?
How does that change when a future spec defines them as used? Do we have to keep whatever was in the previously reserved fields for ever in order to maintain backwards compatibility even though we've added the same data to the end of the trace point?

I don't think they should be here at all. Given a userspace stack has to cope with out knowing the contents of those fields as userspace might not be up to date, I see no problem with requiring a newer kernel to support stuff added in future specs.

Steven, Mauro, you probably have a better idea of history of similar cases. How have other people handled reserved fields in underlying hardware reports that may become used in later revisions?

Also, probably makes sense to cc linux-edac on v3 of this series as the experts in these flows tend to read that list. Obviously intent so far is not to pass these into the edac subsystem, but I'd still let them know this work is going on.

Quoted 35 lines
include/trace/events/cxl.h new file +35 -0
+
+
+#define CXL_EVT_TP_printk(fmt, ...) \
+	TP_printk("%s log=%s : time=%llu uuid=%pUl len=%d flags='%s' "		\
+		"handle=%x related_handle=%x maint_op_class=%u res='%s' "	\
+		" : " fmt,							\
+		__get_str(dev_name), cxl_event_log_type_str(__entry->log),	\
+		__entry->hdr_timestamp, __entry->hdr_uuid, __entry->hdr_length,	\
+		show_hdr_flags(__entry->hdr_flags), __entry->hdr_handle,	\
+		__entry->hdr_related_handle, __entry->hdr_maint_op_class,	\
+		__print_hex(__entry->hdr_res, CXL_EVENT_REC_HDR_RES_LEN),	\
+		##__VA_ARGS__)
+
+TRACE_EVENT(generic_event,
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_record_raw *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		__array(u8, data, CXL_EVENT_RECORD_DATA_LENGTH)
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+		memcpy(__entry->data, &rec->data, CXL_EVENT_RECORD_DATA_LENGTH);
+	),
+
+	CXL_EVT_TP_printk("%s",
+		__print_hex(__entry->data, CXL_EVENT_RECORD_DATA_LENGTH))
+);
+
+#endif /* _CXL_TRACE_EVENTS_H */
author
#8 lore raw

Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command

Quoted 30 lines · Jonathan Cameron
Quoted 17 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

Event records are defined for CXL devices. Each record is reported in one event log. Devices are required to support the storage of at least one event record in each event log type.

Devices track event log overflow by incrementing a counter and tracking the time of the first and last overflow event seen.

Software queries events via the Get Event Record mailbox command; CXL rev 3.0 section 8.2.9.2.2.

Issue the Get Event Record mailbox command on driver load. Trace each record found, as well as any overflow conditions. Only 1 event is requested for each query. Optimization of multiple record queries is deferred.

Quoted 7 lines

This patch traces a raw event record only and leaves the specific event record types to subsequent patches.

Macros are created to use for the common CXL Event header fields.

Cc Steven Rostedt <[email protected]>

Hi Ira,

Quoted 7 lines

This looks basically fine, but I'm not convinced that it is a good or sustainable idea to report reserved fields from the underlying events in the trace events.
If they are defined to have meaning later, we can't remove them from the 'reserved' field with out breaking backwards compatibility. So we end up with a weird mixture of some fields in both reserved and new entries in the TP and some not.

I removed the reserved fields from the data but I forgot the header. I think I had a reason they would be useful in the header but I forget now. Which means I was probably wrong. ;-)

Quoted 4 lines

I've cc'd Mauro as he's probably more experienced in how to handle this sort of stuff than anyone else I know.

Fair enough but I agree the reserved fields should be removed.

Quoted 44 lines

Jonathan

Quoted 26 lines

---
Change from RFC:
Remove redundant error message in get event records loop s/EVENT_RECORD_DATA_LENGTH/CXL_EVENT_RECORD_DATA_LENGTH Use hdr_uuid for the header UUID field
Use cxl_event_log_type_str() for the trace events
Create macros for the header fields and common entries of each event Add reserved buffer output dump
Report error if event query fails
Remove unused record_cnt variable
Steven - reorder overflow record
Remove NOTE about checkpatch
Jonathan
check for exactly 1 record
s/v3.0/rev 3.0
Use 3 byte fields for 24bit fields
Add 3.0 Maintenance Operation Class
Add Dynamic Capacity log type
Fix spelling
Dave Jiang/Dan/Alison
s/cxl-event/cxl
trace/events/cxl-events => trace/events/cxl.h s/cxl_event_overflow/overflow s/cxl_event/generic_event
---

...

Quoted 12 lines
drivers/cxl/core/mbox.c +12 -0
+/**
+ * cxl_mem_get_event_records - Get Event Records from the device
+ * @cxlds: The device data for the operation
+ *
+ * Retrieve all event records available on the device and report them as trace
+ * events.
+ *
+ * See CXL rev 3.0 @8.2.9.2.2 Get Event Records
+ */
+void cxl_mem_get_event_records(struct cxl_dev_state *cxlds)
+{
+	enum cxl_event_log_type log_type;
Quoted 8 lines

drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 757

Quoted 4 lines
drivers/cxl/core/mbox.c +4 -0
+
+	dev_dbg(cxlds->dev, "Reading event logs\n");
+
+	for (log_type = CXL_EVENT_TYPE_INFO;

I'd start at 0. To my mind that's clearer than this which basically says there is a contiguous range that may or may not be 0 based.

Ok. But weather or not it is 0 based does not matter.

I'll change it. It takes out a line of code! ;-)

Quoted 49 lines
Quoted 49 lines
drivers/cxl/core/mbox.c +5 -0
+	     log_type < CXL_EVENT_TYPE_MAX; log_type++)
+		cxl_mem_get_records_log(cxlds, log_type);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, CXL);
+
 /**
  * cxl_mem_get_partition_info - Get partition info
  * @cxlds: The device data for the operation
drivers/cxl/cxlmem.h +18 -0
index 88e3a8e54b6a..fa8d248fb299 100644
@@ -4,6 +4,7 @@
 #define __CXL_MEM_H__
 #include <uapi/linux/cxl_mem.h>
 #include <linux/cdev.h>
+#include <linux/uuid.h>
 #include "cxl.h"
 
 /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
@@ -253,6 +254,7 @@ struct cxl_dev_state {
 enum cxl_opcode {
 	CXL_MBOX_OP_INVALID		= 0x0000,
 	CXL_MBOX_OP_RAW			= CXL_MBOX_OP_INVALID,
+	CXL_MBOX_OP_GET_EVENT_RECORD	= 0x0100,
 	CXL_MBOX_OP_GET_FW_INFO		= 0x0200,
 	CXL_MBOX_OP_ACTIVATE_FW		= 0x0202,
 	CXL_MBOX_OP_GET_SUPPORTED_LOGS	= 0x0400,
@@ -322,6 +324,78 @@ struct cxl_mbox_identify {
 	u8 qos_telemetry_caps;
 } __packed;
 
+/*
+ * Common Event Record Format
+ * CXL rev 3.0 section 8.2.9.2.1; Table 8-42
+ */
+#define CXL_EVENT_REC_HDR_RES_LEN 0xf
+struct cxl_event_record_hdr {
+	uuid_t id;
+	u8 length;
+	u8 flags[3];
+	__le16 handle;
+	__le16 related_handle;
+	__le64 timestamp;
+	u8 maint_op_class;
+	u8 reserved[CXL_EVENT_REC_HDR_RES_LEN];
+} __packed;
+
Quoted 8 lines

drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 344

Quoted 2 lines
drivers/cxl/cxlmem.h +2 -0
+#define CXL_EVENT_RECORD_DATA_LENGTH 0x50
+struct cxl_event_record_raw {

I'd like some comments here on 'why' this makes sense. Is expectation that it's here for future CXL spec definitions or is it intended to allow some reporting of vendor defined records? Actually maybe a comment down at the TP would make more sense than here. Either way comment somewhere :)

This comes from a discussion I had with Dan a while back where we decided that any unknown record would just be dumped to user space as data. The idea was based on the kernel ignoring vendor specified events but does allow for some future CXL spec definitios to get through.

This may cause similar confusion to the reserved fields but I don't think it is quite the same.

I'll add a comment.

Quoted 7 lines
Quoted 5 lines
drivers/cxl/cxlmem.h +4 -0
+	struct cxl_event_record_hdr hdr;
+	u8 data[CXL_EVENT_RECORD_DATA_LENGTH];
+} __packed;
+
Quoted 15 lines

drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 392

Quoted 13 lines
drivers/cxl/cxlmem.h +13 -0
+static inline const char *cxl_event_log_type_str(enum cxl_event_log_type type)
+{
+	switch (type) {
+	case CXL_EVENT_TYPE_INFO:
+		return "Informational";
+	case CXL_EVENT_TYPE_WARN:
+		return "Warning";
+	case CXL_EVENT_TYPE_FAIL:
+		return "Failure";
+	case CXL_EVENT_TYPE_FATAL:
+		return "Fatal";
+	case CXL_EVENT_TYPE_DYNAMIC_CAP:
+		return "Dynamic Capacity";

Array of const char * that you pick from (with limit check) maybe rather than a switch? Doesn't matter that much though.

I've seen it both ways. This forces users to use the helper. And I don't think there is much concern about the speed of the lookup.

Quoted 39 lines
Quoted 39 lines
drivers/cxl/cxlmem.h +7 -0
+	default:
+		break;
+	}
+	return "<unknown>";
+}
+
 struct cxl_mbox_get_partition_info {
 	__le64 active_volatile_cap;
 	__le64 active_persistent_cap;
@@ -381,6 +455,7 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
 struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
 void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
 void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
+void cxl_mem_get_event_records(struct cxl_dev_state *cxlds);
 #ifdef CONFIG_CXL_SUSPEND
 void cxl_mem_active_inc(void);
 void cxl_mem_active_dec(void);
include/trace/events/cxl.h new file +15 -0
new file mode 100644
index 000000000000..318ba5fe046e
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM cxl
+
+#if !defined(_CXL_TRACE_EVENTS_H) ||  defined(TRACE_HEADER_MULTI_READ)
+#define _CXL_TRACE_EVENTS_H
+
+#include <asm-generic/unaligned.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(overflow,
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_get_event_payload *payload),
+
Quoted 8 lines

include/trace/events/cxl.h line 21

Quoted 6 lines
include/trace/events/cxl.h new file +6 -0
+	TP_ARGS(dev_name, log, payload),
+
+	TP_STRUCT__entry(
+		__string(dev_name, dev_name)
+		__field(int, log)
+		__field(u64, first)

first_ts maybe? first alone is a bit vague.

Agreed. Changed to first_ts and last_ts.

Quoted 10 lines
Quoted 10 lines
include/trace/events/cxl.h new file +9 -0
+		__field(u64, last)
+		__field(u16, count)
+	),
+
+	TP_fast_assign(
+		__assign_str(dev_name, dev_name);
+		__entry->log = log;
+		__entry->count = le16_to_cpu(payload->overflow_err_count);
+		__entry->first = le64_to_cpu(payload->first_overflow_timestamp);
Quoted 8 lines

include/trace/events/cxl.h line 35

Quoted 5 lines
include/trace/events/cxl.h new file +5 -0
+		__entry->last = le64_to_cpu(payload->last_overflow_timestamp);
+	),
+
+	TP_printk("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu",
+		__get_str(dev_name), cxl_event_log_type_str(__entry->log),

So we have a potential header include ordering issue given this header doesn't include the definition of cxl_event_log_type_str()?

The RFC had a __print_symbolic() conversion which was redundant. I really don't like having 2 conversion routines for this string. I think it will result in maintenance issues down the line.

I'll try and resolve it. Nothing I have tried this morning seems to work. I may just go back to what I had. I need to double check the user space code too.

Quoted 50 lines
Quoted 50 lines
include/trace/events/cxl.h new file +49 -0
+		__entry->count, __entry->first, __entry->last)
+
+);
+
+/*
+ * Common Event Record Format
+ * CXL 3.0 section 8.2.9.2.1; Table 8-42
+ */
+#define CXL_EVENT_RECORD_FLAG_PERMANENT		BIT(2)
+#define CXL_EVENT_RECORD_FLAG_MAINT_NEEDED	BIT(3)
+#define CXL_EVENT_RECORD_FLAG_PERF_DEGRADED	BIT(4)
+#define CXL_EVENT_RECORD_FLAG_HW_REPLACE	BIT(5)
+#define show_hdr_flags(flags)	__print_flags(flags, " | ",			   \
+	{ CXL_EVENT_RECORD_FLAG_PERMANENT,	"Permanent Condition"		}, \
+	{ CXL_EVENT_RECORD_FLAG_MAINT_NEEDED,	"Maintenance Needed"		}, \
+	{ CXL_EVENT_RECORD_FLAG_PERF_DEGRADED,	"Performance Degraded"		}, \
+	{ CXL_EVENT_RECORD_FLAG_HW_REPLACE,	"Hardware Replacement Needed"	}  \
+)
+
+/*
+ * Define macros for the common header of each CXL event.
+ *
+ * Tracepoints using these macros must do 3 things:
+ *
+ *	1) Add CXL_EVT_TP_entry to TP_STRUCT__entry
+ *	2) Use CXL_EVT_TP_fast_assign within TP_fast_assign;
+ *	   pass the dev_name, log, and CXL event header
+ *	3) Use CXL_EVT_TP_printk() instead of TP_printk()
+ *
+ * See the generic_event tracepoint as an example.
+ */
+#define CXL_EVT_TP_entry					\
+	__string(dev_name, dev_name)				\
+	__field(int, log)					\
+	__array(u8, hdr_uuid, UUID_SIZE)			\
+	__field(u32, hdr_flags)					\
+	__field(u16, hdr_handle)				\
+	__field(u16, hdr_related_handle)			\
+	__field(u64, hdr_timestamp)				\
+	__array(u8, hdr_res, CXL_EVENT_REC_HDR_RES_LEN)		\
+	__field(u8, hdr_length)					\
+	__field(u8, hdr_maint_op_class)
+
+#define CXL_EVT_TP_fast_assign(dname, l, hdr)					\
+	__assign_str(dev_name, (dname));					\
+	__entry->log = (l);							\
+	memcpy(__entry->hdr_uuid, &(hdr).id, UUID_SIZE);			\
+	__entry->hdr_length = (hdr).length;					\
+	__entry->hdr_flags = get_unaligned_le24((hdr).flags);			\
Quoted 8 lines

include/trace/events/cxl.h line 90

Quoted 6 lines
include/trace/events/cxl.h new file +6 -0
+	__entry->hdr_handle = le16_to_cpu((hdr).handle);			\
+	__entry->hdr_related_handle = le16_to_cpu((hdr).related_handle);	\
+	__entry->hdr_timestamp = le64_to_cpu((hdr).timestamp);			\
+	__entry->hdr_maint_op_class = (hdr).maint_op_class;			\
+	memcpy(__entry->hdr_res, &(hdr).reserved,				\
+		CXL_EVENT_REC_HDR_RES_LEN)

What's the logic behind having the reserved fields here?

I forget.

Quoted 4 lines

How does that change when a future spec defines them as used? Do we have to keep whatever was in the previously reserved fields for ever in order to maintain backwards compatibility even though we've added the same data to the end of the trace point?

I don't expect the header to change as much as the records themselves. So I'm ok removing the reserved fields.

Quoted 6 lines

I don't think they should be here at all. Given a userspace stack has to cope with out knowing the contents of those fields as userspace might not be up to date, I see no problem with requiring a newer kernel to support stuff added in future specs.

Quoted 8 lines

Steven, Mauro, you probably have a better idea of history of similar cases. How have other people handled reserved fields in underlying hardware reports that may become used in later revisions?

Also, probably makes sense to cc linux-edac on v3 of this series as the experts in these flows tend to read that list. Obviously intent so far is not to pass these into the edac subsystem, but I'd still let them know this work is going on.

Sounds good.

The struggle I'm having is in how flexible these records are compared to the other RAS trace events.

I'll drop the reserved fields.
Ira

maintainer
#9 lore raw

Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command

In reply to [email protected]
Quoted 6 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:

new file mode 100644
index 000000000000..318ba5fe046e

include/trace/events/cxl.h new file +1 -0
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */

include/trace/events/cxl.h line 11

Quoted 10 lines
include/trace/events/cxl.h new file +10 -0
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM cxl
+
+#if !defined(_CXL_TRACE_EVENTS_H) ||  defined(TRACE_HEADER_MULTI_READ)
+#define _CXL_TRACE_EVENTS_H
+
+#include <asm-generic/unaligned.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(overflow,

Please do not use generic names for grouped events. As most tooling can use the name and not associate it with the group, it makes it ambiguous for what event it wants to enable.

That is, call it "cxl_overflow" and not "overflow".

Quoted 20 lines
include/trace/events/cxl.h new file +20 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_get_event_payload *payload),
+
+	TP_ARGS(dev_name, log, payload),
+
+	TP_STRUCT__entry(
+		__string(dev_name, dev_name)
+		__field(int, log)
+		__field(u64, first)
+		__field(u64, last)
+		__field(u16, count)
+	),
+
+	TP_fast_assign(
+		__assign_str(dev_name, dev_name);
+		__entry->log = log;
+		__entry->count = le16_to_cpu(payload->overflow_err_count);
+		__entry->first = le64_to_cpu(payload->first_overflow_timestamp);
+		__entry->last = le64_to_cpu(payload->last_overflow_timestamp);

include/trace/events/cxl.h

Quoted 10 lines
include/trace/events/cxl.h new file +9 -0
+	),
+
+	TP_printk("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu",
+		__get_str(dev_name), cxl_event_log_type_str(__entry->log),
+		__entry->count, __entry->first, __entry->last)
+
+);
+
 
+TRACE_EVENT(generic_event,

Same here. It's a "cxl_generic_event" not a "generic_event" that will clash with any other subsystem that would (but shouldn't) use the same name.

-- Steve

Quoted 22 lines
include/trace/events/cxl.h new file +21 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_record_raw *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		__array(u8, data, CXL_EVENT_RECORD_DATA_LENGTH)
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+		memcpy(__entry->data, &rec->data, CXL_EVENT_RECORD_DATA_LENGTH);
+	),
+
+	CXL_EVT_TP_printk("%s",
+		__print_hex(__entry->data, CXL_EVENT_RECORD_DATA_LENGTH))
+);
+
+#endif /* _CXL_TRACE_EVENTS_H */
author
#10 lore raw

Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command

Quoted 24 lines · Steven Rostedt
Quoted 6 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:

new file mode 100644
index 000000000000..318ba5fe046e

include/trace/events/cxl.h new file +1 -0
@@ -0,0 +1,130 @@
+/* SPDX-License-Identifier: GPL-2.0 */

include/trace/events/cxl.h line 11

Quoted 10 lines
include/trace/events/cxl.h new file +10 -0
+#undef TRACE_SYSTEM
+#define TRACE_SYSTEM cxl
+
+#if !defined(_CXL_TRACE_EVENTS_H) ||  defined(TRACE_HEADER_MULTI_READ)
+#define _CXL_TRACE_EVENTS_H
+
+#include <asm-generic/unaligned.h>
+#include <linux/tracepoint.h>
+
+TRACE_EVENT(overflow,

Please do not use generic names for grouped events. As most tooling can use the name and not associate it with the group, it makes it ambiguous for what event it wants to enable.

Quoted 1 line

That is, call it "cxl_overflow" and not "overflow".

I did not realize that.[1]

[1] https://lore.kernel.org/linux-cxl/[email protected]/

Quoted 30 lines
Quoted 29 lines
include/trace/events/cxl.h new file +28 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_get_event_payload *payload),
+
+	TP_ARGS(dev_name, log, payload),
+
+	TP_STRUCT__entry(
+		__string(dev_name, dev_name)
+		__field(int, log)
+		__field(u64, first)
+		__field(u64, last)
+		__field(u16, count)
+	),
+
+	TP_fast_assign(
+		__assign_str(dev_name, dev_name);
+		__entry->log = log;
+		__entry->count = le16_to_cpu(payload->overflow_err_count);
+		__entry->first = le64_to_cpu(payload->first_overflow_timestamp);
+		__entry->last = le64_to_cpu(payload->last_overflow_timestamp);
+	),
+
+	TP_printk("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu",
+		__get_str(dev_name), cxl_event_log_type_str(__entry->log),
+		__entry->count, __entry->first, __entry->last)
+
+);
+
Quoted 5 lines

include/trace/events/cxl.h line 104

Quoted 1 line
include/trace/events/cxl.h new file +1 -0
+TRACE_EVENT(generic_event,

Same here. It's a "cxl_generic_event" not a "generic_event" that will clash with any other subsystem that would (but shouldn't) use the same name.

Sure I'll change it back.

Thanks for setting us straight,
Ira

Quoted 25 lines

-- Steve

Quoted 22 lines
include/trace/events/cxl.h new file +21 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_record_raw *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		__array(u8, data, CXL_EVENT_RECORD_DATA_LENGTH)
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+		memcpy(__entry->data, &rec->data, CXL_EVENT_RECORD_DATA_LENGTH);
+	),
+
+	CXL_EVT_TP_printk("%s",
+		__print_hex(__entry->data, CXL_EVENT_RECORD_DATA_LENGTH))
+);
+
+#endif /* _CXL_TRACE_EVENTS_H */
#11 lore raw

Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command

In reply to [email protected]

Hi Ira,

Quoted 39 lines · On 10/10/22 5:41 PM, [email protected] wrote:

From: Ira Weiny <[email protected]>

drivers/cxl/core/mbox.c +35 -0
+static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
+				    enum cxl_event_log_type type)
+{
+	struct cxl_get_event_payload payload;
+
+	do {
+		u8 log_type = type;
+		int rc;
+
+		rc = cxl_mbox_send_cmd(cxlds, CXL_MBOX_OP_GET_EVENT_RECORD,
+				       &log_type, sizeof(log_type),
+				       &payload, sizeof(payload));
+		if (rc) {
+			dev_err(cxlds->dev, "Event log '%s': Failed to query event records : %d",
+				cxl_event_log_type_str(type), rc);
+			return;
+		}
+
+		if (le16_to_cpu(payload.record_count) == 1)
+			trace_generic_event(dev_name(cxlds->dev), type,
+					    &payload.record);
+
+		if (payload.flags & CXL_GET_EVENT_FLAG_OVERFLOW)
+			trace_overflow(dev_name(cxlds->dev), type, &payload);
+
+	} while (payload.flags & CXL_GET_EVENT_FLAG_MORE_RECORDS);
+}
+
+/**
+ * cxl_mem_get_event_records - Get Event Records from the device
+ * @cxlds: The device data for the operation
+ *
+ * Retrieve all event records available on the device and report them as trace
+ * events.
+ *

drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 758

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+ * See CXL rev 3.0 @8.2.9.2.2 Get Event Records
+ */
+void cxl_mem_get_event_records(struct cxl_dev_state *cxlds)
+{
+	enum cxl_event_log_type log_type;
+
+	dev_dbg(cxlds->dev, "Reading event logs\n");
+
+	for (log_type = CXL_EVENT_TYPE_INFO;
+	     log_type < CXL_EVENT_TYPE_MAX; log_type++)

Why should we loop through each event log here? What if the event record doesn't exist in the event log?

I got some Mailbox error messages like this while bootup..
[  346.387010] cxl_pci 0000:7f:00.0: Sending command
[  346.387181] cxl_pci 0000:7f:00.0: Doorbell wait took 0ms
[  346.387197] cxl_pci 0000:7f:00.0: Mailbox operation had an error: cmd input was invalid [  346.387205] cxl_pci 0000:7f:00.0: Event log 'Warning': Failed to query event records : -6 ..

Can we just read the "Event Status" field from Event Status Register (Device Status Registers Capability Offset + 00h) 8.2.8.3.1 in CXL Spec, determine if the records exist and just query those event logs?

Thanks,
Smita

Quoted 7 lines
drivers/cxl/core/mbox.c +4 -0
+		cxl_mem_get_records_log(cxlds, log_type);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, CXL);
+
  /**
   * cxl_mem_get_partition_info - Get partition info
   * @cxlds: The device data for the operation
author
#12 lore raw

Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command

Quoted 6 lines · Smita Koralahalli

Hi Ira,

Quoted 3 lines · On 10/10/22 5:41 PM, [email protected] wrote:

From: Ira Weiny <[email protected]>

[snip]

Quoted 12 lines
Quoted 12 lines
drivers/cxl/core/mbox.c +12 -0
+
+/**
+ * cxl_mem_get_event_records - Get Event Records from the device
+ * @cxlds: The device data for the operation
+ *
+ * Retrieve all event records available on the device and report them as trace
+ * events.
+ *
+ * See CXL rev 3.0 @8.2.9.2.2 Get Event Records
+ */
+void cxl_mem_get_event_records(struct cxl_dev_state *cxlds)
+{
Quoted 8 lines

drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 758

Quoted 6 lines
drivers/cxl/core/mbox.c +6 -0
+	enum cxl_event_log_type log_type;
+
+	dev_dbg(cxlds->dev, "Reading event logs\n");
+
+	for (log_type = CXL_EVENT_TYPE_INFO;
+	     log_type < CXL_EVENT_TYPE_MAX; log_type++)

Why should we loop through each event log here?

The idea was to clear all the event logs. I think this made more sense before the addition of the optional dynamic capacity log.

Quoted 3 lines

What if the event
record doesn't exist in the event log?

An empty log does not cause any issue. The query will simply return 0 records which is valid.

Quoted 10 lines

I got some Mailbox error messages like this while bootup..
[  346.387010] cxl_pci 0000:7f:00.0: Sending command
[  346.387181] cxl_pci 0000:7f:00.0: Doorbell wait took 0ms
[  346.387197] cxl_pci 0000:7f:00.0: Mailbox operation had an error: cmd input was invalid
[  346.387205] cxl_pci 0000:7f:00.0: Event log 'Warning': Failed to query event records : -6
..

Quoted 3 lines

Can we just read the "Event Status" field from Event Status Register (Device Status Registers Capability Offset + 00h) 8.2.8.3.1 in CXL Spec, determine if the records exist and just query those event logs?

Likely the hardware does not have the dynamic capacity log and so the code is asking for something invalid. I did not think of that when I added that new log. Checking status register looks to be the proper solution.

I'll throw in some testing in QEMU for this. I'll also have to implement the status register in QEMU to fully test.

Thanks for the testing! :-D

Ira

Quoted 11 lines

Thanks,
Smita

Quoted 7 lines
drivers/cxl/core/mbox.c +4 -0
+		cxl_mem_get_records_log(cxlds, log_type);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, CXL);
+
  /**
   * cxl_mem_get_partition_info - Get partition info
   * @cxlds: The device data for the operation
#13 lore raw

RFC PATCH v2 3/11 cxl/mem: Implement Clear Event Records command

In reply to [email protected]

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.3 defines the Clear Event Records mailbox command. After an event record is read it needs to be cleared from the event log.

Implement cxl_clear_event_record() and call it for each record retrieved from the device.

Each record is cleared individually. A clear all bit is specified but events could arrive between a get and the final clear all operation. Therefore each event is cleared specifically.

---
Changes from RFC:
Jonathan
Clean up init of payload and use return code. Also report any error to clear the event. s/v3.0/rev 3.0

---
 drivers/cxl/core/mbox.c      | 31 ++++++++++++++++++++++++++++---
 drivers/cxl/cxlmem.h         | 15 +++++++++++++++
 include/uapi/linux/cxl_mem.h |  1 +
 3 files changed, 44 insertions(+), 3 deletions(-)
drivers/cxl/core/mbox.c +28 -3
index 5f258c3f2190..bc4e42b3e01b 100644
@@ -52,6 +52,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 #endif
 	CXL_CMD(GET_SUPPORTED_LOGS, 0, CXL_VARIABLE_PAYLOAD, CXL_CMD_FLAG_FORCE_ENABLE),
 	CXL_CMD(GET_EVENT_RECORD, 1, CXL_VARIABLE_PAYLOAD, 0),
+	CXL_CMD(CLEAR_EVENT_RECORD, CXL_VARIABLE_PAYLOAD, 0, 0),
 	CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
 	CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
 	CXL_CMD(GET_LSA, 0x8, CXL_VARIABLE_PAYLOAD, 0),
@@ -711,6 +712,20 @@ int cxl_enumerate_cmds(struct cxl_dev_state *cxlds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_enumerate_cmds, CXL);
 
+static int cxl_clear_event_record(struct cxl_dev_state *cxlds,
+				  enum cxl_event_log_type log,
+				  __le16 handle)
+{
+	struct cxl_mbox_clear_event_payload payload = {
+		.event_log = log,
+		.nr_recs = 1,
+		.handle = handle,
+	};
+
+	return cxl_mbox_send_cmd(cxlds, CXL_MBOX_OP_CLEAR_EVENT_RECORD,
+				 &payload, sizeof(payload), NULL, 0);
+}
+
 static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
 				    enum cxl_event_log_type type)
 {
@@ -729,9 +744,18 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
 			return;
 		}
 
-		if (le16_to_cpu(payload.record_count) == 1)
+		if (le16_to_cpu(payload.record_count) == 1) {
 			trace_generic_event(dev_name(cxlds->dev), type,
 					    &payload.record);
+			rc = cxl_clear_event_record(cxlds, type,
+						    payload.record.hdr.handle);
+			if (rc) {
+				dev_err(cxlds->dev, "Event log '%s': Failed to clear event (%u) : %d",
+					cxl_event_log_type_str(type),
+					payload.record.hdr.handle, rc);
+				return;
+			}
+		}
 
 		if (payload.flags & CXL_GET_EVENT_FLAG_OVERFLOW)
 			trace_overflow(dev_name(cxlds->dev), type, &payload);
@@ -743,10 +767,11 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
  * cxl_mem_get_event_records - Get Event Records from the device
  * @cxlds: The device data for the operation
  *
- * Retrieve all event records available on the device and report them as trace
- * events.
+ * Retrieve all event records available on the device, report them as trace
+ * events, and clear them.
  *
  * See CXL rev 3.0 @8.2.9.2.2 Get Event Records
+ * See CXL rev 3.0 @8.2.9.2.3 Clear Event Records
  */
 void cxl_mem_get_event_records(struct cxl_dev_state *cxlds)
 {
drivers/cxl/cxlmem.h +15 -0
index fa8d248fb299..75aea34f3ffb 100644
@@ -255,6 +255,7 @@ enum cxl_opcode {
 	CXL_MBOX_OP_INVALID		= 0x0000,
 	CXL_MBOX_OP_RAW			= CXL_MBOX_OP_INVALID,
 	CXL_MBOX_OP_GET_EVENT_RECORD	= 0x0100,
+	CXL_MBOX_OP_CLEAR_EVENT_RECORD	= 0x0101,
 	CXL_MBOX_OP_GET_FW_INFO		= 0x0200,
 	CXL_MBOX_OP_ACTIVATE_FW		= 0x0202,
 	CXL_MBOX_OP_GET_SUPPORTED_LOGS	= 0x0400,
@@ -396,6 +397,20 @@ static inline const char *cxl_event_log_type_str(enum cxl_event_log_type type)
 	return "<unknown>";
 }
 
+/*
+ * Clear Event Records input payload
+ * CXL rev 3.0 section 8.2.9.2.3; Table 8-51
+ *
+ * Space given for 1 record
+ */
+struct cxl_mbox_clear_event_payload {
+	u8 event_log;		/* enum cxl_event_log_type */
+	u8 clear_flags;
+	u8 nr_recs;		/* 1 for this struct */
+	u8 reserved[3];
+	__le16 handle;
+};
+
 struct cxl_mbox_get_partition_info {
 	__le64 active_volatile_cap;
 	__le64 active_persistent_cap;
include/uapi/linux/cxl_mem.h +1 -0
index 70459be5bdd4..7c1ad8062792 100644
@@ -25,6 +25,7 @@
 	___C(RAW, "Raw device command"),                                  \
 	___C(GET_SUPPORTED_LOGS, "Get Supported Logs"),                   \
 	___C(GET_EVENT_RECORD, "Get Event Record"),                       \
+	___C(CLEAR_EVENT_RECORD, "Clear Event Record"),                   \
 	___C(GET_FW_INFO, "Get FW Info"),                                 \
 	___C(GET_PARTITION_INFO, "Get Partition Information"),            \
 	___C(GET_LSA, "Get Label Storage Area"),                          \
Signature · 1 line
2.37.2
#14 lore raw

RFC PATCH v2 4/11 cxl/mem: Clear events on driver load

In reply to [email protected]

From: Ira Weiny <[email protected]>

The information contained in the events prior to the driver loading can be queried at any time through other mailbox commands.

Ensure a clean slate of events by reading and clearing the events. The events are sent to the trace buffer but it is not anticipated to have anyone listening to it at driver load time.

---
 drivers/cxl/pci.c            | 2 ++
 tools/testing/cxl/test/mem.c | 2 ++
 2 files changed, 4 insertions(+)
drivers/cxl/pci.c +2 -0
index faeb5d9d7a7a..5f1b492bd388 100644
@@ -498,6 +498,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
+	cxl_mem_get_event_records(cxlds);
+
 	if (resource_size(&cxlds->pmem_res) && IS_ENABLED(CONFIG_CXL_PMEM))
 		rc = devm_cxl_add_nvdimm(&pdev->dev, cxlmd);
 
tools/testing/cxl/test/mem.c +2 -0
index aa2df3a15051..e2f5445d24ff 100644
@@ -285,6 +285,8 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
+	cxl_mem_get_event_records(cxlds);
+
 	if (resource_size(&cxlds->pmem_res) && IS_ENABLED(CONFIG_CXL_PMEM))
 		rc = devm_cxl_add_nvdimm(dev, cxlmd);
 
Signature · 1 line
2.37.2
Reviewed-by on v2 4/11
#15 lore raw

Re: [RFC V2 PATCH 04/11] cxl/mem: Clear events on driver load

Quoted 10 lines · On Mon, 10 Oct 2022 15:41:24 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

The information contained in the events prior to the driver loading can be queried at any time through other mailbox commands.

Ensure a clean slate of events by reading and clearing the events. The events are sent to the trace buffer but it is not anticipated to have anyone listening to it at driver load time.

Makes sense I think.

Quoted 31 lines
---
 drivers/cxl/pci.c            | 2 ++
 tools/testing/cxl/test/mem.c | 2 ++
 2 files changed, 4 insertions(+)
drivers/cxl/pci.c +2 -0
index faeb5d9d7a7a..5f1b492bd388 100644
@@ -498,6 +498,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
+	cxl_mem_get_event_records(cxlds);
+
 	if (resource_size(&cxlds->pmem_res) && IS_ENABLED(CONFIG_CXL_PMEM))
 		rc = devm_cxl_add_nvdimm(&pdev->dev, cxlmd);
 
tools/testing/cxl/test/mem.c +2 -0
index aa2df3a15051..e2f5445d24ff 100644
@@ -285,6 +285,8 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
+	cxl_mem_get_event_records(cxlds);
+
 	if (resource_size(&cxlds->pmem_res) && IS_ENABLED(CONFIG_CXL_PMEM))
 		rc = devm_cxl_add_nvdimm(dev, cxlmd);
 
#16 lore raw

RFC PATCH v2 5/11 cxl/mem: Trace General Media Event Record

In reply to [email protected]

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.1 defines the General Media Event Record.

Determine if the event read is a general media record and if so trace the record.

---
Changes from RFC:
Add reserved byte array
Use common CXL event header record macros Jonathan
Use unaligned_le{24,16} for unaligned fields Don't use the inverse of phy addr mask
Dave Jiang
s/cxl_gen_media_event/general_media s/cxl_evt_gen_media/cxl_event_gen_media

---
 drivers/cxl/core/mbox.c    |  30 ++++++++++-
 drivers/cxl/cxlmem.h       |  20 +++++++
 include/trace/events/cxl.h | 108 +++++++++++++++++++++++++++++++++++++
 3 files changed, 156 insertions(+), 2 deletions(-)
drivers/cxl/core/mbox.c +28 -2
index bc4e42b3e01b..1097250c115a 100644
@@ -712,6 +712,32 @@ int cxl_enumerate_cmds(struct cxl_dev_state *cxlds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_enumerate_cmds, CXL);
 
+/*
+ * General Media Event Record
+ * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
+ */
+static const uuid_t gen_media_event_uuid =
+	UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
+		  0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6);
+
+static void cxl_trace_event_record(const char *dev_name,
+				   enum cxl_event_log_type type,
+				   struct cxl_get_event_payload *payload)
+{
+	uuid_t *id = &payload->record.hdr.id;
+
+	if (uuid_equal(id, &gen_media_event_uuid)) {
+		struct cxl_event_gen_media *rec =
+				(struct cxl_event_gen_media *)&payload->record;
+
+		trace_general_media(dev_name, type, rec);
+		return;
+	}
+
+	/* For unknown record types print just the header */
+	trace_generic_event(dev_name, type, &payload->record);
+}
+
 static int cxl_clear_event_record(struct cxl_dev_state *cxlds,
 				  enum cxl_event_log_type log,
 				  __le16 handle)
@@ -745,8 +771,8 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
 		}
 
 		if (le16_to_cpu(payload.record_count) == 1) {
-			trace_generic_event(dev_name(cxlds->dev), type,
-					    &payload.record);
+			cxl_trace_event_record(dev_name(cxlds->dev), type,
+					       &payload);
 			rc = cxl_clear_event_record(cxlds, type,
 						    payload.record.hdr.handle);
 			if (rc) {
drivers/cxl/cxlmem.h +20 -0
index 75aea34f3ffb..b5c120bd4068 100644
@@ -411,6 +411,26 @@ struct cxl_mbox_clear_event_payload {
 	__le16 handle;
 };
 
+/*
+ * General Media Event Record
+ * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
+ */
+#define CXL_EVENT_GEN_MED_COMP_ID_SIZE	0x10
+#define CXL_EVENT_GEN_MED_RES_SIZE	0x2e
+struct cxl_event_gen_media {
+	struct cxl_event_record_hdr hdr;
+	__le64 phys_addr;
+	u8 descriptor;
+	u8 type;
+	u8 transaction_type;
+	u8 validity_flags[2];
+	u8 channel;
+	u8 rank;
+	u8 device[3];
+	u8 component_id[CXL_EVENT_GEN_MED_COMP_ID_SIZE];
+	u8 reserved[CXL_EVENT_GEN_MED_RES_SIZE];
+} __packed;
+
 struct cxl_mbox_get_partition_info {
 	__le64 active_volatile_cap;
 	__le64 active_persistent_cap;
include/trace/events/cxl.h +108 -0

114 lines not on this page · raw message

Signature · 1 line
2.37.2
Reviewed-by on v2 5/11
#17 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

Quoted 8 lines · On Mon, 10 Oct 2022 15:41:25 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.1 defines the General Media Event Record.

Determine if the event read is a general media record and if so trace the record.

I'll review the rest of these with the assumption that the question of reserved bytes in tracepoints will get resolved before these go in.

One minor comment on a comment inline. Other than those lgtm

Quoted 39 lines

---
Changes from RFC:
Add reserved byte array
Use common CXL event header record macros Jonathan
Use unaligned_le{24,16} for unaligned fields Don't use the inverse of phy addr mask
Dave Jiang
s/cxl_gen_media_event/general_media s/cxl_evt_gen_media/cxl_event_gen_media

---
 drivers/cxl/core/mbox.c    |  30 ++++++++++-
 drivers/cxl/cxlmem.h       |  20 +++++++
 include/trace/events/cxl.h | 108 +++++++++++++++++++++++++++++++++++++
 3 files changed, 156 insertions(+), 2 deletions(-)
drivers/cxl/core/mbox.c +14 -0
index bc4e42b3e01b..1097250c115a 100644
@@ -712,6 +712,32 @@ int cxl_enumerate_cmds(struct cxl_dev_state *cxlds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_enumerate_cmds, CXL);
 
+/*
+ * General Media Event Record
+ * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
+ */
+static const uuid_t gen_media_event_uuid =
+	UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
+		  0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6);
+
+static void cxl_trace_event_record(const char *dev_name,
+				   enum cxl_event_log_type type,
+				   struct cxl_get_event_payload *payload)
+{
+	uuid_t *id = &payload->record.hdr.id;
+

drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 738

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+	if (uuid_equal(id, &gen_media_event_uuid)) {
+		struct cxl_event_gen_media *rec =
+				(struct cxl_event_gen_media *)&payload->record;
+
+		trace_general_media(dev_name, type, rec);
+		return;
+	}
+
+	/* For unknown record types print just the header */
+	trace_generic_event(dev_name, type, &payload->record);

Looks like it prints the whole thing now...

Quoted 11 lines
drivers/cxl/core/mbox.c +2 -0
+}
+
 
 
include/trace/events/cxl.h +0 -0
index 318ba5fe046e..82a8d3b750a2 100644
@@ -122,6 +122,114 @@ TRACE_EVENT(generic_event,
 
 

include/trace/events/cxl.h TRACE_EVENT(generic_event, · line 133

Quoted 10 lines
include/trace/events/cxl.h +10 -0
+#define CXL_GMER_VALID_CHANNEL				BIT(0)
+#define CXL_GMER_VALID_RANK				BIT(1)
+#define CXL_GMER_VALID_DEVICE				BIT(2)
+#define CXL_GMER_VALID_COMPONENT			BIT(3)
+#define show_valid_flags(flags)	__print_flags(flags, "|",		   \
+	{ CXL_GMER_VALID_CHANNEL,			"CHANNEL"	}, \
+	{ CXL_GMER_VALID_RANK,				"RANK"		}, \
+	{ CXL_GMER_VALID_DEVICE,			"DEVICE"	}, \
+	{ CXL_GMER_VALID_COMPONENT,			"COMPONENT"	}  \
+)

I'd still rather we only had the TP_printk only print those parts that are valid rather than the mess of having to go check the validity bit before deciding whether to take notice of the field. But meh, not that important given thats not the main intended way to consume this data.

Quoted 40 lines
include/trace/events/cxl.h +40 -0
+
+TRACE_EVENT(general_media,
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_gen_media *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		/* General Media */
+		__field(u64, phys_addr)
+		__field(u8, descriptor)
+		__field(u8, type)
+		__field(u8, transaction_type)
+		__field(u8, channel)
+		__field(u32, device)
+		__array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE)
+		__array(u8, reserved, CXL_EVENT_GEN_MED_RES_SIZE)
+		__field(u16, validity_flags)
+		__field(u8, rank) /* Out of order to pack trace record */
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+
+		/* General Media */
+		__entry->phys_addr = le64_to_cpu(rec->phys_addr);
+		__entry->descriptor = rec->descriptor;
+		__entry->type = rec->type;
+		__entry->transaction_type = rec->transaction_type;
+		__entry->channel = rec->channel;
+		__entry->rank = rec->rank;
+		__entry->device = get_unaligned_le24(rec->device);
+		memcpy(__entry->comp_id, &rec->component_id,
+			CXL_EVENT_GEN_MED_COMP_ID_SIZE);
+		memcpy(__entry->reserved, &rec->reserved,
+			CXL_EVENT_GEN_MED_RES_SIZE);
+		__entry->validity_flags = get_unaligned_le16(&rec->validity_flags);
+	),
author
#18 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Mon, 10 Oct 2022 15:41:25 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.1 defines the General Media Event Record.

Quoted 7 lines
Quoted 4 lines

Determine if the event read is a general media record and if so trace the record.

I'll review the rest of these with the assumption that the question of reserved bytes in tracepoints will get resolved before these go in.

Yea I'm removing them. I think I messed up somehow because I thought I had removed the reserved fields from the records. But perhaps that was only in my dreams... :-/ Sorry. :-)

Quoted 5 lines

One minor comment on a comment inline. Other than those lgtm

Thanks!

[snip]

Quoted 19 lines
Quoted 19 lines
(quoted diff) +18 -0
+/*
+ * General Media Event Record
+ * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43
+ */
+static const uuid_t gen_media_event_uuid =
+	UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
+		  0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6);
+
+static void cxl_trace_event_record(const char *dev_name,
+				   enum cxl_event_log_type type,
+				   struct cxl_get_event_payload *payload)
+{
+	uuid_t *id = &payload->record.hdr.id;
+
+	if (uuid_equal(id, &gen_media_event_uuid)) {
+		struct cxl_event_gen_media *rec =
+				(struct cxl_event_gen_media *)&payload->record;
+
Quoted 8 lines

Review

Quoted 6 lines
(quoted diff) +6 -0
+		trace_general_media(dev_name, type, rec);
+		return;
+	}
+
+	/* For unknown record types print just the header */
+	trace_generic_event(dev_name, type, &payload->record);

Looks like it prints the whole thing now...

An unknown record is dumped yes. I'm ok with skipping this but it was discussed early on that any unknown record would be passed through.

Quoted 13 lines
Quoted 11 lines
(quoted diff) +2 -0
+}
+
include/trace/events/cxl.h +0 -0
index 318ba5fe046e..82a8d3b750a2 100644
@@ -122,6 +122,114 @@ TRACE_EVENT(generic_event,
Quoted 16 lines

include/trace/events/cxl.h TRACE_EVENT(generic_event, · line 131

Quoted 10 lines
include/trace/events/cxl.h +10 -0
+#define CXL_GMER_VALID_CHANNEL				BIT(0)
+#define CXL_GMER_VALID_RANK				BIT(1)
+#define CXL_GMER_VALID_DEVICE				BIT(2)
+#define CXL_GMER_VALID_COMPONENT			BIT(3)
+#define show_valid_flags(flags)	__print_flags(flags, "|",		   \
+	{ CXL_GMER_VALID_CHANNEL,			"CHANNEL"	}, \
+	{ CXL_GMER_VALID_RANK,				"RANK"		}, \
+	{ CXL_GMER_VALID_DEVICE,			"DEVICE"	}, \
+	{ CXL_GMER_VALID_COMPONENT,			"COMPONENT"	}  \
+)

I'd still rather we only had the TP_printk only print those parts that are valid rather than the mess of having to go check the validity bit before deciding whether to take notice of the field. But meh, not that important given thats not the main intended way to consume this data.

Ok I spent some time really thinking about this and below is an attempt at that.

However, I must be missing something in what you are proposing because I don't like having extra space in the trace buffer to print into like this and I can't figure out where else to put a print buffer.

Also note that this will have no impact on most of the user space tools which use libtracefs as they will see all the fields and will need to do a similar decode.

Do you really think this is worth the effort?

Ira

commit 54c4ee67bcac6a38cbc9b0ea2ef952e197356dee

Author: Ira Weiny <[email protected]>
Date:   Fri Oct 14 16:15:27 2022 -0700

squash: Attempt to print only valid fields per Jonathan suggestion

drivers/cxl/core/mbox.c +22 -0
index 2add473fc168..9e15028af79c 100644
@@ -720,6 +720,28 @@ static const uuid_t gen_media_event_uuid =
        UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
                  0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6);
 
+const char *cxl_gem_print_valid(u8 *buf, int nr, u16 valid_flags, u8 channel,
+                               u8 rank, u32 device, u8 *comp_id)
+{
+       char *str = buf;
+       int rc = 0;
+
+       if (valid_flags & CXL_GMER_VALID_CHANNEL)
+               rc = snprintf(str, nr, "channel=%u ", channel);
+
+       if (valid_flags & CXL_GMER_VALID_RANK)
+               rc = snprintf(str + rc, nr - rc, "rank=%u ", rank);
+
+       if (valid_flags & CXL_GMER_VALID_DEVICE)
+               rc = snprintf(str + rc, nr - rc, "device=%u ", device);
+
+       if (valid_flags & CXL_GMER_VALID_COMPONENT)
+               rc = snprintf(str + rc, nr - rc, "comp_id=%*ph ",
+                             CXL_EVENT_GEN_MED_COMP_ID_SIZE, comp_id);
+
+       return str;
+}
+
 static void cxl_trace_event_record(const char *dev_name,
                                   enum cxl_event_log_type type,
                                   struct cxl_get_event_payload *payload)
drivers/cxl/cxlmem.h +7 -0
index 8fbd20d8a0b2..3d3bfef69d32 100644
@@ -429,6 +429,13 @@ struct cxl_event_gen_media {
        u8 reserved[0x2e];
 } __packed;
 
+#define CXL_GMER_VALID_CHANNEL                         BIT(0)
+#define CXL_GMER_VALID_RANK                            BIT(1)
+#define CXL_GMER_VALID_DEVICE                          BIT(2)
+#define CXL_GMER_VALID_COMPONENT                       BIT(3)
+const char *cxl_gem_print_valid(u8 *buf, int nr, u16 valid_flags, u8 channel,
+                               u8 rank, u32 device, u8 *comp_id);
+
 struct cxl_mbox_get_partition_info {
        __le64 active_volatile_cap;
        __le64 active_persistent_cap;
include/trace/events/cxl.h +8 -16
index e3e11c9055ba..27bb790cb685 100644
@@ -161,16 +161,7 @@ TRACE_EVENT(generic_event,
        { CXL_GMER_TRANS_INTERNAL_MEDIA_MANAGEMENT,     "Internal Media Management" }   \
 )
 
-#define CXL_GMER_VALID_CHANNEL                         BIT(0)
-#define CXL_GMER_VALID_RANK                            BIT(1)
-#define CXL_GMER_VALID_DEVICE                          BIT(2)
-#define CXL_GMER_VALID_COMPONENT                       BIT(3)
-#define show_valid_flags(flags)        __print_flags(flags, "|",                  \
-       { CXL_GMER_VALID_CHANNEL,                       "CHANNEL"       }, \
-       { CXL_GMER_VALID_RANK,                          "RANK"          }, \
-       { CXL_GMER_VALID_DEVICE,                        "DEVICE"        }, \
-       { CXL_GMER_VALID_COMPONENT,                     "COMPONENT"     }  \
-)
+#define CXL_VALID_PRINT_STR_LEN 512
 
 TRACE_EVENT(general_media,
 
@@ -191,6 +182,7 @@ TRACE_EVENT(general_media,
                __array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE)
                __field(u16, validity_flags)
                __field(u8, rank) /* Out of order to pack trace record */
+               __array(u8, str, CXL_VALID_PRINT_STR_LEN)
        ),
 
        TP_fast_assign(
@@ -209,17 +201,17 @@ TRACE_EVENT(general_media,
                __entry->validity_flags = get_unaligned_le16(&rec->validity_flags);
        ),
 
-       CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \
-               "trans_type='%s' channel=%u rank=%u device=%x comp_id=%s " \
-               "valid_flags='%s'",
+       CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' "     \
+               "trans_type='%s' %s",
                __entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK,
                (__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE",
                show_event_desc_flags(__entry->descriptor),
                show_mem_event_type(__entry->type),
                show_trans_type(__entry->transaction_type),
-               __entry->channel, __entry->rank, __entry->device,
-               __print_hex(__entry->comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE),
-               show_valid_flags(__entry->validity_flags)
+               cxl_gem_print_valid(__entry->str, CXL_VALID_PRINT_STR_LEN,
+                                   __entry->validity_flags, __entry->channel,
+                                   __entry->rank, __entry->device,
+                                   __entry->comp_id)
        )
 );
 
#19 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

Quoted 25 lines
Quoted 8 lines
Quoted 6 lines
include/trace/events/cxl.h +0 -0
index 318ba5fe046e..82a8d3b750a2 100644
@@ -122,6 +122,114 @@ TRACE_EVENT(generic_event,  
Quoted 16 lines

include/trace/events/cxl.h TRACE_EVENT(generic_event, · line 131

Quoted 10 lines
include/trace/events/cxl.h +10 -0
+#define CXL_GMER_VALID_CHANNEL				BIT(0)
+#define CXL_GMER_VALID_RANK				BIT(1)
+#define CXL_GMER_VALID_DEVICE				BIT(2)
+#define CXL_GMER_VALID_COMPONENT			BIT(3)
+#define show_valid_flags(flags)	__print_flags(flags, "|",		   \
+	{ CXL_GMER_VALID_CHANNEL,			"CHANNEL"	}, \
+	{ CXL_GMER_VALID_RANK,				"RANK"		}, \
+	{ CXL_GMER_VALID_DEVICE,			"DEVICE"	}, \
+	{ CXL_GMER_VALID_COMPONENT,			"COMPONENT"	}  \
+)  

I'd still rather we only had the TP_printk only print those parts that are valid rather than the mess of having to go check the validity bit before deciding whether to take notice of the field. But meh, not that important given thats not the main intended way to consume this data.

Quoted 6 lines

Ok I spent some time really thinking about this and below is an attempt at that.

However, I must be missing something in what you are proposing because I don't like having extra space in the trace buffer to print into like this and I can't figure out where else to put a print buffer.

Putting the space in the trace buffer definitely doesn't makes sense. Ah. Looking back the CCIX code I assumed it was serialized so could use a static buffer.

Looking at other similar cases though and we have a lot of use of trace_seq_printf() e.g. libata_trace_parse_status() though note there is some magic macro stuff in include/trace/events/libata.h to tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14

That seems to get you access to the actual buffer we are printing into in similar cases.

Quoted 6 lines

Also note that this will have no impact on most of the user space tools which use libtracefs as they will see all the fields and will need to do a similar decode.

Do you really think this is worth the effort?

With the allocation problem, definitely not. With that solved, it's not a huge amount of extra code. Also rather nicely 'documents' meaning of the valid bits.

I'm a kernel hacker. I like to not need much beyond echo and cat :)

Jonathan

Quoted 116 lines

Ira

commit 54c4ee67bcac6a38cbc9b0ea2ef952e197356dee

Author: Ira Weiny <[email protected]>
Date:   Fri Oct 14 16:15:27 2022 -0700

squash: Attempt to print only valid fields per Jonathan suggestion

drivers/cxl/core/mbox.c +22 -0
index 2add473fc168..9e15028af79c 100644
@@ -720,6 +720,28 @@ static const uuid_t gen_media_event_uuid =
        UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
                  0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6);
 
+const char *cxl_gem_print_valid(u8 *buf, int nr, u16 valid_flags, u8 channel,
+                               u8 rank, u32 device, u8 *comp_id)
+{
+       char *str = buf;
+       int rc = 0;
+
+       if (valid_flags & CXL_GMER_VALID_CHANNEL)
+               rc = snprintf(str, nr, "channel=%u ", channel);
+
+       if (valid_flags & CXL_GMER_VALID_RANK)
+               rc = snprintf(str + rc, nr - rc, "rank=%u ", rank);
+
+       if (valid_flags & CXL_GMER_VALID_DEVICE)
+               rc = snprintf(str + rc, nr - rc, "device=%u ", device);
+
+       if (valid_flags & CXL_GMER_VALID_COMPONENT)
+               rc = snprintf(str + rc, nr - rc, "comp_id=%*ph ",
+                             CXL_EVENT_GEN_MED_COMP_ID_SIZE, comp_id);
+
+       return str;
+}
+
 static void cxl_trace_event_record(const char *dev_name,
                                   enum cxl_event_log_type type,
                                   struct cxl_get_event_payload *payload)
drivers/cxl/cxlmem.h +7 -0
index 8fbd20d8a0b2..3d3bfef69d32 100644
@@ -429,6 +429,13 @@ struct cxl_event_gen_media {
        u8 reserved[0x2e];
 } __packed;
 
+#define CXL_GMER_VALID_CHANNEL                         BIT(0)
+#define CXL_GMER_VALID_RANK                            BIT(1)
+#define CXL_GMER_VALID_DEVICE                          BIT(2)
+#define CXL_GMER_VALID_COMPONENT                       BIT(3)
+const char *cxl_gem_print_valid(u8 *buf, int nr, u16 valid_flags, u8 channel,
+                               u8 rank, u32 device, u8 *comp_id);
+
 struct cxl_mbox_get_partition_info {
        __le64 active_volatile_cap;
        __le64 active_persistent_cap;
include/trace/events/cxl.h +8 -16
index e3e11c9055ba..27bb790cb685 100644
@@ -161,16 +161,7 @@ TRACE_EVENT(generic_event,
        { CXL_GMER_TRANS_INTERNAL_MEDIA_MANAGEMENT,     "Internal Media Management" }   \
 )
 
-#define CXL_GMER_VALID_CHANNEL                         BIT(0)
-#define CXL_GMER_VALID_RANK                            BIT(1)
-#define CXL_GMER_VALID_DEVICE                          BIT(2)
-#define CXL_GMER_VALID_COMPONENT                       BIT(3)
-#define show_valid_flags(flags)        __print_flags(flags, "|",                  \
-       { CXL_GMER_VALID_CHANNEL,                       "CHANNEL"       }, \
-       { CXL_GMER_VALID_RANK,                          "RANK"          }, \
-       { CXL_GMER_VALID_DEVICE,                        "DEVICE"        }, \
-       { CXL_GMER_VALID_COMPONENT,                     "COMPONENT"     }  \
-)
+#define CXL_VALID_PRINT_STR_LEN 512
 
 TRACE_EVENT(general_media,
 
@@ -191,6 +182,7 @@ TRACE_EVENT(general_media,
                __array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE)
                __field(u16, validity_flags)
                __field(u8, rank) /* Out of order to pack trace record */
+               __array(u8, str, CXL_VALID_PRINT_STR_LEN)
        ),
 
        TP_fast_assign(
@@ -209,17 +201,17 @@ TRACE_EVENT(general_media,
                __entry->validity_flags = get_unaligned_le16(&rec->validity_flags);
        ),
 
-       CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \
-               "trans_type='%s' channel=%u rank=%u device=%x comp_id=%s " \
-               "valid_flags='%s'",
+       CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' "     \
+               "trans_type='%s' %s",
                __entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK,
                (__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE",
                show_event_desc_flags(__entry->descriptor),
                show_mem_event_type(__entry->type),
                show_trans_type(__entry->transaction_type),
-               __entry->channel, __entry->rank, __entry->device,
-               __print_hex(__entry->comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE),
-               show_valid_flags(__entry->validity_flags)
+               cxl_gem_print_valid(__entry->str, CXL_VALID_PRINT_STR_LEN,
+                                   __entry->validity_flags, __entry->channel,
+                                   __entry->rank, __entry->device,
+                                   __entry->comp_id)
        )
 );
 
maintainer
#20 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

Quoted 8 lines · On Mon, 17 Oct 2022 17:37:17 +0100 Jonathan Cameron…

Looking at other similar cases though and we have a lot of use of trace_seq_printf() e.g. libata_trace_parse_status() though note there is some magic macro stuff in include/trace/events/libata.h to tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14

That seems to get you access to the actual buffer we are printing into in similar cases.

Looking at the code you linked to, I wonder why __print_flags() wasn't used?

For instance, you have:

const char *
libata_trace_parse_status(struct trace_seq *p, unsigned char status)

{
        const char *ret = trace_seq_buffer_ptr(p);

        trace_seq_printf(p, "{ ");
        if (status & ATA_BUSY)
                trace_seq_printf(p, "BUSY ");
        if (status & ATA_DRDY)
                trace_seq_printf(p, "DRDY ");
        if (status & ATA_DF)
                trace_seq_printf(p, "DF ");
        if (status & ATA_DSC)
                trace_seq_printf(p, "DSC ");
        if (status & ATA_DRQ)
                trace_seq_printf(p, "DRQ ");
        if (status & ATA_CORR)
                trace_seq_printf(p, "CORR ");
        if (status & ATA_SENSE)
                trace_seq_printf(p, "SENSE ");
        if (status & ATA_ERR)
                trace_seq_printf(p, "ERR ");
        trace_seq_putc(p, '}');
        trace_seq_putc(p, 0);

        return ret;
}

Which is just a re-implementation of:

__print_flags(status, " ",

	{ ATA_BUSY, "BUSY" },
	{ ATA_DRDY, "DRDY" },
	{ ATA_DF, "DF" },
	{ ATA_DSC, "DSC" },
	{ ATA_DRQ, "DRQ" },
	{ ATA_CORR, "CORR" },
	{ ATA_SENSE, "SENSE" },
	{ ATA_ERR, "ERR" })

The major difference between the two, is that libtraceevent will be able to parse the above and convert the status bits into strings, whereas using libata_trace_parse_status() will just give you a parsing error.

That is, perf and trace-cmd will not be able to parse it unless you write a separate plugin for libtraceevent to do it but that means you'll have duplicate code.

I know you just want echo and cat, but that will still work, and this will make it work for the tooling as well.

-- Steve

#21 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

Quoted 62 lines · On Mon, 17 Oct 2022 13:21:43 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 8 lines · On Mon, 17 Oct 2022 17:37:17 +0100 Jonathan Cameron…

Looking at other similar cases though and we have a lot of use of trace_seq_printf() e.g. libata_trace_parse_status() though note there is some magic macro stuff in include/trace/events/libata.h to tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14

That seems to get you access to the actual buffer we are printing into in similar cases.

Looking at the code you linked to, I wonder why __print_flags() wasn't used?

For instance, you have:

const char *
libata_trace_parse_status(struct trace_seq *p, unsigned char status)

{
        const char *ret = trace_seq_buffer_ptr(p);

        trace_seq_printf(p, "{ ");
        if (status & ATA_BUSY)
                trace_seq_printf(p, "BUSY ");
        if (status & ATA_DRDY)
                trace_seq_printf(p, "DRDY ");
        if (status & ATA_DF)
                trace_seq_printf(p, "DF ");
        if (status & ATA_DSC)
                trace_seq_printf(p, "DSC ");
        if (status & ATA_DRQ)
                trace_seq_printf(p, "DRQ ");
        if (status & ATA_CORR)
                trace_seq_printf(p, "CORR ");
        if (status & ATA_SENSE)
                trace_seq_printf(p, "SENSE ");
        if (status & ATA_ERR)
                trace_seq_printf(p, "ERR ");
        trace_seq_putc(p, '}');
        trace_seq_putc(p, 0);

        return ret;
}

Which is just a re-implementation of:

__print_flags(status, " ",

	{ ATA_BUSY, "BUSY" },
	{ ATA_DRDY, "DRDY" },
	{ ATA_DF, "DF" },
	{ ATA_DSC, "DSC" },
	{ ATA_DRQ, "DRQ" },
	{ ATA_CORR, "CORR" },
	{ ATA_SENSE, "SENSE" },
	{ ATA_ERR, "ERR" })

The major difference between the two, is that libtraceevent will be able to parse the above and convert the status bits into strings, whereas using libata_trace_parse_status() will just give you a parsing error.

Quoted 6 lines

That is, perf and trace-cmd will not be able to parse it unless you write a separate plugin for libtraceevent to do it but that means you'll have duplicate code.

I know you just want echo and cat, but that will still work, and this will make it work for the tooling as well.

Excellent point, though in the case we are interested in for CXL, __print_flags() is not enough.

We have a mass of fields that only contain something useful to print if the valid bits in a mask are set. I just pulled that example to show how trace_seq_printf() could be used to achieve optional printing as opposed to current situation where the reader of the print has to interpret the mask to work out if fields contain anything useful.

To do something nice with them in perf (well probably ras-daemon in this case) we'll have to parse the valid bits anyway so effectively write such a plugin. There we need to do a bunch of mangling to get the events stored in a DB anyway, so this isn't a huge overhead.

Jonathan

Quoted 3 lines

-- Steve

author
#22 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

Quoted 81 lines · Jonathan Cameron
Quoted 62 lines · On Mon, 17 Oct 2022 13:21:43 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 8 lines · On Mon, 17 Oct 2022 17:37:17 +0100 Jonathan Cameron…

Looking at other similar cases though and we have a lot of use of trace_seq_printf() e.g. libata_trace_parse_status() though note there is some magic macro stuff in include/trace/events/libata.h to tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14

That seems to get you access to the actual buffer we are printing into in similar cases.

Looking at the code you linked to, I wonder why __print_flags() wasn't used?

For instance, you have:

const char *
libata_trace_parse_status(struct trace_seq *p, unsigned char status)

{
        const char *ret = trace_seq_buffer_ptr(p);

        trace_seq_printf(p, "{ ");
        if (status & ATA_BUSY)
                trace_seq_printf(p, "BUSY ");
        if (status & ATA_DRDY)
                trace_seq_printf(p, "DRDY ");
        if (status & ATA_DF)
                trace_seq_printf(p, "DF ");
        if (status & ATA_DSC)
                trace_seq_printf(p, "DSC ");
        if (status & ATA_DRQ)
                trace_seq_printf(p, "DRQ ");
        if (status & ATA_CORR)
                trace_seq_printf(p, "CORR ");
        if (status & ATA_SENSE)
                trace_seq_printf(p, "SENSE ");
        if (status & ATA_ERR)
                trace_seq_printf(p, "ERR ");
        trace_seq_putc(p, '}');
        trace_seq_putc(p, 0);

        return ret;
}

Which is just a re-implementation of:

__print_flags(status, " ",

	{ ATA_BUSY, "BUSY" },
	{ ATA_DRDY, "DRDY" },
	{ ATA_DF, "DF" },
	{ ATA_DSC, "DSC" },
	{ ATA_DRQ, "DRQ" },
	{ ATA_CORR, "CORR" },
	{ ATA_SENSE, "SENSE" },
	{ ATA_ERR, "ERR" })

The major difference between the two, is that libtraceevent will be able to parse the above and convert the status bits into strings, whereas using libata_trace_parse_status() will just give you a parsing error.

Quoted 6 lines

That is, perf and trace-cmd will not be able to parse it unless you write a separate plugin for libtraceevent to do it but that means you'll have duplicate code.

I know you just want echo and cat, but that will still work, and this will make it work for the tooling as well.

Excellent point, though in the case we are interested in for CXL, __print_flags() is not enough.

We have a mass of fields that only contain something useful to print if the valid bits in a mask are set. I just pulled that example to show how trace_seq_printf() could be used to achieve optional printing as opposed to current situation where the reader of the print has to interpret the mask to work out if fields contain anything useful.

Quoted 4 lines

To do something nice with them in perf (well probably ras-daemon in this case) we'll have to parse the valid bits anyway so effectively write such a plugin. There we need to do a bunch of mangling to get the events stored in a DB anyway, so this isn't a huge overhead.

Given this information I think I'm going to punt on this and take your reviewed by on the code as it is.

We can certainly try to change it later but for now I think it serves our purpose. Better to focus on getting the code working with irq's.

Ira

maintainer
#23 lore raw

Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record

In reply to [email protected]

drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 728

Quoted 6 lines · On Mon, 10 Oct 2022 15:41:25 -0700 [email protected] wrote:
drivers/cxl/core/mbox.c +6 -0
+static void cxl_trace_event_record(const char *dev_name,
+				   enum cxl_event_log_type type,
+				   struct cxl_get_event_payload *payload)
+{
+	uuid_t *id = &payload->record.hdr.id;
+

Perhaps you want to add here:

	if (!trace_cxl_general_media_enabled() && !trace_clx_generic_event_enabled())
		return;

This way the below logic is only executed if either event is enabled. The above uses static_branches, so if the architecture supports them, they are not compare and branch, but jumps and/or nops.

-- Steve

Quoted 11 lines
drivers/cxl/core/mbox.c +11 -0
+	if (uuid_equal(id, &gen_media_event_uuid)) {
+		struct cxl_event_gen_media *rec =
+				(struct cxl_event_gen_media *)&payload->record;
+
+		trace_general_media(dev_name, type, rec);
+		return;
+	}
+
+	/* For unknown record types print just the header */
+	trace_generic_event(dev_name, type, &payload->record);
+}
#24 lore raw

RFC PATCH v2 6/11 cxl/mem: Trace DRAM Event Record

In reply to [email protected]

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.2 defines the DRAM Event Record.

Determine if the event read is a DRAM event record and if so trace the record.

---
Changes from RFC:
Add reserved byte data
Use new CXL header macros
Jonathan
Use get_unaligned_le{24,16}() for unaligned fields Use 'else if'
Dave Jiang
s/cxl_dram_event/dram s/cxl_evt_dram_rec/cxl_event_dram
Adjust for new phys addr mask

---
 drivers/cxl/core/mbox.c    | 14 ++++++
 drivers/cxl/cxlmem.h       | 24 ++++++++++
 include/trace/events/cxl.h | 94 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 132 insertions(+)
drivers/cxl/core/mbox.c +14 -0
index 1097250c115a..72b589edc074 100644
@@ -720,6 +720,14 @@ static const uuid_t gen_media_event_uuid =
 	UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
 		  0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6);
 
+/*
+ * DRAM Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44
+ */
+static const uuid_t dram_event_uuid =
+	UUID_INIT(0x601dcbb3, 0x9c06, 0x4eab,
+		  0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24);
+
 static void cxl_trace_event_record(const char *dev_name,
 				   enum cxl_event_log_type type,
 				   struct cxl_get_event_payload *payload)
@@ -732,6 +740,12 @@ static void cxl_trace_event_record(const char *dev_name,
 
 		trace_general_media(dev_name, type, rec);
 		return;
+	} else if (uuid_equal(id, &dram_event_uuid)) {
+		struct cxl_event_dram *rec =
+				(struct cxl_event_dram *)&payload->record;
+
+		trace_dram(dev_name, type, rec);
+		return;
 	}
 
 	/* For unknown record types print just the header */
drivers/cxl/cxlmem.h +24 -0
index b5c120bd4068..d0253e5f1187 100644
@@ -431,6 +431,30 @@ struct cxl_event_gen_media {
 	u8 reserved[CXL_EVENT_GEN_MED_RES_SIZE];
 } __packed;
 
+/*
+ * DRAM Event Record - DER
+ * CXL rev 3.0 section 8.2.9.2.1.2; Table 3-44
+ */
+#define CXL_EVENT_DER_CORRECTION_MASK_SIZE	0x20
+#define CXL_EVENT_DER_RES_SIZE			0x17
+struct cxl_event_dram {
+	struct cxl_event_record_hdr hdr;
+	__le64 phys_addr;
+	u8 descriptor;
+	u8 type;
+	u8 transaction_type;
+	u8 validity_flags[2];
+	u8 channel;
+	u8 rank;
+	u8 nibble_mask[3];
+	u8 bank_group;
+	u8 bank;
+	u8 row[3];
+	u8 column[2];
+	u8 correction_mask[CXL_EVENT_DER_CORRECTION_MASK_SIZE];
+	u8 reserved[CXL_EVENT_DER_RES_SIZE];
+} __packed;
+
 struct cxl_mbox_get_partition_info {
 	__le64 active_volatile_cap;
 	__le64 active_persistent_cap;
include/trace/events/cxl.h +94 -0

100 lines not on this page · raw message

Signature · 1 line
2.37.2
Reviewed-by on v2 6/11
#25 lore raw

Re: [RFC V2 PATCH 06/11] cxl/mem: Trace DRAM Event Record

Quoted 9 lines · On Mon, 10 Oct 2022 15:41:26 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.2 defines the DRAM Event Record.

Determine if the event read is a DRAM event record and if so trace the record.

Trivial comments inline

Quoted 21 lines
include/trace/events/cxl.h +12 -0
index 82a8d3b750a2..7a90cfea348b 100644
@@ -230,6 +230,100 @@ TRACE_EVENT(general_media,
 		)
 );
 
 
+
+TRACE_EVENT(dram,
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_dram *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		/* DRAM */
+		__field(u64, phys_addr)

include/trace/events/cxl.h TRACE_EVENT(general_media, · line 255

Quoted 10 lines
include/trace/events/cxl.h +10 -0
+		__field(u8, descriptor)
+		__field(u8, type)
+		__field(u8, transaction_type)
+		__field(u8, channel)
+		__field(u16, validity_flags)
+		__field(u16, column)	/* Out of order to pack trace record */
+		__field(u32, nibble_mask)
+		__field(u32, row)
+		__array(u8, cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE)
+		__array(u8, reserved, CXL_EVENT_DER_RES_SIZE)

If we are going to have this, why not put it at the end? Will that affect the packing badly?

Quoted 33 lines
include/trace/events/cxl.h +33 -0
+		__field(u8, rank)	/* Out of order to pack trace record */
+		__field(u8, bank_group)	/* Out of order to pack trace record */
+		__field(u8, bank)	/* Out of order to pack trace record */
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+
+		/* DRAM */
+		__entry->phys_addr = le64_to_cpu(rec->phys_addr);
+		__entry->descriptor = rec->descriptor;
+		__entry->type = rec->type;
+		__entry->transaction_type = rec->transaction_type;
+		__entry->validity_flags = get_unaligned_le16(rec->validity_flags);
+		__entry->channel = rec->channel;
+		__entry->rank = rec->rank;
+		__entry->nibble_mask = get_unaligned_le24(rec->nibble_mask);
+		__entry->bank_group = rec->bank_group;
+		__entry->bank = rec->bank;
+		__entry->row = get_unaligned_le24(rec->row);
+		__entry->column = get_unaligned_le16(rec->column);
+		memcpy(__entry->cor_mask, &rec->correction_mask,
+			CXL_EVENT_DER_CORRECTION_MASK_SIZE);
+		memcpy(__entry->reserved, &rec->reserved,
+			CXL_EVENT_DER_RES_SIZE);
+	),
+
+	CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \
+		"trans_type='%s' channel=%u rank=%u nibble_mask=%x " \
+		"bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \
+		"valid_flags='%s' reserved=%s",
+		__entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK,
+		(__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE",

include/trace/events/cxl.h TRACE_EVENT(general_media, · line 324

Quoted 10 lines
include/trace/events/cxl.h +10 -0
+		show_event_desc_flags(__entry->descriptor),
+		show_mem_event_type(__entry->type),
+		show_trans_type(__entry->transaction_type),
+		__entry->channel, __entry->rank, __entry->nibble_mask,
+		__entry->bank_group, __entry->bank,
+		__entry->row, __entry->column,
+		__print_hex(__entry->cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE),
+		show_dram_valid_flags(__entry->validity_flags),
+		__print_hex(__entry->reserved, CXL_EVENT_DER_RES_SIZE)
+		)

Probably one less tab on that trailing )?

Quoted 5 lines
include/trace/events/cxl.h +2 -0
+);
+
 #endif /* _CXL_TRACE_EVENTS_H */
 
 /* This part must be outside protection */
author
#26 lore raw

Re: [RFC V2 PATCH 06/11] cxl/mem: Trace DRAM Event Record

Quoted 43 lines · Jonathan Cameron
Quoted 9 lines · On Mon, 10 Oct 2022 15:41:26 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.2 defines the DRAM Event Record.

Determine if the event read is a DRAM event record and if so trace the record.

Trivial comments inline

Quoted 26 lines
include/trace/events/cxl.h +17 -0
index 82a8d3b750a2..7a90cfea348b 100644
@@ -230,6 +230,100 @@ TRACE_EVENT(general_media,
 		)
 );
 
 
+
+TRACE_EVENT(dram,
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_dram *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		/* DRAM */
+		__field(u64, phys_addr)
+		__field(u8, descriptor)
+		__field(u8, type)
+		__field(u8, transaction_type)
+		__field(u8, channel)
+		__field(u16, validity_flags)
Quoted 8 lines

include/trace/events/cxl.h TRACE_EVENT(general_media, · line 281

Quoted 5 lines
include/trace/events/cxl.h +5 -0
+		__field(u16, column)	/* Out of order to pack trace record */
+		__field(u32, nibble_mask)
+		__field(u32, row)
+		__array(u8, cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE)
+		__array(u8, reserved, CXL_EVENT_DER_RES_SIZE)

If we are going to have this, why not put it at the end? Will that affect the packing badly?

I removed it.

[snip]

Quoted 10 lines
Quoted 10 lines
include/trace/events/cxl.h +10 -0
+
+	CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \
+		"trans_type='%s' channel=%u rank=%u nibble_mask=%x " \
+		"bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \
+		"valid_flags='%s' reserved=%s",
+		__entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK,
+		(__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE",
+		show_event_desc_flags(__entry->descriptor),
+		show_mem_event_type(__entry->type),
+		show_trans_type(__entry->transaction_type),
Quoted 8 lines

include/trace/events/cxl.h TRACE_EVENT(general_media, · line 324

Quoted 7 lines
include/trace/events/cxl.h +7 -0
+		__entry->channel, __entry->rank, __entry->nibble_mask,
+		__entry->bank_group, __entry->bank,
+		__entry->row, __entry->column,
+		__print_hex(__entry->cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE),
+		show_dram_valid_flags(__entry->validity_flags),
+		__print_hex(__entry->reserved, CXL_EVENT_DER_RES_SIZE)
+		)

Probably one less tab on that trailing )?

Done.

Thanks!
Ira

maintainer
#27 lore raw

Re: [RFC V2 PATCH 06/11] cxl/mem: Trace DRAM Event Record

In reply to [email protected]
Quoted 1 line · On Mon, 10 Oct 2022 15:41:26 -0700 [email protected] wrote:

+TRACE_EVENT(dram,

Call this "cxl_dram"

-- Steve

Quoted 65 lines
include/trace/events/cxl.h +65 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_dram *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+		/* DRAM */
+		__field(u64, phys_addr)
+		__field(u8, descriptor)
+		__field(u8, type)
+		__field(u8, transaction_type)
+		__field(u8, channel)
+		__field(u16, validity_flags)
+		__field(u16, column)	/* Out of order to pack trace record */
+		__field(u32, nibble_mask)
+		__field(u32, row)
+		__array(u8, cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE)
+		__array(u8, reserved, CXL_EVENT_DER_RES_SIZE)
+		__field(u8, rank)	/* Out of order to pack trace record */
+		__field(u8, bank_group)	/* Out of order to pack trace record */
+		__field(u8, bank)	/* Out of order to pack trace record */
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+
+		/* DRAM */
+		__entry->phys_addr = le64_to_cpu(rec->phys_addr);
+		__entry->descriptor = rec->descriptor;
+		__entry->type = rec->type;
+		__entry->transaction_type = rec->transaction_type;
+		__entry->validity_flags = get_unaligned_le16(rec->validity_flags);
+		__entry->channel = rec->channel;
+		__entry->rank = rec->rank;
+		__entry->nibble_mask = get_unaligned_le24(rec->nibble_mask);
+		__entry->bank_group = rec->bank_group;
+		__entry->bank = rec->bank;
+		__entry->row = get_unaligned_le24(rec->row);
+		__entry->column = get_unaligned_le16(rec->column);
+		memcpy(__entry->cor_mask, &rec->correction_mask,
+			CXL_EVENT_DER_CORRECTION_MASK_SIZE);
+		memcpy(__entry->reserved, &rec->reserved,
+			CXL_EVENT_DER_RES_SIZE);
+	),
+
+	CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \
+		"trans_type='%s' channel=%u rank=%u nibble_mask=%x " \
+		"bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \
+		"valid_flags='%s' reserved=%s",
+		__entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK,
+		(__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE",
+		show_event_desc_flags(__entry->descriptor),
+		show_mem_event_type(__entry->type),
+		show_trans_type(__entry->transaction_type),
+		__entry->channel, __entry->rank, __entry->nibble_mask,
+		__entry->bank_group, __entry->bank,
+		__entry->row, __entry->column,
+		__print_hex(__entry->cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE),
+		show_dram_valid_flags(__entry->validity_flags),
+		__print_hex(__entry->reserved, CXL_EVENT_DER_RES_SIZE)
+		)
+);
+
#28 lore raw

RFC PATCH v2 7/11 cxl/mem: Trace Memory Module Event Record

In reply to [email protected]

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.3 defines the Memory Module Event Record.

Determine if the event read is memory module record and if so trace the record.

---
Changes from RFC:
Clean up spec reference
Add reserved data
Use new CXL header macros
Jonathan
Use else if
Use get_unaligned_le*() for unaligned fields
Dave Jiang
s/cxl_mem_mod_event/memory_module s/cxl_evt_mem_mod_rec/cxl_event_mem_module

---
 drivers/cxl/core/mbox.c    |  14 ++++
 drivers/cxl/cxlmem.h       |  27 +++++++
 include/trace/events/cxl.h | 146 +++++++++++++++++++++++++++++++++++++
 3 files changed, 187 insertions(+)
drivers/cxl/core/mbox.c +14 -0
index 72b589edc074..6b3119bc83d2 100644
@@ -728,6 +728,14 @@ static const uuid_t dram_event_uuid =
 	UUID_INIT(0x601dcbb3, 0x9c06, 0x4eab,
 		  0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24);
 
+/*
+ * Memory Module Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
+ */
+static const uuid_t mem_mod_event_uuid =
+	UUID_INIT(0xfe927475, 0xdd59, 0x4339,
+		  0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74);
+
 static void cxl_trace_event_record(const char *dev_name,
 				   enum cxl_event_log_type type,
 				   struct cxl_get_event_payload *payload)
@@ -746,6 +754,12 @@ static void cxl_trace_event_record(const char *dev_name,
 
 		trace_dram(dev_name, type, rec);
 		return;
+	} else if (uuid_equal(id, &mem_mod_event_uuid)) {
+		struct cxl_event_mem_module *rec =
+				(struct cxl_event_mem_module *)&payload->record;
+
+		trace_memory_module(dev_name, type, rec);
+		return;
 	}
 
 	/* For unknown record types print just the header */
drivers/cxl/cxlmem.h +27 -0
index d0253e5f1187..79b3fac6d9ef 100644
@@ -455,6 +455,33 @@ struct cxl_event_dram {
 	u8 reserved[CXL_EVENT_DER_RES_SIZE];
 } __packed;
 
+/*
+ * Get Health Info Record
+ * CXL rev 3.0 section 8.2.9.8.3.1; Table 8-100
+ */
+struct cxl_get_health_info {
+	u8 health_status;
+	u8 media_status;
+	u8 add_status;
+	u8 life_used;
+	u8 device_temp[2];
+	u8 dirty_shutdown_cnt[4];
+	u8 cor_vol_err_cnt[4];
+	u8 cor_per_err_cnt[4];
+} __packed;
+
+/*
+ * Memory Module Event Record
+ * CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45
+ */
+#define CXL_EVENT_MEM_MOD_RES_SIZE	0x3d
+struct cxl_event_mem_module {
+	struct cxl_event_record_hdr hdr;
+	u8 event_type;
+	struct cxl_get_health_info info;
+	u8 reserved[CXL_EVENT_MEM_MOD_RES_SIZE];
+} __packed;
+
 struct cxl_mbox_get_partition_info {
 	__le64 active_volatile_cap;
 	__le64 active_persistent_cap;
include/trace/events/cxl.h +146 -0

152 lines not on this page · raw message

Signature · 1 line
2.37.2
Reviewed-by on v2 7/11
#29 lore raw

Re: [RFC V2 PATCH 07/11] cxl/mem: Trace Memory Module Event Record

Quoted 9 lines · On Mon, 10 Oct 2022 15:41:27 -0700 [email protected] wrote:

From: Ira Weiny <[email protected]>

CXL rev 3.0 section 8.2.9.2.1.3 defines the Memory Module Event Record.

Determine if the event read is memory module record and if so trace the record.

A few trivial comments inline. I'm happy either way

Quoted 16 lines
include/trace/events/cxl.h +14 -0
+#define show_add_status(as) __print_symbolic(as,	   \
+	{ CXL_DHI_AS_NORMAL,		"Normal"	}, \
+	{ CXL_DHI_AS_WARNING,		"Warning"	}, \
+	{ CXL_DHI_AS_CRITICAL,		"Critical"	}  \
+)
+
+#define CXL_DHI_AS_LIFE_USED(as)			(as & 0x3)
+#define CXL_DHI_AS_DEV_TEMP(as)				((as & 0xC) >> 2)
+#define CXL_DHI_AS_COR_VOL_ERR_CNT(as)			((as & 0x10) >> 4)
+#define CXL_DHI_AS_COR_PER_ERR_CNT(as)			((as & 0x20) >> 5)
 
 
+	),
+
+	CXL_EVT_TP_printk("evt_type='%s' health_status='%s' media_status='%s' " \
+		"as_life_used=%s as_dev_temp=%s as_cor_vol_err_cnt=%s " \

include/trace/events/cxl.h TRACE_EVENT(dram, · line 464

Quoted 10 lines
include/trace/events/cxl.h +10 -0
+		"as_cor_per_err_cnt=%s life_used=%u dev_temp=%d " \
+		"dirty_shutdown_cnt=%u cor_vol_err_cnt=%u cor_per_err_cnt=%u " \
+		"reserved=%s",
+		show_dev_evt_type(__entry->event_type),
+		show_health_status_flags(__entry->health_status),
+		show_media_status(__entry->media_status),
+		show_add_status(CXL_DHI_AS_LIFE_USED(__entry->add_status)),
+		show_add_status(CXL_DHI_AS_DEV_TEMP(__entry->add_status)),
+		show_add_status(CXL_DHI_AS_COR_VOL_ERR_CNT(__entry->add_status)),
+		show_add_status(CXL_DHI_AS_COR_PER_ERR_CNT(__entry->add_status)),

A little nasty to use same show_add_status() for the 2 bit and 1 bit versions. Obviously it works, but maybe it's worth two macros for the 1 and 2 bit version?

include/trace/events/cxl.h TRACE_EVENT(dram, · line 469

Quoted 5 lines
include/trace/events/cxl.h +5 -0
+		__entry->life_used, __entry->device_temp,
+		__entry->dirty_shutdown_cnt, __entry->cor_vol_err_cnt,
+		__entry->cor_per_err_cnt,
+		__print_hex(__entry->reserved, CXL_EVENT_MEM_MOD_RES_SIZE)
+		)

Aligned one tab too far?

maintainer
#30 lore raw

Re: [RFC V2 PATCH 07/11] cxl/mem: Trace Memory Module Event Record

In reply to [email protected]
Quoted 1 line · On Mon, 10 Oct 2022 15:41:27 -0700 [email protected] wrote:

+TRACE_EVENT(memory_module,

Make sure all your new events have the "cxl_" prefix. "cxl_memory_module".

This goes for all events in this series.

Thanks,

-- Steve

Quoted 64 lines
include/trace/events/cxl.h +64 -0
+
+	TP_PROTO(const char *dev_name, enum cxl_event_log_type log,
+		 struct cxl_event_mem_module *rec),
+
+	TP_ARGS(dev_name, log, rec),
+
+	TP_STRUCT__entry(
+		CXL_EVT_TP_entry
+
+		/* Memory Module Event */
+		__field(u8, event_type)
+
+		/* Device Health Info */
+		__field(u8, health_status)
+		__field(u8, media_status)
+		__field(u8, life_used)
+		__field(u32, dirty_shutdown_cnt)
+		__field(u32, cor_vol_err_cnt)
+		__field(u32, cor_per_err_cnt)
+		__field(s16, device_temp)
+		__field(u8, add_status)
+
+		__array(u8, reserved, CXL_EVENT_MEM_MOD_RES_SIZE)
+	),
+
+	TP_fast_assign(
+		CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr);
+
+		/* Memory Module Event */
+		__entry->event_type = rec->event_type;
+
+		/* Device Health Info */
+		__entry->health_status = rec->info.health_status;
+		__entry->media_status = rec->info.media_status;
+		__entry->life_used = rec->info.life_used;
+		__entry->dirty_shutdown_cnt = get_unaligned_le32(rec->info.dirty_shutdown_cnt);
+		__entry->cor_vol_err_cnt = get_unaligned_le32(rec->info.cor_vol_err_cnt);
+		__entry->cor_per_err_cnt = get_unaligned_le32(rec->info.cor_per_err_cnt);
+		__entry->device_temp = get_unaligned_le16(rec->info.device_temp);
+		__entry->add_status = rec->info.add_status;
+		memcpy(__entry->reserved, &rec->reserved,
+			CXL_EVENT_MEM_MOD_RES_SIZE);
+	),
+
+	CXL_EVT_TP_printk("evt_type='%s' health_status='%s' media_status='%s' " \
+		"as_life_used=%s as_dev_temp=%s as_cor_vol_err_cnt=%s " \
+		"as_cor_per_err_cnt=%s life_used=%u dev_temp=%d " \
+		"dirty_shutdown_cnt=%u cor_vol_err_cnt=%u cor_per_err_cnt=%u " \
+		"reserved=%s",
+		show_dev_evt_type(__entry->event_type),
+		show_health_status_flags(__entry->health_status),
+		show_media_status(__entry->media_status),
+		show_add_status(CXL_DHI_AS_LIFE_USED(__entry->add_status)),
+		show_add_status(CXL_DHI_AS_DEV_TEMP(__entry->add_status)),
+		show_add_status(CXL_DHI_AS_COR_VOL_ERR_CNT(__entry->add_status)),
+		show_add_status(CXL_DHI_AS_COR_PER_ERR_CNT(__entry->add_status)),
+		__entry->life_used, __entry->device_temp,
+		__entry->dirty_shutdown_cnt, __entry->cor_vol_err_cnt,
+		__entry->cor_per_err_cnt,
+		__print_hex(__entry->reserved, CXL_EVENT_MEM_MOD_RES_SIZE)
+		)
+);
+
+
#31 lore raw

RFC PATCH v2 8/11 cxl/test: Add generic mock events

In reply to [email protected]

From: Ira Weiny <[email protected]>

Facilitate testing basic Get/Clear Event functionality by creating multiple logs and generic events with made up UUID's.

Data is completely made up with data patterns which should be easy to spot in trace output.

A single sysfs entry resets the event data and triggers collecting the events for testing.

Test traces are easy to obtain with a small script such as this:

#!/bin/bash -x

devices=find /sys/devices/platform -name cxl_mem*

# Turn on tracing
echo "" > /sys/kernel/tracing/trace echo 1 > /sys/kernel/tracing/events/cxl/enable echo 1 > /sys/kernel/tracing/tracing_on

# Generate fake interrupt
for device in $devices; do
echo 1 > $device/event_trigger
done

# Turn off tracing and report events
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace

---
Changes from RFC:
Separate out the event code
Adjust for struct changes.
Clean up devm_cxl_mock_event_logs()
Clean up naming and comments
Jonathan
Remove dynamic allocation of event logs Clean up comment
Remove unneeded xarray
Ensure event_trigger sysfs is valid prior to the driver going active.
Dan
Remove the fill/reset event sysfs as these operations can be done together

---
 tools/testing/cxl/test/Kbuild   |   2 +-
 tools/testing/cxl/test/events.c | 223 ++++++++++++++++++++++++++++++++
 tools/testing/cxl/test/events.h |   9 ++
 tools/testing/cxl/test/mem.c    |  32 +++++
 4 files changed, 265 insertions(+), 1 deletion(-)
 create mode 100644 tools/testing/cxl/test/events.c
 create mode 100644 tools/testing/cxl/test/events.h
tools/testing/cxl/test/Kbuild +1 -1
index 4e59e2c911f6..64b14b83d8d9 100644
@@ -7,4 +7,4 @@ obj-m += cxl_mock_mem.o
 
 cxl_test-y := cxl.o
 cxl_mock-y := mock.o
-cxl_mock_mem-y := mem.o
+cxl_mock_mem-y := mem.o events.o
tools/testing/cxl/test/events.c new file +223 -0

223 lines not on this page · raw message

tools/testing/cxl/test/events.h new file +9 -0
new file mode 100644
index 000000000000..b3149b38d6df
@@ -0,0 +1,9 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+
+#include <cxlmem.h>
+
+int mock_get_event(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd);
+int mock_clear_event(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd);
+void cxl_mock_add_event_logs(struct cxl_dev_state *cxlds);
+void cxl_mock_remove_event_logs(struct device *dev);
+void cxl_mock_event_trigger(struct device *dev);
tools/testing/cxl/test/mem.c +32 -0
index e2f5445d24ff..9768f5132a5c 100644
@@ -8,6 +8,7 @@
 #include <linux/sizes.h>
 #include <linux/bits.h>
 #include <cxlmem.h>
+#include "events.h"
 
 #define LSA_SIZE SZ_128K
 #define DEV_SIZE SZ_2G
@@ -224,6 +225,12 @@ static int cxl_mock_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *
 	case CXL_MBOX_OP_GET_PARTITION_INFO:
 		rc = mock_partition_info(cxlds, cmd);
 		break;
+	case CXL_MBOX_OP_GET_EVENT_RECORD:
+		rc = mock_get_event(cxlds, cmd);
+		break;
+	case CXL_MBOX_OP_CLEAR_EVENT_RECORD:
+		rc = mock_clear_event(cxlds, cmd);
+		break;
 	case CXL_MBOX_OP_SET_LSA:
 		rc = mock_set_lsa(cxlds, cmd);
 		break;
@@ -245,6 +252,21 @@ static void label_area_release(void *lsa)
 	vfree(lsa);
 }
 
+static ssize_t event_trigger_store(struct device *dev,
+				   struct device_attribute *attr,
+				   const char *buf, size_t count)
+{
+	cxl_mock_event_trigger(dev);
+	return count;
+}
+static DEVICE_ATTR_WO(event_trigger);
+
+static struct attribute *cxl_mock_event_attrs[] = {
+	&dev_attr_event_trigger.attr,
+	NULL
+};
+ATTRIBUTE_GROUPS(cxl_mock_event);
+
 static int cxl_mock_mem_probe(struct platform_device *pdev)
 {
 	struct device *dev = &pdev->dev;
@@ -281,6 +303,8 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	if (rc)
 		return rc;
 
+	cxl_mock_add_event_logs(cxlds);
+
 	cxlmd = devm_cxl_add_memdev(cxlds);
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
@@ -293,6 +317,12 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)
 	return 0;
 }
 
+static int cxl_mock_mem_remove(struct platform_device *pdev)
+{
+	cxl_mock_remove_event_logs(&pdev->dev);
+	return 0;
+}
+
 static const struct platform_device_id cxl_mock_mem_ids[] = {
 	{ .name = "cxl_mem", },
 	{ },
@@ -301,9 +331,11 @@ MODULE_DEVICE_TABLE(platform, cxl_mock_mem_ids);
 
 static struct platform_driver cxl_mock_mem_driver = {
 	.probe = cxl_mock_mem_probe,
+	.remove = cxl_mock_mem_remove,
 	.id_table = cxl_mock_mem_ids,
 	.driver = {
 		.name = KBUILD_MODNAME,
+		.dev_groups = cxl_mock_event_groups,
 	},
 };
 
Signature · 1 line
2.37.2
#32 lore raw

RFC PATCH v2 9/11 cxl/test: Add specific events

In reply to [email protected]

From: Ira Weiny <[email protected]>

Each type of event has different trace point outputs.

Add mock General Media Event, DRAM event, and Memory Module Event records to the mock list of events returned.

---
Changes from RFC:
Adjust for struct changes
adjust for unaligned fields

---
 tools/testing/cxl/test/events.c | 70 +++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
tools/testing/cxl/test/events.c +70 -0
index 1913c321d16c..1d3dbeaf7794 100644
@@ -189,6 +189,70 @@ struct cxl_event_record_raw hardware_replace = {
 	.data = { 0xDE, 0xAD, 0xBE, 0xEF },
 };
 
+struct cxl_event_gen_media gen_media = {
+	.hdr = {
+		.id = UUID_INIT(0xfbcd0a77, 0xc260, 0x417f,
+				0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6),
+		.length = sizeof(struct cxl_event_gen_media),
+		.flags[0] = CXL_EVENT_RECORD_FLAG_PERMANENT,
+		/* .handle = Set dynamically */
+		.related_handle = cpu_to_le16(0),
+	},
+	.phys_addr = cpu_to_le64(0x2000),
+	.descriptor = CXL_GMER_EVT_DESC_UNCORECTABLE_EVENT,
+	.type = CXL_GMER_MEM_EVT_TYPE_DATA_PATH_ERROR,
+	.transaction_type = CXL_GMER_TRANS_HOST_WRITE,
+	.validity_flags = { CXL_GMER_VALID_CHANNEL |
+			    CXL_GMER_VALID_RANK, 0 },
+	.channel = 1,
+	.rank = 30
+};
+
+struct cxl_event_dram dram = {
+	.hdr = {
+		.id = UUID_INIT(0x601dcbb3, 0x9c06, 0x4eab,
+				0xb8, 0xaf, 0x4e, 0x9b, 0xfb, 0x5c, 0x96, 0x24),
+		.length = sizeof(struct cxl_event_dram),
+		.flags[0] = CXL_EVENT_RECORD_FLAG_PERF_DEGRADED,
+		/* .handle = Set dynamically */
+		.related_handle = cpu_to_le16(0),
+	},
+	.phys_addr = cpu_to_le64(0x8000),
+	.descriptor = CXL_GMER_EVT_DESC_THRESHOLD_EVENT,
+	.type = CXL_GMER_MEM_EVT_TYPE_INV_ADDR,
+	.transaction_type = CXL_GMER_TRANS_INTERNAL_MEDIA_SCRUB,
+	.validity_flags = { CXL_DER_VALID_CHANNEL |
+			    CXL_DER_VALID_BANK_GROUP |
+			    CXL_DER_VALID_BANK |
+			    CXL_DER_VALID_COLUMN, 0 },
+	.channel = 1,
+	.bank_group = 5,
+	.bank = 2,
+	.column = { 0xDE, 0xAD},
+};
+
+struct cxl_event_mem_module mem_module = {
+	.hdr = {
+		.id = UUID_INIT(0xfe927475, 0xdd59, 0x4339,
+				0xa5, 0x86, 0x79, 0xba, 0xb1, 0x13, 0xb7, 0x74),
+		.length = sizeof(struct cxl_event_mem_module),
+		/* .handle = Set dynamically */
+		.related_handle = cpu_to_le16(0),
+	},
+	.event_type = CXL_MMER_TEMP_CHANGE,
+	.info = {
+		.health_status = CXL_DHI_HS_PERFORMANCE_DEGRADED,
+		.media_status = CXL_DHI_MS_ALL_DATA_LOST,
+		.add_status = (CXL_DHI_AS_CRITICAL << 2) |
+			      (CXL_DHI_AS_WARNING << 4) |
+			      (CXL_DHI_AS_WARNING << 5),
+		.device_temp = { 0xDE, 0xAD},
+		.dirty_shutdown_cnt = { 0xde, 0xad, 0xbe, 0xef },
+		.cor_vol_err_cnt = { 0xde, 0xad, 0xbe, 0xef },
+		.cor_per_err_cnt = { 0xde, 0xad, 0xbe, 0xef },
+	}
+};
+
 void cxl_mock_add_event_logs(struct cxl_dev_state *cxlds)
 {
 	struct device *dev = cxlds->dev;
@@ -207,8 +271,14 @@ void cxl_mock_add_event_logs(struct cxl_dev_state *cxlds)
 	}
 
 	event_store_add_event(mes, CXL_EVENT_TYPE_INFO, &maint_needed);
+	event_store_add_event(mes, CXL_EVENT_TYPE_INFO,
+			      (struct cxl_event_record_raw *)&gen_media);
+	event_store_add_event(mes, CXL_EVENT_TYPE_INFO,
+			      (struct cxl_event_record_raw *)&mem_module);
 
 	event_store_add_event(mes, CXL_EVENT_TYPE_FATAL, &hardware_replace);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FATAL,
+			      (struct cxl_event_record_raw *)&dram);
 
 }
 EXPORT_SYMBOL_GPL(cxl_mock_add_event_logs);
Signature · 1 line
2.37.2
#33 lore raw

RFC PATCH v2 10/11 cxl/test: Simulate event log overflow

In reply to [email protected]

From: Ira Weiny <[email protected]>

Log overflow is marked by a separate trace message.

Simulate a log with lots of messages and flag overflow until it is drained a bit.

---
Changes from RFC
Adjust for new struct changes

---
 tools/testing/cxl/test/events.c | 36 +++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)
tools/testing/cxl/test/events.c +36 -0
index 1d3dbeaf7794..21cbd2990bca 100644
@@ -68,11 +68,21 @@ static void event_store_add_event(struct mock_event_store *mes,
 	log->nr_events++;
 }
 
+static u16 log_overflow(struct mock_event_log *log)
+{
+	int cnt = log_rec_left(log) - 5;
+
+	if (cnt < 0)
+		return 0;
+	return cnt;
+}
+
 int mock_get_event(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd)
 {
 	struct cxl_get_event_payload *pl;
 	struct mock_event_log *log;
 	u8 log_type;
+	u16 nr_overflow;
 
 	/* Valid request? */
 	if (cmd->size_in != sizeof(log_type))
@@ -98,6 +108,20 @@ int mock_get_event(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd)
 	if (log_rec_left(log) > 1)
 		pl->flags |= CXL_GET_EVENT_FLAG_MORE_RECORDS;
 
+	nr_overflow = log_overflow(log);
+	if (nr_overflow) {
+		u64 ns;
+
+		pl->flags |= CXL_GET_EVENT_FLAG_OVERFLOW;
+		pl->overflow_err_count = cpu_to_le16(nr_overflow);
+		ns = ktime_get_real_ns();
+		ns -= 5000000000; /* 5s ago */
+		pl->first_overflow_timestamp = cpu_to_le64(ns);
+		ns = ktime_get_real_ns();
+		ns -= 1000000000; /* 1s ago */
+		pl->last_overflow_timestamp = cpu_to_le64(ns);
+	}
+
 	memcpy(&pl->record, get_cur_event(log), sizeof(pl->record));
 	pl->record.hdr.handle = get_cur_event_handle(log);
 	return 0;
@@ -276,6 +300,18 @@ void cxl_mock_add_event_logs(struct cxl_dev_state *cxlds)
 	event_store_add_event(mes, CXL_EVENT_TYPE_INFO,
 			      (struct cxl_event_record_raw *)&mem_module);
 
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL, &maint_needed);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL,
+			      (struct cxl_event_record_raw *)&dram);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL,
+			      (struct cxl_event_record_raw *)&gen_media);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL,
+			      (struct cxl_event_record_raw *)&mem_module);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL, &hardware_replace);
+	event_store_add_event(mes, CXL_EVENT_TYPE_FAIL,
+			      (struct cxl_event_record_raw *)&dram);
+
 	event_store_add_event(mes, CXL_EVENT_TYPE_FATAL, &hardware_replace);
 	event_store_add_event(mes, CXL_EVENT_TYPE_FATAL,
 			      (struct cxl_event_record_raw *)&dram);
Signature · 1 line
2.37.2
#34 lore raw

RFC PATCH v2 11/11 cxl/mem: Wire up event interrupts

In reply to [email protected]

From: Ira Weiny <[email protected]>

CXL device events are signaled via interrupts. Each event log may have a different interrupt message number. These message numbers are reported in the Get Event Interrupt Policy mailbox command.

Create an infrastructure to query the max vectors required for the CXL device. Add event interrupt information that infrastructure. Set up a handler for each event log.

Davidlohr suggested the generic vector code.

---
 drivers/cxl/core/mbox.c      |  53 +++++++++++-
 drivers/cxl/cxlmem.h         |  32 ++++++++
 drivers/cxl/pci.c            | 152 +++++++++++++++++++++++++++++++++++
 include/uapi/linux/cxl_mem.h |   2 +
 4 files changed, 237 insertions(+), 2 deletions(-)
drivers/cxl/core/mbox.c +51 -2
index 6b3119bc83d2..ffd58da95df3 100644
@@ -53,6 +53,8 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {
 	CXL_CMD(GET_SUPPORTED_LOGS, 0, CXL_VARIABLE_PAYLOAD, CXL_CMD_FLAG_FORCE_ENABLE),
 	CXL_CMD(GET_EVENT_RECORD, 1, CXL_VARIABLE_PAYLOAD, 0),
 	CXL_CMD(CLEAR_EVENT_RECORD, CXL_VARIABLE_PAYLOAD, 0, 0),
+	CXL_CMD(GET_EVT_INT_POLICY, 0, 0x5, 0),
+	CXL_CMD(SET_EVT_INT_POLICY, 0x5, 0, 0),
 	CXL_CMD(GET_FW_INFO, 0, 0x50, 0),
 	CXL_CMD(GET_PARTITION_INFO, 0, 0x20, 0),
 	CXL_CMD(GET_LSA, 0x8, CXL_VARIABLE_PAYLOAD, 0),
@@ -780,8 +782,8 @@ static int cxl_clear_event_record(struct cxl_dev_state *cxlds,
 				 &payload, sizeof(payload), NULL, 0);
 }
 
-static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
-				    enum cxl_event_log_type type)
+void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
+			     enum cxl_event_log_type type)
 {
 	struct cxl_get_event_payload payload;
 
@@ -816,6 +818,7 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
 
 	} while (payload.flags & CXL_GET_EVENT_FLAG_MORE_RECORDS);
 }
+EXPORT_SYMBOL_NS_GPL(cxl_mem_get_records_log, CXL);
 
 /**
  * cxl_mem_get_event_records - Get Event Records from the device
@@ -839,6 +842,52 @@ void cxl_mem_get_event_records(struct cxl_dev_state *cxlds)
 }
 EXPORT_SYMBOL_NS_GPL(cxl_mem_get_event_records, CXL);
 
+static int cxl_event_msgnum(u8 setting)
+{
+	if (!cxl_evt_int_is_msi(setting))
+		return -1;
+
+	return CXL_EVENT_INT_MSGNUM(setting);
+}
+
+int cxl_event_get_max_msgnum(struct cxl_dev_state *cxlds)
+{
+	struct cxl_event_interrupt_policy *policy = &cxlds->evt_int_policy;
+	int rc;
+
+	policy->info_settings = CXL_INT_MSI_MSIX;
+	policy->warn_settings = CXL_INT_MSI_MSIX;
+	policy->failure_settings = CXL_INT_MSI_MSIX;
+	policy->fatal_settings = CXL_INT_MSI_MSIX;
+	policy->dyn_cap_settings = CXL_INT_MSI_MSIX;
+
+	rc = cxl_mbox_send_cmd(cxlds, CXL_MBOX_OP_SET_EVT_INT_POLICY,
+			       policy, sizeof(*policy),
+			       NULL, 0);
+	if (rc) {
+		dev_err(cxlds->dev, "Failed to set event interrupt policy : %d",
+			rc);
+		memset(policy, CXL_INT_NONE, sizeof(*policy));
+		return -1;
+	}
+
+	rc = cxl_mbox_send_cmd(cxlds, CXL_MBOX_OP_GET_EVT_INT_POLICY, NULL, 0,
+			       policy, sizeof(*policy));
+	if (rc) {
+		dev_err(cxlds->dev, "Failed to get event interrupt policy : %d",
+			rc);
+		return -1;
+	}
+
+	rc = max_t(int, rc, cxl_event_msgnum(policy->info_settings));
+	rc = max_t(int, rc, cxl_event_msgnum(policy->warn_settings));
+	rc = max_t(int, rc, cxl_event_msgnum(policy->failure_settings));
+	rc = max_t(int, rc, cxl_event_msgnum(policy->fatal_settings));
+
+	return rc;
+}
+EXPORT_SYMBOL_NS_GPL(cxl_event_get_max_msgnum, CXL);
+
 /**
  * cxl_mem_get_partition_info - Get partition info
  * @cxlds: The device data for the operation
drivers/cxl/cxlmem.h +32 -0
index 79b3fac6d9ef..27132ed2bdd3 100644
@@ -179,6 +179,32 @@ struct cxl_endpoint_dvsec_info {
 	struct range dvsec_range[2];
 };
 
+/**
+ * Event Interrupt Policy
+ *
+ * CXL rev 3.0 section 8.2.9.2.4; Table 8-52
+ */
+enum cxl_event_int_mode {
+	CXL_INT_NONE		= 0x00,
+	CXL_INT_MSI_MSIX	= 0x01,
+	CXL_INT_FW		= 0x02,
+	CXL_INT_RES		= 0x03,
+};
+#define CXL_EVENT_INT_MODE_MASK 0x3
+#define CXL_EVENT_INT_MSGNUM(setting) (((setting) & 0xf0) >> 4)
+struct cxl_event_interrupt_policy {
+	u8 info_settings;
+	u8 warn_settings;
+	u8 failure_settings;
+	u8 fatal_settings;
+	u8 dyn_cap_settings;
+} __packed;
+
+static inline bool cxl_evt_int_is_msi(u8 setting)
+{
+	return CXL_INT_MSI_MSIX == (setting & CXL_EVENT_INT_MODE_MASK);
+}
+
 /**
  * struct cxl_dev_state - The driver device state
  *
@@ -245,6 +271,7 @@ struct cxl_dev_state {
 
 	resource_size_t component_reg_phys;
 	u64 serial;
+	struct cxl_event_interrupt_policy evt_int_policy;
 
 	struct xarray doe_mbs;
 
@@ -256,6 +283,8 @@ enum cxl_opcode {
 	CXL_MBOX_OP_RAW			= CXL_MBOX_OP_INVALID,
 	CXL_MBOX_OP_GET_EVENT_RECORD	= 0x0100,
 	CXL_MBOX_OP_CLEAR_EVENT_RECORD	= 0x0101,
+	CXL_MBOX_OP_GET_EVT_INT_POLICY	= 0x0102,
+	CXL_MBOX_OP_SET_EVT_INT_POLICY	= 0x0103,
 	CXL_MBOX_OP_GET_FW_INFO		= 0x0200,
 	CXL_MBOX_OP_ACTIVATE_FW		= 0x0202,
 	CXL_MBOX_OP_GET_SUPPORTED_LOGS	= 0x0400,
@@ -541,7 +570,10 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
 struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
 void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
 void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
+void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,
+			     enum cxl_event_log_type type);
 void cxl_mem_get_event_records(struct cxl_dev_state *cxlds);
+int cxl_event_get_max_msgnum(struct cxl_dev_state *cxlds);
 #ifdef CONFIG_CXL_SUSPEND
 void cxl_mem_active_inc(void);
 void cxl_mem_active_dec(void);
drivers/cxl/pci.c +152 -0

164 lines not on this page · raw message

include/uapi/linux/cxl_mem.h +2 -0
index 7c1ad8062792..a8204802fcca 100644
@@ -26,6 +26,8 @@
 	___C(GET_SUPPORTED_LOGS, "Get Supported Logs"),                   \
 	___C(GET_EVENT_RECORD, "Get Event Record"),                       \
 	___C(CLEAR_EVENT_RECORD, "Clear Event Record"),                   \
+	___C(GET_EVT_INT_POLICY, "Get Event Interrupt Policy"),           \
+	___C(SET_EVT_INT_POLICY, "Set Event Interrupt Policy"),           \
 	___C(GET_FW_INFO, "Get FW Info"),                                 \
 	___C(GET_PARTITION_INFO, "Get Partition Information"),            \
 	___C(GET_LSA, "Get Label Storage Area"),                          \
Signature · 1 line
2.37.2
maintainer
#35 lore raw

Re: [RFC V2 PATCH 11/11] cxl/mem: Wire up event interrupts

Quoted 11 lines · On Mon, 10 Oct 2022, [email protected] wrote:

From: Ira Weiny <[email protected]>

CXL device events are signaled via interrupts. Each event log may have a different interrupt message number. These message numbers are reported in the Get Event Interrupt Policy mailbox command.

Create an infrastructure to query the max vectors required for the CXL device. Add event interrupt information that infrastructure. Set up a handler for each event log.

Davidlohr suggested the generic vector code.

So this should be a separate patch, and out of the event series altogether. There are a number of interested parties for irq support, and imo the generic vector stuff should be a first dependency to all of them.

I've sent out a patch with some updates you did. I also kept the whole table populated with nil callbacks as I believe this documents the TODO nicely.

Thanks,
Davidlohr