CXL: Process event logs
v2 cover CXL: Process event logs
From: Ira Weiny <[email protected]>
Changes from RFC v1
Add event irqs
General simplification of the code. Resolve field alignment questions Update to rev 3.0 for comments and structures Add reserved fields and output them
Event records inform the OS of various device events. Events are not needed for any kernel operation but various user level software will want to track events.
Add event reporting through the trace event mechanism. On driver load read and clear all device events.
Enable all event logs for interrupts and process each log on interrupt.
Testing of this was performed with additions to QEMU posted here:
https://lore.kernel.org/all/[email protected]/
Ira Weiny (11):
cxl/mbox: Add debug of hardware error code
cxl/mem: Implement Get Event Records command
cxl/mem: Implement Clear Event Records command
cxl/mem: Clear events on driver load
cxl/mem: Trace General Media Event Record
cxl/mem: Trace DRAM Event Record
cxl/mem: Trace Memory Module Event Record
cxl/test: Add generic mock events
cxl/test: Add specific events
cxl/test: Simulate event log overflow
cxl/mem: Wire up event interrupts
MAINTAINERS | 1 + drivers/cxl/core/mbox.c | 186 ++++++++++++- drivers/cxl/cxlmem.h | 193 +++++++++++++ drivers/cxl/pci.c | 154 ++++++++++ include/trace/events/cxl.h | 478 ++++++++++++++++++++++++++++++++ include/uapi/linux/cxl_mem.h | 4 + tools/testing/cxl/test/Kbuild | 2 +- tools/testing/cxl/test/events.c | 329 ++++++++++++++++++++++ tools/testing/cxl/test/events.h | 9 + tools/testing/cxl/test/mem.c | 34 +++ 10 files changed, 1388 insertions(+), 2 deletions(-) create mode 100644 include/trace/events/cxl.h create mode 100644 tools/testing/cxl/test/events.c create mode 100644 tools/testing/cxl/test/events.h
base-commit: e2302539dd4f1c62d96651c07ddb05aa2461d29c
Signature · 1 line
2.37.2
RFC PATCH v2 1/11 cxl/mbox: Add debug of hardware error code
From: Ira Weiny <[email protected]>
If a mailbox command fails the driver always reports ENXIO. But this may not be enough information to understand why the hardware, or in my case Qemu, was failing.
Add a debug print of the error code returned from the hardware.
--- drivers/cxl/core/mbox.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
drivers/cxl/core/mbox.c +4 -1
@@ -181,8 +181,11 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (rc)return rc; -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { +dev_dbg (cxlds->dev,"MB error : %s\n" , +cxl_mbox_cmd_rc2str (&mbox_cmd));return cxl_mbox_cmd_rc2errno (&mbox_cmd); + }/* * Variable sized commands can't be validated and so it's up to the
Signature · 1 line
2.37.2
Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code
Quoted 9 lines · On Mon, 10 Oct 2022 15:41:21 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
If a mailbox command fails the driver always reports ENXIO. But this may not be enough information to understand why the hardware, or in my case Qemu, was failing.
Add a debug print of the error code returned from the hardware.
Signed-off-by Ira Weiny <[email protected]>
Seems very sensible to me.
Quoted 21 lines
--- drivers/cxl/core/mbox.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)drivers/cxl/core/mbox.c +4
-1@@ -181,8 +181,11 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (rc)return rc; -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { +dev_dbg (cxlds->dev,"MB error : %s\n" , +cxl_mbox_cmd_rc2str (&mbox_cmd));return cxl_mbox_cmd_rc2errno (&mbox_cmd); + }/* * Variable sized commands can't be validated and so it's up to the
Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code
In reply to [email protected]Quoted 7 lines · On Mon, 10 Oct 2022, [email protected] wrote:
From: Ira Weiny <[email protected]>
If a mailbox command fails the driver always reports ENXIO. But this may not be enough information to understand why the hardware, or in my case Qemu, was failing.
Add a debug print of the error code returned from the hardware.
with a nit below.
Quoted 11 lines
Signed-off-by Ira Weiny <[email protected]>---
drivers/cxl/core/mbox.c| 5 ++++-1 file changed, 4 insertions(+), 1 deletion(-)drivers/cxl/core/mbox.c +0
-0
drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 185
Quoted 6 lines
drivers/cxl/core/mbox.c +2
-1@@ -181,8 +181,11 @@ int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in,if (rc)return rc; -if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { + dev_dbg(cxlds->dev, "MB error : %s\n",
Maybe s/MB/mbox?
Quoted 9 lines
drivers/cxl/core/mbox.c +2
-1+ cxl_mbox_cmd_rc2str(&mbox_cmd));return cxl_mbox_cmd_rc2errno (&mbox_cmd); + }/* * Variable sized commands can't be validated and so it's up to the -- 2.37.2
Davidlohr Bueso
maintainerRe: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code
Quoted 5 lines · Davidlohr Bueso
drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 185
Quoted 3 lines
drivers/cxl/core/mbox.c +2
-1-if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { + dev_dbg(cxlds->dev, "MB error : %s\n",Maybe s/MB/mbox?
Actually 'Mailbox' seems to be the standard:
core/regs.c: dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset);core/regs.c: dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset);pci.c: dev_dbg(dev, "Mailbox operation had an error\n");pci.c: dev_err(cxlds->dev, "Mailbox is too small (%zub)",pci.c: dev_dbg(cxlds->dev, "Mailbox payload sized %zu",
Re: [RFC V2 PATCH 01/11] cxl/mbox: Add debug of hardware error code
Quoted 8 lines · Davidlohr Bueso
Quoted 5 lines · Davidlohr Bueso
drivers/cxl/core/mbox.c int cxl_mbox_send_cmd(struct cxl_dev_state *cxlds, u16 opcode, void *in, · line 185
Quoted 3 lines
drivers/cxl/core/mbox.c +2
-1-if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) +if (mbox_cmd.return_code !=CXL_MBOX_CMD_RC_SUCCESS ) { + dev_dbg(cxlds->dev, "MB error : %s\n",Maybe s/MB/mbox?
Actually 'Mailbox' seems to be the standard:
Good point! Changed.
Thanks!
Ira
Quoted 6 lines
core/regs.c:dev_dbg(dev, "found Mailbox capability (0x%x)\n", offset);core/regs.c:dev_dbg(dev, "found Secondary Mailbox capability (0x%x)\n", offset);pci.c:dev_dbg(dev, "Mailbox operation had an error\n");pci.c: dev_err(cxlds->dev, "Mailbox is too small (%zub)",pci.c: dev_dbg(cxlds->dev, "Mailbox payload sized %zu",
RFC PATCH v2 2/11 cxl/mem: Implement Get Event Records command
In reply to [email protected]From: Ira Weiny <[email protected]>
Event records are defined for CXL devices. Each record is reported in one event log. Devices are required to support the storage of at least one event record in each event log type.
Devices track event log overflow by incrementing a counter and tracking the time of the first and last overflow event seen.
Software queries events via the Get Event Record mailbox command; CXL rev 3.0 section 8.2.9.2.2.
Issue the Get Event Record mailbox command on driver load. Trace each record found, as well as any overflow conditions. Only 1 event is requested for each query. Optimization of multiple record queries is deferred.
This patch traces a raw event record only and leaves the specific event record types to subsequent patches.
Macros are created to use for the common CXL Event header fields.
---
Change from RFC:
Remove redundant error message in get event records loop s/EVENT_RECORD_DATA_LENGTH/CXL_EVENT_RECORD_DATA_LENGTH Use hdr_uuid for the header UUID field
Use cxl_event_log_type_str() for the trace events
Create macros for the header fields and common entries of each event Add reserved buffer output dump
Report error if event query fails
Remove unused record_cnt variable
Steven - reorder overflow record
Remove NOTE about checkpatch
Jonathan
check for exactly 1 record
s/v3.0/rev 3.0
Use 3 byte fields for 24bit fields
Add 3.0 Maintenance Operation Class
Add Dynamic Capacity log type
Fix spelling
Dave Jiang/Dan/Alison
s/cxl-event/cxl
trace/events/cxl-events => trace/events/cxl.h s/cxl_event_overflow/overflow s/cxl_event/generic_event
--- MAINTAINERS | 1 + drivers/cxl/core/mbox.c | 53 ++++++++++++++ drivers/cxl/cxlmem.h | 75 ++++++++++++++++++++ include/trace/events/cxl.h | 130 +++++++++++++++++++++++++++++++++++ include/uapi/linux/cxl_mem.h | 1 + 5 files changed, 260 insertions(+) create mode 100644 include/trace/events/cxl.h
MAINTAINERS +1 -0
@@ -5219,6 +5219,7 @@ M: Dan Williams <[email protected]> L: [email protected] S: Maintained F: drivers/cxl/ +F: include/trace/events/cxl.h F: include/uapi/linux/cxl_mem.h CONEXANT ACCESSRUNNER USB DRIVER
drivers/cxl/core/mbox.c +53 -0
71 lines not on this page · raw message
drivers/cxl/cxlmem.h +75 -0
99 lines not on this page · raw message
include/trace/events/cxl.h new file +130 -0
130 lines not on this page · raw message
include/uapi/linux/cxl_mem.h +1 -0
@@ -24,6 +24,7 @@___C (IDENTIFY ,"Identify Command" ), \___C (RAW ,"Raw device command" ), \___C (GET_SUPPORTED_LOGS ,"Get Supported Logs" ), \ +___C (GET_EVENT_RECORD ,"Get Event Record" ), \___C (GET_FW_INFO ,"Get FW Info" ), \___C (GET_PARTITION_INFO ,"Get Partition Information" ), \ ___C(GET_LSA, "Get Label Storage Area"), \
Signature · 1 line
2.37.2
Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command
Quoted 17 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
Event records are defined for CXL devices. Each record is reported in one event log. Devices are required to support the storage of at least one event record in each event log type.
Devices track event log overflow by incrementing a counter and tracking the time of the first and last overflow event seen.
Software queries events via the Get Event Record mailbox command; CXL rev 3.0 section 8.2.9.2.2.
Issue the Get Event Record mailbox command on driver load. Trace each record found, as well as any overflow conditions. Only 1 event is requested for each query. Optimization of multiple record queries is deferred.
Quoted 7 lines
This patch traces a raw event record only and leaves the specific event record types to subsequent patches.
Macros are created to use for the common CXL Event header fields.
Cc Steven Rostedt <[email protected]>Signed-off-by Ira Weiny <[email protected]>
Hi Ira,
This looks basically fine, but I'm not convinced that it is a good or sustainable idea to report reserved fields from the underlying events in the trace events.
If they are defined to have meaning later, we can't remove them from the 'reserved' field with out breaking backwards compatibility. So we end up with a weird mixture of some fields in both reserved and new entries in the TP and some not.
I've cc'd Mauro as he's probably more experienced in how to handle this sort of stuff than anyone else I know.
Jonathan
Quoted 26 lines
---
Change from RFC:
Remove redundant error message in get event records loop s/EVENT_RECORD_DATA_LENGTH/CXL_EVENT_RECORD_DATA_LENGTH Usehdr_uuidfor the header UUID field
Usecxl_event_log_type_str()for the trace events
Create macros for the header fields and common entries of each event Add reserved buffer output dump
Report error if event query fails
Remove unusedrecord_cntvariable
Steven - reorder overflow record
Remove NOTE about checkpatch
Jonathan
check for exactly 1 record
s/v3.0/rev 3.0
Use 3 byte fields for 24bit fields
Add 3.0 Maintenance Operation Class
Add Dynamic Capacity log type
Fix spelling
Dave Jiang/Dan/Alison
s/cxl-event/cxl
trace/events/cxl-events =>trace/events/cxl.hs/cxl_event_overflow/overflow s/cxl_event/generic_event
---
...
Quoted 6 lines
drivers/cxl/core/mbox.c +6
-0+/** +* cxl_mem_get_event_records - Get Event Records from the device +* @cxlds: The device data for the operation +* +* Retrieve all event records available on the device and report them as trace +* events.
drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 757
Quoted 10 lines
drivers/cxl/core/mbox.c +10
-0+* +* See CXL rev 3.0 @8.2.9.2.2 Get Event Records +*/ +void cxl_mem_get_event_records (struct cxl_dev_state *cxlds) +{ +enum cxl_event_log_type log_type; + +dev_dbg (cxlds->dev,"Reading event logs\n" ); + + for (log_type = CXL_EVENT_TYPE_INFO;
I'd start at 0. To my mind that's clearer than this which basically says there is a contiguous range that may or may not be 0 based.
Quoted 40 lines
drivers/cxl/core/mbox.c +5
-0+ log_type < CXL_EVENT_TYPE_MAX; log_type++) +cxl_mem_get_records_log (cxlds, log_type); +} +EXPORT_SYMBOL_NS_GPL (cxl_mem_get_event_records,CXL ); +/** * cxl_mem_get_partition_info - Get partition info * @cxlds: The device data for the operation drivers/cxl/cxlmem.h +10
-0@@ -4,6 +4,7 @@#define __CXL_MEM_H__#include <uapi/linux/cxl_mem.h> #include <linux/cdev.h> +#include <linux/uuid.h> #include "cxl.h" /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ @@ -253,6 +254,7 @@ struct cxl_dev_state {enum cxl_opcode {CXL_MBOX_OP_INVALID =0x0000 ,CXL_MBOX_OP_RAW =CXL_MBOX_OP_INVALID , +CXL_MBOX_OP_GET_EVENT_RECORD =0x0100 ,CXL_MBOX_OP_GET_FW_INFO =0x0200 ,CXL_MBOX_OP_ACTIVATE_FW =0x0202 ,CXL_MBOX_OP_GET_SUPPORTED_LOGS =0x0400 ,@@ -322,6 +324,78 @@ struct cxl_mbox_identify {u8 qos_telemetry_caps; } __packed; +/* +* Common Event Record Format +* CXL rev 3.0 section 8.2.9.2.1; Table 8-42 +*/ +#define CXL_EVENT_REC_HDR_RES_LEN 0xf +struct cxl_event_record_hdr { +uuid_t id; +u8 length;
drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 344
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+u8 flags[3 ]; +__le16 handle; +__le16 related_handle; +__le64 timestamp; +u8 maint_op_class; +u8 reserved[CXL_EVENT_REC_HDR_RES_LEN ]; +} __packed; + +#define CXL_EVENT_RECORD_DATA_LENGTH 0x50 +struct cxl_event_record_raw {
I'd like some comments here on 'why' this makes sense. Is expectation that it's here for future CXL spec definitions or is it intended to allow some reporting of vendor defined records? Actually maybe a comment down at the TP would make more sense than here. Either way comment somewhere :)
Quoted 9 lines
drivers/cxl/cxlmem.h +7
-0+struct cxl_event_record_hdr hdr; +u8 data[CXL_EVENT_RECORD_DATA_LENGTH ]; +} __packed; + +static inline const char *cxl_event_log_type_str (enum cxl_event_log_type type) +{ +switch (type) {
drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 392
Quoted 10 lines
drivers/cxl/cxlmem.h +10
-0+case CXL_EVENT_TYPE_INFO : +return "Informational" ; +case CXL_EVENT_TYPE_WARN : +return "Warning" ; +case CXL_EVENT_TYPE_FAIL : +return "Failure" ; +case CXL_EVENT_TYPE_FATAL : +return "Fatal" ; +case CXL_EVENT_TYPE_DYNAMIC_CAP : +return "Dynamic Capacity" ;
Array of const char * that you pick from (with limit check) maybe rather than a switch? Doesn't matter that much though.
Quoted 34 lines
drivers/cxl/cxlmem.h +7
-0+default : +break ; + } +return "<unknown>" ; +} +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;@@ -381,6 +455,7 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);struct cxl_dev_state *cxl_dev_state_create (struct device *dev);void set_exclusive_cxl_commands (struct cxl_dev_state *cxlds,unsigned long *cmds);void clear_exclusive_cxl_commands (struct cxl_dev_state *cxlds,unsigned long *cmds); +void cxl_mem_get_event_records (struct cxl_dev_state *cxlds);#ifdef CONFIG_CXL_SUSPEND void cxl_mem_active_inc (void );void cxl_mem_active_dec (void );include/trace/events/cxl.h new file +11
-0@@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM cxl + +#if !defined(_CXL_TRACE_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _CXL_TRACE_EVENTS_H + +#include <asm-generic/unaligned.h> +#include <linux/tracepoint.h> + +TRACE_EVENT(overflow,
include/trace/events/cxl.h line 21
Quoted 10 lines
include/trace/events/cxl.h new file +10
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_get_event_payload *payload), + + TP_ARGS(dev_name, log, payload), + + TP_STRUCT__entry( + __string(dev_name, dev_name) + __field(int, log) + __field(u64, first)
first_ts maybe? first alone is a bit vague.
Quoted 4 lines
include/trace/events/cxl.h new file +4
-0+ __field(u64, last) +__field (u16, count) + ), +
include/trace/events/cxl.h line 35
Quoted 10 lines
include/trace/events/cxl.h new file +10
-0+TP_fast_assign ( + __assign_str(dev_name, dev_name); + __entry->log = log; + __entry->count =le16_to_cpu (payload->overflow_err_count); + __entry->first =le64_to_cpu (payload->first_overflow_timestamp); + __entry->last =le64_to_cpu (payload->last_overflow_timestamp); + ), + + TP_printk("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu", + __get_str(dev_name), cxl_event_log_type_str(__entry->log),
So we have a potential header include ordering issue given this header doesn't include the definition of cxl_event_log_type_str()?
Quoted 45 lines
include/trace/events/cxl.h new file +45
-0+ __entry->count, __entry->first, __entry->last) + +); + +/* +* Common Event Record Format +* CXL 3.0 section 8.2.9.2.1; Table 8-42 +*/ +#define CXL_EVENT_RECORD_FLAG_PERMANENT BIT(2) +#define CXL_EVENT_RECORD_FLAG_MAINT_NEEDED BIT(3) +#define CXL_EVENT_RECORD_FLAG_PERF_DEGRADED BIT(4) +#define CXL_EVENT_RECORD_FLAG_HW_REPLACE BIT(5) +#define show_hdr_flags (flags) __print_flags(flags, " | ", \ + { CXL_EVENT_RECORD_FLAG_PERMANENT, "Permanent Condition" }, \ + { CXL_EVENT_RECORD_FLAG_MAINT_NEEDED, "Maintenance Needed" }, \ + { CXL_EVENT_RECORD_FLAG_PERF_DEGRADED, "Performance Degraded" }, \ + { CXL_EVENT_RECORD_FLAG_HW_REPLACE, "Hardware Replacement Needed" } \ +) + +/* +* Define macros for the common header of each CXL event. +* +* Tracepoints using these macros must do 3 things: +* +* 1) Add CXL_EVT_TP_entry to TP_STRUCT__entry +* 2) Use CXL_EVT_TP_fast_assign within TP_fast_assign; +* pass the dev_name, log, and CXL event header +* 3) Use CXL_EVT_TP_printk() instead of TP_printk() +* +* See the generic_event tracepoint as an example. +*/ +#define CXL_EVT_TP_entry \ + __string(dev_name, dev_name) \ + __field(int, log) \ + __array(u8, hdr_uuid, UUID_SIZE) \ + __field(u32, hdr_flags) \ + __field(u16, hdr_handle) \ + __field(u16, hdr_related_handle) \ + __field(u64, hdr_timestamp) \ + __array(u8, hdr_res, CXL_EVENT_REC_HDR_RES_LEN) \ + __field(u8, hdr_length) \ + __field(u8, hdr_maint_op_class) + +#define CXL_EVT_TP_fast_assign (dname, l, hdr) \ + __assign_str(dev_name, (dname)); \
include/trace/events/cxl.h line 90
Quoted 10 lines
include/trace/events/cxl.h new file +10
-0+ __entry->log = (l); \ +memcpy (__entry->hdr_uuid, &(hdr).id,UUID_SIZE ); \ + __entry->hdr_length = (hdr).length; \ + __entry->hdr_flags =get_unaligned_le24 ((hdr).flags); \ + __entry->hdr_handle =le16_to_cpu ((hdr).handle); \ + __entry->hdr_related_handle =le16_to_cpu ((hdr).related_handle); \ + __entry->hdr_timestamp =le64_to_cpu ((hdr).timestamp); \ + __entry->hdr_maint_op_class = (hdr).maint_op_class; \ +memcpy (__entry->hdr_res, &(hdr).reserved, \ +CXL_EVENT_REC_HDR_RES_LEN )
What's the logic behind having the reserved fields here?
How does that change when a future spec defines them as used? Do we have to keep whatever was in the previously reserved fields for ever in order to maintain backwards compatibility even though we've added the same data to the end of the trace point?
I don't think they should be here at all. Given a userspace stack has to cope with out knowing the contents of those fields as userspace might not be up to date, I see no problem with requiring a newer kernel to support stuff added in future specs.
Steven, Mauro, you probably have a better idea of history of similar cases. How have other people handled reserved fields in underlying hardware reports that may become used in later revisions?
Also, probably makes sense to cc linux-edac on v3 of this series as the experts in these flows tend to read that list. Obviously intent so far is not to pass these into the edac subsystem, but I'd still let them know this work is going on.
Quoted 35 lines
include/trace/events/cxl.h new file +35
-0+ + +#define CXL_EVT_TP_printk (fmt, ...) \ + TP_printk("%s log=%s : time=%llu uuid=%pUl len=%d flags='%s' " \ + "handle=%x related_handle=%x maint_op_class=%u res='%s' " \ + " : " fmt, \ + __get_str(dev_name), cxl_event_log_type_str(__entry->log), \ + __entry->hdr_timestamp, __entry->hdr_uuid, __entry->hdr_length, \ + show_hdr_flags(__entry->hdr_flags), __entry->hdr_handle, \ + __entry->hdr_related_handle, __entry->hdr_maint_op_class, \ + __print_hex(__entry->hdr_res, CXL_EVENT_REC_HDR_RES_LEN), \ + ##__VA_ARGS__) + +TRACE_EVENT (generic_event, + + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_record_raw *rec), + +TP_ARGS (dev_name, log, rec), + +TP_STRUCT__entry ( + CXL_EVT_TP_entry +__array (u8, data,CXL_EVENT_RECORD_DATA_LENGTH ) + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); + memcpy(__entry->data, &rec->data, CXL_EVENT_RECORD_DATA_LENGTH); + ), + +CXL_EVT_TP_printk ("%s" , +__print_hex (__entry->data,CXL_EVENT_RECORD_DATA_LENGTH )) +); + +#endif /* _CXL_TRACE_EVENTS_H */
Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command
Quoted 30 lines · Jonathan Cameron
Quoted 17 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
Event records are defined for CXL devices. Each record is reported in one event log. Devices are required to support the storage of at least one event record in each event log type.
Devices track event log overflow by incrementing a counter and tracking the time of the first and last overflow event seen.
Software queries events via the Get Event Record mailbox command; CXL rev 3.0 section 8.2.9.2.2.
Issue the Get Event Record mailbox command on driver load. Trace each record found, as well as any overflow conditions. Only 1 event is requested for each query. Optimization of multiple record queries is deferred.
Quoted 7 lines
This patch traces a raw event record only and leaves the specific event record types to subsequent patches.
Macros are created to use for the common CXL Event header fields.
Cc Steven Rostedt <[email protected]>Signed-off-by Ira Weiny <[email protected]>Hi Ira,
Quoted 7 lines
This looks basically fine, but I'm not convinced that it is a good or sustainable idea to report reserved fields from the underlying events in the trace events.
If they are defined to have meaning later, we can't remove them from the 'reserved' field with out breaking backwards compatibility. So we end up with a weird mixture of some fields in both reserved and new entries in the TP and some not.
I removed the reserved fields from the data but I forgot the header. I think I had a reason they would be useful in the header but I forget now. Which means I was probably wrong. ;-)
Quoted 4 lines
I've cc'd Mauro as he's probably more experienced in how to handle this sort of stuff than anyone else I know.
Fair enough but I agree the reserved fields should be removed.
Quoted 44 lines
Jonathan
Quoted 26 lines
---
Change from RFC:
Remove redundant error message in get event records loop s/EVENT_RECORD_DATA_LENGTH/CXL_EVENT_RECORD_DATA_LENGTH Usehdr_uuidfor the header UUID field
Usecxl_event_log_type_str()for the trace events
Create macros for the header fields and common entries of each event Add reserved buffer output dump
Report error if event query fails
Remove unusedrecord_cntvariable
Steven - reorder overflow record
Remove NOTE about checkpatch
Jonathan
check for exactly 1 record
s/v3.0/rev 3.0
Use 3 byte fields for 24bit fields
Add 3.0 Maintenance Operation Class
Add Dynamic Capacity log type
Fix spelling
Dave Jiang/Dan/Alison
s/cxl-event/cxl
trace/events/cxl-events =>trace/events/cxl.hs/cxl_event_overflow/overflow s/cxl_event/generic_event
---...
Quoted 12 lines
drivers/cxl/core/mbox.c +12
-0+/** +* cxl_mem_get_event_records - Get Event Records from the device +* @cxlds: The device data for the operation +* +* Retrieve all event records available on the device and report them as trace +* events. +* +* See CXL rev 3.0 @8.2.9.2.2 Get Event Records +*/ +void cxl_mem_get_event_records (struct cxl_dev_state *cxlds) +{ +enum cxl_event_log_type log_type;
Quoted 8 lines
drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 757
Quoted 4 lines
drivers/cxl/core/mbox.c +4
-0+ +dev_dbg (cxlds->dev,"Reading event logs\n" ); + + for (log_type = CXL_EVENT_TYPE_INFO;I'd start at 0. To my mind that's clearer than this which basically says there is a contiguous range that may or may not be 0 based.
Ok. But weather or not it is 0 based does not matter.
I'll change it. It takes out a line of code! ;-)
Quoted 49 lines
Quoted 49 lines
drivers/cxl/core/mbox.c +5
-0+ log_type < CXL_EVENT_TYPE_MAX; log_type++) +cxl_mem_get_records_log (cxlds, log_type); +} +EXPORT_SYMBOL_NS_GPL (cxl_mem_get_event_records,CXL ); +/** * cxl_mem_get_partition_info - Get partition info * @cxlds: The device data for the operation drivers/cxl/cxlmem.h +18
-0@@ -4,6 +4,7 @@#define __CXL_MEM_H__#include <uapi/linux/cxl_mem.h> #include <linux/cdev.h> +#include <linux/uuid.h> #include "cxl.h" /* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */ @@ -253,6 +254,7 @@ struct cxl_dev_state {enum cxl_opcode {CXL_MBOX_OP_INVALID =0x0000 ,CXL_MBOX_OP_RAW =CXL_MBOX_OP_INVALID , +CXL_MBOX_OP_GET_EVENT_RECORD =0x0100 ,CXL_MBOX_OP_GET_FW_INFO =0x0200 ,CXL_MBOX_OP_ACTIVATE_FW =0x0202 ,CXL_MBOX_OP_GET_SUPPORTED_LOGS =0x0400 ,@@ -322,6 +324,78 @@ struct cxl_mbox_identify {u8 qos_telemetry_caps; } __packed; +/* +* Common Event Record Format +* CXL rev 3.0 section 8.2.9.2.1; Table 8-42 +*/ +#define CXL_EVENT_REC_HDR_RES_LEN 0xf +struct cxl_event_record_hdr { +uuid_t id; +u8 length; +u8 flags[3 ]; +__le16 handle; +__le16 related_handle; +__le64 timestamp; +u8 maint_op_class; +u8 reserved[CXL_EVENT_REC_HDR_RES_LEN ]; +} __packed; +
Quoted 8 lines
drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 344
Quoted 2 lines
drivers/cxl/cxlmem.h +2
-0+#define CXL_EVENT_RECORD_DATA_LENGTH 0x50 +struct cxl_event_record_raw {I'd like some comments here on 'why' this makes sense. Is expectation that it's here for future CXL spec definitions or is it intended to allow some reporting of vendor defined records? Actually maybe a comment down at the TP would make more sense than here. Either way comment somewhere :)
This comes from a discussion I had with Dan a while back where we decided that any unknown record would just be dumped to user space as data. The idea was based on the kernel ignoring vendor specified events but does allow for some future CXL spec definitios to get through.
This may cause similar confusion to the reserved fields but I don't think it is quite the same.
I'll add a comment.
Quoted 7 lines
Quoted 5 lines
drivers/cxl/cxlmem.h +4
-0+struct cxl_event_record_hdr hdr; +u8 data[CXL_EVENT_RECORD_DATA_LENGTH ]; +} __packed; +
Quoted 15 lines
drivers/cxl/cxlmem.h struct cxl_mbox_identify { · line 392
Quoted 13 lines
drivers/cxl/cxlmem.h +13
-0+static inline const char *cxl_event_log_type_str (enum cxl_event_log_type type) +{ +switch (type) { +case CXL_EVENT_TYPE_INFO : +return "Informational" ; +case CXL_EVENT_TYPE_WARN : +return "Warning" ; +case CXL_EVENT_TYPE_FAIL : +return "Failure" ; +case CXL_EVENT_TYPE_FATAL : +return "Fatal" ; +case CXL_EVENT_TYPE_DYNAMIC_CAP : +return "Dynamic Capacity" ;Array of const char * that you pick from (with limit check) maybe rather than a switch? Doesn't matter that much though.
I've seen it both ways. This forces users to use the helper. And I don't think there is much concern about the speed of the lookup.
Quoted 39 lines
Quoted 39 lines
drivers/cxl/cxlmem.h +7
-0+default : +break ; + } +return "<unknown>" ; +} +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;@@ -381,6 +455,7 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);struct cxl_dev_state *cxl_dev_state_create (struct device *dev);void set_exclusive_cxl_commands (struct cxl_dev_state *cxlds,unsigned long *cmds);void clear_exclusive_cxl_commands (struct cxl_dev_state *cxlds,unsigned long *cmds); +void cxl_mem_get_event_records (struct cxl_dev_state *cxlds);#ifdef CONFIG_CXL_SUSPEND void cxl_mem_active_inc (void );void cxl_mem_active_dec (void );include/trace/events/cxl.h new file +15
-0@@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#undef TRACE_SYSTEM +#define TRACE_SYSTEM cxl + +#if !defined(_CXL_TRACE_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _CXL_TRACE_EVENTS_H + +#include <asm-generic/unaligned.h> +#include <linux/tracepoint.h> + +TRACE_EVENT(overflow, + + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_get_event_payload *payload), +
Quoted 8 lines
include/trace/events/cxl.h line 21
Quoted 6 lines
include/trace/events/cxl.h new file +6
-0+TP_ARGS (dev_name, log, payload), + +TP_STRUCT__entry ( + __string(dev_name, dev_name) + __field(int, log) + __field(u64, first)
first_tsmaybe? first alone is a bit vague.
Agreed. Changed to first_ts and last_ts.
Quoted 10 lines
Quoted 10 lines
include/trace/events/cxl.h new file +9
-0+ __field(u64, last) +__field (u16, count) + ), + +TP_fast_assign ( + __assign_str(dev_name, dev_name); + __entry->log = log; + __entry->count =le16_to_cpu (payload->overflow_err_count); + __entry->first =le64_to_cpu (payload->first_overflow_timestamp);
Quoted 8 lines
include/trace/events/cxl.h line 35
Quoted 5 lines
include/trace/events/cxl.h new file +5
-0+ __entry->last =le64_to_cpu (payload->last_overflow_timestamp); + ), + + TP_printk("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu", + __get_str(dev_name), cxl_event_log_type_str(__entry->log),So we have a potential header include ordering issue given this header doesn't include the definition of
cxl_event_log_type_str()?
The RFC had a __print_symbolic() conversion which was redundant. I really don't like having 2 conversion routines for this string. I think it will result in maintenance issues down the line.
I'll try and resolve it. Nothing I have tried this morning seems to work. I may just go back to what I had. I need to double check the user space code too.
Quoted 50 lines
Quoted 50 lines
include/trace/events/cxl.h new file +49
-0+ __entry->count, __entry->first, __entry->last) + +); + +/* +* Common Event Record Format +* CXL 3.0 section 8.2.9.2.1; Table 8-42 +*/ +#define CXL_EVENT_RECORD_FLAG_PERMANENT BIT(2) +#define CXL_EVENT_RECORD_FLAG_MAINT_NEEDED BIT(3) +#define CXL_EVENT_RECORD_FLAG_PERF_DEGRADED BIT(4) +#define CXL_EVENT_RECORD_FLAG_HW_REPLACE BIT(5) +#define show_hdr_flags (flags) __print_flags(flags, " | ", \ + { CXL_EVENT_RECORD_FLAG_PERMANENT, "Permanent Condition" }, \ + { CXL_EVENT_RECORD_FLAG_MAINT_NEEDED, "Maintenance Needed" }, \ + { CXL_EVENT_RECORD_FLAG_PERF_DEGRADED, "Performance Degraded" }, \ + { CXL_EVENT_RECORD_FLAG_HW_REPLACE, "Hardware Replacement Needed" } \ +) + +/* +* Define macros for the common header of each CXL event. +* +* Tracepoints using these macros must do 3 things: +* +* 1) Add CXL_EVT_TP_entry to TP_STRUCT__entry +* 2) Use CXL_EVT_TP_fast_assign within TP_fast_assign; +* pass the dev_name, log, and CXL event header +* 3) Use CXL_EVT_TP_printk() instead of TP_printk() +* +* See the generic_event tracepoint as an example. +*/ +#define CXL_EVT_TP_entry \ + __string(dev_name, dev_name) \ + __field(int, log) \ + __array(u8, hdr_uuid, UUID_SIZE) \ + __field(u32, hdr_flags) \ + __field(u16, hdr_handle) \ + __field(u16, hdr_related_handle) \ + __field(u64, hdr_timestamp) \ + __array(u8, hdr_res, CXL_EVENT_REC_HDR_RES_LEN) \ + __field(u8, hdr_length) \ + __field(u8, hdr_maint_op_class) + +#define CXL_EVT_TP_fast_assign (dname, l, hdr) \ + __assign_str(dev_name, (dname)); \ + __entry->log = (l); \ + memcpy(__entry->hdr_uuid, &(hdr).id, UUID_SIZE); \ + __entry->hdr_length = (hdr).length; \ + __entry->hdr_flags = get_unaligned_le24((hdr).flags); \
Quoted 8 lines
include/trace/events/cxl.h line 90
Quoted 6 lines
include/trace/events/cxl.h new file +6
-0+ __entry->hdr_handle =le16_to_cpu ((hdr).handle); \ + __entry->hdr_related_handle =le16_to_cpu ((hdr).related_handle); \ + __entry->hdr_timestamp =le64_to_cpu ((hdr).timestamp); \ + __entry->hdr_maint_op_class = (hdr).maint_op_class; \ +memcpy (__entry->hdr_res, &(hdr).reserved, \ +CXL_EVENT_REC_HDR_RES_LEN )What's the logic behind having the reserved fields here?
I forget.
Quoted 4 lines
How does that change when a future spec defines them as used? Do we have to keep whatever was in the previously reserved fields for ever in order to maintain backwards compatibility even though we've added the same data to the end of the trace point?
I don't expect the header to change as much as the records themselves. So I'm ok removing the reserved fields.
Quoted 6 lines
I don't think they should be here at all. Given a userspace stack has to cope with out knowing the contents of those fields as userspace might not be up to date, I see no problem with requiring a newer kernel to support stuff added in future specs.
Quoted 8 lines
Steven, Mauro, you probably have a better idea of history of similar cases. How have other people handled reserved fields in underlying hardware reports that may become used in later revisions?
Also, probably makes sense to cc linux-edac on v3 of this series as the experts in these flows tend to read that list. Obviously intent so far is not to pass these into the edac subsystem, but I'd still let them know this work is going on.
Sounds good.
The struggle I'm having is in how flexible these records are compared to the other RAS trace events.
I'll drop the reserved fields.
Ira
Steven Rostedt
maintainerRe: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command
In reply to [email protected]Quoted 6 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:
new file mode 100644
index 000000000000..318ba5fe046einclude/trace/events/cxl.h new file +1
-0@@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0 */
include/trace/events/cxl.h line 11
Quoted 10 lines
include/trace/events/cxl.h new file +10
-0+#undef TRACE_SYSTEM +#define TRACE_SYSTEM cxl + +#if !defined(_CXL_TRACE_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _CXL_TRACE_EVENTS_H + +#include <asm-generic/unaligned.h> +#include <linux/tracepoint.h> + +TRACE_EVENT(overflow,
Please do not use generic names for grouped events. As most tooling can use the name and not associate it with the group, it makes it ambiguous for what event it wants to enable.
That is, call it "cxl_overflow" and not "overflow".
Quoted 20 lines
include/trace/events/cxl.h new file +20
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_get_event_payload *payload), + +TP_ARGS (dev_name, log, payload), + +TP_STRUCT__entry ( + __string(dev_name, dev_name) +__field (int ,log ) +__field (u64, first) +__field (u64, last) +__field (u16, count) + ), + +TP_fast_assign ( + __assign_str(dev_name, dev_name); + __entry->log = log; + __entry->count =le16_to_cpu (payload->overflow_err_count); + __entry->first =le64_to_cpu (payload->first_overflow_timestamp); + __entry->last =le64_to_cpu (payload->last_overflow_timestamp);
include/trace/events/cxl.h
Quoted 10 lines
include/trace/events/cxl.h new file +9
-0+ ), + +TP_printk ("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu" , +__get_str (dev_name),cxl_event_log_type_str (__entry->log), + __entry->count, __entry->first, __entry->last) + +); + +TRACE_EVENT(generic_event,
Same here. It's a "cxl_generic_event" not a "generic_event" that will clash with any other subsystem that would (but shouldn't) use the same name.
-- Steve
Quoted 22 lines
include/trace/events/cxl.h new file +21
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_record_raw *rec), + +TP_ARGS (dev_name, log, rec), + +TP_STRUCT__entry ( + CXL_EVT_TP_entry +__array (u8 ,data ,CXL_EVENT_RECORD_DATA_LENGTH ) + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); +memcpy (__entry->data, &rec->data,CXL_EVENT_RECORD_DATA_LENGTH ); + ), + +CXL_EVT_TP_printk ("%s" , +__print_hex (__entry->data,CXL_EVENT_RECORD_DATA_LENGTH )) +); + +#endif /* _CXL_TRACE_EVENTS_H */
Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command
Quoted 24 lines · Steven Rostedt
Quoted 6 lines · On Mon, 10 Oct 2022 15:41:22 -0700 [email protected] wrote:
new file mode 100644
index 000000000000..318ba5fe046einclude/trace/events/cxl.h new file +1
-0@@ -0,0 +1,130 @@ +/* SPDX-License-Identifier: GPL-2.0 */ include/trace/events/cxl.h line 11
Quoted 10 lines
include/trace/events/cxl.h new file +10
-0+#undef TRACE_SYSTEM +#define TRACE_SYSTEM cxl + +#if !defined(_CXL_TRACE_EVENTS_H) || defined(TRACE_HEADER_MULTI_READ ) +#define _CXL_TRACE_EVENTS_H + +#include <asm-generic/unaligned.h> +#include <linux/tracepoint.h> + +TRACE_EVENT(overflow,Please do not use generic names for grouped events. As most tooling can use the name and not associate it with the group, it makes it ambiguous for what event it wants to enable.
Quoted 1 line
That is, call it "
cxl_overflow" and not "overflow".
I did not realize that.[1]
[1] https://lore.kernel.org/linux-cxl/[email protected]/
Quoted 30 lines
Quoted 29 lines
include/trace/events/cxl.h new file +28
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_get_event_payload *payload), + +TP_ARGS (dev_name, log, payload), + +TP_STRUCT__entry ( + __string(dev_name, dev_name) +__field (int ,log ) +__field (u64, first) +__field (u64, last) +__field (u16, count) + ), + +TP_fast_assign ( + __assign_str(dev_name, dev_name); + __entry->log = log; + __entry->count =le16_to_cpu (payload->overflow_err_count); + __entry->first =le64_to_cpu (payload->first_overflow_timestamp); + __entry->last =le64_to_cpu (payload->last_overflow_timestamp); + ), + +TP_printk ("%s: EVENT LOG OVERFLOW log=%s : %u records from %llu to %llu" , +__get_str (dev_name),cxl_event_log_type_str (__entry->log), + __entry->count, __entry->first, __entry->last) + +); +
Quoted 5 lines
include/trace/events/cxl.h line 104
Quoted 1 line
include/trace/events/cxl.h new file +1
-0+TRACE_EVENT(generic_event,Same here. It's a "
cxl_generic_event" not a "generic_event" that will clash with any other subsystem that would (but shouldn't) use the same name.
Sure I'll change it back.
Thanks for setting us straight,
Ira
Quoted 25 lines
-- Steve
Quoted 22 lines
include/trace/events/cxl.h new file +21
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_record_raw *rec), + +TP_ARGS (dev_name, log, rec), + +TP_STRUCT__entry ( + CXL_EVT_TP_entry +__array (u8 ,data ,CXL_EVENT_RECORD_DATA_LENGTH ) + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); +memcpy (__entry->data, &rec->data,CXL_EVENT_RECORD_DATA_LENGTH ); + ), + +CXL_EVT_TP_printk ("%s" , +__print_hex (__entry->data,CXL_EVENT_RECORD_DATA_LENGTH )) +); + +#endif /* _CXL_TRACE_EVENTS_H */
Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command
In reply to [email protected]Hi Ira,
Quoted 39 lines · On 10/10/22 5:41 PM, [email protected] wrote:
From: Ira Weiny <[email protected]>
drivers/cxl/core/mbox.c +35
-0+static void cxl_mem_get_records_log (struct cxl_dev_state *cxlds, +enum cxl_event_log_type type) +{ +struct cxl_get_event_payload payload; + +do { +u8 log_type = type; +int rc; + + rc =cxl_mbox_send_cmd (cxlds,CXL_MBOX_OP_GET_EVENT_RECORD , + &log_type,sizeof (log_type), + &payload,sizeof (payload)); +if (rc) { +dev_err (cxlds->dev,"Event log '%s': Failed to query event records : %d" , +cxl_event_log_type_str (type), rc); +return ; + } + +if (le16_to_cpu (payload.record_count) ==1 ) +trace_generic_event (dev_name (cxlds->dev), type, + &payload.record); + +if (payload.flags &CXL_GET_EVENT_FLAG_OVERFLOW ) +trace_overflow (dev_name (cxlds->dev), type, &payload); + + }while (payload.flags &CXL_GET_EVENT_FLAG_MORE_RECORDS ); +} + +/** +* cxl_mem_get_event_records - Get Event Records from the device +* @cxlds: The device data for the operation +* +* Retrieve all event records available on the device and report them as trace +* events. +*
drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 758
Quoted 10 lines
drivers/cxl/core/mbox.c +10
-0+* See CXL rev 3.0 @8.2.9.2.2 Get Event Records +*/ +void cxl_mem_get_event_records (struct cxl_dev_state *cxlds) +{ +enum cxl_event_log_type log_type; + +dev_dbg (cxlds->dev,"Reading event logs\n" ); + +for (log_type =CXL_EVENT_TYPE_INFO ; + log_type <CXL_EVENT_TYPE_MAX ; log_type++)
Why should we loop through each event log here? What if the event record doesn't exist in the event log?
I got some Mailbox error messages like this while bootup..
[ 346.387010] cxl_pci 0000:7f:00.0: Sending command
[ 346.387181] cxl_pci 0000:7f:00.0: Doorbell wait took 0ms
[ 346.387197] cxl_pci 0000:7f:00.0: Mailbox operation had an error: cmd input was invalid [ 346.387205] cxl_pci 0000:7f:00.0: Event log 'Warning': Failed to query event records : -6 ..
Can we just read the "Event Status" field from Event Status Register (Device Status Registers Capability Offset + 00h) 8.2.8.3.1 in CXL Spec, determine if the records exist and just query those event logs?
Thanks,
Smita
Quoted 7 lines
drivers/cxl/core/mbox.c +4
-0+cxl_mem_get_records_log (cxlds, log_type); +} +EXPORT_SYMBOL_NS_GPL (cxl_mem_get_event_records,CXL ); +/** * cxl_mem_get_partition_info - Get partition info * @cxlds: The device data for the operation
Re: [RFC V2 PATCH 02/11] cxl/mem: Implement Get Event Records command
Quoted 6 lines · Smita Koralahalli
Hi Ira,
Quoted 3 lines · On 10/10/22 5:41 PM, [email protected] wrote:
From: Ira Weiny <[email protected]>
[snip]
Quoted 12 lines
Quoted 12 lines
drivers/cxl/core/mbox.c +12
-0+ +/** +* cxl_mem_get_event_records - Get Event Records from the device +* @cxlds: The device data for the operation +* +* Retrieve all event records available on the device and report them as trace +* events. +* +* See CXL rev 3.0 @8.2.9.2.2 Get Event Records +*/ +void cxl_mem_get_event_records (struct cxl_dev_state *cxlds) +{
Quoted 8 lines
drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 758
Quoted 6 lines
drivers/cxl/core/mbox.c +6
-0+enum cxl_event_log_type log_type; + +dev_dbg (cxlds->dev,"Reading event logs\n" ); + +for (log_type =CXL_EVENT_TYPE_INFO ; + log_type <CXL_EVENT_TYPE_MAX ; log_type++)Why should we loop through each event log here?
The idea was to clear all the event logs. I think this made more sense before the addition of the optional dynamic capacity log.
Quoted 3 lines
What if the event
record doesn't exist in the event log?
An empty log does not cause any issue. The query will simply return 0 records which is valid.
Quoted 10 lines
I got some Mailbox error messages like this while bootup..
[ 346.387010]cxl_pci0000:7f:00.0: Sending command
[ 346.387181]cxl_pci0000:7f:00.0: Doorbell wait took 0ms
[ 346.387197]cxl_pci0000:7f:00.0: Mailbox operation had an error: cmd input was invalid
[ 346.387205]cxl_pci0000:7f:00.0: Event log 'Warning': Failed to query event records : -6
..
Quoted 3 lines
Can we just read the "Event Status" field from Event Status Register (Device Status Registers Capability Offset + 00h) 8.2.8.3.1 in CXL Spec, determine if the records exist and just query those event logs?
Likely the hardware does not have the dynamic capacity log and so the code is asking for something invalid. I did not think of that when I added that new log. Checking status register looks to be the proper solution.
I'll throw in some testing in QEMU for this. I'll also have to implement the status register in QEMU to fully test.
Thanks for the testing! :-D
Ira
Quoted 11 lines
Thanks,
SmitaQuoted 7 lines
drivers/cxl/core/mbox.c +4
-0+cxl_mem_get_records_log (cxlds, log_type); +} +EXPORT_SYMBOL_NS_GPL (cxl_mem_get_event_records,CXL ); +/** * cxl_mem_get_partition_info - Get partition info * @cxlds: The device data for the operation
RFC PATCH v2 3/11 cxl/mem: Implement Clear Event Records command
In reply to [email protected]From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.3 defines the Clear Event Records mailbox command. After an event record is read it needs to be cleared from the event log.
Implement cxl_clear_event_record() and call it for each record retrieved from the device.
Each record is cleared individually. A clear all bit is specified but events could arrive between a get and the final clear all operation. Therefore each event is cleared specifically.
---
Changes from RFC:
Jonathan
Clean up init of payload and use return code. Also report any error to clear the event. s/v3.0/rev 3.0
--- drivers/cxl/core/mbox.c | 31 ++++++++++++++++++++++++++++--- drivers/cxl/cxlmem.h | 15 +++++++++++++++ include/uapi/linux/cxl_mem.h | 1 + 3 files changed, 44 insertions(+), 3 deletions(-)
drivers/cxl/core/mbox.c +28 -3
@@ -52,6 +52,7 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {#endif CXL_CMD (GET_SUPPORTED_LOGS ,0 ,CXL_VARIABLE_PAYLOAD ,CXL_CMD_FLAG_FORCE_ENABLE ),CXL_CMD (GET_EVENT_RECORD ,1 ,CXL_VARIABLE_PAYLOAD ,0 ), +CXL_CMD (CLEAR_EVENT_RECORD ,CXL_VARIABLE_PAYLOAD ,0 ,0 ),CXL_CMD (GET_FW_INFO ,0 ,0x50 ,0 ),CXL_CMD (GET_PARTITION_INFO ,0 ,0x20 ,0 ), CXL_CMD(GET_LSA, 0x8, CXL_VARIABLE_PAYLOAD, 0),
@@ -711,6 +712,20 @@ int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) }EXPORT_SYMBOL_NS_GPL (cxl_enumerate_cmds,CXL ); +static int cxl_clear_event_record (struct cxl_dev_state *cxlds, +enum cxl_event_log_type log, +__le16 handle) +{ +struct cxl_mbox_clear_event_payload payload = { + .event_log = log, + .nr_recs =1 , + .handle = handle, + }; + +return cxl_mbox_send_cmd (cxlds,CXL_MBOX_OP_CLEAR_EVENT_RECORD , + &payload,sizeof (payload),NULL ,0 ); +} +static void cxl_mem_get_records_log (struct cxl_dev_state *cxlds,enum cxl_event_log_type type) {
@@ -729,9 +744,18 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,return ; } -if (le16_to_cpu (payload.record_count) ==1 ) +if (le16_to_cpu (payload.record_count) ==1 ) {trace_generic_event (dev_name (cxlds->dev), type, &payload.record); + rc =cxl_clear_event_record (cxlds, type, + payload.record.hdr.handle); +if (rc) { +dev_err (cxlds->dev,"Event log '%s': Failed to clear event (%u) : %d" , +cxl_event_log_type_str (type), + payload.record.hdr.handle, rc); +return ; + } + }if (payload.flags &CXL_GET_EVENT_FLAG_OVERFLOW )trace_overflow (dev_name (cxlds->dev), type, &payload);
@@ -743,10 +767,11 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds,* cxl_mem_get_event_records - Get Event Records from the device * @cxlds: The device data for the operation * -* Retrieve all event records available on the device and report them as trace -* events. +* Retrieve all event records available on the device, report them as trace +* events, and clear them. * * See CXL rev 3.0 @8.2.9.2.2 Get Event Records +* See CXL rev 3.0 @8.2.9.2.3 Clear Event Records */ void cxl_mem_get_event_records (struct cxl_dev_state *cxlds) {
drivers/cxl/cxlmem.h +15 -0
@@ -255,6 +255,7 @@ enum cxl_opcode {CXL_MBOX_OP_INVALID =0x0000 ,CXL_MBOX_OP_RAW =CXL_MBOX_OP_INVALID ,CXL_MBOX_OP_GET_EVENT_RECORD =0x0100 , +CXL_MBOX_OP_CLEAR_EVENT_RECORD =0x0101 ,CXL_MBOX_OP_GET_FW_INFO =0x0200 ,CXL_MBOX_OP_ACTIVATE_FW =0x0202 , CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
@@ -396,6 +397,20 @@ static inline const char *cxl_event_log_type_str(enum cxl_event_log_type type)return "<unknown>" ; } +/* +* Clear Event Records input payload +* CXL rev 3.0 section 8.2.9.2.3; Table 8-51 +* +* Space given for 1 record +*/ +struct cxl_mbox_clear_event_payload { +u8 event_log;/* enum cxl_event_log_type */ +u8 clear_flags; +u8 nr_recs;/* 1 for this struct */ +u8 reserved[3 ]; +__le16 handle; +}; +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;
include/uapi/linux/cxl_mem.h +1 -0
@@ -25,6 +25,7 @@___C (RAW ,"Raw device command" ), \___C (GET_SUPPORTED_LOGS ,"Get Supported Logs" ), \___C (GET_EVENT_RECORD ,"Get Event Record" ), \ +___C (CLEAR_EVENT_RECORD ,"Clear Event Record" ), \___C (GET_FW_INFO ,"Get FW Info" ), \___C (GET_PARTITION_INFO ,"Get Partition Information" ), \ ___C(GET_LSA, "Get Label Storage Area"), \
Signature · 1 line
2.37.2
RFC PATCH v2 4/11 cxl/mem: Clear events on driver load
In reply to [email protected]From: Ira Weiny <[email protected]>
The information contained in the events prior to the driver loading can be queried at any time through other mailbox commands.
Ensure a clean slate of events by reading and clearing the events. The events are sent to the trace buffer but it is not anticipated to have anyone listening to it at driver load time.
--- drivers/cxl/pci.c | 2 ++ tools/testing/cxl/test/mem.c | 2 ++ 2 files changed, 4 insertions(+)
drivers/cxl/pci.c +2 -0
@@ -498,6 +498,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (IS_ERR (cxlmd))return PTR_ERR (cxlmd); +cxl_mem_get_event_records (cxlds); +if (resource_size (&cxlds->pmem_res) &&IS_ENABLED (CONFIG_CXL_PMEM )) rc =devm_cxl_add_nvdimm (&pdev->dev, cxlmd);
tools/testing/cxl/test/mem.c +2 -0
@@ -285,6 +285,8 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)if (IS_ERR (cxlmd))return PTR_ERR (cxlmd); +cxl_mem_get_event_records (cxlds); +if (resource_size (&cxlds->pmem_res) &&IS_ENABLED (CONFIG_CXL_PMEM )) rc =devm_cxl_add_nvdimm (dev, cxlmd);
Signature · 1 line
2.37.2
Re: [RFC V2 PATCH 04/11] cxl/mem: Clear events on driver load
Quoted 10 lines · On Mon, 10 Oct 2022 15:41:24 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
The information contained in the events prior to the driver loading can be queried at any time through other mailbox commands.
Ensure a clean slate of events by reading and clearing the events. The events are sent to the trace buffer but it is not anticipated to have anyone listening to it at driver load time.
Signed-off-by Ira Weiny <[email protected]>
Makes sense I think.
Quoted 31 lines
--- drivers/cxl/pci.c | 2 ++ tools/testing/cxl/test/mem.c | 2 ++ 2 files changed, 4 insertions(+)drivers/cxl/pci.c +2
-0@@ -498,6 +498,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (IS_ERR (cxlmd))return PTR_ERR (cxlmd); +cxl_mem_get_event_records (cxlds); +if (resource_size (&cxlds->pmem_res) &&IS_ENABLED (CONFIG_CXL_PMEM )) rc =devm_cxl_add_nvdimm (&pdev->dev, cxlmd);tools/testing/cxl/test/mem.c +2
-0@@ -285,6 +285,8 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)if (IS_ERR (cxlmd))return PTR_ERR (cxlmd); +cxl_mem_get_event_records (cxlds); +if (resource_size (&cxlds->pmem_res) &&IS_ENABLED (CONFIG_CXL_PMEM )) rc =devm_cxl_add_nvdimm (dev, cxlmd);
RFC PATCH v2 5/11 cxl/mem: Trace General Media Event Record
In reply to [email protected]From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.1 defines the General Media Event Record.
Determine if the event read is a general media record and if so trace the record.
---
Changes from RFC:
Add reserved byte array
Use common CXL event header record macros Jonathan
Use unaligned_le{24,16} for unaligned fields Don't use the inverse of phy addr mask
Dave Jiang
s/cxl_gen_media_event/general_media s/cxl_evt_gen_media/cxl_event_gen_media
--- drivers/cxl/core/mbox.c | 30 ++++++++++- drivers/cxl/cxlmem.h | 20 +++++++ include/trace/events/cxl.h | 108 +++++++++++++++++++++++++++++++++++++ 3 files changed, 156 insertions(+), 2 deletions(-)
drivers/cxl/core/mbox.c +28 -2
@@ -712,6 +712,32 @@ int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) }EXPORT_SYMBOL_NS_GPL (cxl_enumerate_cmds,CXL ); +/* +* General Media Event Record +* CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 +*/ +static const uuid_t gen_media_event_uuid = +UUID_INIT (0xfbcd0a77 ,0xc260 ,0x417f , +0x85 ,0xa9 ,0x08 ,0x8b ,0x16 ,0x21 ,0xeb ,0xa6 ); + +static void cxl_trace_event_record (const char *dev_name, +enum cxl_event_log_type type, +struct cxl_get_event_payload *payload) +{ +uuid_t *id = &payload->record.hdr.id; + +if (uuid_equal (id, &gen_media_event_uuid)) { +struct cxl_event_gen_media *rec = + (struct cxl_event_gen_media *)&payload->record; + +trace_general_media (dev_name, type, rec); +return ; + } + +/* For unknown record types print just the header */ +trace_generic_event (dev_name, type, &payload->record); +} +static int cxl_clear_event_record (struct cxl_dev_state *cxlds,enum cxl_event_log_type log,__le16 handle)
@@ -745,8 +771,8 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds, }if (le16_to_cpu (payload.record_count) ==1 ) { -trace_generic_event (dev_name (cxlds->dev), type, - &payload.record); +cxl_trace_event_record (dev_name (cxlds->dev), type, + &payload); rc =cxl_clear_event_record (cxlds, type, payload.record.hdr.handle);if (rc) {
drivers/cxl/cxlmem.h +20 -0
@@ -411,6 +411,26 @@ struct cxl_mbox_clear_event_payload {__le16 handle; }; +/* +* General Media Event Record +* CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 +*/ +#define CXL_EVENT_GEN_MED_COMP_ID_SIZE 0x10 +#define CXL_EVENT_GEN_MED_RES_SIZE 0x2e +struct cxl_event_gen_media { +struct cxl_event_record_hdr hdr; +__le64 phys_addr; +u8 descriptor; +u8 type; +u8 transaction_type; +u8 validity_flags[2 ]; +u8 channel; +u8 rank; +u8 device[3 ]; +u8 component_id[CXL_EVENT_GEN_MED_COMP_ID_SIZE ]; +u8 reserved[CXL_EVENT_GEN_MED_RES_SIZE ]; +} __packed; +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;
include/trace/events/cxl.h +108 -0
114 lines not on this page · raw message
Signature · 1 line
2.37.2
Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
Quoted 8 lines · On Mon, 10 Oct 2022 15:41:25 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.1 defines the General Media Event Record.
Determine if the event read is a general media record and if so trace the record.
Signed-off-by Ira Weiny <[email protected]>
I'll review the rest of these with the assumption that the question of reserved bytes in tracepoints will get resolved before these go in.
One minor comment on a comment inline. Other than those lgtm
Quoted 39 lines
---
Changes from RFC:
Add reserved byte array
Use common CXL event header record macros Jonathan
Use unaligned_le{24,16} for unaligned fields Don't use the inverse of phy addr mask
Dave Jiang
s/cxl_gen_media_event/general_media s/cxl_evt_gen_media/cxl_event_gen_media--- drivers/cxl/core/mbox.c | 30 ++++++++++- drivers/cxl/cxlmem.h | 20 +++++++ include/trace/events/cxl.h | 108 +++++++++++++++++++++++++++++++++++++ 3 files changed, 156 insertions(+), 2 deletions(-)drivers/cxl/core/mbox.c +14
-0@@ -712,6 +712,32 @@ int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) }EXPORT_SYMBOL_NS_GPL (cxl_enumerate_cmds,CXL ); +/* +* General Media Event Record +* CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 +*/ +static const uuid_t gen_media_event_uuid = +UUID_INIT (0xfbcd0a77 ,0xc260 ,0x417f , +0x85 ,0xa9 ,0x08 ,0x8b ,0x16 ,0x21 ,0xeb ,0xa6 ); + +static void cxl_trace_event_record (const char *dev_name, +enum cxl_event_log_type type, +struct cxl_get_event_payload *payload) +{ +uuid_t *id = &payload->record.hdr.id; +
drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 738
Quoted 10 lines
drivers/cxl/core/mbox.c +10
-0+if (uuid_equal (id, &gen_media_event_uuid)) { +struct cxl_event_gen_media *rec = + (struct cxl_event_gen_media *)&payload->record; + +trace_general_media (dev_name, type, rec); +return ; + } + +/* For unknown record types print just the header */ +trace_generic_event (dev_name, type, &payload->record);
Looks like it prints the whole thing now...
Quoted 11 lines
drivers/cxl/core/mbox.c +2
-0+} +include/trace/events/cxl.h +0
-0@@ -122,6 +122,114 @@ TRACE_EVENT(generic_event,
include/trace/events/cxl.h TRACE_EVENT(generic_event, · line 133
Quoted 10 lines
include/trace/events/cxl.h +10
-0+#define CXL_GMER_VALID_CHANNEL BIT(0) +#define CXL_GMER_VALID_RANK BIT(1) +#define CXL_GMER_VALID_DEVICE BIT(2) +#define CXL_GMER_VALID_COMPONENT BIT(3) +#define show_valid_flags (flags) __print_flags(flags, "|", \ + { CXL_GMER_VALID_CHANNEL, "CHANNEL" }, \ + { CXL_GMER_VALID_RANK, "RANK" }, \ + { CXL_GMER_VALID_DEVICE, "DEVICE" }, \ + { CXL_GMER_VALID_COMPONENT, "COMPONENT" } \ +)
I'd still rather we only had the TP_printk only print those parts that are valid rather than the mess of having to go check the validity bit before deciding whether to take notice of the field. But meh, not that important given thats not the main intended way to consume this data.
Quoted 40 lines
include/trace/events/cxl.h +40
-0+ +TRACE_EVENT (general_media, + + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_gen_media *rec), + +TP_ARGS (dev_name, log, rec), + +TP_STRUCT__entry ( + CXL_EVT_TP_entry +/* General Media */ + __field(u64, phys_addr) + __field(u8, descriptor) + __field(u8, type) + __field(u8, transaction_type) + __field(u8, channel) + __field(u32, device) + __array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE) + __array(u8, reserved, CXL_EVENT_GEN_MED_RES_SIZE) + __field(u16, validity_flags) +__field (u8, rank)/* Out of order to pack trace record */ + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); + +/* General Media */ + __entry->phys_addr = le64_to_cpu(rec->phys_addr); + __entry->descriptor = rec->descriptor; + __entry->type = rec->type; + __entry->transaction_type = rec->transaction_type; + __entry->channel = rec->channel; + __entry->rank = rec->rank; + __entry->device = get_unaligned_le24(rec->device); + memcpy(__entry->comp_id, &rec->component_id, + CXL_EVENT_GEN_MED_COMP_ID_SIZE); +memcpy (__entry->reserved, &rec->reserved, +CXL_EVENT_GEN_MED_RES_SIZE ); + __entry->validity_flags =get_unaligned_le16 (&rec->validity_flags); + ),
Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
Quoted 7 lines · Jonathan Cameron
Quoted 4 lines · On Mon, 10 Oct 2022 15:41:25 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.1 defines the General Media Event Record.
Quoted 7 lines
Quoted 4 lines
Determine if the event read is a general media record and if so trace the record.
Signed-off-by Ira Weiny <[email protected]>I'll review the rest of these with the assumption that the question of reserved bytes in tracepoints will get resolved before these go in.
Yea I'm removing them. I think I messed up somehow because I thought I had removed the reserved fields from the records. But perhaps that was only in my dreams... :-/ Sorry. :-)
Quoted 5 lines
One minor comment on a comment inline. Other than those lgtm
Reviewed-by Jonathan Cameron <[email protected]>
Thanks!
[snip]
Quoted 19 lines
Quoted 19 lines
(quoted diff) +18
-0+/* + * General Media Event Record + * CXL rev 3.0 Section 8.2.9.2.1.1; Table 8-43 + */ +static const uuid_t gen_media_event_uuid = + UUID_INIT(0xfbcd0a77, 0xc260, 0x417f, + 0x85, 0xa9, 0x08, 0x8b, 0x16, 0x21, 0xeb, 0xa6); + +static void cxl_trace_event_record(const char *dev_name, + enum cxl_event_log_type type, + struct cxl_get_event_payload *payload) +{ + uuid_t *id = &payload->record.hdr.id; + + if (uuid_equal(id, &gen_media_event_uuid)) { + struct cxl_event_gen_media *rec = + (struct cxl_event_gen_media *)&payload->record; +
Quoted 8 lines
Review
Quoted 6 lines
(quoted diff) +6
-0+ trace_general_media(dev_name, type, rec); + return; + } + + /* For unknown record types print just the header */ + trace_generic_event(dev_name, type, &payload->record);Looks like it prints the whole thing now...
An unknown record is dumped yes. I'm ok with skipping this but it was discussed early on that any unknown record would be passed through.
Quoted 13 lines
Quoted 11 lines
(quoted diff) +2
-0+} +include/trace/events/cxl.h +0
-0@@ -122,6 +122,114 @@ TRACE_EVENT(generic_event,
Quoted 16 lines
include/trace/events/cxl.h TRACE_EVENT(generic_event, · line 131
Quoted 10 lines
include/trace/events/cxl.h +10
-0+#define CXL_GMER_VALID_CHANNEL BIT(0) +#define CXL_GMER_VALID_RANK BIT(1) +#define CXL_GMER_VALID_DEVICE BIT(2) +#define CXL_GMER_VALID_COMPONENT BIT(3) +#define show_valid_flags (flags) __print_flags(flags, "|", \ + { CXL_GMER_VALID_CHANNEL, "CHANNEL" }, \ + { CXL_GMER_VALID_RANK, "RANK" }, \ + { CXL_GMER_VALID_DEVICE, "DEVICE" }, \ + { CXL_GMER_VALID_COMPONENT, "COMPONENT" } \ +)I'd still rather we only had the
TP_printkonly print those parts that are valid rather than the mess of having to go check the validity bit before deciding whether to take notice of the field. But meh, not that important given thats not the main intended way to consume this data.
Ok I spent some time really thinking about this and below is an attempt at that.
However, I must be missing something in what you are proposing because I don't like having extra space in the trace buffer to print into like this and I can't figure out where else to put a print buffer.
Also note that this will have no impact on most of the user space tools which use libtracefs as they will see all the fields and will need to do a similar decode.
Do you really think this is worth the effort?
Ira
commit 54c4ee67bcac6a38cbc9b0ea2ef952e197356dee
Author: Ira Weiny <[email protected]> Date: Fri Oct 14 16:15:27 2022 -0700
squash: Attempt to print only valid fields per Jonathan suggestion
drivers/cxl/core/mbox.c +22 -0
@@ -720,6 +720,28 @@ static const uuid_t gen_media_event_uuid =UUID_INIT (0xfbcd0a77 ,0xc260 ,0x417f ,0x85 ,0xa9 ,0x08 ,0x8b ,0x16 ,0x21 ,0xeb ,0xa6 ); +const char *cxl_gem_print_valid (u8 *buf,int nr,u16 valid_flags,u8 channel, +u8 rank,u32 device,u8 *comp_id) +{ +char *str = buf; +int rc =0 ; + +if (valid_flags &CXL_GMER_VALID_CHANNEL ) + rc =snprintf (str, nr,"channel=%u " , channel); + +if (valid_flags &CXL_GMER_VALID_RANK ) + rc =snprintf (str + rc, nr - rc,"rank=%u " , rank); + +if (valid_flags &CXL_GMER_VALID_DEVICE ) + rc =snprintf (str + rc, nr - rc,"device=%u " , device); + +if (valid_flags &CXL_GMER_VALID_COMPONENT ) + rc =snprintf (str + rc, nr - rc,"comp_id=%*ph " , +CXL_EVENT_GEN_MED_COMP_ID_SIZE , comp_id); + +return str; +} +static void cxl_trace_event_record (const char *dev_name,enum cxl_event_log_type type,struct cxl_get_event_payload *payload)
drivers/cxl/cxlmem.h +7 -0
@@ -429,6 +429,13 @@ struct cxl_event_gen_media {u8 reserved[0x2e ]; } __packed; +#define CXL_GMER_VALID_CHANNEL BIT(0) +#define CXL_GMER_VALID_RANK BIT(1) +#define CXL_GMER_VALID_DEVICE BIT(2) +#define CXL_GMER_VALID_COMPONENT BIT(3) +const char *cxl_gem_print_valid (u8 *buf,int nr,u16 valid_flags,u8 channel, +u8 rank,u32 device,u8 *comp_id); +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;
include/trace/events/cxl.h +8 -16
@@ -161,16 +161,7 @@ TRACE_EVENT(generic_event, { CXL_GMER_TRANS_INTERNAL_MEDIA_MANAGEMENT, "Internal Media Management" } \ ) -#define CXL_GMER_VALID_CHANNEL BIT(0) -#define CXL_GMER_VALID_RANK BIT(1) -#define CXL_GMER_VALID_DEVICE BIT(2) -#define CXL_GMER_VALID_COMPONENT BIT(3) -#define show_valid_flags (flags) __print_flags(flags, "|", \ - { CXL_GMER_VALID_CHANNEL, "CHANNEL" }, \ - { CXL_GMER_VALID_RANK, "RANK" }, \ - { CXL_GMER_VALID_DEVICE, "DEVICE" }, \ - { CXL_GMER_VALID_COMPONENT, "COMPONENT" } \ -) +#define CXL_VALID_PRINT_STR_LEN 512 TRACE_EVENT(general_media,
@@ -191,6 +182,7 @@ TRACE_EVENT(general_media, __array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE) __field(u16, validity_flags) __field(u8, rank) /* Out of order to pack trace record */ +__array (u8, str,CXL_VALID_PRINT_STR_LEN ) ), TP_fast_assign(
@@ -209,17 +201,17 @@ TRACE_EVENT(general_media, __entry->validity_flags =get_unaligned_le16 (&rec->validity_flags); ), -CXL_EVT_TP_printk ("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ -"trans_type='%s' channel=%u rank=%u device=%x comp_id=%s " \ -"valid_flags='%s'" , +CXL_EVT_TP_printk ("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ +"trans_type='%s' %s" , __entry->phys_addr &CXL_GMER_PHYS_ADDR_MASK , (__entry->phys_addr &CXL_GMER_PHYS_ADDR_VOLATILE ) ?"TRUE" :"FALSE" ,show_event_desc_flags (__entry->descriptor),show_mem_event_type (__entry->type),show_trans_type (__entry->transaction_type), - __entry->channel, __entry->rank, __entry->device, -__print_hex (__entry->comp_id,CXL_EVENT_GEN_MED_COMP_ID_SIZE ), -show_valid_flags (__entry->validity_flags) +cxl_gem_print_valid (__entry->str,CXL_VALID_PRINT_STR_LEN , + __entry->validity_flags, __entry->channel, + __entry->rank, __entry->device, + __entry->comp_id) ) );
Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
Quoted 25 lines
Quoted 8 lines
Quoted 6 lines
include/trace/events/cxl.h +0
-0@@ -122,6 +122,114 @@ TRACE_EVENT(generic_event,Quoted 16 lines
include/trace/events/cxl.h TRACE_EVENT(generic_event, · line 131
Quoted 10 lines
include/trace/events/cxl.h +10
-0+#define CXL_GMER_VALID_CHANNEL BIT(0) +#define CXL_GMER_VALID_RANK BIT(1) +#define CXL_GMER_VALID_DEVICE BIT(2) +#define CXL_GMER_VALID_COMPONENT BIT(3) +#define show_valid_flags (flags) __print_flags(flags, "|", \ + { CXL_GMER_VALID_CHANNEL, "CHANNEL" }, \ + { CXL_GMER_VALID_RANK, "RANK" }, \ + { CXL_GMER_VALID_DEVICE, "DEVICE" }, \ + { CXL_GMER_VALID_COMPONENT, "COMPONENT" } \ +)I'd still rather we only had the
TP_printkonly print those parts that are valid rather than the mess of having to go check the validity bit before deciding whether to take notice of the field. But meh, not that important given thats not the main intended way to consume this data.
Quoted 6 lines
Ok I spent some time really thinking about this and below is an attempt at that.
However, I must be missing something in what you are proposing because I don't like having extra space in the trace buffer to print into like this and I can't figure out where else to put a print buffer.
Putting the space in the trace buffer definitely doesn't makes sense. Ah. Looking back the CCIX code I assumed it was serialized so could use a static buffer.
Looking at other similar cases though and we have a lot of use of trace_seq_printf() e.g. libata_trace_parse_status() though note there is some magic macro stuff in include/trace/events/libata.h to tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14
That seems to get you access to the actual buffer we are printing into in similar cases.
Quoted 6 lines
Also note that this will have no impact on most of the user space tools which use libtracefs as they will see all the fields and will need to do a similar decode.
Do you really think this is worth the effort?
With the allocation problem, definitely not. With that solved, it's not a huge amount of extra code. Also rather nicely 'documents' meaning of the valid bits.
I'm a kernel hacker. I like to not need much beyond echo and cat :)
Jonathan
Quoted 116 lines
Ira
commit
54c4ee67bcac6a38cbc9b0ea2ef952e197356deeAuthor: Ira Weiny <[email protected]> Date: Fri Oct 14 16:15:27 2022 -0700squash: Attempt to print only valid fields per Jonathan suggestion
drivers/cxl/core/mbox.c +22
-0@@ -720,6 +720,28 @@ static const uuid_t gen_media_event_uuid =UUID_INIT (0xfbcd0a77 ,0xc260 ,0x417f ,0x85 ,0xa9 ,0x08 ,0x8b ,0x16 ,0x21 ,0xeb ,0xa6 ); +const char *cxl_gem_print_valid (u8 *buf,int nr,u16 valid_flags,u8 channel, +u8 rank,u32 device,u8 *comp_id) +{ +char *str = buf; +int rc =0 ; + +if (valid_flags &CXL_GMER_VALID_CHANNEL ) + rc =snprintf (str, nr,"channel=%u " , channel); + +if (valid_flags &CXL_GMER_VALID_RANK ) + rc =snprintf (str + rc, nr - rc,"rank=%u " , rank); + +if (valid_flags &CXL_GMER_VALID_DEVICE ) + rc =snprintf (str + rc, nr - rc,"device=%u " , device); + +if (valid_flags &CXL_GMER_VALID_COMPONENT ) + rc =snprintf (str + rc, nr - rc,"comp_id=%*ph " , +CXL_EVENT_GEN_MED_COMP_ID_SIZE , comp_id); + +return str; +} +static void cxl_trace_event_record (const char *dev_name,enum cxl_event_log_type type,struct cxl_get_event_payload *payload)drivers/cxl/cxlmem.h +7
-0@@ -429,6 +429,13 @@ struct cxl_event_gen_media {u8 reserved[0x2e ]; } __packed; +#define CXL_GMER_VALID_CHANNEL BIT(0) +#define CXL_GMER_VALID_RANK BIT(1) +#define CXL_GMER_VALID_DEVICE BIT(2) +#define CXL_GMER_VALID_COMPONENT BIT(3) +const char *cxl_gem_print_valid (u8 *buf,int nr,u16 valid_flags,u8 channel, +u8 rank,u32 device,u8 *comp_id); +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;include/trace/events/cxl.h +8
-16@@ -161,16 +161,7 @@ TRACE_EVENT(generic_event, { CXL_GMER_TRANS_INTERNAL_MEDIA_MANAGEMENT, "Internal Media Management" } \ ) -#define CXL_GMER_VALID_CHANNEL BIT(0) -#define CXL_GMER_VALID_RANK BIT(1) -#define CXL_GMER_VALID_DEVICE BIT(2) -#define CXL_GMER_VALID_COMPONENT BIT(3) -#define show_valid_flags (flags) __print_flags(flags, "|", \ - { CXL_GMER_VALID_CHANNEL, "CHANNEL" }, \ - { CXL_GMER_VALID_RANK, "RANK" }, \ - { CXL_GMER_VALID_DEVICE, "DEVICE" }, \ - { CXL_GMER_VALID_COMPONENT, "COMPONENT" } \ -) +#define CXL_VALID_PRINT_STR_LEN 512 TRACE_EVENT(general_media,@@ -191,6 +182,7 @@ TRACE_EVENT(general_media, __array(u8, comp_id, CXL_EVENT_GEN_MED_COMP_ID_SIZE) __field(u16, validity_flags) __field(u8, rank) /* Out of order to pack trace record */ +__array (u8, str,CXL_VALID_PRINT_STR_LEN ) ), TP_fast_assign(@@ -209,17 +201,17 @@ TRACE_EVENT(general_media, __entry->validity_flags =get_unaligned_le16 (&rec->validity_flags); ), -CXL_EVT_TP_printk ("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ -"trans_type='%s' channel=%u rank=%u device=%x comp_id=%s " \ -"valid_flags='%s'" , +CXL_EVT_TP_printk ("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ +"trans_type='%s' %s" , __entry->phys_addr &CXL_GMER_PHYS_ADDR_MASK , (__entry->phys_addr &CXL_GMER_PHYS_ADDR_VOLATILE ) ?"TRUE" :"FALSE" ,show_event_desc_flags (__entry->descriptor),show_mem_event_type (__entry->type),show_trans_type (__entry->transaction_type), - __entry->channel, __entry->rank, __entry->device, -__print_hex (__entry->comp_id,CXL_EVENT_GEN_MED_COMP_ID_SIZE ), -show_valid_flags (__entry->validity_flags) +cxl_gem_print_valid (__entry->str,CXL_VALID_PRINT_STR_LEN , + __entry->validity_flags, __entry->channel, + __entry->rank, __entry->device, + __entry->comp_id) ) );
Steven Rostedt
maintainerRe: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
Quoted 8 lines · On Mon, 17 Oct 2022 17:37:17 +0100 Jonathan Cameron…
Looking at other similar cases though and we have a lot of use of
trace_seq_printf()e.g.libata_trace_parse_status()though note there is some magic macro stuff ininclude/trace/events/libata.hto tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14That seems to get you access to the actual buffer we are printing into in similar cases.
Looking at the code you linked to, I wonder why __print_flags() wasn't used?
For instance, you have:
const char *libata_trace_parse_status(struct trace_seq *p, unsigned char status)
{
const char *ret = trace_seq_buffer_ptr(p);
trace_seq_printf(p, "{ ");
if (status & ATA_BUSY)
trace_seq_printf(p, "BUSY ");
if (status & ATA_DRDY)
trace_seq_printf(p, "DRDY ");
if (status & ATA_DF)
trace_seq_printf(p, "DF ");
if (status & ATA_DSC)
trace_seq_printf(p, "DSC ");
if (status & ATA_DRQ)
trace_seq_printf(p, "DRQ ");
if (status & ATA_CORR)
trace_seq_printf(p, "CORR ");
if (status & ATA_SENSE)
trace_seq_printf(p, "SENSE ");
if (status & ATA_ERR)
trace_seq_printf(p, "ERR ");
trace_seq_putc(p, '}');
trace_seq_putc(p, 0);
return ret;
}Which is just a re-implementation of:
__print_flags(status, " ",
{ ATA_BUSY, "BUSY" },
{ ATA_DRDY, "DRDY" },
{ ATA_DF, "DF" },
{ ATA_DSC, "DSC" },
{ ATA_DRQ, "DRQ" },
{ ATA_CORR, "CORR" },
{ ATA_SENSE, "SENSE" },
{ ATA_ERR, "ERR" })The major difference between the two, is that libtraceevent will be able to parse the above and convert the status bits into strings, whereas using libata_trace_parse_status() will just give you a parsing error.
That is, perf and trace-cmd will not be able to parse it unless you write a separate plugin for libtraceevent to do it but that means you'll have duplicate code.
I know you just want echo and cat, but that will still work, and this will make it work for the tooling as well.
-- Steve
Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
Quoted 62 lines · On Mon, 17 Oct 2022 13:21:43 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 8 lines · On Mon, 17 Oct 2022 17:37:17 +0100 Jonathan Cameron…
Looking at other similar cases though and we have a lot of use of
trace_seq_printf()e.g.libata_trace_parse_status()though note there is some magic macro stuff ininclude/trace/events/libata.hto tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14That seems to get you access to the actual buffer we are printing into in similar cases.
Looking at the code you linked to, I wonder why
__print_flags()wasn't used?For instance, you have:
const char *
libata_trace_parse_status(struct trace_seq *p, unsigned char status){ const char *ret = trace_seq_buffer_ptr(p); trace_seq_printf(p, "{ "); if (status & ATA_BUSY) trace_seq_printf(p, "BUSY "); if (status & ATA_DRDY) trace_seq_printf(p, "DRDY "); if (status & ATA_DF) trace_seq_printf(p, "DF "); if (status & ATA_DSC) trace_seq_printf(p, "DSC "); if (status & ATA_DRQ) trace_seq_printf(p, "DRQ "); if (status & ATA_CORR) trace_seq_printf(p, "CORR "); if (status & ATA_SENSE) trace_seq_printf(p, "SENSE "); if (status & ATA_ERR) trace_seq_printf(p, "ERR "); trace_seq_putc(p, '}'); trace_seq_putc(p, 0); return ret; }Which is just a re-implementation of:
__print_flags(status, " ",
{ ATA_BUSY, "BUSY" }, { ATA_DRDY, "DRDY" }, { ATA_DF, "DF" }, { ATA_DSC, "DSC" }, { ATA_DRQ, "DRQ" }, { ATA_CORR, "CORR" }, { ATA_SENSE, "SENSE" }, { ATA_ERR, "ERR" })The major difference between the two, is that libtraceevent will be able to parse the above and convert the status bits into strings, whereas using
libata_trace_parse_status()will just give you a parsing error.
Quoted 6 lines
That is, perf and trace-cmd will not be able to parse it unless you write a separate plugin for libtraceevent to do it but that means you'll have duplicate code.
I know you just want echo and cat, but that will still work, and this will make it work for the tooling as well.
Excellent point, though in the case we are interested in for CXL, __print_flags() is not enough.
We have a mass of fields that only contain something useful to print if the valid bits in a mask are set. I just pulled that example to show how trace_seq_printf() could be used to achieve optional printing as opposed to current situation where the reader of the print has to interpret the mask to work out if fields contain anything useful.
To do something nice with them in perf (well probably ras-daemon in this case) we'll have to parse the valid bits anyway so effectively write such a plugin. There we need to do a bunch of mangling to get the events stored in a DB anyway, so this isn't a huge overhead.
Jonathan
Quoted 3 lines
-- Steve
Re: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
Quoted 81 lines · Jonathan Cameron
Quoted 62 lines · On Mon, 17 Oct 2022 13:21:43 -0400 Steven Rostedt <[email protected]> wrote:
Quoted 8 lines · On Mon, 17 Oct 2022 17:37:17 +0100 Jonathan Cameron…
Looking at other similar cases though and we have a lot of use of
trace_seq_printf()e.g.libata_trace_parse_status()though note there is some magic macro stuff ininclude/trace/events/libata.hto tie that together.
https://elixir.bootlin.com/linux/latest/source/drivers/ata/libata-trace.c#L14That seems to get you access to the actual buffer we are printing into in similar cases.
Looking at the code you linked to, I wonder why
__print_flags()wasn't used?For instance, you have:
const char *
libata_trace_parse_status(struct trace_seq *p, unsigned char status){ const char *ret = trace_seq_buffer_ptr(p); trace_seq_printf(p, "{ "); if (status & ATA_BUSY) trace_seq_printf(p, "BUSY "); if (status & ATA_DRDY) trace_seq_printf(p, "DRDY "); if (status & ATA_DF) trace_seq_printf(p, "DF "); if (status & ATA_DSC) trace_seq_printf(p, "DSC "); if (status & ATA_DRQ) trace_seq_printf(p, "DRQ "); if (status & ATA_CORR) trace_seq_printf(p, "CORR "); if (status & ATA_SENSE) trace_seq_printf(p, "SENSE "); if (status & ATA_ERR) trace_seq_printf(p, "ERR "); trace_seq_putc(p, '}'); trace_seq_putc(p, 0); return ret; }Which is just a re-implementation of:
__print_flags(status, " ",
{ ATA_BUSY, "BUSY" }, { ATA_DRDY, "DRDY" }, { ATA_DF, "DF" }, { ATA_DSC, "DSC" }, { ATA_DRQ, "DRQ" }, { ATA_CORR, "CORR" }, { ATA_SENSE, "SENSE" }, { ATA_ERR, "ERR" })The major difference between the two, is that libtraceevent will be able to parse the above and convert the status bits into strings, whereas using
libata_trace_parse_status()will just give you a parsing error.Quoted 6 lines
That is, perf and trace-cmd will not be able to parse it unless you write a separate plugin for libtraceevent to do it but that means you'll have duplicate code.
I know you just want echo and cat, but that will still work, and this will make it work for the tooling as well.
Excellent point, though in the case we are interested in for CXL,
__print_flags()is not enough.We have a mass of fields that only contain something useful to print if the valid bits in a mask are set. I just pulled that example to show how
trace_seq_printf()could be used to achieve optional printing as opposed to current situation where the reader of the print has to interpret the mask to work out if fields contain anything useful.
Quoted 4 lines
To do something nice with them in perf (well probably ras-daemon in this case) we'll have to parse the valid bits anyway so effectively write such a plugin. There we need to do a bunch of mangling to get the events stored in a DB anyway, so this isn't a huge overhead.
Given this information I think I'm going to punt on this and take your reviewed by on the code as it is.
We can certainly try to change it later but for now I think it serves our purpose. Better to focus on getting the code working with irq's.
Ira
Steven Rostedt
maintainerRe: [RFC V2 PATCH 05/11] cxl/mem: Trace General Media Event Record
In reply to [email protected]drivers/cxl/core/mbox.c int cxl_enumerate_cmds(struct cxl_dev_state *cxlds) · line 728
Quoted 6 lines · On Mon, 10 Oct 2022 15:41:25 -0700 [email protected] wrote:
drivers/cxl/core/mbox.c +6
-0+static void cxl_trace_event_record (const char *dev_name, +enum cxl_event_log_type type, +struct cxl_get_event_payload *payload) +{ +uuid_t *id = &payload->record.hdr.id; +
Perhaps you want to add here:
if (!trace_cxl_general_media_enabled() && !trace_clx_generic_event_enabled()) return;
This way the below logic is only executed if either event is enabled. The above uses static_branches, so if the architecture supports them, they are not compare and branch, but jumps and/or nops.
-- Steve
Quoted 11 lines
drivers/cxl/core/mbox.c +11
-0+if (uuid_equal (id, &gen_media_event_uuid)) { +struct cxl_event_gen_media *rec = + (struct cxl_event_gen_media *)&payload->record; + +trace_general_media (dev_name, type, rec); +return ; + } + +/* For unknown record types print just the header */ +trace_generic_event (dev_name, type, &payload->record); +}
RFC PATCH v2 6/11 cxl/mem: Trace DRAM Event Record
In reply to [email protected]From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.2 defines the DRAM Event Record.
Determine if the event read is a DRAM event record and if so trace the record.
---
Changes from RFC:
Add reserved byte data
Use new CXL header macros
Jonathan
Use get_unaligned_le{24,16}() for unaligned fields Use 'else if'
Dave Jiang
s/cxl_dram_event/dram s/cxl_evt_dram_rec/cxl_event_dram
Adjust for new phys addr mask
--- drivers/cxl/core/mbox.c | 14 ++++++ drivers/cxl/cxlmem.h | 24 ++++++++++ include/trace/events/cxl.h | 94 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 132 insertions(+)
drivers/cxl/core/mbox.c +14 -0
@@ -720,6 +720,14 @@ static const uuid_t gen_media_event_uuid =UUID_INIT (0xfbcd0a77 ,0xc260 ,0x417f ,0x85 ,0xa9 ,0x08 ,0x8b ,0x16 ,0x21 ,0xeb ,0xa6 ); +/* +* DRAM Event Record +* CXL rev 3.0 section 8.2.9.2.1.2; Table 8-44 +*/ +static const uuid_t dram_event_uuid = +UUID_INIT (0x601dcbb3 ,0x9c06 ,0x4eab , +0xb8 ,0xaf ,0x4e ,0x9b ,0xfb ,0x5c ,0x96 ,0x24 ); +static void cxl_trace_event_record (const char *dev_name,enum cxl_event_log_type type,struct cxl_get_event_payload *payload)
@@ -732,6 +740,12 @@ static void cxl_trace_event_record(const char *dev_name,trace_general_media (dev_name, type, rec);return ; + }else if (uuid_equal (id, &dram_event_uuid)) { +struct cxl_event_dram *rec = + (struct cxl_event_dram *)&payload->record; + +trace_dram (dev_name, type, rec); +return ; }/* For unknown record types print just the header */
drivers/cxl/cxlmem.h +24 -0
@@ -431,6 +431,30 @@ struct cxl_event_gen_media {u8 reserved[CXL_EVENT_GEN_MED_RES_SIZE ]; } __packed; +/* +* DRAM Event Record - DER +* CXL rev 3.0 section 8.2.9.2.1.2; Table 3-44 +*/ +#define CXL_EVENT_DER_CORRECTION_MASK_SIZE 0x20 +#define CXL_EVENT_DER_RES_SIZE 0x17 +struct cxl_event_dram { +struct cxl_event_record_hdr hdr; +__le64 phys_addr; +u8 descriptor; +u8 type; +u8 transaction_type; +u8 validity_flags[2 ]; +u8 channel; +u8 rank; +u8 nibble_mask[3 ]; +u8 bank_group; +u8 bank; +u8 row[3 ]; +u8 column[2 ]; +u8 correction_mask[CXL_EVENT_DER_CORRECTION_MASK_SIZE ]; +u8 reserved[CXL_EVENT_DER_RES_SIZE ]; +} __packed; +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;
include/trace/events/cxl.h +94 -0
100 lines not on this page · raw message
Signature · 1 line
2.37.2
Re: [RFC V2 PATCH 06/11] cxl/mem: Trace DRAM Event Record
Quoted 9 lines · On Mon, 10 Oct 2022 15:41:26 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.2 defines the DRAM Event Record.
Determine if the event read is a DRAM event record and if so trace the record.
Signed-off-by Ira Weiny <[email protected]>
Trivial comments inline
Quoted 21 lines
include/trace/events/cxl.h +12
-0@@ -230,6 +230,100 @@ TRACE_EVENT(general_media, ) ); + +TRACE_EVENT(dram, + + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_dram *rec), + + TP_ARGS(dev_name, log, rec), + + TP_STRUCT__entry( + CXL_EVT_TP_entry + /* DRAM */ + __field(u64, phys_addr)
include/trace/events/cxl.h TRACE_EVENT(general_media, · line 255
Quoted 10 lines
include/trace/events/cxl.h +10
-0+ __field(u8, descriptor) + __field(u8, type) + __field(u8, transaction_type) + __field(u8, channel) + __field(u16, validity_flags) + __field(u16, column) /* Out of order to pack trace record */ + __field(u32, nibble_mask) + __field(u32, row) + __array(u8, cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE) +__array (u8, reserved,CXL_EVENT_DER_RES_SIZE )
If we are going to have this, why not put it at the end? Will that affect the packing badly?
Quoted 33 lines
include/trace/events/cxl.h +33
-0+ __field(u8, rank) /* Out of order to pack trace record */ + __field(u8, bank_group) /* Out of order to pack trace record */ +__field (u8, bank)/* Out of order to pack trace record */ + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); + +/* DRAM */ + __entry->phys_addr =le64_to_cpu (rec->phys_addr); + __entry->descriptor = rec->descriptor; + __entry->type = rec->type; + __entry->transaction_type = rec->transaction_type; + __entry->validity_flags =get_unaligned_le16 (rec->validity_flags); + __entry->channel = rec->channel; + __entry->rank = rec->rank; + __entry->nibble_mask =get_unaligned_le24 (rec->nibble_mask); + __entry->bank_group = rec->bank_group; + __entry->bank = rec->bank; + __entry->row =get_unaligned_le24 (rec->row); + __entry->column =get_unaligned_le16 (rec->column); +memcpy (__entry->cor_mask, &rec->correction_mask, +CXL_EVENT_DER_CORRECTION_MASK_SIZE ); +memcpy (__entry->reserved, &rec->reserved, +CXL_EVENT_DER_RES_SIZE ); + ), + + CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ + "trans_type='%s' channel=%u rank=%u nibble_mask=%x " \ + "bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \ + "valid_flags='%s' reserved=%s", + __entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK, + (__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE",
include/trace/events/cxl.h TRACE_EVENT(general_media, · line 324
Quoted 10 lines
include/trace/events/cxl.h +10
-0+show_event_desc_flags (__entry->descriptor), +show_mem_event_type (__entry->type), +show_trans_type (__entry->transaction_type), + __entry->channel, __entry->rank, __entry->nibble_mask, + __entry->bank_group, __entry->bank, + __entry->row, __entry->column, +__print_hex (__entry->cor_mask,CXL_EVENT_DER_CORRECTION_MASK_SIZE ), +show_dram_valid_flags (__entry->validity_flags), +__print_hex (__entry->reserved,CXL_EVENT_DER_RES_SIZE ) + )
Probably one less tab on that trailing )?
Quoted 5 lines
include/trace/events/cxl.h +2
-0+); +#endif /* _CXL_TRACE_EVENTS_H */ /* This part must be outside protection */
Re: [RFC V2 PATCH 06/11] cxl/mem: Trace DRAM Event Record
Quoted 43 lines · Jonathan Cameron
Quoted 9 lines · On Mon, 10 Oct 2022 15:41:26 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.2 defines the DRAM Event Record.
Determine if the event read is a DRAM event record and if so trace the record.
Signed-off-by Ira Weiny <[email protected]>Trivial comments inline
Reviewed-by Jonathan Cameron <[email protected]>Quoted 26 lines
include/trace/events/cxl.h +17
-0@@ -230,6 +230,100 @@ TRACE_EVENT(general_media, ) ); + +TRACE_EVENT(dram, + + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_dram *rec), + + TP_ARGS(dev_name, log, rec), + + TP_STRUCT__entry( + CXL_EVT_TP_entry + /* DRAM */ + __field(u64, phys_addr) + __field(u8, descriptor) + __field(u8, type) + __field(u8, transaction_type) + __field(u8, channel) + __field(u16, validity_flags)
Quoted 8 lines
include/trace/events/cxl.h TRACE_EVENT(general_media, · line 281
Quoted 5 lines
include/trace/events/cxl.h +5
-0+ __field(u16, column) /* Out of order to pack trace record */ + __field(u32, nibble_mask) + __field(u32, row) + __array(u8, cor_mask, CXL_EVENT_DER_CORRECTION_MASK_SIZE) +__array (u8, reserved,CXL_EVENT_DER_RES_SIZE )If we are going to have this, why not put it at the end? Will that affect the packing badly?
I removed it.
[snip]
Quoted 10 lines
Quoted 10 lines
include/trace/events/cxl.h +10
-0+ + CXL_EVT_TP_printk("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ + "trans_type='%s' channel=%u rank=%u nibble_mask=%x " \ + "bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \ + "valid_flags='%s' reserved=%s", + __entry->phys_addr & CXL_GMER_PHYS_ADDR_MASK, + (__entry->phys_addr & CXL_GMER_PHYS_ADDR_VOLATILE) ? "TRUE" : "FALSE", + show_event_desc_flags(__entry->descriptor), + show_mem_event_type(__entry->type), + show_trans_type(__entry->transaction_type),
Quoted 8 lines
include/trace/events/cxl.h TRACE_EVENT(general_media, · line 324
Quoted 7 lines
include/trace/events/cxl.h +7
-0+ __entry->channel, __entry->rank, __entry->nibble_mask, + __entry->bank_group, __entry->bank, + __entry->row, __entry->column, +__print_hex (__entry->cor_mask,CXL_EVENT_DER_CORRECTION_MASK_SIZE ), +show_dram_valid_flags (__entry->validity_flags), +__print_hex (__entry->reserved,CXL_EVENT_DER_RES_SIZE ) + )Probably one less tab on that trailing )?
Done.
Thanks!
Ira
Steven Rostedt
maintainerRe: [RFC V2 PATCH 06/11] cxl/mem: Trace DRAM Event Record
In reply to [email protected]Quoted 1 line · On Mon, 10 Oct 2022 15:41:26 -0700 [email protected] wrote:
+TRACE_EVENT(dram,
Call this "cxl_dram"
-- Steve
Quoted 65 lines
include/trace/events/cxl.h +65
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_dram *rec), + +TP_ARGS (dev_name, log, rec), + +TP_STRUCT__entry ( + CXL_EVT_TP_entry +/* DRAM */ +__field (u64 ,phys_addr ) +__field (u8, descriptor) +__field (u8, type) +__field (u8, transaction_type) +__field (u8, channel) +__field (u16, validity_flags) +__field (u16, column)/* Out of order to pack trace record */ +__field (u32, nibble_mask) +__field (u32, row) +__array (u8, cor_mask,CXL_EVENT_DER_CORRECTION_MASK_SIZE ) +__array (u8, reserved,CXL_EVENT_DER_RES_SIZE ) +__field (u8, rank)/* Out of order to pack trace record */ +__field (u8, bank_group)/* Out of order to pack trace record */ +__field (u8, bank)/* Out of order to pack trace record */ + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); + +/* DRAM */ + __entry->phys_addr =le64_to_cpu (rec->phys_addr); + __entry->descriptor = rec->descriptor; + __entry->type = rec->type; + __entry->transaction_type = rec->transaction_type; + __entry->validity_flags =get_unaligned_le16 (rec->validity_flags); + __entry->channel = rec->channel; + __entry->rank = rec->rank; + __entry->nibble_mask =get_unaligned_le24 (rec->nibble_mask); + __entry->bank_group = rec->bank_group; + __entry->bank = rec->bank; + __entry->row =get_unaligned_le24 (rec->row); + __entry->column =get_unaligned_le16 (rec->column); +memcpy (__entry->cor_mask, &rec->correction_mask, +CXL_EVENT_DER_CORRECTION_MASK_SIZE ); +memcpy (__entry->reserved, &rec->reserved, +CXL_EVENT_DER_RES_SIZE ); + ), + +CXL_EVT_TP_printk ("phys_addr=%llx volatile=%s desc='%s' type='%s' " \ +"trans_type='%s' channel=%u rank=%u nibble_mask=%x " \ +"bank_group=%u bank=%u row=%u column=%u cor_mask=%s " \ +"valid_flags='%s' reserved=%s" , + __entry->phys_addr &CXL_GMER_PHYS_ADDR_MASK , + (__entry->phys_addr &CXL_GMER_PHYS_ADDR_VOLATILE ) ?"TRUE" :"FALSE" , +show_event_desc_flags (__entry->descriptor), +show_mem_event_type (__entry->type), +show_trans_type (__entry->transaction_type), + __entry->channel, __entry->rank, __entry->nibble_mask, + __entry->bank_group, __entry->bank, + __entry->row, __entry->column, +__print_hex (__entry->cor_mask,CXL_EVENT_DER_CORRECTION_MASK_SIZE ), +show_dram_valid_flags (__entry->validity_flags), +__print_hex (__entry->reserved,CXL_EVENT_DER_RES_SIZE ) + ) +); +
RFC PATCH v2 7/11 cxl/mem: Trace Memory Module Event Record
In reply to [email protected]From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.3 defines the Memory Module Event Record.
Determine if the event read is memory module record and if so trace the record.
---
Changes from RFC:
Clean up spec reference
Add reserved data
Use new CXL header macros
Jonathan
Use else if
Use get_unaligned_le*() for unaligned fields
Dave Jiang
s/cxl_mem_mod_event/memory_module s/cxl_evt_mem_mod_rec/cxl_event_mem_module
--- drivers/cxl/core/mbox.c | 14 ++++ drivers/cxl/cxlmem.h | 27 +++++++ include/trace/events/cxl.h | 146 +++++++++++++++++++++++++++++++++++++ 3 files changed, 187 insertions(+)
drivers/cxl/core/mbox.c +14 -0
@@ -728,6 +728,14 @@ static const uuid_t dram_event_uuid =UUID_INIT (0x601dcbb3 ,0x9c06 ,0x4eab ,0xb8 ,0xaf ,0x4e ,0x9b ,0xfb ,0x5c ,0x96 ,0x24 ); +/* +* Memory Module Event Record +* CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45 +*/ +static const uuid_t mem_mod_event_uuid = +UUID_INIT (0xfe927475 ,0xdd59 ,0x4339 , +0xa5 ,0x86 ,0x79 ,0xba ,0xb1 ,0x13 ,0xb7 ,0x74 ); +static void cxl_trace_event_record (const char *dev_name,enum cxl_event_log_type type,struct cxl_get_event_payload *payload)
@@ -746,6 +754,12 @@ static void cxl_trace_event_record(const char *dev_name,trace_dram (dev_name, type, rec);return ; + }else if (uuid_equal (id, &mem_mod_event_uuid)) { +struct cxl_event_mem_module *rec = + (struct cxl_event_mem_module *)&payload->record; + +trace_memory_module (dev_name, type, rec); +return ; }/* For unknown record types print just the header */
drivers/cxl/cxlmem.h +27 -0
@@ -455,6 +455,33 @@ struct cxl_event_dram {u8 reserved[CXL_EVENT_DER_RES_SIZE ]; } __packed; +/* +* Get Health Info Record +* CXL rev 3.0 section 8.2.9.8.3.1; Table 8-100 +*/ +struct cxl_get_health_info { +u8 health_status; +u8 media_status; +u8 add_status; +u8 life_used; +u8 device_temp[2 ]; +u8 dirty_shutdown_cnt[4 ]; +u8 cor_vol_err_cnt[4 ]; +u8 cor_per_err_cnt[4 ]; +} __packed; + +/* +* Memory Module Event Record +* CXL rev 3.0 section 8.2.9.2.1.3; Table 8-45 +*/ +#define CXL_EVENT_MEM_MOD_RES_SIZE 0x3d +struct cxl_event_mem_module { +struct cxl_event_record_hdr hdr; +u8 event_type; +struct cxl_get_health_info info; +u8 reserved[CXL_EVENT_MEM_MOD_RES_SIZE ]; +} __packed; +struct cxl_mbox_get_partition_info {__le64 active_volatile_cap;__le64 active_persistent_cap;
include/trace/events/cxl.h +146 -0
152 lines not on this page · raw message
Signature · 1 line
2.37.2
Re: [RFC V2 PATCH 07/11] cxl/mem: Trace Memory Module Event Record
Quoted 9 lines · On Mon, 10 Oct 2022 15:41:27 -0700 [email protected] wrote:
From: Ira Weiny <[email protected]>
CXL rev 3.0 section 8.2.9.2.1.3 defines the Memory Module Event Record.
Determine if the event read is memory module record and if so trace the record.
Signed-off-by Ira Weiny <[email protected]>
A few trivial comments inline. I'm happy either way
Quoted 16 lines
include/trace/events/cxl.h +14
-0+#define show_add_status (as) __print_symbolic(as, \ + { CXL_DHI_AS_NORMAL, "Normal" }, \ + { CXL_DHI_AS_WARNING, "Warning" }, \ + { CXL_DHI_AS_CRITICAL, "Critical" } \ +) + +#define CXL_DHI_AS_LIFE_USED (as) (as & 0x3) +#define CXL_DHI_AS_DEV_TEMP (as) ((as & 0xC) >> 2) +#define CXL_DHI_AS_COR_VOL_ERR_CNT (as) ((as & 0x10) >> 4) +#define CXL_DHI_AS_COR_PER_ERR_CNT (as) ((as & 0x20) >> 5) + ), + +CXL_EVT_TP_printk ("evt_type='%s' health_status='%s' media_status='%s' " \ + "as_life_used=%s as_dev_temp=%s as_cor_vol_err_cnt=%s " \
include/trace/events/cxl.h TRACE_EVENT(dram, · line 464
Quoted 10 lines
include/trace/events/cxl.h +10
-0+"as_cor_per_err_cnt=%s life_used=%u dev_temp=%d " \ +"dirty_shutdown_cnt=%u cor_vol_err_cnt=%u cor_per_err_cnt=%u " \ +"reserved=%s" , +show_dev_evt_type (__entry->event_type), +show_health_status_flags (__entry->health_status), +show_media_status (__entry->media_status), +show_add_status (CXL_DHI_AS_LIFE_USED (__entry->add_status)), +show_add_status (CXL_DHI_AS_DEV_TEMP (__entry->add_status)), +show_add_status (CXL_DHI_AS_COR_VOL_ERR_CNT (__entry->add_status)), + show_add_status(CXL_DHI_AS_COR_PER_ERR_CNT(__entry->add_status)),
A little nasty to use same show_add_status() for the 2 bit and 1 bit versions. Obviously it works, but maybe it's worth two macros for the 1 and 2 bit version?
include/trace/events/cxl.h TRACE_EVENT(dram, · line 469
Quoted 5 lines
include/trace/events/cxl.h +5
-0+ __entry->life_used, __entry->device_temp, + __entry->dirty_shutdown_cnt, __entry->cor_vol_err_cnt, + __entry->cor_per_err_cnt, +__print_hex (__entry->reserved,CXL_EVENT_MEM_MOD_RES_SIZE ) + )
Aligned one tab too far?
Steven Rostedt
maintainerRe: [RFC V2 PATCH 07/11] cxl/mem: Trace Memory Module Event Record
In reply to [email protected]Quoted 1 line · On Mon, 10 Oct 2022 15:41:27 -0700 [email protected] wrote:
+TRACE_EVENT(memory_module,
Make sure all your new events have the "cxl_" prefix. "cxl_memory_module".
This goes for all events in this series.
Thanks,
-- Steve
Quoted 64 lines
include/trace/events/cxl.h +64
-0+ + TP_PROTO(const char *dev_name, enum cxl_event_log_type log, + struct cxl_event_mem_module *rec), + +TP_ARGS (dev_name, log, rec), + +TP_STRUCT__entry ( + CXL_EVT_TP_entry + +/* Memory Module Event */ +__field (u8 ,event_type ) + +/* Device Health Info */ +__field (u8, health_status) +__field (u8, media_status) +__field (u8, life_used) +__field (u32, dirty_shutdown_cnt) +__field (u32, cor_vol_err_cnt) +__field (u32, cor_per_err_cnt) +__field (s16, device_temp) +__field (u8, add_status) + +__array (u8, reserved,CXL_EVENT_MEM_MOD_RES_SIZE ) + ), + +TP_fast_assign ( + CXL_EVT_TP_fast_assign(dev_name, log, rec->hdr); + +/* Memory Module Event */ + __entry->event_type = rec->event_type; + +/* Device Health Info */ + __entry->health_status = rec->info.health_status; + __entry->media_status = rec->info.media_status; + __entry->life_used = rec->info.life_used; + __entry->dirty_shutdown_cnt =get_unaligned_le32 (rec->info.dirty_shutdown_cnt); + __entry->cor_vol_err_cnt =get_unaligned_le32 (rec->info.cor_vol_err_cnt); + __entry->cor_per_err_cnt =get_unaligned_le32 (rec->info.cor_per_err_cnt); + __entry->device_temp =get_unaligned_le16 (rec->info.device_temp); + __entry->add_status = rec->info.add_status; +memcpy (__entry->reserved, &rec->reserved, +CXL_EVENT_MEM_MOD_RES_SIZE ); + ), + +CXL_EVT_TP_printk ("evt_type='%s' health_status='%s' media_status='%s' " \ +"as_life_used=%s as_dev_temp=%s as_cor_vol_err_cnt=%s " \ +"as_cor_per_err_cnt=%s life_used=%u dev_temp=%d " \ +"dirty_shutdown_cnt=%u cor_vol_err_cnt=%u cor_per_err_cnt=%u " \ +"reserved=%s" , +show_dev_evt_type (__entry->event_type), +show_health_status_flags (__entry->health_status), +show_media_status (__entry->media_status), +show_add_status (CXL_DHI_AS_LIFE_USED (__entry->add_status)), +show_add_status (CXL_DHI_AS_DEV_TEMP (__entry->add_status)), +show_add_status (CXL_DHI_AS_COR_VOL_ERR_CNT (__entry->add_status)), +show_add_status (CXL_DHI_AS_COR_PER_ERR_CNT (__entry->add_status)), + __entry->life_used, __entry->device_temp, + __entry->dirty_shutdown_cnt, __entry->cor_vol_err_cnt, + __entry->cor_per_err_cnt, +__print_hex (__entry->reserved,CXL_EVENT_MEM_MOD_RES_SIZE ) + ) +); + +
RFC PATCH v2 8/11 cxl/test: Add generic mock events
In reply to [email protected]From: Ira Weiny <[email protected]>
Facilitate testing basic Get/Clear Event functionality by creating multiple logs and generic events with made up UUID's.
Data is completely made up with data patterns which should be easy to spot in trace output.
A single sysfs entry resets the event data and triggers collecting the events for testing.
Test traces are easy to obtain with a small script such as this:
#!/bin/bash -x
devices=find /sys/devices/platform -name cxl_mem*
# Turn on tracing
echo "" > /sys/kernel/tracing/trace echo 1 > /sys/kernel/tracing/events/cxl/enable echo 1 > /sys/kernel/tracing/tracing_on
# Generate fake interrupt
for device in $devices; do
echo 1 > $device/event_trigger
done
# Turn off tracing and report events
echo 0 > /sys/kernel/tracing/tracing_on
cat /sys/kernel/tracing/trace
---
Changes from RFC:
Separate out the event code
Adjust for struct changes.
Clean up devm_cxl_mock_event_logs()
Clean up naming and comments
Jonathan
Remove dynamic allocation of event logs Clean up comment
Remove unneeded xarray
Ensure event_trigger sysfs is valid prior to the driver going active.
Dan
Remove the fill/reset event sysfs as these operations can be done together
--- tools/testing/cxl/test/Kbuild | 2 +- tools/testing/cxl/test/events.c | 223 ++++++++++++++++++++++++++++++++ tools/testing/cxl/test/events.h | 9 ++ tools/testing/cxl/test/mem.c | 32 +++++ 4 files changed, 265 insertions(+), 1 deletion(-) create mode 100644 tools/testing/cxl/test/events.c create mode 100644 tools/testing/cxl/test/events.h
tools/testing/cxl/test/Kbuild +1 -1
@@ -7,4 +7,4 @@ obj-m += cxl_mock_mem.o cxl_test-y := cxl.o cxl_mock-y := mock.o -cxl_mock_mem-y := mem.o +cxl_mock_mem-y := mem.o events.o
tools/testing/cxl/test/events.c new file +223 -0
223 lines not on this page · raw message
tools/testing/cxl/test/events.h new file +9 -0
@@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ + +#include <cxlmem.h> + +int mock_get_event (struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +int mock_clear_event (struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd); +void cxl_mock_add_event_logs (struct cxl_dev_state *cxlds); +void cxl_mock_remove_event_logs (struct device *dev); +void cxl_mock_event_trigger (struct device *dev);
tools/testing/cxl/test/mem.c +32 -0
@@ -8,6 +8,7 @@#include <linux/sizes.h> #include <linux/bits.h> #include <cxlmem.h> +#include "events.h" #define LSA_SIZE SZ_128K#define DEV_SIZE SZ_2G
@@ -224,6 +225,12 @@ static int cxl_mock_mbox_send(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *case CXL_MBOX_OP_GET_PARTITION_INFO : rc =mock_partition_info (cxlds, cmd);break ; +case CXL_MBOX_OP_GET_EVENT_RECORD : + rc =mock_get_event (cxlds, cmd); +break ; +case CXL_MBOX_OP_CLEAR_EVENT_RECORD : + rc =mock_clear_event (cxlds, cmd); +break ;case CXL_MBOX_OP_SET_LSA : rc =mock_set_lsa (cxlds, cmd);break ;
@@ -245,6 +252,21 @@ static void label_area_release(void *lsa)vfree (lsa); } +static ssize_t event_trigger_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +cxl_mock_event_trigger (dev); +return count; +} +static DEVICE_ATTR_WO (event_trigger); + +static struct attribute *cxl_mock_event_attrs[] = { + &dev_attr_event_trigger.attr, +NULL +}; +ATTRIBUTE_GROUPS (cxl_mock_event); +static int cxl_mock_mem_probe (struct platform_device *pdev) {struct device *dev = &pdev->dev;
@@ -281,6 +303,8 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)if (rc)return rc; +cxl_mock_add_event_logs (cxlds); + cxlmd =devm_cxl_add_memdev (cxlds);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
@@ -293,6 +317,12 @@ static int cxl_mock_mem_probe(struct platform_device *pdev)return 0 ; } +static int cxl_mock_mem_remove (struct platform_device *pdev) +{ +cxl_mock_remove_event_logs (&pdev->dev); +return 0 ; +} +static const struct platform_device_id cxl_mock_mem_ids[] = { { .name ="cxl_mem" , }, { },
@@ -301,9 +331,11 @@ MODULE_DEVICE_TABLE(platform, cxl_mock_mem_ids);static struct platform_driver cxl_mock_mem_driver = { .probe = cxl_mock_mem_probe, + .remove = cxl_mock_mem_remove, .id_table = cxl_mock_mem_ids, .driver = { .name =KBUILD_MODNAME , + .dev_groups = cxl_mock_event_groups, }, };
Signature · 1 line
2.37.2
RFC PATCH v2 9/11 cxl/test: Add specific events
In reply to [email protected]From: Ira Weiny <[email protected]>
Each type of event has different trace point outputs.
Add mock General Media Event, DRAM event, and Memory Module Event records to the mock list of events returned.
---
Changes from RFC:
Adjust for struct changes
adjust for unaligned fields
--- tools/testing/cxl/test/events.c | 70 +++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+)
tools/testing/cxl/test/events.c +70 -0
@@ -189,6 +189,70 @@ struct cxl_event_record_raw hardware_replace = { .data = { 0xDE, 0xAD, 0xBE, 0xEF }, }; +struct cxl_event_gen_media gen_media = { + .hdr = { + .id =UUID_INIT (0xfbcd0a77 ,0xc260 ,0x417f , +0x85 ,0xa9 ,0x08 ,0x8b ,0x16 ,0x21 ,0xeb ,0xa6 ), + .length =sizeof (struct cxl_event_gen_media ), + .flags[0 ] =CXL_EVENT_RECORD_FLAG_PERMANENT , +/* .handle = Set dynamically */ + .related_handle =cpu_to_le16 (0 ), + }, + .phys_addr =cpu_to_le64 (0x2000 ), + .descriptor =CXL_GMER_EVT_DESC_UNCORECTABLE_EVENT , + .type =CXL_GMER_MEM_EVT_TYPE_DATA_PATH_ERROR , + .transaction_type =CXL_GMER_TRANS_HOST_WRITE , + .validity_flags = {CXL_GMER_VALID_CHANNEL | +CXL_GMER_VALID_RANK ,0 }, + .channel =1 , + .rank =30 +}; + +struct cxl_event_dram dram = { + .hdr = { + .id =UUID_INIT (0x601dcbb3 ,0x9c06 ,0x4eab , +0xb8 ,0xaf ,0x4e ,0x9b ,0xfb ,0x5c ,0x96 ,0x24 ), + .length =sizeof (struct cxl_event_dram ), + .flags[0 ] =CXL_EVENT_RECORD_FLAG_PERF_DEGRADED , +/* .handle = Set dynamically */ + .related_handle =cpu_to_le16 (0 ), + }, + .phys_addr =cpu_to_le64 (0x8000 ), + .descriptor =CXL_GMER_EVT_DESC_THRESHOLD_EVENT , + .type =CXL_GMER_MEM_EVT_TYPE_INV_ADDR , + .transaction_type =CXL_GMER_TRANS_INTERNAL_MEDIA_SCRUB , + .validity_flags = {CXL_DER_VALID_CHANNEL | +CXL_DER_VALID_BANK_GROUP | +CXL_DER_VALID_BANK | +CXL_DER_VALID_COLUMN ,0 }, + .channel =1 , + .bank_group =5 , + .bank =2 , + .column = {0xDE ,0xAD }, +}; + +struct cxl_event_mem_module mem_module = { + .hdr = { + .id =UUID_INIT (0xfe927475 ,0xdd59 ,0x4339 , +0xa5 ,0x86 ,0x79 ,0xba ,0xb1 ,0x13 ,0xb7 ,0x74 ), + .length =sizeof (struct cxl_event_mem_module ), +/* .handle = Set dynamically */ + .related_handle =cpu_to_le16 (0 ), + }, + .event_type =CXL_MMER_TEMP_CHANGE , + .info = { + .health_status =CXL_DHI_HS_PERFORMANCE_DEGRADED , + .media_status =CXL_DHI_MS_ALL_DATA_LOST , + .add_status = (CXL_DHI_AS_CRITICAL <<2 ) | + (CXL_DHI_AS_WARNING <<4 ) | + (CXL_DHI_AS_WARNING <<5 ), + .device_temp = {0xDE ,0xAD }, + .dirty_shutdown_cnt = {0xde ,0xad ,0xbe ,0xef }, + .cor_vol_err_cnt = {0xde ,0xad ,0xbe ,0xef }, + .cor_per_err_cnt = {0xde ,0xad ,0xbe ,0xef }, + } +}; +void cxl_mock_add_event_logs (struct cxl_dev_state *cxlds) {struct device *dev = cxlds->dev;
@@ -207,8 +271,14 @@ void cxl_mock_add_event_logs(struct cxl_dev_state *cxlds) }event_store_add_event (mes,CXL_EVENT_TYPE_INFO , &maint_needed); +event_store_add_event (mes,CXL_EVENT_TYPE_INFO , + (struct cxl_event_record_raw *)&gen_media); +event_store_add_event (mes,CXL_EVENT_TYPE_INFO , + (struct cxl_event_record_raw *)&mem_module);event_store_add_event (mes,CXL_EVENT_TYPE_FATAL , &hardware_replace); +event_store_add_event (mes,CXL_EVENT_TYPE_FATAL , + (struct cxl_event_record_raw *)&dram); }EXPORT_SYMBOL_GPL (cxl_mock_add_event_logs);
Signature · 1 line
2.37.2
RFC PATCH v2 10/11 cxl/test: Simulate event log overflow
In reply to [email protected]From: Ira Weiny <[email protected]>
Log overflow is marked by a separate trace message.
Simulate a log with lots of messages and flag overflow until it is drained a bit.
---
Changes from RFC
Adjust for new struct changes
--- tools/testing/cxl/test/events.c | 36 +++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+)
tools/testing/cxl/test/events.c +36 -0
@@ -68,11 +68,21 @@ static void event_store_add_event(struct mock_event_store *mes, log->nr_events++; } +static u16 log_overflow (struct mock_event_log *log) +{ +int cnt =log_rec_left (log) -5 ; + +if (cnt <0 ) +return 0 ; +return cnt; +} +int mock_get_event (struct cxl_dev_state *cxlds,struct cxl_mbox_cmd *cmd) {struct cxl_get_event_payload *pl;struct mock_event_log *log;u8 log_type; +u16 nr_overflow;/* Valid request? */ if (cmd->size_in !=sizeof (log_type))
@@ -98,6 +108,20 @@ int mock_get_event(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd)if (log_rec_left (log) >1 ) pl->flags |=CXL_GET_EVENT_FLAG_MORE_RECORDS ; + nr_overflow =log_overflow (log); +if (nr_overflow) { +u64 ns; + + pl->flags |=CXL_GET_EVENT_FLAG_OVERFLOW ; + pl->overflow_err_count =cpu_to_le16 (nr_overflow); + ns =ktime_get_real_ns (); + ns -=5000000000 ;/* 5s ago */ + pl->first_overflow_timestamp =cpu_to_le64 (ns); + ns =ktime_get_real_ns (); + ns -=1000000000 ;/* 1s ago */ + pl->last_overflow_timestamp =cpu_to_le64 (ns); + } +memcpy (&pl->record,get_cur_event (log),sizeof (pl->record)); pl->record.hdr.handle =get_cur_event_handle (log);return 0 ;
@@ -276,6 +300,18 @@ void cxl_mock_add_event_logs(struct cxl_dev_state *cxlds)event_store_add_event (mes,CXL_EVENT_TYPE_INFO , (struct cxl_event_record_raw *)&mem_module); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , &maint_needed); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , &hardware_replace); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , + (struct cxl_event_record_raw *)&dram); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , + (struct cxl_event_record_raw *)&gen_media); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , + (struct cxl_event_record_raw *)&mem_module); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , &hardware_replace); +event_store_add_event (mes,CXL_EVENT_TYPE_FAIL , + (struct cxl_event_record_raw *)&dram); +event_store_add_event (mes,CXL_EVENT_TYPE_FATAL , &hardware_replace);event_store_add_event (mes,CXL_EVENT_TYPE_FATAL , (struct cxl_event_record_raw *)&dram);
Signature · 1 line
2.37.2
RFC PATCH v2 11/11 cxl/mem: Wire up event interrupts
In reply to [email protected]From: Ira Weiny <[email protected]>
CXL device events are signaled via interrupts. Each event log may have a different interrupt message number. These message numbers are reported in the Get Event Interrupt Policy mailbox command.
Create an infrastructure to query the max vectors required for the CXL device. Add event interrupt information that infrastructure. Set up a handler for each event log.
Davidlohr suggested the generic vector code.
--- drivers/cxl/core/mbox.c | 53 +++++++++++- drivers/cxl/cxlmem.h | 32 ++++++++ drivers/cxl/pci.c | 152 +++++++++++++++++++++++++++++++++++ include/uapi/linux/cxl_mem.h | 2 + 4 files changed, 237 insertions(+), 2 deletions(-)
drivers/cxl/core/mbox.c +51 -2
@@ -53,6 +53,8 @@ static struct cxl_mem_command cxl_mem_commands[CXL_MEM_COMMAND_ID_MAX] = {CXL_CMD (GET_SUPPORTED_LOGS ,0 ,CXL_VARIABLE_PAYLOAD ,CXL_CMD_FLAG_FORCE_ENABLE ),CXL_CMD (GET_EVENT_RECORD ,1 ,CXL_VARIABLE_PAYLOAD ,0 ),CXL_CMD (CLEAR_EVENT_RECORD ,CXL_VARIABLE_PAYLOAD ,0 ,0 ), +CXL_CMD (GET_EVT_INT_POLICY ,0 ,0x5 ,0 ), +CXL_CMD (SET_EVT_INT_POLICY ,0x5 ,0 ,0 ),CXL_CMD (GET_FW_INFO ,0 ,0x50 ,0 ),CXL_CMD (GET_PARTITION_INFO ,0 ,0x20 ,0 ), CXL_CMD(GET_LSA, 0x8, CXL_VARIABLE_PAYLOAD, 0),
@@ -780,8 +782,8 @@ static int cxl_clear_event_record(struct cxl_dev_state *cxlds, &payload, sizeof(payload), NULL, 0); } -static void cxl_mem_get_records_log (struct cxl_dev_state *cxlds, -enum cxl_event_log_type type) +void cxl_mem_get_records_log (struct cxl_dev_state *cxlds, +enum cxl_event_log_type type) {struct cxl_get_event_payload payload;
@@ -816,6 +818,7 @@ static void cxl_mem_get_records_log(struct cxl_dev_state *cxlds, }while (payload.flags &CXL_GET_EVENT_FLAG_MORE_RECORDS ); } +EXPORT_SYMBOL_NS_GPL (cxl_mem_get_records_log,CXL );/** * cxl_mem_get_event_records - Get Event Records from the device
@@ -839,6 +842,52 @@ void cxl_mem_get_event_records(struct cxl_dev_state *cxlds) }EXPORT_SYMBOL_NS_GPL (cxl_mem_get_event_records,CXL ); +static int cxl_event_msgnum (u8 setting) +{ +if (!cxl_evt_int_is_msi (setting)) +return -1 ; + +return CXL_EVENT_INT_MSGNUM (setting); +} + +int cxl_event_get_max_msgnum (struct cxl_dev_state *cxlds) +{ +struct cxl_event_interrupt_policy *policy = &cxlds->evt_int_policy; +int rc; + + policy->info_settings =CXL_INT_MSI_MSIX ; + policy->warn_settings =CXL_INT_MSI_MSIX ; + policy->failure_settings =CXL_INT_MSI_MSIX ; + policy->fatal_settings =CXL_INT_MSI_MSIX ; + policy->dyn_cap_settings =CXL_INT_MSI_MSIX ; + + rc =cxl_mbox_send_cmd (cxlds,CXL_MBOX_OP_SET_EVT_INT_POLICY , + policy,sizeof (*policy), +NULL ,0 ); +if (rc) { +dev_err (cxlds->dev,"Failed to set event interrupt policy : %d" , + rc); +memset (policy,CXL_INT_NONE ,sizeof (*policy)); +return -1 ; + } + + rc =cxl_mbox_send_cmd (cxlds,CXL_MBOX_OP_GET_EVT_INT_POLICY ,NULL ,0 , + policy,sizeof (*policy)); +if (rc) { +dev_err (cxlds->dev,"Failed to get event interrupt policy : %d" , + rc); +return -1 ; + } + + rc =max_t (int, rc,cxl_event_msgnum (policy->info_settings)); + rc =max_t (int, rc,cxl_event_msgnum (policy->warn_settings)); + rc =max_t (int, rc,cxl_event_msgnum (policy->failure_settings)); + rc =max_t (int, rc,cxl_event_msgnum (policy->fatal_settings)); + +return rc; +} +EXPORT_SYMBOL_NS_GPL (cxl_event_get_max_msgnum,CXL ); +/** * cxl_mem_get_partition_info - Get partition info * @cxlds: The device data for the operation
drivers/cxl/cxlmem.h +32 -0
@@ -179,6 +179,32 @@ struct cxl_endpoint_dvsec_info {struct range dvsec_range[2 ]; }; +/** +* Event Interrupt Policy +* +* CXL rev 3.0 section 8.2.9.2.4; Table 8-52 +*/ +enum cxl_event_int_mode { +CXL_INT_NONE =0x00 , +CXL_INT_MSI_MSIX =0x01 , +CXL_INT_FW =0x02 , +CXL_INT_RES =0x03 , +}; +#define CXL_EVENT_INT_MODE_MASK 0x3 +#define CXL_EVENT_INT_MSGNUM (setting) (((setting) & 0xf0) >> 4) +struct cxl_event_interrupt_policy { +u8 info_settings; +u8 warn_settings; +u8 failure_settings; +u8 fatal_settings; +u8 dyn_cap_settings; +} __packed; + +static inline bool cxl_evt_int_is_msi (u8 setting) +{ +return CXL_INT_MSI_MSIX == (setting &CXL_EVENT_INT_MODE_MASK ); +} +/** * struct cxl_dev_state - The driver device state *
@@ -245,6 +271,7 @@ struct cxl_dev_state {resource_size_t component_reg_phys;u64 serial; +struct cxl_event_interrupt_policy evt_int_policy;struct xarray doe_mbs;
@@ -256,6 +283,8 @@ enum cxl_opcode {CXL_MBOX_OP_RAW =CXL_MBOX_OP_INVALID ,CXL_MBOX_OP_GET_EVENT_RECORD =0x0100 ,CXL_MBOX_OP_CLEAR_EVENT_RECORD =0x0101 , +CXL_MBOX_OP_GET_EVT_INT_POLICY =0x0102 , +CXL_MBOX_OP_SET_EVT_INT_POLICY =0x0103 ,CXL_MBOX_OP_GET_FW_INFO =0x0200 ,CXL_MBOX_OP_ACTIVATE_FW =0x0202 , CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
@@ -541,7 +570,10 @@ int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);struct cxl_dev_state *cxl_dev_state_create (struct device *dev);void set_exclusive_cxl_commands (struct cxl_dev_state *cxlds,unsigned long *cmds);void clear_exclusive_cxl_commands (struct cxl_dev_state *cxlds,unsigned long *cmds); +void cxl_mem_get_records_log (struct cxl_dev_state *cxlds, +enum cxl_event_log_type type);void cxl_mem_get_event_records (struct cxl_dev_state *cxlds); +int cxl_event_get_max_msgnum (struct cxl_dev_state *cxlds);#ifdef CONFIG_CXL_SUSPEND void cxl_mem_active_inc (void );void cxl_mem_active_dec (void );
drivers/cxl/pci.c +152 -0
164 lines not on this page · raw message
include/uapi/linux/cxl_mem.h +2 -0
@@ -26,6 +26,8 @@___C (GET_SUPPORTED_LOGS ,"Get Supported Logs" ), \___C (GET_EVENT_RECORD ,"Get Event Record" ), \___C (CLEAR_EVENT_RECORD ,"Clear Event Record" ), \ +___C (GET_EVT_INT_POLICY ,"Get Event Interrupt Policy" ), \ +___C (SET_EVT_INT_POLICY ,"Set Event Interrupt Policy" ), \___C (GET_FW_INFO ,"Get FW Info" ), \___C (GET_PARTITION_INFO ,"Get Partition Information" ), \ ___C(GET_LSA, "Get Label Storage Area"), \
Signature · 1 line
2.37.2
Davidlohr Bueso
maintainerRe: [RFC V2 PATCH 11/11] cxl/mem: Wire up event interrupts
Quoted 11 lines · On Mon, 10 Oct 2022, [email protected] wrote:
From: Ira Weiny <[email protected]>
CXL device events are signaled via interrupts. Each event log may have a different interrupt message number. These message numbers are reported in the Get Event Interrupt Policy mailbox command.
Create an infrastructure to query the max vectors required for the CXL device. Add event interrupt information that infrastructure. Set up a handler for each event log.
Davidlohr suggested the generic vector code.
So this should be a separate patch, and out of the event series altogether. There are a number of interested parties for irq support, and imo the generic vector stuff should be a first dependency to all of them.
I've sent out a patch with some updates you did. I also kept the whole table populated with nil callbacks as I believe this documents the TODO nicely.
Thanks,
Davidlohr