Skip to contentSkip to conversation

Add DPA->HPA translation to dram & general_media events

PATCH Superseded [email protected] 2 participants 6 messages Apr 26, 2024 – Apr 29, 2024 CXL

v4 of 6 Superseded Latest is v6 →

lore
Share & tools
lore

[email protected]

author maintainer
#0 lore raw

v4 cover Add DPA->HPA translation to dram & general_media events

From: Alison Schofield <[email protected]>

Changes in v4:
- Pick up Dan's style suggestions I missed in v3 (Dan)
- Lookup hpa earlier (within trace_enabled()) and pass to the event handler (Dan)
- Add and use a new member in cxl_event union to access the common phys_addr field. Link to v3: https://lore.kernel.org/[email protected]/

Changes in v3:
- Harvest the cxlr before calling trace handler (Dan)
- Replace macros w inline fast assigns (made easier w available cxlr)
- Drop prior review (Jonathan) on Patch 3 due to churn.
- Remove Patch 4 which was making cxl_poison use the (now gone) macros. (A uuid_t change was tucked in Patch 4 and can be posted as a separate tidy-up patch.)
Link to v2: https://lore.kernel.org/[email protected]/

An update to the cxl_events unit test is in review here:
https://lore.kernel.org/[email protected]/

Begin Cover Letter:
Add HPA translations to CXL events: cxl_dram and cxl_general_media

Patches 1 & 2:
Before adding the new support, do some housekeeping and move related helpers to the region driver because there is no looking up region related info without CONFIG_CXL_REGION.

Patch 3:
The new functionality is introduced - cxl_dram & cxl_general_media events. Lookup and log the DPA->HPA translation along with the region name and region uuid.

Alison Schofield (3):
cxl/region: Move cxl_dpa_to_region() work to the region driver cxl/region: Move cxl_trace_hpa() work to the region driver cxl/core: Add region info to cxl_general_media and cxl_dram events

 drivers/cxl/core/core.h   |  14 ++++
 drivers/cxl/core/mbox.c   |  36 ++++++++--
 drivers/cxl/core/memdev.c |  44 -------------
 drivers/cxl/core/region.c | 135 ++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/trace.c  |  91 -------------------------
 drivers/cxl/core/trace.h  |  46 +++++++++----
 include/linux/cxl-event.h |  10 +++
 7 files changed, 224 insertions(+), 152 deletions(-)

base-commit: 4cece764965020c22cff7665b18a012006359095

Signature · 1 line
2.37.3
author maintainer
#1 lore raw

PATCH v4 1/3 cxl/region: Move cxl_dpa_to_region() work to the region driver

From: Alison Schofield <[email protected]>

This helper belongs in the region driver as it is only useful with CONFIG_CXL_REGION. Add a stub in core.h for when the region driver is not built.

---
 drivers/cxl/core/core.h   |  7 +++++++
 drivers/cxl/core/memdev.c | 44 ---------------------------------------
 drivers/cxl/core/region.c | 44 +++++++++++++++++++++++++++++++++++++++
 3 files changed, 51 insertions(+), 44 deletions(-)
drivers/cxl/core/core.h +7 -0
index bc5a95665aa0..87008505f8a9 100644
@@ -27,7 +27,14 @@ void cxl_decoder_kill_region(struct cxl_endpoint_decoder *cxled);
 int cxl_region_init(void);
 void cxl_region_exit(void);
 int cxl_get_poison_by_endpoint(struct cxl_port *port);
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa);
+
 #else
+static inline
+struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
+{
+	return NULL;
+}
 static inline int cxl_get_poison_by_endpoint(struct cxl_port *port)
 {
 	return 0;
drivers/cxl/core/memdev.c +0 -44

50 lines not on this page · raw message

drivers/cxl/core/region.c +44 -0

50 lines not on this page · raw message

Signature · 1 line
2.37.3
author maintainer
#2 lore raw

PATCH v4 2/3 cxl/region: Move cxl_trace_hpa() work to the region driver

In reply to [email protected]

From: Alison Schofield <[email protected]>

This work belongs in the region driver as it is only useful with CONFIG_CXL_REGION. Add a stub in core.h for when the region driver is not built.

---
 drivers/cxl/core/core.h   |  7 +++
 drivers/cxl/core/region.c | 91 +++++++++++++++++++++++++++++++++++++++
 drivers/cxl/core/trace.c  | 91 ---------------------------------------
 drivers/cxl/core/trace.h  |  2 -
 4 files changed, 98 insertions(+), 93 deletions(-)
drivers/cxl/core/core.h +7 -0
index 87008505f8a9..625394486459 100644
@@ -28,8 +28,15 @@ int cxl_region_init(void);
 void cxl_region_exit(void);
 int cxl_get_poison_by_endpoint(struct cxl_port *port);
 struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa);
+u64 cxl_trace_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd,
+		  u64 dpa);
 
 #else
+static inline u64
+cxl_trace_hpa(struct cxl_region *cxlr, const struct cxl_memdev *cxlmd, u64 dpa)
+{
+	return ULLONG_MAX;
+}
 static inline
 struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa)
 {
drivers/cxl/core/region.c +91 -0

97 lines not on this page · raw message

drivers/cxl/core/trace.c +0 -91

94 lines not on this page · raw message

drivers/cxl/core/trace.h +0 -2
index e5f13260fc52..161bdb5734b0 100644
@@ -642,8 +642,6 @@ TRACE_EVENT(cxl_memory_module,
 #define cxl_poison_overflow(flags, time)				\
 	(flags & CXL_POISON_FLAG_OVERFLOW ? le64_to_cpu(time) : 0)
 
-u64 cxl_trace_hpa(struct cxl_region *cxlr, struct cxl_memdev *memdev, u64 dpa);
-
 TRACE_EVENT(cxl_poison,
 
 	TP_PROTO(struct cxl_memdev *cxlmd, struct cxl_region *cxlr,
Signature · 1 line
2.37.3
author maintainer
#3 lore raw

PATCH v4 3/3 cxl/core: Add region info to cxl_general_media and cxl_dram events

In reply to [email protected]

From: Alison Schofield <[email protected]>

User space may need to know which region, if any, maps the DPAs (device physical addresses) reported in a cxl_general_media or cxl_dram event. Since the mapping can change, the kernel provides this information at the time the event occurs. This informs user space that at event <timestamp> this <region> mapped this <DPA> to this <HPA>.

Add the same region info that is included in the cxl_poison trace
event: the DPA->HPA translation, region name, and region uuid.

The new fields are inserted in the trace event and no existing fields are modified. If the DPA is not mapped, user will see:
hpa=ULLONG_MAX, region="", and uuid=0

This work must be protected by dpa_rwsem & region_rwsem since it is looking up region mappings.

---
 drivers/cxl/core/mbox.c   | 36 ++++++++++++++++++++++++++------
 drivers/cxl/core/trace.h  | 44 +++++++++++++++++++++++++++++++--------
 include/linux/cxl-event.h | 10 +++++++++
 3 files changed, 75 insertions(+), 15 deletions(-)
drivers/cxl/core/mbox.c +30 -6
index 9adda4795eb7..df0fc2a4570f 100644
@@ -842,14 +842,38 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 			    enum cxl_event_type event_type,
 			    const uuid_t *uuid, union cxl_event *evt)
 {
-	if (event_type == CXL_CPER_EVENT_GEN_MEDIA)
-		trace_cxl_general_media(cxlmd, type, &evt->gen_media);
-	else if (event_type == CXL_CPER_EVENT_DRAM)
-		trace_cxl_dram(cxlmd, type, &evt->dram);
-	else if (event_type == CXL_CPER_EVENT_MEM_MODULE)
+	if (event_type == CXL_CPER_EVENT_MEM_MODULE) {
 		trace_cxl_memory_module(cxlmd, type, &evt->mem_module);
-	else
+		return;
+	}
+	if (event_type == CXL_CPER_EVENT_GENERIC) {
 		trace_cxl_generic_event(cxlmd, type, uuid, &evt->generic);
+		return;
+	}
+
+	if (trace_cxl_general_media_enabled() || trace_cxl_dram_enabled()) {
+		u64 dpa, hpa = ULLONG_MAX;
+		struct cxl_region *cxlr;
+
+		/*
+		 * These trace points are annotated with HPA and region
+		 * translations. Take topology mutation locks and lookup
+		 * { HPA, REGION } from { DPA, MEMDEV } in the event record.
+		 */
+		guard(rwsem_read)(&cxl_region_rwsem);
+		guard(rwsem_read)(&cxl_dpa_rwsem);
+
+		dpa = le64_to_cpu(evt->common.phys_addr) & CXL_DPA_MASK;
+		cxlr = cxl_dpa_to_region(cxlmd, dpa);
+		if (cxlr)
+			hpa = cxl_trace_hpa(cxlr, cxlmd, dpa);
+
+		if (event_type == CXL_CPER_EVENT_GEN_MEDIA)
+			trace_cxl_general_media(cxlmd, type, cxlr, hpa,
+						&evt->gen_media);
+		else if (event_type == CXL_CPER_EVENT_DRAM)
+			trace_cxl_dram(cxlmd, type, cxlr, hpa, &evt->dram);
+	}
 }
 EXPORT_SYMBOL_NS_GPL(cxl_event_trace_record, CXL);
 
drivers/cxl/core/trace.h +35 -9

110 lines not on this page · raw message

include/linux/cxl-event.h +10 -0
index 03fa6d50d46f..5342755777cc 100644
@@ -91,11 +91,21 @@ struct cxl_event_mem_module {
 	u8 reserved[0x3d];
 } __packed;
 
+/*
+ * General Media or DRAM Event Common Fields
+ * - provides common access to phys_addr
+ */
+struct cxl_event_common {
+	struct cxl_event_record_hdr hdr;
+	__le64 phys_addr;
+} __packed;
+
 union cxl_event {
 	struct cxl_event_generic generic;
 	struct cxl_event_gen_media gen_media;
 	struct cxl_event_dram dram;
 	struct cxl_event_mem_module mem_module;
+	struct cxl_event_common common;
 } __packed;
 
 /*
Signature · 1 line
2.37.3
Reviewed-by on v4 3/3
#4 lore raw

Re: [PATCH v4 3/3] cxl/core: Add region info to cxl_general_media and cxl_dram events

Quoted 37 lines · alison.schofield@ wrote:

From: Alison Schofield <[email protected]>

User space may need to know which region, if any, maps the DPAs (device physical addresses) reported in a cxl_general_media or cxl_dram event. Since the mapping can change, the kernel provides this information at the time the event occurs. This informs user space that at event <timestamp> this <region> mapped this <DPA> to this <HPA>.

Add the same region info that is included in the cxl_poison trace
event: the DPA->HPA translation, region name, and region uuid.

The new fields are inserted in the trace event and no existing fields are modified. If the DPA is not mapped, user will see:
hpa=ULLONG_MAX, region="", and uuid=0

This work must be protected by dpa_rwsem & region_rwsem since it is looking up region mappings.

---
 drivers/cxl/core/mbox.c   | 36 ++++++++++++++++++++++++++------
 drivers/cxl/core/trace.h  | 44 +++++++++++++++++++++++++++++++--------
 include/linux/cxl-event.h | 10 +++++++++
 3 files changed, 75 insertions(+), 15 deletions(-)
drivers/cxl/core/mbox.c +0 -3
index 9adda4795eb7..df0fc2a4570f 100644
@@ -842,14 +842,38 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 			    enum cxl_event_type event_type,
 			    const uuid_t *uuid, union cxl_event *evt)
 {
-	if (event_type == CXL_CPER_EVENT_GEN_MEDIA)
-		trace_cxl_general_media(cxlmd, type, &evt->gen_media);
-	else if (event_type == CXL_CPER_EVENT_DRAM)

drivers/cxl/core/mbox.c void cxl_event_trace_record(const struct cxl_memdev *cxlmd, · line 851

Quoted 10 lines
drivers/cxl/core/mbox.c +5 -3
-		trace_cxl_dram(cxlmd, type, &evt->dram);
-	else if (event_type == CXL_CPER_EVENT_MEM_MODULE)
+	if (event_type == CXL_CPER_EVENT_MEM_MODULE) {
 		trace_cxl_memory_module(cxlmd, type, &evt->mem_module);
-	else
+		return;
+	}
+	if (event_type == CXL_CPER_EVENT_GENERIC) {
 		trace_cxl_generic_event(cxlmd, type, uuid, &evt->generic);
+		return;

Minor, but I think you could shave a couple more lines by keeping the else-if tree going and skip the early "return" calls with a final:

	else if (trace_cxl_general_media_enabled() || trace_cxl_dram_enabled())
Quoted 5 lines
drivers/cxl/core/mbox.c +5 -0
+	}
+
+	if (trace_cxl_general_media_enabled() || trace_cxl_dram_enabled()) {
+		u64 dpa, hpa = ULLONG_MAX;
+		struct cxl_region *cxlr;

drivers/cxl/core/mbox.c void cxl_event_trace_record(const struct cxl_memdev *cxlmd, · line 866

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+
+		/*
+		 * These trace points are annotated with HPA and region
+		 * translations. Take topology mutation locks and lookup
+		 * { HPA, REGION } from { DPA, MEMDEV } in the event record.
+		 */
+		guard(rwsem_read)(&cxl_region_rwsem);
+		guard(rwsem_read)(&cxl_dpa_rwsem);
+
+		dpa = le64_to_cpu(evt->common.phys_addr) & CXL_DPA_MASK;

A merge issue here for Dave to handle here since CXL_DPA_MASK is known broken. Do you want to wait until that bikeshedding on CXL_DPA_MASK replacement name completes, or just take the simple:

-#define CXL_DPA_FLAGS_MASK 0x3F
+#define CXL_DPA_FLAGS_MASK 0x3FULL

...for now?

Otherwise,

author maintainer
#5 lore raw

Re: [PATCH v4 3/3] cxl/core: Add region info to cxl_general_media and cxl_dram events

Quoted 52 lines · Dan Williams
Quoted 37 lines · alison.schofield@ wrote:

From: Alison Schofield <[email protected]>

User space may need to know which region, if any, maps the DPAs (device physical addresses) reported in a cxl_general_media or cxl_dram event. Since the mapping can change, the kernel provides this information at the time the event occurs. This informs user space that at event <timestamp> this <region> mapped this <DPA> to this <HPA>.

Add the same region info that is included in the cxl_poison trace
event: the DPA->HPA translation, region name, and region uuid.

The new fields are inserted in the trace event and no existing fields are modified. If the DPA is not mapped, user will see:
hpa=ULLONG_MAX, region="", and uuid=0

This work must be protected by dpa_rwsem & region_rwsem since it is looking up region mappings.

---
 drivers/cxl/core/mbox.c   | 36 ++++++++++++++++++++++++++------
 drivers/cxl/core/trace.h  | 44 +++++++++++++++++++++++++++++++--------
 include/linux/cxl-event.h | 10 +++++++++
 3 files changed, 75 insertions(+), 15 deletions(-)
drivers/cxl/core/mbox.c +0 -3
index 9adda4795eb7..df0fc2a4570f 100644
@@ -842,14 +842,38 @@ void cxl_event_trace_record(const struct cxl_memdev *cxlmd,
 			    enum cxl_event_type event_type,
 			    const uuid_t *uuid, union cxl_event *evt)
 {
-	if (event_type == CXL_CPER_EVENT_GEN_MEDIA)
-		trace_cxl_general_media(cxlmd, type, &evt->gen_media);
-	else if (event_type == CXL_CPER_EVENT_DRAM)

drivers/cxl/core/mbox.c void cxl_event_trace_record(const struct cxl_memdev *cxlmd, · line 851

Quoted 10 lines
drivers/cxl/core/mbox.c +5 -3
-		trace_cxl_dram(cxlmd, type, &evt->dram);
-	else if (event_type == CXL_CPER_EVENT_MEM_MODULE)
+	if (event_type == CXL_CPER_EVENT_MEM_MODULE) {
 		trace_cxl_memory_module(cxlmd, type, &evt->mem_module);
-	else
+		return;
+	}
+	if (event_type == CXL_CPER_EVENT_GENERIC) {
 		trace_cxl_generic_event(cxlmd, type, uuid, &evt->generic);
+		return;

Minor, but I think you could shave a couple more lines by keeping the else-if tree going and skip the early "return" calls with a final:

Quoted 2 lines
	else if (trace_cxl_general_media_enabled() || trace_cxl_dram_enabled())

I think I didn't do that because this last 'if' breaks the pattern since it's not based on event_type. We're doing something different in this chunk of code.

A switch would be well suited, but it uses even more LOC - like this:

+
+
+ switch (event_type) {
+ case CXL_CPER_EVENT_MEM_MODULE:
+ trace_cxl_memory_module(cxlmd, type, &evt->mem_module);
+ break;
+
+ case CXL_CPER_EVENT_GENERIC:
+ trace_cxl_generic_event(cxlmd, type, uuid, &evt->generic);
+ break;
+
+ case CXL_CPER_EVENT_GEN_MEDIA:
+ case CXL_CPER_EVENT_DRAM:
+ u64 dpa, hpa = ULLONG_MAX;
+ struct cxl_region *cxlr;
+
+ if (!trace_cxl_general_media_enabled() &&
+ !trace_cxl_dram_enabled())
+ break;
+ /*
+ * These trace points are annotated with HPA and region
+ * translations. Take topology mutation locks and lookup
+ * { HPA, REGION } from { DPA, MEMDEV } in the event record.
+ */
+ guard(rwsem_read)(&cxl_region_rwsem);
+ guard(rwsem_read)(&cxl_dpa_rwsem);
+
+ dpa = le64_to_cpu(evt->common.phys_addr) & CXL_DPA_MASK;
+ cxlr = cxl_dpa_to_region(cxlmd, dpa);
+ if (cxlr)
+ hpa = cxl_trace_hpa(cxlr, cxlmd, dpa);
+
+ if (event_type == CXL_CPER_EVENT_GEN_MEDIA)
+ trace_cxl_general_media(cxlmd, type, cxlr, hpa,
+ &evt->gen_media);
+ else if (event_type == CXL_CPER_EVENT_DRAM)
+ trace_cxl_dram(cxlmd, type, cxlr, hpa, &evt->dram);
+ break;
+ default:
+ break;
+ }
+

Maybe if I just move the comment before the changed chunk of code, the break in flow will be easier to follow.

Quoted 1 line

drivers/cxl/core/mbox.c void cxl_event_trace_record(const struct cxl_memdev *cxlmd, · line 852

Quoted 1 line
drivers/cxl/core/mbox.c +1 -0
+	}
	/* Move the comment below to here */
Quoted 19 lines
Quoted 4 lines
drivers/cxl/core/mbox.c +4 -0
+
+	if (trace_cxl_general_media_enabled() || trace_cxl_dram_enabled()) {
+		u64 dpa, hpa = ULLONG_MAX;
+		struct cxl_region *cxlr;

drivers/cxl/core/mbox.c void cxl_event_trace_record(const struct cxl_memdev *cxlmd, · line 866

Quoted 10 lines
drivers/cxl/core/mbox.c +10 -0
+
+		/*
+		 * These trace points are annotated with HPA and region
+		 * translations. Take topology mutation locks and lookup
+		 * { HPA, REGION } from { DPA, MEMDEV } in the event record.
+		 */
+		guard(rwsem_read)(&cxl_region_rwsem);
+		guard(rwsem_read)(&cxl_dpa_rwsem);
+
+		dpa = le64_to_cpu(evt->common.phys_addr) & CXL_DPA_MASK;

A merge issue here for Dave to handle here since CXL_DPA_MASK is known broken. Do you want to wait until that bikeshedding on CXL_DPA_MASK replacement name completes, or just take the simple:

Quoted 4 lines

-#define CXL_DPA_FLAGS_MASK 0x3F
+#define CXL_DPA_FLAGS_MASK 0x3FULL

...for now?

I'm optimistic we can get that fixes patch in. Don't worry about it ;)

Quoted 4 lines

Otherwise,