Skip to contentSkip to conversation

cxl/port: Unify RAS setup across port types

PATCH Applied Dan Williams 4 participants 13 messages Jan 31 – Feb 2 CXL

lore
Share & tools
lore
#0 lore raw

v2 cover cxl/port: Unify RAS setup across port types

Changes since v1 [1]:
- Cleanup the diff by keeping the order of dport_exists() relative to the dev->driver check in cxl_port_add_dport() (Jonathan)
- Drop some repetitive de-referencing with a local @port variable in dport_to_host() (Jonathan)
- s/group/dr_group/ throughout to clarify "devres group" (Jonathan)
- Reuse del_dport() for cxl_dport_release_dr_group() (Jonathan)
- Drop the thin wrappers for devres_{open,close}_group() for the port group.
- Add a comment and a cleanup TODO for the 'add_dport()' operation in 'struct cxl_driver'. (Jonathan)
- Change patch 7 subject to just: "cxl/port: Map Port RAS registers" since it is only introducing a helper that is later used in the endpoint case. (Jonathan)

[1]: http://lore.kernel.org/[email protected]

Original cover letter:

---

The CXL Port Protocol error handling series grew to be over 30 patches which is too much to handle at once given the various topics involved. One of the sub-threads of the v14 review was confusion about the new devres groups to manage port setup unwind failures [2].

[2]: http://lore.kernel.org/[email protected]

Given that review indicated a need to break up and better explain the conversion, do that in a separate patch set. Build on top of the first 18 patches of that series [3] that are ready to merge.

[3]: https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-7.0/cxl-aer-prep

The wider goals of the port protocol handling series are:

1/ Be minimally invasive to the ongoing maintenance burden of PCIe error
handling. Just do the minimal enlightenment to forward "internal" errors for device with active CXL links to the CXL core.

2/ Build a framework for any driver that registers a 'struct cxl_memdev'
(or in the future a 'struct cxl_cachedev') gets protocol error handling support.

This "Unify RAS setup across port types" set supports goal 2/. It enables a model where all CXL error handling is relative to the common 'struct cxl_port' and 'struct cxl_dport' objects and is agnostic to whether those objects are in support of the memory expansion class device (driven by cxl_pci) or any other CXL endpoint in the system that supports CXL.cachemem operation.

In support of that unification, the setup of RAS registers needs to be centralized. That in turn requires new handling for early exit setup failures and additional teardown support for resources optionally acquired at port / dport creation time.

The devres group mechanism is deployed to cleanup some open coded devm_release_action() calls. The devres group facility also comes in handy for unwinding conditional setup steps in the port creation process. Recall that ports defer probing their CXL resources until after they are known to have a downstream CXL connection. So, early exit during setup of a new dport may have more or less work to do depending on whether the first or subsequent dport is being added.

Given probing port resources is a 'probe' action it fits more naturally as a driver operation. If cxl_port_add_dport() then moves to cxl_port driver operation alongside ->probe(), it enables a cxl_test cleanup. The cxl_test approach has a hard time mocking interfaces that are internal to the cxl_core.

The rest of the patches in this set finish off the conversion of 'struct cxl_port' and 'struct cxl_dport' to be the only CXL objects that interact with the CXL RAS.

Dan Williams (8):
cxl/port: Cleanup handling of the nr_dports 0 -> 1 transition cxl/port: Reduce number of @dport variables in cxl_port_add_dport() cxl/port: Cleanup dport removal with a devres group cxl/port: Move decoder setup before dport creation cxl/port: Move dport probe operations to a driver event cxl/port: Move dport RAS setup to dport add time cxl/port: Move endpoint component register management to cxl_port cxl/port: Unify endpoint and switch port lookup

Terry Bowman (1):
cxl/port: Map Port RAS registers

 drivers/cxl/core/core.h              |  10 ++
 drivers/cxl/cxl.h                    |  33 +++---
 drivers/cxl/cxlmem.h                 |   4 +-
 drivers/cxl/cxlpci.h                 |  12 +-
 tools/testing/cxl/exports.h          |  13 ---
 drivers/cxl/core/hdm.c               |   6 +-
 drivers/cxl/core/pci.c               |   8 +-
 drivers/cxl/core/port.c              | 159 +++++++++++++++++----------
 drivers/cxl/core/ras.c               |  50 ++++++---
 drivers/cxl/mem.c                    |   2 -
 drivers/cxl/pci.c                    |  63 +----------
 drivers/cxl/port.c                   | 122 ++++++++++++++++++++
 tools/testing/cxl/cxl_core_exports.c |  22 ----
 tools/testing/cxl/test/mock.c        |  36 ++----
 tools/testing/cxl/Kbuild             |   3 +-
 15 files changed, 310 insertions(+), 233 deletions(-)
 delete mode 100644 tools/testing/cxl/exports.h

base-commit: 9a8920ca8ebfb99604f639e7fbc681d0d04518a0

Signature · 1 line
2.52.0
#1 lore raw

PATCH v2 1/9 cxl/port: Cleanup handling of the nr_dports 0 -> 1 transition

There are multiple setup actions that can occur for a switch port after it is known that it has at least one active downstream link. That work is currently split between __devm_cxl_add_dport(), the add_dport() helper, and cxl_port_add_dport() where decoder setup occurs.

Clean this up by moving all @dport object setup responsibilities into add_dport() and all port effects into cxl_port_add_dport().

add_dport() handles taking a reference on @dport->dport_dev, and cxl_port_add_dport() grows the awareness to setup the port component registers. This removes an awkward open-coded xa_erase() from the middle of __devm_cxl_add_dport() and instead tasks cxl_port_add_dport() with calling the common @dport destruction path if anything goes wrong.

After this @port->nr_dports is always the count of @dports in the @port->dports xarray, and cxl_dport_remove() is symmetric with add_dport(). With ->nr_dports now reliably tracking the number of dports the use of ida_is_empty() can be dropped. Recall that the ida is only cleared on "release" of decoder objects, and release can be arbitrarily delayed past unregistration.

Lastly port->component_reg_phys is no longer reset to CXL_RESOURCE_NONE post setup, no reason is seen to carry that forward.

---
 drivers/cxl/core/port.c | 31 +++++++++++++++----------------
 1 file changed, 15 insertions(+), 16 deletions(-)
drivers/cxl/core/port.c +15 -16
index fef3aa0c6680..ff899c690d85 100644
@@ -1066,11 +1066,15 @@ static int add_dport(struct cxl_port *port, struct cxl_dport *dport)
 		return -EBUSY;
 	}
 
+	/* Arrange for dport_dev to be valid through remove_dport() */
+	struct device *dev __free(put_device) = get_device(dport->dport_dev);
+
 	rc = xa_insert(&port->dports, (unsigned long)dport->dport_dev, dport,
 		       GFP_KERNEL);
 	if (rc)
 		return rc;
 
+	retain_and_null_ptr(dev);
 	port->nr_dports++;
 	return 0;
 }
@@ -1099,6 +1103,7 @@ static void cxl_dport_remove(void *data)
 	struct cxl_dport *dport = data;
 	struct cxl_port *port = dport->port;
 
+	port->nr_dports--;
 	xa_erase(&port->dports, (unsigned long) dport->dport_dev);
 	put_device(dport->dport_dev);
 }
@@ -1181,21 +1186,6 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 	if (rc)
 		return ERR_PTR(rc);
 
-	/*
-	 * Setup port register if this is the first dport showed up. Having
-	 * a dport also means that there is at least 1 active link.
-	 */
-	if (port->nr_dports == 1 &&
-	    port->component_reg_phys != CXL_RESOURCE_NONE) {
-		rc = cxl_port_setup_regs(port, port->component_reg_phys);
-		if (rc) {
-			xa_erase(&port->dports, (unsigned long)dport->dport_dev);
-			return ERR_PTR(rc);
-		}
-		port->component_reg_phys = CXL_RESOURCE_NONE;
-	}
-
-	get_device(dport_dev);
 	rc = devm_add_action_or_reset(host, cxl_dport_remove, dport);
 	if (rc)
 		return ERR_PTR(rc);
@@ -1622,7 +1612,16 @@ static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 
 	cxl_switch_parse_cdat(new_dport);
 
-	if (ida_is_empty(&port->decoder_ida)) {
+	if (port->nr_dports == 1) {
+		/*
+		 * Some host bridges are known to not have component regsisters
+		 * available until a root port has trained CXL. Perform that
+		 * setup now.
+		 */
+		rc = cxl_port_setup_regs(port, port->component_reg_phys);
+		if (rc)
+			return ERR_PTR(rc);
+
 		rc = devm_cxl_switch_port_decoders_setup(port);
 		if (rc)
 			return ERR_PTR(rc);
Signature · 1 line
2.52.0
#2 lore raw

PATCH v2 2/9 cxl/port: Reduce number of @dport variables in cxl_port_add_dport()

In reply to Dan Williams

In preparation for refactoring cxl_port_add_dport() to add RAS register setup, cleanup the number of dport variables with a dport_exists() helper.

Kill the @dport needed to check for duplicates, rename @new_dport to @dport.

---
 drivers/cxl/core/port.c | 38 +++++++++++++++++++++++---------------
 1 file changed, 23 insertions(+), 15 deletions(-)
drivers/cxl/core/port.c +23 -15
index ff899c690d85..d7b6f52d0adc 100644
@@ -1587,30 +1587,38 @@ static int update_decoder_targets(struct device *dev, void *data)
 	return 0;
 }
 
+static bool dport_exists(struct cxl_port *port, struct device *dport_dev)
+{
+	struct cxl_dport *dport = cxl_find_dport_by_dev(port, dport_dev);
+
+	if (dport) {
+		dev_dbg(&port->dev, "dport%d:%s already exists\n",
+			dport->port_id, dev_name(dport_dev));
+		return true;
+	}
+
+	return false;
+}
+
 DEFINE_FREE(del_cxl_dport, struct cxl_dport *, if (!IS_ERR_OR_NULL(_T)) del_dport(_T))
 static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 					    struct device *dport_dev)
 {
-	struct cxl_dport *dport;
 	int rc;
 
 	device_lock_assert(&port->dev);
 	if (!port->dev.driver)
 		return ERR_PTR(-ENXIO);
 
-	dport = cxl_find_dport_by_dev(port, dport_dev);
-	if (dport) {
-		dev_dbg(&port->dev, "dport%d:%s already exists\n",
-			dport->port_id, dev_name(dport_dev));
+	if (dport_exists(port, dport_dev))
 		return ERR_PTR(-EBUSY);
-	}
 
-	struct cxl_dport *new_dport __free(del_cxl_dport) =
+	struct cxl_dport *dport __free(del_cxl_dport) =
 		devm_cxl_add_dport_by_dev(port, dport_dev);
-	if (IS_ERR(new_dport))
-		return new_dport;
+	if (IS_ERR(dport))
+		return dport;
 
-	cxl_switch_parse_cdat(new_dport);
+	cxl_switch_parse_cdat(dport);
 
 	if (port->nr_dports == 1) {
 		/*
@@ -1626,17 +1634,17 @@ static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 		if (rc)
 			return ERR_PTR(rc);
 		dev_dbg(&port->dev, "first dport%d:%s added with decoders\n",
-			new_dport->port_id, dev_name(dport_dev));
-		return no_free_ptr(new_dport);
+			dport->port_id, dev_name(dport_dev));
+		return no_free_ptr(dport);
 	}
 
 	/* New dport added, update the decoder targets */
-	device_for_each_child(&port->dev, new_dport, update_decoder_targets);
+	device_for_each_child(&port->dev, dport, update_decoder_targets);
 
-	dev_dbg(&port->dev, "dport%d:%s added\n", new_dport->port_id,
+	dev_dbg(&port->dev, "dport%d:%s added\n", dport->port_id,
 		dev_name(dport_dev));
 
-	return no_free_ptr(new_dport);
+	return no_free_ptr(dport);
 }
 
 static struct cxl_dport *devm_cxl_create_port(struct device *ep_dev,
Signature · 1 line
2.52.0
#3 lore raw

PATCH v2 3/9 cxl/port: Cleanup dport removal with a devres group

In reply to Dan Williams

In preparation for adding more setup actions like RAS register mapping, introduce a devres group to collect all the dport creation / registration actions. This replaces the maintenance tedium of open coding several devm_release_action() calls in del_dport().

---
 drivers/cxl/core/port.c | 71 +++++++++++++++++++++++++++++++++++------
 1 file changed, 61 insertions(+), 10 deletions(-)
drivers/cxl/core/port.c +61 -10
index d7b6f52d0adc..99bbcf9cf236 100644
@@ -1118,6 +1118,57 @@ static void cxl_dport_unlink(void *data)
 	sysfs_remove_link(&port->dev.kobj, link_name);
 }
 
+static struct device *dport_to_host(struct cxl_dport *dport)
+{
+	struct cxl_port *port = dport->port;
+
+	if (is_cxl_root(port))
+		return port->uport_dev;
+	return &port->dev;
+}
+
+static void free_dport(void *dport)
+{
+	kfree(dport);
+}
+
+/*
+ * Upon return either a group is established with one action (free_dport()), or
+ * no group established and @dport is freed.
+ */
+static void *cxl_dport_open_dr_group_or_free(struct cxl_dport *dport)
+{
+	int rc;
+	struct device *host = dport_to_host(dport);
+	void *group = devres_open_group(host, dport, GFP_KERNEL);
+
+	if (!group) {
+		kfree(dport);
+		return NULL;
+	}
+
+	rc = devm_add_action_or_reset(host, free_dport, dport);
+	if (rc) {
+		devres_release_group(host, group);
+		return NULL;
+	}
+
+	return group;
+}
+
+static void cxl_dport_close_dr_group(struct cxl_dport *dport, void *group)
+{
+	devres_close_group(dport_to_host(dport), group);
+}
+
+static void del_dport(struct cxl_dport *dport)
+{
+	devres_release_group(dport_to_host(dport), dport);
+}
+
+/* The dport group id is the dport */
+DEFINE_FREE(cxl_dport_release_dr_group, void *, if (_T) del_dport(_T))
+
 static struct cxl_dport *
 __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 		     int port_id, resource_size_t component_reg_phys,
@@ -1143,14 +1194,20 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 	    CXL_TARGET_STRLEN)
 		return ERR_PTR(-EINVAL);
 
-	dport = devm_kzalloc(host, sizeof(*dport), GFP_KERNEL);
+	dport = kzalloc(sizeof(*dport), GFP_KERNEL);
 	if (!dport)
 		return ERR_PTR(-ENOMEM);
 
+	/* Just enough init to manage the devres group */
 	dport->dport_dev = dport_dev;
 	dport->port_id = port_id;
 	dport->port = port;
 
+	void *dport_dr_group __free(cxl_dport_release_dr_group) =
+		cxl_dport_open_dr_group_or_free(dport);
+	if (!dport_dr_group)
+		return ERR_PTR(-ENOMEM);
+
 	if (rcrb == CXL_RESOURCE_NONE) {
 		rc = cxl_dport_setup_regs(&port->dev, dport,
 					  component_reg_phys);
@@ -1203,6 +1260,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 
 	cxl_debugfs_create_dport_dir(dport);
 
+	/* keep the group, and mark the end of devm actions */
+	cxl_dport_close_dr_group(dport, no_free_ptr(dport_dr_group));
+
 	return dport;
 }
 
@@ -1429,15 +1489,6 @@ static void delete_switch_port(struct cxl_port *port)
 	devm_release_action(port->dev.parent, unregister_port, port);
 }
 
-static void del_dport(struct cxl_dport *dport)
-{
-	struct cxl_port *port = dport->port;
-
-	devm_release_action(&port->dev, cxl_dport_unlink, dport);
-	devm_release_action(&port->dev, cxl_dport_remove, dport);
-	devm_kfree(&port->dev, dport);
-}
-
 static void del_dports(struct cxl_port *port)
 {
 	struct cxl_dport *dport;
Signature · 1 line
2.52.0
Reviewed-by on v2 3/9
#4 lore raw

Re: [PATCH v2 3/9] cxl/port: Cleanup dport removal with a devres group

Quoted 7 lines · On Fri, 30 Jan 2026 16:03:57 -0800 Dan Williams <[email protected]>…

In preparation for adding more setup actions like RAS register mapping, introduce a devres group to collect all the dport creation / registration actions. This replaces the maintenance tedium of open coding several devm_release_action() calls in del_dport().

I think this is on the right side of the 'too clever' boundary as it is reasonably easy to follow so

#5 lore raw

PATCH v2 4/9 cxl/port: Move decoder setup before dport creation

In reply to Dan Williams

There are port setup actions that run on first dport arrival, and there are setup actions that run per dport.

RAS register setup is a future additional setup action to run per-port (once the first dport arrives), and each dport also has RAS registers to map.

Before adding that, flip the order of "first dport" and "per-dport" actions. This makes allocation symmetric with teardown, "first dport" actions unwind after last dport removed. It also allows for using a devres group to collect the unrelated decoder, RAS, and dport setup actions into one group release action.

The new cxl_port_open_group() collects "first dport" and "per-dport" into one group that can be released on any failure. This group's lifetime only needs to span the short duration of cxl_port_add_dport() to cleanup all potential damage from failing to add a dport. Contrast that to the "dport" devres group that is called upon to destruct fully formed dport objects.

---
 drivers/cxl/core/port.c | 33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)
drivers/cxl/core/port.c +21 -12
index 99bbcf9cf236..6a554d0466a1 100644
@@ -1651,10 +1651,14 @@ static bool dport_exists(struct cxl_port *port, struct device *dport_dev)
 	return false;
 }
 
-DEFINE_FREE(del_cxl_dport, struct cxl_dport *, if (!IS_ERR_OR_NULL(_T)) del_dport(_T))
+/* note this implicitly casts the group back to its @port */
+DEFINE_FREE(cxl_port_release_dr_group, struct cxl_port *,
+	    if (_T) devres_release_group(&_T->dev, _T))
+
 static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 					    struct device *dport_dev)
 {
+	struct cxl_dport *dport;
 	int rc;
 
 	device_lock_assert(&port->dev);
@@ -1664,14 +1668,13 @@ static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 	if (dport_exists(port, dport_dev))
 		return ERR_PTR(-EBUSY);
 
-	struct cxl_dport *dport __free(del_cxl_dport) =
-		devm_cxl_add_dport_by_dev(port, dport_dev);
-	if (IS_ERR(dport))
-		return dport;
-
-	cxl_switch_parse_cdat(dport);
+	/* Temp group for all "first dport" and "per dport" setup actions */
+	void *port_dr_group __free(cxl_port_release_dr_group) =
+		devres_open_group(&port->dev, port, GFP_KERNEL);
+	if (!port_dr_group)
+		return ERR_PTR(-ENOMEM);
 
-	if (port->nr_dports == 1) {
+	if (port->nr_dports == 0) {
 		/*
 		 * Some host bridges are known to not have component regsisters
 		 * available until a root port has trained CXL. Perform that
@@ -1684,18 +1687,24 @@ static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 		rc = devm_cxl_switch_port_decoders_setup(port);
 		if (rc)
 			return ERR_PTR(rc);
-		dev_dbg(&port->dev, "first dport%d:%s added with decoders\n",
-			dport->port_id, dev_name(dport_dev));
-		return no_free_ptr(dport);
 	}
 
+	dport = devm_cxl_add_dport_by_dev(port, dport_dev);
+	if (IS_ERR(dport))
+		return dport;
+
+	/* This group was only needed for early exit above */
+	devres_remove_group(&port->dev, no_free_ptr(port_dr_group));
+
+	cxl_switch_parse_cdat(dport);
+
 	/* New dport added, update the decoder targets */
 	device_for_each_child(&port->dev, dport, update_decoder_targets);
 
 	dev_dbg(&port->dev, "dport%d:%s added\n", dport->port_id,
 		dev_name(dport_dev));
 
-	return no_free_ptr(dport);
+	return dport;
 }
 
 static struct cxl_dport *devm_cxl_create_port(struct device *ep_dev,
Signature · 1 line
2.52.0
#6 lore raw

PATCH v2 5/9 cxl/port: Move dport probe operations to a driver event

In reply to Dan Williams

In preparation for adding more register setup to the cxl_port_add_dport() path (for RAS register mapping), move the dport creation event to a driver callback. This achieves two goals, it puts driver operations logically where they belong, in a driver, and it obviates the gymnastics of DECLARE_TESTABLE() which just makes a mess of grepping for CXL symbols.

In other words, a driver callback is less of an ongoing maintenance burden than this DECLARE_TESTABLE arrangement that does not scale and diminishes the grep-ability of the codebase.

cxl_port_add_dport() moves mostly unmodified from drivers/cxl/core/port.c. The only deliberate change is that it now assumes that the device_lock is held on entry and the driver is attached (just like cxl_port_probe()).

---
 drivers/cxl/cxl.h                    | 31 +++++--------
 tools/testing/cxl/exports.h          | 13 ------
 drivers/cxl/core/hdm.c               |  6 +--
 drivers/cxl/core/pci.c               |  8 ++--
 drivers/cxl/core/port.c              | 68 ++++++++--------------------
 drivers/cxl/port.c                   | 50 ++++++++++++++++++++
 tools/testing/cxl/cxl_core_exports.c | 22 ---------
 tools/testing/cxl/test/mock.c        | 24 +++-------
 tools/testing/cxl/Kbuild             |  2 +
 9 files changed, 98 insertions(+), 126 deletions(-)
 delete mode 100644 tools/testing/cxl/exports.h
drivers/cxl/cxl.h +12 -19
index 6f3741a57932..4479d632a687 100644
@@ -840,8 +840,11 @@ struct cxl_endpoint_dvsec_info {
 };
 
 int devm_cxl_switch_port_decoders_setup(struct cxl_port *port);
-int __devm_cxl_switch_port_decoders_setup(struct cxl_port *port);
 int devm_cxl_endpoint_decoders_setup(struct cxl_port *port);
+void cxl_port_update_decoder_targets(struct cxl_port *port,
+				     struct cxl_dport *dport);
+int cxl_port_setup_regs(struct cxl_port *port,
+			resource_size_t component_reg_phys);
 
 struct cxl_dev_state;
 int cxl_dvsec_rr_decode(struct cxl_dev_state *cxlds,
@@ -851,10 +854,18 @@ bool is_cxl_region(struct device *dev);
 
 extern const struct bus_type cxl_bus_type;
 
+/*
+ * Note, add_dport() is expressly for the cxl_port driver. TODO: investigate a
+ * type-safe driver model where probe()/remove() take the type of object implied
+ * by @id and the add_dport() op only defined for the CXL_DEVICE_PORT driver
+ * template.
+ */
 struct cxl_driver {
 	const char *name;
 	int (*probe)(struct device *dev);
 	void (*remove)(struct device *dev);
+	struct cxl_dport *(*add_dport)(struct cxl_port *port,
+				       struct device *dport_dev);
 	struct device_driver drv;
 	int id;
 };
@@ -939,8 +950,6 @@ void cxl_coordinates_combine(struct access_coordinate *out,
 bool cxl_endpoint_decoder_reset_detected(struct cxl_port *port);
 struct cxl_dport *devm_cxl_add_dport_by_dev(struct cxl_port *port,
 					    struct device *dport_dev);
-struct cxl_dport *__devm_cxl_add_dport_by_dev(struct cxl_port *port,
-					      struct device *dport_dev);
 
 /*
  * Unit test builds overrides this to __weak, find the 'strong' version
@@ -952,20 +961,4 @@ struct cxl_dport *__devm_cxl_add_dport_by_dev(struct cxl_port *port,
 
 u16 cxl_gpf_get_dvsec(struct device *dev);
 
-/*
- * Declaration for functions that are mocked by cxl_test that are called by
- * cxl_core. The respective functions are defined as __foo() and called by
- * cxl_core as foo(). The macros below ensures that those functions would
- * exist as foo(). See tools/testing/cxl/cxl_core_exports.c and
- * tools/testing/cxl/exports.h for setting up the mock functions. The dance
- * is done to avoid a circular dependency where cxl_core calls a function that
- * ends up being a mock function and goes to * cxl_test where it calls a
- * cxl_core function.
- */
-#ifndef CXL_TEST_ENABLE
-#define DECLARE_TESTABLE(x) __##x
-#define devm_cxl_add_dport_by_dev DECLARE_TESTABLE(devm_cxl_add_dport_by_dev)
-#define devm_cxl_switch_port_decoders_setup DECLARE_TESTABLE(devm_cxl_switch_port_decoders_setup)
-#endif
-
 #endif /* __CXL_H__ */
tools/testing/cxl/exports.h deleted +0 -13
deleted file mode 100644
index 7ebee7c0bd67..000000000000
@@ -1,13 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-/* Copyright(c) 2025 Intel Corporation */
-#ifndef __MOCK_CXL_EXPORTS_H_
-#define __MOCK_CXL_EXPORTS_H_
-
-typedef struct cxl_dport *(*cxl_add_dport_by_dev_fn)(struct cxl_port *port,
-							  struct device *dport_dev);
-extern cxl_add_dport_by_dev_fn _devm_cxl_add_dport_by_dev;
-
-typedef int(*cxl_switch_decoders_setup_fn)(struct cxl_port *port);
-extern cxl_switch_decoders_setup_fn _devm_cxl_switch_port_decoders_setup;
-
-#endif
drivers/cxl/core/hdm.c +3 -3
index 1c5d2022c87a..365b02b7a241 100644
@@ -1219,12 +1219,12 @@ static int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
 }
 
 /**
- * __devm_cxl_switch_port_decoders_setup - allocate and setup switch decoders
+ * devm_cxl_switch_port_decoders_setup - allocate and setup switch decoders
  * @port: CXL port context
  *
  * Return 0 or -errno on error
  */
-int __devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
+int devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
 {
 	struct cxl_hdm *cxlhdm;
 
@@ -1248,7 +1248,7 @@ int __devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
 	dev_err(&port->dev, "HDM decoder capability not found\n");
 	return -ENXIO;
 }
-EXPORT_SYMBOL_NS_GPL(__devm_cxl_switch_port_decoders_setup, "CXL");
+EXPORT_SYMBOL_NS_GPL(devm_cxl_switch_port_decoders_setup, "CXL");
 
 /**
  * devm_cxl_endpoint_decoders_setup - allocate and setup endpoint decoders
drivers/cxl/core/pci.c +4 -4
index b838c59d7a3c..f96ce884a213 100644
@@ -41,14 +41,14 @@ static int pci_get_port_num(struct pci_dev *pdev)
 }
 
 /**
- * __devm_cxl_add_dport_by_dev - allocate a dport by dport device
+ * devm_cxl_add_dport_by_dev - allocate a dport by dport device
  * @port: cxl_port that hosts the dport
  * @dport_dev: 'struct device' of the dport
  *
  * Returns the allocated dport on success or ERR_PTR() of -errno on error
  */
-struct cxl_dport *__devm_cxl_add_dport_by_dev(struct cxl_port *port,
-					      struct device *dport_dev)
+struct cxl_dport *devm_cxl_add_dport_by_dev(struct cxl_port *port,
+					    struct device *dport_dev)
 {
 	struct cxl_register_map map;
 	struct pci_dev *pdev;
@@ -69,7 +69,7 @@ struct cxl_dport *__devm_cxl_add_dport_by_dev(struct cxl_port *port,
 	device_lock_assert(&port->dev);
 	return devm_cxl_add_dport(port, dport_dev, port_num, map.resource);
 }
-EXPORT_SYMBOL_NS_GPL(__devm_cxl_add_dport_by_dev, "CXL");
+EXPORT_SYMBOL_NS_GPL(devm_cxl_add_dport_by_dev, "CXL");
 
 static int cxl_dvsec_mem_range_valid(struct cxl_dev_state *cxlds, int id)
 {
drivers/cxl/core/port.c +20 -48

118 lines not on this page · raw message

drivers/cxl/port.c +50 -0

59 lines not on this page · raw message

tools/testing/cxl/cxl_core_exports.c +0 -22
index 6754de35598d..f088792a8925 100644
@@ -2,28 +2,6 @@
 /* Copyright(c) 2022 Intel Corporation. All rights reserved. */
 
 #include "cxl.h"
-#include "exports.h"
 
 /* Exporting of cxl_core symbols that are only used by cxl_test */
 EXPORT_SYMBOL_NS_GPL(cxl_num_decoders_committed, "CXL");
-
-cxl_add_dport_by_dev_fn _devm_cxl_add_dport_by_dev =
-	__devm_cxl_add_dport_by_dev;
-EXPORT_SYMBOL_NS_GPL(_devm_cxl_add_dport_by_dev, "CXL");
-
-struct cxl_dport *devm_cxl_add_dport_by_dev(struct cxl_port *port,
-					    struct device *dport_dev)
-{
-	return _devm_cxl_add_dport_by_dev(port, dport_dev);
-}
-EXPORT_SYMBOL_NS_GPL(devm_cxl_add_dport_by_dev, "CXL");
-
-cxl_switch_decoders_setup_fn _devm_cxl_switch_port_decoders_setup =
-	__devm_cxl_switch_port_decoders_setup;
-EXPORT_SYMBOL_NS_GPL(_devm_cxl_switch_port_decoders_setup, "CXL");
-
-int devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
-{
-	return _devm_cxl_switch_port_decoders_setup(port);
-}
-EXPORT_SYMBOL_NS_GPL(devm_cxl_switch_port_decoders_setup, "CXL");
tools/testing/cxl/test/mock.c +7 -17
index 44bce80ef3ff..f307c5b39184 100644
@@ -10,21 +10,12 @@
 #include <cxlmem.h>
 #include <cxlpci.h>
 #include "mock.h"
-#include "../exports.h"
 
 static LIST_HEAD(mock);
 
-static struct cxl_dport *
-redirect_devm_cxl_add_dport_by_dev(struct cxl_port *port,
-				   struct device *dport_dev);
-static int redirect_devm_cxl_switch_port_decoders_setup(struct cxl_port *port);
-
 void register_cxl_mock_ops(struct cxl_mock_ops *ops)
 {
 	list_add_rcu(&ops->list, &mock);
-	_devm_cxl_add_dport_by_dev = redirect_devm_cxl_add_dport_by_dev;
-	_devm_cxl_switch_port_decoders_setup =
-		redirect_devm_cxl_switch_port_decoders_setup;
 }
 EXPORT_SYMBOL_GPL(register_cxl_mock_ops);
 
@@ -32,9 +23,6 @@ DEFINE_STATIC_SRCU(cxl_mock_srcu);
 
 void unregister_cxl_mock_ops(struct cxl_mock_ops *ops)
 {
-	_devm_cxl_switch_port_decoders_setup =
-		__devm_cxl_switch_port_decoders_setup;
-	_devm_cxl_add_dport_by_dev = __devm_cxl_add_dport_by_dev;
 	list_del_rcu(&ops->list);
 	synchronize_srcu(&cxl_mock_srcu);
 }
@@ -163,7 +151,7 @@ __wrap_nvdimm_bus_register(struct device *dev,
 }
 EXPORT_SYMBOL_GPL(__wrap_nvdimm_bus_register);
 
-int redirect_devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
+int __wrap_devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
 {
 	int rc, index;
 	struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
@@ -171,11 +159,12 @@ int redirect_devm_cxl_switch_port_decoders_setup(struct cxl_port *port)
 	if (ops && ops->is_mock_port(port->uport_dev))
 		rc = ops->devm_cxl_switch_port_decoders_setup(port);
 	else
-		rc = __devm_cxl_switch_port_decoders_setup(port);
+		rc = devm_cxl_switch_port_decoders_setup(port);
 	put_cxl_mock_ops(index);
 
 	return rc;
 }
+EXPORT_SYMBOL_NS_GPL(__wrap_devm_cxl_switch_port_decoders_setup, "CXL");
 
 int __wrap_devm_cxl_endpoint_decoders_setup(struct cxl_port *port)
 {
@@ -257,8 +246,8 @@ void __wrap_cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device
 }
 EXPORT_SYMBOL_NS_GPL(__wrap_cxl_dport_init_ras_reporting, "CXL");
 
-struct cxl_dport *redirect_devm_cxl_add_dport_by_dev(struct cxl_port *port,
-						     struct device *dport_dev)
+struct cxl_dport *__wrap_devm_cxl_add_dport_by_dev(struct cxl_port *port,
+						   struct device *dport_dev)
 {
 	int index;
 	struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
@@ -267,11 +256,12 @@ struct cxl_dport *redirect_devm_cxl_add_dport_by_dev(struct cxl_port *port,
 	if (ops && ops->is_mock_port(port->uport_dev))
 		dport = ops->devm_cxl_add_dport_by_dev(port, dport_dev);
 	else
-		dport = __devm_cxl_add_dport_by_dev(port, dport_dev);
+		dport = devm_cxl_add_dport_by_dev(port, dport_dev);
 	put_cxl_mock_ops(index);
 
 	return dport;
 }
+EXPORT_SYMBOL_NS_GPL(__wrap_devm_cxl_add_dport_by_dev, "CXL");
 
 MODULE_LICENSE("GPL v2");
 MODULE_DESCRIPTION("cxl_test: emulation module");
tools/testing/cxl/Kbuild +2 -0
index 6eceefefb0e0..9b2d514a867e 100644
@@ -10,6 +10,8 @@ ldflags-y += --wrap=cxl_endpoint_parse_cdat
 ldflags-y += --wrap=cxl_dport_init_ras_reporting
 ldflags-y += --wrap=devm_cxl_endpoint_decoders_setup
 ldflags-y += --wrap=hmat_get_extended_linear_cache_size
+ldflags-y += --wrap=devm_cxl_add_dport_by_dev
+ldflags-y += --wrap=devm_cxl_switch_port_decoders_setup
 
 DRIVERS := ../../../drivers
 CXL_SRC := $(DRIVERS)/cxl
Signature · 1 line
2.52.0
#7 lore raw

PATCH v2 6/9 cxl/port: Move dport RAS setup to dport add time

In reply to Dan Williams

Towards the end goal of making all CXL RAS capability handling uniform across host bridge ports, upstream switch ports, and endpoint ports, move dport RAS setup. Move it to cxl_switch_port_probe() context for switch / VH dports (via cxl_port_add_dport()) and cxl_endpoint_port_probe() context for an RCH dport. Rename the RAS setup helper to devm_cxl_dport_ras_setup() for symmetry with devm_cxl_switch_port_decoders_setup().

Only the RCH version needs to be exported and the cxl_test mocking can be deleted with a dev_is_pci() check on the dport_dev.

---
 drivers/cxl/core/core.h       | 10 ++++++++++
 drivers/cxl/cxlpci.h          |  8 ++++----
 drivers/cxl/core/port.c       | 12 +++---------
 drivers/cxl/core/ras.c        | 30 ++++++++++++++++++------------
 drivers/cxl/mem.c             |  2 --
 drivers/cxl/port.c            | 12 ++++++++++++
 tools/testing/cxl/test/mock.c | 12 ------------
 tools/testing/cxl/Kbuild      |  1 -
 8 files changed, 47 insertions(+), 40 deletions(-)
drivers/cxl/core/core.h +10 -0
index 422531799af2..be3c7b137115 100644
@@ -144,6 +144,14 @@ int cxl_pci_get_bandwidth(struct pci_dev *pdev, struct access_coordinate *c);
 int cxl_port_get_switch_dport_bandwidth(struct cxl_port *port,
 					struct access_coordinate *c);
 
+static inline struct device *dport_to_host(struct cxl_dport *dport)
+{
+	struct cxl_port *port = dport->port;
+
+	if (is_cxl_root(port))
+		return port->uport_dev;
+	return &port->dev;
+}
 #ifdef CONFIG_CXL_RAS
 int cxl_ras_init(void);
 void cxl_ras_exit(void);
@@ -152,6 +160,7 @@ void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base);
 void cxl_dport_map_rch_aer(struct cxl_dport *dport);
 void cxl_disable_rch_root_ints(struct cxl_dport *dport);
 void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds);
+void devm_cxl_dport_ras_setup(struct cxl_dport *dport);
 #else
 static inline int cxl_ras_init(void)
 {
@@ -166,6 +175,7 @@ static inline void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base
 static inline void cxl_dport_map_rch_aer(struct cxl_dport *dport) { }
 static inline void cxl_disable_rch_root_ints(struct cxl_dport *dport) { }
 static inline void cxl_handle_rdport_errors(struct cxl_dev_state *cxlds) { }
+static inline void devm_cxl_dport_ras_setup(struct cxl_dport *dport) { }
 #endif /* CONFIG_CXL_RAS */
 
 int cxl_gpf_port_setup(struct cxl_dport *dport);
drivers/cxl/cxlpci.h +4 -4
index 6f9c78886fd9..0db3d73548aa 100644
@@ -81,7 +81,7 @@ void read_cdat_data(struct cxl_port *port);
 void cxl_cor_error_detected(struct pci_dev *pdev);
 pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
 				    pci_channel_state_t state);
-void cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host);
+void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport);
 #else
 static inline void cxl_cor_error_detected(struct pci_dev *pdev) { }
 
@@ -90,9 +90,9 @@ static inline pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
 {
 	return PCI_ERS_RESULT_NONE;
 }
-
-static inline void cxl_dport_init_ras_reporting(struct cxl_dport *dport,
-						struct device *host) { }
+static inline void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport)
+{
+}
 #endif
 
 #endif /* __CXL_PCI_H__ */
drivers/cxl/core/port.c +3 -9
index 7356e1725db8..9f56f7e75e81 100644
@@ -1119,15 +1119,6 @@ static void cxl_dport_unlink(void *data)
 	sysfs_remove_link(&port->dev.kobj, link_name);
 }
 
-static struct device *dport_to_host(struct cxl_dport *dport)
-{
-	struct cxl_port *port = dport->port;
-
-	if (is_cxl_root(port))
-		return port->uport_dev;
-	return &port->dev;
-}
-
 static void free_dport(void *dport)
 {
 	kfree(dport);
@@ -1261,6 +1252,9 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
 
 	cxl_debugfs_create_dport_dir(dport);
 
+	if (!dport->rch)
+		devm_cxl_dport_ras_setup(dport);
+
 	/* keep the group, and mark the end of devm actions */
 	cxl_dport_close_dr_group(dport, no_free_ptr(dport_dr_group));
 
drivers/cxl/core/ras.c +18 -12
index 72908f3ced77..e90b7a91bf5d 100644
@@ -139,26 +139,32 @@ static void cxl_dport_map_ras(struct cxl_dport *dport)
 }
 
 /**
- * cxl_dport_init_ras_reporting - Setup CXL RAS report on this dport
+ * devm_cxl_dport_ras_setup - Setup CXL RAS report on this dport
  * @dport: the cxl_dport that needs to be initialized
- * @host: host device for devm operations
  */
-void cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host)
+void devm_cxl_dport_ras_setup(struct cxl_dport *dport)
 {
-	dport->reg_map.host = host;
+	dport->reg_map.host = dport_to_host(dport);
 	cxl_dport_map_ras(dport);
+}
 
-	if (dport->rch) {
-		struct pci_host_bridge *host_bridge = to_pci_host_bridge(dport->dport_dev);
+void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport)
+{
+	struct pci_host_bridge *host_bridge;
 
-		if (!host_bridge->native_aer)
-			return;
+	if (!dev_is_pci(dport->dport_dev))
+		return;
 
-		cxl_dport_map_rch_aer(dport);
-		cxl_disable_rch_root_ints(dport);
-	}
+	devm_cxl_dport_ras_setup(dport);
+
+	host_bridge = to_pci_host_bridge(dport->dport_dev);
+	if (!host_bridge->native_aer)
+		return;
+
+	cxl_dport_map_rch_aer(dport);
+	cxl_disable_rch_root_ints(dport);
 }
-EXPORT_SYMBOL_NS_GPL(cxl_dport_init_ras_reporting, "CXL");
+EXPORT_SYMBOL_NS_GPL(devm_cxl_dport_rch_ras_setup, "CXL");
 
 void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
 {
drivers/cxl/mem.c +0 -2
index c2ee7f7f6320..e25c33f8c6cf 100644
@@ -166,8 +166,6 @@ static int cxl_mem_probe(struct device *dev)
 	else
 		endpoint_parent = &parent_port->dev;
 
-	cxl_dport_init_ras_reporting(dport, dev);
-
 	scoped_guard(device, endpoint_parent) {
 		if (!endpoint_parent->driver) {
 			dev_err(dev, "CXL port topology %s not enabled\n",
drivers/cxl/port.c +12 -0
index 913c469e067a..929f7e259f0d 100644
@@ -71,6 +71,7 @@ static int cxl_switch_port_probe(struct cxl_port *port)
 static int cxl_endpoint_port_probe(struct cxl_port *port)
 {
 	struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
+	struct cxl_dport *dport = port->parent_dport;
 	int rc;
 
 	/* Cache the data early to ensure is_visible() works */
@@ -86,6 +87,17 @@ static int cxl_endpoint_port_probe(struct cxl_port *port)
 	if (rc)
 		return rc;
 
+	/*
+	 * With VH (CXL Virtual Host) topology the cxl_port::add_dport() method
+	 * handles RAS setup for downstream ports. With RCH (CXL Restricted CXL
+	 * Host) topologies the downstream port is enumerated early by platform
+	 * firmware, but the RCRB (root complex register block) is not mapped
+	 * until after the cxl_pci driver attaches to the RCIeP (root complex
+	 * integrated endpoint).
+	 */
+	if (dport->rch)
+		devm_cxl_dport_rch_ras_setup(dport);
+
 	/*
 	 * Now that all endpoint decoders are successfully enumerated, try to
 	 * assemble regions from committed decoders
tools/testing/cxl/test/mock.c +0 -12
index f307c5b39184..b8fcb50c1027 100644
@@ -234,18 +234,6 @@ void __wrap_cxl_endpoint_parse_cdat(struct cxl_port *port)
 }
 EXPORT_SYMBOL_NS_GPL(__wrap_cxl_endpoint_parse_cdat, "CXL");
 
-void __wrap_cxl_dport_init_ras_reporting(struct cxl_dport *dport, struct device *host)
-{
-	int index;
-	struct cxl_mock_ops *ops = get_cxl_mock_ops(&index);
-
-	if (!ops || !ops->is_mock_port(dport->dport_dev))
-		cxl_dport_init_ras_reporting(dport, host);
-
-	put_cxl_mock_ops(index);
-}
-EXPORT_SYMBOL_NS_GPL(__wrap_cxl_dport_init_ras_reporting, "CXL");
-
 struct cxl_dport *__wrap_devm_cxl_add_dport_by_dev(struct cxl_port *port,
 						   struct device *dport_dev)
 {
tools/testing/cxl/Kbuild +0 -1
index 9b2d514a867e..982e8ea28b92 100644
@@ -7,7 +7,6 @@ ldflags-y += --wrap=nvdimm_bus_register
 ldflags-y += --wrap=cxl_await_media_ready
 ldflags-y += --wrap=devm_cxl_add_rch_dport
 ldflags-y += --wrap=cxl_endpoint_parse_cdat
-ldflags-y += --wrap=cxl_dport_init_ras_reporting
 ldflags-y += --wrap=devm_cxl_endpoint_decoders_setup
 ldflags-y += --wrap=hmat_get_extended_linear_cache_size
 ldflags-y += --wrap=devm_cxl_add_dport_by_dev
Signature · 1 line
2.52.0
#8 lore raw

PATCH v2 7/9 cxl/port: Map Port RAS registers

In reply to Dan Williams

From: Terry Bowman <[email protected]>

In preparation for CXL VH (Virtual Host) topology protocol error handling, add RAS capability registered mapping for all ports in a CXL VH topology. This includes the RAS capabilities of Switch Upstream Ports, Switch Downstream Ports, Host Bridge Ports ("upstream"), and Root Ports ("downstream")

Update cxl_port_add_dport() to map the upstream RAS capability on first 'dport' attach.

---
 drivers/cxl/cxl.h      |  2 ++
 drivers/cxl/cxlpci.h   |  4 ++++
 drivers/cxl/core/ras.c | 16 ++++++++++++++++
 drivers/cxl/port.c     |  6 ++++++
 4 files changed, 28 insertions(+)
drivers/cxl/cxl.h +2 -0
index 4479d632a687..626a37b72fc3 100644
@@ -607,6 +607,7 @@ struct cxl_dax_region {
  * @parent_dport: dport that points to this port in the parent
  * @decoder_ida: allocator for decoder ids
  * @reg_map: component and ras register mapping parameters
+ * @regs: mapped component registers
  * @nr_dports: number of entries in @dports
  * @hdm_end: track last allocated HDM decoder instance for allocation ordering
  * @commit_end: cursor to track highest committed decoder for commit ordering
@@ -628,6 +629,7 @@ struct cxl_port {
 	struct cxl_dport *parent_dport;
 	struct ida decoder_ida;
 	struct cxl_register_map reg_map;
+	struct cxl_component_regs regs;
 	int nr_dports;
 	int hdm_end;
 	int commit_end;
drivers/cxl/cxlpci.h +4 -0
index 0db3d73548aa..970add0256e9 100644
@@ -82,6 +82,7 @@ void cxl_cor_error_detected(struct pci_dev *pdev);
 pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
 				    pci_channel_state_t state);
 void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport);
+void devm_cxl_port_ras_setup(struct cxl_port *port);
 #else
 static inline void cxl_cor_error_detected(struct pci_dev *pdev) { }
 
@@ -93,6 +94,9 @@ static inline pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
 static inline void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport)
 {
 }
+static inline void devm_cxl_port_ras_setup(struct cxl_port *port)
+{
+}
 #endif
 
 #endif /* __CXL_PCI_H__ */
drivers/cxl/core/ras.c +16 -0
index e90b7a91bf5d..b4be9c5715a6 100644
@@ -166,6 +166,22 @@ void devm_cxl_dport_rch_ras_setup(struct cxl_dport *dport)
 }
 EXPORT_SYMBOL_NS_GPL(devm_cxl_dport_rch_ras_setup, "CXL");
 
+void devm_cxl_port_ras_setup(struct cxl_port *port)
+{
+	struct cxl_register_map *map = &port->reg_map;
+
+	if (!map->component_map.ras.valid) {
+		dev_dbg(&port->dev, "RAS registers not found\n");
+		return;
+	}
+
+	map->host = &port->dev;
+	if (cxl_map_component_regs(map, &port->regs,
+				   BIT(CXL_CM_CAP_CAP_ID_RAS)))
+		dev_dbg(&port->dev, "Failed to map RAS capability\n");
+}
+EXPORT_SYMBOL_NS_GPL(devm_cxl_port_ras_setup, "CXL");
+
 void cxl_handle_cor_ras(struct device *dev, void __iomem *ras_base)
 {
 	void __iomem *addr;
drivers/cxl/port.c +6 -0
index 929f7e259f0d..6ebd665fb347 100644
@@ -192,6 +192,12 @@ static struct cxl_dport *cxl_port_add_dport(struct cxl_port *port,
 		rc = devm_cxl_switch_port_decoders_setup(port);
 		if (rc)
 			return ERR_PTR(rc);
+
+		/*
+		 * RAS setup is optional, either driver operation can continue
+		 * on failure, or the device does not implement RAS registers.
+		 */
+		devm_cxl_port_ras_setup(port);
 	}
 
 	dport = devm_cxl_add_dport_by_dev(port, dport_dev);
Signature · 1 line
2.52.0
#9 lore raw

PATCH v2 8/9 cxl/port: Move endpoint component register management to cxl_port

In reply to Dan Williams

In preparation for generic protocol error handling across CXL endpoints, whether they be memory expander class devices or accelerators, drop the endpoint component management from cxl_dev_state.

Organize all CXL port component management through the common cxl_port driver.

Note that the end game is that drivers/cxl/core/ras.c loses all dependencies on a 'struct cxl_dev_state' parameter and operates only on port resources. The removal of component register mapping from cxl_pci is an incremental step towards that.

---
 drivers/cxl/cxlmem.h   |  4 +--
 drivers/cxl/core/ras.c |  6 ++--
 drivers/cxl/pci.c      | 63 +-----------------------------------------
 drivers/cxl/port.c     | 54 ++++++++++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 67 deletions(-)
drivers/cxl/cxlmem.h +2 -2
index 434031a0c1f7..ab7201ef3ea6 100644
@@ -415,7 +415,7 @@ struct cxl_dpa_partition {
  * @dev: The device associated with this CXL state
  * @cxlmd: The device representing the CXL.mem capabilities of @dev
  * @reg_map: component and ras register mapping parameters
- * @regs: Parsed register blocks
+ * @regs: Class device "Device" registers
  * @cxl_dvsec: Offset to the PCIe device DVSEC
  * @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
  * @media_ready: Indicate whether the device media is usable
@@ -431,7 +431,7 @@ struct cxl_dev_state {
 	struct device *dev;
 	struct cxl_memdev *cxlmd;
 	struct cxl_register_map reg_map;
-	struct cxl_regs regs;
+	struct cxl_device_regs regs;
 	int cxl_dvsec;
 	bool rcd;
 	bool media_ready;
drivers/cxl/core/ras.c +3 -3
index b4be9c5715a6..f6a8f4a355f1 100644
@@ -255,6 +255,7 @@ bool cxl_handle_ras(struct device *dev, void __iomem *ras_base)
 void cxl_cor_error_detected(struct pci_dev *pdev)
 {
 	struct cxl_dev_state *cxlds = pci_get_drvdata(pdev);
+	struct cxl_memdev *cxlmd = cxlds->cxlmd;
 	struct device *dev = &cxlds->cxlmd->dev;
 
 	scoped_guard(device, dev) {
@@ -268,7 +269,7 @@ void cxl_cor_error_detected(struct pci_dev *pdev)
 		if (cxlds->rcd)
 			cxl_handle_rdport_errors(cxlds);
 
-		cxl_handle_cor_ras(&cxlds->cxlmd->dev, cxlds->regs.ras);
+		cxl_handle_cor_ras(&cxlds->cxlmd->dev, cxlmd->endpoint->regs.ras);
 	}
 }
 EXPORT_SYMBOL_NS_GPL(cxl_cor_error_detected, "CXL");
@@ -297,10 +298,9 @@ pci_ers_result_t cxl_error_detected(struct pci_dev *pdev,
 		 * chance the situation is recoverable dump the status of the RAS
 		 * capability registers and bounce the active state of the memdev.
 		 */
-		ue = cxl_handle_ras(&cxlds->cxlmd->dev, cxlds->regs.ras);
+		ue = cxl_handle_ras(&cxlds->cxlmd->dev, cxlmd->endpoint->regs.ras);
 	}
 
-
 	switch (state) {
 	case pci_channel_io_normal:
 		if (ue) {
drivers/cxl/pci.c +1 -62

93 lines not on this page · raw message

drivers/cxl/port.c +54 -0

71 lines not on this page · raw message

Signature · 1 line
2.52.0
#10 lore raw

PATCH v2 9/9 cxl/port: Unify endpoint and switch port lookup

In reply to Dan Williams

In support of generic CXL protocol error handling across various 'struct cxl_port' types, update find_cxl_port_by_uport() to retrieve endpoint CXL port companions from endpoint PCIe device instances.

The end result is that upstream switch ports and endpoint ports can share error handling and eventually delete the misplaced cxl_error_handlers from the cxl_pci class driver.

---
 drivers/cxl/core/port.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)
drivers/cxl/core/port.c +11 -1
index 9f56f7e75e81..ee7d14528867 100644
@@ -1590,10 +1590,20 @@ static int match_port_by_uport(struct device *dev, const void *data)
 		return 0;
 
 	port = to_cxl_port(dev);
+	/* Endpoint ports are hosted by memdevs */
+	if (is_cxl_memdev(port->uport_dev))
+		return uport_dev == port->uport_dev->parent;
 	return uport_dev == port->uport_dev;
 }
 
-/*
+/**
+ * find_cxl_port_by_uport - Find a CXL port device companion
+ * @uport_dev: Device that acts as a switch or endpoint in the CXL hierarchy
+ *
+ * In the case of endpoint ports recall that port->uport_dev points to a 'struct
+ * cxl_memdev' device. So, the @uport_dev argument is the parent device of the
+ * 'struct cxl_memdev' in that case.
+ *
  * Function takes a device reference on the port device. Caller should do a
  * put_device() when done.
  */
Signature · 1 line
2.52.0
Tested-by on the v2 series
#11 lore raw
In reply to Dan Williams
Quoted 106 lines · Dan Williams

Changes since v1 [1]:
- Cleanup the diff by keeping the order of dport_exists() relative to the dev->driver check in cxl_port_add_dport() (Jonathan)
- Drop some repetitive de-referencing with a local @port variable in dport_to_host() (Jonathan)
- s/group/dr_group/ throughout to clarify "devres group" (Jonathan)
- Reuse del_dport() for cxl_dport_release_dr_group() (Jonathan)
- Drop the thin wrappers for devres_{open,close}_group() for the port group.
- Add a comment and a cleanup TODO for the 'add_dport()' operation in 'struct cxl_driver'. (Jonathan)
- Change patch 7 subject to just: "cxl/port: Map Port RAS registers" since it is only introducing a helper that is later used in the endpoint case. (Jonathan)

[1]: http://lore.kernel.org/[email protected]

Original cover letter:

---

The CXL Port Protocol error handling series grew to be over 30 patches which is too much to handle at once given the various topics involved. One of the sub-threads of the v14 review was confusion about the new devres groups to manage port setup unwind failures [2].

[2]: http://lore.kernel.org/[email protected]

Given that review indicated a need to break up and better explain the conversion, do that in a separate patch set. Build on top of the first 18 patches of that series [3] that are ready to merge.

[3]: https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-7.0/cxl-aer-prep

The wider goals of the port protocol handling series are:

1/ Be minimally invasive to the ongoing maintenance burden of PCIe error
handling. Just do the minimal enlightenment to forward "internal" errors for device with active CXL links to the CXL core.

2/ Build a framework for any driver that registers a 'struct cxl_memdev'
(or in the future a 'struct cxl_cachedev') gets protocol error handling support.

This "Unify RAS setup across port types" set supports goal 2/. It enables a model where all CXL error handling is relative to the common 'struct cxl_port' and 'struct cxl_dport' objects and is agnostic to whether those objects are in support of the memory expansion class device (driven by cxl_pci) or any other CXL endpoint in the system that supports CXL.cachemem operation.

In support of that unification, the setup of RAS registers needs to be centralized. That in turn requires new handling for early exit setup failures and additional teardown support for resources optionally acquired at port / dport creation time.

The devres group mechanism is deployed to cleanup some open coded devm_release_action() calls. The devres group facility also comes in handy for unwinding conditional setup steps in the port creation process. Recall that ports defer probing their CXL resources until after they are known to have a downstream CXL connection. So, early exit during setup of a new dport may have more or less work to do depending on whether the first or subsequent dport is being added.

Given probing port resources is a 'probe' action it fits more naturally as a driver operation. If cxl_port_add_dport() then moves to cxl_port driver operation alongside ->probe(), it enables a cxl_test cleanup. The cxl_test approach has a hard time mocking interfaces that are internal to the cxl_core.

The rest of the patches in this set finish off the conversion of 'struct cxl_port' and 'struct cxl_dport' to be the only CXL objects that interact with the CXL RAS.

Dan Williams (8):
cxl/port: Cleanup handling of the nr_dports 0 -> 1 transition cxl/port: Reduce number of @dport variables in cxl_port_add_dport() cxl/port: Cleanup dport removal with a devres group cxl/port: Move decoder setup before dport creation cxl/port: Move dport probe operations to a driver event cxl/port: Move dport RAS setup to dport add time cxl/port: Move endpoint component register management to cxl_port cxl/port: Unify endpoint and switch port lookup

Terry Bowman (1):
cxl/port: Map Port RAS registers

 drivers/cxl/core/core.h              |  10 ++
 drivers/cxl/cxl.h                    |  33 +++---
 drivers/cxl/cxlmem.h                 |   4 +-
 drivers/cxl/cxlpci.h                 |  12 +-
 tools/testing/cxl/exports.h          |  13 ---
 drivers/cxl/core/hdm.c               |   6 +-
 drivers/cxl/core/pci.c               |   8 +-
 drivers/cxl/core/port.c              | 159 +++++++++++++++++----------
 drivers/cxl/core/ras.c               |  50 ++++++---
 drivers/cxl/mem.c                    |   2 -
 drivers/cxl/pci.c                    |  63 +----------
 drivers/cxl/port.c                   | 122 ++++++++++++++++++++
 tools/testing/cxl/cxl_core_exports.c |  22 ----
 tools/testing/cxl/test/mock.c        |  36 ++----
 tools/testing/cxl/Kbuild             |   3 +-
 15 files changed, 310 insertions(+), 233 deletions(-)
 delete mode 100644 tools/testing/cxl/exports.h
Quoted 1 line

base-commit: 9a8920ca8ebfb99604f639e7fbc681d0d04518a0

Hi Dan,

I rebased my CXL port error handling series onto your series and completed testing with no issues.

I tested using a single‑switch topology consisting of one root port (RP), one upstream port (USP), four downstream ports (DSPs), and four endpoints (EPs). Testing followed the error‑handling series test procedure. I injected AER errors using the aer_inject tool and simulated RAS error presence with a debug patch.

All CXL devices were correctly discovered and populated in the topology. Protocol error tests passed on each CXL device, and I did not observe any regressions.

Please feel free to include my tested sign‑off:

-Terry

maintainer Applied
#12 lore raw
In reply to Dan Williams
Quoted 15 lines · Dan Williams

Changes since v1 [1]:
- Cleanup the diff by keeping the order of dport_exists() relative to the dev->driver check in cxl_port_add_dport() (Jonathan)
- Drop some repetitive de-referencing with a local @port variable in dport_to_host() (Jonathan)
- s/group/dr_group/ throughout to clarify "devres group" (Jonathan)
- Reuse del_dport() for cxl_dport_release_dr_group() (Jonathan)
- Drop the thin wrappers for devres_{open,close}_group() for the port group.
- Add a comment and a cleanup TODO for the 'add_dport()' operation in 'struct cxl_driver'. (Jonathan)
- Change patch 7 subject to just: "cxl/port: Map Port RAS registers" since it is only introducing a helper that is later used in the endpoint case. (Jonathan)

Quoted 1 line

[1]: http://lore.kernel.org/[email protected]

Applied to cxl/next
0da3050bdded5f121aaca6b5247ea50681d7129e

Quoted 91 lines

Original cover letter:

---

The CXL Port Protocol error handling series grew to be over 30 patches which is too much to handle at once given the various topics involved. One of the sub-threads of the v14 review was confusion about the new devres groups to manage port setup unwind failures [2].

[2]: http://lore.kernel.org/[email protected]

Given that review indicated a need to break up and better explain the conversion, do that in a separate patch set. Build on top of the first 18 patches of that series [3] that are ready to merge.

[3]: https://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl.git/log/?h=for-7.0/cxl-aer-prep

The wider goals of the port protocol handling series are:

1/ Be minimally invasive to the ongoing maintenance burden of PCIe error
handling. Just do the minimal enlightenment to forward "internal" errors for device with active CXL links to the CXL core.

2/ Build a framework for any driver that registers a 'struct cxl_memdev'
(or in the future a 'struct cxl_cachedev') gets protocol error handling support.

This "Unify RAS setup across port types" set supports goal 2/. It enables a model where all CXL error handling is relative to the common 'struct cxl_port' and 'struct cxl_dport' objects and is agnostic to whether those objects are in support of the memory expansion class device (driven by cxl_pci) or any other CXL endpoint in the system that supports CXL.cachemem operation.

In support of that unification, the setup of RAS registers needs to be centralized. That in turn requires new handling for early exit setup failures and additional teardown support for resources optionally acquired at port / dport creation time.

The devres group mechanism is deployed to cleanup some open coded devm_release_action() calls. The devres group facility also comes in handy for unwinding conditional setup steps in the port creation process. Recall that ports defer probing their CXL resources until after they are known to have a downstream CXL connection. So, early exit during setup of a new dport may have more or less work to do depending on whether the first or subsequent dport is being added.

Given probing port resources is a 'probe' action it fits more naturally as a driver operation. If cxl_port_add_dport() then moves to cxl_port driver operation alongside ->probe(), it enables a cxl_test cleanup. The cxl_test approach has a hard time mocking interfaces that are internal to the cxl_core.

The rest of the patches in this set finish off the conversion of 'struct cxl_port' and 'struct cxl_dport' to be the only CXL objects that interact with the CXL RAS.

Dan Williams (8):
cxl/port: Cleanup handling of the nr_dports 0 -> 1 transition cxl/port: Reduce number of @dport variables in cxl_port_add_dport() cxl/port: Cleanup dport removal with a devres group cxl/port: Move decoder setup before dport creation cxl/port: Move dport probe operations to a driver event cxl/port: Move dport RAS setup to dport add time cxl/port: Move endpoint component register management to cxl_port cxl/port: Unify endpoint and switch port lookup

Terry Bowman (1):
cxl/port: Map Port RAS registers

 drivers/cxl/core/core.h              |  10 ++
 drivers/cxl/cxl.h                    |  33 +++---
 drivers/cxl/cxlmem.h                 |   4 +-
 drivers/cxl/cxlpci.h                 |  12 +-
 tools/testing/cxl/exports.h          |  13 ---
 drivers/cxl/core/hdm.c               |   6 +-
 drivers/cxl/core/pci.c               |   8 +-
 drivers/cxl/core/port.c              | 159 +++++++++++++++++----------
 drivers/cxl/core/ras.c               |  50 ++++++---
 drivers/cxl/mem.c                    |   2 -
 drivers/cxl/pci.c                    |  63 +----------
 drivers/cxl/port.c                   | 122 ++++++++++++++++++++
 tools/testing/cxl/cxl_core_exports.c |  22 ----
 tools/testing/cxl/test/mock.c        |  36 ++----
 tools/testing/cxl/Kbuild             |   3 +-
 15 files changed, 310 insertions(+), 233 deletions(-)
 delete mode 100644 tools/testing/cxl/exports.h

base-commit: 9a8920ca8ebfb99604f639e7fbc681d0d04518a0