Skip to contentSkip to conversation

CXL core reorganization

PATCH Superseded Ben Widawsky 2 participants 14 messages Jul 15, 2021 – Jul 28, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover CXL core reorganization

The main motivation of the patch series is to establish the cxl_core driver in its own directory and modularize it. Specifically, the patch series aims to achieve three things:
1. Move existing core functionality to a new directory.
2. Split existing core functionality into multiple files.
3. Migrate memdev functionality into core.

#1 is trivially accomplished with git mv. The file itself is renamed back to bus.c since the goal is to break up core functionality into multiple files, and so the name core.c doesn't make sense in that context.

#2 is also trivially accomplished via cut/paste.

#3 is slightly invasive in that it has certain functional changes to improve the existing interfaces and make them more generic. The rest of the change is cut/paste. This is also the only part of the series which has runtime functional change in that some interfaces are removed from cxl_pci, moved into cxl_core, and exported for other drivers to use.

Ben Widawsky (6):
cxl: Move cxl_core to new directory cxl/core: Improve CXL core kernel docs cxl/core: Extract register and pmem functionality cxl/mem: Move character device region creation
cxl: Pass fops and shutdown to memdev creation cxl/core: Move memdev management to core

 .../driver-api/cxl/memory-devices.rst         |   8 +-
 drivers/cxl/Makefile                          |   2 +-
 drivers/cxl/{core.c => core/bus.c}            | 458 +-----------------
 drivers/cxl/core/core.h                       |  20 +
 drivers/cxl/core/memdev.c                     | 224 +++++++++
 drivers/cxl/core/pmem.c                       | 201 ++++++++
 drivers/cxl/core/regs.c                       | 235 +++++++++
 drivers/cxl/mem.h                             |   9 +
 drivers/cxl/pci.c                             | 239 +--------
 9 files changed, 731 insertions(+), 665 deletions(-)
 rename drivers/cxl/{core.c => core/bus.c} (58%)
 create mode 100644 drivers/cxl/core/core.h
 create mode 100644 drivers/cxl/core/memdev.c
 create mode 100644 drivers/cxl/core/pmem.c
 create mode 100644 drivers/cxl/core/regs.c
Signature · 1 line
2.32.0
#1 lore raw

PATCH 1/6 cxl: Move cxl_core to new directory

CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.

---
 Documentation/driver-api/cxl/memory-devices.rst | 2 +-
 drivers/cxl/Makefile                            | 2 +-
 drivers/cxl/{core.c => core/bus.c}              | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)
 rename drivers/cxl/{core.c => core/bus.c} (99%)
Documentation/driver-api/cxl/memory-devices.rst +1 -1
index 487ce4f41d77..a86e2c7c551a 100644
@@ -36,7 +36,7 @@ CXL Core
 .. kernel-doc:: drivers/cxl/cxl.h
    :internal:
 
-.. kernel-doc:: drivers/cxl/core.c
+.. kernel-doc:: drivers/cxl/core/bus.c
    :doc: cxl core
 
 External Interfaces
drivers/cxl/Makefile +1 -1
index 32954059b37b..ad5a4f8f4511 100644
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
-cxl_core-y := core.o
+cxl_core-y := core/bus.o
 cxl_pci-y := pci.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +2 -2
similarity index 99%
rename from drivers/cxl/core.c
rename to drivers/cxl/core/bus.c
index a2e4d54fc7bc..00b759ff92d3 100644
@@ -6,8 +6,8 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/idr.h>
-#include "cxl.h"
-#include "mem.h"
+#include "../cxl.h"
+#include "../mem.h"
 
 /**
  * DOC: cxl core
Signature · 1 line
2.32.0
#2 lore raw

Re: [PATCH 1/6] cxl: Move cxl_core to new directory

Quoted 6 lines · Ben Widawsky

CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.

I think this effort wants a drivers/cxl/core/Makefile, more below:

Quoted 26 lines
---
 Documentation/driver-api/cxl/memory-devices.rst | 2 +-
 drivers/cxl/Makefile                            | 2 +-
 drivers/cxl/{core.c => core/bus.c}              | 4 ++--
 3 files changed, 4 insertions(+), 4 deletions(-)
 rename drivers/cxl/{core.c => core/bus.c} (99%)
Documentation/driver-api/cxl/memory-devices.rst +1 -1
index 487ce4f41d77..a86e2c7c551a 100644
@@ -36,7 +36,7 @@ CXL Core
 .. kernel-doc:: drivers/cxl/cxl.h
    :internal:
 
-.. kernel-doc:: drivers/cxl/core.c
+.. kernel-doc:: drivers/cxl/core/bus.c
    :doc: cxl core
 
 External Interfaces
drivers/cxl/Makefile +0 -0
index 32954059b37b..ad5a4f8f4511 100644

drivers/cxl/Makefile obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o · line 7

Quoted 3 lines
drivers/cxl/Makefile +0 -0
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL

Only the core has exports so this is not relevant for the drivers (pci, acpi, pmem...)

drivers/cxl/Makefile obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o · line 8

Quoted 2 lines
drivers/cxl/Makefile +1 -1
-cxl_core-y := core.o
+cxl_core-y := core/bus.o

Not typical to include directory names in Makefiles.

Quoted 11 lines
drivers/cxl/Makefile +0 -0
 cxl_pci-y := pci.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +0 -0
similarity index 99%
rename from drivers/cxl/core.c
rename to drivers/cxl/core/bus.c
index a2e4d54fc7bc..00b759ff92d3 100644

drivers/cxl/core/bus.c line 10

Quoted 7 lines
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +2 -2
@@ -6,8 +6,8 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/idr.h>
-#include "cxl.h"
-#include "mem.h"
+#include "../cxl.h"
+#include "../mem.h"

I'd prefer <cxl.h> and <mem.h>.

Proposed changes to fold:

drivers/cxl/Makefile +1 -3
index ad5a4f8f4511..d1aaabc940f3 100644
@@ -1,11 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_CXL_BUS) += cxl_core.o
+obj-$(CONFIG_CXL_BUS) += core/
 obj-$(CONFIG_CXL_MEM) += cxl_pci.o
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 
-ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
-cxl_core-y := core/bus.o
 cxl_pci-y := pci.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o
drivers/cxl/core/Makefile new file +5 -0
new file mode 100644
index 000000000000..c65e9f61abe9
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS) += cxl_core.o
+
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -I./drivers/cxl
+cxl_core-y := bus.o
drivers/cxl/core/bus.c +2 -2
index 00b759ff92d3..6ce04e3976d2 100644
@@ -6,8 +6,8 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/idr.h>
-#include "../cxl.h"
-#include "../mem.h"
+#include <cxl.h>
+#include <mem.h>
 
 /**
  * DOC: cxl core
#3 lore raw

PATCH v2 1/6 cxl: Move cxl_core to new directory

CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.

---
v2: Create new Makefile for core (Dan)

---
 Documentation/driver-api/cxl/memory-devices.rst | 2 +-
 drivers/cxl/Makefile                            | 4 +---
 drivers/cxl/core/Makefile                       | 5 +++++
 drivers/cxl/{core.c => core/bus.c}              | 4 ++--
 4 files changed, 9 insertions(+), 6 deletions(-)
 create mode 100644 drivers/cxl/core/Makefile
 rename drivers/cxl/{core.c => core/bus.c} (99%)
Documentation/driver-api/cxl/memory-devices.rst +1 -1
index 487ce4f41d77..a86e2c7c551a 100644
@@ -36,7 +36,7 @@ CXL Core
 .. kernel-doc:: drivers/cxl/cxl.h
    :internal:
 
-.. kernel-doc:: drivers/cxl/core.c
+.. kernel-doc:: drivers/cxl/core/bus.c
    :doc: cxl core
 
 External Interfaces
drivers/cxl/Makefile +1 -3
index 32954059b37b..d1aaabc940f3 100644
@@ -1,11 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_CXL_BUS) += cxl_core.o
+obj-$(CONFIG_CXL_BUS) += core/
 obj-$(CONFIG_CXL_MEM) += cxl_pci.o
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 
-ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
-cxl_core-y := core.o
 cxl_pci-y := pci.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o
drivers/cxl/core/Makefile new file +5 -0
new file mode 100644
index 000000000000..c65e9f61abe9
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_BUS) += cxl_core.o
+
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -I./drivers/cxl
+cxl_core-y := bus.o
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +2 -2
similarity index 99%
rename from drivers/cxl/core.c
rename to drivers/cxl/core/bus.c
index a2e4d54fc7bc..6ce04e3976d2 100644
@@ -6,8 +6,8 @@
 #include <linux/pci.h>
 #include <linux/slab.h>
 #include <linux/idr.h>
-#include "cxl.h"
-#include "mem.h"
+#include <cxl.h>
+#include <mem.h>
 
 /**
  * DOC: cxl core
Signature · 1 line
2.32.0
#4 lore raw

Re: [PATCH v2 1/6] cxl: Move cxl_core to new directory

Quoted 9 lines · Ben Widawsky

CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.

---
v2: Create new Makefile for core (Dan)

Dan, there's some minor conflicts as a result of this change. Let me know if you'd like me to resend the series.

[snip]

#5 lore raw

PATCH 2/6 cxl/core: Improve CXL core kernel docs

In reply to Ben Widawsky

Now that CXL core's role is well understood, the documentation should reflect that information.

---
 drivers/cxl/core/bus.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/cxl/core/bus.c +9 -2
index 00b759ff92d3..f50872e8e7af 100644
@@ -12,8 +12,15 @@
 /**
  * DOC: cxl core
  *
- * The CXL core provides a sysfs hierarchy for control devices and a rendezvous
- * point for cross-device interleave coordination through cxl ports.
+ * The CXL core provides a set of interfaces that can be consumed by CXL aware
+ * drivers. The interfaces allow for creation, modification, and destruction of
+ * regions, memory devices, ports, and decoders. CXL aware drivers must register
+ * with the CXL core via these interfaces in order to be able to participate in
+ * cross-device interleave coordination. The CXL core also establishes and
+ * maintains the bridge to the nvdimm subsystem.
+ *
+ * CXL core introduces sysfs hierarchy to control the devices that are
+ * instantiated by the core.
  */
 
 static DEFINE_IDA(cxl_port_ida);
Signature · 1 line
2.32.0
Reviewed-by on v1 2/6
#6 lore raw

Re: [PATCH 2/6] cxl/core: Improve CXL core kernel docs

Quoted 3 lines · Ben Widawsky

Now that CXL core's role is well understood, the documentation should reflect that information.

Looks good to me.

Quoted 31 lines
---
 drivers/cxl/core/bus.c | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/cxl/core/bus.c +9 -3
index 00b759ff92d3..f50872e8e7af 100644
@@ -12,8 +12,15 @@
 /**
  * DOC: cxl core
  *
- * The CXL core provides a sysfs hierarchy for control devices and a rendezvous
- * point for cross-device interleave coordination through cxl ports.
+ * The CXL core provides a set of interfaces that can be consumed by CXL aware
+ * drivers. The interfaces allow for creation, modification, and destruction of
+ * regions, memory devices, ports, and decoders. CXL aware drivers must register
+ * with the CXL core via these interfaces in order to be able to participate in
+ * cross-device interleave coordination. The CXL core also establishes and
+ * maintains the bridge to the nvdimm subsystem.
+ *
+ * CXL core introduces sysfs hierarchy to control the devices that are
+ * instantiated by the core.
  */
 
 static DEFINE_IDA(cxl_port_ida);
--

2.32.0

#7 lore raw

PATCH 3/6 cxl/core: Extract register and pmem functionality

In reply to Ben Widawsky

Register mapping and pmem/nvdimm integration are distinct enough from basic CXL bus functionality that it warrants being moved out of bus.c Additionally, this aims to modularize for the sake of reducing the size of bus.c

pmem and register programming have very clear separation and are done together for that reason. Other parts of core, like ports and decoders should be pulled out as well, but those are more integrated with core and therefore saved for later.

---
 .../driver-api/cxl/memory-devices.rst         |   6 +
 drivers/cxl/Makefile                          |   2 +-
 drivers/cxl/core/bus.c                        | 435 +-----------------
 drivers/cxl/core/core.h                       |  20 +
 drivers/cxl/core/pmem.c                       | 201 ++++++++
 drivers/cxl/core/regs.c                       | 235 ++++++++++
 6 files changed, 466 insertions(+), 433 deletions(-)
 create mode 100644 drivers/cxl/core/core.h
 create mode 100644 drivers/cxl/core/pmem.c
 create mode 100644 drivers/cxl/core/regs.c
Documentation/driver-api/cxl/memory-devices.rst +6 -0
index a86e2c7c551a..46847d8c70a0 100644
@@ -39,6 +39,12 @@ CXL Core
 .. kernel-doc:: drivers/cxl/core/bus.c
    :doc: cxl core
 
+.. kernel-doc:: drivers/cxl/core/pmem.c
+   :internal:
+
+.. kernel-doc:: drivers/cxl/core/regs.c
+   :internal:
+
 External Interfaces
 ===================
 
drivers/cxl/Makefile +1 -1
index ad5a4f8f4511..607d8e6ab199 100644
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
-cxl_core-y := core/bus.o
+cxl_core-y := core/bus.o core/pmem.o core/regs.o
 cxl_pci-y := pci.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o
drivers/cxl/core/bus.c +3 -432

465 lines not on this page · raw message

drivers/cxl/core/core.h new file +20 -0
new file mode 100644
index 000000000000..74011c40801d
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. */
+
+#ifndef __CXL_CORE_H__
+#define __CXL_CORE_H__
+
+#include "../cxl.h"
+#include "../mem.h"
+
+extern const struct device_type cxl_nvdimm_bridge_type;
+extern const struct device_type cxl_nvdimm_type;
+
+extern struct attribute_group cxl_base_attribute_group;
+
+static inline void unregister_cxl_dev(void *dev)
+{
+	device_unregister(dev);
+}
+
+#endif /* __CXL_CORE_H__ */
drivers/cxl/core/pmem.c new file +201 -0

201 lines not on this page · raw message

drivers/cxl/core/regs.c new file +235 -0

235 lines not on this page · raw message

Signature · 1 line
2.32.0
#8 lore raw

Re: [PATCH 3/6] cxl/core: Extract register and pmem functionality

In general "and" in the patch title is a red flag that the patch can be split. I'll go ahead and do that when folding in the collision fixes.

Quoted 24 lines · Ben Widawsky

Register mapping and pmem/nvdimm integration are distinct enough from basic CXL bus functionality that it warrants being moved out of bus.c Additionally, this aims to modularize for the sake of reducing the size of bus.c

pmem and register programming have very clear separation and are done together for that reason. Other parts of core, like ports and decoders should be pulled out as well, but those are more integrated with core and therefore saved for later.

---
 .../driver-api/cxl/memory-devices.rst         |   6 +
 drivers/cxl/Makefile                          |   2 +-
 drivers/cxl/core/bus.c                        | 435 +-----------------
 drivers/cxl/core/core.h                       |  20 +
 drivers/cxl/core/pmem.c                       | 201 ++++++++
 drivers/cxl/core/regs.c                       | 235 ++++++++++
 6 files changed, 466 insertions(+), 433 deletions(-)
 create mode 100644 drivers/cxl/core/core.h
 create mode 100644 drivers/cxl/core/pmem.c
 create mode 100644 drivers/cxl/core/regs.c

[..]

Quoted 6 lines
drivers/cxl/core/core.h new file +0 -0
new file mode 100644
index 000000000000..74011c40801d

drivers/cxl/core/core.h line 8

Quoted 8 lines
drivers/cxl/core/core.h new file +8 -0
@@ -0,0 +1,20 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2020 Intel Corporation. */
+
+#ifndef __CXL_CORE_H__
+#define __CXL_CORE_H__
+
+#include "../cxl.h"
+#include "../mem.h"

This results in unnecessary includes. Each compilation unit should only include the headers it absolutely needs. In this case, making each compilation unit responsible for its own includes means that bus.c does not need mem.h, and regs.c does not need either cxl.h or core.h

#9 lore raw

PATCH 4/6 cxl/mem: Move character device region creation

In reply to Ben Widawsky

In order to merge the memdev specific functionality of cxl_pci into core, the character device creation currently handled by cxl_pci need to be moved. The rest of the changes are largely cut/paste, the actual functional change is done here in preparation.

---
 drivers/cxl/core/bus.c | 12 ++++++++++++
 drivers/cxl/mem.h      |  2 ++
 drivers/cxl/pci.c      | 14 +-------------
 3 files changed, 15 insertions(+), 13 deletions(-)
drivers/cxl/core/bus.c +12 -0
index 33196813ebcb..8c2351c52d2b 100644
@@ -22,6 +22,8 @@
  * instantiated by the core.
  */
 
+int cxl_mem_major;
+EXPORT_SYMBOL_GPL(cxl_mem_major);
 static DEFINE_IDA(cxl_port_ida);
 
 static ssize_t devtype_show(struct device *dev, struct device_attribute *attr,
@@ -632,12 +634,22 @@ EXPORT_SYMBOL_GPL(cxl_bus_type);
 
 static __init int cxl_core_init(void)
 {
+	dev_t devt;
+	int rc;
+
+	rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
+	if (rc)
+		return rc;
+
+	cxl_mem_major = MAJOR(devt);
+
 	return bus_register(&cxl_bus_type);
 }
 
 static void cxl_core_exit(void)
 {
 	bus_unregister(&cxl_bus_type);
+	unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
 }
 
 module_init(cxl_core_init);
drivers/cxl/mem.h +2 -0
index 8f02d02b26b4..2092f86beeb8 100644
@@ -34,6 +34,8 @@
  */
 #define CXL_MEM_MAX_DEVS 65536
 
+extern int cxl_mem_major;
+
 /**
  * struct cxl_memdev - CXL bus object representing a Type-3 Memory Device
  * @dev: driver core device object
drivers/cxl/pci.c +1 -13
index fe240e1504d5..6f10b19c9c83 100644
@@ -94,7 +94,6 @@ struct mbox_cmd {
 #define CXL_MBOX_SUCCESS 0
 };
 
-static int cxl_mem_major;
 static DEFINE_IDA(cxl_memdev_ida);
 static DECLARE_RWSEM(cxl_memdev_rwsem);
 static struct dentry *cxl_debugfs;
@@ -1640,25 +1639,15 @@ static struct pci_driver cxl_mem_driver = {
 static __init int cxl_mem_init(void)
 {
 	struct dentry *mbox_debugfs;
-	dev_t devt;
 	int rc;
 
 	/* Double check the anonymous union trickery in struct cxl_regs */
 	BUILD_BUG_ON(offsetof(struct cxl_regs, memdev) !=
 		     offsetof(struct cxl_regs, device_regs.memdev));
 
-	rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
-	if (rc)
-		return rc;
-
-	cxl_mem_major = MAJOR(devt);
-
 	rc = pci_register_driver(&cxl_mem_driver);
-	if (rc) {
-		unregister_chrdev_region(MKDEV(cxl_mem_major, 0),
-					 CXL_MEM_MAX_DEVS);
+	if (rc)
 		return rc;
-	}
 
 	cxl_debugfs = debugfs_create_dir("cxl", NULL);
 	mbox_debugfs = debugfs_create_dir("mbox", cxl_debugfs);
@@ -1672,7 +1661,6 @@ static __exit void cxl_mem_exit(void)
 {
 	debugfs_remove_recursive(cxl_debugfs);
 	pci_unregister_driver(&cxl_mem_driver);
-	unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
 }
 
 MODULE_LICENSE("GPL v2");
Signature · 1 line
2.32.0
#10 lore raw

Re: [PATCH 4/6] cxl/mem: Move character device region creation

Quoted 18 lines · Ben Widawsky

In order to merge the memdev specific functionality of cxl_pci into core, the character device creation currently handled by cxl_pci need to be moved. The rest of the changes are largely cut/paste, the actual functional change is done here in preparation.

---
 drivers/cxl/core/bus.c | 12 ++++++++++++
 drivers/cxl/mem.h      |  2 ++
 drivers/cxl/pci.c      | 14 +-------------
 3 files changed, 15 insertions(+), 13 deletions(-)
drivers/cxl/core/bus.c +0 -0
index 33196813ebcb..8c2351c52d2b 100644

drivers/cxl/core/bus.c line 26

Quoted 5 lines
drivers/cxl/core/bus.c +2 -0
@@ -22,6 +22,8 @@
  * instantiated by the core.
  */
 
+int cxl_mem_major;
+EXPORT_SYMBOL_GPL(cxl_mem_major);

This symbol export is necessary here, but it's not needed by the end of the series which leads me to question this fine grained split. I think the shutdown patch is suitable to split, but this one and the last can be squashed.

Quoted 5 lines
drivers/cxl/core/bus.c +0 -0
 static DEFINE_IDA(cxl_port_ida);
 
 static ssize_t devtype_show(struct device *dev, struct device_attribute *attr,
@@ -632,12 +634,22 @@ EXPORT_SYMBOL_GPL(cxl_bus_type);
 

drivers/cxl/core/bus.c EXPORT_SYMBOL_GPL(cxl_bus_type); · line 644

Quoted 10 lines
drivers/cxl/core/bus.c +8 -0
 static __init int cxl_core_init(void)
 {
+       dev_t devt;
+       int rc;
+
+       rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
+       if (rc)
+               return rc;
+
+       cxl_mem_major = MAJOR(devt);

I'd rather have cxl_core_{init,exit}() call something like cxl_memdev_{init,exit}() that way cxl_mem_major can become static again.

#11 lore raw

PATCH 5/6 cxl: Pass fops and shutdown to memdev creation

In reply to Ben Widawsky

Drivers that use cxl_core for registering a cxl_memdev will likely require synchronization with the core for shutdown so as to not race against the device going away. The main example currently is with the ioctl interface. Through the duration of the ioctl it's expected that the underlying memdev will not disappear.

Additionally, it may be desirable to have the fops be passed along as well for drivers which do not want the standard handler for the character device's ioctl.

As memdev is being migrated to core, this separation must be made.

---
 drivers/cxl/mem.h |  2 ++
 drivers/cxl/pci.c | 23 +++++++++++++++--------
 2 files changed, 17 insertions(+), 8 deletions(-)
drivers/cxl/mem.h +2 -0
index 2092f86beeb8..2b7481376621 100644
@@ -42,12 +42,14 @@ extern int cxl_mem_major;
  * @cdev: char dev core object for ioctl operations
  * @cxlm: pointer to the parent device driver data
  * @id: id number of this memdev instance.
+ * @shutdown: Optional function to call on memory device shutdown.
  */
 struct cxl_memdev {
 	struct device dev;
 	struct cdev cdev;
 	struct cxl_mem *cxlm;
 	int id;
+	void (*shutdown)(struct cxl_memdev *cxlmd);
 };
 
 /**
drivers/cxl/pci.c +15 -8
index 6f10b19c9c83..418ae0eac188 100644
@@ -1293,11 +1293,13 @@ static void cxl_memdev_unregister(void *_cxlmd)
 	struct device *dev = &cxlmd->dev;
 
 	cdev_device_del(&cxlmd->cdev, dev);
-	cxl_memdev_shutdown(cxlmd);
+	if (cxlmd->shutdown)
+		cxlmd->shutdown(cxlmd);
 	put_device(dev);
 }
 
-static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
+static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm,
+					   const struct file_operations *fops)
 {
 	struct pci_dev *pdev = cxlm->pdev;
 	struct cxl_memdev *cxlmd;
@@ -1323,7 +1325,7 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
 	device_set_pm_not_required(dev);
 
 	cdev = &cxlmd->cdev;
-	cdev_init(cdev, &cxl_memdev_fops);
+	cdev_init(cdev, fops);
 	return cxlmd;
 
 err:
@@ -1331,15 +1333,17 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
 	return ERR_PTR(rc);
 }
 
-static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
-					      struct cxl_mem *cxlm)
+static struct cxl_memdev *
+devm_cxl_add_memdev(struct device *host, struct cxl_mem *cxlm,
+		    const struct file_operations *fops,
+		    void (*shutdown)(struct cxl_memdev *cxlmd))
 {
 	struct cxl_memdev *cxlmd;
 	struct device *dev;
 	struct cdev *cdev;
 	int rc;
 
-	cxlmd = cxl_memdev_alloc(cxlm);
+	cxlmd = cxl_memdev_alloc(cxlm, fops);
 	if (IS_ERR(cxlmd))
 		return cxlmd;
 
@@ -1362,6 +1366,7 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
 	rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
 	if (rc)
 		return ERR_PTR(rc);
+	cxlmd->shutdown = shutdown;
 	return cxlmd;
 
 err:
@@ -1369,7 +1374,8 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
 	 * The cdev was briefly live, shutdown any ioctl operations that
 	 * saw that state.
 	 */
-	cxl_memdev_shutdown(cxlmd);
+	if (shutdown)
+		shutdown(cxlmd);
 	put_device(dev);
 	return ERR_PTR(rc);
 }
@@ -1610,7 +1616,8 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
-	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlm);
+	cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlm, &cxl_memdev_fops,
+				    cxl_memdev_shutdown);
 	if (IS_ERR(cxlmd))
 		return PTR_ERR(cxlmd);
 
Signature · 1 line
2.32.0
#12 lore raw

Re: [PATCH 5/6] cxl: Pass fops and shutdown to memdev creation

Quoted 4 lines · Ben Widawsky

Drivers that use cxl_core for registering a cxl_memdev will likely require synchronization with the core for shutdown so as to not race against the device going away.

This isn't quite right. There are three independent lifetimes to worry
about: the liveness of the file_operations (bounded by userspace last close), the driver attach lifetime (bounded by ->remove()), and the device lifetime (bounded by last put_device()). The devm vs file_operations lifetime collision is a common problem [1]. I'd go so far as to define something like:

struct devres_file_operations {
    struct file_operations *fops;
    (int)(*shutdown)(struct device *);
};

...where the definition of @shutdown is:

"invoked in the devres release path to disconnect any driver instance data from @dev. It assumes synchronization with any fops operation that requires driver data. After @shutdown an operation may only reference @device data."

I'd define it locally for now, unless / until other subsystems want to share the CXL device allocation scheme.

[1]: https://lore.kernel.org/r/[email protected]

Quoted 3 lines

The main example currently is with the
ioctl interface. Through the duration of the ioctl it's expected that the underlying memdev will not disappear.

It's not cxlmd that's the concern, it's cxlm.

Quoted 3 lines

Additionally, it may be desirable to have the fops be passed along as well for drivers which do not want the standard handler for the character device's ioctl.

Let's give a concrete example. "The unit test code for CXL will also use a mock driver that provides its own devres_file_operations instance."

I'll take a shot at reworking this.

Quoted 109 lines

As memdev is being migrated to core, this separation must be made.

---
 drivers/cxl/mem.h |  2 ++
 drivers/cxl/pci.c | 23 +++++++++++++++--------
 2 files changed, 17 insertions(+), 8 deletions(-)
drivers/cxl/mem.h +2 -0
index 2092f86beeb8..2b7481376621 100644
@@ -42,12 +42,14 @@ extern int cxl_mem_major;
  * @cdev: char dev core object for ioctl operations
  * @cxlm: pointer to the parent device driver data
  * @id: id number of this memdev instance.
+ * @shutdown: Optional function to call on memory device shutdown.
  */
 struct cxl_memdev {
        struct device dev;
        struct cdev cdev;
        struct cxl_mem *cxlm;
        int id;
+       void (*shutdown)(struct cxl_memdev *cxlmd);
 };
 
 /**
drivers/cxl/pci.c +15 -9
index 6f10b19c9c83..418ae0eac188 100644
@@ -1293,11 +1293,13 @@ static void cxl_memdev_unregister(void *_cxlmd)
        struct device *dev = &cxlmd->dev;
 
        cdev_device_del(&cxlmd->cdev, dev);
-       cxl_memdev_shutdown(cxlmd);
+       if (cxlmd->shutdown)
+               cxlmd->shutdown(cxlmd);
        put_device(dev);
 }
 
-static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
+static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm,
+                                          const struct file_operations *fops)
 {
        struct pci_dev *pdev = cxlm->pdev;
        struct cxl_memdev *cxlmd;
@@ -1323,7 +1325,7 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
        device_set_pm_not_required(dev);
 
        cdev = &cxlmd->cdev;
-       cdev_init(cdev, &cxl_memdev_fops);
+       cdev_init(cdev, fops);
        return cxlmd;
 
 err:
@@ -1331,15 +1333,17 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
        return ERR_PTR(rc);
 }
 
-static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
-                                             struct cxl_mem *cxlm)
+static struct cxl_memdev *
+devm_cxl_add_memdev(struct device *host, struct cxl_mem *cxlm,
+                   const struct file_operations *fops,
+                   void (*shutdown)(struct cxl_memdev *cxlmd))
 {
        struct cxl_memdev *cxlmd;
        struct device *dev;
        struct cdev *cdev;
        int rc;
 
-       cxlmd = cxl_memdev_alloc(cxlm);
+       cxlmd = cxl_memdev_alloc(cxlm, fops);
        if (IS_ERR(cxlmd))
                return cxlmd;
 
@@ -1362,6 +1366,7 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
        rc = devm_add_action_or_reset(host, cxl_memdev_unregister, cxlmd);
        if (rc)
                return ERR_PTR(rc);
+       cxlmd->shutdown = shutdown;
        return cxlmd;
 
 err:
@@ -1369,7 +1374,8 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,
         * The cdev was briefly live, shutdown any ioctl operations that
         * saw that state.
         */
-       cxl_memdev_shutdown(cxlmd);
+       if (shutdown)
+               shutdown(cxlmd);
        put_device(dev);
        return ERR_PTR(rc);
 }
@@ -1610,7 +1616,8 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
        if (rc)
                return rc;
 
-       cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlm);
+       cxlmd = devm_cxl_add_memdev(&pdev->dev, cxlm, &cxl_memdev_fops,
+                                   cxl_memdev_shutdown);
        if (IS_ERR(cxlmd))
                return PTR_ERR(cxlmd);
 
--

2.32.0

#13 lore raw

PATCH 6/6 cxl/core: Move memdev management to core

In reply to Ben Widawsky

The CXL core must own memdev management so that CXL drivers may register for services. At present, only cxl_pci cares about this functionality, which is why the functionality was implemented there. The immediate need for this is a memdev driver which will be needed to manage CXL regions. Potentially, drivers for other types of devices may desire this functionality as well - an accelerator wishing to present itself as a memory device for example.

---
 drivers/cxl/Makefile      |   2 +-
 drivers/cxl/core/memdev.c | 224 +++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.h         |   5 +
 drivers/cxl/pci.c         | 228 ++------------------------------------
 4 files changed, 237 insertions(+), 222 deletions(-)
 create mode 100644 drivers/cxl/core/memdev.c
drivers/cxl/Makefile +1 -1
index 607d8e6ab199..60941699c9ac 100644
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
-cxl_core-y := core/bus.o core/pmem.o core/regs.o
+cxl_core-y := core/bus.o core/memdev.o core/pmem.o core/regs.o
 cxl_pci-y := pci.o
 cxl_acpi-y := acpi.o
 cxl_pmem-y := pmem.o
drivers/cxl/core/memdev.c new file +224 -0

224 lines not on this page · raw message

drivers/cxl/mem.h +5 -0
index 2b7481376621..3730e3509ab6 100644
@@ -52,6 +52,11 @@ struct cxl_memdev {
 	void (*shutdown)(struct cxl_memdev *cxlmd);
 };
 
+struct cxl_memdev *
+devm_cxl_add_memdev(struct device *host, struct cxl_mem *cxlm,
+		    const struct file_operations *fops,
+		    void (*shutdown)(struct cxl_memdev *cxlmd));
+
 /**
  * struct cxl_mem - A CXL memory device
  * @pdev: The PCI device associated with this CXL device.
drivers/cxl/pci.c +7 -221

246 lines not on this page · raw message

Signature · 1 line
2.32.0