CXL core reorganization
Subsystems (2)
cover CXL core reorganization
The main motivation of the patch series is to establish the cxl_core driver in its own directory and modularize it. Specifically, the patch series aims to achieve three things:
1. Move existing core functionality to a new directory.
2. Split existing core functionality into multiple files.
3. Migrate memdev functionality into core.
#1 is trivially accomplished with git mv. The file itself is renamed back to bus.c since the goal is to break up core functionality into multiple files, and so the name core.c doesn't make sense in that context.
#2 is also trivially accomplished via cut/paste.
#3 is slightly invasive in that it has certain functional changes to improve the existing interfaces and make them more generic. The rest of the change is cut/paste. This is also the only part of the series which has runtime functional change in that some interfaces are removed from cxl_pci, moved into cxl_core, and exported for other drivers to use.
Ben Widawsky (6):
cxl: Move cxl_core to new directory cxl/core: Improve CXL core kernel docs cxl/core: Extract register and pmem functionality cxl/mem: Move character device region creation
cxl: Pass fops and shutdown to memdev creation cxl/core: Move memdev management to core
.../driver-api/cxl/memory-devices.rst | 8 +-
drivers/cxl/Makefile | 2 +-
drivers/cxl/{core.c => core/bus.c} | 458 +-----------------
drivers/cxl/core/core.h | 20 +
drivers/cxl/core/memdev.c | 224 +++++++++
drivers/cxl/core/pmem.c | 201 ++++++++
drivers/cxl/core/regs.c | 235 +++++++++
drivers/cxl/mem.h | 9 +
drivers/cxl/pci.c | 239 +--------
9 files changed, 731 insertions(+), 665 deletions(-)
rename drivers/cxl/{core.c => core/bus.c} (58%)
create mode 100644 drivers/cxl/core/core.h
create mode 100644 drivers/cxl/core/memdev.c
create mode 100644 drivers/cxl/core/pmem.c
create mode 100644 drivers/cxl/core/regs.cSignature · 1 line
2.32.0
PATCH 1/6 cxl: Move cxl_core to new directory
CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.
---
Documentation/driver-api/cxl/memory-devices.rst | 2 +-
drivers/cxl/Makefile | 2 +-
drivers/cxl/{core.c => core/bus.c} | 4 ++--
3 files changed, 4 insertions(+), 4 deletions(-)
rename drivers/cxl/{core.c => core/bus.c} (99%)Documentation/driver-api/cxl/memory-devices.rst +1 -1
@@ -36,7 +36,7 @@ CXL Core .. kernel-doc:: drivers/cxl/cxl.h :internal: -.. kernel-doc:: drivers/cxl/core.c +.. kernel-doc:: drivers/cxl/core/bus.c :doc: cxl core External Interfaces
drivers/cxl/Makefile +1 -1
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -cxl_core-y := core.o +cxl_core-y := core/bus.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +2 -2
@@ -6,8 +6,8 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> -#include "cxl.h" -#include "mem.h" +#include "../cxl.h" +#include "../mem.h" /** * DOC: cxl core
Signature · 1 line
2.32.0
Re: [PATCH 1/6] cxl: Move cxl_core to new directory
Quoted 6 lines · Ben Widawsky
CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.
I think this effort wants a drivers/cxl/core/Makefile, more below:
Quoted 26 lines
Signed-off-by Ben Widawsky <[email protected]>--- Documentation/driver-api/cxl/memory-devices.rst | 2 +- drivers/cxl/Makefile | 2 +- drivers/cxl/{core.c => core/bus.c} | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) rename drivers/cxl/{core.c => core/bus.c} (99%)Documentation/driver-api/cxl/memory-devices.rst +1
-1@@ -36,7 +36,7 @@ CXL Core .. kernel-doc:: drivers/cxl/cxl.h :internal: -.. kernel-doc:: drivers/cxl/core.c +.. kernel-doc:: drivers/cxl/core/bus.c :doc: cxl core External Interfacesdrivers/cxl/Makefile +0
-0
drivers/cxl/Makefile obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o · line 7
Quoted 3 lines
drivers/cxl/Makefile +0
-0@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
Only the core has exports so this is not relevant for the drivers (pci, acpi, pmem...)
drivers/cxl/Makefile obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o · line 8
Quoted 2 lines
drivers/cxl/Makefile +1
-1-cxl_core-y := core.o +cxl_core-y := core/bus.o
Not typical to include directory names in Makefiles.
Quoted 11 lines
drivers/cxl/Makefile +0
-0cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.odrivers/cxl/core/bus.c renamed from drivers/cxl/core.c +0
-0
drivers/cxl/core/bus.c line 10
Quoted 7 lines
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +2
-2@@ -6,8 +6,8 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> -#include "cxl.h" -#include "mem.h" +#include "../cxl.h" +#include "../mem.h"
I'd prefer <cxl.h> and <mem.h>.
Proposed changes to fold:
drivers/cxl/Makefile +1 -3
@@ -1,11 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 -obj-$(CONFIG_CXL_BUS) += cxl_core.o +obj-$(CONFIG_CXL_BUS) += core/ obj-$(CONFIG_CXL_MEM) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o -ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -cxl_core-y := core/bus.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o
drivers/cxl/core/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_CXL_BUS) += cxl_core.o + +ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -I./drivers/cxl +cxl_core-y := bus.o
drivers/cxl/core/bus.c +2 -2
@@ -6,8 +6,8 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> -#include "../cxl.h" -#include "../mem.h" +#include <cxl.h> +#include <mem.h> /** * DOC: cxl core
PATCH v2 1/6 cxl: Move cxl_core to new directory
CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.
---
v2: Create new Makefile for core (Dan)
---
Documentation/driver-api/cxl/memory-devices.rst | 2 +-
drivers/cxl/Makefile | 4 +---
drivers/cxl/core/Makefile | 5 +++++
drivers/cxl/{core.c => core/bus.c} | 4 ++--
4 files changed, 9 insertions(+), 6 deletions(-)
create mode 100644 drivers/cxl/core/Makefile
rename drivers/cxl/{core.c => core/bus.c} (99%)Documentation/driver-api/cxl/memory-devices.rst +1 -1
@@ -36,7 +36,7 @@ CXL Core .. kernel-doc:: drivers/cxl/cxl.h :internal: -.. kernel-doc:: drivers/cxl/core.c +.. kernel-doc:: drivers/cxl/core/bus.c :doc: cxl core External Interfaces
drivers/cxl/Makefile +1 -3
@@ -1,11 +1,9 @@ # SPDX-License-Identifier: GPL-2.0 -obj-$(CONFIG_CXL_BUS) += cxl_core.o +obj-$(CONFIG_CXL_BUS) += core/ obj-$(CONFIG_CXL_MEM) += cxl_pci.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o -ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -cxl_core-y := core.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o
drivers/cxl/core/Makefile new file +5 -0
@@ -0,0 +1,5 @@ +# SPDX-License-Identifier: GPL-2.0 +obj-$(CONFIG_CXL_BUS) += cxl_core.o + +ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -I./drivers/cxl +cxl_core-y := bus.o
drivers/cxl/core/bus.c renamed from drivers/cxl/core.c +2 -2
@@ -6,8 +6,8 @@#include <linux/pci.h> #include <linux/slab.h> #include <linux/idr.h> -#include "cxl.h" -#include "mem.h" +#include <cxl.h> +#include <mem.h> /** * DOC: cxl core
Signature · 1 line
2.32.0
Re: [PATCH v2 1/6] cxl: Move cxl_core to new directory
Quoted 9 lines · Ben Widawsky
CXL core is growing, and it's already arguably unmanageable. To support future growth, move core functionality to a new directory and rename the file to represent just bus support. Future work will remove non-bus functionality.
Signed-off-by Ben Widawsky <[email protected]>---
v2: Create newMakefilefor core (Dan)
Dan, there's some minor conflicts as a result of this change. Let me know if you'd like me to resend the series.
[snip]
PATCH 2/6 cxl/core: Improve CXL core kernel docs
In reply to Ben WidawskyNow that CXL core's role is well understood, the documentation should reflect that information.
--- drivers/cxl/core/bus.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/cxl/core/bus.c +9 -2
@@ -12,8 +12,15 @@/** * DOC: cxl core * -* The CXL core provides a sysfs hierarchy for control devices and a rendezvous -* point for cross-device interleave coordination through cxl ports. +* The CXL core provides a set of interfaces that can be consumed by CXL aware +* drivers. The interfaces allow for creation, modification, and destruction of +* regions, memory devices, ports, and decoders. CXL aware drivers must register +* with the CXL core via these interfaces in order to be able to participate in +* cross-device interleave coordination. The CXL core also establishes and +* maintains the bridge to the nvdimm subsystem. +* +* CXL core introduces sysfs hierarchy to control the devices that are +* instantiated by the core. */ static DEFINE_IDA (cxl_port_ida);
Signature · 1 line
2.32.0
Re: [PATCH 2/6] cxl/core: Improve CXL core kernel docs
Quoted 3 lines · Ben Widawsky
Now that CXL core's role is well understood, the documentation should reflect that information.
Looks good to me.
Quoted 31 lines
Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/core/bus.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)drivers/cxl/core/bus.c +9
-3@@ -12,8 +12,15 @@/** * DOC: cxl core * -* The CXL core provides a sysfs hierarchy for control devices and a rendezvous -* point for cross-device interleave coordination through cxl ports. +* The CXL core provides a set of interfaces that can be consumed by CXL aware +* drivers. The interfaces allow for creation, modification, and destruction of +* regions, memory devices, ports, and decoders. CXL aware drivers must register +* with the CXL core via these interfaces in order to be able to participate in +* cross-device interleave coordination. The CXL core also establishes and +* maintains the bridge to the nvdimm subsystem. +* +* CXL core introduces sysfs hierarchy to control the devices that are +* instantiated by the core. */ static DEFINE_IDA (cxl_port_ida); --2.32.0
PATCH 3/6 cxl/core: Extract register and pmem functionality
In reply to Ben WidawskyRegister mapping and pmem/nvdimm integration are distinct enough from basic CXL bus functionality that it warrants being moved out of bus.c Additionally, this aims to modularize for the sake of reducing the size of bus.c
pmem and register programming have very clear separation and are done together for that reason. Other parts of core, like ports and decoders should be pulled out as well, but those are more integrated with core and therefore saved for later.
--- .../driver-api/cxl/memory-devices.rst | 6 + drivers/cxl/Makefile | 2 +- drivers/cxl/core/bus.c | 435 +----------------- drivers/cxl/core/core.h | 20 + drivers/cxl/core/pmem.c | 201 ++++++++ drivers/cxl/core/regs.c | 235 ++++++++++ 6 files changed, 466 insertions(+), 433 deletions(-) create mode 100644 drivers/cxl/core/core.h create mode 100644 drivers/cxl/core/pmem.c create mode 100644 drivers/cxl/core/regs.c
Documentation/driver-api/cxl/memory-devices.rst +6 -0
@@ -39,6 +39,12 @@ CXL Core .. kernel-doc:: drivers/cxl/core/bus.c :doc: cxl core +.. kernel-doc:: drivers/cxl/core/pmem.c + :internal: + +.. kernel-doc:: drivers/cxl/core/regs.c + :internal: + External Interfaces ===================
drivers/cxl/Makefile +1 -1
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -cxl_core-y := core/bus.o +cxl_core-y := core/bus.o core/pmem.o core/regs.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o
drivers/cxl/core/bus.c +3 -432
465 lines not on this page · raw message
drivers/cxl/core/core.h new file +20 -0
@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. */ + +#ifndef __CXL_CORE_H__ +#define __CXL_CORE_H__ + +#include "../cxl.h" +#include "../mem.h" + +extern const struct device_type cxl_nvdimm_bridge_type; +extern const struct device_type cxl_nvdimm_type; + +extern struct attribute_group cxl_base_attribute_group; + +static inline void unregister_cxl_dev (void *dev) +{ +device_unregister (dev); +} + +#endif /* __CXL_CORE_H__ */
drivers/cxl/core/pmem.c new file +201 -0
201 lines not on this page · raw message
drivers/cxl/core/regs.c new file +235 -0
235 lines not on this page · raw message
Signature · 1 line
2.32.0
Re: [PATCH 3/6] cxl/core: Extract register and pmem functionality
In general "and" in the patch title is a red flag that the patch can be split. I'll go ahead and do that when folding in the collision fixes.
Quoted 24 lines · Ben Widawsky
Register mapping and pmem/nvdimm integration are distinct enough from basic CXL bus functionality that it warrants being moved out of
bus.cAdditionally, this aims to modularize for the sake of reducing the size ofbus.cpmem and register programming have very clear separation and are done together for that reason. Other parts of core, like ports and decoders should be pulled out as well, but those are more integrated with core and therefore saved for later.
Signed-off-by Ben Widawsky <[email protected]>--- .../driver-api/cxl/memory-devices.rst | 6 + drivers/cxl/Makefile | 2 +- drivers/cxl/core/bus.c | 435 +----------------- drivers/cxl/core/core.h | 20 + drivers/cxl/core/pmem.c | 201 ++++++++ drivers/cxl/core/regs.c | 235 ++++++++++ 6 files changed, 466 insertions(+), 433 deletions(-) create mode 100644 drivers/cxl/core/core.h create mode 100644 drivers/cxl/core/pmem.c create mode 100644 drivers/cxl/core/regs.c
[..]
Quoted 6 lines
drivers/cxl/core/core.h new file +0
-0
drivers/cxl/core/core.h line 8
Quoted 8 lines
drivers/cxl/core/core.h new file +8
-0@@ -0,0 +1,20 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2020 Intel Corporation. */ + +#ifndef __CXL_CORE_H__ +#define __CXL_CORE_H__ + +#include "../cxl.h" +#include "../mem.h"
This results in unnecessary includes. Each compilation unit should only include the headers it absolutely needs. In this case, making each compilation unit responsible for its own includes means that bus.c does not need mem.h, and regs.c does not need either cxl.h or core.h
PATCH 4/6 cxl/mem: Move character device region creation
In reply to Ben WidawskyIn order to merge the memdev specific functionality of cxl_pci into core, the character device creation currently handled by cxl_pci need to be moved. The rest of the changes are largely cut/paste, the actual functional change is done here in preparation.
--- drivers/cxl/core/bus.c | 12 ++++++++++++ drivers/cxl/mem.h | 2 ++ drivers/cxl/pci.c | 14 +------------- 3 files changed, 15 insertions(+), 13 deletions(-)
drivers/cxl/core/bus.c +12 -0
@@ -22,6 +22,8 @@* instantiated by the core. */ +int cxl_mem_major; +EXPORT_SYMBOL_GPL (cxl_mem_major);static DEFINE_IDA (cxl_port_ida); static ssize_t devtype_show(struct device *dev, struct device_attribute *attr,
@@ -632,12 +634,22 @@ EXPORT_SYMBOL_GPL(cxl_bus_type);static __init int cxl_core_init (void ) { +dev_t devt; +int rc; + + rc =alloc_chrdev_region (&devt,0 ,CXL_MEM_MAX_DEVS ,"cxl" ); +if (rc) +return rc; + + cxl_mem_major =MAJOR (devt); +return bus_register (&cxl_bus_type); }static void cxl_core_exit (void ) {bus_unregister (&cxl_bus_type); +unregister_chrdev_region (MKDEV (cxl_mem_major,0 ),CXL_MEM_MAX_DEVS ); }module_init (cxl_core_init);
drivers/cxl/mem.h +2 -0
@@ -34,6 +34,8 @@*/ #define CXL_MEM_MAX_DEVS 65536 +extern int cxl_mem_major; +/** * struct cxl_memdev - CXL bus object representing a Type-3 Memory Device * @dev: driver core device object
drivers/cxl/pci.c +1 -13
@@ -94,7 +94,6 @@ struct mbox_cmd {#define CXL_MBOX_SUCCESS 0 }; -static int cxl_mem_major;static DEFINE_IDA (cxl_memdev_ida);static DECLARE_RWSEM (cxl_memdev_rwsem);static struct dentry *cxl_debugfs;
@@ -1640,25 +1639,15 @@ static struct pci_driver cxl_mem_driver = {static __init int cxl_mem_init (void ) {struct dentry *mbox_debugfs; -dev_t devt;int rc;/* Double check the anonymous union trickery in struct cxl_regs */ BUILD_BUG_ON (offsetof(struct cxl_regs , memdev) != offsetof(struct cxl_regs, device_regs.memdev)); - rc =alloc_chrdev_region (&devt,0 ,CXL_MEM_MAX_DEVS ,"cxl" ); -if (rc) -return rc; - - cxl_mem_major =MAJOR (devt); - rc =pci_register_driver (&cxl_mem_driver); -if (rc) { -unregister_chrdev_region (MKDEV (cxl_mem_major,0 ), -CXL_MEM_MAX_DEVS ); +if (rc)return rc; - } cxl_debugfs =debugfs_create_dir ("cxl" ,NULL ); mbox_debugfs =debugfs_create_dir ("mbox" , cxl_debugfs);
@@ -1672,7 +1661,6 @@ static __exit void cxl_mem_exit(void) {debugfs_remove_recursive (cxl_debugfs);pci_unregister_driver (&cxl_mem_driver); -unregister_chrdev_region (MKDEV (cxl_mem_major,0 ),CXL_MEM_MAX_DEVS ); }MODULE_LICENSE ("GPL v2" );
Signature · 1 line
2.32.0
Re: [PATCH 4/6] cxl/mem: Move character device region creation
Quoted 18 lines · Ben Widawsky
In order to merge the memdev specific functionality of
cxl_pciinto core, the character device creation currently handled bycxl_pcineed to be moved. The rest of the changes are largely cut/paste, the actual functional change is done here in preparation.Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/core/bus.c | 12 ++++++++++++ drivers/cxl/mem.h | 2 ++ drivers/cxl/pci.c | 14 +------------- 3 files changed, 15 insertions(+), 13 deletions(-)drivers/cxl/core/bus.c +0
-0
drivers/cxl/core/bus.c line 26
Quoted 5 lines
drivers/cxl/core/bus.c +2
-0@@ -22,6 +22,8 @@* instantiated by the core. */ +int cxl_mem_major; +EXPORT_SYMBOL_GPL (cxl_mem_major);
This symbol export is necessary here, but it's not needed by the end of the series which leads me to question this fine grained split. I think the shutdown patch is suitable to split, but this one and the last can be squashed.
Quoted 5 lines
drivers/cxl/core/bus.c +0
-0static DEFINE_IDA (cxl_port_ida); static ssize_t devtype_show(struct device *dev, struct device_attribute *attr,@@ -632,12 +634,22 @@ EXPORT_SYMBOL_GPL(cxl_bus_type);
drivers/cxl/core/bus.c EXPORT_SYMBOL_GPL(cxl_bus_type); · line 644
Quoted 10 lines
drivers/cxl/core/bus.c +8
-0static __init int cxl_core_init (void ) { +dev_t devt; +int rc; + + rc =alloc_chrdev_region (&devt,0 ,CXL_MEM_MAX_DEVS ,"cxl" ); +if (rc) +return rc; + + cxl_mem_major =MAJOR (devt);
I'd rather have cxl_core_{init,exit}() call something like cxl_memdev_{init,exit}() that way cxl_mem_major can become static again.
PATCH 5/6 cxl: Pass fops and shutdown to memdev creation
In reply to Ben WidawskyDrivers that use cxl_core for registering a cxl_memdev will likely require synchronization with the core for shutdown so as to not race against the device going away. The main example currently is with the ioctl interface. Through the duration of the ioctl it's expected that the underlying memdev will not disappear.
Additionally, it may be desirable to have the fops be passed along as well for drivers which do not want the standard handler for the character device's ioctl.
As memdev is being migrated to core, this separation must be made.
--- drivers/cxl/mem.h | 2 ++ drivers/cxl/pci.c | 23 +++++++++++++++-------- 2 files changed, 17 insertions(+), 8 deletions(-)
drivers/cxl/mem.h +2 -0
@@ -42,12 +42,14 @@ extern int cxl_mem_major;* @cdev: char dev core object for ioctl operations * @cxlm: pointer to the parent device driver data * @id: id number of this memdev instance. +* @shutdown: Optional function to call on memory device shutdown. */ struct cxl_memdev {struct device dev;struct cdev cdev;struct cxl_mem *cxlm;int id; +void (*shutdown)(struct cxl_memdev *cxlmd); };/**
drivers/cxl/pci.c +15 -8
@@ -1293,11 +1293,13 @@ static void cxl_memdev_unregister(void *_cxlmd)struct device *dev = &cxlmd->dev;cdev_device_del (&cxlmd->cdev, dev); -cxl_memdev_shutdown (cxlmd); +if (cxlmd->shutdown) + cxlmd->shutdown (cxlmd);put_device (dev); } -static struct cxl_memdev *cxl_memdev_alloc (struct cxl_mem *cxlm) +static struct cxl_memdev *cxl_memdev_alloc (struct cxl_mem *cxlm, +const struct file_operations *fops) {struct pci_dev *pdev = cxlm->pdev;struct cxl_memdev *cxlmd;
@@ -1323,7 +1325,7 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)device_set_pm_not_required (dev); cdev = &cxlmd->cdev; -cdev_init (cdev, &cxl_memdev_fops); +cdev_init (cdev, fops);return cxlmd; err:
@@ -1331,15 +1333,17 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)return ERR_PTR (rc); } -static struct cxl_memdev *devm_cxl_add_memdev (struct device *host, -struct cxl_mem *cxlm) +static struct cxl_memdev * +devm_cxl_add_memdev (struct device *host,struct cxl_mem *cxlm, +const struct file_operations *fops, +void (*shutdown)(struct cxl_memdev *cxlmd)) {struct cxl_memdev *cxlmd;struct device *dev;struct cdev *cdev;int rc; - cxlmd =cxl_memdev_alloc (cxlm); + cxlmd =cxl_memdev_alloc (cxlm, fops);if (IS_ERR (cxlmd))return cxlmd;
@@ -1362,6 +1366,7 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host, rc =devm_add_action_or_reset (host, cxl_memdev_unregister, cxlmd);if (rc)return ERR_PTR (rc); + cxlmd->shutdown = shutdown;return cxlmd; err:
@@ -1369,7 +1374,8 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,* The cdev was briefly live, shutdown any ioctl operations that * saw that state. */ -cxl_memdev_shutdown (cxlmd); +if (shutdown) +shutdown (cxlmd);put_device (dev);return ERR_PTR (rc); }
@@ -1610,7 +1616,8 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; - cxlmd =devm_cxl_add_memdev (&pdev->dev, cxlm); + cxlmd =devm_cxl_add_memdev (&pdev->dev, cxlm, &cxl_memdev_fops, + cxl_memdev_shutdown);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd);
Signature · 1 line
2.32.0
Re: [PATCH 5/6] cxl: Pass fops and shutdown to memdev creation
Quoted 4 lines · Ben Widawsky
Drivers that use
cxl_corefor registering acxl_memdevwill likely require synchronization with the core for shutdown so as to not race against the device going away.
This isn't quite right. There are three independent lifetimes to worry
about: the liveness of the file_operations (bounded by userspace last close), the driver attach lifetime (bounded by ->remove()), and the device lifetime (bounded by last put_device()). The devm vs file_operations lifetime collision is a common problem [1]. I'd go so far as to define something like:
struct devres_file_operations {
struct file_operations *fops;
(int)(*shutdown)(struct device *);
};...where the definition of @shutdown is:
"invoked in the devres release path to disconnect any driver instance data from @dev. It assumes synchronization with any fops operation that requires driver data. After @shutdown an operation may only reference @device data."
I'd define it locally for now, unless / until other subsystems want to share the CXL device allocation scheme.
[1]: https://lore.kernel.org/r/[email protected]
Quoted 3 lines
The main example currently is with the
ioctl interface. Through the duration of the ioctl it's expected that the underlying memdev will not disappear.
It's not cxlmd that's the concern, it's cxlm.
Quoted 3 lines
Additionally, it may be desirable to have the fops be passed along as well for drivers which do not want the standard handler for the character device's ioctl.
Let's give a concrete example. "The unit test code for CXL will also use a mock driver that provides its own devres_file_operations instance."
I'll take a shot at reworking this.
Quoted 109 lines
As memdev is being migrated to core, this separation must be made.
Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/mem.h | 2 ++ drivers/cxl/pci.c | 23 +++++++++++++++-------- 2 files changed, 17 insertions(+), 8 deletions(-)drivers/cxl/mem.h +2
-0@@ -42,12 +42,14 @@ extern int cxl_mem_major;* @cdev: char dev core object for ioctl operations * @cxlm: pointer to the parent device driver data * @id: id number of this memdev instance. +* @shutdown: Optional function to call on memory device shutdown. */ struct cxl_memdev {struct device dev;struct cdev cdev;struct cxl_mem *cxlm;int id; +void (*shutdown)(struct cxl_memdev *cxlmd); };/** drivers/cxl/pci.c +15
-9@@ -1293,11 +1293,13 @@ static void cxl_memdev_unregister(void *_cxlmd)struct device *dev = &cxlmd->dev;cdev_device_del (&cxlmd->cdev, dev); -cxl_memdev_shutdown (cxlmd); +if (cxlmd->shutdown) + cxlmd->shutdown (cxlmd);put_device (dev); } -static struct cxl_memdev *cxl_memdev_alloc (struct cxl_mem *cxlm) +static struct cxl_memdev *cxl_memdev_alloc (struct cxl_mem *cxlm, +const struct file_operations *fops) {struct pci_dev *pdev = cxlm->pdev;struct cxl_memdev *cxlmd;@@ -1323,7 +1325,7 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)device_set_pm_not_required (dev); cdev = &cxlmd->cdev; -cdev_init (cdev, &cxl_memdev_fops); +cdev_init (cdev, fops);return cxlmd; err:@@ -1331,15 +1333,17 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)return ERR_PTR (rc); } -static struct cxl_memdev *devm_cxl_add_memdev (struct device *host, -struct cxl_mem *cxlm) +static struct cxl_memdev * +devm_cxl_add_memdev (struct device *host,struct cxl_mem *cxlm, +const struct file_operations *fops, +void (*shutdown)(struct cxl_memdev *cxlmd)) {struct cxl_memdev *cxlmd;struct device *dev;struct cdev *cdev;int rc; - cxlmd =cxl_memdev_alloc (cxlm); + cxlmd =cxl_memdev_alloc (cxlm, fops);if (IS_ERR (cxlmd))return cxlmd;@@ -1362,6 +1366,7 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host, rc =devm_add_action_or_reset (host, cxl_memdev_unregister, cxlmd);if (rc)return ERR_PTR (rc); + cxlmd->shutdown = shutdown;return cxlmd; err:@@ -1369,7 +1374,8 @@ static struct cxl_memdev *devm_cxl_add_memdev(struct device *host,* The cdev was briefly live, shutdown any ioctl operations that * saw that state. */ -cxl_memdev_shutdown (cxlmd); +if (shutdown) +shutdown (cxlmd);put_device (dev);return ERR_PTR (rc); }@@ -1610,7 +1616,8 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)if (rc)return rc; - cxlmd =devm_cxl_add_memdev (&pdev->dev, cxlm); + cxlmd =devm_cxl_add_memdev (&pdev->dev, cxlm, &cxl_memdev_fops, + cxl_memdev_shutdown);if (IS_ERR (cxlmd))return PTR_ERR (cxlmd); --2.32.0
PATCH 6/6 cxl/core: Move memdev management to core
In reply to Ben WidawskyThe CXL core must own memdev management so that CXL drivers may register for services. At present, only cxl_pci cares about this functionality, which is why the functionality was implemented there. The immediate need for this is a memdev driver which will be needed to manage CXL regions. Potentially, drivers for other types of devices may desire this functionality as well - an accelerator wishing to present itself as a memory device for example.
--- drivers/cxl/Makefile | 2 +- drivers/cxl/core/memdev.c | 224 +++++++++++++++++++++++++++++++++++++ drivers/cxl/mem.h | 5 + drivers/cxl/pci.c | 228 ++------------------------------------ 4 files changed, 237 insertions(+), 222 deletions(-) create mode 100644 drivers/cxl/core/memdev.c
drivers/cxl/Makefile +1 -1
@@ -5,7 +5,7 @@ obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL -cxl_core-y := core/bus.o core/pmem.o core/regs.o +cxl_core-y := core/bus.o core/memdev.o core/pmem.o core/regs.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o
drivers/cxl/core/memdev.c new file +224 -0
224 lines not on this page · raw message
drivers/cxl/mem.h +5 -0
@@ -52,6 +52,11 @@ struct cxl_memdev {void (*shutdown)(struct cxl_memdev *cxlmd); }; +struct cxl_memdev * +devm_cxl_add_memdev (struct device *host,struct cxl_mem *cxlm, +const struct file_operations *fops, +void (*shutdown)(struct cxl_memdev *cxlmd)); +/** * struct cxl_mem - A CXL memory device * @pdev: The PCI device associated with this CXL device.
drivers/cxl/pci.c +7 -221
246 lines not on this page · raw message
Signature · 1 line
2.32.0