CXL Region driver
Subsystems (5)
v3 cover CXL Region driver
Major changes since v2:
- Clarify encoded region/granularity from raw values
- Rename "region" to cxlr everywhere
- Kconfig for the region driver
- Several small bug fixes
https://gitlab.com/bwidawsk/linux/-/tree/cxl_region-v3
Original commit message follows with minor updates for correctness.
---
This patch series introduces the CXL region driver as well as associated APIs in CXL core. The region driver enables the creation of "regions" which is a concept defined by the CXL 2.0 specification [1]. Region verification and programming state are owned by the cxl_region driver (implemented in the cxl_region module). It relies on cxl_mem to determine if devices are CXL routed, and cxl_port to actually handle the programming of the HDM decoders. Much of the region driver is an implementation of algorithms described in the CXL Type 3 Memory Device Software Guide [2].
The region driver will be responsible for configuring regions found on persistent capacities in the Label Storage Area (LSA), it will also enumerate regions configured by BIOS, usually volatile capacities, and will allow for dynamic region creation (which can then be stored in the LSA). It is the primary consumer of the CXL Port [3] and CXL Mem drivers introduced previously [4].
The patches for the region driver could be squashed. They're broken out to aid review and because that's the order they were implemented in. My preference is to keep those as they are.
Some things are still missing and will be worked on while these are reviewed (in priority order):
1. Volatile regions creation and enumeration (Have a plan)
2. multi-level switches
3. Decoder programming restrictions (No plan). The one know restriction I've missed is to disallow programming HDM decoders that aren't in incremental system physical address ranges.
4. CXL region teardown -> nd_region teardown
5. Stress testing
[1]: https://www.computeexpresslink.org/download-the-specification
[2]: https://cdrdv2.intel.com/v1/dl/getContent/643805?wapkw=CXL%20memory%20device%20sw%20guide
[3]: https://lore.kernel.org/linux-cxl/164298424635.3018233.9356036382052246767.stgit@dwillia2-desk3.amr.corp.intel.com/T/#u
[4]: https://lore.kernel.org/linux-cxl/164298429450.3018233.13269591903486669825.stgit@dwillia2-desk3.amr.corp.intel.com/T/#u
---
Ben Widawsky (14):
cxl/region: Add region creation ABI cxl/region: Introduce concept of region configuration cxl/mem: Cache port created by the mem dev cxl/region: Introduce a cxl_region driver cxl/acpi: Handle address space allocation cxl/region: Address space allocation cxl/region: Implement XHB verification cxl/region: HB port config verification cxl/region: Add infrastructure for decoder programming cxl/region: Collect host bridge decoders cxl/region: Add support for single switch level
cxl: Program decoders for regions
cxl/pmem: Convert nvdimm bridge API to use dev cxl/region: Create an nd_region
.clang-format | 3 + Documentation/ABI/testing/sysfs-bus-cxl | 64 ++ .../driver-api/cxl/memory-devices.rst | 14 + drivers/cxl/Kconfig | 5 + drivers/cxl/Makefile | 2 + drivers/cxl/acpi.c | 30 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 4 + drivers/cxl/core/hdm.c | 209 +++++ drivers/cxl/core/pmem.c | 28 +- drivers/cxl/core/port.c | 105 ++- drivers/cxl/core/region.c | 529 +++++++++++ drivers/cxl/cxl.h | 76 +- drivers/cxl/cxlmem.h | 9 + drivers/cxl/mem.c | 35 +- drivers/cxl/pmem.c | 2 +- drivers/cxl/port.c | 62 +- drivers/cxl/region.c | 866 ++++++++++++++++++ drivers/cxl/region.h | 47 + tools/testing/cxl/Kbuild | 1 + 20 files changed, 2077 insertions(+), 15 deletions(-) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.c create mode 100644 drivers/cxl/region.h
base-commit: e783362eb54cd99b2cac8b3a9aeac942e6f6ac07
prerequisite-patch-id: 90de8aefc2999f55c7534fefa971d95653c4220c
prerequisite-patch-id: 32a5b56d83bf3372b6ed4b40f621eafb33a7201b
prerequisite-patch-id: f827831bb7a23e0789d16d7b8979b165253c6301
prerequisite-patch-id: 08b8febd42d3ab508b618937473807e553589e36
prerequisite-patch-id: 18049f47c948582c1dc26348d9765c934eb82a75
prerequisite-patch-id: 8f66d52af297449fa007a0ba963c5239b153ef5b
prerequisite-patch-id: 3e2e86cbc2631b99c1b5c0179f35799d3df31f91
prerequisite-patch-id: b88becd4997320a34e918cdef1b620e6dea14917
prerequisite-patch-id: c61df81018f2a93b87d10965b418afa659d9d6d6
prerequisite-patch-id: 73b31df62e00bb7af7082e2ca4d40023a7962abd
prerequisite-patch-id: 207abfcd5028c41df8875ee795a8ab697cd7c688
prerequisite-patch-id: 26978f021b3b0f4a6734ef8c0100c724dc88742e
prerequisite-patch-id: bf229ca5aab5c5dffe69ba5b9380749a66cf20ba
prerequisite-patch-id: 20ebefe1acfdecf184d048cb605368e1863646c1
prerequisite-patch-id: f34c26e902dd868dc1c3ef8ba8246cc063cf991a
prerequisite-patch-id: bcc59db1c6528244b649ced35eab015699c410fa
prerequisite-patch-id: 2f9f6cfbd6b73a563498c6b6d721bbc169a0a414
prerequisite-patch-id: dc8fb216dc8ff4f813bfc689273d9c5f5124e789
prerequisite-patch-id: da83e8074d339426c886c481070366afb189b561
prerequisite-patch-id: 501fe71f19065ba9f31cabd86756fedda853c414
prerequisite-patch-id: ceeef31c2ca85a426d507563b886347d28acc322
prerequisite-patch-id: f876c09942ae5a3223a36329c23262a05b2669f4
prerequisite-patch-id: 44fa61c5569614c8d9df854cde6fedfc2bc78c12
prerequisite-patch-id: 04ad90e1bbb5646125c4633fbe5341f572bc9548
prerequisite-patch-id: f4dbf89d99917f50c30e1ee56bfeff8d8dd6b0f3
prerequisite-patch-id: 2d7c3aacefcb8133897e3256ed6f76952555c2f1
prerequisite-patch-id: 7454df4bdb07381f02717845eb3b17011a89ab18
prerequisite-patch-id: 52ec0dfd506bb6a3f8d11a914cfc7320193a6445
prerequisite-patch-id: 9de14fa54cfba412e09d7b41f392c0f6d55d6a01
prerequisite-patch-id: ae39a482c2067a1f04baee5ce9131901e6d359ec
prerequisite-patch-id: 446240d2ed24d9e55ac9edfc65b511495659464a
prerequisite-patch-id: ba6bf6450e47df5e95e2fb1780d9edd126bc0eb2
prerequisite-patch-id: 3c0865b6dd062e677ef8e160e14f823622eafb9f
prerequisite-patch-id: 4503f5507cbdeb0770b420b4c26d87be2b173813
prerequisite-patch-id: c5a8cbda77c95b052040770eca0dc5b99876dc66
prerequisite-patch-id: e064003a6c48131fac401d9a48d4d6204fea6123
prerequisite-patch-id: b4c7213971c981dd5ca0fda992643a7c61548fef
prerequisite-patch-id: 2bd09e27f8a8df144a8ad386822390c87ef46ec5
prerequisite-patch-id: 60b3fafbd3bfa225405a6762bdb6b89c044b0b86
prerequisite-patch-id: 620068ae417bf0784809107e0dae3ec9793632df
prerequisite-patch-id: c3415fe92e29cd4afc508f8caf31cb914be09261
prerequisite-patch-id: 4c01f305244036afa9aaa918c8215659327dd0f3
prerequisite-patch-id: 034aeb7e124c5a34785c963bf014aa5380f00a2e
prerequisite-patch-id: 26f18c2ca586e6d734cd319e0e7f24398b17217f
prerequisite-patch-id: ef97136efb8c077232fe39a0465389565803a7b7
prerequisite-patch-id: 6a63e03117287b748cfec00e2c16a41ed38f4f9a
Signature · 1 line
2.35.0
PATCH v3 1/14 cxl/region: Add region creation ABI
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region)
// Create a new region by name
echo $region > /sys/bus/cxl/devices/decoder0.0/create_region
// Region now exists in sysfs
stat -t /sys/bus/cxl/devices/decoder0.0/$region
// Delete the region, and name
echo $region > /sys/bus/cxl/devices/decoder0.0/delete_region
---
Changes since v2:
- Rename 'region' variables to 'cxlr'
- Update ABI docs for possible actual upstream version
--- Documentation/ABI/testing/sysfs-bus-cxl | 24 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 16 ++ drivers/cxl/core/region.c | 208 ++++++++++++++++++ drivers/cxl/cxl.h | 9 + drivers/cxl/region.h | 38 ++++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 311 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.h
Documentation/ABI/testing/sysfs-bus-cxl +24 -0
@@ -163,3 +163,27 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Creates a new CXL region. Writing a value of the form + "regionX.Y:Z" will create a new uninitialized region that will + be mapped by the CXL decoderX.Y. Reading from this node will + return a newly allocated region name. In order to create a + region (writing) you must use a value returned from reading the + node. Regions must be created for root decoders, and must + subsequently configured and bound to a region driver before they + can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. A region must be unbound from the + region driver before being deleted. The attributes expects a + region in the form "regionX.Y:Z". The region's name, allocated + by reading create_region, will also be released.
Documentation/driver-api/cxl/memory-devices.rst +11 -0
@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================
drivers/cxl/core/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.o
drivers/cxl/core/core.h +3 -0
@@ -10,6 +10,9 @@ extern const struct device_type cxl_memdev_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,
drivers/cxl/core/port.c +16 -0
@@ -215,6 +215,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,
@@ -267,11 +269,23 @@ static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { NULL, }; +static int delete_region (struct device *dev,void *arg) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); + +return cxl_delete_region (cxld,dev_name (dev)); +} +static void cxl_decoder_release (struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +device_for_each_child (&cxld->dev, cxld, delete_region); + +dev_WARN_ONCE (dev, !ida_is_empty (&cxld->region_ida), +"Lost track of a region" ); +ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }
@@ -1194,6 +1208,8 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +ida_init (&cxld->region_ida); +return cxld; err:kfree (cxld);
drivers/cxl/core/region.c new file +208 -0
208 lines not on this page · raw message
drivers/cxl/cxl.h +9 -0
@@ -221,6 +221,7 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */
@@ -236,6 +237,7 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct ida region_ida;int nr_targets;struct cxl_dport *target[]; };
@@ -323,6 +325,13 @@ struct cxl_ep {struct list_head list; }; +bool is_cxl_region (struct device *dev); +struct cxl_region *to_cxl_region (struct device *dev); +struct cxl_region *cxl_alloc_region (struct cxl_decoder *cxld, +int interleave_ways); +int cxl_add_region (struct cxl_decoder *cxld,struct cxl_region *cxlr); +int cxl_delete_region (struct cxl_decoder *cxld,const char *region); +static inline bool is_cxl_root (struct cxl_port *port) {return port->uport == port->dev.parent;
drivers/cxl/region.h new file +38 -0
@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This regions id. Id is globally unique across all regions. +* @list: Node in decoder's region list. +* @res: Resource this region carves out of the platform decode range. +* @config: HDM decoder program config +* @config.size: Size of the region determined from LSA or userspace. +* @config.uuid: The UUID for this region. +* @config.interleave_ways: Number of interleave ways this region is configured for. +* @config.interleave_granularity: Interleave granularity of region +* @config.targets: The memory devices comprising the region. +*/ +struct cxl_region { +struct device dev; +int id; +struct list_head list; +struct resource *res; + +struct { +u64 size; +uuid_t uuid; +int interleave_ways; +int interleave_granularity; +struct cxl_memdev *targets[CXL_DECODER_MAX_INTERLEAVE ]; + } config; +}; + +#endif
tools/testing/cxl/Kbuild +1 -0
@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/
Signature · 1 line
2.35.0
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 11 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
Quoted 5 lines
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
echo $region >/sys/bus/cxl/devices/decoder0.0/create_region
Were I someone coming in cold to this the immediate question about this example would be "what if userspace races to create the region?". How about showing the example that this interface requires looping until the kernel returns success in case userspace races itself to create the next region? I think this would work for that purpose.
---
while
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region > /sys/bus/cxl/devices/decoder0.0/create_region do true; done
---
Quoted 31 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>---
Changes since v2:
- Rename 'region' variables to 'cxlr'
- Update ABI docs for possible actual upstream version--- Documentation/ABI/testing/sysfs-bus-cxl | 24 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 16 ++ drivers/cxl/core/region.c | 208 ++++++++++++++++++ drivers/cxl/cxl.h | 9 + drivers/cxl/region.h | 38 ++++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 311 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +0
-0
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 168
Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +3
-0@@ -163,3 +163,27 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: August, 2021
Maybe move this to January, 2022?
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +6
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + Creates a new CXL region. Writing a value of the form + "regionX.Y:Z" will create a new uninitialized region that will + be mapped by the CXL decoderX.Y.
"Write a value of the form 'regionX.Y:Z' to instantiate a new region within the decode range bounded by decoderX.Y."
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ Reading from this node will + return a newly allocated region name. In order to create a + region (writing) you must use a value returned from reading the + node.
"The value written must match the current value returned from reading this attribute. This behavior lets the kernel arbitrate racing attempts to create a region. The thread that fails to write loops and tries the next value."
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ subsequently configured and bound to a region driver before they + can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. A region must be unbound from the + region driver before being deleted.
....why does it need to be unbound first? device_unregister() triggers device_release_driver()?
Side note: I am more and more thinking that even though the BIOS may try to lock some configurations down, if the system owner wants the region deleted the kernel should probably do everything in its power to oblige and override the BIOS including secondary bus resets to get the decoders unlocked. I.e. either the driver needs to hide the delete region attribute for locked regions, or it needs to give as much power to root as someone who has physical access and can rip out devices that are decoding locked ranges. Something we can discuss later, but every 'disable' and 'delete' interface requires answering the question "what about 'locked' configs and hot-remove?".
Quoted 70 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ region in the form "regionX.Y:Z". The region's name, allocated + by reading create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -10,6 +10,9 @@ extern const struct device_type cxl_memdev_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +5
-0@@ -215,6 +215,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -267,11 +269,23 @@ static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { NULL, }; +static int delete_region (struct device *dev,void *arg) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);
drivers/cxl/core/port.c static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { · line 284
Quoted 10 lines
drivers/cxl/core/port.c +5
-0+ +return cxl_delete_region (cxld,dev_name (dev)); +} +static void cxl_decoder_release (struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +device_for_each_child (&cxld->dev, cxld, delete_region);
This is too late. Regions should be deleted before the decoder is unregistered, and I think it happens naturally due to the root port association with memdevs. I.e. a root decoder is unregistered by its parent port being unregistered which is triggered by cxl_acpi ->remove(). That ->remove() event triggers all memdevs to disconnect, albeit in a workqueue. So as long as cxl_acpi ->remove flushes that workqueue then it knows that all memdevs have triggered ->remove().
If the behavior of a region is that it gets deleted upon the last memdev being unmapped from it then there should not be any regions to clean up at decoder release time.
Quoted 22 lines
drivers/cxl/core/port.c +6
-0+ +dev_WARN_ONCE (dev, !ida_is_empty (&cxld->region_ida), +"Lost track of a region" ); +ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1194,6 +1208,8 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +ida_init (&cxld->region_ida); +return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +0
-0
drivers/cxl/core/region.c line 2
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0@@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */
Happy New Year! Let's go to 2022 here.
Quoted 7 lines
drivers/cxl/core/region.c new file +7
-0+#include <linux/io-64-nonatomic-lo-hi.h> +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h>
drivers/cxl/core/region.c line 19
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+#include "core.h" + +/** +* DOC: cxl core region +* +* Regions are managed through the Linux device model. Each region instance is a +* unique struct device. CXL core provides functionality to create, destroy, and +* configure regions. This is all implemented here. Binding a region +* (programming the hardware) is handled by a separate region driver. +*/
Somewhat information lite, how about:
"CXL Regions represent mapped memory capacity in system physical address space. Whereas the CXL Root Decoders identify the bounds of potential CXL Memory ranges, Regions represent the active mapped capacity by the HDM Decoder Capability structures throughout the Host Bridges, Switches, and Endpoints in the topology."
Quoted 9 lines
drivers/cxl/core/region.c new file +9
-0+ +static void cxl_region_release (struct device *dev); + +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static ssize_t create_region_show(struct device *dev,
drivers/cxl/core/region.c line 38
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ struct device_attribute *attr, char *buf) +{ + struct cxl_port *port = to_cxl_port(dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +int rc; + +if (dev_WARN_ONCE (dev, !is_root_decoder (dev), +"Invalid decoder selected for region." )) { +return -ENODEV ; + }
This can go, it's already the case that this attribute is only listed in 'cxl_decoder_root_attrs'
drivers/cxl/core/region.c line 40
Quoted 1 line
drivers/cxl/core/region.c new file +1
-0+ rc =ida_alloc (&cxld->region_ida,GFP_KERNEL );
This looks broken. What if userspace does:
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region) region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_region
...i.e. it should only advance to create a new name after the previous one was instantiated / confirmed via a write.
Also, sysfs values are world readable by default, so non-root can burn up region_ida.
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+if (rc <0 ) { +dev_err (&cxld->dev,"Couldn't get a new id\n" ); +return rc; + } + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, rc); +} + +static ssize_t create_region_store(struct device *dev, + struct device_attribute *attr,
drivers/cxl/core/region.c line 60
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ const char *buf, size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +int decoder_id, port_id, region_id; +struct cxl_region *cxlr; +ssize_t rc; + +if (sscanf (buf,"region%d.%d:%d" , &port_id, &decoder_id, ®ion_id) !=3 ) +return -EINVAL ;
With the proposed change above to cache the current 'next' region name this can just be something like:
sysfs_streq(buf, cxld->next);
Quoted 4 lines
drivers/cxl/core/region.c new file +4
-0+ +if (decoder_id != cxld->id) +return -EINVAL ; +
drivers/cxl/core/region.c line 74
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+if (port_id != port->id) +return -EINVAL ; + + cxlr =cxl_alloc_region (cxld, region_id); +if (IS_ERR (cxlr)) +return PTR_ERR (cxlr); + + rc =cxl_add_region (cxld, cxlr); +if (rc) { +kfree (cxlr);
...'add' failures usually require a put_device(), are you sure kfree() is correct here.
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+return rc; + } + +return len; +}
drivers/cxl/core/region.c line 89
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+DEVICE_ATTR_RW (create_region); + +static ssize_t delete_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev); +int rc; + + rc =cxl_delete_region (cxld, buf);
I would have expected symmetry with cxl_add_region() i.e. convert @buf to @cxlr and keep the function signatures between add and delete aligned.
Quoted 23 lines
drivers/cxl/core/region.c new file +23
-0+if (rc) +return rc; + +return len; +} +DEVICE_ATTR_WO (delete_region); + +struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} +EXPORT_SYMBOL_GPL (to_cxl_region); + +static void cxl_region_release (struct device *dev) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); + +ida_free (&cxld->region_ida, cxlr->id);
drivers/cxl/core/region.c line 122
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+kfree (cxlr); +} + +struct cxl_region *cxl_alloc_region (struct cxl_decoder *cxld,int id) +{ +struct cxl_region *cxlr; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM );
To keep symmetry with other device object allocations in the cxl/core I would expect to see the device_initialize() dev->type and dev->bus setup occur here as well.
Quoted 34 lines
drivers/cxl/core/region.c new file +34
-0+ + cxlr->id = id; + +return cxlr; +} + +/** +* cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +int cxl_add_region (struct cxl_decoder *cxld,struct cxl_region *cxlr) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct device *dev = &cxlr->dev; +int rc; + +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err; +
drivers/cxl/core/region.c line 166
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ rc =device_add (dev); +if (rc) + goto err; + +dev_dbg (dev,"Added to %s\n" ,dev_name (&cxld->dev)); + +return 0 ; + +err: +put_device (dev);
Here is that put_device() I was expecting, that kfree() earlier was a double-free it seems.
Also, I would have expected a devm action to remove this. Something like:
struct cxl_port *port = to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)
devm_cxl_add_region(port->uport, cxld, id);
else
rc = -ENXIO;cxl_device_unlock(&port->dev);
...then no matter what you know the region will be unregistered when the root port goes away.
Quoted 15 lines
drivers/cxl/core/region.c new file +15
-0+return rc; +} + +static struct cxl_region *cxl_find_region_by_name (struct cxl_decoder *cxld, +const char *name) +{ +struct device *region_dev; + + region_dev =device_find_child_by_name (&cxld->dev, name); +if (!region_dev) +return ERR_PTR (-ENOENT ); + +return to_cxl_region (region_dev); +} +
drivers/cxl/core/region.c line 191
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+/** +* cxl_delete_region - Deletes a region +* @cxld: Parent decoder +* @region_name: Named region, ie. regionX.Y:Z +*/ +int cxl_delete_region (struct cxl_decoder *cxld,const char *region_name) +{ +struct cxl_region *cxlr; + +device_lock (&cxld->dev);
cxl_device_lock()
...if the lock is needed, but I don't see why the lock is needed?
drivers/cxl/core/region.c line 203
Quoted 12 lines
drivers/cxl/core/region.c new file +12
-0+ + cxlr =cxl_find_region_by_name (cxld, region_name); +if (IS_ERR (cxlr)) { +device_unlock (&cxld->dev); +return PTR_ERR (cxlr); + } + +dev_dbg (&cxld->dev,"Requested removal of %s from %s\n" , +dev_name (&cxlr->dev),dev_name (&cxld->dev)); + +device_unregister (&cxlr->dev); +device_unlock (&cxld->dev);
This would need to change to devm_release_action() of course if the add side changes to devm_cxl_add_region().
Quoted 48 lines
drivers/cxl/core/region.c new file +5
-0+ +put_device (&cxlr->dev); + +return 0 ; +}drivers/cxl/cxl.h +9
-0@@ -221,6 +221,7 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +237,7 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct ida region_ida;int nr_targets;struct cxl_dport *target[]; };@@ -323,6 +325,13 @@ struct cxl_ep {struct list_head list; }; +bool is_cxl_region (struct device *dev); +struct cxl_region *to_cxl_region (struct device *dev); +struct cxl_region *cxl_alloc_region (struct cxl_decoder *cxld, +int interleave_ways); +int cxl_add_region (struct cxl_decoder *cxld,struct cxl_region *cxlr); +int cxl_delete_region (struct cxl_decoder *cxld,const char *region); +static inline bool is_cxl_root (struct cxl_port *port) {return port->uport == port->dev.parent;drivers/cxl/region.h new file +3
-0@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__
drivers/cxl/region.h line 13
Quoted 10 lines
drivers/cxl/region.h new file +10
-0+#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This regions id. Id is globally unique across all regions.
s/regions/region's/
Quoted 12 lines
drivers/cxl/region.h new file +12
-0+* @list: Node in decoder's region list. +* @res: Resource this region carves out of the platform decode range. +* @config: HDM decoder program config +* @config.size: Size of the region determined from LSA or userspace. +* @config.uuid: The UUID for this region. +* @config.interleave_ways: Number of interleave ways this region is configured for. +* @config.interleave_granularity: Interleave granularity of region +* @config.targets: The memory devices comprising the region. +*/ +struct cxl_region { +struct device dev; +int id;
drivers/cxl/region.h line 35
Quoted 10 lines
drivers/cxl/region.h new file +10
-0+struct list_head list; +struct resource *res; + +struct { +u64 size; +uuid_t uuid; +int interleave_ways; +int interleave_granularity; +struct cxl_memdev *targets[CXL_DECODER_MAX_INTERLEAVE ]; + } config;
Why a sub-struct?
Quoted 18 lines
drivers/cxl/region.h new file +3
-0+}; + +#endif tools/testing/cxl/Kbuild +1
-1@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/ --2.35.0
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]
Quoted 9 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)
Quoted 7 lines
devm_cxl_add_region(port->uport, cxld, id); else rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the cxl_acpi driver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 12 lines · Dan Williams
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]Quoted 10 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)devm_cxl_add_region(port->uport, cxld, id);
I assume you mean devm_cxl_delete_region(), yes?
Quoted 11 lines
Quoted 6 lines
else
rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the
cxl_acpidriver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.
I'm somewhat confused when you say devm action to remove this. The current auto region deletion happens when the ->release() is called. Are you suggesting when the root decoder is removed I delete the regions at that point?
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 23 lines · Ben Widawsky
Quoted 12 lines · Dan Williams
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]Quoted 10 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)devm_cxl_add_region(port->uport, cxld, id);I assume you mean
devm_cxl_delete_region(), yes?Quoted 7 lines
Quoted 6 lines
else
rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
Quoted 8 lines
Quoted 4 lines
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the
cxl_acpidriver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.I'm somewhat confused when you say devm action to remove this. The current auto region deletion happens when the ->release() is called. Are you suggesting when the root decoder is removed I delete the regions at that point?
Hmm. I went back and looked and I had changed this functionality at some point... So forget I said that, it isn't how it's working currently. But the question remains, are you suggesting I delete in the root decoder unregistration?
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 29 lines · Ben Widawsky
Quoted 28 lines · Ben Widawsky
Quoted 12 lines · Dan Williams
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]Quoted 10 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)devm_cxl_add_region(port->uport, cxld, id);I assume you mean
devm_cxl_delete_region(), yes?Quoted 11 lines
Quoted 6 lines
else
rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the
cxl_acpidriver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.
Quoted 8 lines
Quoted 3 lines
I'm somewhat confused when you say devm action to remove this. The current auto region deletion happens when the ->release() is called. Are you suggesting when the root decoder is removed I delete the regions at that point?
Hmm. I went back and looked and I had changed this functionality at some point... So forget I said that, it isn't how it's working currently. But the question remains, are you suggesting I delete in the root decoder unregistration?
I think it's easier if I write what I think you mean.... Here are the relevant parts:
devm_cxl_region_delete() is removed entirely.
static void unregister_region(void *_cxlr)
{
struct cxl_region *cxlr = _cxlr;
device_unregister(&cxlr->dev);
}static int devm_cxl_region_add(struct cxl_decoder *cxld, struct cxl_region *cxlr)
{
struct cxl_port *port = to_cxl_port(cxld->dev.parent);
struct device *dev = &cxlr->dev;
int rc;
rc = dev_set_name(dev, "region%d.%d:%d", port->id, cxld->id, cxlr->id);
if (rc)
return rc;
rc = device_add(dev);
if (rc)
return rc;
return devm_add_action_or_reset(&cxld->dev, unregister_region, cxlr);
}static ssize_t delete_region_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
struct cxl_decoder *cxld = to_cxl_decoder(dev);
struct cxl_region *cxlr;
cxlr = cxl_find_region_by_name(cxld, buf);
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
devm_release_action(dev, unregister_region, cxlr);
return len;
}DEVICE_ATTR_WO(delete_region);
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 54 lines · Ben Widawsky
Quoted 29 lines · Ben Widawsky
Quoted 28 lines · Ben Widawsky
Quoted 12 lines · Dan Williams
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]Quoted 10 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)devm_cxl_add_region(port->uport, cxld, id);I assume you mean
devm_cxl_delete_region(), yes?Quoted 11 lines
Quoted 6 lines
else
rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the
cxl_acpidriver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.Quoted 8 lines
Quoted 3 lines
I'm somewhat confused when you say devm action to remove this. The current auto region deletion happens when the ->release() is called. Are you suggesting when the root decoder is removed I delete the regions at that point?
Hmm. I went back and looked and I had changed this functionality at some point... So forget I said that, it isn't how it's working currently. But the question remains, are you suggesting I delete in the root decoder unregistration?
I think it's easier if I write what I think you mean.... Here are the relevant parts:
devm_cxl_region_delete()is removed entirely.static void
unregister_region(void *_cxlr){ struct cxl_region *cxlr = _cxlr; device_unregister(&cxlr->dev); }static int
devm_cxl_region_add(struct cxl_decoder *cxld, struct cxl_region *cxlr)
Quoted 14 lines
{
struct cxl_port *port = to_cxl_port(cxld->dev.parent);
struct device *dev = &cxlr->dev;
int rc;
rc = dev_set_name(dev, "region%d.%d:%d", port->id, cxld->id, cxlr->id);
if (rc)
return rc;
rc = device_add(dev);
if (rc)
return rc;
return devm_add_action_or_reset(&cxld->dev, unregister_region, cxlr);Decoders can't host devm actions. The host for this action would need to be the parent port.
Quoted 5 lines
}
static
ssize_tdelete_region_store(struct device *dev,
structdevice_attribute*attr, const char *buf,size_tlen)
Quoted 9 lines
{
struct cxl_decoder *cxld = to_cxl_decoder(dev);
struct cxl_region *cxlr;
cxlr = cxl_find_region_by_name(cxld, buf);
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
devm_release_action(dev, unregister_region, cxlr);Yes, modulo the same comment as before that the decoder object is not a suitable devm host. This also needs a solution for the race between these 2 actions:
echo "ACPI0017:00" > /sys/bus/platform/drivers/cxl_acpi/unbind echo $region > /sys/bus/cxl/devices/$decoder/delete_region
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 64 lines · Dan Williams
Quoted 63 lines · Ben Widawsky
Quoted 29 lines · Ben Widawsky
Quoted 28 lines · Ben Widawsky
Quoted 12 lines · Dan Williams
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]Quoted 10 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)devm_cxl_add_region(port->uport, cxld, id);I assume you mean
devm_cxl_delete_region(), yes?Quoted 11 lines
Quoted 6 lines
else
rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the
cxl_acpidriver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.Quoted 8 lines
Quoted 3 lines
I'm somewhat confused when you say devm action to remove this. The current auto region deletion happens when the ->release() is called. Are you suggesting when the root decoder is removed I delete the regions at that point?
Hmm. I went back and looked and I had changed this functionality at some point... So forget I said that, it isn't how it's working currently. But the question remains, are you suggesting I delete in the root decoder unregistration?
I think it's easier if I write what I think you mean.... Here are the relevant parts:
devm_cxl_region_delete()is removed entirely.static void
unregister_region(void *_cxlr){ struct cxl_region *cxlr = _cxlr; device_unregister(&cxlr->dev); }static int
devm_cxl_region_add(struct cxl_decoder *cxld, struct cxl_region *cxlr){ struct cxl_port *port = to_cxl_port(cxld->dev.parent); struct device *dev = &cxlr->dev; int rc; rc = dev_set_name(dev, "region%d.%d:%d", port->id, cxld->id, cxlr->id); if (rc) return rc;
Quoted 8 lines
Quoted 5 lines
rc = device_add(dev); if (rc) return rc; return devm_add_action_or_reset(&cxld->dev, unregister_region, cxlr);Decoders can't host devm actions. The host for this action would need to be the parent port.
Happy to change it since I can't imagine a decoder would go down without the port also going down. Can you please explain why a decoder can't host a devm action though. I'd like to understand that better.
Quoted 14 lines
Quoted 14 lines
}
static
ssize_tdelete_region_store(struct device *dev,
structdevice_attribute*attr, const char *buf,size_tlen){ struct cxl_decoder *cxld = to_cxl_decoder(dev); struct cxl_region *cxlr; cxlr = cxl_find_region_by_name(cxld, buf); if (IS_ERR(cxlr)) return PTR_ERR(cxlr);
Quoted 8 lines
Quoted 1 line
devm_release_action(dev, unregister_region, cxlr);Yes, modulo the same comment as before that the decoder object is not a suitable devm host. This also needs a solution for the race between these 2 actions:
echo "ACPI0017:00" >
/sys/bus/platform/drivers/cxl_acpi/unbindecho $region >/sys/bus/cxl/devices/$decoder/delete_region
Is there a better solution than taking the root port lock?
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
Quoted 70 lines · Ben Widawsky
Quoted 68 lines · Dan Williams
Quoted 67 lines · Ben Widawsky
Quoted 29 lines · Ben Widawsky
Quoted 28 lines · Ben Widawsky
Quoted 12 lines · Dan Williams
On Fri, Jan 28, 2022 at 10:14 AM Dan Williams <[email protected]> wrote:
[..]Quoted 10 lines
Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)devm_cxl_add_region(port->uport, cxld, id);I assume you mean
devm_cxl_delete_region(), yes?Quoted 11 lines
Quoted 6 lines
else
rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
...actually, the lock and ->dev.driver check here are not needed because this attribute is only registered while the
cxl_acpidriver is bound. So, it is safe to assume this is protected as decoder remove synchronizes against active sysfs users.Quoted 8 lines
Quoted 3 lines
I'm somewhat confused when you say devm action to remove this. The current auto region deletion happens when the ->release() is called. Are you suggesting when the root decoder is removed I delete the regions at that point?
Hmm. I went back and looked and I had changed this functionality at some point... So forget I said that, it isn't how it's working currently. But the question remains, are you suggesting I delete in the root decoder unregistration?
I think it's easier if I write what I think you mean.... Here are the relevant parts:
devm_cxl_region_delete()is removed entirely.static void
unregister_region(void *_cxlr){ struct cxl_region *cxlr = _cxlr; device_unregister(&cxlr->dev); }static int
devm_cxl_region_add(struct cxl_decoder *cxld, struct cxl_region *cxlr){ struct cxl_port *port = to_cxl_port(cxld->dev.parent); struct device *dev = &cxlr->dev; int rc; rc = dev_set_name(dev, "region%d.%d:%d", port->id, cxld->id, cxlr->id); if (rc) return rc; rc = device_add(dev); if (rc) return rc;
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
return devm_add_action_or_reset(&cxld->dev, unregister_region, cxlr);Decoders can't host devm actions. The host for this action would need to be the parent port.
Happy to change it since I can't imagine a decoder would go down without the port also going down. Can you please explain why a decoder can't host a devm action though. I'd like to understand that better.
So, devm releases resources at 2 times one of which is "too late". The natural / expected point at which they are released is by the driver core at $driver->remove($dev) time. There is also a backstop release point at $dev->${type,class,bus}->release() time. The latter one is "too late" because it effectively leave the device registered indefinitely which is broken because the parent sysfs directory hierarchy for that device will have already been removed, so the late release may crash depending on when the last put_device($dev) is performed. The decoder never experiences a ->remove() event, but it's parent port does (at least for non-root ports). For this case it will likely need to reach further up and use the same devm host as the decoder itself which is the ACPI0017 device.
Quoted 17 lines
Quoted 17 lines
Quoted 16 lines
}
static
ssize_tdelete_region_store(struct device *dev,
structdevice_attribute*attr, const char *buf,size_tlen){ struct cxl_decoder *cxld = to_cxl_decoder(dev); struct cxl_region *cxlr; cxlr = cxl_find_region_by_name(cxld, buf); if (IS_ERR(cxlr)) return PTR_ERR(cxlr); devm_release_action(dev, unregister_region, cxlr);
Quoted 8 lines
Quoted 6 lines
Yes, modulo the same comment as before that the decoder object is not a suitable devm host. This also needs a solution for the race between these 2 actions:
echo "ACPI0017:00" >
/sys/bus/platform/drivers/cxl_acpi/unbindecho $region >/sys/bus/cxl/devices/$decoder/delete_regionIs there a better solution than taking the root port lock?
Depends what lockdep says. The first choice lock to synchronize those 2 actions would be the ACPI0017 device_lock, but lockdep might point out a problem with that vs sysfs teardown synchronization.
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
In reply to Dan WilliamsQuoted 24 lines · Dan Williams
Quoted 11 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
Quoted 5 lines
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
echo $region >/sys/bus/cxl/devices/decoder0.0/create_regionWere I someone coming in cold to this the immediate question about this example would be "what if userspace races to create the region?". How about showing the example that this interface requires looping until the kernel returns success in case userspace races itself to create the next region? I think this would work for that purpose.
Quoted 9 lines
---
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; done---
See below... TL;DR the interface was designed to not work this way because I misunderstood you.
Quoted 26 lines
Quoted 26 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>---
Changes since v2:
- Rename 'region' variables to 'cxlr'
- Update ABI docs for possible actual upstream version--- Documentation/ABI/testing/sysfs-bus-cxl | 24 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 16 ++ drivers/cxl/core/region.c | 208 ++++++++++++++++++ drivers/cxl/cxl.h | 9 + drivers/cxl/region.h | 38 ++++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 311 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.h
Quoted 14 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 168
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +3
-0@@ -163,3 +163,27 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: August, 2021Maybe move this to January, 2022?
Okay.
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-cxl +6
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + Creates a new CXL region. Writing a value of the form + "regionX.Y:Z" will create a new uninitialized region that will + be mapped by the CXL decoderX.Y."Write a value of the form 'regionX.Y:Z' to instantiate a new region within the decode range bounded by decoderX.Y."
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ Reading from this node will + return a newly allocated region name. In order to create a + region (writing) you must use a value returned from reading the + node.
Quoted 4 lines
"The value written must match the current value returned from reading this attribute. This behavior lets the kernel arbitrate racing attempts to create a region. The thread that fails to write loops and tries the next value."
Okay.
Quoted 14 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ subsequently configured and bound to a region driver before they + can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. A region must be unbound from the + region driver before being deleted.....why does it need to be unbound first?
device_unregister()triggersdevice_release_driver()?
It shouldn't. This must be old verbiage from when the operation was expected to be different.
Quoted 12 lines
Side note: I am more and more thinking that even though the BIOS may try to lock some configurations down, if the system owner wants the region deleted the kernel should probably do everything in its power to oblige and override the BIOS including secondary bus resets to get the decoders unlocked. I.e. either the driver needs to hide the delete region attribute for locked regions, or it needs to give as much power to root as someone who has physical access and can rip out devices that are decoding locked ranges. Something we can discuss later, but every 'disable' and 'delete' interface requires answering the question "what about 'locked' configs and hot-remove?".
When I originally started on these patches, I hadn't been planning to enumerate regions based on the current system programming. So the original answer would have been, the region is hidden. With that change of mind, I think it makes sense to do everything in the driver's power to obey, however, it starts to get really messy if/when switch ports are decoding to other devices and need a reset to unlock decoders. With that in mind, first attempt should be to disallow deleting locked regions. Perhaps later we can add an "unlock" interface which must be used before delete.
Quoted 79 lines
Quoted 79 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ region in the form "regionX.Y:Z". The region's name, allocated + by reading create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -10,6 +10,9 @@ extern const struct device_type cxl_memdev_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +9
-0@@ -215,6 +215,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -267,11 +269,23 @@ static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { NULL, }; +static int delete_region (struct device *dev,void *arg) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); + +return cxl_delete_region (cxld,dev_name (dev)); +} +static void cxl_decoder_release (struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent);
Quoted 10 lines
drivers/cxl/core/port.c static const struct attribute_group *cxl_decoder_endpoint_attribute_groups[] = { · line 284
Quoted 1 line
drivers/cxl/core/port.c +1
-0+device_for_each_child (&cxld->dev, cxld, delete_region);This is too late. Regions should be deleted before the decoder is unregistered, and I think it happens naturally due to the root port association with memdevs. I.e. a root decoder is unregistered by its parent port being unregistered which is triggered by
cxl_acpi->remove(). That ->remove() event triggers all memdevs to disconnect, albeit in a workqueue. So as long ascxl_acpi->remove flushes that workqueue then it knows that all memdevs have triggered ->remove().
Yes, I agree with this. I wasn't entirely certain of what I was doing when I originally wrote this. However, it should probably become a WARN of some sort that no regions are left.
Quoted 4 lines
If the behavior of a region is that it gets deleted upon the last memdev being unmapped from it then there should not be any regions to clean up at decoder release time.
This is the goal, but not implemented yet.
Quoted 11 lines
Quoted 11 lines
drivers/cxl/core/port.c +4
-0+ +dev_WARN_ONCE (dev, !ida_is_empty (&cxld->region_ida), +"Lost track of a region" ); +ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1194,6 +1208,8 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 );
Quoted 16 lines
drivers/cxl/core/region.c line 2
Quoted 13 lines
drivers/cxl/core/port.c +2
-0+ida_init (&cxld->region_ida); +return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +2
-0@@ -0,0 +1,208 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ Happy New Year! Let's go to 2022 here.
Sure. I've always wondered if it's supposed to be when it was written, or when it was merged.
Quoted 20 lines
Quoted 7 lines
drivers/cxl/core/region.c new file +7
-0+#include <linux/io-64-nonatomic-lo-hi.h> +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> drivers/cxl/core/region.c line 19
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+#include "core.h" + +/** +* DOC: cxl core region +* +* Regions are managed through the Linux device model. Each region instance is a +* unique struct device. CXL core provides functionality to create, destroy, and +* configure regions. This is all implemented here. Binding a region +* (programming the hardware) is handled by a separate region driver. +*/ Somewhat information lite, how about:
Quoted 6 lines
"CXL Regions represent mapped memory capacity in system physical address space. Whereas the CXL Root Decoders identify the bounds of potential CXL Memory ranges, Regions represent the active mapped capacity by the HDM Decoder Capability structures throughout the Host Bridges, Switches, and Endpoints in the topology."
Okay.
Quoted 31 lines
Quoted 9 lines
drivers/cxl/core/region.c new file +9
-0+ +static void cxl_region_release (struct device *dev); + +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static ssize_t create_region_show(struct device *dev,drivers/cxl/core/region.c line 38
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ struct device_attribute *attr, char *buf) +{ + struct cxl_port *port = to_cxl_port(dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +int rc; + +if (dev_WARN_ONCE (dev, !is_root_decoder (dev), +"Invalid decoder selected for region." )) { +return -ENODEV ; + }This can go, it's already the case that this attribute is only listed in '
cxl_decoder_root_attrs'drivers/cxl/core/region.c line 40
Quoted 1 line
drivers/cxl/core/region.c new file +1
-0+ rc =ida_alloc (&cxld->region_ida,GFP_KERNEL );This looks broken. What if userspace does:
region=$(cat
/sys/bus/cxl/devices/decoder0.0/create_region) region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) echo $region >/sys/bus/cxl/devices/decoder0.0/create_region
Quoted 6 lines
...i.e. it should only advance to create a new name after the previous one was instantiated / confirmed via a write.
Also, sysfs values are world readable by default, so non-root can burn up
region_ida.
I misunderstood your original request for this interface. How it's working is by design. I'll change it to work the way you describe.
Quoted 24 lines
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+if (rc <0 ) { +dev_err (&cxld->dev,"Couldn't get a new id\n" ); +return rc; + } + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, rc); +} + +static ssize_t create_region_store(struct device *dev, + struct device_attribute *attr,drivers/cxl/core/region.c line 60
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ const char *buf, size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +int decoder_id, port_id, region_id; +struct cxl_region *cxlr; +ssize_t rc; + +if (sscanf (buf,"region%d.%d:%d" , &port_id, &decoder_id, ®ion_id) !=3 ) +return -EINVAL ;With the proposed change above to cache the current 'next' region name this can just be something like:
Quoted 2 lines
sysfs_streq(buf, cxld->next);
Sounds good.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/region.c new file +9
-0+ +if (decoder_id != cxld->id) +return -EINVAL ; + +if (port_id != port->id) +return -EINVAL ; + + cxlr =cxl_alloc_region (cxld, region_id); +if (IS_ERR (cxlr))
Quoted 9 lines
drivers/cxl/core/region.c line 74
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+return PTR_ERR (cxlr); + + rc =cxl_add_region (cxld, cxlr); +if (rc) { +kfree (cxlr);...'add' failures usually require a
put_device(), are you surekfree()is correct here.
You had requested, and I can't recall why, that regions are not devm managed. It sounds like you're asking for it to be devm managed. I wish I took more notes of our conversations...
Quoted 49 lines
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+return rc; + } + +return len; +}drivers/cxl/core/region.c line 89
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+DEVICE_ATTR_RW (create_region); + +static ssize_t delete_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev); +int rc; + + rc =cxl_delete_region (cxld, buf);I would have expected symmetry with
cxl_add_region()i.e. convert @buf to @cxlr and keep the function signatures between add and delete aligned.Quoted 29 lines
drivers/cxl/core/region.c new file +29
-0+if (rc) +return rc; + +return len; +} +DEVICE_ATTR_WO (delete_region); + +struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} +EXPORT_SYMBOL_GPL (to_cxl_region); + +static void cxl_region_release (struct device *dev) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); + +ida_free (&cxld->region_ida, cxlr->id); +kfree (cxlr); +} + +struct cxl_region *cxl_alloc_region (struct cxl_decoder *cxld,int id) +{ +struct cxl_region *cxlr;
Quoted 9 lines
drivers/cxl/core/region.c line 122
Quoted 4 lines
drivers/cxl/core/region.c new file +4
-0+ + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM );To keep symmetry with other device object allocations in the cxl/core I would expect to see the
device_initialize()dev->typeanddev->bussetup occur here as well.
Okay.
Quoted 54 lines
Quoted 34 lines
drivers/cxl/core/region.c new file +34
-0+ + cxlr->id = id; + +return cxlr; +} + +/** +* cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +int cxl_add_region (struct cxl_decoder *cxld,struct cxl_region *cxlr) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct device *dev = &cxlr->dev; +int rc; + +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err; +drivers/cxl/core/region.c line 166
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ rc =device_add (dev); +if (rc) + goto err; + +dev_dbg (dev,"Added to %s\n" ,dev_name (&cxld->dev)); + +return 0 ; + +err: +put_device (dev);Here is that
put_device()I was expecting, thatkfree()earlier was a double-free it seems.Also, I would have expected a devm action to remove this. Something like:
struct
cxl_port*port =to_cxl_port(cxld->dev.parent);
cxl_device_lock(&port->dev);
if (port->dev.driver)
Quoted 8 lines
devm_cxl_add_region(port->uport, cxld, id); else rc = -ENXIO;
cxl_device_unlock(&port->dev);...then no matter what you know the region will be unregistered when the root port goes away.
Right. So as I said above, you had suggested devm wasn't the right interface. I can't recall why. I can rework it to use devm, and then it should work that way.
Quoted 28 lines
Quoted 15 lines
drivers/cxl/core/region.c new file +15
-0+return rc; +} + +static struct cxl_region *cxl_find_region_by_name (struct cxl_decoder *cxld, +const char *name) +{ +struct device *region_dev; + + region_dev =device_find_child_by_name (&cxld->dev, name); +if (!region_dev) +return ERR_PTR (-ENOENT ); + +return to_cxl_region (region_dev); +} +drivers/cxl/core/region.c line 191
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+/** +* cxl_delete_region - Deletes a region +* @cxld: Parent decoder +* @region_name: Named region, ie. regionX.Y:Z +*/ +int cxl_delete_region (struct cxl_decoder *cxld,const char *region_name) +{ +struct cxl_region *cxlr; + +device_lock (&cxld->dev);
cxl_device_lock()
Quoted 2 lines
...if the lock is needed, but I don't see why the lock is needed?
I don't see one either. I can't recall if I had one previously.
Quoted 7 lines
Quoted 7 lines
drivers/cxl/core/region.c new file +7
-0+ + cxlr =cxl_find_region_by_name (cxld, region_name); +if (IS_ERR (cxlr)) { +device_unlock (&cxld->dev); +return PTR_ERR (cxlr); + } +
Quoted 9 lines
drivers/cxl/core/region.c line 203
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+dev_dbg (&cxld->dev,"Requested removal of %s from %s\n" , +dev_name (&cxlr->dev),dev_name (&cxld->dev)); + +device_unregister (&cxlr->dev); +device_unlock (&cxld->dev);This would need to change to
devm_release_action()of course if the add side changes todevm_cxl_add_region().
Yes.
Quoted 77 lines
Quoted 48 lines
drivers/cxl/core/region.c new file +5
-0+ +put_device (&cxlr->dev); + +return 0 ; +}drivers/cxl/cxl.h +9
-0@@ -221,6 +221,7 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +237,7 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct ida region_ida;int nr_targets;struct cxl_dport *target[]; };@@ -323,6 +325,13 @@ struct cxl_ep {struct list_head list; }; +bool is_cxl_region (struct device *dev); +struct cxl_region *to_cxl_region (struct device *dev); +struct cxl_region *cxl_alloc_region (struct cxl_decoder *cxld, +int interleave_ways); +int cxl_add_region (struct cxl_decoder *cxld,struct cxl_region *cxlr); +int cxl_delete_region (struct cxl_decoder *cxld,const char *region); +static inline bool is_cxl_root (struct cxl_port *port) {return port->uport == port->dev.parent;drivers/cxl/region.h new file +3
-0@@ -0,0 +1,38 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__drivers/cxl/region.h line 13
Quoted 10 lines
drivers/cxl/region.h new file +10
-0+#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This regions id. Id is globally unique across all regions. s/regions/region's/
Quoted 16 lines
drivers/cxl/region.h new file +16
-0+* @list: Node in decoder's region list. +* @res: Resource this region carves out of the platform decode range. +* @config: HDM decoder program config +* @config.size: Size of the region determined from LSA or userspace. +* @config.uuid: The UUID for this region. +* @config.interleave_ways: Number of interleave ways this region is configured for. +* @config.interleave_granularity: Interleave granularity of region +* @config.targets: The memory devices comprising the region. +*/ +struct cxl_region { +struct device dev; +int id; +struct list_head list; +struct resource *res; + +struct {
Quoted 9 lines
drivers/cxl/region.h line 35
Quoted 6 lines
drivers/cxl/region.h new file +6
-0+u64 size; +uuid_t uuid; +int interleave_ways; +int interleave_granularity; +struct cxl_memdev *targets[CXL_DECODER_MAX_INTERLEAVE ]; + } config;Why a sub-struct?
Just stylistic. I originally did have this as a named struct and used it as such. I will change it.
Quoted 18 lines
Quoted 18 lines
drivers/cxl/region.h new file +3
-0+}; + +#endif tools/testing/cxl/Kbuild +1
-1@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/ --2.35.0
Re: [PATCH v3 01/14] cxl/region: Add region creation ABI
In reply to Ben WidawskyQuoted 16 lines · On Thu, 27 Jan 2022 16:26:54 -0800 Ben Widawsky <[email protected]> wrote:
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
echo $region >/sys/bus/cxl/devices/decoder0.0/create_region
Quoted 8 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>
One trivial comment below.
Quoted 21 lines
drivers/cxl/cxl.h +2
-0@@ -221,6 +221,7 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +237,7 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct ida region_ida;int nr_targets;struct cxl_dport *target[]; };
drivers/cxl/cxl.h struct cxl_ep { · line 328
Quoted 4 lines
drivers/cxl/cxl.h +1
-0@@ -323,6 +325,13 @@ struct cxl_ep {struct list_head list; }; +bool is_cxl_region (struct device *dev);
Not in this patch. Looks like it is in patch 4.
Quoted 9 lines
drivers/cxl/cxl.h +6
-0+struct cxl_region *to_cxl_region (struct device *dev); +struct cxl_region *cxl_alloc_region (struct cxl_decoder *cxld, +int interleave_ways); +int cxl_add_region (struct cxl_decoder *cxld,struct cxl_region *cxlr); +int cxl_delete_region (struct cxl_decoder *cxld,const char *region); +static inline bool is_cxl_root (struct cxl_port *port) {return port->uport == port->dev.parent;
PATCH v4 1/14 cxl/region: Add region creation ABI
In reply to Ben WidawskyRegions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region)
// Create a new region by name
while
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region > /sys/bus/cxl/devices/decoder0.0/create_region do true; done
// Region now exists in sysfs
stat -t /sys/bus/cxl/devices/decoder0.0/$region
// Delete the region, and name
echo $region > /sys/bus/cxl/devices/decoder0.0/delete_region
---
Changes since v3:
- Change ABI for creation (cache a next) (Dan)
- For above, Use ratelimited dev_err for new ABI
- Use devm management for regions (Dan)
- Move device initialization to alloc (Dan)
- Move device alloc to add
- Change functions naming to use foo_bar_alloc instead of foo_alloc_bar
- Update commit message example for ABI change (Dan)
- Update ABI documentation (Dan)
- Update copyright date (Dan)
- Update region kdoc (Dan)
- Remove unnecessary WARN in region creation (Dan)
- Remove is_cxl_region() (for now) (Jonathan)
- Check driver binding to avoid double free of region (Dan)
--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.h
Documentation/ABI/testing/sysfs-bus-cxl +23 -0
@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.
Documentation/driver-api/cxl/memory-devices.rst +11 -0
@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================
drivers/cxl/core/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.o
drivers/cxl/core/core.h +3 -0
@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,
drivers/cxl/core/port.c +11 -0
@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,
@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }
@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);
drivers/cxl/core/region.c new file +213 -0
213 lines not on this page · raw message
drivers/cxl/cxl.h +5 -0
@@ -221,6 +221,8 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. +* @next_region_id: Cached region id for next region. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */
@@ -236,6 +238,9 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct mutex id_lock; +struct ida region_ida; +int next_region_id;int nr_targets;struct cxl_dport *target[]; };
drivers/cxl/region.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This region's id. Id is globally unique across all regions. +* @flags: Flags representing the current state of the region. +*/ +struct cxl_region { +struct device dev; +int id; +unsigned long flags; +#define REGION_DEAD 0 +}; + +#endif
tools/testing/cxl/Kbuild +1 -0
@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/
base-commit: 3bdf187d313e067de2a81109f9a1dd3da7f3dc2c
Signature · 1 line
2.35.1
PATCH v5 1/15 cxl/region: Add region creation ABI
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region)
// Create a new region by name
while
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region > /sys/bus/cxl/devices/decoder0.0/create_region do true; done
// Region now exists in sysfs
stat -t /sys/bus/cxl/devices/decoder0.0/$region
// Delete the region, and name
echo $region > /sys/bus/cxl/devices/decoder0.0/delete_region
---
Changes since v4:
- Add the missed base attributes addition
--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.h
Documentation/ABI/testing/sysfs-bus-cxl +23 -0
@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.
Documentation/driver-api/cxl/memory-devices.rst +11 -0
@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================
drivers/cxl/core/Makefile +1 -0
@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.o
drivers/cxl/core/core.h +3 -0
@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,
drivers/cxl/core/port.c +11 -0
@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,
@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }
@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);
drivers/cxl/core/region.c new file +213 -0
213 lines not on this page · raw message
drivers/cxl/cxl.h +5 -0
@@ -221,6 +221,8 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. +* @next_region_id: Cached region id for next region. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */
@@ -236,6 +238,9 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct mutex id_lock; +struct ida region_ida; +int next_region_id;int nr_targets;struct cxl_dport *target[]; };
drivers/cxl/region.h new file +23 -0
@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This region's id. Id is globally unique across all regions. +* @flags: Flags representing the current state of the region. +*/ +struct cxl_region { +struct device dev; +int id; +unsigned long flags; +#define REGION_DEAD 0 +}; + +#endif
tools/testing/cxl/Kbuild +1 -0
@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/
Signature · 1 line
2.35.1
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
Quoted 320 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; done// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>---
Changes since v4:
- Add the missed base attributes addition--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +23
-0@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +11
-0@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +154
-0@@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> +#include "core.h" + +/** +* DOC: cxl core region +* +* CXL Regions represent mapped memory capacity in system physical address +* space. Whereas the CXL Root Decoders identify the bounds of potential CXL +* Memory ranges, Regions represent the active mapped capacity by the HDM +* Decoder Capability structures throughout the Host Bridges, Switches, and +* Endpoints in the topology. +*/ + +static void cxl_region_release (struct device *dev); + +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} + +static struct cxl_region *cxl_region_alloc (struct cxl_decoder *cxld) +{ +struct cxl_region *cxlr; +struct device *dev; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM ); + + dev = &cxlr->dev; +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type; + +return cxlr; +} + +static void unregister_region (void *_cxlr) +{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev); +} + +/** +* devm_cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +static struct cxl_region *devm_cxl_add_region (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_region *cxlr; +struct device *dev; +int rc; + + cxlr =cxl_region_alloc (cxld); +if (IS_ERR (cxlr)) +return cxlr; + + dev = &cxlr->dev; + + cxlr->id = cxld->next_region_id; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err_out; + +/* affirm that release will have access to the decoder's region ida */ +get_device (&cxld->dev); + + rc =device_add (dev); +if (!rc) + rc =devm_add_action_or_reset (port->uport, unregister_region, + cxlr); +if (rc) + goto err_out; + +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr); +return ERR_PTR (rc); +} + +static ssize_t create_region_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, + cxld->next_region_id); +} + +static ssize_t create_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; +int d, p, r, rc =0 ; + +if (sscanf (buf,"region%d.%d:%d" , &p, &d, &r) !=3 ) +return -EINVAL ; + +if (port->id != p || cxld->id != d) +return -EINVAL ; + + rc =mutex_lock_interruptible (&cxld->id_lock); +if (rc) +return rc; + +if (cxld->next_region_id != r) { + rc = -EINVAL ; + goto out; + } + + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) { +dev_dbg (dev,"Failed to get next cached id (%d)\n" , rc); + goto out; + } + + cxlr =devm_cxl_add_region (cxld); +if (IS_ERR (cxlr)) {
drivers/cxl/core/region.c line 164
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ rc =PTR_ERR (cxlr); + goto out; + } + + cxld->next_region_id = rc; +dev_dbg (dev,"Created %s\n" ,dev_name (&cxlr->dev)); + +out: +mutex_unlock (&cxld->id_lock); +return rc ? rc : len;
Sigh. This is incorrect after a last minute change I made. While the regions do get created you get back an errno here based on the ID allocated. Will not bother resending v6 until there's some other review to avoid more thrash.
Quoted 116 lines
drivers/cxl/core/region.c new file +49
-0+} +DEVICE_ATTR_RW (create_region); + +static struct cxl_region *cxl_find_region_by_name (struct cxl_decoder *cxld, +const char *name) +{ +struct device *region_dev; + + region_dev =device_find_child_by_name (&cxld->dev, name); +if (!region_dev) +return ERR_PTR (-ENOENT ); + +return to_cxl_region (region_dev); +} + +static ssize_t delete_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; + + cxlr =cxl_find_region_by_name (cxld, buf); +if (IS_ERR (cxlr)) +return PTR_ERR (cxlr); + +/* After this, the region is no longer a child of the decoder. */ +devm_release_action (port->uport, unregister_region, cxlr); + +/* Release is likely called here, so cxlr is not safe to reference. */ +put_device (&cxlr->dev); + cxlr =NULL ; + +dev_dbg (dev,"Deleted %s\n" , buf); +return len; +} +DEVICE_ATTR_WO (delete_region); + +static void cxl_region_release (struct device *dev) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); + +dev_dbg (&cxld->dev,"Releasing %s\n" ,dev_name (dev)); +ida_free (&cxld->region_ida, cxlr->id); +kfree (cxlr); +put_device (&cxld->dev); +}drivers/cxl/cxl.h +5
-0@@ -221,6 +221,8 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. +* @next_region_id: Cached region id for next region. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +238,9 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct mutex id_lock; +struct ida region_ida; +int next_region_id;int nr_targets;struct cxl_dport *target[]; };drivers/cxl/region.h new file +23
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This region's id. Id is globally unique across all regions. +* @flags: Flags representing the current state of the region. +*/ +struct cxl_region { +struct device dev; +int id; +unsigned long flags; +#define REGION_DEAD 0 +}; + +#endif tools/testing/cxl/Kbuild +1
-0@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/Signature · 1 line
2.35.1
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
In reply to Ben WidawskyQuoted 20 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; done
Quoted 7 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>
Looking good, a few more fixes and cleanups identified below.
Quoted 151 lines
---
Changes since v4:
- Add the missed base attributes addition--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +23
-0@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +11
-0@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +11
-0@@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> +#include "core.h" + +/**
drivers/cxl/core/region.c line 21
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+* DOC: cxl core region +* +* CXL Regions represent mapped memory capacity in system physical address +* space. Whereas the CXL Root Decoders identify the bounds of potential CXL +* Memory ranges, Regions represent the active mapped capacity by the HDM +* Decoder Capability structures throughout the Host Bridges, Switches, and +* Endpoints in the topology. +*/ + +static void cxl_region_release (struct device *dev);
Why forward declare this versus move cxl_region_type after the definition?
No other CXL object release functions are forward declared.
Quoted 30 lines
drivers/cxl/core/region.c new file +30
-0+ +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} + +static struct cxl_region *cxl_region_alloc (struct cxl_decoder *cxld) +{ +struct cxl_region *cxlr; +struct device *dev; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM ); + + dev = &cxlr->dev; +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type;
drivers/cxl/core/region.c line 61
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +return cxlr; +} + +static void unregister_region (void *_cxlr) +{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev);
I thought REGION_DEAD was needed to prevent double devm_release_action(), not double unregister?
Quoted 32 lines
drivers/cxl/core/region.c new file +32
-0+} + +/** +* devm_cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +static struct cxl_region *devm_cxl_add_region (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_region *cxlr; +struct device *dev; +int rc; + + cxlr =cxl_region_alloc (cxld); +if (IS_ERR (cxlr)) +return cxlr; + + dev = &cxlr->dev; + + cxlr->id = cxld->next_region_id; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err_out;
drivers/cxl/core/region.c line 103
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +/* affirm that release will have access to the decoder's region ida */ +get_device (&cxld->dev); + + rc =device_add (dev); +if (!rc) + rc =devm_add_action_or_reset (port->uport, unregister_region, + cxlr); +if (rc) + goto err_out;
All the other usages in device_add() in the subsystem follow the style of:
rc = device_add(dev);
if (rc)
goto err;
...any reason to be unique here and indent the success case?
drivers/cxl/core/region.c line 109
Quoted 6 lines
drivers/cxl/core/region.c new file +6
-0+ +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr);
This is a double-free of cxlr;
Quoted 40 lines
drivers/cxl/core/region.c new file +40
-0+return ERR_PTR (rc); +} + +static ssize_t create_region_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, + cxld->next_region_id); +} + +static ssize_t create_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; +int d, p, r, rc =0 ; + +if (sscanf (buf,"region%d.%d:%d" , &p, &d, &r) !=3 ) +return -EINVAL ; + +if (port->id != p || cxld->id != d) +return -EINVAL ; + + rc =mutex_lock_interruptible (&cxld->id_lock); +if (rc) +return rc; + +if (cxld->next_region_id != r) { + rc = -EINVAL ; + goto out; + } + + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) { +dev_dbg (dev,"Failed to get next cached id (%d)\n" , rc);
drivers/cxl/core/region.c line 159
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ goto out; + } + + cxlr =devm_cxl_add_region (cxld); +if (IS_ERR (cxlr)) { + rc =PTR_ERR (cxlr); + goto out; + } + + cxld->next_region_id = rc;
This looks like a leak in the case when devm_cxl_add_region() fails, so just move it above that call.
drivers/cxl/core/region.c line 164
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+dev_dbg (dev,"Created %s\n" ,dev_name (&cxlr->dev)); + +out: +mutex_unlock (&cxld->id_lock); +return rc ? rc : len;
if (rc)
return rc;
return len;
Quoted 19 lines
drivers/cxl/core/region.c new file +19
-0+} +DEVICE_ATTR_RW (create_region); + +static struct cxl_region *cxl_find_region_by_name (struct cxl_decoder *cxld, +const char *name) +{ +struct device *region_dev; + + region_dev =device_find_child_by_name (&cxld->dev, name); +if (!region_dev) +return ERR_PTR (-ENOENT ); + +return to_cxl_region (region_dev); +} + +static ssize_t delete_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{
drivers/cxl/core/region.c line 193
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; + + cxlr =cxl_find_region_by_name (cxld, buf); +if (IS_ERR (cxlr)) +return PTR_ERR (cxlr); + +/* After this, the region is no longer a child of the decoder. */ +devm_release_action (port->uport, unregister_region, cxlr);
This may trigger a WARN in the case where 2 threads race to trigger the release action. I think the DEAD check is needed to gate this call, not device_unregister().
drivers/cxl/core/region.c line 195
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+ +/* Release is likely called here, so cxlr is not safe to reference. */
This is always the case with any put_device(), so no need for this comment.
drivers/cxl/core/region.c line 197
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+put_device (&cxlr->dev); + cxlr =NULL ;
This NULL assignment has no value.
drivers/cxl/core/region.c line 199
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+ +dev_dbg (dev,"Deleted %s\n" , buf);
Not sure a debug statement is needed for something userspace can directly view itself with the result code from the sysfs write.
Quoted 81 lines
drivers/cxl/core/region.c new file +14
-0+return len; +} +DEVICE_ATTR_WO (delete_region); + +static void cxl_region_release (struct device *dev) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); + +dev_dbg (&cxld->dev,"Releasing %s\n" ,dev_name (dev)); +ida_free (&cxld->region_ida, cxlr->id); +kfree (cxlr); +put_device (&cxld->dev); +}drivers/cxl/cxl.h +5
-0@@ -221,6 +221,8 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. +* @next_region_id: Cached region id for next region. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +238,9 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct mutex id_lock; +struct ida region_ida; +int next_region_id;int nr_targets;struct cxl_dport *target[]; };drivers/cxl/region.h new file +23
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This region's id. Id is globally unique across all regions. +* @flags: Flags representing the current state of the region. +*/ +struct cxl_region { +struct device dev; +int id; +unsigned long flags; +#define REGION_DEAD 0 +}; + +#endif tools/testing/cxl/Kbuild +1
-1@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/ --2.35.1
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
Quoted 195 lines · Dan Williams
Quoted 20 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; doneQuoted 7 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>Looking good, a few more fixes and cleanups identified below.
Quoted 151 lines
---
Changes since v4:
- Add the missed base attributes addition--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +23
-0@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +11
-0@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +11
-0@@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> +#include "core.h" + +/** drivers/cxl/core/region.c line 21
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+* DOC: cxl core region +* +* CXL Regions represent mapped memory capacity in system physical address +* space. Whereas the CXL Root Decoders identify the bounds of potential CXL +* Memory ranges, Regions represent the active mapped capacity by the HDM +* Decoder Capability structures throughout the Host Bridges, Switches, and +* Endpoints in the topology. +*/ + +static void cxl_region_release (struct device *dev);Why forward declare this versus move
cxl_region_typeafter the definition?
Quoted 2 lines
No other CXL object release functions are forward declared.
I liked having the device_type declared at the top. I will change it to match the other code.
Quoted 35 lines
Quoted 35 lines
drivers/cxl/core/region.c new file +35
-0+ +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} + +static struct cxl_region *cxl_region_alloc (struct cxl_decoder *cxld) +{ +struct cxl_region *cxlr; +struct device *dev; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM ); + + dev = &cxlr->dev; +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type; + +return cxlr; +} + +static void unregister_region (void *_cxlr)
Quoted 9 lines
drivers/cxl/core/region.c line 61
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev);I thought
REGION_DEADwas needed to prevent doubledevm_release_action(), not double unregister?
I believe that's correct, repeating what you said on our internal list:
On 22-02-14 14:11:41, Dan Williams wrote:
True, you do need to solve the race between multiple writers racing to do the unregistration, but that could be done with something like:
if (!test_and_set_bit(REGION_DEAD, &cxlr->flags))
device_unregister(&cxlr->dev);So I was just trying to implement what you said. Remainder of the discussion below...
Quoted 45 lines
Quoted 32 lines
drivers/cxl/core/region.c new file +32
-0+} + +/** +* devm_cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +static struct cxl_region *devm_cxl_add_region (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_region *cxlr; +struct device *dev; +int rc; + + cxlr =cxl_region_alloc (cxld); +if (IS_ERR (cxlr)) +return cxlr; + + dev = &cxlr->dev; + + cxlr->id = cxld->next_region_id; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err_out;drivers/cxl/core/region.c line 103
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +/* affirm that release will have access to the decoder's region ida */ +get_device (&cxld->dev); + + rc =device_add (dev); +if (!rc) + rc =devm_add_action_or_reset (port->uport, unregister_region, + cxlr); +if (rc) + goto err_out;All the other usages in
device_add()in the subsystem follow the style of:
Quoted 6 lines
rc =
device_add(dev);
if (rc)goto err;...any reason to be unique here and indent the success case?
It allowed skipping a goto. I can change it.
Quoted 10 lines
drivers/cxl/core/region.c line 109
Quoted 7 lines
drivers/cxl/core/region.c new file +6
-0+ +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr);This is a double-free of cxlr;
Because of release()? How does release get called if the region device wasn't added? Or is there something else?
Quoted 61 lines
Quoted 40 lines
drivers/cxl/core/region.c new file +40
-0+return ERR_PTR (rc); +} + +static ssize_t create_region_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, + cxld->next_region_id); +} + +static ssize_t create_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; +int d, p, r, rc =0 ; + +if (sscanf (buf,"region%d.%d:%d" , &p, &d, &r) !=3 ) +return -EINVAL ; + +if (port->id != p || cxld->id != d) +return -EINVAL ; + + rc =mutex_lock_interruptible (&cxld->id_lock); +if (rc) +return rc; + +if (cxld->next_region_id != r) { + rc = -EINVAL ; + goto out; + } + + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) { +dev_dbg (dev,"Failed to get next cached id (%d)\n" , rc);drivers/cxl/core/region.c line 159
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ goto out; + } + + cxlr =devm_cxl_add_region (cxld); +if (IS_ERR (cxlr)) { + rc =PTR_ERR (cxlr); + goto out; + } + + cxld->next_region_id = rc;This looks like a leak in the case when
devm_cxl_add_region()fails, so just move it above that call.drivers/cxl/core/region.c line 164
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+dev_dbg (dev,"Created %s\n" ,dev_name (&cxlr->dev)); + +out: +mutex_unlock (&cxld->id_lock); +return rc ? rc : len;if (rc)
Quoted 3 lines
return rc;return len;
Yeah. This was a bug I pointed out already. I have it fixed locally.
Quoted 25 lines
Quoted 25 lines
drivers/cxl/core/region.c new file +25
-0+} +DEVICE_ATTR_RW (create_region); + +static struct cxl_region *cxl_find_region_by_name (struct cxl_decoder *cxld, +const char *name) +{ +struct device *region_dev; + + region_dev =device_find_child_by_name (&cxld->dev, name); +if (!region_dev) +return ERR_PTR (-ENOENT ); + +return to_cxl_region (region_dev); +} + +static ssize_t delete_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; + + cxlr =cxl_find_region_by_name (cxld, buf); +if (IS_ERR (cxlr))
Quoted 8 lines
drivers/cxl/core/region.c line 193
Quoted 4 lines
drivers/cxl/core/region.c new file +4
-0+return PTR_ERR (cxlr); + +/* After this, the region is no longer a child of the decoder. */ +devm_release_action (port->uport, unregister_region, cxlr);This may trigger a WARN in the case where 2 threads race to trigger the release action. I think the DEAD check is needed to gate this call, not
device_unregister().
Continuing from above use of REGION_DEAD....
Dang. I actually had added a mutex to protect this and I cherry-picked the wrong patch.
This looks good to me though:
cxlr = cxl_find_region_by_name(cxld, buf);
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
if (!test_and_set_bit(REGION_DEAD, &cxlr->flags)) {
devm_release_action(...)
}put_device(dev);
return len;
Quoted 10 lines
drivers/cxl/core/region.c line 195
Quoted 3 lines
drivers/cxl/core/region.c new file +2
-0+ +/* Release is likely called here, so cxlr is not safe to reference. */ This is always the case with any
put_device(), so no need for this comment.drivers/cxl/core/region.c line 197
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+put_device (&cxlr->dev); + cxlr =NULL ;This NULL assignment has no value.
It was meant to prevent future me from trying to use cxlr after this point, which I've done. I can remove it.
Quoted 7 lines
drivers/cxl/core/region.c line 199
Quoted 3 lines
drivers/cxl/core/region.c new file +2
-0+ +dev_dbg (dev,"Deleted %s\n" , buf);Not sure a debug statement is needed for something userspace can directly view itself with the result code from the sysfs write.
It was helpful for my driver development, but I can remove it.
Quoted 81 lines
Quoted 81 lines
drivers/cxl/core/region.c new file +14
-0+return len; +} +DEVICE_ATTR_WO (delete_region); + +static void cxl_region_release (struct device *dev) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); + +dev_dbg (&cxld->dev,"Releasing %s\n" ,dev_name (dev)); +ida_free (&cxld->region_ida, cxlr->id); +kfree (cxlr); +put_device (&cxld->dev); +}drivers/cxl/cxl.h +5
-0@@ -221,6 +221,8 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. +* @next_region_id: Cached region id for next region. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +238,9 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct mutex id_lock; +struct ida region_ida; +int next_region_id;int nr_targets;struct cxl_dport *target[]; };drivers/cxl/region.h new file +23
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This region's id. Id is globally unique across all regions. +* @flags: Flags representing the current state of the region. +*/ +struct cxl_region { +struct device dev; +int id; +unsigned long flags; +#define REGION_DEAD 0 +}; + +#endif tools/testing/cxl/Kbuild +1
-1@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/ --2.35.1
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
Quoted 30 lines · Ben Widawsky
Quoted 28 lines · Dan Williams
Quoted 27 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; done// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>
[..]
Quoted 17 lines
Quoted 10 lines
drivers/cxl/core/region.c line 61
Quoted 6 lines
drivers/cxl/core/region.c new file +6
-0+static void unregister_region (void *_cxlr) +{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev);I thought
REGION_DEADwas needed to prevent doubledevm_release_action(), not double unregister?I believe that's correct, repeating what you said on our internal list:
On 22-02-14 14:11:41, Dan Williams wrote:
True, you do need to solve the race between multiple writers racing to do the unregistration, but that could be done with something like:
Quoted 5 lines
if (!test_and_set_bit(REGION_DEAD, &cxlr->flags)) device_unregister(&cxlr->dev);So I was just trying to implement what you said. Remainder of the discussion below...
That was in the context of moving the unregistration to a workqueue and taking the device lock to validate whether the device has already been unbound. In this case keeping the devm_release_action() inline in the sysfs attribute the flag needs to protect against racing devm_release_action(). I am not saying that a workqueue is now needed, just clarifying the context of that suggestion.
[..]
Quoted 10 lines
Quoted 9 lines
drivers/cxl/core/region.c line 109
Quoted 6 lines
drivers/cxl/core/region.c new file +6
-0+ +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr);This is a double-free of cxlr;
Quoted 2 lines
Because of
release()? How does release get called if the region device wasn't added? Or is there something else?
->release() is always called at final put_device() regardless of whether the device was registered with device_add() or not. I.e. see all the other dev_set_name() error handling in the core that just does put_device().
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
In reply to Dan WilliamsQuoted 347 lines · Dan Williams
Quoted 20 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; doneQuoted 7 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>Looking good, a few more fixes and cleanups identified below.
Quoted 151 lines
---
Changes since v4:
- Add the missed base attributes addition--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +23
-0@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +11
-0@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +11
-0@@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> +#include "core.h" + +/** drivers/cxl/core/region.c line 21
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+* DOC: cxl core region +* +* CXL Regions represent mapped memory capacity in system physical address +* space. Whereas the CXL Root Decoders identify the bounds of potential CXL +* Memory ranges, Regions represent the active mapped capacity by the HDM +* Decoder Capability structures throughout the Host Bridges, Switches, and +* Endpoints in the topology. +*/ + +static void cxl_region_release (struct device *dev);Why forward declare this versus move
cxl_region_typeafter the definition?No other CXL object release functions are forward declared.
Quoted 30 lines
drivers/cxl/core/region.c new file +30
-0+ +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} + +static struct cxl_region *cxl_region_alloc (struct cxl_decoder *cxld) +{ +struct cxl_region *cxlr; +struct device *dev; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM ); + + dev = &cxlr->dev; +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type;drivers/cxl/core/region.c line 61
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +return cxlr; +} + +static void unregister_region (void *_cxlr) +{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev);I thought
REGION_DEADwas needed to prevent doubledevm_release_action(), not double unregister?Quoted 32 lines
drivers/cxl/core/region.c new file +32
-0+} + +/** +* devm_cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +static struct cxl_region *devm_cxl_add_region (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_region *cxlr; +struct device *dev; +int rc; + + cxlr =cxl_region_alloc (cxld); +if (IS_ERR (cxlr)) +return cxlr; + + dev = &cxlr->dev; + + cxlr->id = cxld->next_region_id; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err_out;drivers/cxl/core/region.c line 103
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +/* affirm that release will have access to the decoder's region ida */ +get_device (&cxld->dev); + + rc =device_add (dev); +if (!rc) + rc =devm_add_action_or_reset (port->uport, unregister_region, + cxlr); +if (rc) + goto err_out;All the other usages in
device_add()in the subsystem follow the style of:rc =
device_add(dev);
if (rc)goto err;...any reason to be unique here and indent the success case?
drivers/cxl/core/region.c line 109
Quoted 6 lines
drivers/cxl/core/region.c new file +6
-0+ +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr);This is a double-free of cxlr;
Quoted 45 lines
drivers/cxl/core/region.c new file +45
-0+return ERR_PTR (rc); +} + +static ssize_t create_region_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, + cxld->next_region_id); +} + +static ssize_t create_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; +int d, p, r, rc =0 ; + +if (sscanf (buf,"region%d.%d:%d" , &p, &d, &r) !=3 ) +return -EINVAL ; + +if (port->id != p || cxld->id != d) +return -EINVAL ; + + rc =mutex_lock_interruptible (&cxld->id_lock); +if (rc) +return rc; + +if (cxld->next_region_id != r) { + rc = -EINVAL ; + goto out; + } + + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) { +dev_dbg (dev,"Failed to get next cached id (%d)\n" , rc); + goto out; + } + + cxlr =devm_cxl_add_region (cxld); +if (IS_ERR (cxlr)) {
Quoted 9 lines
drivers/cxl/core/region.c line 159
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+ rc =PTR_ERR (cxlr); + goto out; + } + + cxld->next_region_id = rc;This looks like a leak in the case when
devm_cxl_add_region()fails, so just move it above that call.
It's not super simple with the current pre-caching. If you move this above devm_cxl_add_region(), then you lose the previously pre-cached region. I think the cleaner solution is to just free the ida on failure. Pretty sure no matter what method you go, you need an ida_free in there somewhere. Do you see another way?
Quoted 141 lines
drivers/cxl/core/region.c line 164
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+dev_dbg (dev,"Created %s\n" ,dev_name (&cxlr->dev)); + +out: +mutex_unlock (&cxld->id_lock); +return rc ? rc : len;if (rc)
return rc;return len;
Quoted 19 lines
drivers/cxl/core/region.c new file +19
-0+} +DEVICE_ATTR_RW (create_region); + +static struct cxl_region *cxl_find_region_by_name (struct cxl_decoder *cxld, +const char *name) +{ +struct device *region_dev; + + region_dev =device_find_child_by_name (&cxld->dev, name); +if (!region_dev) +return ERR_PTR (-ENOENT ); + +return to_cxl_region (region_dev); +} + +static ssize_t delete_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{drivers/cxl/core/region.c line 193
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; + + cxlr =cxl_find_region_by_name (cxld, buf); +if (IS_ERR (cxlr)) +return PTR_ERR (cxlr); + +/* After this, the region is no longer a child of the decoder. */ +devm_release_action (port->uport, unregister_region, cxlr);This may trigger a WARN in the case where 2 threads race to trigger the release action. I think the DEAD check is needed to gate this call, not
device_unregister().drivers/cxl/core/region.c line 195
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+ +/* Release is likely called here, so cxlr is not safe to reference. */ This is always the case with any
put_device(), so no need for this comment.drivers/cxl/core/region.c line 197
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+put_device (&cxlr->dev); + cxlr =NULL ;This NULL assignment has no value.
drivers/cxl/core/region.c line 199
Quoted 2 lines
drivers/cxl/core/region.c new file +2
-0+ +dev_dbg (dev,"Deleted %s\n" , buf);Not sure a debug statement is needed for something userspace can directly view itself with the result code from the sysfs write.
Quoted 81 lines
drivers/cxl/core/region.c new file +14
-0+return len; +} +DEVICE_ATTR_WO (delete_region); + +static void cxl_region_release (struct device *dev) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); + +dev_dbg (&cxld->dev,"Releasing %s\n" ,dev_name (dev)); +ida_free (&cxld->region_ida, cxlr->id); +kfree (cxlr); +put_device (&cxld->dev); +}drivers/cxl/cxl.h +5
-0@@ -221,6 +221,8 @@ enum cxl_decoder_type {* @target_type: accelerator vs expander (type2 vs type3) selector * @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list +* @region_ida: allocator for region ids. +* @next_region_id: Cached region id for next region. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -236,6 +238,9 @@ struct cxl_decoder {enum cxl_decoder_type target_type;unsigned long flags;seqlock_t target_lock; +struct mutex id_lock; +struct ida region_ida; +int next_region_id;int nr_targets;struct cxl_dport *target[]; };drivers/cxl/region.h new file +23
-0@@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* Copyright(c) 2021 Intel Corporation. */ +#ifndef __CXL_REGION_H__ +#define __CXL_REGION_H__ + +#include <linux/uuid.h> + +#include "cxl.h" + +/** +* struct cxl_region - CXL region +* @dev: This region's device. +* @id: This region's id. Id is globally unique across all regions. +* @flags: Flags representing the current state of the region. +*/ +struct cxl_region { +struct device dev; +int id; +unsigned long flags; +#define REGION_DEAD 0 +}; + +#endif tools/testing/cxl/Kbuild +1
-1@@ -46,6 +46,7 @@ cxl_core-y += $(CXL_CORE_SRC)/memdev.o cxl_core-y += $(CXL_CORE_SRC)/mbox.o cxl_core-y += $(CXL_CORE_SRC)/pci.o cxl_core-y += $(CXL_CORE_SRC)/hdm.o +cxl_core-y += $(CXL_CORE_SRC)/region.o cxl_core-y += config_check.o obj-m += test/ --2.35.1
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
Quoted 359 lines · Ben Widawsky
Quoted 347 lines · Dan Williams
Quoted 20 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; doneQuoted 7 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>Looking good, a few more fixes and cleanups identified below.
Quoted 151 lines
---
Changes since v4:
- Add the missed base attributes addition--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +23
-0@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +11
-0@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +11
-0@@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> +#include "core.h" + +/** drivers/cxl/core/region.c line 21
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+* DOC: cxl core region +* +* CXL Regions represent mapped memory capacity in system physical address +* space. Whereas the CXL Root Decoders identify the bounds of potential CXL +* Memory ranges, Regions represent the active mapped capacity by the HDM +* Decoder Capability structures throughout the Host Bridges, Switches, and +* Endpoints in the topology. +*/ + +static void cxl_region_release (struct device *dev);Why forward declare this versus move
cxl_region_typeafter the definition?No other CXL object release functions are forward declared.
Quoted 30 lines
drivers/cxl/core/region.c new file +30
-0+ +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} + +static struct cxl_region *cxl_region_alloc (struct cxl_decoder *cxld) +{ +struct cxl_region *cxlr; +struct device *dev; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM ); + + dev = &cxlr->dev; +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type;drivers/cxl/core/region.c line 61
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +return cxlr; +} + +static void unregister_region (void *_cxlr) +{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev);I thought
REGION_DEADwas needed to prevent doubledevm_release_action(), not double unregister?Quoted 32 lines
drivers/cxl/core/region.c new file +32
-0+} + +/** +* devm_cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +static struct cxl_region *devm_cxl_add_region (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_region *cxlr; +struct device *dev; +int rc; + + cxlr =cxl_region_alloc (cxld); +if (IS_ERR (cxlr)) +return cxlr; + + dev = &cxlr->dev; + + cxlr->id = cxld->next_region_id; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err_out;drivers/cxl/core/region.c line 103
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +/* affirm that release will have access to the decoder's region ida */ +get_device (&cxld->dev); + + rc =device_add (dev); +if (!rc) + rc =devm_add_action_or_reset (port->uport, unregister_region, + cxlr); +if (rc) + goto err_out;All the other usages in
device_add()in the subsystem follow the style of:rc =
device_add(dev);
if (rc)goto err;...any reason to be unique here and indent the success case?
drivers/cxl/core/region.c line 109
Quoted 6 lines
drivers/cxl/core/region.c new file +6
-0+ +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr);This is a double-free of cxlr;
Quoted 45 lines
drivers/cxl/core/region.c new file +45
-0+return ERR_PTR (rc); +} + +static ssize_t create_region_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, + cxld->next_region_id); +} + +static ssize_t create_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; +int d, p, r, rc =0 ; + +if (sscanf (buf,"region%d.%d:%d" , &p, &d, &r) !=3 ) +return -EINVAL ; + +if (port->id != p || cxld->id != d) +return -EINVAL ; + + rc =mutex_lock_interruptible (&cxld->id_lock); +if (rc) +return rc; + +if (cxld->next_region_id != r) { + rc = -EINVAL ; + goto out; + } + + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) { +dev_dbg (dev,"Failed to get next cached id (%d)\n" , rc); + goto out; + } + + cxlr =devm_cxl_add_region (cxld); +if (IS_ERR (cxlr)) {Quoted 9 lines
drivers/cxl/core/region.c line 159
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+ rc =PTR_ERR (cxlr); + goto out; + } + + cxld->next_region_id = rc;This looks like a leak in the case when
devm_cxl_add_region()fails, so just move it above that call.
Quoted 5 lines
It's not super simple with the current pre-caching. If you move this above
devm_cxl_add_region(), then you lose the previously pre-cached region. I think the cleaner solution is to just free the ida on failure. Pretty sure no matter what method you go, you need anida_freein there somewhere. Do you see another way?
As soon as one thread has successfully acquired the next_region_id then it is safe to advance and assume that the devm_cxl_add_region() owns releasing that id.
In fact, just make that implicit. Move the ida_alloc() and the next_region_id advancement internal to cxl_region_alloc() with a device_lock_assert() for the id_lock. Then the recovery of the allocated id happens naturally like all the other ids in the subsystem i.e. at release time.
Re: [PATCH v5 01/15] cxl/region: Add region creation ABI
Quoted 370 lines · Dan Williams
Quoted 359 lines · Ben Widawsky
Quoted 347 lines · Dan Williams
Quoted 20 lines · Ben Widawsky
Regions are created as a child of the decoder that encompasses an address space with constraints. Regions have a number of attributes that must be configured before the region can be activated.
The ABI is not meant to be secure, but is meant to avoid accidental races. As a result, a buggy process may create a region by name that was allocated by a different process. However, multiple processes which are trying not to race with each other shouldn't need special synchronization to do so.
// Allocate a new region name
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region)// Create a new region by name
while
region=$(cat/sys/bus/cxl/devices/decoder0.0/create_region) ! echo $region >/sys/bus/cxl/devices/decoder0.0/create_regiondo true; doneQuoted 7 lines
// Region now exists in sysfs
stat -t/sys/bus/cxl/devices/decoder0.0/$region// Delete the region, and name
echo $region >/sys/bus/cxl/devices/decoder0.0/delete_regionSigned-off-by Ben Widawsky <[email protected]>Looking good, a few more fixes and cleanups identified below.
Quoted 151 lines
---
Changes since v4:
- Add the missed base attributes addition--- Documentation/ABI/testing/sysfs-bus-cxl | 23 ++ .../driver-api/cxl/memory-devices.rst | 11 + drivers/cxl/core/Makefile | 1 + drivers/cxl/core/core.h | 3 + drivers/cxl/core/port.c | 11 + drivers/cxl/core/region.c | 213 ++++++++++++++++++ drivers/cxl/cxl.h | 5 + drivers/cxl/region.h | 23 ++ tools/testing/cxl/Kbuild | 1 + 9 files changed, 291 insertions(+) create mode 100644 drivers/cxl/core/region.c create mode 100644 drivers/cxl/region.hDocumentation/ABI/testing/sysfs-bus-cxl +23
-0@@ -163,3 +163,26 @@ Description: memory (type-3). The 'target_type' attribute indicates the current setting which may dynamically change based on what memory regions are activated in this decode hierarchy. + +What: /sys/bus/cxl/devices/decoderX.Y/create_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Write a value of the form 'regionX.Y:Z' to instantiate a new + region within the decode range bounded by decoderX.Y. The value + written must match the current value returned from reading this + attribute. This behavior lets the kernel arbitrate racing + attempts to create a region. The thread that fails to write + loops and tries the next value. Regions must be created for root + decoders, and must subsequently configured and bound to a region + driver before they can be used. + +What: /sys/bus/cxl/devices/decoderX.Y/delete_region +Date: January, 2022 +KernelVersion: v5.18 +Contact: [email protected] +Description: + Deletes the named region. The attribute expects a region in the + form "regionX.Y:Z". The region's name, allocated by reading + create_region, will also be released.Documentation/driver-api/cxl/memory-devices.rst +11
-0@@ -362,6 +362,17 @@ CXL Core .. kernel-doc:: drivers/cxl/core/mbox.c :doc: cxl mbox +CXL Regions +----------- +.. kernel-doc:: drivers/cxl/region.h + :identifiers: + +.. kernel-doc:: drivers/cxl/core/region.c + :doc: cxl core region + +.. kernel-doc:: drivers/cxl/core/region.c + :identifiers: + External Interfaces ===================drivers/cxl/core/Makefile +1
-0@@ -4,6 +4,7 @@ obj-$(CONFIG_CXL_BUS) += cxl_core.o ccflags-y += -I$(srctree)/drivers/cxl cxl_core-y := port.o cxl_core-y += pmem.o +cxl_core-y += region.o cxl_core-y += regs.o cxl_core-y += memdev.o cxl_core-y += mbox.odrivers/cxl/core/core.h +3
-0@@ -9,6 +9,9 @@ extern const struct device_type cxl_nvdimm_type;extern struct attribute_group cxl_base_attribute_group; +extern struct device_attribute dev_attr_create_region; +extern struct device_attribute dev_attr_delete_region; +struct cxl_send_command ;struct cxl_mem_query_commands ; int cxl_query_cmd(struct cxl_memdev *cxlmd,drivers/cxl/core/port.c +11
-0@@ -213,6 +213,8 @@ static struct attribute_group cxl_decoder_base_attribute_group = { };static struct attribute *cxl_decoder_root_attrs[] = { + &dev_attr_create_region.attr, + &dev_attr_delete_region.attr, &dev_attr_cap_pmem.attr, &dev_attr_cap_ram.attr, &dev_attr_cap_type2.attr,@@ -270,6 +272,8 @@ static void cxl_decoder_release(struct device *dev)struct cxl_decoder *cxld =to_cxl_decoder (dev);struct cxl_port *port =to_cxl_port (dev->parent); +ida_free (&cxld->region_ida, cxld->next_region_id); +ida_destroy (&cxld->region_ida);ida_free (&port->decoder_ida, cxld->id);kfree (cxld); }@@ -1244,6 +1248,13 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +mutex_init (&cxld->id_lock); +ida_init (&cxld->region_ida); + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) + goto err; + + cxld->next_region_id = rc;return cxld; err:kfree (cxld);drivers/cxl/core/region.c new file +11
-0@@ -0,0 +1,213 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2022 Intel Corporation. All rights reserved. */ +#include <linux/device.h> +#include <linux/module.h> +#include <linux/slab.h> +#include <linux/idr.h> +#include <region.h> +#include <cxl.h> +#include "core.h" + +/** drivers/cxl/core/region.c line 21
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+* DOC: cxl core region +* +* CXL Regions represent mapped memory capacity in system physical address +* space. Whereas the CXL Root Decoders identify the bounds of potential CXL +* Memory ranges, Regions represent the active mapped capacity by the HDM +* Decoder Capability structures throughout the Host Bridges, Switches, and +* Endpoints in the topology. +*/ + +static void cxl_region_release (struct device *dev);Why forward declare this versus move
cxl_region_typeafter the definition?No other CXL object release functions are forward declared.
Quoted 30 lines
drivers/cxl/core/region.c new file +30
-0+ +static const struct device_type cxl_region_type = { + .name ="cxl_region" , + .release = cxl_region_release, +}; + +static struct cxl_region *to_cxl_region (struct device *dev) +{ +if (dev_WARN_ONCE (dev, dev->type != &cxl_region_type, +"not a cxl_region device\n" )) +return NULL ; + + return container_of(dev, struct cxl_region, dev); +} + +static struct cxl_region *cxl_region_alloc (struct cxl_decoder *cxld) +{ +struct cxl_region *cxlr; +struct device *dev; + + cxlr =kzalloc (sizeof (*cxlr),GFP_KERNEL ); +if (!cxlr) +return ERR_PTR (-ENOMEM ); + + dev = &cxlr->dev; +device_initialize (dev); + dev->parent = &cxld->dev; +device_set_pm_not_required (dev); + dev->bus = &cxl_bus_type; + dev->type = &cxl_region_type;drivers/cxl/core/region.c line 61
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +return cxlr; +} + +static void unregister_region (void *_cxlr) +{ +struct cxl_region *cxlr = _cxlr; + +if (!test_and_set_bit (REGION_DEAD , &cxlr->flags)) +device_unregister (&cxlr->dev);I thought
REGION_DEADwas needed to prevent doubledevm_release_action(), not double unregister?Quoted 32 lines
drivers/cxl/core/region.c new file +32
-0+} + +/** +* devm_cxl_add_region - Adds a region to a decoder +* @cxld: Parent decoder. +* @cxlr: Region to be added to the decoder. +* +* This is the second step of region initialization. Regions exist within an +* address space which is mapped by a @cxld. That @cxld must be a root decoder, +* and it enforces constraints upon the region as it is configured. +* +* Return: 0 if the region was added to the @cxld, else returns negative error +* code. The region will be named "regionX.Y.Z" where X is the port, Y is the +* decoder id, and Z is the region number. +*/ +static struct cxl_region *devm_cxl_add_region (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_region *cxlr; +struct device *dev; +int rc; + + cxlr =cxl_region_alloc (cxld); +if (IS_ERR (cxlr)) +return cxlr; + + dev = &cxlr->dev; + + cxlr->id = cxld->next_region_id; + rc =dev_set_name (dev,"region%d.%d:%d" , port->id, cxld->id, cxlr->id); +if (rc) + goto err_out;drivers/cxl/core/region.c line 103
Quoted 10 lines
drivers/cxl/core/region.c new file +10
-0+ +/* affirm that release will have access to the decoder's region ida */ +get_device (&cxld->dev); + + rc =device_add (dev); +if (!rc) + rc =devm_add_action_or_reset (port->uport, unregister_region, + cxlr); +if (rc) + goto err_out;All the other usages in
device_add()in the subsystem follow the style of:rc =
device_add(dev);
if (rc)goto err;...any reason to be unique here and indent the success case?
drivers/cxl/core/region.c line 109
Quoted 6 lines
drivers/cxl/core/region.c new file +6
-0+ +return cxlr; + +err_out: +put_device (dev); +kfree (cxlr);This is a double-free of cxlr;
Quoted 45 lines
drivers/cxl/core/region.c new file +45
-0+return ERR_PTR (rc); +} + +static ssize_t create_region_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); + +return sysfs_emit (buf,"region%d.%d:%d\n" , port->id, cxld->id, + cxld->next_region_id); +} + +static ssize_t create_region_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_port *port =to_cxl_port (dev->parent); +struct cxl_decoder *cxld =to_cxl_decoder (dev); +struct cxl_region *cxlr; +int d, p, r, rc =0 ; + +if (sscanf (buf,"region%d.%d:%d" , &p, &d, &r) !=3 ) +return -EINVAL ; + +if (port->id != p || cxld->id != d) +return -EINVAL ; + + rc =mutex_lock_interruptible (&cxld->id_lock); +if (rc) +return rc; + +if (cxld->next_region_id != r) { + rc = -EINVAL ; + goto out; + } + + rc =ida_alloc (&cxld->region_ida,GFP_KERNEL ); +if (rc <0 ) { +dev_dbg (dev,"Failed to get next cached id (%d)\n" , rc); + goto out; + } + + cxlr =devm_cxl_add_region (cxld); +if (IS_ERR (cxlr)) {Quoted 9 lines
drivers/cxl/core/region.c line 159
Quoted 5 lines
drivers/cxl/core/region.c new file +5
-0+ rc =PTR_ERR (cxlr); + goto out; + } + + cxld->next_region_id = rc;This looks like a leak in the case when
devm_cxl_add_region()fails, so just move it above that call.Quoted 5 lines
It's not super simple with the current pre-caching. If you move this above
devm_cxl_add_region(), then you lose the previously pre-cached region. I think the cleaner solution is to just free the ida on failure. Pretty sure no matter what method you go, you need anida_freein there somewhere. Do you see another way?As soon as one thread has successfully acquired the
next_region_idthen it is safe to advance and assume that thedevm_cxl_add_region()owns releasing that id.
Quoted 5 lines
In fact, just make that implicit. Move the
ida_alloc()and thenext_region_idadvancement internal tocxl_region_alloc()with adevice_lock_assert()for theid_lock. Then the recovery of the allocated id happens naturally like all the other ids in the subsystem i.e. at release time.
This is the right answer. Regions created from LSA/BIOS will need this anyway.
PATCH v3 2/14 cxl/region: Introduce concept of region configuration
In reply to Ben WidawskyThe region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
A example is provided below:
/sys/bus/cxl/devices/region0.0:0
├── interleave_granularity ├── interleave_ways ├── offset ├── size ├── subsystem -> ../../../../../../bus/cxl ├── target0 ├── uevent └── uuid
--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)
Documentation/ABI/testing/sysfs-bus-cxl +40 -0
@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offset +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder. Region space allocation is handled by the driver, but + the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions. + +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined: + + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume. + target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for. Each target must be set with a memdev device ie. + 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one. + == ========================================================
drivers/cxl/core/region.c +300 -0
328 lines not on this page · raw message
Signature · 1 line
2.35.0
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative interleave_granularity values to specified, just return -EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation to cxl_region_probe().
Quoted 4 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0
Quoted 8 lines
├── interleave_granularity ├── interleave_ways ├── offset ├── size ├── subsystem -> ../../../../../../bus/cxl ├── target0 ├── uevent └── uuid
As mentioned off-list, it looks like devtype and modalias are missing.
Quoted 13 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)Documentation/ABI/testing/sysfs-bus-cxl +0
-0
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offset
This is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 1 line
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021
Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder.
"A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.
I would also expect, before / along with these new region attributes, there would be 'available' and 'max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:
Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.
s/Phsyical/Physical/
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.
That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 228
Quoted 5 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.
Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Quoted 19 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h"
drivers/cxl/core/region.c line 30
Quoted 10 lines
drivers/cxl/core/region.c +7
-0@@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */ +struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;
This function seems redundant with just checking "cxlr->dev.driver != NULL"? The benefit of that is there is no need to carry a TODO in the series.
drivers/cxl/core/region.c line 39
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{ +struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);
A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
Quoted 8 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +
drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;
I would expect:
if (dev->driver)
return -EBUSY;
...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.
Quoted 5 lines
drivers/cxl/core/region.c +5
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret;
drivers/cxl/core/region.c line 72
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val; + + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );
Why?
drivers/cxl/core/region.c line 75
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);
To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
Quoted 19 lines
drivers/cxl/core/region.c +19
-0+ +return len; + +err: + cxlr->config.interleave_ways = prev_iw; +return ret; +} +static DEVICE_ATTR_RW (interleave_ways); + +static ssize_t interleave_granularity_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_granularity); +} + +static ssize_t interleave_granularity_store(struct device *dev,
drivers/cxl/core/region.c line 104
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *cxlr = to_cxl_region(dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + cxlr->config.interleave_granularity = val;
This wants minimum input validation and synchronization against an active region.
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return len; +} +static DEVICE_ATTR_RW (interleave_granularity);
drivers/cxl/core/region.c line 118
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ +static ssize_t offset_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); +resource_size_t offset; + +if (!cxlr->res) +return sysfs_emit (buf,"\n" );
Should be an error I would think. I.e. require size to be set before s/offset/resource/ can be read.
drivers/cxl/core/region.c line 120
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + offset = cxld->platform_res.start - cxlr->res->start;
Why make usersapce do the offset math?
drivers/cxl/core/region.c line 124
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return sysfs_emit (buf,"%pa\n" , &offset); +} +static DEVICE_ATTR_RO (offset);
This can be DEVICE_ATTR_ADMIN_RO() to hide physical address layout information from non-root.
drivers/cxl/core/region.c line 131
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+ +static ssize_t size_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%llu\n" , cxlr->config.size);
Perhaps no need to store size separately if this becomes:
sysfs_emit(buf, "%llu\n", (unsigned long long) resource_size(cxlr->res));
...?
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+} + +static ssize_t size_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +unsigned long long val; +ssize_t rc; +
drivers/cxl/core/region.c line 150
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ rc =kstrtoull (buf,0 , &val); +if (rc) +return rc; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);
I think lockdep will complain about device_lock() usage in an attribute. Try changing this to cxl_device_lock() with CONFIG_PROVE_CXL_LOCKING=y.
Quoted 47 lines
drivers/cxl/core/region.c +47
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n)
drivers/cxl/core/region.c line 207
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else + ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);
The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like dax_mapping_attributes or REGION_MAPPING() from drivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.
Quoted 12 lines
drivers/cxl/core/region.c +12
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); +
drivers/cxl/core/region.c line 229
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + } + + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);
I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 97 lines
drivers/cxl/core/region.c +88
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,
drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 8 lines
drivers/cxl/core/region.c +3
-0@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);
Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
Quoted 6 lines
drivers/cxl/core/region.c +0
-1kfree (cxlr); } --2.35.0
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
I will cut to the part that effects ABI so tool development can continue. I'll get back to the other bits later.
On 22-01-28 16:25:34, Dan Williams wrote:
[snip]
Quoted 21 lines
Quoted 21 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }
Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
You've put two points in here:
1. Handle decoder allocation at sysfs boundary. I'll respond to this when I come back around to the rest of the review comments.
2. Take a decoder for target instead of a memdev. I don't agree with this direction as it's asymmetric to how LSA processing works. The goal was to model the LSA for configuration. The kernel will have to be in the business of reserving and enumerating decoders out of memdevs for both LSA (where we have a list of memdevs) and volatile (where we use the memdevs in the system to enumerate populated decoders). I don't see much value in making userspace do the same.
I'd like to ask you reconsider if you still think it's preferable to use decoders as part of the ABI and if you still feel that way I can go change it since it has minimal impact overall.
Quoted 15 lines
Quoted 15 lines
drivers/cxl/core/region.c +15
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} +
[snip]
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 52 lines · Ben Widawsky
I will cut to the part that effects ABI so tool development can continue. I'll get back to the other bits later.
On 22-01-28 16:25:34, Dan Williams wrote:
[snip]
Quoted 21 lines
Quoted 21 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
You've put two points in here:
1. Handle decoder allocation at sysfs boundary. I'll respond to this when I come back around to the rest of the review comments.
2. Take a decoder for target instead of a memdev. I don't agree with this direction as it's asymmetric to how LSA processing works. The goal was to model the LSA for configuration. The kernel will have to be in the business of reserving and enumerating decoders out of memdevs for both LSA (where we have a list of memdevs) and volatile (where we use the memdevs in the system to enumerate populated decoders). I don't see much value in making userspace do the same.
Quoted 3 lines
I'd like to ask you reconsider if you still think it's preferable to use decoders as part of the ABI and if you still feel that way I can go change it since it has minimal impact overall.
It's more than a preference. I think there are fundamental recovery scenarios where the kernel needs userspace help to resolve decoder / DPA assignment and conflicts.
PMEM interleaves behave similarly to RAID where you have multiple devices in a set that can each fail independently, and because they are hotplug capable the chances of migrating devices from one system to another are higher than PMEM devices today where hotplug is mostly non-existent. If you lurk on linux-raid long enough you will inevitably encounter someone coming to the list saying, "help a drive in my RAID array was dying. I managed to save it off, help me reassemble my array". The story often gets worse when they say "I managed to corrupt my metadata block, so I don't know what order the drives are supposed to be in". There are several breadcrumbs and trial and error steps that one takes to try to get the data back online:
https://raid.wiki.kernel.org/index.php/RAID_Recovery.
Now imagine that scenario with CXL where there are additional complicating factors like label-storage-area can fail independently of the data area, there are region labels with HPA fields that mandate assembly at a given address, decoders must be programmed in increasing DPA order, volatile memory and locked/fixed decoders complicate decoder selection. Considering all the ways that CXL region assembly can fail it seems inevitable that someone will get into a situation where they need to pick the decoder and the DPA to map while also not clobbering the LSA. I.e. I see a "CXL Recovery" wiki in our future.
The requirements that I think fall out from that are:
1/ Region assembly needs to be possible without updating labels. So, in contrast to the way that nvdimm does it, instead of updating the region label on every attribute write there would be a commit step that realizes the current region configuration in the labels, or is ommitted in recovery scenarios where you are not ready to clobber the labels.
2/ Userspace needs the flexibility to be able to select/override which DPA gets mapped in which decoder (kernel would handle DPA skip).
All the ways I can think of to augment the ABI to allow for this style of recovery devolve to just assigning decoders to regions in the first instance.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
In reply to Dan WilliamsQuoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().
Two points:c
1. How do we distinguish "basic input validation". It'd be good if we could define "basic input validation". For instance, when I first wrote these patches, x3 would have been EINVAL, but today it's allowed. Can you help enumerate what you consider basic.
2. I like the idea that all validation takes place in one place. Obviously you do not. So, see #1 and I will rework.
Quoted 13 lines
Quoted 4 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0Quoted 8 lines
├── interleave_granularity ├── interleave_ways ├── offset ├── size ├── subsystem -> ../../../../../../bus/cxl ├── target0 ├── uevent └── uuid
Quoted 2 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Thanks.
Quoted 8 lines
Quoted 8 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
It is offset. I can change it to physical base if you'd like but I thought that information wasn't critically important for userspace to have. Does userspace care about the physical base?
Quoted 21 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 1 line
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.
Quoted 7 lines
I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".
Sounds good.
Quoted 18 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Okay. Order doesn't matter if you do validation all in one place as it is, but sounds like we're changing that. So I can split it when we figure out what validation is actually occurring at the sysfs attr boundary.
Quoted 14 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Yes This is wrong. I thought I had fixed it, but I guess not.
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 228
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.
Quoted 5 lines
Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Well the kernel fills it in if userspace leaves it out. I'm guessing you're going to ask me to change that, so I will remove Mandatory/Optional.
Quoted 22 lines
Quoted 22 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" @@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */
Quoted 12 lines
drivers/cxl/core/region.c line 30
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.
Yeah. I think checking driver bind status is sufficient to replace this.
Quoted 4 lines
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{
Quoted 8 lines
drivers/cxl/core/region.c line 39
Quoted 5 lines
drivers/cxl/core/region.c +5
-0+struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Is this referring to the later question about whether targets are decoders or memdevs? The thought was each region would hold a reference to all memdevs in the interleave set.
Quoted 5 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
I'll defer to you on that. I'll remove holding the reference, but I definitely haven't solved the interaction when a memdev goes away. I had been thinking the inverse originally, a memdev can't go away until the region is gone. According to the spec, these devices can't be hot removed, only managed remove, so if things blew up, not our problem. However, if we have decent infrastructure to support better than that, we should.
Quoted 21 lines
Quoted 8 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
Quoted 8 lines
if (
dev->driver)
return-EBUSY;...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.
Okay.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;
Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
I copied it from another driver. I didn't check if it was actually needed or not.
Quoted 8 lines
drivers/cxl/core/region.c line 75
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
I'm fine with that.
Quoted 157 lines
Quoted 19 lines
drivers/cxl/core/region.c +19
-0+ +return len; + +err: + cxlr->config.interleave_ways = prev_iw; +return ret; +} +static DEVICE_ATTR_RW (interleave_ways); + +static ssize_t interleave_granularity_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_granularity); +} + +static ssize_t interleave_granularity_store(struct device *dev,drivers/cxl/core/region.c line 104
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *cxlr = to_cxl_region(dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + cxlr->config.interleave_granularity = val;This wants minimum input validation and synchronization against an active region.
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return len; +} +static DEVICE_ATTR_RW (interleave_granularity);drivers/cxl/core/region.c line 118
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ +static ssize_t offset_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); +resource_size_t offset; + +if (!cxlr->res) +return sysfs_emit (buf,"\n" );Should be an error I would think. I.e. require size to be set before s/offset/resource/ can be read.
drivers/cxl/core/region.c line 120
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + offset = cxld->platform_res.start - cxlr->res->start;Why make usersapce do the offset math?
drivers/cxl/core/region.c line 124
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return sysfs_emit (buf,"%pa\n" , &offset); +} +static DEVICE_ATTR_RO (offset);This can be
DEVICE_ATTR_ADMIN_RO()to hide physical address layout information from non-root.drivers/cxl/core/region.c line 131
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+ +static ssize_t size_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%llu\n" , cxlr->config.size);Perhaps no need to store size separately if this becomes:
sysfs_emit(buf, "%llu\n", (unsigned long long) resource_size(cxlr->res));...?
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+} + +static ssize_t size_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +unsigned long long val; +ssize_t rc; +drivers/cxl/core/region.c line 150
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ rc =kstrtoull (buf,0 , &val); +if (rc) +return rc; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.Quoted 54 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else
Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.
I had been thinking the kernel would manage the DPS spans of a memdev (and create the mappings). I can make this look like dax_mapping_attributes.
Quoted 20 lines
Quoted 20 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }
Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
My preference is obviously how it is, using memdevs and having the decoders allocated at bind time. I don't have an objective argument why one is better than the other so I will change it. I will make the interface take a set of decoders.
Quoted 102 lines
Quoted 102 lines
drivers/cxl/core/region.c +89
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;
Quoted 7 lines
drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 3 lines
drivers/cxl/core/region.c +2
-0ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
I think I already explained why it works this way. I will change it.
Quoted 6 lines
Quoted 6 lines
drivers/cxl/core/region.c +0
-1kfree (cxlr); } --2.35.0
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 27 lines · Ben Widawsky
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().
Quoted 5 lines
Two points:c
1. How do we distinguish "basic input validation". It'd be good if we could define "basic input validation". For instance, when I first wrote these patches, x3 would have been EINVAL, but today it's allowed. Can you help enumerate what you consider basic.
I internalized this kernel design principle from Dave Miller many years ago paraphrasing "push decision making out to leaf code as much as possible", and centralizing all validation in cxl_region_probe() violates. The software that makes the mistake does not know it made a mistake until much later and "probe failed" is less descriptive than "EINVAL writing interleave_ways" . I wish I could find the thread because it also talked about his iteration process.
Basic input validation to me is things like:
- Don't allow writes while the region is active
- Check that values are in bound. So yes, the interleave-ways value of 3 would fail until the kernel supports it, and granularity values > 16K would also fail.
- Check that memdevs are actually downstream targets of the given decoder
- Check that the region uuid is unique
- Check that decoder has capacity
- Check that the memdev has capacity
- Check that the decoder to map the DPA is actually available given decoders must be programmed in increasing DPA order
Essentially any validation short of walking the topology to program upstream decoders since those errors are only resolved by racing region probes that try to grab upstream decoder resources.
Quoted 3 lines
2. I like the idea that all validation takes place in one place. Obviously you do not. So, see #1 and I will rework.
The validation helpers need to be written once, where they are called does not much matter, does it?
Quoted 43 lines
Quoted 14 lines
Quoted 5 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0Quoted 8 lines
├── interleave_granularity ├── interleave_ways ├── offset ├── size ├── subsystem -> ../../../../../../bus/cxl ├── target0 ├── uevent └── uuidQuoted 2 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Thanks.
Quoted 8 lines
Quoted 8 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
Quoted 3 lines
It is offset. I can change it to physical base if you'd like but I thought that information wasn't critically important for userspace to have. Does userspace care about the physical base?
Yes, similar use case as /proc/iomem. Error handling comes to mind as you can see physical address data in the messages like machine check notification and go immediately match that to a CXL region. PCI, NVDIMM, and DAX all emit a "resource" attribute to identify the physical address base.
Quoted 51 lines
Quoted 22 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.Quoted 7 lines
I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".Sounds good.
Quoted 18 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Quoted 3 lines
Okay. Order doesn't matter if you do validation all in one place as it is, but sounds like we're changing that. So I can split it when we figure out what validation is actually occurring at the sysfs attr boundary.
Forcing a write order simplifies the validation matrix. Consider the reduction in test surface if the kernel is more strict about what it allows into the kernel early. Let's make syzbot's job harder.
Quoted 33 lines
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Yes This is wrong. I thought I had fixed it, but I guess not.
Quoted 7 lines
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.
Quoted 8 lines
Quoted 5 lines
Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Well the kernel fills it in if userspace leaves it out. I'm guessing you're going to ask me to change that, so I will remove Mandatory/Optional.
Yeah, why carry unnecessary code in the kernel? Userspace is well equipped to meet the requirement that it writes the UUID.
Quoted 51 lines
Quoted 23 lines
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" @@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */ Quoted 12 lines
drivers/cxl/core/region.c line 30
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.Yeah. I think checking driver bind status is sufficient to replace this.
Quoted 4 lines
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{Quoted 8 lines
drivers/cxl/core/region.c line 39
Quoted 5 lines
drivers/cxl/core/region.c +5
-0+struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Quoted 2 lines
Is this referring to the later question about whether targets are decoders or memdevs?
Yes.
Quoted 2 lines
The thought was each region would hold a reference to all memdevs in the interleave set.
It's not clear that a region needs to hold a reference if a memdev self removes itself from the region before it is unregistered. I am open to being convinced this is needed but it would need come with an explanation of what can a region do with a memdev reference after that memdev has experienced a ->remove() event.
Quoted 11 lines
Quoted 4 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
I'll defer to you on that. I'll remove holding the reference, but I definitely haven't solved the interaction when a memdev goes away. I had been thinking the inverse originally, a memdev can't go away until the region is gone. According to the spec, these devices can't be hot removed, only managed remove, so if things blew up, not our problem. However, if we have decent infrastructure to support better than that, we should.
It turns out there's no such thing as "managed remove" as far as the kernel is concerned, it's all up to userspace. ->remove() can happen at any time and ->remove() can not fail. Per Bjorn Linux does not support PCIe hotplug latching so there is no kernel mechanism to block hotplug. Unless and until the PCIe native hotplug code picks up a mechanism to deny unplug events CXL needs to be prepared for any memdev to experience ->remove() regardless of region status.
For example, I expect the "managed remove" flow is something like this:
# cxl disable-memdev mem3 cxl memdev: action_disable: mem3 is part of an active region cxl memdev: cmd_disable_memdev: disabled 0 mem
...where the tool tries to enforce safety, but if someone really wants that device gone:
# cxl disable-memdev mem3 --force cxl memdev: cmd_disable_memdev: disabled 1 mem
...and the CXL sub-system will need to trigger memory-failure across all the regions that were impacted by that violent event.
Quoted 52 lines
Quoted 22 lines
Quoted 9 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
Quoted 8 lines
if (
dev->driver)
return-EBUSY;...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.Okay.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
Quoted 2 lines
I copied it from another driver. I didn't check if it was actually needed or not.
It's not needed since the agent that wrote interleave ways is also expected to be the agent that is configuring the rest of the parameters.
Quoted 179 lines
Quoted 9 lines
drivers/cxl/core/region.c line 75
Quoted 4 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
I'm fine with that.
Quoted 157 lines
Quoted 19 lines
drivers/cxl/core/region.c +19
-0+ +return len; + +err: + cxlr->config.interleave_ways = prev_iw; +return ret; +} +static DEVICE_ATTR_RW (interleave_ways); + +static ssize_t interleave_granularity_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_granularity); +} + +static ssize_t interleave_granularity_store(struct device *dev,drivers/cxl/core/region.c line 104
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *cxlr = to_cxl_region(dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + cxlr->config.interleave_granularity = val;This wants minimum input validation and synchronization against an active region.
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return len; +} +static DEVICE_ATTR_RW (interleave_granularity);drivers/cxl/core/region.c line 118
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ +static ssize_t offset_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); +resource_size_t offset; + +if (!cxlr->res) +return sysfs_emit (buf,"\n" );Should be an error I would think. I.e. require size to be set before s/offset/resource/ can be read.
drivers/cxl/core/region.c line 120
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + offset = cxld->platform_res.start - cxlr->res->start;Why make usersapce do the offset math?
drivers/cxl/core/region.c line 124
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return sysfs_emit (buf,"%pa\n" , &offset); +} +static DEVICE_ATTR_RO (offset);This can be
DEVICE_ATTR_ADMIN_RO()to hide physical address layout information from non-root.drivers/cxl/core/region.c line 131
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+ +static ssize_t size_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%llu\n" , cxlr->config.size);Perhaps no need to store size separately if this becomes:
sysfs_emit(buf, "%llu\n", (unsigned long long) resource_size(cxlr->res));...?
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+} + +static ssize_t size_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +unsigned long long val; +ssize_t rc; +drivers/cxl/core/region.c line 150
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ rc =kstrtoull (buf,0 , &val); +if (rc) +return rc; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.Quoted 54 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.
Quoted 2 lines
I had been thinking the kernel would manage the DPS spans of a memdev (and create the mappings). I can make this look like
dax_mapping_attributes.
I think we get this for free by just linking the region to each endpoint decoder in use and then userspace can walk that to get DPA info from the decoder (modulo extending endpoint decoders with DPA extent info).
Quoted 31 lines
Quoted 21 lines
Quoted 21 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 4 lines
My preference is obviously how it is, using memdevs and having the decoders allocated at bind time. I don't have an objective argument why one is better than the other so I will change it. I will make the interface take a set of decoders.
Let me know if the arguments I have made for why more granularity is necessary have changed your preference. I'm open to hearing alternate ideas.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 34 lines · Dan Williams
Quoted 27 lines · Ben Widawsky
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 5 lines
Two points:c
1. How do we distinguish "basic input validation". It'd be good if we could define "basic input validation". For instance, when I first wrote these patches, x3 would have been EINVAL, but today it's allowed. Can you help enumerate what you consider basic.
Quoted 7 lines
I internalized this kernel design principle from Dave Miller many years ago paraphrasing "push decision making out to leaf code as much as possible", and centralizing all validation in
cxl_region_probe()violates. The software that makes the mistake does not know it made a mistake until much later and "probe failed" is less descriptive than "EINVAL writinginterleave_ways" . I wish I could find the thread because it also talked about his iteration process.
It would definitely be interesting to understand why pushing decision making into the leaf code is a violation. Was it primary around the descriptiveness of the error?
Quoted 9 lines
Basic input validation to me is things like:
- Don't allow writes while the region is active
- Check that values are in bound. So yes, the interleave-ways value of 3 would fail until the kernel supports it, and granularity values > 16K would also fail.
- Check that memdevs are actually downstream targets of the given decoder
- Check that the region uuid is unique
These are obviously easy and informative at attr store time (in fact, active was meant to be checked already for many cases). So if we agree to codify this at probe via WARN, and add it to kdoc, I've no problem with it.
Quoted 9 lines
- Check that decoder has capacity
- Check that the memdev has capacity
- Check that the decoder to map the DPA is actually available given decoders must be programmed in increasing DPA orderEssentially any validation short of walking the topology to program upstream decoders since those errors are only resolved by racing region probes that try to grab upstream decoder resources.
I intentionally avoided doing a lot of these until probe because it seemed like not a great policy to deny regions from being populated if another region utilizing those resources hasn't been bound yes. For a simple example, if x1 region A is created and utilizes all of memdev ɑ's capacity you block out any other region setup using memdev ɑ, even if region A wasn't bound. There's a similar problem with specifying decoders as part of configuration.
I'll infer from your comment that you are fine with this tradeoff, or you have some other way to manage this in mind.
I really see any validation which requires removal of resources from the system to be more fit for bind time. I suppose if the proposal is to move the region attributes to be DEVICE_ATTR_ADMIN, that pushes the problem onto the system administrator. It just seemed like most of the interface could be non-root.
Quoted 7 lines
Quoted 3 lines
2. I like the idea that all validation takes place in one place. Obviously you do not. So, see #1 and I will rework.
The validation helpers need to be written once, where they are called does not much matter, does it?
Somewhat addressed above too...
I think that depends on whether the full list is established as mentioned. If in the region driver we can put several assertions that a variety of things don't need [re]validation, then it doesn't matter. Without this, when trying to debug or add code you need to figure out which place is doing the validation and which place should do it.
At the very least I think the plan should be established in a kdoc.
Quoted 542 lines
Quoted 43 lines
Quoted 14 lines
Quoted 5 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0Quoted 8 lines
├── interleave_granularity ├── interleave_ways ├── offset ├── size ├── subsystem -> ../../../../../../bus/cxl ├── target0 ├── uevent └── uuidQuoted 2 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Thanks.
Quoted 8 lines
Quoted 8 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
Quoted 3 lines
It is offset. I can change it to physical base if you'd like but I thought that information wasn't critically important for userspace to have. Does userspace care about the physical base?
Yes, similar use case as
/proc/iomem. Error handling comes to mind as you can see physical address data in the messages like machine check notification and go immediately match that to a CXL region. PCI, NVDIMM, and DAX all emit a "resource" attribute to identify the physical address base.Quoted 51 lines
Quoted 22 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.Quoted 7 lines
I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".Sounds good.
Quoted 18 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Quoted 3 lines
Okay. Order doesn't matter if you do validation all in one place as it is, but sounds like we're changing that. So I can split it when we figure out what validation is actually occurring at the sysfs attr boundary.
Forcing a write order simplifies the validation matrix. Consider the reduction in test surface if the kernel is more strict about what it allows into the kernel early. Let's make syzbot's job harder.
Quoted 33 lines
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Yes This is wrong. I thought I had fixed it, but I guess not.
Quoted 7 lines
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.Quoted 8 lines
Quoted 5 lines
Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Well the kernel fills it in if userspace leaves it out. I'm guessing you're going to ask me to change that, so I will remove Mandatory/Optional.
Yeah, why carry unnecessary code in the kernel? Userspace is well equipped to meet the requirement that it writes the UUID.
Quoted 51 lines
Quoted 23 lines
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" @@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */ Quoted 12 lines
drivers/cxl/core/region.c line 30
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.Yeah. I think checking driver bind status is sufficient to replace this.
Quoted 4 lines
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{Quoted 8 lines
drivers/cxl/core/region.c line 39
Quoted 5 lines
drivers/cxl/core/region.c +5
-0+struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Quoted 2 lines
Is this referring to the later question about whether targets are decoders or memdevs?
Yes.
Quoted 2 lines
The thought was each region would hold a reference to all memdevs in the interleave set.
It's not clear that a region needs to hold a reference if a memdev self removes itself from the region before it is unregistered. I am open to being convinced this is needed but it would need come with an explanation of what can a region do with a memdev reference after that memdev has experienced a ->remove() event.
Quoted 11 lines
Quoted 4 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
I'll defer to you on that. I'll remove holding the reference, but I definitely haven't solved the interaction when a memdev goes away. I had been thinking the inverse originally, a memdev can't go away until the region is gone. According to the spec, these devices can't be hot removed, only managed remove, so if things blew up, not our problem. However, if we have decent infrastructure to support better than that, we should.
It turns out there's no such thing as "managed remove" as far as the kernel is concerned, it's all up to userspace. ->remove() can happen at any time and ->remove() can not fail. Per Bjorn Linux does not support PCIe hotplug latching so there is no kernel mechanism to block hotplug. Unless and until the PCIe native hotplug code picks up a mechanism to deny unplug events CXL needs to be prepared for any memdev to experience ->remove() regardless of region status.
For example, I expect the "managed remove" flow is something like this:
# cxl disable-memdev mem3 cxl memdev: action_disable: mem3 is part of an active region cxl memdev: cmd_disable_memdev: disabled 0 mem...where the tool tries to enforce safety, but if someone really wants that device gone:
# cxl disable-memdev mem3 --force cxl memdev: cmd_disable_memdev: disabled 1 mem...and the CXL sub-system will need to trigger memory-failure across all the regions that were impacted by that violent event.
Quoted 52 lines
Quoted 22 lines
Quoted 9 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
Quoted 8 lines
if (
dev->driver)
return-EBUSY;...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.Okay.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
Quoted 2 lines
I copied it from another driver. I didn't check if it was actually needed or not.
It's not needed since the agent that wrote interleave ways is also expected to be the agent that is configuring the rest of the parameters.
Quoted 179 lines
Quoted 9 lines
drivers/cxl/core/region.c line 75
Quoted 4 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
I'm fine with that.
Quoted 157 lines
Quoted 19 lines
drivers/cxl/core/region.c +19
-0+ +return len; + +err: + cxlr->config.interleave_ways = prev_iw; +return ret; +} +static DEVICE_ATTR_RW (interleave_ways); + +static ssize_t interleave_granularity_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_granularity); +} + +static ssize_t interleave_granularity_store(struct device *dev,drivers/cxl/core/region.c line 104
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+struct device_attribute *attr, + const char *buf, size_t len) +{ + struct cxl_region *cxlr = to_cxl_region(dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + cxlr->config.interleave_granularity = val;This wants minimum input validation and synchronization against an active region.
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return len; +} +static DEVICE_ATTR_RW (interleave_granularity);drivers/cxl/core/region.c line 118
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ +static ssize_t offset_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent); +struct cxl_region *cxlr =to_cxl_region (dev); +resource_size_t offset; + +if (!cxlr->res) +return sysfs_emit (buf,"\n" );Should be an error I would think. I.e. require size to be set before s/offset/resource/ can be read.
drivers/cxl/core/region.c line 120
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + offset = cxld->platform_res.start - cxlr->res->start;Why make usersapce do the offset math?
drivers/cxl/core/region.c line 124
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return sysfs_emit (buf,"%pa\n" , &offset); +} +static DEVICE_ATTR_RO (offset);This can be
DEVICE_ATTR_ADMIN_RO()to hide physical address layout information from non-root.drivers/cxl/core/region.c line 131
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+ +static ssize_t size_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%llu\n" , cxlr->config.size);Perhaps no need to store size separately if this becomes:
sysfs_emit(buf, "%llu\n", (unsigned long long) resource_size(cxlr->res));...?
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+} + +static ssize_t size_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +unsigned long long val; +ssize_t rc; +drivers/cxl/core/region.c line 150
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+ rc =kstrtoull (buf,0 , &val); +if (rc) +return rc; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.Quoted 54 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.Quoted 2 lines
I had been thinking the kernel would manage the DPS spans of a memdev (and create the mappings). I can make this look like
dax_mapping_attributes.I think we get this for free by just linking the region to each endpoint decoder in use and then userspace can walk that to get DPA info from the decoder (modulo extending endpoint decoders with DPA extent info).
Quoted 31 lines
Quoted 21 lines
Quoted 21 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 4 lines
My preference is obviously how it is, using memdevs and having the decoders allocated at bind time. I don't have an objective argument why one is better than the other so I will change it. I will make the interface take a set of decoders.
Let me know if the arguments I have made for why more granularity is necessary have changed your preference. I'm open to hearing alternate ideas.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 44 lines · Ben Widawsky
Quoted 34 lines · Dan Williams
Quoted 27 lines · Ben Widawsky
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 5 lines
Two points:c
1. How do we distinguish "basic input validation". It'd be good if we could define "basic input validation". For instance, when I first wrote these patches, x3 would have been EINVAL, but today it's allowed. Can you help enumerate what you consider basic.Quoted 7 lines
I internalized this kernel design principle from Dave Miller many years ago paraphrasing "push decision making out to leaf code as much as possible", and centralizing all validation in
cxl_region_probe()violates. The software that makes the mistake does not know it made a mistake until much later and "probe failed" is less descriptive than "EINVAL writinginterleave_ways" . I wish I could find the thread because it also talked about his iteration process.
Quoted 3 lines
It would definitely be interesting to understand why pushing decision making into the leaf code is a violation. Was it primary around the descriptiveness of the error?
You mean the other way round, why is it a violation to move decision making into the core? It was a comment about the inflexibility of the core logic vs leaf logic, in the case of CXL it's about the observability of errors at the right granularity which the core can not do because the core is disconnected from the transaction that injected the error.
Quoted 9 lines
Quoted 8 lines
Basic input validation to me is things like:
- Don't allow writes while the region is active
- Check that values are in bound. So yes, the interleave-ways value of 3 would fail until the kernel supports it, and granularity values > 16K would also fail.
- Check that memdevs are actually downstream targets of the given decoder
- Check that the region uuid is unique
Quoted 3 lines
These are obviously easy and informative at attr store time (in fact, active was meant to be checked already for many cases). So if we agree to codify this at probe via WARN, and add it to kdoc, I've no problem with it.
Why is WARN needed? Either the sysfs validation does it job correctly or it doesn't. Also if sysfs didn't WARN when the bad input is specified why would the core do anything higher than dev_err()? Basically I think the bar for WARN is obvious kernel programming error where only a kernel-developer will see it vs EINVAL at runtime scenarios. I have seen Greg raise the bar for WARN in his reviews given how many deployments turn on 'panic_on_warn'.
Quoted 17 lines
Quoted 9 lines
- Check that decoder has capacity
- Check that the memdev has capacity
- Check that the decoder to map the DPA is actually available given decoders must be programmed in increasing DPA orderEssentially any validation short of walking the topology to program upstream decoders since those errors are only resolved by racing region probes that try to grab upstream decoder resources.
I intentionally avoided doing a lot of these until probe because it seemed like not a great policy to deny regions from being populated if another region utilizing those resources hasn't been bound yes. For a simple example, if x1 region A is created and utilizes all of memdev ɑ's capacity you block out any other region setup using memdev ɑ, even if region A wasn't bound. There's a similar problem with specifying decoders as part of configuration.
Quoted 2 lines
I'll infer from your comment that you are fine with this tradeoff, or you have some other way to manage this in mind.
It comes back to observability if threadA allocates all the DPA then yes all other threads should see -ENOSPC. No different than if 3 fdisk threads all tried to create a partition, the first one to the kernel wins. If threadA does not end up activating that regionA's capacity that's userspace's fault, and the admin needs to make sure that configuration does not race itself. The kernel allocating DPA immediately lets those races be found early such that threadB finds all the DPA gone and stops trying to create the region.
Quoted 4 lines
I really see any validation which requires removal of resources from the system to be more fit for bind time. I suppose if the proposal is to move the region attributes to be
DEVICE_ATTR_ADMIN, that pushes the problem onto the system administrator. It just seemed like most of the interface could be non-root.
None of the sysfs entries for CXL are writable by non-root.
DEVICE_ATTR_RW() is 0644DEVICE_ATTR_ADMIN_RW() is 0600
Yes, pushing the problem onto the sysadmin is the only option. Only CAP_SYS_ADMIN can be trusted to muck with the physical address layout of the system. Even then CONFIG_LOCKDOWN_KERNEL wants to limit what CAP_SYS_ADMIN can to do the memory configuration, so I don't see any room for non-root to be considered in this ABI.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
2. I like the idea that all validation takes place in one place. Obviously you do not. So, see #1 and I will rework.
The validation helpers need to be written once, where they are called does not much matter, does it?
Quoted 7 lines
Somewhat addressed above too...
I think that depends on whether the full list is established as mentioned. If in the region driver we can put several assertions that a variety of things don't need [re]validation, then it doesn't matter. Without this, when trying to debug or add code you need to figure out which place is doing the validation and which place should do it.
All I can say it has not been a problem in practice for NVDIMM debug scenarios which does validation at probe for pre-existing namespaces and validation at sysfs write for namespace creation.
Quoted 1 line
At the very least I think the plan should be established in a kdoc.
Sure, a "CXL Region: Theory of Operation" would be a good document to lead into the patch series as a follow-on to "CXL Bus: Theory of Operation".
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 49 lines · Dan Williams
Quoted 44 lines · Ben Widawsky
Quoted 34 lines · Dan Williams
Quoted 27 lines · Ben Widawsky
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 5 lines
Two points:c
1. How do we distinguish "basic input validation". It'd be good if we could define "basic input validation". For instance, when I first wrote these patches, x3 would have been EINVAL, but today it's allowed. Can you help enumerate what you consider basic.Quoted 7 lines
I internalized this kernel design principle from Dave Miller many years ago paraphrasing "push decision making out to leaf code as much as possible", and centralizing all validation in
cxl_region_probe()violates. The software that makes the mistake does not know it made a mistake until much later and "probe failed" is less descriptive than "EINVAL writinginterleave_ways" . I wish I could find the thread because it also talked about his iteration process.Quoted 3 lines
It would definitely be interesting to understand why pushing decision making into the leaf code is a violation. Was it primary around the descriptiveness of the error?
Quoted 6 lines
You mean the other way round, why is it a violation to move decision making into the core? It was a comment about the inflexibility of the core logic vs leaf logic, in the case of CXL it's about the observability of errors at the right granularity which the core can not do because the core is disconnected from the transaction that injected the error.
I did mean the other way around. The thing that gets tricky if you do it at the sysfs boundary is you do have to start seeing the interface as stateful. Perhaps the complexity I see arising from this won't materialize, so I'll try it and see. It seems like it can get messy quickly though.
Quoted 14 lines
Quoted 10 lines
Quoted 9 lines
Basic input validation to me is things like:
- Don't allow writes while the region is active
- Check that values are in bound. So yes, the interleave-ways value of 3 would fail until the kernel supports it, and granularity values > 16K would also fail.
- Check that memdevs are actually downstream targets of the given decoder
- Check that the region uuid is uniqueQuoted 3 lines
These are obviously easy and informative at attr store time (in fact, active was meant to be checked already for many cases). So if we agree to codify this at probe via WARN, and add it to kdoc, I've no problem with it.
Quoted 7 lines
Why is WARN needed? Either the sysfs validation does it job correctly or it doesn't. Also if sysfs didn't WARN when the bad input is specified why would the core do anything higher than
dev_err()? Basically I think the bar for WARN is obvious kernel programming error where only a kernel-developer will see it vs EINVAL at runtime scenarios. I have seen Greg raise the bar for WARN in his reviews given how many deployments turn on 'panic_on_warn'.
Ultimately some checking will need to occur in one form or another in region probe(). Either explicit via conditional: if (!is_valid(interleave_ways)) return -EINVAL, or implicitly, for example 1 << (rootd_ig - cxlr_ig) is some invalid nonsense which later fails host bridge programming verification. Before discussing further, which are you suggesting?
Quoted 21 lines
Quoted 18 lines
Quoted 10 lines
- Check that decoder has capacity
- Check that the memdev has capacity
- Check that the decoder to map the DPA is actually available given decoders must be programmed in increasing DPA orderEssentially any validation short of walking the topology to program upstream decoders since those errors are only resolved by racing region probes that try to grab upstream decoder resources.
I intentionally avoided doing a lot of these until probe because it seemed like not a great policy to deny regions from being populated if another region utilizing those resources hasn't been bound yes. For a simple example, if x1 region A is created and utilizes all of memdev ɑ's capacity you block out any other region setup using memdev ɑ, even if region A wasn't bound. There's a similar problem with specifying decoders as part of configuration.
Quoted 2 lines
I'll infer from your comment that you are fine with this tradeoff, or you have some other way to manage this in mind.
Quoted 8 lines
It comes back to observability if threadA allocates all the DPA then yes all other threads should see
-ENOSPC. No different than if 3 fdisk threads all tried to create a partition, the first one to the kernel wins. If threadA does not end up activating that regionA's capacity that's userspace's fault, and the admin needs to make sure that configuration does not race itself. The kernel allocating DPA immediately lets those races be found early such that threadB finds all the DPA gone and stops trying to create the region.
Okay. I don't have a strong opinion on how userspace should or shouldn't use this interface. It seems less friendly to do it this way, but per the following comment, if it's root only, it doesn't really matter.
I was under the impression you expected userspace to manage the DPA as well. I don't really see any reason why the kernel should manage it if userspace is already handling all these other bits. Let userspace set the offset and size (can make a single device attr for it), and upon doing so it gets reserved.
Quoted 10 lines
Quoted 5 lines
I really see any validation which requires removal of resources from the system to be more fit for bind time. I suppose if the proposal is to move the region attributes to be
DEVICE_ATTR_ADMIN, that pushes the problem onto the system administrator. It just seemed like most of the interface could be non-root.None of the sysfs entries for CXL are writable by non-root.
DEVICE_ATTR_RW()is 0644DEVICE_ATTR_ADMIN_RW()is 0600
My mistake. I forgot about that.
Quoted 6 lines
Yes, pushing the problem onto the sysadmin is the only option. Only
CAP_SYS_ADMINcan be trusted to muck with the physical address layout of the system. Even thenCONFIG_LOCKDOWN_KERNELwants to limit whatCAP_SYS_ADMINcan to do the memory configuration, so I don't see any room for non-root to be considered in this ABI.
That's fine. As the interface is today (before your requested changes) only region->probe() is something that mucks with the physical address layout. It could theoretically be entirely configured (not bound) by userspace.
Quoted 22 lines
Quoted 10 lines
Quoted 9 lines
Quoted 5 lines
2. I like the idea that all validation takes place in one place. Obviously you do not. So, see #1 and I will rework.
The validation helpers need to be written once, where they are called does not much matter, does it?
Quoted 7 lines
Somewhat addressed above too...
I think that depends on whether the full list is established as mentioned. If in the region driver we can put several assertions that a variety of things don't need [re]validation, then it doesn't matter. Without this, when trying to debug or add code you need to figure out which place is doing the validation and which place should do it.
All I can say it has not been a problem in practice for NVDIMM debug scenarios which does validation at probe for pre-existing namespaces and validation at sysfs write for namespace creation.
Quoted 5 lines
Quoted 1 line
At the very least I think the plan should be established in a kdoc.
Sure, a "CXL Region: Theory of Operation" would be a good document to lead into the patch series as a follow-on to "CXL Bus: Theory of Operation".
Yeah. I will write it once we close this discussion.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
On Thu, Feb 3, 2022 at 4:20 PM Ben Widawsky <[email protected]> wrote:
[..]
Quoted 21 lines
Quoted 13 lines
Quoted 9 lines
Quoted 8 lines
Basic input validation to me is things like:
- Don't allow writes while the region is active
- Check that values are in bound. So yes, the interleave-ways value of 3 would fail until the kernel supports it, and granularity values > 16K would also fail.
- Check that memdevs are actually downstream targets of the given decoder
- Check that the region uuid is uniqueQuoted 3 lines
These are obviously easy and informative at attr store time (in fact, active was meant to be checked already for many cases). So if we agree to codify this at probe via WARN, and add it to kdoc, I've no problem with it.
Quoted 7 lines
Why is WARN needed? Either the sysfs validation does it job correctly or it doesn't. Also if sysfs didn't WARN when the bad input is specified why would the core do anything higher than
dev_err()? Basically I think the bar for WARN is obvious kernel programming error where only a kernel-developer will see it vs EINVAL at runtime scenarios. I have seen Greg raise the bar for WARN in his reviews given how many deployments turn on 'panic_on_warn'.
Quoted 5 lines
Ultimately some checking will need to occur in one form or another in region
probe(). Either explicit via conditional: if (!is_valid(interleave_ways)) return-EINVAL, or implicitly, for example 1 << (rootd_ig-cxlr_ig) is some invalid nonsense which later fails host bridge programming verification. Before discussing further, which are you suggesting?
Explicit validation at probe in addition to the explicit validation at the sysfs boundary (as much as possible to report errors early). The "at probe time" validation does not know if this was a new region, or one enumerated from LSA or the configuration that the BIOS specified. So I do expect validation overlap, but there will also be distinct checks for those different scenarios. For example, see how NVDIMM validates namespace configuration writes via sysfs, but does not validate the LSA because it's writing the label and had better be prepared to read what it writes.
Quoted 31 lines
Quoted 22 lines
Quoted 19 lines
Quoted 11 lines
- Check that decoder has capacity
- Check that the memdev has capacity
- Check that the decoder to map the DPA is actually available given decoders must be programmed in increasing DPA orderEssentially any validation short of walking the topology to program upstream decoders since those errors are only resolved by racing region probes that try to grab upstream decoder resources.
I intentionally avoided doing a lot of these until probe because it seemed like not a great policy to deny regions from being populated if another region utilizing those resources hasn't been bound yes. For a simple example, if x1 region A is created and utilizes all of memdev ɑ's capacity you block out any other region setup using memdev ɑ, even if region A wasn't bound. There's a similar problem with specifying decoders as part of configuration.
Quoted 2 lines
I'll infer from your comment that you are fine with this tradeoff, or you have some other way to manage this in mind.
Quoted 8 lines
It comes back to observability if threadA allocates all the DPA then yes all other threads should see
-ENOSPC. No different than if 3 fdisk threads all tried to create a partition, the first one to the kernel wins. If threadA does not end up activating that regionA's capacity that's userspace's fault, and the admin needs to make sure that configuration does not race itself. The kernel allocating DPA immediately lets those races be found early such that threadB finds all the DPA gone and stops trying to create the region.
Quoted 8 lines
Okay. I don't have a strong opinion on how userspace should or shouldn't use this interface. It seems less friendly to do it this way, but per the following comment, if it's root only, it doesn't really matter.
I was under the impression you expected userspace to manage the DPA as well. I don't really see any reason why the kernel should manage it if userspace is already handling all these other bits. Let userspace set the offset and size (can make a single device attr for it), and upon doing so it gets reserved.
Userspace sense requests, kernel allocates or denies that request after resolving races with other requesters. Yes, this makes the interface stateful. Sysfs is not suited to stateless operation.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
In reply to Dan WilliamsConsolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Quoted 13 lines
Quoted 4 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0Quoted 8 lines
├── interleave_granularity ├── interleave_ways ├── offset ├── size ├── subsystem -> ../../../../../../bus/cxl ├── target0 ├── uevent └── uuid
Quoted 2 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Yep. This belongs in the previous patch though.
Quoted 8 lines
Quoted 8 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
It was meant to be relative. I can make it absolute if that's preferable but the physical base is known at the decoder level already.
Quoted 21 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 1 line
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.
Quoted 8 lines
I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".
Since this is the configuration part of the ABI, I'd rather add that information when the plumbing to report them exists. I'm struggling to understand the balance (as mentioned also earlier in this mail thread) as to what userspace does and what the kernel does. I will add these as you request.
Quoted 17 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Okay. Since we're moving toward making the sysfs ABI stateful, would you like me to make the attrs only visible when they can actually be set?
Quoted 14 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Correct.
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 228
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Okay.
Quoted 23 lines
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" @@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */
Quoted 12 lines
drivers/cxl/core/region.c line 30
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.
The idea behind this was to give the reviewer somewhat of a bigger picture as to how things should work in the code rather than in a commit message. I will remove this.
Quoted 13 lines
drivers/cxl/core/region.c line 39
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{ +struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Quoted 4 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
This needs some work. The concern I have is region operations will need to operate on memdevs/decoders at various points in time. When the memdev goes away, the region will also need to go away. None of that plumbing was in place in v3 and the reference on the memdev was just a half-hearted attempt at doing the right thing.
For now if you prefer I remove the reference, but perhaps the decoder reference would buy us some safety?
Quoted 21 lines
Quoted 8 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
Quoted 7 lines
if (
dev->driver)
return-EBUSY;...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.
Initially my thought here is that this is a problem for userspace to deal with. If userspace can't figure out how to synchronously configure and bind the region, that's not a kernel problem. However, we've put some effort into protecting userspace from itself in the create ABI, so it might be more in line to do that here.
In summary, I'm fine to add it, but I think I really need to get more in your brain about the userspace/kernel divide sooner rather than later.
Quoted 10 lines
Quoted 10 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;
Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
copypasta
Quoted 8 lines
drivers/cxl/core/region.c line 75
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
Okay. One of the earlier versions did this implicitly since the #ways was needed to create the region. I thought from the ABI perspective, flexibility was good. Userspace may choose not to utilize it.
Quoted 24 lines
Quoted 24 lines
drivers/cxl/core/region.c +24
-0+ +return len; + +err: + cxlr->config.interleave_ways = prev_iw; +return ret; +} +static DEVICE_ATTR_RW (interleave_ways); + +static ssize_t interleave_granularity_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_granularity); +} + +static ssize_t interleave_granularity_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int val, ret;
Quoted 9 lines
drivers/cxl/core/region.c line 104
Quoted 5 lines
drivers/cxl/core/region.c +5
-0+ + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + cxlr->config.interleave_granularity = val;This wants minimum input validation and synchronization against an active region.
Okay. Already comprehended above.
Quoted 9 lines
Quoted 9 lines
drivers/cxl/core/region.c +9
-0+ +return len; +} +static DEVICE_ATTR_RW (interleave_granularity); + +static ssize_t offset_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);
Quoted 9 lines
drivers/cxl/core/region.c line 118
Quoted 5 lines
drivers/cxl/core/region.c +5
-0+struct cxl_region *cxlr =to_cxl_region (dev); +resource_size_t offset; + +if (!cxlr->res) +return sysfs_emit (buf,"\n" );Should be an error I would think. I.e. require size to be set before s/offset/resource/ can be read.
Okay. Already comprehended above.
Quoted 5 lines
drivers/cxl/core/region.c line 120
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + offset = cxld->platform_res.start - cxlr->res->start;Why make usersapce do the offset math?
Okay. Already comprehended above.
Quoted 18 lines
drivers/cxl/core/region.c line 124
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ +return sysfs_emit (buf,"%pa\n" , &offset); +} +static DEVICE_ATTR_RO (offset);This can be
DEVICE_ATTR_ADMIN_RO()to hide physical address layout information from non-root.drivers/cxl/core/region.c line 131
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+ +static ssize_t size_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%llu\n" , cxlr->config.size);Perhaps no need to store size separately if this becomes:
Quoted 1 line
sysfs_emit(buf, "%llu\n", (unsigned long long) resource_size(cxlr->res));
iirc that broke the 80 character limit so I opted to do it this way. I can change it.
Quoted 19 lines
...?
Quoted 15 lines
drivers/cxl/core/region.c +15
-0+} + +static ssize_t size_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +unsigned long long val; +ssize_t rc; + + rc =kstrtoull (buf,0 , &val); +if (rc) +return rc; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr))
Quoted 9 lines
drivers/cxl/core/region.c line 150
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+ rc = -EBUSY ; + else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.
I might have messed it up, but I didn't seem to run into an issue. With the driver bound check though, it can go away.
I think it would be really good to add this kind of detail to sysfs.rst. Quick grep finds me arm64/kernel/mte and the nfit driver taking the device lock in an attr.
Quoted 54 lines
Quoted 54 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else
Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.
I think show_target should just return the chosen decoder and then the decoder attributes will tell the rest, wouldn't they?
Quoted 20 lines
Quoted 20 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }
Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Per above, I think making this decoders makes sense. I could make it flexible for ease of use, like if you specify memX, the kernel will pick a decoder for you however I suspect you won't like that.
Quoted 102 lines
Quoted 102 lines
drivers/cxl/core/region.c +89
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;
Quoted 7 lines
drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 3 lines
drivers/cxl/core/region.c +2
-0ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
Would remove() be more appropriate?
Quoted 6 lines
Quoted 6 lines
drivers/cxl/core/region.c +0
-1kfree (cxlr); } --2.35.0
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 29 lines · Ben Widawsky
Consolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().
Quoted 3 lines
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Input validation. Every value that gets written to a sysfs attribute should be checked for validity, more below:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0
├──interleave_granularity
...validate granularity is within spec and can be supported by the root decoder.
Quoted 1 line
Quoted 1 line
Quoted 1 line
├──
interleave_ways
...validate ways is within spec and can be supported by the root decoder.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── offset ├── size
...try to reserve decoder capacity to validate that there is available space.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── subsystem -> ../../../../../../bus/cxl ├── target0
...validate that the target maps to the decoder.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── uevent └── uuid
...validate that the uuid is unique relative to other regions.
Quoted 30 lines
Quoted 3 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Yep. This belongs in the previous patch though.
Quoted 8 lines
Quoted 8 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
Quoted 2 lines
It was meant to be relative. I can make it absolute if that's preferable but the physical base is known at the decoder level already.
Yes, but it saves userspace a step to get the absolute value here and matches what happens in PCI sysfs where the user is not required to look up the bridge base to calculate the absolute value.
Quoted 31 lines
Quoted 22 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.Quoted 8 lines
I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".
Quoted 4 lines
Since this is the configuration part of the ABI, I'd rather add that information when the plumbing to report them exists. I'm struggling to understand the balance (as mentioned also earlier in this mail thread) as to what userspace does and what the kernel does. I will add these as you request.
Userspace asks by DPA size and SPA size and the kernel validates / performs the allocation on its behalf.
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl +7
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description:Quoted 9 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 3 lines
Documentation/ABI/testing/sysfs-bus-cxl +3
-0+ (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Quoted 1 line
Okay. Since we're moving toward making the sysfs ABI stateful,
sysfs is always stateful. Stateless would be an ioctl.
Quoted 2 lines
would you like me
to make the attrs only visible when they can actually be set?
No, that's a bit too much magic, and it would be racy.
Quoted 75 lines
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Correct.
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 228
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Okay.
Quoted 23 lines
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" @@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */ Quoted 12 lines
drivers/cxl/core/region.c line 30
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.
Quoted 3 lines
The idea behind this was to give the reviewer somewhat of a bigger picture as to how things should work in the code rather than in a commit message. I will remove this.
They look premature to me.
Quoted 19 lines
Quoted 14 lines
drivers/cxl/core/region.c line 39
Quoted 10 lines
drivers/cxl/core/region.c +9
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{ +struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Quoted 4 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
Quoted 8 lines
This needs some work. The concern I have is region operations will need to operate on memdevs/decoders at various points in time. When the memdev goes away, the region will also need to go away. None of that plumbing was in place in v3 and the reference on the memdev was just a half-hearted attempt at doing the right thing.
For now if you prefer I remove the reference, but perhaps the decoder reference would buy us some safety?
So, I don't want to merge an interim solution. I think this series needs to prove out the end to end final ABI with all the lifetime issues worked out before committing to it upstream because lifetime issues get much harder to fix when they also need to conform to a legacy ABI.
Quoted 25 lines
Quoted 25 lines
Quoted 9 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
if (
dev->driver)
return-EBUSY;
Quoted 8 lines
Quoted 4 lines
...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.Initially my thought here is that this is a problem for userspace to deal with. If userspace can't figure out how to synchronously configure and bind the region, that's not a kernel problem.
The kernel always needs to protect itself. Userspace is free to race itself, but it can not be allowed to trigger a kernel race. So there needs to be protection against userspace writing interleave_ways and the kernel being able to trust that interleave_ways is now static for the life of the region.
Quoted 3 lines
However, we've put some effort into
protecting userspace from itself in the create ABI, so it might be more in line to do that here.
That safety was about preventing userspace from leaking kernel memory, not about protecting userspace from itself. It's still the case that userspace racing itself will get horribly confused when it collides region creation, but the kernel protects itself by resolving the race.
Quoted 2 lines
In summary, I'm fine to add it, but I think I really need to get more in your brain about the userspace/kernel divide sooner rather than later.
Don't let userspace break the kernel, that's it.
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
copypasta
Quoted 4 lines
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);
Quoted 8 lines
Quoted 4 lines
To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
Okay. One of the earlier versions did this implicitly since the #ways was needed to create the region. I thought from the ABI perspective, flexibility was good. Userspace may choose not to utilize it.
More flexibility == more maintenance burden. If it's not strictly necessary, don't expose it, so making this read-only seems simpler to me.
[..]
Quoted 12 lines
Quoted 11 lines
drivers/cxl/core/region.c line 150
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.
Quoted 6 lines
I might have messed it up, but I didn't seem to run into an issue. With the driver bound check though, it can go away.
I think it would be really good to add this kind of detail to
sysfs.rst. Quick grep finds me arm64/kernel/mte and the nfit driver taking the device lock in an attr.
Yeah, CONFIG_PROVE_{NVDIMM,CXL}_LOCKING needs to annotate the driver-core as well. I'm concerned there's a class of deadlocks that lockdep just can't see.
Quoted 66 lines
Quoted 56 lines
Quoted 56 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.
Quoted 2 lines
I think
show_targetshould just return the chosen decoder and then the decoder attributes will tell the rest, wouldn't they?
Given the conflicts that can arise between HDM decoders needing to map increasing DPA values and other conflicts that there will be situations where the kernel auto-picking a decoder will get in the way. Exposing the decoder selection to userspace also gives one more place to do leaf validation. I.e. at decoder-to-region assignment time the kernel can validate that the DPA is available and can be mapped by the given decoder given the state of other decoders on that device.
Quoted 31 lines
Quoted 21 lines
Quoted 21 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 3 lines
Per above, I think making this decoders makes sense. I could make it flexible for ease of use, like if you specify memX, the kernel will pick a decoder for you however I suspect you won't like that.
Right, put the user friendliness in the tooling, not sysfs ABI.
Quoted 111 lines
Quoted 103 lines
Quoted 103 lines
drivers/cxl/core/region.c +89
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;Quoted 7 lines
drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 3 lines
drivers/cxl/core/region.c +2
-0ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
Quoted 1 line
Would
remove()be more appropriate?
->remove() does not seem a good fit since it may be the case that someone wants do "echo $region > /sys/bus/cxl/drivers/cxl_region/unbind; echo $region > /sys/bus/cxl/drivers/cxl_region/bind;" without needing to go reconfigure the targets. I am suggesting that before device_unregister(&cxlr->dev) the targets are released.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 58 lines · Dan Williams
Quoted 29 lines · Ben Widawsky
Consolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 3 lines
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Input validation. Every value that gets written to a sysfs attribute should be checked for validity, more below:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0
├──interleave_granularity...validate granularity is within spec and can be supported by the root decoder.
Quoted 1 line
Quoted 1 line
Quoted 1 line
├──
interleave_ways...validate ways is within spec and can be supported by the root decoder.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── offset ├── size...try to reserve decoder capacity to validate that there is available space.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── subsystem -> ../../../../../../bus/cxl ├── target0
Quoted 7 lines
...validate that the target maps to the decoder.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── uevent └── uuid...validate that the uuid is unique relative to other regions.
Okay.
Quoted 30 lines
Quoted 30 lines
Quoted 3 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Yep. This belongs in the previous patch though.
Quoted 23 lines
Quoted 13 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)Documentation/ABI/testing/sysfs-bus-cxl +0
-0Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
Quoted 7 lines
Quoted 2 lines
It was meant to be relative. I can make it absolute if that's preferable but the physical base is known at the decoder level already.
Yes, but it saves userspace a step to get the absolute value here and matches what happens in PCI sysfs where the user is not required to look up the bridge base to calculate the absolute value.
Okay.
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".
Quoted 7 lines
Quoted 4 lines
Since this is the configuration part of the ABI, I'd rather add that information when the plumbing to report them exists. I'm struggling to understand the balance (as mentioned also earlier in this mail thread) as to what userspace does and what the kernel does. I will add these as you request.
Userspace asks by DPA size and SPA size and the kernel validates / performs the allocation on its behalf.
Okay, I guess that addresses below regarding the tuple.
Quoted 102 lines
Quoted 18 lines
Quoted 17 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl +10
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description: + (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Quoted 1 line
Okay. Since we're moving toward making the sysfs ABI stateful,
sysfs is always stateful. Stateless would be an ioctl.
Quoted 2 lines
would you like me
to make the attrs only visible when they can actually be set?No, that's a bit too much magic, and it would be racy.
Quoted 75 lines
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Correct.
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 228
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Okay.
Quoted 35 lines
Quoted 20 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" drivers/cxl/core/region.c line 30
Quoted 10 lines
drivers/cxl/core/region.c +7
-0@@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */ +struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.
Quoted 6 lines
Quoted 3 lines
The idea behind this was to give the reviewer somewhat of a bigger picture as to how things should work in the code rather than in a commit message. I will remove this.
They look premature to me.
Given that you don't want me to reference the DWG, it is. The steps outlined with TODOs were all based on the DWG's overall flow.
Quoted 25 lines
Quoted 25 lines
Quoted 14 lines
drivers/cxl/core/region.c line 39
Quoted 10 lines
drivers/cxl/core/region.c +9
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{ +struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Quoted 4 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
This needs some work. The concern I have is region operations will need to operate on memdevs/decoders at various points in time. When the memdev goes away, the region will also need to go away. None of that plumbing was in place in v3 and the reference on the memdev was just a half-hearted attempt at doing the right thing.
Quoted 9 lines
Quoted 2 lines
For now if you prefer I remove the reference, but perhaps the decoder reference would buy us some safety?
So, I don't want to merge an interim solution. I think this series needs to prove out the end to end final ABI with all the lifetime issues worked out before committing to it upstream because lifetime issues get much harder to fix when they also need to conform to a legacy ABI.
I should have been clearer. I had been planning to send at least one more version before promising to fix lifetime issues. I will skip that step.
Quoted 34 lines
Quoted 25 lines
Quoted 25 lines
Quoted 9 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
if (
dev->driver)
return-EBUSY;Quoted 8 lines
Quoted 4 lines
...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.Initially my thought here is that this is a problem for userspace to deal with. If userspace can't figure out how to synchronously configure and bind the region, that's not a kernel problem.
Quoted 5 lines
The kernel always needs to protect itself. Userspace is free to race itself, but it can not be allowed to trigger a kernel race. So there needs to be protection against userspace writing
interleave_waysand the kernel being able to trust thatinterleave_waysis now static for the life of the region.
Yeah - originally I was relying on the device_lock for this, but that now doesn't work. seqlock is fine. I could also copy all the config information at the beginning of probe and simply use that.
If we're going the route of making interleave_ways write-once, why not make all attributes the same?
Quoted 148 lines
Quoted 4 lines
However, we've put some effort into
protecting userspace from itself in the create ABI, so it might be more in line to do that here.That safety was about preventing userspace from leaking kernel memory, not about protecting userspace from itself. It's still the case that userspace racing itself will get horribly confused when it collides region creation, but the kernel protects itself by resolving the race.
Quoted 2 lines
In summary, I'm fine to add it, but I think I really need to get more in your brain about the userspace/kernel divide sooner rather than later.
Don't let userspace break the kernel, that's it.
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
copypasta
Quoted 4 lines
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);Quoted 8 lines
Quoted 4 lines
To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
Okay. One of the earlier versions did this implicitly since the #ways was needed to create the region. I thought from the ABI perspective, flexibility was good. Userspace may choose not to utilize it.
More flexibility == more maintenance burden. If it's not strictly necessary, don't expose it, so making this read-only seems simpler to me.
[..]
Quoted 12 lines
Quoted 11 lines
drivers/cxl/core/region.c line 150
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.Quoted 6 lines
I might have messed it up, but I didn't seem to run into an issue. With the driver bound check though, it can go away.
I think it would be really good to add this kind of detail to
sysfs.rst. Quick grep finds me arm64/kernel/mte and the nfit driver taking the device lock in an attr.Yeah, CONFIG_PROVE_{NVDIMM,CXL}_LOCKING needs to annotate the driver-core as well. I'm concerned there's a class of deadlocks that lockdep just can't see.
Quoted 66 lines
Quoted 56 lines
Quoted 56 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.Quoted 2 lines
I think
show_targetshould just return the chosen decoder and then the decoder attributes will tell the rest, wouldn't they?
Quoted 8 lines
Given the conflicts that can arise between HDM decoders needing to map increasing DPA values and other conflicts that there will be situations where the kernel auto-picking a decoder will get in the way. Exposing the decoder selection to userspace also gives one more place to do leaf validation. I.e. at decoder-to-region assignment time the kernel can validate that the DPA is available and can be mapped by the given decoder given the state of other decoders on that device.
Okay, but per below, these are associated with setting the target. The attribute show does only need to provide the decoder, then userspace can look at the decoder to find if it's active/DPAs/etc.
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
Quoted 13 lines
drivers/cxl/core/region.c +12
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); +drivers/cxl/core/region.c line 229
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + } + + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 6 lines
Quoted 3 lines
Per above, I think making this decoders makes sense. I could make it flexible for ease of use, like if you specify memX, the kernel will pick a decoder for you however I suspect you won't like that.
Right, put the user friendliness in the tooling, not sysfs ABI.
Okay.
Quoted 111 lines
Quoted 111 lines
Quoted 110 lines
Quoted 98 lines
drivers/cxl/core/region.c +88
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 8 lines
drivers/cxl/core/region.c +3
-0@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
Quoted 8 lines
Quoted 1 line
Would
remove()be more appropriate?->remove() does not seem a good fit since it may be the case that someone wants do "echo $region >
/sys/bus/cxl/drivers/cxl_region/unbind; echo $region >/sys/bus/cxl/drivers/cxl_region/bind;" without needing to go reconfigure the targets. I am suggesting that beforedevice_unregister(&cxlr->dev)the targets are released.
Okay.
Why would one want to do this? I acknowledge someone *may* do that. I'd like to know what value you see there.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
On Thu, Feb 17, 2022 at 12:20 PM Ben Widawsky <[email protected]> wrote:
[..]
Quoted 14 lines
Quoted 14 lines
Quoted 13 lines
Quoted 9 lines
drivers/cxl/core/region.c line 30
Quoted 4 lines
drivers/cxl/core/region.c +4
-0+static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.The idea behind this was to give the reviewer somewhat of a bigger picture as to how things should work in the code rather than in a commit message. I will remove this.
Quoted 5 lines
Quoted 2 lines
They look premature to me.
Given that you don't want me to reference the DWG, it is. The steps outlined with TODOs were all based on the DWG's overall flow.
Right, the DWG is a good bootstrap, but the Linux implementation is going to go beyond it so might as well couch all the language in terms of base spec references and Linux documentation and not have this indirection to a 3rd document.
[..]
Quoted 15 lines
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.Initially my thought here is that this is a problem for userspace to deal with. If userspace can't figure out how to synchronously configure and bind the region, that's not a kernel problem.
Quoted 5 lines
The kernel always needs to protect itself. Userspace is free to race itself, but it can not be allowed to trigger a kernel race. So there needs to be protection against userspace writing
interleave_waysand the kernel being able to trust thatinterleave_waysis now static for the life of the region.
Quoted 6 lines
Yeah - originally I was relying on the
device_lockfor this, but that now doesn't work. seqlock is fine. I could also copy all the config information at the beginning of probe and simply use that.If we're going the route of making
interleave_wayswrite-once, why not make all attributes the same?
Sure. It could always be relaxed later if there was a convincing need to modify an existing region without tearing it down first. In fact, that reconfigure flexibility was a source of bugs in NVDIMM sysfs ABI that the tooling did not leverage because "ndctl create-namespace --reconfigure" internally did: read namespace attributes, destroy namepsace, create new namespace with saved attributes.
[..]
Quoted 30 lines
Quoted 21 lines
Quoted 18 lines
Quoted 8 lines
Quoted 8 lines
drivers/cxl/core/region.c +8
-0+static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.Quoted 2 lines
I think
show_targetshould just return the chosen decoder and then the decoder attributes will tell the rest, wouldn't they?Quoted 8 lines
Given the conflicts that can arise between HDM decoders needing to map increasing DPA values and other conflicts that there will be situations where the kernel auto-picking a decoder will get in the way. Exposing the decoder selection to userspace also gives one more place to do leaf validation. I.e. at decoder-to-region assignment time the kernel can validate that the DPA is available and can be mapped by the given decoder given the state of other decoders on that device.
Quoted 3 lines
Okay, but per below, these are associated with setting the target. The attribute show does only need to provide the decoder, then userspace can look at the decoder to find if it's active/DPAs/etc.
Yes.
Quoted 161 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
Quoted 14 lines
drivers/cxl/core/region.c +12
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); +drivers/cxl/core/region.c line 229
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + } + + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 6 lines
Quoted 3 lines
Per above, I think making this decoders makes sense. I could make it flexible for ease of use, like if you specify memX, the kernel will pick a decoder for you however I suspect you won't like that.
Right, put the user friendliness in the tooling, not sysfs ABI.
Okay.
Quoted 111 lines
Quoted 111 lines
Quoted 110 lines
Quoted 98 lines
drivers/cxl/core/region.c +88
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 8 lines
drivers/cxl/core/region.c +3
-0@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
Quoted 8 lines
Quoted 1 line
Would
remove()be more appropriate?->remove() does not seem a good fit since it may be the case that someone wants do "echo $region >
/sys/bus/cxl/drivers/cxl_region/unbind; echo $region >/sys/bus/cxl/drivers/cxl_region/bind;" without needing to go reconfigure the targets. I am suggesting that beforedevice_unregister(&cxlr->dev)the targets are released.
Quoted 4 lines
Okay.
Why would one want to do this? I acknowledge someone *may* do that. I'd like to know what value you see there.
There are several debug and error handling scenarios that say "quiesce CXL.mem". Seems reasonable to map those to "cxl disable-region", and seems unreasonable that "cxl disable-region" requires "cxl create-region" to get back to operational state.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
In reply to Dan WilliamsQuoted 41 lines · Dan Williams
Quoted 29 lines · Ben Widawsky
Consolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 3 lines
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Input validation. Every value that gets written to a sysfs attribute should be checked for validity, more below:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
A example is provided below:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
/sys/bus/cxl/devices/region0.0:0
├──interleave_granularity...validate granularity is within spec and can be supported by the root decoder.
Quoted 1 line
Quoted 1 line
Quoted 1 line
├──
interleave_ways...validate ways is within spec and can be supported by the root decoder.
I'm not sure how to do this one. Validation requires device positions and we can't set the targets until ways is set. Can you please provide some more insight on what you'd like me to check in addition to the value being within spec?
Quoted 584 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
├── offset ├── size...try to reserve decoder capacity to validate that there is available space.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── subsystem -> ../../../../../../bus/cxl ├── target0...validate that the target maps to the decoder.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
├── uevent └── uuid...validate that the uuid is unique relative to other regions.
Quoted 30 lines
Quoted 3 lines
As mentioned off-list, it looks like devtype and modalias are missing.
Yep. This belongs in the previous patch though.
Quoted 8 lines
Quoted 8 lines
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-cxl | 40 ++++ drivers/cxl/core/region.c | 300 ++++++++++++++++++++++++ 2 files changed, 340 insertions(+)Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 191
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0@@ -187,3 +187,43 @@ Description: region driver before being deleted. The attributes expects a region in the form "regionX.Y:Z". The region's name, allocated by reading create_region, will also be released. + +What: /sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/offsetThis is just another 'resource' attribute for the physical base address of the region, right? 'offset' sounds like something that would be relative instead of absolute.
Quoted 2 lines
It was meant to be relative. I can make it absolute if that's preferable but the physical base is known at the decoder level already.
Yes, but it saves userspace a step to get the absolute value here and matches what happens in PCI sysfs where the user is not required to look up the bridge base to calculate the absolute value.
Quoted 31 lines
Quoted 22 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 192
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+Date: August, 2021Same date update comment here.
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-cxl +5
-0+KernelVersion: v5.18 +Contact: [email protected] +Description: + (RO) A region resides within an address space that is claimed by + a decoder."A region is a contiguous partition of a CXL Root decoder address space."
Quoted 1 line
Region space allocation is handled by the driver, but
"Region capacity is allocated by writing to the size attribute, the resulting physical address base determined by the driver is reflected here."
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 199
Quoted 2 lines
Documentation/ABI/testing/sysfs-bus-cxl +2
-0+ the offset may be read by userspace tooling in order to + determine fragmentation, and available size for new regions.Quoted 8 lines
I would also expect, before / along with these new region attributes, there would be 'available' and '
max_extent_available' at the decoder level to indicate how much free space the decoder has and how big the next region creation can be. User tooling can walk the decoder and the regions together to determine fragmentation if necessary, but for the most part the tool likely only cares about "how big can the next region be?" and "how full is this decoder?".Quoted 4 lines
Since this is the configuration part of the ABI, I'd rather add that information when the plumbing to report them exists. I'm struggling to understand the balance (as mentioned also earlier in this mail thread) as to what userspace does and what the kernel does. I will add these as you request.
Userspace asks by DPA size and SPA size and the kernel validates / performs the allocation on its behalf.
Quoted 17 lines
Quoted 7 lines
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl +7
-0+ +What: +/sys/bus/cxl/devices/decoderX.Y/regionX.Y:Z/{interleave,size,uuid,target[0-15]} +Date: August, 2021 +KernelVersion: v5.18 +Contact: [email protected] +Description:Quoted 9 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 209
Quoted 3 lines
Documentation/ABI/testing/sysfs-bus-cxl +3
-0+ (RW) Configuring regions requires a minimal set of parameters in + order for the subsequent bind operation to succeed. The + following parameters are defined:Let's split up the descriptions into individual sections. That can also document the order that attributes must be written. For example, doesn't size need to be set before targets are added so that targets can be validated whether they have sufficient capacity?
Quoted 1 line
Okay. Since we're moving toward making the sysfs ABI stateful,
sysfs is always stateful. Stateless would be an ioctl.
Quoted 2 lines
would you like me
to make the attrs only visible when they can actually be set?No, that's a bit too much magic, and it would be racy.
Quoted 75 lines
Quoted 15 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 220
Quoted 12 lines
Documentation/ABI/testing/sysfs-bus-cxl +11
-0+ + == ======================================================== + interleave_granularity Mandatory. Number of consecutive bytes + each device in the interleave set will claim. The + possible interleave granularity values are determined by + the CXL spec and the participating devices. + interleave_ways Mandatory. Number of devices participating in the + region. Each device will provide 1/interleave of storage + for the region. + size Manadatory. Phsyical address space the region will + consume.s/Phsyical/Physical/
Quoted 7 lines
Documentation/ABI/testing/sysfs-bus-cxl Description:
Quoted 4 lines
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ target Mandatory. Memory devices are the backing storage for a + region. There will be N targets based on the number of + interleave ways that the top level decoder is configured + for.That doesn't sound right, IW at the root != IW at the endpoint level and the region needs to record all the endpoint level targets.
Correct.
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-cxl Description: · line 228
Quoted 6 lines
Each target must be set with a memdev device ie.
Documentation/ABI/testing/sysfs-bus-cxl +4
-0+ 'mem1'. This attribute only becomes available after + setting the 'interleave' attribute. + uuid Optional. A unique identifier for the region. If none is + selected, the kernel will create one.Let's drop the Mandatory / Optional distinction, or I am otherwise not understanding what this is trying to document. For example 'uuid' is "mandatory" for PMEM regions and "omitted" for volatile regions not optional.
Okay.
Quoted 23 lines
Quoted 23 lines
Documentation/ABI/testing/sysfs-bus-cxl +1
-0+ == ========================================================drivers/cxl/core/region.c +3
-0@@ -3,9 +3,12 @@#include <linux/io-64-nonatomic-lo-hi.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sizes.h> #include <linux/slab.h> +#include <linux/uuid.h> #include <linux/idr.h> #include <region.h> +#include <cxlmem.h> #include <cxl.h> #include "core.h" @@ -18,11 +21,305 @@* (programming the hardware) is handled by a separate region driver. */ Quoted 12 lines
drivers/cxl/core/region.c line 30
Quoted 7 lines
drivers/cxl/core/region.c +7
-0+struct cxl_region *to_cxl_region (struct device *dev); +static const struct attribute_group region_interleave_group; + +static bool is_region_active (struct cxl_region *cxlr) +{ +/* TODO: Regions can't be activated yet. */ +return false;This function seems redundant with just checking "
cxlr->dev.driver!= NULL"? The benefit of that is there is no need to carry a TODO in the series.Quoted 3 lines
The idea behind this was to give the reviewer somewhat of a bigger picture as to how things should work in the code rather than in a commit message. I will remove this.
They look premature to me.
Quoted 19 lines
Quoted 14 lines
drivers/cxl/core/region.c line 39
Quoted 10 lines
drivers/cxl/core/region.c +9
-0+} + +static void remove_target (struct cxl_region *cxlr,int target) +{ +struct cxl_memdev *cxlmd; + + cxlmd = cxlr->config.targets[target]; +if (cxlmd) +put_device (&cxlmd->dev);A memdev can be a member of multiple regions at once, shouldn't this be an endpoint decoder or similar, not the entire memdev?
Quoted 4 lines
Also, if memdevs autoremove themselves from regions at memdev ->remove() time then I don't think the region needs to hold references on memdevs.
Quoted 8 lines
This needs some work. The concern I have is region operations will need to operate on memdevs/decoders at various points in time. When the memdev goes away, the region will also need to go away. None of that plumbing was in place in v3 and the reference on the memdev was just a half-hearted attempt at doing the right thing.
For now if you prefer I remove the reference, but perhaps the decoder reference would buy us some safety?
So, I don't want to merge an interim solution. I think this series needs to prove out the end to end final ABI with all the lifetime issues worked out before committing to it upstream because lifetime issues get much harder to fix when they also need to conform to a legacy ABI.
Quoted 25 lines
Quoted 25 lines
Quoted 9 lines
drivers/cxl/core/region.c +8
-0+ cxlr->config.targets[target] =NULL ; +} + +static ssize_t interleave_ways_show (struct device *dev, +struct device_attribute *attr,char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +drivers/cxl/core/region.c line 57
Quoted 10 lines
drivers/cxl/core/region.c +10
-0+return sysfs_emit (buf,"%d\n" , cxlr->config.interleave_ways); +} + +static ssize_t interleave_ways_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +int ret, prev_iw; +int val;I would expect:
if (
dev->driver)
return-EBUSY;Quoted 8 lines
Quoted 4 lines
...to shutdown configuration writes once the region is active. Might also need a region-wide seqlock like
target_list_show. So that region probe drains all active sysfs writers before assuming the configuration is stable.Initially my thought here is that this is a problem for userspace to deal with. If userspace can't figure out how to synchronously configure and bind the region, that's not a kernel problem.
The kernel always needs to protect itself. Userspace is free to race itself, but it can not be allowed to trigger a kernel race. So there needs to be protection against userspace writing
interleave_waysand the kernel being able to trust thatinterleave_waysis now static for the life of the region.Quoted 3 lines
However, we've put some effort into
protecting userspace from itself in the create ABI, so it might be more in line to do that here.That safety was about preventing userspace from leaking kernel memory, not about protecting userspace from itself. It's still the case that userspace racing itself will get horribly confused when it collides region creation, but the kernel protects itself by resolving the race.
Quoted 2 lines
In summary, I'm fine to add it, but I think I really need to get more in your brain about the userspace/kernel divide sooner rather than later.
Don't let userspace break the kernel, that's it.
Quoted 27 lines
Quoted 11 lines
Quoted 11 lines
drivers/cxl/core/region.c +9
-0+ + prev_iw = cxlr->config.interleave_ways; + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; +if (ret <0 || ret >CXL_DECODER_MAX_INTERLEAVE ) +return -EINVAL ; + + cxlr->config.interleave_ways = val;Quoted 9 lines
drivers/cxl/core/region.c line 72
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+ + ret =sysfs_update_group (&dev->kobj, ®ion_interleave_group); +if (ret <0 ) + goto err; + +sysfs_notify (&dev->kobj,NULL ,"target_interleave" );Why?
copypasta
Quoted 4 lines
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ +while (prev_iw > cxlr->config.interleave_ways) +remove_target (cxlr, --prev_iw);Quoted 8 lines
Quoted 4 lines
To make the kernel side simpler this attribute could just require that setting interleave ways is a one way street, if you want to change it you need to delete the region and start over.
Okay. One of the earlier versions did this implicitly since the #ways was needed to create the region. I thought from the ABI perspective, flexibility was good. Userspace may choose not to utilize it.
More flexibility == more maintenance burden. If it's not strictly necessary, don't expose it, so making this read-only seems simpler to me.
[..]
Quoted 12 lines
Quoted 11 lines
drivers/cxl/core/region.c line 150
Quoted 6 lines
drivers/cxl/core/region.c +6
-0+device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + cxlr->config.size = val; +device_unlock (&cxlr->dev);I think lockdep will complain about
device_lock()usage in an attribute. Try changing this tocxl_device_lock()withCONFIG_PROVE_CXL_LOCKING=y.Quoted 6 lines
I might have messed it up, but I didn't seem to run into an issue. With the driver bound check though, it can go away.
I think it would be really good to add this kind of detail to
sysfs.rst. Quick grep finds me arm64/kernel/mte and the nfit driver taking the device lock in an attr.Yeah, CONFIG_PROVE_{NVDIMM,CXL}_LOCKING needs to annotate the driver-core as well. I'm concerned there's a class of deadlocks that lockdep just can't see.
Quoted 66 lines
Quoted 56 lines
Quoted 56 lines
drivers/cxl/core/region.c +54
-0+ +return rc ? rc : len; +} +static DEVICE_ATTR_RW (size); + +static ssize_t uuid_show (struct device *dev,struct device_attribute *attr, +char *buf) +{ +struct cxl_region *cxlr =to_cxl_region (dev); + +return sysfs_emit (buf,"%pUb\n" , &cxlr->config.uuid); +} + +static ssize_t uuid_store (struct device *dev,struct device_attribute *attr, +const char *buf,size_t len) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +ssize_t rc; + +if (len !=UUID_STRING_LEN +1 ) +return -EINVAL ; + +device_lock (&cxlr->dev); +if (is_region_active (cxlr)) + rc = -EBUSY ; +else + rc =uuid_parse (buf, &cxlr->config.uuid); +device_unlock (&cxlr->dev); + +return rc ? rc : len; +} +static DEVICE_ATTR_RW (uuid); + +static struct attribute *region_attrs[] = { + &dev_attr_interleave_ways.attr, + &dev_attr_interleave_granularity.attr, + &dev_attr_offset.attr, + &dev_attr_size.attr, + &dev_attr_uuid.attr, +NULL , +}; + +static const struct attribute_group region_group = { + .attrs = region_attrs, +}; + +static size_t show_targetN (struct cxl_region *cxlr,char *buf,int n) +{ +int ret; + +device_lock (&cxlr->dev); +if (!cxlr->config.targets[n]) + ret =sysfs_emit (buf,"\n" ); +else Quoted 9 lines
drivers/cxl/core/region.c line 207
Quoted 3 lines
drivers/cxl/core/region.c +3
-0+ ret =sysfs_emit (buf,"%s\n" , +dev_name (&cxlr->config.targets[n]->dev)); +device_unlock (&cxlr->dev);The component contribution of a memdev to a region is a DPA-span, not the whole memdev. I would expect something like
dax_mapping_attributesorREGION_MAPPING()fromdrivers/nvdimm/region_devs.c. A tuple of information about the component contribution of a memdev to a region.Quoted 2 lines
I think
show_targetshould just return the chosen decoder and then the decoder attributes will tell the rest, wouldn't they?Given the conflicts that can arise between HDM decoders needing to map increasing DPA values and other conflicts that there will be situations where the kernel auto-picking a decoder will get in the way. Exposing the decoder selection to userspace also gives one more place to do leaf validation. I.e. at decoder-to-region assignment time the kernel can validate that the DPA is available and can be mapped by the given decoder given the state of other decoders on that device.
Quoted 31 lines
Quoted 21 lines
Quoted 21 lines
drivers/cxl/core/region.c +20
-0+ +return ret; +} + +static size_t set_targetN (struct cxl_region *cxlr,const char *buf,int n, +size_t len) +{ +struct device *memdev_dev; +struct cxl_memdev *cxlmd; + +device_lock (&cxlr->dev); + +if (len ==1 || cxlr->config.targets[n]) +remove_target (cxlr, n); + +/* Remove target special case */ +if (len ==1 ) { +device_unlock (&cxlr->dev); +return len; + }Quoted 9 lines
drivers/cxl/core/region.c line 229
Quoted 2 lines
drivers/cxl/core/region.c +2
-0+ + memdev_dev =bus_find_device_by_name (&cxl_bus_type,NULL , buf);I think this wants to be an endpoint decoder, not a memdev. Because it's the decoder that joins a memdev to a region, or at least a decoder should be picked when the memdev is assigned so that the DPA mapping can be registered. If all the decoders are allocated then fail here.
Quoted 3 lines
Per above, I think making this decoders makes sense. I could make it flexible for ease of use, like if you specify memX, the kernel will pick a decoder for you however I suspect you won't like that.
Right, put the user friendliness in the tooling, not sysfs ABI.
Quoted 111 lines
Quoted 103 lines
Quoted 103 lines
drivers/cxl/core/region.c +89
-0+if (!memdev_dev) { +device_unlock (&cxlr->dev); +return -ENOENT ; + } + +/* reference to memdev held until target is unset or region goes away */ + + cxlmd =to_cxl_memdev (memdev_dev); + cxlr->config.targets[n] = cxlmd; + +device_unlock (&cxlr->dev); + +return len; +} + +#define TARGET_ATTR_RW (n) \ + static ssize_t target##n##_show( \ + struct device *dev, struct device_attribute *attr, char *buf) \ + { \ + return show_targetN(to_cxl_region(dev), buf, (n)); \ + } \ + static ssize_t target##n##_store(struct device *dev, \ + struct device_attribute *attr, \ + const char *buf, size_t len) \ + { \ + return set_targetN(to_cxl_region(dev), buf, (n), len); \ + } \ + static DEVICE_ATTR_RW(target##n) + +TARGET_ATTR_RW (0 ); +TARGET_ATTR_RW (1 ); +TARGET_ATTR_RW (2 ); +TARGET_ATTR_RW (3 ); +TARGET_ATTR_RW (4 ); +TARGET_ATTR_RW (5 ); +TARGET_ATTR_RW (6 ); +TARGET_ATTR_RW (7 ); +TARGET_ATTR_RW (8 ); +TARGET_ATTR_RW (9 ); +TARGET_ATTR_RW (10 ); +TARGET_ATTR_RW (11 ); +TARGET_ATTR_RW (12 ); +TARGET_ATTR_RW (13 ); +TARGET_ATTR_RW (14 ); +TARGET_ATTR_RW (15 ); + +static struct attribute *interleave_attrs[] = { + &dev_attr_target0.attr, + &dev_attr_target1.attr, + &dev_attr_target2.attr, + &dev_attr_target3.attr, + &dev_attr_target4.attr, + &dev_attr_target5.attr, + &dev_attr_target6.attr, + &dev_attr_target7.attr, + &dev_attr_target8.attr, + &dev_attr_target9.attr, + &dev_attr_target10.attr, + &dev_attr_target11.attr, + &dev_attr_target12.attr, + &dev_attr_target13.attr, + &dev_attr_target14.attr, + &dev_attr_target15.attr, +NULL , +}; + +static umode_t visible_targets (struct kobject *kobj,struct attribute *a,int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); +struct cxl_region *cxlr =to_cxl_region (dev); + +if (n < cxlr->config.interleave_ways) +return a->mode; +return 0 ; +} + +static const struct attribute_group region_interleave_group = { + .attrs = interleave_attrs, + .is_visible = visible_targets, +}; + +static const struct attribute_group *region_groups[] = { + ®ion_group, + ®ion_interleave_group, +NULL , +}; +static void cxl_region_release (struct device *dev);static const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, + .groups = region_groups }; static ssize_t create_region_show(struct device *dev,@@ -108,8 +405,11 @@ static void cxl_region_release(struct device *dev) {struct cxl_decoder *cxld =to_cxl_decoder (dev->parent);struct cxl_region *cxlr =to_cxl_region (dev); +int i;Quoted 7 lines
drivers/cxl/core/region.c static void cxl_region_release(struct device *dev) · line 412
Quoted 3 lines
drivers/cxl/core/region.c +2
-0ida_free (&cxld->region_ida, cxlr->id); +for (i =0 ; i < cxlr->config.interleave_ways; i++) +remove_target (cxlr, i);Like the last patch this feels too late. I expect whatever unregisters the region should have already handled removing the targets.
Quoted 1 line
Would
remove()be more appropriate?->remove() does not seem a good fit since it may be the case that someone wants do "echo $region >
/sys/bus/cxl/drivers/cxl_region/unbind; echo $region >/sys/bus/cxl/drivers/cxl_region/bind;" without needing to go reconfigure the targets. I am suggesting that beforedevice_unregister(&cxlr->dev)the targets are released.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 48 lines · Ben Widawsky
Quoted 46 lines · Dan Williams
Quoted 29 lines · Ben Widawsky
Consolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 3 lines
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Input validation. Every value that gets written to a sysfs attribute should be checked for validity, more below:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0
├──interleave_granularity...validate granularity is within spec and can be supported by the root decoder.
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
├──
interleave_ways...validate ways is within spec and can be supported by the root decoder.
I'm not sure how to do this one. Validation requires device positions and we can't set the targets until ways is set. Can you please provide some more insight on what you'd like me to check in addition to the value being within spec?
For example you could check that interleave_ways is >= to the root level interleave. I.e. it would be invalid to attempt a x1 interleave on a decoder that is x2 interleaved at the host-bridge level.
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 53 lines · Dan Williams
Quoted 52 lines · Ben Widawsky
Quoted 49 lines · Dan Williams
Quoted 29 lines · Ben Widawsky
Consolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 3 lines
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Input validation. Every value that gets written to a sysfs attribute should be checked for validity, more below:
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
A example is provided below:
/sys/bus/cxl/devices/region0.0:0
├──interleave_granularity...validate granularity is within spec and can be supported by the root decoder.
Quoted 1 line
Quoted 1 line
Quoted 1 line
├──
interleave_ways...validate ways is within spec and can be supported by the root decoder.
Quoted 8 lines
Quoted 4 lines
I'm not sure how to do this one. Validation requires device positions and we can't set the targets until ways is set. Can you please provide some more insight on what you'd like me to check in addition to the value being within spec?
For example you could check that
interleave_waysis >= to the root level interleave. I.e. it would be invalid to attempt a x1 interleave on a decoder that is x2 interleaved at the host-bridge level.
I tried to convince myself that that assertion always holds and didn't feel super comfortable. If you do, I can add those kinds of checks.
Thanks.
Ben
Re: [PATCH v3 02/14] cxl/region: Introduce concept of region configuration
Quoted 60 lines · Ben Widawsky
Quoted 58 lines · Dan Williams
Quoted 56 lines · Ben Widawsky
Quoted 41 lines · Dan Williams
Quoted 29 lines · Ben Widawsky
Consolidating earlier discussions...
Quoted 16 lines · Dan Williams
Quoted 8 lines · Ben Widawsky
The region creation APIs create a vacant region. Configuring the region works in the same way as similar subsystems such as devdax. Sysfs attrs will be provided to allow userspace to configure the region. Finally once all configuration is complete, userspace may activate the region.
Introduced here are the most basic attributes needed to configure a region. Details of these attribute are described in the ABI
s/attribute/attributes/
Quoted 3 lines
Documentation. Sanity checking of configuration parameters are done at region binding time. This consolidates all such logic in one place, rather than being strewn across multiple places.
Quoted 8 lines
I think that's too late for some of the validation. The complex validation that the region driver does throughout the topology is different from the basic input validation that can be done at the sysfs write time. For example ,this patch allows negative
interleave_granularityvalues to specified, just return-EINVAL. I agree that sysfs should not validate everything, I disagree with pushing all validation tocxl_region_probe().Quoted 3 lines
Okay. It might save us some back and forth if you could outline everything you'd expect to be validated, but I can also make an attempt to figure out the reasonable set of things.
Input validation. Every value that gets written to a sysfs attribute should be checked for validity, more below:
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
A example is provided below:
Quoted 8 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
/sys/bus/cxl/devices/region0.0:0
├──interleave_granularity...validate granularity is within spec and can be supported by the root decoder.
Quoted 1 line
Quoted 1 line
Quoted 1 line
├──
interleave_ways...validate ways is within spec and can be supported by the root decoder.
I'm not sure how to do this one. Validation requires device positions and we can't set the targets until ways is set. Can you please provide some more insight on what you'd like me to check in addition to the value being within spec?
Quoted 6 lines
Quoted 3 lines
For example you could check that
interleave_waysis >= to the root level interleave. I.e. it would be invalid to attempt a x1 interleave on a decoder that is x2 interleaved at the host-bridge level.I tried to convince myself that that assertion always holds and didn't feel super comfortable. If you do, I can add those kinds of checks.
The only way to support a x1 region on an x2 interleave is to have the size be equal to interleave granularity so that accesses stay contained to that one device.
In fact that's another validation step, which you might already have, region size must be >= and aligned to interleave_granularity * interleave_ways.
PATCH v3 3/14 cxl/mem: Cache port created by the mem dev
In reply to Ben WidawskySince region programming sees all components in the topology as a port, it's required that endpoints are treated equally. The easiest way to go from endpoint to port is to simply cache it at creation time.
---
Changes since v2:
- Rebased on Dan's latest port/mem changes
- Keep a reference to the port until the memdev goes away
- add action to release device reference for the port
--- drivers/cxl/cxlmem.h | 2 ++ drivers/cxl/mem.c | 35 ++++++++++++++++++++++++++++------- 2 files changed, 30 insertions(+), 7 deletions(-)
drivers/cxl/cxlmem.h +2 -0
@@ -37,6 +37,7 @@* @id: id number of this memdev instance. * @detach_work: active memdev lost a port in its ancestry * @component_reg_phys: register base of component registers +* @port: The port created by this device */ struct cxl_memdev {struct device dev;
@@ -44,6 +45,7 @@ struct cxl_memdev {struct cxl_dev_state *cxlds;struct work_struct detach_work;int id; +struct cxl_port *port; };static inline struct cxl_memdev *to_cxl_memdev (struct device *dev)
drivers/cxl/mem.c +28 -7
@@ -45,26 +45,31 @@ static int wait_for_media(struct cxl_memdev *cxlmd)return 0 ; } -static int create_endpoint (struct cxl_memdev *cxlmd, -struct cxl_port *parent_port) +static struct cxl_port *create_endpoint (struct cxl_memdev *cxlmd, +struct cxl_port *parent_port) {struct cxl_dev_state *cxlds = cxlmd->cxlds;struct cxl_port *endpoint; +int rc; endpoint =devm_cxl_add_port (&parent_port->dev, &cxlmd->dev, cxlds->component_reg_phys, parent_port);if (IS_ERR (endpoint)) -return PTR_ERR (endpoint); +return endpoint;dev_dbg (&cxlmd->dev,"add: %s\n" ,dev_name (&endpoint->dev));if (!endpoint->dev.driver) {dev_err (&cxlmd->dev,"%s failed probe\n" ,dev_name (&endpoint->dev)); -return -ENXIO ; +return ERR_PTR (-ENXIO ); } -return cxl_endpoint_autoremove (cxlmd, endpoint); + rc =cxl_endpoint_autoremove (cxlmd, endpoint); +if (rc) +return ERR_PTR (rc); + +return endpoint; }/**
@@ -127,11 +132,18 @@ __mock bool cxl_dvsec_decode_init(struct cxl_dev_state *cxlds)return do_hdm_init; } +static void delete_memdev (void *dev) +{ +struct cxl_memdev *cxlmd = dev; + +put_device (&cxlmd->port->dev); +} +static int cxl_mem_probe (struct device *dev) {struct cxl_memdev *cxlmd =to_cxl_memdev (dev);struct cxl_dev_state *cxlds = cxlmd->cxlds; -struct cxl_port *parent_port; +struct cxl_port *parent_port, *ep_port;int rc;/*
@@ -201,7 +213,16 @@ static int cxl_mem_probe(struct device *dev) goto out; } - rc =create_endpoint (cxlmd, parent_port); + ep_port =create_endpoint (cxlmd, parent_port); +if (IS_ERR (ep_port)) { + rc =PTR_ERR (ep_port); + goto out; + } + +get_device (&ep_port->dev); + cxlmd->port = ep_port; + + rc =devm_add_action_or_reset (dev, delete_memdev, cxlmd); out:cxl_device_unlock (&parent_port->dev);put_device (&parent_port->dev);
Signature · 1 line
2.35.0
Re: [PATCH v3 03/14] cxl/mem: Cache port created by the mem dev
Quoted 4 lines · Ben Widawsky
Since region programming sees all components in the topology as a port, it's required that endpoints are treated equally. The easiest way to go from endpoint to port is to simply cache it at creation time.
As of 8dd2bc0f8e02 ("cxl/mem: Add the cxl_mem driver"), cxl_endpoint_autoremove() already sets cxlmd drvdata to @endpoint, so this patch isn't needed.
PATCH v3 4/14 cxl/region: Introduce a cxl_region driver
In reply to Ben WidawskyThe cxl_region driver is responsible for managing the HDM decoder programming in the CXL topology. Once a region is created it must be configured and bound to the driver in order to activate it.
The following is a sample of how such controls might work:
region=$(cat /sys/bus/cxl/devices/decoder0.0/create_region) echo $region > /sys/bus/cxl/devices/decoder0.0/create_region echo 2 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/interleave echo $((256<<20)) > /sys/bus/cxl/devices/decoder0.0/region0.0:0/size echo mem0 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/target0 echo mem1 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/target1 echo region0.0:0 > /sys/bus/cxl/drivers/cxl_region/bind
In order to handle the eventual rise in failure modes of binding a region, a new trace event is created to help track these failures for debug and reconfiguration paths in userspace.
---
Changes since v2:
- Add CONFIG_CXL_REGION
- Check ways/granularity in sanitize
--- .../driver-api/cxl/memory-devices.rst | 3 + drivers/cxl/Kconfig | 4 + drivers/cxl/Makefile | 2 + drivers/cxl/core/core.h | 1 + drivers/cxl/core/port.c | 17 +- drivers/cxl/core/region.c | 25 +- drivers/cxl/cxl.h | 31 ++ drivers/cxl/region.c | 349 ++++++++++++++++++ drivers/cxl/region.h | 4 + 9 files changed, 431 insertions(+), 5 deletions(-) create mode 100644 drivers/cxl/region.c
Documentation/driver-api/cxl/memory-devices.rst +3 -0
@@ -364,6 +364,9 @@ CXL Core CXL Regions ----------- +.. kernel-doc:: drivers/cxl/region.c + :doc: cxl region + .. kernel-doc:: drivers/cxl/region.h :identifiers:
drivers/cxl/Kconfig +4 -0
@@ -98,4 +98,8 @@ config CXL_PORT default CXL_BUS tristate +config CXL_REGION + default CXL_PORT + tristate + endif
drivers/cxl/Makefile +2 -0
@@ -5,9 +5,11 @@ obj-$(CONFIG_CXL_MEM) += cxl_mem.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o obj-$(CONFIG_CXL_PORT) += cxl_port.o +obj-$(CONFIG_CXL_REGION) += cxl_region.o cxl_mem-y := mem.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o cxl_port-y := port.o +cxl_region-y := region.o
drivers/cxl/core/core.h +1 -0
@@ -7,6 +7,7 @@extern const struct device_type cxl_nvdimm_bridge_type;extern const struct device_type cxl_nvdimm_type;extern const struct device_type cxl_memdev_type; +extern const struct device_type cxl_region_type;extern struct attribute_group cxl_base_attribute_group;
drivers/cxl/core/port.c +15 -2
@@ -9,6 +9,7 @@#include <linux/idr.h> #include <cxlmem.h> #include <cxlpci.h> +#include <region.h> #include <cxl.h> #include "core.h"
@@ -49,6 +50,8 @@ static int cxl_device_id(struct device *dev) }if (dev->type == &cxl_memdev_type)return CXL_DEVICE_MEMORY_EXPANDER ; +if (dev->type == &cxl_region_type) +return CXL_DEVICE_REGION ;return 0 ; }
@@ -1425,13 +1428,23 @@ static int cxl_bus_match(struct device *dev, struct device_driver *drv)static int cxl_bus_probe (struct device *dev) { -int rc; +int id =cxl_device_id (dev); +int rc = -ENODEV ;cxl_nested_lock (dev); - rc =to_cxl_drv (dev->driver)->probe (dev); +if (id ==CXL_DEVICE_REGION ) { +/* Regions cannot bind until parameters are set */ +struct cxl_region *cxlr =to_cxl_region (dev); + +if (is_cxl_region_configured (cxlr)) + rc =to_cxl_drv (dev->driver)->probe (dev); + }else { + rc =to_cxl_drv (dev->driver)->probe (dev); + }cxl_nested_unlock (dev);dev_dbg (dev,"probe: %d\n" , rc); +return rc; }
drivers/cxl/core/region.c +22 -3
@@ -12,6 +12,8 @@#include <cxl.h> #include "core.h" +#include "core.h" +/** * DOC: cxl core region *
@@ -26,10 +28,27 @@ static const struct attribute_group region_interleave_group;static bool is_region_active (struct cxl_region *cxlr) { -/* TODO: Regions can't be activated yet. */ -return false; +return cxlr->active; } +/* +* Most sanity checking is left up to region binding. This does the most basic +* check to determine whether or not the core should try probing the driver. +*/ +bool is_cxl_region_configured (const struct cxl_region *cxlr) +{ +/* zero sized regions aren't a thing. */ +if (cxlr->config.size <=0 ) +return false; + +/* all regions have at least 1 target */ +if (!cxlr->config.targets[0 ]) +return false; + +return true; +} +EXPORT_SYMBOL_GPL (is_cxl_region_configured); +static void remove_target (struct cxl_region *cxlr,int target) {struct cxl_memdev *cxlmd;
@@ -316,7 +335,7 @@ static const struct attribute_group *region_groups[] = {static void cxl_region_release (struct device *dev); -static const struct device_type cxl_region_type = { +const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, .groups = region_groups
drivers/cxl/cxl.h +31 -0
@@ -81,6 +81,31 @@ static inline int cxl_to_interleave_ways(u8 eniw) } } +static inline bool cxl_is_interleave_ways_valid (int iw) +{ +switch (iw) { + case 0 ... 4: +case 6 : +case 8 : +case 12 : +case 16 : +return true; +default : +return false; + } + +unreachable (); +} + +static inline bool cxl_is_interleave_granularity_valid (int ig) +{ +if (!is_power_of_2 (ig)) +return false; + +/* 16K is the max */ +return ((ig >>15 ) ==0 ); +} +/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ #define CXLDEV_CAP_ARRAY_OFFSET 0x0#define CXLDEV_CAP_ARRAY_CAP_ID 0
@@ -199,6 +224,10 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,#define CXL_DECODER_F_ENABLE BIT(5)#define CXL_DECODER_F_MASK GENMASK(5, 0) +#define cxl_is_pmem_t3 (flags) \ + (((flags) & (CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM)) == \ + (CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM)) +enum cxl_decoder_type {CXL_DECODER_ACCELERATOR =2 ,CXL_DECODER_EXPANDER =3 ,
@@ -357,6 +386,7 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port, resource_size_t component_reg_phys);struct cxl_dport *cxl_find_dport_by_dev (struct cxl_port *port,const struct device *dev); +struct cxl_port *ep_find_cxl_port (struct cxl_memdev *cxlmd,unsigned int depth);struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);
@@ -404,6 +434,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);#define CXL_DEVICE_PORT 3#define CXL_DEVICE_ROOT 4#define CXL_DEVICE_MEMORY_EXPANDER 5 +#define CXL_DEVICE_REGION 6#define MODULE_ALIAS_CXL (type) MODULE_ALIAS("cxl:t" __stringify(type) "*")#define CXL_MODALIAS_FMT "cxl:t%d"
drivers/cxl/region.c new file +349 -0
349 lines not on this page · raw message
drivers/cxl/region.h +4 -0
@@ -13,6 +13,7 @@* @id: This regions id. Id is globally unique across all regions. * @list: Node in decoder's region list. * @res: Resource this region carves out of the platform decode range. +* @active: If the region has been activated. * @config: HDM decoder program config * @config.size: Size of the region determined from LSA or userspace. * @config.uuid: The UUID for this region.
@@ -25,6 +26,7 @@ struct cxl_region {int id;struct list_head list;struct resource *res; +bool active;struct {u64 size;
@@ -35,4 +37,6 @@ struct cxl_region { } config; }; +bool is_cxl_region_configured (const struct cxl_region *cxlr); +#endif
Signature · 1 line
2.35.0
Re: [PATCH v3 04/14] cxl/region: Introduce a cxl_region driver
Quoted 14 lines · On Thu, 27 Jan 2022 16:26:57 -0800 Ben Widawsky <[email protected]> wrote:
The
cxl_regiondriver is responsible for managing the HDM decoder programming in the CXL topology. Once a region is created it must be configured and bound to the driver in order to activate it.The following is a sample of how such controls might work:
region=$(cat
/sys/bus/cxl/devices/decoder0.0/create_region) echo $region >/sys/bus/cxl/devices/decoder0.0/create_regionecho 2 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/interleave echo $((256<<20)) > /sys/bus/cxl/devices/decoder0.0/region0.0:0/size echo mem0 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/target0 echo mem1 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/target1 echo region0.0:0 >/sys/bus/cxl/drivers/cxl_region/bind
Quoted 6 lines
In order to handle the eventual rise in failure modes of binding a region, a new trace event is created to help track these failures for debug and reconfiguration paths in userspace.
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>
A few minor comments inline.
Thanks,
Jonathan
Quoted 106 lines
---
Changes since v2:
- AddCONFIG_CXL_REGION
- Check ways/granularity in sanitize--- .../driver-api/cxl/memory-devices.rst | 3 + drivers/cxl/Kconfig | 4 + drivers/cxl/Makefile | 2 + drivers/cxl/core/core.h | 1 + drivers/cxl/core/port.c | 17 +- drivers/cxl/core/region.c | 25 +- drivers/cxl/cxl.h | 31 ++ drivers/cxl/region.c | 349 ++++++++++++++++++ drivers/cxl/region.h | 4 + 9 files changed, 431 insertions(+), 5 deletions(-) create mode 100644 drivers/cxl/region.cDocumentation/driver-api/cxl/memory-devices.rst +3
-0@@ -364,6 +364,9 @@ CXL Core CXL Regions ----------- +.. kernel-doc:: drivers/cxl/region.c + :doc: cxl region + .. kernel-doc:: drivers/cxl/region.h :identifiers:drivers/cxl/Kconfig +4
-0@@ -98,4 +98,8 @@ config CXL_PORT default CXL_BUS tristate +config CXL_REGION + default CXL_PORT + tristate + endifdrivers/cxl/Makefile +2
-0@@ -5,9 +5,11 @@ obj-$(CONFIG_CXL_MEM) += cxl_mem.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o obj-$(CONFIG_CXL_PORT) += cxl_port.o +obj-$(CONFIG_CXL_REGION) += cxl_region.o cxl_mem-y := mem.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o cxl_port-y := port.o +cxl_region-y := region.odrivers/cxl/core/core.h +1
-0@@ -7,6 +7,7 @@extern const struct device_type cxl_nvdimm_bridge_type;extern const struct device_type cxl_nvdimm_type;extern const struct device_type cxl_memdev_type; +extern const struct device_type cxl_region_type;extern struct attribute_group cxl_base_attribute_group;drivers/cxl/core/port.c +8
-2@@ -9,6 +9,7 @@#include <linux/idr.h> #include <cxlmem.h> #include <cxlpci.h> +#include <region.h> #include <cxl.h> #include "core.h" @@ -49,6 +50,8 @@ static int cxl_device_id(struct device *dev) }if (dev->type == &cxl_memdev_type)return CXL_DEVICE_MEMORY_EXPANDER ; +if (dev->type == &cxl_region_type) +return CXL_DEVICE_REGION ;return 0 ; }@@ -1425,13 +1428,23 @@ static int cxl_bus_match(struct device *dev, struct device_driver *drv)static int cxl_bus_probe (struct device *dev) { -int rc; +int id =cxl_device_id (dev); +int rc = -ENODEV ;cxl_nested_lock (dev); - rc =to_cxl_drv (dev->driver)->probe (dev); +if (id ==CXL_DEVICE_REGION ) { +/* Regions cannot bind until parameters are set */ +struct cxl_region *cxlr =to_cxl_region (dev);
drivers/cxl/core/port.c static int cxl_bus_match(struct device *dev, struct device_driver *drv) · line 1447
Quoted 10 lines
drivers/cxl/core/port.c +7
-0+ +if (is_cxl_region_configured (cxlr)) + rc =to_cxl_drv (dev->driver)->probe (dev); + }else { + rc =to_cxl_drv (dev->driver)->probe (dev); + }cxl_nested_unlock (dev);dev_dbg (dev,"probe: %d\n" , rc); +
Shouldn't be in this patch.
Quoted 67 lines
drivers/cxl/core/port.c +0
-0return rc; }drivers/cxl/core/region.c +22
-3@@ -12,6 +12,8 @@#include <cxl.h> #include "core.h" +#include "core.h" +/** * DOC: cxl core region * @@ -26,10 +28,27 @@ static const struct attribute_group region_interleave_group;static bool is_region_active (struct cxl_region *cxlr) { -/* TODO: Regions can't be activated yet. */ -return false; +return cxlr->active; } +/* +* Most sanity checking is left up to region binding. This does the most basic +* check to determine whether or not the core should try probing the driver. +*/ +bool is_cxl_region_configured (const struct cxl_region *cxlr) +{ +/* zero sized regions aren't a thing. */ +if (cxlr->config.size <=0 ) +return false; + +/* all regions have at least 1 target */ +if (!cxlr->config.targets[0 ]) +return false; + +return true; +} +EXPORT_SYMBOL_GPL (is_cxl_region_configured); +static void remove_target (struct cxl_region *cxlr,int target) {struct cxl_memdev *cxlmd;@@ -316,7 +335,7 @@ static const struct attribute_group *region_groups[] = {static void cxl_region_release (struct device *dev); -static const struct device_type cxl_region_type = { +const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, .groups = region_groupsdrivers/cxl/cxl.h +4
-0@@ -81,6 +81,31 @@ static inline int cxl_to_interleave_ways(u8 eniw) } } +static inline bool cxl_is_interleave_ways_valid (int iw) +{ +switch (iw) { + case 0 ... 4:
drivers/cxl/cxl.h static inline int cxl_to_interleave_ways(u8 eniw) · line 97
Quoted 10 lines
drivers/cxl/cxl.h +10
-0+case 6 : +case 8 : +case 12 : +case 16 : +return true; +default : +return false; + } + +unreachable ();
Why does this need marking?
Quoted 26 lines
drivers/cxl/cxl.h +15
-0+} + +static inline bool cxl_is_interleave_granularity_valid (int ig) +{ +if (!is_power_of_2 (ig)) +return false; + +/* 16K is the max */ +return ((ig >>15 ) ==0 ); +} +/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ #define CXLDEV_CAP_ARRAY_OFFSET 0x0#define CXLDEV_CAP_ARRAY_CAP_ID 0@@ -199,6 +224,10 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,#define CXL_DECODER_F_ENABLE BIT(5)#define CXL_DECODER_F_MASK GENMASK(5, 0) +#define cxl_is_pmem_t3 (flags) \ + (((flags) & (CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM)) == \ + (CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM)) +enum cxl_decoder_type {CXL_DECODER_ACCELERATOR =2 ,CXL_DECODER_EXPANDER =3 ,
drivers/cxl/cxl.h struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port, · line 389
Quoted 4 lines
drivers/cxl/cxl.h +1
-0@@ -357,6 +386,7 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port, resource_size_t component_reg_phys);struct cxl_dport *cxl_find_dport_by_dev (struct cxl_port *port,const struct device *dev); +struct cxl_port *ep_find_cxl_port (struct cxl_memdev *cxlmd,unsigned int depth);
Doesn't seem to be in this patch or indeed anywhere in the series.
Quoted 37 lines
drivers/cxl/cxl.h +1
-0struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);@@ -404,6 +434,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);#define CXL_DEVICE_PORT 3#define CXL_DEVICE_ROOT 4#define CXL_DEVICE_MEMORY_EXPANDER 5 +#define CXL_DEVICE_REGION 6#define MODULE_ALIAS_CXL (type) MODULE_ALIAS("cxl:t" __stringify(type) "*")#define CXL_MODALIAS_FMT "cxl:t%d"drivers/cxl/region.c new file +20
-0@@ -0,0 +1,349 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include <linux/platform_device.h> +#include <linux/device.h> +#include <linux/module.h> +#include <linux/pci.h> +#include "cxlmem.h" +#include "region.h" +#include "cxl.h" + +/** +* DOC: cxl region +* +* This module implements a region driver that is capable of programming CXL +* hardware to setup regions. +* +* A CXL region encompasses a chunk of host physical address space that may be +* consumed by a single device (x1 interleave aka linear) or across multiple +* devices (xN interleaved). The region driver has the following +* responsibilities:
drivers/cxl/region.c line 30
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+* +* * Walk topology to obtain decoder resources for region configuration. +* * Program decoder resources based on region configuration. +* * Bridge CXL regions to LIBNVDIMM +* * Initiates reading and configuring LSA regions +* * Enumerates regions created by BIOS (typically volatile) +*/ + +#define region_ways (region) ((region)->config.interleave_ways) +#define region_granularity (region) ((region)->config.interleave_granularity)
If you drop the 'config' level of the structure as I think Dan suggested then I don't think these macros will be justified vs region->interleave_xxx They are a bit borderline now for whether they help or hinder readability!
Quoted 13 lines
drivers/cxl/region.c new file +13
-0+ +static struct cxl_decoder *rootd_from_region (struct cxl_region *cxlr) +{ +struct device *d = cxlr->dev.parent; + +if (WARN_ONCE (!is_root_decoder (d), +"Corrupt topology for root region\n" )) +return NULL ; + +return to_cxl_decoder (d); +} + +static struct cxl_port *get_hostbridge (const struct cxl_memdev *ep)
drivers/cxl/region.c line 53
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+{ +struct cxl_port *port = ep->port; + +while (!is_cxl_root (port)) { + port =to_cxl_port (port->dev.parent); +if (port->depth ==1 ) +return port; + } + +BUG ();
Add a comment on why. For anyone hitting this they are in a weird state, so give them a hint!
Quoted 9 lines
drivers/cxl/region.c new file +9
-0+return NULL ; +} + +static struct cxl_port *get_root_decoder (const struct cxl_memdev *endpoint) +{ +struct cxl_port *hostbridge =get_hostbridge (endpoint); + +if (hostbridge) +return to_cxl_port (hostbridge->dev.parent);
drivers/cxl/region.c line 72
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+ +return NULL ; +} + +/** +* sanitize_region() - Check is region is reasonably configured +* @cxlr: The region to check +* +* Determination as to whether or not a region can possibly be configured is +* described in CXL Memory Device SW Guide. In order to implement the algorithms
Lines are a tiny bit long given no loss of readability in bringing them under 80 chars.
Quoted 157 lines
drivers/cxl/region.c new file +157
-0+* described there, certain more basic configuration parameters must first need +* to be validated. That is accomplished by this function. +* +* Returns 0 if the region is reasonably configured, else returns a negative +* error code. +*/ +static int sanitize_region (const struct cxl_region *cxlr) +{ +const int ig =region_granularity (cxlr); +const int iw =region_ways (cxlr); +int i; + +if (dev_WARN_ONCE (&cxlr->dev, !is_cxl_region_configured (cxlr), +"unconfigured regions can't be probed (race?)\n" )) { +return -ENXIO ; + } + +/* +* Interleave attributes should be caught by later math, but it's +* easiest to find those issues here, now. +*/ +if (!cxl_is_interleave_ways_valid (iw)) { +dev_dbg (&cxlr->dev,"Invalid number of ways\n" ); +return -ENXIO ; + } + +if (!cxl_is_interleave_granularity_valid (ig)) { +dev_dbg (&cxlr->dev,"Invalid interleave granularity\n" ); +return -ENXIO ; + } + +if (cxlr->config.size % (SZ_256M * iw)) { +dev_dbg (&cxlr->dev,"Invalid size. Must be multiple of %uM\n" , +256 * iw); +return -ENXIO ; + } + +for (i =0 ; i < iw; i++) { +if (!cxlr->config.targets[i]) { +dev_dbg (&cxlr->dev,"Missing memory device target%u" , + i); +return -ENXIO ; + } +if (!cxlr->config.targets[i]->dev.driver) { +dev_dbg (&cxlr->dev,"%s isn't CXL.mem capable\n" , +dev_name (&cxlr->config.targets[i]->dev)); +return -ENODEV ; + } + } + +return 0 ; +} + +/** +* allocate_address_space() - Gets address space for the region. +* @cxlr: The region that will consume the address space +*/ +static int allocate_address_space (struct cxl_region *cxlr) +{ +/* TODO */ +return 0 ; +} + +/** +* find_cdat_dsmas() - Find a valid DSMAS for the region +* @cxlr: The region +*/ +static bool find_cdat_dsmas (const struct cxl_region *cxlr) +{ +return true; +} + +/** +* qtg_match() - Does this root decoder have desirable QTG for the endpoint +* @rootd: The root decoder for the region +* @endpoint: Endpoint whose QTG is being compared +* +* Prior to calling this function, the caller should verify that all endpoints +* in the region have the same QTG ID. +* +* Returns true if the QTG ID of the root decoder matches the endpoint +*/ +static bool qtg_match (const struct cxl_decoder *rootd, +const struct cxl_memdev *endpoint) +{ +/* TODO: */ +return true; +} + +/** +* region_xhb_config_valid() - determine cross host bridge validity +* @cxlr: The region being programmed +* @rootd: The root decoder to check against +* +* The algorithm is outlined in 2.13.14 "Verify XHB configuration sequence" of +* the CXL Memory Device SW Guide (Rev1p0). +* +* Returns true if the configuration is valid. +*/ +static bool region_xhb_config_valid (const struct cxl_region *cxlr, +const struct cxl_decoder *rootd) +{ +/* TODO: */ +return true; +} + +/** +* region_hb_rp_config_valid() - determine root port ordering is correct +* @cxlr: Region to validate +* @rootd: root decoder for this @cxlr +* +* The algorithm is outlined in 2.13.15 "Verify HB root port configuration +* sequence" of the CXL Memory Device SW Guide (Rev1p0). +* +* Returns true if the configuration is valid. +*/ +static bool region_hb_rp_config_valid (const struct cxl_region *cxlr, +const struct cxl_decoder *rootd) +{ +/* TODO: */ +return true; +} + +/** +* rootd_contains() - determine if this region can exist in the root decoder +* @rootd: root decoder that potentially decodes to this region +* @cxlr: region to be routed by the @rootd +*/ +static bool rootd_contains (const struct cxl_region *cxlr, +const struct cxl_decoder *rootd) +{ +/* TODO: */ +return true; +} + +static bool rootd_valid (const struct cxl_region *cxlr, +const struct cxl_decoder *rootd) +{ +const struct cxl_memdev *endpoint = cxlr->config.targets[0 ]; + +if (!qtg_match (rootd, endpoint)) +return false; + +if (!cxl_is_pmem_t3 (rootd->flags)) +return false; + +if (!region_xhb_config_valid (cxlr, rootd)) +return false; + +if (!region_hb_rp_config_valid (cxlr, rootd)) +return false; + +if (!rootd_contains (cxlr, rootd)) +return false; + +return true; +}
drivers/cxl/region.c line 239
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+ +struct rootd_context { +const struct cxl_region *cxlr; +struct cxl_port *hbs[CXL_DECODER_MAX_INTERLEAVE ]; +int count; +}; + +static int rootd_match (struct device *dev,void *data) +{ +struct rootd_context *ctx = (struct rootd_context *)data;
Shouldn't need to cast from a void * to another pointer type.
drivers/cxl/region.c line 245
Quoted 6 lines
drivers/cxl/region.c new file +6
-0+const struct cxl_region *cxlr = ctx->cxlr; + +if (!is_root_decoder (dev)) +return 0 ; + +return !!rootd_valid (cxlr,to_cxl_decoder (dev));
Extreme given rootd_valid returns a bool...
Quoted 6 lines
drivers/cxl/region.c new file +6
-0+} + +/* +* This is a roughly equivalent implementation to "Figure 45 - High-level +* sequence: Finding CFMWS for region" from the CXL Memory Device SW Guide +* Rev1p0.
drivers/cxl/region.c line 261
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+*/ +static struct cxl_decoder *find_rootd (const struct cxl_region *cxlr, +const struct cxl_port *root) +{ +struct rootd_context ctx; +struct device *ret; + + ctx.cxlr = cxlr; + + ret =device_find_child ((struct device *)&root->dev, &ctx, rootd_match);
Why do you need a cast?
Quoted 41 lines
drivers/cxl/region.c new file +39
-0+if (ret) +return to_cxl_decoder (ret); + +return NULL ; +} +static int cxl_region_probe (struct device *dev) +{ +struct cxl_region *cxlr =to_cxl_region (dev); +struct cxl_port *root_port; +struct cxl_decoder *rootd, *ours; +int ret; + +device_lock_assert (&cxlr->dev); + +if (cxlr->active) +return 0 ; + +if (uuid_is_null (&cxlr->config.uuid)) +uuid_gen (&cxlr->config.uuid); + +/* TODO: What about volatile, and LSA generated regions? */ + + ret =sanitize_region (cxlr); +if (ret) +return ret; + + ret =allocate_address_space (cxlr); +if (ret) +return ret; + +if (!find_cdat_dsmas (cxlr)) +return -ENXIO ; + + rootd =rootd_from_region (cxlr); +if (!rootd) { +dev_err (dev,"Couldn't find root decoder\n" ); +return -ENXIO ; + } +
drivers/cxl/region.c line 323
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+if (!rootd_valid (cxlr, rootd)) { +dev_err (dev,"Picked invalid rootd\n" ); +return -ENXIO ; + } + + root_port =get_root_decoder (cxlr->config.targets[0 ]); + ours =find_rootd (cxlr, root_port); +if (ours != rootd) +dev_dbg (dev,"Picked different rootd %s %s\n" , +dev_name (&rootd->dev),dev_name (&ours->dev));
Good to add a comment here on why it is fine to continue after what the dbg statement and check makes sound like a bad thing!
drivers/cxl/region.c line 334
Quoted 11 lines
drivers/cxl/region.c new file +11
-0+if (ours) +put_device (&ours->dev); + + ret =collect_ep_decoders (cxlr); +if (ret) +return ret; + + ret =bind_region (cxlr); +if (!ret) { + cxlr->active = true; +dev_info (dev,"Bound" );
Looks like left over debug info...
Quoted 15 lines
drivers/cxl/region.c new file +15
-0+ } + +return ret; +} + +static struct cxl_driver cxl_region_driver = { + .name ="cxl_region" , + .probe = cxl_region_probe, + .id =CXL_DEVICE_REGION , +}; +module_cxl_driver (cxl_region_driver); + +MODULE_LICENSE ("GPL v2" ); +MODULE_IMPORT_NS (CXL ); +MODULE_ALIAS_CXL (CXL_DEVICE_REGION );
Re: [PATCH v3 04/14] cxl/region: Introduce a cxl_region driver
In reply to Ben WidawskyQuoted 120 lines · Ben Widawsky
The
cxl_regiondriver is responsible for managing the HDM decoder programming in the CXL topology. Once a region is created it must be configured and bound to the driver in order to activate it.The following is a sample of how such controls might work:
region=$(cat
/sys/bus/cxl/devices/decoder0.0/create_region) echo $region >/sys/bus/cxl/devices/decoder0.0/create_regionecho 2 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/interleave echo $((256<<20)) > /sys/bus/cxl/devices/decoder0.0/region0.0:0/size echo mem0 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/target0 echo mem1 > /sys/bus/cxl/devices/decoder0.0/region0.0:0/target1 echo region0.0:0 >/sys/bus/cxl/drivers/cxl_region/bindIn order to handle the eventual rise in failure modes of binding a region, a new trace event is created to help track these failures for debug and reconfiguration paths in userspace.
Reported-by kernel test robot <[email protected]> (v2)Signed-off-by Ben Widawsky <[email protected]>---
Changes since v2:
- AddCONFIG_CXL_REGION
- Check ways/granularity in sanitize--- .../driver-api/cxl/memory-devices.rst | 3 + drivers/cxl/Kconfig | 4 + drivers/cxl/Makefile | 2 + drivers/cxl/core/core.h | 1 + drivers/cxl/core/port.c | 17 +- drivers/cxl/core/region.c | 25 +- drivers/cxl/cxl.h | 31 ++ drivers/cxl/region.c | 349 ++++++++++++++++++ drivers/cxl/region.h | 4 + 9 files changed, 431 insertions(+), 5 deletions(-) create mode 100644 drivers/cxl/region.cDocumentation/driver-api/cxl/memory-devices.rst +3
-0@@ -364,6 +364,9 @@ CXL Core CXL Regions ----------- +.. kernel-doc:: drivers/cxl/region.c + :doc: cxl region + .. kernel-doc:: drivers/cxl/region.h :identifiers:drivers/cxl/Kconfig +4
-0@@ -98,4 +98,8 @@ config CXL_PORT default CXL_BUS tristate +config CXL_REGION + default CXL_PORT + tristate + endifdrivers/cxl/Makefile +2
-0@@ -5,9 +5,11 @@ obj-$(CONFIG_CXL_MEM) += cxl_mem.o obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o obj-$(CONFIG_CXL_PMEM) += cxl_pmem.o obj-$(CONFIG_CXL_PORT) += cxl_port.o +obj-$(CONFIG_CXL_REGION) += cxl_region.o cxl_mem-y := mem.o cxl_pci-y := pci.o cxl_acpi-y := acpi.o cxl_pmem-y := pmem.o cxl_port-y := port.o +cxl_region-y := region.odrivers/cxl/core/core.h +1
-0@@ -7,6 +7,7 @@extern const struct device_type cxl_nvdimm_bridge_type;extern const struct device_type cxl_nvdimm_type;extern const struct device_type cxl_memdev_type; +extern const struct device_type cxl_region_type;extern struct attribute_group cxl_base_attribute_group;drivers/cxl/core/port.c +4
-1@@ -9,6 +9,7 @@#include <linux/idr.h> #include <cxlmem.h> #include <cxlpci.h> +#include <region.h> #include <cxl.h> #include "core.h" @@ -49,6 +50,8 @@ static int cxl_device_id(struct device *dev) }if (dev->type == &cxl_memdev_type)return CXL_DEVICE_MEMORY_EXPANDER ; +if (dev->type == &cxl_region_type) +return CXL_DEVICE_REGION ;return 0 ; }@@ -1425,13 +1428,23 @@ static int cxl_bus_match(struct device *dev, struct device_driver *drv)static int cxl_bus_probe (struct device *dev) { -int rc; +int id =cxl_device_id (dev);
drivers/cxl/core/port.c static int cxl_bus_match(struct device *dev, struct device_driver *drv) · line 1440
Quoted 10 lines
drivers/cxl/core/port.c +7
-1+int rc = -ENODEV ;cxl_nested_lock (dev); - rc =to_cxl_drv (dev->driver)->probe (dev); +if (id ==CXL_DEVICE_REGION ) { +/* Regions cannot bind until parameters are set */ +struct cxl_region *cxlr =to_cxl_region (dev); + +if (is_cxl_region_configured (cxlr)) + rc =to_cxl_drv (dev->driver)->probe (dev);
Setting aside whether is_cxl_region_configured() is sufficient, all probe failures of the region belong to cxl_region_probe(), no special casing in cxl_bus_probe() required. I would only expect special casing here if there were responsibilities that the core needed to manage relative to region probe, but I think all of that can be safely pushed out to leaf functions.
Quoted 24 lines
drivers/cxl/core/port.c +4
-0+ }else { + rc =to_cxl_drv (dev->driver)->probe (dev); + }cxl_nested_unlock (dev);dev_dbg (dev,"probe: %d\n" , rc); +return rc; }drivers/cxl/core/region.c +2
-0@@ -12,6 +12,8 @@#include <cxl.h> #include "core.h" +#include "core.h" +/** * DOC: cxl core region *
drivers/cxl/core/region.c static const struct attribute_group region_interleave_group; · line 31
Quoted 6 lines
drivers/cxl/core/region.c +1
-2@@ -26,10 +28,27 @@ static const struct attribute_group region_interleave_group;static bool is_region_active (struct cxl_region *cxlr) { -/* TODO: Regions can't be activated yet. */ -return false; +return cxlr->active;
I think we already talked about this being redundant with the state of cxlr->dev.driver...
Quoted 54 lines
drivers/cxl/core/region.c +19
-1} +/* +* Most sanity checking is left up to region binding. This does the most basic +* check to determine whether or not the core should try probing the driver. +*/ +bool is_cxl_region_configured (const struct cxl_region *cxlr) +{ +/* zero sized regions aren't a thing. */ +if (cxlr->config.size <=0 ) +return false; + +/* all regions have at least 1 target */ +if (!cxlr->config.targets[0 ]) +return false; + +return true; +} +EXPORT_SYMBOL_GPL (is_cxl_region_configured); +static void remove_target (struct cxl_region *cxlr,int target) {struct cxl_memdev *cxlmd;@@ -316,7 +335,7 @@ static const struct attribute_group *region_groups[] = {static void cxl_region_release (struct device *dev); -static const struct device_type cxl_region_type = { +const struct device_type cxl_region_type = { .name ="cxl_region" , .release = cxl_region_release, .groups = region_groupsdrivers/cxl/cxl.h +14
-0@@ -81,6 +81,31 @@ static inline int cxl_to_interleave_ways(u8 eniw) } } +static inline bool cxl_is_interleave_ways_valid (int iw) +{ +switch (iw) { + case 0 ... 4: +case 6 : +case 8 : +case 12 : +case 16 : +return true; +default : +return false; + } + +unreachable ();
drivers/cxl/cxl.h static inline int cxl_to_interleave_ways(u8 eniw) · line 107
Quoted 10 lines
drivers/cxl/cxl.h +10
-0+} + +static inline bool cxl_is_interleave_granularity_valid (int ig) +{ +if (!is_power_of_2 (ig)) +return false; + +/* 16K is the max */ +return ((ig >>15 ) ==0 ); +}
...already discussed that this validation needs to happen at input time.
drivers/cxl/cxl.h void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr, · line 229
Quoted 11 lines
drivers/cxl/cxl.h +4
-0+/* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ #define CXLDEV_CAP_ARRAY_OFFSET 0x0#define CXLDEV_CAP_ARRAY_CAP_ID 0@@ -199,6 +224,10 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,#define CXL_DECODER_F_ENABLE BIT(5)#define CXL_DECODER_F_MASK GENMASK(5, 0) +#define cxl_is_pmem_t3 (flags) \ + (((flags) & (CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM)) == \ + (CXL_DECODER_F_TYPE3 | CXL_DECODER_F_PMEM))
It's not clear that this macro is more readable than open coding the flag checking. For example all the decoders that this driver cares about will have CXL_DECODER_F_TYPE3, right?
Quoted 31 lines
drivers/cxl/cxl.h +3
-0+enum cxl_decoder_type {CXL_DECODER_ACCELERATOR =2 ,CXL_DECODER_EXPANDER =3 ,@@ -357,6 +386,7 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port, resource_size_t component_reg_phys);struct cxl_dport *cxl_find_dport_by_dev (struct cxl_port *port,const struct device *dev); +struct cxl_port *ep_find_cxl_port (struct cxl_memdev *cxlmd,unsigned int depth);struct cxl_decoder *to_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);@@ -404,6 +434,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);#define CXL_DEVICE_PORT 3#define CXL_DEVICE_ROOT 4#define CXL_DEVICE_MEMORY_EXPANDER 5 +#define CXL_DEVICE_REGION 6#define MODULE_ALIAS_CXL (type) MODULE_ALIAS("cxl:t" __stringify(type) "*")#define CXL_MODALIAS_FMT "cxl:t%d"drivers/cxl/region.c new file +5
-0@@ -0,0 +1,349 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* Copyright(c) 2021 Intel Corporation. All rights reserved. */ +#include <linux/platform_device.h> +#include <linux/device.h> +#include <linux/module.h>
drivers/cxl/region.c line 15
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+#include <linux/pci.h> +#include "cxlmem.h" +#include "region.h" +#include "cxl.h" + +/** +* DOC: cxl region +* +* This module implements a region driver that is capable of programming CXL +* hardware to setup regions.
This lead-in does not clarify anything, I'd just delete it to jump straight to this goodness in the next paragraph.
Quoted 5 lines
drivers/cxl/region.c new file +5
-0+* +* A CXL region encompasses a chunk of host physical address space that may be +* consumed by a single device (x1 interleave aka linear) or across multiple +* devices (xN interleaved). The region driver has the following +* responsibilities:
drivers/cxl/region.c line 30
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+* +* * Walk topology to obtain decoder resources for region configuration. +* * Program decoder resources based on region configuration. +* * Bridge CXL regions to LIBNVDIMM +* * Initiates reading and configuring LSA regions +* * Enumerates regions created by BIOS (typically volatile) +*/ + +#define region_ways (region) ((region)->config.interleave_ways) +#define region_granularity (region) ((region)->config.interleave_granularity)
This seems to lend credence to just dropping the "config." indirection and just open-coding region->interleave_{ways,granularity}.
I can read region->interleave_ways and not worry whereas region_ways(region) might have side effects.
drivers/cxl/region.c line 37
Quoted 7 lines
drivers/cxl/region.c new file +7
-0+ +static struct cxl_decoder *rootd_from_region (struct cxl_region *cxlr) +{ +struct device *d = cxlr->dev.parent; + +if (WARN_ONCE (!is_root_decoder (d), +"Corrupt topology for root region\n" ))
This can't happen. The WARN's in the other to_cxl_<object> helpers are due to missing type-safety of the argument. A cxl_region is an explicit type with an implicit relationship to its parent device.
drivers/cxl/region.c line 43
Quoted 6 lines
drivers/cxl/region.c new file +6
-0+return NULL ; + +return to_cxl_decoder (d); +} + +static struct cxl_port *get_hostbridge (const struct cxl_memdev *ep)
I'd prefer to call this walk_to_port_level() and take a @depth arg instead of get_hostbridge(). The walk_ instead of get_ prefix since this helper does not take a reference (get_device()) on the returned object, and _to_port_level instead of _hostbridge because this is CXL subsystem topology independent of the PCIe naming, and will need the depth support for switch support.
drivers/cxl/region.c line 53
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+{ +struct cxl_port *port = ep->port; + +while (!is_cxl_root (port)) { + port =to_cxl_port (port->dev.parent); +if (port->depth ==1 ) +return port; + } + +BUG ();
No need to crash, just have the caller handle NULL.
drivers/cxl/region.c line 57
Quoted 4 lines
drivers/cxl/region.c new file +4
-0+return NULL ; +} + +static struct cxl_port *get_root_decoder (const struct cxl_memdev *endpoint)
Why is a function named _decoder returning a 'struct cxl_port *'?
Quoted 20 lines
drivers/cxl/region.c new file +20
-0+{ +struct cxl_port *hostbridge =get_hostbridge (endpoint); + +if (hostbridge) +return to_cxl_port (hostbridge->dev.parent); + +return NULL ; +} + +/** +* sanitize_region() - Check is region is reasonably configured +* @cxlr: The region to check +* +* Determination as to whether or not a region can possibly be configured is +* described in CXL Memory Device SW Guide. In order to implement the algorithms +* described there, certain more basic configuration parameters must first need +* to be validated. That is accomplished by this function. +* +* Returns 0 if the region is reasonably configured, else returns a negative +* error code.
drivers/cxl/region.c line 87
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+*/ +static int sanitize_region (const struct cxl_region *cxlr) +{ +const int ig =region_granularity (cxlr); +const int iw =region_ways (cxlr); +int i; + +if (dev_WARN_ONCE (&cxlr->dev, !is_cxl_region_configured (cxlr), +"unconfigured regions can't be probed (race?)\n" )) { +return -ENXIO ;
Just let unconfigured regions be probed and then fail the probe.
Quoted 12 lines
drivers/cxl/region.c new file +12
-0+ } + +/* +* Interleave attributes should be caught by later math, but it's +* easiest to find those issues here, now. +*/ +if (!cxl_is_interleave_ways_valid (iw)) { +dev_dbg (&cxlr->dev,"Invalid number of ways\n" ); +return -ENXIO ; + } + +if (!cxl_is_interleave_granularity_valid (ig)) {
drivers/cxl/region.c line 109
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+dev_dbg (&cxlr->dev,"Invalid interleave granularity\n" ); +return -ENXIO ; + } + +if (cxlr->config.size % (SZ_256M * iw)) { +dev_dbg (&cxlr->dev,"Invalid size. Must be multiple of %uM\n" , +256 * iw); +return -ENXIO ; + } +
All of the above looks ok modulo also validating at input time.
drivers/cxl/region.c line 116
Quoted 7 lines
drivers/cxl/region.c new file +7
-0+for (i =0 ; i < iw; i++) { +if (!cxlr->config.targets[i]) { +dev_dbg (&cxlr->dev,"Missing memory device target%u" , + i); +return -ENXIO ; + } +if (!cxlr->config.targets[i]->dev.driver) {
The above is an argument for why the region should track endpoint decoders as targets not memdevs, because endpoint decoders only exist while memdevs are enabled. The endpoint decoder unregistration path can take care to trigger driver detachment of all associated regions first.
Quoted 7 lines
drivers/cxl/region.c new file +7
-0+dev_dbg (&cxlr->dev,"%s isn't CXL.mem capable\n" , +dev_name (&cxlr->config.targets[i]->dev)); +return -ENODEV ; + } + } + +return 0 ;
drivers/cxl/region.c line 133
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+} + +/** +* allocate_address_space() - Gets address space for the region. +* @cxlr: The region that will consume the address space +*/ +static int allocate_address_space (struct cxl_region *cxlr) +{ +/* TODO */ +return 0 ;
I'd prefer this function definition just move to the later patch where this gets implemented.
drivers/cxl/region.c line 140
Quoted 7 lines
drivers/cxl/region.c new file +7
-0+} + +/** +* find_cdat_dsmas() - Find a valid DSMAS for the region +* @cxlr: The region +*/ +static bool find_cdat_dsmas (const struct cxl_region *cxlr)
CDAT support is a long ways off, too early to include this. \
Quoted 8 lines
drivers/cxl/region.c new file +8
-0+{ +return true; +} + +/** +* qtg_match() - Does this root decoder have desirable QTG for the endpoint +* @rootd: The root decoder for the region +* @endpoint: Endpoint whose QTG is being compared
drivers/cxl/region.c line 158
Quoted 10 lines
drivers/cxl/region.c new file +10
-0+* +* Prior to calling this function, the caller should verify that all endpoints +* in the region have the same QTG ID. +* +* Returns true if the QTG ID of the root decoder matches the endpoint +*/ +static bool qtg_match (const struct cxl_decoder *rootd, +const struct cxl_memdev *endpoint) +{ +/* TODO: */
239 lines not shown · raw message
Part of this message is not shown here. View the raw message.
PATCH v3 5/14 cxl/acpi: Handle address space allocation
In reply to Ben WidawskyRegions are carved out of an addresses space which is claimed by top level decoders, and subsequently their children decoders. Regions are created with a size and therefore must fit, with proper alignment, in that address space. The support for doing this fitting is handled by the driver automatically.
As an example, a platform might configure a top level decoder to claim 1TB of address space @ 0x800000000 -> 0x10800000000; it would be possible to create M regions with appropriate alignment to occupy that address space. Each of those regions would have a host physical address somewhere in the range between 32G and 1.3TB, and the location will be determined by the logic added here.
The request_region() usage is not strictly mandatory at this point as the actual handling of the address space is done with genpools. It is highly likely however that the resource/region APIs will become useful in the not too distant future.
All decoders manage a host physical address space while active. Only the root decoder has constraints on location and size. As a result, it makes most sense for the root decoder to be responsible for managing the entire address space, and mid-level decoders and endpoints can ask the root decoder for suballocations.
--- drivers/cxl/acpi.c | 30 ++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 32 insertions(+)
drivers/cxl/acpi.c +30 -0
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ #include <linux/platform_device.h> +#include <linux/genalloc.h> #include <linux/module.h> #include <linux/device.h> #include <linux/kernel.h>
@@ -73,6 +74,27 @@ static int cxl_acpi_cfmws_verify(struct device *dev,return 0 ; } +/* +* Every decoder while active has an address space that it is decoding. However, +* only the root level decoders have fixed host physical address space ranges. +*/ +static int cxl_create_cfmws_address_space (struct cxl_decoder *cxld, +struct acpi_cedt_cfmws *cfmws) +{ +const int order =ilog2 (SZ_256M * cxld->interleave_ways); +struct device *dev = &cxld->dev; +struct gen_pool *pool; + + pool =devm_gen_pool_create (dev, order,NUMA_NO_NODE ,dev_name (dev)); +if (IS_ERR (pool)) +return PTR_ERR (pool); + + cxld->address_space = pool; + +return gen_pool_add (cxld->address_space, cfmws->base_hpa, + cfmws->window_size,NUMA_NO_NODE ); +} +struct cxl_cfmws_context {struct device *dev;struct cxl_port *root_port;
@@ -113,6 +135,14 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, cxld->interleave_ways =CFMWS_INTERLEAVE_WAYS (cfmws); cxld->interleave_granularity =CFMWS_INTERLEAVE_GRANULARITY (cfmws); + rc =cxl_create_cfmws_address_space (cxld, cfmws); +if (rc) { +dev_err (dev, +"Failed to create CFMWS address space for decoder\n" ); +put_device (&cxld->dev); +return 0 ; + } + rc =cxl_decoder_add (cxld, target_map);if (rc)put_device (&cxld->dev);
drivers/cxl/cxl.h +2 -0
@@ -251,6 +251,7 @@ enum cxl_decoder_type {* @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list * @region_ida: allocator for region ids. +* @address_space: Used/free address space for regions. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */
@@ -267,6 +268,7 @@ struct cxl_decoder {unsigned long flags;seqlock_t target_lock;struct ida region_ida; +struct gen_pool *address_space;int nr_targets;struct cxl_dport *target[]; };
Signature · 1 line
2.35.0
Re: [PATCH v3 05/14] cxl/acpi: Handle address space allocation
Quoted 3 lines · Ben Widawsky
Regions are carved out of an addresses space which is claimed by top level decoders, and subsequently their children decoders. Regions are
s/children/descendant/
Quoted 11 lines
created with a size and therefore must fit, with proper alignment, in that address space. The support for doing this fitting is handled by the driver automatically.
As an example, a platform might configure a top level decoder to claim 1TB of address space @
0x800000000->0x10800000000; it would be possible to create M regions with appropriate alignment to occupy that address space. Each of those regions would have a host physical address somewhere in the range between 32G and 1.3TB, and the location will be determined by the logic added here.
Quoted 4 lines
The
request_region()usage is not strictly mandatory at this point as the actual handling of the address space is done with genpools. It is highly likely however that the resource/region APIs will become useful in the not too distant future.
More on this below, but I think resource APIs are critical for the pre-existing / BIOS created region case and I have a feeling gen_pool is not a good fit.
Quoted 30 lines
All decoders manage a host physical address space while active. Only the root decoder has constraints on location and size. As a result, it makes most sense for the root decoder to be responsible for managing the entire address space, and mid-level decoders and endpoints can ask the root decoder for suballocations.
Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/acpi.c | 30 ++++++++++++++++++++++++++++++ drivers/cxl/cxl.h | 2 ++ 2 files changed, 32 insertions(+)drivers/cxl/acpi.c +3
-0@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ #include <linux/platform_device.h> +#include <linux/genalloc.h> #include <linux/module.h> #include <linux/device.h> #include <linux/kernel.h> @@ -73,6 +74,27 @@ static int cxl_acpi_cfmws_verify(struct device *dev,return 0 ; } +/* +* Every decoder while active has an address space that it is decoding. However,
drivers/cxl/acpi.c static int cxl_acpi_cfmws_verify(struct device *dev, · line 88
Quoted 10 lines
drivers/cxl/acpi.c +10
-0+* only the root level decoders have fixed host physical address space ranges. +*/ +static int cxl_create_cfmws_address_space (struct cxl_decoder *cxld, +struct acpi_cedt_cfmws *cfmws) +{ +const int order =ilog2 (SZ_256M * cxld->interleave_ways); +struct device *dev = &cxld->dev; +struct gen_pool *pool; + + pool =devm_gen_pool_create (dev, order,NUMA_NO_NODE ,dev_name (dev));
The cxld dev is not a suitable devm host.
Moreover, the address space is a generic property of root decoders, it belongs in the core not in cxl_acpi.
As for the data structure / APIs to manage the address space I'm not sure gen_pool is the right answer, because the capacity tracking will be done in terms of __request_region() and resource trees. The infrastructure to keep the gen_pool aligned with the resource tree drops away if there was an interface for allocating free space out of a resource tree to augment the base API of requesting space with known addresses. In fact, there is already the request_free_mem_region() helper. Did you consider that vs gen_pool? Otherwise, how to solve the problem of pre-populating the busy areas of the gen_pool relative to capacity that the BIOS may have consumed out of the decoder range? That comes for free with just walking decoders at boot and doing __request_region() against the root decoders. Then the allocation helper can just walk that free space similar to request_free_mem_region().
Quoted 50 lines
drivers/cxl/acpi.c +17
-0+if (IS_ERR (pool)) +return PTR_ERR (pool); + + cxld->address_space = pool; + +return gen_pool_add (cxld->address_space, cfmws->base_hpa, + cfmws->window_size,NUMA_NO_NODE ); +} +struct cxl_cfmws_context {struct device *dev;struct cxl_port *root_port;@@ -113,6 +135,14 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg, cxld->interleave_ways =CFMWS_INTERLEAVE_WAYS (cfmws); cxld->interleave_granularity =CFMWS_INTERLEAVE_GRANULARITY (cfmws); + rc =cxl_create_cfmws_address_space (cxld, cfmws); +if (rc) { +dev_err (dev, +"Failed to create CFMWS address space for decoder\n" ); +put_device (&cxld->dev); +return 0 ; + } + rc =cxl_decoder_add (cxld, target_map);if (rc)put_device (&cxld->dev);drivers/cxl/cxl.h +2
-1@@ -251,6 +251,7 @@ enum cxl_decoder_type {* @flags: memory type capabilities and locking * @target_lock: coordinate coherent reads of the target list * @region_ida: allocator for region ids. +* @address_space: Used/free address space for regions. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */ @@ -267,6 +268,7 @@ struct cxl_decoder {unsigned long flags;seqlock_t target_lock;struct ida region_ida; +struct gen_pool *address_space;int nr_targets;struct cxl_dport *target[]; }; --2.35.0
PATCH v3 6/14 cxl/region: Address space allocation
In reply to Ben WidawskyWhen a region is not assigned a host physical address, one is picked by the driver. As the address will determine which CFMWS contains the region, it's usually a better idea to let the driver make this determination.
--- drivers/cxl/region.c | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-)
drivers/cxl/region.c +38 -2
@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ #include <linux/platform_device.h> +#include <linux/genalloc.h> #include <linux/device.h> #include <linux/module.h> #include <linux/pci.h>
@@ -64,6 +65,20 @@ static struct cxl_port *get_root_decoder(const struct cxl_memdev *endpoint)return NULL ; } +static void release_cxl_region (void *r) +{ +struct cxl_region *cxlr = (struct cxl_region *)r; +struct cxl_decoder *rootd =rootd_from_region (cxlr); +struct resource *res = &rootd->platform_res; +resource_size_t start, size; + + start = cxlr->res->start; + size =resource_size (cxlr->res); + +__release_region (res, start, size); +gen_pool_free (rootd->address_space, start, size); +} +/** * sanitize_region() - Check is region is reasonably configured * @cxlr: The region to check
@@ -129,8 +144,29 @@ static int sanitize_region(const struct cxl_region *cxlr)*/ static int allocate_address_space (struct cxl_region *cxlr) { -/* TODO */ -return 0 ; +struct cxl_decoder *rootd =rootd_from_region (cxlr); +unsigned long start; + + start =gen_pool_alloc (rootd->address_space, cxlr->config.size); +if (!start) { +dev_dbg (&cxlr->dev,"Couldn't allocate %lluM of address space" , + cxlr->config.size >>20 ); +return -ENOMEM ; + } + + cxlr->res = +__request_region (&rootd->platform_res, start, cxlr->config.size, +dev_name (&cxlr->dev),IORESOURCE_MEM ); +if (!cxlr->res) { +dev_dbg (&cxlr->dev,"Couldn't obtain region from %s (%pR)\n" , +dev_name (&rootd->dev), &rootd->platform_res); +gen_pool_free (rootd->address_space, start, cxlr->config.size); +return -ENOMEM ; + } + +dev_dbg (&cxlr->dev,"resource %pR" , cxlr->res); + +return devm_add_action_or_reset (&cxlr->dev, release_cxl_region, cxlr); }/**
Signature · 1 line
2.35.0
Re: [PATCH v3 06/14] cxl/region: Address space allocation
Quoted 2 lines · Ben Widawsky
When a region is not assigned a host physical address,
Curious, is there a way to pick one in the current ABI? Not that I want one, in fact I think Linux should make it as difficult as possible to create a region with a fixed address (per the 'HPA' field of the region label) given all the problems it can cause with decoder allocation ordering. Unless and until someone identifies a solid use case for that capability it should be de-emphasized.
Quoted 28 lines
one is picked by
the driver. As the address will determine which CFMWS contains the region, it's usually a better idea to let the driver make this determination.Signed-off-by Ben Widawsky <[email protected]>--- drivers/cxl/region.c | 40 ++++++++++++++++++++++++++++++++++++++-- 1 file changed, 38 insertions(+), 2 deletions(-)drivers/cxl/region.c +3
-0@@ -1,6 +1,7 @@// SPDX-License-Identifier: GPL-2.0-only /* Copyright(c) 2021 Intel Corporation. All rights reserved. */ #include <linux/platform_device.h> +#include <linux/genalloc.h> #include <linux/device.h> #include <linux/module.h> #include <linux/pci.h> @@ -64,6 +65,20 @@ static struct cxl_port *get_root_decoder(const struct cxl_memdev *endpoint)return NULL ; } +static void release_cxl_region (void *r) +{
drivers/cxl/region.c static struct cxl_port *get_root_decoder(const struct cxl_memdev *endpoint) · line 79
Quoted 10 lines
drivers/cxl/region.c +10
-0+struct cxl_region *cxlr = (struct cxl_region *)r; +struct cxl_decoder *rootd =rootd_from_region (cxlr); +struct resource *res = &rootd->platform_res; +resource_size_t start, size; + + start = cxlr->res->start; + size =resource_size (cxlr->res); + +__release_region (res, start, size); +gen_pool_free (rootd->address_space, start, size);
If the need to keep the gen_pool in sync is dropped then this open-coded devm release handler can be replaced with __devm_request_region().
drivers/cxl/region.c static int sanitize_region(const struct cxl_region *cxlr) · line 132
Quoted 10 lines
drivers/cxl/region.c +2
-1+} +/** * sanitize_region() - Check is region is reasonably configured * @cxlr: The region to check @@ -129,8 +144,29 @@ static int sanitize_region(const struct cxl_region *cxlr)*/ static int allocate_address_space (struct cxl_region *cxlr) { -/* TODO */
The problem with TODOs is now I forget which context calls allocate_address_space(). If the caller was added in this patch it would be reviewable, as is, I need to go to another window to search "allocate_address_space" to recall that it is called from cxl_region_probe(). That's too late as someone defining a region should know upfront a region creation time that space has been reserved, or not.
drivers/cxl/region.c static int sanitize_region(const struct cxl_region *cxlr) · line 148
Quoted 3 lines
drivers/cxl/region.c +2
-1-return 0 ; +struct cxl_decoder *rootd =rootd_from_region (cxlr); +unsigned long start;
s/unsigned long/resource_size_t/?
Quoted 27 lines
drivers/cxl/region.c +21
-1+ + start =gen_pool_alloc (rootd->address_space, cxlr->config.size); +if (!start) { +dev_dbg (&cxlr->dev,"Couldn't allocate %lluM of address space" , + cxlr->config.size >>20 ); +return -ENOMEM ; + } + + cxlr->res = +__request_region (&rootd->platform_res, start, cxlr->config.size, +dev_name (&cxlr->dev),IORESOURCE_MEM ); +if (!cxlr->res) { +dev_dbg (&cxlr->dev,"Couldn't obtain region from %s (%pR)\n" , +dev_name (&rootd->dev), &rootd->platform_res); +gen_pool_free (rootd->address_space, start, cxlr->config.size); +return -ENOMEM ; + } + +dev_dbg (&cxlr->dev,"resource %pR" , cxlr->res); + +return devm_add_action_or_reset (&cxlr->dev, release_cxl_region, cxlr); }/** -- 2.35.0
PATCH v3 7/14 cxl/region: Implement XHB verification
In reply to Ben WidawskyCross host bridge verification primarily determines if the requested interleave ordering can be achieved by the root decoder, which isn't as programmable as other decoders.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.14
---
Changes since v2:
- Fail earlier on lack of host bridges. This should only be capable as of now with cxl_test memdevs.
--- .clang-format | 2 + drivers/cxl/cxl.h | 13 +++++++ drivers/cxl/region.c | 89 +++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 103 insertions(+), 1 deletion(-)
.clang-format +2 -0
@@ -169,6 +169,8 @@ ForEachMacros: - 'for_each_cpu_and' - 'for_each_cpu_not' - 'for_each_cpu_wrap' + - 'for_each_cxl_decoder_target' + - 'for_each_cxl_endpoint' - 'for_each_dapm_widgets' - 'for_each_dev_addr' - 'for_each_dev_scope'
drivers/cxl/cxl.h +13 -0
@@ -81,6 +81,19 @@ static inline int cxl_to_interleave_ways(u8 eniw) } } +static inline u8 cxl_to_eniw (u8 ways) +{ +if (is_power_of_2 (ways)) +return ilog2 (ways); + +return ways /3 +8 ; +} + +static inline u8 cxl_to_ig (u16 g) +{ +return ilog2 (g) -8 ; +} +static inline bool cxl_is_interleave_ways_valid (int iw) {switch (iw) {
drivers/cxl/region.c +88 -1
107 lines not on this page · raw message
Signature · 1 line
2.35.0
Re: [PATCH v3 07/14] cxl/region: Implement XHB verification
Quoted 4 lines · Ben Widawsky
Cross host bridge verification primarily determines if the requested interleave ordering can be achieved by the root decoder, which isn't as programmable as other decoders.
I don't understand that comment. Are you talking about the CFMWS static decoders that can not be programmed at all, or the 1st level decoders beneath that.
Quoted 2 lines
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.14
Just spell out the support here and don't require the reader to read that other doc to understand if it follows it exactly, or takes some liberties. I.e. the assumptions and tradeoffs built into the design choices in the patch need to be spelled out here.
Quoted 35 lines
Signed-off-by Ben Widawsky <[email protected]>---
Changes since v2:
- Fail earlier on lack of host bridges. This should only be capable as of now withcxl_testmemdevs.--- .clang-format | 2 + drivers/cxl/cxl.h | 13 +++++++ drivers/cxl/region.c | 89 +++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 103 insertions(+), 1 deletion(-).clang-format +2
-0@@ -169,6 +169,8 @@ ForEachMacros: - 'for_each_cpu_and' - 'for_each_cpu_not' - 'for_each_cpu_wrap' + - 'for_each_cxl_decoder_target' + - 'for_each_cxl_endpoint' - 'for_each_dapm_widgets' - 'for_each_dev_addr' - 'for_each_dev_scope'drivers/cxl/cxl.h +2
-0@@ -81,6 +81,19 @@ static inline int cxl_to_interleave_ways(u8 eniw) } } +static inline u8 cxl_to_eniw (u8 ways) +{
drivers/cxl/cxl.h static inline int cxl_to_interleave_ways(u8 eniw) · line 95
Quoted 10 lines
drivers/cxl/cxl.h +10
-0+if (is_power_of_2 (ways)) +return ilog2 (ways); + +return ways /3 +8 ; +} + +static inline u8 cxl_to_ig (u16 g) +{ +return ilog2 (g) -8 ; +}
These need better names to not be confused with the reverse helpers. How about interleave_ways_to_cxl_iw() and interleave_granularity_to_cxl_ig()?
Quoted 9 lines
drivers/cxl/cxl.h +1
-0+static inline bool cxl_is_interleave_ways_valid (int iw) {switch (iw) {drivers/cxl/region.c +0
-0
drivers/cxl/region.c line 33
Quoted 5 lines
drivers/cxl/region.c +2
-0@@ -29,6 +29,17 @@#define region_ways (region) ((region)->config.interleave_ways)#define region_granularity (region) ((region)->config.interleave_granularity) +#define region_eniw (region) (cxl_to_eniw(region_ways(region))) +#define region_ig (region) (cxl_to_ig(region_granularity(region)))
This feels like too much indirection...
drivers/cxl/region.c line 37
Quoted 4 lines
drivers/cxl/region.c +4
-0+ +#define for_each_cxl_endpoint (ep, region, idx) \ + for (idx = 0, ep = (region)->config.targets[idx]; \ + idx < region_ways(region); ep = (region)->config.targets[++idx])
Is the macro really buying anything in terms of readability?
for (i = 0; i < region->interleave_ways; i++)
...looks ok to me.
drivers/cxl/region.c line 42
Quoted 5 lines
drivers/cxl/region.c +5
-0+ +#define for_each_cxl_decoder_target (dport, decoder, idx) \ + for (idx = 0, dport = (decoder)->target[idx]; \ + idx < (decoder)->nr_targets - 1; \ + dport = (decoder)->target[++idx])
Doesn't this need locking to protect against target array updates? Another detail that might be less obfuscated with an open coded loop. I would only expect a new for_each() macro when the iterator is a function call, not a simple array de-reference with an incremented index.
Quoted 9 lines
drivers/cxl/region.c +2
-0static struct cxl_decoder *rootd_from_region (struct cxl_region *cxlr) {@@ -195,6 +206,30 @@ static bool qtg_match(const struct cxl_decoder *rootd,return true; } +static int get_unique_hostbridges (const struct cxl_region *cxlr, +struct cxl_port **hbs)
drivers/cxl/region.c static bool qtg_match(const struct cxl_decoder *rootd, · line 220
Quoted 10 lines
drivers/cxl/region.c +10
-0+{ +struct cxl_memdev *ep; +int i, hb_count =0 ; + +for_each_cxl_endpoint (ep, cxlr, i) { +struct cxl_port *hb =get_hostbridge (ep); +bool found = false; +int j; + +BUG_ON (!hb);
Doesn't seem like a reason to crash the kernel.
Quoted 16 lines
drivers/cxl/region.c +12
-0+ +for (j =0 ; j < hb_count; j++) { +if (hbs[j] == hb) + found = true; + } +if (!found) + hbs[hb_count++] = hb; + } + +return hb_count; +} +/** * region_xhb_config_valid() - determine cross host bridge validity * @cxlr: The region being programmed
drivers/cxl/region.c static bool qtg_match(const struct cxl_decoder *rootd, · line 250
Quoted 9 lines
drivers/cxl/region.c +5
-1@@ -208,7 +243,59 @@ static bool qtg_match(const struct cxl_decoder *rootd,static bool region_xhb_config_valid (const struct cxl_region *cxlr,const struct cxl_decoder *rootd) { -/* TODO: */ +const int rootd_eniw =cxl_to_eniw (rootd->interleave_ways); +const int rootd_ig =cxl_to_ig (rootd->interleave_granularity); +const int cxlr_ig =region_ig (cxlr); +const int cxlr_iw =region_ways (cxlr); +struct cxl_port *hbs[CXL_DECODER_MAX_INTERLEAVE ];
I'm worried about the stack usage of this. 0day has not complained yet, but is it necessary to collect everything into an array versus having a helper that does the lookup based on an index? It's not like cxl_region_probe() is a fast path.
drivers/cxl/region.c static bool qtg_match(const struct cxl_decoder *rootd, · line 255
Quoted 5 lines
drivers/cxl/region.c +5
-0+struct cxl_dport *target; +int i; + + i =get_unique_hostbridges (cxlr, hbs); +if (dev_WARN_ONCE (&cxlr->dev, i ==0 ,"Cannot find a valid host bridge\n" ))
Doesn't seem like a reason to crash the kernel. At least the topology basics like this can be validated at target assignment time.
drivers/cxl/region.c static bool qtg_match(const struct cxl_decoder *rootd, · line 260
Quoted 5 lines
drivers/cxl/region.c +5
-0+return false; + +/* Are all devices in this region on the same CXL host bridge */ +if (i ==1 ) +return true;
Doesn't this also need to check that the decoder is not interleaved across host bridges?
Quoted 6 lines
drivers/cxl/region.c +6
-0+ +/* CFMWS.HBIG >= Device.Label.IG */ +if (rootd_ig < cxlr_ig) { +dev_dbg (&cxlr->dev, +"%s HBIG must be greater than region IG (%d < %d)\n" , +dev_name (&rootd->dev), rootd_ig, cxlr_ig);
drivers/cxl/region.c static bool qtg_match(const struct cxl_decoder *rootd, · line 276
Quoted 10 lines
drivers/cxl/region.c +10
-0+return false; + } + +/* +* ((2^(CFMWS.HBIG - Device.RLabel.IG) * (2^CFMWS.ENIW)) > Device.RLabel.NLabel) +* +* XXX: 2^CFMWS.ENIW is trying to decode the NIW. Instead, use the look +* up function which supports non power of 2 interleave configurations. +*/ +if (((1 << (rootd_ig - cxlr_ig)) * (1 << rootd_eniw)) > cxlr_iw) {
Now here is where some helper macros could make things more readable. This looks like an acronym soup despite me being familiar with the CXL spec.
Quoted 28 lines
drivers/cxl/region.c +22
-1+dev_dbg (&cxlr->dev, +"granularity ratio requires a larger number of devices (%d) than currently configured (%d)\n" , + ((1 << (rootd_ig - cxlr_ig)) * (1 << rootd_eniw)), + cxlr_iw); +return false; + } + +/* +* CFMWS.InterleaveTargetList[n] must contain all devices, x where: +* (Device[x],RegionLabel.Position >> (CFMWS.HBIG - +* Device[x].RegionLabel.InterleaveGranularity)) & +* ((2^CFMWS.ENIW) - 1) = n +*/ +for_each_cxl_decoder_target (target, rootd, i) { +if (((i >> (rootd_ig - cxlr_ig))) & + (((1 << rootd_eniw) -1 ) != target->port_id)) { +dev_dbg (&cxlr->dev, +"One or more devices are not connected to the correct hostbridge.\n" ); +return false; + } + } +return true; } --2.35.0
PATCH v3 8/14 cxl/region: HB port config verification
In reply to Ben WidawskyHost bridge root port verification determines if the device ordering in an interleave set can be programmed through the host bridges and switches.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.15. The current version of the guide does not yet support x3 interleave configurations, and so that's not supported here either.
--- .clang-format | 1 + drivers/cxl/core/port.c | 1 + drivers/cxl/cxl.h | 2 + drivers/cxl/region.c | 127 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 130 insertions(+), 1 deletion(-)
.clang-format +1 -0
@@ -171,6 +171,7 @@ ForEachMacros: - 'for_each_cpu_wrap' - 'for_each_cxl_decoder_target' - 'for_each_cxl_endpoint' + - 'for_each_cxl_endpoint_hb' - 'for_each_dapm_widgets' - 'for_each_dev_addr' - 'for_each_dev_scope'
drivers/cxl/core/port.c +1 -0
@@ -706,6 +706,7 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port,return ERR_PTR (-ENOMEM );INIT_LIST_HEAD (&dport->list); +INIT_LIST_HEAD (&dport->verify_link); dport->dport = dport_dev; dport->port_id = port_id; dport->component_reg_phys = component_reg_phys;
drivers/cxl/cxl.h +2 -0
@@ -350,6 +350,7 @@ struct cxl_port {* @component_reg_phys: downstream port component registers * @port: reference to cxl_port that contains this downstream port * @list: node for a cxl_port's list of cxl_dport instances +* @verify_link: node used for hb root port verification */ struct cxl_dport {struct device *dport;
@@ -357,6 +358,7 @@ struct cxl_dport {resource_size_t component_reg_phys;struct cxl_port *port;struct list_head list; +struct list_head verify_link; };/**
drivers/cxl/region.c +126 -1
151 lines not on this page · raw message
Signature · 1 line
2.35.0
Re: [PATCH v3 08/14] cxl/region: HB port config verification
Quoted 17 lines · On Thu, 27 Jan 2022 16:27:01 -0800 Ben Widawsky <[email protected]> wrote:
Host bridge root port verification determines if the device ordering in an interleave set can be programmed through the host bridges and switches.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.15. The current version of the guide does not yet support x3 interleave configurations, and so that's not supported here either.
Signed-off-by Ben Widawsky <[email protected]>drivers/cxl/region.c +5
-0+static struct cxl_dport *get_rp (struct cxl_memdev *ep) +{ +struct cxl_port *port, *parent_port = port = ep->port; +struct cxl_dport *dport; +
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 323
Quoted 10 lines
drivers/cxl/region.c +10
-0+while (!is_cxl_root (port)) { + parent_port =to_cxl_port (port->dev.parent); +if (parent_port->depth ==1 ) +list_for_each_entry (dport, &parent_port->dports, list) +if (dport->dport == port->uport->parent->parent) +return dport; + port = parent_port; + } + +BUG ();
I know you mentioned you were reworking this patch set anyway, but I thought I'd give some quick debugging related feedback.
When running against a single switch in qemu (patches out once things are actually working), I hit this BUG() printing dev_name for the port->uport->parent->parent gives pci0000:0c but the matches are sort against 0000:0c:00.0 etc
So looks like one too many levels of parent in this case at least.
The other bug I haven't chased down yet is that if we happen to have downstream ports of the switch with duplicate ids (far too easy to do in QEMU as port_num is an optional parameter for switch DS ports) it's detected and the probe fails
- but then it tries again and we get an infinite loop of new ports being created and failing to probe...
I'll get back this one once I have it working with
a valid switch config.
Jonathan
Quoted 2 lines
(quoted diff) +2
-0+ return NULL; +}
Re: [PATCH v3 08/14] cxl/region: HB port config verification
Quoted 34 lines · Jonathan Cameron
Quoted 17 lines · On Thu, 27 Jan 2022 16:27:01 -0800 Ben Widawsky <[email protected]> wrote:
Host bridge root port verification determines if the device ordering in an interleave set can be programmed through the host bridges and switches.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.15. The current version of the guide does not yet support x3 interleave configurations, and so that's not supported here either.
Signed-off-by Ben Widawsky <[email protected]>drivers/cxl/region.c +5
-0+static struct cxl_dport *get_rp (struct cxl_memdev *ep) +{ +struct cxl_port *port, *parent_port = port = ep->port; +struct cxl_dport *dport; +drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 323
Quoted 10 lines
drivers/cxl/region.c +10
-0+while (!is_cxl_root (port)) { + parent_port =to_cxl_port (port->dev.parent); +if (parent_port->depth ==1 ) +list_for_each_entry (dport, &parent_port->dports, list) +if (dport->dport == port->uport->parent->parent) +return dport; + port = parent_port; + } + +BUG ();I know you mentioned you were reworking this patch set anyway, but I thought I'd give some quick debugging related feedback.
Quoted 7 lines
When running against a single switch in qemu (patches out once things are actually working), I hit this
BUG()printingdev_namefor theport->uport->parent->parentgives pci0000:0c but the matches are sort against 0000:0c:00.0 etcSo looks like one too many levels of parent in this case at least.
Hmm. This definitely looks dubious now that I see it again. Let me try to figure out how to rework it. I think it would be good to ask Dan as well. Much of the topology relationship works from bottom up, but top down is less easy. Previously I had used pci-isms to do this but Dan has been working on keeping the two domains isolated, which I agree is a good idea.
Quoted 7 lines
The other bug I haven't chased down yet is that if we happen to have downstream ports of the switch with duplicate ids (far too easy to do in QEMU as
port_numis an optional parameter for switch DS ports) it's detected and the probe fails
- but then it tries again and we get an infinite loop of new ports being created and failing to probe...
Is this allowed by spec? We shouldn't infinite loop, but I can't imagine the driver could do anything saner than fail to probe for such a case.
Quoted 2 lines
I'll get back this one once I have it working with a valid switch config.
Thanks.
Quoted 5 lines
Jonathan
Quoted 2 lines
(quoted diff) +2
-0+ return NULL; +}
Re: [PATCH v3 08/14] cxl/region: HB port config verification
Quoted 57 lines · On Mon, 14 Feb 2022 09:51:55 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 34 lines · Jonathan Cameron
Quoted 17 lines · On Thu, 27 Jan 2022 16:27:01 -0800 Ben Widawsky <[email protected]> wrote:
Host bridge root port verification determines if the device ordering in an interleave set can be programmed through the host bridges and switches.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.15. The current version of the guide does not yet support x3 interleave configurations, and so that's not supported here either.
Signed-off-by Ben Widawsky <[email protected]>drivers/cxl/region.c +5
-0+static struct cxl_dport *get_rp (struct cxl_memdev *ep) +{ +struct cxl_port *port, *parent_port = port = ep->port; +struct cxl_dport *dport; +drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 323
Quoted 10 lines
drivers/cxl/region.c +10
-0+while (!is_cxl_root (port)) { + parent_port =to_cxl_port (port->dev.parent); +if (parent_port->depth ==1 ) +list_for_each_entry (dport, &parent_port->dports, list) +if (dport->dport == port->uport->parent->parent) +return dport; + port = parent_port; + } + +BUG ();I know you mentioned you were reworking this patch set anyway, but I thought I'd give some quick debugging related feedback.
Quoted 7 lines
When running against a single switch in qemu (patches out once things are actually working), I hit this
BUG()printingdev_namefor theport->uport->parent->parentgives pci0000:0c but the matches are sort against 0000:0c:00.0 etcSo looks like one too many levels of parent in this case at least.
Hmm. This definitely looks dubious now that I see it again. Let me try to figure out how to rework it. I think it would be good to ask Dan as well. Much of the topology relationship works from bottom up, but top down is less easy. Previously I had used pci-isms to do this but Dan has been working on keeping the two domains isolated, which I agree is a good idea.
Quoted 7 lines
The other bug I haven't chased down yet is that if we happen to have downstream ports of the switch with duplicate ids (far too easy to do in QEMU as
port_numis an optional parameter for switch DS ports) it's detected and the probe fails
- but then it tries again and we get an infinite loop of new ports being created and failing to probe...
Quoted 2 lines
Is this allowed by spec? We shouldn't infinite loop, but I can't imagine the driver could do anything saner than fail to probe for such a case.
It would be a hardware bug, however I suspect any failure to probe will cause it rather that this specific case. I'll inject another failure when I get back to this properly.
Jonathan
Quoted 11 lines
Quoted 3 lines
I'll get back this one once I have it working with a valid switch config.
Thanks.
Quoted 5 lines
Jonathan
Quoted 2 lines
(quoted diff) +2
-0+ return NULL; +}
Re: [PATCH v3 08/14] cxl/region: HB port config verification
In reply to Ben WidawskyQuoted 173 lines · On Thu, 27 Jan 2022 16:27:01 -0800 Ben Widawsky <[email protected]> wrote:
Host bridge root port verification determines if the device ordering in an interleave set can be programmed through the host bridges and switches.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.15. The current version of the guide does not yet support x3 interleave configurations, and so that's not supported here either.
Signed-off-by Ben Widawsky <[email protected]>--- .clang-format | 1 + drivers/cxl/core/port.c | 1 + drivers/cxl/cxl.h | 2 + drivers/cxl/region.c | 127 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 130 insertions(+), 1 deletion(-).clang-format +1
-0@@ -171,6 +171,7 @@ ForEachMacros: - 'for_each_cpu_wrap' - 'for_each_cxl_decoder_target' - 'for_each_cxl_endpoint' + - 'for_each_cxl_endpoint_hb' - 'for_each_dapm_widgets' - 'for_each_dev_addr' - 'for_each_dev_scope'drivers/cxl/core/port.c +1
-0@@ -706,6 +706,7 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port,return ERR_PTR (-ENOMEM );INIT_LIST_HEAD (&dport->list); +INIT_LIST_HEAD (&dport->verify_link); dport->dport = dport_dev; dport->port_id = port_id; dport->component_reg_phys = component_reg_phys;drivers/cxl/cxl.h +2
-0@@ -350,6 +350,7 @@ struct cxl_port {* @component_reg_phys: downstream port component registers * @port: reference to cxl_port that contains this downstream port * @list: node for a cxl_port's list of cxl_dport instances +* @verify_link: node used for hb root port verification */ struct cxl_dport {struct device *dport;@@ -357,6 +358,7 @@ struct cxl_dport {resource_size_t component_reg_phys;struct cxl_port *port;struct list_head list; +struct list_head verify_link; };/** drivers/cxl/region.c +82
-1@@ -4,6 +4,7 @@#include <linux/genalloc.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sort.h> #include <linux/pci.h> #include "cxlmem.h" #include "region.h" @@ -36,6 +37,12 @@for (idx =0 , ep = (region)->config.targets[idx]; \ idx < region_ways(region); ep = (region)->config.targets[++idx]) +#define for_each_cxl_endpoint_hb (ep, region, hb, idx) \ + for (idx = 0, (ep) = (region)->config.targets[idx]; \ + idx < region_ways(region); \ + idx++, (ep) = (region)->config.targets[idx]) \ + if (get_hostbridge(ep) == (hb)) +#define for_each_cxl_decoder_target (dport, decoder, idx) \ for (idx = 0, dport = (decoder)->target[idx]; \ idx < (decoder)->nr_targets - 1; \@@ -299,6 +306,59 @@ static bool region_xhb_config_valid(const struct cxl_region *cxlr,return true; } +static struct cxl_dport *get_rp (struct cxl_memdev *ep) +{ +struct cxl_port *port, *parent_port = port = ep->port; +struct cxl_dport *dport; + +while (!is_cxl_root (port)) { + parent_port =to_cxl_port (port->dev.parent); +if (parent_port->depth ==1 ) +list_for_each_entry (dport, &parent_port->dports, list) +if (dport->dport == port->uport->parent->parent) +return dport; + port = parent_port; + } + +BUG (); +return NULL ; +} + +static int get_num_root_ports (const struct cxl_region *cxlr) +{ +struct cxl_memdev *endpoint; +struct cxl_dport *dport, *tmp; +int num_root_ports =0 ; +LIST_HEAD (root_ports); +int idx; + +for_each_cxl_endpoint (endpoint, cxlr, idx) { +struct cxl_dport *root_port =get_rp (endpoint); + +if (list_empty (&root_port->verify_link)) { +list_add_tail (&root_port->verify_link, &root_ports); + num_root_ports++; + } + } + +list_for_each_entry_safe (dport, tmp, &root_ports, verify_link) +list_del_init (&dport->verify_link); + +return num_root_ports; +} + +static bool has_switch (const struct cxl_region *cxlr) +{ +struct cxl_memdev *ep; +int i; + +for_each_cxl_endpoint (ep, cxlr, i) +if (ep->port->depth >2 ) +return true; + +return false; +} +/** * region_hb_rp_config_valid() - determine root port ordering is correct * @cxlr: Region to validate @@ -312,7 +372,72 @@ static bool region_xhb_config_valid(const struct cxl_region *cxlr,static bool region_hb_rp_config_valid (const struct cxl_region *cxlr,const struct cxl_decoder *rootd) { -/* TODO: */ +const int num_root_ports =get_num_root_ports (cxlr); +struct cxl_port *hbs[CXL_DECODER_MAX_INTERLEAVE ]; +int hb_count, i; + + hb_count =get_unique_hostbridges (cxlr, hbs); + +/* TODO: Switch support */ +if (has_switch (cxlr)) +return false; + +/* +* Are all devices in this region on the same CXL Host Bridge +* Root Port? +*/ +if (num_root_ports ==1 && !has_switch (cxlr)) +return true; + +for (i =0 ; i < hb_count; i++) { +int idx, position_mask; +struct cxl_dport *rp; +struct cxl_port *hb; +
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 406
Quoted 10 lines
drivers/cxl/region.c +10
-0+/* Get next CXL Host Bridge this region spans */ + hb = hbs[i]; + +/* +* Calculate the position mask: NumRootPorts = 2^PositionMask +* for this region. +* +* XXX: pos_mask is actually (1 << PositionMask) - 1 +*/ + position_mask = (1 << (ilog2 (num_root_ports))) -1 ;
Needs to account for the root ports potentially being spread over multiple host bridges. For now I'm assuming some symmetry to move my own testing forwards but that's not strictly required if we want to be really flexible.
So I've been using
position_mask = (1 << (ilog2(num_root_ports/hb_count)) - 1;
Quoted 37 lines
drivers/cxl/region.c +34
-0+ +/* +* Calculate the PortGrouping for each device on this CXL Host +* Bridge Root Port: +* PortGrouping = RegionLabel.Position & PositionMask +* +* The following nest iterators effectively iterate over each +* root port in the region. +* for_each_unique_rootport(rp, cxlr) +*/ +list_for_each_entry (rp, &hb->dports, list) { +struct cxl_memdev *ep; +int port_grouping =-1 ; + +for_each_cxl_endpoint_hb (ep, cxlr, hb, idx) { +if (get_rp (ep) != rp) +continue ; + +if (port_grouping ==-1 ) + port_grouping = idx & position_mask; + +/* +* Do all devices in the region connected to this CXL +* Host Bridge Root Port have the same PortGrouping? +*/ +if ((idx & position_mask) != port_grouping) { +dev_dbg (&cxlr->dev, +"One or more devices are not connected to the correct Host Bridge Root Port\n" ); +return false; + } + } + } + } +return true; }
Re: [PATCH v3 08/14] cxl/region: HB port config verification
In reply to Ben WidawskyQuoted 9 lines · Ben Widawsky
Host bridge root port verification determines if the device ordering in an interleave set can be programmed through the host bridges and switches.
The algorithm implemented here is based on the CXL Type 3 Memory Device Software Guide, chapter 2.13.15. The current version of the guide does not yet support x3 interleave configurations, and so that's not supported here either.
Given x3 is in a released ECN lets go ahead and include it because it may have a material effect on the design, but more importantly the ABI.
Quoted 46 lines
Signed-off-by Ben Widawsky <[email protected]>--- .clang-format | 1 + drivers/cxl/core/port.c | 1 + drivers/cxl/cxl.h | 2 + drivers/cxl/region.c | 127 +++++++++++++++++++++++++++++++++++++++- 4 files changed, 130 insertions(+), 1 deletion(-).clang-format +1
-0@@ -171,6 +171,7 @@ ForEachMacros: - 'for_each_cpu_wrap' - 'for_each_cxl_decoder_target' - 'for_each_cxl_endpoint' + - 'for_each_cxl_endpoint_hb' - 'for_each_dapm_widgets' - 'for_each_dev_addr' - 'for_each_dev_scope'drivers/cxl/core/port.c +1
-0@@ -706,6 +706,7 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port,return ERR_PTR (-ENOMEM );INIT_LIST_HEAD (&dport->list); +INIT_LIST_HEAD (&dport->verify_link); dport->dport = dport_dev; dport->port_id = port_id; dport->component_reg_phys = component_reg_phys;drivers/cxl/cxl.h +1
-0@@ -350,6 +350,7 @@ struct cxl_port {* @component_reg_phys: downstream port component registers * @port: reference to cxl_port that contains this downstream port * @list: node for a cxl_port's list of cxl_dport instances +* @verify_link: node used for hb root port verification */ struct cxl_dport {struct device *dport;
drivers/cxl/cxl.h struct cxl_dport { · line 361
Quoted 4 lines
drivers/cxl/cxl.h +1
-0@@ -357,6 +358,7 @@ struct cxl_dport {resource_size_t component_reg_phys;struct cxl_port *port;struct list_head list; +struct list_head verify_link;
Is this list also protected by the port device_lock?
Quoted 31 lines
drivers/cxl/cxl.h +0
-0};/** drivers/cxl/region.c +7
-0@@ -4,6 +4,7 @@#include <linux/genalloc.h> #include <linux/device.h> #include <linux/module.h> +#include <linux/sort.h> #include <linux/pci.h> #include "cxlmem.h" #include "region.h" @@ -36,6 +37,12 @@for (idx =0 , ep = (region)->config.targets[idx]; \ idx < region_ways(region); ep = (region)->config.targets[++idx]) +#define for_each_cxl_endpoint_hb (ep, region, hb, idx) \ + for (idx = 0, (ep) = (region)->config.targets[idx]; \ + idx < region_ways(region); \ + idx++, (ep) = (region)->config.targets[idx]) \ + if (get_hostbridge(ep) == (hb)) +#define for_each_cxl_decoder_target (dport, decoder, idx) \ for (idx = 0, dport = (decoder)->target[idx]; \ idx < (decoder)->nr_targets - 1; \@@ -299,6 +306,59 @@ static bool region_xhb_config_valid(const struct cxl_region *cxlr,return true; }
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 317
Quoted 10 lines
drivers/cxl/region.c +9
-0+static struct cxl_dport *get_rp (struct cxl_memdev *ep) +{ +struct cxl_port *port, *parent_port = port = ep->port; +struct cxl_dport *dport; + +while (!is_cxl_root (port)) { + parent_port =to_cxl_port (port->dev.parent); +if (parent_port->depth ==1 ) +list_for_each_entry (dport, &parent_port->dports, list)
Locking?
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 318
Quoted 1 line
drivers/cxl/region.c +1
-0+if (dport->dport == port->uport->parent->parent)
This assumes no switches.
Effectively it is identical to what devm_cxl_enumerate_ports(), which does support switches, is doing. To reduce maintenance burden it could follow the same pattern of:
for (iter = dev; iter; iter = grandparent(iter))
...
if (dev_is_cxl_root_child(&port->dev))
...
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 323
Quoted 5 lines
drivers/cxl/region.c +5
-0+return dport; + port = parent_port; + } + +BUG ();
more kernel crashing... why?
Quoted 6 lines
drivers/cxl/region.c +6
-0+return NULL ; +} + +static int get_num_root_ports (const struct cxl_region *cxlr) +{ +struct cxl_memdev *endpoint;
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 339
Quoted 10 lines
drivers/cxl/region.c +10
-0+struct cxl_dport *dport, *tmp; +int num_root_ports =0 ; +LIST_HEAD (root_ports); +int idx; + +for_each_cxl_endpoint (endpoint, cxlr, idx) { +struct cxl_dport *root_port =get_rp (endpoint); + +if (list_empty (&root_port->verify_link)) { +list_add_tail (&root_port->verify_link, &root_ports);
Doesn't this run into problems when there are multiple regions per root port?
Quoted 26 lines
drivers/cxl/region.c +22
-0+ num_root_ports++; + } + } + +list_for_each_entry_safe (dport, tmp, &root_ports, verify_link) +list_del_init (&dport->verify_link); + +return num_root_ports; +} + +static bool has_switch (const struct cxl_region *cxlr) +{ +struct cxl_memdev *ep; +int i; + +for_each_cxl_endpoint (ep, cxlr, i) +if (ep->port->depth >2 ) +return true; + +return false; +} +/** * region_hb_rp_config_valid() - determine root port ordering is correct * @cxlr: Region to validate
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 376
Quoted 6 lines
drivers/cxl/region.c +2
-1@@ -312,7 +372,72 @@ static bool region_xhb_config_valid(const struct cxl_region *cxlr,static bool region_hb_rp_config_valid (const struct cxl_region *cxlr,const struct cxl_decoder *rootd) { -/* TODO: */ +const int num_root_ports =get_num_root_ports (cxlr); +struct cxl_port *hbs[CXL_DECODER_MAX_INTERLEAVE ];
In terms of stack usage, doesn't the caller also have one of these on the stack when this is called?
Quoted 4 lines
drivers/cxl/region.c +4
-0+int hb_count, i; + + hb_count =get_unique_hostbridges (cxlr, hbs); +
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 390
Quoted 10 lines
drivers/cxl/region.c +10
-0+/* TODO: Switch support */ +if (has_switch (cxlr)) +return false; + +/* +* Are all devices in this region on the same CXL Host Bridge +* Root Port? +*/ +if (num_root_ports ==1 && !has_switch (cxlr)) +return true;
How can this happen without any intervening switch?
Quoted 6 lines
drivers/cxl/region.c +6
-0+ +for (i =0 ; i < hb_count; i++) { +int idx, position_mask; +struct cxl_dport *rp; +struct cxl_port *hb; +
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 406
Quoted 10 lines
drivers/cxl/region.c +10
-0+/* Get next CXL Host Bridge this region spans */ + hb = hbs[i]; + +/* +* Calculate the position mask: NumRootPorts = 2^PositionMask +* for this region. +* +* XXX: pos_mask is actually (1 << PositionMask) - 1 +*/ + position_mask = (1 << (ilog2 (num_root_ports))) -1 ;
Isn't "1 << ilog2(num_root_ports)" just "num_root_ports"?
drivers/cxl/region.c static bool region_xhb_config_valid(const struct cxl_region *cxlr, · line 411
Quoted 5 lines
drivers/cxl/region.c +5
-0+ +/* +* Calculate the PortGrouping for each device on this CXL Host +* Bridge Root Port: +* PortGrouping = RegionLabel.Position & PositionMask
Still confused what a port grouping is and what it means for the algorithm especially since RegionLabels are not relevant to this part of the algorithm. This assumes someone is familiar with "guide" terminology?
Quoted 35 lines
drivers/cxl/region.c +29
-1+* +* The following nest iterators effectively iterate over each +* root port in the region. +* for_each_unique_rootport(rp, cxlr) +*/ +list_for_each_entry (rp, &hb->dports, list) { +struct cxl_memdev *ep; +int port_grouping =-1 ; + +for_each_cxl_endpoint_hb (ep, cxlr, hb, idx) { +if (get_rp (ep) != rp) +continue ; + +if (port_grouping ==-1 ) + port_grouping = idx & position_mask; + +/* +* Do all devices in the region connected to this CXL +* Host Bridge Root Port have the same PortGrouping? +*/ +if ((idx & position_mask) != port_grouping) { +dev_dbg (&cxlr->dev, +"One or more devices are not connected to the correct Host Bridge Root Port\n" ); +return false; + } + } + } + } +return true; } --2.35.0
PATCH v3 9/14 cxl/region: Add infrastructure for decoder programming
In reply to Ben WidawskyThere are 3 steps in handling region programming once it has been configured by userspace.
1. Sanitize the parameters against the system.
2. Collect decoder resources from the topology
3. Program decoder resources
The infrastructure added here addresses #2. Two new APIs are introduced to allow collecting and returning decoder resources. Additionally the infrastructure includes two lists managed by the region driver, a staged list, and a commit list. The staged list contains those collected in step #2, and the commit list are all the decoders programmed in step #3.
--- drivers/cxl/core/port.c | 71 ++++++++++++++++++++++ drivers/cxl/core/region.c | 2 + drivers/cxl/cxl.h | 8 +++ drivers/cxl/cxlmem.h | 7 +++ drivers/cxl/port.c | 62 ++++++++++++++++++- drivers/cxl/region.c | 125 +++++++++++++++++++++++++++++++++----- drivers/cxl/region.h | 5 ++ 7 files changed, 263 insertions(+), 17 deletions(-)
drivers/cxl/core/port.c +71 -0
@@ -1212,6 +1212,8 @@ static struct cxl_decoder *cxl_decoder_alloc(struct cxl_port *port, cxld->target_type =CXL_DECODER_EXPANDER ; cxld->platform_res = (struct resource )DEFINE_RES_MEM (0 ,0 ); +INIT_LIST_HEAD (&cxld->region_link); +ida_init (&cxld->region_ida);return cxld;
@@ -1366,6 +1368,75 @@ int cxl_decoder_add(struct cxl_decoder *cxld, int *target_map) }EXPORT_SYMBOL_NS_GPL (cxl_decoder_add,CXL ); +/** +* cxl_get_decoder() - Get an unused decoder from the port. +* @port: The port to obtain a decoder from. +* +* Region programming requires obtaining decoder resources from all ports that +* participate in the interleave set. This function shall be used to pull the +* decoder resource out of the list of available. +* +* Context: Process context. Takes and releases the device lock of the port. +* +* Return: A cxl_decoder that can be used for programming if successful, else a +* negative error code. +*/ +struct cxl_decoder *cxl_get_decoder (struct cxl_port *port) +{ +struct cxl_hdm *cxlhdm; +int dec; + + cxlhdm =dev_get_drvdata (&port->dev); +if (dev_WARN_ONCE (&port->dev, !cxlhdm,"No port drvdata\n" )) +return ERR_PTR (-ENXIO ); + +device_lock (&port->dev); + dec =find_first_bit (cxlhdm->decoders.free_mask, + cxlhdm->decoders.count); +if (dec == cxlhdm->decoders.count) { +device_unlock (&port->dev); +return ERR_PTR (-ENODEV ); + } + +clear_bit (dec, cxlhdm->decoders.free_mask); +device_unlock (&port->dev); + +return cxlhdm->decoders.cxld[dec]; +} +EXPORT_SYMBOL_NS_GPL (cxl_get_decoder,CXL ); + +/** +* cxl_put_decoder() - Return an inactive decoder to the port. +* @cxld: The decoder being returned. +*/ +void cxl_put_decoder (struct cxl_decoder *cxld) +{ +struct cxl_port *port =to_cxl_port (cxld->dev.parent); +struct cxl_hdm *cxlhdm; +int i; + + cxlhdm =dev_get_drvdata (&port->dev); +if (dev_WARN_ONCE (&port->dev, !cxlhdm,"No port drvdata\n" )) +return ; + +device_lock (&port->dev); + +for (i =0 ; i <CXL_DECODER_MAX_INSTANCES ; i++) { +struct cxl_decoder *d = cxlhdm->decoders.cxld[i]; + +if (!d) +continue ; + +if (d == cxld) { +set_bit (i, cxlhdm->decoders.free_mask); +break ; + } + } + +device_unlock (&port->dev); +} +EXPORT_SYMBOL_NS_GPL (cxl_put_decoder,CXL ); +static void cxld_unregister (void *dev) {device_unregister (dev);
drivers/cxl/core/region.c +2 -0
@@ -440,6 +440,8 @@ struct cxl_region *cxl_alloc_region(struct cxl_decoder *cxld, int id)if (!cxlr)return ERR_PTR (-ENOMEM ); +INIT_LIST_HEAD (&cxlr->staged_list); +INIT_LIST_HEAD (&cxlr->commit_list); cxlr->id = id;return cxlr;
drivers/cxl/cxl.h +8 -0
@@ -35,6 +35,8 @@#define CXL_CM_CAP_CAP_ID_HDM 0x5#define CXL_CM_CAP_CAP_HDM_VERSION 1 +#define CXL_DECODER_MAX_INSTANCES 10 +/* HDM decoders CXL 2.0 8.2.5.12 CXL HDM Decoder Capability Structure */ #define CXL_HDM_DECODER_CAP_OFFSET 0x0#define CXL_HDM_DECODER_COUNT_MASK GENMASK(3, 0)
@@ -265,6 +267,7 @@ enum cxl_decoder_type {* @target_lock: coordinate coherent reads of the target list * @region_ida: allocator for region ids. * @address_space: Used/free address space for regions. +* @region_link: This decoder's place on either the staged, or commit list. * @nr_targets: number of elements in @target * @target: active ordered target list in current decoder configuration */
@@ -282,6 +285,7 @@ struct cxl_decoder {seqlock_t target_lock;struct ida region_ida;struct gen_pool *address_space; +struct list_head region_link;int nr_targets;struct cxl_dport *target[]; };
@@ -326,6 +330,7 @@ struct cxl_nvdimm {* @id: id for port device-name * @dports: cxl_dport instances referenced by decoders * @endpoints: cxl_ep instances, endpoints that are a descendant of this port +* @region_link: this port's node on the region's list of ports * @decoder_ida: allocator for decoder ids * @component_reg_phys: component register capability base address (optional) * @dead: last ep has been removed, force port re-creation
@@ -396,6 +401,8 @@ struct cxl_port *find_cxl_root(struct device *dev);int devm_cxl_enumerate_ports (struct cxl_memdev *cxlmd);int cxl_bus_rescan (void );struct cxl_port *cxl_mem_find_port (struct cxl_memdev *cxlmd); +struct cxl_decoder *cxl_get_decoder (struct cxl_port *port); +void cxl_put_decoder (struct cxl_decoder *cxld);bool schedule_cxl_memdev_detach (struct cxl_memdev *cxlmd); struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port,
@@ -406,6 +413,7 @@ struct cxl_dport *cxl_find_dport_by_dev(struct cxl_port *port,struct cxl_port *ep_find_cxl_port (struct cxl_memdev *cxlmd,unsigned int depth);struct cxl_decoder *to_cxl_decoder (struct device *dev); +bool is_cxl_decoder (struct device *dev);bool is_root_decoder (struct device *dev);bool is_cxl_decoder (struct device *dev); struct cxl_decoder *cxl_root_decoder_alloc(struct cxl_port *port,
drivers/cxl/cxlmem.h +7 -0
@@ -305,5 +305,12 @@ struct cxl_hdm {unsigned int target_count;unsigned int interleave_mask;struct cxl_port *port; + +struct port_decoders { +unsigned long *free_mask; +int count; + +struct cxl_decoder *cxld[CXL_DECODER_MAX_INSTANCES ]; + } decoders; };#endif /* __CXL_MEM_H__ */
drivers/cxl/port.c +61 -1
78 lines not on this page · raw message
drivers/cxl/region.c +109 -16
209 lines not on this page · raw message
drivers/cxl/region.h +5 -0
@@ -14,6 +14,9 @@* @list: Node in decoder's region list. * @res: Resource this region carves out of the platform decode range. * @active: If the region has been activated. +* @staged_list: All decoders staged for programming. +* @commit_list: All decoders programmed for this region's parameters. +* * @config: HDM decoder program config * @config.size: Size of the region determined from LSA or userspace. * @config.uuid: The UUID for this region.
@@ -27,6 +30,8 @@ struct cxl_region {struct list_head list;struct resource *res;bool active; +struct list_head staged_list; +struct list_head commit_list;struct {u64 size;
Signature · 1 line
2.35.0