Skip to contentSkip to conversation

CXL 2.0 Support

RFC PATCH Superseded Ben Widawsky 10 participants 70 messages Nov 11, 2020 – Dec 7, 2020

Subsystems (4)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover CXL 2.0 Support

Introduce support for “type-3” memory devices defined in the recently released Compute Express Link (CXL) 2.0 specification. Specifically, these are the memory devices defined by section 8.2.8.5 of the CXL 2.0 spec. A reference implementation emulating these devices is being submitted to the QEMU mailing list. “Type-3” is a CXL device that acts as a memory expander for RAM or PMEM. It might be interleaved with other CXL devices in a given physical address range.

These changes allow for foundational enumeration of CXL 2.0 memory devices. The functionality present is:
- Initial driver bring-up
- Device enumeration and an initial sysfs representation
- Submit a basic firmware command via ‘mailbox’ to an emulated memory device
with non-volatile capacity.

Some of the functionality that is still missing includes:
- Memory interleaving at the host bridge, root port, or switch level
- CXL 1.1 Root Complex Integrated Endpoint Support
- CXL 2.0 Hot plug support

In addition to the core functionality of discovering the spec defined registers and resources, introduce a CXL device model that will be the foundation for translating CXL capabilities into existing Linux infrastructure for Persistent Memory and other memory devices. For now, this only includes support for the management command mailbox that type-3 devices surface. These control devices fill the role of “DIMMs” / nmemX memory-devices in LIBNVDIMM terms.

Now, while implementing the driver some feedback for the specification was generated to cover perceived gaps and address conflicts. The feedback is presented as a reference implementation in the driver and QEMU emulation. Specifically the following concepts are original to the Linux implementation and feedback / collaboration is requested to develop these into specification proposals:
1. Top level ACPI object (ACPI0017)
2. HW imposed address space and interleave constraints
3. _OSC UUID A4D1629D-FF52-4888-BE96-E5CADE548DB1

ACPI0017
--------
Introduce a new ACPI namespace device with an _HID of ACPI0017. The purpose of this object is twofold, support a legacy OS with a set of out-of-tree CXL modules, and establish an attach point for a driver that knows about interleaving. Both of these boil down to the same point, to centralize Operating System support for resources described by the CXL Early Discovery Table (CEDT).

The legacy OS problem stems from the spec's description of a host bridge, ACPI0016 is denoted as the _HID for host bridges, with a _CID of PNP0A08. In a CXL unaware version of Linux, the core ACPI subsystem will bind a driver to PNP0A08 and preclude a CXL-aware driver from binding to ACPI0016. An ACPI0017 device allows a standalone CXL-aware driver to register for handling / coordinating CEDT and CXL-specific _OSC control.

Similarly when managing interleaving there needs to be some management layer above the ACPI0016 device that is capable of assembling leaf nodes into interleave sets. As is the case with ACPI0012 that does this central coordination for NFIT defined resources, ACPI0017 does the same for CEDT described resources.

Memory Windows
-------
For CXL.mem capable platforms, there is a need for a mechanism for platform firmware to make the Operating System aware of any restrictions that hardware might have in address space. For example, in a system with 4 host bridges all participating in an interleave set, the firmware needs to provide some description of this. That information is missing from the CXL 2.0 spec as of today and it also is not implemented in the driver. A variety of ACPI based mechanisms, for example _CRS fields on the ACPI0017 device, were considered.

CXL Exclusive _OSC
-----------------
CXL 2.0 definition provides new fields to _OSC for host bridges to allow for new services provided by CXL - error handling, hot plug, capabilities, etc. This is built on top of PCIe _OSC via a new UUID. A CXL unaware OS will use the old UUID to configure the PCIe host bridge. The expectation is that a CXL aware OS uses the new UUID and to modify both CXL and PCIE capabilities in one shot. The issue arises when trying to define a standalone CXL driver. The core OS will configure the PCIe _OSC, but when the CXL driver attempts to set CXL _OSC the current definition makes that driver re-specify PCIE capabilities. An isolated CXL-only _OSC allows the PCIE core to be unchanged and let a CXL driver stack manage CXL _OSC without the possibility of clobbering / colliding with PCIE core OSC management. The proposal moves the new _OSC dwords (SUPC and CTRC) to their own _OSC UUID.

Next steps after this basic foundation is expanded command support and LIBNVDIMM integration. This is the initial “release early / release often” version of the Linux CXL enabling.

Ben Widawsky (5):
cxl/mem: Map memory device registers
cxl/mem: Find device capabilities
cxl/mem: Initialize the mailbox interface
cxl/mem: Implement polled mode mailbox
MAINTAINERS: Add maintainers of the CXL driver

Dan Williams (2):
cxl/mem: Add a driver for the type-3 mailbox
cxl/mem: Register CXL memX devices

Vishal Verma (2):
cxl/acpi: Add an acpi_cxl module for the CXL interconnect cxl/acpi: add OSC support

 MAINTAINERS           |   9 +
 drivers/Kconfig       |   1 +
 drivers/Makefile      |   1 +
 drivers/cxl/Kconfig   |  50 ++++
 drivers/cxl/Makefile  |   9 +
 drivers/cxl/acpi.c    | 325 ++++++++++++++++++++++
 drivers/cxl/acpi.h    |  33 +++
 drivers/cxl/bus.c     |  35 +++
 drivers/cxl/bus.h     |   8 +
 drivers/cxl/cxl.h     | 166 +++++++++++
 drivers/cxl/mem.c     | 631 ++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h     |  21 ++
 include/acpi/actbl1.h |  52 ++++
 13 files changed, 1341 insertions(+)
 create mode 100644 drivers/cxl/Kconfig
 create mode 100644 drivers/cxl/Makefile
 create mode 100644 drivers/cxl/acpi.c
 create mode 100644 drivers/cxl/acpi.h
 create mode 100644 drivers/cxl/bus.c
 create mode 100644 drivers/cxl/bus.h
 create mode 100644 drivers/cxl/cxl.h
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
Signature · 1 line
2.29.2
#1 lore raw

RFC PATCH 1/9 cxl/acpi: Add an acpi_cxl module for the CXL interconnect

From: Vishal Verma <[email protected]>

Add an acpi_cxl module to coordinate the ACPI portions of the CXL (Compute eXpress Link) interconnect. This driver binds to ACPI0017 objects in the ACPI tree, and coordinates access to the resources provided by the ACPI CEDT (CXL Early Discovery Table).

It also coordinates operations of the root port _OSC object to notify platform firmware that the OS has native support for the CXL capabilities of endpoints.

Note: the actbl1.h changes are speculative. The expectation is that they will arrive through the ACPICA tree in due time.

---
 drivers/Kconfig       |   1 +
 drivers/Makefile      |   1 +
 drivers/cxl/Kconfig   |  30 +++++++++++
 drivers/cxl/Makefile  |   5 ++
 drivers/cxl/acpi.c    | 119 ++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/acpi.h    |  15 ++++++
 include/acpi/actbl1.h |  52 ++++++++++++++++++
 7 files changed, 223 insertions(+)
 create mode 100644 drivers/cxl/Kconfig
 create mode 100644 drivers/cxl/Makefile
 create mode 100644 drivers/cxl/acpi.c
 create mode 100644 drivers/cxl/acpi.h
drivers/Kconfig +1 -0
index dcecc9f6e33f..62c753a73651 100644
@@ -6,6 +6,7 @@ menu "Device Drivers"
 source "drivers/amba/Kconfig"
 source "drivers/eisa/Kconfig"
 source "drivers/pci/Kconfig"
+source "drivers/cxl/Kconfig"
 source "drivers/pcmcia/Kconfig"
 source "drivers/rapidio/Kconfig"
 
drivers/Makefile +1 -0
index c0cd1b9075e3..5dad349de73b 100644
@@ -73,6 +73,7 @@ obj-$(CONFIG_NVM)		+= lightnvm/
 obj-y				+= base/ block/ misc/ mfd/ nfc/
 obj-$(CONFIG_LIBNVDIMM)		+= nvdimm/
 obj-$(CONFIG_DAX)		+= dax/
+obj-$(CONFIG_CXL_BUS)		+= cxl/
 obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/
 obj-$(CONFIG_NUBUS)		+= nubus/
 obj-y				+= macintosh/
drivers/cxl/Kconfig new file +30 -0
new file mode 100644
index 000000000000..dd724bd364df
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig CXL_BUS
+	tristate "CXL (Compute Express Link) Devices Support"
+	help
+	  CXL is a bus that is electrically compatible with PCI-E, but layers
+	  three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The
+	  CXL.cache protocol allows devices to hold cachelines locally, the
+	  CXL.mem protocol allows devices to be fully coherent memory targets, the
+	  CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for
+	  the configuration and management of devices supporting these protocols.
+
+if CXL_BUS
+
+config CXL_BUS_PROVIDER
+	tristate
+
+config CXL_ACPI
+	tristate "CXL Platform Support"
+	depends on ACPI
+	default CXL_BUS
+	select CXL_BUS_PROVIDER
+	help
+	  CXL Platform Support is a prerequisite for any CXL device driver that
+	  wants to claim ownership of the component register space. By default
+	  platform firmware assumes Linux is unaware of CXL capabilities and
+	  requires explicit opt-in. This platform component also mediates
+	  resources described by the CEDT (CXL Early Discovery Table)
+
+	  Say 'y' to enable CXL (Compute Express Link) drivers.
+endif
drivers/cxl/Makefile new file +5 -0
new file mode 100644
index 000000000000..d38cd34a2582
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_acpi-y := acpi.o
drivers/cxl/acpi.c new file +119 -0

119 lines not on this page · raw message

drivers/cxl/acpi.h new file +15 -0
new file mode 100644
index 000000000000..011505475cc6
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+
+#ifndef __CXL_ACPI_H__
+#define __CXL_ACPI_H__
+#include <linux/acpi.h>
+
+struct acpi_cxl_desc {
+	struct acpi_table_header acpi_header;
+	struct device *dev;
+};
+
+int cxl_bus_prepared(struct pci_dev *pci_dev);
+
+#endif	/* __CXL_ACPI_H__ */
include/acpi/actbl1.h +52 -0

64 lines not on this page · raw message

Signature · 1 line
2.29.2
#2 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

Hi,

Quoted 26 lines · Ben Widawsky
---
 drivers/Kconfig       |   1 +
 drivers/Makefile      |   1 +
 drivers/cxl/Kconfig   |  30 +++++++++++
 drivers/cxl/Makefile  |   5 ++
 drivers/cxl/acpi.c    | 119 ++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/acpi.h    |  15 ++++++
 include/acpi/actbl1.h |  52 ++++++++++++++++++
 7 files changed, 223 insertions(+)
drivers/cxl/Kconfig new file +10 -0
new file mode 100644
index 000000000000..dd724bd364df
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig CXL_BUS
+	tristate "CXL (Compute Express Link) Devices Support"
+	help
+	  CXL is a bus that is electrically compatible with PCI-E, but layers
+	  three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The
+	  CXL.cache protocol allows devices to hold cachelines locally, the
+	  CXL.mem protocol allows devices to be fully coherent memory targets, the
+	  CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for
+	  the configuration and management of devices supporting these protocols.

drivers/cxl/Kconfig line 20

Quoted 10 lines
drivers/cxl/Kconfig new file +10 -0
+
+if CXL_BUS
+
+config CXL_BUS_PROVIDER
+	tristate
+
+config CXL_ACPI
+	tristate "CXL Platform Support"
+	depends on ACPI
+	default CXL_BUS

Please provide some justification for something other than the default default of 'n'. We try hard not to add drivers/modules that are not required for bootup.

drivers/cxl/Kconfig line 27

Quoted 7 lines
drivers/cxl/Kconfig new file +7 -0
+	select CXL_BUS_PROVIDER
+	help
+	  CXL Platform Support is a prerequisite for any CXL device driver that
+	  wants to claim ownership of the component register space. By default
+	  platform firmware assumes Linux is unaware of CXL capabilities and
+	  requires explicit opt-in. This platform component also mediates
+	  resources described by the CEDT (CXL Early Discovery Table)

end sentence with '.'

drivers/cxl/Kconfig line 29

Quoted 2 lines
drivers/cxl/Kconfig new file +2 -0
+
+	  Say 'y' to enable CXL (Compute Express Link) drivers.
	  or 'm'
Quoted 1 line
drivers/cxl/Kconfig new file +1 -0
+endif
Signature · 1 line
~Randy
#3 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

In reply to Ben Widawsky

drivers/cxl/Kconfig line 11

Quoted 10 lines · Ben Widawsky
drivers/cxl/Kconfig new file +10 -0
+menuconfig CXL_BUS
+	tristate "CXL (Compute Express Link) Devices Support"
+	help
+	  CXL is a bus that is electrically compatible with PCI-E, but layers
+	  three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The
+	  CXL.cache protocol allows devices to hold cachelines locally, the
+	  CXL.mem protocol allows devices to be fully coherent memory targets, the
+	  CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for
+	  the configuration and management of devices supporting these protocols.
+

Please fix the overly long lines.

Quoted 1 line

+static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)

Another overly long line.

drivers/cxl/acpi.c line 19

Quoted 4 lines
drivers/cxl/acpi.c new file +4 -0
+{
+	dev_set_drvdata(dev, acpi_desc);
+	acpi_desc->dev = dev;
+}

But this helper seems pretty pointless to start with.

drivers/cxl/acpi.c line 60

Quoted 4 lines
drivers/cxl/acpi.c new file +4 -0
+static int acpi_cxl_remove(struct acpi_device *adev)
+{
+	return 0;
+}

The emptry remove callback is not needed.

Quoted 18 lines
drivers/cxl/acpi.c new file +18 -0
+/*
+ * If/when CXL support is defined by other platform firmware the kernel
+ * will need a mechanism to select between the platform specific version
+ * of this routine, until then, hard-code ACPI assumptions
+ */
+int cxl_bus_prepared(struct pci_dev *pdev)
+{
+	struct acpi_device *adev;
+	struct pci_dev *root_port;
+	struct device *root;
+
+	root_port = pcie_find_root_port(pdev);
+	if (!root_port)
+		return -ENXIO;
+
+	root = root_port->dev.parent;
+	if (!root)
+		return -ENXIO;

drivers/cxl/acpi.c line 104

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+
+	adev = ACPI_COMPANION(root);
+	if (!adev)
+		return -ENXIO;
+
+	/* TODO: OSC enabling */
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_bus_prepared);

What is the point of this function? I doesn't realy do anything, not even a CXL specific check.

include/acpi/actbl1.h struct acpi_ibft_target { · line 1634

Quoted 8 lines
include/acpi/actbl1.h +7 -0
+/*******************************************************************************
+ *
+ * CEDT - CXL Early Discovery Table (ACPI 6.4)
+ *        Version 1
+ *
+ ******************************************************************************/
+

Pleae use the normal Linux comment style.

include/acpi/actbl1.h struct acpi_ibft_target { · line 1677

Quoted 7 lines
include/acpi/actbl1.h +7 -0
+#define ACPI_CEDT_CHBS_VERSION_CXL11    (0)
+#define ACPI_CEDT_CHBS_VERSION_CXL20    (1)
+
+/* Values for length field above */
+
+#define ACPI_CEDT_CHBS_LENGTH_CXL11     (0x2000)
+#define ACPI_CEDT_CHBS_LENGTH_CXL20     (0x10000)

No need for the braces.

maintainer
#4 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

Quoted 14 lines · Christoph Hellwig

drivers/cxl/Kconfig line 11

Quoted 10 lines · Ben Widawsky
drivers/cxl/Kconfig new file +10 -0
+menuconfig CXL_BUS
+	tristate "CXL (Compute Express Link) Devices Support"
+	help
+	  CXL is a bus that is electrically compatible with PCI-E, but layers
+	  three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The
+	  CXL.cache protocol allows devices to hold cachelines locally, the
+	  CXL.mem protocol allows devices to be fully coherent memory targets, the
+	  CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for
+	  the configuration and management of devices supporting these protocols.
+

Please fix the overly long lines.

Quoted 3 lines
Quoted 1 line

+static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)

Another overly long line.

Hi Christpph,

I thought 100 col. lines were acceptable now.

Quoted 8 lines

drivers/cxl/acpi.c line 19

Quoted 5 lines
drivers/cxl/acpi.c new file +4 -0
+{
+	dev_set_drvdata(dev, acpi_desc);
+	acpi_desc->dev = dev;
+}

But this helper seems pretty pointless to start with.

Quoted 6 lines

drivers/cxl/acpi.c line 60

Quoted 4 lines
drivers/cxl/acpi.c new file +4 -0
+static int acpi_cxl_remove(struct acpi_device *adev)
+{
+	return 0;
+}

The emptry remove callback is not needed.

Agreed on both of the above comments - these are just boilerplate for now, I expect they will get filled in in the next revision as more functionality gets fleshed out. If they are still empty/no-op by then I will remove them.

Quoted 24 lines
Quoted 24 lines
drivers/cxl/acpi.c new file +23 -0
+/*
+ * If/when CXL support is defined by other platform firmware the kernel
+ * will need a mechanism to select between the platform specific version
+ * of this routine, until then, hard-code ACPI assumptions
+ */
+int cxl_bus_prepared(struct pci_dev *pdev)
+{
+	struct acpi_device *adev;
+	struct pci_dev *root_port;
+	struct device *root;
+
+	root_port = pcie_find_root_port(pdev);
+	if (!root_port)
+		return -ENXIO;
+
+	root = root_port->dev.parent;
+	if (!root)
+		return -ENXIO;
+
+	adev = ACPI_COMPANION(root);
+	if (!adev)
+		return -ENXIO;
+
Quoted 8 lines

drivers/cxl/acpi.c line 104

Quoted 5 lines
drivers/cxl/acpi.c new file +5 -0
+	/* TODO: OSC enabling */
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_bus_prepared);

What is the point of this function? I doesn't realy do anything, not even a CXL specific check.

This gets a bit more fleshed out in patch 2. I kept that separate so that it is easier to review the bulk of the _OSC work in that patch without this driver boilerplate getting in the way.

Quoted 13 lines

include/acpi/actbl1.h struct acpi_ibft_target { · line 1634

Quoted 9 lines
include/acpi/actbl1.h +7 -0
+/*******************************************************************************
+ *
+ * CEDT - CXL Early Discovery Table (ACPI 6.4)
+ *        Version 1
+ *
+ ******************************************************************************/
+

Pleae use the normal Linux comment style.

Quoted 9 lines

include/acpi/actbl1.h struct acpi_ibft_target { · line 1677

Quoted 7 lines
include/acpi/actbl1.h +7 -0
+#define ACPI_CEDT_CHBS_VERSION_CXL11    (0)
+#define ACPI_CEDT_CHBS_VERSION_CXL20    (1)
+
+/* Values for length field above */
+
+#define ACPI_CEDT_CHBS_LENGTH_CXL11     (0x2000)
+#define ACPI_CEDT_CHBS_LENGTH_CXL20     (0x10000)

No need for the braces.

For both of these - see the note in the commit message. I just followed the ACPI header's style, and these hunks are only in this series to make it usable. I expect the 'actual' struct definitions, naming etc will come through ACPICA.

#5 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

Quoted 3 lines · Vishal L

Hi Christpph,

I thought 100 col. lines were acceptable now.

Quote from the coding style document:

"The preferred limit on the length of a single line is 80 columns.

Statements longer than 80 columns should be broken into sensible chunks, unless exceeding 80 columns significantly increases readability and does not hide information."

So yes, they are acceptable as an expception. Not for crap like this.

maintainer
#6 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

Quoted 7 lines · On Wed, 2020-11-11 at 07:34 +0000, [email protected] wrote:
Quoted 3 lines · Vishal L

Hi Christpph,

I thought 100 col. lines were acceptable now.

Quote from the coding style document:

Quoted 7 lines

"The preferred limit on the length of a single line is 80 columns.

Statements longer than 80 columns should be broken into sensible chunks, unless exceeding 80 columns significantly increases readability and does not hide information."

So yes, they are acceptable as an expception. Not for crap like this.

Ah fair enough, I'll reflow all of these for the next revision.

#7 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

In reply to Ben Widawsky

drivers/cxl/acpi.c line 31

Quoted 6 lines · Ben Widawsky
drivers/cxl/acpi.c new file +6 -0
+static int acpi_cxl_add(struct acpi_device *adev)
+{
+	struct acpi_cxl_desc *acpi_desc;
+	struct device *dev = &adev->dev;
+	struct acpi_table_header *tbl;
+	acpi_status status = AE_OK;

Pointless init.

drivers/cxl/acpi.c line 33

Quoted 2 lines
drivers/cxl/acpi.c new file +2 -0
+	acpi_size sz;
+	int rc = 0;

Pointless init.

Quoted 9 lines
drivers/cxl/acpi.c new file +9 -0
+	status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl);
+	if (ACPI_FAILURE(status)) {
+		dev_err(dev, "failed to find CEDT at startup\n");
+		return 0;
+	}
+
+	rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl);
+	if (rc)
+		return rc;
#8 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

In reply to Ben Widawsky
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Ben,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on pm/linux-next]
[also build test WARNING on linus/master linux/master v5.10-rc3 next-20201113] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Ben-Widawsky/CXL-2-0-Support/20201111-134547
base:   https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git linux-next
config: i386-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/a138b84a65bc7931b234fc14497d8cfe53d736f2
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Ben-Widawsky/CXL-2-0-Support/20201111-134547
        git checkout a138b84a65bc7931b234fc14497d8cfe53d736f2
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

In file included from include/linux/device.h:15,

                    from include/linux/acpi.h:15,
                    from drivers/cxl/acpi.c:10:
   drivers/cxl/acpi.c: In function 'acpi_cxl_add':
>> drivers/cxl/acpi.c:45:16: warning: format '%lld' expects argument of type 'long long int', but argument 3 has type 'acpi_size' {aka 'unsigned int'} [-Wformat=]
      45 |  dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
         |                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   include/linux/dev_printk.h:19:22: note: in definition of macro 'dev_fmt'
      19 | #define dev_fmt(fmt) fmt
         |                      ^~~
   drivers/cxl/acpi.c:45:2: note: in expansion of macro 'dev_info'
      45 |  dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
         |  ^~~~~~~~
   drivers/cxl/acpi.c:45:43: note: format string is defined here
      45 |  dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
         |                                        ~~~^
         |                                           |
         |                                           long long int
         |                                        %d

vim +45 drivers/cxl/acpi.c

    25	
    26	static int acpi_cxl_add(struct acpi_device *adev)
    27	{
    28		struct acpi_cxl_desc *acpi_desc;
    29		struct device *dev = &adev->dev;
    30		struct acpi_table_header *tbl;
    31		acpi_status status = AE_OK;
    32		acpi_size sz;
    33		int rc = 0;
    34	
    35		status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl);
    36		if (ACPI_FAILURE(status)) {
    37			dev_err(dev, "failed to find CEDT at startup\n");
    38			return 0;
    39		}
    40	
    41		rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl);
    42		if (rc)
    43			return rc;
    44		sz = tbl->length;
  > 45		dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
    46	
    47		acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL);
    48		if (!acpi_desc)
    49			return -ENOMEM;
    50		acpi_cxl_desc_init(acpi_desc, &adev->dev);
    51	
    52		acpi_desc->acpi_header = *tbl;
    53	
    54		return 0;
    55	}
    56	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#9 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

In reply to Ben Widawsky
Quoted 6 lines · On Tue, 10 Nov 2020 21:43:48 -0800 Ben Widawsky <[email protected]> wrote:

From: Vishal Verma <[email protected]>

Add an acpi_cxl module to coordinate the ACPI portions of the CXL (Compute eXpress Link) interconnect. This driver binds to ACPI0017 objects in the ACPI tree, and coordinates access to the resources provided by the ACPI CEDT (CXL Early Discovery Table).

I think the qemu series notes that this ACPI0017 is just a proposal at this stage. Please make sure that's highlighted here as well unless that status is out of date.

Quoted 110 lines

It also coordinates operations of the root port _OSC object to notify platform firmware that the OS has native support for the CXL capabilities of endpoints.

Note: the actbl1.h changes are speculative. The expectation is that they will arrive through the ACPICA tree in due time.

---
 drivers/Kconfig       |   1 +
 drivers/Makefile      |   1 +
 drivers/cxl/Kconfig   |  30 +++++++++++
 drivers/cxl/Makefile  |   5 ++
 drivers/cxl/acpi.c    | 119 ++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/acpi.h    |  15 ++++++
 include/acpi/actbl1.h |  52 ++++++++++++++++++
 7 files changed, 223 insertions(+)
 create mode 100644 drivers/cxl/Kconfig
 create mode 100644 drivers/cxl/Makefile
 create mode 100644 drivers/cxl/acpi.c
 create mode 100644 drivers/cxl/acpi.h
drivers/Kconfig +1 -0
index dcecc9f6e33f..62c753a73651 100644
@@ -6,6 +6,7 @@ menu "Device Drivers"
 source "drivers/amba/Kconfig"
 source "drivers/eisa/Kconfig"
 source "drivers/pci/Kconfig"
+source "drivers/cxl/Kconfig"
 source "drivers/pcmcia/Kconfig"
 source "drivers/rapidio/Kconfig"
 
drivers/Makefile +1 -0
index c0cd1b9075e3..5dad349de73b 100644
@@ -73,6 +73,7 @@ obj-$(CONFIG_NVM)		+= lightnvm/
 obj-y				+= base/ block/ misc/ mfd/ nfc/
 obj-$(CONFIG_LIBNVDIMM)		+= nvdimm/
 obj-$(CONFIG_DAX)		+= dax/
+obj-$(CONFIG_CXL_BUS)		+= cxl/
 obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/
 obj-$(CONFIG_NUBUS)		+= nubus/
 obj-y				+= macintosh/
drivers/cxl/Kconfig new file +30 -0
new file mode 100644
index 000000000000..dd724bd364df
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig CXL_BUS
+	tristate "CXL (Compute Express Link) Devices Support"
+	help
+	  CXL is a bus that is electrically compatible with PCI-E, but layers
+	  three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The
+	  CXL.cache protocol allows devices to hold cachelines locally, the
+	  CXL.mem protocol allows devices to be fully coherent memory targets, the
+	  CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for
+	  the configuration and management of devices supporting these protocols.
+
+if CXL_BUS
+
+config CXL_BUS_PROVIDER
+	tristate
+
+config CXL_ACPI
+	tristate "CXL Platform Support"
+	depends on ACPI
+	default CXL_BUS
+	select CXL_BUS_PROVIDER
+	help
+	  CXL Platform Support is a prerequisite for any CXL device driver that
+	  wants to claim ownership of the component register space. By default
+	  platform firmware assumes Linux is unaware of CXL capabilities and
+	  requires explicit opt-in. This platform component also mediates
+	  resources described by the CEDT (CXL Early Discovery Table)
+
+	  Say 'y' to enable CXL (Compute Express Link) drivers.
+endif
drivers/cxl/Makefile new file +5 -0
new file mode 100644
index 000000000000..d38cd34a2582
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_acpi-y := acpi.o
drivers/cxl/acpi.c new file +7 -0
new file mode 100644
index 000000000000..26e4f73838a7
@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright(c) 2020 Intel Corporation. All rights reserved.
+ */
+#include <linux/list_sort.h>
+#include <linux/module.h>
+#include <linux/mutex.h>

drivers/cxl/acpi.c line 17

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+#include <linux/sysfs.h>
+#include <linux/list.h>
+#include <linux/acpi.h>
+#include <linux/sort.h>
+#include <linux/pci.h>
+#include "acpi.h"
+
+static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)
+{
+	dev_set_drvdata(dev, acpi_desc);

No need to have this wrapper + it hides the fact you are not just initialsing the acpi_desc structure.

Quoted 4 lines
drivers/cxl/acpi.c new file +4 -0
+	acpi_desc->dev = dev;
+}
+
+static void acpi_cedt_put_table(void *table)

drivers/cxl/acpi.c line 31

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+{
+	acpi_put_table(table);
+}
+
+static int acpi_cxl_add(struct acpi_device *adev)
+{
+	struct acpi_cxl_desc *acpi_desc;
+	struct device *dev = &adev->dev;
+	struct acpi_table_header *tbl;
+	acpi_status status = AE_OK;

Set below, so don't do it here.

drivers/cxl/acpi.c line 33

Quoted 2 lines
drivers/cxl/acpi.c new file +2 -0
+	acpi_size sz;
+	int rc = 0;

Set in paths in which it's used so don't do it here.

drivers/cxl/acpi.c line 43

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+
+	status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl);
+	if (ACPI_FAILURE(status)) {
+		dev_err(dev, "failed to find CEDT at startup\n");
+		return 0;
+	}
+
+	rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl);
+	if (rc)
+		return rc;

blank line here preferred for readability (do something, then check errors as block)

drivers/cxl/acpi.c line 49

Quoted 6 lines
drivers/cxl/acpi.c new file +6 -0
+	sz = tbl->length;
+	dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
+
+	acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL);
+	if (!acpi_desc)
+		return -ENOMEM;

blank line here slightly helps readability.

drivers/cxl/acpi.c line 59

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+	acpi_cxl_desc_init(acpi_desc, &adev->dev);
+
+	acpi_desc->acpi_header = *tbl;
+
+	return 0;
+}
+
+static int acpi_cxl_remove(struct acpi_device *adev)
+{
+	return 0;

Don't think empty remove is needed.

Quoted 104 lines
drivers/cxl/acpi.c new file +60 -0
+}
+
+static const struct acpi_device_id acpi_cxl_ids[] = {
+	{ "ACPI0017", 0 },
+	{ "", 0 },
+};
+MODULE_DEVICE_TABLE(acpi, acpi_cxl_ids);
+
+static struct acpi_driver acpi_cxl_driver = {
+	.name = KBUILD_MODNAME,
+	.ids = acpi_cxl_ids,
+	.ops = {
+		.add = acpi_cxl_add,
+		.remove = acpi_cxl_remove,
+	},
+};
+
+/*
+ * If/when CXL support is defined by other platform firmware the kernel
+ * will need a mechanism to select between the platform specific version
+ * of this routine, until then, hard-code ACPI assumptions
+ */
+int cxl_bus_prepared(struct pci_dev *pdev)
+{
+	struct acpi_device *adev;
+	struct pci_dev *root_port;
+	struct device *root;
+
+	root_port = pcie_find_root_port(pdev);
+	if (!root_port)
+		return -ENXIO;
+
+	root = root_port->dev.parent;
+	if (!root)
+		return -ENXIO;
+
+	adev = ACPI_COMPANION(root);
+	if (!adev)
+		return -ENXIO;
+
+	/* TODO: OSC enabling */
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_bus_prepared);
+
+static __init int acpi_cxl_init(void)
+{
+	return acpi_bus_register_driver(&acpi_cxl_driver);
+}
+
+static __exit void acpi_cxl_exit(void)
+{
+	acpi_bus_unregister_driver(&acpi_cxl_driver);
+}
+
+module_init(acpi_cxl_init);
+module_exit(acpi_cxl_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
drivers/cxl/acpi.h new file +15 -0
new file mode 100644
index 000000000000..011505475cc6
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+
+#ifndef __CXL_ACPI_H__
+#define __CXL_ACPI_H__
+#include <linux/acpi.h>
+
+struct acpi_cxl_desc {
+	struct acpi_table_header acpi_header;
+	struct device *dev;
+};
+
+int cxl_bus_prepared(struct pci_dev *pci_dev);
+
+#endif	/* __CXL_ACPI_H__ */
include/acpi/actbl1.h +8 -0
index 43549547ed3e..70f745f526e3 100644
@@ -28,6 +28,7 @@
 #define ACPI_SIG_BERT           "BERT"	/* Boot Error Record Table */
 #define ACPI_SIG_BGRT           "BGRT"	/* Boot Graphics Resource Table */
 #define ACPI_SIG_BOOT           "BOOT"	/* Simple Boot Flag Table */
+#define ACPI_SIG_CEDT           "CEDT"	/* CXL Early Discovery Table */
 #define ACPI_SIG_CPEP           "CPEP"	/* Corrected Platform Error Polling table */
 #define ACPI_SIG_CSRT           "CSRT"	/* Core System Resource Table */
 #define ACPI_SIG_DBG2           "DBG2"	/* Debug Port table type 2 */
@@ -1624,6 +1625,57 @@ struct acpi_ibft_target {
 	u16 reverse_chap_secret_offset;
 };
 
+/*******************************************************************************
+ *
+ * CEDT - CXL Early Discovery Table (ACPI 6.4)
+ *        Version 1
+ *
+ ******************************************************************************/
+

include/acpi/actbl1.h struct acpi_ibft_target { · line 1644

Quoted 10 lines
include/acpi/actbl1.h +10 -0
+struct acpi_table_cedt {
+	struct acpi_table_header header;	/* Common ACPI table header */
+	u32 reserved;
+};
+
+/* Values for CEDT structure types */
+
+enum acpi_cedt_type {
+	ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */
+	ACPI_CEDT_TYPE_CFMWS = 1, 	/* CFMWS - CXL Fixed Memory Window Structure */

This isn't in the 2.0 spec, so I guess also part of some proposed changes.

Quoted 37 lines
include/acpi/actbl1.h +34 -0
+};
+
+struct acpi_cedt_structure {
+	u8 type;
+	u8 reserved;
+	u16 length;
+};
+
+/*
+ * CEDT Structures, correspond to Type in struct acpi_cedt_structure
+ */
+
+/* 0: CXL Host Bridge Structure */
+
+struct acpi_cedt_chbs {
+	struct acpi_cedt_structure header;
+	u32 uid;
+	u32 version;
+	u32 reserved1;
+	u64 base;
+	u32 length;
+	u32 reserved2;
+};
+
+/* Values for version field above */
+
+#define ACPI_CEDT_CHBS_VERSION_CXL11    (0)
+#define ACPI_CEDT_CHBS_VERSION_CXL20    (1)
+
+/* Values for length field above */
+
+#define ACPI_CEDT_CHBS_LENGTH_CXL11     (0x2000)
+#define ACPI_CEDT_CHBS_LENGTH_CXL20     (0x10000)
+
 /* Reset to default packing */
 
 #pragma pack()
maintainer
#10 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

Quoted 5 lines · Jonathan Cameron
Quoted 2 lines · On Tue, 10 Nov 2020 21:43:48 -0800 Ben Widawsky <[email protected]> wrote:

From: Vishal Verma <[email protected]>

Quoted 8 lines
Quoted 4 lines

Add an acpi_cxl module to coordinate the ACPI portions of the CXL (Compute eXpress Link) interconnect. This driver binds to ACPI0017 objects in the ACPI tree, and coordinates access to the resources provided by the ACPI CEDT (CXL Early Discovery Table).

I think the qemu series notes that this ACPI0017 is just a proposal at this stage. Please make sure that's highlighted here as well unless that status is out of date.

Hi Jonathan,

Thank you for the review. The cover letter talks about this, but I agree it would be worth repeating in this patch briefly as I did with the OSC patch. If it is still in a proposal state by the next posting, I'll make sure to add a note here too.

Quoted 54 lines

drivers/cxl/acpi.c line 17

Quoted 5 lines
drivers/cxl/acpi.c new file +4 -0
+
+static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)
+{
+	dev_set_drvdata(dev, acpi_desc);

No need to have this wrapper + it hides the fact you are not just initialsing the acpi_desc structure.

Quoted 4 lines
drivers/cxl/acpi.c new file +4 -0
+	acpi_desc->dev = dev;
+}
+
+static void acpi_cedt_put_table(void *table)

drivers/cxl/acpi.c line 31

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+{
+	acpi_put_table(table);
+}
+
+static int acpi_cxl_add(struct acpi_device *adev)
+{
+	struct acpi_cxl_desc *acpi_desc;
+	struct device *dev = &adev->dev;
+	struct acpi_table_header *tbl;
+	acpi_status status = AE_OK;

Set below, so don't do it here.

drivers/cxl/acpi.c line 33

Quoted 2 lines
drivers/cxl/acpi.c new file +2 -0
+	acpi_size sz;
+	int rc = 0;

Set in paths in which it's used so don't do it here.

drivers/cxl/acpi.c line 43

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+
+	status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl);
+	if (ACPI_FAILURE(status)) {
+		dev_err(dev, "failed to find CEDT at startup\n");
+		return 0;
+	}
+
+	rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl);
+	if (rc)
+		return rc;

blank line here preferred for readability (do something, then check errors as block)

drivers/cxl/acpi.c line 49

Quoted 6 lines
drivers/cxl/acpi.c new file +6 -0
+	sz = tbl->length;
+	dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
+
+	acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL);
+	if (!acpi_desc)
+		return -ENOMEM;

blank line here slightly helps readability.

Quoted 13 lines

drivers/cxl/acpi.c line 59

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+	acpi_cxl_desc_init(acpi_desc, &adev->dev);
+
+	acpi_desc->acpi_header = *tbl;
+
+	return 0;
+}
+
+static int acpi_cxl_remove(struct acpi_device *adev)
+{
+	return 0;

Don't think empty remove is needed.

Agreed with all of the above, I'll fix for the next posting.

Quoted 8 lines

include/acpi/actbl1.h struct acpi_ibft_target { · line 1644

Quoted 6 lines
include/acpi/actbl1.h +6 -0
+
+/* Values for CEDT structure types */
+
+enum acpi_cedt_type {
+	ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */
+	ACPI_CEDT_TYPE_CFMWS = 1, 	/* CFMWS - CXL Fixed Memory Window Structure */

This isn't in the 2.0 spec, so I guess also part of some proposed changes.

Yes this got in accidentally, will remove.

#11 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

In reply to Ben Widawsky
Quoted 168 lines · Ben Widawsky

From: Vishal Verma <[email protected]>

Add an acpi_cxl module to coordinate the ACPI portions of the CXL (Compute eXpress Link) interconnect. This driver binds to ACPI0017 objects in the ACPI tree, and coordinates access to the resources provided by the ACPI CEDT (CXL Early Discovery Table).

It also coordinates operations of the root port _OSC object to notify platform firmware that the OS has native support for the CXL capabilities of endpoints.

Note: the actbl1.h changes are speculative. The expectation is that they will arrive through the ACPICA tree in due time.

---
 drivers/Kconfig       |   1 +
 drivers/Makefile      |   1 +
 drivers/cxl/Kconfig   |  30 +++++++++++
 drivers/cxl/Makefile  |   5 ++
 drivers/cxl/acpi.c    | 119 ++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/acpi.h    |  15 ++++++
 include/acpi/actbl1.h |  52 ++++++++++++++++++
 7 files changed, 223 insertions(+)
 create mode 100644 drivers/cxl/Kconfig
 create mode 100644 drivers/cxl/Makefile
 create mode 100644 drivers/cxl/acpi.c
 create mode 100644 drivers/cxl/acpi.h
drivers/Kconfig +1 -0
index dcecc9f6e33f..62c753a73651 100644
@@ -6,6 +6,7 @@ menu "Device Drivers"
 source "drivers/amba/Kconfig"
 source "drivers/eisa/Kconfig"
 source "drivers/pci/Kconfig"
+source "drivers/cxl/Kconfig"
 source "drivers/pcmcia/Kconfig"
 source "drivers/rapidio/Kconfig"
 
drivers/Makefile +1 -0
index c0cd1b9075e3..5dad349de73b 100644
@@ -73,6 +73,7 @@ obj-$(CONFIG_NVM)             += lightnvm/
 obj-y                          += base/ block/ misc/ mfd/ nfc/
 obj-$(CONFIG_LIBNVDIMM)                += nvdimm/
 obj-$(CONFIG_DAX)              += dax/
+obj-$(CONFIG_CXL_BUS)          += cxl/
 obj-$(CONFIG_DMA_SHARED_BUFFER) += dma-buf/
 obj-$(CONFIG_NUBUS)            += nubus/
 obj-y                          += macintosh/
drivers/cxl/Kconfig new file +30 -0
new file mode 100644
index 000000000000..dd724bd364df
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig CXL_BUS
+       tristate "CXL (Compute Express Link) Devices Support"
+       help
+         CXL is a bus that is electrically compatible with PCI-E, but layers
+         three protocols on that signalling (CXL.io, CXL.cache, and CXL.mem). The
+         CXL.cache protocol allows devices to hold cachelines locally, the
+         CXL.mem protocol allows devices to be fully coherent memory targets, the
+         CXL.io protocol is equivalent to PCI-E. Say 'y' to enable support for
+         the configuration and management of devices supporting these protocols.
+
+if CXL_BUS
+
+config CXL_BUS_PROVIDER
+       tristate
+
+config CXL_ACPI
+       tristate "CXL Platform Support"
+       depends on ACPI
+       default CXL_BUS
+       select CXL_BUS_PROVIDER
+       help
+         CXL Platform Support is a prerequisite for any CXL device driver that
+         wants to claim ownership of the component register space. By default
+         platform firmware assumes Linux is unaware of CXL capabilities and
+         requires explicit opt-in. This platform component also mediates
+         resources described by the CEDT (CXL Early Discovery Table)
+
+         Say 'y' to enable CXL (Compute Express Link) drivers.
+endif
drivers/cxl/Makefile new file +5 -0
new file mode 100644
index 000000000000..d38cd34a2582
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+
+ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
+cxl_acpi-y := acpi.o
drivers/cxl/acpi.c new file +58 -0
new file mode 100644
index 000000000000..26e4f73838a7
@@ -0,0 +1,119 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright(c) 2020 Intel Corporation. All rights reserved.
+ */
+#include <linux/list_sort.h>
+#include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/sysfs.h>
+#include <linux/list.h>
+#include <linux/acpi.h>
+#include <linux/sort.h>
+#include <linux/pci.h>
+#include "acpi.h"
+
+static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)
+{
+       dev_set_drvdata(dev, acpi_desc);
+       acpi_desc->dev = dev;
+}
+
+static void acpi_cedt_put_table(void *table)
+{
+       acpi_put_table(table);
+}
+
+static int acpi_cxl_add(struct acpi_device *adev)
+{
+       struct acpi_cxl_desc *acpi_desc;
+       struct device *dev = &adev->dev;
+       struct acpi_table_header *tbl;
+       acpi_status status = AE_OK;
+       acpi_size sz;
+       int rc = 0;
+
+       status = acpi_get_table(ACPI_SIG_CEDT, 0, &tbl);
+       if (ACPI_FAILURE(status)) {
+               dev_err(dev, "failed to find CEDT at startup\n");
+               return 0;
+       }
+
+       rc = devm_add_action_or_reset(dev, acpi_cedt_put_table, tbl);
+       if (rc)
+               return rc;
+       sz = tbl->length;
+       dev_info(dev, "found CEDT at startup: %lld bytes\n", sz);
+
+       acpi_desc = devm_kzalloc(dev, sizeof(*acpi_desc), GFP_KERNEL);
+       if (!acpi_desc)
+               return -ENOMEM;
+       acpi_cxl_desc_init(acpi_desc, &adev->dev);
+
+       acpi_desc->acpi_header = *tbl;
+
+       return 0;
+}
+
+static int acpi_cxl_remove(struct acpi_device *adev)
+{

drivers/cxl/acpi.c line 68

Quoted 10 lines
drivers/cxl/acpi.c new file +10 -0
+       return 0;
+}
+
+static const struct acpi_device_id acpi_cxl_ids[] = {
+       { "ACPI0017", 0 },
+       { "", 0 },
+};
+MODULE_DEVICE_TABLE(acpi, acpi_cxl_ids);
+
+static struct acpi_driver acpi_cxl_driver = {

First of all, no new acpi_driver instances, pretty please!

acpi_default_enumeration() should create a platform device with the ACPI0017 ID for you. Can't you provide a driver for this one?

Quoted 145 lines
drivers/cxl/acpi.c new file +51 -0
+       .name = KBUILD_MODNAME,
+       .ids = acpi_cxl_ids,
+       .ops = {
+               .add = acpi_cxl_add,
+               .remove = acpi_cxl_remove,
+       },
+};
+
+/*
+ * If/when CXL support is defined by other platform firmware the kernel
+ * will need a mechanism to select between the platform specific version
+ * of this routine, until then, hard-code ACPI assumptions
+ */
+int cxl_bus_prepared(struct pci_dev *pdev)
+{
+       struct acpi_device *adev;
+       struct pci_dev *root_port;
+       struct device *root;
+
+       root_port = pcie_find_root_port(pdev);
+       if (!root_port)
+               return -ENXIO;
+
+       root = root_port->dev.parent;
+       if (!root)
+               return -ENXIO;
+
+       adev = ACPI_COMPANION(root);
+       if (!adev)
+               return -ENXIO;
+
+       /* TODO: OSC enabling */
+
+       return 0;
+}
+EXPORT_SYMBOL_GPL(cxl_bus_prepared);
+
+static __init int acpi_cxl_init(void)
+{
+       return acpi_bus_register_driver(&acpi_cxl_driver);
+}
+
+static __exit void acpi_cxl_exit(void)
+{
+       acpi_bus_unregister_driver(&acpi_cxl_driver);
+}
+
+module_init(acpi_cxl_init);
+module_exit(acpi_cxl_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
drivers/cxl/acpi.h new file +15 -0
new file mode 100644
index 000000000000..011505475cc6
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+
+#ifndef __CXL_ACPI_H__
+#define __CXL_ACPI_H__
+#include <linux/acpi.h>
+
+struct acpi_cxl_desc {
+       struct acpi_table_header acpi_header;
+       struct device *dev;
+};
+
+int cxl_bus_prepared(struct pci_dev *pci_dev);
+
+#endif /* __CXL_ACPI_H__ */
include/acpi/actbl1.h +52 -1
index 43549547ed3e..70f745f526e3 100644
@@ -28,6 +28,7 @@
 #define ACPI_SIG_BERT           "BERT" /* Boot Error Record Table */
 #define ACPI_SIG_BGRT           "BGRT" /* Boot Graphics Resource Table */
 #define ACPI_SIG_BOOT           "BOOT" /* Simple Boot Flag Table */
+#define ACPI_SIG_CEDT           "CEDT" /* CXL Early Discovery Table */
 #define ACPI_SIG_CPEP           "CPEP" /* Corrected Platform Error Polling table */
 #define ACPI_SIG_CSRT           "CSRT" /* Core System Resource Table */
 #define ACPI_SIG_DBG2           "DBG2" /* Debug Port table type 2 */
@@ -1624,6 +1625,57 @@ struct acpi_ibft_target {
        u16 reverse_chap_secret_offset;
 };
 
+/*******************************************************************************
+ *
+ * CEDT - CXL Early Discovery Table (ACPI 6.4)
+ *        Version 1
+ *
+ ******************************************************************************/
+
+struct acpi_table_cedt {
+       struct acpi_table_header header;        /* Common ACPI table header */
+       u32 reserved;
+};
+
+/* Values for CEDT structure types */
+
+enum acpi_cedt_type {
+       ACPI_CEDT_TYPE_HOST_BRIDGE = 0, /* CHBS - CXL Host Bridge Structure */
+       ACPI_CEDT_TYPE_CFMWS = 1,       /* CFMWS - CXL Fixed Memory Window Structure */
+};
+
+struct acpi_cedt_structure {
+       u8 type;
+       u8 reserved;
+       u16 length;
+};
+
+/*
+ * CEDT Structures, correspond to Type in struct acpi_cedt_structure
+ */
+
+/* 0: CXL Host Bridge Structure */
+
+struct acpi_cedt_chbs {
+       struct acpi_cedt_structure header;
+       u32 uid;
+       u32 version;
+       u32 reserved1;
+       u64 base;
+       u32 length;
+       u32 reserved2;
+};
+
+/* Values for version field above */
+
+#define ACPI_CEDT_CHBS_VERSION_CXL11    (0)
+#define ACPI_CEDT_CHBS_VERSION_CXL20    (1)
+
+/* Values for length field above */
+
+#define ACPI_CEDT_CHBS_LENGTH_CXL11     (0x2000)
+#define ACPI_CEDT_CHBS_LENGTH_CXL20     (0x10000)
+
 /* Reset to default packing */
 
 #pragma pack()
--

2.29.2

#12 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

On Tue, Nov 17, 2020 at 6:33 AM Rafael J. Wysocki <[email protected]> wrote:
[..]

Quoted 7 lines
Quoted 1 line

+static struct acpi_driver acpi_cxl_driver = {

First of all, no new acpi_driver instances, pretty please!

acpi_default_enumeration() should create a platform device with the ACPI0017 ID for you. Can't you provide a driver for this one?

Ah, yes, I recall we had this discussion around the time the ACPI0012 NFIT driver was developed. IIRC the reason ACPI0012 remaining an acpi_driver was due to a need to handle ACPI notifications, is that the deciding factor? ACPI0017 does not have any notifications so it seems like platform driver is the way to go.

#13 lore raw

Re: [RFC PATCH 1/9] cxl/acpi: Add an acpi_cxl module for the CXL interconnect

Quoted 7 lines · Dan Williams

On Tue, Nov 17, 2020 at 6:33 AM Rafael J. Wysocki <[email protected]> wrote:
[..]

Quoted 4 lines
Quoted 1 line

+static struct acpi_driver acpi_cxl_driver = {

First of all, no new acpi_driver instances, pretty please!

Quoted 8 lines
Quoted 3 lines

acpi_default_enumeration() should create a platform device with the ACPI0017 ID for you. Can't you provide a driver for this one?

Ah, yes, I recall we had this discussion around the time the ACPI0012 NFIT driver was developed. IIRC the reason ACPI0012 remaining an acpi_driver was due to a need to handle ACPI notifications, is that the deciding factor?

Sort of. In fact, a platform device driver can still handle ACPI notifications just fine, it just needs to install a notify handler for that.

The cases when an acpi_driver is really needed are basically when creating the platform device during the enumeration is not desirable, like in the PCI or PNP cases (because they both create device objects of a different type to represent the "physical" device).

It doesn't look like it is really needed for ACPI0012, but since it is there already, well ...

Quoted 1 line

ACPI0017 does not have any notifications so it seems like platform driver is the way to go.

Indeed.

#14 lore raw

RFC PATCH 2/9 cxl/acpi: add OSC support

In reply to Ben Widawsky

From: Vishal Verma <[email protected]>

Add support to advertise OS capabilities, and request OS control for CXL features using the ACPI _OSC mechanism. Advertise support for all possible CXL features, and attempt to request control too for all possible features.

Based on a patch by Sean Kelley.

---

This uses a non-standard UUID for CXL and is meant as a change proposal to the CXL specification. The definition for _OSC intermixes PCIe and CXL Dwords, which makes a clean separation of CXL capabilities difficult, if not impossible. This is therefore subject to change.

---
 drivers/cxl/acpi.c | 210 ++++++++++++++++++++++++++++++++++++++++++++-
 drivers/cxl/acpi.h |  18 ++++
 2 files changed, 226 insertions(+), 2 deletions(-)
drivers/cxl/acpi.c +208 -2
index 26e4f73838a7..c3e2f7f6ea02 100644
@@ -12,6 +12,16 @@
 #include <linux/pci.h>
 #include "acpi.h"
 
+/*
+ * TODO: These are global for now to save the OSC state.
+ * This works if OSC can be expected to be uniform for every ACPI0016 object.
+ * If that is not the case, revisit this. These need to be saved because
+ * relinquishing control of a previously granted capability is disallowed.
+ */
+static u32 cxl_osc_support_set;
+static u32 cxl_osc_control_set;
+static DEFINE_MUTEX(acpi_desc_lock);
+
 static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)
 {
 	dev_set_drvdata(dev, acpi_desc);
@@ -74,6 +84,199 @@ static struct acpi_driver acpi_cxl_driver = {
 	},
 };
 
+struct pci_osc_bit_struct {
+	u32 bit;
+	char *desc;
+};
+
+static struct pci_osc_bit_struct cxl_osc_support_bit[] = {
+	{ CXL_OSC_PORT_REG_ACCESS_SUPPORT, "CXLPortRegAccess" },
+	{ CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT, "CXLPortDevRegAccess" },
+	{ CXL_OSC_PER_SUPPORT, "CXLProtocolErrorReporting" },
+	{ CXL_OSC_NATIVE_HP_SUPPORT, "CXLNativeHotPlug" },
+};
+
+static struct pci_osc_bit_struct cxl_osc_control_bit[] = {
+	{ CXL_OSC_MEM_ERROR_CONTROL, "CXLMemErrorReporting" },
+};
+
+/*
+ * CXL 2.0 spec UUID - unusable as it PCI and CXL OSC are mixed
+ * static u8 cxl_osc_uuid_str[] = "68F2D50B-C469-4d8A-BD3D-941A103FD3FC";
+ */
+/* New, proposed UUID for CXL-only OSC */
+static u8 cxl_osc_uuid_str[] = "A4D1629D-FF52-4888-BE96-E5CADE548DB1";
+
+static void decode_osc_bits(struct device *dev, char *msg, u32 word,
+			    struct pci_osc_bit_struct *table, int size)
+{
+	char buf[80];
+	int i, len = 0;
+	struct pci_osc_bit_struct *entry;
+
+	buf[0] = '\0';
+	for (i = 0, entry = table; i < size; i++, entry++)
+		if (word & entry->bit)
+			len += scnprintf(buf + len, sizeof(buf) - len, "%s%s",
+					len ? " " : "", entry->desc);
+
+	dev_info(dev, "_OSC: %s [%s]\n", msg, buf);
+}
+
+static void decode_cxl_osc_support(struct device *dev, char *msg, u32 word)
+{
+	decode_osc_bits(dev, msg, word, cxl_osc_support_bit,
+			ARRAY_SIZE(cxl_osc_support_bit));
+}
+
+static void decode_cxl_osc_control(struct device *dev, char *msg, u32 word)
+{
+	decode_osc_bits(dev, msg, word, cxl_osc_control_bit,
+			ARRAY_SIZE(cxl_osc_control_bit));
+}
+
+static acpi_status acpi_cap_run_osc(acpi_handle handle,
+				    const u32 *capbuf, u8 *uuid_str, u32 *retval)
+{
+	struct acpi_osc_context context = {
+		.uuid_str = uuid_str,
+		.rev = 1,
+		.cap.length = 20,
+		.cap.pointer = (void *)capbuf,
+	};
+	acpi_status status;
+
+	status = acpi_run_osc(handle, &context);
+	if (ACPI_SUCCESS(status)) {
+		*retval = *((u32 *)(context.ret.pointer + 8));
+		kfree(context.ret.pointer);
+	}
+	return status;
+}
+
+static acpi_status cxl_query_osc(acpi_handle handle, u32 support, u32 *control)
+{
+	acpi_status status;
+	u32 result, capbuf[5];
+
+	support &= CXL_OSC_SUPPORT_VALID_MASK;
+	support |= cxl_osc_support_set;
+
+	capbuf[OSC_QUERY_DWORD] = OSC_QUERY_ENABLE;
+	capbuf[CXL_OSC_SUPPORT_DWORD] = support;
+	if (control) {
+		*control &= CXL_OSC_CONTROL_VALID_MASK;
+		capbuf[CXL_OSC_CONTROL_DWORD] = *control | cxl_osc_control_set;
+	} else {
+		/* Run _OSC query only with existing controls. */
+		capbuf[CXL_OSC_CONTROL_DWORD] = cxl_osc_control_set;
+	}
+
+	status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, &result);
+	if (ACPI_SUCCESS(status)) {
+		cxl_osc_support_set = support;
+		if (control)
+			*control = result;
+	}
+	return status;
+}
+
+static acpi_status cxl_osc_advertise_support(acpi_handle handle, u32 flags)
+{
+	acpi_status status;
+
+	mutex_lock(&acpi_desc_lock);
+	status = cxl_query_osc(handle, flags, NULL);
+	mutex_unlock(&acpi_desc_lock);
+	return status;
+}
+
+/**
+ * cxl_osc_request_control - Request control of CXL root _OSC features.
+ * @adev: ACPI device for the PCI root bridge (or PCIe Root Complex).
+ * @mask: Mask of _OSC bits to request control of, place to store control mask.
+ * @req: Mask of _OSC bits the control of is essential to the caller.
+ **/
+static acpi_status cxl_osc_request_control(struct acpi_device *adev, u32 *mask, u32 req)
+{
+	acpi_handle handle = adev->handle;
+	struct device *dev = &adev->dev;
+	acpi_status status = AE_OK;
+	u32 ctrl, capbuf[5];
+
+	if (!mask)
+		return AE_BAD_PARAMETER;
+
+	ctrl = *mask & CXL_OSC_MEM_ERROR_CONTROL;
+	if ((ctrl & req) != req)
+		return AE_TYPE;
+
+	mutex_lock(&acpi_desc_lock);
+
+	*mask = ctrl | cxl_osc_control_set;
+	/* No need to evaluate _OSC if the control was already granted. */
+	if ((cxl_osc_control_set & ctrl) == ctrl)
+		goto out;
+
+	/* Need to check the available controls bits before requesting them. */
+	while (*mask) {
+		status = cxl_query_osc(handle, cxl_osc_support_set, mask);
+		if (ACPI_FAILURE(status))
+			goto out;
+		if (ctrl == *mask)
+			break;
+		decode_cxl_osc_control(dev, "platform does not support", ctrl & ~(*mask));
+		ctrl = *mask;
+	}
+
+	if ((ctrl & req) != req) {
+		decode_cxl_osc_control(dev, "not requesting control; platform does not support",
+				       req & ~(ctrl));
+		status = AE_SUPPORT;
+		goto out;
+	}
+
+	capbuf[OSC_QUERY_DWORD] = 0;
+	capbuf[CXL_OSC_SUPPORT_DWORD] = cxl_osc_support_set;
+	capbuf[CXL_OSC_CONTROL_DWORD] = ctrl;
+	status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, mask);
+	if (ACPI_SUCCESS(status))
+		cxl_osc_control_set = *mask;
+out:
+	mutex_unlock(&acpi_desc_lock);
+	return status;
+}
+
+static int cxl_negotiate_osc(struct acpi_device *adev)
+{
+	u32 cxl_support, cxl_control, requested;
+	acpi_handle handle = adev->handle;
+	struct device *dev = &adev->dev;
+	acpi_status status;
+
+	/* Declare support for everything */
+	cxl_support = CXL_OSC_SUPPORT_VALID_MASK;
+	decode_cxl_osc_support(dev, "OS supports", cxl_support);
+	status = cxl_osc_advertise_support(handle, cxl_support);
+	if (ACPI_FAILURE(status)) {
+		dev_info(dev, "CXL_OSC failed (%s)\n", acpi_format_exception(status));
+		return -ENXIO;
+	}
+
+	/* Request control for everything */
+	cxl_control = CXL_OSC_CONTROL_VALID_MASK;
+	requested = cxl_control;
+	status = cxl_osc_request_control(adev, &cxl_control, CXL_OSC_MEM_ERROR_CONTROL);
+	if (ACPI_SUCCESS(status)) {
+		decode_cxl_osc_control(dev, "OS now controls", cxl_control);
+	} else {
+		decode_cxl_osc_control(dev, "OS requested", requested);
+		decode_cxl_osc_control(dev, "platform willing to grant", cxl_control);
+		dev_info(dev, "_OSC failed (%s)\n", acpi_format_exception(status));
+	}
+	return 0;
+}
+
 /*
  * If/when CXL support is defined by other platform firmware the kernel
  * will need a mechanism to select between the platform specific version
@@ -84,6 +287,7 @@ int cxl_bus_prepared(struct pci_dev *pdev)
 	struct acpi_device *adev;
 	struct pci_dev *root_port;
 	struct device *root;
+	int rc;
 
 	root_port = pcie_find_root_port(pdev);
 	if (!root_port)
@@ -97,9 +301,11 @@ int cxl_bus_prepared(struct pci_dev *pdev)
 	if (!adev)
 		return -ENXIO;
 
-	/* TODO: OSC enabling */
+	rc = cxl_negotiate_osc(adev);
+	if (rc)
+		dev_err(&pdev->dev, "Failed to negotiate OSC\n");
 
-	return 0;
+	return rc;
 }
 EXPORT_SYMBOL_GPL(cxl_bus_prepared);
 
drivers/cxl/acpi.h +18 -0
index 011505475cc6..bf8d078e1b2a 100644
@@ -10,6 +10,24 @@ struct acpi_cxl_desc {
 	struct device *dev;
 };
 
+/* Indexes into _OSC Capabilities Buffer */
+#define CXL_OSC_SUPPORT_DWORD			1	/* DWORD 2 */
+#define CXL_OSC_CONTROL_DWORD			2	/* DWORD 3 */
+
+/* CXL Host Bridge _OSC: Capabilities DWORD 2: Support Field */
+#define CXL_OSC_PORT_REG_ACCESS_SUPPORT		0x00000001
+#define CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT	0x00000002
+#define CXL_OSC_PER_SUPPORT			0x00000004
+#define CXL_OSC_NATIVE_HP_SUPPORT		0x00000008
+#define CXL_OSC_SUPPORT_VALID_MASK		(CXL_OSC_PORT_REG_ACCESS_SUPPORT |	\
+						 CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT |	\
+						 CXL_OSC_PER_SUPPORT |			\
+						 CXL_OSC_NATIVE_HP_SUPPORT)
+
+/* CXL Host Bridge _OSC: Capabilities DWORD 3: Control Field */
+#define CXL_OSC_MEM_ERROR_CONTROL		0x00000001
+#define CXL_OSC_CONTROL_VALID_MASK		(CXL_OSC_MEM_ERROR_CONTROL)
+
 int cxl_bus_prepared(struct pci_dev *pci_dev);
 
 #endif	/* __CXL_ACPI_H__ */
Signature · 1 line
2.29.2
#15 lore raw

Re: [RFC PATCH 2/9] cxl/acpi: add OSC support

Quoted 11 lines · On Tue, 10 Nov 2020 21:43:49 -0800 Ben Widawsky <[email protected]> wrote:

From: Vishal Verma <[email protected]>

Add support to advertise OS capabilities, and request OS control for CXL features using the ACPI _OSC mechanism. Advertise support for all possible CXL features, and attempt to request control too for all possible features.

Based on a patch by Sean Kelley.

I guess unsurprisingly a lot of this is cut and paste from PCIe so can we share some of the code?

Thanks,

Jonathan

Quoted 65 lines

---

This uses a non-standard UUID for CXL and is meant as a change proposal to the CXL specification. The definition for _OSC intermixes PCIe and CXL Dwords, which makes a clean separation of CXL capabilities difficult, if not impossible. This is therefore subject to change.

---
 drivers/cxl/acpi.c | 210 ++++++++++++++++++++++++++++++++++++++++++++-
 drivers/cxl/acpi.h |  18 ++++
 2 files changed, 226 insertions(+), 2 deletions(-)
drivers/cxl/acpi.c +37 -0
index 26e4f73838a7..c3e2f7f6ea02 100644
@@ -12,6 +12,16 @@
 #include <linux/pci.h>
 #include "acpi.h"
 
+/*
+ * TODO: These are global for now to save the OSC state.
+ * This works if OSC can be expected to be uniform for every ACPI0016 object.
+ * If that is not the case, revisit this. These need to be saved because
+ * relinquishing control of a previously granted capability is disallowed.
+ */
+static u32 cxl_osc_support_set;
+static u32 cxl_osc_control_set;
+static DEFINE_MUTEX(acpi_desc_lock);
+
 static void acpi_cxl_desc_init(struct acpi_cxl_desc *acpi_desc, struct device *dev)
 {
 	dev_set_drvdata(dev, acpi_desc);
@@ -74,6 +84,199 @@ static struct acpi_driver acpi_cxl_driver = {
 	},
 };
 
+struct pci_osc_bit_struct {
+	u32 bit;
+	char *desc;
+};
+
+static struct pci_osc_bit_struct cxl_osc_support_bit[] = {
+	{ CXL_OSC_PORT_REG_ACCESS_SUPPORT, "CXLPortRegAccess" },
+	{ CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT, "CXLPortDevRegAccess" },
+	{ CXL_OSC_PER_SUPPORT, "CXLProtocolErrorReporting" },
+	{ CXL_OSC_NATIVE_HP_SUPPORT, "CXLNativeHotPlug" },
+};
+
+static struct pci_osc_bit_struct cxl_osc_control_bit[] = {
+	{ CXL_OSC_MEM_ERROR_CONTROL, "CXLMemErrorReporting" },
+};
+
+/*
+ * CXL 2.0 spec UUID - unusable as it PCI and CXL OSC are mixed
+ * static u8 cxl_osc_uuid_str[] = "68F2D50B-C469-4d8A-BD3D-941A103FD3FC";
+ */
+/* New, proposed UUID for CXL-only OSC */
+static u8 cxl_osc_uuid_str[] = "A4D1629D-FF52-4888-BE96-E5CADE548DB1";
+
+static void decode_osc_bits(struct device *dev, char *msg, u32 word,
+			    struct pci_osc_bit_struct *table, int size)
+{
+	char buf[80];

drivers/cxl/acpi.c static struct acpi_driver acpi_cxl_driver = { · line 123

Quoted 10 lines
drivers/cxl/acpi.c +10 -0
+	int i, len = 0;
+	struct pci_osc_bit_struct *entry;
+
+	buf[0] = '\0';
+	for (i = 0, entry = table; i < size; i++, entry++)
+		if (word & entry->bit)
+			len += scnprintf(buf + len, sizeof(buf) - len, "%s%s",
+					len ? " " : "", entry->desc);
+
+	dev_info(dev, "_OSC: %s [%s]\n", msg, buf);

Can we look at sharing code with PCI?

Quoted 210 lines
drivers/cxl/acpi.c +161 -2
+}
+
+static void decode_cxl_osc_support(struct device *dev, char *msg, u32 word)
+{
+	decode_osc_bits(dev, msg, word, cxl_osc_support_bit,
+			ARRAY_SIZE(cxl_osc_support_bit));
+}
+
+static void decode_cxl_osc_control(struct device *dev, char *msg, u32 word)
+{
+	decode_osc_bits(dev, msg, word, cxl_osc_control_bit,
+			ARRAY_SIZE(cxl_osc_control_bit));
+}
+
+static acpi_status acpi_cap_run_osc(acpi_handle handle,
+				    const u32 *capbuf, u8 *uuid_str, u32 *retval)
+{
+	struct acpi_osc_context context = {
+		.uuid_str = uuid_str,
+		.rev = 1,
+		.cap.length = 20,
+		.cap.pointer = (void *)capbuf,
+	};
+	acpi_status status;
+
+	status = acpi_run_osc(handle, &context);
+	if (ACPI_SUCCESS(status)) {
+		*retval = *((u32 *)(context.ret.pointer + 8));
+		kfree(context.ret.pointer);
+	}
+	return status;
+}
+
+static acpi_status cxl_query_osc(acpi_handle handle, u32 support, u32 *control)
+{
+	acpi_status status;
+	u32 result, capbuf[5];
+
+	support &= CXL_OSC_SUPPORT_VALID_MASK;
+	support |= cxl_osc_support_set;
+
+	capbuf[OSC_QUERY_DWORD] = OSC_QUERY_ENABLE;
+	capbuf[CXL_OSC_SUPPORT_DWORD] = support;
+	if (control) {
+		*control &= CXL_OSC_CONTROL_VALID_MASK;
+		capbuf[CXL_OSC_CONTROL_DWORD] = *control | cxl_osc_control_set;
+	} else {
+		/* Run _OSC query only with existing controls. */
+		capbuf[CXL_OSC_CONTROL_DWORD] = cxl_osc_control_set;
+	}
+
+	status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, &result);
+	if (ACPI_SUCCESS(status)) {
+		cxl_osc_support_set = support;
+		if (control)
+			*control = result;
+	}
+	return status;
+}
+
+static acpi_status cxl_osc_advertise_support(acpi_handle handle, u32 flags)
+{
+	acpi_status status;
+
+	mutex_lock(&acpi_desc_lock);
+	status = cxl_query_osc(handle, flags, NULL);
+	mutex_unlock(&acpi_desc_lock);
+	return status;
+}
+
+/**
+ * cxl_osc_request_control - Request control of CXL root _OSC features.
+ * @adev: ACPI device for the PCI root bridge (or PCIe Root Complex).
+ * @mask: Mask of _OSC bits to request control of, place to store control mask.
+ * @req: Mask of _OSC bits the control of is essential to the caller.
+ **/
+static acpi_status cxl_osc_request_control(struct acpi_device *adev, u32 *mask, u32 req)
+{
+	acpi_handle handle = adev->handle;
+	struct device *dev = &adev->dev;
+	acpi_status status = AE_OK;
+	u32 ctrl, capbuf[5];
+
+	if (!mask)
+		return AE_BAD_PARAMETER;
+
+	ctrl = *mask & CXL_OSC_MEM_ERROR_CONTROL;
+	if ((ctrl & req) != req)
+		return AE_TYPE;
+
+	mutex_lock(&acpi_desc_lock);
+
+	*mask = ctrl | cxl_osc_control_set;
+	/* No need to evaluate _OSC if the control was already granted. */
+	if ((cxl_osc_control_set & ctrl) == ctrl)
+		goto out;
+
+	/* Need to check the available controls bits before requesting them. */
+	while (*mask) {
+		status = cxl_query_osc(handle, cxl_osc_support_set, mask);
+		if (ACPI_FAILURE(status))
+			goto out;
+		if (ctrl == *mask)
+			break;
+		decode_cxl_osc_control(dev, "platform does not support", ctrl & ~(*mask));
+		ctrl = *mask;
+	}
+
+	if ((ctrl & req) != req) {
+		decode_cxl_osc_control(dev, "not requesting control; platform does not support",
+				       req & ~(ctrl));
+		status = AE_SUPPORT;
+		goto out;
+	}
+
+	capbuf[OSC_QUERY_DWORD] = 0;
+	capbuf[CXL_OSC_SUPPORT_DWORD] = cxl_osc_support_set;
+	capbuf[CXL_OSC_CONTROL_DWORD] = ctrl;
+	status = acpi_cap_run_osc(handle, capbuf, cxl_osc_uuid_str, mask);
+	if (ACPI_SUCCESS(status))
+		cxl_osc_control_set = *mask;
+out:
+	mutex_unlock(&acpi_desc_lock);
+	return status;
+}
+
+static int cxl_negotiate_osc(struct acpi_device *adev)
+{
+	u32 cxl_support, cxl_control, requested;
+	acpi_handle handle = adev->handle;
+	struct device *dev = &adev->dev;
+	acpi_status status;
+
+	/* Declare support for everything */
+	cxl_support = CXL_OSC_SUPPORT_VALID_MASK;
+	decode_cxl_osc_support(dev, "OS supports", cxl_support);
+	status = cxl_osc_advertise_support(handle, cxl_support);
+	if (ACPI_FAILURE(status)) {
+		dev_info(dev, "CXL_OSC failed (%s)\n", acpi_format_exception(status));
+		return -ENXIO;
+	}
+
+	/* Request control for everything */
+	cxl_control = CXL_OSC_CONTROL_VALID_MASK;
+	requested = cxl_control;
+	status = cxl_osc_request_control(adev, &cxl_control, CXL_OSC_MEM_ERROR_CONTROL);
+	if (ACPI_SUCCESS(status)) {
+		decode_cxl_osc_control(dev, "OS now controls", cxl_control);
+	} else {
+		decode_cxl_osc_control(dev, "OS requested", requested);
+		decode_cxl_osc_control(dev, "platform willing to grant", cxl_control);
+		dev_info(dev, "_OSC failed (%s)\n", acpi_format_exception(status));
+	}
+	return 0;
+}
+
 /*
  * If/when CXL support is defined by other platform firmware the kernel
  * will need a mechanism to select between the platform specific version
@@ -84,6 +287,7 @@ int cxl_bus_prepared(struct pci_dev *pdev)
 	struct acpi_device *adev;
 	struct pci_dev *root_port;
 	struct device *root;
+	int rc;
 
 	root_port = pcie_find_root_port(pdev);
 	if (!root_port)
@@ -97,9 +301,11 @@ int cxl_bus_prepared(struct pci_dev *pdev)
 	if (!adev)
 		return -ENXIO;
 
-	/* TODO: OSC enabling */
+	rc = cxl_negotiate_osc(adev);
+	if (rc)
+		dev_err(&pdev->dev, "Failed to negotiate OSC\n");
 
-	return 0;
+	return rc;
 }
 EXPORT_SYMBOL_GPL(cxl_bus_prepared);
 
drivers/cxl/acpi.h +18 -0
index 011505475cc6..bf8d078e1b2a 100644
@@ -10,6 +10,24 @@ struct acpi_cxl_desc {
 	struct device *dev;
 };
 
+/* Indexes into _OSC Capabilities Buffer */
+#define CXL_OSC_SUPPORT_DWORD			1	/* DWORD 2 */
+#define CXL_OSC_CONTROL_DWORD			2	/* DWORD 3 */
+
+/* CXL Host Bridge _OSC: Capabilities DWORD 2: Support Field */
+#define CXL_OSC_PORT_REG_ACCESS_SUPPORT		0x00000001
+#define CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT	0x00000002
+#define CXL_OSC_PER_SUPPORT			0x00000004
+#define CXL_OSC_NATIVE_HP_SUPPORT		0x00000008
+#define CXL_OSC_SUPPORT_VALID_MASK		(CXL_OSC_PORT_REG_ACCESS_SUPPORT |	\
+						 CXL_OSC_PORT_DEV_REG_ACCESS_SUPPORT |	\
+						 CXL_OSC_PER_SUPPORT |			\
+						 CXL_OSC_NATIVE_HP_SUPPORT)
+
+/* CXL Host Bridge _OSC: Capabilities DWORD 3: Control Field */
+#define CXL_OSC_MEM_ERROR_CONTROL		0x00000001
+#define CXL_OSC_CONTROL_VALID_MASK		(CXL_OSC_MEM_ERROR_CONTROL)
+
 int cxl_bus_prepared(struct pci_dev *pci_dev);
 
 #endif	/* __CXL_ACPI_H__ */
#16 lore raw

Re: [RFC PATCH 2/9] cxl/acpi: add OSC support

Quoted 11 lines · Jonathan Cameron
Quoted 7 lines · On Tue, 10 Nov 2020 21:43:49 -0800 Ben Widawsky <[email protected]> wrote:

From: Vishal Verma <[email protected]>

Add support to advertise OS capabilities, and request OS control for CXL features using the ACPI _OSC mechanism. Advertise support for all possible CXL features, and attempt to request control too for all possible features.

Quoted 8 lines
Quoted 4 lines

Based on a patch by Sean Kelley.

I guess unsurprisingly a lot of this is cut and paste from PCIe so can we share some of the code?

I do not see a refactoring effort for these bit being all that fruitful. The backport pressure for this driver stack I expect will be higher than most, so I'm sensitive to avoiding unnecessary core entanglements.

#17 lore raw

Re: [RFC PATCH 2/9] cxl/acpi: add OSC support

Quoted 19 lines · Dan Williams
Quoted 16 lines · Jonathan Cameron
Quoted 11 lines · On Tue, 10 Nov 2020 21:43:49 -0800 Ben Widawsky <[email protected]> wrote:

From: Vishal Verma <[email protected]>

Add support to advertise OS capabilities, and request OS control for CXL features using the ACPI _OSC mechanism. Advertise support for all possible CXL features, and attempt to request control too for all possible features.

Based on a patch by Sean Kelley.

Quoted 6 lines
Quoted 3 lines

I guess unsurprisingly a lot of this is cut and paste from PCIe so can we share some of the code?

I do not see a refactoring effort for these bit being all that fruitful.

Well, that depends on how much code duplication could be avoided this way.

Quoted 3 lines

The backport pressure for this driver stack I expect will be higher than most, so I'm sensitive to avoiding unnecessary core entanglements.

If two pieces of code are based on the same underlying common code, it is immediately clear to the reader how similar to each other they are. Otherwise, they need to be carefully compared with each other to find out what the differences are and whether or not they are arbitrary or vitally important. That is essential both from the revirem perspective today and to anyone wanting to understand the given code in the future (possibly in order to modify it without breaking it). It outweighs the convenience by far IMV, with all due respect.

Recall how much effort it took to combine x86 with x86_64 and why it turned out to be necessary to do that work, for one example.

#18 lore raw

Re: [RFC PATCH 2/9] cxl/acpi: add OSC support

Quoted 43 lines · Rafael J. Wysocki
Quoted 19 lines · Dan Williams
Quoted 16 lines · Jonathan Cameron
Quoted 11 lines · On Tue, 10 Nov 2020 21:43:49 -0800 Ben Widawsky <[email protected]> wrote:

From: Vishal Verma <[email protected]>

Add support to advertise OS capabilities, and request OS control for CXL features using the ACPI _OSC mechanism. Advertise support for all possible CXL features, and attempt to request control too for all possible features.

Based on a patch by Sean Kelley.

Quoted 6 lines
Quoted 3 lines

I guess unsurprisingly a lot of this is cut and paste from PCIe so can we share some of the code?

I do not see a refactoring effort for these bit being all that fruitful.

Well, that depends on how much code duplication could be avoided this way.

Quoted 3 lines

The backport pressure for this driver stack I expect will be higher than most, so I'm sensitive to avoiding unnecessary core entanglements.

If two pieces of code are based on the same underlying common code, it is immediately clear to the reader how similar to each other they are. Otherwise, they need to be carefully compared with each other to find out what the differences are and whether or not they are arbitrary or vitally important. That is essential both from the revirem perspective today and to anyone wanting to understand the given code in the future (possibly in order to modify it without breaking it). It outweighs the convenience by far IMV, with all due respect.

Quoted 2 lines

Recall how much effort it took to combine x86 with x86_64 and why it turned out to be necessary to do that work, for one example.

I agree with above, but the degree of potential code sharing and refactoring for CXL is nowhere near approaching the x86_64 situation. There's also the counter example in ext3 and ext4 where a split is maintained for good reason. All I'm saying is that let's judge patches and not theory when it comes to refactoring CXL, my expectation is that those opportunities will be few and far between. CXL is a superset of PCIE functionality so it should not put much pressure on common core PCIE code to change vs incremental CXL extensions.

#19 lore raw

RFC PATCH 3/9 cxl/mem: Add a driver for the type-3 mailbox

In reply to Ben Widawsky

From: Dan Williams <[email protected]>

The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller.

The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices.

A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the "type-3 mailbox" protocol.

For now just land the driver boiler-plate and fill it in with functionality in subsequent commits.

---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
drivers/cxl/Kconfig +20 -0
index dd724bd364df..15548f5c77ff 100644
@@ -27,4 +27,24 @@ config CXL_ACPI
 	  resources described by the CEDT (CXL Early Discovery Table)
 
 	  Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
+        default m if CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
drivers/cxl/Makefile +2 -0
index d38cd34a2582..97fdffb00f2d 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
drivers/cxl/mem.c new file +82 -0

82 lines not on this page · raw message

drivers/cxl/pci.h new file +15 -0
new file mode 100644
index 000000000000..beb03921e6da
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL	0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC	0x23
+#define PCI_DVSEC_VENDOR_CXL	0x1E98
+#define PCI_DVSEC_VENDOR_OFFSET	0x4
+#define PCI_DVSEC_ID_OFFSET	0x8
+#define PCI_DVSEC_ID_CXL	0x0
+#define PCI_DVSEC_ID_CXL_REGLOC	0x8
+
+#endif /* __CXL_PCI_H__ */
Signature · 1 line
2.29.2
#20 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

Hi,

Quoted 12 lines · Ben Widawsky
---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
drivers/cxl/Kconfig +0 -0
index dd724bd364df..15548f5c77ff 100644

drivers/cxl/Kconfig config CXL_ACPI · line 34

Quoted 8 lines
drivers/cxl/Kconfig +5 -0
@@ -27,4 +27,24 @@ config CXL_ACPI
 	  resources described by the CEDT (CXL Early Discovery Table)
 
 	  Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
+        default m if CXL_BUS_PROVIDER

The "if CXL_BUS_PROVIDER" should be redundant due to the "depends on" clause. However, having any default or 'm' or 'y' needs to be justified.

drivers/cxl/Kconfig config CXL_ACPI · line 41

Quoted 7 lines
drivers/cxl/Kconfig +7 -0
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach

The builtin driver will not be named cxl-mem.ko.

drivers/cxl/Kconfig config CXL_ACPI · line 45

Quoted 4 lines
drivers/cxl/Kconfig +4 -0
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but

prerequisite

Quoted 5 lines
drivers/cxl/Kconfig +4 -0
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
Signature · 1 line
~Randy
#21 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

In reply to Ben Widawsky

drivers/cxl/Kconfig config CXL_ACPI · line 33

Quoted 3 lines · Ben Widawsky
drivers/cxl/Kconfig +3 -0
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n

depend on PCI && CXL_BUS_PROVIDER

Quoted 1 line

+ default m if CXL_BUS_PROVIDER

Please don't set weird defaults for new code. Especially not default to module crap like this.

Quoted 1 line

+// Copyright(c) 2020 Intel Corporation. All rights reserved.

Please don't use '//' for anything but the SPDX header.

drivers/cxl/mem.c line 29

Quoted 7 lines
drivers/cxl/mem.c new file +7 -0
+
+		pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+		pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+		if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+			return pos;
+
+		pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);

Overly long lines again.

drivers/cxl/mem.c line 62

Quoted 3 lines
drivers/cxl/mem.c new file +3 -0
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}

No need for the empty remove callback.

Quoted 1 line

+MODULE_AUTHOR("Intel Corporation");

A module author is not a company.

#22 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

Quoted 6 lines · Christoph Hellwig

drivers/cxl/Kconfig config CXL_ACPI · line 33

Quoted 3 lines · Ben Widawsky
drivers/cxl/Kconfig +3 -0
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
Quoted 6 lines

depend on PCI && CXL_BUS_PROVIDER

Quoted 1 line

+ default m if CXL_BUS_PROVIDER

Please don't set weird defaults for new code. Especially not default to module crap like this.

This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, a way to blanket turn on a subsystem without needing to go hunt down individual configs. All of CXL is "default n", but if someone turns on a piece of it they get all of it by default. The user can then opt-out on pieces after that first opt-in. If there's a better way to turn on suggested configs I'm open to switch to that style. As for the "default m" I was worried that it would be "default y" without the specificity, but I did not test that... will check. There have been times when I wished that distros defaulted bleeding edge new enabling to 'm' and putting that default in the Kconfig maybe saves me from needing to file individual config changes to distros after the fact.

Quoted 4 lines
Quoted 2 lines

+// Copyright(c) 2020 Intel Corporation. All rights reserved.

Please don't use '//' for anything but the SPDX header.

Ok, I find // following by /* */ a bit ugly, but I don't care enough to fight.

Quoted 10 lines

drivers/cxl/mem.c line 29

Quoted 8 lines
drivers/cxl/mem.c new file +7 -0
+
+             pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+             pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+             if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+                     return pos;
+
+             pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);

Overly long lines again.

I thought 100 is the new 80 these days?

Quoted 5 lines

drivers/cxl/mem.c line 62

Quoted 3 lines
drivers/cxl/mem.c new file +3 -0
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}

No need for the empty remove callback.

True, will fix.

Quoted 4 lines
Quoted 2 lines

+MODULE_AUTHOR("Intel Corporation");

A module author is not a company.

At least I don't have a copyright assignment clause, I don't agree with the vanity of listing multiple people here especially when MAINTAINERS has the contact info, and I don't want to maintain a list as people do drive-by contributions and we need to figure out at what level of contribution mandates a new MODULE_AUTHOR line. Now, that said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR lines, but I otherwise expect MAINTAINERS is the central source for module contact info.

#23 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

On Wed, Nov 11, 2020 at 9:17 AM Dan Williams <[email protected]> wrote:
[..]

Quoted 11 lines
Quoted 9 lines

drivers/cxl/mem.c line 29

Quoted 7 lines
drivers/cxl/mem.c new file +7 -0
+
+             pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+             pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+             if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+                     return pos;
+
+             pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);

Overly long lines again.

I thought 100 is the new 80 these days?

Saw your clarification to Vishal, I had missed that. Will trim.

#24 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

In reply to Dan Williams
Quoted 14 lines · Dan Williams
Quoted 6 lines · Christoph Hellwig

drivers/cxl/Kconfig config CXL_ACPI · line 33

Quoted 3 lines · Ben Widawsky
drivers/cxl/Kconfig +3 -0
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
Quoted 6 lines

depend on PCI && CXL_BUS_PROVIDER

Quoted 1 line

+ default m if CXL_BUS_PROVIDER

Please don't set weird defaults for new code. Especially not default to module crap like this.

Quoted 11 lines

This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, a way to blanket turn on a subsystem without needing to go hunt down individual configs. All of CXL is "default n", but if someone turns on a piece of it they get all of it by default. The user can then opt-out on pieces after that first opt-in. If there's a better way to turn on suggested configs I'm open to switch to that style. As for the "default m" I was worried that it would be "default y" without the specificity, but I did not test that... will check. There have been times when I wished that distros defaulted bleeding edge new enabling to 'm' and putting that default in the Kconfig maybe saves me from needing to file individual config changes to distros after the fact.

What we as developers put into mainline kernel Kconfig files has nothing to do with what distros use in their distro config files.
Or at least it shouldn't. Maybe your experience has been different.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

+// Copyright(c) 2020 Intel Corporation. All rights reserved.

Please don't use '//' for anything but the SPDX header.

Ok, I find // following by /* */ a bit ugly, but I don't care enough to fight.

Hm, it's not in coding-style AFAICT but Linus has OK-ed C99 style comments:
http://lkml.iu.edu/hypermail/linux/kernel/1607.1/00627.html

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

+MODULE_AUTHOR("Intel Corporation");

A module author is not a company.

Quoted 8 lines

At least I don't have a copyright assignment clause, I don't agree with the vanity of listing multiple people here especially when MAINTAINERS has the contact info, and I don't want to maintain a list as people do drive-by contributions and we need to figure out at what level of contribution mandates a new MODULE_AUTHOR line. Now, that said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR lines, but I otherwise expect MAINTAINERS is the central source for module contact info.

Sure, MAINTAINERS is fine, but the MODULE_AUTHOR() above provides no useful information.
Even saying (made up) [email protected] would be slightly better, but some kind of contact info would be great. Otherwise just delete that line.

Signature · 1 line
~Randy
#25 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

Quoted 28 lines · Randy Dunlap
Quoted 14 lines · Dan Williams
Quoted 6 lines · Christoph Hellwig

drivers/cxl/Kconfig config CXL_ACPI · line 33

Quoted 3 lines · Ben Widawsky
drivers/cxl/Kconfig +3 -0
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
Quoted 6 lines

depend on PCI && CXL_BUS_PROVIDER

Quoted 1 line

+ default m if CXL_BUS_PROVIDER

Please don't set weird defaults for new code. Especially not default to module crap like this.

Quoted 11 lines

This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, a way to blanket turn on a subsystem without needing to go hunt down individual configs. All of CXL is "default n", but if someone turns on a piece of it they get all of it by default. The user can then opt-out on pieces after that first opt-in. If there's a better way to turn on suggested configs I'm open to switch to that style. As for the "default m" I was worried that it would be "default y" without the specificity, but I did not test that... will check. There have been times when I wished that distros defaulted bleeding edge new enabling to 'm' and putting that default in the Kconfig maybe saves me from needing to file individual config changes to distros after the fact.

Quoted 3 lines

What we as developers put into mainline kernel Kconfig files has nothing to do with what distros use in their distro config files.
Or at least it shouldn't. Maybe your experience has been different.

I agree with that sentiment, but walk it back through the requirement I mentioned above... *if* we want a top-level CXL option (default n) that goes and enables many CXL sub-options the default for those sub-options is something that needs to be listed in the Kconfig. 'm' is more flexible than 'y', so if a user wants CXL at all, and doesn't care about how, I'd prefer it's 'm' rather than 'y'.

I have had to go submit distro config fixes when Kconfig defaulted to 'y' when 'm' was available, and the reasoning for why it was 'y' was "oh, that was the Kconfig default when I flipped this other option".

Quoted 24 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

+// Copyright(c) 2020 Intel Corporation. All rights reserved.

Please don't use '//' for anything but the SPDX header.

Ok, I find // following by /* */ a bit ugly, but I don't care enough to fight.

Hm, it's not in coding-style AFAICT but Linus has OK-ed C99 style comments:
http://lkml.iu.edu/hypermail/linux/kernel/1607.1/00627.html

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

+MODULE_AUTHOR("Intel Corporation");

A module author is not a company.

Quoted 8 lines

At least I don't have a copyright assignment clause, I don't agree with the vanity of listing multiple people here especially when MAINTAINERS has the contact info, and I don't want to maintain a list as people do drive-by contributions and we need to figure out at what level of contribution mandates a new MODULE_AUTHOR line. Now, that said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR lines, but I otherwise expect MAINTAINERS is the central source for module contact info.

Quoted 4 lines

Sure, MAINTAINERS is fine, but the MODULE_AUTHOR() above provides no useful information.
Even saying (made up) [email protected] would be slightly better, but some kind of contact info would be great. Otherwise just delete that line.

True, if the goal is to allow random end users to email support questions about this module I'd rather not put my email there. Instead, if it's someone that has kernel development questions then they should be able to use MAINTAINERS for that contact.

#26 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

In reply to Dan Williams
Quoted 6 lines · Dan Williams
Quoted 6 lines

drivers/cxl/Kconfig config CXL_ACPI · line 33

Quoted 3 lines
drivers/cxl/Kconfig +3 -0
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n

depend on PCI && CXL_BUS_PROVIDER

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

+ default m if CXL_BUS_PROVIDER

Please don't set weird defaults for new code. Especially not default to module crap like this.

This goes back to what people like Dave C. asked for LIBNVDIMM / DAX, a way to blanket turn on a subsystem without needing to go hunt down individual configs.

Then at least do a

default CXL_BUS_PROVIDER

but we really don't do this elsewhere. E.g. we don't default the scsi disk driver on if there is some host adapter selected.

Quoted 4 lines
Quoted 3 lines
Quoted 1 line

+MODULE_AUTHOR("Intel Corporation");

A module author is not a company.

Quoted 8 lines

At least I don't have a copyright assignment clause, I don't agree with the vanity of listing multiple people here especially when MAINTAINERS has the contact info, and I don't want to maintain a list as people do drive-by contributions and we need to figure out at what level of contribution mandates a new MODULE_AUTHOR line. Now, that said I would be ok to duplicate the MAINTAINERS as MODULE_AUTHOR lines, but I otherwise expect MAINTAINERS is the central source for module contact info.

IMHO MODULE_AUTHOR is completely pointless. I haven't used for ~15 years. Especially as the concept that a module has a single author is a rather strange one.

#27 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

In reply to Ben Widawsky
Quoted 6 lines · Ben Widawsky

From: Dan Williams <[email protected]>

The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller.

Quoted 6 lines

The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices.

A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the "type-3 mailbox" protocol.

"Type 3" to indicate that this is a proper adjective that can be looked up in the spec and to match the usage there.

The r1.1 spec I have doesn't mention "mailbox". Is that also something defined in the 2.0 spec?

A URL or similar citation for the spec would be nice somewhere.

Quoted 26 lines

For now just land the driver boiler-plate and fill it in with functionality in subsequent commits.

---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
drivers/cxl/Kconfig +4 -0
index dd724bd364df..15548f5c77ff 100644
@@ -27,4 +27,24 @@ config CXL_ACPI
 	  resources described by the CEDT (CXL Early Discovery Table)
 
 	  Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n

drivers/cxl/Kconfig config CXL_ACPI · line 43

Quoted 10 lines
drivers/cxl/Kconfig +10 -0
+        default m if CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver

"Type 3"

Quoted 26 lines
drivers/cxl/Kconfig +6 -0
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
drivers/cxl/Makefile +2 -0
index d38cd34a2582..97fdffb00f2d 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
drivers/cxl/mem.c new file +1 -0
new file mode 100644
index 000000000000..aa7d881fa47b
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only

drivers/cxl/mem.c line 11

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/io.h>
+#include "acpi.h"
+#include "pci.h"
+
+struct cxl_mem {
+	void __iomem *regs;
+};

Unused, maybe move it to the patch that adds the use?

Quoted 11 lines
drivers/cxl/mem.c new file +11 -0
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+	int pos;
+
+	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+	if (!pos)
+		return 0;
+
+	while (pos) {
+		u16 vendor, id;
+

drivers/cxl/mem.c line 33

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+		pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+		pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+		if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+			return pos;
+
+		pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);
+	}
+
+	return 0;
+}

I assume we'll refactor and move this into the PCI core after we resolve the several places this is needed. When we do that, the vendor would be passed in, so maybe we should do that here to make it simpler to move this to the PCI core.

drivers/cxl/mem.c line 43

Quoted 9 lines
drivers/cxl/mem.c new file +9 -0
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	int rc, regloc;
+
+	rc = cxl_bus_prepared(pdev);
+	if (rc != 0) {
+		dev_err(dev, "failed to acquire interface\n");

Interesting naming: apparently when cxl_bus_prepared() returns a non-zero ("true") value, it is actually *not* prepared?

drivers/cxl/mem.c line 55

Quoted 12 lines
drivers/cxl/mem.c new file +12 -0
+		return rc;
+	}
+
+	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+	if (!regloc) {
+		dev_err(dev, "register location dvsec not found\n");
+		return -ENXIO;
+	}
+
+	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm)
+		return -ENOMEM;

Unused. And [4/9] removes it before it's *ever* used :)

Quoted 33 lines
drivers/cxl/mem.c new file +26 -0
+	return 0;
+}
+
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+	/* PCI class code for CXL.mem Type-3 Devices */
+	{ PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+	  PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+	{ /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+	.name			= KBUILD_MODNAME,
+	.id_table		= cxl_mem_pci_tbl,
+	.probe			= cxl_mem_probe,
+	.remove			= cxl_mem_remove,
+};
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +1 -1
new file mode 100644
index 000000000000..beb03921e6da
--- /dev/null
 
+++ b/drivers/cxl/pci.h

drivers/cxl/pci.h line 2

Quoted 2 lines
drivers/cxl/pci.h new file +2 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.

/* SPDX-... */
/* Copyright ...*/

The SPDX rules are a bit arcane and annoyingly hard to grep for, but I found them in Documentation/process/license-rules.rst

drivers/cxl/pci.h line 13

Quoted 11 lines
drivers/cxl/pci.h new file +11 -0
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL	0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC	0x23
+#define PCI_DVSEC_VENDOR_CXL	0x1E98
+#define PCI_DVSEC_VENDOR_OFFSET	0x4
+#define PCI_DVSEC_ID_OFFSET	0x8
+#define PCI_DVSEC_ID_CXL	0x0
+#define PCI_DVSEC_ID_CXL_REGLOC	0x8

I assume these will go in include/linux/pci_ids.h (PCI_CLASS_...) and include/uapi/linux/pci_regs.h (the rest) eventually, after we get the merge issues sorted out. But if they're only used in cxl/mem.c, I'd put them there for now.

Quoted 4 lines
drivers/cxl/pci.h new file +1 -0
+#endif /* __CXL_PCI_H__ */
Signature · 1 line
2.29.2
#28 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

Quoted 11 lines · Bjorn Helgaas
Quoted 10 lines · Ben Widawsky

From: Dan Williams <[email protected]>

The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller.

The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices.

Quoted 8 lines
Quoted 2 lines

A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the "type-3 mailbox" protocol.

"Type 3" to indicate that this is a proper adjective that can be looked up in the spec and to match the usage there.

The r1.1 spec I have doesn't mention "mailbox". Is that also something defined in the 2.0 spec?

Yes, these device types are new to 2.0.

Quoted 3 lines

A URL or similar citation for the spec would be nice somewhere.

Agreed. For the patches I authored at least, it seemed repetitive to put a Link:
in each one to the spec. It was meant to be in the cover letter, but obviously I missed that. Do you have a suggestion there, is cover letter good enough?

Quoted 69 lines
Quoted 26 lines

For now just land the driver boiler-plate and fill it in with functionality in subsequent commits.

---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
drivers/cxl/Kconfig +4 -0
index dd724bd364df..15548f5c77ff 100644
@@ -27,4 +27,24 @@ config CXL_ACPI
 	  resources described by the CEDT (CXL Early Discovery Table)
 
 	  Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n

drivers/cxl/Kconfig config CXL_ACPI · line 43

Quoted 10 lines
drivers/cxl/Kconfig +10 -0
+        default m if CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver

"Type 3"

Quoted 30 lines
drivers/cxl/Kconfig +6 -0
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
drivers/cxl/Makefile +2 -0
index d38cd34a2582..97fdffb00f2d 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
drivers/cxl/mem.c new file +5 -0
new file mode 100644
index 000000000000..aa7d881fa47b
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/io.h>
Quoted 9 lines

drivers/cxl/mem.c line 11

Quoted 6 lines
drivers/cxl/mem.c new file +6 -0
+#include "acpi.h"
+#include "pci.h"
+
+struct cxl_mem {
+	void __iomem *regs;
+};

Unused, maybe move it to the patch that adds the use?

This is a remnant from when Dan gave me the basis to do the mmio work. I agree it can be removed now.

Quoted 18 lines
Quoted 18 lines
drivers/cxl/mem.c new file +18 -0
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+	int pos;
+
+	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+	if (!pos)
+		return 0;
+
+	while (pos) {
+		u16 vendor, id;
+
+		pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+		pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+		if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+			return pos;
+
+		pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);
+	}
Quoted 9 lines

drivers/cxl/mem.c line 30

Quoted 3 lines
drivers/cxl/mem.c new file +3 -0
+
+	return 0;
+}

I assume we'll refactor and move this into the PCI core after we resolve the several places this is needed. When we do that, the vendor would be passed in, so maybe we should do that here to make it simpler to move this to the PCI core.

I think we'll need to keep this in order to try to keep the dream alive of loading a CXL kernel module on an older kernel. However, PCI code would benefit from having it (in an ideal world, it'd only be there).

Quoted 4 lines
Quoted 4 lines
drivers/cxl/mem.c new file +4 -0
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
Quoted 9 lines

drivers/cxl/mem.c line 43

Quoted 5 lines
drivers/cxl/mem.c new file +5 -0
+	int rc, regloc;
+
+	rc = cxl_bus_prepared(pdev);
+	if (rc != 0) {
+		dev_err(dev, "failed to acquire interface\n");

Interesting naming: apparently when cxl_bus_prepared() returns a non-zero ("true") value, it is actually *not* prepared?

This looks like a rebase fail to me, but I'll let Dan answer.

Quoted 6 lines
Quoted 6 lines
drivers/cxl/mem.c new file +6 -0
+		return rc;
+	}
+
+	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+	if (!regloc) {
+		dev_err(dev, "register location dvsec not found\n");
Quoted 9 lines

drivers/cxl/mem.c line 55

Quoted 6 lines
drivers/cxl/mem.c new file +6 -0
+		return -ENXIO;
+	}
+
+	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm)
+		return -ENOMEM;

Unused. And [4/9] removes it before it's *ever* used :)

Same as a few above, I think Dan was providing this for me to implement the reset. It could go away...

Quoted 63 lines
Quoted 33 lines
drivers/cxl/mem.c new file +26 -0
+	return 0;
+}
+
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+	/* PCI class code for CXL.mem Type-3 Devices */
+	{ PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+	  PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+	{ /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+	.name			= KBUILD_MODNAME,
+	.id_table		= cxl_mem_pci_tbl,
+	.probe			= cxl_mem_probe,
+	.remove			= cxl_mem_remove,
+};
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +1 -1
new file mode 100644
index 000000000000..beb03921e6da
--- /dev/null
 
+++ b/drivers/cxl/pci.h

drivers/cxl/pci.h line 2

Quoted 2 lines
drivers/cxl/pci.h new file +2 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.

/* SPDX-... */
/* Copyright ...*/

The SPDX rules are a bit arcane and annoyingly hard to grep for, but I found them in Documentation/process/license-rules.rst

drivers/cxl/pci.h line 13

Quoted 11 lines
drivers/cxl/pci.h new file +11 -0
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL	0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC	0x23
+#define PCI_DVSEC_VENDOR_CXL	0x1E98
+#define PCI_DVSEC_VENDOR_OFFSET	0x4
+#define PCI_DVSEC_ID_OFFSET	0x8
+#define PCI_DVSEC_ID_CXL	0x0
+#define PCI_DVSEC_ID_CXL_REGLOC	0x8

I assume these will go in include/linux/pci_ids.h (PCI_CLASS_...) and include/uapi/linux/pci_regs.h (the rest) eventually, after we get the merge issues sorted out. But if they're only used in cxl/mem.c, I'd put them there for now.

Quoted 4 lines
drivers/cxl/pci.h new file +1 -0
+#endif /* __CXL_PCI_H__ */
Signature · 1 line
2.29.2
#29 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

On Fri, Nov 13, 2020 at 5:09 PM Ben Widawsky <[email protected]> wrote:
[..]

Quoted 5 lines
Quoted 2 lines

Unused, maybe move it to the patch that adds the use?

This is a remnant from when Dan gave me the basis to do the mmio work. I agree it can be removed now.

Yes.

Quoted 28 lines
Quoted 18 lines
Quoted 18 lines
drivers/cxl/mem.c new file +18 -0
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+   int pos;
+
+   pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+   if (!pos)
+           return 0;
+
+   while (pos) {
+           u16 vendor, id;
+
+           pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+           pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+           if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+                   return pos;
+
+           pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);
+   }
Quoted 9 lines

drivers/cxl/mem.c line 30

Quoted 3 lines
drivers/cxl/mem.c new file +3 -0
+
+   return 0;
+}

I assume we'll refactor and move this into the PCI core after we resolve the several places this is needed. When we do that, the vendor would be passed in, so maybe we should do that here to make it simpler to move this to the PCI core.

Quoted 3 lines

I think we'll need to keep this in order to try to keep the dream alive of loading a CXL kernel module on an older kernel. However, PCI code would benefit from having it (in an ideal world, it'd only be there).

So I think this is fine / expected to move standalone common code like this to the PCI core. What I'm aiming to avoid with "the dream" Ben references is unnecessary dependencies on core changes. CXL is large enough that it will generate more backport pressure than ACPI NFIT / LIBNVDIMM ever did. From a self interest perspective maximizing how much of CXL can be enabled without core dependencies is a goal just to lighten my own backport load. The internals of cxl_mem_dvsec() are simple enough to backport.

Quoted 15 lines
Quoted 14 lines

drivers/cxl/mem.c line 43

Quoted 10 lines
drivers/cxl/mem.c new file +9 -0
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+   struct device *dev = &pdev->dev;
+   struct cxl_mem *cxlm;
+   int rc, regloc;
+
+   rc = cxl_bus_prepared(pdev);
+   if (rc != 0) {
+           dev_err(dev, "failed to acquire interface\n");

Interesting naming: apparently when cxl_bus_prepared() returns a non-zero ("true") value, it is actually *not* prepared?

Quoted 1 line

This looks like a rebase fail to me, but I'll let Dan answer.

Yeah, I originally envisioned this as a ternary result with -EPROBE_DEFER as a possible return value, but now that we've found a way to handle CXL _OSC without colliding with legacy PCIE _OSC this can indeed move to a boolean result.

Will fix up.

Quoted 17 lines
Quoted 16 lines

drivers/cxl/mem.c line 55

Quoted 13 lines
drivers/cxl/mem.c new file +12 -0
+           return rc;
+   }
+
+   regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+   if (!regloc) {
+           dev_err(dev, "register location dvsec not found\n");
+           return -ENXIO;
+   }
+
+   cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+   if (!cxlm)
+           return -ENOMEM;

Unused. And [4/9] removes it before it's *ever* used :)

Quoted 2 lines

Same as a few above, I think Dan was providing this for me to implement the reset. It could go away...

Yes, a collaboration artifact that we can clean up.

Quoted 40 lines
Quoted 40 lines
Quoted 34 lines
drivers/cxl/mem.c new file +26 -0
+   return 0;
+}
+
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+   /* PCI class code for CXL.mem Type-3 Devices */
+   { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+     PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+   { /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+   .name                   = KBUILD_MODNAME,
+   .id_table               = cxl_mem_pci_tbl,
+   .probe                  = cxl_mem_probe,
+   .remove                 = cxl_mem_remove,
+};
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +1 -1
new file mode 100644
index 000000000000..beb03921e6da
--- /dev/null
 
+++ b/drivers/cxl/pci.h

drivers/cxl/pci.h line 2

Quoted 2 lines
drivers/cxl/pci.h new file +2 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.

/* SPDX-... */
/* Copyright ...*/

Quoted 2 lines
Quoted 2 lines

The SPDX rules are a bit arcane and annoyingly hard to grep for, but I found them in Documentation/process/license-rules.rst

Yes, I did not realize the header vs source /* */ vs // SPDX style.

Quoted 17 lines
Quoted 17 lines

drivers/cxl/pci.h line 13

Quoted 12 lines
drivers/cxl/pci.h new file +11 -0
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL       0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC       0x23
+#define PCI_DVSEC_VENDOR_CXL       0x1E98
+#define PCI_DVSEC_VENDOR_OFFSET    0x4
+#define PCI_DVSEC_ID_OFFSET        0x8
+#define PCI_DVSEC_ID_CXL   0x0
+#define PCI_DVSEC_ID_CXL_REGLOC    0x8

I assume these will go in include/linux/pci_ids.h (PCI_CLASS_...) and include/uapi/linux/pci_regs.h (the rest) eventually, after we get the merge issues sorted out. But if they're only used in cxl/mem.c, I'd put them there for now.

Yes, I assume they'll move eventually. I'm cheating a standalone backport driver organization in the meantime.

#30 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

In reply to Ben Widawsky
Quoted 10 lines · On Tue, 10 Nov 2020 21:43:50 -0800 Ben Widawsky <[email protected]> wrote:

From: Dan Williams <[email protected]>

The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller.

The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices.

Quoted 8 lines

A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the "type-3 mailbox" protocol.

For now just land the driver boiler-plate and fill it in with functionality in subsequent commits.

I've tried to avoid repeats, so mostly this is me moaning about naming!

Jonathan

Quoted 75 lines
---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
drivers/cxl/Kconfig +20 -0
index dd724bd364df..15548f5c77ff 100644
@@ -27,4 +27,24 @@ config CXL_ACPI
 	  resources described by the CEDT (CXL Early Discovery Table)
 
 	  Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
+        default m if CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
drivers/cxl/Makefile +2 -0
index d38cd34a2582..97fdffb00f2d 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
drivers/cxl/mem.c new file +19 -0
new file mode 100644
index 000000000000..aa7d881fa47b
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/io.h>
+#include "acpi.h"
+#include "pci.h"
+
+struct cxl_mem {
+	void __iomem *regs;
+};
+
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+	int pos;
+
+	pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+	if (!pos)
+		return 0;

drivers/cxl/mem.c line 29

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+
+	while (pos) {
+		u16 vendor, id;
+
+		pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+		pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+		if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+			return pos;
+
+		pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);

This is good generic code and wouldn't cause much backport effort (even if needed to bring in a local copy), so perhaps make it a generic function and move to core PCI code?

Mind you I guess that can happen the 'second' time someone wants to find a DVSEC.

Quoted 23 lines
drivers/cxl/mem.c new file +23 -0
+	}
+
+	return 0;
+}
+
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	int rc, regloc;
+
+	rc = cxl_bus_prepared(pdev);
+	if (rc != 0) {
+		dev_err(dev, "failed to acquire interface\n");
+		return rc;
+	}
+
+	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+	if (!regloc) {
+		dev_err(dev, "register location dvsec not found\n");
+		return -ENXIO;
+	}
+

drivers/cxl/mem.c line 62

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm)
+		return -ENOMEM;
+
+	return 0;
+}
+
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}

I'd bring this in only when needed in later patch.

Quoted 26 lines
drivers/cxl/mem.c new file +20 -0
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+	/* PCI class code for CXL.mem Type-3 Devices */
+	{ PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+	  PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+	{ /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+	.name			= KBUILD_MODNAME,
+	.id_table		= cxl_mem_pci_tbl,
+	.probe			= cxl_mem_probe,
+	.remove			= cxl_mem_remove,
+};
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +0 -0
new file mode 100644
index 000000000000..beb03921e6da

drivers/cxl/pci.h line 9

Quoted 9 lines
drivers/cxl/pci.h new file +9 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL	0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC	0x23
+#define PCI_DVSEC_VENDOR_CXL	0x1E98

Hmm. The magic question of what to call a vendor ID that isn't a vendor ID but just a magic number that talks like a duck and quacks like a duck (for anyone wondering what I'm talking about, there is a nice bit of legal boilerplate on this in the CXL spec)

This name is definitely not accurate however.

PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), though possibly this is the only software visible use.

drivers/cxl/pci.h line 11

Quoted 2 lines
drivers/cxl/pci.h new file +2 -0
+#define PCI_DVSEC_VENDOR_OFFSET	0x4
+#define PCI_DVSEC_ID_OFFSET	0x8

Put a line break here perhaps and maybe a spec reference to where to find the various DVSEC IDs.

drivers/cxl/pci.h line 12

Quoted 1 line
drivers/cxl/pci.h new file +1 -0
+#define PCI_DVSEC_ID_CXL	0x0

That's definitely a confusing name as well.

PCI_DEVSEC_ID_CXL_DEVICE maybe?

Quoted 3 lines
drivers/cxl/pci.h new file +3 -0
+#define PCI_DVSEC_ID_CXL_REGLOC	0x8
+
+#endif /* __CXL_PCI_H__ */
#31 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

Quoted 196 lines · Jonathan Cameron
Quoted 10 lines · On Tue, 10 Nov 2020 21:43:50 -0800 Ben Widawsky <[email protected]> wrote:

From: Dan Williams <[email protected]>

The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller.

The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices.

Quoted 8 lines

A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the "type-3 mailbox" protocol.

For now just land the driver boiler-plate and fill it in with functionality in subsequent commits.

I've tried to avoid repeats, so mostly this is me moaning about naming!

Jonathan

Quoted 75 lines
---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
drivers/cxl/Kconfig +20 -0
index dd724bd364df..15548f5c77ff 100644
@@ -27,4 +27,24 @@ config CXL_ACPI
        resources described by the CEDT (CXL Early Discovery Table)
 
        Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
+        default m if CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
drivers/cxl/Makefile +2 -0
index d38cd34a2582..97fdffb00f2d 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
drivers/cxl/mem.c new file +19 -0
new file mode 100644
index 000000000000..aa7d881fa47b
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/io.h>
+#include "acpi.h"
+#include "pci.h"
+
+struct cxl_mem {
+     void __iomem *regs;
+};
+
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+     int pos;
+
+     pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+     if (!pos)
+             return 0;

drivers/cxl/mem.c line 29

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+
+     while (pos) {
+             u16 vendor, id;
+
+             pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+             pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+             if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+                     return pos;
+
+             pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);

This is good generic code and wouldn't cause much backport effort (even if needed to bring in a local copy), so perhaps make it a generic function and move to core PCI code?

Mind you I guess that can happen the 'second' time someone wants to find a DVSEC.

Quoted 23 lines
drivers/cxl/mem.c new file +23 -0
+     }
+
+     return 0;
+}
+
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+     struct device *dev = &pdev->dev;
+     struct cxl_mem *cxlm;
+     int rc, regloc;
+
+     rc = cxl_bus_prepared(pdev);
+     if (rc != 0) {
+             dev_err(dev, "failed to acquire interface\n");
+             return rc;
+     }
+
+     regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+     if (!regloc) {
+             dev_err(dev, "register location dvsec not found\n");
+             return -ENXIO;
+     }
+

drivers/cxl/mem.c line 62

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+     cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+     if (!cxlm)
+             return -ENOMEM;
+
+     return 0;
+}
+
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}

I'd bring this in only when needed in later patch.

Quoted 26 lines
drivers/cxl/mem.c new file +20 -0
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+     /* PCI class code for CXL.mem Type-3 Devices */
+     { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+       PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+     { /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+     .name                   = KBUILD_MODNAME,
+     .id_table               = cxl_mem_pci_tbl,
+     .probe                  = cxl_mem_probe,
+     .remove                 = cxl_mem_remove,
+};
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +0 -0
new file mode 100644
index 000000000000..beb03921e6da

drivers/cxl/pci.h line 9

Quoted 9 lines
drivers/cxl/pci.h new file +9 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL 0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC 0x23
+#define PCI_DVSEC_VENDOR_CXL 0x1E98

Hmm. The magic question of what to call a vendor ID that isn't a vendor ID but just a magic number that talks like a duck and quacks like a duck (for anyone wondering what I'm talking about, there is a nice bit of legal boilerplate on this in the CXL spec)

Quoted 4 lines

This name is definitely not accurate however.

PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), though possibly this is the only software visible use.

Finally working my way back through this review to make the changes. If 0x1E98 becomes visible to software somewhere else then this can become something like the following:

#define PCI_DVSEC_VENDOR_CXL PCI_UNIQUE_VALUE_CXL

...or whatever the generic name is, but this field per the specification is the DVSEC-vendor-id and calling it PCI_UNIQUE_VALUE_CXL does not have any basis in the spec.

I will rename it though to:

PCI_DVSEC_VENDOR_ID_CXL

...since include/linux/pci_ids.h includes the _ID_ part.

Quoted 7 lines

drivers/cxl/pci.h line 11

Quoted 4 lines
drivers/cxl/pci.h new file +2 -0
+#define PCI_DVSEC_VENDOR_OFFSET      0x4
+#define PCI_DVSEC_ID_OFFSET  0x8

Put a line break here perhaps and maybe a spec reference to where to find the various DVSEC IDs.

Ok.

Quoted 4 lines

drivers/cxl/pci.h line 12

Quoted 2 lines
drivers/cxl/pci.h new file +1 -0
+#define PCI_DVSEC_ID_CXL     0x0

That's definitely a confusing name as well.

Yeah, should be PCI_DVSEC_DEVICE_ID_CXL

Quoted 8 lines

PCI_DEVSEC_ID_CXL_DEVICE maybe?

Quoted 3 lines
drivers/cxl/pci.h new file +3 -0
+#define PCI_DVSEC_ID_CXL_REGLOC      0x8
+
+#endif /* __CXL_PCI_H__ */
#32 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

Quoted 228 lines · Dan Williams
Quoted 196 lines · Jonathan Cameron
Quoted 10 lines · On Tue, 10 Nov 2020 21:43:50 -0800 Ben Widawsky <[email protected]> wrote:

From: Dan Williams <[email protected]>

The CXL.mem protocol allows a device to act as a provider of "System RAM" and/or "Persistent Memory" that is fully coherent as if the memory was attached to the typical CPU memory controller.

The memory range exported by the device may optionally be described by the platform firmware memory map, or by infrastructure like LIBNVDIMM to provision persistent memory capacity from one, or more, CXL.mem devices.

Quoted 8 lines

A pre-requisite for Linux-managed memory-capacity provisioning is this cxl_mem driver that can speak the "type-3 mailbox" protocol.

For now just land the driver boiler-plate and fill it in with functionality in subsequent commits.

I've tried to avoid repeats, so mostly this is me moaning about naming!

Jonathan

Quoted 75 lines
---
 drivers/cxl/Kconfig  | 20 +++++++++++
 drivers/cxl/Makefile |  2 ++
 drivers/cxl/mem.c    | 82 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/pci.h    | 15 ++++++++
 4 files changed, 119 insertions(+)
 create mode 100644 drivers/cxl/mem.c
 create mode 100644 drivers/cxl/pci.h
drivers/cxl/Kconfig +20 -0
index dd724bd364df..15548f5c77ff 100644
@@ -27,4 +27,24 @@ config CXL_ACPI
        resources described by the CEDT (CXL Early Discovery Table)
 
        Say 'y' to enable CXL (Compute Express Link) drivers.
+
+config CXL_MEM
+        tristate "CXL.mem Device Support"
+        depends on PCI && CXL_BUS_PROVIDER != n
+        default m if CXL_BUS_PROVIDER
+        help
+          The CXL.mem protocol allows a device to act as a provider of
+          "System RAM" and/or "Persistent Memory" that is fully coherent
+          as if the memory was attached to the typical CPU memory
+          controller.
+
+          Say 'y/m' to enable a driver named "cxl_mem.ko" that will attach
+          to CXL.mem devices for configuration, provisioning, and health
+          monitoring, the so called "type-3 mailbox". Note, this driver
+          is required for dynamic provisioning of CXL.mem attached
+          memory, a pre-requisite for persistent memory support, but
+          devices that provide volatile memory may be fully described by
+          existing platform firmware memory enumeration.
+
+          If unsure say 'n'.
 endif
drivers/cxl/Makefile +2 -0
index d38cd34a2582..97fdffb00f2d 100644
@@ -1,5 +1,7 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
+obj-$(CONFIG_CXL_MEM) += cxl_mem.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
+cxl_mem-y := mem.o
drivers/cxl/mem.c new file +19 -0
new file mode 100644
index 000000000000..aa7d881fa47b
@@ -0,0 +1,82 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/io.h>
+#include "acpi.h"
+#include "pci.h"
+
+struct cxl_mem {
+     void __iomem *regs;
+};
+
+static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
+{
+     int pos;
+
+     pos = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DVSEC);
+     if (!pos)
+             return 0;

drivers/cxl/mem.c line 29

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+
+     while (pos) {
+             u16 vendor, id;
+
+             pci_read_config_word(pdev, pos + PCI_DVSEC_VENDOR_OFFSET, &vendor);
+             pci_read_config_word(pdev, pos + PCI_DVSEC_ID_OFFSET, &id);
+             if (vendor == PCI_DVSEC_VENDOR_CXL && dvsec == id)
+                     return pos;
+
+             pos = pci_find_next_ext_capability(pdev, pos, PCI_EXT_CAP_ID_DVSEC);

This is good generic code and wouldn't cause much backport effort (even if needed to bring in a local copy), so perhaps make it a generic function and move to core PCI code?

Mind you I guess that can happen the 'second' time someone wants to find a DVSEC.

Quoted 23 lines
drivers/cxl/mem.c new file +23 -0
+     }
+
+     return 0;
+}
+
+static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+     struct device *dev = &pdev->dev;
+     struct cxl_mem *cxlm;
+     int rc, regloc;
+
+     rc = cxl_bus_prepared(pdev);
+     if (rc != 0) {
+             dev_err(dev, "failed to acquire interface\n");
+             return rc;
+     }
+
+     regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
+     if (!regloc) {
+             dev_err(dev, "register location dvsec not found\n");
+             return -ENXIO;
+     }
+

drivers/cxl/mem.c line 62

Quoted 10 lines
drivers/cxl/mem.c new file +10 -0
+     cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
+     if (!cxlm)
+             return -ENOMEM;
+
+     return 0;
+}
+
+static void cxl_mem_remove(struct pci_dev *pdev)
+{
+}

I'd bring this in only when needed in later patch.

Quoted 26 lines
drivers/cxl/mem.c new file +20 -0
+
+static const struct pci_device_id cxl_mem_pci_tbl[] = {
+     /* PCI class code for CXL.mem Type-3 Devices */
+     { PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID, PCI_ANY_ID,
+       PCI_CLASS_MEMORY_CXL, 0xffffff, 0 },
+     { /* terminate list */ },
+};
+MODULE_DEVICE_TABLE(pci, cxl_mem_pci_tbl);
+
+static struct pci_driver cxl_mem_driver = {
+     .name                   = KBUILD_MODNAME,
+     .id_table               = cxl_mem_pci_tbl,
+     .probe                  = cxl_mem_probe,
+     .remove                 = cxl_mem_remove,
+};
+
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
+module_pci_driver(cxl_mem_driver);
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +0 -0
new file mode 100644
index 000000000000..beb03921e6da

drivers/cxl/pci.h line 9

Quoted 9 lines
drivers/cxl/pci.h new file +9 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL 0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC 0x23
+#define PCI_DVSEC_VENDOR_CXL 0x1E98

Hmm. The magic question of what to call a vendor ID that isn't a vendor ID but just a magic number that talks like a duck and quacks like a duck (for anyone wondering what I'm talking about, there is a nice bit of legal boilerplate on this in the CXL spec)

Quoted 4 lines

This name is definitely not accurate however.

PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), though possibly this is the only software visible use.

Finally working my way back through this review to make the changes. If 0x1E98 becomes visible to software somewhere else then this can become something like the following:

#define PCI_DVSEC_VENDOR_CXL PCI_UNIQUE_VALUE_CXL

...or whatever the generic name is, but this field per the specification is the DVSEC-vendor-id and calling it PCI_UNIQUE_VALUE_CXL does not have any basis in the spec.

I will rename it though to:

PCI_DVSEC_VENDOR_ID_CXL

...since include/linux/pci_ids.h includes the _ID_ part.

Quoted 7 lines

drivers/cxl/pci.h line 11

Quoted 4 lines
drivers/cxl/pci.h new file +2 -0
+#define PCI_DVSEC_VENDOR_OFFSET      0x4
+#define PCI_DVSEC_ID_OFFSET  0x8

Put a line break here perhaps and maybe a spec reference to where to find the various DVSEC IDs.

Quoted 8 lines

Ok.

Quoted 4 lines

drivers/cxl/pci.h line 12

Quoted 2 lines
drivers/cxl/pci.h new file +1 -0
+#define PCI_DVSEC_ID_CXL     0x0

That's definitely a confusing name as well.

Yeah, should be PCI_DVSEC_DEVICE_ID_CXL

Actually, no, the spec calls this the "DVSEC id" so PCI_DVSEC_ID_CXL seems suitable to me. This is from:

Table 126. PCI Express DVSEC Register Settings for CXL Device

In the CXL 2.0 Specification.

#33 lore raw

Re: [RFC PATCH 3/9] cxl/mem: Add a driver for the type-3 mailbox

...

Quoted 31 lines
Quoted 31 lines
Quoted 22 lines
Quoted 7 lines
drivers/cxl/mem.c new file +1 -0
+MODULE_IMPORT_NS(CXL);
drivers/cxl/pci.h new file +0 -0
new file mode 100644
index 000000000000..beb03921e6da

drivers/cxl/pci.h line 9

Quoted 9 lines
drivers/cxl/pci.h new file +9 -0
@@ -0,0 +1,15 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_PCI_H__
+#define __CXL_PCI_H__
+
+#define PCI_CLASS_MEMORY_CXL 0x050210
+
+#define PCI_EXT_CAP_ID_DVSEC 0x23
+#define PCI_DVSEC_VENDOR_CXL 0x1E98  

Hmm. The magic question of what to call a vendor ID that isn't a vendor ID but just a magic number that talks like a duck and quacks like a duck (for anyone wondering what I'm talking about, there is a nice bit of legal boilerplate on this in the CXL spec)

Quoted 4 lines

This name is definitely not accurate however.

PCI_UNIQUE_VALUE_CXL maybe? It is used for other things than DVSEC (VDMs etc), though possibly this is the only software visible use.

Finally working my way back through this review to make the changes. If 0x1E98 becomes visible to software somewhere else then this can become something like the following:

Quoted 5 lines
Quoted 5 lines

#define PCI_DVSEC_VENDOR_CXL PCI_UNIQUE_VALUE_CXL

...or whatever the generic name is, but this field per the specification is the DVSEC-vendor-id and calling it PCI_UNIQUE_VALUE_CXL does not have any basis in the spec.

There is a big statement about it as a footnote to 3.1.2 in CXL 2.0 "The Unique Value that is provided in this specification for use in ... Designated Vendor Specific Extended Capabilities.." And for extra amusement in the "Notice Regarding PCI-SIG Unique Value" that forms part of the click through
https://www.computeexpresslink.org/download-the-specification
(that's the only use of "PCI-SIG Unique Value" that Google finds
but I know of one other similar statement)

However, I agree it's being used in DVSEC field only (from software point of view) so fair enough to name it after where it is used rather than what it is.

Quoted 22 lines
Quoted 22 lines

I will rename it though to:

PCI_DVSEC_VENDOR_ID_CXL

...since include/linux/pci_ids.h includes the _ID_ part.

Quoted 7 lines

drivers/cxl/pci.h line 11

Quoted 4 lines
drivers/cxl/pci.h new file +2 -0
+#define PCI_DVSEC_VENDOR_OFFSET      0x4
+#define PCI_DVSEC_ID_OFFSET  0x8  

Put a line break here perhaps and maybe a spec reference to where to find the various DVSEC IDs.

Ok.

Quoted 4 lines

drivers/cxl/pci.h line 12

Quoted 2 lines
drivers/cxl/pci.h new file +1 -0
+#define PCI_DVSEC_ID_CXL     0x0  

That's definitely a confusing name as well.

Quoted 8 lines
Quoted 1 line

Yeah, should be PCI_DVSEC_DEVICE_ID_CXL

Actually, no, the spec calls this the "DVSEC id" so PCI_DVSEC_ID_CXL seems suitable to me. This is from:

Table 126. PCI Express DVSEC Register Settings for CXL Device

In the CXL 2.0 Specification.

The DVSEC ID naming is straight from the PCI spec so that part is fine, my issue is this is one of a whole bunch of CXL related DVSEC ID so it needs a more specific name.

PCI_DVSEC_ID_CXL_DEVICE would work in line with table 124.

I'm not that bothered though.

Jonathan

#34 lore raw

RFC PATCH 4/9 cxl/mem: Map memory device registers

In reply to Ben Widawsky

All the necessary bits are initialized in order to find and map the register space for CXL Memory Devices. This is accomplished by using the Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to use, and how much of an offset from that BAR should be added.

If the memory device registers are found and mapped a new internal data structure tracking device state is allocated.

---
 drivers/cxl/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/pci.h |  6 +++++
 2 files changed, 69 insertions(+), 5 deletions(-)
drivers/cxl/mem.c +63 -5
index aa7d881fa47b..8d9b9ab6c5ea 100644
@@ -7,9 +7,49 @@
 #include "pci.h"
 
 struct cxl_mem {
+	struct pci_dev *pdev;
 	void __iomem *regs;
 };
 
+static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	void __iomem *regs;
+	u64 offset;
+	u8 bar;
+	int rc;
+
+	offset = ((u64)reg_hi << 32) | (reg_lo & 0xffff0000);
+	bar = reg_lo & 0x7;
+
+	/* Basic sanity check that BAR is big enough */
+	if (pci_resource_len(pdev, bar) < offset) {
+		dev_err(dev, "bar%d: %pr: too small (offset: %#llx)\n",
+				bar, &pdev->resource[bar], (unsigned long long) offset);
+		return ERR_PTR(-ENXIO);
+	}
+
+	rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev));
+	if (rc != 0) {
+		dev_err(dev, "failed to map registers\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm) {
+		dev_err(dev, "No memory available\n");
+		return ERR_PTR(-ENOMEM);
+	}
+
+	regs = pcim_iomap_table(pdev)[bar];
+	cxlm->pdev = pdev;
+	cxlm->regs = regs + offset;
+
+	dev_dbg(dev, "Mapped CXL Memory Device resource\n");
+	return cxlm;
+}
+
 static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 {
 	int pos;
@@ -34,9 +74,9 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
+	struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
 	struct device *dev = &pdev->dev;
-	struct cxl_mem *cxlm;
-	int rc, regloc;
+	int rc, regloc, i;
 
 	rc = cxl_bus_prepared(pdev);
 	if (rc != 0) {
@@ -44,15 +84,33 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return rc;
 	}
 
+	rc = pcim_enable_device(pdev);
+	if (rc)
+		return rc;
+
 	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
 	if (!regloc) {
 		dev_err(dev, "register location dvsec not found\n");
 		return -ENXIO;
 	}
+	regloc += 0xc; /* Skip DVSEC + reserved fields */
+
+	for (i = regloc; i < regloc + 0x24; i += 8) {
+		u32 reg_lo, reg_hi;
+
+		pci_read_config_dword(pdev, i, &reg_lo);
+		pci_read_config_dword(pdev, i + 4, &reg_hi);
+
+		if (CXL_REGLOG_IS_MEMDEV(reg_lo)) {
+			cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
+			break;
+		}
+	}
+
+	if (IS_ERR(cxlm))
+		return -ENXIO;
 
-	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
-	if (!cxlm)
-		return -ENOMEM;
+	pci_set_drvdata(pdev, cxlm);
 
 	return 0;
 }
drivers/cxl/pci.h +6 -0
index beb03921e6da..be87f62e9132 100644
@@ -12,4 +12,10 @@
 #define PCI_DVSEC_ID_CXL	0x0
 #define PCI_DVSEC_ID_CXL_REGLOC	0x8
 
+#define CXL_REGLOG_RBI_EMPTY 0
+#define CXL_REGLOG_RBI_COMPONENT 1
+#define CXL_REGLOG_RBI_VIRT 2
+#define CXL_REGLOG_RBI_MEMDEV 3
+#define CXL_REGLOG_IS_MEMDEV(x) ((((x) >> 8) & 0xff) == CXL_REGLOG_RBI_MEMDEV)
+
 #endif /* __CXL_PCI_H__ */
Signature · 1 line
2.29.2
#35 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

Quoted 4 lines · Ben Widawsky

All the necessary bits are initialized in order to find and map the register space for CXL Memory Devices. This is accomplished by using the Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to use, and how much of an offset from that BAR should be added.

"Initialize the necessary bits ..." to use the usual imperative sentence structure, as you did in the subject.

Quoted 2 lines

If the memory device registers are found and mapped a new internal data structure tracking device state is allocated.

"Allocate device state if we find device registers" or similar.

Quoted 24 lines
---
 drivers/cxl/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/pci.h |  6 +++++
 2 files changed, 69 insertions(+), 5 deletions(-)
drivers/cxl/mem.c +7 -0
index aa7d881fa47b..8d9b9ab6c5ea 100644
@@ -7,9 +7,49 @@
 #include "pci.h"
 
 struct cxl_mem {
+	struct pci_dev *pdev;
 	void __iomem *regs;
 };
 
+static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	void __iomem *regs;
+	u64 offset;

drivers/cxl/mem.c line 29

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	u8 bar;
+	int rc;
+
+	offset = ((u64)reg_hi << 32) | (reg_lo & 0xffff0000);
+	bar = reg_lo & 0x7;
+
+	/* Basic sanity check that BAR is big enough */
+	if (pci_resource_len(pdev, bar) < offset) {
+		dev_err(dev, "bar%d: %pr: too small (offset: %#llx)\n",
+				bar, &pdev->resource[bar], (unsigned long long) offset);

s/bar/BAR/

Quoted 27 lines
drivers/cxl/mem.c +23 -0
+		return ERR_PTR(-ENXIO);
+	}
+
+	rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev));
+	if (rc != 0) {
+		dev_err(dev, "failed to map registers\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm) {
+		dev_err(dev, "No memory available\n");
+		return ERR_PTR(-ENOMEM);
+	}
+
+	regs = pcim_iomap_table(pdev)[bar];
+	cxlm->pdev = pdev;
+	cxlm->regs = regs + offset;
+
+	dev_dbg(dev, "Mapped CXL Memory Device resource\n");
+	return cxlm;
+}
+
 static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 {
 	int pos;

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 38

Quoted 6 lines
drivers/cxl/mem.c +1 -1
@@ -34,9 +74,9 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
+	struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
 	struct device *dev = &pdev->dev;
-	struct cxl_mem *cxlm;

The order was better before ("dev", then "clxm"). Oh, I suppose this is a "reverse Christmas tree" thing.

Quoted 24 lines
drivers/cxl/mem.c +11 -1
-	int rc, regloc;
+	int rc, regloc, i;
 
 	rc = cxl_bus_prepared(pdev);
 	if (rc != 0) {
@@ -44,15 +84,33 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return rc;
 	}
 
+	rc = pcim_enable_device(pdev);
+	if (rc)
+		return rc;
+
 	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
 	if (!regloc) {
 		dev_err(dev, "register location dvsec not found\n");
 		return -ENXIO;
 	}
+	regloc += 0xc; /* Skip DVSEC + reserved fields */
+
+	for (i = regloc; i < regloc + 0x24; i += 8) {
+		u32 reg_lo, reg_hi;
+
+		pci_read_config_dword(pdev, i, &reg_lo);

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 111

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+		pci_read_config_dword(pdev, i + 4, &reg_hi);
+
+		if (CXL_REGLOG_IS_MEMDEV(reg_lo)) {
+			cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
+			break;
+		}
+	}
+
+	if (IS_ERR(cxlm))
+		return -ENXIO;

I think this would be easier to read if cxl_mem_create() returned NULL on failure (it prints error messages and we throw away -ENXIO/-ENOMEM distinction here anyway) so you could do:

  struct cxl_mem *cxlm = NULL;

  for (...) {
    if (...) {
      cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
      break;
    }
  }

if (!cxlm)
return -ENXIO; /* -ENODEV might be more natural? */

Quoted 12 lines
drivers/cxl/mem.c +1 -3
-	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
-	if (!cxlm)
-		return -ENOMEM;
+	pci_set_drvdata(pdev, cxlm);
 
 	return 0;
 }
drivers/cxl/pci.h +0 -0
index beb03921e6da..be87f62e9132 100644

drivers/cxl/pci.h line 18

Quoted 7 lines
drivers/cxl/pci.h +4 -0
@@ -12,4 +12,10 @@
 #define PCI_DVSEC_ID_CXL	0x0
 #define PCI_DVSEC_ID_CXL_REGLOC	0x8
 
+#define CXL_REGLOG_RBI_EMPTY 0
+#define CXL_REGLOG_RBI_COMPONENT 1
+#define CXL_REGLOG_RBI_VIRT 2
+#define CXL_REGLOG_RBI_MEMDEV 3

Maybe line these values up.

drivers/cxl/pci.h line 19

Quoted 1 line
drivers/cxl/pci.h +1 -0
+#define CXL_REGLOG_IS_MEMDEV(x) ((((x) >> 8) & 0xff) == CXL_REGLOG_RBI_MEMDEV)

If these are only needed in cxl/mem.c, they could go there. Do you expect code outside of drivers/cxl to need these?

Quoted 4 lines
drivers/cxl/pci.h +0 -0
 #endif /* __CXL_PCI_H__ */
Signature · 1 line
2.29.2
#36 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

Quoted 79 lines · Bjorn Helgaas
Quoted 4 lines · Ben Widawsky

All the necessary bits are initialized in order to find and map the register space for CXL Memory Devices. This is accomplished by using the Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to use, and how much of an offset from that BAR should be added.

"Initialize the necessary bits ..." to use the usual imperative sentence structure, as you did in the subject.

Quoted 2 lines

If the memory device registers are found and mapped a new internal data structure tracking device state is allocated.

"Allocate device state if we find device registers" or similar.

Quoted 24 lines
---
 drivers/cxl/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/pci.h |  6 +++++
 2 files changed, 69 insertions(+), 5 deletions(-)
drivers/cxl/mem.c +7 -0
index aa7d881fa47b..8d9b9ab6c5ea 100644
@@ -7,9 +7,49 @@
 #include "pci.h"
 
 struct cxl_mem {
+	struct pci_dev *pdev;
 	void __iomem *regs;
 };
 
+static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	void __iomem *regs;
+	u64 offset;

drivers/cxl/mem.c line 29

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	u8 bar;
+	int rc;
+
+	offset = ((u64)reg_hi << 32) | (reg_lo & 0xffff0000);
+	bar = reg_lo & 0x7;
+
+	/* Basic sanity check that BAR is big enough */
+	if (pci_resource_len(pdev, bar) < offset) {
+		dev_err(dev, "bar%d: %pr: too small (offset: %#llx)\n",
+				bar, &pdev->resource[bar], (unsigned long long) offset);

s/bar/BAR/

Quoted 28 lines
drivers/cxl/mem.c +23 -0
+		return ERR_PTR(-ENXIO);
+	}
+
+	rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev));
+	if (rc != 0) {
+		dev_err(dev, "failed to map registers\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm) {
+		dev_err(dev, "No memory available\n");
+		return ERR_PTR(-ENOMEM);
+	}
+
+	regs = pcim_iomap_table(pdev)[bar];
+	cxlm->pdev = pdev;
+	cxlm->regs = regs + offset;
+
+	dev_dbg(dev, "Mapped CXL Memory Device resource\n");
+	return cxlm;
+}
+
 static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 {
 	int pos;
@@ -34,9 +74,9 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 
Quoted 9 lines

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 38

Quoted 5 lines
drivers/cxl/mem.c +1 -1
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
+	struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
 	struct device *dev = &pdev->dev;
-	struct cxl_mem *cxlm;

The order was better before ("dev", then "clxm"). Oh, I suppose this is a "reverse Christmas tree" thing.

I don't actually care either way as long as it's consistent. I tend to do reverse Christmas tree for no particular reason.

Quoted 48 lines
Quoted 24 lines
drivers/cxl/mem.c +11 -1
-	int rc, regloc;
+	int rc, regloc, i;
 
 	rc = cxl_bus_prepared(pdev);
 	if (rc != 0) {
@@ -44,15 +84,33 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return rc;
 	}
 
+	rc = pcim_enable_device(pdev);
+	if (rc)
+		return rc;
+
 	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
 	if (!regloc) {
 		dev_err(dev, "register location dvsec not found\n");
 		return -ENXIO;
 	}
+	regloc += 0xc; /* Skip DVSEC + reserved fields */
+
+	for (i = regloc; i < regloc + 0x24; i += 8) {
+		u32 reg_lo, reg_hi;
+
+		pci_read_config_dword(pdev, i, &reg_lo);

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 111

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+		pci_read_config_dword(pdev, i + 4, &reg_hi);
+
+		if (CXL_REGLOG_IS_MEMDEV(reg_lo)) {
+			cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
+			break;
+		}
+	}
+
+	if (IS_ERR(cxlm))
+		return -ENXIO;

I think this would be easier to read if cxl_mem_create() returned NULL on failure (it prints error messages and we throw away -ENXIO/-ENOMEM distinction here anyway) so you could do:

  struct cxl_mem *cxlm = NULL;

  for (...) {
    if (...) {
      cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
      break;
    }
  }
Quoted 3 lines

if (!cxlm)
return -ENXIO; /* -ENODEV might be more natural? */

I agree on both counts. Both of these came from Dan, so I will let him explain.

Quoted 22 lines
Quoted 12 lines
drivers/cxl/mem.c +1 -3
-	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
-	if (!cxlm)
-		return -ENOMEM;
+	pci_set_drvdata(pdev, cxlm);
 
 	return 0;
 }
drivers/cxl/pci.h +0 -0
index beb03921e6da..be87f62e9132 100644

drivers/cxl/pci.h line 18

Quoted 7 lines
drivers/cxl/pci.h +4 -0
@@ -12,4 +12,10 @@
 #define PCI_DVSEC_ID_CXL	0x0
 #define PCI_DVSEC_ID_CXL_REGLOC	0x8
 
+#define CXL_REGLOG_RBI_EMPTY 0
+#define CXL_REGLOG_RBI_COMPONENT 1
+#define CXL_REGLOG_RBI_VIRT 2
+#define CXL_REGLOG_RBI_MEMDEV 3

Maybe line these values up.

Quoted 4 lines

drivers/cxl/pci.h line 19

Quoted 1 line
drivers/cxl/pci.h +1 -0
+#define CXL_REGLOG_IS_MEMDEV(x) ((((x) >> 8) & 0xff) == CXL_REGLOG_RBI_MEMDEV)

If these are only needed in cxl/mem.c, they could go there. Do you expect code outside of drivers/cxl to need these?

Will do.

I'll suck in everything else as they seem like improvements.

Quoted 5 lines
Quoted 5 lines
drivers/cxl/pci.h +0 -0
 #endif /* __CXL_PCI_H__ */
Signature · 1 line
2.29.2
#37 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

Quoted 91 lines · Ben Widawsky
Quoted 79 lines · Bjorn Helgaas
Quoted 4 lines · Ben Widawsky

All the necessary bits are initialized in order to find and map the register space for CXL Memory Devices. This is accomplished by using the Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to use, and how much of an offset from that BAR should be added.

"Initialize the necessary bits ..." to use the usual imperative sentence structure, as you did in the subject.

Quoted 2 lines

If the memory device registers are found and mapped a new internal data structure tracking device state is allocated.

"Allocate device state if we find device registers" or similar.

Quoted 24 lines
---
 drivers/cxl/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/pci.h |  6 +++++
 2 files changed, 69 insertions(+), 5 deletions(-)
drivers/cxl/mem.c +7 -0
index aa7d881fa47b..8d9b9ab6c5ea 100644
@@ -7,9 +7,49 @@
 #include "pci.h"
 
 struct cxl_mem {
+   struct pci_dev *pdev;
    void __iomem *regs;
 };
 
+static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
+{
+   struct device *dev = &pdev->dev;
+   struct cxl_mem *cxlm;
+   void __iomem *regs;
+   u64 offset;

drivers/cxl/mem.c line 29

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+   u8 bar;
+   int rc;
+
+   offset = ((u64)reg_hi << 32) | (reg_lo & 0xffff0000);
+   bar = reg_lo & 0x7;
+
+   /* Basic sanity check that BAR is big enough */
+   if (pci_resource_len(pdev, bar) < offset) {
+           dev_err(dev, "bar%d: %pr: too small (offset: %#llx)\n",
+                           bar, &pdev->resource[bar], (unsigned long long) offset);

s/bar/BAR/

Quoted 28 lines
drivers/cxl/mem.c +23 -0
+           return ERR_PTR(-ENXIO);
+   }
+
+   rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev));
+   if (rc != 0) {
+           dev_err(dev, "failed to map registers\n");
+           return ERR_PTR(-ENXIO);
+   }
+
+   cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
+   if (!cxlm) {
+           dev_err(dev, "No memory available\n");
+           return ERR_PTR(-ENOMEM);
+   }
+
+   regs = pcim_iomap_table(pdev)[bar];
+   cxlm->pdev = pdev;
+   cxlm->regs = regs + offset;
+
+   dev_dbg(dev, "Mapped CXL Memory Device resource\n");
+   return cxlm;
+}
+
 static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 {
    int pos;
@@ -34,9 +74,9 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 
Quoted 9 lines

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 38

Quoted 5 lines
drivers/cxl/mem.c +1 -1
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
+   struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
    struct device *dev = &pdev->dev;
-   struct cxl_mem *cxlm;

The order was better before ("dev", then "clxm"). Oh, I suppose this is a "reverse Christmas tree" thing.

Quoted 2 lines

I don't actually care either way as long as it's consistent. I tend to do reverse Christmas tree for no particular reason.

Yeah, reverse Christmas tree for no particular reason.

Quoted 49 lines
Quoted 49 lines
Quoted 25 lines
drivers/cxl/mem.c +11 -1
-   int rc, regloc;
+   int rc, regloc, i;
 
    rc = cxl_bus_prepared(pdev);
    if (rc != 0) {
@@ -44,15 +84,33 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
            return rc;
    }
 
+   rc = pcim_enable_device(pdev);
+   if (rc)
+           return rc;
+
    regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
    if (!regloc) {
            dev_err(dev, "register location dvsec not found\n");
            return -ENXIO;
    }
+   regloc += 0xc; /* Skip DVSEC + reserved fields */
+
+   for (i = regloc; i < regloc + 0x24; i += 8) {
+           u32 reg_lo, reg_hi;
+
+           pci_read_config_dword(pdev, i, &reg_lo);

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 111

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+           pci_read_config_dword(pdev, i + 4, &reg_hi);
+
+           if (CXL_REGLOG_IS_MEMDEV(reg_lo)) {
+                   cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
+                   break;
+           }
+   }
+
+   if (IS_ERR(cxlm))
+           return -ENXIO;

I think this would be easier to read if cxl_mem_create() returned NULL on failure (it prints error messages and we throw away -ENXIO/-ENOMEM distinction here anyway) so you could do:

  struct cxl_mem *cxlm = NULL;

  for (...) {
    if (...) {
      cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
      break;
    }
  }
Quoted 5 lines
Quoted 3 lines

if (!cxlm)
return -ENXIO; /* -ENODEV might be more natural? */

I agree on both counts. Both of these came from Dan, so I will let him explain.

I'm not attached to differentiating -ENOMEM from -ENXIO and am ok to drop the ERR_PTR() return. I do tend to use -ENXIO for failure to perform an initialization action vs failure to even find the device, but if -ENODEV seems more idiomatic to Bjorn, I won't argue.

#38 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

Quoted 10 lines · Dan Williams
Quoted 9 lines · Ben Widawsky
Quoted 8 lines · Bjorn Helgaas
Quoted 5 lines · Ben Widawsky

static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) {

drivers/cxl/mem.c +1 -1
+   struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
    struct device *dev = &pdev->dev;
-   struct cxl_mem *cxlm;
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The order was better before ("dev", then "clxm"). Oh, I suppose this is a "reverse Christmas tree" thing.

I don't actually care either way as long as it's consistent. I tend to do reverse Christmas tree for no particular reason.

Yeah, reverse Christmas tree for no particular reason.

FWIW, the usual drivers/pci style is to order the decls in the order the variables are used in the code. But this isn't drivers/pci, so it's up to you. I only noticed because changing the order made the diff bigger than it needed to be.

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines

I think this would be easier to read if cxl_mem_create() returned NULL on failure (it prints error messages and we throw away -ENXIO/-ENOMEM distinction here anyway) so you could do:

  struct cxl_mem *cxlm = NULL;

  for (...) {
    if (...) {
      cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
      break;
    }
  }

if (!cxlm)
return -ENXIO; /* -ENODEV might be more natural? */

Quoted 6 lines
Quoted 1 line

I agree on both counts. Both of these came from Dan, so I will let him explain.

I'm not attached to differentiating -ENOMEM from -ENXIO and am ok to drop the ERR_PTR() return. I do tend to use -ENXIO for failure to perform an initialization action vs failure to even find the device, but if -ENODEV seems more idiomatic to Bjorn, I won't argue.

-ENXIO is fine with me. I just don't see it as often so I don't really know what it is.

Bjorn

#39 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

Quoted 49 lines · Bjorn Helgaas
Quoted 10 lines · Dan Williams
Quoted 9 lines · Ben Widawsky
Quoted 8 lines · Bjorn Helgaas
Quoted 5 lines · Ben Widawsky

static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) {

drivers/cxl/mem.c +1 -1
+   struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
    struct device *dev = &pdev->dev;
-   struct cxl_mem *cxlm;
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

The order was better before ("dev", then "clxm"). Oh, I suppose this is a "reverse Christmas tree" thing.

I don't actually care either way as long as it's consistent. I tend to do reverse Christmas tree for no particular reason.

Yeah, reverse Christmas tree for no particular reason.

FWIW, the usual drivers/pci style is to order the decls in the order the variables are used in the code. But this isn't drivers/pci, so it's up to you. I only noticed because changing the order made the diff bigger than it needed to be.

Quoted 17 lines
Quoted 17 lines
Quoted 16 lines

I think this would be easier to read if cxl_mem_create() returned NULL on failure (it prints error messages and we throw away -ENXIO/-ENOMEM distinction here anyway) so you could do:

  struct cxl_mem *cxlm = NULL;

  for (...) {
    if (...) {
      cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
      break;
    }
  }

if (!cxlm)
return -ENXIO; /* -ENODEV might be more natural? */

Quoted 6 lines
Quoted 1 line

I agree on both counts. Both of these came from Dan, so I will let him explain.

I'm not attached to differentiating -ENOMEM from -ENXIO and am ok to drop the ERR_PTR() return. I do tend to use -ENXIO for failure to perform an initialization action vs failure to even find the device, but if -ENODEV seems more idiomatic to Bjorn, I won't argue.

Quoted 4 lines

-ENXIO is fine with me. I just don't see it as often so I don't really know what it is.

Bjorn

Dan, Bjorn, I did a fairly randomized look at various probe functions and ENODEV seems to be more common. My sort of historical use has been
- ENODEV: General, couldn't establish device presence
- ENXIO: Device was there but something is totally misconfigured
- E*: A matching errno for exactly what went wrong

My question though is, would it be useful to propagate the error up through probe?

#40 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

On Mon, Nov 23, 2020 at 11:20 AM Ben Widawsky <[email protected]> wrote:
[..]

Quoted 5 lines
Quoted 4 lines

-ENXIO is fine with me. I just don't see it as often so I don't really know what it is.

Bjorn

Quoted 8 lines

Dan, Bjorn, I did a fairly randomized look at various probe functions and ENODEV seems to be more common. My sort of historical use has been
- ENODEV: General, couldn't establish device presence
- ENXIO: Device was there but something is totally misconfigured
- E*: A matching errno for exactly what went wrong

My question though is, would it be useful to propagate the error up through probe?

The error from probe becomes the modprobe exit code, or the write to the 'bind' attribute errno. So, it's a choice between "No such device or address", or "No such device". The "or address" mention makes a small bit more sense to me since the device is obviously present as it is visible in lspci, but either error code clearly indicates a driver problem so ENODEV is fine.

For the other error codes I think it would be confusing to return something like EINVAL from probe as that would be mistaken as an invalid argument to the modprobe without stracing to see that it came from the result of a sysfs write

#41 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

Quoted 23 lines · Dan Williams

On Mon, Nov 23, 2020 at 11:20 AM Ben Widawsky <[email protected]> wrote:
[..]

Quoted 5 lines
Quoted 4 lines

-ENXIO is fine with me. I just don't see it as often so I don't really know what it is.

Bjorn

Quoted 8 lines

Dan, Bjorn, I did a fairly randomized look at various probe functions and ENODEV seems to be more common. My sort of historical use has been
- ENODEV: General, couldn't establish device presence
- ENXIO: Device was there but something is totally misconfigured
- E*: A matching errno for exactly what went wrong

My question though is, would it be useful to propagate the error up through probe?

The error from probe becomes the modprobe exit code, or the write to the 'bind' attribute errno. So, it's a choice between "No such device or address", or "No such device". The "or address" mention makes a small bit more sense to me since the device is obviously present as it is visible in lspci, but either error code clearly indicates a driver problem so ENODEV is fine.

Quoted 4 lines

For the other error codes I think it would be confusing to return something like EINVAL from probe as that would be mistaken as an invalid argument to the modprobe without stracing to see that it came from the result of a sysfs write

Currently in this path there are 2 general reasons for failure:
1. Driver internal problem, ENOMEM or some such.
2. Device problem (the memory device capability isn't present).

I think I'll return ENODEV for the former and ENXIO for the latter. If that sounds good to everyone else.

#42 lore raw

Re: [RFC PATCH 4/9] cxl/mem: Map memory device registers

In reply to Ben Widawsky
Quoted 87 lines · On Tue, 10 Nov 2020 21:43:51 -0800 Ben Widawsky <[email protected]> wrote:

All the necessary bits are initialized in order to find and map the register space for CXL Memory Devices. This is accomplished by using the Register Locator DVSEC (CXL 2.0 - 8.1.9.1) to determine which PCI BAR to use, and how much of an offset from that BAR should be added.

If the memory device registers are found and mapped a new internal data structure tracking device state is allocated.

---
 drivers/cxl/mem.c | 68 +++++++++++++++++++++++++++++++++++++++++++----
 drivers/cxl/pci.h |  6 +++++
 2 files changed, 69 insertions(+), 5 deletions(-)
drivers/cxl/mem.c +45 -2
index aa7d881fa47b..8d9b9ab6c5ea 100644
@@ -7,9 +7,49 @@
 #include "pci.h"
 
 struct cxl_mem {
+	struct pci_dev *pdev;
 	void __iomem *regs;
 };
 
+static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
+{
+	struct device *dev = &pdev->dev;
+	struct cxl_mem *cxlm;
+	void __iomem *regs;
+	u64 offset;
+	u8 bar;
+	int rc;
+
+	offset = ((u64)reg_hi << 32) | (reg_lo & 0xffff0000);
+	bar = reg_lo & 0x7;
+
+	/* Basic sanity check that BAR is big enough */
+	if (pci_resource_len(pdev, bar) < offset) {
+		dev_err(dev, "bar%d: %pr: too small (offset: %#llx)\n",
+				bar, &pdev->resource[bar], (unsigned long long) offset);
+		return ERR_PTR(-ENXIO);
+	}
+
+	rc = pcim_iomap_regions(pdev, 1 << bar, pci_name(pdev));
+	if (rc != 0) {
+		dev_err(dev, "failed to map registers\n");
+		return ERR_PTR(-ENXIO);
+	}
+
+	cxlm = devm_kzalloc(&pdev->dev, sizeof(*cxlm), GFP_KERNEL);
+	if (!cxlm) {
+		dev_err(dev, "No memory available\n");
+		return ERR_PTR(-ENOMEM);
+	}
+
+	regs = pcim_iomap_table(pdev)[bar];
+	cxlm->pdev = pdev;
+	cxlm->regs = regs + offset;
+
+	dev_dbg(dev, "Mapped CXL Memory Device resource\n");
+	return cxlm;
+}
+
 static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 {
 	int pos;
@@ -34,9 +74,9 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 
 static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 {
+	struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
 	struct device *dev = &pdev->dev;
-	struct cxl_mem *cxlm;
-	int rc, regloc;
+	int rc, regloc, i;
 
 	rc = cxl_bus_prepared(pdev);
 	if (rc != 0) {
@@ -44,15 +84,33 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 		return rc;
 	}
 
+	rc = pcim_enable_device(pdev);
+	if (rc)
+		return rc;

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 99

Quoted 10 lines
drivers/cxl/mem.c +5 -0
+
 	regloc = cxl_mem_dvsec(pdev, PCI_DVSEC_ID_CXL_REGLOC);
 	if (!regloc) {
 		dev_err(dev, "register location dvsec not found\n");
 		return -ENXIO;
 	}
+	regloc += 0xc; /* Skip DVSEC + reserved fields */
+
+	for (i = regloc; i < regloc + 0x24; i += 8) {
+		u32 reg_lo, reg_hi;

Hmm. That "register offset low" naming in the spec is just designed to confuse given lots of other things packed in the register.
Perhaps a comment here to say it contains other information?
Also possibly some docs for cxl_mem_create to make the same point there.

Quoted 7 lines
drivers/cxl/mem.c +7 -0
+
+		pci_read_config_dword(pdev, i, &reg_lo);
+		pci_read_config_dword(pdev, i + 4, &reg_hi);
+
+		if (CXL_REGLOG_IS_MEMDEV(reg_lo)) {
+			cxlm = cxl_mem_create(pdev, reg_lo, reg_hi);
+			break;

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 113

Quoted 10 lines
drivers/cxl/mem.c +6 -3
+		}
+	}
+
+	if (IS_ERR(cxlm))
+		return -ENXIO;
 
-	cxlm = devm_kzalloc(dev, sizeof(*cxlm), GFP_KERNEL);
-	if (!cxlm)
-		return -ENOMEM;
+	pci_set_drvdata(pdev, cxlm);

I could be wrong but don't think this is used yet. I'd prefer to see it introduced only when it is. Makes it easy to match up without having to search back in earlier patches.

drivers/cxl/pci.h line 15

Quoted 12 lines
drivers/cxl/mem.c +0 -0
 	return 0;
 }
drivers/cxl/pci.h +1 -0
index beb03921e6da..be87f62e9132 100644
@@ -12,4 +12,10 @@
 #define PCI_DVSEC_ID_CXL	0x0
 #define PCI_DVSEC_ID_CXL_REGLOC	0x8
 
+#define CXL_REGLOG_RBI_EMPTY 0

As in the QEMU patches, please add a comment on what RBI means here. It's non obvious even just after you've read through the spec!

Quoted 6 lines
drivers/cxl/pci.h +5 -0
+#define CXL_REGLOG_RBI_COMPONENT 1
+#define CXL_REGLOG_RBI_VIRT 2
+#define CXL_REGLOG_RBI_MEMDEV 3
+#define CXL_REGLOG_IS_MEMDEV(x) ((((x) >> 8) & 0xff) == CXL_REGLOG_RBI_MEMDEV)
+
 #endif /* __CXL_PCI_H__ */
#43 lore raw

RFC PATCH 5/9 cxl/mem: Find device capabilities

In reply to Ben Widawsky

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +89 -0
new file mode 100644
index 000000000000..02858ae63d6d
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+
+#ifndef __CXL_H__
+#define __CXL_H__
+
+/* Device */
+#define CXLDEV_CAP_ARRAY_REG 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff)
+#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff)
+
+#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1
+#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2
+#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3
+#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000
+
+/* Mailbox */
+#define CXLDEV_MB_CAPS 0x00
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)
+#define CXLDEV_MB_CTRL 0x04
+#define CXLDEV_MB_CMD 0x08
+#define CXLDEV_MB_STATUS 0x10
+#define CXLDEV_MB_BG_CMD_STATUS 0x18
+
+struct cxl_mem {
+	struct pci_dev *pdev;
+	void __iomem *regs;
+
+	/* Cap 0000h */
+	struct {
+		void __iomem *regs;
+	} status;
+
+	/* Cap 0002h */
+	struct {
+		void __iomem *regs;
+		size_t payload_size;
+	} mbox;
+
+	/* Cap 0040h */
+	struct {
+		void __iomem *regs;
+	} mem;
+};
+
+#define cxl_reg(type)                                                          \
+	static inline void cxl_write_##type##_reg32(struct cxl_mem *cxlm,      \
+						    u32 reg, u32 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		writel(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline void cxl_write_##type##_reg64(struct cxl_mem *cxlm,      \
+						    u32 reg, u64 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		writeq(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline u32 cxl_read_##type##_reg32(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		return readl(reg_addr + reg);                                  \
+	}                                                                      \
+	static inline u64 cxl_read_##type##_reg64(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		return readq(reg_addr + reg);                                  \
+	}
+
+cxl_reg(status)
+cxl_reg(mbox)
+
+static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = READ_ONCE(cxlm->regs);
+
+	return readl(reg_addr + reg);
+}
+
+static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = READ_ONCE(cxlm->regs);
+
+	return readq(reg_addr + reg);
+}
+#endif /* __CXL_H__ */
drivers/cxl/mem.c +54 -4

71 lines not on this page · raw message

Signature · 1 line
2.29.2
#44 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

Quoted 22 lines · Ben Widawsky

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +0 -0
new file mode 100644
index 000000000000..02858ae63d6d

drivers/cxl/cxl.h line 2

Quoted 2 lines
drivers/cxl/cxl.h new file +2 -0
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.

Fix comment usage (I think SPDX in .h needs "/* */")

drivers/cxl/cxl.h line 16

Quoted 13 lines
drivers/cxl/cxl.h new file +13 -0
+#ifndef __CXL_H__
+#define __CXL_H__
+
+/* Device */
+#define CXLDEV_CAP_ARRAY_REG 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff)
+#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff)
+
+#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1
+#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2
+#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3
+#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000

Strange that the first three are decimal and the last is hex.

drivers/cxl/cxl.h line 20

Quoted 3 lines
drivers/cxl/cxl.h new file +3 -0
+/* Mailbox */
+#define CXLDEV_MB_CAPS 0x00
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)

Use upper- or lower-case hex consistently. Add tabs to line things up.

Quoted 15 lines
drivers/cxl/cxl.h new file +15 -0
+#define CXLDEV_MB_CTRL 0x04
+#define CXLDEV_MB_CMD 0x08
+#define CXLDEV_MB_STATUS 0x10
+#define CXLDEV_MB_BG_CMD_STATUS 0x18
+
+struct cxl_mem {
+	struct pci_dev *pdev;
+	void __iomem *regs;
+
+	/* Cap 0000h */
+	struct {
+		void __iomem *regs;
+	} status;
+
+	/* Cap 0002h */

drivers/cxl/cxl.h line 45

Quoted 10 lines
drivers/cxl/cxl.h new file +10 -0
+	struct {
+		void __iomem *regs;
+		size_t payload_size;
+	} mbox;
+
+	/* Cap 0040h */
+	struct {
+		void __iomem *regs;
+	} mem;
+};

Maybe a note about why READ_ONCE() is required?

Quoted 32 lines
drivers/cxl/cxl.h new file +32 -0
+#define cxl_reg(type)                                                          \
+	static inline void cxl_write_##type##_reg32(struct cxl_mem *cxlm,      \
+						    u32 reg, u32 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		writel(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline void cxl_write_##type##_reg64(struct cxl_mem *cxlm,      \
+						    u32 reg, u64 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		writeq(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline u32 cxl_read_##type##_reg32(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		return readl(reg_addr + reg);                                  \
+	}                                                                      \
+	static inline u64 cxl_read_##type##_reg64(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		return readq(reg_addr + reg);                                  \
+	}
+
+cxl_reg(status)
+cxl_reg(mbox)
+
+static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = READ_ONCE(cxlm->regs);

drivers/cxl/cxl.h line 88

Quoted 10 lines
drivers/cxl/cxl.h new file +10 -0
+
+	return readl(reg_addr + reg);
+}
+
+static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = READ_ONCE(cxlm->regs);
+
+	return readq(reg_addr + reg);
+}

Are the "__" prefixes here to leave space for something else in the future? "__" typically means something like "raw", so right now it sort of reads like "raw cxl raw read". So if you don't *need* the "__" prefix, I'd drop it.

Quoted 6 lines
drivers/cxl/cxl.h new file +1 -0
+#endif /* __CXL_H__ */
drivers/cxl/mem.c +0 -0
index 8d9b9ab6c5ea..4109ef7c3ecb 100644

drivers/cxl/mem.c line 12

Quoted 9 lines
drivers/cxl/mem.c +1 -4
@@ -5,11 +5,57 @@
 #include <linux/io.h>
 #include "acpi.h"
 #include "pci.h"
+#include "cxl.h"
 
-struct cxl_mem {
-	struct pci_dev *pdev;
-	void __iomem *regs;
-};

Probably nicer if you put "struct cxl_mem" in its ultimate destination (drivers/cxl/cxl.h) from the beginning. Then it's easier to see what this patch adds because it's not moving at the same time.

Quoted 11 lines
drivers/cxl/mem.c +11 -0
+static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
+{
+	u64 cap_array;
+	int cap;
+
+	cap_array = __cxl_raw_read_reg64(cxlm, CXLDEV_CAP_ARRAY_REG);
+	if (CXLDEV_CAP_ARRAY_ID(cap_array) != CXLDEV_CAP_ARRAY_CAP_ID)
+		return -ENODEV;
+
+	for (cap = 1; cap <= CXLDEV_CAP_ARRAY_COUNT(cap_array); cap++) {
+		void *__iomem register_block;

drivers/cxl/mem.c line 30

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+		u32 offset;
+		u16 cap_id;
+
+		cap_id = __cxl_raw_read_reg32(cxlm, cap * 0x10) & 0xffff;
+		offset = __cxl_raw_read_reg32(cxlm, cap * 0x10 + 0x4);
+		register_block = cxlm->regs + offset;
+
+		switch (cap_id) {
+		case CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS:
+			dev_dbg(&cxlm->pdev->dev, "found Status capability\n");

Consider including the address or offset in these messages to help debug? Printing a completely constant string always seems like a missed opportunity to me.

Quoted 45 lines
drivers/cxl/mem.c +32 -0
+			cxlm->status.regs = register_block;
+			break;
+		case CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				 "found Mailbox capability\n");
+			cxlm->mbox.regs = register_block;
+			cxlm->mbox.payload_size = CXLDEV_MB_CAP_PAYLOAD_SIZE(cap_id);
+			break;
+		case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				   "found UNSUPPORTED Secondary Mailbox capability\n");
+			break;
+		case CXL_CAPABILITIES_CAP_ID_MEMDEV:
+			dev_dbg(&cxlm->pdev->dev,
+				 "found Memory Device capability\n");
+			cxlm->mem.regs = register_block;
+			break;
+		default:
+			dev_err(&cxlm->pdev->dev, "Unknown cap ID: %d\n", cap_id);
+			return -ENXIO;
+		}
+	}
+
+	if (!cxlm->status.regs || !cxlm->mbox.regs || !cxlm->mem.regs)
+		return -ENXIO;
+
+	return 0;
+}
 
 static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
 {
@@ -110,6 +156,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (IS_ERR(cxlm))
 		return -ENXIO;
 
+	rc = cxl_mem_setup_regs(cxlm);
+	if (rc)
+		return rc;
+
 	pci_set_drvdata(pdev, cxlm);
 
 	return 0;
Signature · 1 line
2.29.2
#45 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

Quoted 70 lines · Bjorn Helgaas
Quoted 22 lines · Ben Widawsky

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +0 -0
new file mode 100644
index 000000000000..02858ae63d6d

drivers/cxl/cxl.h line 2

Quoted 2 lines
drivers/cxl/cxl.h new file +2 -0
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.

Fix comment usage (I think SPDX in .h needs "/* */")

drivers/cxl/cxl.h line 16

Quoted 13 lines
drivers/cxl/cxl.h new file +13 -0
+#ifndef __CXL_H__
+#define __CXL_H__
+
+/* Device */
+#define CXLDEV_CAP_ARRAY_REG 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff)
+#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff)
+
+#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1
+#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2
+#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3
+#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000

Strange that the first three are decimal and the last is hex.

drivers/cxl/cxl.h line 20

Quoted 3 lines
drivers/cxl/cxl.h new file +3 -0
+/* Mailbox */
+#define CXLDEV_MB_CAPS 0x00
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)

Use upper- or lower-case hex consistently. Add tabs to line things up.

Quoted 19 lines
drivers/cxl/cxl.h new file +19 -0
+#define CXLDEV_MB_CTRL 0x04
+#define CXLDEV_MB_CMD 0x08
+#define CXLDEV_MB_STATUS 0x10
+#define CXLDEV_MB_BG_CMD_STATUS 0x18
+
+struct cxl_mem {
+	struct pci_dev *pdev;
+	void __iomem *regs;
+
+	/* Cap 0000h */
+	struct {
+		void __iomem *regs;
+	} status;
+
+	/* Cap 0002h */
+	struct {
+		void __iomem *regs;
+		size_t payload_size;
+	} mbox;
Quoted 9 lines

drivers/cxl/cxl.h line 45

Quoted 6 lines
drivers/cxl/cxl.h new file +6 -0
+
+	/* Cap 0040h */
+	struct {
+		void __iomem *regs;
+	} mem;
+};

Maybe a note about why READ_ONCE() is required?

I don't believe it's actually necessary. I will drop it.

Quoted 39 lines
Quoted 39 lines
drivers/cxl/cxl.h new file +39 -0
+#define cxl_reg(type)                                                          \
+	static inline void cxl_write_##type##_reg32(struct cxl_mem *cxlm,      \
+						    u32 reg, u32 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		writel(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline void cxl_write_##type##_reg64(struct cxl_mem *cxlm,      \
+						    u32 reg, u64 value)        \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		writeq(value, reg_addr + reg);                                 \
+	}                                                                      \
+	static inline u32 cxl_read_##type##_reg32(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		return readl(reg_addr + reg);                                  \
+	}                                                                      \
+	static inline u64 cxl_read_##type##_reg64(struct cxl_mem *cxlm,        \
+						  u32 reg)                     \
+	{                                                                      \
+		void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
+		return readq(reg_addr + reg);                                  \
+	}
+
+cxl_reg(status)
+cxl_reg(mbox)
+
+static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = READ_ONCE(cxlm->regs);
+
+	return readl(reg_addr + reg);
+}
+
+static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
+{
+	void __iomem *reg_addr = READ_ONCE(cxlm->regs);
Quoted 9 lines

drivers/cxl/cxl.h line 88

Quoted 3 lines
drivers/cxl/cxl.h new file +3 -0
+
+	return readq(reg_addr + reg);
+}

Are the "__" prefixes here to leave space for something else in the future? "__" typically means something like "raw", so right now it sort of reads like "raw cxl raw read". So if you don't *need* the "__" prefix, I'd drop it.

The "__" prefix is because those functions really shouldn't be used except in early initialization and perhaps for debugfs kinds of things. I can remove the 'raw' from the name, but I do consider this a raw read as it isn't going to read/write to any particular function of a CXL device.

So unless you're deeply offended by it, I'd like to make it

__cxl_read/write_reg64()

Quoted 11 lines
Quoted 11 lines
drivers/cxl/cxl.h new file +1 -0
+#endif /* __CXL_H__ */
drivers/cxl/mem.c +1 -0
index 8d9b9ab6c5ea..4109ef7c3ecb 100644
@@ -5,11 +5,57 @@
 #include <linux/io.h>
 #include "acpi.h"
 #include "pci.h"
+#include "cxl.h"
 
Quoted 9 lines

drivers/cxl/mem.c line 12

Quoted 4 lines
drivers/cxl/mem.c +0 -4
-struct cxl_mem {
-	struct pci_dev *pdev;
-	void __iomem *regs;
-};

Probably nicer if you put "struct cxl_mem" in its ultimate destination (drivers/cxl/cxl.h) from the beginning. Then it's easier to see what this patch adds because it's not moving at the same time.

Yes, this is sort of the wart again of 3 of us all working on the code at the same time. Dan, you want to squash it into yours?

Quoted 17 lines
Quoted 17 lines
drivers/cxl/mem.c +17 -0
+static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
+{
+	u64 cap_array;
+	int cap;
+
+	cap_array = __cxl_raw_read_reg64(cxlm, CXLDEV_CAP_ARRAY_REG);
+	if (CXLDEV_CAP_ARRAY_ID(cap_array) != CXLDEV_CAP_ARRAY_CAP_ID)
+		return -ENODEV;
+
+	for (cap = 1; cap <= CXLDEV_CAP_ARRAY_COUNT(cap_array); cap++) {
+		void *__iomem register_block;
+		u32 offset;
+		u16 cap_id;
+
+		cap_id = __cxl_raw_read_reg32(cxlm, cap * 0x10) & 0xffff;
+		offset = __cxl_raw_read_reg32(cxlm, cap * 0x10 + 0x4);
+		register_block = cxlm->regs + offset;
Quoted 9 lines

drivers/cxl/mem.c line 30

Quoted 4 lines
drivers/cxl/mem.c +4 -0
+
+		switch (cap_id) {
+		case CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS:
+			dev_dbg(&cxlm->pdev->dev, "found Status capability\n");

Consider including the address or offset in these messages to help debug? Printing a completely constant string always seems like a missed opportunity to me.

Sure. The main thing the debug message is trying to help notify is textual versions of the caps, compared to what one might expect. I don't see offsets as immediately useful, but they definitely do not hurt.

Quoted 45 lines
Quoted 45 lines
drivers/cxl/mem.c +32 -0
+			cxlm->status.regs = register_block;
+			break;
+		case CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				 "found Mailbox capability\n");
+			cxlm->mbox.regs = register_block;
+			cxlm->mbox.payload_size = CXLDEV_MB_CAP_PAYLOAD_SIZE(cap_id);
+			break;
+		case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				   "found UNSUPPORTED Secondary Mailbox capability\n");
+			break;
+		case CXL_CAPABILITIES_CAP_ID_MEMDEV:
+			dev_dbg(&cxlm->pdev->dev,
+				 "found Memory Device capability\n");
+			cxlm->mem.regs = register_block;
+			break;
+		default:
+			dev_err(&cxlm->pdev->dev, "Unknown cap ID: %d\n", cap_id);
+			return -ENXIO;
+		}
+	}
+
+	if (!cxlm->status.regs || !cxlm->mbox.regs || !cxlm->mem.regs)
+		return -ENXIO;
+
+	return 0;
+}
 
 static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi)
 {
@@ -110,6 +156,10 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (IS_ERR(cxlm))
 		return -ENXIO;
 
+	rc = cxl_mem_setup_regs(cxlm);
+	if (rc)
+		return rc;
+
 	pci_set_drvdata(pdev, cxlm);
 
 	return 0;
Signature · 1 line
2.29.2
#46 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

In reply to Ben Widawsky
[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Ben,

[FYI, it's a private test report for your RFC patch.]
[auto build test ERROR on pm/linux-next]
[also build test ERROR on linus/master linux/master v5.10-rc3 next-20201113] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Ben-Widawsky/CXL-2-0-Support/20201111-134547
base:   https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git linux-next
config: i386-allyesconfig (attached as .config)
compiler: gcc-9 (Debian 9.3.0-15) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/03316f5eab47413b046c08f6b977867dfade36f9
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Ben-Widawsky/CXL-2-0-Support/20201111-134547
        git checkout 03316f5eab47413b046c08f6b977867dfade36f9
        # save the attached .config to linux build tree
        make W=1 ARCH=i386 

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

In file included from drivers/cxl/mem.c:8:
drivers/cxl/cxl.h: In function 'cxl_write_status_reg64':

>> drivers/cxl/cxl.h:58:3: error: implicit declaration of function 'writeq'; did you mean 'writel'? [-Werror=implicit-function-declaration]
      58 |   writeq(value, reg_addr + reg);                                 \
         |   ^~~~~~
   drivers/cxl/cxl.h:73:1: note: in expansion of macro 'cxl_reg'
      73 | cxl_reg(status)
         | ^~~~~~~
   drivers/cxl/cxl.h: In function 'cxl_read_status_reg64':
>> drivers/cxl/cxl.h:70:10: error: implicit declaration of function 'readq'; did you mean 'readl'? [-Werror=implicit-function-declaration]
      70 |   return readq(reg_addr + reg);                                  \
         |          ^~~~~
   drivers/cxl/cxl.h:73:1: note: in expansion of macro 'cxl_reg'
      73 | cxl_reg(status)
         | ^~~~~~~

cc1: some warnings being treated as errors

vim +58 drivers/cxl/cxl.h

    46	
    47	#define cxl_reg(type)                                                          \
    48		static inline void cxl_write_##type##_reg32(struct cxl_mem *cxlm,      \
    49							    u32 reg, u32 value)        \
    50		{                                                                      \
    51			void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
    52			writel(value, reg_addr + reg);                                 \
    53		}                                                                      \
    54		static inline void cxl_write_##type##_reg64(struct cxl_mem *cxlm,      \
    55							    u32 reg, u64 value)        \
    56		{                                                                      \
    57			void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
  > 58			writeq(value, reg_addr + reg);                                 \
    59		}                                                                      \
    60		static inline u32 cxl_read_##type##_reg32(struct cxl_mem *cxlm,        \
    61							  u32 reg)                     \
    62		{                                                                      \
    63			void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
    64			return readl(reg_addr + reg);                                  \
    65		}                                                                      \
    66		static inline u64 cxl_read_##type##_reg64(struct cxl_mem *cxlm,        \
    67							  u32 reg)                     \
    68		{                                                                      \
    69			void __iomem *reg_addr = READ_ONCE(cxlm->type.regs);           \
  > 70			return readq(reg_addr + reg);                                  \
    71		}
    72	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#47 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

In reply to Ben Widawsky
Quoted 10 lines · On Tue, 10 Nov 2020 21:43:52 -0800 Ben Widawsky <[email protected]> wrote:

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

A few really minor things in this one.

Jonathan

Quoted 15 lines
---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +3 -0
new file mode 100644
index 000000000000..02858ae63d6d
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+

drivers/cxl/cxl.h line 13

Quoted 10 lines
drivers/cxl/cxl.h new file +10 -0
+#ifndef __CXL_H__
+#define __CXL_H__
+
+/* Device */
+#define CXLDEV_CAP_ARRAY_REG 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff)
+#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff)
+
+#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1

I'm not sure what you can do about consistent naming, but perhaps this really does need to be CXLDEV_CAP_CAP_ID_x however silly that looks.

It's a funny exercise I've only seen done once in a spec, but I wish everyone would try working out what their fully expanded field names will end up as and make sure the long form naming shortens to something sensible. It definitely helps with clarity!

drivers/cxl/cxl.h line 19

Quoted 6 lines
drivers/cxl/cxl.h new file +6 -0
+#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2
+#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3
+#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000
+
+/* Mailbox */
+#define CXLDEV_MB_CAPS 0x00

Elsewhere you've used _OFFSET postfix. That's useful so I'd do it here as well. Cross references to the spec section always appreciated as well!

drivers/cxl/cxl.h line 30

Quoted 11 lines
drivers/cxl/cxl.h new file +11 -0
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)
+#define CXLDEV_MB_CTRL 0x04
+#define CXLDEV_MB_CMD 0x08
+#define CXLDEV_MB_STATUS 0x10
+#define CXLDEV_MB_BG_CMD_STATUS 0x18
+
+struct cxl_mem {
+	struct pci_dev *pdev;
+	void __iomem *regs;
+
+	/* Cap 0000h */

What are the numbers here? These capabilities have too many indexes associated with them in different ways for it to be obvious, so perhaps more detail in the comments?

Quoted 15 lines
drivers/cxl/cxl.h new file +15 -0
+	struct {
+		void __iomem *regs;
+	} status;
+
+	/* Cap 0002h */
+	struct {
+		void __iomem *regs;
+		size_t payload_size;
+	} mbox;
+
+	/* Cap 0040h */
+	struct {
+		void __iomem *regs;
+	} mem;
+};
#48 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

Quoted 63 lines · Jonathan Cameron
Quoted 10 lines · On Tue, 10 Nov 2020 21:43:52 -0800 Ben Widawsky <[email protected]> wrote:

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

A few really minor things in this one.

Jonathan

Quoted 15 lines
---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +3 -0
new file mode 100644
index 000000000000..02858ae63d6d
@@ -0,0 +1,89 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+

drivers/cxl/cxl.h line 13

Quoted 10 lines
drivers/cxl/cxl.h new file +10 -0
+#ifndef __CXL_H__
+#define __CXL_H__
+
+/* Device */
+#define CXLDEV_CAP_ARRAY_REG 0x0
+#define CXLDEV_CAP_ARRAY_CAP_ID 0
+#define CXLDEV_CAP_ARRAY_ID(x) ((x) & 0xffff)
+#define CXLDEV_CAP_ARRAY_COUNT(x) (((x) >> 32) & 0xffff)
+
+#define CXL_CAPABILITIES_CAP_ID_DEVICE_STATUS 1

I'm not sure what you can do about consistent naming, but perhaps this really does need to be CXLDEV_CAP_CAP_ID_x however silly that looks.

It's a funny exercise I've only seen done once in a spec, but I wish everyone would try working out what their fully expanded field names will end up as and make sure the long form naming shortens to something sensible. It definitely helps with clarity!

drivers/cxl/cxl.h line 19

Quoted 6 lines
drivers/cxl/cxl.h new file +6 -0
+#define CXL_CAPABILITIES_CAP_ID_PRIMARY_MAILBOX 2
+#define CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX 3
+#define CXL_CAPABILITIES_CAP_ID_MEMDEV 0x4000
+
+/* Mailbox */
+#define CXLDEV_MB_CAPS 0x00

Elsewhere you've used _OFFSET postfix. That's useful so I'd do it here as well. Cross references to the spec section always appreciated as well!

Quoted 15 lines

drivers/cxl/cxl.h line 30

Quoted 11 lines
drivers/cxl/cxl.h new file +11 -0
+#define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)
+#define CXLDEV_MB_CTRL 0x04
+#define CXLDEV_MB_CMD 0x08
+#define CXLDEV_MB_STATUS 0x10
+#define CXLDEV_MB_BG_CMD_STATUS 0x18
+
+struct cxl_mem {
+	struct pci_dev *pdev;
+	void __iomem *regs;
+
+	/* Cap 0000h */

What are the numbers here? These capabilities have too many indexes associated with them in different ways for it to be obvious, so perhaps more detail in the comments?

This comment was a bug. The cap is 0001h actually. I've added the hash define for its cap id as part of the comment.

Everything else is accepted.

Quoted 16 lines
Quoted 16 lines
drivers/cxl/cxl.h new file +15 -0
+	struct {
+		void __iomem *regs;
+	} status;
+
+	/* Cap 0002h */
+	struct {
+		void __iomem *regs;
+		size_t payload_size;
+	} mbox;
+
+	/* Cap 0040h */
+	struct {
+		void __iomem *regs;
+	} mem;
+};
#49 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

In reply to Ben Widawsky

drivers/cxl/mem.c line 41

Quoted 3 lines · Ben Widawsky
drivers/cxl/mem.c +3 -0
+		case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				   "found UNSUPPORTED Secondary Mailbox capability\n");

Per spec, the secondary mailbox is intended for use by platform firmware, so Linux should never be using it anyway. Maybe that message is slightly misleading?

Jon.

P.S. Related - I've severe doubts about the mailbox approach being proposed by CXL and have begun to push back through the spec org.

Signature · 1 line
Computer Architect
#50 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

Quoted 9 lines · Jon Masters

drivers/cxl/mem.c line 41

Quoted 3 lines · Ben Widawsky
drivers/cxl/mem.c +3 -0
+		case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX:
+			dev_dbg(&cxlm->pdev->dev,
+				   "found UNSUPPORTED Secondary Mailbox capability\n");

Per spec, the secondary mailbox is intended for use by platform firmware, so Linux should never be using it anyway. Maybe that message is slightly misleading?

Yeah, I think the message could be reworded, but it is dev_dbg, so I wasn't too worried about the wording in the first place. I think it is a mistake in this case for the spec to describe the intended purpose. If the expectation is for platform firmware to use it, but there is no negotiation mechanism in place, it's essentially useless.

Quoted 5 lines

Jon.

P.S. Related - I've severe doubts about the mailbox approach being proposed by CXL and have begun to push back through the spec org.

Any reason not to articulate that here? Now that the spec is public, I don't see any reason not to disclose that publicly.

#51 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

In reply to Jon Masters
Quoted 7 lines · Jon Masters

drivers/cxl/mem.c line 41

Quoted 3 lines · Ben Widawsky
drivers/cxl/mem.c +3 -0
+             case CXL_CAPABILITIES_CAP_ID_SECONDARY_MAILBOX:
+                     dev_dbg(&cxlm->pdev->dev,
+                                "found UNSUPPORTED Secondary Mailbox capability\n");
Quoted 8 lines

Per spec, the secondary mailbox is intended for use by platform firmware, so Linux should never be using it anyway. Maybe that message is slightly misleading?

Jon.

P.S. Related - I've severe doubts about the mailbox approach being proposed by CXL and have begun to push back through the spec org.

The more Linux software voices the better. At the same time the spec is released so we're into xkcd territory [1] of what the driver will be expected to support for any future improvements.

[1]: https://xkcd.com/927/

#52 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

In reply to Ben Widawsky
Quoted 22 lines · Ben Widawsky

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +0 -0
new file mode 100644
index 000000000000..02858ae63d6d

[..]

Quoted 1 line

+static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)

Going through my reworks and the "raw" jumped out at me. My typical interpretation of "raw" in respect to register access macros is the difference between readl() and __raw_readl() which means "don't do bus endian swizzling, and don't do a memory clobber barrier". Any heartburn to drop the "raw"?

...is it only me that reacts that way?

#53 lore raw

Re: [RFC PATCH 5/9] cxl/mem: Find device capabilities

Quoted 26 lines · Dan Williams
Quoted 22 lines · Ben Widawsky

CXL devices contain an array of capabilities that describe the interactions software can interact with the device, or firmware running on the device. A CXL compliant device must implement the device status and the mailbox capability. A CXL compliant memory device must implement the memory device capability.

Each of the capabilities can [will] provide an offset within the MMIO region for interacting with the CXL device.

---
 drivers/cxl/cxl.h | 89 +++++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.c | 58 +++++++++++++++++++++++++++---
 2 files changed, 143 insertions(+), 4 deletions(-)
 create mode 100644 drivers/cxl/cxl.h
drivers/cxl/cxl.h new file +0 -0
new file mode 100644
index 000000000000..02858ae63d6d

[..]

Quoted 1 line

+static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)

Quoted 7 lines

Going through my reworks and the "raw" jumped out at me. My typical interpretation of "raw" in respect to register access macros is the difference between readl() and __raw_readl() which means "don't do bus endian swizzling, and don't do a memory clobber barrier". Any heartburn to drop the "raw"?

...is it only me that reacts that way?

I will drop "raw". Especially given that I intend to reuse the word in v2 for something entirely different, it makes sense.

My idea of "raw" was that it's just unfettered access to the device's MMIO space. No offsets, no checks. I'm not sure of a better adjective to describe that, but if you have any in mind, I'd like to add it.

#54 lore raw

RFC PATCH 6/9 cxl/mem: Initialize the mailbox interface

In reply to Ben Widawsky

Provide enough functionality to utilize the mailbox of a memory device. The mailbox is used to interact with the firmware running on the memory device.

The CXL specification defines separate capabilities for the mailbox and the memory device. While we can confirm the mailbox is ready, in order to actually interact with the memory device, the driver must also confirm the device's firmware is ready.

---
 drivers/cxl/cxl.h | 28 ++++++++++++++++++++++
 drivers/cxl/mem.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+)
drivers/cxl/cxl.h +28 -0
index 02858ae63d6d..482fc9cdc890 100644
@@ -19,14 +19,41 @@
 #define CXLDEV_MB_CAPS 0x00
 #define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)
 #define CXLDEV_MB_CTRL 0x04
+#define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
 #define CXLDEV_MB_STATUS 0x10
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
 
+/* Memory Device */
+#define CXLMDEV_STATUS 0
+#define CXLMDEV_DEV_FATAL BIT(0)
+#define CXLMDEV_FW_HALT BIT(1)
+#define CXLMDEV_MEDIA_STATUS_SHIFT 2
+#define CXLMDEV_MEDIA_STATUS_MASK 0x3
+#define CXLMDEV_READY(status)                                                  \
+	((((status) >> CXLMDEV_MEDIA_STATUS_SHIFT) &                           \
+	  CXLMDEV_MEDIA_STATUS_MASK) == CXLMDEV_MS_READY)
+#define CXLMDEV_MS_NOT_READY 0
+#define CXLMDEV_MS_READY 1
+#define CXLMDEV_MS_ERROR 2
+#define CXLMDEV_MS_DISABLED 3
+#define CXLMDEV_MBOX_IF_READY BIT(4)
+#define CXLMDEV_RESET_NEEDED_SHIFT 5
+#define CXLMDEV_RESET_NEEDED_MASK 0x7
+#define CXLMDEV_RESET_NEEDED(status)                                           \
+	(((status) >> CXLMDEV_RESET_NEEDED_SHIFT) & CXLMDEV_RESET_NEEDED_MASK)
+#define CXLMDEV_RESET_NEEDED_NOT 0
+#define CXLMDEV_RESET_NEEDED_COLD 1
+#define CXLMDEV_RESET_NEEDED_WARM 2
+#define CXLMDEV_RESET_NEEDED_HOT 3
+#define CXLMDEV_RESET_NEEDED_CXL 4
+
 struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
 
+	spinlock_t mbox_lock; /* Protects device mailbox and firmware */
+
 	/* Cap 0000h */
 	struct {
 		void __iomem *regs;
@@ -72,6 +99,7 @@ struct cxl_mem {
 
 cxl_reg(status)
 cxl_reg(mbox)
+cxl_reg(mem)
 
 static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)
 {
drivers/cxl/mem.c +59 -0

77 lines not on this page · raw message

Signature · 1 line
2.29.2
#55 lore raw

Re: [RFC PATCH 6/9] cxl/mem: Initialize the mailbox interface

Quoted 10 lines · On Tue, 10 Nov 2020 21:43:53 -0800 Ben Widawsky <[email protected]> wrote:

Provide enough functionality to utilize the mailbox of a memory device. The mailbox is used to interact with the firmware running on the memory device.

The CXL specification defines separate capabilities for the mailbox and the memory device. While we can confirm the mailbox is ready, in order to actually interact with the memory device, the driver must also confirm the device's firmware is ready.

Hi Ben,

A few minor suggestions.

J

Quoted 10 lines
---
 drivers/cxl/cxl.h | 28 ++++++++++++++++++++++
 drivers/cxl/mem.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+)
drivers/cxl/cxl.h +0 -0
index 02858ae63d6d..482fc9cdc890 100644

drivers/cxl/cxl.h line 28

Quoted 10 lines
drivers/cxl/cxl.h +3 -0
@@ -19,14 +19,41 @@
 #define CXLDEV_MB_CAPS 0x00
 #define   CXLDEV_MB_CAP_PAYLOAD_SIZE(cap) ((cap) & 0x1F)
 #define CXLDEV_MB_CTRL 0x04
+#define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
 #define CXLDEV_MB_STATUS 0x10
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
 
+/* Memory Device */
+#define CXLMDEV_STATUS 0

As mentioned earlier, I'd make sure there is a clear way of telling what is a register/memory offset and what is a field.

drivers/cxl/cxl.h line 32

Quoted 4 lines
drivers/cxl/cxl.h +4 -0
+#define CXLMDEV_DEV_FATAL BIT(0)
+#define CXLMDEV_FW_HALT BIT(1)
+#define CXLMDEV_MEDIA_STATUS_SHIFT 2
+#define CXLMDEV_MEDIA_STATUS_MASK 0x3

Problem with masks defined like this is it's not apparent from name if they are pre or post shift. Could we use FIELD_GET etc and GENMASK to ensure there is only one unambiguous definition?

Quoted 43 lines
drivers/cxl/cxl.h +21 -0
+#define CXLMDEV_READY(status)                                                  \
+	((((status) >> CXLMDEV_MEDIA_STATUS_SHIFT) &                           \
+	  CXLMDEV_MEDIA_STATUS_MASK) == CXLMDEV_MS_READY)
+#define CXLMDEV_MS_NOT_READY 0
+#define CXLMDEV_MS_READY 1
+#define CXLMDEV_MS_ERROR 2
+#define CXLMDEV_MS_DISABLED 3
+#define CXLMDEV_MBOX_IF_READY BIT(4)
+#define CXLMDEV_RESET_NEEDED_SHIFT 5
+#define CXLMDEV_RESET_NEEDED_MASK 0x7
+#define CXLMDEV_RESET_NEEDED(status)                                           \
+	(((status) >> CXLMDEV_RESET_NEEDED_SHIFT) & CXLMDEV_RESET_NEEDED_MASK)
+#define CXLMDEV_RESET_NEEDED_NOT 0
+#define CXLMDEV_RESET_NEEDED_COLD 1
+#define CXLMDEV_RESET_NEEDED_WARM 2
+#define CXLMDEV_RESET_NEEDED_HOT 3
+#define CXLMDEV_RESET_NEEDED_CXL 4
+
 struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
 
+	spinlock_t mbox_lock; /* Protects device mailbox and firmware */
+
 	/* Cap 0000h */
 	struct {
 		void __iomem *regs;
@@ -72,6 +99,7 @@ struct cxl_mem {
 
 cxl_reg(status)
 cxl_reg(mbox)
+cxl_reg(mem)
 
 static inline u32 __cxl_raw_read_reg32(struct cxl_mem *cxlm, u32 reg)
 {
drivers/cxl/mem.c +0 -0
index 4109ef7c3ecb..9fd2d1daa534 100644
@@ -7,6 +7,56 @@
 #include "pci.h"
 #include "cxl.h"
 

drivers/cxl/mem.c line 19

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+static int cxl_mem_mbox_get(struct cxl_mem *cxlm)
+{
+	u64 md_status;
+	u32 ctrl;
+	int rc = -EBUSY;
+
+	spin_lock(&cxlm->mbox_lock);
+
+	ctrl = cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL);
+	if (ctrl & CXLDEV_MB_CTRL_DOORBELL)

Perhaps a comment on what this path means? I know from the spec, but not super obvious from the code here. If we do hit this path the device will fail to come up and we won't have a clue why.

Quoted 66 lines
drivers/cxl/mem.c +49 -0
+		goto out;
+
+	md_status = cxl_read_mem_reg64(cxlm, CXLMDEV_STATUS);
+	if (md_status & CXLMDEV_MBOX_IF_READY && CXLMDEV_READY(md_status)) {
+		/*
+		 * Hardware shouldn't allow a ready status but also have failure
+		 * bits set. Spit out an error, this should be a bug report
+		 */
+		if (md_status & CXLMDEV_DEV_FATAL) {
+			dev_err(&cxlm->pdev->dev,
+				"CXL device reporting ready and fatal\n");
+			rc = -EFAULT;
+			goto out;
+		}
+		if (md_status & CXLMDEV_FW_HALT) {
+			dev_err(&cxlm->pdev->dev,
+				"CXL device reporting ready and halted\n");
+			rc = -EFAULT;
+			goto out;
+		}
+		if (CXLMDEV_RESET_NEEDED(md_status)) {
+			dev_err(&cxlm->pdev->dev,
+				"CXL device reporting ready and reset needed\n");
+			rc = -EFAULT;
+			goto out;
+		}
+
+		return 0;
+	}
+
+out:
+	spin_unlock(&cxlm->mbox_lock);
+	return rc;
+}
+
+static void cxl_mem_mbox_put(struct cxl_mem *cxlm)
+{
+	spin_unlock(&cxlm->mbox_lock);
+}
+
 static int cxl_mem_setup_regs(struct cxl_mem *cxlm)
 {
 	u64 cap_array;
@@ -88,6 +138,8 @@ static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, u32 reg_
 		return ERR_PTR(-ENOMEM);
 	}
 
+	spin_lock_init(&cxlm->mbox_lock);
+
 	regs = pcim_iomap_table(pdev)[bar];
 	cxlm->pdev = pdev;
 	cxlm->regs = regs + offset;
@@ -160,6 +212,13 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
+	/* Check that hardware "looks" okay. */
+	rc = cxl_mem_mbox_get(cxlm);
+	if (rc)
+		return rc;
+
+	cxl_mem_mbox_put(cxlm);
+	dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n");
 	pci_set_drvdata(pdev, cxlm);
 
 	return 0;
#56 lore raw

RFC PATCH 7/9 cxl/mem: Implement polled mode mailbox

In reply to Ben Widawsky

Create a function to handle sending a command, optionally with a payload, to the memory device, polling on a result, and then optionally copying out the payload. The algorithm for doing this come straight out of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting a command in the background. That implementation is saved for a later time.

Secondary mailboxes aren't implemented at this time.

WARNING: This is untested with actual timeouts occurring.

---
 drivers/cxl/cxl.h |  16 +++++++
 drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
drivers/cxl/cxl.h +16 -0
index 482fc9cdc890..f49ab80f68bd 100644
@@ -21,8 +21,12 @@
 #define CXLDEV_MB_CTRL 0x04
 #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
 #define CXLDEV_MB_STATUS 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
+#define CXLDEV_MB_PAYLOAD 0x20
 
 /* Memory Device */
 #define CXLMDEV_STATUS 0
@@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
 	return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+					    unsigned int length)
+{
+	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
+					     u8 *output, unsigned int length)
+{
+	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
+}
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +107 -0

118 lines not on this page · raw message

Signature · 1 line
2.29.2
#57 lore raw

Re: [RFC PATCH 7/9] cxl/mem: Implement polled mode mailbox

Quoted 58 lines · Ben Widawsky

Create a function to handle sending a command, optionally with a payload, to the memory device, polling on a result, and then optionally copying out the payload. The algorithm for doing this come straight out of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting a command in the background. That implementation is saved for a later time.

Secondary mailboxes aren't implemented at this time.

WARNING: This is untested with actual timeouts occurring.

---
 drivers/cxl/cxl.h |  16 +++++++
 drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
drivers/cxl/cxl.h +16 -0
index 482fc9cdc890..f49ab80f68bd 100644
@@ -21,8 +21,12 @@
 #define CXLDEV_MB_CTRL 0x04
 #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
 #define CXLDEV_MB_STATUS 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
+#define CXLDEV_MB_PAYLOAD 0x20
 
 /* Memory Device */
 #define CXLMDEV_STATUS 0
@@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
 	return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+					    unsigned int length)
+{
+	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
+					     u8 *output, unsigned int length)
+{
+	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
+}
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +0 -0
index 9fd2d1daa534..08913360d500 100644

drivers/cxl/mem.c line 2

Quoted 2 lines
drivers/cxl/mem.c +0 -0
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 // Copyright(c) 2020 Intel Corporation. All rights reserved.

/* Copyright ... */

Quoted 9 lines
drivers/cxl/mem.c +2 -0
+#include <linux/sched/clock.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/io.h>
@@ -7,6 +8,112 @@
 #include "pci.h"
 #include "cxl.h"
 
+struct mbox_cmd {

drivers/cxl/mem.c line 21

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	u16 cmd;
+	u8 *payload;
+	size_t payload_size;
+	u16 return_code;
+};
+
+static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
+{
+	u64 start, now;
+	int cpu, ret, timeout = 2000000000;

It'd be nice to have a hint about where this timeout comes from and what the units are. local_clock(), sched_clock_cpu(), etc don't have any hints either and I got tired of following the chain.

Several callers use ns_to_ktime(local_clock()), so I guess it must be in ns?

Quoted 25 lines
drivers/cxl/mem.c +25 -0
+	start = local_clock();
+	preempt_disable();
+	cpu = smp_processor_id();
+	for (;;) {
+		now = local_clock();
+		preempt_enable();
+		if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) &
+		     CXLDEV_MB_CTRL_DOORBELL) == 0) {
+			ret = 0;
+			break;
+		}
+
+		if (now - start >= timeout) {
+			ret = -ETIMEDOUT;
+			break;
+		}
+
+		cpu_relax();
+		preempt_disable();
+		if (unlikely(cpu != smp_processor_id())) {
+			timeout -= (now - start);
+			cpu = smp_processor_id();
+			start = local_clock();
+		}
+	}
#58 lore raw

Re: [RFC PATCH 7/9] cxl/mem: Implement polled mode mailbox

In reply to Ben Widawsky
Quoted 9 lines · On Tue, 10 Nov 2020 21:43:54 -0800 Ben Widawsky <[email protected]> wrote:

Create a function to handle sending a command, optionally with a payload, to the memory device, polling on a result, and then optionally copying out the payload. The algorithm for doing this come straight out of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting a command in the background. That implementation is saved for a later time.

Quoted 5 lines

Secondary mailboxes aren't implemented at this time.

WARNING: This is untested with actual timeouts occurring.

Question inline for why the preempt / local timer dance is worth bothering with. What am I missing?

Thanks,

Jonathan

Quoted 62 lines
---
 drivers/cxl/cxl.h |  16 +++++++
 drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
drivers/cxl/cxl.h +16 -0
index 482fc9cdc890..f49ab80f68bd 100644
@@ -21,8 +21,12 @@
 #define CXLDEV_MB_CTRL 0x04
 #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
 #define CXLDEV_MB_STATUS 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
+#define CXLDEV_MB_PAYLOAD 0x20
 
 /* Memory Device */
 #define CXLMDEV_STATUS 0
@@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
 	return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+					    unsigned int length)
+{
+	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
+					     u8 *output, unsigned int length)
+{
+	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
+}
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +9 -0
index 9fd2d1daa534..08913360d500 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 // Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/sched/clock.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/io.h>
@@ -7,6 +8,112 @@
 #include "pci.h"
 #include "cxl.h"
 
+struct mbox_cmd {
+	u16 cmd;
+	u8 *payload;
+	size_t payload_size;
+	u16 return_code;
+};
+
+static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)

drivers/cxl/mem.c line 28

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+{
+	u64 start, now;
+	int cpu, ret, timeout = 2000000000;
+
+	start = local_clock();
+	preempt_disable();
+	cpu = smp_processor_id();
+	for (;;) {
+		now = local_clock();
+		preempt_enable();

What do we ever do with this mailbox that is particularly performance critical? I'd like to understand why we care enough to mess around with the preemption changes and local clock etc.

Quoted 39 lines
drivers/cxl/mem.c +39 -0
+		if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) &
+		     CXLDEV_MB_CTRL_DOORBELL) == 0) {
+			ret = 0;
+			break;
+		}
+
+		if (now - start >= timeout) {
+			ret = -ETIMEDOUT;
+			break;
+		}
+
+		cpu_relax();
+		preempt_disable();
+		if (unlikely(cpu != smp_processor_id())) {
+			timeout -= (now - start);
+			cpu = smp_processor_id();
+			start = local_clock();
+		}
+	}
+
+	return ret;
+}
+
+/*
+ * Returns 0 if the doorbell transaction was successful from a protocol level.
+ * Caller should check the return code in @mbox_cmd to make sure it succeeded.
+ */
+static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
+{
+	u64 cmd, status;
+	int rc;
+
+	lockdep_assert_held(&cxlm->mbox_lock);
+
+	/*
+	 * Here are the steps from 8.2.8.4 of the CXL 2.0 spec.
+	 *   1. Caller reads MB Control Register to verify doorbell is clear
+	 *   2. Caller writes Command Register
+	 *   3. Caller writes Command Payload Registers if input payload is non-empty

drivers/cxl/mem.c line 77

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	 *   4. Caller writes MB Control Register to set doorbell
+	 *   5. Caller either polls for doorbell to be clear or waits for interrupt if configured
+	 *   6. Caller reads MB Status Register to fetch Return code
+	 *   7. If command successful, Caller reads Command Register to get Payload Length
+	 *   8. If output payload is non-empty, host reads Command Payload Registers
+	 */
+
+	cmd = mbox_cmd->cmd;
+	if (mbox_cmd->payload_size) {
+		/* #3 */

Having just given the steps above, having them out of order feels like it needs a comment to state why.

Quoted 42 lines
drivers/cxl/mem.c +39 -0
+		cmd |= mbox_cmd->payload_size
+		       << CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT;
+		cxl_mbox_payload_fill(cxlm, mbox_cmd->payload, mbox_cmd->payload_size);
+	}
+
+	/* #2 */
+	cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD, cmd);
+
+	/* #4 */
+	cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL, CXLDEV_MB_CTRL_DOORBELL);
+
+	/* #5 */
+	rc = cxldev_wait_for_doorbell(cxlm);
+	if (rc == -ETIMEDOUT) {
+		dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n");
+		return rc;
+	}
+
+	/* #6 */
+	status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS);
+	cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD);
+
+	mbox_cmd->return_code = (status >> CXLDEV_MB_STATUS_RET_CODE_SHIFT) &
+				CXLDEV_MB_STATUS_RET_CODE_MASK;
+
+	/* There was a problem, let the caller deal with it */
+	if (mbox_cmd->return_code != 0)
+		return 0;
+
+	/* #7 */
+	mbox_cmd->payload_size = cmd >> CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT;
+
+	/* #8 */
+	if (mbox_cmd->payload_size)
+		cxl_mbox_payload_drain(cxlm, mbox_cmd->payload, mbox_cmd->payload_size);
+
+	return 0;
+}
+
 static int cxl_mem_mbox_get(struct cxl_mem *cxlm)
 {
 	u64 md_status;
#59 lore raw

Re: [RFC PATCH 7/9] cxl/mem: Implement polled mode mailbox

Quoted 93 lines · Jonathan Cameron
Quoted 9 lines · On Tue, 10 Nov 2020 21:43:54 -0800 Ben Widawsky <[email protected]> wrote:

Create a function to handle sending a command, optionally with a payload, to the memory device, polling on a result, and then optionally copying out the payload. The algorithm for doing this come straight out of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting a command in the background. That implementation is saved for a later time.

Quoted 5 lines

Secondary mailboxes aren't implemented at this time.

WARNING: This is untested with actual timeouts occurring.

Question inline for why the preempt / local timer dance is worth bothering with. What am I missing?

Thanks,

Jonathan

Quoted 68 lines
---
 drivers/cxl/cxl.h |  16 +++++++
 drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
drivers/cxl/cxl.h +16 -0
index 482fc9cdc890..f49ab80f68bd 100644
@@ -21,8 +21,12 @@
 #define CXLDEV_MB_CTRL 0x04
 #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
 #define CXLDEV_MB_STATUS 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
+#define CXLDEV_MB_PAYLOAD 0x20
 
 /* Memory Device */
 #define CXLMDEV_STATUS 0
@@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
 	return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+					    unsigned int length)
+{
+	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
+					     u8 *output, unsigned int length)
+{
+	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
+}
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +15 -0
index 9fd2d1daa534..08913360d500 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 // Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/sched/clock.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/io.h>
@@ -7,6 +8,112 @@
 #include "pci.h"
 #include "cxl.h"
 
+struct mbox_cmd {
+	u16 cmd;
+	u8 *payload;
+	size_t payload_size;
+	u16 return_code;
+};
+
+static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
+{
+	u64 start, now;
+	int cpu, ret, timeout = 2000000000;
+
+	start = local_clock();
+	preempt_disable();
Quoted 9 lines

drivers/cxl/mem.c line 28

Quoted 4 lines
drivers/cxl/mem.c +4 -0
+	cpu = smp_processor_id();
+	for (;;) {
+		now = local_clock();
+		preempt_enable();

What do we ever do with this mailbox that is particularly performance critical? I'd like to understand why we care enough to mess around with the preemption changes and local clock etc.

It is quite obviously a premature optimization at this point (since we only support a single command in QEMU). However, the polling can be anywhere from instant to 2 seconds. QEMU implementation aside again, some devices may never support interrupts on completion, and so I thought providing a poll function now that is capable of working for most [all?] cases was wise.

Quoted 96 lines
Quoted 39 lines
drivers/cxl/mem.c +39 -0
+		if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) &
+		     CXLDEV_MB_CTRL_DOORBELL) == 0) {
+			ret = 0;
+			break;
+		}
+
+		if (now - start >= timeout) {
+			ret = -ETIMEDOUT;
+			break;
+		}
+
+		cpu_relax();
+		preempt_disable();
+		if (unlikely(cpu != smp_processor_id())) {
+			timeout -= (now - start);
+			cpu = smp_processor_id();
+			start = local_clock();
+		}
+	}
+
+	return ret;
+}
+
+/*
+ * Returns 0 if the doorbell transaction was successful from a protocol level.
+ * Caller should check the return code in @mbox_cmd to make sure it succeeded.
+ */
+static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
+{
+	u64 cmd, status;
+	int rc;
+
+	lockdep_assert_held(&cxlm->mbox_lock);
+
+	/*
+	 * Here are the steps from 8.2.8.4 of the CXL 2.0 spec.
+	 *   1. Caller reads MB Control Register to verify doorbell is clear
+	 *   2. Caller writes Command Register
+	 *   3. Caller writes Command Payload Registers if input payload is non-empty

drivers/cxl/mem.c line 77

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	 *   4. Caller writes MB Control Register to set doorbell
+	 *   5. Caller either polls for doorbell to be clear or waits for interrupt if configured
+	 *   6. Caller reads MB Status Register to fetch Return code
+	 *   7. If command successful, Caller reads Command Register to get Payload Length
+	 *   8. If output payload is non-empty, host reads Command Payload Registers
+	 */
+
+	cmd = mbox_cmd->cmd;
+	if (mbox_cmd->payload_size) {
+		/* #3 */

Having just given the steps above, having them out of order feels like it needs a comment to state why.

Quoted 42 lines
drivers/cxl/mem.c +39 -0
+		cmd |= mbox_cmd->payload_size
+		       << CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT;
+		cxl_mbox_payload_fill(cxlm, mbox_cmd->payload, mbox_cmd->payload_size);
+	}
+
+	/* #2 */
+	cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD, cmd);
+
+	/* #4 */
+	cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL, CXLDEV_MB_CTRL_DOORBELL);
+
+	/* #5 */
+	rc = cxldev_wait_for_doorbell(cxlm);
+	if (rc == -ETIMEDOUT) {
+		dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n");
+		return rc;
+	}
+
+	/* #6 */
+	status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS);
+	cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD);
+
+	mbox_cmd->return_code = (status >> CXLDEV_MB_STATUS_RET_CODE_SHIFT) &
+				CXLDEV_MB_STATUS_RET_CODE_MASK;
+
+	/* There was a problem, let the caller deal with it */
+	if (mbox_cmd->return_code != 0)
+		return 0;
+
+	/* #7 */
+	mbox_cmd->payload_size = cmd >> CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT;
+
+	/* #8 */
+	if (mbox_cmd->payload_size)
+		cxl_mbox_payload_drain(cxlm, mbox_cmd->payload, mbox_cmd->payload_size);
+
+	return 0;
+}
+
 static int cxl_mem_mbox_get(struct cxl_mem *cxlm)
 {
 	u64 md_status;
#60 lore raw

Re: [RFC PATCH 7/9] cxl/mem: Implement polled mode mailbox

Quoted 104 lines · On Tue, 17 Nov 2020 08:34:38 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 93 lines · Jonathan Cameron
Quoted 9 lines · On Tue, 10 Nov 2020 21:43:54 -0800 Ben Widawsky <[email protected]> wrote:

Create a function to handle sending a command, optionally with a payload, to the memory device, polling on a result, and then optionally copying out the payload. The algorithm for doing this come straight out of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting a command in the background. That implementation is saved for a later time.

Quoted 5 lines

Secondary mailboxes aren't implemented at this time.

WARNING: This is untested with actual timeouts occurring.

Question inline for why the preempt / local timer dance is worth bothering with. What am I missing?

Thanks,

Jonathan

Quoted 68 lines
---
 drivers/cxl/cxl.h |  16 +++++++
 drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
drivers/cxl/cxl.h +16 -0
index 482fc9cdc890..f49ab80f68bd 100644
@@ -21,8 +21,12 @@
 #define CXLDEV_MB_CTRL 0x04
 #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
 #define CXLDEV_MB_STATUS 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
+#define CXLDEV_MB_PAYLOAD 0x20
 
 /* Memory Device */
 #define CXLMDEV_STATUS 0
@@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
 	return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+					    unsigned int length)
+{
+	memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
+					     u8 *output, unsigned int length)
+{
+	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
+}
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +15 -0
index 9fd2d1daa534..08913360d500 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 // Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/sched/clock.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/io.h>
@@ -7,6 +8,112 @@
 #include "pci.h"
 #include "cxl.h"
 
+struct mbox_cmd {
+	u16 cmd;
+	u8 *payload;
+	size_t payload_size;
+	u16 return_code;
+};
+
+static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
+{
+	u64 start, now;
+	int cpu, ret, timeout = 2000000000;
+
+	start = local_clock();
+	preempt_disable();
Quoted 9 lines

drivers/cxl/mem.c line 28

Quoted 4 lines
drivers/cxl/mem.c +4 -0
+	cpu = smp_processor_id();
+	for (;;) {
+		now = local_clock();
+		preempt_enable();  

What do we ever do with this mailbox that is particularly performance critical? I'd like to understand why we care enough to mess around with the preemption changes and local clock etc.

Quoted 5 lines

It is quite obviously a premature optimization at this point (since we only support a single command in QEMU). However, the polling can be anywhere from instant to 2 seconds. QEMU implementation aside again, some devices may never support interrupts on completion, and so I thought providing a poll function now that is capable of working for most [all?] cases was wise.

Definitely seems premature. I'd want to see real numbers on hardware to justify this sort of complexity. Maybe others disagree though.

Jonathan

Quoted 97 lines
Quoted 97 lines
Quoted 40 lines
drivers/cxl/mem.c +39 -0
+		if ((cxl_read_mbox_reg32(cxlm, CXLDEV_MB_CTRL) &
+		     CXLDEV_MB_CTRL_DOORBELL) == 0) {
+			ret = 0;
+			break;
+		}
+
+		if (now - start >= timeout) {
+			ret = -ETIMEDOUT;
+			break;
+		}
+
+		cpu_relax();
+		preempt_disable();
+		if (unlikely(cpu != smp_processor_id())) {
+			timeout -= (now - start);
+			cpu = smp_processor_id();
+			start = local_clock();
+		}
+	}
+
+	return ret;
+}
+
+/*
+ * Returns 0 if the doorbell transaction was successful from a protocol level.
+ * Caller should check the return code in @mbox_cmd to make sure it succeeded.
+ */
+static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
+{
+	u64 cmd, status;
+	int rc;
+
+	lockdep_assert_held(&cxlm->mbox_lock);
+
+	/*
+	 * Here are the steps from 8.2.8.4 of the CXL 2.0 spec.
+	 *   1. Caller reads MB Control Register to verify doorbell is clear
+	 *   2. Caller writes Command Register
+	 *   3. Caller writes Command Payload Registers if input payload is non-empty

drivers/cxl/mem.c line 77

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	 *   4. Caller writes MB Control Register to set doorbell
+	 *   5. Caller either polls for doorbell to be clear or waits for interrupt if configured
+	 *   6. Caller reads MB Status Register to fetch Return code
+	 *   7. If command successful, Caller reads Command Register to get Payload Length
+	 *   8. If output payload is non-empty, host reads Command Payload Registers
+	 */
+
+	cmd = mbox_cmd->cmd;
+	if (mbox_cmd->payload_size) {
+		/* #3 */  

Having just given the steps above, having them out of order feels like it needs a comment to state why.

Quoted 42 lines
drivers/cxl/mem.c +39 -0
+		cmd |= mbox_cmd->payload_size
+		       << CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT;
+		cxl_mbox_payload_fill(cxlm, mbox_cmd->payload, mbox_cmd->payload_size);
+	}
+
+	/* #2 */
+	cxl_write_mbox_reg64(cxlm, CXLDEV_MB_CMD, cmd);
+
+	/* #4 */
+	cxl_write_mbox_reg32(cxlm, CXLDEV_MB_CTRL, CXLDEV_MB_CTRL_DOORBELL);
+
+	/* #5 */
+	rc = cxldev_wait_for_doorbell(cxlm);
+	if (rc == -ETIMEDOUT) {
+		dev_warn(&cxlm->pdev->dev, "Mailbox command timed out\n");
+		return rc;
+	}
+
+	/* #6 */
+	status = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_STATUS);
+	cmd = cxl_read_mbox_reg64(cxlm, CXLDEV_MB_CMD);
+
+	mbox_cmd->return_code = (status >> CXLDEV_MB_STATUS_RET_CODE_SHIFT) &
+				CXLDEV_MB_STATUS_RET_CODE_MASK;
+
+	/* There was a problem, let the caller deal with it */
+	if (mbox_cmd->return_code != 0)
+		return 0;
+
+	/* #7 */
+	mbox_cmd->payload_size = cmd >> CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT;
+
+	/* #8 */
+	if (mbox_cmd->payload_size)
+		cxl_mbox_payload_drain(cxlm, mbox_cmd->payload, mbox_cmd->payload_size);
+
+	return 0;
+}
+
 static int cxl_mem_mbox_get(struct cxl_mem *cxlm)
 {
 	u64 md_status;  
#61 lore raw

Re: [RFC PATCH 7/9] cxl/mem: Implement polled mode mailbox

Quoted 108 lines · Jonathan Cameron
Quoted 104 lines · On Tue, 17 Nov 2020 08:34:38 -0800 Ben Widawsky <[email protected]> wrote:
Quoted 102 lines · Jonathan Cameron
Quoted 9 lines · On Tue, 10 Nov 2020 21:43:54 -0800 Ben Widawsky <[email protected]> wrote:

Create a function to handle sending a command, optionally with a payload, to the memory device, polling on a result, and then optionally copying out the payload. The algorithm for doing this come straight out of the CXL 2.0 specification.

Primary mailboxes are capable of generating an interrupt when submitting a command in the background. That implementation is saved for a later time.

Quoted 5 lines

Secondary mailboxes aren't implemented at this time.

WARNING: This is untested with actual timeouts occurring.

Question inline for why the preempt / local timer dance is worth bothering with. What am I missing?

Thanks,

Jonathan

Quoted 62 lines
---
 drivers/cxl/cxl.h |  16 +++++++
 drivers/cxl/mem.c | 107 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 123 insertions(+)
drivers/cxl/cxl.h +16 -0
index 482fc9cdc890..f49ab80f68bd 100644
@@ -21,8 +21,12 @@
 #define CXLDEV_MB_CTRL 0x04
 #define   CXLDEV_MB_CTRL_DOORBELL BIT(0)
 #define CXLDEV_MB_CMD 0x08
+#define   CXLDEV_MB_CMD_PAYLOAD_LENGTH_SHIFT 16
 #define CXLDEV_MB_STATUS 0x10
+#define   CXLDEV_MB_STATUS_RET_CODE_SHIFT 32
+#define   CXLDEV_MB_STATUS_RET_CODE_MASK 0xffff
 #define CXLDEV_MB_BG_CMD_STATUS 0x18
+#define CXLDEV_MB_PAYLOAD 0x20
 
 /* Memory Device */
 #define CXLMDEV_STATUS 0
@@ -114,4 +118,16 @@ static inline u64 __cxl_raw_read_reg64(struct cxl_mem *cxlm, u32 reg)
 
  return readq(reg_addr + reg);
 }
+
+static inline void cxl_mbox_payload_fill(struct cxl_mem *cxlm, u8 *input,
+                                     unsigned int length)
+{
+ memcpy_toio(cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, input, length);
+}
+
+static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
+                                      u8 *output, unsigned int length)
+{
+ memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
+}
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +9 -0
index 9fd2d1daa534..08913360d500 100644
@@ -1,5 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-only
 // Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/sched/clock.h>
 #include <linux/module.h>
 #include <linux/pci.h>
 #include <linux/io.h>
@@ -7,6 +8,112 @@
 #include "pci.h"
 #include "cxl.h"
 
+struct mbox_cmd {
+ u16 cmd;
+ u8 *payload;
+ size_t payload_size;
+ u16 return_code;
+};
+
+static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)

drivers/cxl/mem.c line 28

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+{
+ u64 start, now;
+ int cpu, ret, timeout = 2000000000;
+
+ start = local_clock();
+ preempt_disable();
+ cpu = smp_processor_id();
+ for (;;) {
+         now = local_clock();
+         preempt_enable();

What do we ever do with this mailbox that is particularly performance critical? I'd like to understand why we care enough to mess around with the preemption changes and local clock etc.

Quoted 8 lines
Quoted 5 lines

It is quite obviously a premature optimization at this point (since we only support a single command in QEMU). However, the polling can be anywhere from instant to 2 seconds. QEMU implementation aside again, some devices may never support interrupts on completion, and so I thought providing a poll function now that is capable of working for most [all?] cases was wise.

Definitely seems premature. I'd want to see real numbers on hardware to justify this sort of complexity. Maybe others disagree though.

The polling is definitely needed, but I think it can be a simple jiffies based loop and avoid this sched_clock() complexity.

#62 lore raw

RFC PATCH 8/9 cxl/mem: Register CXL memX devices

In reply to Ben Widawsky

From: Dan Williams <[email protected]>

Create the /sys/bus/cxl hierarchy to enumerate memory devices (per-endpoint control devices), memory address space devices (platform address ranges with interleaving, performance, and persistence attributes), and memory regions (active provisioned memory from an address space device that is in use as System RAM or delegated to libnvdimm as Persistent Memory regions).

For now, only the per-endpoint control devices are registered on the 'cxl' bus.

---
 drivers/cxl/Makefile |   2 +
 drivers/cxl/bus.c    |  35 ++++++
 drivers/cxl/bus.h    |   8 ++
 drivers/cxl/cxl.h    |  33 +++++
 drivers/cxl/mem.c    | 287 ++++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 359 insertions(+), 6 deletions(-)
 create mode 100644 drivers/cxl/bus.c
 create mode 100644 drivers/cxl/bus.h
drivers/cxl/Makefile +2 -0
index 97fdffb00f2d..1cc032092852 100644
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+obj-$(CONFIG_CXL_BUS_PROVIDER) += cxl_bus.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
 cxl_mem-y := mem.o
+cxl_bus-y := bus.o
drivers/cxl/bus.c new file +35 -0
new file mode 100644
index 000000000000..8594366955f7
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/device.h>
+#include <linux/module.h>
+
+static struct bus_type cxl_bus_type = {
+	.name = "cxl",
+};
+
+int cxl_register(struct device *dev)
+{
+	int rc;
+
+	dev->bus = &cxl_bus_type;
+	rc = device_add(dev);
+	if (rc)
+		put_device(dev);
+	return rc;
+}
+EXPORT_SYMBOL_GPL(cxl_register);
+
+static __init int cxl_bus_init(void)
+{
+	return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+	bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
drivers/cxl/bus.h new file +8 -0
new file mode 100644
index 000000000000..fe2bea2bbc3c
@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_BUS_H__
+#define __CXL_BUS_H__
+
+int cxl_register(struct device *dev);
+
+#endif /* __CXL_BUS_H__ */
drivers/cxl/cxl.h +33 -0
index f49ab80f68bd..cef5fd9ea68b 100644
@@ -3,6 +3,7 @@
 
 #ifndef __CXL_H__
 #define __CXL_H__
+#include <linux/range.h>
 
 /* Device */
 #define CXLDEV_CAP_ARRAY_REG 0x0
@@ -52,12 +53,24 @@
 #define CXLMDEV_RESET_NEEDED_HOT 3
 #define CXLMDEV_RESET_NEEDED_CXL 4
 
+struct cxl_memdev;
 struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
+	struct cxl_memdev *cxlmd;
 
 	spinlock_t mbox_lock; /* Protects device mailbox and firmware */
 
+	struct {
+		struct range range;
+	} pmem;
+
+	struct {
+		struct range range;
+	} ram;
+
+	char firmware_version[0x10];
+
 	/* Cap 0000h */
 	struct {
 		void __iomem *regs;
@@ -130,4 +143,24 @@ static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
 {
 	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
 }
+
+#define CXL_MBOX_IDENTIFY 0x4000
+
+struct cxl_mbox_identify {
+	char fw_revision[0x10];
+	__le64 total_capacity;
+	__le64 volatile_capacity;
+	__le64 persistent_capacity;
+	__le64 partition_align;
+	__le16 info_event_log_size;
+	__le16 warning_event_log_size;
+	__le16 failure_event_log_size;
+	__le16 fatal_event_log_size;
+	__le32 lsa_size;
+	u8 poison_list_max_mer[3];
+	__le16 inject_poison_limit;
+	u8 poison_caps;
+	u8 qos_telemetry_caps;
+} __packed;
+
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +281 -6

342 lines not on this page · raw message

Signature · 1 line
2.29.2
#63 lore raw

Re: [RFC PATCH 8/9] cxl/mem: Register CXL memX devices

[If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in https://git-scm.com/docs/git-format-patch]

Hi Ben,

[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on pm/linux-next]
[also build test WARNING on linus/master linux/master v5.10-rc3 next-20201111] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch]

url:    https://github.com/0day-ci/linux/commits/Ben-Widawsky/CXL-2-0-Support/20201111-134547
base:   https://git.kernel.org/pub/scm/linux/kernel/git/rafael/linux-pm.git linux-next
config: ia64-allmodconfig (attached as .config)
compiler: ia64-linux-gcc (GCC) 9.3.0

reproduce (this is a W=1 build):
wget https://raw.githubusercontent.com/intel/lkp-tests/master/sbin/make.cross -O ~/bin/make.cross chmod +x ~/bin/make.cross
# https://github.com/0day-ci/linux/commit/69b8720d497b9e7dfdd05e6a17f49ab41b6b681f
git remote add linux-review https://github.com/0day-ci/linux
git fetch --no-tags linux-review Ben-Widawsky/CXL-2-0-Support/20201111-134547 git checkout 69b8720d497b9e7dfdd05e6a17f49ab41b6b681f
# save the attached .config to linux build tree
COMPILER_INSTALL_PATH=$HOME/0day COMPILER=gcc-9.3.0 make.cross ARCH=ia64

If you fix the issue, kindly add following tag as appropriate

All warnings (new ones prefixed by >>):

>> drivers/cxl/bus.c:10:5: warning: no previous prototype for 'cxl_register' [-Wmissing-prototypes]

10 | int cxl_register(struct device *dev)
| ^~~~~~~~~~~~

vim +/cxl_register +10 drivers/cxl/bus.c

     9	
  > 10	int cxl_register(struct device *dev)
    11	{
    12		int rc;
    13	
    14		dev->bus = &cxl_bus_type;
    15		rc = device_add(dev);
    16		if (rc)
    17			put_device(dev);
    18		return rc;
    19	}
    20	EXPORT_SYMBOL_GPL(cxl_register);
    21	

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#64 lore raw

Re: [RFC PATCH 8/9] cxl/mem: Register CXL memX devices

In reply to Ben Widawsky
Quoted 11 lines · On Tue, 10 Nov 2020 21:43:55 -0800 Ben Widawsky <[email protected]> wrote:

From: Dan Williams <[email protected]>

Create the /sys/bus/cxl hierarchy to enumerate memory devices (per-endpoint control devices), memory address space devices (platform address ranges with interleaving, performance, and persistence attributes), and memory regions (active provisioned memory from an address space device that is in use as System RAM or delegated to libnvdimm as Persistent Memory regions).

For now, only the per-endpoint control devices are registered on the 'cxl' bus.

Reviewing ABI without documentation is challenging even when it's simple so please add that for v2.

This patch feels somewhat unpolished, but I guess it is mainly here to give an illustration of how stuff might fit together rather than any expectation of detailed review.

So in that spirit I've just pointed out stuff that jumped out at me during a quick read through.

Thanks,

Jonathan

Quoted 174 lines
---
 drivers/cxl/Makefile |   2 +
 drivers/cxl/bus.c    |  35 ++++++
 drivers/cxl/bus.h    |   8 ++
 drivers/cxl/cxl.h    |  33 +++++
 drivers/cxl/mem.c    | 287 ++++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 359 insertions(+), 6 deletions(-)
 create mode 100644 drivers/cxl/bus.c
 create mode 100644 drivers/cxl/bus.h
drivers/cxl/Makefile +2 -0
index 97fdffb00f2d..1cc032092852 100644
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+obj-$(CONFIG_CXL_BUS_PROVIDER) += cxl_bus.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
 cxl_mem-y := mem.o
+cxl_bus-y := bus.o
drivers/cxl/bus.c new file +35 -0
new file mode 100644
index 000000000000..8594366955f7
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/device.h>
+#include <linux/module.h>
+
+static struct bus_type cxl_bus_type = {
+	.name = "cxl",
+};
+
+int cxl_register(struct device *dev)
+{
+	int rc;
+
+	dev->bus = &cxl_bus_type;
+	rc = device_add(dev);
+	if (rc)
+		put_device(dev);
+	return rc;
+}
+EXPORT_SYMBOL_GPL(cxl_register);
+
+static __init int cxl_bus_init(void)
+{
+	return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+	bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
drivers/cxl/bus.h new file +8 -0
new file mode 100644
index 000000000000..fe2bea2bbc3c
@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_BUS_H__
+#define __CXL_BUS_H__
+
+int cxl_register(struct device *dev);
+
+#endif /* __CXL_BUS_H__ */
drivers/cxl/cxl.h +33 -0
index f49ab80f68bd..cef5fd9ea68b 100644
@@ -3,6 +3,7 @@
 
 #ifndef __CXL_H__
 #define __CXL_H__
+#include <linux/range.h>
 
 /* Device */
 #define CXLDEV_CAP_ARRAY_REG 0x0
@@ -52,12 +53,24 @@
 #define CXLMDEV_RESET_NEEDED_HOT 3
 #define CXLMDEV_RESET_NEEDED_CXL 4
 
+struct cxl_memdev;
 struct cxl_mem {
 	struct pci_dev *pdev;
 	void __iomem *regs;
+	struct cxl_memdev *cxlmd;
 
 	spinlock_t mbox_lock; /* Protects device mailbox and firmware */
 
+	struct {
+		struct range range;
+	} pmem;
+
+	struct {
+		struct range range;
+	} ram;
+
+	char firmware_version[0x10];
+
 	/* Cap 0000h */
 	struct {
 		void __iomem *regs;
@@ -130,4 +143,24 @@ static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
 {
 	memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
 }
+
+#define CXL_MBOX_IDENTIFY 0x4000
+
+struct cxl_mbox_identify {
+	char fw_revision[0x10];
+	__le64 total_capacity;
+	__le64 volatile_capacity;
+	__le64 persistent_capacity;
+	__le64 partition_align;
+	__le16 info_event_log_size;
+	__le16 warning_event_log_size;
+	__le16 failure_event_log_size;
+	__le16 fatal_event_log_size;
+	__le32 lsa_size;
+	u8 poison_list_max_mer[3];
+	__le16 inject_poison_limit;
+	u8 poison_caps;
+	u8 qos_telemetry_caps;
+} __packed;
+
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +10 -0
index 08913360d500..54743d196feb 100644
@@ -2,11 +2,15 @@
 // Copyright(c) 2020 Intel Corporation. All rights reserved.
 #include <linux/sched/clock.h>
 #include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/cdev.h>
+#include <linux/idr.h>
 #include <linux/pci.h>
 #include <linux/io.h>
 #include "acpi.h"
 #include "pci.h"
 #include "cxl.h"
+#include "bus.h"
 
 struct mbox_cmd {
 	u16 cmd;
@@ -15,6 +19,53 @@ struct mbox_cmd {
 	u16 return_code;
 };
 
+/*
+ * An entire PCI topology full of devices should be enough for any
+ * config
+ */
+#define CXL_MEM_MAX_DEVS 65536
+

drivers/cxl/mem.c struct mbox_cmd { · line 37

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+struct cxl_memdev {
+	struct device dev;
+	struct cxl_mem *cxlm;
+	int id;
+};
+
+static int cxl_mem_major;
+static struct cdev cxl_mem_cdev;
+static DEFINE_IDR(cxl_mem_idr);
+static DEFINE_MUTEX(cxl_memdev_lock);

Define scope of this lock with a comment.

Quoted 92 lines
drivers/cxl/mem.c +77 -1
+
+static int cxl_mem_open(struct inode *inode, struct file *file)
+{
+	long minor = iminor(inode);
+	struct cxl_memdev *cxlmd;
+
+	rcu_read_lock();
+	cxlmd = idr_find(&cxl_mem_idr, minor);
+	rcu_read_unlock();
+
+	if (!cxlmd)
+		return -ENXIO;
+
+	file->private_data = cxlmd;
+
+	return 0;
+}
+
+static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+	return -ENOTTY;
+}
+
+static const struct file_operations cxl_mem_fops = {
+	.owner = THIS_MODULE,
+	.open = cxl_mem_open,
+	.unlocked_ioctl = cxl_mem_ioctl,
+	.compat_ioctl = compat_ptr_ioctl,
+	.llseek = noop_llseek,
+};
+
 static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
 {
 	u64 start, now;
@@ -53,7 +104,7 @@ static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
  * Returns 0 if the doorbell transaction was successful from a protocol level.
  * Caller should check the return code in @mbox_cmd to make sure it succeeded.
  */
-static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
+static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
 {
 	u64 cmd, status;
 	int rc;
@@ -277,10 +328,185 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
 	return 0;
 }
 
+static struct cxl_memdev *to_cxl_memdev(struct device *dev)
+{
+	return container_of(dev, struct cxl_memdev, dev);
+}
+
+static void cxl_memdev_release(struct device *dev)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+
+	mutex_lock(&cxl_memdev_lock);
+	idr_remove(&cxl_mem_idr, cxlmd->id);
+	mutex_unlock(&cxl_memdev_lock);
+
+	kfree(cxlmd);
+}
+
+static char *cxl_memdev_devnode(struct device *dev, umode_t *mode, kuid_t *uid, kgid_t *gid)
+{
+	return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
+}
+
+static ssize_t firmware_version_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+	struct cxl_mem *cxlm = cxlmd->cxlm;
+
+	return sprintf(buf, "%.16s\n", cxlm->firmware_version);
+}
+static DEVICE_ATTR_RO(firmware_version);
+
+static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+	struct cxl_mem *cxlm = cxlmd->cxlm;
+
+	return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->ram.range));
+}
+static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL);
+
+static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+	struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+	struct cxl_mem *cxlm = cxlmd->cxlm;
+
+	return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->pmem.range));

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 385

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+}
+static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL);
+
+static struct attribute *cxl_memdev_attributes[] = {
+	&dev_attr_firmware_version.attr,
+	NULL,
+};
+
+static struct attribute *cxl_memdev_pmem_attributes[] = {
+	&dev_attr_pmem_size.attr,

It's simple, but should still have docs in Documentation/ABI/testing/sysfs...

Quoted 34 lines
drivers/cxl/mem.c +34 -0
+	NULL,
+};
+
+static struct attribute *cxl_memdev_ram_attributes[] = {
+	&dev_attr_ram_size.attr,
+	NULL,
+};
+
+static struct attribute_group cxl_memdev_attribute_group = {
+	.attrs = cxl_memdev_attributes,
+};
+
+static struct attribute_group cxl_memdev_ram_attribute_group = {
+	.name = "ram",
+	.attrs = cxl_memdev_ram_attributes,
+};
+
+static struct attribute_group cxl_memdev_pmem_attribute_group = {
+	.name = "pmem",
+	.attrs = cxl_memdev_pmem_attributes,
+};
+
+static const struct attribute_group *cxl_memdev_attribute_groups[] = {
+	&cxl_memdev_attribute_group,
+	&cxl_memdev_ram_attribute_group,
+	&cxl_memdev_pmem_attribute_group,
+	NULL,
+};
+
+static const struct device_type cxl_memdev_type = {
+	.name = "cxl_memdev",
+	.release = cxl_memdev_release,
+	.devnode = cxl_memdev_devnode,
+	.groups = cxl_memdev_attribute_groups,

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 429

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+};
+
+static struct cxl_memdev *cxl_mem_add_memdev(struct cxl_mem *cxlm)
+{
+	struct pci_dev *pdev = cxlm->pdev;
+	struct cxl_memdev *cxlmd;
+	struct device *dev;
+	int id, rc;
+
+	cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);

Maybe I missed it, but I'm not seeing this freed anywhere.

Quoted 13 lines
drivers/cxl/mem.c +13 -0
+	if (!cxlmd)
+		return ERR_PTR(-ENOMEM);
+
+	cxlmd->cxlm = cxlm;
+	cxlm->cxlmd = cxlmd;
+
+	mutex_lock(&cxl_memdev_lock);
+	id = idr_alloc(&cxl_mem_idr, cxlmd, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL);
+	mutex_unlock(&cxl_memdev_lock);
+	if (id < 0) {
+		rc = id;
+		goto err_idr;
+	}

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 452

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+
+	cxlmd->id = id;
+
+	dev = &cxlmd->dev;
+
+	device_initialize(dev);
+	dev->parent = &pdev->dev;
+	dev->devt = MKDEV(cxl_mem_major, id);
+	dev->type = &cxl_memdev_type;
+	dev_set_name(dev, "mem%d", id);

blank line here

Quoted 12 lines
drivers/cxl/mem.c +12 -0
+	rc = cxl_register(dev);
+	if (rc)
+		return ERR_PTR(rc);
+
+	return cxlmd;
+
+err_idr:
+	kfree(cxlmd);
+
+	return ERR_PTR(rc);
+}
+

...

Quoted 9 lines

static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) {

drivers/cxl/mem.c +1 -0
 	struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
 	struct device *dev = &pdev->dev;
+	struct cxl_memdev *cxlmd;
 	int rc, regloc, i;
 
 	rc = cxl_bus_prepared(pdev);

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 327

Quoted 10 lines
drivers/cxl/mem.c +1 -4
@@ -319,20 +545,31 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	if (rc)
 		return rc;
 
-	/* Check that hardware "looks" okay. */
-	rc = cxl_mem_mbox_get(cxlm);
+	rc = cxl_mem_identify(cxlm);
 	if (rc)
 		return rc;
-
-	cxl_mem_mbox_put(cxlm);

It was kind of nice to see the flow earlier, but I'm also thinking it made a slightly harder to read patch. Hmm. Maybe just drop the version earlier in favour of a todo comment that you then do here?

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 552

Quoted 2 lines
drivers/cxl/mem.c +1 -0
 	dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n");
+

Nice to tidy that up by moving to earlier patch.

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 557

Quoted 5 lines
drivers/cxl/mem.c +3 -0
 	pci_set_drvdata(pdev, cxlm);
 
+	cxlmd = cxl_mem_add_memdev(cxlm);
+	if (IS_ERR(cxlmd))
+		return PTR_ERR(cxlmd);

Given we don't actually use cxlmd perhaps a simple return value of 0 or error would be better from cxl_mem_add_memdev()

(I guess you may have follow up patches that do something with it
here, though it feels wrong to ever do so given it is now registered and hence exposed to the system).

Quoted 5 lines
drivers/cxl/mem.c +1 -0
+
 	return 0;
 }
 
 static void cxl_mem_remove(struct pci_dev *pdev)

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 572

Quoted 10 lines
drivers/cxl/mem.c +9 -0
 {
+	struct cxl_mem *cxlm = pci_get_drvdata(pdev);
+	struct cxl_memdev *cxlmd = cxlm->cxlmd;
+
+	device_lock(&cxlmd->dev);
+	cxlm->cxlmd = NULL;
+	cxlmd->cxlm = NULL;
+	device_unlock(&cxlmd->dev);
+
+	device_unregister(&cxlmd->dev);

Why device_unregister last? Normally removing exposure to the system is the first thing you do in a remove() call. Particularly as you'll get NULL ptr dereferences if anyone manages a sysfs read between the pointers being set to NULL above and the device_unregister() taking away the sysfs files.

Quoted 32 lines
drivers/cxl/mem.c +25 -0
 }
 
 static const struct pci_device_id cxl_mem_pci_tbl[] = {
@@ -350,7 +587,45 @@ static struct pci_driver cxl_mem_driver = {
 	.remove			= cxl_mem_remove,
 };
 
+static __init int cxl_mem_init(void)
+{
+	int rc;
+	dev_t devt;
+
+	rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
+	if (rc)
+		return rc;
+
+	cxl_mem_major = MAJOR(devt);
+
+	cdev_init(&cxl_mem_cdev, &cxl_mem_fops);
+	rc = cdev_add(&cxl_mem_cdev, MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+	if (rc)
+		goto err_cdev;
+
+	rc = pci_register_driver(&cxl_mem_driver);
+	if (rc)
+		goto err_driver;
+
+	return 0;
+
+err_driver:
+	cdev_del(&cxl_mem_cdev);
+err_cdev:

drivers/cxl/mem.c static struct pci_driver cxl_mem_driver = { · line 624

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+	unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+
+	return rc;
+}
+
+static __exit void cxl_mem_exit(void)
+{
+	pci_unregister_driver(&cxl_mem_driver);
+	unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+	cdev_del(&cxl_mem_cdev);

Ordering? cdev_dev should be before unregister_chrdev_region to match error handling in init()

Quoted 8 lines
drivers/cxl/mem.c +4 -1
+}
+
 MODULE_LICENSE("GPL v2");
 MODULE_AUTHOR("Intel Corporation");
-module_pci_driver(cxl_mem_driver);
+module_init(cxl_mem_init);
+module_exit(cxl_mem_exit);
 MODULE_IMPORT_NS(CXL);
#65 lore raw

Re: [RFC PATCH 8/9] cxl/mem: Register CXL memX devices

Quoted 16 lines · Jonathan Cameron
Quoted 11 lines · On Tue, 10 Nov 2020 21:43:55 -0800 Ben Widawsky <[email protected]> wrote:

From: Dan Williams <[email protected]>

Create the /sys/bus/cxl hierarchy to enumerate memory devices (per-endpoint control devices), memory address space devices (platform address ranges with interleaving, performance, and persistence attributes), and memory regions (active provisioned memory from an address space device that is in use as System RAM or delegated to libnvdimm as Persistent Memory regions).

For now, only the per-endpoint control devices are registered on the 'cxl' bus.

Quoted 6 lines

Reviewing ABI without documentation is challenging even when it's simple so please add that for v2.

This patch feels somewhat unpolished, but I guess it is mainly here to give an illustration of how stuff might fit together rather than any expectation of detailed review.

Yeah, this is definitely an early look in the spirit of "Release early / release often".

Quoted 187 lines

So in that spirit I've just pointed out stuff that jumped out at me during a quick read through.

Thanks,

Jonathan

Quoted 178 lines
---
 drivers/cxl/Makefile |   2 +
 drivers/cxl/bus.c    |  35 ++++++
 drivers/cxl/bus.h    |   8 ++
 drivers/cxl/cxl.h    |  33 +++++
 drivers/cxl/mem.c    | 287 ++++++++++++++++++++++++++++++++++++++++++-
 5 files changed, 359 insertions(+), 6 deletions(-)
 create mode 100644 drivers/cxl/bus.c
 create mode 100644 drivers/cxl/bus.h
drivers/cxl/Makefile +2 -0
index 97fdffb00f2d..1cc032092852 100644
@@ -1,7 +1,9 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_CXL_ACPI) += cxl_acpi.o
 obj-$(CONFIG_CXL_MEM) += cxl_mem.o
+obj-$(CONFIG_CXL_BUS_PROVIDER) += cxl_bus.o
 
 ccflags-y += -DDEFAULT_SYMBOL_NAMESPACE=CXL
 cxl_acpi-y := acpi.o
 cxl_mem-y := mem.o
+cxl_bus-y := bus.o
drivers/cxl/bus.c new file +35 -0
new file mode 100644
index 000000000000..8594366955f7
@@ -0,0 +1,35 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#include <linux/device.h>
+#include <linux/module.h>
+
+static struct bus_type cxl_bus_type = {
+     .name = "cxl",
+};
+
+int cxl_register(struct device *dev)
+{
+     int rc;
+
+     dev->bus = &cxl_bus_type;
+     rc = device_add(dev);
+     if (rc)
+             put_device(dev);
+     return rc;
+}
+EXPORT_SYMBOL_GPL(cxl_register);
+
+static __init int cxl_bus_init(void)
+{
+     return bus_register(&cxl_bus_type);
+}
+
+static void cxl_bus_exit(void)
+{
+     bus_unregister(&cxl_bus_type);
+}
+
+module_init(cxl_bus_init);
+module_exit(cxl_bus_exit);
+MODULE_LICENSE("GPL v2");
+MODULE_AUTHOR("Intel Corporation");
drivers/cxl/bus.h new file +8 -0
new file mode 100644
index 000000000000..fe2bea2bbc3c
@@ -0,0 +1,8 @@
+// SPDX-License-Identifier: GPL-2.0-only
+// Copyright(c) 2020 Intel Corporation. All rights reserved.
+#ifndef __CXL_BUS_H__
+#define __CXL_BUS_H__
+
+int cxl_register(struct device *dev);
+
+#endif /* __CXL_BUS_H__ */
drivers/cxl/cxl.h +33 -0
index f49ab80f68bd..cef5fd9ea68b 100644
@@ -3,6 +3,7 @@
 
 #ifndef __CXL_H__
 #define __CXL_H__
+#include <linux/range.h>
 
 /* Device */
 #define CXLDEV_CAP_ARRAY_REG 0x0
@@ -52,12 +53,24 @@
 #define CXLMDEV_RESET_NEEDED_HOT 3
 #define CXLMDEV_RESET_NEEDED_CXL 4
 
+struct cxl_memdev;
 struct cxl_mem {
      struct pci_dev *pdev;
      void __iomem *regs;
+     struct cxl_memdev *cxlmd;
 
      spinlock_t mbox_lock; /* Protects device mailbox and firmware */
 
+     struct {
+             struct range range;
+     } pmem;
+
+     struct {
+             struct range range;
+     } ram;
+
+     char firmware_version[0x10];
+
      /* Cap 0000h */
      struct {
              void __iomem *regs;
@@ -130,4 +143,24 @@ static inline void cxl_mbox_payload_drain(struct cxl_mem *cxlm,
 {
      memcpy_fromio(output, cxlm->mbox.regs + CXLDEV_MB_PAYLOAD, length);
 }
+
+#define CXL_MBOX_IDENTIFY 0x4000
+
+struct cxl_mbox_identify {
+     char fw_revision[0x10];
+     __le64 total_capacity;
+     __le64 volatile_capacity;
+     __le64 persistent_capacity;
+     __le64 partition_align;
+     __le16 info_event_log_size;
+     __le16 warning_event_log_size;
+     __le16 failure_event_log_size;
+     __le16 fatal_event_log_size;
+     __le32 lsa_size;
+     u8 poison_list_max_mer[3];
+     __le16 inject_poison_limit;
+     u8 poison_caps;
+     u8 qos_telemetry_caps;
+} __packed;
+
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +14 -0
index 08913360d500..54743d196feb 100644
@@ -2,11 +2,15 @@
 // Copyright(c) 2020 Intel Corporation. All rights reserved.
 #include <linux/sched/clock.h>
 #include <linux/module.h>
+#include <linux/mutex.h>
+#include <linux/cdev.h>
+#include <linux/idr.h>
 #include <linux/pci.h>
 #include <linux/io.h>
 #include "acpi.h"
 #include "pci.h"
 #include "cxl.h"
+#include "bus.h"
 
 struct mbox_cmd {
      u16 cmd;
@@ -15,6 +19,53 @@ struct mbox_cmd {
      u16 return_code;
 };
 
+/*
+ * An entire PCI topology full of devices should be enough for any
+ * config
+ */
+#define CXL_MEM_MAX_DEVS 65536
+
+struct cxl_memdev {
+     struct device dev;
+     struct cxl_mem *cxlm;
+     int id;
Quoted 8 lines

drivers/cxl/mem.c struct mbox_cmd { · line 37

Quoted 6 lines
drivers/cxl/mem.c +6 -0
+};
+
+static int cxl_mem_major;
+static struct cdev cxl_mem_cdev;
+static DEFINE_IDR(cxl_mem_idr);
+static DEFINE_MUTEX(cxl_memdev_lock);

Define scope of this lock with a comment.

Will do.

Quoted 97 lines
Quoted 97 lines
drivers/cxl/mem.c +81 -1
+
+static int cxl_mem_open(struct inode *inode, struct file *file)
+{
+     long minor = iminor(inode);
+     struct cxl_memdev *cxlmd;
+
+     rcu_read_lock();
+     cxlmd = idr_find(&cxl_mem_idr, minor);
+     rcu_read_unlock();
+
+     if (!cxlmd)
+             return -ENXIO;
+
+     file->private_data = cxlmd;
+
+     return 0;
+}
+
+static long cxl_mem_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
+{
+     return -ENOTTY;
+}
+
+static const struct file_operations cxl_mem_fops = {
+     .owner = THIS_MODULE,
+     .open = cxl_mem_open,
+     .unlocked_ioctl = cxl_mem_ioctl,
+     .compat_ioctl = compat_ptr_ioctl,
+     .llseek = noop_llseek,
+};
+
 static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
 {
      u64 start, now;
@@ -53,7 +104,7 @@ static int cxldev_wait_for_doorbell(struct cxl_mem *cxlm)
  * Returns 0 if the doorbell transaction was successful from a protocol level.
  * Caller should check the return code in @mbox_cmd to make sure it succeeded.
  */
-static int __maybe_unused cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
+static int cxl_mem_mbox_send_cmd(struct cxl_mem *cxlm, struct mbox_cmd *mbox_cmd)
 {
      u64 cmd, status;
      int rc;
@@ -277,10 +328,185 @@ static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec)
      return 0;
 }
 
+static struct cxl_memdev *to_cxl_memdev(struct device *dev)
+{
+     return container_of(dev, struct cxl_memdev, dev);
+}
+
+static void cxl_memdev_release(struct device *dev)
+{
+     struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+
+     mutex_lock(&cxl_memdev_lock);
+     idr_remove(&cxl_mem_idr, cxlmd->id);
+     mutex_unlock(&cxl_memdev_lock);
+
+     kfree(cxlmd);
+}
+
+static char *cxl_memdev_devnode(struct device *dev, umode_t *mode, kuid_t *uid, kgid_t *gid)
+{
+     return kasprintf(GFP_KERNEL, "cxl/%s", dev_name(dev));
+}
+
+static ssize_t firmware_version_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+     struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+     struct cxl_mem *cxlm = cxlmd->cxlm;
+
+     return sprintf(buf, "%.16s\n", cxlm->firmware_version);
+}
+static DEVICE_ATTR_RO(firmware_version);
+
+static ssize_t ram_size_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+     struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+     struct cxl_mem *cxlm = cxlmd->cxlm;
+
+     return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->ram.range));
+}
+static struct device_attribute dev_attr_ram_size = __ATTR(size, 0444, ram_size_show, NULL);
+
+static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr, char *buf)
+{
+     struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+     struct cxl_mem *cxlm = cxlmd->cxlm;
+
+     return sprintf(buf, "%#llx\n", (unsigned long long) range_len(&cxlm->pmem.range));
+}
+static struct device_attribute dev_attr_pmem_size = __ATTR(size, 0444, pmem_size_show, NULL);
+
+static struct attribute *cxl_memdev_attributes[] = {
Quoted 8 lines

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 385

Quoted 6 lines
drivers/cxl/mem.c +6 -0
+     &dev_attr_firmware_version.attr,
+     NULL,
+};
+
+static struct attribute *cxl_memdev_pmem_attributes[] = {
+     &dev_attr_pmem_size.attr,

It's simple, but should still have docs in Documentation/ABI/testing/sysfs...

Agree.

Quoted 39 lines
Quoted 39 lines
drivers/cxl/mem.c +38 -0
+     NULL,
+};
+
+static struct attribute *cxl_memdev_ram_attributes[] = {
+     &dev_attr_ram_size.attr,
+     NULL,
+};
+
+static struct attribute_group cxl_memdev_attribute_group = {
+     .attrs = cxl_memdev_attributes,
+};
+
+static struct attribute_group cxl_memdev_ram_attribute_group = {
+     .name = "ram",
+     .attrs = cxl_memdev_ram_attributes,
+};
+
+static struct attribute_group cxl_memdev_pmem_attribute_group = {
+     .name = "pmem",
+     .attrs = cxl_memdev_pmem_attributes,
+};
+
+static const struct attribute_group *cxl_memdev_attribute_groups[] = {
+     &cxl_memdev_attribute_group,
+     &cxl_memdev_ram_attribute_group,
+     &cxl_memdev_pmem_attribute_group,
+     NULL,
+};
+
+static const struct device_type cxl_memdev_type = {
+     .name = "cxl_memdev",
+     .release = cxl_memdev_release,
+     .devnode = cxl_memdev_devnode,
+     .groups = cxl_memdev_attribute_groups,
+};
+
+static struct cxl_memdev *cxl_mem_add_memdev(struct cxl_mem *cxlm)
+{
Quoted 8 lines

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 429

Quoted 6 lines
drivers/cxl/mem.c +6 -0
+     struct pci_dev *pdev = cxlm->pdev;
+     struct cxl_memdev *cxlmd;
+     struct device *dev;
+     int id, rc;
+
+     cxlmd = kzalloc(sizeof(*cxlmd), GFP_KERNEL);

Maybe I missed it, but I'm not seeing this freed anywhere.

See: cxl_memdev_release()

Quoted 18 lines
Quoted 18 lines
drivers/cxl/mem.c +17 -0
+     if (!cxlmd)
+             return ERR_PTR(-ENOMEM);
+
+     cxlmd->cxlm = cxlm;
+     cxlm->cxlmd = cxlmd;
+
+     mutex_lock(&cxl_memdev_lock);
+     id = idr_alloc(&cxl_mem_idr, cxlmd, 0, CXL_MEM_MAX_DEVS, GFP_KERNEL);
+     mutex_unlock(&cxl_memdev_lock);
+     if (id < 0) {
+             rc = id;
+             goto err_idr;
+     }
+
+     cxlmd->id = id;
+
+     dev = &cxlmd->dev;
Quoted 8 lines

drivers/cxl/mem.c static int cxl_mem_dvsec(struct pci_dev *pdev, int dvsec) · line 452

Quoted 6 lines
drivers/cxl/mem.c +6 -0
+
+     device_initialize(dev);
+     dev->parent = &pdev->dev;
+     dev->devt = MKDEV(cxl_mem_major, id);
+     dev->type = &cxl_memdev_type;
+     dev_set_name(dev, "mem%d", id);

blank line here

ok

Quoted 28 lines
Quoted 13 lines
drivers/cxl/mem.c +12 -0
+     rc = cxl_register(dev);
+     if (rc)
+             return ERR_PTR(rc);
+
+     return cxlmd;
+
+err_idr:
+     kfree(cxlmd);
+
+     return ERR_PTR(rc);
+}
+

...

Quoted 12 lines

static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)

 {
      struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
      struct device *dev = &pdev->dev;
drivers/cxl/mem.c +1 -0
+     struct cxl_memdev *cxlmd;
      int rc, regloc, i;
 
      rc = cxl_bus_prepared(pdev);
@@ -319,20 +545,31 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
      if (rc)
              return rc;
 
Quoted 11 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 327

Quoted 7 lines
drivers/cxl/mem.c +1 -4
-     /* Check that hardware "looks" okay. */
-     rc = cxl_mem_mbox_get(cxlm);
+     rc = cxl_mem_identify(cxlm);
      if (rc)
              return rc;
-
-     cxl_mem_mbox_put(cxlm);

It was kind of nice to see the flow earlier, but I'm also thinking it made a slightly harder to read patch. Hmm. Maybe just drop the version earlier in favour of a todo comment that you then do here?

Not sure I follow, but I think you're saying don't bother with an initial patch introducing just doing the raw cxl_mem_mbox_get() in this path, jump straight to cxl_mem_identify()?

Quoted 5 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 552

Quoted 3 lines
drivers/cxl/mem.c +1 -0
      dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n");
+

Nice to tidy that up by moving to earlier patch.

Sure.

Quoted 7 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 557

Quoted 6 lines
drivers/cxl/mem.c +3 -0
      pci_set_drvdata(pdev, cxlm);
 
+     cxlmd = cxl_mem_add_memdev(cxlm);
+     if (IS_ERR(cxlmd))
+             return PTR_ERR(cxlmd);
Quoted 6 lines

Given we don't actually use cxlmd perhaps a simple return value of 0 or error would be better from cxl_mem_add_memdev()

(I guess you may have follow up patches that do something with it
here, though it feels wrong to ever do so given it is now registered and hence exposed to the system).

It's not added if IS_ERR() is true, but it would be simpler to just have cxl_mem_add_memdev() return an int since ->probe() doesn't use it.

Quoted 14 lines
Quoted 14 lines
drivers/cxl/mem.c +8 -0
+
      return 0;
 }
 
 static void cxl_mem_remove(struct pci_dev *pdev)
 {
+     struct cxl_mem *cxlm = pci_get_drvdata(pdev);
+     struct cxl_memdev *cxlmd = cxlm->cxlmd;
+
+     device_lock(&cxlmd->dev);
+     cxlm->cxlmd = NULL;
+     cxlmd->cxlm = NULL;
+     device_unlock(&cxlmd->dev);
Quoted 8 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 572

Quoted 2 lines
drivers/cxl/mem.c +2 -0
+
+     device_unregister(&cxlmd->dev);

Why device_unregister last? Normally removing exposure to the system is the first thing you do in a remove() call. Particularly as you'll get NULL ptr dereferences if anyone manages a sysfs read between the pointers being set to NULL above and the device_unregister() taking away the sysfs files.

Yes, the unregister should be before the invalidation in this case. In fact I'll likely drop the invalidation or do some other synchronization requirement for any races between the ioctl path and the remove path.

Quoted 40 lines
Quoted 40 lines
drivers/cxl/mem.c +30 -0
 }
 
 static const struct pci_device_id cxl_mem_pci_tbl[] = {
@@ -350,7 +587,45 @@ static struct pci_driver cxl_mem_driver = {
      .remove                 = cxl_mem_remove,
 };
 
+static __init int cxl_mem_init(void)
+{
+     int rc;
+     dev_t devt;
+
+     rc = alloc_chrdev_region(&devt, 0, CXL_MEM_MAX_DEVS, "cxl");
+     if (rc)
+             return rc;
+
+     cxl_mem_major = MAJOR(devt);
+
+     cdev_init(&cxl_mem_cdev, &cxl_mem_fops);
+     rc = cdev_add(&cxl_mem_cdev, MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+     if (rc)
+             goto err_cdev;
+
+     rc = pci_register_driver(&cxl_mem_driver);
+     if (rc)
+             goto err_driver;
+
+     return 0;
+
+err_driver:
+     cdev_del(&cxl_mem_cdev);
+err_cdev:
+     unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+
+     return rc;
+}
+
Quoted 8 lines

drivers/cxl/mem.c static struct pci_driver cxl_mem_driver = { · line 624

Quoted 5 lines
drivers/cxl/mem.c +5 -0
+static __exit void cxl_mem_exit(void)
+{
+     pci_unregister_driver(&cxl_mem_driver);
+     unregister_chrdev_region(MKDEV(cxl_mem_major, 0), CXL_MEM_MAX_DEVS);
+     cdev_del(&cxl_mem_cdev);

Ordering? cdev_dev should be before unregister_chrdev_region to match error handling in init()

Yes.

Thanks for taking a look!

#66 lore raw

Re: [RFC PATCH 8/9] cxl/mem: Register CXL memX devices

Quoted 21 lines · On Thu, 19 Nov 2020 18:16:19 -0800 Dan Williams <[email protected]>…
Quoted 19 lines · Jonathan Cameron
Quoted 11 lines · On Tue, 10 Nov 2020 21:43:55 -0800 Ben Widawsky <[email protected]> wrote:

From: Dan Williams <[email protected]>

Create the /sys/bus/cxl hierarchy to enumerate memory devices (per-endpoint control devices), memory address space devices (platform address ranges with interleaving, performance, and persistence attributes), and memory regions (active provisioned memory from an address space device that is in use as System RAM or delegated to libnvdimm as Persistent Memory regions).

For now, only the per-endpoint control devices are registered on the 'cxl' bus.

Reviewing ABI without documentation is challenging even when it's simple so please add that for v2.

Quoted 7 lines
Quoted 3 lines

This patch feels somewhat unpolished, but I guess it is mainly here to give an illustration of how stuff might fit together rather than any expectation of detailed review.

Yeah, this is definitely an early look in the spirit of "Release early / release often".

Definitely a good idea.

...

Quoted 25 lines
Quoted 13 lines
Quoted 13 lines

static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)

 {
      struct cxl_mem *cxlm = ERR_PTR(-ENXIO);
      struct device *dev = &pdev->dev;
drivers/cxl/mem.c +1 -0
+     struct cxl_memdev *cxlmd;
      int rc, regloc, i;
 
      rc = cxl_bus_prepared(pdev);
@@ -319,20 +545,31 @@ static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id)
      if (rc)
              return rc;
 
Quoted 11 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 327

Quoted 7 lines
drivers/cxl/mem.c +1 -4
-     /* Check that hardware "looks" okay. */
-     rc = cxl_mem_mbox_get(cxlm);
+     rc = cxl_mem_identify(cxlm);
      if (rc)
              return rc;
-
-     cxl_mem_mbox_put(cxlm);  

It was kind of nice to see the flow earlier, but I'm also thinking it made a slightly harder to read patch. Hmm. Maybe just drop the version earlier in favour of a todo comment that you then do here?

Quoted 3 lines

Not sure I follow, but I think you're saying don't bother with an initial patch introducing just doing the raw cxl_mem_mbox_get() in this path, jump straight to cxl_mem_identify()?

Exactly.

Quoted 19 lines
Quoted 6 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 552

Quoted 4 lines
drivers/cxl/mem.c +1 -0
      dev_dbg(&pdev->dev, "CXL Memory Device Interface Up\n");
+  

Nice to tidy that up by moving to earlier patch.

Sure.

Quoted 10 lines

drivers/cxl/mem.c static int cxl_mem_probe(struct pci_dev *pdev, const struct pci_device_id *id) · line 557

Quoted 6 lines
drivers/cxl/mem.c +3 -0
      pci_set_drvdata(pdev, cxlm);
 
+     cxlmd = cxl_mem_add_memdev(cxlm);
+     if (IS_ERR(cxlmd))
+             return PTR_ERR(cxlmd);  

Given we don't actually use cxlmd perhaps a simple return value of 0 or error would be better from cxl_mem_add_memdev()

Quoted 7 lines
Quoted 3 lines

(I guess you may have follow up patches that do something with it
here, though it feels wrong to ever do so given it is now registered and hence exposed to the system).

It's not added if IS_ERR() is true, but it would be simpler to just have cxl_mem_add_memdev() return an int since ->probe() doesn't use it.

Agreed.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
drivers/cxl/mem.c +1 -0
+
      return 0;
 }
 

...

#67 lore raw

RFC PATCH 9/9 MAINTAINERS: Add maintainers of the CXL driver

In reply to Ben Widawsky
Cc Dan Williams <[email protected]>
Cc Vishal Verma <[email protected]>
Cc Ira Weiny <[email protected]>
---
 MAINTAINERS | 9 +++++++++
 1 file changed, 9 insertions(+)
MAINTAINERS +9 -0
index b516bb34a8d5..25e294031376 100644
@@ -4428,6 +4428,15 @@ F:	fs/configfs/
 F:	include/linux/configfs.h
 F:	samples/configfs/
 
+COMPUTE EXPRESS LINK (CXL)
+M:	Vishal Verma <[email protected]>
+M:	Ira Weiny <[email protected]>
+M:	Ben Widawsky <[email protected]>
+M:	Dan Williams <[email protected]>
+L:	[email protected]
+S:	Maintained
+F:	drivers/cxl/
+
 CONSOLE SUBSYSTEM
 M:	Greg Kroah-Hartman <[email protected]>
 S:	Supported
Signature · 1 line
2.29.2
#69 lore raw
In reply to Ben Widawsky
Quoted 10 lines · Ben Widawsky

...
Ben Widawsky (5):
cxl/mem: Map memory device registers
cxl/mem: Find device capabilities
cxl/mem: Initialize the mailbox interface
cxl/mem: Implement polled mode mailbox
MAINTAINERS: Add maintainers of the CXL driver

Dan Williams (2):
cxl/mem: Add a driver for the type-3 mailbox

To include important words first and use "Type 3" as in spec:

cxl/mem: Add Type 3 mailbox driver

Quoted 5 lines

cxl/mem: Register CXL memX devices

Vishal Verma (2):
cxl/acpi: Add an acpi_cxl module for the CXL interconnect cxl/acpi: add OSC support

For consistency:

cxl/acpi: Add _OSC support

It's conventional in drivers/acpi and drivers/pci to capitalize the
"ACPI" and "PCI" initialisms except in actual C code. Seems like
you're mostly doing the same with "CXL", except in the subject lines above. Since you're making a new directory, I guess you get to choose.

I use "PCIe" (not "PCIE" or "PCI-E"; you have a mix) because that seems to be the official abbreviation.