Skip to contentSkip to conversation

PCI Data Object Exchange support + CXL CDAT

PATCH In review Jonathan Cameron 4 participants 16 messages May 24, 2021 – Jul 8, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

v4 cover PCI Data Object Exchange support + CXL CDAT

Series first introduces generic support for DOE mailboxes as defined in the ECN to the PCIe r5.0 specification available from the PCI SIG [0]

A user is then introduced in the form of the table access protocol defined in the CXL 2.0 specification [1] used to access the
Coherent Device Attribute Table (CDAT) defined in [2]

Finally, in two patches that are not intended for merging, an example of a generic IOCTL interface to perform synchronous exchanges from user space in a kernel mediated fashion is introduced. The current consensus seems in favour of not introducing such an interface, but instead providing per protocol interfaces where appropriate. As this code was developed in parallel with that discussion and may be of use to someone it is included here.

Open questions.
* Do we need to prevent userspace access to the DOE when the kernel is managing it (registered by driver)? Discussion in thread:
https://lore.kernel.org/linux-pci/[email protected]/
* Does a generic userspace mediation interface make sense? Currently it seems not, but as it was nearly ready, I've included it in this patch set anyway.
* Move CXL cdat handling to a separate file, or wait until we can see how it is going to be used?

Changes since v3
Thanks to Ira, Bjorn and Dan for feedback.
* Addition of a generic IOCTL interface and demo program for discussion.
* Fixed an issue with accidentally disabling interrupts.
* Ensure IRQ_HANDLED always returned as clearing of BUSY can result in an interrupt, but there is no means of identifying this. The best that can be done is to eat the interrupt.
* Edited comments to more directly tie them to the code the referred to.
* Lock scope documentation update.
* Dropped the CDAT dump to log patch as we are hopefully getting closer to this being applied.
* Fixed the references to pcie_ missed when renaming to pci_ in v3.

Set based on cxl/next

All testing conducted against QEMU emulation of a CXL type 3 device in conjunction with DOE mailbox patches v5 [3, 4]

[0] https://pcisig.com/specifications
[1] https://www.computeexpresslink.org/download-the-specification
[2] https://uefi.org/node/4093
[3] https://lore.kernel.org/qemu-devel/[email protected]/
[4] https://lore.kernel.org/qemu-devel/[email protected]/

Jonathan Cameron (5):
PCI: Add vendor ID for the PCI SIG
PCI/DOE: Add Data Object Exchange support
cxl/mem: Add CDAT table reading from DOE
DONOTMERGE PCI/DOE Add per DOE chrdev for ioctl based access
DONOTMERGE: PCI/DOE: Add userspace example program to tools/pci

 drivers/cxl/Kconfig           |   1 +
 drivers/cxl/cxl.h             |  21 +
 drivers/cxl/mem.c             | 174 ++++++++
 drivers/cxl/mem.h             |   6 +
 drivers/pci/Kconfig           |   8 +
 drivers/pci/Makefile          |   1 +
 drivers/pci/doe.c             | 794 ++++++++++++++++++++++++++++++++++
 drivers/pci/pci-driver.c      |   3 +-
 include/linux/pci-doe.h       | 100 +++++
 include/linux/pci.h           |   3 +
 include/linux/pci_ids.h       |   1 +
 include/uapi/linux/pci_doe.h  |  32 ++
 include/uapi/linux/pci_regs.h |  29 +-
 tools/pci/Build               |   1 +
 tools/pci/Makefile            |   9 +-
 tools/pci/doetest.c           | 131 ++++++
 16 files changed, 1311 insertions(+), 3 deletions(-)
 create mode 100644 drivers/pci/doe.c
 create mode 100644 include/linux/pci-doe.h
 create mode 100644 include/uapi/linux/pci_doe.h
 create mode 100644 tools/pci/doetest.c

base-commit: 35c32e3095d396c750f5cdfdaa94cba83d9b23c6

Signature · 1 line
2.19.1
#1 lore raw

PATCH v4 1/5 PCI: Add vendor ID for the PCI SIG

This ID is used in DOE headers to identify protocols that are defined within the PCI Express Base Specification.

Specified in Table 7-x2 of the Data Object Exchange ECN (approved 12 March 2020) available from https://members.pcisig.com/wg/PCI-SIG/document/14143

---
 include/linux/pci_ids.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/pci_ids.h +1 -0
index 4c3fa5293d76..dcc8b4b14198 100644
@@ -149,6 +149,7 @@
 #define PCI_CLASS_OTHERS		0xff
 
 /* Vendors and devices.  Sort key: vendor first, device next. */
+#define PCI_VENDOR_ID_PCI_SIG		0x0001
 
 #define PCI_VENDOR_ID_LOONGSON		0x0014
 
Signature · 1 line
2.19.1
#2 lore raw

Re: [PATCH v4 1/5] PCI: Add vendor ID for the PCI SIG

Quoted 17 lines · Jonathan Cameron

This ID is used in DOE headers to identify protocols that are defined within the PCI Express Base Specification.

Specified in Table 7-x2 of the Data Object Exchange ECN (approved 12 March 2020) available from https://members.pcisig.com/wg/PCI-SIG/document/14143

---
 include/linux/pci_ids.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/pci_ids.h +0 -0
index 4c3fa5293d76..dcc8b4b14198 100644

include/linux/pci_ids.h line 152

Quoted 4 lines
include/linux/pci_ids.h +1 -0
@@ -149,6 +149,7 @@
 #define PCI_CLASS_OTHERS               0xff
 
 /* Vendors and devices.  Sort key: vendor first, device next. */
+#define PCI_VENDOR_ID_PCI_SIG          0x0001

Should this not be:

PCI_DOE_VENDOR_ID_PCI_SIG?

...because I don't think this value will ever show up at the typical config-offset 0 vendor-id, will it?

#3 lore raw

Re: [PATCH v4 1/5] PCI: Add vendor ID for the PCI SIG

Quoted 26 lines · On Thu, 10 Jun 2021 08:17:23 -0700 Dan Williams <[email protected]>…
Quoted 17 lines · Jonathan Cameron

This ID is used in DOE headers to identify protocols that are defined within the PCI Express Base Specification.

Specified in Table 7-x2 of the Data Object Exchange ECN (approved 12 March 2020) available from https://members.pcisig.com/wg/PCI-SIG/document/14143

---
 include/linux/pci_ids.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/pci_ids.h +0 -0
index 4c3fa5293d76..dcc8b4b14198 100644

include/linux/pci_ids.h line 152

Quoted 4 lines
include/linux/pci_ids.h +1 -0
@@ -149,6 +149,7 @@
 #define PCI_CLASS_OTHERS               0xff
 
 /* Vendors and devices.  Sort key: vendor first, device next. */
+#define PCI_VENDOR_ID_PCI_SIG          0x0001  

Should this not be:

Quoted 4 lines

PCI_DOE_VENDOR_ID_PCI_SIG?

...because I don't think this value will ever show up at the typical config-offset 0 vendor-id, will it?

Good question.

Whilst I agree it is unlikely to turn up as a conventional vendor-id (though I've not found any text ruling it out) it already turns up in locations other than DOE.

Many of them aren't software visible, but potentially places like SPDM are in which you would have a registry ID of 0x3 (PCI-SIG) followed by the PCI vendor ID (this one). Those are used in SPDM vendor defined requests / responses.

That SPDM feature is then used in IDE establishment.
The IDE ECN (via pcisig.com) has the following:
"The VendorID field of the VENDOR_DEFINED_REQUEST/
VENDOR_DEFINED_RESPONSE must contain the value 0001h, which is assigned to the PCI-SIG."

Which to my reading, isn't quite the same as saying it's a vendor ID, but nearly so.

Now, I argued the *_DVSEC_* naming in the CXL one based on the spec saying that was all it could be used for but I may well have been wrong longer term.

I'm fine with renaming it to the PCI_DOE_* version then dropping the DOE when it gets used for something else though if that works for people.

At least this time naming isn't made awkward by legalese.

Jonathan

Reviewed-by
#4 lore raw

Re: [PATCH v4 1/5] PCI: Add vendor ID for the PCI SIG

Quoted 55 lines · Jonathan Cameron
Quoted 26 lines · On Thu, 10 Jun 2021 08:17:23 -0700 Dan Williams <[email protected]>…
Quoted 17 lines · Jonathan Cameron

This ID is used in DOE headers to identify protocols that are defined within the PCI Express Base Specification.

Specified in Table 7-x2 of the Data Object Exchange ECN (approved 12 March 2020) available from https://members.pcisig.com/wg/PCI-SIG/document/14143

---
 include/linux/pci_ids.h | 1 +
 1 file changed, 1 insertion(+)
include/linux/pci_ids.h +0 -0
index 4c3fa5293d76..dcc8b4b14198 100644

include/linux/pci_ids.h line 152

Quoted 4 lines
include/linux/pci_ids.h +1 -0
@@ -149,6 +149,7 @@
 #define PCI_CLASS_OTHERS               0xff
 
 /* Vendors and devices.  Sort key: vendor first, device next. */
+#define PCI_VENDOR_ID_PCI_SIG          0x0001

Should this not be:

Quoted 4 lines

PCI_DOE_VENDOR_ID_PCI_SIG?

...because I don't think this value will ever show up at the typical config-offset 0 vendor-id, will it?

Good question.

Whilst I agree it is unlikely to turn up as a conventional vendor-id (though I've not found any text ruling it out) it already turns up in locations other than DOE.

Many of them aren't software visible, but potentially places like SPDM are in which you would have a registry ID of 0x3 (PCI-SIG) followed by the PCI vendor ID (this one). Those are used in SPDM vendor defined requests / responses.

That SPDM feature is then used in IDE establishment.
The IDE ECN (via pcisig.com) has the following:
"The VendorID field of the VENDOR_DEFINED_REQUEST/
VENDOR_DEFINED_RESPONSE must contain the value 0001h, which is assigned to the PCI-SIG."

Which to my reading, isn't quite the same as saying it's a vendor ID, but nearly so.

Quoted 8 lines

Now, I argued the *_DVSEC_* naming in the CXL one based on the spec saying that was all it could be used for but I may well have been wrong longer term.

I'm fine with renaming it to the PCI_DOE_* version then dropping the DOE when it gets used for something else though if that works for people.

At least this time naming isn't made awkward by legalese.

For fun I did a lookup for vendor-id 1 and it came back "Fry's Electronics Counterfeit Flash Drive"

https://pcilookup.com/?ven=0001&dev=&action=submit

The potential for it to be used in other places outside of DOE makes me think the way you have it here is fine.

#5 lore raw

PATCH v4 2/5 PCI/DOE: Add Data Object Exchange support

In reply to Jonathan Cameron

Introduced in a PCI ECN [1], DOE provides a config space based mailbox with standard protocol discovery. Each mailbox is accessed through a DOE Extended Capability.

A device may have 1 or more DOE mailboxes, each of which is allowed to support any number of protocols (some DOE protocol specifications apply additional restrictions). A given protocol may be supported on more than one DOE mailbox on a given function.

If a driver wishes to access any number of DOE instances / protocols it makes a single call to pci_doe_register_all() which will find available DOEs, create the required infrastructure and cache the protocols they support. pci_doe_find() can then retrieve a pointer to an appropriate DOE instance.

A synchronous interface is provided in pci_doe_exchange_sync() to perform a single query / response exchange.

Testing conducted against QEMU using:

https://lore.kernel.org/qemu-devel/[email protected]/

[1] https://members.pcisig.com/wg/PCI-SIG/document/14143
Data Object Exchange (DOE) - Approved 12 March 2020

---
 drivers/pci/Kconfig           |   8 +
 drivers/pci/Makefile          |   1 +
 drivers/pci/doe.c             | 626 ++++++++++++++++++++++++++++++++++
 include/linux/pci-doe.h       |  87 +++++
 include/linux/pci.h           |   3 +
 include/uapi/linux/pci_regs.h |  29 +-
 6 files changed, 753 insertions(+), 1 deletion(-)
drivers/pci/Kconfig +8 -0
index 0c473d75e625..a30c59cf5e27 100644
@@ -190,6 +190,14 @@ config PCI_HYPERV
 	  The PCI device frontend driver allows the kernel to import arbitrary
 	  PCI devices from a PCI backend to support PCI driver domains.
 
+config PCI_DOE
+	bool
+	help
+	  This enables library support for the PCI Data Object Exchange
+	  capability. DOE provides a simple mailbox in PCI config space that is
+	  used by a number of different protocols.
+	  DOE is defined in the Data Object Exchange ECN to the PCIe r5.0 spec.
+
 choice
 	prompt "PCI Express hierarchy optimization setting"
 	default PCIE_BUS_DEFAULT
drivers/pci/Makefile +1 -0
index d62c4ac4ae1b..1b61c1a1c232 100644
@@ -28,6 +28,7 @@ obj-$(CONFIG_PCI_STUB)		+= pci-stub.o
 obj-$(CONFIG_PCI_PF_STUB)	+= pci-pf-stub.o
 obj-$(CONFIG_PCI_ECAM)		+= ecam.o
 obj-$(CONFIG_PCI_P2PDMA)	+= p2pdma.o
+obj-$(CONFIG_PCI_DOE)		+= doe.o
 obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o
 
 # Endpoint library must be initialized before its users
drivers/pci/doe.c new file +626 -0

626 lines not on this page · raw message

include/linux/pci-doe.h new file +87 -0

87 lines not on this page · raw message

include/linux/pci.h +3 -0
index c20211e59a57..2250a00ad8c2 100644
@@ -332,6 +332,9 @@ struct pci_dev {
 #ifdef CONFIG_PCIEPORTBUS
 	struct rcec_ea	*rcec_ea;	/* RCEC cached endpoint association */
 	struct pci_dev  *rcec;          /* Associated RCEC device */
+#endif
+#ifdef CONFIG_PCI_DOE
+	struct list_head doe_list;	/* Data Object Exchange mailboxes */
 #endif
 	u8		pcie_cap;	/* PCIe capability offset */
 	u8		msi_cap;	/* MSI capability offset */
include/uapi/linux/pci_regs.h +28 -1
index e709ae8235e7..b97df1d8bd19 100644
@@ -730,7 +730,8 @@
 #define PCI_EXT_CAP_ID_DVSEC	0x23	/* Designated Vendor-Specific */
 #define PCI_EXT_CAP_ID_DLF	0x25	/* Data Link Feature */
 #define PCI_EXT_CAP_ID_PL_16GT	0x26	/* Physical Layer 16.0 GT/s */
-#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_PL_16GT
+#define PCI_EXT_CAP_ID_DOE	0x2E	/* Data Object Exchange */
+#define PCI_EXT_CAP_ID_MAX	PCI_EXT_CAP_ID_DOE
 
 #define PCI_EXT_CAP_DSN_SIZEOF	12
 #define PCI_EXT_CAP_MCAST_ENDPOINT_SIZEOF 40
@@ -1092,4 +1093,30 @@
 #define  PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_MASK		0x000000F0
 #define  PCI_PL_16GT_LE_CTRL_USP_TX_PRESET_SHIFT	4
 
+/* Data Object Exchange */
+#define PCI_DOE_CAP		0x04	/* DOE Capabilities Register */
+#define  PCI_DOE_CAP_INT			0x00000001  /* Interrupt Support */
+#define  PCI_DOE_CAP_IRQ			0x00000ffe  /* Interrupt Message Number */
+#define PCI_DOE_CTRL		0x08	/* DOE Control Register */
+#define  PCI_DOE_CTRL_ABORT			0x00000001  /* DOE Abort */
+#define  PCI_DOE_CTRL_INT_EN			0x00000002  /* DOE Interrupt Enable */
+#define  PCI_DOE_CTRL_GO			0x80000000  /* DOE Go */
+#define PCI_DOE_STATUS		0x0c	/* DOE Status Register */
+#define  PCI_DOE_STATUS_BUSY			0x00000001  /* DOE Busy */
+#define  PCI_DOE_STATUS_INT_STATUS		0x00000002  /* DOE Interrupt Status */
+#define  PCI_DOE_STATUS_ERROR			0x00000004  /* DOE Error */
+#define  PCI_DOE_STATUS_DATA_OBJECT_READY	0x80000000  /* Data Object Ready */
+#define PCI_DOE_WRITE		0x10	/* DOE Write Data Mailbox Register */
+#define PCI_DOE_READ		0x14	/* DOE Read Data Mailbox Register */
+
+/* DOE Data Object - note not actually registers */
+#define PCI_DOE_DATA_OBJECT_HEADER_1_VID	0x0000ffff
+#define PCI_DOE_DATA_OBJECT_HEADER_1_TYPE	0x00ff0000
+#define PCI_DOE_DATA_OBJECT_HEADER_2_LENGTH	0x0003ffff
+
+#define PCI_DOE_DATA_OBJECT_DISC_REQ_3_INDEX	0x000000ff
+#define PCI_DOE_DATA_OBJECT_DISC_RSP_3_VID	0x0000ffff
+#define PCI_DOE_DATA_OBJECT_DISC_RSP_3_PROTOCOL	0x00ff0000
+#define PCI_DOE_DATA_OBJECT_DISC_RSP_3_NEXT_INDEX 0xff000000
+
 #endif /* LINUX_PCI_REGS_H */
Signature · 1 line
2.19.1
#6 lore raw

Re: [PATCH v4 2/5] PCI/DOE: Add Data Object Exchange support

Quoted 16 lines · Jonathan Cameron

Introduced in a PCI ECN [1], DOE provides a config space based mailbox with standard protocol discovery. Each mailbox is accessed through a DOE Extended Capability.

A device may have 1 or more DOE mailboxes, each of which is allowed to support any number of protocols (some DOE protocol specifications apply additional restrictions). A given protocol may be supported on more than one DOE mailbox on a given function.

If a driver wishes to access any number of DOE instances / protocols it makes a single call to pci_doe_register_all() which will find available DOEs, create the required infrastructure and cache the protocols they support. pci_doe_find() can then retrieve a pointer to an appropriate DOE instance.

Quoted 6 lines

A synchronous interface is provided in pci_doe_exchange_sync() to perform a single query / response exchange.

Testing conducted against QEMU using:

https://lore.kernel.org/qemu-devel/[email protected]/

Nice.

I was hoping that by now QEMU upstream would have given us some indication that this useful work that has a chance of being merged. I fear it's only us CXL practitioner's that care. Perhaps the PCI IDE support will get them to move on at least the DOE patches?

Quoted 5 lines

[1] https://members.pcisig.com/wg/PCI-SIG/document/14143
Data Object Exchange (DOE) - Approved 12 March 2020

The core logic of this looks good to me. The interfaces for other code to make use of this I feel can lean heavier on existing mechanics. A few points come to mind:

- Does this need to support anything more than queue depth 1? I know the specification seems to allow for some overlapping and queueing, but I don't think there are any use cases that are precluded if the max number of tasks in flight for a given DOE is one.

- Once its queue depth 1 then the list of tasks can be replaced with a wait_queue_head_t where submitters wait for the previous task to finish.

- This appears to be the prototypical scenario for deploying the new auxiliary bus facility. Rather than custom code device-like facilities (lists and parents etc) in 'struct pci_doe' just make pci_doe a device directly (auxiliary-device) and separate the infrastructure that drives that device to a driver (auxiliary-driver). That makes the lifetime management more idiomatic, allows for user space to have typical driver-binding controls to manage kernel-user DOE conflicts, and it allows for typical driver services like devm.

#7 lore raw

Re: [PATCH v4 2/5] PCI/DOE: Add Data Object Exchange support

Quoted 52 lines · Dan Williams
Quoted 16 lines · Jonathan Cameron

Introduced in a PCI ECN [1], DOE provides a config space based mailbox with standard protocol discovery. Each mailbox is accessed through a DOE Extended Capability.

A device may have 1 or more DOE mailboxes, each of which is allowed to support any number of protocols (some DOE protocol specifications apply additional restrictions). A given protocol may be supported on more than one DOE mailbox on a given function.

If a driver wishes to access any number of DOE instances / protocols it makes a single call to pci_doe_register_all() which will find available DOEs, create the required infrastructure and cache the protocols they support. pci_doe_find() can then retrieve a pointer to an appropriate DOE instance.

Quoted 6 lines

A synchronous interface is provided in pci_doe_exchange_sync() to perform a single query / response exchange.

Testing conducted against QEMU using:

https://lore.kernel.org/qemu-devel/[email protected]/

Nice.

I was hoping that by now QEMU upstream would have given us some indication that this useful work that has a chance of being merged. I fear it's only us CXL practitioner's that care. Perhaps the PCI IDE support will get them to move on at least the DOE patches?

Quoted 5 lines

[1] https://members.pcisig.com/wg/PCI-SIG/document/14143
Data Object Exchange (DOE) - Approved 12 March 2020

The core logic of this looks good to me. The interfaces for other code to make use of this I feel can lean heavier on existing mechanics. A few points come to mind:

- Does this need to support anything more than queue depth 1? I know the specification seems to allow for some overlapping and queueing, but I don't think there are any use cases that are precluded if the max number of tasks in flight for a given DOE is one.

- Once its queue depth 1 then the list of tasks can be replaced with a wait_queue_head_t where submitters wait for the previous task to finish.

Quoted 8 lines

- This appears to be the prototypical scenario for deploying the new auxiliary bus facility. Rather than custom code device-like facilities (lists and parents etc) in 'struct pci_doe' just make pci_doe a device directly (auxiliary-device) and separate the infrastructure that drives that device to a driver (auxiliary-driver). That makes the lifetime management more idiomatic, allows for user space to have typical driver-binding controls to manage kernel-user DOE conflicts, and it allows for typical driver services like devm.

Hi Jonathan,

Are you waiting on me to take a shot at refactoring the DOE driver into this proposed auxiliary device/driver organization? I am happy to do that if you've moved on to looking at the kernel-side SPDM implementation [1].

I would expect DOE, SPDM, and IDE would be a useful topic to discuss at the the Plumbers PCI Microconference assuming we do not solve all the open issues before September.

[1]: https://lore.kernel.org/r/[email protected]

#8 lore raw

Re: [PATCH v4 2/5] PCI/DOE: Add Data Object Exchange support

Quoted 64 lines · Dan Williams
Quoted 52 lines · Dan Williams
Quoted 16 lines · Jonathan Cameron

Introduced in a PCI ECN [1], DOE provides a config space based mailbox with standard protocol discovery. Each mailbox is accessed through a DOE Extended Capability.

A device may have 1 or more DOE mailboxes, each of which is allowed to support any number of protocols (some DOE protocol specifications apply additional restrictions). A given protocol may be supported on more than one DOE mailbox on a given function.

If a driver wishes to access any number of DOE instances / protocols it makes a single call to pci_doe_register_all() which will find available DOEs, create the required infrastructure and cache the protocols they support. pci_doe_find() can then retrieve a pointer to an appropriate DOE instance.

Quoted 6 lines

A synchronous interface is provided in pci_doe_exchange_sync() to perform a single query / response exchange.

Testing conducted against QEMU using:

https://lore.kernel.org/qemu-devel/[email protected]/

Nice.

I was hoping that by now QEMU upstream would have given us some indication that this useful work that has a chance of being merged. I fear it's only us CXL practitioner's that care. Perhaps the PCI IDE support will get them to move on at least the DOE patches?

Quoted 5 lines

[1] https://members.pcisig.com/wg/PCI-SIG/document/14143
Data Object Exchange (DOE) - Approved 12 March 2020

The core logic of this looks good to me. The interfaces for other code to make use of this I feel can lean heavier on existing mechanics. A few points come to mind:

- Does this need to support anything more than queue depth 1? I know the specification seems to allow for some overlapping and queueing, but I don't think there are any use cases that are precluded if the max number of tasks in flight for a given DOE is one.

- Once its queue depth 1 then the list of tasks can be replaced with a wait_queue_head_t where submitters wait for the previous task to finish.

Quoted 8 lines

- This appears to be the prototypical scenario for deploying the new auxiliary bus facility. Rather than custom code device-like facilities (lists and parents etc) in 'struct pci_doe' just make pci_doe a device directly (auxiliary-device) and separate the infrastructure that drives that device to a driver (auxiliary-driver). That makes the lifetime management more idiomatic, allows for user space to have typical driver-binding controls to manage kernel-user DOE conflicts, and it allows for typical driver services like devm.

Hi Jonathan,

Quoted 8 lines

Are you waiting on me to take a shot at refactoring the DOE driver into this proposed auxiliary device/driver organization? I am happy to do that if you've moved on to looking at the kernel-side SPDM implementation [1].

I would expect DOE, SPDM, and IDE would be a useful topic to discuss at the the Plumbers PCI Microconference assuming we do not solve all the open issues before September.

Definitely, I will make sure we schedule a slot on these topics.

Thanks,
Lorenzo

Quoted 1 line

[1]: https://lore.kernel.org/r/[email protected]

#9 lore raw

Re: [PATCH v4 2/5] PCI/DOE: Add Data Object Exchange support

Quoted 41 lines · On Thu, 8 Jul 2021 09:48:15 +0100 Lorenzo Pieralisi…
Quoted 40 lines · Dan Williams
Quoted 39 lines · Dan Williams
Quoted 16 lines · Jonathan Cameron

Introduced in a PCI ECN [1], DOE provides a config space based mailbox with standard protocol discovery. Each mailbox is accessed through a DOE Extended Capability.

A device may have 1 or more DOE mailboxes, each of which is allowed to support any number of protocols (some DOE protocol specifications apply additional restrictions). A given protocol may be supported on more than one DOE mailbox on a given function.

If a driver wishes to access any number of DOE instances / protocols it makes a single call to pci_doe_register_all() which will find available DOEs, create the required infrastructure and cache the protocols they support. pci_doe_find() can then retrieve a pointer to an appropriate DOE instance.

Quoted 6 lines

A synchronous interface is provided in pci_doe_exchange_sync() to perform a single query / response exchange.

Testing conducted against QEMU using:

https://lore.kernel.org/qemu-devel/[email protected]/

Nice.

I was hoping that by now QEMU upstream would have given us some indication that this useful work that has a chance of being merged. I fear it's only us CXL practitioner's that care. Perhaps the PCI IDE support will get them to move on at least the DOE patches?

Quoted 5 lines

[1] https://members.pcisig.com/wg/PCI-SIG/document/14143
Data Object Exchange (DOE) - Approved 12 March 2020

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

The core logic of this looks good to me. The interfaces for other code to make use of this I feel can lean heavier on existing mechanics. A few points come to mind:

- Does this need to support anything more than queue depth 1? I know the specification seems to allow for some overlapping and queueing, but I don't think there are any use cases that are precluded if the max number of tasks in flight for a given DOE is one.

We aren't handling overlapping today, so a queue of one and wait on access to that would work fine. Queuing is more likely to be about multiple protocols sharing a DOE (where allowed) as I'm not seeing a need in any individual protocol to queue lots of things up at once. You could queue a few things in CMA as they aren't all dependent on each other (though the order matters for building the measurements) but I'm not sure why you would.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

- Once its queue depth 1 then the list of tasks can be replaced with a wait_queue_head_t where submitters wait for the previous task to finish.

I'll go with a resounding 'maybe' on that... I'm not sure I like that the async path in the submitter could be waiting a very long time (if we think anyone will actually do async...) I'm probably missing something.

Quoted 10 lines
Quoted 10 lines
Quoted 9 lines

- This appears to be the prototypical scenario for deploying the new auxiliary bus facility. Rather than custom code device-like facilities (lists and parents etc) in 'struct pci_doe' just make pci_doe a device directly (auxiliary-device) and separate the infrastructure that drives that device to a driver (auxiliary-driver). That makes the lifetime management more idiomatic, allows for user space to have typical driver-binding controls to manage kernel-user DOE conflicts, and it allows for typical driver services like devm.

Quoted 6 lines
Quoted 6 lines

Hi Jonathan,

Are you waiting on me to take a shot at refactoring the DOE driver into this proposed auxiliary device/driver organization? I am happy to do that if you've moved on to looking at the kernel-side SPDM implementation [1].

I wasn't really expecting you to take a shot, but if you have time, that would be great. I've not looked at the auxiliary bus stuff yet.

Whilst I have a PoC up and running for SPDM, it's a bunch of dirty hacks and I suspect that, even when cleaned up, it's going to keep me busy for a while.

Mostly I've developed a steadily growing hatred for the different ways in which signatures can be stored and hand decoding records or digging in openspdm to figure out what order and encoding things are in this time...

Quoted 6 lines
Quoted 4 lines

I would expect DOE, SPDM, and IDE would be a useful topic to discuss at the the Plumbers PCI Microconference assuming we do not solve all the open issues before September.

Definitely, I will make sure we schedule a slot on these topics.

+1

I'm sure there will be some issues left :)

Quoted 5 lines

Thanks,
Lorenzo

Quoted 1 line

[1]: https://lore.kernel.org/r/[email protected]

#10 lore raw

PATCH v4 3/5 cxl/mem: Add CDAT table reading from DOE

In reply to Jonathan Cameron

This patch provides a sysfs binary attribute to allow dumping of the whole table.

Binary dumping is modeled on /sys/firmware/ACPI/tables/

The ability to dump this table will be very useful for emulation of real devices once they become available as QEMU CXL type 3 device emulation will be able to load this file in.

This does not support table updates at runtime. It will always provide whatever was there when first cached. Handling of table updates can be implemented later.

Once we have more users, this code can move out to driver/cxl/cdat.c or similar.

---
 drivers/cxl/Kconfig |   1 +
 drivers/cxl/cxl.h   |  21 ++++++
 drivers/cxl/mem.c   | 174 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.h   |   6 ++
 4 files changed, 202 insertions(+)
drivers/cxl/Kconfig +1 -0
index 97dc4d751651..26cad9fa29f7 100644
@@ -15,6 +15,7 @@ if CXL_BUS
 
 config CXL_MEM
 	tristate "CXL.mem: Memory Devices"
+	select PCI_DOE
 	help
 	  The CXL.mem protocol allows a device to act as a provider of
 	  "System RAM" and/or "Persistent Memory" that is fully coherent
drivers/cxl/cxl.h +21 -0
index d49e0cb679fa..e649a286aace 100644
@@ -7,6 +7,7 @@
 #include <linux/bitfield.h>
 #include <linux/bitops.h>
 #include <linux/io.h>
+#include <linux/pci-doe.h>
 
 /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
 #define CXLDEV_CAP_ARRAY_OFFSET 0x0
@@ -69,5 +70,25 @@ struct cxl_regs {
 void cxl_setup_device_regs(struct device *dev, void __iomem *base,
 			   struct cxl_device_regs *regs);
 
+/*
+ * Address space properties derived from:
+ * CXL 2.0 8.2.5.12.7 CXL HDM Decoder 0 Control Register
+ */
+#define CXL_ADDRSPACE_RAM   BIT(0)
+#define CXL_ADDRSPACE_PMEM  BIT(1)
+#define CXL_ADDRSPACE_TYPE2 BIT(2)
+#define CXL_ADDRSPACE_TYPE3 BIT(3)
+#define CXL_ADDRSPACE_MASK  GENMASK(3, 0)
+
+#define CXL_DOE_PROTOCOL_COMPLIANCE 0
+#define CXL_DOE_PROTOCOL_TABLE_ACCESS 2
+
+/* Common to request and response */
+#define CXL_DOE_TABLE_ACCESS_3_CODE GENMASK(7, 0)
+#define   CXL_DOE_TABLE_ACCESS_3_CODE_READ 0
+#define CXL_DOE_TABLE_ACCESS_3_TYPE GENMASK(15, 8)
+#define   CXL_DOE_TABLE_ACCESS_3_TYPE_CDAT 0
+#define CXL_DOE_TABLE_ACCESS_3_ENTRY_HANDLE GENMASK(31, 16)
+
 extern struct bus_type cxl_bus_type;
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +174 -0

228 lines not on this page · raw message

drivers/cxl/mem.h +6 -0
index 0a3f70316872..fb26155a8fb3 100644
@@ -38,12 +38,16 @@
  * @cdev: char dev core object for ioctl operations
  * @cxlm: pointer to the parent device driver data
  * @id: id number of this memdev instance.
+ * @cdat_table: cache of CDAT table
+ * @cdat_length: length of cached CDAT table
  */
 struct cxl_memdev {
 	struct device dev;
 	struct cdev cdev;
 	struct cxl_mem *cxlm;
 	int id;
+	void *cdat_table;
+	size_t cdat_length;
 };
 
 /**
@@ -51,6 +55,7 @@ struct cxl_memdev {
  * @pdev: The PCI device associated with this CXL device.
  * @base: IO mappings to the device's MMIO
  * @cxlmd: Logical memory device chardev / interface
+ * @table_doe: Data exchange object mailbox used to read tables
  * @regs: Parsed register blocks
  * @payload_size: Size of space for payload
  *                (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
@@ -65,6 +70,7 @@ struct cxl_mem {
 	void __iomem *base;
 	struct cxl_memdev *cxlmd;
 
+	struct pci_doe *table_doe;
 	struct cxl_regs regs;
 
 	size_t payload_size;
Signature · 1 line
2.19.1
#11 lore raw

Re: [PATCH v4 3/5] cxl/mem: Add CDAT table reading from DOE

Quoted 52 lines · Jonathan Cameron

This patch provides a sysfs binary attribute to allow dumping of the whole table.

Binary dumping is modeled on /sys/firmware/ACPI/tables/

The ability to dump this table will be very useful for emulation of real devices once they become available as QEMU CXL type 3 device emulation will be able to load this file in.

This does not support table updates at runtime. It will always provide whatever was there when first cached. Handling of table updates can be implemented later.

Once we have more users, this code can move out to driver/cxl/cdat.c or similar.

---
 drivers/cxl/Kconfig |   1 +
 drivers/cxl/cxl.h   |  21 ++++++
 drivers/cxl/mem.c   | 174 ++++++++++++++++++++++++++++++++++++++++++++
 drivers/cxl/mem.h   |   6 ++
 4 files changed, 202 insertions(+)
drivers/cxl/Kconfig +1 -0
index 97dc4d751651..26cad9fa29f7 100644
@@ -15,6 +15,7 @@ if CXL_BUS
 
 config CXL_MEM
        tristate "CXL.mem: Memory Devices"
+       select PCI_DOE
        help
          The CXL.mem protocol allows a device to act as a provider of
          "System RAM" and/or "Persistent Memory" that is fully coherent
drivers/cxl/cxl.h +1 -0
index d49e0cb679fa..e649a286aace 100644
@@ -7,6 +7,7 @@
 #include <linux/bitfield.h>
 #include <linux/bitops.h>
 #include <linux/io.h>
+#include <linux/pci-doe.h>
 
 /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */
 #define CXLDEV_CAP_ARRAY_OFFSET 0x0
@@ -69,5 +70,25 @@ struct cxl_regs {
 void cxl_setup_device_regs(struct device *dev, void __iomem *base,
                           struct cxl_device_regs *regs);

drivers/cxl/cxl.h struct cxl_regs { · line 81

Quoted 10 lines
drivers/cxl/cxl.h +9 -0
 
+/*
+ * Address space properties derived from:
+ * CXL 2.0 8.2.5.12.7 CXL HDM Decoder 0 Control Register
+ */
+#define CXL_ADDRSPACE_RAM   BIT(0)
+#define CXL_ADDRSPACE_PMEM  BIT(1)
+#define CXL_ADDRSPACE_TYPE2 BIT(2)
+#define CXL_ADDRSPACE_TYPE3 BIT(3)
+#define CXL_ADDRSPACE_MASK  GENMASK(3, 0)

Looks like this got picked up from a rebase... they're now decoder flags, but no need to include them in the CDAT patch.

Quoted 74 lines
drivers/cxl/cxl.h +11 -0
+
+#define CXL_DOE_PROTOCOL_COMPLIANCE 0
+#define CXL_DOE_PROTOCOL_TABLE_ACCESS 2
+
+/* Common to request and response */
+#define CXL_DOE_TABLE_ACCESS_3_CODE GENMASK(7, 0)
+#define   CXL_DOE_TABLE_ACCESS_3_CODE_READ 0
+#define CXL_DOE_TABLE_ACCESS_3_TYPE GENMASK(15, 8)
+#define   CXL_DOE_TABLE_ACCESS_3_TYPE_CDAT 0
+#define CXL_DOE_TABLE_ACCESS_3_ENTRY_HANDLE GENMASK(31, 16)
+
 extern struct bus_type cxl_bus_type;
 #endif /* __CXL_H__ */
drivers/cxl/mem.c +46 -0
index c5fdf2c57181..4224d1de311e 100644
@@ -14,6 +14,7 @@
 #include "pci.h"
 #include "cxl.h"
 #include "mem.h"
+#include "cdat.h"
 
 /**
  * DOC: cxl mem
@@ -926,6 +927,85 @@ static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm)
        return 0;
 }
 
+#define CDAT_DOE_REQ(entry_handle)                                     \
+       (FIELD_PREP(CXL_DOE_TABLE_ACCESS_REQ_CODE,                      \
+                   CXL_DOE_TABLE_ACCESS_REQ_CODE_READ) |               \
+        FIELD_PREP(CXL_DOE_TABLE_ACCESS_TABLE_TYPE,                    \
+                   CXL_DOE_TABLE_ACCESS_TABLE_TYPE_CDATA) |            \
+        FIELD_PREP(CXL_DOE_TABLE_ACCESS_ENTRY_HANDLE, (entry_handle)))
+
+static ssize_t cdat_get_length(struct pci_doe *doe)
+{
+       u32 cdat_request_pl = CDAT_DOE_REQ(0);
+       u32 cdat_response_pl[32];
+       struct pci_doe_exchange ex = {
+               .vid = PCI_DVSEC_VENDOR_ID_CXL,
+               .protocol = CXL_DOE_PROTOCOL_TABLE_ACCESS,
+               .request_pl = &cdat_request_pl,
+               .request_pl_sz = sizeof(cdat_request_pl),
+               .response_pl = cdat_response_pl,
+               .response_pl_sz = sizeof(cdat_response_pl),
+       };
+
+       ssize_t rc;
+
+       rc = pci_doe_exchange_sync(doe, &ex);
+       if (rc < 0)
+               return rc;
+       if (rc < 1)
+               return -EIO;
+
+       return cdat_response_pl[1];
+}
+
+static int cdat_to_buffer(struct pci_doe *doe, u32 *buffer, size_t length)
+{
+       int entry_handle = 0;
+       int rc;
+
+       do {
+               u32 cdat_request_pl = CDAT_DOE_REQ(entry_handle);
+               u32 cdat_response_pl[32];
+               struct pci_doe_exchange ex = {
+                       .vid = PCI_DVSEC_VENDOR_ID_CXL,
+                       .protocol = CXL_DOE_PROTOCOL_TABLE_ACCESS,
+                       .request_pl = &cdat_request_pl,
+                       .request_pl_sz = sizeof(cdat_request_pl),
+                       .response_pl = cdat_response_pl,

drivers/cxl/mem.c static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm) · line 984

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+                       .response_pl_sz = sizeof(cdat_response_pl),
+               };
+               size_t entry_dw;
+               u32 *entry;
+
+               rc = pci_doe_exchange_sync(doe, &ex);
+               if (rc < 0)
+                       return rc;
+
+               entry = cdat_response_pl + 1;

I think:

entry = &cdat_response_pl[1];

...is less ambiguous, otherwise you need to backtrack and remember that cdat_response_pl is an array not a struct. Perhaps a comment or a symbol name for "1" would help here too?

drivers/cxl/mem.c static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm) · line 988

Quoted 4 lines
drivers/cxl/mem.c +4 -0
+               entry_dw = rc / sizeof(u32);
+               /* Skip Header */
+               entry_dw -= 1;
+               entry_dw = min(length / 4, entry_dw);

sometimes sizeof(u32) sometimes "4"?

drivers/cxl/mem.c static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm) · line 990

Quoted 2 lines
drivers/cxl/mem.c +2 -0
+               memcpy(buffer, entry, entry_dw * sizeof(u32));
+               length -= entry_dw * sizeof(u32);

Why not keep entry_dw in bytes, it seems this conversion to a word count is causing more trouble than it is worth above.

drivers/cxl/mem.c static int cxl_mem_setup_mailbox(struct cxl_mem *cxlm) · line 994

Quoted 4 lines
drivers/cxl/mem.c +4 -0
+               buffer += entry_dw;
+               entry_handle = FIELD_GET(CXL_DOE_TABLE_ACCESS_ENTRY_HANDLE, cdat_response_pl[0]);
+
+       } while (entry_handle != 0xFFFF);

Shouldn't this also break out if length goes to zero?

Quoted 57 lines
drivers/cxl/mem.c +43 -0
+
+       return 0;
+}
+
+static void cxl_mem_free_irq_vectors(void *data)
+{
+       pci_free_irq_vectors(data);
+}
+
+static void cxl_mem_doe_unregister_all(void *data)
+{
+       pci_doe_unregister_all(data);
+}
+
 static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo,
                                      u32 reg_hi)
 {
@@ -933,6 +1013,7 @@ static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo,
        struct cxl_mem *cxlm;
        void __iomem *regs;
        u64 offset;
+       int irqs;
        u8 bar;
        int rc;
 
@@ -971,6 +1052,44 @@ static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo,
                return NULL;
        }
 
+       /*
+        * An implementation of a cxl type3 device may support an unknown
+        * number of interrupts. Assume that number is not that large and
+        * request them all.
+        */
+       irqs = pci_msix_vec_count(pdev);
+       rc = pci_alloc_irq_vectors(pdev, irqs, irqs, PCI_IRQ_MSIX);
+       if (rc != irqs) {
+               /* No interrupt available - carry on */
+               dev_dbg(dev, "No interrupts available for DOE\n");
+       } else {
+               /*
+                * Enabling bus mastering could be done within the DOE
+                * initialization, but as it potentially has other impacts
+                * keep it within the driver.
+                */
+               pci_set_master(pdev);
+               rc = devm_add_action_or_reset(dev, cxl_mem_free_irq_vectors,
+                                              pdev);
+               if (rc)
+                       return NULL;
+       }
+
+       /*
+        * Find a DOE mailbox that supports CDAT.
+        * Supporting other DOE protocols will require more complexity.
+        */
+       rc = pci_doe_register_all(pdev);

drivers/cxl/mem.c static struct cxl_mem *cxl_mem_create(struct pci_dev *pdev, u32 reg_lo, · line 1092

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+       if (rc < 0)
+               return NULL;
+
+       rc = devm_add_action_or_reset(dev, cxl_mem_doe_unregister_all, pdev);
+       if (rc)
+               return NULL;
+
+       cxlm->table_doe = pci_doe_find(pdev, PCI_DVSEC_VENDOR_ID_CXL,
+                                      CXL_DOE_PROTOCOL_TABLE_ACCESS);
+

cxl_mem_create() is about allocating / initializing the @cxlm object. I think the above belongs in its own hardware init function. Well the interrupt init belongs in its own init function before cxl_mem_create() and the DOE / CDAT registration can come sometime later after cxl_mem_identify() succeeds. I.e. it's not until the driver has successfully setup mailbox communications should it worry about talking to the DOE.

Quoted 65 lines
drivers/cxl/mem.c +37 -0
        dev_dbg(dev, "Mapped CXL Memory Device resource\n");
        return cxlm;
 }
@@ -1060,6 +1179,31 @@ static ssize_t pmem_size_show(struct device *dev, struct device_attribute *attr,
        return sysfs_emit(buf, "%#llx\n", len);
 }
 
+static ssize_t CDAT_read(struct file *filp, struct kobject *kobj,
+                        struct bin_attribute *bin_attr, char *buf,
+                        loff_t offset, size_t count)
+{
+       struct device *dev = kobj_to_dev(kobj);
+       struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+
+       return memory_read_from_buffer(buf, count, &offset, cxlmd->cdat_table,
+                                      cxlmd->cdat_length);
+}
+
+static BIN_ATTR_RO(CDAT, 0);
+
+static umode_t cxl_memdev_bin_attr_is_visible(struct kobject *kobj,
+                                             struct bin_attribute *attr, int i)
+{
+       struct device *dev = kobj_to_dev(kobj);
+       struct cxl_memdev *cxlmd = to_cxl_memdev(dev);
+
+       if ((attr == &bin_attr_CDAT) && cxlmd->cdat_table)
+               return 0400;
+
+       return 0;
+}
+
 static struct device_attribute dev_attr_pmem_size =
        __ATTR(size, 0444, pmem_size_show, NULL);
 
@@ -1069,6 +1213,11 @@ static struct attribute *cxl_memdev_attributes[] = {
        NULL,
 };
 
+static struct bin_attribute *cxl_memdev_bin_attributes[] = {
+       &bin_attr_CDAT,
+       NULL,
+};
+
 static struct attribute *cxl_memdev_pmem_attributes[] = {
        &dev_attr_pmem_size.attr,
        NULL,
@@ -1081,6 +1230,8 @@ static struct attribute *cxl_memdev_ram_attributes[] = {
 
 static struct attribute_group cxl_memdev_attribute_group = {
        .attrs = cxl_memdev_attributes,
+       .bin_attrs = cxl_memdev_bin_attributes,
+       .is_bin_visible = cxl_memdev_bin_attr_is_visible,
 };
 
 static struct attribute_group cxl_memdev_ram_attribute_group = {
@@ -1158,6 +1309,25 @@ static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm)
        return ERR_PTR(rc);
 }
 
+static int cxl_cache_cdat_table(struct cxl_memdev *cxlmd)
+{
+       struct cxl_mem *cxlm = cxlmd->cxlm;
+       struct device *dev = &cxlmd->dev;
+       ssize_t cdat_length;

drivers/cxl/mem.c static struct cxl_memdev *cxl_memdev_alloc(struct cxl_mem *cxlm) · line 1326

Quoted 10 lines
drivers/cxl/mem.c +10 -0
+
+       if (cxlm->table_doe == NULL)
+               return 0;
+
+       cdat_length = cdat_get_length(cxlm->table_doe);
+       if (cdat_length < 0)
+               return cdat_length;
+
+       cxlmd->cdat_length = cdat_length;
+       cxlmd->cdat_table = devm_kzalloc(dev->parent, cdat_length, GFP_KERNEL);

I'm not sure how big these CDATs can get, but I don't think there is any requirement for the memory to be contiguous, so perhaps this should be kvzalloc() to be kind to the page allocator.

Quoted 55 lines
drivers/cxl/mem.c +8 -0
+
+       return cdat_to_buffer(cxlm->table_doe, cxlmd->cdat_table, cxlmd->cdat_length);
+}
+
 static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
 {
        struct cxl_memdev *cxlmd;
@@ -1180,6 +1350,10 @@ static int cxl_mem_add_memdev(struct cxl_mem *cxlm)
         */
        cxlmd->cxlm = cxlm;
 
+       rc = cxl_cache_cdat_table(cxlmd);
+       if (rc)
+               goto err;
+
        cdev = &cxlmd->cdev;
        rc = cdev_device_add(cdev, dev);
        if (rc)
drivers/cxl/mem.h +6 -0
index 0a3f70316872..fb26155a8fb3 100644
@@ -38,12 +38,16 @@
  * @cdev: char dev core object for ioctl operations
  * @cxlm: pointer to the parent device driver data
  * @id: id number of this memdev instance.
+ * @cdat_table: cache of CDAT table
+ * @cdat_length: length of cached CDAT table
  */
 struct cxl_memdev {
        struct device dev;
        struct cdev cdev;
        struct cxl_mem *cxlm;
        int id;
+       void *cdat_table;
+       size_t cdat_length;
 };
 
 /**
@@ -51,6 +55,7 @@ struct cxl_memdev {
  * @pdev: The PCI device associated with this CXL device.
  * @base: IO mappings to the device's MMIO
  * @cxlmd: Logical memory device chardev / interface
+ * @table_doe: Data exchange object mailbox used to read tables
  * @regs: Parsed register blocks
  * @payload_size: Size of space for payload
  *                (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
@@ -65,6 +70,7 @@ struct cxl_mem {
        void __iomem *base;
        struct cxl_memdev *cxlmd;
 
+       struct pci_doe *table_doe;
        struct cxl_regs regs;
 
        size_t payload_size;
#12 lore raw

PATCH v4 4/5 DONOTMERGE: PCI/DOE: Add per DOE chrdev for ioctl based access

In reply to Jonathan Cameron

It is not safe to access DOE mailboxes directly from userspace at the same time as the kernel may be accessing them. An implementation note in the ECN suggest use of a lock for this purpose, but in general, mediation is needed. Here we provide that mediation by providing a simple IOCTL interface allowing userspace to issue requests to the DOE and in a synchronous fashion receive the response.

There is no sanity checking of the messages sent so this is not an appropriate interface to expose to userspace, but may be of use to others.

Current discussions suggstion that we will instead implement protocol specific interfaces where needed. The first of these is the CDAT interface.

---
 drivers/pci/doe.c            | 188 +++++++++++++++++++++++++++++++++--
 drivers/pci/pci-driver.c     |   3 +-
 include/linux/pci-doe.h      |  13 +++
 include/uapi/linux/pci_doe.h |  32 ++++++
 4 files changed, 225 insertions(+), 11 deletions(-)
drivers/pci/doe.c +178 -10
index 27514313ed6a..2d20f59e42c6 100644
@@ -15,6 +15,10 @@
 #include <linux/pci.h>
 #include <linux/pci-doe.h>
 #include <linux/workqueue.h>
+#include <uapi/linux/pci_doe.h>
+
+/* Maximum number of DOE instances in the system */
+#define PCI_DOE_MAX_CNT 65536
 
 #define PCI_DOE_PROTOCOL_DISCOVERY 0
 
@@ -24,6 +28,10 @@
 /* Timeout of 1 second from 6.xx.1 (Operation), ECN - Data Object Exchange */
 #define PCI_DOE_TIMEOUT HZ
 
+static int pci_doe_major;
+static DEFINE_IDA(pci_doe_ida);
+static DECLARE_RWSEM(pci_doe_rwsem);
+
 static irqreturn_t pci_doe_irq(int irq, void *data)
 {
 	struct pci_doe *doe = data;
@@ -479,6 +487,126 @@ static int pci_doe_abort(struct pci_doe *doe)
 	return 0;
 }
 
+static void pci_doe_release(struct device *dev)
+{
+	struct pci_doe *doe = container_of(dev, struct pci_doe, dev);
+
+	ida_free(&pci_doe_ida, MINOR(doe->dev.devt));
+	kfree(doe);
+}
+
+static char *pci_doe_devnode(struct device *dev, umode_t *mode, kuid_t *uid,
+			     kgid_t *gid)
+{
+	return kasprintf(GFP_KERNEL, "pcidoe/%s", dev_name(dev));
+}
+
+static const struct device_type pci_doe_type = {
+	.name = "pci_doe",
+	.release = pci_doe_release,
+	.devnode = pci_doe_devnode,
+};
+
+static long __pci_doe_ioctl(struct pci_doe *doe, unsigned int cmd,
+			    unsigned long arg)
+{
+	struct pci_doe_uexchange __user *uex;
+	struct pci_doe_uexchange ex;
+	struct pci_doe_exchange exchange;
+	u32 *request_pl;
+	u32 *response_pl;
+	int ret;
+
+	if (cmd != PCI_DOE_EXCHANGE)
+		return -ENOTTY;
+
+	uex = (void __user *)arg;
+	if (copy_from_user(&ex, uex, sizeof(ex)))
+		return -EFAULT;
+
+	/* Cap size at something sensible */
+	request_pl = vmemdup_user(u64_to_user_ptr(ex.in.payload), ex.in.size);
+	if (!request_pl)
+		return -ENOMEM;
+
+	response_pl = kvzalloc(ex.out.size, GFP_KERNEL);
+	if (!response_pl) {
+		ret = -ENOMEM;
+		goto free_request;
+	}
+
+	exchange.vid = ex.vid;
+	exchange.protocol = ex.protocol;
+	exchange.request_pl = request_pl;
+	exchange.request_pl_sz = ex.in.size;
+	exchange.response_pl = response_pl;
+	exchange.response_pl_sz = ex.out.size;
+	ret = pci_doe_exchange_sync(doe, &exchange);
+	if (ret < 0)
+		goto free_response;
+	ret = 0;
+
+	if (copy_to_user(u64_to_user_ptr(ex.out.payload), response_pl, ex.out.size)) {
+		ret = -EFAULT;
+		goto free_response;
+	}
+
+	/* No useful value to return currently */
+	ex.retval = 0;
+	if (copy_to_user(uex, &ex, sizeof(ex))) {
+		ret = -EFAULT;
+		goto free_response;
+	}
+
+free_response:
+	kvfree(response_pl);
+free_request:
+	kvfree(request_pl);
+
+	return ret;
+}
+
+static long pci_doe_ioctl(struct file *file, unsigned int cmd,
+			    unsigned long arg)
+{
+	struct pci_doe *doe = file->private_data;
+	int rc = -ENXIO;
+
+	down_read(&pci_doe_rwsem);
+	if (!doe->going_down)
+		rc = __pci_doe_ioctl(doe, cmd, arg);
+	up_read(&pci_doe_rwsem);
+
+	return rc;
+}
+
+static int pci_doe_open(struct inode *inode, struct file *file)
+{
+	struct pci_doe *doe = container_of(inode->i_cdev, typeof(*doe), cdev);
+
+	get_device(&doe->dev);
+	file->private_data = doe;
+
+	return 0;
+}
+
+static int pci_doe_file_release(struct inode *inode, struct file *file)
+{
+	struct pci_doe *doe = container_of(inode->i_cdev, typeof(*doe), cdev);
+
+	put_device(&doe->dev);
+
+	return 0;
+}
+static const struct file_operations pci_doe_fops = {
+	.owner = THIS_MODULE,
+	.unlocked_ioctl = pci_doe_ioctl,
+	.open = pci_doe_open,
+	.release = pci_doe_file_release,
+	.compat_ioctl = compat_ptr_ioctl,
+	.llseek = noop_llseek,
+};
+
 static int pci_doe_register(struct pci_doe *doe)
 {
 	struct pci_dev *pdev = doe->pdev;
@@ -486,17 +614,35 @@ static int pci_doe_register(struct pci_doe *doe)
 	int rc, irq;
 	u32 val;
 
+	rc = ida_alloc_range(&pci_doe_ida, 0, PCI_DOE_MAX_CNT - 1, GFP_KERNEL);
+	if (rc < 0)
+		return rc;
+
+	device_initialize(&doe->dev);
+	doe->dev.parent = &pdev->dev;
+	doe->dev.devt = MKDEV(pci_doe_major, rc);
+	doe->dev.type = &pci_doe_type;
+	device_set_pm_not_required(&doe->dev);
+	rc = dev_set_name(&doe->dev, "doe[%s]_%x", dev_name(&pdev->dev), doe->cap);
+	if (rc)
+		goto err_put_device;
+
+	cdev_init(&doe->cdev, &pci_doe_fops);
+
 	pci_read_config_dword(pdev, doe->cap + PCI_DOE_CAP, &val);
 
 	if (!poll && FIELD_GET(PCI_DOE_CAP_INT, val)) {
-		irq = pci_irq_vector(pdev, FIELD_GET(PCI_DOE_CAP_IRQ, val));
-		if (irq < 0)
-			return irq;
+		rc = pci_irq_vector(pdev, FIELD_GET(PCI_DOE_CAP_IRQ, val));
+		if (rc < 0)
+			goto err_put_device;
+		irq = rc;
 
 		doe->irq_name = kasprintf(GFP_KERNEL, "DOE[%s]_%x",
 					  dev_name(&pdev->dev), doe->cap);
-		if (!doe->irq_name)
-			return -ENOMEM;
+		if (!doe->irq_name) {
+			rc = -ENOMEM;
+			goto err_put_device;
+		}
 
 		rc = request_irq(irq, pci_doe_irq, 0, doe->irq_name, doe);
 		if (rc)
@@ -512,6 +658,10 @@ static int pci_doe_register(struct pci_doe *doe)
 	if (rc)
 		goto err_free_irqs;
 
+	rc = cdev_device_add(&doe->cdev, &doe->dev);
+	if (rc)
+		goto err_free_irqs;
+
 	return 0;
 
 err_free_irqs:
@@ -519,15 +669,22 @@ static int pci_doe_register(struct pci_doe *doe)
 		free_irq(doe->irq, doe);
 err_free_name:
 	kfree(doe->irq_name);
+err_put_device:
+	put_device(&doe->dev);
 
 	return rc;
 }
 
 static void pci_doe_unregister(struct pci_doe *doe)
 {
+	cdev_device_del(&doe->cdev, &doe->dev);
+	down_write(&pci_doe_rwsem);
+	doe->going_down = 1;
+	up_write(&pci_doe_rwsem);
 	if (doe->irq > 0)
 		free_irq(doe->irq, doe);
 	kfree(doe->irq_name);
+	put_device(&doe->dev);
 }
 
 void pci_doe_unregister_all(struct pci_dev *pdev)
@@ -539,7 +696,6 @@ void pci_doe_unregister_all(struct pci_dev *pdev)
 		cancel_delayed_work_sync(&doe->statemachine);
 		kfree(doe->prots);
 		pci_doe_unregister(doe);
-		kfree(doe);
 	}
 }
 EXPORT_SYMBOL_GPL(pci_doe_unregister_all);
@@ -559,6 +715,7 @@ int pci_doe_register_all(struct pci_dev *pdev)
 {
 	struct pci_doe *doe;
 	int pos = 0;
+
 	int rc;
 
 	INIT_LIST_HEAD(&pdev->doe_list);
@@ -578,15 +735,12 @@ int pci_doe_register_all(struct pci_dev *pdev)
 
 		pci_doe_init(doe, pdev, pos);
 		rc = pci_doe_register(doe);
-		if (rc) {
-			kfree(doe);
+		if (rc)
 			goto err_free_does;
-		}
 
 		rc = pci_doe_cache_protocols(doe);
 		if (rc) {
 			pci_doe_unregister(doe);
-			kfree(doe);
 			goto err_free_does;
 		}
 
@@ -624,3 +778,17 @@ struct pci_doe *pci_doe_find(struct pci_dev *pdev, u16 vid, u8 type)
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(pci_doe_find);
+
+int pci_doe_sys_init(void)
+{
+	dev_t devt;
+	int rc;
+
+	rc = alloc_chrdev_region(&devt, 0, PCI_DOE_MAX_CNT, "pcidoe");
+	if (rc)
+		return rc;
+	pci_doe_major = MAJOR(devt);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(pci_doe_sys_init);
drivers/pci/pci-driver.c +2 -1
index ec44a79e951a..e2077a2b866f 100644
@@ -20,6 +20,7 @@
 #include <linux/of_device.h>
 #include <linux/acpi.h>
 #include <linux/dma-map-ops.h>
+#include <linux/pci-doe.h>
 #include "pci.h"
 #include "pcie/portdrv.h"
 
@@ -1655,6 +1656,6 @@ static int __init pci_driver_init(void)
 		return ret;
 #endif
 	dma_debug_add_bus(&pci_bus_type);
-	return 0;
+	return pci_doe_sys_init();
 }
 postcore_initcall(pci_driver_init);
include/linux/pci-doe.h +13 -0
index b2624e505458..bdc5f15f14ab 100644
@@ -6,6 +6,7 @@
  *     Jonathan Cameron <[email protected]>
  */
 
+#include <linux/cdev.h>
 #include <linux/completion.h>
 #include <linux/list.h>
 #include <linux/mutex.h>
@@ -56,8 +57,11 @@ struct pci_doe_exchange {
  * @abort: Request a manual abort (e.g. on init)
  * @dead: Used to mark a DOE for which an ABORT has timed out. Further messages
  *        will immediately be aborted with error
+ * @going_down: Mark DOE as removed
  */
 struct pci_doe {
+	struct device dev;
+	struct cdev cdev;
 	int cap;
 	struct pci_dev *pdev;
 	struct completion abort_c;
@@ -76,6 +80,7 @@ struct pci_doe {
 	unsigned int busy_retries;
 	unsigned int abort:1;
 	unsigned int dead:1;
+	unsigned int going_down:1;
 };
 
 int pci_doe_register_all(struct pci_dev *pdev);
@@ -84,4 +89,12 @@ struct pci_doe *pci_doe_find(struct pci_dev *pdev, u16 vid, u8 type);
 
 int pci_doe_exchange_sync(struct pci_doe *doe, struct pci_doe_exchange *ex);
 
+#ifdef CONFIG_PCI_DOE
+int pci_doe_sys_init(void);
+#else
+static inline int pci_doe_sys_init(void)
+{
+	return 0;
+}
+#endif /* CONFIG_PCI_DOE */
 #endif
include/uapi/linux/pci_doe.h new file +32 -0
new file mode 100644
index 000000000000..d01a27561df7
@@ -0,0 +1,32 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Userspace interface for safely accessing a PCI Data Exchange Object
+ * mailbox that has been registered by a driver.
+ */
+
+#ifndef LINUX_PCI_DOE_UAPI_H
+#define LINUX_PCI_DOE_UAPI_H
+
+#include <linux/ioctl.h>
+#include <linux/types.h>
+
+struct pci_doe_uexchange {
+	__u16 vid;
+	__u8 protocol;
+	__u8 rsvd;
+	__u32 retval;
+	struct {
+		__s32 size;
+		__u32 rsvd;
+		__u64 payload;
+	} in;
+	struct {
+		__s32 size;
+		__u32 rsvd;
+		__u64 payload;
+	} out;
+};
+
+#define PCI_DOE_EXCHANGE _IOWR(0xDA, 1, struct pci_doe_uexchange)
+
+#endif /* LINUX_PCI_DOE_UAPI_H */
Signature · 1 line
2.19.1
#13 lore raw

Re: [PATCH v4 4/5] DONOTMERGE: PCI/DOE: Add per DOE chrdev for ioctl based access

I love your patch! Yet something to improve:

Hi Jonathan,

I love your patch! Yet something to improve:

[auto build test ERROR on 35c32e3095d396c750f5cdfdaa94cba83d9b23c6]

url:    https://github.com/0day-ci/linux/commits/Jonathan-Cameron/PCI-Data-Object-Exchange-support-CXL-CDAT/20210524-214403
base:   35c32e3095d396c750f5cdfdaa94cba83d9b23c6
config: x86_64-rhel-8.3-kselftests (attached as .config)
compiler: gcc-9 (Debian 9.3.0-22) 9.3.0
reproduce (this is a W=1 build):
        # https://github.com/0day-ci/linux/commit/22a6e8bee6206990d309c9c61941f67868c3a12c
        git remote add linux-review https://github.com/0day-ci/linux
        git fetch --no-tags linux-review Jonathan-Cameron/PCI-Data-Object-Exchange-support-CXL-CDAT/20210524-214403
        git checkout 22a6e8bee6206990d309c9c61941f67868c3a12c
        # save the attached .config to linux build tree
        make W=1 ARCH=x86_64 

If you fix the issue, kindly add following tag as appropriate

All errors (new ones prefixed by >>):

>> error: include/uapi/linux/pci_doe.h: missing "WITH Linux-syscall-note" for SPDX-License-Identifier

make[2]: *** [scripts/Makefile.headersinst:63: usr/include/linux/pci_doe.h] Error 1 make[2]: Target '__headers' not remade because of errors.
make[1]: *** [Makefile:1333: headers] Error 2
make[1]: Target 'headers_install' not remade because of errors.
make: *** [Makefile:215: __sub-make] Error 2
make: Target 'headers_install' not remade because of errors.

---
0-DAY CI Kernel Test Service, Intel Corporation
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org

#14 lore raw

PATCH v4 5/5 DONOTMERGE: PCI/DOE: Add userspace example program to tools/pci

In reply to Jonathan Cameron

The example uses the Discovery Protocol to illustrate the use of the IOCTL interface to access the DOE mailboxes form userspace.

---
 tools/pci/Build     |   1 +
 tools/pci/Makefile  |   9 ++-
 tools/pci/doetest.c | 131 ++++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 140 insertions(+), 1 deletion(-)
tools/pci/Build +1 -0
index c375aea21790..af4521bebf93 100644
@@ -1 +1,2 @@
 pcitest-y += pcitest.o
+doetest-y += doetest.o
tools/pci/Makefile +8 -1
index 4b95a5176355..b2e54afe583c 100644
@@ -14,7 +14,7 @@ MAKEFLAGS += -r
 
 CFLAGS += -O2 -Wall -g -D_GNU_SOURCE -I$(OUTPUT)include
 
-ALL_TARGETS := pcitest
+ALL_TARGETS := pcitest doetest
 ALL_PROGRAMS := $(patsubst %,$(OUTPUT)%,$(ALL_TARGETS))
 
 SCRIPTS := pcitest.sh
@@ -30,6 +30,7 @@ include $(srctree)/tools/build/Makefile.include
 $(OUTPUT)include/linux/: ../../include/uapi/linux/
 	mkdir -p $(OUTPUT)include/linux/ 2>&1 || true
 	ln -sf $(CURDIR)/../../include/uapi/linux/pcitest.h $@
+	ln -sf $(CURDIR)/../../include/uapi/linux/pci_doe.h $@
 
 prepare: $(OUTPUT)include/linux/
 
@@ -39,6 +40,12 @@ $(PCITEST_IN): prepare FORCE
 $(OUTPUT)pcitest: $(PCITEST_IN)
 	$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) $< -o $@
 
+DOETEST_IN := $(OUTPUT)doetest-in.o
+$(DOETEST_IN): prepare FORCE
+	$(Q)$(MAKE) $(build)=doetest
+$(OUTPUT)doetest: $(DOETEST_IN)
+	$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) $< -o $@
+
 clean:
 	rm -f $(ALL_PROGRAMS)
 	rm -rf $(OUTPUT)include/
tools/pci/doetest.c new file +131 -0

131 lines not on this page · raw message

Signature · 1 line
2.19.1
#15 lore raw
In reply to Jonathan Cameron
Quoted 7 lines · On Mon, 24 May 2021 21:39:33 +0800 Jonathan Cameron…

Series first introduces generic support for DOE mailboxes as defined in the ECN to the PCIe r5.0 specification available from the PCI SIG [0]

A user is then introduced in the form of the table access protocol defined in the CXL 2.0 specification [1] used to access the
Coherent Device Attribute Table (CDAT) defined in [2]

Quoted 8 lines

Finally, in two patches that are not intended for merging, an example of a generic IOCTL interface to perform synchronous exchanges from user space in a kernel mediated fashion is introduced. The current consensus seems in favour of not introducing such an interface, but instead providing per protocol interfaces where appropriate. As this code was developed in parallel with that discussion and may be of use to someone it is included here.

Gentle, not particularly urgent, request for review on this set.

I am currently missing ABI docs for the CDAT binary attribute, but that could be handled in a follow up patch.

Thanks,

Jonathan

Quoted 65 lines

Open questions.
* Do we need to prevent userspace access to the DOE when the kernel is managing it (registered by driver)? Discussion in thread:
https://lore.kernel.org/linux-pci/[email protected]/
* Does a generic userspace mediation interface make sense? Currently it seems not, but as it was nearly ready, I've included it in this patch set anyway.
* Move CXL cdat handling to a separate file, or wait until we can see how it is going to be used?

Changes since v3
Thanks to Ira, Bjorn and Dan for feedback.
* Addition of a generic IOCTL interface and demo program for discussion.
* Fixed an issue with accidentally disabling interrupts.
* Ensure IRQ_HANDLED always returned as clearing of BUSY can result in an interrupt, but there is no means of identifying this. The best that can be done is to eat the interrupt.
* Edited comments to more directly tie them to the code the referred to.
* Lock scope documentation update.
* Dropped the CDAT dump to log patch as we are hopefully getting closer to this being applied.
* Fixed the references to pcie_ missed when renaming to pci_ in v3.

Set based on cxl/next

All testing conducted against QEMU emulation of a CXL type 3 device in conjunction with DOE mailbox patches v5 [3, 4]

[0] https://pcisig.com/specifications
[1] https://www.computeexpresslink.org/download-the-specification
[2] https://uefi.org/node/4093
[3] https://lore.kernel.org/qemu-devel/[email protected]/
[4] https://lore.kernel.org/qemu-devel/[email protected]/

Jonathan Cameron (5):
PCI: Add vendor ID for the PCI SIG
PCI/DOE: Add Data Object Exchange support
cxl/mem: Add CDAT table reading from DOE
DONOTMERGE PCI/DOE Add per DOE chrdev for ioctl based access
DONOTMERGE: PCI/DOE: Add userspace example program to tools/pci

 drivers/cxl/Kconfig           |   1 +
 drivers/cxl/cxl.h             |  21 +
 drivers/cxl/mem.c             | 174 ++++++++
 drivers/cxl/mem.h             |   6 +
 drivers/pci/Kconfig           |   8 +
 drivers/pci/Makefile          |   1 +
 drivers/pci/doe.c             | 794 ++++++++++++++++++++++++++++++++++
 drivers/pci/pci-driver.c      |   3 +-
 include/linux/pci-doe.h       | 100 +++++
 include/linux/pci.h           |   3 +
 include/linux/pci_ids.h       |   1 +
 include/uapi/linux/pci_doe.h  |  32 ++
 include/uapi/linux/pci_regs.h |  29 +-
 tools/pci/Build               |   1 +
 tools/pci/Makefile            |   9 +-
 tools/pci/doetest.c           | 131 ++++++
 16 files changed, 1311 insertions(+), 3 deletions(-)
 create mode 100644 drivers/pci/doe.c
 create mode 100644 include/linux/pci-doe.h
 create mode 100644 include/uapi/linux/pci_doe.h
 create mode 100644 tools/pci/doetest.c

base-commit: 35c32e3095d396c750f5cdfdaa94cba83d9b23c6