Copy Offload in NVMe Fabrics with P2P PCI Memory
Subsystems (10)
Logan Gunthorpe
maintainerv6 cover Copy Offload in NVMe Fabrics with P2P PCI Memory
Hi Everyone,
Here is version 6 of the PCI P2PDMA patch set. This version makes a few minor changes from v5 and is based on v4.19-rc3. A git repo is here:
https://github.com/sbates130272/linux-p2pmem pci-p2p-v6
Thanks,
Logan
--
Changes in v6:
* Rebased on v4.19-rc3
* Remove the changes to the common submit_bio() path and instead set REQ_NOMERGE in the NVME target driver, when appropriate. Per discussions with Jens and Christoph.
* Some minor grammar changes in the documentation as spotted by Randy.
Changes in v5:
* Rebased on v4.19-rc1
* Drop changing ACS settings in this patchset. Now, the code will only allow P2P transactions between devices whos downstream ports do not restrict P2P TLPs.
* Drop the REQ_PCI_P2PDMA block flag and instead use is_pci_p2pdma_page() to tell if a request is P2P or not. In that case we check for queue support and enforce using REQ_NOMERGE. Per feedback from Christoph.
* Drop the pci_p2pdma_unmap_sg() function as it was empty and only there for symmetry and compatibility with dma_unmap_sg. Per feedback from Christoph.
* Split off the logic to handle enabling P2P in NVMe fabrics' configfs into specific helpers in the p2pdma code. Per feedback from Christoph.
* A number of other minor cleanups and fixes as pointed out by Christoph and others.
Changes in v4:
* Change the original upstream_bridges_match() function to upstream_bridge_distance() which calculates the distance between two devices as long as they are behind the same root port. This should address Bjorn's concerns that the code was to focused on being behind a single switch.
* The disable ACS function now disables ACS for all bridge ports instead of switch ports (ie. those that had two upstream_bridge ports).
* Change the pci_p2pmem_alloc_sgl() and pci_p2pmem_free_sgl() API to be more like sgl_alloc() in that the alloc function returns the allocated scatterlist and nents is not required bythe free function.
* Moved the new documentation into the driver-api tree as requested by Jonathan
* Add SGL alloc and free helpers in the nvmet code so that the individual drivers can share the code that allocates P2P memory. As requested by Christoph.
* Cleanup the nvmet_p2pmem_store() function as Christoph thought my first attempt was ugly.
* Numerous commit message and comment fix-ups
Changes in v3:
* Many more fixes and minor cleanups that were spotted by Bjorn
* Additional explanation of the ACS change in both the commit message and Kconfig doc. Also, the code that disables the ACS bits is surrounded explicitly by an #ifdef
* Removed the flag we added to rdma_rw_ctx() in favour of using is_pci_p2pdma_page(), as suggested by Sagi.
* Adjust pci_p2pmem_find() so that it prefers P2P providers that are closest to (or the same as) the clients using them. In cases of ties, the provider is randomly chosen.
* Modify the NVMe Target code so that the PCI device name of the provider may be explicitly specified, bypassing the logic in pci_p2pmem_find(). (Note: it's still enforced that the provider must be behind the same switch as the clients).
* As requested by Bjorn, added documentation for driver writers.
Changes in v2:
* Renamed everything to 'p2pdma' per the suggestion from Bjorn as well as a bunch of cleanup and spelling fixes he pointed out in the last series.
* To address Alex's ACS concerns, we change to a simpler method of just disabling ACS behind switches for any kernel that has CONFIG_PCI_P2PDMA.
* We also reject using devices that employ 'dma_virt_ops' which should fairly simply handle Jason's concerns that this work might break with the HFI, QIB and rxe drivers that use the virtual ops to implement their own special DMA operations.
--
This is a continuation of our work to enable using Peer-to-Peer PCI memory in the kernel with initial support for the NVMe fabrics target subsystem. Many thanks go to Christoph Hellwig who provided valuable feedback to get these patches to where they are today.
The concept here is to use memory that's exposed on a PCI BAR as data buffers in the NVMe target code such that data can be transferred from an RDMA NIC to the special memory and then directly to an NVMe device avoiding system memory entirely. The upside of this is better QoS for applications running on the CPU utilizing memory and lower PCI bandwidth required to the CPU (such that systems could be designed with fewer lanes connected to the CPU).
Due to these trade-offs we've designed the system to only enable using the PCI memory in cases where the NIC, NVMe devices and memory are all behind the same PCI switch hierarchy. This will mean many setups that could likely work well will not be supported so that we can be more confident it will work and not place any responsibility on the user to understand their topology. (We chose to go this route based on feedback we received at the last LSF). Future work may enable these transfers using a white list of known good root complexes. However, at this time, there is no reliable way to ensure that Peer-to-Peer transactions are permitted between PCI Root Ports.
For PCI P2P DMA transfers to work in this situation the ACS bits must be disabled on the downstream ports (DSPs) for all devices involved in the transfer. This can be done using the "disable_acs_redir" PCI kernel command line option which was introduced in v4.19.
In order to enable PCI P2P functionality, we introduce a few new PCI functions such that a driver can register P2P memory with the system. Struct pages are created for this memory using devm_memremap_pages() and the PCI bus offset is stored in the corresponding pagemap structure.
Another set of functions allow a client driver to create a list of client devices that will be used in a given P2P transactions and then use that list to find any P2P memory that is supported by all the client devices.
In the block layer, we also introduce a flag for a request queue to indicate a given queue supports targeting P2P memory. The driver submitting bios must ensure that the queue supports P2P before attempting to submit BIOs backed by P2P memory. Also, P2P requests are marked to not be merged seeing a non-homogenous request would complicate the DMA mapping requirements.
In the PCI NVMe driver, we modify the existing CMB support to utilize the new PCI P2P memory infrastructure and also add support for P2P memory in its request queue. When a P2P request is received it uses the pci_p2pmem_map_sg() function which applies the necessary transformation to get the corrent pci_bus_addr_t for the DMA transactions.
In the RDMA core, we also adjust rdma_rw_ctx_init() and rdma_rw_ctx_destroy() to take a flags argument which indicates whether to use the PCI P2P mapping functions or not. To avoid odd RDMA devices that don't use the proper DMA infrastructure this code rejects using any device that employs the virt_dma_ops implementation.
Finally, in the NVMe fabrics target port we introduce a new configuration attribute: 'p2pmem'. When set to a true boolean, the port will attempt to find P2P memory supported by the RDMA NIC and all namespaces. It may also be set to a PCI device name to select a specific P2P memory to use. If supported memory is found, it will be used in all IO transfers. And if a port is using P2P memory, adding new namespaces that are not supported by that memory will fail.
These patches have been tested on a number of Intel based systems and for a variety of RDMA NICs (Mellanox, Broadcomm, Chelsio) and NVMe SSDs (Intel, Seagate, Samsung) and p2pdma devices (Eideticom, Microsemi, Chelsio and Everspin) using switches from both Microsemi and Broadcomm.
--
Logan Gunthorpe (13):
PCI/P2PDMA: Support peer-to-peer memory
PCI/P2PDMA: Add sysfs group to display p2pmem stats PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers
docs-rst: Add a new directory for PCI documentation PCI/P2PDMA: Add P2P DMA driver writer's documentation
block: Add PCI P2P flag for request queue and check support for
requests
IB/core: Ensure we map P2P memory correctly in
rdma_rw_ctx_[init|destroy]()
nvme-pci: Use PCI p2pmem subsystem to manage the CMB
nvme-pci: Add support for P2P memory in requests
nvme-pci: Add a quirk for a pseudo CMB
nvmet: Introduce helper functions to allocate and free request SGLs
nvmet: Optionally use PCI P2P memory
Documentation/ABI/testing/sysfs-bus-pci | 25 +
Documentation/driver-api/index.rst | 2 +-
Documentation/driver-api/pci/index.rst | 21 +
Documentation/driver-api/pci/p2pdma.rst | 170 ++++
Documentation/driver-api/{ => pci}/pci.rst | 0
drivers/infiniband/core/rw.c | 11 +-
drivers/nvme/host/core.c | 4 +
drivers/nvme/host/nvme.h | 8 +
drivers/nvme/host/pci.c | 121 ++-
drivers/nvme/target/configfs.c | 36 +
drivers/nvme/target/core.c | 154 ++++
drivers/nvme/target/io-cmd-bdev.c | 3 +
drivers/nvme/target/nvmet.h | 15 +
drivers/nvme/target/rdma.c | 22 +-
drivers/pci/Kconfig | 17 +
drivers/pci/Makefile | 1 +
drivers/pci/p2pdma.c | 941 +++++++++++++++++++++
include/linux/blkdev.h | 3 +
include/linux/memremap.h | 6 +
include/linux/mm.h | 18 +
include/linux/pci-p2pdma.h | 124 +++
include/linux/pci.h | 4 +
22 files changed, 1652 insertions(+), 54 deletions(-)
create mode 100644 Documentation/driver-api/pci/index.rst
create mode 100644 Documentation/driver-api/pci/p2pdma.rst
rename Documentation/driver-api/{ => pci}/pci.rst (100%)
create mode 100644 drivers/pci/p2pdma.c
create mode 100644 include/linux/pci-p2pdma.h--
2.19.0
Logan Gunthorpe
maintainerPATCH v6 1/13 PCI/P2PDMA: Support peer-to-peer memory
Some PCI devices may have memory mapped in a BAR space that's intended for use in peer-to-peer transactions. In order to enable such transactions the memory must be registered with ZONE_DEVICE pages so it can be used by DMA interfaces in existing drivers.
Add an interface for other subsystems to find and allocate chunks of P2P memory as necessary to facilitate transfers between two PCI peers:
int pci_p2pdma_add_client();
struct pci_dev *pci_p2pmem_find();
void *pci_alloc_p2pmem();
The new interface requires a driver to collect a list of client devices involved in the transaction with the pci_p2pmem_add_client*() functions then call pci_p2pmem_find() to obtain any suitable P2P memory. Once this is done the list is bound to the memory and the calling driver is free to add and remove clients as necessary (adding incompatible clients will fail). With a suitable p2pmem device, memory can then be allocated with pci_alloc_p2pmem() for use in DMA transactions.
Depending on hardware, using peer-to-peer memory may reduce the bandwidth of the transfer but can significantly reduce pressure on system memory. This may be desirable in many cases: for example a system could be designed with a small CPU connected to a PCIe switch by a small number of lanes which would maximize the number of lanes available to connect to NVMe devices.
The code is designed to only utilize the p2pmem device if all the devices involved in a transfer are behind the same PCI bridge. This is because we have no way of knowing whether peer-to-peer routing between PCIe Root Ports is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P transfers that go through the RC is limited to only reducing DRAM usage and, in some cases, coding convenience. The PCI-SIG may be exploring adding a new capability bit to advertise whether this is possible for future hardware.
This commit includes significant rework and feedback from Christoph Hellwig.
--- drivers/pci/Kconfig | 17 + drivers/pci/Makefile | 1 + drivers/pci/p2pdma.c | 761 +++++++++++++++++++++++++++++++++++++ include/linux/memremap.h | 5 + include/linux/mm.h | 18 + include/linux/pci-p2pdma.h | 102 +++++ include/linux/pci.h | 4 + 7 files changed, 908 insertions(+) create mode 100644 drivers/pci/p2pdma.c create mode 100644 include/linux/pci-p2pdma.h
drivers/pci/Kconfig +17 -0
@@ -132,6 +132,23 @@ config PCI_PASID If unsure, say N. +config PCI_P2PDMA + bool "PCI peer-to-peer transfer support" + depends on PCI && ZONE_DEVICE + select GENERIC_ALLOCATOR + help + Enableѕ drivers to do PCI peer-to-peer transactions to and from + BARs that are exposed in other devices that are the part of + the hierarchy where peer-to-peer DMA is guaranteed by the PCI + specification to work (ie. anything below a single PCI bridge). + + Many PCIe root complexes do not support P2P transactions and + it's hard to tell which support it at all, so at this time, + P2P DMA transations must be between devices behind the same root + port. + + If unsure, say N. + config PCI_LABEL def_bool y if (DMI || ACPI) depends on PCI
drivers/pci/Makefile +1 -0
@@ -26,6 +26,7 @@ obj-$(CONFIG_PCI_SYSCALL) += syscall.o obj-$(CONFIG_PCI_STUB) += pci-stub.o obj-$(CONFIG_PCI_PF_STUB) += pci-pf-stub.o obj-$(CONFIG_PCI_ECAM) += ecam.o +obj-$(CONFIG_PCI_P2PDMA) += p2pdma.o obj-$(CONFIG_XEN_PCIDEV_FRONTEND) += xen-pcifront.o # Endpoint library must be initialized before its users
drivers/pci/p2pdma.c new file +761 -0
761 lines not on this page · raw message
include/linux/memremap.h +5 -0
@@ -53,11 +53,16 @@ struct vmem_altmap {* wakeup event whenever a page is unpinned and becomes idle. This * wakeup is used to coordinate physical address space management (ex: * fs truncate/hole punch) vs pinned pages (ex: device dma). +* +* MEMORY_DEVICE_PCI_P2PDMA: +* Device memory residing in a PCI BAR intended for use with Peer-to-Peer +* transactions. */ enum memory_type {MEMORY_DEVICE_PRIVATE =1 ,MEMORY_DEVICE_PUBLIC ,MEMORY_DEVICE_FS_DAX , +MEMORY_DEVICE_PCI_P2PDMA , };/*
include/linux/mm.h +18 -0
@@ -890,6 +890,19 @@ static inline bool is_device_public_page(const struct page *page) page->pgmap->type == MEMORY_DEVICE_PUBLIC; } +#ifdef CONFIG_PCI_P2PDMA +static inline bool is_pci_p2pdma_page (const struct page *page) +{ +return is_zone_device_page (page) && + page->pgmap->type ==MEMORY_DEVICE_PCI_P2PDMA ; +} +#else /* CONFIG_PCI_P2PDMA */ +static inline bool is_pci_p2pdma_page (const struct page *page) +{ +return false; +} +#endif /* CONFIG_PCI_P2PDMA */ +#else /* CONFIG_DEV_PAGEMAP_OPS */ static inline void dev_pagemap_get_ops (void ) {
@@ -913,6 +926,11 @@ static inline bool is_device_public_page(const struct page *page) {return false; } + +static inline bool is_pci_p2pdma_page (const struct page *page) +{ +return false; +}#endif /* CONFIG_DEV_PAGEMAP_OPS */ static inline void get_page (struct page *page)
include/linux/pci-p2pdma.h new file +102 -0
102 lines not on this page · raw message
include/linux/pci.h +4 -0
@@ -281,6 +281,7 @@ struct pcie_link_state;struct pci_vpd ;struct pci_sriov ;struct pci_ats ; +struct pci_p2pdma ;/* The pci_dev structure describes PCI devices */ struct pci_dev {
@@ -438,6 +439,9 @@ struct pci_dev {#endif #ifdef CONFIG_PCI_PASID u16 pasid_features; +#endif +#ifdef CONFIG_PCI_P2PDMA +struct pci_p2pdma *p2pdma;#endif phys_addr_t rom;/* Physical address if not from BAR */ size_t romlen;/* Length if not from BAR */
Signature · 1 line
2.19.0
Re: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory
Quoted 36 lines · Logan Gunthorpe
Some PCI devices may have memory mapped in a BAR space that's intended for use in peer-to-peer transactions. In order to enable such transactions the memory must be registered with
ZONE_DEVICEpages so it can be used by DMA interfaces in existing drivers.Add an interface for other subsystems to find and allocate chunks of P2P memory as necessary to facilitate transfers between two PCI peers:
int
pci_p2pdma_add_client();
structpci_dev*pci_p2pmem_find();
void*pci_alloc_p2pmem();The new interface requires a driver to collect a list of client devices involved in the transaction with the pci_p2pmem_add_client*() functions then call
pci_p2pmem_find()to obtain any suitable P2P memory. Once this is done the list is bound to the memory and the calling driver is free to add and remove clients as necessary (adding incompatible clients will fail). With a suitable p2pmem device, memory can then be allocated withpci_alloc_p2pmem()for use in DMA transactions.Depending on hardware, using peer-to-peer memory may reduce the bandwidth of the transfer but can significantly reduce pressure on system memory. This may be desirable in many cases: for example a system could be designed with a small CPU connected to a PCIe switch by a small number of lanes which would maximize the number of lanes available to connect to NVMe devices.
The code is designed to only utilize the p2pmem device if all the devices involved in a transfer are behind the same PCI bridge. This is because we have no way of knowing whether peer-to-peer routing between PCIe Root Ports is supported (PCIe r4.0, sec 1.3.1). Additionally, the benefits of P2P transfers that go through the RC is limited to only reducing DRAM usage and, in some cases, coding convenience. The PCI-SIG may be exploring adding a new capability bit to advertise whether this is possible for future hardware.
Quoted 5 lines
This commit includes significant rework and feedback from Christoph Hellwig.
Signed-off-by Christoph Hellwig <[email protected]>Signed-off-by Logan Gunthorpe <[email protected]>
Mostly trivial comments below. Thanks for persevering with this!
Quoted 12 lines
--- drivers/pci/Kconfig | 17 + drivers/pci/Makefile | 1 + drivers/pci/p2pdma.c | 761 +++++++++++++++++++++++++++++++++++++ include/linux/memremap.h | 5 + include/linux/mm.h | 18 + include/linux/pci-p2pdma.h | 102 +++++ include/linux/pci.h | 4 + 7 files changed, 908 insertions(+) create mode 100644 drivers/pci/p2pdma.c create mode 100644 include/linux/pci-p2pdma.h
Quoted 1 line
+#define
pr_fmt(fmt)"pci-p2pdma: " fmt
Is pr_fmt() actually used anywhere?
Quoted 1 line
+ * Check if a PCI bridge has it's ACS redirection bits set to redirect P2P
s/it's/its/
Quoted 4 lines
+ * TLPs upstream via ACS. Returns 1 if the packets will be redirected
+ * upstream, 0 otherwise.
+ */
+static intpci_bridge_has_acs_redir(struct pci_dev *dev)
Most of your code uses "pdev" for a struct pci_dev *.
drivers/pci/p2pdma.c line 216
Quoted 8 lines
drivers/pci/p2pdma.c new file +8
-0+static void seq_buf_print_bus_devfn (struct seq_buf *buf,struct pci_dev *dev) +{ +if (!buf) +return ; + +seq_buf_printf (buf,"%04x:%02x:%02x.%x;" ,pci_domain_nr (dev->bus), + dev->bus->number,PCI_SLOT (dev->devfn), +PCI_FUNC (dev->devfn));
dev vs pdev?
I think you could use pci_name() here (see pci_setup_device()).
Quoted 3 lines
+ * In the case where two devices are connected to different PCIe switches,
+ * this function will still return a positive distance as long as both
+ * switches evenutally have a common upstream bridge. Note this covers
s/evenutally/eventually/
drivers/pci/p2pdma.c line 328
Quoted 7 lines
drivers/pci/p2pdma.c new file +7
-0+static int upstream_bridge_distance_warn (struct pci_dev *provider, +struct pci_dev *client) +{ +struct seq_buf acs_list; +int ret; + +seq_buf_init (&acs_list,kmalloc (PAGE_SIZE ,GFP_KERNEL ),PAGE_SIZE );
Check for kmalloc() failure here? Failure would mean acs_list->buffer is NULL, but I gave up following the chain to see how that would be handled.
drivers/pci/p2pdma.c line 332
Quoted 4 lines
drivers/pci/p2pdma.c new file +4
-0+ + ret =upstream_bridge_distance (provider, client, &acs_list); +if (ret ==-2 ) { +pci_warn (client,"cannot be used for peer-to-peer DMA as ACS redirect is set between the client and provider\n" );
Maybe include pci_name(provider) in these messages?
drivers/pci/p2pdma.c
Quoted 9 lines
drivers/pci/p2pdma.c new file +8
-0+/* Drop final semicolon */ + acs_list.buffer[acs_list.len-1 ] =0 ; +pci_warn (client,"to disable ACS redirect for this path, add the kernel parameter: pci=disable_acs_redir=%s\n" , + acs_list.buffer); + + }else if (ret <0 ) { +pci_warn (client,"cannot be used for peer-to-peer DMA as the client and provider do not share an upstream bridge\n" ); + * pci_p2pdma_assign_provider - Check compatibily (as per pci_p2pdma_distance)
s/compatibily/compatibility/
drivers/pci/p2pdma.c line 587
Quoted 12 lines
drivers/pci/p2pdma.c new file +12
-0+struct pci_dev *pci_p2pmem_find (struct list_head *clients) +{ +struct pci_dev *pdev =NULL ; +struct pci_p2pdma_client *pos; +int distance; +int closest_distance =INT_MAX ; +struct pci_dev **closest_pdevs; +int dev_cnt =0 ; +const int max_devs =PAGE_SIZE /sizeof (*closest_pdevs); +int i; + + closest_pdevs =kmalloc (PAGE_SIZE ,GFP_KERNEL );
Check for kmalloc() failure?
include/linux/pci-p2pdma.h line 9
Quoted 11 lines
+++
b/include/linux/pci-p2pdma.hinclude/linux/pci-p2pdma.h new file +9
-0@@ -0,0 +1,102 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* +* PCI Peer 2 Peer DMA support. +* +* Copyright (c) 2016-2018, Logan Gunthorpe +* Copyright (c) 2016-2017, Microsemi Corporation +* Copyright (c) 2017, Christoph Hellwig +* Copyright (c) 2018, Eideticom Inc. +*
Spurious blank line.
Quoted 1 line
(quoted diff) +1
-0+ */
Logan Gunthorpe
maintainerRe: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory
Quoted 3 lines · Bjorn Helgaas
Mostly trivial comments below. Thanks for persevering with this!
Thanks Bjorn. I'll make these fixes for v7 next week.
Quoted 3 lines
Quoted 1 line
+#define
pr_fmt(fmt)"pci-p2pdma: " fmtIs
pr_fmt()actually used anywhere?
It's used in a Patch 4 to print errors in the configfs helpers.
Logan
Re: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory
Quoted 6 lines · Logan Gunthorpe
Quoted 3 lines · Bjorn Helgaas
Quoted 1 line
+#define
pr_fmt(fmt)"pci-p2pdma: " fmtIs
pr_fmt()actually used anywhere?It's used in a Patch 4 to print errors in the configfs helpers.
Ideally pr_fmt() would be added in the same patch that uses it, but obviously pretty trivial.
Logan Gunthorpe
maintainerRe: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory
Quoted 10 lines · Bjorn Helgaas
Quoted 6 lines · Logan Gunthorpe
Quoted 3 lines · Bjorn Helgaas
Quoted 1 line
+#define
pr_fmt(fmt)"pci-p2pdma: " fmtIs
pr_fmt()actually used anywhere?It's used in a Patch 4 to print errors in the configfs helpers.
Ideally
pr_fmt()would be added in the same patch that uses it, but obviously pretty trivial.
Hmm, well Christoph asked me to do it this way. ;)
Logan
Christoph Hellwig
maintainerRe: [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory
Quoted 5 lines · Logan Gunthorpe
Quoted 4 lines · Bjorn Helgaas
Quoted 3 lines · Logan Gunthorpe
Quoted 2 lines · Bjorn Helgaas
Quoted 1 line
+#define
pr_fmt(fmt)"pci-p2pdma: " fmt
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line
Is
pr_fmt()actually used anywhere?It's used in a Patch 4 to print errors in the configfs helpers.
Ideally
pr_fmt()would be added in the same patch that uses it, but obviously pretty trivial.Hmm, well Christoph asked me to do it this way. ;)
Sorry. I somehow was under the impression we already had a printk in the earlier patch, but it looks like I was confused.
_______________________________________________
Linux-nvdimm mailing list
[email protected]
https://lists.01.org/mailman/listinfo/linux-nvdimm
Logan Gunthorpe
maintainerPATCH v6 2/13 PCI/P2PDMA: Add sysfs group to display p2pmem stats
In reply to Logan GunthorpeAdd a sysfs group to display statistics about P2P memory that is registered in each PCI device.
Attributes in the group display the total amount of P2P memory, the amount available and whether it is published or not.
--- Documentation/ABI/testing/sysfs-bus-pci | 25 ++++++++++++ drivers/pci/p2pdma.c | 54 +++++++++++++++++++++++++ 2 files changed, 79 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +25 -0
@@ -323,3 +323,28 @@ Description: This is similar to /sys/bus/pci/drivers_autoprobe, but affects only the VFs associated with a specific PF. + +What: /sys/bus/pci/devices/.../p2pmem/available +Date: November 2017 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: + If the device has any Peer-to-Peer memory registered, this + file contains the amount of memory that has not been + allocated (in decimal). + +What: /sys/bus/pci/devices/.../p2pmem/size +Date: November 2017 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: + If the device has any Peer-to-Peer memory registered, this + file contains the total amount of memory that the device + provides (in decimal). + +What: /sys/bus/pci/devices/.../p2pmem/published +Date: November 2017 +Contact: Logan Gunthorpe <logang at deltatee.com> +Description: + If the device has any Peer-to-Peer memory registered, this + file contains a '1' if the memory has been published for + use inside the kernel or a '0' if it is only intended + for use within the driver that published it.
drivers/pci/p2pdma.c +54 -0
75 lines not on this page · raw message
Signature · 1 line
2.19.0
Re: [PATCH v6 02/13] PCI/P2PDMA: Add sysfs group to display p2pmem stats
Quoted 7 lines · Logan Gunthorpe
Add a sysfs group to display statistics about P2P memory that is registered in each PCI device.
Attributes in the group display the total amount of P2P memory, the amount available and whether it is published or not.
Signed-off-by Logan Gunthorpe <[email protected]>
Quoted 5 lines
Documentation/ABI/testing/sysfs-bus-pci +5
-0+What: /sys/bus/pci/devices/.../p2pmem/available +Date: November 2017 +Contact: Logan Gunthorpe <[email protected]> +Description: + If the device has any Peer-to-Peer memory registered, this
Documentation/ABI/testing/sysfs-bus-pci Description: · line 341
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +10
-0+ file contains the amount of memory that has not been + allocated (in decimal). + +What: /sys/bus/pci/devices/.../p2pmem/size +Date: November 2017 +Contact: Logan Gunthorpe <[email protected]> +Description: + If the device has any Peer-to-Peer memory registered, this + file contains the total amount of memory that the device + provides (in decimal).
Maybe reorder this so the "size" (total amount) is documented before "available" (some subset of "size")?
Documentation/ABI/testing/sysfs-bus-pci Description: · line 350
Quoted 9 lines
Documentation/ABI/testing/sysfs-bus-pci +9
-0+ +What: /sys/bus/pci/devices/.../p2pmem/published +Date: November 2017 +Contact: Logan Gunthorpe <[email protected]> +Description: + If the device has any Peer-to-Peer memory registered, this + file contains a '1' if the memory has been published for + use inside the kernel or a '0' if it is only intended + for use within the driver that published it.
It doesn't read quite right to talk about "use within the driver that *published* it". Is it really published in that case? That sounds more like "private". I expected something like the following (but I don't claim to understand the whole use model here):
... this file contains a '1' if the memory has been published for use outside the driver that owns the device.
Logan Gunthorpe
maintainerPATCH v6 3/13 PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
In reply to Logan GunthorpeThe DMA address used when mapping PCI P2P memory must be the PCI bus address. Thus, introduce pci_p2pmem_map_sg() to map the correct addresses when using P2P memory. Memory mapped in this way does not need to be unmapped.
For this, we assume that an SGL passed to these functions contain all P2P memory or no P2P memory.
--- drivers/pci/p2pdma.c | 43 ++++++++++++++++++++++++++++++++++++++ include/linux/memremap.h | 1 + include/linux/pci-p2pdma.h | 7 +++++++ 3 files changed, 51 insertions(+)
drivers/pci/p2pdma.c +43 -0
@@ -191,6 +191,8 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->res.flags =pci_resource_flags (pdev, bar); pgmap->ref = &pdev->p2pdma->devmap_ref; pgmap->type =MEMORY_DEVICE_PCI_P2PDMA ; + pgmap->pci_p2pdma_bus_offset =pci_bus_address (pdev, bar) - +pci_resource_start (pdev, bar); addr =devm_memremap_pages (&pdev->dev, pgmap);if (IS_ERR (addr)) {
@@ -813,3 +815,44 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) pdev->p2pdma->p2pmem_published = publish; }EXPORT_SYMBOL_GPL (pci_p2pmem_publish); + +/** +* pci_p2pdma_map_sg - map a PCI peer-to-peer scatterlist for DMA +* @dev: device doing the DMA request +* @sg: scatter list to map +* @nents: elements in the scatterlist +* @dir: DMA direction +* +* Scatterlists mapped with this function should not be unmapped in any way. +* +* Returns the number of SG entries mapped or 0 on error. +*/ +int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents, +enum dma_data_direction dir) +{ +struct dev_pagemap *pgmap; +struct scatterlist *s; +phys_addr_t paddr; +int i; + +/* +* p2pdma mappings are not compatible with devices that use +* dma_virt_ops. If the upper layers do the right thing +* this should never happen because it will be prevented +* by the check in pci_p2pdma_add_client() +*/ +if (WARN_ON_ONCE (IS_ENABLED (CONFIG_DMA_VIRT_OPS ) && + dev->dma_ops == &dma_virt_ops)) +return 0 ; + +for_each_sg (sg, s, nents, i) { + pgmap =sg_page (s)->pgmap; + paddr =sg_phys (s); + + s->dma_address = paddr - pgmap->pci_p2pdma_bus_offset; +sg_dma_len (s) = s->length; + } + +return nents; +} +EXPORT_SYMBOL_GPL (pci_p2pdma_map_sg);
include/linux/memremap.h +1 -0
@@ -125,6 +125,7 @@ struct dev_pagemap {struct device *dev;void *data;enum memory_type type; +u64 pci_p2pdma_bus_offset; };#ifdef CONFIG_ZONE_DEVICE
include/linux/pci-p2pdma.h +7 -0
@@ -36,6 +36,8 @@ struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, unsigned int *nents, u32 length);void pci_p2pmem_free_sgl (struct pci_dev *pdev,struct scatterlist *sgl);void pci_p2pmem_publish (struct pci_dev *pdev,bool publish); +int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents, +enum dma_data_direction dir);#else /* CONFIG_PCI_P2PDMA */ static inline int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset)
@@ -98,5 +100,10 @@ static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev,static inline void pci_p2pmem_publish (struct pci_dev *pdev,bool publish) { } +static inline int pci_p2pdma_map_sg (struct device *dev, +struct scatterlist *sg,int nents,enum dma_data_direction dir) +{ +return 0 ; +}#endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_PCI_P2P_H */
Signature · 1 line
2.19.0
Re: [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
Quoted 4 lines · Logan Gunthorpe
The DMA address used when mapping PCI P2P memory must be the PCI bus address. Thus, introduce
pci_p2pmem_map_sg()to map the correct addresses when using P2P memory. Memory mapped in this way does not need to be unmapped.
I think the use of "map" in this context is slightly confusing because the general expectation is that map/unmap must be balanced.
I assume it's because the "mapping" consumes no resources, e.g., requires no page table entries. Possibly there's a better verb than "map", e.g., "convert", "convert_to_p2pdma", etc?
If you keep "map", maybe add a sentence or two about why there's no corresponding unmap?
Quoted 4 lines
For this, we assume that an SGL passed to these functions contain all P2P memory or no P2P memory.
Signed-off-by Logan Gunthorpe <[email protected]>
Quoted 103 lines
--- drivers/pci/p2pdma.c | 43 ++++++++++++++++++++++++++++++++++++++ include/linux/memremap.h | 1 + include/linux/pci-p2pdma.h | 7 +++++++ 3 files changed, 51 insertions(+)drivers/pci/p2pdma.c +43
-0@@ -191,6 +191,8 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size, pgmap->res.flags =pci_resource_flags (pdev, bar); pgmap->ref = &pdev->p2pdma->devmap_ref; pgmap->type =MEMORY_DEVICE_PCI_P2PDMA ; + pgmap->pci_p2pdma_bus_offset =pci_bus_address (pdev, bar) - +pci_resource_start (pdev, bar); addr =devm_memremap_pages (&pdev->dev, pgmap);if (IS_ERR (addr)) {@@ -813,3 +815,44 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish) pdev->p2pdma->p2pmem_published = publish; }EXPORT_SYMBOL_GPL (pci_p2pmem_publish); + +/** +* pci_p2pdma_map_sg - map a PCI peer-to-peer scatterlist for DMA +* @dev: device doing the DMA request +* @sg: scatter list to map +* @nents: elements in the scatterlist +* @dir: DMA direction +* +* Scatterlists mapped with this function should not be unmapped in any way. +* +* Returns the number of SG entries mapped or 0 on error. +*/ +int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents, +enum dma_data_direction dir) +{ +struct dev_pagemap *pgmap; +struct scatterlist *s; +phys_addr_t paddr; +int i; + +/* +* p2pdma mappings are not compatible with devices that use +* dma_virt_ops. If the upper layers do the right thing +* this should never happen because it will be prevented +* by the check in pci_p2pdma_add_client() +*/ +if (WARN_ON_ONCE (IS_ENABLED (CONFIG_DMA_VIRT_OPS ) && + dev->dma_ops == &dma_virt_ops)) +return 0 ; + +for_each_sg (sg, s, nents, i) { + pgmap =sg_page (s)->pgmap; + paddr =sg_phys (s); + + s->dma_address = paddr - pgmap->pci_p2pdma_bus_offset; +sg_dma_len (s) = s->length; + } + +return nents; +} +EXPORT_SYMBOL_GPL (pci_p2pdma_map_sg);include/linux/memremap.h +1
-0@@ -125,6 +125,7 @@ struct dev_pagemap {struct device *dev;void *data;enum memory_type type; +u64 pci_p2pdma_bus_offset; };#ifdef CONFIG_ZONE_DEVICE include/linux/pci-p2pdma.h +7
-0@@ -36,6 +36,8 @@ struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, unsigned int *nents, u32 length);void pci_p2pmem_free_sgl (struct pci_dev *pdev,struct scatterlist *sgl);void pci_p2pmem_publish (struct pci_dev *pdev,bool publish); +int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents, +enum dma_data_direction dir);#else /* CONFIG_PCI_P2PDMA */ static inline int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset)@@ -98,5 +100,10 @@ static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev,static inline void pci_p2pmem_publish (struct pci_dev *pdev,bool publish) { } +static inline int pci_p2pdma_map_sg (struct device *dev, +struct scatterlist *sg,int nents,enum dma_data_direction dir) +{ +return 0 ; +}#endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_PCI_P2P_H */ Signature · 1 line
2.19.0
Re: [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
Quoted 9 lines · Bjorn Helgaas
Quoted 4 lines · Logan Gunthorpe
The DMA address used when mapping PCI P2P memory must be the PCI bus address. Thus, introduce
pci_p2pmem_map_sg()to map the correct addresses when using P2P memory. Memory mapped in this way does not need to be unmapped.I think the use of "map" in this context is slightly confusing because the general expectation is that map/unmap must be balanced.
Quoted 6 lines
I assume it's because the "mapping" consumes no resources, e.g., requires no page table entries. Possibly there's a better verb than "map", e.g., "convert", "
convert_to_p2pdma", etc?If you keep "map", maybe add a sentence or two about why there's no corresponding unmap?
Another wrinkle is that "map" usually takes an A and gives you back a B. Now the caller has both A and B and both are still valid.
Here we pass in an SGL and the SGL is transformed, so the caller only has B and A has been destroyed, i.e., the SGL can no longer be used as it was before, and there's no way to get A back.
Maybe this is pointless bikeshedding, so if your color is the best, don't change anything.
Bjorn
Logan Gunthorpe
maintainerRe: [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
Quoted 2 lines · Bjorn Helgaas
Quoted 2 lines
I think the use of "map" in this context is slightly confusing because the general expectation is that map/unmap must be balanced.
Yeah, Jason said the same thing, but having an empty unmap function seems wasteful and Christoph said to just remove it. My opinion is that it's not that big an issue one way or another -- if we have to add an unmap later it's not really that hard.
Quoted 2 lines
Quoted 2 lines
If you keep "map", maybe add a sentence or two about why there's no corresponding unmap?
Will do.
Quoted 5 lines
Another wrinkle is that "map" usually takes an A and gives you back a B. Now the caller has both A and B and both are still valid. Here we pass in an SGL and the SGL is transformed, so the caller only has B and A has been destroyed, i.e., the SGL can no longer be used as it was before, and there's no way to get A back.
I wouldn't say that. Our map_sg function is doing the same thing dma_map_sg is: it sets the DMA address and length in the scatter list. So B is still A just with other fields set. If the caller wanted to map this SG in a different way they can still do so and the new DMA address/length would override the old values. (Normally, you'd want to unmap before doing something like that, but seeing our unmap is an empty operation, we wouldn't have to do that.)
Logan
Re: [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
Quoted 20 lines · Logan Gunthorpe
Quoted 2 lines · Bjorn Helgaas
Quoted 2 lines
I think the use of "map" in this context is slightly confusing because the general expectation is that map/unmap must be balanced.
Yeah, Jason said the same thing, but having an empty unmap function seems wasteful and Christoph said to just remove it. My opinion is that it's not that big an issue one way or another -- if we have to add an unmap later it's not really that hard.
Quoted 2 lines
Quoted 2 lines
If you keep "map", maybe add a sentence or two about why there's no corresponding unmap?
Will do.
Quoted 5 lines
Another wrinkle is that "map" usually takes an A and gives you back a B. Now the caller has both A and B and both are still valid. Here we pass in an SGL and the SGL is transformed, so the caller only has B and A has been destroyed, i.e., the SGL can no longer be used as it was before, and there's no way to get A back.
Quoted 7 lines
I wouldn't say that. Our
map_sgfunction is doing the same thingdma_map_sgis: it sets the DMA address and length in the scatter list. So B is still A just with other fields set. If the caller wanted to map this SG in a different way they can still do so and the new DMA address/length would override the old values. (Normally, you'd want to unmap before doing something like that, but seeing our unmap is an empty operation, we wouldn't have to do that.)
Ok. I was assuming s->dma_address would have been already set before the call and would be overwritten by pci_p2pmem_map_sg(). But I guess that's not the case -- sounds like s->dma_address is undefined before the call.
Bjorn
Logan Gunthorpe
maintainerRe: [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
Quoted 4 lines · Bjorn Helgaas
Ok. I was assuming
s->dma_addresswould have been already set before the call and would be overwritten bypci_p2pmem_map_sg(). But I guess that's not the case -- sounds likes->dma_addressis undefined before the call.
Correct. It's set by whatever dma_map function operates on it.
Logan
Logan Gunthorpe
maintainerPATCH v6 4/13 PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers
In reply to Logan GunthorpeUsers of the P2PDMA infrastructure will typically need a way for the user to tell the kernel to use P2P resources. Typically this will be a simple on/off boolean operation but sometimes it may be desirable for the user to specify the exact device to use for the P2P operation.
Add new helpers for attributes which take a boolean or a PCI device. Any boolean, or the word 'auto' turn P2P on or off. Specifying a full PCI device name/BDF will select the specific device.
--- drivers/pci/p2pdma.c | 83 ++++++++++++++++++++++++++++++++++++++ include/linux/pci-p2pdma.h | 15 +++++++ 2 files changed, 98 insertions(+)
drivers/pci/p2pdma.c +83 -0
@@ -9,6 +9,7 @@*/ #define pr_fmt (fmt) "pci-p2pdma: " fmt +#include <linux/ctype.h> #include <linux/pci-p2pdma.h> #include <linux/module.h> #include <linux/slab.h>
@@ -856,3 +857,85 @@ int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,return nents; }EXPORT_SYMBOL_GPL (pci_p2pdma_map_sg); + +/** +* pci_p2pdma_enable_store - parse a configfs/sysfs attribute store +* to enable p2pdma +* @page: contents of the value to be stored +* @p2p_dev: returns the PCI device that was selected to be used +* (if 'auto', 'none or a boolean isn't the store value) +* @use_p2pdma: returns whether to enable p2pdma or not +* +* Parses an attribute value to decide whether to enable p2pdma. +* The value can select a PCI device (using it's full BDF device +* name), a boolean, or 'auto'. 'auto' and a true boolean value +* have the same meaning. A false value disables p2pdma and +* a PCI device enables it to use a specific device as the +* backing provider. +* +* pci_p2pdma_enable_show() should be used as the show operation for +* the attribute. +* +* Returns 0 on success +*/ +int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev, +bool *use_p2pdma) +{ +struct device *dev; + + dev =bus_find_device_by_name (&pci_bus_type,NULL , page); +if (dev) { + *use_p2pdma = true; + *p2p_dev =to_pci_dev (dev); + +if (!pci_has_p2pmem (*p2p_dev)) { +pr_err ("PCI device has no peer-to-peer memory: %s\n" , + page); +pci_dev_put (*p2p_dev); +return -ENODEV ; + } + +return 0 ; + }else if (sysfs_streq (page,"auto" )) { + *use_p2pdma = true; +return 0 ; + }else if ((page[0 ] =='0' || page[0 ] =='1' ) && !iscntrl (page[1 ])) { +/* +* If the user enters a PCI device that doesn't exist +* like "0000:01:00.1", we don't want strtobool to think +* it's a '0' when it's clearly not what the user wanted. +* So we require 0's and 1's to be exactly one character. +*/ + }else if (!strtobool (page, use_p2pdma)) { +return 0 ; + } + +pr_err ("No such PCI device: %.*s\n" , (int )strcspn (page,"\n" ), page); +return -ENODEV ; +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable_store); + +/** +* pci_p2pdma_enable_show - show a configfs/sysfs attribute indicating +* whether p2pdma is enabled +* @page: contents of the stored value +* @p2p_dev: the selected p2p device (NULL if no device is selected) +* @use_p2pdma: whether p2pdme has been enabled +* +* Attributes that use pci_p2pdma_enable_store() should use this function +* to show the value of the attribute. +* +* Returns 0 on success +*/ +ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev, +bool use_p2pdma) +{ +if (!use_p2pdma) +return sprintf (page,"none\n" ); + +if (!p2p_dev) +return sprintf (page,"auto\n" ); + +return sprintf (page,"%s\n" ,pci_name (p2p_dev)); +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable_show);
include/linux/pci-p2pdma.h +15 -0
@@ -38,6 +38,10 @@ void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl);void pci_p2pmem_publish (struct pci_dev *pdev,bool publish);int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir); +int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev, +bool *use_p2pdma); +ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev, +bool use_p2pdma);#else /* CONFIG_PCI_P2PDMA */ static inline int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset)
@@ -105,5 +109,16 @@ static inline int pci_p2pdma_map_sg(struct device *dev, {return 0 ; } +static inline int pci_p2pdma_enable_store (const char *page, +struct pci_dev **p2p_dev,bool *use_p2pdma) +{ + *use_p2pdma = false; +return 0 ; +} +static inline ssize_t pci_p2pdma_enable_show (char *page, +struct pci_dev *p2p_dev,bool use_p2pdma) +{ +return sprintf (page,"none\n" ); +}#endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_PCI_P2P_H */
Signature · 1 line
2.19.0
Re: [PATCH v6 04/13] PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers
Quoted 9 lines · Logan Gunthorpe
Users of the P2PDMA infrastructure will typically need a way for the user to tell the kernel to use P2P resources. Typically this will be a simple on/off boolean operation but sometimes it may be desirable for the user to specify the exact device to use for the P2P operation.
Add new helpers for attributes which take a boolean or a PCI device. Any boolean, or the word 'auto' turn P2P on or off. Specifying a full PCI device name/BDF will select the specific device.
I think examples of the valid booleans would be useful. "Boolean" suggests true/false, but AFAICT strtobool() only works for things like 0/1/on/off.
What's the need for "auto"? Sounds like it's equivalent to "on" but it's not obvious why we need both.
It'd be sort of nice if the contents of the sysfs file were valid things you could put *back* into it. But it looks like the contents might be "none", which isn't valid for the store. And the contents are never "on" or "off".
Where do these store/show functions get connected to sysfs? I know there's a lot of magic there, so apologies if I'm missing it.
Quoted 45 lines
Signed-off-by Logan Gunthorpe <[email protected]>--- drivers/pci/p2pdma.c | 83 ++++++++++++++++++++++++++++++++++++++ include/linux/pci-p2pdma.h | 15 +++++++ 2 files changed, 98 insertions(+)drivers/pci/p2pdma.c +24
-0@@ -9,6 +9,7 @@*/ #define pr_fmt (fmt) "pci-p2pdma: " fmt +#include <linux/ctype.h> #include <linux/pci-p2pdma.h> #include <linux/module.h> #include <linux/slab.h> @@ -856,3 +857,85 @@ int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,return nents; }EXPORT_SYMBOL_GPL (pci_p2pdma_map_sg); + +/** +* pci_p2pdma_enable_store - parse a configfs/sysfs attribute store +* to enable p2pdma +* @page: contents of the value to be stored +* @p2p_dev: returns the PCI device that was selected to be used +* (if 'auto', 'none or a boolean isn't the store value) +* @use_p2pdma: returns whether to enable p2pdma or not +* +* Parses an attribute value to decide whether to enable p2pdma. +* The value can select a PCI device (using it's full BDF device +* name), a boolean, or 'auto'. 'auto' and a true boolean value +* have the same meaning. A false value disables p2pdma and +* a PCI device enables it to use a specific device as the +* backing provider. +* +* pci_p2pdma_enable_show() should be used as the show operation for +* the attribute. +* +* Returns 0 on success +*/ +int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev, +bool *use_p2pdma)
drivers/pci/p2pdma.c int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents, · line 892
Quoted 10 lines
drivers/pci/p2pdma.c +10
-0+{ +struct device *dev; + + dev =bus_find_device_by_name (&pci_bus_type,NULL , page); +if (dev) { + *use_p2pdma = true; + *p2p_dev =to_pci_dev (dev); + +if (!pci_has_p2pmem (*p2p_dev)) { + pr_err("PCI device has no peer-to-peer memory: %s\n",
pci_err()?
Quoted 84 lines
drivers/pci/p2pdma.c +49
-0+ page); +pci_dev_put (*p2p_dev); +return -ENODEV ; + } + +return 0 ; + }else if (sysfs_streq (page,"auto" )) { + *use_p2pdma = true; +return 0 ; + }else if ((page[0 ] =='0' || page[0 ] =='1' ) && !iscntrl (page[1 ])) { +/* +* If the user enters a PCI device that doesn't exist +* like "0000:01:00.1", we don't want strtobool to think +* it's a '0' when it's clearly not what the user wanted. +* So we require 0's and 1's to be exactly one character. +*/ + }else if (!strtobool (page, use_p2pdma)) { +return 0 ; + } + +pr_err ("No such PCI device: %.*s\n" , (int )strcspn (page,"\n" ), page); +return -ENODEV ; +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable_store); + +/** +* pci_p2pdma_enable_show - show a configfs/sysfs attribute indicating +* whether p2pdma is enabled +* @page: contents of the stored value +* @p2p_dev: the selected p2p device (NULL if no device is selected) +* @use_p2pdma: whether p2pdme has been enabled +* +* Attributes that use pci_p2pdma_enable_store() should use this function +* to show the value of the attribute. +* +* Returns 0 on success +*/ +ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev, +bool use_p2pdma) +{ +if (!use_p2pdma) +return sprintf (page,"none\n" ); + +if (!p2p_dev) +return sprintf (page,"auto\n" ); + +return sprintf (page,"%s\n" ,pci_name (p2p_dev)); +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable_show);include/linux/pci-p2pdma.h +15
-0@@ -38,6 +38,10 @@ void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl);void pci_p2pmem_publish (struct pci_dev *pdev,bool publish);int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir); +int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev, +bool *use_p2pdma); +ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev, +bool use_p2pdma);#else /* CONFIG_PCI_P2PDMA */ static inline int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset)@@ -105,5 +109,16 @@ static inline int pci_p2pdma_map_sg(struct device *dev, {return 0 ; } +static inline int pci_p2pdma_enable_store (const char *page, +struct pci_dev **p2p_dev,bool *use_p2pdma) +{ + *use_p2pdma = false; +return 0 ; +} +static inline ssize_t pci_p2pdma_enable_show (char *page, +struct pci_dev *p2p_dev,bool use_p2pdma) +{ +return sprintf (page,"none\n" ); +}#endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_PCI_P2P_H */ Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerRe: [PATCH v6 04/13] PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers
Quoted 2 lines · Bjorn Helgaas
Where do these store/show functions get connected to sysfs? I know there's a lot of magic there, so apologies if I'm missing it.
Well in this patch set they are used in configfs not sysfs. It's not magic, they are just helpers for other drivers. They are called in Patch 13 by NVMe target's configfs functions.
Logan
Logan Gunthorpe
maintainerRe: [PATCH v6 04/13] PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers
In reply to Bjorn HelgaasQuoted 4 lines · Bjorn Helgaas
It'd be sort of nice if the contents of the sysfs file were valid things you could put *back* into it. But it looks like the contents might be "none", which isn't valid for the store. And the contents are never "on" or "off".
In fact, "none" would be accepted by strtobool() as a false (seeing it starts with 'n' and strtobool() only looks at the first character). However, per the rest of your feedback I've changed show() to report "0", "1" or pci_name() and dropped "auto". This matches what the other strtobool configfs attributes are doing in NVMe target.
Logan
Re: [PATCH v6 04/13] PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers
In reply to Logan GunthorpeQuoted 11 lines · Logan Gunthorpe
Users of the P2PDMA infrastructure will typically need a way for the user to tell the kernel to use P2P resources. Typically this will be a simple on/off boolean operation but sometimes it may be desirable for the user to specify the exact device to use for the P2P operation.
Add new helpers for attributes which take a boolean or a PCI device. Any boolean, or the word 'auto' turn P2P on or off. Specifying a full PCI device name/BDF will select the specific device.
Signed-off-by Logan Gunthorpe <[email protected]>
I appreciate your update to drop "auto" and show 0/1/pci_name(). When you post that, include my:
Quoted 138 lines
--- drivers/pci/p2pdma.c | 83 ++++++++++++++++++++++++++++++++++++++ include/linux/pci-p2pdma.h | 15 +++++++ 2 files changed, 98 insertions(+)drivers/pci/p2pdma.c +83
-0@@ -9,6 +9,7 @@*/ #define pr_fmt (fmt) "pci-p2pdma: " fmt +#include <linux/ctype.h> #include <linux/pci-p2pdma.h> #include <linux/module.h> #include <linux/slab.h> @@ -856,3 +857,85 @@ int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,return nents; }EXPORT_SYMBOL_GPL (pci_p2pdma_map_sg); + +/** +* pci_p2pdma_enable_store - parse a configfs/sysfs attribute store +* to enable p2pdma +* @page: contents of the value to be stored +* @p2p_dev: returns the PCI device that was selected to be used +* (if 'auto', 'none or a boolean isn't the store value) +* @use_p2pdma: returns whether to enable p2pdma or not +* +* Parses an attribute value to decide whether to enable p2pdma. +* The value can select a PCI device (using it's full BDF device +* name), a boolean, or 'auto'. 'auto' and a true boolean value +* have the same meaning. A false value disables p2pdma and +* a PCI device enables it to use a specific device as the +* backing provider. +* +* pci_p2pdma_enable_show() should be used as the show operation for +* the attribute. +* +* Returns 0 on success +*/ +int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev, +bool *use_p2pdma) +{ +struct device *dev; + + dev =bus_find_device_by_name (&pci_bus_type,NULL , page); +if (dev) { + *use_p2pdma = true; + *p2p_dev =to_pci_dev (dev); + +if (!pci_has_p2pmem (*p2p_dev)) { +pr_err ("PCI device has no peer-to-peer memory: %s\n" , + page); +pci_dev_put (*p2p_dev); +return -ENODEV ; + } + +return 0 ; + }else if (sysfs_streq (page,"auto" )) { + *use_p2pdma = true; +return 0 ; + }else if ((page[0 ] =='0' || page[0 ] =='1' ) && !iscntrl (page[1 ])) { +/* +* If the user enters a PCI device that doesn't exist +* like "0000:01:00.1", we don't want strtobool to think +* it's a '0' when it's clearly not what the user wanted. +* So we require 0's and 1's to be exactly one character. +*/ + }else if (!strtobool (page, use_p2pdma)) { +return 0 ; + } + +pr_err ("No such PCI device: %.*s\n" , (int )strcspn (page,"\n" ), page); +return -ENODEV ; +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable_store); + +/** +* pci_p2pdma_enable_show - show a configfs/sysfs attribute indicating +* whether p2pdma is enabled +* @page: contents of the stored value +* @p2p_dev: the selected p2p device (NULL if no device is selected) +* @use_p2pdma: whether p2pdme has been enabled +* +* Attributes that use pci_p2pdma_enable_store() should use this function +* to show the value of the attribute. +* +* Returns 0 on success +*/ +ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev, +bool use_p2pdma) +{ +if (!use_p2pdma) +return sprintf (page,"none\n" ); + +if (!p2p_dev) +return sprintf (page,"auto\n" ); + +return sprintf (page,"%s\n" ,pci_name (p2p_dev)); +} +EXPORT_SYMBOL_GPL (pci_p2pdma_enable_show);include/linux/pci-p2pdma.h +15
-0@@ -38,6 +38,10 @@ void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl);void pci_p2pmem_publish (struct pci_dev *pdev,bool publish);int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir); +int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev, +bool *use_p2pdma); +ssize_t pci_p2pdma_enable_show (char *page,struct pci_dev *p2p_dev, +bool use_p2pdma);#else /* CONFIG_PCI_P2PDMA */ static inline int pci_p2pdma_add_resource (struct pci_dev *pdev,int bar,size_t size,u64 offset)@@ -105,5 +109,16 @@ static inline int pci_p2pdma_map_sg(struct device *dev, {return 0 ; } +static inline int pci_p2pdma_enable_store (const char *page, +struct pci_dev **p2p_dev,bool *use_p2pdma) +{ + *use_p2pdma = false; +return 0 ; +} +static inline ssize_t pci_p2pdma_enable_show (char *page, +struct pci_dev *p2p_dev,bool use_p2pdma) +{ +return sprintf (page,"none\n" ); +}#endif /* CONFIG_PCI_P2PDMA */ #endif /* _LINUX_PCI_P2P_H */ Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 5/13 docs-rst: Add a new directory for PCI documentation
In reply to Logan GunthorpeAdd a new directory in the driver API guide for PCI specific documentation.
This is in preparation for adding a new PCI P2P DMA driver writers guide which will go in this directory.
---
Documentation/driver-api/index.rst | 2 +-
Documentation/driver-api/pci/index.rst | 20 ++++++++++++++++++++
Documentation/driver-api/{ => pci}/pci.rst | 0
3 files changed, 21 insertions(+), 1 deletion(-)
create mode 100644 Documentation/driver-api/pci/index.rst
rename Documentation/driver-api/{ => pci}/pci.rst (100%)Documentation/driver-api/index.rst +1 -1
@@ -29,7 +29,7 @@ available subsections can be seen below. iio/index input usb/index - pci + pci/index spi i2c hsi
Documentation/driver-api/pci/index.rst new file +20 -0
@@ -0,0 +1,20 @@ +.. SPDX-License-Identifier: GPL-2.0 +============================================ +The Linux PCI driver implementer's API guide +============================================ + +.. class:: toc-title + + Table of contents + +.. toctree:: + :maxdepth: 2 + + pci + +.. only:: subproject and html + + Indices + ======= + + * :ref:`genindex`
Documentation/driver-api/pci/pci.rst renamed from Documentation/driver-api/pci.rst +0 -0
Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 6/13 PCI/P2PDMA: Add P2P DMA driver writer's documentation
In reply to Logan GunthorpeAdd a restructured text file describing how to write drivers with support for P2P DMA transactions. The document describes how to use the APIs that were added in the previous few commits.
Also adds an index for the PCI documentation tree even though this is the only PCI document that has been converted to restructured text at this time.
--- Documentation/driver-api/pci/index.rst | 1 + Documentation/driver-api/pci/p2pdma.rst | 170 ++++++++++++++++++++++++ 2 files changed, 171 insertions(+) create mode 100644 Documentation/driver-api/pci/p2pdma.rst
Documentation/driver-api/pci/index.rst +1 -0
@@ -11,6 +11,7 @@ The Linux PCI driver implementer's API guide :maxdepth: 2 pci + p2pdma .. only:: subproject and html
Documentation/driver-api/pci/p2pdma.rst new file +170 -0
170 lines not on this page · raw message
Signature · 1 line
2.19.0
Re: [PATCH v6 06/13] PCI/P2PDMA: Add P2P DMA driver writer's documentation
Quoted 10 lines · Logan Gunthorpe
Add a restructured text file describing how to write drivers with support for P2P DMA transactions. The document describes how to use the APIs that were added in the previous few commits.
Also adds an index for the PCI documentation tree even though this is the only PCI document that has been converted to restructured text at this time.
Signed-off-by Logan Gunthorpe <[email protected]>
Documentation/driver-api/pci/p2pdma.rst line 118
Quoted 5 lines
Documentation/driver-api/pci/p2pdma.rst new file +5
-0+With the client list in hand, the orchestrator may then call +:c:func:`pci_p2pmem_find()` to obtain a published P2P memory provider +that is supported (behind the same root port) as all the clients. If more +than one provider is supported, the one nearest to all the clients will +be chosen first. If there are more than one provider is an equal distance
s/If there are more/If more/
Quoted 1 line
+away, the one returned will be chosen at random. This function returns the PCI
s/the one returned will be chosen at random/one will be chosen arbitrarily/ ? (I doubt it's really random)
Documentation/driver-api/pci/p2pdma.rst line 155
Quoted 10 lines
Documentation/driver-api/pci/p2pdma.rst new file +9
-0+device to use for the provider with a reference taken and therefore +when it's no longer needed it should be returned with pci_dev_put(). +Struct Page Caveats +------------------- + +Driver writers should be very careful about not passing these special +struct pages to code that isn't prepared for it. At this time, the kernel +interfaces do not have any checks for ensuring this. This obviously +precludes passing these pages to userspace.
Sounds like landmines here since the reader probably can't translate "code that isn't prepared for it" into a list of interfaces that are off-limits. But that's a VM issue that is above my pay grade, so I'm not suggesting any change; just pointing out something that makes me wonder "hmmm..., how would I act on this?"
Quoted 7 lines
Documentation/driver-api/pci/p2pdma.rst new file +7
-0+P2P memory is also technically IO memory but should never have any side +effects behind it. Thus, the order of loads and stores should not be important +and ioreadX(), iowriteX() and friends should not be necessary. +However, as the memory is not cache coherent, if access ever needs to +be protected by a spinlock then :c:func:`mmiowb()` must be used before +unlocking the lock. (See ACQUIRES VS I/O ACCESSES in +Documentation/memory-barriers.txt)
Logan Gunthorpe
maintainerRe: [PATCH v6 06/13] PCI/P2PDMA: Add P2P DMA driver writer's documentation
Quoted 4 lines · Bjorn Helgaas
Quoted 1 line
+away, the one returned will be chosen at random. This function returns the PCI
s/the one returned will be chosen at random/one will be chosen arbitrarily/ ? (I doubt it's really random)
You're the second person to ask this, but yes, we really do it randomly. Such that if there are multiple drivers getting resources they should be evenly distributed. It's not the ideal solution but if it were to do it arbitrarily then the code would likely return the same device all the time and any additional devices would not get used.
Quoted 5 lines
Quoted 5 lines
Documentation/driver-api/pci/p2pdma.rst new file +5
-0+Struct Page Caveats +------------------- + +Driver writers should be very careful about not passing these special +struct pages to code that isn't prepared for it. At this time, the kernel
Quoted 8 lines
Documentation/driver-api/pci/p2pdma.rst line 155
Quoted 2 lines
Documentation/driver-api/pci/p2pdma.rst new file +2
-0+interfaces do not have any checks for ensuring this. This obviously +precludes passing these pages to userspace.Sounds like landmines here since the reader probably can't translate "code that isn't prepared for it" into a list of interfaces that are off-limits. But that's a VM issue that is above my pay grade, so I'm not suggesting any change; just pointing out something that makes me wonder "hmmm..., how would I act on this?"
Yes, these are big landmines. These are the problems we have been fighting with trying to get P2P memory into the kernel and the first step seems to have settled on the developer using them has to be careful and ensure they are only sent to paths that are used correctly. This series does these things in the block layer and RDMA interface, but due to Jens not wanting to add warnings to the common code, it's still possible for a developer to code something that sends these pages to block devices that are not prepared for it.
Logan
Re: [PATCH v6 06/13] PCI/P2PDMA: Add P2P DMA driver writer's documentation
Quoted 11 lines · Logan Gunthorpe
Quoted 4 lines · Bjorn Helgaas
Quoted 1 line
+away, the one returned will be chosen at random. This function returns the PCI
s/the one returned will be chosen at random/one will be chosen arbitrarily/ ? (I doubt it's really random)
You're the second person to ask this, but yes, we really do it randomly. Such that if there are multiple drivers getting resources they should be evenly distributed. It's not the ideal solution but if it were to do it arbitrarily then the code would likely return the same device all the time and any additional devices would not get used.
Ah, I see, I should have searched for "random" in that patch and answered my own question :) Sorry for the noise.
Bjorn
Logan Gunthorpe
maintainerPATCH v6 7/13 block: Add PCI P2P flag for request queue and check support for requests
In reply to Logan GunthorpeQUEUE_FLAG_PCI_P2P is introduced meaning a driver's request queue supports targeting P2P memory.
When a request is submitted we check if PCI P2PDMA memory is assigned to the first page in the bio. If it is, we ensure the queue it's submitted to supports it, and enforce REQ_NOMERGE.
--- include/linux/blkdev.h | 3 +++ 1 file changed, 3 insertions(+)
include/linux/blkdev.h +3 -0
@@ -699,6 +699,7 @@ struct request_queue {#define QUEUE_FLAG_SCSI_PASSTHROUGH 27/* queue supports SCSI commands */ #define QUEUE_FLAG_QUIESCED 28/* queue has been quiesced */ #define QUEUE_FLAG_PREEMPT_ONLY 29/* only process REQ_PREEMPT requests */ +#define QUEUE_FLAG_PCI_P2PDMA 30/* device supports pci p2p requests */ #define QUEUE_FLAG_DEFAULT ((1 << QUEUE_FLAG_IO_STAT) | \ (1 << QUEUE_FLAG_SAME_COMP) | \
@@ -731,6 +732,8 @@ bool blk_queue_flag_test_and_clear(unsigned int flag, struct request_queue *q);#define blk_queue_dax (q) test_bit(QUEUE_FLAG_DAX, &(q)->queue_flags)#define blk_queue_scsi_passthrough (q) \ test_bit(QUEUE_FLAG_SCSI_PASSTHROUGH, &(q)->queue_flags) +#define blk_queue_pci_p2pdma (q) \ + test_bit(QUEUE_FLAG_PCI_P2PDMA, &(q)->queue_flags)#define blk_noretry_request (rq) \ ((rq)->cmd_flags & (REQ_FAILFAST_DEV|REQ_FAILFAST_TRANSPORT| \
Signature · 1 line
2.19.0
Jens Axboe
maintainerRe: [PATCH v6 07/13] block: Add PCI P2P flag for request queue and check support for requests
Quoted 6 lines · Logan Gunthorpe
QUEUE_FLAG_PCI_P2Pis introduced meaning a driver's request queue supports targeting P2P memory.When a request is submitted we check if PCI P2PDMA memory is assigned to the first page in the bio. If it is, we ensure the queue it's submitted to supports it, and enforce
REQ_NOMERGE.
This changelog is no longer accurate.
Signature · 1 line
Jens Axboe
Logan Gunthorpe
maintainerRe: [PATCH v6 07/13] block: Add PCI P2P flag for request queue and check support for requests
Quoted 9 lines · Jens Axboe
Quoted 6 lines · Logan Gunthorpe
QUEUE_FLAG_PCI_P2Pis introduced meaning a driver's request queue supports targeting P2P memory.When a request is submitted we check if PCI P2PDMA memory is assigned to the first page in the bio. If it is, we ensure the queue it's submitted to supports it, and enforce
REQ_NOMERGE.This changelog is no longer accurate.
Oops, I'll fix this for v7.
Thanks,
Logan
Logan Gunthorpe
maintainerPATCH v6 8/13 IB/core: Ensure we map P2P memory correctly in rdma_rw_ctx_[init|destroy]()
In reply to Logan GunthorpeIn order to use PCI P2P memory the pci_p2pmem_map_sg() function must be called to map the correct PCI bus address.
To do this, check the first page in the scatter list to see if it is P2P memory or not. At the moment, scatter lists that contain P2P memory must be homogeneous so if the first page is P2P the entire SGL should be P2P.
--- drivers/infiniband/core/rw.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-)
drivers/infiniband/core/rw.c +9 -2
@@ -12,6 +12,7 @@*/ #include <linux/moduleparam.h> #include <linux/slab.h> +#include <linux/pci-p2pdma.h> #include <rdma/mr_pool.h> #include <rdma/rw.h>
@@ -280,7 +281,11 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,struct ib_device *dev = qp->pd->device;int ret; - ret =ib_dma_map_sg (dev, sg, sg_cnt, dir); +if (is_pci_p2pdma_page (sg_page (sg))) + ret =pci_p2pdma_map_sg (dev->dma_device, sg, sg_cnt, dir); +else + ret =ib_dma_map_sg (dev, sg, sg_cnt, dir); +if (!ret)return -ENOMEM ; sg_cnt = ret;
@@ -602,7 +607,9 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u8 port_num,break ; } -ib_dma_unmap_sg (qp->pd->device, sg, sg_cnt, dir); +/* P2PDMA contexts do not need to be unmapped */ +if (!is_pci_p2pdma_page (sg_page (sg))) +ib_dma_unmap_sg (qp->pd->device, sg, sg_cnt, dir); }EXPORT_SYMBOL (rdma_rw_ctx_destroy);
Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 9/13 nvme-pci: Use PCI p2pmem subsystem to manage the CMB
In reply to Logan GunthorpeRegister the CMB buffer as p2pmem and use the appropriate allocation functions to create and destroy the IO submission queues.
If the CMB supports WDS and RDS, publish it for use as P2P memory by other devices.
Kernels without CONFIG_PCI_P2PDMA will also no longer support NVMe CMB. However, seeing the main use-cases for the CMB is P2P operations, this seems like a reasonable dependency.
We drop the __iomem safety on the buffer seeing that, by convention, it's safe to directly access memory mapped by memremap()/devm_memremap_pages(). Architectures where this is not safe will not be supported by memremap() and therefore will not be support PCI P2P and have no support for CMB.
--- drivers/nvme/host/pci.c | 80 +++++++++++++++++++++++------------------ 1 file changed, 45 insertions(+), 35 deletions(-)
drivers/nvme/host/pci.c +45 -35
@@ -30,6 +30,7 @@#include <linux/types.h> #include <linux/io-64-nonatomic-lo-hi.h> #include <linux/sed-opal.h> +#include <linux/pci-p2pdma.h> #include "nvme.h"
@@ -99,9 +100,8 @@ struct nvme_dev {struct work_struct remove_work;struct mutex shutdown_lock;bool subsystem; -void __iomem *cmb; -pci_bus_addr_t cmb_bus_addr;u64 cmb_size; +bool cmb_use_sqes;u32 cmbsz;u32 cmbloc;struct nvme_ctrl ctrl;
@@ -158,7 +158,7 @@ struct nvme_queue {struct nvme_dev *dev;spinlock_t sq_lock;struct nvme_command *sq_cmds; -struct nvme_command __iomem *sq_cmds_io; +bool sq_cmds_is_io;spinlock_t cq_lock____cacheline_aligned_in_smp ;volatile struct nvme_completion *cqes;struct blk_mq_tags **tags;
@@ -447,11 +447,8 @@ static int nvme_pci_map_queues(struct blk_mq_tag_set *set)static void nvme_submit_cmd (struct nvme_queue *nvmeq,struct nvme_command *cmd) {spin_lock (&nvmeq->sq_lock); -if (nvmeq->sq_cmds_io) -memcpy_toio (&nvmeq->sq_cmds_io[nvmeq->sq_tail], cmd, -sizeof (*cmd)); -else -memcpy (&nvmeq->sq_cmds[nvmeq->sq_tail], cmd,sizeof (*cmd)); + +memcpy (&nvmeq->sq_cmds[nvmeq->sq_tail], cmd,sizeof (*cmd));if (++nvmeq->sq_tail == nvmeq->q_depth) nvmeq->sq_tail =0 ;
@@ -1232,9 +1229,18 @@ static void nvme_free_queue(struct nvme_queue *nvmeq) {dma_free_coherent (nvmeq->q_dmadev,CQ_SIZE (nvmeq->q_depth), (void *)nvmeq->cqes, nvmeq->cq_dma_addr); -if (nvmeq->sq_cmds) -dma_free_coherent (nvmeq->q_dmadev,SQ_SIZE (nvmeq->q_depth), - nvmeq->sq_cmds, nvmeq->sq_dma_addr); + +if (nvmeq->sq_cmds) { +if (nvmeq->sq_cmds_is_io) +pci_free_p2pmem (to_pci_dev (nvmeq->q_dmadev), + nvmeq->sq_cmds, +SQ_SIZE (nvmeq->q_depth)); +else +dma_free_coherent (nvmeq->q_dmadev, +SQ_SIZE (nvmeq->q_depth), + nvmeq->sq_cmds, + nvmeq->sq_dma_addr); + } }static void nvme_free_queues (struct nvme_dev *dev,int lowest)
@@ -1323,12 +1329,21 @@ static int nvme_cmb_qdepth(struct nvme_dev *dev, int nr_io_queues,static int nvme_alloc_sq_cmds (struct nvme_dev *dev,struct nvme_queue *nvmeq,int qid,int depth) { -/* CMB SQEs will be mapped before creation */ -if (qid && dev->cmb && use_cmb_sqes && (dev->cmbsz &NVME_CMBSZ_SQS )) -return 0 ; +struct pci_dev *pdev =to_pci_dev (dev->dev); + +if (qid && dev->cmb_use_sqes && (dev->cmbsz &NVME_CMBSZ_SQS )) { + nvmeq->sq_cmds =pci_alloc_p2pmem (pdev,SQ_SIZE (depth)); + nvmeq->sq_dma_addr =pci_p2pmem_virt_to_bus (pdev, + nvmeq->sq_cmds); + nvmeq->sq_cmds_is_io = true; + } + +if (!nvmeq->sq_cmds) { + nvmeq->sq_cmds =dma_alloc_coherent (dev->dev,SQ_SIZE (depth), + &nvmeq->sq_dma_addr,GFP_KERNEL ); + nvmeq->sq_cmds_is_io = false; + } - nvmeq->sq_cmds =dma_alloc_coherent (dev->dev,SQ_SIZE (depth), - &nvmeq->sq_dma_addr,GFP_KERNEL );if (!nvmeq->sq_cmds)return -ENOMEM ;return 0 ;
@@ -1405,13 +1420,6 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid)int result;s16 vector; -if (dev->cmb && use_cmb_sqes && (dev->cmbsz &NVME_CMBSZ_SQS )) { -unsigned offset = (qid -1 ) *roundup (SQ_SIZE (nvmeq->q_depth), - dev->ctrl.page_size); - nvmeq->sq_dma_addr = dev->cmb_bus_addr + offset; - nvmeq->sq_cmds_io = dev->cmb + offset; - } -/* * A queue's vector matches the queue identifier unless the controller * has only one vector available.
@@ -1652,9 +1660,6 @@ static void nvme_map_cmb(struct nvme_dev *dev)return ; dev->cmbloc =readl (dev->bar +NVME_REG_CMBLOC ); -if (!use_cmb_sqes) -return ; - size =nvme_cmb_size_unit (dev) *nvme_cmb_size (dev); offset =nvme_cmb_size_unit (dev) *NVME_CMB_OFST (dev->cmbloc); bar =NVME_CMB_BIR (dev->cmbloc);
@@ -1671,11 +1676,18 @@ static void nvme_map_cmb(struct nvme_dev *dev)if (size > bar_size - offset) size = bar_size - offset; - dev->cmb =ioremap_wc (pci_resource_start (pdev, bar) + offset, size); -if (!dev->cmb) +if (pci_p2pdma_add_resource (pdev, bar, size, offset)) { +dev_warn (dev->ctrl.device, +"failed to register the CMB\n" );return ; - dev->cmb_bus_addr =pci_bus_address (pdev, bar) + offset; + } + dev->cmb_size = size; + dev->cmb_use_sqes = use_cmb_sqes && (dev->cmbsz &NVME_CMBSZ_SQS ); + +if ((dev->cmbsz & (NVME_CMBSZ_WDS |NVME_CMBSZ_RDS )) == + (NVME_CMBSZ_WDS |NVME_CMBSZ_RDS )) +pci_p2pmem_publish (pdev, true);if (sysfs_add_file_to_group (&dev->ctrl.device->kobj, &dev_attr_cmb.attr,NULL ))
@@ -1685,12 +1697,10 @@ static void nvme_map_cmb(struct nvme_dev *dev)static inline void nvme_release_cmb (struct nvme_dev *dev) { -if (dev->cmb) { -iounmap (dev->cmb); - dev->cmb =NULL ; +if (dev->cmb_size) {sysfs_remove_file_from_group (&dev->ctrl.device->kobj, &dev_attr_cmb.attr,NULL ); - dev->cmbsz =0 ; + dev->cmb_size =0 ; } }
@@ -1889,13 +1899,13 @@ static int nvme_setup_io_queues(struct nvme_dev *dev)if (nr_io_queues ==0 )return 0 ; -if (dev->cmb && (dev->cmbsz &NVME_CMBSZ_SQS )) { +if (dev->cmb_use_sqes) { result =nvme_cmb_qdepth (dev, nr_io_queues,sizeof (struct nvme_command ));if (result >0 ) dev->q_depth = result;else -nvme_release_cmb (dev); + dev->cmb_use_sqes = false; } do {
Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 10/13 nvme-pci: Add support for P2P memory in requests
In reply to Logan GunthorpeFor P2P requests, we must use the pci_p2pmem_map_sg() function instead of the dma_map_sg functions.
With that, we can then indicate PCI_P2P support in the request queue. For this, we create an NVME_F_PCI_P2P flag which tells the core to set QUEUE_FLAG_PCI_P2P in the request queue.
--- drivers/nvme/host/core.c | 4 ++++ drivers/nvme/host/nvme.h | 1 + drivers/nvme/host/pci.c | 17 +++++++++++++---- 3 files changed, 18 insertions(+), 4 deletions(-)
drivers/nvme/host/core.c +4 -0
@@ -3051,7 +3051,11 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid) ns->queue =blk_mq_init_queue (ctrl->tagset);if (IS_ERR (ns->queue)) goto out_free_ns; +blk_queue_flag_set (QUEUE_FLAG_NONROT , ns->queue); +if (ctrl->ops->flags &NVME_F_PCI_P2PDMA ) +blk_queue_flag_set (QUEUE_FLAG_PCI_P2PDMA , ns->queue); + ns->queue->queuedata = ns; ns->ctrl = ctrl;
drivers/nvme/host/nvme.h +1 -0
@@ -343,6 +343,7 @@ struct nvme_ctrl_ops {unsigned int flags;#define NVME_F_FABRICS (1 << 0)#define NVME_F_METADATA_SUPPORTED (1 << 1) +#define NVME_F_PCI_P2PDMA (1 << 2)int (*reg_read32)(struct nvme_ctrl *ctrl,u32 off,u32 *val);int (*reg_write32)(struct nvme_ctrl *ctrl,u32 off,u32 val);int (*reg_read64)(struct nvme_ctrl *ctrl,u32 off,u64 *val);
drivers/nvme/host/pci.c +13 -4
@@ -745,8 +745,13 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, goto out; ret =BLK_STS_RESOURCE ; - nr_mapped =dma_map_sg_attrs (dev->dev, iod->sg, iod->nents, dma_dir, -DMA_ATTR_NO_WARN ); + +if (is_pci_p2pdma_page (sg_page (iod->sg))) + nr_mapped =pci_p2pdma_map_sg (dev->dev, iod->sg, iod->nents, + dma_dir); +else + nr_mapped =dma_map_sg_attrs (dev->dev, iod->sg, iod->nents, + dma_dir,DMA_ATTR_NO_WARN );if (!nr_mapped) goto out;
@@ -788,7 +793,10 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req) DMA_TO_DEVICE :DMA_FROM_DEVICE ;if (iod->nents) { -dma_unmap_sg (dev->dev, iod->sg, iod->nents, dma_dir); +/* P2PDMA requests do not need to be unmapped */ +if (!is_pci_p2pdma_page (sg_page (iod->sg))) +dma_unmap_sg (dev->dev, iod->sg, iod->nents, dma_dir); +if (blk_integrity_rq (req))dma_unmap_sg (dev->dev, &iod->meta_sg,1 , dma_dir); }
@@ -2400,7 +2408,8 @@ static int nvme_pci_get_address(struct nvme_ctrl *ctrl, char *buf, int size)static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name ="pcie" , .module =THIS_MODULE , - .flags =NVME_F_METADATA_SUPPORTED , + .flags =NVME_F_METADATA_SUPPORTED | +NVME_F_PCI_P2PDMA , .reg_read32 = nvme_pci_reg_read32, .reg_write32 = nvme_pci_reg_write32, .reg_read64 = nvme_pci_reg_read64,
Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 11/13 nvme-pci: Add a quirk for a pseudo CMB
In reply to Logan GunthorpeIntroduce a quirk to use CMB-like memory on older devices that have an exposed BAR but do not advertise support for using CMBLOC and CMBSIZE.
We'd like to use some of these older cards to test P2P memory.
--- drivers/nvme/host/nvme.h | 7 +++++++ drivers/nvme/host/pci.c | 24 ++++++++++++++++++++---- 2 files changed, 27 insertions(+), 4 deletions(-)
drivers/nvme/host/nvme.h +7 -0
@@ -90,6 +90,13 @@ enum nvme_quirks {* Set MEDIUM priority on SQ creation */ NVME_QUIRK_MEDIUM_PRIO_SQ = (1 <<7 ), + +/* +* Pseudo CMB Support on BAR 4. For adapters like the Microsemi +* NVRAM that have CMB-like memory on a BAR but does not set +* CMBLOC or CMBSZ. +*/ + NVME_QUIRK_PSEUDO_CMB_BAR4 = (1 << 8), };/*
drivers/nvme/host/pci.c +20 -4
@@ -1644,6 +1644,13 @@ static ssize_t nvme_cmb_show(struct device *dev, } static DEVICE_ATTR(cmb, S_IRUGO, nvme_cmb_show, NULL); +static u32 nvme_pseudo_cmbsz (struct pci_dev *pdev,int bar) +{ +return NVME_CMBSZ_WDS |NVME_CMBSZ_RDS | + (((ilog2 (SZ_16M ) -12 ) /4 ) <<NVME_CMBSZ_SZU_SHIFT ) | + ((pci_resource_len (pdev, bar) /SZ_16M ) <<NVME_CMBSZ_SZ_SHIFT ); +} +static u64 nvme_cmb_size_unit (struct nvme_dev *dev) {u8 szu = (dev->cmbsz >>NVME_CMBSZ_SZU_SHIFT ) &NVME_CMBSZ_SZU_MASK ;
@@ -1663,10 +1670,15 @@ static void nvme_map_cmb(struct nvme_dev *dev)struct pci_dev *pdev =to_pci_dev (dev->dev);int bar; - dev->cmbsz =readl (dev->bar +NVME_REG_CMBSZ ); -if (!dev->cmbsz) -return ; - dev->cmbloc =readl (dev->bar +NVME_REG_CMBLOC ); +if (dev->ctrl.quirks &NVME_QUIRK_PSEUDO_CMB_BAR4 ) { + dev->cmbsz =nvme_pseudo_cmbsz (pdev,4 ); + dev->cmbloc =4 ; + }else { + dev->cmbsz =readl (dev->bar +NVME_REG_CMBSZ ); +if (!dev->cmbsz) +return ; + dev->cmbloc =readl (dev->bar +NVME_REG_CMBLOC ); + } size =nvme_cmb_size_unit (dev) *nvme_cmb_size (dev); offset =nvme_cmb_size_unit (dev) *NVME_CMB_OFST (dev->cmbloc);
@@ -2715,6 +2727,10 @@ static const struct pci_device_id nvme_id_table[] = { .driver_data = NVME_QUIRK_LIGHTNVM, }, { PCI_DEVICE(0x1d1d, 0x2601), /* CNEX Granby */ .driver_data = NVME_QUIRK_LIGHTNVM, }, + { PCI_DEVICE(0x11f8, 0xf117), /* Microsemi NVRAM adaptor */ + .driver_data = NVME_QUIRK_PSEUDO_CMB_BAR4, }, + { PCI_DEVICE(0x1db1, 0x0002), /* Everspin nvNitro adaptor */ + .driver_data = NVME_QUIRK_PSEUDO_CMB_BAR4, }, { PCI_DEVICE_CLASS(PCI_CLASS_STORAGE_EXPRESS, 0xffffff) }, { PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2001) }, { PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2003) },
Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 12/13 nvmet: Introduce helper functions to allocate and free request SGLs
In reply to Logan GunthorpeAdd helpers to allocate and free the SGL in a struct nvmet_req:
int nvmet_req_alloc_sgl(struct nvmet_req *req, struct nvmet_sq *sq) void nvmet_req_free_sgl(struct nvmet_req *req)
This will be expanded in a future patch to implement peer-to-peer memory DMAs and should be common with all target drivers. The presently unused 'sq' argument in the alloc function will be necessary to decide whether to use peer-to-peer memory and obtain the correct provider to allocate the memory.
The new helpers are used in nvmet-rdma. Seeing we use req.transfer_len as the length of the SGL it is set earlier and cleared on any error. It also seems to be unnecessary to accumulate the length as the map_sgl functions should only ever be called once per request.
--- drivers/nvme/target/core.c | 18 ++++++++++++++++++ drivers/nvme/target/nvmet.h | 2 ++ drivers/nvme/target/rdma.c | 20 ++++++++++++-------- 3 files changed, 32 insertions(+), 8 deletions(-)
drivers/nvme/target/core.c +18 -0
@@ -725,6 +725,24 @@ void nvmet_req_execute(struct nvmet_req *req) }EXPORT_SYMBOL_GPL (nvmet_req_execute); +int nvmet_req_alloc_sgl (struct nvmet_req *req,struct nvmet_sq *sq) +{ + req->sg =sgl_alloc (req->transfer_len,GFP_KERNEL , &req->sg_cnt); +if (!req->sg) +return -ENOMEM ; + +return 0 ; +} +EXPORT_SYMBOL_GPL (nvmet_req_alloc_sgl); + +void nvmet_req_free_sgl (struct nvmet_req *req) +{ +sgl_free (req->sg); + req->sg =NULL ; + req->sg_cnt =0 ; +} +EXPORT_SYMBOL_GPL (nvmet_req_free_sgl); +static inline bool nvmet_cc_en (u32 cc) {return (cc >>NVME_CC_EN_SHIFT ) &0x1 ;
drivers/nvme/target/nvmet.h +2 -0
@@ -336,6 +336,8 @@ bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,void nvmet_req_uninit (struct nvmet_req *req);void nvmet_req_execute (struct nvmet_req *req);void nvmet_req_complete (struct nvmet_req *req,u16 status); +int nvmet_req_alloc_sgl (struct nvmet_req *req,struct nvmet_sq *sq); +void nvmet_req_free_sgl (struct nvmet_req *req);void nvmet_cq_setup (struct nvmet_ctrl *ctrl,struct nvmet_cq *cq,u16 qid,u16 size);
drivers/nvme/target/rdma.c +12 -8
@@ -489,7 +489,7 @@ static void nvmet_rdma_release_rsp(struct nvmet_rdma_rsp *rsp) }if (rsp->req.sg != rsp->cmd->inline_sg) -sgl_free (rsp->req.sg); +nvmet_req_free_sgl (&rsp->req);if (unlikely (!list_empty_careful (&queue->rsp_wr_wait_list)))nvmet_rdma_process_wr_wait_list (queue);
@@ -638,24 +638,24 @@ static u16 nvmet_rdma_map_sgl_keyed(struct nvmet_rdma_rsp *rsp, {struct rdma_cm_id *cm_id = rsp->queue->cm_id;u64 addr =le64_to_cpu (sgl->addr); -u32 len =get_unaligned_le24 (sgl->length);u32 key =get_unaligned_le32 (sgl->key);int ret; + rsp->req.transfer_len =get_unaligned_le24 (sgl->length); +/* no data command? */ -if (!len) +if (!rsp->req.transfer_len)return 0 ; - rsp->req.sg =sgl_alloc (len,GFP_KERNEL , &rsp->req.sg_cnt); -if (!rsp->req.sg) -return NVME_SC_INTERNAL ; + ret =nvmet_req_alloc_sgl (&rsp->req, &rsp->queue->nvme_sq); +if (ret <0 ) + goto error_out; ret =rdma_rw_ctx_init (&rsp->rw, cm_id->qp, cm_id->port_num, rsp->req.sg, rsp->req.sg_cnt,0 , addr, key,nvmet_data_dir (&rsp->req));if (ret <0 ) -return NVME_SC_INTERNAL ; - rsp->req.transfer_len += len; + goto error_out; rsp->n_rdma += ret;if (invalidate) {
@@ -664,6 +664,10 @@ static u16 nvmet_rdma_map_sgl_keyed(struct nvmet_rdma_rsp *rsp, }return 0 ; + +error_out: + rsp->req.transfer_len =0 ; +return NVME_SC_INTERNAL ; }static u16 nvmet_rdma_map_sgl (struct nvmet_rdma_rsp *rsp)
Signature · 1 line
2.19.0
Logan Gunthorpe
maintainerPATCH v6 13/13 nvmet: Optionally use PCI P2P memory
In reply to Logan GunthorpeWe create a configfs attribute in each nvme-fabrics target port to enable p2p memory use. When enabled, the port will only then use the p2p memory if a p2p memory device can be found which is behind the same switch hierarchy as the RDMA port and all the block devices in use. If the user enabled it and no devices are found, then the system will silently fall back on using regular memory.
If appropriate, that port will allocate memory for the RDMA buffers for queues from the p2pmem device falling back to system memory should anything fail.
Ideally, we'd want to use an NVME CMB buffer as p2p memory. This would save an extra PCI transfer as the NVME card could just take the data out of it's own memory. However, at this time, only a limited number of cards with CMB buffers seem to be available.
--- drivers/nvme/target/configfs.c | 36 ++++++++ drivers/nvme/target/core.c | 138 +++++++++++++++++++++++++++++- drivers/nvme/target/io-cmd-bdev.c | 3 + drivers/nvme/target/nvmet.h | 13 +++ drivers/nvme/target/rdma.c | 2 + 5 files changed, 191 insertions(+), 1 deletion(-)
drivers/nvme/target/configfs.c +36 -0
@@ -17,6 +17,8 @@#include <linux/slab.h> #include <linux/stat.h> #include <linux/ctype.h> +#include <linux/pci.h> +#include <linux/pci-p2pdma.h> #include "nvmet.h"
@@ -1094,6 +1096,37 @@ static void nvmet_port_release(struct config_item *item)kfree (port); } +#ifdef CONFIG_PCI_P2PDMA +static ssize_t nvmet_p2pmem_show (struct config_item *item,char *page) +{ +struct nvmet_port *port =to_nvmet_port (item); + +return pci_p2pdma_enable_show (page, port->p2p_dev, port->use_p2pmem); +} + +static ssize_t nvmet_p2pmem_store (struct config_item *item, +const char *page,size_t count) +{ +struct nvmet_port *port =to_nvmet_port (item); +struct pci_dev *p2p_dev =NULL ; +bool use_p2pmem; +int error; + + error =pci_p2pdma_enable_store (page, &p2p_dev, &use_p2pmem); +if (error) +return error; + +down_write (&nvmet_config_sem); + port->use_p2pmem = use_p2pmem; +pci_dev_put (port->p2p_dev); + port->p2p_dev = p2p_dev; +up_write (&nvmet_config_sem); + +return count; +} +CONFIGFS_ATTR (nvmet_, p2pmem); +#endif /* CONFIG_PCI_P2PDMA */ +static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_adrfam, &nvmet_attr_addr_treq,
@@ -1101,6 +1134,9 @@ static struct configfs_attribute *nvmet_port_attrs[] = { &nvmet_attr_addr_trsvcid, &nvmet_attr_addr_trtype, &nvmet_attr_param_inline_data_size, +#ifdef CONFIG_PCI_P2PDMA + &nvmet_attr_p2pmem, +#endif NULL, };
drivers/nvme/target/core.c +137 -1
213 lines not on this page · raw message
drivers/nvme/target/io-cmd-bdev.c +3 -0
@@ -78,6 +78,9 @@ static void nvmet_bdev_execute_rw(struct nvmet_req *req) op =REQ_OP_READ ; } +if (is_pci_p2pdma_page (sg_page (req->sg))) + op_flags |=REQ_NOMERGE ; + sector =le64_to_cpu (req->cmd->rw.slba); sector <<= (req->ns->blksize_shift -9 );
drivers/nvme/target/nvmet.h +13 -0
@@ -84,6 +84,11 @@ static inline struct nvmet_ns *to_nvmet_ns(struct config_item *item) return container_of(to_config_group(item), struct nvmet_ns, group); } +static inline struct device *nvmet_ns_dev (struct nvmet_ns *ns) +{ +return disk_to_dev (ns->bdev->bd_disk); +} +struct nvmet_cq {u16 qid;u16 size;
@@ -134,6 +139,8 @@ struct nvmet_port {void *priv;bool enabled;int inline_data_size; +bool use_p2pmem; +struct pci_dev *p2p_dev; };static inline struct nvmet_port *to_nvmet_port (struct config_item *item)
@@ -182,6 +189,9 @@ struct nvmet_ctrl {__le32 *changed_ns_list;u32 nr_changed_ns; +struct pci_dev *p2p_dev; +struct list_head p2p_clients; +char subsysnqn[NVMF_NQN_FIELD_LEN ];char hostnqn[NVMF_NQN_FIELD_LEN ]; };
@@ -294,6 +304,9 @@ struct nvmet_req {void (*execute)(struct nvmet_req *req);const struct nvmet_fabrics_ops *ops; + +struct pci_dev *p2p_dev; +struct device *p2p_client; };extern struct workqueue_struct *buffered_io_wq;
drivers/nvme/target/rdma.c +2 -0
@@ -735,6 +735,8 @@ static void nvmet_rdma_handle_command(struct nvmet_rdma_queue *queue, cmd->send_sge.addr, cmd->send_sge.length, DMA_TO_DEVICE); + cmd->req.p2p_client = &queue->dev->device->dev; +if (!nvmet_req_init (&cmd->req, &queue->nvme_cq, &queue->nvme_sq, &nvmet_rdma_ops))return ;
Signature · 1 line
2.19.0