DMA Mapping P2PDMA Pages
Subsystems (10)
Logan Gunthorpe
maintainerv8 cover DMA Mapping P2PDMA Pages
Hi,
This series is the first 13 patches from my previous Userspace P2P with the CMB series[1]. These patches just do the first part which is to allow passing P2PDMA pages into dma_map_sg() and support it with both dma-direct and dma-iommu. The series ends with dropping the old pci_p2pdma_[un]map_sg() interface.
This will be followed up later with the rest of the series which has more issues to work out.
This series is based on v5.19-rc5. A git branch is available here:
https://github.com/sbates130272/linux-p2pmem/ p2pdma_map_v8
Logan
[1] https://lkml.kernel.org/r/[email protected]
--
Changes since v7 (first half of the series only):
- Rebase onto v5.19-rc5
- Collect reviewed-by tags from Christoph
- Change the first patch to use a dma_flags field in the scatterlist instead of a bit in the page link, per an idea from Robin
- Rework the iommu-dma implementation to store the bus addresses in iommu_dma_map_sg() and only copy them in __finalise_sg. This allows the code to use the same pci_p2pdma_map_segment() helper that the dma-direct code uses and we can drop the two specialized helpers. (Per Robin)
- Instead of using a zero length segment which __iommu_map_sg() ignores, explicitly skip segments marked with as a dma bus address. (per Robin)
- Rework the iommu_dma_unmap_sg() function per a suggestion from Robin.
Changes since v6:
- Rebase onto v5.19-rc1
- Rework how the pages are stored in the VMA per Jason's suggestion
Changes since v5:
- Rebased onto v5.18-rc1 which includes Christophs cleanup to free_zone_device_page() (similar to Ralph's patch).
- Fix bug with concurrent first calls to pci_p2pdma_vma_fault() that caused a double allocation and lost p2p memory. Noticed by Andrew Maier.
- Collected a Reviewed-by tag from Chaitanya.
- Numerous minor fixes to commit messages
Changes since v4:
- Rebase onto v5.17-rc1.
- Included Ralph Cambell's patches which removes the ZONE_DEVICE page reference count offset. This is just to demonstrate that this series is compatible with that direction.
- Added a comment in pci_p2pdma_map_sg_attrs(), per Chaitanya and included his Reviewed-by tags.
- Patch 1 in the last series which cleaned up scatterlist.h has been upstreamed.
- Dropped NEED_SG_DMA_BUS_ADDR_FLAG seeing depends on doesn't work with selected symbols, per Christoph.
- Switched iov_iter_get_pages_[alloc_]flags to be exported with EXPORT_SYMBOL_GPL, per Christoph.
- Renamed zone_device_pages_are_mergeable() to zone_device_pages_have_same_pgmap(), per Christoph.
- Renamed .mmap_file_open operation in nvme_ctrl_ops to cdev_file_open(), per Christoph.
Changes since v3:
- Add some comment and commit message cleanups I had missed for v3, also moved the prototypes for some of the p2pdma helpers to dma-map-ops.h (which I missed in v3 and was suggested in v2).
- Add separate cleanup patch for scatterlist.h and change the macros to functions. (Suggested by Chaitanya and Jason, respectively)
- Rename sg_dma_mark_pci_p2pdma() and sg_is_dma_pci_p2pdma() to sg_dma_mark_bus_address() and sg_is_dma_bus_address() which is a more generic name (As requested by Jason)
- Fixes to some comments and commit messages as suggested by Bjorn and Jason.
- Ensure swiotlb is not used with P2PDMA pages. (Per Jason)
- The sgtable coversion in RDMA was split out and sent upstream separately, the new patch is only the removal. (Per Jason)
- Moved the FOLL_PCI_P2PDMA check outside of get_dev_pagemap() as Jason suggested this will be removed in the near term.
- Add two patches to ensure that zone device pages with different pgmaps are never merged in the block layer or sg_alloc_append_table_from_pages() (Per Jason)
- Ensure synchronize_rcu() or call_rcu() is used before returning pages to the genalloc. (Jason pointed out that pages are not gauranteed to be unused in all architectures until at least after an RCU grace period, and that synchronize_rcu() was likely too slow to use in the vma close operation.
- Collected Acks and Reviews by Bjorn, Jason and Max.
--
Logan Gunthorpe (13):
lib/scatterlist: add flag for indicating P2PDMA segments in an SGL PCI/P2PDMA: Attempt to set map_type if it has not been set PCI/P2PDMA: Introduce helpers for dma_map_sg implementations
dma-mapping: allow EREMOTEIO return code for P2PDMA transfers
dma-direct: support PCI P2PDMA pages in dma-direct map_sg
dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support
iommu: Explicitly skip bus address marked segments in
__iommu_map_sg()
iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg
nvme-pci: check DMA ops when indicating support for PCI P2PDMA
nvme-pci: convert to using dma_map_sgtable() RDMA/core: introduce ib_dma_pci_p2p_dma_supported() RDMA/rw: drop pci_p2pdma_[un]map_sg()
PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg()
drivers/infiniband/core/rw.c | 45 ++++------------ drivers/iommu/dma-iommu.c | 99 +++++++++++++++++++++++++++++++----- drivers/iommu/iommu.c | 4 ++ drivers/nvme/host/core.c | 3 +- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 81 ++++++++++++++--------------- drivers/nvme/target/rdma.c | 2 +- drivers/pci/Kconfig | 5 ++ drivers/pci/p2pdma.c | 93 +++++++++++---------------------- include/linux/dma-map-ops.h | 63 +++++++++++++++++++++++ include/linux/dma-mapping.h | 5 ++ include/linux/pci-p2pdma.h | 27 ---------- include/linux/scatterlist.h | 69 +++++++++++++++++++++++++ include/rdma/ib_verbs.h | 11 ++++ kernel/dma/direct.c | 43 +++++++++++++--- kernel/dma/direct.h | 8 ++- kernel/dma/mapping.c | 23 ++++++++- 17 files changed, 390 insertions(+), 193 deletions(-)
base-commit: 88084a3df1672e131ddc1b4e39eeacfd39864acf --
2.30.2
Logan Gunthorpe
maintainerPATCH v8 1/13 lib/scatterlist: add flag for indicating P2PDMA segments in an SGL
Introduce a dma_flags field in struct scatterlist. These flags will be used by dma_[un]map_sg_p2pdma() to determine when a given SGL segments dma_address points to a PCI bus address. dma_unmap_sg_p2pdma() will need to perform different cleanup when a segment is marked as a bus address.
The dma_flags field will fit in the existing padding on 64BIT systems (assuming CONFIG_NEED_SG_DMA_LENGTH is also set).
The new bit will only be used when CONFIG_PCI_P2PDMA is set; this means PCI P2PDMA will require CONFIG_64BIT. This should be acceptable as the majority of P2PDMA use cases are restricted to newer root complexes and roughly require the extra address space for memory BARs used in the transactions.
--- drivers/pci/Kconfig | 5 +++ include/linux/scatterlist.h | 69 +++++++++++++++++++++++++++++++++++++ 2 files changed, 74 insertions(+)
drivers/pci/Kconfig +5 -0
@@ -164,6 +164,11 @@ config PCI_PASID config PCI_P2PDMA bool "PCI peer-to-peer transfer support" depends on ZONE_DEVICE + # + # The need for the scatterlist DMA bus address flag means PCI P2PDMA + # requires 64bit + # + depends on 64BIT select GENERIC_ALLOCATOR help Enableѕ drivers to do PCI peer-to-peer transactions to and from
include/linux/scatterlist.h +69 -0
81 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v8 01/13] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL
Do we need the dma_ prefix for the flags field? That's just what it is used for now.
Otherwise looks good:
Logan Gunthorpe
maintainerPATCH v8 2/13 PCI/P2PDMA: Attempt to set map_type if it has not been set
In reply to Logan GunthorpeAttempt to find the mapping type for P2PDMA pages on the first DMA map attempt if it has not been done ahead of time.
Previously, the mapping type was expected to be calculated ahead of time, but if pages are to come from userspace then there's no way to ensure the path was checked ahead of time.
This change will calculate the mapping type if it hasn't pre-calculated so it is no longer invalid to call pci_p2pdma_map_sg() before the mapping type is calculated, so drop the WARN_ON when that is the case.
--- drivers/pci/p2pdma.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
drivers/pci/p2pdma.c +6 -1
@@ -854,6 +854,7 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap,struct pci_dev *provider =to_p2p_pgmap (pgmap)->provider;struct pci_dev *client;struct pci_p2pdma *p2pdma; +int dist;if (!provider->p2pdma)return PCI_P2PDMA_MAP_NOT_SUPPORTED ;
@@ -870,6 +871,10 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap, type =xa_to_value (xa_load (&p2pdma->map_types,map_types_idx (client)));rcu_read_unlock (); + +if (type ==PCI_P2PDMA_MAP_UNKNOWN ) +return calc_map_type_and_dist (provider, client, &dist, true); +return type; }
@@ -912,7 +917,7 @@ int pci_p2pdma_map_sg_attrs(struct device *dev, struct scatterlist *sg,case PCI_P2PDMA_MAP_BUS_ADDR :return __pci_p2pdma_map_sg (p2p_pgmap, dev, sg, nents);default : -WARN_ON_ONCE (1 ); +/* Mapping is not Supported */ return 0 ; } }
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 3/13 PCI/P2PDMA: Introduce helpers for dma_map_sg implementations
In reply to Logan GunthorpeAdd pci_p2pdma_map_segment() as a helper for dma_map_sg() implementations. It takes an scatterlist segment that must point to a pci_p2pdma struct page and will map it if the mapping requires a bus address.
The return value indicates whether the mapping required a bus address or whether the caller still needs to map the segment normally. If the segment should not be mapped, -EREMOTEIO is returned.
This helper uses a state structure to track the changes to the pgmap across calls and avoid needing to lookup into the xarray for every page.
The prototype for the helper is added to dma-map-ops.h as it is only useful to dma map implementations and don't need to pollute the public pci-p2pdma header.
--- drivers/pci/p2pdma.c | 44 +++++++++++++++++++++++++----- include/linux/dma-map-ops.h | 53 +++++++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+), 7 deletions(-)
drivers/pci/p2pdma.c +37 -7
@@ -10,6 +10,7 @@#define pr_fmt (fmt) "pci-p2pdma: " fmt#include <linux/ctype.h> +#include <linux/dma-map-ops.h> #include <linux/pci-p2pdma.h> #include <linux/module.h> #include <linux/slab.h>
@@ -20,13 +21,6 @@#include <linux/seq_buf.h> #include <linux/xarray.h> -enum pci_p2pdma_map_type { -PCI_P2PDMA_MAP_UNKNOWN =0 , -PCI_P2PDMA_MAP_NOT_SUPPORTED , -PCI_P2PDMA_MAP_BUS_ADDR , -PCI_P2PDMA_MAP_THRU_HOST_BRIDGE , -}; -struct pci_p2pdma {struct gen_pool *pool;bool p2pmem_published;
@@ -944,6 +938,42 @@ void pci_p2pdma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, }EXPORT_SYMBOL_GPL (pci_p2pdma_unmap_sg_attrs); +/** +* pci_p2pdma_map_segment - map an sg segment determining the mapping type +* @state: State structure that should be declared outside of the for_each_sg() +* loop and initialized to zero. +* @dev: DMA device that's doing the mapping operation +* @sg: scatterlist segment to map +* +* This is a helper to be used by non-IOMMU dma_map_sg() implementations where +* the sg segment is the same for the page_link and the dma_address. +* +* Attempt to map a single segment in an SGL with the PCI bus address. +* The segment must point to a PCI P2PDMA page and thus must be +* wrapped in a is_pci_p2pdma_page(sg_page(sg)) check. +* +* Returns the type of mapping used and maps the page if the type is +* PCI_P2PDMA_MAP_BUS_ADDR. +*/ +enum pci_p2pdma_map_type +pci_p2pdma_map_segment (struct pci_p2pdma_map_state *state,struct device *dev, +struct scatterlist *sg) +{ +if (state->pgmap !=sg_page (sg)->pgmap) { + state->pgmap =sg_page (sg)->pgmap; + state->map =pci_p2pdma_map_type (state->pgmap, dev); + state->bus_off =to_p2p_pgmap (state->pgmap)->bus_offset; + } + +if (state->map ==PCI_P2PDMA_MAP_BUS_ADDR ) { + sg->dma_address =sg_phys (sg) + state->bus_off; +sg_dma_len (sg) = sg->length; +sg_dma_mark_bus_address (sg); + } + +return state->map; +} +/** * pci_p2pdma_enable_store - parse a configfs/sysfs attribute store * to enable p2pdma
include/linux/dma-map-ops.h +53 -0
57 lines not on this page · raw message
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 4/13 dma-mapping: allow EREMOTEIO return code for P2PDMA transfers
In reply to Logan GunthorpeAdd EREMOTEIO error return to dma_map_sgtable() which will be used by .map_sg() implementations that detect P2PDMA pages that the underlying DMA device cannot access.
--- kernel/dma/mapping.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
kernel/dma/mapping.c +4 -1
@@ -197,7 +197,7 @@ static int __dma_map_sg_attrs(struct device *dev, struct scatterlist *sg,if (ents >0 )debug_dma_map_sg (dev, sg, nents, ents, dir, attrs);else if (WARN_ON_ONCE (ents != -EINVAL && ents != -ENOMEM && - ents != -EIO )) + ents != -EIO && ents != -EREMOTEIO ))return -EIO ;return ents;
@@ -255,6 +255,9 @@ EXPORT_SYMBOL(dma_map_sg_attrs);* complete the mapping. Should succeed if retried later. * -EIO Legacy error code with an unknown meaning. eg. this is * returned if a lower level call returned DMA_MAPPING_ERROR. +* -EREMOTEIO The DMA device cannot access P2PDMA memory specified in +* the sg_table. This will not succeed if retried. +* */ int dma_map_sgtable (struct device *dev,struct sg_table *sgt,enum dma_data_direction dir,unsigned long attrs)
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 5/13 dma-direct: support PCI P2PDMA pages in dma-direct map_sg
In reply to Logan GunthorpeAdd PCI P2PDMA support for dma_direct_map_sg() so that it can map PCI P2PDMA pages directly without a hack in the callers. This allows for heterogeneous SGLs that contain both P2PDMA and regular pages.
A P2PDMA page may have three possible outcomes when being mapped:
1) If the data path between the two devices doesn't go through the root port, then it should be mapped with a PCI bus address
2) If the data path goes through the host bridge, it should be mapped normally, as though it were a CPU physical address
3) It is not possible for the two devices to communicate and thus the mapping operation should fail (and it will return -EREMOTEIO).
SGL segments that contain PCI bus addresses are marked with sg_dma_mark_pci_p2pdma() and are ignored when unmapped.
P2PDMA mappings are also failed if swiotlb needs to be used on the mapping.
--- kernel/dma/direct.c | 43 +++++++++++++++++++++++++++++++++++++------ kernel/dma/direct.h | 8 +++++++- 2 files changed, 44 insertions(+), 7 deletions(-)
kernel/dma/direct.c +37 -6
@@ -453,29 +453,60 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } +/* +* Unmaps segments, except for ones marked as pci_p2pdma which do not +* require any further action as they contain a bus address. +*/ void dma_direct_unmap_sg (struct device *dev,struct scatterlist *sgl,int nents,enum dma_data_direction dir,unsigned long attrs) {struct scatterlist *sg;int i; -for_each_sg (sgl, sg, nents, i) -dma_direct_unmap_page (dev, sg->dma_address,sg_dma_len (sg), dir, - attrs); +for_each_sg (sgl, sg, nents, i) { +if (sg_is_dma_bus_address (sg)) +sg_dma_unmark_bus_address (sg); +else +dma_direct_unmap_page (dev, sg->dma_address, +sg_dma_len (sg), dir, attrs); + } }#endif int dma_direct_map_sg (struct device *dev,struct scatterlist *sgl,int nents,enum dma_data_direction dir,unsigned long attrs) { -int i; +struct pci_p2pdma_map_state p2pdma_state = {}; +enum pci_p2pdma_map_type map;struct scatterlist *sg; +int i, ret;for_each_sg (sgl, sg, nents, i) { +if (is_pci_p2pdma_page (sg_page (sg))) { + map =pci_p2pdma_map_segment (&p2pdma_state, dev, sg); +switch (map) { +case PCI_P2PDMA_MAP_BUS_ADDR : +continue ; +case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE : +/* +* Any P2P mapping that traverses the PCI +* host bridge must be mapped with CPU physical +* address and not PCI bus addresses. This is +* done with dma_direct_map_page() below. +*/ +break ; +default : + ret = -EREMOTEIO ; + goto out_unmap; + } + } + sg->dma_address =dma_direct_map_page (dev,sg_page (sg), sg->offset, sg->length, dir, attrs); -if (sg->dma_address ==DMA_MAPPING_ERROR ) +if (sg->dma_address ==DMA_MAPPING_ERROR ) { + ret = -EIO ; goto out_unmap; + }sg_dma_len (sg) = sg->length; }
@@ -483,7 +514,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, out_unmap:dma_direct_unmap_sg (dev, sgl, i, dir, attrs |DMA_ATTR_SKIP_CPU_SYNC ); -return -EIO ; +return ret; } dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr,
kernel/dma/direct.h +7 -1
@@ -8,6 +8,7 @@#define _KERNEL_DMA_DIRECT_H#include <linux/dma-direct.h> +#include <linux/memremap.h> int dma_direct_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size,
@@ -87,10 +88,15 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev,phys_addr_t phys =page_to_phys (page) + offset;dma_addr_t dma_addr =phys_to_dma (dev, phys); -if (is_swiotlb_force_bounce (dev)) +if (is_swiotlb_force_bounce (dev)) { +if (is_pci_p2pdma_page (page)) +return DMA_MAPPING_ERROR ;return swiotlb_map (dev, phys, size, dir, attrs); + }if (unlikely (!dma_capable (dev, dma_addr, size, true))) { +if (is_pci_p2pdma_page (page)) +return DMA_MAPPING_ERROR ;if (is_swiotlb_active (dev))return swiotlb_map (dev, phys, size, dir, attrs);
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 6/13 dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support
In reply to Logan GunthorpeAdd a flags member to the dma_map_ops structure with one flag to indicate support for PCI P2PDMA.
Also, add a helper to check if a device supports PCI P2PDMA.
--- include/linux/dma-map-ops.h | 10 ++++++++++ include/linux/dma-mapping.h | 5 +++++ kernel/dma/mapping.c | 18 ++++++++++++++++++ 3 files changed, 33 insertions(+)
include/linux/dma-map-ops.h +10 -0
@@ -11,7 +11,17 @@struct cma ; +/* +* Values for struct dma_map_ops.flags: +* +* DMA_F_PCI_P2PDMA_SUPPORTED: Indicates the dma_map_ops implementation can +* handle PCI P2PDMA pages in the map_sg/unmap_sg operation. +*/ +#define DMA_F_PCI_P2PDMA_SUPPORTED (1 << 0) +struct dma_map_ops { +unsigned int flags; +void *(*alloc)(struct device *dev,size_t size,dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs);
include/linux/dma-mapping.h +5 -0
@@ -140,6 +140,7 @@ int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma, unsigned long attrs);bool dma_can_mmap (struct device *dev);int dma_supported (struct device *dev,u64 mask); +bool dma_pci_p2pdma_supported (struct device *dev);int dma_set_mask (struct device *dev,u64 mask);int dma_set_coherent_mask (struct device *dev,u64 mask);u64 dma_get_required_mask (struct device *dev);
@@ -250,6 +251,10 @@ static inline int dma_supported(struct device *dev, u64 mask) {return 0 ; } +static inline bool dma_pci_p2pdma_supported (struct device *dev) +{ +return false; +}static inline int dma_set_mask (struct device *dev,u64 mask) {return -EIO ;
kernel/dma/mapping.c +18 -0
@@ -723,6 +723,24 @@ int dma_supported(struct device *dev, u64 mask) }EXPORT_SYMBOL (dma_supported); +bool dma_pci_p2pdma_supported (struct device *dev) +{ +const struct dma_map_ops *ops =get_dma_ops (dev); + +/* if ops is not set, dma direct will be used which supports P2PDMA */ +if (!ops) +return true; + +/* +* Note: dma_ops_bypass is not checked here because P2PDMA should +* not be used with dma mapping ops that do not have support even +* if the specific device is bypassing them. +*/ + +return ops->flags &DMA_F_PCI_P2PDMA_SUPPORTED ; +} +EXPORT_SYMBOL_GPL (dma_pci_p2pdma_supported); +#ifdef CONFIG_ARCH_HAS_DMA_SET_MASK void arch_dma_set_mask (struct device *dev,u64 mask);#else
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 7/13 iommu: Explicitly skip bus address marked segments in __iommu_map_sg()
In reply to Logan GunthorpeIn order to support PCI P2PDMA mappings with dma-iommu, explicitly skip any segments marked with sg_dma_mark_bus_address() in __iommu_map_sg().
These segments should not be mapped into the IOVA and will be handled separately in as subsequent patch for dma-iommu.
--- drivers/iommu/iommu.c | 4 ++++ 1 file changed, 4 insertions(+)
drivers/iommu/iommu.c +4 -0
@@ -2457,6 +2457,9 @@ static ssize_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, len =0 ; } +if (sg_is_dma_bus_address (sg)) + goto next; +if (len) { len += sg->length; }else {
@@ -2464,6 +2467,7 @@ static ssize_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, start = s_phys; } +next:if (++i < nents) sg =sg_next (sg); }
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 8/13 iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg
In reply to Logan GunthorpeCall pci_p2pdma_map_segment() when a PCI P2PDMA page is seen so the bus address is set in the dma address and the segment is marked with sg_dma_mark_bus_address(). iommu_map_sg() will then skip these segments. Then, in __finalise_sg(), copy the dma address from the input segment to the output segment. __invalidate_sg() must also learn to skip these segments.
A P2PDMA page may have three possible outcomes when being mapped:
1) If the data path between the two devices doesn't go through the root port, then it should be mapped with a PCI bus address
2) If the data path goes through the host bridge, it should be mapped normally with an IOMMU IOVA.
3) It is not possible for the two devices to communicate and thus the mapping operation should fail (and it will return -EREMOTEIO).
Similar to dma-direct, the sg_dma_mark_pci_p2pdma() flag is used to indicate bus address segments. On unmap, P2PDMA segments are skipped over when determining the start and end IOVA addresses.
With this change, the flags variable in the dma_map_ops is set to DMA_F_PCI_P2PDMA_SUPPORTED to indicate support for P2PDMA pages.
--- drivers/iommu/dma-iommu.c | 99 +++++++++++++++++++++++++++++++++------ 1 file changed, 85 insertions(+), 14 deletions(-)
drivers/iommu/dma-iommu.c +85 -14
@@ -21,6 +21,7 @@#include <linux/iova.h> #include <linux/irq.h> #include <linux/list_sort.h> +#include <linux/memremap.h> #include <linux/mm.h> #include <linux/mutex.h> #include <linux/pci.h>
@@ -1053,15 +1054,30 @@ static int __finalise_sg(struct device *dev, struct scatterlist *sg, int nents,for_each_sg (sg, s, nents, i) {/* Restore this segment's original unaligned fields first */ +dma_addr_t s_dma_addr =sg_dma_address (s);unsigned int s_iova_off =sg_dma_address (s);unsigned int s_length =sg_dma_len (s);unsigned int s_iova_len = s->length; - s->offset += s_iova_off; - s->length = s_length;sg_dma_address (s) =DMA_MAPPING_ERROR ;sg_dma_len (s) =0 ; +if (sg_is_dma_bus_address (s)) { +if (i >0 ) + cur =sg_next (cur); + +sg_dma_unmark_bus_address (s); +sg_dma_address (cur) = s_dma_addr; +sg_dma_len (cur) = s_length; +sg_dma_mark_bus_address (cur); + count++; + cur_len =0 ; +continue ; + } + + s->offset += s_iova_off; + s->length = s_length; +/* * Now fill in the real DMA data. If... * - there is a valid output segment to append to
@@ -1102,10 +1118,14 @@ static void __invalidate_sg(struct scatterlist *sg, int nents)int i;for_each_sg (sg, s, nents, i) { -if (sg_dma_address (s) !=DMA_MAPPING_ERROR ) - s->offset +=sg_dma_address (s); -if (sg_dma_len (s)) - s->length =sg_dma_len (s); +if (sg_is_dma_bus_address (s)) { +sg_dma_unmark_bus_address (s); + }else { +if (sg_dma_address (s) !=DMA_MAPPING_ERROR ) + s->offset +=sg_dma_address (s); +if (sg_dma_len (s)) + s->length =sg_dma_len (s); + }sg_dma_address (s) =DMA_MAPPING_ERROR ;sg_dma_len (s) =0 ; }
@@ -1158,6 +1178,8 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,struct iova_domain *iovad = &cookie->iovad;struct scatterlist *s, *prev =NULL ;int prot =dma_info_to_prot (dir,dev_is_dma_coherent (dev), attrs); +struct pci_p2pdma_map_state p2pdma_state = {}; +enum pci_p2pdma_map_type map;dma_addr_t iova;size_t iova_len =0 ;unsigned long mask =dma_get_seg_boundary (dev);
@@ -1187,6 +1209,30 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,size_t s_length = s->length;size_t pad_len = (mask - iova_len +1 ) & mask; +if (is_pci_p2pdma_page (sg_page (s))) { + map =pci_p2pdma_map_segment (&p2pdma_state, dev, s); +switch (map) { +case PCI_P2PDMA_MAP_BUS_ADDR : +/* +* iommu_map_sg() will skip this segment as +* it is marked as a bus address, +* __finalise_sg() will copy the dma address +* into the output segment. +*/ +continue ; +case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE : +/* +* Mapping through host bridge should be +* mapped with regular IOVAs, thus we +* do nothing here and continue below. +*/ +break ; +default : + ret = -EREMOTEIO ; + goto out_restore_sg; + } + } +sg_dma_address (s) = s_iova_off;sg_dma_len (s) = s_length; s->offset -= s_iova_off;
@@ -1215,6 +1261,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, prev = s; } +if (!iova_len) +return __finalise_sg (dev, sg, nents,0 ); + iova =iommu_dma_alloc_iova (domain, iova_len,dma_get_mask (dev), dev);if (!iova) { ret = -ENOMEM ;
@@ -1236,7 +1285,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, out_restore_sg:__invalidate_sg (sg, nents); out: -if (ret != -ENOMEM ) +if (ret != -ENOMEM && ret != -EREMOTEIO )return -EINVAL ;return ret; }
@@ -1244,7 +1293,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg,static void iommu_dma_unmap_sg (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs) { -dma_addr_t start, end; +dma_addr_t end =0 , start;struct scatterlist *tmp;int i;
@@ -1258,16 +1307,37 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg,/* * The scatterlist segments are mapped into a single -* contiguous IOVA allocation, so this is incredibly easy. +* contiguous IOVA allocation, the start and end points +* just have to be determined. */ - start =sg_dma_address (sg); -for_each_sg (sg_next (sg), tmp, nents -1 , i) { +for_each_sg (sg, tmp, nents, i) { +if (sg_is_dma_bus_address (tmp)) { +sg_dma_unmark_bus_address (tmp); +continue ; + } + +if (sg_dma_len (tmp) ==0 ) +break ; + + start =sg_dma_address (tmp); +break ; + } + + nents -= i; +for_each_sg (tmp, tmp, nents, i) { +if (sg_is_dma_bus_address (tmp)) { +sg_dma_unmark_bus_address (tmp); +continue ; + } +if (sg_dma_len (tmp) ==0 )break ; - sg = tmp; + + end =sg_dma_address (tmp) +sg_dma_len (tmp); } - end =sg_dma_address (sg) +sg_dma_len (sg); -__iommu_dma_unmap (dev, start, end - start); + +if (end) +__iommu_dma_unmap (dev, start, end - start); } static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys,
@@ -1460,6 +1530,7 @@ static unsigned long iommu_dma_get_merge_boundary(struct device *dev) }static const struct dma_map_ops iommu_dma_ops = { + .flags =DMA_F_PCI_P2PDMA_SUPPORTED , .alloc = iommu_dma_alloc, .free = iommu_dma_free, .alloc_pages = dma_common_alloc_pages,
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 9/13 nvme-pci: check DMA ops when indicating support for PCI P2PDMA
In reply to Logan GunthorpeIntroduce a supports_pci_p2pdma() operation in nvme_ctrl_ops to replace the fixed NVME_F_PCI_P2PDMA flag such that the dma_map_ops flags can be checked for PCI P2PDMA support.
--- drivers/nvme/host/core.c | 3 ++- drivers/nvme/host/nvme.h | 2 +- drivers/nvme/host/pci.c | 12 +++++++++--- 3 files changed, 12 insertions(+), 5 deletions(-)
drivers/nvme/host/core.c +2 -1
@@ -3996,7 +3996,8 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, unsigned nsid,blk_queue_flag_set (QUEUE_FLAG_STABLE_WRITES , ns->queue);blk_queue_flag_set (QUEUE_FLAG_NONROT , ns->queue); -if (ctrl->ops->flags &NVME_F_PCI_P2PDMA ) +if (ctrl->ops->supports_pci_p2pdma && + ctrl->ops->supports_pci_p2pdma (ctrl))blk_queue_flag_set (QUEUE_FLAG_PCI_P2PDMA , ns->queue); ns->ctrl = ctrl;
drivers/nvme/host/nvme.h +1 -1
@@ -495,7 +495,6 @@ struct nvme_ctrl_ops {unsigned int flags;#define NVME_F_FABRICS (1 << 0)#define NVME_F_METADATA_SUPPORTED (1 << 1) -#define NVME_F_PCI_P2PDMA (1 << 2)int (*reg_read32)(struct nvme_ctrl *ctrl,u32 off,u32 *val);int (*reg_write32)(struct nvme_ctrl *ctrl,u32 off,u32 val);int (*reg_read64)(struct nvme_ctrl *ctrl,u32 off,u64 *val);
@@ -505,6 +504,7 @@ struct nvme_ctrl_ops {void (*stop_ctrl)(struct nvme_ctrl *ctrl);int (*get_address)(struct nvme_ctrl *ctrl,char *buf,int size);void (*print_device_info)(struct nvme_ctrl *ctrl); +bool (*supports_pci_p2pdma)(struct nvme_ctrl *ctrl); };/*
drivers/nvme/host/pci.c +9 -3
@@ -2984,7 +2984,6 @@ static int nvme_pci_get_address(struct nvme_ctrl *ctrl, char *buf, int size)return snprintf (buf, size,"%s\n" ,dev_name (&pdev->dev)); } -static void nvme_pci_print_device_info (struct nvme_ctrl *ctrl) {struct pci_dev *pdev =to_pci_dev (to_nvme_dev (ctrl)->dev);
@@ -2999,11 +2998,17 @@ static void nvme_pci_print_device_info(struct nvme_ctrl *ctrl) subsys->firmware_rev); } +static bool nvme_pci_supports_pci_p2pdma (struct nvme_ctrl *ctrl) +{ +struct nvme_dev *dev =to_nvme_dev (ctrl); + +return dma_pci_p2pdma_supported (dev->dev); +} +static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .name ="pcie" , .module =THIS_MODULE , - .flags =NVME_F_METADATA_SUPPORTED | -NVME_F_PCI_P2PDMA , + .flags =NVME_F_METADATA_SUPPORTED , .reg_read32 = nvme_pci_reg_read32, .reg_write32 = nvme_pci_reg_write32, .reg_read64 = nvme_pci_reg_read64,
@@ -3011,6 +3016,7 @@ static const struct nvme_ctrl_ops nvme_pci_ctrl_ops = { .submit_async_event = nvme_pci_submit_async_event, .get_address = nvme_pci_get_address, .print_device_info = nvme_pci_print_device_info, + .supports_pci_p2pdma = nvme_pci_supports_pci_p2pdma, };static int nvme_dev_map (struct nvme_dev *dev)
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 10/13 nvme-pci: convert to using dma_map_sgtable()
In reply to Logan GunthorpeThe dma_map operations now support P2PDMA pages directly. So remove the calls to pci_p2pdma_[un]map_sg_attrs() and replace them with calls to dma_map_sgtable().
dma_map_sgtable() returns more complete error codes than dma_map_sg() and allows differentiating EREMOTEIO errors in case an unsupported P2PDMA transfer is requested. When this happens, return BLK_STS_TARGET so the request isn't retried.
--- drivers/nvme/host/pci.c | 69 +++++++++++++++++------------------------ 1 file changed, 29 insertions(+), 40 deletions(-)
drivers/nvme/host/pci.c +29 -40
@@ -230,11 +230,10 @@ struct nvme_iod {bool use_sgl;int aborted;int npages;/* In the PRP list. 0 means small pool in use */ -int nents;/* Used in scatterlist */ dma_addr_t first_dma;unsigned int dma_len;/* length of single DMA segment mapping */ dma_addr_t meta_dma; -struct scatterlist *sg; +struct sg_table sgt; };static inline unsigned int nvme_dbbuf_size (struct nvme_dev *dev)
@@ -524,7 +523,7 @@ static void nvme_commit_rqs(struct blk_mq_hw_ctx *hctx)static void **nvme_pci_iod_list (struct request *req) {struct nvme_iod *iod =blk_mq_rq_to_pdu (req); -return (void **)(iod->sg +blk_rq_nr_phys_segments (req)); +return (void **)(iod->sgt.sgl +blk_rq_nr_phys_segments (req)); }static inline bool nvme_pci_use_sgls (struct nvme_dev *dev,struct request *req)
@@ -576,17 +575,6 @@ static void nvme_free_sgls(struct nvme_dev *dev, struct request *req) } } -static void nvme_unmap_sg (struct nvme_dev *dev,struct request *req) -{ -struct nvme_iod *iod =blk_mq_rq_to_pdu (req); - -if (is_pci_p2pdma_page (sg_page (iod->sg))) -pci_p2pdma_unmap_sg (dev->dev, iod->sg, iod->nents, -rq_dma_dir (req)); -else -dma_unmap_sg (dev->dev, iod->sg, iod->nents,rq_dma_dir (req)); -} -static void nvme_unmap_data (struct nvme_dev *dev,struct request *req) {struct nvme_iod *iod =blk_mq_rq_to_pdu (req);
@@ -597,9 +585,10 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req)return ; } -WARN_ON_ONCE (!iod->nents); +WARN_ON_ONCE (!iod->sgt.nents); + +dma_unmap_sgtable (dev->dev, &iod->sgt,rq_dma_dir (req),0 ); -nvme_unmap_sg (dev, req);if (iod->npages ==0 )dma_pool_free (dev->prp_small_pool,nvme_pci_iod_list (req)[0 ], iod->first_dma);
@@ -607,7 +596,7 @@ static void nvme_unmap_data(struct nvme_dev *dev, struct request *req)nvme_free_sgls (dev, req); elsenvme_free_prps (dev ,req ); -mempool_free (iod->sg, dev->iod_mempool); +mempool_free (iod->sgt.sgl, dev->iod_mempool); }static void nvme_print_sgl (struct scatterlist *sgl,int nents)
@@ -630,7 +619,7 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev,struct nvme_iod *iod =blk_mq_rq_to_pdu (req);struct dma_pool *pool;int length =blk_rq_payload_bytes (req); -struct scatterlist *sg = iod->sg; +struct scatterlist *sg = iod->sgt.sgl;int dma_len =sg_dma_len (sg);u64 dma_addr =sg_dma_address (sg);int offset = dma_addr & (NVME_CTRL_PAGE_SIZE -1 );
@@ -703,16 +692,16 @@ static blk_status_t nvme_pci_setup_prps(struct nvme_dev *dev, dma_len =sg_dma_len (sg); } done: - cmnd->dptr.prp1 =cpu_to_le64 (sg_dma_address (iod->sg)); + cmnd->dptr.prp1 =cpu_to_le64 (sg_dma_address (iod->sgt.sgl)); cmnd->dptr.prp2 =cpu_to_le64 (iod->first_dma);return BLK_STS_OK ; free_prps:nvme_free_prps (dev, req);return BLK_STS_RESOURCE ; bad_sgl: -WARN (DO_ONCE (nvme_print_sgl, iod->sg, iod->nents), +WARN (DO_ONCE (nvme_print_sgl, iod->sgt.sgl, iod->sgt.nents),"Invalid SGL for payload:%d nents:%d\n" , -blk_rq_payload_bytes (req), iod->nents); +blk_rq_payload_bytes (req), iod->sgt.nents);return BLK_STS_IOERR ; }
@@ -738,12 +727,13 @@ static void nvme_pci_sgl_set_seg(struct nvme_sgl_desc *sge, }static blk_status_t nvme_pci_setup_sgls (struct nvme_dev *dev, -struct request *req,struct nvme_rw_command *cmd,int entries) +struct request *req,struct nvme_rw_command *cmd) {struct nvme_iod *iod =blk_mq_rq_to_pdu (req);struct dma_pool *pool;struct nvme_sgl_desc *sg_list; -struct scatterlist *sg = iod->sg; +struct scatterlist *sg = iod->sgt.sgl; +unsigned int entries = iod->sgt.nents;dma_addr_t sgl_dma;int i =0 ;
@@ -841,7 +831,7 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, {struct nvme_iod *iod =blk_mq_rq_to_pdu (req);blk_status_t ret =BLK_STS_RESOURCE ; -int nr_mapped; +int rc;if (blk_rq_nr_phys_segments (req) ==1 ) {struct bio_vec bv =req_bvec (req);
@@ -859,26 +849,25 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req, } iod->dma_len =0 ; - iod->sg =mempool_alloc (dev->iod_mempool,GFP_ATOMIC ); -if (!iod->sg) + iod->sgt.sgl =mempool_alloc (dev->iod_mempool,GFP_ATOMIC ); +if (!iod->sgt.sgl)return BLK_STS_RESOURCE ; -sg_init_table (iod->sg,blk_rq_nr_phys_segments (req)); - iod->nents =blk_rq_map_sg (req->q, req, iod->sg); -if (!iod->nents) +sg_init_table (iod->sgt.sgl,blk_rq_nr_phys_segments (req)); + iod->sgt.orig_nents =blk_rq_map_sg (req->q, req, iod->sgt.sgl); +if (!iod->sgt.orig_nents) goto out_free_sg; -if (is_pci_p2pdma_page (sg_page (iod->sg))) - nr_mapped =pci_p2pdma_map_sg_attrs (dev->dev, iod->sg, - iod->nents,rq_dma_dir (req),DMA_ATTR_NO_WARN ); -else - nr_mapped =dma_map_sg_attrs (dev->dev, iod->sg, iod->nents, -rq_dma_dir (req),DMA_ATTR_NO_WARN ); -if (!nr_mapped) + rc =dma_map_sgtable (dev->dev, &iod->sgt,rq_dma_dir (req), +DMA_ATTR_NO_WARN ); +if (rc) { +if (rc == -EREMOTEIO ) + ret =BLK_STS_TARGET ; goto out_free_sg; + } iod->use_sgl =nvme_pci_use_sgls (dev, req);if (iod->use_sgl) - ret =nvme_pci_setup_sgls (dev, req, &cmnd->rw, nr_mapped); + ret =nvme_pci_setup_sgls (dev, req, &cmnd->rw);else ret =nvme_pci_setup_prps (dev, req, &cmnd->rw);if (ret !=BLK_STS_OK )
@@ -886,9 +875,9 @@ static blk_status_t nvme_map_data(struct nvme_dev *dev, struct request *req,return BLK_STS_OK ; out_unmap_sg: -nvme_unmap_sg (dev, req); +dma_unmap_sgtable (dev->dev, &iod->sgt,rq_dma_dir (req),0 ); out_free_sg: -mempool_free (iod->sg, dev->iod_mempool); +mempool_free (iod->sgt.sgl, dev->iod_mempool);return ret; }
@@ -912,7 +901,7 @@ static blk_status_t nvme_prep_rq(struct nvme_dev *dev, struct request *req) iod->aborted =0 ; iod->npages =-1 ; - iod->nents =0 ; + iod->sgt.nents =0 ; ret =nvme_setup_cmd (req->q->queuedata, req);if (ret)
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 11/13 RDMA/core: introduce ib_dma_pci_p2p_dma_supported()
In reply to Logan GunthorpeIntroduce the helper function ib_dma_pci_p2p_dma_supported() to check if a given ib_device can be used in P2PDMA transfers. This ensures the ib_device is not using virt_dma and also that the underlying dma_device supports P2PDMA.
Use the new helper in nvme-rdma to replace the existing check for ib_uses_virt_dma(). Adding the dma_pci_p2pdma_supported() check allows switching away from pci_p2pdma_[un]map_sg().
--- drivers/nvme/target/rdma.c | 2 +- include/rdma/ib_verbs.h | 11 +++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-)
drivers/nvme/target/rdma.c +1 -1
@@ -415,7 +415,7 @@ static int nvmet_rdma_alloc_rsp(struct nvmet_rdma_device *ndev,if (ib_dma_mapping_error (ndev->device, r->send_sge.addr)) goto out_free_rsp; -if (!ib_uses_virt_dma (ndev->device)) +if (ib_dma_pci_p2p_dma_supported (ndev->device)) r->req.p2p_client = &ndev->device->dev; r->send_sge.length =sizeof (*r->req.cqe); r->send_sge.lkey = ndev->pd->local_dma_lkey;
include/rdma/ib_verbs.h +11 -0
@@ -4013,6 +4013,17 @@ static inline bool ib_uses_virt_dma(struct ib_device *dev)return IS_ENABLED (CONFIG_INFINIBAND_VIRT_DMA ) && !dev->dma_device; } +/* +* Check if a IB device's underlying DMA mapping supports P2PDMA transfers. +*/ +static inline bool ib_dma_pci_p2p_dma_supported (struct ib_device *dev) +{ +if (ib_uses_virt_dma (dev)) +return false; + +return dma_pci_p2pdma_supported (dev->dma_device); +} +/** * ib_dma_mapping_error - check a DMA addr for error * @dev: The device for which the dma_addr was created
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 12/13 RDMA/rw: drop pci_p2pdma_[un]map_sg()
In reply to Logan Gunthorpedma_map_sg() now supports the use of P2PDMA pages so pci_p2pdma_map_sg() is no longer necessary and may be dropped. This means the rdma_rw_[un]map_sg() helpers are no longer necessary. Remove it all.
--- drivers/infiniband/core/rw.c | 45 ++++++++---------------------------- 1 file changed, 9 insertions(+), 36 deletions(-)
drivers/infiniband/core/rw.c +9 -36
@@ -274,33 +274,6 @@ static int rdma_rw_init_single_wr(struct rdma_rw_ctx *ctx, struct ib_qp *qp,return 1 ; } -static void rdma_rw_unmap_sg (struct ib_device *dev,struct scatterlist *sg, -u32 sg_cnt,enum dma_data_direction dir) -{ -if (is_pci_p2pdma_page (sg_page (sg))) -pci_p2pdma_unmap_sg (dev->dma_device, sg, sg_cnt, dir); -else -ib_dma_unmap_sg (dev, sg, sg_cnt, dir); -} - -static int rdma_rw_map_sgtable (struct ib_device *dev,struct sg_table *sgt, -enum dma_data_direction dir) -{ -int nents; - -if (is_pci_p2pdma_page (sg_page (sgt->sgl))) { -if (WARN_ON_ONCE (ib_uses_virt_dma (dev))) -return 0 ; - nents =pci_p2pdma_map_sg (dev->dma_device, sgt->sgl, - sgt->orig_nents, dir); -if (!nents) -return -EIO ; - sgt->nents = nents; -return 0 ; - } -return ib_dma_map_sgtable_attrs (dev, sgt, dir,0 ); -} -/** * rdma_rw_ctx_init - initialize a RDMA READ/WRITE context * @ctx: context to initialize
@@ -327,7 +300,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u32 port_num, };int ret; - ret =rdma_rw_map_sgtable (dev, &sgt, dir); + ret =ib_dma_map_sgtable_attrs (dev, &sgt, dir,0 );if (ret)return ret; sg_cnt = sgt.nents;
@@ -366,7 +339,7 @@ int rdma_rw_ctx_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp, u32 port_num,return ret; out_unmap_sg: -rdma_rw_unmap_sg (dev, sgt.sgl, sgt.orig_nents, dir); +ib_dma_unmap_sgtable_attrs (dev, &sgt, dir,0 );return ret; }EXPORT_SYMBOL (rdma_rw_ctx_init);
@@ -414,12 +387,12 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,return -EINVAL ; } - ret =rdma_rw_map_sgtable (dev, &sgt, dir); + ret =ib_dma_map_sgtable_attrs (dev, &sgt, dir,0 );if (ret)return ret;if (prot_sg_cnt) { - ret =rdma_rw_map_sgtable (dev, &prot_sgt, dir); + ret =ib_dma_map_sgtable_attrs (dev, &prot_sgt, dir,0 );if (ret) goto out_unmap_sg; }
@@ -486,9 +459,9 @@ int rdma_rw_ctx_signature_init(struct rdma_rw_ctx *ctx, struct ib_qp *qp,kfree (ctx->reg); out_unmap_prot_sg:if (prot_sgt.nents) -rdma_rw_unmap_sg (dev, prot_sgt.sgl, prot_sgt.orig_nents, dir); +ib_dma_unmap_sgtable_attrs (dev, &prot_sgt, dir,0 ); out_unmap_sg: -rdma_rw_unmap_sg (dev, sgt.sgl, sgt.orig_nents, dir); +ib_dma_unmap_sgtable_attrs (dev, &sgt, dir,0 );return ret; }EXPORT_SYMBOL (rdma_rw_ctx_signature_init);
@@ -621,7 +594,7 @@ void rdma_rw_ctx_destroy(struct rdma_rw_ctx *ctx, struct ib_qp *qp,break ; } -rdma_rw_unmap_sg (qp->pd->device, sg, sg_cnt, dir); +ib_dma_unmap_sg (qp->pd->device, sg, sg_cnt, dir); }EXPORT_SYMBOL (rdma_rw_ctx_destroy);
@@ -649,8 +622,8 @@ void rdma_rw_ctx_destroy_signature(struct rdma_rw_ctx *ctx, struct ib_qp *qp,kfree (ctx->reg);if (prot_sg_cnt) -rdma_rw_unmap_sg (qp->pd->device, prot_sg, prot_sg_cnt, dir); -rdma_rw_unmap_sg (qp->pd->device, sg, sg_cnt, dir); +ib_dma_unmap_sg (qp->pd->device, prot_sg, prot_sg_cnt, dir); +ib_dma_unmap_sg (qp->pd->device, sg, sg_cnt, dir); }EXPORT_SYMBOL (rdma_rw_ctx_destroy_signature);
Signature · 1 line
2.30.2
Logan Gunthorpe
maintainerPATCH v8 13/13 PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg()
In reply to Logan GunthorpeThis interface is superseded by support in dma_map_sg() which now supports heterogeneous scatterlists. There are no longer any users, so remove it.
--- drivers/pci/p2pdma.c | 66 -------------------------------------- include/linux/pci-p2pdma.h | 27 ---------------- 2 files changed, 93 deletions(-)
drivers/pci/p2pdma.c +0 -66
@@ -872,72 +872,6 @@ static enum pci_p2pdma_map_type pci_p2pdma_map_type(struct dev_pagemap *pgmap,return type; } -static int __pci_p2pdma_map_sg (struct pci_p2pdma_pagemap *p2p_pgmap, -struct device *dev,struct scatterlist *sg,int nents) -{ -struct scatterlist *s; -int i; - -for_each_sg (sg, s, nents, i) { - s->dma_address =sg_phys (s) + p2p_pgmap->bus_offset; -sg_dma_len (s) = s->length; - } - -return nents; -} - -/** -* pci_p2pdma_map_sg_attrs - map a PCI peer-to-peer scatterlist for DMA -* @dev: device doing the DMA request -* @sg: scatter list to map -* @nents: elements in the scatterlist -* @dir: DMA direction -* @attrs: DMA attributes passed to dma_map_sg() (if called) -* -* Scatterlists mapped with this function should be unmapped using -* pci_p2pdma_unmap_sg_attrs(). -* -* Returns the number of SG entries mapped or 0 on error. -*/ -int pci_p2pdma_map_sg_attrs (struct device *dev,struct scatterlist *sg, -int nents,enum dma_data_direction dir,unsigned long attrs) -{ -struct pci_p2pdma_pagemap *p2p_pgmap = -to_p2p_pgmap (sg_page (sg)->pgmap); - -switch (pci_p2pdma_map_type (sg_page (sg)->pgmap, dev)) { -case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE : -return dma_map_sg_attrs (dev, sg, nents, dir, attrs); -case PCI_P2PDMA_MAP_BUS_ADDR : -return __pci_p2pdma_map_sg (p2p_pgmap, dev, sg, nents); -default : -/* Mapping is not Supported */ -return 0 ; - } -} -EXPORT_SYMBOL_GPL (pci_p2pdma_map_sg_attrs); - -/** -* pci_p2pdma_unmap_sg_attrs - unmap a PCI peer-to-peer scatterlist that was -* mapped with pci_p2pdma_map_sg() -* @dev: device doing the DMA request -* @sg: scatter list to map -* @nents: number of elements returned by pci_p2pdma_map_sg() -* @dir: DMA direction -* @attrs: DMA attributes passed to dma_unmap_sg() (if called) -*/ -void pci_p2pdma_unmap_sg_attrs (struct device *dev,struct scatterlist *sg, -int nents,enum dma_data_direction dir,unsigned long attrs) -{ -enum pci_p2pdma_map_type map_type; - - map_type =pci_p2pdma_map_type (sg_page (sg)->pgmap, dev); - -if (map_type ==PCI_P2PDMA_MAP_THRU_HOST_BRIDGE ) -dma_unmap_sg_attrs (dev, sg, nents, dir, attrs); -} -EXPORT_SYMBOL_GPL (pci_p2pdma_unmap_sg_attrs); -/** * pci_p2pdma_map_segment - map an sg segment determining the mapping type * @state: State structure that should be declared outside of the for_each_sg()
include/linux/pci-p2pdma.h +0 -27
@@ -30,10 +30,6 @@ struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev, unsigned int *nents, u32 length);void pci_p2pmem_free_sgl (struct pci_dev *pdev,struct scatterlist *sgl);void pci_p2pmem_publish (struct pci_dev *pdev,bool publish); -int pci_p2pdma_map_sg_attrs (struct device *dev,struct scatterlist *sg, -int nents,enum dma_data_direction dir,unsigned long attrs); -void pci_p2pdma_unmap_sg_attrs (struct device *dev,struct scatterlist *sg, -int nents,enum dma_data_direction dir,unsigned long attrs);int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev,bool *use_p2pdma); ssize_t pci_p2pdma_enable_show(char *page, struct pci_dev *p2p_dev,
@@ -83,17 +79,6 @@ static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev,static inline void pci_p2pmem_publish (struct pci_dev *pdev,bool publish) { } -static inline int pci_p2pdma_map_sg_attrs (struct device *dev, -struct scatterlist *sg,int nents,enum dma_data_direction dir, -unsigned long attrs) -{ -return 0 ; -} -static inline void pci_p2pdma_unmap_sg_attrs (struct device *dev, -struct scatterlist *sg,int nents,enum dma_data_direction dir, -unsigned long attrs) -{ -}static inline int pci_p2pdma_enable_store (const char *page,struct pci_dev **p2p_dev,bool *use_p2pdma) {
@@ -119,16 +104,4 @@ static inline struct pci_dev *pci_p2pmem_find(struct device *client) return pci_p2pmem_find_many(&client, 1); } -static inline int pci_p2pdma_map_sg (struct device *dev,struct scatterlist *sg, -int nents,enum dma_data_direction dir) -{ -return pci_p2pdma_map_sg_attrs (dev, sg, nents, dir,0 ); -} - -static inline void pci_p2pdma_unmap_sg (struct device *dev, -struct scatterlist *sg,int nents,enum dma_data_direction dir) -{ -pci_p2pdma_unmap_sg_attrs (dev, sg, nents, dir,0 ); -} -#endif /* _LINUX_PCI_P2P_H */
Signature · 1 line
2.30.2
Christoph Hellwig
maintainerI'd like to hear from Robin on the dma-iommu parts, but otherwise this looks fine to me modulo a little filed naming change I could handle when applying this.
I've applied this to the dma-mapping for-next branch.
I'd have liked a review from Robin, but as he seems to be out and the rework was to fit his comments I'd rather have us unblocked before the end of the merge window.
Logan Gunthorpe
maintainerQuoted 5 lines · Christoph Hellwig
I've applied this to the dma-mapping for-next branch.
I'd have liked a review from Robin, but as he seems to be out and the rework was to fit his comments I'd rather have us unblocked before the end of the merge window.
Great thanks! If Robin has some feedback when he gets back I'd be happy to make a couple more cleanup patches.
Logan
Robin Murphy
maintainerQuoted 11 lines · Logan Gunthorpe
Quoted 5 lines · Christoph Hellwig
I've applied this to the dma-mapping for-next branch.
I'd have liked a review from Robin, but as he seems to be out and the rework was to fit his comments I'd rather have us unblocked before the end of the merge window.
Great thanks! If Robin has some feedback when he gets back I'd be happy to make a couple more cleanup patches.
Indeed I've been taking some (rather overdue) holiday the last couple of weeks, sorry I forgot to take a look at this version beforehand. From a quick skim, nothing stands out as objectionable; thanks for taking my suggestions on board!
At some point once this lands I'll have a go at using the new flags to improve iommu-dma bounce buffering, so if I do spot anything else while I'm in there I'm sure could clean it up myself.
Cheers,
Robin.