implement generic dma_map_ops for IOMMUs v6
Subsystems (3)
Hi Robin and Joerg,
I think we are finally ready for the generic dma-iommu series. I have various DMA API changes pending, and Tom has patches ready to convert the AMD and Intel iommu drivers over to it. I'd love to have this in a stable branch shared between the dma-mapping and iommu trees the day after rc2 is released. I volunteer to create the branch, but I'm fine with it living in the iommu tree as well.
A git tree is also available at:
git://git.infradead.org/users/hch/misc.git dma-iommu-ops.6
Gitweb:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dma-iommu-ops.6
Changes since v5:
- rebased to latest linus tree and dropped a patch now merged
- remove the now pointless __dma_iommu_mmap function
- restore a cleanup from Robin that would have caused a conflict in last merge winow, but is fine now
Changes since v4:
- rebased to 5.2-rc1
Changes since v3:
- fold the separate patch to refactor mmap bounds checking
- don't warn on not finding a vm_area
- improve a commit log
- refactor __dma_iommu_free a little differently
- remove a minor MSI map cleanup to avoid a conflict with the "Split iommu_dma_map_msi_msg" series
Changes since v2:
- address various review comments and include patches from Robin
Changes since v1:
- only include other headers in dma-iommu.h if CONFIG_DMA_IOMMU is enabled
- keep using a scatterlist in iommu_dma_alloc
- split out mmap/sgtable fixes and move them early in the series
- updated a few commit logs
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
_______________________________________________
linux-arm-kernel mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
PATCH 1/23 iommu/dma: Cleanup dma-iommu.h
No need for a __KERNEL__ guard outside uapi and add a missing comment describing the #else cpp statement. Last but not least include <linux/errno.h> instead of the asm version, which is frowned upon.
--- include/linux/dma-iommu.h | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
include/linux/dma-iommu.h +2 -4
@@ -16,9 +16,8 @@#ifndef __DMA_IOMMU_H#define __DMA_IOMMU_H -#ifdef __KERNEL__ +#include <linux/errno.h> #include <linux/types.h> -#include <asm/errno.h> #ifdef CONFIG_IOMMU_DMA #include <linux/dma-mapping.h>
@@ -86,7 +85,7 @@ void iommu_dma_compose_msi_msg(struct msi_desc *desc,void iommu_dma_get_resv_regions (struct device *dev,struct list_head *list); -#else +#else /* CONFIG_IOMMU_DMA */ struct iommu_domain ;struct msi_desc ;
@@ -128,5 +127,4 @@ static inline void iommu_dma_get_resv_regions(struct device *dev, struct list_he }#endif /* CONFIG_IOMMU_DMA */ -#endif /* __KERNEL__ */ #endif /* __DMA_IOMMU_H */
Signature · 1 line
2.20.1
PATCH 2/23 iommu/dma: Remove the flush_page callback
In reply to Christoph HellwigWe now have a arch_dma_prep_coherent architecture hook that is used for the generic DMA remap allocator, and we should use the same interface for the dma-iommu code.
--- arch/arm64/mm/dma-mapping.c | 8 +------- drivers/iommu/dma-iommu.c | 8 +++----- include/linux/dma-iommu.h | 3 +-- 3 files changed, 5 insertions(+), 14 deletions(-)
arch/arm64/mm/dma-mapping.c +1 -7
@@ -104,12 +104,6 @@ arch_initcall(arm64_dma_init);#include <linux/platform_device.h> #include <linux/amba/bus.h> -/* Thankfully, all cache ops are by VA so we can ignore phys here */ -static void flush_page (struct device *dev,const void *virt,phys_addr_t phys) -{ -__dma_flush_area (virt,PAGE_SIZE ); -} -static void *__iommu_alloc_attrs (struct device *dev,size_t size,dma_addr_t *handle,gfp_t gfp,unsigned long attrs)
@@ -186,7 +180,7 @@ static void *__iommu_alloc_attrs(struct device *dev, size_t size,struct page **pages; pages =iommu_dma_alloc (dev, iosize, gfp, attrs, ioprot, - handle, flush_page); + handle);if (!pages)return NULL ;
drivers/iommu/dma-iommu.c +3 -5
@@ -22,6 +22,7 @@#include <linux/acpi_iort.h> #include <linux/device.h> #include <linux/dma-iommu.h> +#include <linux/dma-noncoherent.h> #include <linux/gfp.h> #include <linux/huge_mm.h> #include <linux/iommu.h>
@@ -560,8 +561,6 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size,* @attrs: DMA attributes for this allocation * @prot: IOMMU mapping flags * @handle: Out argument for allocated DMA handle -* @flush_page: Arch callback which must ensure PAGE_SIZE bytes from the -* given VA/PA are visible to the given non-coherent device. * * If @size is less than PAGE_SIZE, then a full CPU page will be allocated, * but an IOMMU which supports smaller pages might not map the whole thing.
@@ -570,8 +569,7 @@ void iommu_dma_free(struct device *dev, struct page **pages, size_t size,* or NULL on failure. */ struct page **iommu_dma_alloc (struct device *dev,size_t size,gfp_t gfp, -unsigned long attrs,int prot,dma_addr_t *handle, -void (*flush_page)(struct device *,const void *,phys_addr_t )) +unsigned long attrs,int prot,dma_addr_t *handle) {struct iommu_domain *domain =iommu_get_dma_domain (dev);struct iommu_dma_cookie *cookie = domain->iova_cookie;
@@ -615,7 +613,7 @@ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp,*/ sg_miter_start (&miter, sgt.sgl, sgt.orig_nents,SG_MITER_FROM_SG );while (sg_miter_next (&miter)) -flush_page (dev, miter.addr,page_to_phys (miter.page)); +arch_dma_prep_coherent (miter.page,PAGE_SIZE );sg_miter_stop (&miter); }
include/linux/dma-iommu.h +1 -2
@@ -44,8 +44,7 @@ int dma_info_to_prot(enum dma_data_direction dir, bool coherent,* the arch code to take care of attributes and cache maintenance */ struct page **iommu_dma_alloc (struct device *dev,size_t size,gfp_t gfp, -unsigned long attrs,int prot,dma_addr_t *handle, -void (*flush_page)(struct device *,const void *,phys_addr_t )); +unsigned long attrs,int prot,dma_addr_t *handle);void iommu_dma_free (struct device *dev,struct page **pages,size_t size,dma_addr_t *handle);
Signature · 6 lines
2.20.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 3/23 iommu/dma: Use for_each_sg in iommu_dma_alloc
In reply to Christoph Hellwigarch_dma_prep_coherent can handle physically contiguous ranges larger than PAGE_SIZE just fine, which means we don't need a page-based iterator.
--- drivers/iommu/dma-iommu.c | 14 +++++--------- 1 file changed, 5 insertions(+), 9 deletions(-)
drivers/iommu/dma-iommu.c +5 -9
@@ -606,15 +606,11 @@ struct page **iommu_dma_alloc(struct device *dev, size_t size, gfp_t gfp, goto out_free_iova;if (!(prot &IOMMU_CACHE )) { -struct sg_mapping_iter miter; -/* -* The CPU-centric flushing implied by SG_MITER_TO_SG isn't -* sufficient here, so skip it by using the "wrong" direction. -*/ -sg_miter_start (&miter, sgt.sgl, sgt.orig_nents,SG_MITER_FROM_SG ); -while (sg_miter_next (&miter)) -arch_dma_prep_coherent (miter.page,PAGE_SIZE ); -sg_miter_stop (&miter); +struct scatterlist *sg; +int i; + +for_each_sg (sgt.sgl, sg, sgt.orig_nents, i) +arch_dma_prep_coherent (sg_page (sg), sg->length); } if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, prot)
Signature · 1 line
2.20.1
PATCH 4/23 iommu/dma: move the arm64 wrappers to common code
In reply to Christoph HellwigThere is nothing really arm64 specific in the iommu_dma_ops implementation, so move it to dma-iommu.c and keep a lot of symbols self-contained. Note the implementation does depend on the DMA_DIRECT_REMAP infrastructure for now, so we'll have to make the DMA_IOMMU support depend on it, but this will be relaxed soon.
--- arch/arm64/mm/dma-mapping.c | 394 +---------------------------------- drivers/iommu/Kconfig | 1 + drivers/iommu/dma-iommu.c | 398 +++++++++++++++++++++++++++++++++--- include/linux/dma-iommu.h | 42 +--- 4 files changed, 378 insertions(+), 457 deletions(-)
arch/arm64/mm/dma-mapping.c +4 -390
@@ -27,6 +27,7 @@#include <linux/dma-direct.h> #include <linux/dma-noncoherent.h> #include <linux/dma-contiguous.h> +#include <linux/dma-iommu.h> #include <linux/vmalloc.h> #include <linux/swiotlb.h> #include <linux/pci.h>
@@ -58,37 +59,6 @@ void arch_dma_prep_coherent(struct page *page, size_t size)__dma_flush_area (page_address (page), size); } -#ifdef CONFIG_IOMMU_DMA -static int __swiotlb_get_sgtable_page (struct sg_table *sgt, -struct page *page,size_t size) -{ -int ret =sg_alloc_table (sgt,1 ,GFP_KERNEL ); - -if (!ret) -sg_set_page (sgt->sgl, page,PAGE_ALIGN (size),0 ); - -return ret; -} - -static int __swiotlb_mmap_pfn (struct vm_area_struct *vma, -unsigned long pfn,size_t size) -{ -int ret = -ENXIO ; -unsigned long nr_vma_pages =vma_pages (vma); -unsigned long nr_pages =PAGE_ALIGN (size) >>PAGE_SHIFT ; -unsigned long off = vma->vm_pgoff; - -if (off < nr_pages && nr_vma_pages <= (nr_pages - off)) { - ret =remap_pfn_range (vma, vma->vm_start, - pfn + off, - vma->vm_end - vma->vm_start, - vma->vm_page_prot); - } - -return ret; -} -#endif /* CONFIG_IOMMU_DMA */ -static int __init arm64_dma_init (void ) { WARN_TAINT(ARCH_DMA_MINALIGN < cache_line_size(),
@@ -100,374 +70,18 @@ static int __init arm64_dma_init(void)arch_initcall (arm64_dma_init);#ifdef CONFIG_IOMMU_DMA -#include <linux/dma-iommu.h> -#include <linux/platform_device.h> -#include <linux/amba/bus.h> - -static void *__iommu_alloc_attrs (struct device *dev,size_t size, -dma_addr_t *handle,gfp_t gfp, -unsigned long attrs) -{ -bool coherent =dev_is_dma_coherent (dev); -int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs); -size_t iosize = size; -void *addr; - -if (WARN (!dev,"cannot create IOMMU mapping for unknown device\n" )) -return NULL ; - - size =PAGE_ALIGN (size); - -/* -* Some drivers rely on this, and we probably don't want the -* possibility of stale kernel data being read by devices anyway. -*/ - gfp |= __GFP_ZERO; - -if (!gfpflags_allow_blocking (gfp)) { -struct page *page; -/* -* In atomic context we can't remap anything, so we'll only -* get the virtually contiguous buffer we need by way of a -* physically contiguous allocation. -*/ -if (coherent) { - page =alloc_pages (gfp,get_order (size)); - addr = page ?page_address (page) :NULL ; - }else { - addr =dma_alloc_from_pool (size, &page, gfp); - } -if (!addr) -return NULL ; - - *handle =iommu_dma_map_page (dev, page,0 , iosize, ioprot); -if (*handle ==DMA_MAPPING_ERROR ) { -if (coherent) -__free_pages (page,get_order (size)); -else -dma_free_from_pool (addr, size); - addr =NULL ; - } - }else if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); -struct page *page; - - page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , -get_order (size), gfp & __GFP_NOWARN); -if (!page) -return NULL ; - - *handle =iommu_dma_map_page (dev, page,0 , iosize, ioprot); -if (*handle ==DMA_MAPPING_ERROR ) { -dma_release_from_contiguous (dev, page, - size >>PAGE_SHIFT ); -return NULL ; - } - addr =dma_common_contiguous_remap (page, size,VM_USERMAP , - prot, -__builtin_return_address (0 )); -if (addr) { -if (!coherent) -__dma_flush_area (page_to_virt (page), iosize); -memset (addr,0 , size); - }else { -iommu_dma_unmap_page (dev, *handle, iosize,0 , attrs); -dma_release_from_contiguous (dev, page, - size >>PAGE_SHIFT ); - } - }else { -pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); -struct page **pages; - - pages =iommu_dma_alloc (dev, iosize, gfp, attrs, ioprot, - handle); -if (!pages) -return NULL ; - - addr =dma_common_pages_remap (pages, size,VM_USERMAP , prot, -__builtin_return_address (0 )); -if (!addr) -iommu_dma_free (dev, pages, iosize, handle); - } -return addr; -} - -static void __iommu_free_attrs (struct device *dev,size_t size,void *cpu_addr, -dma_addr_t handle,unsigned long attrs) -{ -size_t iosize = size; - - size =PAGE_ALIGN (size); -/* -* @cpu_addr will be one of 4 things depending on how it was allocated: -* - A remapped array of pages for contiguous allocations. -* - A remapped array of pages from iommu_dma_alloc(), for all -* non-atomic allocations. -* - A non-cacheable alias from the atomic pool, for atomic -* allocations by non-coherent devices. -* - A normal lowmem address, for atomic allocations by -* coherent devices. -* Hence how dodgy the below logic looks... -*/ -if (dma_in_atomic_pool (cpu_addr, size)) { -iommu_dma_unmap_page (dev, handle, iosize,0 ,0 ); -dma_free_from_pool (cpu_addr, size); - }else if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -struct page *page =vmalloc_to_page (cpu_addr); - -iommu_dma_unmap_page (dev, handle, iosize,0 , attrs); -dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); -dma_common_free_remap (cpu_addr, size,VM_USERMAP ); - }else if (is_vmalloc_addr (cpu_addr)){ -struct vm_struct *area =find_vm_area (cpu_addr); - -if (WARN_ON (!area || !area->pages)) -return ; -iommu_dma_free (dev, area->pages, iosize, &handle); -dma_common_free_remap (cpu_addr, size,VM_USERMAP ); - }else { -iommu_dma_unmap_page (dev, handle, iosize,0 ,0 ); -__free_pages (virt_to_page (cpu_addr),get_order (size)); - } -} - -static int __iommu_mmap_attrs (struct device *dev,struct vm_area_struct *vma, -void *cpu_addr,dma_addr_t dma_addr,size_t size, -unsigned long attrs) -{ -struct vm_struct *area; -int ret; - - vma->vm_page_prot =arch_dma_mmap_pgprot (dev, vma->vm_page_prot, attrs); - -if (dma_mmap_from_dev_coherent (dev, vma, cpu_addr, size, &ret)) -return ret; - -if (!is_vmalloc_addr (cpu_addr)) { -unsigned long pfn =page_to_pfn (virt_to_page (cpu_addr)); -return __swiotlb_mmap_pfn (vma, pfn, size); - } - -if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -/* -* DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, -* hence in the vmalloc space. -*/ -unsigned long pfn =vmalloc_to_pfn (cpu_addr); -return __swiotlb_mmap_pfn (vma, pfn, size); - } - - area =find_vm_area (cpu_addr); -if (WARN_ON (!area || !area->pages)) -return -ENXIO ; - -return iommu_dma_mmap (area->pages, size, vma); -} - -static int __iommu_get_sgtable (struct device *dev,struct sg_table *sgt, -void *cpu_addr,dma_addr_t dma_addr, -size_t size,unsigned long attrs) -{ -unsigned int count =PAGE_ALIGN (size) >>PAGE_SHIFT ; -struct vm_struct *area =find_vm_area (cpu_addr); - -if (!is_vmalloc_addr (cpu_addr)) { -struct page *page =virt_to_page (cpu_addr); -return __swiotlb_get_sgtable_page (sgt, page, size); - } - -if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -/* -* DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, -* hence in the vmalloc space. -*/ -struct page *page =vmalloc_to_page (cpu_addr); -return __swiotlb_get_sgtable_page (sgt, page, size); - } - -if (WARN_ON (!area || !area->pages)) -return -ENXIO ; - -return sg_alloc_table_from_pages (sgt, area->pages, count,0 , size, -GFP_KERNEL ); -} - -static void __iommu_sync_single_for_cpu (struct device *dev, -dma_addr_t dev_addr,size_t size, -enum dma_data_direction dir) -{ -phys_addr_t phys; - -if (dev_is_dma_coherent (dev)) -return ; - - phys =iommu_iova_to_phys (iommu_get_dma_domain (dev), dev_addr); -arch_sync_dma_for_cpu (dev, phys, size, dir); -} - -static void __iommu_sync_single_for_device (struct device *dev, -dma_addr_t dev_addr,size_t size, -enum dma_data_direction dir) -{ -phys_addr_t phys; - -if (dev_is_dma_coherent (dev)) -return ; - - phys =iommu_iova_to_phys (iommu_get_dma_domain (dev), dev_addr); -arch_sync_dma_for_device (dev, phys, size, dir); -} - -static dma_addr_t __iommu_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size, -enum dma_data_direction dir, -unsigned long attrs) -{ -bool coherent =dev_is_dma_coherent (dev); -int prot =dma_info_to_prot (dir, coherent, attrs); -dma_addr_t dev_addr =iommu_dma_map_page (dev, page, offset, size, prot); - -if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC ) && - dev_addr !=DMA_MAPPING_ERROR ) -__dma_map_area (page_address (page) + offset, size, dir); - -return dev_addr; -} - -static void __iommu_unmap_page (struct device *dev,dma_addr_t dev_addr, -size_t size,enum dma_data_direction dir, -unsigned long attrs) -{ -if ((attrs &DMA_ATTR_SKIP_CPU_SYNC ) ==0 ) -__iommu_sync_single_for_cpu (dev, dev_addr, size, dir); - -iommu_dma_unmap_page (dev, dev_addr, size, dir, attrs); -} - -static void __iommu_sync_sg_for_cpu (struct device *dev, -struct scatterlist *sgl,int nelems, -enum dma_data_direction dir) -{ -struct scatterlist *sg; -int i; - -if (dev_is_dma_coherent (dev)) -return ; - -for_each_sg (sgl, sg, nelems, i) -arch_sync_dma_for_cpu (dev,sg_phys (sg), sg->length, dir); -} - -static void __iommu_sync_sg_for_device (struct device *dev, -struct scatterlist *sgl,int nelems, -enum dma_data_direction dir) -{ -struct scatterlist *sg; -int i; - -if (dev_is_dma_coherent (dev)) -return ; - -for_each_sg (sgl, sg, nelems, i) -arch_sync_dma_for_device (dev,sg_phys (sg), sg->length, dir); -} - -static int __iommu_map_sg_attrs (struct device *dev,struct scatterlist *sgl, -int nelems,enum dma_data_direction dir, -unsigned long attrs) -{ -bool coherent =dev_is_dma_coherent (dev); - -if ((attrs &DMA_ATTR_SKIP_CPU_SYNC ) ==0 ) -__iommu_sync_sg_for_device (dev, sgl, nelems, dir); - -return iommu_dma_map_sg (dev, sgl, nelems, -dma_info_to_prot (dir, coherent, attrs)); -} - -static void __iommu_unmap_sg_attrs (struct device *dev, -struct scatterlist *sgl,int nelems, -enum dma_data_direction dir, -unsigned long attrs) -{ -if ((attrs &DMA_ATTR_SKIP_CPU_SYNC ) ==0 ) -__iommu_sync_sg_for_cpu (dev, sgl, nelems, dir); - -iommu_dma_unmap_sg (dev, sgl, nelems, dir, attrs); -} - -static const struct dma_map_ops iommu_dma_ops = { - .alloc = __iommu_alloc_attrs, - .free = __iommu_free_attrs, - .mmap = __iommu_mmap_attrs, - .get_sgtable = __iommu_get_sgtable, - .map_page = __iommu_map_page, - .unmap_page = __iommu_unmap_page, - .map_sg = __iommu_map_sg_attrs, - .unmap_sg = __iommu_unmap_sg_attrs, - .sync_single_for_cpu = __iommu_sync_single_for_cpu, - .sync_single_for_device = __iommu_sync_single_for_device, - .sync_sg_for_cpu = __iommu_sync_sg_for_cpu, - .sync_sg_for_device = __iommu_sync_sg_for_device, - .map_resource = iommu_dma_map_resource, - .unmap_resource = iommu_dma_unmap_resource, -}; - -static int __init __iommu_dma_init (void ) -{ -return iommu_dma_init (); -} -arch_initcall (__iommu_dma_init); - -static void __iommu_setup_dma_ops (struct device *dev,u64 dma_base,u64 size, -const struct iommu_ops *ops) -{ -struct iommu_domain *domain; - -if (!ops) -return ; - -/* -* The IOMMU core code allocates the default DMA domain, which the -* underlying IOMMU driver needs to support via the dma-iommu layer. -*/ - domain =iommu_get_domain_for_dev (dev); - -if (!domain) - goto out_err; - -if (domain->type ==IOMMU_DOMAIN_DMA ) { -if (iommu_dma_init_domain (domain, dma_base, size, dev)) - goto out_err; - - dev->dma_ops = &iommu_dma_ops; - } - -return ; - -out_err: -pr_warn ("Failed to set up IOMMU for device %s; retaining platform DMA ops\n" , -dev_name (dev)); -} -void arch_teardown_dma_ops (struct device *dev) { dev->dma_ops =NULL ; } - -#else - -static void __iommu_setup_dma_ops (struct device *dev,u64 dma_base,u64 size, -const struct iommu_ops *iommu) -{ } - -#endif /* CONFIG_IOMMU_DMA */ +#endif void arch_setup_dma_ops (struct device *dev,u64 dma_base,u64 size,const struct iommu_ops *iommu,bool coherent) { dev->dma_coherent = coherent; -__iommu_setup_dma_ops (dev, dma_base, size, iommu); +if (iommu) +iommu_setup_dma_ops (dev, dma_base, size);#ifdef CONFIG_XEN if (xen_initial_domain ())
drivers/iommu/Kconfig +1 -0
@@ -97,6 +97,7 @@ config IOMMU_DMA select IOMMU_IOVA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH + depends on DMA_DIRECT_REMAP config FSL_PAMU bool "Freescale IOMMU support"
drivers/iommu/dma-iommu.c +370 -28
531 lines not on this page · raw message
include/linux/dma-iommu.h +3 -39
61 lines not on this page · raw message
Signature · 1 line
2.20.1
PATCH 5/23 iommu/dma: Move __iommu_dma_map
In reply to Christoph HellwigMoving this function up to its unmap counterpart helps to keep related code together for the following changes.
--- drivers/iommu/dma-iommu.c | 46 +++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 23 deletions(-)
drivers/iommu/dma-iommu.c +23 -23
@@ -464,6 +464,29 @@ static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr,iommu_dma_free_iova (cookie, dma_addr, size); } +static dma_addr_t __iommu_dma_map (struct device *dev,phys_addr_t phys, +size_t size,int prot,struct iommu_domain *domain) +{ +struct iommu_dma_cookie *cookie = domain->iova_cookie; +size_t iova_off =0 ; +dma_addr_t iova; + +if (cookie->type ==IOMMU_DMA_IOVA_COOKIE ) { + iova_off =iova_offset (&cookie->iovad, phys); + size =iova_align (&cookie->iovad, size + iova_off); + } + + iova =iommu_dma_alloc_iova (domain, size,dma_get_mask (dev), dev); +if (!iova) +return DMA_MAPPING_ERROR ; + +if (iommu_map (domain, iova, phys - iova_off, size, prot)) { +iommu_dma_free_iova (cookie, iova, size); +return DMA_MAPPING_ERROR ; + } +return iova + iova_off; +} +static void __iommu_dma_free_pages (struct page **pages,int count) {while (count--)
@@ -692,29 +715,6 @@ static void iommu_dma_sync_sg_for_device(struct device *dev,arch_sync_dma_for_device (dev,sg_phys (sg), sg->length, dir); } -static dma_addr_t __iommu_dma_map (struct device *dev,phys_addr_t phys, -size_t size,int prot,struct iommu_domain *domain) -{ -struct iommu_dma_cookie *cookie = domain->iova_cookie; -size_t iova_off =0 ; -dma_addr_t iova; - -if (cookie->type ==IOMMU_DMA_IOVA_COOKIE ) { - iova_off =iova_offset (&cookie->iovad, phys); - size =iova_align (&cookie->iovad, size + iova_off); - } - - iova =iommu_dma_alloc_iova (domain, size,dma_get_mask (dev), dev); -if (!iova) -return DMA_MAPPING_ERROR ; - -if (iommu_map (domain, iova, phys - iova_off, size, prot)) { -iommu_dma_free_iova (cookie, iova, size); -return DMA_MAPPING_ERROR ; - } -return iova + iova_off; -} -static dma_addr_t __iommu_dma_map_page (struct device *dev,struct page *page,unsigned long offset,size_t size,int prot) {
Signature · 1 line
2.20.1
PATCH 6/23 iommu/dma: Move domain lookup into __iommu_dma_{map,unmap}
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
Most of the callers don't care, and the couple that do already have the domain to hand for other reasons are in slow paths where the (trivial) overhead of a repeated lookup will be utterly immaterial.
--- drivers/iommu/dma-iommu.c | 34 ++++++++++++++++------------------ 1 file changed, 16 insertions(+), 18 deletions(-)
drivers/iommu/dma-iommu.c +16 -18
@@ -448,9 +448,10 @@ static void iommu_dma_free_iova(struct iommu_dma_cookie *cookie, size >> iova_shift(iovad)); } -static void __iommu_dma_unmap (struct iommu_domain *domain,dma_addr_t dma_addr, +static void __iommu_dma_unmap (struct device *dev,dma_addr_t dma_addr,size_t size) { +struct iommu_domain *domain =iommu_get_dma_domain (dev);struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad;size_t iova_off =iova_offset (iovad, dma_addr);
@@ -465,8 +466,9 @@ static void __iommu_dma_unmap(struct iommu_domain *domain, dma_addr_t dma_addr, }static dma_addr_t __iommu_dma_map (struct device *dev,phys_addr_t phys, -size_t size,int prot,struct iommu_domain *domain) +size_t size,int prot) { +struct iommu_domain *domain =iommu_get_dma_domain (dev);struct iommu_dma_cookie *cookie = domain->iova_cookie;size_t iova_off =0 ;dma_addr_t iova;
@@ -565,7 +567,7 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,static void __iommu_dma_free (struct device *dev,struct page **pages,size_t size,dma_addr_t *handle) { -__iommu_dma_unmap (iommu_get_dma_domain (dev), *handle, size); +__iommu_dma_unmap (dev, *handle, size);__iommu_dma_free_pages (pages,PAGE_ALIGN (size) >>PAGE_SHIFT ); *handle =DMA_MAPPING_ERROR ; }
@@ -718,14 +720,13 @@ static void iommu_dma_sync_sg_for_device(struct device *dev,static dma_addr_t __iommu_dma_map_page (struct device *dev,struct page *page,unsigned long offset,size_t size,int prot) { -return __iommu_dma_map (dev,page_to_phys (page) + offset, size, prot, -iommu_get_dma_domain (dev)); +return __iommu_dma_map (dev,page_to_phys (page) + offset, size, prot); }static void __iommu_dma_unmap_page (struct device *dev,dma_addr_t handle,size_t size,enum dma_data_direction dir,unsigned long attrs) { -__iommu_dma_unmap (iommu_get_dma_domain (dev), handle, size); +__iommu_dma_unmap (dev, handle, size); } static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,
@@ -734,11 +735,10 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, {phys_addr_t phys =page_to_phys (page) + offset;bool coherent =dev_is_dma_coherent (dev); +int prot =dma_info_to_prot (dir, coherent, attrs);dma_addr_t dma_handle; - dma_handle =__iommu_dma_map (dev, phys, size, -dma_info_to_prot (dir, coherent, attrs), -iommu_get_dma_domain (dev)); + dma_handle =__iommu_dma_map (dev, phys, size, prot);if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC ) && dma_handle !=DMA_MAPPING_ERROR )arch_sync_dma_for_device (dev, phys, size, dir);
@@ -750,7 +750,7 @@ static void iommu_dma_unmap_page(struct device *dev, dma_addr_t dma_handle, {if (!(attrs &DMA_ATTR_SKIP_CPU_SYNC ))iommu_dma_sync_single_for_cpu (dev, dma_handle, size, dir); -__iommu_dma_unmap (iommu_get_dma_domain (dev), dma_handle, size); +__iommu_dma_unmap (dev, dma_handle, size); }/*
@@ -931,21 +931,20 @@ static void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, sg = tmp; } end =sg_dma_address (sg) +sg_dma_len (sg); -__iommu_dma_unmap (iommu_get_dma_domain (dev), start, end - start); +__iommu_dma_unmap (dev, start, end - start); }static dma_addr_t iommu_dma_map_resource (struct device *dev,phys_addr_t phys,size_t size,enum dma_data_direction dir,unsigned long attrs) {return __iommu_dma_map (dev, phys, size, -dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO , -iommu_get_dma_domain (dev)); +dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO ); }static void iommu_dma_unmap_resource (struct device *dev,dma_addr_t handle,size_t size,enum dma_data_direction dir,unsigned long attrs) { -__iommu_dma_unmap (iommu_get_dma_domain (dev), handle, size); +__iommu_dma_unmap (dev, handle, size); } static void *iommu_dma_alloc(struct device *dev, size_t size,
@@ -1205,9 +1204,8 @@ void iommu_setup_dma_ops(struct device *dev, u64 dma_base, u64 size) }static struct iommu_dma_msi_page *iommu_dma_get_msi_page (struct device *dev, -phys_addr_t msi_addr,struct iommu_domain *domain) +phys_addr_t msi_addr,struct iommu_dma_cookie *cookie) { -struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iommu_dma_msi_page *msi_page;dma_addr_t iova;int prot =IOMMU_WRITE |IOMMU_NOEXEC |IOMMU_MMIO ;
@@ -1222,7 +1220,7 @@ static struct iommu_dma_msi_page *iommu_dma_get_msi_page(struct device *dev,if (!msi_page)return NULL ; - iova =__iommu_dma_map (dev, msi_addr, size, prot, domain); + iova =__iommu_dma_map (dev, msi_addr, size, prot);if (iova ==DMA_MAPPING_ERROR ) goto out_free_page;
@@ -1258,7 +1256,7 @@ int iommu_dma_prepare_msi(struct msi_desc *desc, phys_addr_t msi_addr)* of an MSI from within an IPI handler. */ spin_lock_irqsave (&cookie->msi_lock, flags); - msi_page =iommu_dma_get_msi_page (dev, msi_addr, domain); + msi_page =iommu_dma_get_msi_page (dev, msi_addr, cookie);spin_unlock_irqrestore (&cookie->msi_lock, flags);msi_desc_set_iommu_cookie (desc, msi_page);
Signature · 6 lines
2.20.1 _______________________________________________ iommu mailing list [email protected] https://lists.linuxfoundation.org/mailman/listinfo/iommu
PATCH 7/23 iommu/dma: Squash __iommu_dma_{map,unmap}_page helpers
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
The remaining internal callsites don't care about having prototypes compatible with the relevant dma_map_ops callbacks, so the extra level of indirection just wastes space and complictaes things.
--- drivers/iommu/dma-iommu.c | 25 +++++++------------------ 1 file changed, 7 insertions(+), 18 deletions(-)
drivers/iommu/dma-iommu.c +7 -18
@@ -717,18 +717,6 @@ static void iommu_dma_sync_sg_for_device(struct device *dev,arch_sync_dma_for_device (dev,sg_phys (sg), sg->length, dir); } -static dma_addr_t __iommu_dma_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size,int prot) -{ -return __iommu_dma_map (dev,page_to_phys (page) + offset, size, prot); -} - -static void __iommu_dma_unmap_page (struct device *dev,dma_addr_t handle, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -__iommu_dma_unmap (dev, handle, size); -} -static dma_addr_t iommu_dma_map_page (struct device *dev,struct page *page,unsigned long offset,size_t size,enum dma_data_direction dir,unsigned long attrs)
@@ -974,7 +962,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,if (!addr)return NULL ; - *handle =__iommu_dma_map_page (dev, page,0 , iosize, ioprot); + *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, + ioprot);if (*handle ==DMA_MAPPING_ERROR ) {if (coherent)__free_pages (page,get_order (size));
@@ -991,7 +980,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,if (!page)return NULL ; - *handle =__iommu_dma_map_page (dev, page,0 , iosize, ioprot); + *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, ioprot);if (*handle ==DMA_MAPPING_ERROR ) {dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT );
@@ -1005,7 +994,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,arch_dma_prep_coherent (page, iosize);memset (addr,0 , size); }else { -__iommu_dma_unmap_page (dev, *handle, iosize,0 , attrs); +__iommu_dma_unmap (dev, *handle, iosize);dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); }
@@ -1044,12 +1033,12 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,* Hence how dodgy the below logic looks... */ if (dma_in_atomic_pool (cpu_addr, size)) { -__iommu_dma_unmap_page (dev, handle, iosize,0 ,0 ); +__iommu_dma_unmap (dev, handle, iosize);dma_free_from_pool (cpu_addr, size); }else if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) {struct page *page =vmalloc_to_page (cpu_addr); -__iommu_dma_unmap_page (dev, handle, iosize,0 , attrs); +__iommu_dma_unmap (dev, handle, iosize);dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT );dma_common_free_remap (cpu_addr, size,VM_USERMAP ); }else if (is_vmalloc_addr (cpu_addr)){
@@ -1060,7 +1049,7 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,__iommu_dma_free (dev, area->pages, iosize, &handle);dma_common_free_remap (cpu_addr, size,VM_USERMAP ); }else { -__iommu_dma_unmap_page (dev, handle, iosize,0 ,0 ); +__iommu_dma_unmap (dev, handle, iosize);__free_pages (virt_to_page (cpu_addr),get_order (size)); } }
Signature · 1 line
2.20.1
PATCH 8/23 iommu/dma: Factor out remapped pages lookup
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
Since we duplicate the find_vm_area() logic a few times in places where we only care aboute the pages, factor out a helper to abstract it.
--- drivers/iommu/dma-iommu.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-)
drivers/iommu/dma-iommu.c +20 -12
@@ -554,6 +554,15 @@ static struct page **__iommu_dma_alloc_pages(struct device *dev,return pages; } +static struct page **__iommu_dma_get_pages (void *cpu_addr) +{ +struct vm_struct *area =find_vm_area (cpu_addr); + +if (!area || !area->pages) +return NULL ; +return area->pages; +} +/** * iommu_dma_free - Free a buffer allocated by __iommu_dma_alloc() * @dev: Device which owns this buffer
@@ -1042,11 +1051,11 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT );dma_common_free_remap (cpu_addr, size,VM_USERMAP ); }else if (is_vmalloc_addr (cpu_addr)){ -struct vm_struct *area =find_vm_area (cpu_addr); +struct page **pages =__iommu_dma_get_pages (cpu_addr); -if (WARN_ON (!area || !area->pages)) +if (!pages)return ; -__iommu_dma_free (dev, area->pages, iosize, &handle); +__iommu_dma_free (dev, pages, iosize, &handle);dma_common_free_remap (cpu_addr, size,VM_USERMAP ); }else {__iommu_dma_unmap (dev, handle, iosize);
@@ -1078,7 +1087,7 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma, {unsigned long nr_pages =PAGE_ALIGN (size) >>PAGE_SHIFT ;unsigned long off = vma->vm_pgoff; -struct vm_struct *area; +struct page **pages;int ret; vma->vm_page_prot =arch_dma_mmap_pgprot (dev, vma->vm_page_prot, attrs);
@@ -1103,11 +1112,10 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma,return __iommu_dma_mmap_pfn (vma, pfn, size); } - area =find_vm_area (cpu_addr); -if (WARN_ON (!area || !area->pages)) + pages =__iommu_dma_get_pages (cpu_addr); +if (!pages)return -ENXIO ; - -return __iommu_dma_mmap (area->pages, size, vma); +return __iommu_dma_mmap (pages, size, vma); } static int __iommu_dma_get_sgtable_page(struct sg_table *sgt, struct page *page,
@@ -1125,7 +1133,7 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt, unsigned long attrs) {unsigned int count =PAGE_ALIGN (size) >>PAGE_SHIFT ; -struct vm_struct *area =find_vm_area (cpu_addr); +struct page **pages;if (!is_vmalloc_addr (cpu_addr)) {struct page *page =virt_to_page (cpu_addr);
@@ -1141,10 +1149,10 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt,return __iommu_dma_get_sgtable_page (sgt, page, size); } -if (WARN_ON (!area || !area->pages)) + pages =__iommu_dma_get_pages (cpu_addr); +if (!pages)return -ENXIO ; - -return sg_alloc_table_from_pages (sgt, area->pages, count,0 , size, +return sg_alloc_table_from_pages (sgt, pages, count,0 , size,GFP_KERNEL ); }
Signature · 1 line
2.20.1
PATCH 9/23 iommu/dma: Refactor the page array remapping allocator
In reply to Christoph HellwigMove the call to dma_common_pages_remap into __iommu_dma_alloc and rename it to iommu_dma_alloc_remap. This creates a self-contained helper for remapped pages allocation and mapping.
--- drivers/iommu/dma-iommu.c | 54 +++++++++++++++++++-------------------- 1 file changed, 26 insertions(+), 28 deletions(-)
drivers/iommu/dma-iommu.c +26 -28
@@ -564,9 +564,9 @@ static struct page **__iommu_dma_get_pages(void *cpu_addr) }/** -* iommu_dma_free - Free a buffer allocated by __iommu_dma_alloc() +* iommu_dma_free - Free a buffer allocated by iommu_dma_alloc_remap() * @dev: Device which owns this buffer -* @pages: Array of buffer pages as returned by __iommu_dma_alloc() +* @pages: Array of buffer pages as returned by __iommu_dma_alloc_remap() * @size: Size of buffer in bytes * @handle: DMA address of buffer *
@@ -582,33 +582,35 @@ static void __iommu_dma_free(struct device *dev, struct page **pages, }/** -* __iommu_dma_alloc - Allocate and map a buffer contiguous in IOVA space +* iommu_dma_alloc_remap - Allocate and map a buffer contiguous in IOVA space * @dev: Device to allocate memory for. Must be a real device * attached to an iommu_dma_domain * @size: Size of buffer in bytes +* @dma_handle: Out argument for allocated DMA handle * @gfp: Allocation flags * @attrs: DMA attributes for this allocation -* @prot: IOMMU mapping flags -* @handle: Out argument for allocated DMA handle * * If @size is less than PAGE_SIZE, then a full CPU page will be allocated, * but an IOMMU which supports smaller pages might not map the whole thing. * -* Return: Array of struct page pointers describing the buffer, -* or NULL on failure. +* Return: Mapped virtual address, or NULL on failure. */ -static struct page **__iommu_dma_alloc (struct device *dev,size_t size, -gfp_t gfp,unsigned long attrs,int prot,dma_addr_t *handle) +static void *iommu_dma_alloc_remap (struct device *dev,size_t size, +dma_addr_t *dma_handle,gfp_t gfp,unsigned long attrs) {struct iommu_domain *domain =iommu_get_dma_domain (dev);struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad; +bool coherent =dev_is_dma_coherent (dev); +int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs); +pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); +unsigned int count, min_size, alloc_sizes = domain->pgsize_bitmap;struct page **pages;struct sg_table sgt;dma_addr_t iova; -unsigned int count, min_size, alloc_sizes = domain->pgsize_bitmap; +void *vaddr; - *handle =DMA_MAPPING_ERROR ; + *dma_handle =DMA_MAPPING_ERROR ; min_size = alloc_sizes & -alloc_sizes;if (min_size <PAGE_SIZE ) {
@@ -634,7 +636,7 @@ static struct page **__iommu_dma_alloc(struct device *dev, size_t size,if (sg_alloc_table_from_pages (&sgt, pages, count,0 , size,GFP_KERNEL )) goto out_free_iova; -if (!(prot &IOMMU_CACHE )) { +if (!(ioprot &IOMMU_CACHE )) {struct scatterlist *sg;int i;
@@ -642,14 +644,21 @@ static struct page **__iommu_dma_alloc(struct device *dev, size_t size,arch_dma_prep_coherent (sg_page (sg), sg->length); } -if (iommu_map_sg (domain, iova, sgt.sgl, sgt.orig_nents, prot) +if (iommu_map_sg (domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; - *handle = iova; + vaddr =dma_common_pages_remap (pages, size,VM_USERMAP , prot, +__builtin_return_address (0 )); +if (!vaddr) + goto out_unmap; + + *dma_handle = iova;sg_free_table (&sgt); -return pages; +return vaddr; +out_unmap: +__iommu_dma_unmap (dev, iova, size); out_free_sg:sg_free_table (&sgt); out_free_iova:
@@ -1008,18 +1017,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, size >> PAGE_SHIFT); } }else { -pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); -struct page **pages; - - pages =__iommu_dma_alloc (dev, iosize, gfp, attrs, ioprot, - handle); -if (!pages) -return NULL ; - - addr =dma_common_pages_remap (pages, size,VM_USERMAP , prot, -__builtin_return_address (0 )); -if (!addr) -__iommu_dma_free (dev, pages, iosize, handle); + addr =iommu_dma_alloc_remap (dev, iosize, handle, gfp, attrs); }return addr; }
@@ -1033,7 +1031,7 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,/* * @cpu_addr will be one of 4 things depending on how it was allocated: * - A remapped array of pages for contiguous allocations. -* - A remapped array of pages from __iommu_dma_alloc(), for all +* - A remapped array of pages from iommu_dma_alloc_remap(), for all * non-atomic allocations. * - A non-cacheable alias from the atomic pool, for atomic * allocations by non-coherent devices.
Signature · 1 line
2.20.1
PATCH 10/23 iommu/dma: Remove __iommu_dma_free
In reply to Christoph HellwigWe only have a single caller of this function left, so open code it there.
--- drivers/iommu/dma-iommu.c | 21 ++------------------- 1 file changed, 2 insertions(+), 19 deletions(-)
drivers/iommu/dma-iommu.c +2 -19
@@ -563,24 +563,6 @@ static struct page **__iommu_dma_get_pages(void *cpu_addr)return area->pages; } -/** -* iommu_dma_free - Free a buffer allocated by iommu_dma_alloc_remap() -* @dev: Device which owns this buffer -* @pages: Array of buffer pages as returned by __iommu_dma_alloc_remap() -* @size: Size of buffer in bytes -* @handle: DMA address of buffer -* -* Frees both the pages associated with the buffer, and the array -* describing them -*/ -static void __iommu_dma_free (struct device *dev,struct page **pages, -size_t size,dma_addr_t *handle) -{ -__iommu_dma_unmap (dev, *handle, size); -__iommu_dma_free_pages (pages,PAGE_ALIGN (size) >>PAGE_SHIFT ); - *handle =DMA_MAPPING_ERROR ; -} -/** * iommu_dma_alloc_remap - Allocate and map a buffer contiguous in IOVA space * @dev: Device to allocate memory for. Must be a real device
@@ -1053,7 +1035,8 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,if (!pages)return ; -__iommu_dma_free (dev, pages, iosize, &handle); +__iommu_dma_unmap (dev, handle, iosize); +__iommu_dma_free_pages (pages, size >>PAGE_SHIFT );dma_common_free_remap (cpu_addr, size,VM_USERMAP ); }else {__iommu_dma_unmap (dev, handle, iosize);
Signature · 1 line
2.20.1
PATCH 11/23 iommu/dma: Refactor iommu_dma_free
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
The freeing logic was made particularly horrible by part of it being opaque to the arch wrapper, which led to a lot of convoluted repetition to ensure each path did everything in the right order. Now that it's all private, we can pick apart and consolidate the logically-distinct steps of freeing the IOMMU mapping, the underlying pages, and the CPU remap (if necessary) into something much more manageable.
--- drivers/iommu/dma-iommu.c | 73 ++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 40 deletions(-)
drivers/iommu/dma-iommu.c +33 -40
@@ -935,6 +935,39 @@ static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle,__iommu_dma_unmap (dev, handle, size); } +static void iommu_dma_free (struct device *dev,size_t size,void *cpu_addr, +dma_addr_t handle,unsigned long attrs) +{ +size_t alloc_size =PAGE_ALIGN (size); +int count = alloc_size >>PAGE_SHIFT ; +struct page *page =NULL , **pages =NULL ; + +__iommu_dma_unmap (dev, handle, size); + +/* Non-coherent atomic allocation? Easy */ +if (dma_free_from_pool (cpu_addr, alloc_size)) +return ; + +if (is_vmalloc_addr (cpu_addr)) { +/* +* If it the address is remapped, then it's either non-coherent +* or highmem CMA, or an iommu_dma_alloc_remap() construction. +*/ + pages =__iommu_dma_get_pages (cpu_addr); +if (!pages) + page =vmalloc_to_page (cpu_addr); +dma_common_free_remap (cpu_addr, alloc_size,VM_USERMAP ); + }else { +/* Lowmem means a coherent atomic or CMA allocation */ + page =virt_to_page (cpu_addr); + } + +if (pages) +__iommu_dma_free_pages (pages, count); +if (page && !dma_release_from_contiguous (dev, page, count)) +__free_pages (page,get_order (alloc_size)); +} +static void *iommu_dma_alloc (struct device *dev,size_t size,dma_addr_t *handle,gfp_t gfp,unsigned long attrs) {
@@ -1004,46 +1037,6 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,return addr; } -static void iommu_dma_free (struct device *dev,size_t size,void *cpu_addr, -dma_addr_t handle,unsigned long attrs) -{ -size_t iosize = size; - - size =PAGE_ALIGN (size); -/* -* @cpu_addr will be one of 4 things depending on how it was allocated: -* - A remapped array of pages for contiguous allocations. -* - A remapped array of pages from iommu_dma_alloc_remap(), for all -* non-atomic allocations. -* - A non-cacheable alias from the atomic pool, for atomic -* allocations by non-coherent devices. -* - A normal lowmem address, for atomic allocations by -* coherent devices. -* Hence how dodgy the below logic looks... -*/ -if (dma_in_atomic_pool (cpu_addr, size)) { -__iommu_dma_unmap (dev, handle, iosize); -dma_free_from_pool (cpu_addr, size); - }else if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -struct page *page =vmalloc_to_page (cpu_addr); - -__iommu_dma_unmap (dev, handle, iosize); -dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); -dma_common_free_remap (cpu_addr, size,VM_USERMAP ); - }else if (is_vmalloc_addr (cpu_addr)){ -struct page **pages =__iommu_dma_get_pages (cpu_addr); - -if (!pages) -return ; -__iommu_dma_unmap (dev, handle, iosize); -__iommu_dma_free_pages (pages, size >>PAGE_SHIFT ); -dma_common_free_remap (cpu_addr, size,VM_USERMAP ); - }else { -__iommu_dma_unmap (dev, handle, iosize); -__free_pages (virt_to_page (cpu_addr),get_order (size)); - } -} -static int __iommu_dma_mmap_pfn (struct vm_area_struct *vma,unsigned long pfn,size_t size) {
Signature · 1 line
2.20.1
PATCH 12/23 iommu/dma: Refactor iommu_dma_alloc
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
Shuffle around the self-contained atomic and non-contiguous cases to return early and get out of the way of the CMA case that we're about to work on next.
--- drivers/iommu/dma-iommu.c | 60 +++++++++++++++++++-------------------- 1 file changed, 30 insertions(+), 30 deletions(-)
drivers/iommu/dma-iommu.c +30 -30
@@ -973,14 +973,19 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, {bool coherent =dev_is_dma_coherent (dev);int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs); +pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs);size_t iosize = size; +struct page *page;void *addr; size =PAGE_ALIGN (size); gfp |= __GFP_ZERO; +if (gfpflags_allow_blocking (gfp) && + !(attrs &DMA_ATTR_FORCE_CONTIGUOUS )) +return iommu_dma_alloc_remap (dev, iosize, handle, gfp, attrs); +if (!gfpflags_allow_blocking (gfp)) { -struct page *page;/* * In atomic context we can't remap anything, so we'll only * get the virtually contiguous buffer we need by way of a
@@ -1002,39 +1007,34 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,__free_pages (page,get_order (size)); elsedma_free_from_pool (addr ,size ); - addr =NULL ; - } - }else if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); -struct page *page; - - page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , -get_order (size), gfp & __GFP_NOWARN); -if (!page)return NULL ; - - *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, ioprot); -if (*handle ==DMA_MAPPING_ERROR ) { -dma_release_from_contiguous (dev, page, - size >>PAGE_SHIFT ); -return NULL ; - } - addr =dma_common_contiguous_remap (page, size,VM_USERMAP , - prot, -__builtin_return_address (0 )); -if (addr) { -if (!coherent) -arch_dma_prep_coherent (page, iosize); -memset (addr,0 , size); - }else { -__iommu_dma_unmap (dev, *handle, iosize); -dma_release_from_contiguous (dev, page, - size >>PAGE_SHIFT ); } - }else { - addr =iommu_dma_alloc_remap (dev, iosize, handle, gfp, attrs); +return addr; } + + page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , +get_order (size), gfp & __GFP_NOWARN); +if (!page) +return NULL ; + + *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, ioprot); +if (*handle ==DMA_MAPPING_ERROR ) + goto out_free_pages; + + addr =dma_common_contiguous_remap (page, size,VM_USERMAP , prot, +__builtin_return_address (0 )); +if (!addr) + goto out_unmap; + +if (!coherent) +arch_dma_prep_coherent (page, iosize); +memset (addr,0 , size);return addr; +out_unmap: +__iommu_dma_unmap (dev, *handle, iosize); +out_free_pages: +dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); +return NULL ; } static int __iommu_dma_mmap_pfn(struct vm_area_struct *vma,
Signature · 1 line
2.20.1
PATCH 13/23 iommu/dma: Don't remap CMA unnecessarily
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
Always remapping CMA allocations was largely a bodge to keep the freeing logic manageable when it was split between here and an arch wrapper. Now that it's all together and streamlined, we can relax that limitation.
--- drivers/iommu/dma-iommu.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-)
drivers/iommu/dma-iommu.c +12 -7
@@ -973,7 +973,6 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, {bool coherent =dev_is_dma_coherent (dev);int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs); -pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs);size_t iosize = size;struct page *page;void *addr;
@@ -1021,13 +1020,19 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,if (*handle ==DMA_MAPPING_ERROR ) goto out_free_pages; - addr =dma_common_contiguous_remap (page, size,VM_USERMAP , prot, -__builtin_return_address (0 )); -if (!addr) - goto out_unmap; +if (!coherent ||PageHighMem (page)) { +pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); -if (!coherent) -arch_dma_prep_coherent (page, iosize); + addr =dma_common_contiguous_remap (page, size,VM_USERMAP , prot, +__builtin_return_address (0 )); +if (!addr) + goto out_unmap; + +if (!coherent) +arch_dma_prep_coherent (page, iosize); + }else { + addr =page_address (page); + }memset (addr,0 , size);return addr; out_unmap:
Signature · 1 line
2.20.1
PATCH 14/23 iommu/dma: Merge the CMA and alloc_pages allocation paths
In reply to Christoph HellwigInstead of having a separate code path for the non-blocking alloc_pages and CMA allocations paths merge them into one. There is a slight behavior change here in that we try the page allocator if CMA fails. This matches what dma-direct and other iommu drivers do and will be needed to use the dma-iommu code on architectures without DMA remapping later on.
--- drivers/iommu/dma-iommu.c | 32 ++++++++++++-------------------- 1 file changed, 12 insertions(+), 20 deletions(-)
drivers/iommu/dma-iommu.c +12 -20
@@ -974,7 +974,7 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,bool coherent =dev_is_dma_coherent (dev);int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs);size_t iosize = size; -struct page *page; +struct page *page =NULL ;void *addr; size =PAGE_ALIGN (size);
@@ -984,35 +984,26 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, !(attrs & DMA_ATTR_FORCE_CONTIGUOUS))return iommu_dma_alloc_remap (dev, iosize, handle, gfp, attrs); -if (!gfpflags_allow_blocking (gfp)) { -/* -* In atomic context we can't remap anything, so we'll only -* get the virtually contiguous buffer we need by way of a -* physically contiguous allocation. -*/ -if (coherent) { - page =alloc_pages (gfp,get_order (size)); - addr = page ?page_address (page) :NULL ; - }else { - addr =dma_alloc_from_pool (size, &page, gfp); - } +if (!gfpflags_allow_blocking (gfp) && !coherent) { + addr =dma_alloc_from_pool (size, &page, gfp);if (!addr)return NULL ; *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, ioprot);if (*handle ==DMA_MAPPING_ERROR ) { -if (coherent) -__free_pages (page,get_order (size)); -else -dma_free_from_pool (addr, size); +dma_free_from_pool (addr, size);return NULL ; }return addr; } - page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , -get_order (size), gfp & __GFP_NOWARN); +if (gfpflags_allow_blocking (gfp)) + page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , +get_order (size), + gfp & __GFP_NOWARN); +if (!page) + page =alloc_pages (gfp,get_order (size));if (!page)return NULL ;
@@ -1038,7 +1029,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, out_unmap:__iommu_dma_unmap (dev, *handle, iosize); out_free_pages: -dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT ); +if (!dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT )) +__free_pages (page,get_order (size));return NULL ; }
Signature · 1 line
2.20.1
PATCH 15/23 iommu/dma: Split iommu_dma_free
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
Most of it can double up to serve the failure cleanup path for iommu_dma_alloc().
--- drivers/iommu/dma-iommu.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-)
drivers/iommu/dma-iommu.c +8 -4
@@ -935,15 +935,12 @@ static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle,__iommu_dma_unmap (dev, handle, size); } -static void iommu_dma_free (struct device *dev,size_t size,void *cpu_addr, -dma_addr_t handle,unsigned long attrs) +static void __iommu_dma_free (struct device *dev,size_t size,void *cpu_addr) {size_t alloc_size =PAGE_ALIGN (size);int count = alloc_size >>PAGE_SHIFT ;struct page *page =NULL , **pages =NULL ; -__iommu_dma_unmap (dev, handle, size); -/* Non-coherent atomic allocation? Easy */ if (dma_free_from_pool (cpu_addr, alloc_size))return ;
@@ -968,6 +965,13 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,__free_pages (page,get_order (alloc_size)); } +static void iommu_dma_free (struct device *dev,size_t size,void *cpu_addr, +dma_addr_t handle,unsigned long attrs) +{ +__iommu_dma_unmap (dev, handle, size); +__iommu_dma_free (dev, size, cpu_addr); +} +static void *iommu_dma_alloc (struct device *dev,size_t size,dma_addr_t *handle,gfp_t gfp,unsigned long attrs) {
Signature · 1 line
2.20.1
PATCH 16/23 iommu/dma: Cleanup variable naming in iommu_dma_alloc
In reply to Christoph HellwigFrom: Robin Murphy <[email protected]>
Most importantly clear up the size / iosize confusion. Also rename addr to cpu_addr to match the surrounding code and make the intention a little more clear.
--- drivers/iommu/dma-iommu.c | 45 +++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 23 deletions(-)
drivers/iommu/dma-iommu.c +22 -23
@@ -977,64 +977,63 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, {bool coherent =dev_is_dma_coherent (dev);int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs); -size_t iosize = size; +size_t alloc_size =PAGE_ALIGN (size);struct page *page =NULL ; -void *addr; +void *cpu_addr; - size =PAGE_ALIGN (size); gfp |= __GFP_ZERO;if (gfpflags_allow_blocking (gfp) && !(attrs &DMA_ATTR_FORCE_CONTIGUOUS )) -return iommu_dma_alloc_remap (dev, iosize, handle, gfp, attrs); +return iommu_dma_alloc_remap (dev, size, handle, gfp, attrs);if (!gfpflags_allow_blocking (gfp) && !coherent) { - addr =dma_alloc_from_pool (size, &page, gfp); -if (!addr) + cpu_addr =dma_alloc_from_pool (alloc_size, &page, gfp); +if (!cpu_addr)return NULL ; - *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, + *handle =__iommu_dma_map (dev,page_to_phys (page), size, ioprot);if (*handle ==DMA_MAPPING_ERROR ) { -dma_free_from_pool (addr, size); +dma_free_from_pool (cpu_addr, alloc_size);return NULL ; } -return addr; +return cpu_addr; }if (gfpflags_allow_blocking (gfp)) - page =dma_alloc_from_contiguous (dev, size >>PAGE_SHIFT , -get_order (size), + page =dma_alloc_from_contiguous (dev, alloc_size >>PAGE_SHIFT , +get_order (alloc_size), gfp & __GFP_NOWARN);if (!page) - page =alloc_pages (gfp,get_order (size)); + page =alloc_pages (gfp,get_order (alloc_size));if (!page)return NULL ; - *handle =__iommu_dma_map (dev,page_to_phys (page), iosize, ioprot); + *handle =__iommu_dma_map (dev,page_to_phys (page), size, ioprot);if (*handle ==DMA_MAPPING_ERROR ) goto out_free_pages;if (!coherent ||PageHighMem (page)) {pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); - addr =dma_common_contiguous_remap (page, size,VM_USERMAP , prot, -__builtin_return_address (0 )); -if (!addr) + cpu_addr =dma_common_contiguous_remap (page, alloc_size, +VM_USERMAP , prot,__builtin_return_address (0 )); +if (!cpu_addr) goto out_unmap;if (!coherent) -arch_dma_prep_coherent (page, iosize); +arch_dma_prep_coherent (page, size); }else { - addr =page_address (page); + cpu_addr =page_address (page); } -memset (addr,0 , size); -return addr; +memset (cpu_addr,0 , alloc_size); +return cpu_addr; out_unmap: -__iommu_dma_unmap (dev, *handle, iosize); +__iommu_dma_unmap (dev, *handle, size); out_free_pages: -if (!dma_release_from_contiguous (dev, page, size >>PAGE_SHIFT )) -__free_pages (page,get_order (size)); +if (!dma_release_from_contiguous (dev, page, alloc_size >>PAGE_SHIFT )) +__free_pages (page,get_order (alloc_size));return NULL ; }
Signature · 1 line
2.20.1
PATCH 17/23 iommu/dma: Refactor iommu_dma_alloc, part 2
In reply to Christoph HellwigAll the logic in iommu_dma_alloc that deals with page allocation from the CMA or page allocators can be split into a self-contained helper, and we can than map the result of that or the atomic pool allocation with the iommu later. This also allows reusing __iommu_dma_free to tear down the allocations and MMU mappings when the IOMMU mapping fails.
Based on a patch from Robin Murphy.
--- drivers/iommu/dma-iommu.c | 65 +++++++++++++++++++++------------------ 1 file changed, 35 insertions(+), 30 deletions(-)
drivers/iommu/dma-iommu.c +35 -30
@@ -972,35 +972,14 @@ static void iommu_dma_free(struct device *dev, size_t size, void *cpu_addr,__iommu_dma_free (dev, size, cpu_addr); } -static void *iommu_dma_alloc (struct device *dev,size_t size, -dma_addr_t *handle,gfp_t gfp,unsigned long attrs) +static void *iommu_dma_alloc_pages (struct device *dev,size_t size, +struct page **pagep,gfp_t gfp,unsigned long attrs) {bool coherent =dev_is_dma_coherent (dev); -int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs);size_t alloc_size =PAGE_ALIGN (size);struct page *page =NULL ;void *cpu_addr; - gfp |= __GFP_ZERO; - -if (gfpflags_allow_blocking (gfp) && - !(attrs &DMA_ATTR_FORCE_CONTIGUOUS )) -return iommu_dma_alloc_remap (dev, size, handle, gfp, attrs); - -if (!gfpflags_allow_blocking (gfp) && !coherent) { - cpu_addr =dma_alloc_from_pool (alloc_size, &page, gfp); -if (!cpu_addr) -return NULL ; - - *handle =__iommu_dma_map (dev,page_to_phys (page), size, - ioprot); -if (*handle ==DMA_MAPPING_ERROR ) { -dma_free_from_pool (cpu_addr, alloc_size); -return NULL ; - } -return cpu_addr; - } -if (gfpflags_allow_blocking (gfp)) page = dma_alloc_from_contiguous(dev, alloc_size >> PAGE_SHIFT, get_order(alloc_size),
@@ -1010,33 +989,59 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,if (!page)return NULL ; - *handle =__iommu_dma_map (dev,page_to_phys (page), size, ioprot); -if (*handle ==DMA_MAPPING_ERROR ) - goto out_free_pages; -if (!coherent ||PageHighMem (page)) {pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); cpu_addr =dma_common_contiguous_remap (page, alloc_size,VM_USERMAP , prot,__builtin_return_address (0 ));if (!cpu_addr) - goto out_unmap; + goto out_free_pages;if (!coherent)arch_dma_prep_coherent (page, size); }else { cpu_addr =page_address (page); } + + *pagep = page;memset (cpu_addr,0 , alloc_size);return cpu_addr; -out_unmap: -__iommu_dma_unmap (dev, *handle, size); out_free_pages:if (!dma_release_from_contiguous (dev, page, alloc_size >>PAGE_SHIFT ))__free_pages (page,get_order (alloc_size));return NULL ; } +static void *iommu_dma_alloc (struct device *dev,size_t size, +dma_addr_t *handle,gfp_t gfp,unsigned long attrs) +{ +bool coherent =dev_is_dma_coherent (dev); +int ioprot =dma_info_to_prot (DMA_BIDIRECTIONAL , coherent, attrs); +struct page *page =NULL ; +void *cpu_addr; + + gfp |= __GFP_ZERO; + +if (gfpflags_allow_blocking (gfp) && + !(attrs &DMA_ATTR_FORCE_CONTIGUOUS )) +return iommu_dma_alloc_remap (dev, size, handle, gfp, attrs); + +if (!gfpflags_allow_blocking (gfp) && !coherent) + cpu_addr =dma_alloc_from_pool (PAGE_ALIGN (size), &page, gfp); +else + cpu_addr =iommu_dma_alloc_pages (dev, size, &page, gfp, attrs); +if (!cpu_addr) +return NULL ; + + *handle =__iommu_dma_map (dev,page_to_phys (page), size, ioprot); +if (*handle ==DMA_MAPPING_ERROR ) { +__iommu_dma_free (dev, size, cpu_addr); +return NULL ; + } + +return cpu_addr; +} +static int __iommu_dma_mmap_pfn (struct vm_area_struct *vma,unsigned long pfn,size_t size) {
Signature · 1 line
2.20.1
PATCH 18/23 iommu/dma: Refactor iommu_dma_get_sgtable
In reply to Christoph HellwigInline __iommu_dma_get_sgtable_page into the main function, and use the fact that __iommu_dma_get_pages return NULL for remapped contigous allocations to simplify the code flow a bit.
--- drivers/iommu/dma-iommu.c | 45 +++++++++++++++------------------------ 1 file changed, 17 insertions(+), 28 deletions(-)
drivers/iommu/dma-iommu.c +17 -28
@@ -1097,42 +1097,31 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma,return __iommu_dma_mmap (pages, size, vma); } -static int __iommu_dma_get_sgtable_page (struct sg_table *sgt,struct page *page, -size_t size) -{ -int ret =sg_alloc_table (sgt,1 ,GFP_KERNEL ); - -if (!ret) -sg_set_page (sgt->sgl, page,PAGE_ALIGN (size),0 ); -return ret; -} -static int iommu_dma_get_sgtable (struct device *dev,struct sg_table *sgt,void *cpu_addr,dma_addr_t dma_addr,size_t size,unsigned long attrs) { -unsigned int count =PAGE_ALIGN (size) >>PAGE_SHIFT ; -struct page **pages; +struct page *page; +int ret; -if (!is_vmalloc_addr (cpu_addr)) { -struct page *page =virt_to_page (cpu_addr); -return __iommu_dma_get_sgtable_page (sgt, page, size); - } +if (is_vmalloc_addr (cpu_addr)) { +struct page **pages =__iommu_dma_get_pages (cpu_addr); -if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -/* -* DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, -* hence in the vmalloc space. -*/ -struct page *page =vmalloc_to_page (cpu_addr); -return __iommu_dma_get_sgtable_page (sgt, page, size); +if (pages) { +return sg_alloc_table_from_pages (sgt, pages, +PAGE_ALIGN (size) >>PAGE_SHIFT , +0 , size,GFP_KERNEL ); + } + + page =vmalloc_to_page (cpu_addr); + }else { + page =virt_to_page (cpu_addr); } - pages =__iommu_dma_get_pages (cpu_addr); -if (!pages) -return -ENXIO ; -return sg_alloc_table_from_pages (sgt, pages, count,0 , size, -GFP_KERNEL ); + ret =sg_alloc_table (sgt,1 ,GFP_KERNEL ); +if (!ret) +sg_set_page (sgt->sgl, page,PAGE_ALIGN (size),0 ); +return ret; }static const struct dma_map_ops iommu_dma_ops = {
Signature · 1 line
2.20.1
PATCH 19/23 iommu/dma: Refactor iommu_dma_mmap
In reply to Christoph HellwigInline __iommu_dma_mmap and __iommu_dma_mmap_pfn into the main function, and use the fact that __iommu_dma_get_pages return NULL for remapped contigous allocations to simplify the code flow a bit.
--- drivers/iommu/dma-iommu.c | 60 +++++++-------------------------------- 1 file changed, 11 insertions(+), 49 deletions(-)
drivers/iommu/dma-iommu.c +11 -49
@@ -650,21 +650,6 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size,return NULL ; } -/** -* __iommu_dma_mmap - Map a buffer into provided user VMA -* @pages: Array representing buffer from __iommu_dma_alloc() -* @size: Size of buffer in bytes -* @vma: VMA describing requested userspace mapping -* -* Maps the pages of the buffer in @pages into @vma. The caller is responsible -* for verifying the correct size and protection of @vma beforehand. -*/ -static int __iommu_dma_mmap (struct page **pages,size_t size, -struct vm_area_struct *vma) -{ -return vm_map_pages (vma, pages,PAGE_ALIGN (size) >>PAGE_SHIFT ); -} -static void iommu_dma_sync_single_for_cpu (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir) {
@@ -1042,31 +1027,13 @@ static void *iommu_dma_alloc(struct device *dev, size_t size,return cpu_addr; } -static int __iommu_dma_mmap_pfn (struct vm_area_struct *vma, -unsigned long pfn,size_t size) -{ -int ret = -ENXIO ; -unsigned long nr_vma_pages =vma_pages (vma); -unsigned long nr_pages =PAGE_ALIGN (size) >>PAGE_SHIFT ; -unsigned long off = vma->vm_pgoff; - -if (off < nr_pages && nr_vma_pages <= (nr_pages - off)) { - ret =remap_pfn_range (vma, vma->vm_start, - pfn + off, - vma->vm_end - vma->vm_start, - vma->vm_page_prot); - } - -return ret; -} -static int iommu_dma_mmap (struct device *dev,struct vm_area_struct *vma,void *cpu_addr,dma_addr_t dma_addr,size_t size,unsigned long attrs) {unsigned long nr_pages =PAGE_ALIGN (size) >>PAGE_SHIFT ;unsigned long off = vma->vm_pgoff; -struct page **pages; +unsigned long pfn;int ret; vma->vm_page_prot =arch_dma_mmap_pgprot (dev, vma->vm_page_prot, attrs);
@@ -1077,24 +1044,19 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma,if (off >= nr_pages ||vma_pages (vma) > nr_pages - off)return -ENXIO ; -if (!is_vmalloc_addr (cpu_addr)) { -unsigned long pfn =page_to_pfn (virt_to_page (cpu_addr)); -return __iommu_dma_mmap_pfn (vma, pfn, size); - } +if (is_vmalloc_addr (cpu_addr)) { +struct page **pages =__iommu_dma_get_pages (cpu_addr); -if (attrs &DMA_ATTR_FORCE_CONTIGUOUS ) { -/* -* DMA_ATTR_FORCE_CONTIGUOUS allocations are always remapped, -* hence in the vmalloc space. -*/ -unsigned long pfn =vmalloc_to_pfn (cpu_addr); -return __iommu_dma_mmap_pfn (vma, pfn, size); +if (pages) +return vm_map_pages (vma, pages, nr_pages); + pfn =vmalloc_to_pfn (cpu_addr); + }else { + pfn =page_to_pfn (virt_to_page (cpu_addr)); } - pages =__iommu_dma_get_pages (cpu_addr); -if (!pages) -return -ENXIO ; -return __iommu_dma_mmap (pages, size, vma); +return remap_pfn_range (vma, vma->vm_start, pfn + off, + vma->vm_end - vma->vm_start, + vma->vm_page_prot); } static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt,
Signature · 1 line
2.20.1
PATCH 20/23 iommu/dma: Don't depend on CONFIG_DMA_DIRECT_REMAP
In reply to Christoph HellwigFor entirely dma coherent architectures there is no requirement to ever remap dma coherent allocation. Move all the remap and pool code under IS_ENABLED() checks and drop the Kconfig dependency.
--- drivers/iommu/Kconfig | 1 - drivers/iommu/dma-iommu.c | 16 +++++++++------- 2 files changed, 9 insertions(+), 8 deletions(-)
drivers/iommu/Kconfig +0 -1
@@ -97,7 +97,6 @@ config IOMMU_DMA select IOMMU_IOVA select IRQ_MSI_IOMMU select NEED_SG_DMA_LENGTH - depends on DMA_DIRECT_REMAP config FSL_PAMU bool "Freescale IOMMU support"
drivers/iommu/dma-iommu.c +9 -7
@@ -927,10 +927,11 @@ static void __iommu_dma_free(struct device *dev, size_t size, void *cpu_addr)struct page *page =NULL , **pages =NULL ;/* Non-coherent atomic allocation? Easy */ -if (dma_free_from_pool (cpu_addr, alloc_size)) +if (IS_ENABLED (CONFIG_DMA_DIRECT_REMAP ) && +dma_free_from_pool (cpu_addr, alloc_size))return ; -if (is_vmalloc_addr (cpu_addr)) { +if (IS_ENABLED (CONFIG_DMA_REMAP ) &&is_vmalloc_addr (cpu_addr)) {/* * If it the address is remapped, then it's either non-coherent * or highmem CMA, or an iommu_dma_alloc_remap() construction.
@@ -974,7 +975,7 @@ static void *iommu_dma_alloc_pages(struct device *dev, size_t size,if (!page)return NULL ; -if (!coherent ||PageHighMem (page)) { +if (IS_ENABLED (CONFIG_DMA_REMAP ) && (!coherent ||PageHighMem (page))) {pgprot_t prot =arch_dma_mmap_pgprot (dev,PAGE_KERNEL , attrs); cpu_addr = dma_common_contiguous_remap(page, alloc_size,
@@ -1007,11 +1008,12 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, gfp |= __GFP_ZERO; -if (gfpflags_allow_blocking (gfp) && +if (IS_ENABLED (CONFIG_DMA_REMAP ) &&gfpflags_allow_blocking (gfp) && !(attrs &DMA_ATTR_FORCE_CONTIGUOUS ))return iommu_dma_alloc_remap (dev, size, handle, gfp, attrs); -if (!gfpflags_allow_blocking (gfp) && !coherent) +if (IS_ENABLED (CONFIG_DMA_DIRECT_REMAP ) && + !gfpflags_allow_blocking (gfp) && !coherent) cpu_addr =dma_alloc_from_pool (PAGE_ALIGN (size), &page, gfp);else cpu_addr =iommu_dma_alloc_pages (dev, size, &page, gfp, attrs);
@@ -1044,7 +1046,7 @@ static int iommu_dma_mmap(struct device *dev, struct vm_area_struct *vma,if (off >= nr_pages ||vma_pages (vma) > nr_pages - off)return -ENXIO ; -if (is_vmalloc_addr (cpu_addr)) { +if (IS_ENABLED (CONFIG_DMA_REMAP ) &&is_vmalloc_addr (cpu_addr)) {struct page **pages =__iommu_dma_get_pages (cpu_addr);if (pages)
@@ -1066,7 +1068,7 @@ static int iommu_dma_get_sgtable(struct device *dev, struct sg_table *sgt,struct page *page;int ret; -if (is_vmalloc_addr (cpu_addr)) { +if (IS_ENABLED (CONFIG_DMA_REMAP ) &&is_vmalloc_addr (cpu_addr)) {struct page **pages =__iommu_dma_get_pages (cpu_addr);if (pages) {
Signature · 1 line
2.20.1
PATCH 21/23 iommu/dma: Switch copyright boilerplace to SPDX
In reply to Christoph Hellwig--- drivers/iommu/dma-iommu.c | 13 +------------ include/linux/dma-iommu.h | 13 +------------ 2 files changed, 2 insertions(+), 24 deletions(-)
drivers/iommu/dma-iommu.c +1 -12
@@ -1,3 +1,4 @@ +// SPDX-License-Identifier: GPL-2.0 /* * A fairly generic DMA-API to IOMMU-API glue layer. *
@@ -5,18 +6,6 @@* * based in part on arch/arm/mm/dma-mapping.c: * Copyright (C) 2000-2004 Russell King -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 as -* published by the Free Software Foundation. -* -* This program is distributed in the hope that it will be useful, -* but WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -* GNU General Public License for more details. -* -* You should have received a copy of the GNU General Public License -* along with this program. If not, see <http://www.gnu.org/licenses/>. */ #include <linux/acpi_iort.h>
include/linux/dma-iommu.h +1 -12
@@ -1,17 +1,6 @@ +/* SPDX-License-Identifier: GPL-2.0 */ /* * Copyright (C) 2014-2015 ARM Ltd. -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 as -* published by the Free Software Foundation. -* -* This program is distributed in the hope that it will be useful, -* but WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -* GNU General Public License for more details. -* -* You should have received a copy of the GNU General Public License -* along with this program. If not, see <http://www.gnu.org/licenses/>. */ #ifndef __DMA_IOMMU_H#define __DMA_IOMMU_H
Signature · 1 line
2.20.1
PATCH 22/23 arm64: switch copyright boilerplace to SPDX in dma-mapping.c
In reply to Christoph Hellwig--- arch/arm64/mm/dma-mapping.c | 15 +-------------- 1 file changed, 1 insertion(+), 14 deletions(-)
arch/arm64/mm/dma-mapping.c +1 -14
@@ -1,20 +1,7 @@ +// SPDX-License-Identifier: GPL-2.0 /* -* SWIOTLB-based DMA API implementation -* * Copyright (C) 2012 ARM Ltd. * Author: Catalin Marinas <[email protected]> -* -* This program is free software; you can redistribute it and/or modify -* it under the terms of the GNU General Public License version 2 as -* published by the Free Software Foundation. -* -* This program is distributed in the hope that it will be useful, -* but WITHOUT ANY WARRANTY; without even the implied warranty of -* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -* GNU General Public License for more details. -* -* You should have received a copy of the GNU General Public License -* along with this program. If not, see <http://www.gnu.org/licenses/>. */ #include <linux/gfp.h>
Signature · 1 line
2.20.1
PATCH 23/23 arm64: trim includes in dma-mapping.c
In reply to Christoph HellwigWith most of the previous functionality now elsewhere a lot of the headers included in this file are not needed.
--- arch/arm64/mm/dma-mapping.c | 10 ---------- 1 file changed, 10 deletions(-)
arch/arm64/mm/dma-mapping.c +0 -10
@@ -5,19 +5,9 @@*/ #include <linux/gfp.h> -#include <linux/acpi.h> -#include <linux/memblock.h> #include <linux/cache.h> -#include <linux/export.h> -#include <linux/slab.h> -#include <linux/genalloc.h> -#include <linux/dma-direct.h> #include <linux/dma-noncoherent.h> -#include <linux/dma-contiguous.h> #include <linux/dma-iommu.h> -#include <linux/vmalloc.h> -#include <linux/swiotlb.h> -#include <linux/pci.h> #include <asm/cacheflush.h>
Signature · 1 line
2.20.1
Robin Murphy
maintainerQuoted 9 lines · Christoph Hellwig
Hi Robin and Joerg,
I think we are finally ready for the generic dma-iommu series. I have various DMA API changes pending, and Tom has patches ready to convert the AMD and Intel iommu drivers over to it. I'd love to have this in a stable branch shared between the dma-mapping and iommu trees the day after rc2 is released. I volunteer to create the branch, but I'm fine with it living in the iommu tree as well.
Yup, I'd also like to see this in -next ASAP in the hope that board farms and maybe even real users can start chewing on it. Thanks for persevering :)
Robin.
Quoted 30 lines
A git tree is also available at:
git://git.infradead.org/users/hch/misc.gitdma-iommu-ops.6Gitweb:
http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dma-iommu-ops.6
Changes since v5:
- rebased to latest linus tree and dropped a patch now merged
- remove the now pointless__dma_iommu_mmapfunction
- restore a cleanup from Robin that would have caused a conflict in last merge winow, but is fine nowChanges since v4:
- rebased to 5.2-rc1Changes since v3:
- fold the separate patch to refactor mmap bounds checking
- don't warn on not finding avm_area
- improve a commit log
- refactor__dma_iommu_freea little differently
- remove a minor MSI map cleanup to avoid a conflict with the "Splitiommu_dma_map_msi_msg" seriesChanges since v2:
- address various review comments and include patches from Robin
Quoted 10 lines
Changes since v1:
- only include other headers indma-iommu.hifCONFIG_DMA_IOMMUis enabled
- keep using a scatterlist iniommu_dma_alloc
- split out mmap/sgtable fixes and move them early in the series
- updated a few commit logs
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu
_______________________________________________
iommu mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/iommu