dma-mapping: migrate to physical address-based API
Subsystems (16)
cover dma-mapping: migrate to physical address-based API
Changelog:
v1:
* Added new DMA_ATTR_MMIO attribute to indicate PCI_P2PDMA_MAP_THRU_HOST_BRIDGE path.
* Rewrote dma_map_* functions to use thus new attribute
v0: https://lore.kernel.org/all/[email protected]/
------------------------------------------------------------------------
This series refactors the DMA mapping to use physical addresses as the primary interface instead of page+offset parameters. This change aligns the DMA API with the underlying hardware reality where DMA operations work with physical addresses, not page structures.
The series maintains export symbol backward compatibility by keeping the old page-based API as wrapper functions around the new physical address-based implementations.
Thanks
Leon Romanovsky (16):
dma-mapping: introduce new DMA attribute to indicate MMIO memory iommu/dma: handle MMIO path in dma_iova_link
dma-debug: refactor to use physical addresses for page mapping
dma-mapping: rename trace_dma_*map_page to trace_dma_*map_phys iommu/dma: rename iommu_dma_*map_page to iommu_dma_*map_phys iommu/dma: extend iommu_dma_*map_phys API to handle MMIO memory
dma-mapping: convert dma_direct_*map_page to be phys_addr_t based
kmsan: convert kmsan_handle_dma to use physical addresses
dma-mapping: handle MMIO flow in dma_map|unmap_page
xen: swiotlb: Open code map_resource callback
dma-mapping: export new dma_*map_phys() interface mm/hmm: migrate to physical address-based DMA mapping API mm/hmm: properly take MMIO path
block-dma: migrate to dma_map_phys instead of map_page
block-dma: properly take MMIO path
nvme-pci: unmap MMIO pages with appropriate interface
Documentation/core-api/dma-api.rst | 4 +- Documentation/core-api/dma-attributes.rst | 7 ++ arch/powerpc/kernel/dma-iommu.c | 4 +- block/blk-mq-dma.c | 15 ++- drivers/iommu/dma-iommu.c | 69 +++++++------ drivers/nvme/host/pci.c | 18 +++- drivers/virtio/virtio_ring.c | 4 +- drivers/xen/swiotlb-xen.c | 21 +++- include/linux/blk-mq-dma.h | 6 +- include/linux/blk_types.h | 2 + include/linux/dma-direct.h | 2 - include/linux/dma-map-ops.h | 8 +- include/linux/dma-mapping.h | 27 +++++ include/linux/iommu-dma.h | 11 +-- include/linux/kmsan.h | 12 ++- include/trace/events/dma.h | 9 +- kernel/dma/debug.c | 71 ++++--------- kernel/dma/debug.h | 37 ++----- kernel/dma/direct.c | 22 +---- kernel/dma/direct.h | 50 ++++++---- kernel/dma/mapping.c | 115 +++++++++++++--------- kernel/dma/ops_helpers.c | 6 +- mm/hmm.c | 19 ++-- mm/kmsan/hooks.c | 36 +++++-- rust/kernel/dma.rs | 3 + tools/virtio/linux/kmsan.h | 2 +- 26 files changed, 320 insertions(+), 260 deletions(-)
Signature · 1 line
2.50.1
PATCH 2/16 iommu/dma: handle MMIO path in dma_iova_link
From: Leon Romanovsky <[email protected]>
Make sure that CPU is not synced if MMIO path is taken.
--- drivers/iommu/dma-iommu.c | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-)
drivers/iommu/dma-iommu.c +16 -5
@@ -1837,13 +1837,20 @@ static int __dma_iova_link(struct device *dev, dma_addr_t addr, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { - bool coherent = dev_is_dma_coherent(dev); + int prot; -if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC )) -arch_sync_dma_for_device (phys, size, dir); +if (attrs &DMA_ATTR_MMIO ) + prot =dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO ; +else { +bool coherent =dev_is_dma_coherent (dev); + +if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC )) +arch_sync_dma_for_device (phys, size, dir); + prot =dma_info_to_prot (dir, coherent, attrs); + }return iommu_map_nosync (iommu_get_dma_domain (dev), addr, phys, size, -dma_info_to_prot (dir, coherent, attrs),GFP_ATOMIC ); + prot,GFP_ATOMIC ); } static int iommu_dma_iova_bounce_and_link(struct device *dev, dma_addr_t addr,
@@ -1949,9 +1956,13 @@ int dma_iova_link(struct device *dev, struct dma_iova_state *state,return -EIO ;if (dev_use_swiotlb (dev, size, dir) && -iova_unaligned (iovad, phys, size)) +iova_unaligned (iovad, phys, size)) { +if (attrs &DMA_ATTR_MMIO ) +return -EPERM ; +return iommu_dma_iova_link_swiotlb (dev, state, phys, offset, size, dir, attrs); + } return __dma_iova_link(dev, state->addr + offset - iova_start_pad, phys - iova_start_pad,
Signature · 1 line
2.50.1
PATCH 3/16 dma-debug: refactor to use physical addresses for page mapping
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert the DMA debug infrastructure from page-based to physical address-based mapping as a preparation to rely on physical address for DMA mapping routines.
The refactoring renames debug_dma_map_page() to debug_dma_map_phys() and changes its signature to accept a phys_addr_t parameter instead of struct page and offset. Similarly, debug_dma_unmap_page() becomes debug_dma_unmap_phys(). A new dma_debug_phy type is introduced to distinguish physical address mappings from other debug entry types. All callers throughout the codebase are updated to pass physical addresses directly, eliminating the need for page-to-physical conversion in the debug layer.
This refactoring eliminates the need to convert between page pointers and physical addresses in the debug layer, making the code more efficient and consistent with the DMA mapping API's physical address focus.
--- Documentation/core-api/dma-api.rst | 4 ++-- kernel/dma/debug.c | 28 +++++++++++++++++----------- kernel/dma/debug.h | 16 +++++++--------- kernel/dma/mapping.c | 15 ++++++++------- 4 files changed, 34 insertions(+), 29 deletions(-)
Documentation/core-api/dma-api.rst +2 -2
@@ -761,7 +761,7 @@ example warning message may look like this:: [<ffffffff80235177>] find_busiest_group+0x207/0x8a0 [<ffffffff8064784f>] _spin_lock_irqsave+0x1f/0x50 [<ffffffff803c7ea3>] check_unmap+0x203/0x490 - [<ffffffff803c8259>] debug_dma_unmap_page+0x49/0x50 + [<ffffffff803c8259>] debug_dma_unmap_phys+0x49/0x50 [<ffffffff80485f26>] nv_tx_done_optimized+0xc6/0x2c0 [<ffffffff80486c13>] nv_nic_irq_optimized+0x73/0x2b0 [<ffffffff8026df84>] handle_IRQ_event+0x34/0x70
@@ -855,7 +855,7 @@ that a driver may be leaking mappings. dma-debug interface debug_dma_mapping_error() to debug drivers that fail to check DMA mapping errors on addresses returned by dma_map_single() and dma_map_page() interfaces. This interface clears a flag set by -debug_dma_map_page() to indicate that dma_mapping_error() has been called by +debug_dma_map_phys() to indicate that dma_mapping_error() has been called by the driver. When driver does unmap, debug_dma_unmap() checks the flag and if this flag is still set, prints warning message that includes call trace that leads up to the unmap. This interface can be called from dma_mapping_error()
kernel/dma/debug.c +17 -11
@@ -39,6 +39,7 @@ enum { dma_debug_sg, dma_debug_coherent, dma_debug_resource, + dma_debug_phy, }; enum map_err_types {
@@ -141,6 +142,7 @@ static const char *type2name[] = { [dma_debug_sg] = "scatter-gather", [dma_debug_coherent] = "coherent", [dma_debug_resource] = "resource", + [dma_debug_phy] = "phy", };static const char *dir2name[] = {
@@ -1201,9 +1203,8 @@ void debug_dma_map_single(struct device *dev, const void *addr, }EXPORT_SYMBOL (debug_dma_map_single); -void debug_dma_map_page (struct device *dev,struct page *page,size_t offset, -size_t size,int direction,dma_addr_t dma_addr, -unsigned long attrs) +void debug_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +int direction,dma_addr_t dma_addr,unsigned long attrs) {struct dma_debug_entry *entry;
@@ -1218,19 +1219,24 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,return ; entry->dev = dev; - entry->type = dma_debug_single; - entry->paddr =page_to_phys (page) + offset; + entry->type = dma_debug_phy; + entry->paddr = phys; entry->dev_addr = dma_addr; entry->size = size; entry->direction = direction; entry->map_err_type =MAP_ERR_NOT_CHECKED ; -check_for_stack (dev, page, offset); +if (!(attrs &DMA_ATTR_MMIO )) { +struct page *page =phys_to_page (phys); +size_t offset =offset_in_page (page); -if (!PageHighMem (page)) { -void *addr =page_address (page) + offset; +check_for_stack (dev, page, offset); -check_for_illegal_area (dev, addr, size); +if (!PageHighMem (page)) { +void *addr =page_address (page) + offset; + +check_for_illegal_area (dev, addr, size); + } }add_dma_entry (entry, attrs);
@@ -1274,11 +1280,11 @@ void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr) }EXPORT_SYMBOL (debug_dma_mapping_error); -void debug_dma_unmap_page (struct device *dev,dma_addr_t dma_addr, +void debug_dma_unmap_phys (struct device *dev,dma_addr_t dma_addr,size_t size,int direction) {struct dma_debug_entry ref = { - .type = dma_debug_single, + .type = dma_debug_phy, .dev = dev, .dev_addr = dma_addr, .size = size,
kernel/dma/debug.h +7 -9
@@ -9,12 +9,11 @@#define _KERNEL_DMA_DEBUG_H#ifdef CONFIG_DMA_API_DEBUG -extern void debug_dma_map_page (struct device *dev,struct page *page, -size_t offset,size_t size, -int direction,dma_addr_t dma_addr, +extern void debug_dma_map_phys (struct device *dev,phys_addr_t phys, +size_t size,int direction,dma_addr_t dma_addr,unsigned long attrs); -extern void debug_dma_unmap_page (struct device *dev,dma_addr_t addr, +extern void debug_dma_unmap_phys (struct device *dev,dma_addr_t addr,size_t size,int direction); extern void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
@@ -55,14 +54,13 @@ extern void debug_dma_sync_sg_for_device(struct device *dev,struct scatterlist *sg, int nelems, int direction);#else /* CONFIG_DMA_API_DEBUG */ -static inline void debug_dma_map_page (struct device *dev,struct page *page, -size_t offset,size_t size, -int direction,dma_addr_t dma_addr, -unsigned long attrs) +static inline void debug_dma_map_phys (struct device *dev,phys_addr_t phys, +size_t size,int direction, +dma_addr_t dma_addr,unsigned long attrs) { } -static inline void debug_dma_unmap_page (struct device *dev,dma_addr_t addr, +static inline void debug_dma_unmap_phys (struct device *dev,dma_addr_t addr,size_t size,int direction) { }
kernel/dma/mapping.c +8 -7
@@ -157,6 +157,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, unsigned long attrs) {const struct dma_map_ops *ops =get_dma_ops (dev); +phys_addr_t phys =page_to_phys (page) + offset;dma_addr_t addr;BUG_ON (!valid_dma_direction (dir));
@@ -165,16 +166,15 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_page_direct (dev,page_to_phys (page) + offset + size)) +arch_dma_map_page_direct (dev, phys + size)) addr =dma_direct_map_page (dev, page, offset, size, dir, attrs);else if (use_dma_iommu (dev)) addr =iommu_dma_map_page (dev, page, offset, size, dir, attrs);else addr = ops->map_page (dev, page, offset, size, dir, attrs);kmsan_handle_dma (page, offset, size, dir); -trace_dma_map_page (dev,page_to_phys (page) + offset, addr, size, dir, - attrs); -debug_dma_map_page (dev, page, offset, size, dir, addr, attrs); +trace_dma_map_page (dev, phys, addr, size, dir, attrs); +debug_dma_map_phys (dev, phys, size, dir, addr, attrs);return addr; }
@@ -194,7 +194,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,else ops->unmap_page (dev, addr, size, dir, attrs);trace_dma_unmap_page (dev, addr, size, dir, attrs); -debug_dma_unmap_page (dev, addr, size, dir); +debug_dma_unmap_phys (dev, addr, size, dir); }EXPORT_SYMBOL (dma_unmap_page_attrs);
@@ -712,7 +712,8 @@ struct page *dma_alloc_pages(struct device *dev, size_t size,if (page) {trace_dma_alloc_pages (dev,page_to_virt (page), *dma_handle, size, dir, gfp,0 ); -debug_dma_map_page (dev, page,0 , size, dir, *dma_handle,0 ); +debug_dma_map_phys (dev,page_to_phys (page), size, dir, + *dma_handle,0 ); }else {trace_dma_alloc_pages (dev,NULL ,0 , size, dir, gfp,0 ); }
@@ -738,7 +739,7 @@ void dma_free_pages(struct device *dev, size_t size, struct page *page, dma_addr_t dma_handle, enum dma_data_direction dir) { trace_dma_free_pages(dev, page_to_virt(page), dma_handle, size, dir, 0); -debug_dma_unmap_page (dev, dma_handle, size, dir); +debug_dma_unmap_phys (dev, dma_handle, size, dir);__dma_free_pages (dev, size, page, dma_handle, dir); }EXPORT_SYMBOL_GPL (dma_free_pages);
Signature · 1 line
2.50.1
PATCH 1/16 dma-mapping: introduce new DMA attribute to indicate MMIO memory
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
This patch introduces the DMA_ATTR_MMIO attribute to mark DMA buffers that reside in memory-mapped I/O (MMIO) regions, such as device BARs exposed through the host bridge, which are accessible for peer-to-peer (P2P) DMA.
This attribute is especially useful for exporting device memory to other devices for DMA without CPU involvement, and avoids unnecessary or potentially detrimental CPU cache maintenance calls.
--- Documentation/core-api/dma-attributes.rst | 7 +++++++ include/linux/dma-mapping.h | 14 ++++++++++++++ include/trace/events/dma.h | 3 ++- rust/kernel/dma.rs | 3 +++ 4 files changed, 26 insertions(+), 1 deletion(-)
Documentation/core-api/dma-attributes.rst +7 -0
@@ -130,3 +130,10 @@ accesses to DMA buffers in both privileged "supervisor" and unprivileged subsystem that the buffer is fully accessible at the elevated privilege level (and ideally inaccessible or at least read-only at the lesser-privileged levels). + +DMA_ATTR_MMIO +------------- + +This attribute is especially useful for exporting device memory to other +devices for DMA without CPU involvement, and avoids unnecessary or +potentially detrimental CPU cache maintenance calls.
include/linux/dma-mapping.h +14 -0
@@ -58,6 +58,20 @@*/ #define DMA_ATTR_PRIVILEGED (1UL << 9) +/* +* DMA_ATTR_MMIO - Indicates memory-mapped I/O (MMIO) region for DMA mapping +* +* This attribute is used for MMIO memory regions that are exposed through +* the host bridge and are accessible for peer-to-peer (P2P) DMA. Memory +* marked with this attribute is not system RAM and may represent device +* BAR windows or peer-exposed memory. +* +* Typical usage is for mapping hardware memory BARs or exporting device +* memory to other devices for DMA without involving main system RAM. +* The attribute guarantees no CPU cache maintenance calls will be made. +*/ +#define DMA_ATTR_MMIO (1UL << 10) +/* * A dma_addr_t can hold any valid DMA or bus address for the platform. It can * be given to a device to use as a DMA source or target. It is specific to a
include/trace/events/dma.h +2 -1
@@ -31,7 +31,8 @@ TRACE_DEFINE_ENUM(DMA_NONE); { DMA_ATTR_FORCE_CONTIGUOUS, "FORCE_CONTIGUOUS" }, \ { DMA_ATTR_ALLOC_SINGLE_PAGES, "ALLOC_SINGLE_PAGES" }, \ { DMA_ATTR_NO_WARN, "NO_WARN" }, \ - { DMA_ATTR_PRIVILEGED, "PRIVILEGED" }) + { DMA_ATTR_PRIVILEGED, "PRIVILEGED" }, \ + { DMA_ATTR_MMIO, "MMIO" }) DECLARE_EVENT_CLASS(dma_map, TP_PROTO(struct device *dev, phys_addr_t phys_addr, dma_addr_t dma_addr,
rust/kernel/dma.rs +3 -0
@@ -242,6 +242,9 @@ pub mod attrs {/// Indicates that the buffer is fully accessible at an elevated privilege level (and /// ideally inaccessible or at least read-only at lesser-privileged levels). pub const DMA_ATTR_PRIVILEGED :Attrs =Attrs (bindings::DMA_ATTR_PRIVILEGED ); + +/// Indicates that the buffer is MMIO memory. +pub const DMA_ATTR_MMIO :Attrs =Attrs (bindings::DMA_ATTR_MMIO ); }/// An abstraction of the `dma_alloc_coherent` API.
Signature · 1 line
2.50.1
PATCH 4/16 dma-mapping: rename trace_dma_*map_page to trace_dma_*map_phys
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
As a preparation for following map_page -> map_phys API conversion, let's rename trace_dma_*map_page() to be trace_dma_*map_phys().
--- include/trace/events/dma.h | 4 ++-- kernel/dma/mapping.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
include/trace/events/dma.h +2 -2
@@ -72,7 +72,7 @@ DEFINE_EVENT(dma_map, name, \ size_t size, enum dma_data_direction dir, unsigned long attrs), \ TP_ARGS(dev, phys_addr, dma_addr, size, dir, attrs)) -DEFINE_MAP_EVENT(dma_map_page); +DEFINE_MAP_EVENT(dma_map_phys);DEFINE_MAP_EVENT (dma_map_resource); DECLARE_EVENT_CLASS(dma_unmap,
@@ -110,7 +110,7 @@ DEFINE_EVENT(dma_unmap, name, \ enum dma_data_direction dir, unsigned long attrs), \ TP_ARGS(dev, addr, size, dir, attrs)) -DEFINE_UNMAP_EVENT(dma_unmap_page); +DEFINE_UNMAP_EVENT(dma_unmap_phys);DEFINE_UNMAP_EVENT (dma_unmap_resource); DECLARE_EVENT_CLASS(dma_alloc_class,
kernel/dma/mapping.c +2 -2
@@ -173,7 +173,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,else addr = ops->map_page (dev, page, offset, size, dir, attrs);kmsan_handle_dma (page, offset, size, dir); -trace_dma_map_page (dev, phys, addr, size, dir, attrs); +trace_dma_map_phys (dev, phys, addr, size, dir, attrs);debug_dma_map_phys (dev, phys, size, dir, addr, attrs);return addr;
@@ -193,7 +193,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,iommu_dma_unmap_page (dev, addr, size, dir, attrs); else ops->unmap_page(dev, addr, size, dir, attrs); -trace_dma_unmap_page (dev, addr, size, dir, attrs); +trace_dma_unmap_phys (dev, addr, size, dir, attrs);debug_dma_unmap_phys (dev, addr, size, dir); }EXPORT_SYMBOL (dma_unmap_page_attrs);
Signature · 1 line
2.50.1
PATCH 6/16 iommu/dma: extend iommu_dma_*map_phys API to handle MMIO memory
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Combine iommu_dma_*map_phys with iommu_dma_*map_resource interfaces in order to allow single phys_addr_t flow.
--- drivers/iommu/dma-iommu.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
drivers/iommu/dma-iommu.c +16 -4
@@ -1193,12 +1193,17 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size,enum dma_data_direction dir,unsigned long attrs) { -bool coherent =dev_is_dma_coherent (dev); -int prot =dma_info_to_prot (dir, coherent, attrs);struct iommu_domain *domain =iommu_get_dma_domain (dev);struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad;dma_addr_t iova, dma_mask =dma_get_mask (dev); +bool coherent; +int prot; + +if (attrs &DMA_ATTR_MMIO ) +return __iommu_dma_map (dev, phys, size, +dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO , +dma_get_mask (dev));/* * If both the physical buffer start address and size are page aligned,
@@ -1211,6 +1216,9 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,return DMA_MAPPING_ERROR ; } + coherent =dev_is_dma_coherent (dev); + prot =dma_info_to_prot (dir, coherent, attrs); +if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC ))arch_sync_dma_for_device (phys, size, dir);
@@ -1223,10 +1231,14 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,void iommu_dma_unmap_phys (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir,unsigned long attrs) { -struct iommu_domain *domain =iommu_get_dma_domain (dev);phys_addr_t phys; - phys =iommu_iova_to_phys (domain, dma_handle); +if (attrs &DMA_ATTR_MMIO ) { +__iommu_dma_unmap (dev, dma_handle, size); +return ; + } + + phys =iommu_iova_to_phys (iommu_get_dma_domain (dev), dma_handle);if (WARN_ON (!phys))return ;
Signature · 1 line
2.50.1
PATCH 7/16 dma-mapping: convert dma_direct_*map_page to be phys_addr_t based
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert the DMA direct mapping functions to accept physical addresses directly instead of page+offset parameters. The functions were already operating on physical addresses internally, so this change eliminates the redundant page-to-physical conversion at the API boundary.
The functions dma_direct_map_page() and dma_direct_unmap_page() are renamed to dma_direct_map_phys() and dma_direct_unmap_phys() respectively, with their calling convention changed from (struct page *page, unsigned long offset) to (phys_addr_t phys).
Architecture-specific functions arch_dma_map_page_direct() and arch_dma_unmap_page_direct() are similarly renamed to arch_dma_map_phys_direct() and arch_dma_unmap_phys_direct().
The is_pci_p2pdma_page() checks are replaced with DMA_ATTR_MMIO checks to allow integration with dma_direct_map_resource and dma_direct_map_phys() is extended to support MMIO path either.
--- arch/powerpc/kernel/dma-iommu.c | 4 +-- include/linux/dma-map-ops.h | 8 +++--- kernel/dma/direct.c | 6 ++-- kernel/dma/direct.h | 50 ++++++++++++++++++++------------- kernel/dma/mapping.c | 8 +++--- 5 files changed, 44 insertions(+), 32 deletions(-)
arch/powerpc/kernel/dma-iommu.c +2 -2
@@ -14,7 +14,7 @@#define can_map_direct (dev, addr) \ ((dev)->bus_dma_limit >= phys_to_dma((dev), (addr))) -bool arch_dma_map_page_direct (struct device *dev,phys_addr_t addr) +bool arch_dma_map_phys_direct (struct device *dev,phys_addr_t addr) {if (likely (!dev->bus_dma_limit))return false;
@@ -24,7 +24,7 @@ bool arch_dma_map_page_direct(struct device *dev, phys_addr_t addr)#define is_direct_handle (dev, h) ((h) >= (dev)->archdata.dma_offset) -bool arch_dma_unmap_page_direct (struct device *dev,dma_addr_t dma_handle) +bool arch_dma_unmap_phys_direct (struct device *dev,dma_addr_t dma_handle) {if (likely (!dev->bus_dma_limit))return false;
include/linux/dma-map-ops.h +4 -4
@@ -392,15 +392,15 @@ void *arch_dma_set_uncached(void *addr, size_t size);void arch_dma_clear_uncached (void *addr,size_t size);#ifdef CONFIG_ARCH_HAS_DMA_MAP_DIRECT -bool arch_dma_map_page_direct (struct device *dev,phys_addr_t addr); -bool arch_dma_unmap_page_direct (struct device *dev,dma_addr_t dma_handle); +bool arch_dma_map_phys_direct (struct device *dev,phys_addr_t addr); +bool arch_dma_unmap_phys_direct (struct device *dev,dma_addr_t dma_handle);bool arch_dma_map_sg_direct (struct device *dev,struct scatterlist *sg,int nents);bool arch_dma_unmap_sg_direct (struct device *dev,struct scatterlist *sg,int nents);#else -#define arch_dma_map_page_direct (d, a) (false) -#define arch_dma_unmap_page_direct (d, a) (false) +#define arch_dma_map_phys_direct (d, a) (false) +#define arch_dma_unmap_phys_direct (d, a) (false)#define arch_dma_map_sg_direct (d, s, n) (false)#define arch_dma_unmap_sg_direct (d, s, n) (false)#endif
kernel/dma/direct.c +3 -3
@@ -453,7 +453,7 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,if (sg_dma_is_bus_address (sg))sg_dma_unmark_bus_address (sg);else -dma_direct_unmap_page (dev, sg->dma_address, +dma_direct_unmap_phys (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); } }
@@ -476,8 +476,8 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,*/ break ;case PCI_P2PDMA_MAP_NONE : - sg->dma_address =dma_direct_map_page (dev,sg_page (sg), - sg->offset, sg->length, dir, attrs); + sg->dma_address =dma_direct_map_phys (dev,sg_phys (sg), + sg->length, dir, attrs);if (sg->dma_address ==DMA_MAPPING_ERROR ) { ret = -EIO ; goto out_unmap;
kernel/dma/direct.h +31 -19
73 lines not on this page · raw message
kernel/dma/mapping.c +4 -4
@@ -166,8 +166,8 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_page_direct (dev, phys + size)) - addr =dma_direct_map_page (dev, page, offset, size, dir, attrs); +arch_dma_map_phys_direct (dev, phys + size)) + addr =dma_direct_map_phys (dev, phys, size, dir, attrs);else if (use_dma_iommu (dev)) addr =iommu_dma_map_phys (dev, phys, size, dir, attrs);else
@@ -187,8 +187,8 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,BUG_ON (!valid_dma_direction (dir));if (dma_map_direct (dev, ops) || -arch_dma_unmap_page_direct (dev, addr + size)) -dma_direct_unmap_page (dev, addr, size, dir, attrs); +arch_dma_unmap_phys_direct (dev, addr + size)) +dma_direct_unmap_phys (dev, addr, size, dir, attrs);else if (use_dma_iommu (dev))iommu_dma_unmap_phys (dev, addr, size, dir, attrs);else
Signature · 1 line
2.50.1
PATCH 5/16 iommu/dma: rename iommu_dma_*map_page to iommu_dma_*map_phys
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Rename the IOMMU DMA mapping functions to better reflect their actual calling convention. The functions iommu_dma_map_page() and iommu_dma_unmap_page() are renamed to iommu_dma_map_phys() and iommu_dma_unmap_phys() respectively, as they already operate on physical addresses rather than page structures.
The calling convention changes from accepting (struct page *page, unsigned long offset) to (phys_addr_t phys), which eliminates the need for page-to-physical address conversion within the functions. This renaming prepares for the broader DMA API conversion from page-based to physical address-based mapping throughout the kernel.
All callers are updated to pass physical addresses directly, including dma_map_page_attrs(), scatterlist mapping functions, and DMA page allocation helpers. The change simplifies the code by removing the page_to_phys() + offset calculation that was previously done inside the IOMMU functions.
--- drivers/iommu/dma-iommu.c | 14 ++++++-------- include/linux/iommu-dma.h | 7 +++---- kernel/dma/mapping.c | 4 ++-- kernel/dma/ops_helpers.c | 6 +++--- 4 files changed, 14 insertions(+), 17 deletions(-)
drivers/iommu/dma-iommu.c +6 -8
@@ -1190,11 +1190,9 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,return iova_offset (iovad, phys | size); } -dma_addr_t iommu_dma_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size,enum dma_data_direction dir, -unsigned long attrs) +dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs) { -phys_addr_t phys =page_to_phys (page) + offset;bool coherent =dev_is_dma_coherent (dev);int prot =dma_info_to_prot (dir, coherent, attrs);struct iommu_domain *domain =iommu_get_dma_domain (dev);
@@ -1222,7 +1220,7 @@ dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,return iova; } -void iommu_dma_unmap_page (struct device *dev,dma_addr_t dma_handle, +void iommu_dma_unmap_phys (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir,unsigned long attrs) {struct iommu_domain *domain =iommu_get_dma_domain (dev);
@@ -1341,7 +1339,7 @@ static void iommu_dma_unmap_sg_swiotlb(struct device *dev, struct scatterlist *sint i;for_each_sg (sg, s, nents, i) -iommu_dma_unmap_page (dev,sg_dma_address (s), +iommu_dma_unmap_phys (dev,sg_dma_address (s),sg_dma_len (s), dir, attrs); }
@@ -1354,8 +1352,8 @@ static int iommu_dma_map_sg_swiotlb(struct device *dev, struct scatterlist *sg,sg_dma_mark_swiotlb (sg);for_each_sg (sg, s, nents, i) { -sg_dma_address (s) =iommu_dma_map_page (dev,sg_page (s), - s->offset, s->length, dir, attrs); +sg_dma_address (s) =iommu_dma_map_phys (dev,sg_phys (s), + s->length, dir, attrs);if (sg_dma_address (s) ==DMA_MAPPING_ERROR ) goto out_unmap;sg_dma_len (s) = s->length;
include/linux/iommu-dma.h +3 -4
@@ -21,10 +21,9 @@ static inline bool use_dma_iommu(struct device *dev) }#endif /* CONFIG_IOMMU_DMA */ -dma_addr_t iommu_dma_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size,enum dma_data_direction dir, -unsigned long attrs); -void iommu_dma_unmap_page (struct device *dev,dma_addr_t dma_handle, +dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs); +void iommu_dma_unmap_phys (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir,unsigned long attrs);int iommu_dma_map_sg (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs);
kernel/dma/mapping.c +2 -2
@@ -169,7 +169,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, arch_dma_map_page_direct(dev, phys + size)) addr =dma_direct_map_page (dev, page, offset, size, dir, attrs); else if (use_dma_iommu(dev)) - addr =iommu_dma_map_page (dev, page, offset, size, dir, attrs); + addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); else addr = ops->map_page (dev, page, offset, size, dir, attrs);kmsan_handle_dma (page, offset, size, dir);
@@ -190,7 +190,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, arch_dma_unmap_page_direct(dev, addr + size))dma_direct_unmap_page (dev, addr, size, dir, attrs); else if (use_dma_iommu(dev)) - iommu_dma_unmap_page(dev, addr, size, dir, attrs); + iommu_dma_unmap_phys(dev, addr, size, dir, attrs); else ops->unmap_page(dev, addr, size, dir, attrs);trace_dma_unmap_phys (dev, addr, size, dir, attrs);
kernel/dma/ops_helpers.c +3 -3
@@ -72,8 +72,8 @@ struct page *dma_common_alloc_pages(struct device *dev, size_t size,return NULL ;if (use_dma_iommu (dev)) - *dma_handle =iommu_dma_map_page (dev, page,0 , size, dir, -DMA_ATTR_SKIP_CPU_SYNC ); + *dma_handle =iommu_dma_map_phys (dev,page_to_phys (page), size, + dir,DMA_ATTR_SKIP_CPU_SYNC );else *dma_handle = ops->map_page (dev, page,0 , size, dir,DMA_ATTR_SKIP_CPU_SYNC );
@@ -92,7 +92,7 @@ void dma_common_free_pages(struct device *dev, size_t size, struct page *page,const struct dma_map_ops *ops =get_dma_ops (dev);if (use_dma_iommu (dev)) -iommu_dma_unmap_page (dev, dma_handle, size, dir, +iommu_dma_unmap_phys (dev, dma_handle, size, dir,DMA_ATTR_SKIP_CPU_SYNC );else if (ops->unmap_page) ops->unmap_page(dev, dma_handle, size, dir,
Signature · 1 line
2.50.1
PATCH 9/16 dma-mapping: handle MMIO flow in dma_map|unmap_page
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Extend base DMA page API to handle MMIO flow.
--- kernel/dma/mapping.c | 24 ++++++++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-)
kernel/dma/mapping.c +20 -4
@@ -158,6 +158,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, {const struct dma_map_ops *ops =get_dma_ops (dev);phys_addr_t phys =page_to_phys (page) + offset; +bool is_mmio = attrs &DMA_ATTR_MMIO ;dma_addr_t addr;BUG_ON (!valid_dma_direction (dir));
@@ -166,12 +167,23 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_phys_direct (dev, phys + size)) + (!is_mmio &&arch_dma_map_phys_direct (dev, phys + size))) addr =dma_direct_map_phys (dev, phys, size, dir, attrs);else if (use_dma_iommu (dev)) addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); -else +else if (is_mmio) { +if (!ops->map_resource) +return DMA_MAPPING_ERROR ; + + addr = ops->map_resource (dev, phys, size, dir, attrs); + }else { +/* +* All platforms which implement .map_page() don't support +* non-struct page backed addresses. +*/ addr = ops->map_page (dev, page, offset, size, dir, attrs); + } +kmsan_handle_dma (phys, size, dir);trace_dma_map_phys (dev, phys, addr, size, dir, attrs);debug_dma_map_phys (dev, phys, size, dir, addr, attrs);
@@ -184,14 +196,18 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) {const struct dma_map_ops *ops =get_dma_ops (dev); +bool is_mmio = attrs &DMA_ATTR_MMIO ;BUG_ON (!valid_dma_direction (dir));if (dma_map_direct (dev, ops) || -arch_dma_unmap_phys_direct (dev, addr + size)) + (!is_mmio &&arch_dma_unmap_phys_direct (dev, addr + size)))dma_direct_unmap_phys (dev, addr, size, dir, attrs);else if (use_dma_iommu (dev))iommu_dma_unmap_phys (dev, addr, size, dir, attrs); -else +else if (is_mmio) { +if (ops->unmap_resource) + ops->unmap_resource (dev, addr, size, dir, attrs); + }else ops->unmap_page (dev, addr, size, dir, attrs);trace_dma_unmap_phys (dev, addr, size, dir, attrs);debug_dma_unmap_phys (dev, addr, size, dir);
Signature · 1 line
2.50.1
PATCH 8/16 kmsan: convert kmsan_handle_dma to use physical addresses
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert the KMSAN DMA handling function from page-based to physical address-based interface.
The refactoring renames kmsan_handle_dma() parameters from accepting (struct page *page, size_t offset, size_t size) to (phys_addr_t phys, size_t size). A PFN_VALID check is added to prevent KMSAN operations on non-page memory, preventing from non struct page backed address,
As part of this change, support for highmem addresses is implemented using kmap_local_page() to handle both lowmem and highmem regions properly. All callers throughout the codebase are updated to use the new phys_addr_t based interface.
--- drivers/virtio/virtio_ring.c | 4 ++-- include/linux/kmsan.h | 12 +++++++----- kernel/dma/mapping.c | 2 +- mm/kmsan/hooks.c | 36 +++++++++++++++++++++++++++++------- tools/virtio/linux/kmsan.h | 2 +- 5 files changed, 40 insertions(+), 16 deletions(-)
drivers/virtio/virtio_ring.c +2 -2
@@ -378,7 +378,7 @@ static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist* is initialized by the hardware. Explicitly check/unpoison it * depending on the direction. */ -kmsan_handle_dma (sg_page (sg), sg->offset, sg->length, direction); +kmsan_handle_dma (sg_phys (sg), sg->length, direction); *addr = (dma_addr_t )sg_phys (sg);return 0 ; }
@@ -3157,7 +3157,7 @@ dma_addr_t virtqueue_dma_map_single_attrs(struct virtqueue *_vq, void *ptr,struct vring_virtqueue *vq =to_vvq (_vq);if (!vq->use_dma_api) { -kmsan_handle_dma (virt_to_page (ptr),offset_in_page (ptr), size, dir); +kmsan_handle_dma (virt_to_phys (ptr), size, dir);return (dma_addr_t )virt_to_phys (ptr); }
include/linux/kmsan.h +7 -5
@@ -182,8 +182,7 @@ void kmsan_iounmap_page_range(unsigned long start, unsigned long end);/** * kmsan_handle_dma() - Handle a DMA data transfer. -* @page: first page of the buffer. -* @offset: offset of the buffer within the first page. +* @phys: physical address of the buffer. * @size: buffer size. * @dir: one of possible dma_data_direction values. *
@@ -191,8 +190,11 @@ void kmsan_iounmap_page_range(unsigned long start, unsigned long end);* * checks the buffer, if it is copied to device; * * initializes the buffer, if it is copied from device; * * does both, if this is a DMA_BIDIRECTIONAL transfer. +* +* The function handles page lookup internally and supports both lowmem +* and highmem addresses. */ -void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir);/**
@@ -372,8 +374,8 @@ static inline void kmsan_iounmap_page_range(unsigned long start, { } -static inline void kmsan_handle_dma (struct page *page,size_t offset, -size_t size,enum dma_data_direction dir) +static inline void kmsan_handle_dma (phys_addr_t phys,size_t size, +enum dma_data_direction dir) { }
kernel/dma/mapping.c +1 -1
@@ -172,7 +172,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); else addr = ops->map_page (dev, page, offset, size, dir, attrs); -kmsan_handle_dma (page, offset, size, dir); +kmsan_handle_dma (phys, size, dir);trace_dma_map_phys (dev, phys, addr, size, dir, attrs);debug_dma_map_phys (dev, phys, size, dir, addr, attrs);
mm/kmsan/hooks.c +29 -7
@@ -336,25 +336,48 @@ static void kmsan_handle_dma_page(const void *addr, size_t size, }/* Helper function to handle DMA data transfers. */ -void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) {u64 page_offset, to_go, addr; +struct page *page; +void *kaddr; -if (PageHighMem (page)) +if (!pfn_valid (PHYS_PFN (phys)))return ; - addr = (u64 )page_address (page) + offset; + + page =phys_to_page (phys); + page_offset =offset_in_page (phys); +/* * The kernel may occasionally give us adjacent DMA pages not belonging * to the same allocation. Process them separately to avoid triggering * internal KMSAN checks. */ while (size >0 ) { - page_offset =offset_in_page (addr); to_go =min (PAGE_SIZE - page_offset, (u64 )size); + +if (PageHighMem (page)) +/* Handle highmem pages using kmap */ + kaddr =kmap_local_page (page); +else +/* Lowmem pages can be accessed directly */ + kaddr =page_address (page); + + addr = (u64 )kaddr + page_offset;kmsan_handle_dma_page ((void *)addr, to_go, dir); - addr += to_go; + +if (PageHighMem (page)) +kunmap_local (page); + + phys += to_go; size -= to_go; + +/* Move to next page if needed */ +if (size >0 ) { + page =phys_to_page (phys); + page_offset =offset_in_page (phys); + } } }EXPORT_SYMBOL_GPL (kmsan_handle_dma);
@@ -366,8 +389,7 @@ void kmsan_handle_dma_sg(struct scatterlist *sg, int nents,int i;for_each_sg (sg, item, nents, i) -kmsan_handle_dma (sg_page (item), item->offset, item->length, - dir); +kmsan_handle_dma (sg_phys (item), item->length, dir); }/* Functions from kmsan-checks.h follow. */
tools/virtio/linux/kmsan.h +1 -1
@@ -4,7 +4,7 @@#include <linux/gfp.h> -inline void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +inline void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) { }
Signature · 1 line
2.50.1
PATCH 11/16 dma-mapping: export new dma_*map_phys() interface
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Introduce new DMA mapping functions dma_map_phys() and dma_unmap_phys() that operate directly on physical addresses instead of page+offset parameters. This provides a more efficient interface for drivers that already have physical addresses available.
The new functions are implemented as the primary mapping layer, with the existing dma_map_page_attrs() and dma_unmap_page_attrs() functions converted to simple wrappers around the phys-based implementations.
The old page-based API is preserved in mapping.c to ensure that existing code won't be affected by changing EXPORT_SYMBOL to EXPORT_SYMBOL_GPL variant for dma_*map_phys().
--- drivers/iommu/dma-iommu.c | 14 -------- include/linux/dma-direct.h | 2 -- include/linux/dma-mapping.h | 13 +++++++ include/linux/iommu-dma.h | 4 --- include/trace/events/dma.h | 2 -- kernel/dma/debug.c | 43 ----------------------- kernel/dma/debug.h | 21 ------------ kernel/dma/direct.c | 16 --------- kernel/dma/mapping.c | 68 ++++++++++++++++++++----------------- 9 files changed, 49 insertions(+), 134 deletions(-)
drivers/iommu/dma-iommu.c +0 -14
@@ -1556,20 +1556,6 @@ void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,__iommu_dma_unmap (dev, start, end - start); } -dma_addr_t iommu_dma_map_resource (struct device *dev,phys_addr_t phys, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -return __iommu_dma_map (dev, phys, size, -dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO , -dma_get_mask (dev)); -} - -void iommu_dma_unmap_resource (struct device *dev,dma_addr_t handle, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -__iommu_dma_unmap (dev, handle, size); -} -static void __iommu_dma_free (struct device *dev,size_t size,void *cpu_addr) {size_t alloc_size =PAGE_ALIGN (size);
include/linux/dma-direct.h +0 -2
@@ -149,7 +149,5 @@ void dma_direct_free_pages(struct device *dev, size_t size, struct page *page, dma_addr_t dma_addr, enum dma_data_direction dir);int dma_direct_supported (struct device *dev,u64 mask); -dma_addr_t dma_direct_map_resource (struct device *dev,phys_addr_t paddr, -size_t size,enum dma_data_direction dir,unsigned long attrs);#endif /* _LINUX_DMA_DIRECT_H */
include/linux/dma-mapping.h +13 -0
@@ -132,6 +132,10 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, unsigned long attrs);void dma_unmap_page_attrs (struct device *dev,dma_addr_t addr,size_t size,enum dma_data_direction dir,unsigned long attrs); +dma_addr_t dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs); +void dma_unmap_phys (struct device *dev,dma_addr_t addr,size_t size, +enum dma_data_direction dir,unsigned long attrs);unsigned int dma_map_sg_attrs (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs); void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg,
@@ -186,6 +190,15 @@ static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) { } +static inline dma_addr_t dma_map_phys (struct device *dev,phys_addr_t phys, +size_t size,enum dma_data_direction dir,unsigned long attrs) +{ +return DMA_MAPPING_ERROR ; +} +static inline void dma_unmap_phys (struct device *dev,dma_addr_t addr, +size_t size,enum dma_data_direction dir,unsigned long attrs) +{ +}static inline unsigned int dma_map_sg_attrs (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs)
include/linux/iommu-dma.h +0 -4
@@ -42,10 +42,6 @@ size_t iommu_dma_opt_mapping_size(void);size_t iommu_dma_max_mapping_size (struct device *dev);void iommu_dma_free (struct device *dev,size_t size,void *cpu_addr,dma_addr_t handle,unsigned long attrs); -dma_addr_t iommu_dma_map_resource (struct device *dev,phys_addr_t phys, -size_t size,enum dma_data_direction dir,unsigned long attrs); -void iommu_dma_unmap_resource (struct device *dev,dma_addr_t handle, -size_t size,enum dma_data_direction dir,unsigned long attrs);struct sg_table *iommu_dma_alloc_noncontiguous (struct device *dev,size_t size,enum dma_data_direction dir,gfp_t gfp,unsigned long attrs); void iommu_dma_free_noncontiguous(struct device *dev, size_t size,
include/trace/events/dma.h +0 -2
@@ -73,7 +73,6 @@ DEFINE_EVENT(dma_map, name, \ TP_ARGS(dev, phys_addr, dma_addr, size, dir, attrs))DEFINE_MAP_EVENT (dma_map_phys); -DEFINE_MAP_EVENT (dma_map_resource); DECLARE_EVENT_CLASS(dma_unmap, TP_PROTO(struct device *dev, dma_addr_t addr, size_t size,
@@ -111,7 +110,6 @@ DEFINE_EVENT(dma_unmap, name, \ TP_ARGS(dev, addr, size, dir, attrs))DEFINE_UNMAP_EVENT (dma_unmap_phys); -DEFINE_UNMAP_EVENT (dma_unmap_resource); DECLARE_EVENT_CLASS(dma_alloc_class, TP_PROTO(struct device *dev, void *virt_addr, dma_addr_t dma_addr,
kernel/dma/debug.c +0 -43
61 lines not on this page · raw message
kernel/dma/debug.h +0 -21
@@ -30,14 +30,6 @@ extern void debug_dma_alloc_coherent(struct device *dev, size_t size,extern void debug_dma_free_coherent (struct device *dev,size_t size,void *virt,dma_addr_t addr); -extern void debug_dma_map_resource (struct device *dev,phys_addr_t addr, -size_t size,int direction, -dma_addr_t dma_addr, -unsigned long attrs); - -extern void debug_dma_unmap_resource (struct device *dev,dma_addr_t dma_addr, -size_t size,int direction); -extern void debug_dma_sync_single_for_cpu (struct device *dev,dma_addr_t dma_handle,size_t size,int direction);
@@ -88,19 +80,6 @@ static inline void debug_dma_free_coherent(struct device *dev, size_t size, { } -static inline void debug_dma_map_resource (struct device *dev,phys_addr_t addr, -size_t size,int direction, -dma_addr_t dma_addr, -unsigned long attrs) -{ -} - -static inline void debug_dma_unmap_resource (struct device *dev, -dma_addr_t dma_addr,size_t size, -int direction) -{ -} -static inline void debug_dma_sync_single_for_cpu (struct device *dev,dma_addr_t dma_handle,size_t size,int direction)
kernel/dma/direct.c +0 -16
@@ -502,22 +502,6 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,return ret; } -dma_addr_t dma_direct_map_resource (struct device *dev,phys_addr_t paddr, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -dma_addr_t dma_addr = paddr; - -if (unlikely (!dma_capable (dev, dma_addr, size, false))) { -dev_err_once (dev, -"DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n" , - &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); -WARN_ON_ONCE (1 ); -return DMA_MAPPING_ERROR ; - } - -return dma_addr; -} -int dma_direct_get_sgtable (struct device *dev,struct sg_table *sgt,void *cpu_addr,dma_addr_t dma_addr,size_t size,unsigned long attrs)
kernel/dma/mapping.c +36 -32
110 lines not on this page · raw message
Signature · 1 line
2.50.1
PATCH 12/16 mm/hmm: migrate to physical address-based DMA mapping API
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert HMM DMA operations from the legacy page-based API to the new physical address-based dma_map_phys() and dma_unmap_phys() functions. This demonstrates the preferred approach for new code that should use physical addresses directly rather than page+offset parameters.
The change replaces dma_map_page() and dma_unmap_page() calls with dma_map_phys() and dma_unmap_phys() respectively, using the physical address that was already available in the code. This eliminates the redundant page-to-physical address conversion and aligns with the DMA subsystem's move toward physical address-centric interfaces.
This serves as an example of how new code should be written to leverage the more efficient physical address API, which provides cleaner interfaces for drivers that already have access to physical addresses.
--- mm/hmm.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/hmm.c +4 -4
@@ -775,8 +775,8 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map,if (WARN_ON_ONCE (dma_need_unmap (dev) && !dma_addrs)) goto error; - dma_addr =dma_map_page (dev, page,0 , map->dma_entry_size, -DMA_BIDIRECTIONAL ); + dma_addr =dma_map_phys (dev, paddr, map->dma_entry_size, +DMA_BIDIRECTIONAL ,0 );if (dma_mapping_error (dev, dma_addr)) goto error;
@@ -819,8 +819,8 @@ bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx)dma_iova_unlink (dev, state, idx * map->dma_entry_size, map->dma_entry_size,DMA_BIDIRECTIONAL , attrs); }else if (dma_need_unmap (dev)) -dma_unmap_page (dev, dma_addrs[idx], map->dma_entry_size, -DMA_BIDIRECTIONAL ); +dma_unmap_phys (dev, dma_addrs[idx], map->dma_entry_size, +DMA_BIDIRECTIONAL ,0 ); pfns[idx] &= ~(HMM_PFN_DMA_MAPPED |HMM_PFN_P2PDMA |HMM_PFN_P2PDMA_BUS );
Signature · 1 line
2.50.1
PATCH 10/16 xen: swiotlb: Open code map_resource callback
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
General dma_direct_map_resource() is going to be removed in next patch, so simply open-code it in xen driver.
--- drivers/xen/swiotlb-xen.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
drivers/xen/swiotlb-xen.c +20 -1
@@ -392,6 +392,25 @@ xen_swiotlb_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, } } +static dma_addr_t xen_swiotlb_direct_map_resource (struct device *dev, +phys_addr_t paddr, +size_t size, +enum dma_data_direction dir, +unsigned long attrs) +{ +dma_addr_t dma_addr = paddr; + +if (unlikely (!dma_capable (dev, dma_addr, size, false))) { +dev_err_once (dev, +"DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n" , + &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); +WARN_ON_ONCE (1 ); +return DMA_MAPPING_ERROR ; + } + +return dma_addr; +} +/* * Return whether the given device DMA address mask can be supported * properly. For example, if your device can only drive the low 24-bits
@@ -426,5 +445,5 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, - .map_resource = dma_direct_map_resource, + .map_resource = xen_swiotlb_direct_map_resource, };
Signature · 1 line
2.50.1
PATCH 13/16 mm/hmm: properly take MMIO path
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
In case peer-to-peer transaction traverses through host bridge, the IOMMU needs to have IOMMU_MMIO flag, together with skip of CPU sync.
The latter was handled by provided DMA_ATTR_SKIP_CPU_SYNC flag, but IOMMU flag was missed, due to assumption that such memory can be treated as regular one.
Reuse newly introduced DMA attribute to properly take MMIO path.
--- mm/hmm.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
mm/hmm.c +8 -7
@@ -746,7 +746,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map,case PCI_P2PDMA_MAP_NONE :break ;case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE : - attrs |=DMA_ATTR_SKIP_CPU_SYNC ; + attrs |=DMA_ATTR_MMIO ; pfns[idx] |=HMM_PFN_P2PDMA ;break ;case PCI_P2PDMA_MAP_BUS_ADDR :
@@ -776,7 +776,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, goto error; dma_addr =dma_map_phys (dev, paddr, map->dma_entry_size, -DMA_BIDIRECTIONAL ,0 ); +DMA_BIDIRECTIONAL , attrs);if (dma_mapping_error (dev, dma_addr)) goto error;
@@ -811,16 +811,17 @@ bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx)if ((pfns[idx] & valid_dma) != valid_dma)return false; +if (pfns[idx] &HMM_PFN_P2PDMA ) + attrs |=DMA_ATTR_MMIO ; +if (pfns[idx] &HMM_PFN_P2PDMA_BUS ) ;/* no need to unmap bus address P2P mappings */ -else if (dma_use_iova (state)) { -if (pfns[idx] &HMM_PFN_P2PDMA ) - attrs |=DMA_ATTR_SKIP_CPU_SYNC ; +else if (dma_use_iova (state))dma_iova_unlink (dev, state, idx * map->dma_entry_size, map->dma_entry_size,DMA_BIDIRECTIONAL , attrs); - }else if (dma_need_unmap (dev)) +else if (dma_need_unmap (dev))dma_unmap_phys (dev, dma_addrs[idx], map->dma_entry_size, -DMA_BIDIRECTIONAL ,0 ); +DMA_BIDIRECTIONAL , attrs); pfns[idx] &= ~(HMM_PFN_DMA_MAPPED |HMM_PFN_P2PDMA |HMM_PFN_P2PDMA_BUS );
Signature · 1 line
2.50.1
PATCH 14/16 block-dma: migrate to dma_map_phys instead of map_page
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
After introduction of dma_map_phys(), there is no need to convert from physical address to struct page in order to map page. So let's use it directly.
--- block/blk-mq-dma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/blk-mq-dma.c +2 -2
@@ -87,8 +87,8 @@ static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)static bool blk_dma_map_direct (struct request *req,struct device *dma_dev,struct blk_dma_iter *iter,struct phys_vec *vec) { - iter->addr =dma_map_page (dma_dev,phys_to_page (vec->paddr), -offset_in_page (vec->paddr), vec->len,rq_dma_dir (req)); + iter->addr =dma_map_phys (dma_dev, vec->paddr, vec->len, +rq_dma_dir (req),0 );if (dma_mapping_error (dma_dev, iter->addr)) { iter->status =BLK_STS_RESOURCE ;return false;
Signature · 1 line
2.50.1
PATCH 16/16 nvme-pci: unmap MMIO pages with appropriate interface
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Block layer maps MMIO memory through dma_map_phys() interface with help of DMA_ATTR_MMIO attribute. There is a need to unmap that memory with the appropriate unmap function.
--- drivers/nvme/host/pci.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
drivers/nvme/host/pci.c +13 -5
@@ -682,11 +682,15 @@ static void nvme_free_prps(struct request *req) {struct nvme_iod *iod =blk_mq_rq_to_pdu (req);struct nvme_queue *nvmeq = req->mq_hctx->driver_data; +unsigned int attrs =0 ;unsigned int i; +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ; +for (i =0 ; i < iod->nr_dma_vecs; i++) -dma_unmap_page (nvmeq->dev->dev, iod->dma_vecs[i].addr, - iod->dma_vecs[i].len,rq_dma_dir (req)); +dma_unmap_phys (nvmeq->dev->dev, iod->dma_vecs[i].addr, + iod->dma_vecs[i].len,rq_dma_dir (req), attrs);mempool_free (iod->dma_vecs, nvmeq->dev->dmavec_mempool); }
@@ -699,15 +703,19 @@ static void nvme_free_sgls(struct request *req)unsigned int sqe_dma_len =le32_to_cpu (iod->cmd.common.dptr.sgl.length);struct nvme_sgl_desc *sg_list = iod->descriptors[0 ];enum dma_data_direction dir =rq_dma_dir (req); +unsigned int attrs =0 ; + +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ;if (iod->nr_descriptors) {unsigned int nr_entries = sqe_dma_len /sizeof (*sg_list), i;for (i =0 ; i < nr_entries; i++) -dma_unmap_page (dma_dev,le64_to_cpu (sg_list[i].addr), -le32_to_cpu (sg_list[i].length), dir); +dma_unmap_phys (dma_dev,le64_to_cpu (sg_list[i].addr), +le32_to_cpu (sg_list[i].length), dir, attrs); }else { -dma_unmap_page (dma_dev, sqe_dma_addr, sqe_dma_len, dir); +dma_unmap_phys (dma_dev, sqe_dma_addr, sqe_dma_len, dir, attrs); } }
Signature · 1 line
2.50.1
PATCH 15/16 block-dma: properly take MMIO path
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Make sure that CPU is not synced and IOMMU is configured to take MMIO path by providing newly introduced DMA_ATTR_MMIO attribute.
--- block/blk-mq-dma.c | 13 +++++++++++-- include/linux/blk-mq-dma.h | 6 +++++- include/linux/blk_types.h | 2 ++ 3 files changed, 18 insertions(+), 3 deletions(-)
block/blk-mq-dma.c +11 -2
@@ -87,8 +87,13 @@ static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)static bool blk_dma_map_direct (struct request *req,struct device *dma_dev,struct blk_dma_iter *iter,struct phys_vec *vec) { +unsigned int attrs =0 ; + +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ; + iter->addr =dma_map_phys (dma_dev, vec->paddr, vec->len, -rq_dma_dir (req),0 ); +rq_dma_dir (req), attrs);if (dma_mapping_error (dma_dev, iter->addr)) { iter->status =BLK_STS_RESOURCE ;return false;
@@ -103,14 +108,17 @@ static bool blk_rq_dma_map_iova(struct request *req, struct device *dma_dev, {enum dma_data_direction dir =rq_dma_dir (req);unsigned int mapped =0 ; +unsigned int attrs =0 ;int error; iter->addr = state->addr; iter->len =dma_iova_size (state); +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ; do { error = dma_iova_link(dma_dev, state, vec->paddr, mapped, - vec->len, dir, 0); + vec->len, dir, attrs); if (error) break; mapped += vec->len;
@@ -176,6 +184,7 @@ bool blk_rq_dma_map_iter_start(struct request *req, struct device *dma_dev,* same as non-P2P transfers below and during unmap. */ req->cmd_flags &= ~REQ_P2PDMA ; + req->cmd_flags |=REQ_MMIO ;break ;default : iter->status =BLK_STS_INVAL ;
include/linux/blk-mq-dma.h +5 -1
@@ -48,12 +48,16 @@ static inline bool blk_rq_dma_map_coalesce(struct dma_iova_state *state)static inline bool blk_rq_dma_unmap (struct request *req,struct device *dma_dev,struct dma_iova_state *state,size_t mapped_len) { +unsigned int attrs =0 ; +if (req->cmd_flags &REQ_P2PDMA )return true;if (dma_use_iova (state)) { +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ;dma_iova_destroy (dma_dev, state, mapped_len,rq_dma_dir (req), -0 ); + attrs);return true; }
include/linux/blk_types.h +2 -0
@@ -387,6 +387,7 @@ enum req_flag_bits { __REQ_FS_PRIVATE,/* for file system (submitter) use */ __REQ_ATOMIC,/* for atomic write operations */ __REQ_P2PDMA,/* contains P2P DMA pages */ + __REQ_MMIO, /* contains MMIO memory *//* * Command specific flags, keep last: */
@@ -420,6 +421,7 @@ enum req_flag_bits {#define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE)#define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC)#define REQ_P2PDMA (__force blk_opf_t)(1ULL << __REQ_P2PDMA) +#define REQ_MMIO (__force blk_opf_t)(1ULL << __REQ_MMIO)#define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP)
Signature · 1 line
2.50.1
Jason Gunthorpe
maintainerRe: [PATCH v1 01/16] dma-mapping: introduce new DMA attribute to indicate MMIO memory
In reply to Leon RomanovskyQuoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
This patch introduces the
DMA_ATTR_MMIOattribute to mark DMA buffers that reside in memory-mapped I/O (MMIO) regions, such as device BARs exposed through the host bridge, which are accessible for peer-to-peer (P2P) DMA.This attribute is especially useful for exporting device memory to other devices for DMA without CPU involvement, and avoids unnecessary or potentially detrimental CPU cache maintenance calls.
It is worth mentioning here that dma_map_resource() and DMA_ATTR_MMIO are intended to be the same thing.
Documentation/core-api/dma-attributes.rst accesses to DMA buffers in both privileged "supervisor" and unprivileged · line 139
Quoted 13 lines
Documentation/core-api/dma-attributes.rst +7
-0@@ -130,3 +130,10 @@ accesses to DMA buffers in both privileged "supervisor" and unprivileged subsystem that the buffer is fully accessible at the elevated privilege level (and ideally inaccessible or at least read-only at the lesser-privileged levels). + +DMA_ATTR_MMIO +------------- + +This attribute is especially useful for exporting device memory to other +devices for DMA without CPU involvement, and avoids unnecessary or +potentially detrimental CPU cache maintenance calls.
How about
This attribute indicates the physical address is not normal system memory. It may not be used with kmap*()/phys_to_virt()/phys_to_page() functions, it may not be cachable, and access using CPU load/store instructions may not be allowed.
Usually this will be used to describe MMIO addresses, or other non cachable register addresses. When DMA mapping this sort of address we call the operation Peer to Peer as a one device is DMA'ing to another device. For PCI devices the p2pdma APIs must be used to determine if DMA_ATTR_MMIO is appropriate.
For architectures that require cache flushing for DMA coherence DMA_ATTR_MMIO will not perform any cache flushing. The address provided must never be mapped cachable into the CPU.
Quoted 4 lines
+/*
+ *DMA_ATTR_MMIO- Indicates memory-mapped I/O (MMIO) region for DMA mapping
+ *
+ * This attribute is used for MMIO memory regions that are exposed through
Quoted 8 lines
+ * the host bridge and are accessible for peer-to-peer (P2P) DMA. Memory
+ * marked with this attribute is not system RAM and may represent device
+ * BAR windows or peer-exposed memory.
+ *
+ * Typical usage is for mapping hardware memory BARs or exporting device
+ * memory to other devices for DMA without involving main system RAM.
+ * The attribute guarantees no CPU cache maintenance calls will be made.
+ */
I'd copy the Documentation/ text
Jason
Re: [PATCH v1 02/16] iommu/dma: handle MMIO path in dma_iova_link
In reply to Leon RomanovskyQuoted 3 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Make sure that CPU is not synced if MMIO path is taken.
Let's elaborate..
Implement DMA_ATTR_MMIO for dma_iova_link().
This will replace the hacky use of DMA_ATTR_SKIP_CPU_SYNC to avoid touching the possibly non-KVA MMIO memory.
Also correct the incorrect caching attribute for the IOMMU, MMIO memory should not be cachable inside the IOMMU mapping or it can possibly create system problems. Set IOMMU_MMIO for DMA_ATTR_MMIO.
Quoted 5 lines
drivers/iommu/dma-iommu.c +0
-0
drivers/iommu/dma-iommu.c static int __dma_iova_link(struct device *dev, dma_addr_t addr, · line 1843
Quoted 10 lines
drivers/iommu/dma-iommu.c +3
-3@@ -1837,13 +1837,20 @@ static int __dma_iova_link(struct device *dev, dma_addr_t addr, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { - bool coherent = dev_is_dma_coherent(dev); + int prot; -if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC )) -arch_sync_dma_for_device (phys, size, dir); +if (attrs &DMA_ATTR_MMIO ) + prot =dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO ;
Yeah, exactly, we need the IOPTE on ARM to have the right cachability or some systems might go wrong.
drivers/iommu/dma-iommu.c static int __dma_iova_link(struct device *dev, dma_addr_t addr, · line 1854
Quoted 12 lines
drivers/iommu/dma-iommu.c +8
-1+else { +bool coherent =dev_is_dma_coherent (dev); + +if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC )) +arch_sync_dma_for_device (phys, size, dir); + prot =dma_info_to_prot (dir, coherent, attrs); + }return iommu_map_nosync (iommu_get_dma_domain (dev), addr, phys, size, -dma_info_to_prot (dir, coherent, attrs),GFP_ATOMIC ); + prot,GFP_ATOMIC ); }
Hmm, I missed this in prior series, ideally the GFP_ATOMIC should be passed in as a gfp_t here so we can use GFP_KERNEL in callers that are able.
Jason
Re: [PATCH v1 03/16] dma-debug: refactor to use physical addresses for page mapping
In reply to Leon Romanovskykernel/dma/debug.c void debug_dma_map_single(struct device *dev, const void *addr, · line 1209
Quoted 4 lines · Leon Romanovsky
kernel/dma/debug.c +2
-0+void debug_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +int direction,dma_addr_t dma_addr,unsigned long attrs) {struct dma_debug_entry *entry;
Should this patch should also absorb debug_dma_map_resource() into here as well and we can have the caller of dma_dma_map_resource() call debug_dma_map_page with ATTR_MMIO?
If not, this looks OK
Jason
Re: [PATCH v1 03/16] dma-debug: refactor to use physical addresses for page mapping
Quoted 9 lines · Jason Gunthorpe
kernel/dma/debug.c void debug_dma_map_single(struct device *dev, const void *addr, · line 1209
Quoted 4 lines · Leon Romanovsky
kernel/dma/debug.c +2
-0+void debug_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +int direction,dma_addr_t dma_addr,unsigned long attrs) {struct dma_debug_entry *entry;Should this patch should also absorb
debug_dma_map_resource()into here as well and we can have the caller ofdma_dma_map_resource()calldebug_dma_map_pagewithATTR_MMIO?
It is done in "[PATCH v1 11/16] dma-mapping: export new dma_*map_phys() interface".
Thanks
Quoted 7 lines
If not, this looks OK
Reviewed-by Jason Gunthorpe <[email protected]>Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 05/16] iommu/dma: rename iommu_dma_*map_page to iommu_dma_*map_phys
In reply to Leon RomanovskyQuoted 8 lines · Leon Romanovsky
drivers/iommu/dma-iommu.c +0
-0@@ -1190,11 +1190,9 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,return iova_offset (iovad, phys | size); }
drivers/iommu/dma-iommu.c static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys, · line 1198
Quoted 10 lines
drivers/iommu/dma-iommu.c +2
-4-dma_addr_t iommu_dma_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size,enum dma_data_direction dir, -unsigned long attrs) +dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs) { -phys_addr_t phys =page_to_phys (page) + offset;bool coherent =dev_is_dma_coherent (dev);int prot =dma_info_to_prot (dir, coherent, attrs);struct iommu_domain *domain =iommu_get_dma_domain (dev);
No issue with pushing the page_to_phys to the looks like two callers..
It is worth pointing though that today if the page * was a MEMORY_DEVICE_PCI_P2PDMA page then it is illegal to call the swiotlb functions a few lines below this:
phys = iommu_dma_map_swiotlb(dev, phys, size, dir, attrs);
ie struct page alone as a type is not sufficient to make this function safe for a long time now.
So I would add some explanation in the commit message how this will be situated in the final call chains, and maybe leave behind a comment that attrs may not have ATTR_MMIO in this function.
I think the answer is iommu_dma_map_phys() is only called for !ATTR_MMIO addresses, and that iommu_dma_map_resource() will be called for ATTR_MMIO?
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 06/16] iommu/dma: extend iommu_dma_*map_phys API to handle MMIO memory
In reply to Leon RomanovskyQuoted 4 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Combine iommu_dma_*map_phys with iommu_dma_*map_resource interfaces in order to allow single
phys_addr_tflow.
Some later patch deletes iommu_dma_map_resource() ? Mention that plan here?
Quoted 9 lines
drivers/iommu/dma-iommu.c +0
-2@@ -1193,12 +1193,17 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size,enum dma_data_direction dir,unsigned long attrs) { -bool coherent =dev_is_dma_coherent (dev); -int prot =dma_info_to_prot (dir, coherent, attrs);struct iommu_domain *domain =iommu_get_dma_domain (dev);
drivers/iommu/dma-iommu.c static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys, · line 1206
Quoted 10 lines
drivers/iommu/dma-iommu.c +7
-0struct iommu_dma_cookie *cookie = domain->iova_cookie;struct iova_domain *iovad = &cookie->iovad;dma_addr_t iova, dma_mask =dma_get_mask (dev); +bool coherent; +int prot; + +if (attrs &DMA_ATTR_MMIO ) +return __iommu_dma_map (dev, phys, size, +dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO , +dma_get_mask (dev));
I realize that iommu_dma_map_resource() doesn't today, but shouldn't this be checking for swiotlb:
if (dev_use_swiotlb(dev, size, dir) &&
iova_unaligned(iovad, phys, size)) {Except we have to fail for ATTR_MMIO?
Now that we have ATTR_MMIO, should dma_info_to_prot() just handle it directly instead of open coding the | IOMMU_MMIO and messing with the coherent attribute?
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 07/16] dma-mapping: convert dma_direct_*map_page to be phys_addr_t based
In reply to Leon RomanovskyQuoted 15 lines · Leon Romanovsky
kernel/dma/direct.h +3
-5@@ -80,42 +80,54 @@ static inline void dma_direct_sync_single_for_cpu(struct device *dev,arch_dma_mark_clean (paddr, size); } -static inline dma_addr_t dma_direct_map_page (struct device *dev, -struct page *page,unsigned long offset,size_t size, -enum dma_data_direction dir,unsigned long attrs) +static inline dma_addr_t dma_direct_map_phys (struct device *dev, +phys_addr_t phys,size_t size,enum dma_data_direction dir, +unsigned long attrs) { -phys_addr_t phys =page_to_phys (page) + offset; -dma_addr_t dma_addr =phys_to_dma (dev, phys);
kernel/dma/direct.h static inline void dma_direct_sync_single_for_cpu(struct device *dev, · line 96
Quoted 10 lines
kernel/dma/direct.h +10
-0+bool is_mmio = attrs &DMA_ATTR_MMIO ; +dma_addr_t dma_addr; +bool capable; + + dma_addr = (is_mmio) ? phys :phys_to_dma (dev, phys); + capable =dma_capable (dev, dma_addr, size, is_mmio); +if (is_mmio) { +if (unlikely (!capable)) + goto err_overflow; +return dma_addr;
Similar remark here, shouldn't we be checking swiotlb things for ATTR_MMIO and failing if swiotlb is needed?
kernel/dma/direct.h static inline void dma_direct_sync_single_for_cpu(struct device *dev, · line 92
Quoted 3 lines
kernel/dma/direct.h +0
-3-if (is_swiotlb_force_bounce (dev)) { -if (is_pci_p2pdma_page (page)) -return DMA_MAPPING_ERROR ;
This
kernel/dma/direct.h static inline void dma_direct_sync_single_for_cpu(struct device *dev, · line 99
Quoted 4 lines
kernel/dma/direct.h +0
-4-if (unlikely (!dma_capable (dev, dma_addr, size, true)) || -dma_kmalloc_needs_bounce (dev, size, dir)) { -if (is_pci_p2pdma_page (page)) -return DMA_MAPPING_ERROR ;
And this
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
In reply to Leon RomanovskyQuoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Convert the KMSAN DMA handling function from page-based to physical address-based interface.
The refactoring renames
kmsan_handle_dma()parameters from accepting (struct page *page,size_toffset,size_tsize) to (phys_addr_tphys,size_tsize). APFN_VALIDcheck is added to prevent KMSAN operations on non-page memory, preventing from non struct page backed address,
Quoted 4 lines
As part of this change, support for highmem addresses is implemented using
kmap_local_page()to handle both lowmem and highmem regions properly. All callers throughout the codebase are updated to use the newphys_addr_tbased interface.
Use the function Matthew pointed at kmap_local_pfn()
Maybe introduce the kmap_local_phys() he suggested too.
mm/kmsan/hooks.c static void kmsan_handle_dma_page(const void *addr, size_t size, · line 347
Quoted 12 lines
/* Helper function to handle DMA data transfers. */
mm/kmsan/hooks.c +4
-2-void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) {u64 page_offset, to_go, addr; +struct page *page; +void *kaddr; -if (PageHighMem (page)) +if (!pfn_valid (PHYS_PFN (phys)))return ;
Not needed, the caller must pass in a phys that is kmap compatible. Maybe just leave a comment. FWIW today this is also not checking for P2P or DEVICE non-kmap struct pages either, so it should be fine without checks.
Quoted 7 lines
mm/kmsan/hooks.c +4
-1- addr = (u64 )page_address (page) + offset; + + page =phys_to_page (phys); + page_offset =offset_in_page (phys); +/* * The kernel may occasionally give us adjacent DMA pages not belonging
mm/kmsan/hooks.c static void kmsan_handle_dma_page(const void *addr, size_t size, · line 362
Quoted 10 lines
mm/kmsan/hooks.c +4
-1* to the same allocation. Process them separately to avoid triggering * internal KMSAN checks. */ while (size >0 ) { - page_offset =offset_in_page (addr); to_go =min (PAGE_SIZE - page_offset, (u64 )size); + +if (PageHighMem (page)) +/* Handle highmem pages using kmap */ + kaddr =kmap_local_page (page);
No need for the PageHighMem() - just always call kmap_local_pfn().
I'd also propose that any debug/sanitizer checks that the passed phys is valid for kmap (eg pfn valid, not zone_device, etc) should be inside the kmap code.
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 09/16] dma-mapping: handle MMIO flow in dma_map|unmap_page
In reply to Leon RomanovskyQuoted 3 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Extend base DMA page API to handle MMIO flow.
I would mention here this follows the long ago agreement that we don't need to enable P2P in the legacy dma_ops area. Simply failing when getting an ATTR_MMIO is OK.
Quoted 11 lines
kernel/dma/mapping.c +1
-0@@ -158,6 +158,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, {const struct dma_map_ops *ops =get_dma_ops (dev);phys_addr_t phys =page_to_phys (page) + offset; +bool is_mmio = attrs &DMA_ATTR_MMIO ;dma_addr_t addr;BUG_ON (!valid_dma_direction (dir));
kernel/dma/mapping.c dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, · line 171
Quoted 6 lines
kernel/dma/mapping.c +1
-1@@ -166,12 +167,23 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_phys_direct (dev, phys + size)) + (!is_mmio &&arch_dma_map_phys_direct (dev, phys + size))) addr =dma_direct_map_phys (dev, phys, size, dir, attrs);
I don't know this area, maybe explain a bit in the commit message how you see ATTR_MMIO interacts with arch_dma_map_phys_direct ?
Quoted 4 lines
kernel/dma/mapping.c +1
-1else if (use_dma_iommu (dev)) addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); -else +else if (is_mmio) {
kernel/dma/mapping.c dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, · line 184
Quoted 10 lines
kernel/dma/mapping.c +9
-0+if (!ops->map_resource) +return DMA_MAPPING_ERROR ; + + addr = ops->map_resource (dev, phys, size, dir, attrs); + }else { +/* +* All platforms which implement .map_page() don't support +* non-struct page backed addresses. +*/ addr = ops->map_page (dev, page, offset, size, dir, attrs);
Comment could be clearer maybe just:
The dma_ops API contract for ops->map_page() requires kmappable memory, while ops->map_resource() does not.
But this approach looks good to me, it prevents non-kmappable phys from going down to the legacy dma_ops map_page where it cannot work.
From here you could do what Marek and Christoph asked to flush the struct page out of the ops->map_page() and replace it with kmap_local_phys().
Jason
Re: [PATCH v1 13/16] mm/hmm: properly take MMIO path
In reply to Leon RomanovskyQuoted 22 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
In case peer-to-peer transaction traverses through host bridge, the IOMMU needs to have
IOMMU_MMIOflag, together with skip of CPU sync.The latter was handled by provided
DMA_ATTR_SKIP_CPU_SYNCflag, but IOMMU flag was missed, due to assumption that such memory can be treated as regular one.Reuse newly introduced DMA attribute to properly take MMIO path.
Signed-off-by Leon Romanovsky <[email protected]>--- mm/hmm.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)mm/hmm.c +0
-0
mm/hmm.c dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, · line 751
Quoted 7 lines
mm/hmm.c +1
-1@@ -746,7 +746,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map,case PCI_P2PDMA_MAP_NONE :break ;case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE : - attrs |=DMA_ATTR_SKIP_CPU_SYNC ; + attrs |=DMA_ATTR_MMIO ; pfns[idx] |=HMM_PFN_P2PDMA ;break ;
Yeah, this is a lot cleaner
Jason
Re: [PATCH v1 12/16] mm/hmm: migrate to physical address-based DMA mapping API
In reply to Leon RomanovskyQuoted 13 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Convert HMM DMA operations from the legacy page-based API to the new physical address-based
dma_map_phys()anddma_unmap_phys()functions. This demonstrates the preferred approach for new code that should use physical addresses directly rather than page+offset parameters.The change replaces
dma_map_page()anddma_unmap_page()calls withdma_map_phys()anddma_unmap_phys()respectively, using the physical address that was already available in the code. This eliminates the redundant page-to-physical address conversion and aligns with the DMA subsystem's move toward physical address-centric interfaces.
Quoted 8 lines
This serves as an example of how new code should be written to leverage the more efficient physical address API, which provides cleaner interfaces for drivers that already have access to physical addresses.
Signed-off-by Leon Romanovsky <[email protected]>--- mm/hmm.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
Maybe the next patch should be squished into here too if it is going to be a full example
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 11/16] dma-mapping: export new dma_*map_phys() interface
In reply to Leon RomanovskyQuoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Introduce new DMA mapping functions
dma_map_phys()anddma_unmap_phys()that operate directly on physical addresses instead of page+offset parameters. This provides a more efficient interface for drivers that already have physical addresses available.The new functions are implemented as the primary mapping layer, with the existing
dma_map_page_attrs()anddma_unmap_page_attrs()functions converted to simple wrappers around the phys-based implementations.
Briefly explain how the existing functions are remapped into wrappers calling the phys functions.
kernel/dma/mapping.c dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, · line 206
Quoted 11 lines
kernel/dma/mapping.c +11
-0+dma_addr_t dma_map_page_attrs (struct device *dev,struct page *page, +size_t offset,size_t size,enum dma_data_direction dir, +unsigned long attrs) +{ +phys_addr_t phys =page_to_phys (page) + offset; + +if (unlikely (attrs &DMA_ATTR_MMIO )) +return DMA_MAPPING_ERROR ; + +if (IS_ENABLED (CONFIG_DMA_API_DEBUG )) +WARN_ON_ONCE (!pfn_valid (PHYS_PFN (phys)));
This is not useful, if we have a struct page and did page_to_phys then pfn_valid is always true.
Instead this should check for any ZONE_DEVICE page and reject that. And handle the error:
if (WARN_ON_ONCE()) return DMA_MAPPING_ERROR;
I'd add another debug check inside dma_map_phys that if !ATTR_MMIO then pfn_valid, and not zone_device
Quoted 18 lines
kernel/dma/mapping.c +2
-9@@ -337,41 +364,18 @@ EXPORT_SYMBOL(dma_unmap_sg_attrs);dma_addr_t dma_map_resource (struct device *dev,phys_addr_t phys_addr,size_t size,enum dma_data_direction dir,unsigned long attrs) { -const struct dma_map_ops *ops =get_dma_ops (dev); -dma_addr_t addr =DMA_MAPPING_ERROR ; - -BUG_ON (!valid_dma_direction (dir)); - -if (WARN_ON_ONCE (!dev->dma_mask)) +if (IS_ENABLED (CONFIG_DMA_API_DEBUG ) && +WARN_ON_ONCE (pfn_valid (PHYS_PFN (phys_addr))))return DMA_MAPPING_ERROR ; -if (dma_map_direct (dev, ops)) - addr =dma_direct_map_resource (dev, phys_addr, size, dir, attrs); -else if (use_dma_iommu (dev))
kernel/dma/mapping.c EXPORT_SYMBOL(dma_unmap_sg_attrs); · line 374
Quoted 10 lines
kernel/dma/mapping.c +1
-7- addr =iommu_dma_map_resource (dev, phys_addr, size, dir, attrs); - else if (ops->map_resource) - addr = ops->map_resource (dev, phys_addr, size, dir, attrs); - -trace_dma_map_resource (dev, phys_addr, addr, size, dir, attrs); -debug_dma_map_resource (dev, phys_addr, size, dir, addr, attrs); -return addr; +return dma_map_phys (dev, phys_addr, size, dir, attrs |DMA_ATTR_MMIO ); }EXPORT_SYMBOL (dma_map_resource);
I think this makes alot of sense at least.
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 16/16] nvme-pci: unmap MMIO pages with appropriate interface
In reply to Leon RomanovskyQuoted 5 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Block layer maps MMIO memory through
dma_map_phys()interface with help ofDMA_ATTR_MMIOattribute. There is a need to unmap that memory with the appropriate unmap function.
Be specific, AFIACT the issue is that on dma_ops platforms the map will call ops->map_resource for ATTR_MMIO so we must have the unmap call ops->unmap_resournce
Maybe these patches should be swapped then, as adding ATTR_MMIO seems like it created this issue?
Jason
Jason Gunthorpe
maintainerQuoted 8 lines · Leon Romanovsky
Changelog:
v1:
* Added newDMA_ATTR_MMIOattribute to indicatePCI_P2PDMA_MAP_THRU_HOST_BRIDGEpath.
* Rewrote dma_map_* functions to use thus new attribute
v0: https://lore.kernel.org/all/[email protected]/
------------------------------------------------------------------------
Quoted 4 lines
This series refactors the DMA mapping to use physical addresses as the primary interface instead of page+offset parameters. This change aligns the DMA API with the underlying hardware reality where DMA operations work with physical addresses, not page structures.
Lets elaborate this as Robin asked:
This series refactors the DMA mapping API to provide a phys_addr_t based, and struct-page free, external API that can handle all the mapping cases we want in modern systems:
- struct page based cachable DRAM
- struct page MEMORY_DEVICE_PCI_P2PDMA PCI peer to peer non-cachable MMIO
- struct page-less PCI peer to peer non-cachable MMIO
- struct page-less "resource" MMIO
Overall this gets much closer to Matthew's long term wish for struct-pageless IO to cachable DRAM. The remaining primary work would be in the mm side to allow kmap_local_pfn()/phys_to_virt() to work on phys_addr_t without a struct page.
The general design is to remove struct page usage entirely from the DMA API inner layers. For flows that need to have a KVA for the physical address they can use kmap_local_pfn() or phys_to_virt(). This isolates the struct page requirements to MM code only. Long term all removals of struct page usage are supporting Matthew's memdesc project which seeks to substantially transform how struct page works.
Instead make the DMA API internals work on phys_addr_t. Internally there are still dedicated 'page' and 'resource' flows, except they are now distinguished by a new DMA_ATTR_MMIO instead of by callchain. Both flows use the same phys_addr_t.
When DMA_ATTR_MMIO is specified things work similar to the existing 'resource' flow. kmap_local_pfn(), phys_to_virt(), phys_to_page(), pfn_valid(), etc are never called on the phys_addr_t. This requires rejecting any configuration that would need swiotlb. CPU cache flushing is not required, and avoided, as ATTR_MMIO also indicates the address have no cachable mappings. This effectively removes any DMA API side requirement to have struct page when DMA_ATTR_MMIO is used.
In the !DMA_ATTR_MMIO mode things work similarly to the 'page' flow, except on the common path of no cache flush, no swiotlb it never touches a struct page. When cache flushing or swiotlb copying kmap_local_pfn()/phys_to_virt() are used to get a KVA for CPU usage. This was already the case on the unmap side, now the map side is symmetric.
Callers are adjusted to set DMA_ATTR_MMIO. Existing 'resource' users must set it. The existing struct page based MEMORY_DEVICE_PCI_P2PDMA path must also set it. This corrects some existing bugs where iommu mappings for P2P MMIO were improperly marked IOMMU_CACHE.
Since ATTR_MMIO is made to work with all the existing DMA map entry points, particularly dma_iova_link(), this finally allows a way to use the new DMA API to map PCI P2P MMIO without creating struct page. The VFIO DMABUF series demonstrates how this works. This is intended to replace the incorrect driver use of dma_map_resource() on PCI BAR addresses.
This series does the core code and modern flows. A followup series will give the same treatement to the legacy dma_ops implementation.
Jason
Re: [PATCH v1 10/16] xen: swiotlb: Open code map_resource callback
In reply to Leon RomanovskyQuoted 6 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
General
dma_direct_map_resource()is going to be removed in next patch, so simply open-code it in xen driver.Signed-off-by Leon Romanovsky <[email protected]>
Juergen
Marek Szyprowski
maintainerQuoted 69 lines · Jason Gunthorpe
Quoted 8 lines · Leon Romanovsky
Changelog:
v1:
* Added newDMA_ATTR_MMIOattribute to indicatePCI_P2PDMA_MAP_THRU_HOST_BRIDGEpath.
* Rewrote dma_map_* functions to use thus new attribute
v0: https://lore.kernel.org/all/[email protected]/
------------------------------------------------------------------------Quoted 4 lines
This series refactors the DMA mapping to use physical addresses as the primary interface instead of page+offset parameters. This change aligns the DMA API with the underlying hardware reality where DMA operations work with physical addresses, not page structures.
Lets elaborate this as Robin asked:
This series refactors the DMA mapping API to provide a
phys_addr_tbased, and struct-page free, external API that can handle all the mapping cases we want in modern systems:- struct page based cachable DRAM
- struct pageMEMORY_DEVICE_PCI_P2PDMAPCI peer to peer non-cachable MMIO
- struct page-less PCI peer to peer non-cachable MMIO
- struct page-less "resource" MMIOOverall this gets much closer to Matthew's long term wish for struct-pageless IO to cachable DRAM. The remaining primary work would be in the mm side to allow kmap_local_pfn()/phys_to_virt() to work on
phys_addr_twithout a struct page.The general design is to remove struct page usage entirely from the DMA API inner layers. For flows that need to have a KVA for the physical address they can use
kmap_local_pfn()orphys_to_virt(). This isolates the struct page requirements to MM code only. Long term all removals of struct page usage are supporting Matthew's memdesc project which seeks to substantially transform how struct page works.Instead make the DMA API internals work on
phys_addr_t. Internally there are still dedicated 'page' and 'resource' flows, except they are now distinguished by a newDMA_ATTR_MMIOinstead of by callchain. Both flows use the samephys_addr_t.When
DMA_ATTR_MMIOis specified things work similar to the existing 'resource' flow.kmap_local_pfn(),phys_to_virt(),phys_to_page(),pfn_valid(), etc are never called on thephys_addr_t. This requires rejecting any configuration that would need swiotlb. CPU cache flushing is not required, and avoided, asATTR_MMIOalso indicates the address have no cachable mappings. This effectively removes any DMA API side requirement to have struct page whenDMA_ATTR_MMIOis used.In the !DMA_ATTR_MMIO mode things work similarly to the 'page' flow, except on the common path of no cache flush, no swiotlb it never touches a struct page. When cache flushing or swiotlb copying kmap_local_pfn()/phys_to_virt() are used to get a KVA for CPU usage. This was already the case on the unmap side, now the map side is symmetric.
Callers are adjusted to set
DMA_ATTR_MMIO. Existing 'resource' users must set it. The existing struct page basedMEMORY_DEVICE_PCI_P2PDMApath must also set it. This corrects some existing bugs where iommu mappings for P2P MMIO were improperly markedIOMMU_CACHE.Since
ATTR_MMIOis made to work with all the existing DMA map entry points, particularlydma_iova_link(), this finally allows a way to use the new DMA API to map PCI P2P MMIO without creating struct page. The VFIO DMABUF series demonstrates how this works. This is intended to replace the incorrect driver use ofdma_map_resource()on PCI BAR addresses.
Quoted 2 lines
This series does the core code and modern flows. A followup series will give the same treatement to the legacy
dma_opsimplementation.
Thanks for the elaborate description, that's something that was missing in the previous attempt. I read again all the previous discussion and this explanation and there are still two things that imho needs more clarification.
First - basing the API on the phys_addr_t.
Page based API had the advantage that it was really hard to abuse it and call for something that is not 'a normal RAM'. I initially though that phys_addr_t based API will somehow simplify arch specific implementation, as some of them indeed rely on phys_addr_t internally, but I missed other things pointed by Robin. Do we have here any alternative?
Second - making dma_map_phys() a single API to handle all cases.
Do we really need such single function to handle all cases? To handle P2P case, the caller already must pass DMA_ATTR_MMIO, so it must somehow keep such information internally. Cannot it just call existing dma_map_resource(), so there will be clear distinction between these 2 cases (DMA to RAM and P2P DMA)? Do we need additional check for DMA_ATTR_MMIO for every typical DMA user? I know that branching is cheap, but this will probably increase code size for most of the typical users for no reason.
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Jason Gunthorpe
maintainerQuoted 4 lines · Marek Szyprowski
First - basing the API on the
phys_addr_t.Page based API had the advantage that it was really hard to abuse it and call for something that is not 'a normal RAM'.
This is not true anymore. Today we have ZONE_DEVICE as a struct page type with a whole bunch of non-dram sub-types:
enum memory_type {
/* 0 is reserved to catch uninitialized type fields */
MEMORY_DEVICE_PRIVATE = 1,
MEMORY_DEVICE_COHERENT,
MEMORY_DEVICE_FS_DAX,
MEMORY_DEVICE_GENERIC,
MEMORY_DEVICE_PCI_P2PDMA,
};Few of which are kmappable/page_to_virtable() in a way that is useful for the DMA API.
DMA API sort of ignores all of this and relies on the caller to not pass in an incorrect struct page. eg we rely on things like the block stack to do the right stuff when a MEMORY_DEVICE_PCI_P2PDMA is present in a bio_vec.
Which is not really fundamentally different from just using phys_addr_t in the first place.
Sure, this was a stronger argument when this stuff was originally written, before ZONE_DEVICE was invented.
Quoted 4 lines
I initially though that
phys_addr_tbased API will somehow simplify arch specific implementation, as some of them indeed rely onphys_addr_tinternally, but I missed other things pointed by Robin. Do we have here any alternative?
I think it is less of a code simplification, more as a reduction in conceptual load. When we can say directly there is no struct page type anyhwere in the DMA API layers then we only have to reason about kmap/phys_to_virt compatibly.
This is also a weaker overall requirement than needing an actual struct page which allows optimizing other parts of the kernel. Like we aren't forced to create MEMORY_DEVICE_PCI_P2PDMA stuct pages just to use the dma api.
Again, any place in the kernel we can get rid of struct page the smoother the road will be for the MM side struct page restructuring.
For example one of the bigger eventual goes here is to make a bio_vec store phys_addr_t, not struct page pointers.
DMA API is not alone here, we have been de-struct-paging the kernel for a long time now:
netdev: https://lore.kernel.org/linux-mm/[email protected]/
slab: https://lore.kernel.org/linux-mm/[email protected]/
iommmu: https://lore.kernel.org/all/[email protected]/
page tables: https://lore.kernel.org/linux-mm/[email protected]/
zswap: https://lore.kernel.org/all/[email protected]/
With a long term goal that struct page only exists for legacy code, and is maybe entirely compiled out of modern server kernels.
Quoted 3 lines
Second - making
dma_map_phys()a single API to handle all cases.Do we really need such single function to handle all cases?
If we accept the direction to remove struct page then it makes little sense to have a dma_map_ram(phys_addr) and dma_map_resource(phys_addr) and force key callers (like block) to have more ifs - especially if the conditional could become "free" inside the dma API (see below).
Plus if we keep the callchain split then adding a
"dma_link_resource"/etc are now needed as well.
Quoted 3 lines
DMA_ATTR_MMIOfor every typical DMA user? I know that branching is cheap, but this will probably increase code size for most of the typical users for no reason.
Well, having two call chains will increase the code size much more, and 'resource' can't be compiled out. Arguably this unification should reduce the .text size since many of the resource only functions go away.
There are some branches, and I think the push toward re-using DMA_ATTR_SKIP_CPU_SYNC was directly to try to reduce that branch cost.
However, I think we should be looking for a design here that is "free" on the fast no-swiotlb and non-cache-flush path. I think this can be achieved by checking ATTR_MMIO only after seeing swiotlb is needed (like today's is p2p check). And we can probably freely fold it into the existing sync check:
if ((attrs & (DMA_ATTR_SKIP_CPU_SYNC | DMA_ATTR_MMIO)) == 0)
I saw Leon hasn't done these micro optimizations, but it seems like it could work out.
Regards,
Jason
Quoted 56 lines · Jason Gunthorpe
Quoted 4 lines · Marek Szyprowski
First - basing the API on the
phys_addr_t.Page based API had the advantage that it was really hard to abuse it and call for something that is not 'a normal RAM'.
This is not true anymore. Today we have
ZONE_DEVICEas a struct page type with a whole bunch of non-dram sub-types:enum memory_type { /* 0 is reserved to catch uninitialized type fields */ MEMORY_DEVICE_PRIVATE = 1, MEMORY_DEVICE_COHERENT, MEMORY_DEVICE_FS_DAX, MEMORY_DEVICE_GENERIC, MEMORY_DEVICE_PCI_P2PDMA, };Few of which are kmappable/page_to_virtable() in a way that is useful for the DMA API.
DMA API sort of ignores all of this and relies on the caller to not pass in an incorrect struct page. eg we rely on things like the block stack to do the right stuff when a
MEMORY_DEVICE_PCI_P2PDMAis present in abio_vec.Which is not really fundamentally different from just using
phys_addr_tin the first place.Sure, this was a stronger argument when this stuff was originally written, before
ZONE_DEVICEwas invented.Quoted 4 lines
I initially though that
phys_addr_tbased API will somehow simplify arch specific implementation, as some of them indeed rely onphys_addr_tinternally, but I missed other things pointed by Robin. Do we have here any alternative?I think it is less of a code simplification, more as a reduction in conceptual load. When we can say directly there is no struct page type anyhwere in the DMA API layers then we only have to reason about kmap/phys_to_virt compatibly.
This is also a weaker overall requirement than needing an actual struct page which allows optimizing other parts of the kernel. Like we aren't forced to create
MEMORY_DEVICE_PCI_P2PDMAstuct pages just to use the dma api.Again, any place in the kernel we can get rid of struct page the smoother the road will be for the MM side struct page restructuring.
For example one of the bigger eventual goes here is to make a
bio_vecstorephys_addr_t, not struct page pointers.DMA API is not alone here, we have been de-struct-paging the kernel for a long time now:
Quoted 8 lines
netdev: https://lore.kernel.org/linux-mm/[email protected]/
slab: https://lore.kernel.org/linux-mm/[email protected]/
iommmu: https://lore.kernel.org/all/[email protected]/
page tables: https://lore.kernel.org/linux-mm/[email protected]/
zswap: https://lore.kernel.org/all/[email protected]/With a long term goal that struct page only exists for legacy code, and is maybe entirely compiled out of modern server kernels.
Why just server kernels? I suspect client systems actually run newer kernels than servers do.
Signature · 2 lines
Sincerely, Demi Marie Obenour (she/her/hers)
Jason Gunthorpe
maintainerQuoted 5 lines · Demi Marie Obenour
Quoted 2 lines
With a long term goal that struct page only exists for legacy code, and is maybe entirely compiled out of modern server kernels.
Why just server kernels? I suspect client systems actually run newer kernels than servers do.
I would guess this is because of the people who are interested in this work. Frankly there isn't much benifit for small memory client systems. Modern servers have > 1TB of memory and struct page really hurts here.
The flip side of this is the work is enormous and I think there is a general idea that the smaller set of server related drivers and subsystems will get ready well before the wider universe of stuff a client or android might use.
It is not that more can't happen it just ultimately depends on interest and time.
Many modern servers use quite new kernels if you ignore the enterprise distros :\
Jason
Re: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
In reply to Jason GunthorpeQuoted 11 lines · Jason Gunthorpe
Quoted 10 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Convert the KMSAN DMA handling function from page-based to physical address-based interface.
The refactoring renames
kmsan_handle_dma()parameters from accepting (struct page *page,size_toffset,size_tsize) to (phys_addr_tphys,size_tsize). APFN_VALIDcheck is added to prevent KMSAN operations on non-page memory, preventing from non struct page backed address,
Quoted 8 lines
Quoted 4 lines
As part of this change, support for highmem addresses is implemented using
kmap_local_page()to handle both lowmem and highmem regions properly. All callers throughout the codebase are updated to use the newphys_addr_tbased interface.Use the function Matthew pointed at
kmap_local_pfn()Maybe introduce the
kmap_local_phys()he suggested too.
At this point it gives nothing.
Quoted 10 lines
Quoted 10 lines
/* Helper function to handle DMA data transfers. */
mm/kmsan/hooks.c +3
-1-void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) {u64 page_offset, to_go, addr; +struct page *page; +void *kaddr;
Quoted 8 lines
mm/kmsan/hooks.c static void kmsan_handle_dma_page(const void *addr, size_t size, · line 347
Quoted 3 lines
mm/kmsan/hooks.c +1
-1-if (PageHighMem (page)) +if (!pfn_valid (PHYS_PFN (phys)))return ;Not needed, the caller must pass in a phys that is kmap compatible. Maybe just leave a comment. FWIW today this is also not checking for P2P or DEVICE non-kmap struct pages either, so it should be fine without checks.
It is not true as we will call to kmsan_handle_dma() unconditionally in dma_map_phys(). The reason to it is that kmsan_handle_dma() is guarded with debug kconfig options and cost of pfn_valid() can be accommodated in that case. It gives more clean DMA code.
155 dma_addr_t dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,
156 enum dma_data_direction dir, unsigned long attrs)
157 {
<...>
187
188 kmsan_handle_dma(phys, size, dir); 189 trace_dma_map_phys(dev, phys, addr, size, dir, attrs); 190 debug_dma_map_phys(dev, phys, size, dir, addr, attrs);
191
192 return addr;
193 }
194 EXPORT_SYMBOL_GPL(dma_map_phys);
So let's keep this patch as is.
Thanks
Re: [PATCH v1 16/16] nvme-pci: unmap MMIO pages with appropriate interface
In reply to Jason GunthorpeQuoted 7 lines · Jason Gunthorpe
Quoted 5 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Block layer maps MMIO memory through
dma_map_phys()interface with help ofDMA_ATTR_MMIOattribute. There is a need to unmap that memory with the appropriate unmap function.
Quoted 6 lines
Be specific, AFIACT the issue is that on
dma_opsplatforms the map will callops->map_resourceforATTR_MMIOso we must have the unmap callops->unmap_resournceMaybe these patches should be swapped then, as adding
ATTR_MMIOseems like it created this issue?
The best variant will be to squash previous patch "block-dma: properly take MMIO path", but I don't want to mix them as they for different kernel areas.
Thanks
Quoted 3 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
In reply to Leon RomanovskyQuoted 18 lines · Leon Romanovsky
Quoted 9 lines
Quoted 9 lines
/* Helper function to handle DMA data transfers. */
mm/kmsan/hooks.c +3
-1-void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) {u64 page_offset, to_go, addr; +struct page *page; +void *kaddr;Quoted 8 lines
mm/kmsan/hooks.c static void kmsan_handle_dma_page(const void *addr, size_t size, · line 347
Quoted 3 lines
mm/kmsan/hooks.c +1
-1-if (PageHighMem (page)) +if (!pfn_valid (PHYS_PFN (phys)))return ;Not needed, the caller must pass in a phys that is kmap compatible. Maybe just leave a comment. FWIW today this is also not checking for P2P or DEVICE non-kmap struct pages either, so it should be fine without checks.
Quoted 4 lines
It is not true as we will call to
kmsan_handle_dma()unconditionally indma_map_phys(). The reason to it is thatkmsan_handle_dma()is guarded with debug kconfig options and cost ofpfn_valid()can be accommodated in that case. It gives more clean DMA code.
Then check attrs here, not pfn_valid.
Quoted 1 line
So let's keep this patch as is.
Still need to fix the remarks you clipped, do not check PageHighMem just call kmap_local_pfn(). All thie PageHighMem stuff is new to this patch and should not be here, it is the wrong way to use highmem.
Jason
Re: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
Quoted 19 lines · Jason Gunthorpe
Quoted 18 lines · Leon Romanovsky
Quoted 17 lines
mm/kmsan/hooks.c static void kmsan_handle_dma_page(const void *addr, size_t size, · line 347
Quoted 12 lines
/* Helper function to handle DMA data transfers. */
mm/kmsan/hooks.c +4
-2-void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) {u64 page_offset, to_go, addr; +struct page *page; +void *kaddr; -if (PageHighMem (page)) +if (!pfn_valid (PHYS_PFN (phys)))return ;Not needed, the caller must pass in a phys that is kmap compatible. Maybe just leave a comment. FWIW today this is also not checking for P2P or DEVICE non-kmap struct pages either, so it should be fine without checks.
Quoted 6 lines
Quoted 4 lines
It is not true as we will call to
kmsan_handle_dma()unconditionally indma_map_phys(). The reason to it is thatkmsan_handle_dma()is guarded with debug kconfig options and cost ofpfn_valid()can be accommodated in that case. It gives more clean DMA code.Then check attrs here, not
pfn_valid.
attrs are not available in kmsan_handle_dma(). I can add it if you prefer.
Quoted 6 lines
Quoted 2 lines
So let's keep this patch as is.
Still need to fix the remarks you clipped, do not check PageHighMem just call
kmap_local_pfn(). All thie PageHighMem stuff is new to this patch and should not be here, it is the wrong way to use highmem.
Sure, thanks
Quoted 3 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
Quoted 3 lines · Leon Romanovsky
Quoted 1 line
Then check attrs here, not
pfn_valid.attrs are not available in
kmsan_handle_dma(). I can add it if you prefer.
That makes more sense to the overall design. The comments I gave before were driving at a promise to never try to touch a struct page for ATTR_MMIO and think this should be comphrensive to never touching a struct page even if pfnvalid.
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
So let's keep this patch as is.
Still need to fix the remarks you clipped, do not check PageHighMem just call
kmap_local_pfn(). All thie PageHighMem stuff is new to this patch and should not be here, it is the wrong way to use highmem.Sure, thanks
I am wondering if there is some reason it was written like this in the first place. Maybe we can't even do kmap here.. So perhaps if there is not a strong reason to change it just continue to check pagehighmem and fail.
if (!(attrs & ATTR_MMIO) && PageHighMem(phys_to_page(phys)))
return;
Jason
Re: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
Quoted 18 lines · Jason Gunthorpe
Quoted 3 lines · Leon Romanovsky
Quoted 1 line
Then check attrs here, not
pfn_valid.attrs are not available in
kmsan_handle_dma(). I can add it if you prefer.That makes more sense to the overall design. The comments I gave before were driving at a promise to never try to touch a struct page for
ATTR_MMIOand think this should be comphrensive to never touching a struct page even if pfnvalid.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
So let's keep this patch as is.
Still need to fix the remarks you clipped, do not check PageHighMem just call
kmap_local_pfn(). All thie PageHighMem stuff is new to this patch and should not be here, it is the wrong way to use highmem.Sure, thanks
Quoted 7 lines
I am wondering if there is some reason it was written like this in the first place. Maybe we can't even do kmap here.. So perhaps if there is not a strong reason to change it just continue to check pagehighmem and fail.
if (!(attrs &
ATTR_MMIO) && PageHighMem(phys_to_page(phys)))
return;
Does this version good enough? There is no need to call to kmap_local_pfn() if we prevent PageHighMem pages.
mm/kmsan/hooks.c +4 -11
@@ -337,13 +337,13 @@ static void kmsan_handle_dma_page(const void *addr, size_t size,/* Helper function to handle DMA data transfers. */ void kmsan_handle_dma (phys_addr_t phys,size_t size, -enum dma_data_direction dir) +enum dma_data_direction dir,unsigned long attrs) {u64 page_offset, to_go, addr;struct page *page;void *kaddr; -if (!pfn_valid (PHYS_PFN (phys))) +if ((attrs &ATTR_MMIO ) ||PageHighMem (phys_to_page (phys)))return ; page =phys_to_page (phys);
@@ -357,19 +357,12 @@ void kmsan_handle_dma(phys_addr_t phys, size_t size,while (size >0 ) { to_go =min (PAGE_SIZE - page_offset, (u64 )size); -if (PageHighMem (page)) -/* Handle highmem pages using kmap */ - kaddr =kmap_local_page (page); -else -/* Lowmem pages can be accessed directly */ - kaddr =page_address (page); +/* Lowmem pages can be accessed directly */ + kaddr =page_address (page); addr = (u64 )kaddr + page_offset;kmsan_handle_dma_page ((void *)addr, to_go, dir); -if (PageHighMem (page)) -kunmap_local (page); - phys += to_go; size -= to_go;
(END)
Quoted 3 lines
Jason
Jason Gunthorpe
maintainerRe: [PATCH v1 08/16] kmsan: convert kmsan_handle_dma to use physical addresses
Quoted 24 lines · Leon Romanovsky
Quoted 23 lines · Jason Gunthorpe
Quoted 3 lines · Leon Romanovsky
Quoted 1 line
Then check attrs here, not
pfn_valid.attrs are not available in
kmsan_handle_dma(). I can add it if you prefer.That makes more sense to the overall design. The comments I gave before were driving at a promise to never try to touch a struct page for
ATTR_MMIOand think this should be comphrensive to never touching a struct page even if pfnvalid.Quoted 7 lines
Quoted 5 lines
Quoted 1 line
So let's keep this patch as is.
Still need to fix the remarks you clipped, do not check PageHighMem just call
kmap_local_pfn(). All thie PageHighMem stuff is new to this patch and should not be here, it is the wrong way to use highmem.Sure, thanks
I am wondering if there is some reason it was written like this in the first place. Maybe we can't even do kmap here.. So perhaps if there is not a strong reason to change it just continue to check pagehighmem and fail.
Quoted 5 lines
Quoted 2 lines
if (!(attrs &
ATTR_MMIO) && PageHighMem(phys_to_page(phys)))
return;Does this version good enough? There is no need to call to
kmap_local_pfn()if we prevent PageHighMem pages.
Why make the rest of the changes though, isn't it just:
if (PageHighMem(page))
return;Becomes:
if (attrs & ATTR_MMIO))
return;
page = phys_to_page(phys);
if (PageHighMem(page))
return;Leave the rest as is?
Jason