dma-mapping: migrate to physical address-based API
Subsystems (17)
v6 cover dma-mapping: migrate to physical address-based API
From: Leon Romanovsky <[email protected]>
Changelog:
v6:
* Based on "dma-debug: don't enforce dma mapping check on noncoherent allocations" patch.
* Removed some unused variables from kmsan conversion.
* Fixed missed ! in dma check.
v5: https://lore.kernel.org/all/[email protected]
* Added Jason's and Keith's Reviewed-by tags
* Fixed DMA_ATTR_MMIO check in dma_direct_map_phys
* Jason's cleanup suggestions
v4: https://lore.kernel.org/all/[email protected]/
* Fixed kbuild error with mismatch in kmsan function declaration due to rebase error.
v3: https://lore.kernel.org/all/[email protected]
* Fixed typo in "cacheable" word
* Simplified kmsan patch a lot to be simple argument refactoring
v2: https://lore.kernel.org/all/[email protected]
* Used commit messages and cover letter from Jason
* Moved setting IOMMU_MMIO flag to dma_info_to_prot function
* Micro-optimized the code
* Rebased code on v6.17-rc1
v1: https://lore.kernel.org/all/[email protected]
* Added new DMA_ATTR_MMIO attribute to indicate PCI_P2PDMA_MAP_THRU_HOST_BRIDGE path.
* Rewrote dma_map_* functions to use thus new attribute
v0: https://lore.kernel.org/all/[email protected]/
------------------------------------------------------------------------
This series refactors the DMA mapping to use physical addresses as the primary interface instead of page+offset parameters. This change aligns the DMA API with the underlying hardware reality where DMA operations work with physical addresses, not page structures.
The series maintains export symbol backward compatibility by keeping the old page-based API as wrapper functions around the new physical address-based implementations.
This series refactors the DMA mapping API to provide a phys_addr_t based, and struct-page free, external API that can handle all the mapping cases we want in modern systems:
- struct page based cacheable DRAM
- struct page MEMORY_DEVICE_PCI_P2PDMA PCI peer to peer non-cacheable MMIO
- struct page-less PCI peer to peer non-cacheable MMIO
- struct page-less "resource" MMIO
Overall this gets much closer to Matthew's long term wish for struct-pageless IO to cacheable DRAM. The remaining primary work would be in the mm side to allow kmap_local_pfn()/phys_to_virt() to work on phys_addr_t without a struct page.
The general design is to remove struct page usage entirely from the DMA API inner layers. For flows that need to have a KVA for the physical address they can use kmap_local_pfn() or phys_to_virt(). This isolates the struct page requirements to MM code only. Long term all removals of struct page usage are supporting Matthew's memdesc project which seeks to substantially transform how struct page works.
Instead make the DMA API internals work on phys_addr_t. Internally there are still dedicated 'page' and 'resource' flows, except they are now distinguished by a new DMA_ATTR_MMIO instead of by callchain. Both flows use the same phys_addr_t.
When DMA_ATTR_MMIO is specified things work similar to the existing 'resource' flow. kmap_local_pfn(), phys_to_virt(), phys_to_page(), pfn_valid(), etc are never called on the phys_addr_t. This requires rejecting any configuration that would need swiotlb. CPU cache flushing is not required, and avoided, as ATTR_MMIO also indicates the address have no cacheable mappings. This effectively removes any DMA API side requirement to have struct page when DMA_ATTR_MMIO is used.
In the !DMA_ATTR_MMIO mode things work similarly to the 'page' flow, except on the common path of no cache flush, no swiotlb it never touches a struct page. When cache flushing or swiotlb copying kmap_local_pfn()/phys_to_virt() are used to get a KVA for CPU usage. This was already the case on the unmap side, now the map side is symmetric.
Callers are adjusted to set DMA_ATTR_MMIO. Existing 'resource' users must set it. The existing struct page based MEMORY_DEVICE_PCI_P2PDMA path must also set it. This corrects some existing bugs where iommu mappings for P2P MMIO were improperly marked IOMMU_CACHE.
Since ATTR_MMIO is made to work with all the existing DMA map entry points, particularly dma_iova_link(), this finally allows a way to use the new DMA API to map PCI P2P MMIO without creating struct page. The VFIO DMABUF series demonstrates how this works. This is intended to replace the incorrect driver use of dma_map_resource() on PCI BAR addresses.
This series does the core code and modern flows. A followup series will give the same treatment to the legacy dma_ops implementation.
Thanks
Leon Romanovsky (16):
dma-mapping: introduce new DMA attribute to indicate MMIO memory iommu/dma: implement DMA_ATTR_MMIO for dma_iova_link().
dma-debug: refactor to use physical addresses for page mapping
dma-mapping: rename trace_dma_*map_page to trace_dma_*map_phys iommu/dma: rename iommu_dma_*map_page to iommu_dma_*map_phys iommu/dma: implement DMA_ATTR_MMIO for iommu_dma_(un)map_phys()
dma-mapping: convert dma_direct_*map_page to be phys_addr_t based
kmsan: convert kmsan_handle_dma to use physical addresses
dma-mapping: implement DMA_ATTR_MMIO for dma_(un)map_page_attrs()
xen: swiotlb: Open code map_resource callback
dma-mapping: export new dma_*map_phys() interface mm/hmm: migrate to physical address-based DMA mapping API mm/hmm: properly take MMIO path
block-dma: migrate to dma_map_phys instead of map_page
block-dma: properly take MMIO path
nvme-pci: unmap MMIO pages with appropriate interface
Documentation/core-api/dma-api.rst | 4 +- Documentation/core-api/dma-attributes.rst | 18 ++++ arch/powerpc/kernel/dma-iommu.c | 4 +- block/blk-mq-dma.c | 15 ++- drivers/iommu/dma-iommu.c | 61 ++++++------ drivers/nvme/host/pci.c | 18 +++- drivers/virtio/virtio_ring.c | 4 +- drivers/xen/swiotlb-xen.c | 21 +++- include/linux/blk-mq-dma.h | 6 +- include/linux/blk_types.h | 2 + include/linux/dma-direct.h | 2 - include/linux/dma-map-ops.h | 8 +- include/linux/dma-mapping.h | 33 +++++++ include/linux/iommu-dma.h | 11 +-- include/linux/kmsan.h | 9 +- include/linux/page-flags.h | 1 + include/trace/events/dma.h | 9 +- kernel/dma/debug.c | 82 ++++------------ kernel/dma/debug.h | 37 ++----- kernel/dma/direct.c | 22 +---- kernel/dma/direct.h | 57 +++++++---- kernel/dma/mapping.c | 112 +++++++++++++--------- kernel/dma/ops_helpers.c | 6 +- mm/hmm.c | 19 ++-- mm/kmsan/hooks.c | 10 +- rust/kernel/dma.rs | 3 + tools/virtio/linux/kmsan.h | 2 +- 27 files changed, 312 insertions(+), 264 deletions(-)
Signature · 1 line
2.51.0
PATCH v6 1/16 dma-mapping: introduce new DMA attribute to indicate MMIO memory
From: Leon Romanovsky <[email protected]>
This patch introduces the DMA_ATTR_MMIO attribute to mark DMA buffers that reside in memory-mapped I/O (MMIO) regions, such as device BARs exposed through the host bridge, which are accessible for peer-to-peer (P2P) DMA.
This attribute is especially useful for exporting device memory to other devices for DMA without CPU involvement, and avoids unnecessary or potentially detrimental CPU cache maintenance calls.
DMA_ATTR_MMIO is supposed to provide dma_map_resource() functionality without need to call to special function and perform branching when processing generic containers like bio_vec by the callers.
--- Documentation/core-api/dma-attributes.rst | 18 ++++++++++++++++++ include/linux/dma-mapping.h | 20 ++++++++++++++++++++ include/trace/events/dma.h | 3 ++- rust/kernel/dma.rs | 3 +++ 4 files changed, 43 insertions(+), 1 deletion(-)
Documentation/core-api/dma-attributes.rst +18 -0
@@ -130,3 +130,21 @@ accesses to DMA buffers in both privileged "supervisor" and unprivileged subsystem that the buffer is fully accessible at the elevated privilege level (and ideally inaccessible or at least read-only at the lesser-privileged levels). + +DMA_ATTR_MMIO +------------- + +This attribute indicates the physical address is not normal system +memory. It may not be used with kmap*()/phys_to_virt()/phys_to_page() +functions, it may not be cacheable, and access using CPU load/store +instructions may not be allowed. + +Usually this will be used to describe MMIO addresses, or other non-cacheable +register addresses. When DMA mapping this sort of address we call +the operation Peer to Peer as a one device is DMA'ing to another device. +For PCI devices the p2pdma APIs must be used to determine if +DMA_ATTR_MMIO is appropriate. + +For architectures that require cache flushing for DMA coherence +DMA_ATTR_MMIO will not perform any cache flushing. The address +provided must never be mapped cacheable into the CPU.
include/linux/dma-mapping.h +20 -0
@@ -58,6 +58,26 @@*/ #define DMA_ATTR_PRIVILEGED (1UL << 9) +/* +* DMA_ATTR_MMIO - Indicates memory-mapped I/O (MMIO) region for DMA mapping +* +* This attribute indicates the physical address is not normal system +* memory. It may not be used with kmap*()/phys_to_virt()/phys_to_page() +* functions, it may not be cacheable, and access using CPU load/store +* instructions may not be allowed. +* +* Usually this will be used to describe MMIO addresses, or other non-cacheable +* register addresses. When DMA mapping this sort of address we call +* the operation Peer to Peer as a one device is DMA'ing to another device. +* For PCI devices the p2pdma APIs must be used to determine if DMA_ATTR_MMIO +* is appropriate. +* +* For architectures that require cache flushing for DMA coherence +* DMA_ATTR_MMIO will not perform any cache flushing. The address +* provided must never be mapped cacheable into the CPU. +*/ +#define DMA_ATTR_MMIO (1UL << 10) +/* * A dma_addr_t can hold any valid DMA or bus address for the platform. It can * be given to a device to use as a DMA source or target. It is specific to a
include/trace/events/dma.h +2 -1
@@ -31,7 +31,8 @@ TRACE_DEFINE_ENUM(DMA_NONE); { DMA_ATTR_FORCE_CONTIGUOUS, "FORCE_CONTIGUOUS" }, \ { DMA_ATTR_ALLOC_SINGLE_PAGES, "ALLOC_SINGLE_PAGES" }, \ { DMA_ATTR_NO_WARN, "NO_WARN" }, \ - { DMA_ATTR_PRIVILEGED, "PRIVILEGED" }) + { DMA_ATTR_PRIVILEGED, "PRIVILEGED" }, \ + { DMA_ATTR_MMIO, "MMIO" }) DECLARE_EVENT_CLASS(dma_map, TP_PROTO(struct device *dev, phys_addr_t phys_addr, dma_addr_t dma_addr,
rust/kernel/dma.rs +3 -0
@@ -242,6 +242,9 @@ pub mod attrs {/// Indicates that the buffer is fully accessible at an elevated privilege level (and /// ideally inaccessible or at least read-only at lesser-privileged levels). pub const DMA_ATTR_PRIVILEGED :Attrs =Attrs (bindings::DMA_ATTR_PRIVILEGED ); + +/// Indicates that the buffer is MMIO memory. +pub const DMA_ATTR_MMIO :Attrs =Attrs (bindings::DMA_ATTR_MMIO ); }/// An abstraction of the `dma_alloc_coherent` API.
Signature · 1 line
2.51.0
PATCH v6 2/16 iommu/dma: implement DMA_ATTR_MMIO for dma_iova_link().
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
This will replace the hacky use of DMA_ATTR_SKIP_CPU_SYNC to avoid touching the possibly non-KVA MMIO memory.
Also correct the incorrect caching attribute for the IOMMU, MMIO memory should not be cachable inside the IOMMU mapping or it can possibly create system problems. Set IOMMU_MMIO for DMA_ATTR_MMIO.
--- drivers/iommu/dma-iommu.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-)
drivers/iommu/dma-iommu.c +14 -4
@@ -724,7 +724,12 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, struct device *devstatic int dma_info_to_prot (enum dma_data_direction dir,bool coherent,unsigned long attrs) { -int prot = coherent ?IOMMU_CACHE :0 ; +int prot; + +if (attrs &DMA_ATTR_MMIO ) + prot =IOMMU_MMIO ; +else + prot = coherent ?IOMMU_CACHE :0 ;if (attrs &DMA_ATTR_PRIVILEGED ) prot |=IOMMU_PRIV ;
@@ -1838,12 +1843,13 @@ static int __dma_iova_link(struct device *dev, dma_addr_t addr, unsigned long attrs) {bool coherent =dev_is_dma_coherent (dev); +int prot =dma_info_to_prot (dir, coherent, attrs); -if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC )) +if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC |DMA_ATTR_MMIO )))arch_sync_dma_for_device (phys, size, dir);return iommu_map_nosync (iommu_get_dma_domain (dev), addr, phys, size, -dma_info_to_prot (dir, coherent, attrs),GFP_ATOMIC ); + prot,GFP_ATOMIC ); } static int iommu_dma_iova_bounce_and_link(struct device *dev, dma_addr_t addr,
@@ -1949,9 +1955,13 @@ int dma_iova_link(struct device *dev, struct dma_iova_state *state,return -EIO ;if (dev_use_swiotlb (dev, size, dir) && -iova_unaligned (iovad, phys, size)) +iova_unaligned (iovad, phys, size)) { +if (attrs &DMA_ATTR_MMIO ) +return -EPERM ; +return iommu_dma_iova_link_swiotlb (dev, state, phys, offset, size, dir, attrs); + } return __dma_iova_link(dev, state->addr + offset - iova_start_pad, phys - iova_start_pad,
Signature · 1 line
2.51.0
PATCH v6 3/16 dma-debug: refactor to use physical addresses for page mapping
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert the DMA debug infrastructure from page-based to physical address-based mapping as a preparation to rely on physical address for DMA mapping routines.
The refactoring renames debug_dma_map_page() to debug_dma_map_phys() and changes its signature to accept a phys_addr_t parameter instead of struct page and offset. Similarly, debug_dma_unmap_page() becomes debug_dma_unmap_phys(). A new dma_debug_phy type is introduced to distinguish physical address mappings from other debug entry types. All callers throughout the codebase are updated to pass physical addresses directly, eliminating the need for page-to-physical conversion in the debug layer.
This refactoring eliminates the need to convert between page pointers and physical addresses in the debug layer, making the code more efficient and consistent with the DMA mapping API's physical address focus.
--- Documentation/core-api/dma-api.rst | 4 +-- include/linux/page-flags.h | 1 + kernel/dma/debug.c | 39 +++++++++++++++--------------- kernel/dma/debug.h | 16 ++++++------ kernel/dma/mapping.c | 10 ++++---- 5 files changed, 35 insertions(+), 35 deletions(-)
Documentation/core-api/dma-api.rst +2 -2
@@ -761,7 +761,7 @@ example warning message may look like this:: [<ffffffff80235177>] find_busiest_group+0x207/0x8a0 [<ffffffff8064784f>] _spin_lock_irqsave+0x1f/0x50 [<ffffffff803c7ea3>] check_unmap+0x203/0x490 - [<ffffffff803c8259>] debug_dma_unmap_page+0x49/0x50 + [<ffffffff803c8259>] debug_dma_unmap_phys+0x49/0x50 [<ffffffff80485f26>] nv_tx_done_optimized+0xc6/0x2c0 [<ffffffff80486c13>] nv_nic_irq_optimized+0x73/0x2b0 [<ffffffff8026df84>] handle_IRQ_event+0x34/0x70
@@ -855,7 +855,7 @@ that a driver may be leaking mappings. dma-debug interface debug_dma_mapping_error() to debug drivers that fail to check DMA mapping errors on addresses returned by dma_map_single() and dma_map_page() interfaces. This interface clears a flag set by -debug_dma_map_page() to indicate that dma_mapping_error() has been called by +debug_dma_map_phys() to indicate that dma_mapping_error() has been called by the driver. When driver does unmap, debug_dma_unmap() checks the flag and if this flag is still set, prints warning message that includes call trace that leads up to the unmap. This interface can be called from dma_mapping_error()
include/linux/page-flags.h +1 -0
@@ -614,6 +614,7 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)* available at this point. */ #define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE (HighMem, highmem)
kernel/dma/debug.c +20 -19
@@ -40,6 +40,7 @@ enum { dma_debug_coherent, dma_debug_resource, dma_debug_noncoherent, + dma_debug_phy, }; enum map_err_types {
@@ -143,6 +144,7 @@ static const char *type2name[] = { [dma_debug_coherent] = "coherent", [dma_debug_resource] = "resource", [dma_debug_noncoherent] = "noncoherent", + [dma_debug_phy] = "phy", };static const char *dir2name[] = {
@@ -1054,17 +1056,16 @@ static void check_unmap(struct dma_debug_entry *ref)dma_entry_free (entry); } -static void check_for_stack (struct device *dev, -struct page *page,size_t offset) +static void check_for_stack (struct device *dev,phys_addr_t phys) {void *addr;struct vm_struct *stack_vm_area =task_stack_vm_area (current);if (!stack_vm_area) {/* Stack is direct-mapped. */ -if (PageHighMem (page)) +if (PhysHighMem (phys))return ; - addr =page_address (page) + offset; + addr =phys_to_virt (phys);if (object_is_on_stack (addr))err_printk (dev,NULL ,"device driver maps memory from stack [addr=%p]\n" , addr); }else {
@@ -1072,10 +1073,12 @@ static void check_for_stack(struct device *dev,int i;for (i =0 ; i < stack_vm_area->nr_pages; i++) { -if (page != stack_vm_area->pages[i]) +if (__phys_to_pfn (phys) != +page_to_pfn (stack_vm_area->pages[i]))continue ; - addr = (u8 *)current->stack + i *PAGE_SIZE + offset; + addr = (u8 *)current->stack + i *PAGE_SIZE + + (phys %PAGE_SIZE );err_printk (dev,NULL ,"device driver maps memory from stack [probable addr=%p]\n" , addr);break ; }
@@ -1204,9 +1207,8 @@ void debug_dma_map_single(struct device *dev, const void *addr, }EXPORT_SYMBOL (debug_dma_map_single); -void debug_dma_map_page (struct device *dev,struct page *page,size_t offset, -size_t size,int direction,dma_addr_t dma_addr, -unsigned long attrs) +void debug_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +int direction,dma_addr_t dma_addr,unsigned long attrs) {struct dma_debug_entry *entry;
@@ -1221,19 +1223,18 @@ void debug_dma_map_page(struct device *dev, struct page *page, size_t offset,return ; entry->dev = dev; - entry->type = dma_debug_single; - entry->paddr =page_to_phys (page) + offset; + entry->type = dma_debug_phy; + entry->paddr = phys; entry->dev_addr = dma_addr; entry->size = size; entry->direction = direction; entry->map_err_type =MAP_ERR_NOT_CHECKED ; -check_for_stack (dev, page, offset); +if (!(attrs &DMA_ATTR_MMIO )) { +check_for_stack (dev, phys); -if (!PageHighMem (page)) { -void *addr =page_address (page) + offset; - -check_for_illegal_area (dev, addr, size); +if (!PhysHighMem (phys)) +check_for_illegal_area (dev,phys_to_virt (phys), size); }add_dma_entry (entry, attrs);
@@ -1277,11 +1278,11 @@ void debug_dma_mapping_error(struct device *dev, dma_addr_t dma_addr) }EXPORT_SYMBOL (debug_dma_mapping_error); -void debug_dma_unmap_page (struct device *dev,dma_addr_t dma_addr, +void debug_dma_unmap_phys (struct device *dev,dma_addr_t dma_addr,size_t size,int direction) {struct dma_debug_entry ref = { - .type = dma_debug_single, + .type = dma_debug_phy, .dev = dev, .dev_addr = dma_addr, .size = size,
@@ -1305,7 +1306,7 @@ void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,return ;for_each_sg (sg, s, nents, i) { -check_for_stack (dev,sg_page (s), s->offset); +check_for_stack (dev,sg_phys (s));if (!PageHighMem (sg_page (s)))check_for_illegal_area (dev,sg_virt (s), s->length); }
kernel/dma/debug.h +7 -9
@@ -9,12 +9,11 @@#define _KERNEL_DMA_DEBUG_H#ifdef CONFIG_DMA_API_DEBUG -extern void debug_dma_map_page (struct device *dev,struct page *page, -size_t offset,size_t size, -int direction,dma_addr_t dma_addr, +extern void debug_dma_map_phys (struct device *dev,phys_addr_t phys, +size_t size,int direction,dma_addr_t dma_addr,unsigned long attrs); -extern void debug_dma_unmap_page (struct device *dev,dma_addr_t addr, +extern void debug_dma_unmap_phys (struct device *dev,dma_addr_t addr,size_t size,int direction); extern void debug_dma_map_sg(struct device *dev, struct scatterlist *sg,
@@ -62,14 +61,13 @@ extern void debug_dma_free_pages(struct device *dev, struct page *page, size_t size, int direction, dma_addr_t dma_addr);#else /* CONFIG_DMA_API_DEBUG */ -static inline void debug_dma_map_page (struct device *dev,struct page *page, -size_t offset,size_t size, -int direction,dma_addr_t dma_addr, -unsigned long attrs) +static inline void debug_dma_map_phys (struct device *dev,phys_addr_t phys, +size_t size,int direction, +dma_addr_t dma_addr,unsigned long attrs) { } -static inline void debug_dma_unmap_page (struct device *dev,dma_addr_t addr, +static inline void debug_dma_unmap_phys (struct device *dev,dma_addr_t addr,size_t size,int direction) { }
kernel/dma/mapping.c +5 -5
@@ -157,6 +157,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, unsigned long attrs) {const struct dma_map_ops *ops =get_dma_ops (dev); +phys_addr_t phys =page_to_phys (page) + offset;dma_addr_t addr;BUG_ON (!valid_dma_direction (dir));
@@ -165,16 +166,15 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_page_direct (dev,page_to_phys (page) + offset + size)) +arch_dma_map_page_direct (dev, phys + size)) addr =dma_direct_map_page (dev, page, offset, size, dir, attrs);else if (use_dma_iommu (dev)) addr =iommu_dma_map_page (dev, page, offset, size, dir, attrs);else addr = ops->map_page (dev, page, offset, size, dir, attrs);kmsan_handle_dma (page, offset, size, dir); -trace_dma_map_page (dev,page_to_phys (page) + offset, addr, size, dir, - attrs); -debug_dma_map_page (dev, page, offset, size, dir, addr, attrs); +trace_dma_map_page (dev, phys, addr, size, dir, attrs); +debug_dma_map_phys (dev, phys, size, dir, addr, attrs);return addr; }
@@ -194,7 +194,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,else ops->unmap_page (dev, addr, size, dir, attrs);trace_dma_unmap_page (dev, addr, size, dir, attrs); -debug_dma_unmap_page (dev, addr, size, dir); +debug_dma_unmap_phys (dev, addr, size, dir); }EXPORT_SYMBOL (dma_unmap_page_attrs);
Signature · 1 line
2.51.0
Re: [PATCH v6 03/16] dma-debug: refactor to use physical addresses for page mapping
Quoted 1 line · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
<...>
Quoted 1 line
include/linux/page-flags.h | 1 +
<...>
include/linux/page-flags.h FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) · line 617
Quoted 7 lines
include/linux/page-flags.h +1
-0@@ -614,6 +614,7 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)* available at this point. */ #define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))
This was a not so great idea to add PhysHighMem() because of "else" below which unfolds to maze of macros and automatically generated functions with "static inline int Page##uname ..." signature.
include/linux/page-flags.h FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) · line 620
Quoted 3 lines
include/linux/page-flags.h +0
-0#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE (HighMem, highmem)
Thanks
Re: [PATCH v6 03/16] dma-debug: refactor to use physical addresses for page mapping
Quoted 9 lines · Leon Romanovsky
Quoted 1 line · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
<...>
Quoted 1 line
include/linux/page-flags.h | 1 +<...>
Quoted 15 lines
include/linux/page-flags.h FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) · line 617
Quoted 7 lines
include/linux/page-flags.h +1
-0@@ -614,6 +614,7 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)* available at this point. */ #define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))This was a not so great idea to add
PhysHighMem()because of "else" below which unfolds to maze of macros and automatically generated functions with "static inline int Page##uname ..." signature.Quoted 3 lines
include/linux/page-flags.h +0
-0#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE (HighMem, highmem)
After sleeping over it, the following hunk will help:
include/linux/page-flags.h +1 -1
@@ -614,11 +614,11 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)* available at this point. */ #define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) -#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE(HighMem, highmem)#endif +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))/* Does kmap_local_folio() only allow access to one page of the folio? */ #ifdef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
Quoted 3 lines
Thanks
Jason Gunthorpe
maintainerRe: [PATCH v6 03/16] dma-debug: refactor to use physical addresses for page mapping
include/linux/page-flags.h FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) · line 621
Quoted 7 lines · Leon Romanovsky
#define
PageHighMem(__p)is_highmem_idx(page_zonenum(__p))include/linux/page-flags.h +1
-1-#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE(HighMem, highmem)#endif +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))
Yeah, that's what I imagined, and I'd make it a static inline
static inline bool PhysHighMem(phys_addr_t phys)
These existing macros are old fashioned imho.
Jason
Marek Szyprowski
maintainerRe: [PATCH v6 03/16] dma-debug: refactor to use physical addresses for page mapping
In reply to Leon RomanovskyQuoted 31 lines · Leon Romanovsky
Quoted 7 lines · Leon Romanovsky
Quoted 1 line · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
<...>
Quoted 1 line
include/linux/page-flags.h | 1 +<...>
Quoted 14 lines
include/linux/page-flags.h FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) · line 617
Quoted 7 lines
include/linux/page-flags.h +1
-0@@ -614,6 +614,7 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)* available at this point. */ #define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))This was a not so great idea to add
PhysHighMem()because of "else" below which unfolds to maze of macros and automatically generated functions with "static inline int Page##uname ..." signature.Quoted 3 lines
include/linux/page-flags.h +0
-0#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE (HighMem, highmem)After sleeping over it, the following hunk will help:
include/linux/page-flags.h +0
-0@@ -614,11 +614,11 @@ FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE)* available at this point. */
include/linux/page-flags.h FOLIO_FLAG(dropbehind, FOLIO_HEAD_PAGE) · line 624
Quoted 10 lines
include/linux/page-flags.h +1
-1#define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) -#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f))#else PAGEFLAG_FALSE(HighMem, highmem)#endif +#define PhysHighMem (__p) (PageHighMem(phys_to_page(__p)))/* Does kmap_local_folio() only allow access to one page of the folio? */ #ifdef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP
Okay, I will add this fixup while applying the patches.
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
PATCH v6 4/16 dma-mapping: rename trace_dma_*map_page to trace_dma_*map_phys
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
As a preparation for following map_page -> map_phys API conversion, let's rename trace_dma_*map_page() to be trace_dma_*map_phys().
--- include/trace/events/dma.h | 4 ++-- kernel/dma/mapping.c | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-)
include/trace/events/dma.h +2 -2
@@ -72,7 +72,7 @@ DEFINE_EVENT(dma_map, name, \ size_t size, enum dma_data_direction dir, unsigned long attrs), \ TP_ARGS(dev, phys_addr, dma_addr, size, dir, attrs)) -DEFINE_MAP_EVENT(dma_map_page); +DEFINE_MAP_EVENT(dma_map_phys);DEFINE_MAP_EVENT (dma_map_resource); DECLARE_EVENT_CLASS(dma_unmap,
@@ -110,7 +110,7 @@ DEFINE_EVENT(dma_unmap, name, \ enum dma_data_direction dir, unsigned long attrs), \ TP_ARGS(dev, addr, size, dir, attrs)) -DEFINE_UNMAP_EVENT(dma_unmap_page); +DEFINE_UNMAP_EVENT(dma_unmap_phys);DEFINE_UNMAP_EVENT (dma_unmap_resource); DECLARE_EVENT_CLASS(dma_alloc_class,
kernel/dma/mapping.c +2 -2
@@ -173,7 +173,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,else addr = ops->map_page (dev, page, offset, size, dir, attrs);kmsan_handle_dma (page, offset, size, dir); -trace_dma_map_page (dev, phys, addr, size, dir, attrs); +trace_dma_map_phys (dev, phys, addr, size, dir, attrs);debug_dma_map_phys (dev, phys, size, dir, addr, attrs);return addr;
@@ -193,7 +193,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,iommu_dma_unmap_page (dev, addr, size, dir, attrs); else ops->unmap_page(dev, addr, size, dir, attrs); -trace_dma_unmap_page (dev, addr, size, dir, attrs); +trace_dma_unmap_phys (dev, addr, size, dir, attrs);debug_dma_unmap_phys (dev, addr, size, dir); }EXPORT_SYMBOL (dma_unmap_page_attrs);
Signature · 1 line
2.51.0
PATCH v6 5/16 iommu/dma: rename iommu_dma_*map_page to iommu_dma_*map_phys
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Rename the IOMMU DMA mapping functions to better reflect their actual calling convention. The functions iommu_dma_map_page() and iommu_dma_unmap_page() are renamed to iommu_dma_map_phys() and iommu_dma_unmap_phys() respectively, as they already operate on physical addresses rather than page structures.
The calling convention changes from accepting (struct page *page, unsigned long offset) to (phys_addr_t phys), which eliminates the need for page-to-physical address conversion within the functions. This renaming prepares for the broader DMA API conversion from page-based to physical address-based mapping throughout the kernel.
All callers are updated to pass physical addresses directly, including dma_map_page_attrs(), scatterlist mapping functions, and DMA page allocation helpers. The change simplifies the code by removing the page_to_phys() + offset calculation that was previously done inside the IOMMU functions.
--- drivers/iommu/dma-iommu.c | 14 ++++++-------- include/linux/iommu-dma.h | 7 +++---- kernel/dma/mapping.c | 4 ++-- kernel/dma/ops_helpers.c | 6 +++--- 4 files changed, 14 insertions(+), 17 deletions(-)
drivers/iommu/dma-iommu.c +6 -8
@@ -1195,11 +1195,9 @@ static inline size_t iova_unaligned(struct iova_domain *iovad, phys_addr_t phys,return iova_offset (iovad, phys | size); } -dma_addr_t iommu_dma_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size,enum dma_data_direction dir, -unsigned long attrs) +dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs) { -phys_addr_t phys =page_to_phys (page) + offset;bool coherent =dev_is_dma_coherent (dev);int prot =dma_info_to_prot (dir, coherent, attrs);struct iommu_domain *domain =iommu_get_dma_domain (dev);
@@ -1227,7 +1225,7 @@ dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page,return iova; } -void iommu_dma_unmap_page (struct device *dev,dma_addr_t dma_handle, +void iommu_dma_unmap_phys (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir,unsigned long attrs) {struct iommu_domain *domain =iommu_get_dma_domain (dev);
@@ -1346,7 +1344,7 @@ static void iommu_dma_unmap_sg_swiotlb(struct device *dev, struct scatterlist *sint i;for_each_sg (sg, s, nents, i) -iommu_dma_unmap_page (dev,sg_dma_address (s), +iommu_dma_unmap_phys (dev,sg_dma_address (s),sg_dma_len (s), dir, attrs); }
@@ -1359,8 +1357,8 @@ static int iommu_dma_map_sg_swiotlb(struct device *dev, struct scatterlist *sg,sg_dma_mark_swiotlb (sg);for_each_sg (sg, s, nents, i) { -sg_dma_address (s) =iommu_dma_map_page (dev,sg_page (s), - s->offset, s->length, dir, attrs); +sg_dma_address (s) =iommu_dma_map_phys (dev,sg_phys (s), + s->length, dir, attrs);if (sg_dma_address (s) ==DMA_MAPPING_ERROR ) goto out_unmap;sg_dma_len (s) = s->length;
include/linux/iommu-dma.h +3 -4
@@ -21,10 +21,9 @@ static inline bool use_dma_iommu(struct device *dev) }#endif /* CONFIG_IOMMU_DMA */ -dma_addr_t iommu_dma_map_page (struct device *dev,struct page *page, -unsigned long offset,size_t size,enum dma_data_direction dir, -unsigned long attrs); -void iommu_dma_unmap_page (struct device *dev,dma_addr_t dma_handle, +dma_addr_t iommu_dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs); +void iommu_dma_unmap_phys (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir,unsigned long attrs);int iommu_dma_map_sg (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs);
kernel/dma/mapping.c +2 -2
@@ -169,7 +169,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, arch_dma_map_page_direct(dev, phys + size)) addr =dma_direct_map_page (dev, page, offset, size, dir, attrs); else if (use_dma_iommu(dev)) - addr =iommu_dma_map_page (dev, page, offset, size, dir, attrs); + addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); else addr = ops->map_page (dev, page, offset, size, dir, attrs);kmsan_handle_dma (page, offset, size, dir);
@@ -190,7 +190,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, arch_dma_unmap_page_direct(dev, addr + size))dma_direct_unmap_page (dev, addr, size, dir, attrs); else if (use_dma_iommu(dev)) - iommu_dma_unmap_page(dev, addr, size, dir, attrs); + iommu_dma_unmap_phys(dev, addr, size, dir, attrs); else ops->unmap_page(dev, addr, size, dir, attrs);trace_dma_unmap_phys (dev, addr, size, dir, attrs);
kernel/dma/ops_helpers.c +3 -3
@@ -72,8 +72,8 @@ struct page *dma_common_alloc_pages(struct device *dev, size_t size,return NULL ;if (use_dma_iommu (dev)) - *dma_handle =iommu_dma_map_page (dev, page,0 , size, dir, -DMA_ATTR_SKIP_CPU_SYNC ); + *dma_handle =iommu_dma_map_phys (dev,page_to_phys (page), size, + dir,DMA_ATTR_SKIP_CPU_SYNC );else *dma_handle = ops->map_page (dev, page,0 , size, dir,DMA_ATTR_SKIP_CPU_SYNC );
@@ -92,7 +92,7 @@ void dma_common_free_pages(struct device *dev, size_t size, struct page *page,const struct dma_map_ops *ops =get_dma_ops (dev);if (use_dma_iommu (dev)) -iommu_dma_unmap_page (dev, dma_handle, size, dir, +iommu_dma_unmap_phys (dev, dma_handle, size, dir,DMA_ATTR_SKIP_CPU_SYNC );else if (ops->unmap_page) ops->unmap_page(dev, dma_handle, size, dir,
Signature · 1 line
2.51.0
PATCH v6 6/16 iommu/dma: implement DMA_ATTR_MMIO for iommu_dma_(un)map_phys()
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Make iommu_dma_map_phys() and iommu_dma_unmap_phys() respect DMA_ATTR_MMIO.
DMA_ATTR_MMIO makes the functions behave the same as iommu_dma_(un)map_resource():
- No swiotlb is possible
- No cache flushing is done (ATTR_MMIO should not be cached memory)
- prot for iommu_map() has IOMMU_MMIO not IOMMU_CACHE
This is preparation for replacing iommu_dma_map_resource() callers with iommu_dma_map_phys(DMA_ATTR_MMIO) and removing iommu_dma_(un)map_resource().
--- drivers/iommu/dma-iommu.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
drivers/iommu/dma-iommu.c +11 -4
@@ -1211,16 +1211,19 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,*/ if (dev_use_swiotlb (dev, size, dir) &&iova_unaligned (iovad, phys, size)) { +if (attrs &DMA_ATTR_MMIO ) +return DMA_MAPPING_ERROR ; + phys =iommu_dma_map_swiotlb (dev, phys, size, dir, attrs);if (phys == (phys_addr_t )DMA_MAPPING_ERROR )return DMA_MAPPING_ERROR ; } -if (!coherent && !(attrs &DMA_ATTR_SKIP_CPU_SYNC )) +if (!coherent && !(attrs & (DMA_ATTR_SKIP_CPU_SYNC |DMA_ATTR_MMIO )))arch_sync_dma_for_device (phys, size, dir); iova =__iommu_dma_map (dev, phys, size, prot, dma_mask); -if (iova ==DMA_MAPPING_ERROR ) +if (iova ==DMA_MAPPING_ERROR && !(attrs &DMA_ATTR_MMIO ))swiotlb_tbl_unmap_single (dev, phys, size, dir, attrs);return iova; }
@@ -1228,10 +1231,14 @@ dma_addr_t iommu_dma_map_phys(struct device *dev, phys_addr_t phys, size_t size,void iommu_dma_unmap_phys (struct device *dev,dma_addr_t dma_handle,size_t size,enum dma_data_direction dir,unsigned long attrs) { -struct iommu_domain *domain =iommu_get_dma_domain (dev);phys_addr_t phys; - phys =iommu_iova_to_phys (domain, dma_handle); +if (attrs &DMA_ATTR_MMIO ) { +__iommu_dma_unmap (dev, dma_handle, size); +return ; + } + + phys =iommu_iova_to_phys (iommu_get_dma_domain (dev), dma_handle);if (WARN_ON (!phys))return ;
Signature · 1 line
2.51.0
PATCH v6 7/16 dma-mapping: convert dma_direct_*map_page to be phys_addr_t based
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert the DMA direct mapping functions to accept physical addresses directly instead of page+offset parameters. The functions were already operating on physical addresses internally, so this change eliminates the redundant page-to-physical conversion at the API boundary.
The functions dma_direct_map_page() and dma_direct_unmap_page() are renamed to dma_direct_map_phys() and dma_direct_unmap_phys() respectively, with their calling convention changed from (struct page *page, unsigned long offset) to (phys_addr_t phys).
Architecture-specific functions arch_dma_map_page_direct() and arch_dma_unmap_page_direct() are similarly renamed to arch_dma_map_phys_direct() and arch_dma_unmap_phys_direct().
The is_pci_p2pdma_page() checks are replaced with DMA_ATTR_MMIO checks to allow integration with dma_direct_map_resource and dma_direct_map_phys() is extended to support MMIO path either.
--- arch/powerpc/kernel/dma-iommu.c | 4 +-- include/linux/dma-map-ops.h | 8 ++--- kernel/dma/direct.c | 6 ++-- kernel/dma/direct.h | 57 +++++++++++++++++++++------------ kernel/dma/mapping.c | 8 ++--- 5 files changed, 49 insertions(+), 34 deletions(-)
arch/powerpc/kernel/dma-iommu.c +2 -2
@@ -14,7 +14,7 @@#define can_map_direct (dev, addr) \ ((dev)->bus_dma_limit >= phys_to_dma((dev), (addr))) -bool arch_dma_map_page_direct (struct device *dev,phys_addr_t addr) +bool arch_dma_map_phys_direct (struct device *dev,phys_addr_t addr) {if (likely (!dev->bus_dma_limit))return false;
@@ -24,7 +24,7 @@ bool arch_dma_map_page_direct(struct device *dev, phys_addr_t addr)#define is_direct_handle (dev, h) ((h) >= (dev)->archdata.dma_offset) -bool arch_dma_unmap_page_direct (struct device *dev,dma_addr_t dma_handle) +bool arch_dma_unmap_phys_direct (struct device *dev,dma_addr_t dma_handle) {if (likely (!dev->bus_dma_limit))return false;
include/linux/dma-map-ops.h +4 -4
@@ -392,15 +392,15 @@ void *arch_dma_set_uncached(void *addr, size_t size);void arch_dma_clear_uncached (void *addr,size_t size);#ifdef CONFIG_ARCH_HAS_DMA_MAP_DIRECT -bool arch_dma_map_page_direct (struct device *dev,phys_addr_t addr); -bool arch_dma_unmap_page_direct (struct device *dev,dma_addr_t dma_handle); +bool arch_dma_map_phys_direct (struct device *dev,phys_addr_t addr); +bool arch_dma_unmap_phys_direct (struct device *dev,dma_addr_t dma_handle);bool arch_dma_map_sg_direct (struct device *dev,struct scatterlist *sg,int nents);bool arch_dma_unmap_sg_direct (struct device *dev,struct scatterlist *sg,int nents);#else -#define arch_dma_map_page_direct (d, a) (false) -#define arch_dma_unmap_page_direct (d, a) (false) +#define arch_dma_map_phys_direct (d, a) (false) +#define arch_dma_unmap_phys_direct (d, a) (false)#define arch_dma_map_sg_direct (d, s, n) (false)#define arch_dma_unmap_sg_direct (d, s, n) (false)#endif
kernel/dma/direct.c +3 -3
@@ -453,7 +453,7 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl,if (sg_dma_is_bus_address (sg))sg_dma_unmark_bus_address (sg);else -dma_direct_unmap_page (dev, sg->dma_address, +dma_direct_unmap_phys (dev, sg->dma_address,sg_dma_len (sg), dir, attrs); } }
@@ -476,8 +476,8 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,*/ break ;case PCI_P2PDMA_MAP_NONE : - sg->dma_address =dma_direct_map_page (dev,sg_page (sg), - sg->offset, sg->length, dir, attrs); + sg->dma_address =dma_direct_map_phys (dev,sg_phys (sg), + sg->length, dir, attrs);if (sg->dma_address ==DMA_MAPPING_ERROR ) { ret = -EIO ; goto out_unmap;
kernel/dma/direct.h +36 -21
78 lines not on this page · raw message
kernel/dma/mapping.c +4 -4
@@ -166,8 +166,8 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_page_direct (dev, phys + size)) - addr =dma_direct_map_page (dev, page, offset, size, dir, attrs); +arch_dma_map_phys_direct (dev, phys + size)) + addr =dma_direct_map_phys (dev, phys, size, dir, attrs);else if (use_dma_iommu (dev)) addr =iommu_dma_map_phys (dev, phys, size, dir, attrs);else
@@ -187,8 +187,8 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,BUG_ON (!valid_dma_direction (dir));if (dma_map_direct (dev, ops) || -arch_dma_unmap_page_direct (dev, addr + size)) -dma_direct_unmap_page (dev, addr, size, dir, attrs); +arch_dma_unmap_phys_direct (dev, addr + size)) +dma_direct_unmap_phys (dev, addr, size, dir, attrs);else if (use_dma_iommu (dev))iommu_dma_unmap_phys (dev, addr, size, dir, attrs);else
Signature · 1 line
2.51.0
PATCH v6 8/16 kmsan: convert kmsan_handle_dma to use physical addresses
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert the KMSAN DMA handling function from page-based to physical address-based interface.
The refactoring renames kmsan_handle_dma() parameters from accepting (struct page *page, size_t offset, size_t size) to (phys_addr_t phys, size_t size). The existing semantics where callers are expected to provide only kmap memory is continued here.
--- drivers/virtio/virtio_ring.c | 4 ++-- include/linux/kmsan.h | 9 ++++----- kernel/dma/mapping.c | 3 ++- mm/kmsan/hooks.c | 10 ++++++---- tools/virtio/linux/kmsan.h | 2 +- 5 files changed, 15 insertions(+), 13 deletions(-)
drivers/virtio/virtio_ring.c +2 -2
@@ -378,7 +378,7 @@ static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist* is initialized by the hardware. Explicitly check/unpoison it * depending on the direction. */ -kmsan_handle_dma (sg_page (sg), sg->offset, sg->length, direction); +kmsan_handle_dma (sg_phys (sg), sg->length, direction); *addr = (dma_addr_t )sg_phys (sg);return 0 ; }
@@ -3157,7 +3157,7 @@ dma_addr_t virtqueue_dma_map_single_attrs(struct virtqueue *_vq, void *ptr,struct vring_virtqueue *vq =to_vvq (_vq);if (!vq->use_dma_api) { -kmsan_handle_dma (virt_to_page (ptr),offset_in_page (ptr), size, dir); +kmsan_handle_dma (virt_to_phys (ptr), size, dir);return (dma_addr_t )virt_to_phys (ptr); }
include/linux/kmsan.h +4 -5
@@ -182,8 +182,7 @@ void kmsan_iounmap_page_range(unsigned long start, unsigned long end);/** * kmsan_handle_dma() - Handle a DMA data transfer. -* @page: first page of the buffer. -* @offset: offset of the buffer within the first page. +* @phys: physical address of the buffer. * @size: buffer size. * @dir: one of possible dma_data_direction values. *
@@ -192,7 +191,7 @@ void kmsan_iounmap_page_range(unsigned long start, unsigned long end);* * initializes the buffer, if it is copied from device; * * does both, if this is a DMA_BIDIRECTIONAL transfer. */ -void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir);/**
@@ -372,8 +371,8 @@ static inline void kmsan_iounmap_page_range(unsigned long start, { } -static inline void kmsan_handle_dma (struct page *page,size_t offset, -size_t size,enum dma_data_direction dir) +static inline void kmsan_handle_dma (phys_addr_t phys,size_t size, +enum dma_data_direction dir) { }
kernel/dma/mapping.c +2 -1
@@ -172,7 +172,8 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); else addr = ops->map_page (dev, page, offset, size, dir, attrs); -kmsan_handle_dma (page, offset, size, dir); + +kmsan_handle_dma (phys, size, dir);trace_dma_map_phys (dev, phys, addr, size, dir, attrs);debug_dma_map_phys (dev, phys, size, dir, addr, attrs);
mm/kmsan/hooks.c +6 -4
@@ -336,14 +336,16 @@ static void kmsan_handle_dma_page(const void *addr, size_t size, }/* Helper function to handle DMA data transfers. */ -void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) { -u64 page_offset, to_go, addr; +struct page *page =phys_to_page (phys); +u64 page_offset, to_go; +void *addr; -if (PageHighMem (page)) +if (PhysHighMem (phys))return ; - addr = (u64 )page_address (page) + offset; + addr =page_to_virt (page);/* * The kernel may occasionally give us adjacent DMA pages not belonging * to the same allocation. Process them separately to avoid triggering
tools/virtio/linux/kmsan.h +1 -1
@@ -4,7 +4,7 @@#include <linux/gfp.h> -inline void kmsan_handle_dma (struct page *page,size_t offset,size_t size, +inline void kmsan_handle_dma (phys_addr_t phys,size_t size,enum dma_data_direction dir) { }
Signature · 1 line
2.51.0
PATCH v6 9/16 dma-mapping: implement DMA_ATTR_MMIO for dma_(un)map_page_attrs()
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Make dma_map_page_attrs() and dma_map_page_attrs() respect DMA_ATTR_MMIO.
DMA_ATR_MMIO makes the functions behave the same as dma_(un)map_resource():
- No swiotlb is possible
- Legacy dma_ops arches use ops->map_resource()
- No kmsan
- No arch_dma_map_phys_direct()
The prior patches have made the internal functions called here support DMA_ATTR_MMIO.
This is also preparation for turning dma_map_resource() into an inline calling dma_map_phys(DMA_ATTR_MMIO) to consolidate the flows.
--- kernel/dma/mapping.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-)
kernel/dma/mapping.c +21 -5
@@ -158,6 +158,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, {const struct dma_map_ops *ops =get_dma_ops (dev);phys_addr_t phys =page_to_phys (page) + offset; +bool is_mmio = attrs &DMA_ATTR_MMIO ;dma_addr_t addr;BUG_ON (!valid_dma_direction (dir));
@@ -166,14 +167,25 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page,return DMA_MAPPING_ERROR ;if (dma_map_direct (dev, ops) || -arch_dma_map_phys_direct (dev, phys + size)) + (!is_mmio &&arch_dma_map_phys_direct (dev, phys + size))) addr =dma_direct_map_phys (dev, phys, size, dir, attrs);else if (use_dma_iommu (dev)) addr =iommu_dma_map_phys (dev, phys, size, dir, attrs); -else +else if (is_mmio) { +if (!ops->map_resource) +return DMA_MAPPING_ERROR ; + + addr = ops->map_resource (dev, phys, size, dir, attrs); + }else { +/* +* The dma_ops API contract for ops->map_page() requires +* kmappable memory, while ops->map_resource() does not. +*/ addr = ops->map_page (dev, page, offset, size, dir, attrs); + } -kmsan_handle_dma (phys, size, dir); +if (!is_mmio) +kmsan_handle_dma (phys, size, dir);trace_dma_map_phys (dev, phys, addr, size, dir, attrs);debug_dma_map_phys (dev, phys, size, dir, addr, attrs);
@@ -185,14 +197,18 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) {const struct dma_map_ops *ops =get_dma_ops (dev); +bool is_mmio = attrs &DMA_ATTR_MMIO ;BUG_ON (!valid_dma_direction (dir));if (dma_map_direct (dev, ops) || -arch_dma_unmap_phys_direct (dev, addr + size)) + (!is_mmio &&arch_dma_unmap_phys_direct (dev, addr + size)))dma_direct_unmap_phys (dev, addr, size, dir, attrs);else if (use_dma_iommu (dev))iommu_dma_unmap_phys (dev, addr, size, dir, attrs); -else +else if (is_mmio) { +if (ops->unmap_resource) + ops->unmap_resource (dev, addr, size, dir, attrs); + }else ops->unmap_page (dev, addr, size, dir, attrs);trace_dma_unmap_phys (dev, addr, size, dir, attrs);debug_dma_unmap_phys (dev, addr, size, dir);
Signature · 1 line
2.51.0
PATCH v6 10/16 xen: swiotlb: Open code map_resource callback
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
General dma_direct_map_resource() is going to be removed in next patch, so simply open-code it in xen driver.
--- drivers/xen/swiotlb-xen.c | 21 ++++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-)
drivers/xen/swiotlb-xen.c +20 -1
@@ -392,6 +392,25 @@ xen_swiotlb_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, } } +static dma_addr_t xen_swiotlb_direct_map_resource (struct device *dev, +phys_addr_t paddr, +size_t size, +enum dma_data_direction dir, +unsigned long attrs) +{ +dma_addr_t dma_addr = paddr; + +if (unlikely (!dma_capable (dev, dma_addr, size, false))) { +dev_err_once (dev, +"DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n" , + &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); +WARN_ON_ONCE (1 ); +return DMA_MAPPING_ERROR ; + } + +return dma_addr; +} +/* * Return whether the given device DMA address mask can be supported * properly. For example, if your device can only drive the low 24-bits
@@ -426,5 +445,5 @@ const struct dma_map_ops xen_swiotlb_dma_ops = { .alloc_pages_op = dma_common_alloc_pages, .free_pages = dma_common_free_pages, .max_mapping_size = swiotlb_max_mapping_size, - .map_resource = dma_direct_map_resource, + .map_resource = xen_swiotlb_direct_map_resource, };
Signature · 1 line
2.51.0
PATCH v6 11/16 dma-mapping: export new dma_*map_phys() interface
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Introduce new DMA mapping functions dma_map_phys() and dma_unmap_phys() that operate directly on physical addresses instead of page+offset parameters. This provides a more efficient interface for drivers that already have physical addresses available.
The new functions are implemented as the primary mapping layer, with the existing dma_map_page_attrs()/dma_map_resource() and dma_unmap_page_attrs()/dma_unmap_resource() functions converted to simple wrappers around the phys-based implementations.
In case dma_map_page_attrs(), the struct page is converted to physical address with help of page_to_phys() function and dma_map_resource() provides physical address as is together with addition of DMA_ATTR_MMIO attribute.
The old page-based API is preserved in mapping.c to ensure that existing code won't be affected by changing EXPORT_SYMBOL to EXPORT_SYMBOL_GPL variant for dma_*map_phys().
--- drivers/iommu/dma-iommu.c | 14 -------- include/linux/dma-direct.h | 2 -- include/linux/dma-mapping.h | 13 +++++++ include/linux/iommu-dma.h | 4 --- include/trace/events/dma.h | 2 -- kernel/dma/debug.c | 43 ----------------------- kernel/dma/debug.h | 21 ----------- kernel/dma/direct.c | 16 --------- kernel/dma/mapping.c | 69 ++++++++++++++++++++----------------- 9 files changed, 50 insertions(+), 134 deletions(-)
drivers/iommu/dma-iommu.c +0 -14
@@ -1556,20 +1556,6 @@ void iommu_dma_unmap_sg(struct device *dev, struct scatterlist *sg, int nents,__iommu_dma_unmap (dev, start, end - start); } -dma_addr_t iommu_dma_map_resource (struct device *dev,phys_addr_t phys, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -return __iommu_dma_map (dev, phys, size, -dma_info_to_prot (dir, false, attrs) |IOMMU_MMIO , -dma_get_mask (dev)); -} - -void iommu_dma_unmap_resource (struct device *dev,dma_addr_t handle, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -__iommu_dma_unmap (dev, handle, size); -} -static void __iommu_dma_free (struct device *dev,size_t size,void *cpu_addr) {size_t alloc_size =PAGE_ALIGN (size);
include/linux/dma-direct.h +0 -2
@@ -149,7 +149,5 @@ void dma_direct_free_pages(struct device *dev, size_t size, struct page *page, dma_addr_t dma_addr, enum dma_data_direction dir);int dma_direct_supported (struct device *dev,u64 mask); -dma_addr_t dma_direct_map_resource (struct device *dev,phys_addr_t paddr, -size_t size,enum dma_data_direction dir,unsigned long attrs);#endif /* _LINUX_DMA_DIRECT_H */
include/linux/dma-mapping.h +13 -0
@@ -138,6 +138,10 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, unsigned long attrs);void dma_unmap_page_attrs (struct device *dev,dma_addr_t addr,size_t size,enum dma_data_direction dir,unsigned long attrs); +dma_addr_t dma_map_phys (struct device *dev,phys_addr_t phys,size_t size, +enum dma_data_direction dir,unsigned long attrs); +void dma_unmap_phys (struct device *dev,dma_addr_t addr,size_t size, +enum dma_data_direction dir,unsigned long attrs);unsigned int dma_map_sg_attrs (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs); void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg,
@@ -192,6 +196,15 @@ static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) { } +static inline dma_addr_t dma_map_phys (struct device *dev,phys_addr_t phys, +size_t size,enum dma_data_direction dir,unsigned long attrs) +{ +return DMA_MAPPING_ERROR ; +} +static inline void dma_unmap_phys (struct device *dev,dma_addr_t addr, +size_t size,enum dma_data_direction dir,unsigned long attrs) +{ +}static inline unsigned int dma_map_sg_attrs (struct device *dev,struct scatterlist *sg,int nents,enum dma_data_direction dir,unsigned long attrs)
include/linux/iommu-dma.h +0 -4
@@ -42,10 +42,6 @@ size_t iommu_dma_opt_mapping_size(void);size_t iommu_dma_max_mapping_size (struct device *dev);void iommu_dma_free (struct device *dev,size_t size,void *cpu_addr,dma_addr_t handle,unsigned long attrs); -dma_addr_t iommu_dma_map_resource (struct device *dev,phys_addr_t phys, -size_t size,enum dma_data_direction dir,unsigned long attrs); -void iommu_dma_unmap_resource (struct device *dev,dma_addr_t handle, -size_t size,enum dma_data_direction dir,unsigned long attrs);struct sg_table *iommu_dma_alloc_noncontiguous (struct device *dev,size_t size,enum dma_data_direction dir,gfp_t gfp,unsigned long attrs); void iommu_dma_free_noncontiguous(struct device *dev, size_t size,
include/trace/events/dma.h +0 -2
@@ -73,7 +73,6 @@ DEFINE_EVENT(dma_map, name, \ TP_ARGS(dev, phys_addr, dma_addr, size, dir, attrs))DEFINE_MAP_EVENT (dma_map_phys); -DEFINE_MAP_EVENT (dma_map_resource); DECLARE_EVENT_CLASS(dma_unmap, TP_PROTO(struct device *dev, dma_addr_t addr, size_t size,
@@ -111,7 +110,6 @@ DEFINE_EVENT(dma_unmap, name, \ TP_ARGS(dev, addr, size, dir, attrs))DEFINE_UNMAP_EVENT (dma_unmap_phys); -DEFINE_UNMAP_EVENT (dma_unmap_resource); DECLARE_EVENT_CLASS(dma_alloc_class, TP_PROTO(struct device *dev, void *virt_addr, dma_addr_t dma_addr,
kernel/dma/debug.c +0 -43
61 lines not on this page · raw message
kernel/dma/debug.h +0 -21
@@ -30,14 +30,6 @@ extern void debug_dma_alloc_coherent(struct device *dev, size_t size,extern void debug_dma_free_coherent (struct device *dev,size_t size,void *virt,dma_addr_t addr); -extern void debug_dma_map_resource (struct device *dev,phys_addr_t addr, -size_t size,int direction, -dma_addr_t dma_addr, -unsigned long attrs); - -extern void debug_dma_unmap_resource (struct device *dev,dma_addr_t dma_addr, -size_t size,int direction); -extern void debug_dma_sync_single_for_cpu (struct device *dev,dma_addr_t dma_handle,size_t size,int direction);
@@ -95,19 +87,6 @@ static inline void debug_dma_free_coherent(struct device *dev, size_t size, { } -static inline void debug_dma_map_resource (struct device *dev,phys_addr_t addr, -size_t size,int direction, -dma_addr_t dma_addr, -unsigned long attrs) -{ -} - -static inline void debug_dma_unmap_resource (struct device *dev, -dma_addr_t dma_addr,size_t size, -int direction) -{ -} -static inline void debug_dma_sync_single_for_cpu (struct device *dev,dma_addr_t dma_handle,size_t size,int direction)
kernel/dma/direct.c +0 -16
@@ -502,22 +502,6 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents,return ret; } -dma_addr_t dma_direct_map_resource (struct device *dev,phys_addr_t paddr, -size_t size,enum dma_data_direction dir,unsigned long attrs) -{ -dma_addr_t dma_addr = paddr; - -if (unlikely (!dma_capable (dev, dma_addr, size, false))) { -dev_err_once (dev, -"DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n" , - &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); -WARN_ON_ONCE (1 ); -return DMA_MAPPING_ERROR ; - } - -return dma_addr; -} -int dma_direct_get_sgtable (struct device *dev,struct sg_table *sgt,void *cpu_addr,dma_addr_t dma_addr,size_t size,unsigned long attrs)
kernel/dma/mapping.c +37 -32
111 lines not on this page · raw message
Signature · 1 line
2.51.0
PATCH v6 12/16 mm/hmm: migrate to physical address-based DMA mapping API
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Convert HMM DMA operations from the legacy page-based API to the new physical address-based dma_map_phys() and dma_unmap_phys() functions. This demonstrates the preferred approach for new code that should use physical addresses directly rather than page+offset parameters.
The change replaces dma_map_page() and dma_unmap_page() calls with dma_map_phys() and dma_unmap_phys() respectively, using the physical address that was already available in the code. This eliminates the redundant page-to-physical address conversion and aligns with the DMA subsystem's move toward physical address-centric interfaces.
This serves as an example of how new code should be written to leverage the more efficient physical address API, which provides cleaner interfaces for drivers that already have access to physical addresses.
--- mm/hmm.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
mm/hmm.c +4 -4
@@ -775,8 +775,8 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map,if (WARN_ON_ONCE (dma_need_unmap (dev) && !dma_addrs)) goto error; - dma_addr =dma_map_page (dev, page,0 , map->dma_entry_size, -DMA_BIDIRECTIONAL ); + dma_addr =dma_map_phys (dev, paddr, map->dma_entry_size, +DMA_BIDIRECTIONAL ,0 );if (dma_mapping_error (dev, dma_addr)) goto error;
@@ -819,8 +819,8 @@ bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx)dma_iova_unlink (dev, state, idx * map->dma_entry_size, map->dma_entry_size,DMA_BIDIRECTIONAL , attrs); }else if (dma_need_unmap (dev)) -dma_unmap_page (dev, dma_addrs[idx], map->dma_entry_size, -DMA_BIDIRECTIONAL ); +dma_unmap_phys (dev, dma_addrs[idx], map->dma_entry_size, +DMA_BIDIRECTIONAL ,0 ); pfns[idx] &= ~(HMM_PFN_DMA_MAPPED |HMM_PFN_P2PDMA |HMM_PFN_P2PDMA_BUS );
Signature · 1 line
2.51.0
PATCH v6 13/16 mm/hmm: properly take MMIO path
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
In case peer-to-peer transaction traverses through host bridge, the IOMMU needs to have IOMMU_MMIO flag, together with skip of CPU sync.
The latter was handled by provided DMA_ATTR_SKIP_CPU_SYNC flag, but IOMMU flag was missed, due to assumption that such memory can be treated as regular one.
Reuse newly introduced DMA attribute to properly take MMIO path.
--- mm/hmm.c | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-)
mm/hmm.c +8 -7
@@ -746,7 +746,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map,case PCI_P2PDMA_MAP_NONE :break ;case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE : - attrs |=DMA_ATTR_SKIP_CPU_SYNC ; + attrs |=DMA_ATTR_MMIO ; pfns[idx] |=HMM_PFN_P2PDMA ;break ;case PCI_P2PDMA_MAP_BUS_ADDR :
@@ -776,7 +776,7 @@ dma_addr_t hmm_dma_map_pfn(struct device *dev, struct hmm_dma_map *map, goto error; dma_addr =dma_map_phys (dev, paddr, map->dma_entry_size, -DMA_BIDIRECTIONAL ,0 ); +DMA_BIDIRECTIONAL , attrs);if (dma_mapping_error (dev, dma_addr)) goto error;
@@ -811,16 +811,17 @@ bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx)if ((pfns[idx] & valid_dma) != valid_dma)return false; +if (pfns[idx] &HMM_PFN_P2PDMA ) + attrs |=DMA_ATTR_MMIO ; +if (pfns[idx] &HMM_PFN_P2PDMA_BUS ) ;/* no need to unmap bus address P2P mappings */ -else if (dma_use_iova (state)) { -if (pfns[idx] &HMM_PFN_P2PDMA ) - attrs |=DMA_ATTR_SKIP_CPU_SYNC ; +else if (dma_use_iova (state))dma_iova_unlink (dev, state, idx * map->dma_entry_size, map->dma_entry_size,DMA_BIDIRECTIONAL , attrs); - }else if (dma_need_unmap (dev)) +else if (dma_need_unmap (dev))dma_unmap_phys (dev, dma_addrs[idx], map->dma_entry_size, -DMA_BIDIRECTIONAL ,0 ); +DMA_BIDIRECTIONAL , attrs); pfns[idx] &= ~(HMM_PFN_DMA_MAPPED |HMM_PFN_P2PDMA |HMM_PFN_P2PDMA_BUS );
Signature · 1 line
2.51.0
PATCH v6 14/16 block-dma: migrate to dma_map_phys instead of map_page
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
After introduction of dma_map_phys(), there is no need to convert from physical address to struct page in order to map page. So let's use it directly.
--- block/blk-mq-dma.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
block/blk-mq-dma.c +2 -2
@@ -87,8 +87,8 @@ static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)static bool blk_dma_map_direct (struct request *req,struct device *dma_dev,struct blk_dma_iter *iter,struct phys_vec *vec) { - iter->addr =dma_map_page (dma_dev,phys_to_page (vec->paddr), -offset_in_page (vec->paddr), vec->len,rq_dma_dir (req)); + iter->addr =dma_map_phys (dma_dev, vec->paddr, vec->len, +rq_dma_dir (req),0 );if (dma_mapping_error (dma_dev, iter->addr)) { iter->status =BLK_STS_RESOURCE ;return false;
Signature · 1 line
2.51.0
PATCH v6 15/16 block-dma: properly take MMIO path
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Make sure that CPU is not synced and IOMMU is configured to take MMIO path by providing newly introduced DMA_ATTR_MMIO attribute.
--- block/blk-mq-dma.c | 13 +++++++++++-- include/linux/blk-mq-dma.h | 6 +++++- include/linux/blk_types.h | 2 ++ 3 files changed, 18 insertions(+), 3 deletions(-)
block/blk-mq-dma.c +11 -2
@@ -87,8 +87,13 @@ static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)static bool blk_dma_map_direct (struct request *req,struct device *dma_dev,struct blk_dma_iter *iter,struct phys_vec *vec) { +unsigned int attrs =0 ; + +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ; + iter->addr =dma_map_phys (dma_dev, vec->paddr, vec->len, -rq_dma_dir (req),0 ); +rq_dma_dir (req), attrs);if (dma_mapping_error (dma_dev, iter->addr)) { iter->status =BLK_STS_RESOURCE ;return false;
@@ -103,14 +108,17 @@ static bool blk_rq_dma_map_iova(struct request *req, struct device *dma_dev, {enum dma_data_direction dir =rq_dma_dir (req);unsigned int mapped =0 ; +unsigned int attrs =0 ;int error; iter->addr = state->addr; iter->len =dma_iova_size (state); +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ; do { error = dma_iova_link(dma_dev, state, vec->paddr, mapped, - vec->len, dir, 0); + vec->len, dir, attrs); if (error) break; mapped += vec->len;
@@ -176,6 +184,7 @@ bool blk_rq_dma_map_iter_start(struct request *req, struct device *dma_dev,* same as non-P2P transfers below and during unmap. */ req->cmd_flags &= ~REQ_P2PDMA ; + req->cmd_flags |=REQ_MMIO ;break ;default : iter->status =BLK_STS_INVAL ;
include/linux/blk-mq-dma.h +5 -1
@@ -48,12 +48,16 @@ static inline bool blk_rq_dma_map_coalesce(struct dma_iova_state *state)static inline bool blk_rq_dma_unmap (struct request *req,struct device *dma_dev,struct dma_iova_state *state,size_t mapped_len) { +unsigned int attrs =0 ; +if (req->cmd_flags &REQ_P2PDMA )return true;if (dma_use_iova (state)) { +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ;dma_iova_destroy (dma_dev, state, mapped_len,rq_dma_dir (req), -0 ); + attrs);return true; }
include/linux/blk_types.h +2 -0
@@ -387,6 +387,7 @@ enum req_flag_bits { __REQ_FS_PRIVATE,/* for file system (submitter) use */ __REQ_ATOMIC,/* for atomic write operations */ __REQ_P2PDMA,/* contains P2P DMA pages */ + __REQ_MMIO, /* contains MMIO memory *//* * Command specific flags, keep last: */
@@ -420,6 +421,7 @@ enum req_flag_bits {#define REQ_FS_PRIVATE (__force blk_opf_t)(1ULL << __REQ_FS_PRIVATE)#define REQ_ATOMIC (__force blk_opf_t)(1ULL << __REQ_ATOMIC)#define REQ_P2PDMA (__force blk_opf_t)(1ULL << __REQ_P2PDMA) +#define REQ_MMIO (__force blk_opf_t)(1ULL << __REQ_MMIO)#define REQ_NOUNMAP (__force blk_opf_t)(1ULL << __REQ_NOUNMAP)
Signature · 1 line
2.51.0
PATCH v6 16/16 nvme-pci: unmap MMIO pages with appropriate interface
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Block layer maps MMIO memory through dma_map_phys() interface with help of DMA_ATTR_MMIO attribute. There is a need to unmap that memory with the appropriate unmap function, something which wasn't possible before adding new REQ attribute to block layer in previous patch.
--- drivers/nvme/host/pci.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
drivers/nvme/host/pci.c +13 -5
@@ -682,11 +682,15 @@ static void nvme_free_prps(struct request *req) {struct nvme_iod *iod =blk_mq_rq_to_pdu (req);struct nvme_queue *nvmeq = req->mq_hctx->driver_data; +unsigned int attrs =0 ;unsigned int i; +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ; +for (i =0 ; i < iod->nr_dma_vecs; i++) -dma_unmap_page (nvmeq->dev->dev, iod->dma_vecs[i].addr, - iod->dma_vecs[i].len,rq_dma_dir (req)); +dma_unmap_phys (nvmeq->dev->dev, iod->dma_vecs[i].addr, + iod->dma_vecs[i].len,rq_dma_dir (req), attrs);mempool_free (iod->dma_vecs, nvmeq->dev->dmavec_mempool); }
@@ -699,15 +703,19 @@ static void nvme_free_sgls(struct request *req)unsigned int sqe_dma_len =le32_to_cpu (iod->cmd.common.dptr.sgl.length);struct nvme_sgl_desc *sg_list = iod->descriptors[0 ];enum dma_data_direction dir =rq_dma_dir (req); +unsigned int attrs =0 ; + +if (req->cmd_flags &REQ_MMIO ) + attrs =DMA_ATTR_MMIO ;if (iod->nr_descriptors) {unsigned int nr_entries = sqe_dma_len /sizeof (*sg_list), i;for (i =0 ; i < nr_entries; i++) -dma_unmap_page (dma_dev,le64_to_cpu (sg_list[i].addr), -le32_to_cpu (sg_list[i].length), dir); +dma_unmap_phys (dma_dev,le64_to_cpu (sg_list[i].addr), +le32_to_cpu (sg_list[i].length), dir, attrs); }else { -dma_unmap_page (dma_dev, sqe_dma_addr, sqe_dma_len, dir); +dma_unmap_phys (dma_dev, sqe_dma_addr, sqe_dma_len, dir, attrs); } }
Signature · 1 line
2.51.0
Quoted 94 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Changelog:
v6:
* Based on "dma-debug: don't enforce dma mapping check on noncoherent allocations" patch.
* Removed some unused variables from kmsan conversion.
* Fixed missed ! in dma check.
v5: https://lore.kernel.org/all/[email protected]
* Added Jason's and Keith's Reviewed-by tags
* FixedDMA_ATTR_MMIOcheck indma_direct_map_phys
* Jason's cleanup suggestions
v4: https://lore.kernel.org/all/[email protected]/
* Fixed kbuild error with mismatch in kmsan function declaration due to rebase error.
v3: https://lore.kernel.org/all/[email protected]
* Fixed typo in "cacheable" word
* Simplified kmsan patch a lot to be simple argument refactoring
v2: https://lore.kernel.org/all/[email protected]
* Used commit messages and cover letter from Jason
* Moved settingIOMMU_MMIOflag todma_info_to_protfunction
* Micro-optimized the code
* Rebased code on v6.17-rc1
v1: https://lore.kernel.org/all/[email protected]
* Added newDMA_ATTR_MMIOattribute to indicatePCI_P2PDMA_MAP_THRU_HOST_BRIDGEpath.
* Rewrote dma_map_* functions to use thus new attribute
v0: https://lore.kernel.org/all/[email protected]/
------------------------------------------------------------------------This series refactors the DMA mapping to use physical addresses as the primary interface instead of page+offset parameters. This change aligns the DMA API with the underlying hardware reality where DMA operations work with physical addresses, not page structures.
The series maintains export symbol backward compatibility by keeping the old page-based API as wrapper functions around the new physical address-based implementations.
This series refactors the DMA mapping API to provide a
phys_addr_tbased, and struct-page free, external API that can handle all the mapping cases we want in modern systems:- struct page based cacheable DRAM
- struct pageMEMORY_DEVICE_PCI_P2PDMAPCI peer to peer non-cacheable MMIO
- struct page-less PCI peer to peer non-cacheable MMIO
- struct page-less "resource" MMIOOverall this gets much closer to Matthew's long term wish for struct-pageless IO to cacheable DRAM. The remaining primary work would be in the mm side to allow kmap_local_pfn()/phys_to_virt() to work on
phys_addr_twithout a struct page.The general design is to remove struct page usage entirely from the DMA API inner layers. For flows that need to have a KVA for the physical address they can use
kmap_local_pfn()orphys_to_virt(). This isolates the struct page requirements to MM code only. Long term all removals of struct page usage are supporting Matthew's memdesc project which seeks to substantially transform how struct page works.Instead make the DMA API internals work on
phys_addr_t. Internally there are still dedicated 'page' and 'resource' flows, except they are now distinguished by a newDMA_ATTR_MMIOinstead of by callchain. Both flows use the samephys_addr_t.When
DMA_ATTR_MMIOis specified things work similar to the existing 'resource' flow.kmap_local_pfn(),phys_to_virt(),phys_to_page(),pfn_valid(), etc are never called on thephys_addr_t. This requires rejecting any configuration that would need swiotlb. CPU cache flushing is not required, and avoided, asATTR_MMIOalso indicates the address have no cacheable mappings. This effectively removes any DMA API side requirement to have struct page whenDMA_ATTR_MMIOis used.In the !DMA_ATTR_MMIO mode things work similarly to the 'page' flow, except on the common path of no cache flush, no swiotlb it never touches a struct page. When cache flushing or swiotlb copying kmap_local_pfn()/phys_to_virt() are used to get a KVA for CPU usage. This was already the case on the unmap side, now the map side is symmetric.
Callers are adjusted to set
DMA_ATTR_MMIO. Existing 'resource' users must set it. The existing struct page basedMEMORY_DEVICE_PCI_P2PDMApath must also set it. This corrects some existing bugs where iommu mappings for P2P MMIO were improperly markedIOMMU_CACHE.Since
ATTR_MMIOis made to work with all the existing DMA map entry points, particularlydma_iova_link(), this finally allows a way to use the new DMA API to map PCI P2P MMIO without creating struct page. The VFIO DMABUF series demonstrates how this works. This is intended to replace the incorrect driver use ofdma_map_resource()on PCI BAR addresses.
Quoted 2 lines
This series does the core code and modern flows. A followup series will give the same treatment to the legacy
dma_opsimplementation.
Applied patches 1-13 into dma-mapping-for-next branch. Let's check if it works fine in linux-next.
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Quoted 95 lines · Marek Szyprowski
Quoted 94 lines · Leon Romanovsky
From: Leon Romanovsky <[email protected]>
Changelog:
v6:
* Based on "dma-debug: don't enforce dma mapping check on noncoherent allocations" patch.
* Removed some unused variables from kmsan conversion.
* Fixed missed ! in dma check.
v5: https://lore.kernel.org/all/[email protected]
* Added Jason's and Keith's Reviewed-by tags
* FixedDMA_ATTR_MMIOcheck indma_direct_map_phys
* Jason's cleanup suggestions
v4: https://lore.kernel.org/all/[email protected]/
* Fixed kbuild error with mismatch in kmsan function declaration due to rebase error.
v3: https://lore.kernel.org/all/[email protected]
* Fixed typo in "cacheable" word
* Simplified kmsan patch a lot to be simple argument refactoring
v2: https://lore.kernel.org/all/[email protected]
* Used commit messages and cover letter from Jason
* Moved settingIOMMU_MMIOflag todma_info_to_protfunction
* Micro-optimized the code
* Rebased code on v6.17-rc1
v1: https://lore.kernel.org/all/[email protected]
* Added newDMA_ATTR_MMIOattribute to indicatePCI_P2PDMA_MAP_THRU_HOST_BRIDGEpath.
* Rewrote dma_map_* functions to use thus new attribute
v0: https://lore.kernel.org/all/[email protected]/
------------------------------------------------------------------------This series refactors the DMA mapping to use physical addresses as the primary interface instead of page+offset parameters. This change aligns the DMA API with the underlying hardware reality where DMA operations work with physical addresses, not page structures.
The series maintains export symbol backward compatibility by keeping the old page-based API as wrapper functions around the new physical address-based implementations.
This series refactors the DMA mapping API to provide a
phys_addr_tbased, and struct-page free, external API that can handle all the mapping cases we want in modern systems:- struct page based cacheable DRAM
- struct pageMEMORY_DEVICE_PCI_P2PDMAPCI peer to peer non-cacheable MMIO
- struct page-less PCI peer to peer non-cacheable MMIO
- struct page-less "resource" MMIOOverall this gets much closer to Matthew's long term wish for struct-pageless IO to cacheable DRAM. The remaining primary work would be in the mm side to allow kmap_local_pfn()/phys_to_virt() to work on
phys_addr_twithout a struct page.The general design is to remove struct page usage entirely from the DMA API inner layers. For flows that need to have a KVA for the physical address they can use
kmap_local_pfn()orphys_to_virt(). This isolates the struct page requirements to MM code only. Long term all removals of struct page usage are supporting Matthew's memdesc project which seeks to substantially transform how struct page works.Instead make the DMA API internals work on
phys_addr_t. Internally there are still dedicated 'page' and 'resource' flows, except they are now distinguished by a newDMA_ATTR_MMIOinstead of by callchain. Both flows use the samephys_addr_t.When
DMA_ATTR_MMIOis specified things work similar to the existing 'resource' flow.kmap_local_pfn(),phys_to_virt(),phys_to_page(),pfn_valid(), etc are never called on thephys_addr_t. This requires rejecting any configuration that would need swiotlb. CPU cache flushing is not required, and avoided, asATTR_MMIOalso indicates the address have no cacheable mappings. This effectively removes any DMA API side requirement to have struct page whenDMA_ATTR_MMIOis used.In the !DMA_ATTR_MMIO mode things work similarly to the 'page' flow, except on the common path of no cache flush, no swiotlb it never touches a struct page. When cache flushing or swiotlb copying kmap_local_pfn()/phys_to_virt() are used to get a KVA for CPU usage. This was already the case on the unmap side, now the map side is symmetric.
Callers are adjusted to set
DMA_ATTR_MMIO. Existing 'resource' users must set it. The existing struct page basedMEMORY_DEVICE_PCI_P2PDMApath must also set it. This corrects some existing bugs where iommu mappings for P2P MMIO were improperly markedIOMMU_CACHE.Since
ATTR_MMIOis made to work with all the existing DMA map entry points, particularlydma_iova_link(), this finally allows a way to use the new DMA API to map PCI P2P MMIO without creating struct page. The VFIO DMABUF series demonstrates how this works. This is intended to replace the incorrect driver use ofdma_map_resource()on PCI BAR addresses.
Quoted 5 lines
Quoted 2 lines
This series does the core code and modern flows. A followup series will give the same treatment to the legacy
dma_opsimplementation.Applied patches 1-13 into dma-mapping-for-next branch. Let's check if it works fine in linux-next.
Thanks a lot.
Quoted 7 lines
Best regards
Signature · 2 lines
Marek Szyprowski, PhD Samsung R&D Institute Poland
Keith Busch
maintainerQuoted 9 lines · Leon Romanovsky
Quoted 6 lines · Marek Szyprowski
Quoted 3 lines
This series does the core code and modern flows. A followup series will give the same treatment to the legacy
dma_opsimplementation.Applied patches 1-13 into dma-mapping-for-next branch. Let's check if it works fine in linux-next.
Thanks a lot.
Just fyi, when dma debug is enabled, we're seeing this new warning below. I have not had a chance to look into it yet, so I'm just reporting the observation.
DMA-API: nvme 0006:01:00.0: cacheline tracking EEXIST, overlapping mappings aren't supported
WARNING: kernel/dma/debug.c:598 at add_dma_entry+0x26c/0x328, CPU#1: (udev-worker)/773 Modules linked in: acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) CPU: 1 UID: 0 PID: 773 Comm: (udev-worker) Tainted: G E N 6.17.0-rc6-next-20250918-debug #6 PREEMPT(none) Tainted: [E]=UNSIGNED_MODULE, [N]=TEST pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) pc : add_dma_entry+0x26c/0x328 lr : add_dma_entry+0x26c/0x328 sp : ffff80009fe0f460 x29: ffff80009fe0f470 x28: 0000000000000001 x27: 0000000000000001 x26: ffff8000835d7f38 x25: ffff8000835d7000 x24: ffff8000835d7e60 x23: 0000000000000000 x22: 0000000006e2cc00 x21: 0000000000000000 x20: ffff800082e8f218 x19: ffff0000a908ff80 x18: 00000000ffffffff x17: ffff8000801972a0 x16: ffff800080197054 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000004 x12: 0000000000020006 x11: 0000000030e4ef9f x10: ffff800083443358 x9 : ffff80008019499c x8 : 00000000fffeffff x7 : ffff800083443358 x6 : 0000000000000000 x5 : 00000000000bfff4 x4 : 0000000000000000 x3 : ffff0000bb005ac0 x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000bb005ac0 Call trace: add_dma_entry+0x26c/0x328 (P) debug_dma_map_phys+0xc4/0xf0 dma_map_phys+0xe0/0x410 dma_map_page_attrs+0x94/0xf8 blk_dma_map_direct.isra.0+0x64/0xb8 blk_rq_dma_map_iter_next+0x6c/0xc8 nvme_prep_rq+0x894/0xa98 nvme_queue_rqs+0xb0/0x1a0 blk_mq_dispatch_queue_requests+0x268/0x3b8 blk_mq_flush_plug_list+0x90/0x188 __blk_flush_plug+0x104/0x170 blk_finish_plug+0x38/0x50 read_pages+0x1a4/0x3b8 page_cache_ra_unbounded+0x1a0/0x400 force_page_cache_ra+0xa8/0xd8 page_cache_sync_ra+0xa0/0x3f8 filemap_get_pages+0x104/0x950 filemap_read+0xf4/0x498 blkdev_read_iter+0x88/0x180 vfs_read+0x214/0x310 ksys_read+0x70/0x110 __arm64_sys_read+0x20/0x30 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0xc4/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x1a0/0x340 el0t_64_sync_handler+0x98/0xe0 el0t_64_sync+0x17c/0x180 ---[ end trace 0000000000000000 ]---
Quoted 6 lines · Keith Busch
Quoted 5 lines · Leon Romanovsky
Quoted 4 lines · Marek Szyprowski
Quoted 3 lines
This series does the core code and modern flows. A followup series will give the same treatment to the legacy
dma_opsimplementation.
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Applied patches 1-13 into dma-mapping-for-next branch. Let's check if it works fine in linux-next.
Thanks a lot.
Just fyi, when dma debug is enabled, we're seeing this new warning below. I have not had a chance to look into it yet, so I'm just reporting the observation.
Did you apply all patches or only Marek's branch?
I don't get this warning when I run my NVMe tests on current dmabuf-vfio branch.
Thanks
Quoted 52 lines
DMA-API: nvme 0006:01:00.0: cacheline tracking EEXIST, overlapping mappings aren't supported
WARNING: kernel/dma/debug.c:598 at add_dma_entry+0x26c/0x328, CPU#1: (udev-worker)/773 Modules linked in: acpi_power_meter(E) loop(E) efivarfs(E) autofs4(E) CPU: 1 UID: 0 PID: 773 Comm: (udev-worker) Tainted: G E N 6.17.0-rc6-next-20250918-debug #6 PREEMPT(none) Tainted: [E]=UNSIGNED_MODULE, [N]=TEST pstate: 63400009 (nZCv daif +PAN -UAO +TCO +DIT -SSBS BTYPE=--) pc : add_dma_entry+0x26c/0x328 lr : add_dma_entry+0x26c/0x328 sp : ffff80009fe0f460 x29: ffff80009fe0f470 x28: 0000000000000001 x27: 0000000000000001 x26: ffff8000835d7f38 x25: ffff8000835d7000 x24: ffff8000835d7e60 x23: 0000000000000000 x22: 0000000006e2cc00 x21: 0000000000000000 x20: ffff800082e8f218 x19: ffff0000a908ff80 x18: 00000000ffffffff x17: ffff8000801972a0 x16: ffff800080197054 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000004 x12: 0000000000020006 x11: 0000000030e4ef9f x10: ffff800083443358 x9 : ffff80008019499c x8 : 00000000fffeffff x7 : ffff800083443358 x6 : 0000000000000000 x5 : 00000000000bfff4 x4 : 0000000000000000 x3 : ffff0000bb005ac0 x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff0000bb005ac0 Call trace: add_dma_entry+0x26c/0x328 (P) debug_dma_map_phys+0xc4/0xf0 dma_map_phys+0xe0/0x410 dma_map_page_attrs+0x94/0xf8 blk_dma_map_direct.isra.0+0x64/0xb8 blk_rq_dma_map_iter_next+0x6c/0xc8 nvme_prep_rq+0x894/0xa98 nvme_queue_rqs+0xb0/0x1a0 blk_mq_dispatch_queue_requests+0x268/0x3b8 blk_mq_flush_plug_list+0x90/0x188 __blk_flush_plug+0x104/0x170 blk_finish_plug+0x38/0x50 read_pages+0x1a4/0x3b8 page_cache_ra_unbounded+0x1a0/0x400 force_page_cache_ra+0xa8/0xd8 page_cache_sync_ra+0xa0/0x3f8 filemap_get_pages+0x104/0x950 filemap_read+0xf4/0x498 blkdev_read_iter+0x88/0x180 vfs_read+0x214/0x310 ksys_read+0x70/0x110 __arm64_sys_read+0x20/0x30 invoke_syscall+0x4c/0x118 el0_svc_common.constprop.0+0xc4/0xf0 do_el0_svc+0x24/0x38 el0_svc+0x1a0/0x340 el0t_64_sync_handler+0x98/0xe0 el0t_64_sync+0x17c/0x180 ---[ end trace 0000000000000000 ]---
Keith Busch
maintainerQuoted 10 lines · Leon Romanovsky
Quoted 9 lines · Keith Busch
Quoted 8 lines · Leon Romanovsky
Quoted 6 lines · Marek Szyprowski
Quoted 3 lines
This series does the core code and modern flows. A followup series will give the same treatment to the legacy
dma_opsimplementation.Applied patches 1-13 into dma-mapping-for-next branch. Let's check if it works fine in linux-next.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Thanks a lot.
Just fyi, when dma debug is enabled, we're seeing this new warning below. I have not had a chance to look into it yet, so I'm just reporting the observation.
Did you apply all patches or only Marek's branch?
I don't get this warning when I run my NVMe tests on current dmabuf-vfio branch.
This was the snapshot of linux-next from the 20250918 tag. It doesn't have the full patchset applied.
One other thing to note, this was runing on arm64 platform using smmu configured with 64k pages. If your iommu granule is 4k instead, we wouldn't use the blk_dma_map_direct path.
Jason Gunthorpe
maintainerQuoted 20 lines · Keith Busch
Quoted 10 lines · Leon Romanovsky
Quoted 9 lines · Keith Busch
Quoted 8 lines · Leon Romanovsky
Quoted 6 lines · Marek Szyprowski
Quoted 3 lines
This series does the core code and modern flows. A followup series will give the same treatment to the legacy
dma_opsimplementation.Applied patches 1-13 into dma-mapping-for-next branch. Let's check if it works fine in linux-next.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Thanks a lot.
Just fyi, when dma debug is enabled, we're seeing this new warning below. I have not had a chance to look into it yet, so I'm just reporting the observation.
Did you apply all patches or only Marek's branch?
I don't get this warning when I run my NVMe tests on current dmabuf-vfio branch.
Quoted 6 lines
This was the snapshot of linux-next from the 20250918 tag. It doesn't have the full patchset applied.
One other thing to note, this was runing on arm64 platform using smmu configured with 64k pages. If your iommu granule is 4k instead, we wouldn't use the
blk_dma_map_directpath.
I spent some time looking to see if I could guess what this is and came up empty. It seems most likely we are leaking a dma mapping tracking somehow? The DMA API side is pretty simple here though..
Not sure the 64k/4k itself is a cause, but triggering the non-iova flow is probably the issue.
Can you check the output of this debugfs:
/*
* Dump mappings entries on user space via debugfs */
static int dump_show(struct seq_file *seq, void *v)
? If the system is idle and it has lots of entries that is probably confirmation of the theory.
Jason
Keith Busch
maintainerQuoted 9 lines · Jason Gunthorpe
Quoted 4 lines · Keith Busch
One other thing to note, this was runing on arm64 platform using smmu configured with 64k pages. If your iommu granule is 4k instead, we wouldn't use the
blk_dma_map_directpath.I spent some time looking to see if I could guess what this is and came up empty. It seems most likely we are leaking a dma mapping tracking somehow? The DMA API side is pretty simple here though..
Yeah, nothing stood out to me here either.
Quoted 4 lines
Not sure the 64k/4k itself is a cause, but triggering the non-iova flow is probably the issue.
Can you check the output of this debugfs:
I don't have a system in this state at the moment, so we checked previous logs on machines running older kernels. It's extermely uncommon, but this error was happening prior to this series, so I don't think this introduced any new problem here. I'll keeping looking, but I don't think we'll make much progress if I can't find a more reliable reproducer.
Thanks!
Jason Gunthorpe
maintainerQuoted 6 lines · Keith Busch
I don't have a system in this state at the moment, so we checked previous logs on machines running older kernels. It's extermely uncommon, but this error was happening prior to this series, so I don't think this introduced any new problem here. I'll keeping looking, but I don't think we'll make much progress if I can't find a more reliable reproducer.
Okay, that's great. It needs to get resolved but it is not this series at fault.
Very rare is a different perspective, I mis-thought it was happening reproducible all the time..
It seems to me it is actually a legitimate thing for userspace to be able to trigger this cache line debug. If you do concurrent O_DIRECT to the very same memory it should trigger if I read it right..
So it may not even be an actual bug???
Jason
Keith Busch
maintainerQuoted 2 lines · Jason Gunthorpe
Very rare is a different perspective, I mis-thought it was happening reproducible all the time..
Yes, sorry for the false alarm. I think we got unlucky and hit it on one of the first boots from testing linux-next, so knee-jerk reaction was to suspect the new code that showed up in the stack.