Skip to contentSkip to conversation

ntb_transport: replace wmb() with dma_wmb() in TX paths

PATCH In review Xingbang Liu 2 participants 3 messages Apr 24 – Apr 27 NTB driver core

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ntb_transport: replace wmb() with dma_wmb() in TX paths

The TX paths currently use wmb() after memcpy_toio() to ensure that writes to the peer memory window are visible before subsequent notification.

The actual ordering requirement is to ensure that data written to the peer memory is observed before the associated notification or status update. This matches the semantics of dma_wmb(), which orders memory writes with respect to external observers.

The peer memory window is mapped with ioremap_wc(), allowing write-combining. On weakly ordered architectures such as arm64, dma_wmb() ensures that prior writes are committed from write-combining buffers and become visible to the peer before subsequent operations.

This aligns with common patterns in PCIe-based drivers, where it is used to order descriptor or payload writes before notification, while avoiding the stronger ordering semantics of wmb().

The ordering with respect to peer observation is completed by the subsequent notification mechanism.

No functional change is intended.

---
 drivers/ntb/ntb_transport.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ntb/ntb_transport.c +1 -1
index 7cabc82305d6..c8ef46ddcc57 100644
@@ -1791,7 +1791,7 @@ static void ntb_memcpy_tx_on_stack(struct ntb_queue_entry *entry, void __iomem *
 #endif
 
 	/* Ensure that the data is fully copied out before setting the flags */
-	wmb();
+	dma_wmb();
 
 	ntb_tx_copy_callback(entry, NULL);
 }
Signature · 1 line
2.34.1
maintainer
#1 lore raw
Quoted 18 lines · Xingbang Liu

The TX paths currently use wmb() after memcpy_toio() to ensure that writes to the peer memory window are visible before subsequent notification.

The actual ordering requirement is to ensure that data written to the peer memory is observed before the associated notification or status update. This matches the semantics of dma_wmb(), which orders memory writes with respect to external observers.

The peer memory window is mapped with ioremap_wc(), allowing write-combining. On weakly ordered architectures such as arm64, dma_wmb() ensures that prior writes are committed from write-combining buffers and become visible to the peer before subsequent operations.

This aligns with common patterns in PCIe-based drivers, where it is used to order descriptor or payload writes before notification, while avoiding the stronger ordering semantics of wmb().

Quoted 6 lines

The ordering with respect to peer observation is completed by the subsequent notification mechanism.

No functional change is intended.

NAK. dma_wmb() is used for shared memory between CPU and a device. For example:

memcpy(dma_buffer, data, size);
dma_wmb(); /* Ensure CPU writes visible to device DMA */ writel(DOORBELL, device->reg + DB_OFFSET); /* trigger device DMA from dma_buffer */

In this case we are copying data to MMIO and therefore wmb() is needed to ensure all MMIO writes have happened before we proceed.

DJ

Quoted 17 lines
---
 drivers/ntb/ntb_transport.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ntb/ntb_transport.c +1 -1
index 7cabc82305d6..c8ef46ddcc57 100644
@@ -1791,7 +1791,7 @@ static void ntb_memcpy_tx_on_stack(struct ntb_queue_entry *entry, void __iomem *
 #endif
 
 	/* Ensure that the data is fully copied out before setting the flags */
-	wmb();
+	dma_wmb();
 
 	ntb_tx_copy_callback(entry, NULL);
 }
#2 lore raw

Hi DJ ,

Thanks for the review and the explanation.

I understand that dma_wmb() is intended for CPU → device DMA visibility, and since this path writes to MMIO (peer memory window), a full wmb() is required to guarantee that all writes are visible before the notification/doorbell.

I will withdraw this patch and keep the wmb() in place.

Thanks,
Xingbang Liu

Quoted 57 lines · Dave Jiang <[email protected]> 于2026年4月24日周五 23:32写道:
Quoted 18 lines · Xingbang Liu

The TX paths currently use wmb() after memcpy_toio() to ensure that writes to the peer memory window are visible before subsequent notification.

The actual ordering requirement is to ensure that data written to the peer memory is observed before the associated notification or status update. This matches the semantics of dma_wmb(), which orders memory writes with respect to external observers.

The peer memory window is mapped with ioremap_wc(), allowing write-combining. On weakly ordered architectures such as arm64, dma_wmb() ensures that prior writes are committed from write-combining buffers and become visible to the peer before subsequent operations.

This aligns with common patterns in PCIe-based drivers, where it is used to order descriptor or payload writes before notification, while avoiding the stronger ordering semantics of wmb().

Quoted 6 lines

The ordering with respect to peer observation is completed by the subsequent notification mechanism.

No functional change is intended.

NAK. dma_wmb() is used for shared memory between CPU and a device. For example:

memcpy(dma_buffer, data, size);
dma_wmb(); /* Ensure CPU writes visible to device DMA */ writel(DOORBELL, device->reg + DB_OFFSET); /* trigger device DMA from dma_buffer */

In this case we are copying data to MMIO and therefore wmb() is needed to ensure all MMIO writes have happened before we proceed.

DJ

Quoted 17 lines
---
 drivers/ntb/ntb_transport.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
drivers/ntb/ntb_transport.c +1 -1
index 7cabc82305d6..c8ef46ddcc57 100644
@@ -1791,7 +1791,7 @@ static void ntb_memcpy_tx_on_stack(struct ntb_queue_entry *entry, void __iomem *
 #endif
 
      /* Ensure that the data is fully copied out before setting the flags */
-     wmb();
+     dma_wmb();
 
      ntb_tx_copy_callback(entry, NULL);
 }