net: Allow SF devices to be used for ZC DMA
Subsystems (4)
PATCH v2 1/1 net: Allow SF devices to be used for ZC DMA
For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
scalable function netdevs [1] have the DMA device in the grandparent.
This patch adds a helper for getting the DMA device for a netdev from its parent or grandparent if necessary. The NULL case is handled in the callers.
devmem and io_uring are updated accordingly to use this helper instead of directly using the parent.
[1] Documentation/networking/device_drivers/ethernet/mellanox/mlx5/switchdev.rst
----
Changes in v2 [2]:
- Dropped the Fixes tag.
- Added more documentation as requeseted.
- Renamed the patch title to better reflect its purpose.
Changes in v1 [1]:
- Upgraded from RFC status.
- Dropped driver specific bits for generic solution.
- Implemented single patch as a fix as requested in RFC.
- Handling of multi-PF netdevs will be handled in a subsequent patch series.
[1] RFC: https://lore.kernel.org/all/[email protected]/
[2] v2: https://lore.kernel.org/all/[email protected]/
--- include/linux/netdevice.h | 21 +++++++++++++++++++++ io_uring/zcrx.c | 2 +- net/core/devmem.c | 10 +++++++++- 3 files changed, 31 insertions(+), 2 deletions(-)
include/linux/netdevice.h +21 -0
@@ -5560,4 +5560,25 @@ extern struct net_device *blackhole_netdev; atomic_long_add((VAL), &(DEV)->stats.__##FIELD)#define DEV_STATS_READ (DEV ,FIELD ) atomic_long_read(&(DEV)->stats.__##FIELD) +static inline struct device *netdev_get_dma_dev (const struct net_device *dev) +{ +struct device *dma_dev = dev->dev.parent; + +if (!dma_dev) +return NULL ; + +/* Common case: dma device is parent device of netdev. */ +if (dma_dev->dma_mask) +return dma_dev; + +/* SF netdevs have an auxdev device as parent, the dma device being the +* grandparent. +*/ + dma_dev = dma_dev->parent; +if (dma_dev && dma_dev->dma_mask) +return dma_dev; + +return NULL ; +} +#endif /* _LINUX_NETDEVICE_H */
io_uring/zcrx.c +1 -1
@@ -584,7 +584,7 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx, goto err; } - ifq->dev = ifq->netdev->dev.parent; + ifq->dev =netdev_get_dma_dev (ifq->netdev);if (!ifq->dev) { ret = -EOPNOTSUPP ; goto err;
net/core/devmem.c +9 -1
@@ -183,6 +183,7 @@ net_devmem_bind_dmabuf(struct net_device *dev, {struct net_devmem_dmabuf_binding *binding;static u32 id_alloc_next; +struct device *dma_dev;struct scatterlist *sg;struct dma_buf *dmabuf;unsigned int sg_idx, i;
@@ -193,6 +194,13 @@ net_devmem_bind_dmabuf(struct net_device *dev,if (IS_ERR (dmabuf))return ERR_CAST (dmabuf); + dma_dev =netdev_get_dma_dev (dev); +if (!dma_dev) { + err = -EOPNOTSUPP ; +NL_SET_ERR_MSG (extack,"Device doesn't support dma" ); + goto err_put_dmabuf; + } + binding =kzalloc_node (sizeof (*binding),GFP_KERNEL ,dev_to_node (&dev->dev));if (!binding) {
@@ -209,7 +217,7 @@ net_devmem_bind_dmabuf(struct net_device *dev, binding->dmabuf = dmabuf; - binding->attachment =dma_buf_attach (binding->dmabuf, dev->dev.parent); + binding->attachment =dma_buf_attach (binding->dmabuf, dma_dev);if (IS_ERR (binding->attachment)) { err =PTR_ERR (binding->attachment);NL_SET_ERR_MSG (extack,"Failed to bind dmabuf to device" );
Signature · 1 line
2.43.0
Jakub Kicinski
maintainerQuoted 18 lines · On Fri, 11 Jul 2025 09:26:34 +0000 Dragos Tatulea wrote:
include/linux/netdevice.h +10
-0@@ -5560,4 +5560,25 @@ extern struct net_device *blackhole_netdev; atomic_long_add((VAL), &(DEV)->stats.__##FIELD)#define DEV_STATS_READ (DEV ,FIELD ) atomic_long_read(&(DEV)->stats.__##FIELD) +static inline struct device *netdev_get_dma_dev (const struct net_device *dev) +{ +struct device *dma_dev = dev->dev.parent; + +if (!dma_dev) +return NULL ; + +/* Common case: dma device is parent device of netdev. */ +if (dma_dev->dma_mask) +return dma_dev;
include/linux/netdevice.h extern struct net_device *blackhole_netdev; · line 5582
Quoted 10 lines
include/linux/netdevice.h +10
-0+ +/* SF netdevs have an auxdev device as parent, the dma device being the +* grandparent. +*/ + dma_dev = dma_dev->parent; +if (dma_dev && dma_dev->dma_mask) +return dma_dev; + +return NULL ; +}
LGTM, but we need a better place for this function. netdevice.h is included directly by 1.5k files, and indirectly by probably another 5k. It's not a great place to put random helpers with 2 callers. Maybe net/netdev_rx_queue.h and net/core/netdev_rx_queue.c? I don't think it needs to be a static inline either.
Signature · 1 line
pw-bot: cr
Quoted 18 lines · Jakub Kicinski
Quoted 18 lines
include/linux/netdevice.h +18
-0+static inline struct device *netdev_get_dma_dev (const struct net_device *dev) +{ +struct device *dma_dev = dev->dev.parent; + +if (!dma_dev) +return NULL ; + +/* Common case: dma device is parent device of netdev. */ +if (dma_dev->dma_mask) +return dma_dev; + +/* SF netdevs have an auxdev device as parent, the dma device being the +* grandparent. +*/ + dma_dev = dma_dev->parent; +if (dma_dev && dma_dev->dma_mask) +return dma_dev; +
Quoted 8 lines
Review
Quoted 2 lines
(quoted diff) +2
-0+ return NULL; +}LGTM, but we need a better place for this function.
netdevice.his included directly by 1.5k files, and indirectly by probably another 5k. It's not a great place to put random helpers with 2 callers. Maybenet/netdev_rx_queue.handnet/core/netdev_rx_queue.c? I don't think it needs to be a static inline either.
The whole concept is also buggy. Trying to get a dma-able device by walking down from an upper level construct like the netdevice can't work reliably. You'll need to explicitly provide the dma_device using either a method or a pointer to it instead of this guesswork.
Jakub Kicinski
maintainerQuoted 10 lines · On Mon, 14 Jul 2025 21:39:30 -0700 Christoph Hellwig wrote:
Quoted 5 lines
LGTM, but we need a better place for this function.
netdevice.his included directly by 1.5k files, and indirectly by probably another 5k. It's not a great place to put random helpers with 2 callers. Maybenet/netdev_rx_queue.handnet/core/netdev_rx_queue.c? I don't think it needs to be a static inline either.The whole concept is also buggy. Trying to get a dma-able device by walking down from an upper level construct like the netdevice can't work reliably. You'll need to explicitly provide the
dma_deviceusing either a method or a pointer to it instead of this guesswork.
Yeah, I'm pretty sure we'll end up with a method in queue ops. But it's not that deep, an easy thing to change.
Quoted 7 lines · Jakub Kicinski
Quoted 6 lines · On Mon, 14 Jul 2025 21:39:30 -0700 Christoph Hellwig wrote:
Quoted 5 lines
LGTM, but we need a better place for this function.
netdevice.his included directly by 1.5k files, and indirectly by probably another 5k. It's not a great place to put random helpers with 2 callers. Maybenet/netdev_rx_queue.handnet/core/netdev_rx_queue.c? I don't think it needs to be a static inline either.
Quoted 7 lines
Quoted 4 lines
The whole concept is also buggy. Trying to get a dma-able device by walking down from an upper level construct like the netdevice can't work reliably. You'll need to explicitly provide the
dma_deviceusing either a method or a pointer to it instead of this guesswork.Yeah, I'm pretty sure we'll end up with a method in queue ops. But it's not that deep, an easy thing to change.
Why not get this right now instead of adding more of the hacky parent walking?
Quoted 17 lines
From: Christoph Hellwig <[email protected]>
Sent: 16 July 2025 04:44 PMQuoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Mon, 14 Jul 2025 21:39:30 -0700 Christoph Hellwig wrote:
Quoted 5 lines
LGTM, but we need a better place for this function.
netdevice.his included directly by 1.5k files, and indirectly by probably another 5k. It's not a great place to put random helpers with 2 callers. Maybenet/netdev_rx_queue.handnet/core/netdev_rx_queue.c? I don't think it needs to be a static inline either.The whole concept is also buggy. Trying to get a dma-able device by walking down from an upper level construct like the netdevice can't work reliably. You'll need to explicitly provide the
dma_deviceusing either a method or a pointer to it instead of this guesswork.
Quoted 5 lines
Quoted 2 lines
Yeah, I'm pretty sure we'll end up with a method in queue ops. But it's not that deep, an easy thing to change.
Why not get this right now instead of adding more of the hacky parent walking?
The previous RFC version (v1) [1], the driver was explicitly providing dma_dev
at device level.
Queue level is even better; it will address the Netdev with two pci devs socket direct use case too. Not sure how difficult it is.
Dragos can you please evaluate?
I believe the dma_mask check in [1] should be removed regardless.
Quoted 17 lines · Parav Pandit
Quoted 17 lines
From: Christoph Hellwig <[email protected]>
Sent: 16 July 2025 04:44 PMQuoted 12 lines · Jakub Kicinski
Quoted 10 lines · On Mon, 14 Jul 2025 21:39:30 -0700 Christoph Hellwig wrote:
Quoted 5 lines
LGTM, but we need a better place for this function.
netdevice.his included directly by 1.5k files, and indirectly by probably another 5k. It's not a great place to put random helpers with 2 callers. Maybenet/netdev_rx_queue.handnet/core/netdev_rx_queue.c? I don't think it needs to be a static inline either.The whole concept is also buggy. Trying to get a dma-able device by walking down from an upper level construct like the netdevice can't work reliably. You'll need to explicitly provide the
dma_deviceusing either a method or a pointer to it instead of this guesswork.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Yeah, I'm pretty sure we'll end up with a method in queue ops. But it's not that deep, an easy thing to change.
Why not get this right now instead of adding more of the hacky parent walking?
The previous RFC version (v1) [1], the driver was explicitly providing
dma_dev
at device level.
Queue level is even better; it will address the Netdev with two pci devs socket direct use case too.
Yup. This was the second more generic proposal in the RFC [1].
Quoted 4 lines
Not sure how difficult it is.
Dragos can you please evaluate?
It is not difficult. But some changes are required in net_devmem_bind_dmabuf() and its callers.
Will send a v3 with the changes.
Quoted 3 lines
I believe the
dma_maskcheck in [1] should be removed regardless.
Why is that? What if the parent device really doesn't support DMA?
[1] https://lore.kernel.org/netdev/22kf5wtxym5x3zllar7ek3onkav6nfzclf7w2lzifhebjme4jb@h4qycdqmwern/
Thanks,
Dragos