Skip to contentSkip to conversation

devmem/io_uring: Allow devices without parent PCI device

RFC In review Dragos Tatulea 5 participants 24 messages Jul 2, 2025 – Jul 11, 2025

Subsystems (7)
lore
Share & tools
lore
#0 lore raw

cover devmem/io_uring: Allow devices without parent PCI device

The io_uring and devmem code is assuming that the parent device of the netdev is a DMA capable device. This is not always the case.

Some devices do have a DMA capable device that can be used, but not as
parent: mlx5 SFs have an auxdev as parent, but they do have an associated PCI device.

Also, if DMA is not supported the operation should be blocked. Otherwise the mapping will return success with 0 mapped entries and the caller will consider the mapping as succesful.

This RFC is supposed to start the discussion on the best way to:
- Block the binding operation early if not supported.
- Allow devices that support this usecase but don't have a parent device as a PCI device.

Dragos Tatulea (4):
net: Allow non parent devices to be used for ZC DMA io_uring/zcrx: Use the new netdev_get_dma_dev() API
net: devmem: Use the new netdev_get_dma_dev() API net/mlx5e: Enable HDS zerocopy flows for SFs

 drivers/net/ethernet/mellanox/mlx5/core/en_main.c |  3 +++
 include/linux/netdevice.h                         | 13 +++++++++++++
 io_uring/zcrx.c                                   |  2 +-
 net/core/devmem.c                                 | 10 +++++++++-
 4 files changed, 26 insertions(+), 2 deletions(-)
Signature · 1 line
2.50.0
#1 lore raw

RFC 1/4 net: Allow non parent devices to be used for ZC DMA

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

This patch introduces the possibility for the driver to specify another DMA device to be used via the new dma_dev field. The field should be set before register_netdev().

A new helper function is added to get the DMA device or return NULL. The callers can check for NULL and fail early if the device is not capable of DMA.

---
 include/linux/netdevice.h | 13 +++++++++++++
 1 file changed, 13 insertions(+)
include/linux/netdevice.h +13 -0
index 5847c20994d3..83faa2314c30 100644
@@ -2550,6 +2550,9 @@ struct net_device {
 
 	struct hwtstamp_provider __rcu	*hwprov;
 
+	/* To be set by devices that can do DMA but not via parent. */
+	struct device		*dma_dev;
+
 	u8			priv[] ____cacheline_aligned
 				       __counted_by(priv_len);
 } ____cacheline_aligned;
@@ -5560,4 +5563,14 @@ extern struct net_device *blackhole_netdev;
 		atomic_long_add((VAL), &(DEV)->stats.__##FIELD)
 #define DEV_STATS_READ(DEV, FIELD) atomic_long_read(&(DEV)->stats.__##FIELD)
 
+static inline struct device *netdev_get_dma_dev(const struct net_device *dev)
+{
+	struct device *dma_dev = dev->dma_dev ? dev->dma_dev : dev->dev.parent;
+
+	if (!dma_dev->dma_mask)
+		dma_dev = NULL;
+
+	return dma_dev;
+}
+
 #endif	/* _LINUX_NETDEVICE_H */
Signature · 1 line
2.50.0
maintainer
#2 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 3 lines · On Wed, 2 Jul 2025 20:24:23 +0300 Dragos Tatulea wrote:

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

Noob question -- I thought that the point of SFs was that you can pass them thru to a VM. How do they not have DMA support? Is it added on demand by the mediated driver or some such?

#3 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 2 Jul 2025 20:24:23 +0300 Dragos Tatulea wrote:

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

Noob question -- I thought that the point of SFs was that you can pass them thru to a VM. How do they not have DMA support? Is it added on demand by the mediated driver or some such?

They do have DMA support. Maybe didn't state it properly in the commit message. It is just that the the parent device (sf_netdev->dev.parent.device) is not a DMA device. The grandparent device is a DMA device though (PCI dev of parent PFs). But I wanted to keep it generic. Maybe it doesn't need to be so generic?

Regarding SFs and VM passtrhough: my understanding is that SFs are more for passing them to a container.

Thanks,
Dragos

maintainer
#4 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 9 lines · On Wed, 2 Jul 2025 20:01:48 +0000 Dragos Tatulea wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 2 Jul 2025 20:24:23 +0300 Dragos Tatulea wrote:

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

Noob question -- I thought that the point of SFs was that you can pass them thru to a VM. How do they not have DMA support? Is it added on demand by the mediated driver or some such?

Quoted 8 lines

They do have DMA support. Maybe didn't state it properly in the commit message. It is just that the the parent device (sf_netdev->dev.parent.device) is not a DMA device. The grandparent device is a DMA device though (PCI dev of parent PFs). But I wanted to keep it generic. Maybe it doesn't need to be so generic?

Regarding SFs and VM passtrhough: my understanding is that SFs are more for passing them to a container.

Mm. We had macvlan offload for over a decade, there's no need for a fake struct device, auxbus and all them layers to delegate a "subdevice" to a container in netdev world.
In my head subfunctions are a way of configuring a PCIe PASID ergo they _only_ make sense in context of DMA.
Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

#5 lore raw

RE: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 20 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

Quoted 15 lines · On Wed, 2 Jul 2025 20:01:48 +0000 Dragos Tatulea wrote:
Quoted 8 lines · Jakub Kicinski
Quoted 3 lines · On Wed, 2 Jul 2025 20:24:23 +0300 Dragos Tatulea wrote:

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

Noob question -- I thought that the point of SFs was that you can pass them thru to a VM. How do they not have DMA support? Is it added on demand by the mediated driver or some such?

They do have DMA support. Maybe didn't state it properly in the commit message. It is just that the the parent device (sf_netdev->dev.parent.device) is not a DMA device. The grandparent device is a DMA device though (PCI dev of parent PFs). But I wanted to keep it generic. Maybe it doesn't need to be so generic?

Quoted 6 lines
Quoted 2 lines

Regarding SFs and VM passtrhough: my understanding is that SFs are more for passing them to a container.

Mm. We had macvlan offload for over a decade, there's no need for a fake struct device, auxbus and all them layers to delegate a "subdevice" to a container in netdev world.

SFs are full PCI devices except having unique PCI BDF as they utilize the parent PCI Device's BDF (RID).
Presently, SFs are used with and without containers when users need
hw based netdevs.
Some CSPs use them as hot-plug devices from the DPU side too.

Unlike macvlan,
SF netdevs have dedicated hw queues, switchdev representors, mtu, qdiscs, QoS rate limiters.
vdpa of SFs is prominent use too to offload virtio queues. And some are using SFs rdma devices too.

SFs are the pre-SIOV_R2 devices and hence reliance of auxiliary bus and utilizing core driver infrastructure sort of aligns to the kernel core. If I recollect correctly, the Intel ICE SFs are exactly similar.

Quoted 2 lines

In my head subfunctions are a way of configuring a PCIe PASID ergo they _only_ make sense in context of DMA.

SF DMA is on the parent PCI device.

SIOV_R2 will have its own PCI RID which is ratified or getting ratified. When its done, SF (as SIOV_R2 device) instantiation can be extended with its own PCI RID. At that point they can be mapped to a VM.

Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent? So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be mistaken here.

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

#6 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 4 lines · Parav Pandit
Quoted 4 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

[...]

Quoted 19 lines
Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent? So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be mistaken here.

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Quoted 8 lines

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going one parent up if the parent device is not DMA capable would be a good starting point.

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue. io_uring could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

[0] https://docs.kernel.org/networking/multi-pf-netdev.html

Thanks,
Dragos

#7 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 31 lines · Dragos Tatulea
Quoted 4 lines · Parav Pandit
Quoted 4 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

[...]

Quoted 24 lines
Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent? So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be mistaken here.

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

Quoted 6 lines
Quoted 3 lines

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

I think what's being discussed here is pretty straight forward and doesn't need UAPI changes, right? Or were you referring to another API?

Quoted 9 lines

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going one parent up if the parent device is not DMA capable would be a good starting point.

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue.

Agreed these are the 2 options.

Quoted 4 lines

io_uring
could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

Why? AFAICT both io_uring and devmem are in the exact same boat right now, and your patchset seems to show that? Both use dev->dev.parent as the mapping device, and AFAIU you want to use dev->dev.parent.parent or something like that?

Also AFAIU the driver won't need to block the multi PF case, it's actually core that would need to handle that. For example, if devmem wants to bind a dmabuf to 4 queues, but queues 0 & 1 use 1 dma device, but queues 2 & 3 use another dma-device, then core doesn't know what to do, because it can't map the dmabuf to both devices at once. The restriction would be at bind time that all the queues being bound to have the same dma device. Core would need to check that and return an error if the devices diverge. I imagine all of this is the same for io_uring, unless I'm missing something.

Quoted 3 lines

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

At the moment, from your description of the problem, I would lean to going with Jakub's approach and handling the common case via #1. If more use cases that require a very custom dma device to be passed we can always move to #2 later, but FWIW I don't see a reason to come up with a super future proof complicated solution right now, but I'm happy to hear disagreements.

Signature · 2 lines
Thanks,
Mina
#8 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 35 lines · Mina Almasry
Quoted 34 lines · Dragos Tatulea
Quoted 4 lines · Parav Pandit
Quoted 4 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

[...]

Quoted 27 lines
Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent? So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be mistaken here.

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

Quoted 8 lines
Quoted 3 lines

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

I think what's being discussed here is pretty straight forward and doesn't need UAPI changes, right? Or were you referring to another API?

I was referring to the fact that devmem takes one big buffer, maps it for a single device (in net_devmem_bind_dmabuf()) and then assigns it to queues in net_devmem_bind_dmabuf_to_queue(). As the single buffer is part of the API, I don't see how the mapping could be done in a per queue way.

Quoted 17 lines
Quoted 9 lines

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going one parent up if the parent device is not DMA capable would be a good starting point.

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue.

Agreed these are the 2 options.

Quoted 4 lines

io_uring
could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

Quoted 5 lines

Why? AFAICT both io_uring and devmem are in the exact same boat right now, and your patchset seems to show that? Both use dev->dev.parent as the mapping device, and AFAIU you want to use dev->dev.parent.parent or something like that?

Right. My patches show that. But the issue raised by Parav is different:
different queues can belong to different DMA devices from different PFs in the case of Multi PF netdev.

io_uring can do it because it maps individual buffers to individual queues. So it would be trivial to get the DMA device of each queue through a new queue op.

Quoted 10 lines

Also AFAIU the driver won't need to block the multi PF case, it's actually core that would need to handle that. For example, if devmem wants to bind a dmabuf to 4 queues, but queues 0 & 1 use 1 dma device, but queues 2 & 3 use another dma-device, then core doesn't know what to do, because it can't map the dmabuf to both devices at once. The restriction would be at bind time that all the queues being bound to have the same dma device. Core would need to check that and return an error if the devices diverge. I imagine all of this is the same for io_uring, unless I'm missing something.

Agreed. Currently I didn't see an API for Multi PF netdev to expose this information so my thinking defaulted to "let's block it from the driver side".

Quoted 10 lines
Quoted 3 lines

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

At the moment, from your description of the problem, I would lean to going with Jakub's approach and handling the common case via #1. If more use cases that require a very custom dma device to be passed we can always move to #2 later, but FWIW I don't see a reason to come up with a super future proof complicated solution right now, but I'm happy to hear disagreements.

But we also don't want to start off on the left foot when we know of both issues right now. And I think we can wrap it up nicely in a single function similary to how the current patch does it.

Thanks,
Dragos

#9 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 45 lines · Dragos Tatulea
Quoted 35 lines · Mina Almasry
Quoted 34 lines · Dragos Tatulea
Quoted 4 lines · Parav Pandit
Quoted 4 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

[...]

Quoted 27 lines
Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent? So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be mistaken here.

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

Quoted 8 lines
Quoted 3 lines

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

I think what's being discussed here is pretty straight forward and doesn't need UAPI changes, right? Or were you referring to another API?

Quoted 6 lines

I was referring to the fact that devmem takes one big buffer, maps it for a single device (in net_devmem_bind_dmabuf()) and then assigns it to queues in net_devmem_bind_dmabuf_to_queue(). As the single buffer is part of the API, I don't see how the mapping could be done in a per queue way.

Oh, I see. devmem does support mapping a single buffer to multiple queues in a single netlink API call, but there is nothing stopping the user from mapping N buffers to N queues in N netlink API calls.

Quoted 22 lines
Quoted 17 lines
Quoted 9 lines

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going one parent up if the parent device is not DMA capable would be a good starting point.

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue.

Agreed these are the 2 options.

Quoted 4 lines

io_uring
could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

Quoted 5 lines

Why? AFAICT both io_uring and devmem are in the exact same boat right now, and your patchset seems to show that? Both use dev->dev.parent as the mapping device, and AFAIU you want to use dev->dev.parent.parent or something like that?

Quoted 8 lines

Right. My patches show that. But the issue raised by Parav is different:
different queues can belong to different DMA devices from different PFs in the case of Multi PF netdev.

io_uring can do it because it maps individual buffers to individual queues. So it would be trivial to get the DMA device of each queue through a new queue op.

Right, devmem doesn't stop you from mapping individual buffers to individual queues. It just also supports mapping the same buffer to multiple queues. AFAIR, io_uring also supports mapping a single buffer to multiple queues, but I could easily be very wrong about that. It's just a vague recollection from reviewing the iozcrx.c implementation a while back.

In your case, I think, if the user is trying to map a single buffer to multiple queues, and those queues have different dma-devices, then you have to error out. I don't see how to sanely handle that without adding a lot of code. The user would have to fall back onto mapping a single buffer to a single queue (or multiple queues that share the same dma-device).

Quoted 10 lines
Quoted 10 lines

Also AFAIU the driver won't need to block the multi PF case, it's actually core that would need to handle that. For example, if devmem wants to bind a dmabuf to 4 queues, but queues 0 & 1 use 1 dma device, but queues 2 & 3 use another dma-device, then core doesn't know what to do, because it can't map the dmabuf to both devices at once. The restriction would be at bind time that all the queues being bound to have the same dma device. Core would need to check that and return an error if the devices diverge. I imagine all of this is the same for io_uring, unless I'm missing something.

Quoted 4 lines

Agreed. Currently I didn't see an API for Multi PF netdev to expose this information so my thinking defaulted to "let's block it from the driver side".

Agreed.

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

At the moment, from your description of the problem, I would lean to going with Jakub's approach and handling the common case via #1. If more use cases that require a very custom dma device to be passed we can always move to #2 later, but FWIW I don't see a reason to come up with a super future proof complicated solution right now, but I'm happy to hear disagreements.

Quoted 4 lines

But we also don't want to start off on the left foot when we know of both issues right now. And I think we can wrap it up nicely in a single function similary to how the current patch does it.

FWIW I don't have a strong preference. I'm fine with the simple solution for now and I'm fine with the slightly more complicated future proof solution.

Signature · 2 lines
Thanks,
Mina
#10 lore raw

RE: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 136 lines

From: Mina Almasry <[email protected]>
Sent: 08 July 2025 03:25 AM

Quoted 3 lines · Dragos Tatulea
Quoted 1 line · Mina Almasry

On Fri, Jul 4, 2025 at 6:11 AM Dragos Tatulea <[email protected]>

Quoted 11 lines · wrote:
Quoted 11 lines
Quoted 11 lines · Parav Pandit
Quoted 4 lines
Quoted 4 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

[...]

Quoted 4 lines
Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you

suggest.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer

parent->parent?

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be

mistaken here.

Quoted 22 lines
Quoted 14 lines
Quoted 14 lines
Quoted 14 lines

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

Quoted 8 lines
Quoted 3 lines

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

I think what's being discussed here is pretty straight forward and doesn't need UAPI changes, right? Or were you referring to another API?

Quoted 6 lines

I was referring to the fact that devmem takes one big buffer, maps it for a single device (in net_devmem_bind_dmabuf()) and then assigns it to queues in net_devmem_bind_dmabuf_to_queue(). As the single buffer is part of the API, I don't see how the mapping could be done in a per queue way.

Oh, I see. devmem does support mapping a single buffer to multiple queues in a single netlink API call, but there is nothing stopping the user from mapping N buffers to N queues in N netlink API calls.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going

one

Quoted 18 lines
Quoted 13 lines
Quoted 5 lines

parent up if the parent device is not DMA capable would be a good starting point.

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue.

Agreed these are the 2 options.

Quoted 4 lines

io_uring
could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

Quoted 5 lines

Why? AFAICT both io_uring and devmem are in the exact same boat right now, and your patchset seems to show that? Both use dev->dev.parent as the mapping device, and AFAIU you want to use dev->dev.parent.parent or something like that?

Quoted 8 lines

Right. My patches show that. But the issue raised by Parav is different:
different queues can belong to different DMA devices from different PFs in the case of Multi PF netdev.

io_uring can do it because it maps individual buffers to individual queues. So it would be trivial to get the DMA device of each queue through a new queue op.

Right, devmem doesn't stop you from mapping individual buffers to individual queues. It just also supports mapping the same buffer to multiple queues. AFAIR, io_uring also supports mapping a single buffer to multiple queues, but I could easily be very wrong about that. It's just a vague recollection from reviewing the iozcrx.c implementation a while back.

In your case, I think, if the user is trying to map a single buffer to multiple queues, and those queues have different dma-devices, then you have to error out. I don't see how to sanely handle that without adding a lot of code. The user would have to fall back onto mapping a single buffer to a single queue (or multiple queues that share the same dma-device).

Quoted 10 lines
Quoted 10 lines

Also AFAIU the driver won't need to block the multi PF case, it's actually core that would need to handle that. For example, if devmem wants to bind a dmabuf to 4 queues, but queues 0 & 1 use 1 dma device, but queues 2 & 3 use another dma-device, then core doesn't know what to do, because it can't map the dmabuf to both devices at once. The restriction would be at bind time that all the queues being bound to have the same dma device. Core would need to check that and return an error if the devices diverge. I imagine all of this is the same for io_uring, unless I'm missing something.

Quoted 4 lines

Agreed. Currently I didn't see an API for Multi PF netdev to expose this information so my thinking defaulted to "let's block it from the driver side".

Agreed.

Quoted 10 lines
Quoted 10 lines
Quoted 3 lines

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

At the moment, from your description of the problem, I would lean to going with Jakub's approach and handling the common case via #1. If more use cases that require a very custom dma device to be passed we can always move to #2 later, but FWIW I don't see a reason to come up with a super future proof complicated solution right now, but I'm happy to hear disagreements.

Quoted 8 lines
Quoted 4 lines

But we also don't want to start off on the left foot when we know of both issues right now. And I think we can wrap it up nicely in a single function similary to how the current patch does it.

FWIW I don't have a strong preference. I'm fine with the simple solution for now and I'm fine with the slightly more complicated future proof solution.

Looks good to me as well.

maintainer
#11 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

In reply to Mina Almasry
Quoted 1 line · Mina Almasry

On Mon, Jul 7, 2025 at 2:35 PM Dragos Tatulea <[email protected]> wrote:

...>> Right. My patches show that. But the issue raised by Parav is different:

Quoted 8 lines
Quoted 7 lines

different queues can belong to different DMA devices from different PFs in the case of Multi PF netdev.

io_uring can do it because it maps individual buffers to individual queues. So it would be trivial to get the DMA device of each queue through a new queue op.

Quoted 4 lines

Right, devmem doesn't stop you from mapping individual buffers to individual queues. It just also supports mapping the same buffer to multiple queues. AFAIR, io_uring also supports mapping a single buffer to multiple queues, but I could easily be very wrong about that. It's

It doesn't, but it could benefit from sharing depending on userspace, so it might eventually come to the same problem.

Signature · 1 line
Pavel Begunkov
#12 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

In reply to Mina Almasry
Quoted 53 lines · Mina Almasry
Quoted 45 lines · Dragos Tatulea
Quoted 35 lines · Mina Almasry
Quoted 34 lines · Dragos Tatulea
Quoted 4 lines · Parav Pandit
Quoted 4 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

[...]

Quoted 27 lines
Quoted 4 lines

Maybe someone with closer understanding can chime in. If the kind of subfunctions you describe are expected, and there's a generic way of recognizing them -- automatically going to parent of parent would indeed be cleaner and less error prone, as you suggest.

I am not sure when the parent of parent assumption would fail, but can be a good start.

If netdev 8 bytes extension to store dma_dev is concern, probably a netdev IFF_DMA_DEV_PARENT can be elegant to refer parent->parent? So that there is no guess work in devmem layer.

That said, my understanding of devmem is limited, so I could be mistaken here.

In the long term, the devmem infrastructure likely needs to be modernized to support queue-level DMA mapping.
This is useful because drivers like mlx5 already support socket-direct netdev that span across two PCI devices.

Currently, devmem is limited to a single PCI device per netdev. While the buffer pool could be per device, the actual DMA mapping might need to be deferred until buffer posting time to support such multi-device scenarios.

In an offline discussion, Dragos mentioned that io_uring already operates at the queue level, may be some ideas can be picked up from io_uring?

Quoted 8 lines
Quoted 3 lines

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

I think what's being discussed here is pretty straight forward and doesn't need UAPI changes, right? Or were you referring to another API?

Quoted 6 lines

I was referring to the fact that devmem takes one big buffer, maps it for a single device (in net_devmem_bind_dmabuf()) and then assigns it to queues in net_devmem_bind_dmabuf_to_queue(). As the single buffer is part of the API, I don't see how the mapping could be done in a per queue way.

Quoted 3 lines

Oh, I see. devmem does support mapping a single buffer to multiple queues in a single netlink API call, but there is nothing stopping the user from mapping N buffers to N queues in N netlink API calls.

Oh, yes, of course. Why didn't I think of that...

Thanks,
Dragos

maintainer
#13 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

In reply to Dragos Tatulea
Quoted 4 lines · Dragos Tatulea
Quoted 3 lines · Parav Pandit
Quoted 3 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

...>> In an offline discussion, Dragos mentioned that io_uring already

Quoted 12 lines
Quoted 2 lines

operates at the queue level, may be some ideas can be picked up from io_uring?

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going one parent up if the parent device is not DMA capable would be a good starting point.

Quoted 7 lines

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue. io_uring could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

No objection from zcrx for either approach, but it sounds like a good idea to have something simple for 1) sooner than later, and perhaps marked as a fix.

Signature · 1 line
Pavel Begunkov
#14 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 23 lines · Pavel Begunkov
Quoted 4 lines · Dragos Tatulea
Quoted 3 lines · Parav Pandit
Quoted 3 lines

From: Jakub Kicinski <[email protected]>
Sent: 03 July 2025 02:23 AM

...>> In an offline discussion, Dragos mentioned that io_uring already

Quoted 17 lines
Quoted 2 lines

operates at the queue level, may be some ideas can be picked up from io_uring?

The problem for devmem is that the device based API is already set in stone so not sure how we can change this. Maybe Mina can chime in.

To sum the conversation up, there are 2 imperfect and overlapping solutions:

1) For the common case of having a single PCI device per netdev, going one parent up if the parent device is not DMA capable would be a good starting point.

2) For multi-PF netdev [0], a per-queue get_dma_dev() op would be ideal as it provides the right PF device for the given queue. io_uring could use this but devmem can't. Devmem could use 1. but the driver has to detect and block the multi PF case.

Quoted 7 lines
Quoted 2 lines

I think we need both. Either that or a netdev op with an optional queue parameter. Any thoughts?

No objection from zcrx for either approach, but it sounds like a good idea to have something simple for 1) sooner than later, and perhaps marked as a fix.

Sounds good. This is light enough to be a single patch.

Will tackle multi-PF netdev in a subsequent series.

Thanks,
Dragos

maintainer
#15 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

In reply to Parav Pandit
Quoted 7 lines · On Thu, 3 Jul 2025 11:58:50 +0000 Parav Pandit wrote:
Quoted 2 lines

In my head subfunctions are a way of configuring a PCIe PASID ergo they _only_ make sense in context of DMA.

SF DMA is on the parent PCI device.

SIOV_R2 will have its own PCI RID which is ratified or getting ratified. When its done, SF (as SIOV_R2 device) instantiation can be extended with its own PCI RID. At that point they can be mapped to a VM.

AFAIU every PCIe transaction for a queue with a PASID assigned should have a PASID prefix. Why is a different RID necessary? CPUs can't select IOMMU context based on RID+PASID?

#16 lore raw

RE: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 12 lines

From: Jakub Kicinski <[email protected]>
Sent: 11 July 2025 05:29 AM
Subject: Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 4 lines · On Thu, 3 Jul 2025 11:58:50 +0000 Parav Pandit wrote:
Quoted 2 lines

In my head subfunctions are a way of configuring a PCIe PASID ergo they _only_ make sense in context of DMA.

SF DMA is on the parent PCI device.

Quoted 7 lines
Quoted 3 lines

SIOV_R2 will have its own PCI RID which is ratified or getting ratified. When its done, SF (as SIOV_R2 device) instantiation can be extended with its own PCI RID. At that point they can be mapped to a VM.

AFAIU every PCIe transaction for a queue with a PASID assigned should have a PASID prefix. Why is a different RID necessary?
CPUs can't select IOMMU context based on RID+PASID?

It can, however,
PASID is meant to be used for process isolation and not expected to be abused for identify the device. Doing so, would also prohibits using PASID inside the VM. It requires another complex vPASID to pPASID translation.

Tagging MSI-X interrupts with PASID is another challenge. For CC defining isolation boundary with RID+PASID was yet another hack.

There were other issues in splitting PASID for device scaling vs process scaling for dual use.

So it was concluded to opt to avoid that abuse and use the standard RID construct for device identification.

maintainer
#17 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 22 lines · On Fri, 11 Jul 2025 02:52:23 +0000 Parav Pandit wrote:
Quoted 5 lines · On Thu, 3 Jul 2025 11:58:50 +0000 Parav Pandit wrote:
Quoted 4 lines
Quoted 2 lines

In my head subfunctions are a way of configuring a PCIe PASID ergo they _only_ make sense in context of DMA.

SF DMA is on the parent PCI device.

Quoted 7 lines
Quoted 3 lines

SIOV_R2 will have its own PCI RID which is ratified or getting ratified. When its done, SF (as SIOV_R2 device) instantiation can be extended with its own PCI RID. At that point they can be mapped to a VM.

AFAIU every PCIe transaction for a queue with a PASID assigned should have a PASID prefix. Why is a different RID necessary?
CPUs can't select IOMMU context based on RID+PASID?

It can, however,
PASID is meant to be used for process isolation and not expected to be abused for identify the device. Doing so, would also prohibits using PASID inside the VM. It requires another complex vPASID to pPASID translation.

Tagging MSI-X interrupts with PASID is another challenge. For CC defining isolation boundary with RID+PASID was yet another hack.

Quoted 5 lines

There were other issues in splitting PASID for device scaling vs process scaling for dual use.

So it was concluded to opt to avoid that abuse and use the standard RID construct for device identification.

I see, that explains it. Thanks Parav!

maintainer
#18 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

In reply to Dragos Tatulea
Quoted 32 lines · Dragos Tatulea

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

This patch introduces the possibility for the driver to specify another DMA device to be used via the new dma_dev field. The field should be set before register_netdev().

A new helper function is added to get the DMA device or return NULL. The callers can check for NULL and fail early if the device is not capable of DMA.

---
  include/linux/netdevice.h | 13 +++++++++++++
  1 file changed, 13 insertions(+)
include/linux/netdevice.h +3 -0
index 5847c20994d3..83faa2314c30 100644
@@ -2550,6 +2550,9 @@ struct net_device {
 
  	struct hwtstamp_provider __rcu	*hwprov;
 
+	/* To be set by devices that can do DMA but not via parent. */
+	struct device		*dma_dev;
+
  	u8			priv[] ____cacheline_aligned
  				       __counted_by(priv_len);
  } ____cacheline_aligned;

include/linux/netdevice.h extern struct net_device *blackhole_netdev; · line 5570

Quoted 8 lines
include/linux/netdevice.h +5 -0
@@ -5560,4 +5563,14 @@ extern struct net_device *blackhole_netdev;
  		atomic_long_add((VAL), &(DEV)->stats.__##FIELD)
  #define DEV_STATS_READ(DEV, FIELD) atomic_long_read(&(DEV)->stats.__##FIELD)
 
+static inline struct device *netdev_get_dma_dev(const struct net_device *dev)
+{
+	struct device *dma_dev = dev->dma_dev ? dev->dma_dev : dev->dev.parent;
+
+	if (!dma_dev->dma_mask)

dev->dev.parent is NULL for veth and I assume other virtual devices as well.

Mina, can you verify that devmem checks that? Seems like veth is rejected by netdev_need_ops_lock() in netdev_nl_bind_rx_doit(), but IIRC per netdev locking came after devmem got merged, and there are other virt devices that might already be converted.

Quoted 6 lines
include/linux/netdevice.h +5 -0
+		dma_dev = NULL;
+
+	return dma_dev;
+}
+
  #endif	/* _LINUX_NETDEVICE_H */
Signature · 1 line
Pavel Begunkov
#19 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 45 lines · Pavel Begunkov
Quoted 32 lines · Dragos Tatulea

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

This patch introduces the possibility for the driver to specify another DMA device to be used via the new dma_dev field. The field should be set before register_netdev().

A new helper function is added to get the DMA device or return NULL. The callers can check for NULL and fail early if the device is not capable of DMA.

---
  include/linux/netdevice.h | 13 +++++++++++++
  1 file changed, 13 insertions(+)
include/linux/netdevice.h +3 -0
index 5847c20994d3..83faa2314c30 100644
@@ -2550,6 +2550,9 @@ struct net_device {
 
      struct hwtstamp_provider __rcu  *hwprov;
 
+     /* To be set by devices that can do DMA but not via parent. */
+     struct device           *dma_dev;
+
      u8                      priv[] ____cacheline_aligned
                                     __counted_by(priv_len);
  } ____cacheline_aligned;

include/linux/netdevice.h extern struct net_device *blackhole_netdev; · line 5570

Quoted 8 lines
include/linux/netdevice.h +5 -0
@@ -5560,4 +5563,14 @@ extern struct net_device *blackhole_netdev;
              atomic_long_add((VAL), &(DEV)->stats.__##FIELD)
  #define DEV_STATS_READ(DEV, FIELD) atomic_long_read(&(DEV)->stats.__##FIELD)
 
+static inline struct device *netdev_get_dma_dev(const struct net_device *dev)
+{
+     struct device *dma_dev = dev->dma_dev ? dev->dma_dev : dev->dev.parent;
+
+     if (!dma_dev->dma_mask)

dev->dev.parent is NULL for veth and I assume other virtual devices as well.

Quoted 5 lines

Mina, can you verify that devmem checks that? Seems like veth is rejected by netdev_need_ops_lock() in netdev_nl_bind_rx_doit(), but IIRC per netdev locking came after devmem got merged, and there are other virt devices that might already be converted.

We never attempt devmem binding on any devices that don't support the queue API, even before the per netdev locking was merged (there was an explicit ops check).

even then, dev->dev.parent == NULL isn't disasterous, as far as I could surmise from a quick look. Seems to be only used with dma_buf_attach which NULL checks it.

Signature · 2 lines
Thanks,
Mina
maintainer
#20 lore raw

Re: [RFC net-next 1/4] net: Allow non parent devices to be used for ZC DMA

Quoted 52 lines · Mina Almasry
Quoted 45 lines · Pavel Begunkov
Quoted 32 lines · Dragos Tatulea

For zerocopy (io_uring, devmem), there is an assumption that the parent device can do DMA. However that is not always the case:
for example mlx5 SF devices have an auxiliary device as a parent.

This patch introduces the possibility for the driver to specify another DMA device to be used via the new dma_dev field. The field should be set before register_netdev().

A new helper function is added to get the DMA device or return NULL. The callers can check for NULL and fail early if the device is not capable of DMA.

---
   include/linux/netdevice.h | 13 +++++++++++++
   1 file changed, 13 insertions(+)
include/linux/netdevice.h +3 -0
index 5847c20994d3..83faa2314c30 100644
@@ -2550,6 +2550,9 @@ struct net_device {
 
       struct hwtstamp_provider __rcu  *hwprov;
 
+     /* To be set by devices that can do DMA but not via parent. */
+     struct device           *dma_dev;
+
       u8                      priv[] ____cacheline_aligned
                                      __counted_by(priv_len);
   } ____cacheline_aligned;

include/linux/netdevice.h extern struct net_device *blackhole_netdev; · line 5570

Quoted 8 lines
include/linux/netdevice.h +5 -0
@@ -5560,4 +5563,14 @@ extern struct net_device *blackhole_netdev;
               atomic_long_add((VAL), &(DEV)->stats.__##FIELD)
   #define DEV_STATS_READ(DEV, FIELD) atomic_long_read(&(DEV)->stats.__##FIELD)
 
+static inline struct device *netdev_get_dma_dev(const struct net_device *dev)
+{
+     struct device *dma_dev = dev->dma_dev ? dev->dma_dev : dev->dev.parent;
+
+     if (!dma_dev->dma_mask)

dev->dev.parent is NULL for veth and I assume other virtual devices as well.

Quoted 5 lines

Mina, can you verify that devmem checks that? Seems like veth is rejected by netdev_need_ops_lock() in netdev_nl_bind_rx_doit(), but IIRC per netdev locking came after devmem got merged, and there are other virt devices that might already be converted.

Quoted 3 lines

We never attempt devmem binding on any devices that don't support the queue API, even before the per netdev locking was merged (there was an explicit ops check).

great!

io_uring doesn't look at ->queue_mgmt_ops, so the helper from this patch needs to handle it one way or another.

Signature · 1 line
Pavel Begunkov
#21 lore raw

RFC 2/4 io_uring/zcrx: Use the new netdev_get_dma_dev() API

In reply to Dragos Tatulea

Using the new DMA dev helper API, there will be an early failure if the device does not support DMA.

---
 io_uring/zcrx.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
io_uring/zcrx.c +1 -1
index 797247a34cb7..93462e5b2207 100644
@@ -584,7 +584,7 @@ int io_register_zcrx_ifq(struct io_ring_ctx *ctx,
 		goto err;
 	}
 
-	ifq->dev = ifq->netdev->dev.parent;
+	ifq->dev = netdev_get_dma_dev(ifq->netdev);
 	if (!ifq->dev) {
 		ret = -EOPNOTSUPP;
 		goto err;
Signature · 1 line
2.50.0
#22 lore raw

RFC 3/4 net: devmem: Use the new netdev_get_dma_dev() API

In reply to Dragos Tatulea

Using the new DMA dev helper API, there will be an error on buffer binding if the device does not support DMA.

Previously this went through and was returning success event if the mappings were not done. Only a warning was printed.

---
 net/core/devmem.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
net/core/devmem.c +9 -1
index b3a62ca0df65..c6354b47257f 100644
@@ -183,6 +183,7 @@ net_devmem_bind_dmabuf(struct net_device *dev,
 {
 	struct net_devmem_dmabuf_binding *binding;
 	static u32 id_alloc_next;
+	struct device *dma_dev;
 	struct scatterlist *sg;
 	struct dma_buf *dmabuf;
 	unsigned int sg_idx, i;
@@ -193,6 +194,13 @@ net_devmem_bind_dmabuf(struct net_device *dev,
 	if (IS_ERR(dmabuf))
 		return ERR_CAST(dmabuf);
 
+	dma_dev = netdev_get_dma_dev(dev);
+	if (!dma_dev) {
+		err = -EOPNOTSUPP;
+		NL_SET_ERR_MSG(extack, "Parent device can't do dma");
+		goto err_put_dmabuf;
+	}
+
 	binding = kzalloc_node(sizeof(*binding), GFP_KERNEL,
 			       dev_to_node(&dev->dev));
 	if (!binding) {
@@ -209,7 +217,7 @@ net_devmem_bind_dmabuf(struct net_device *dev,
 
 	binding->dmabuf = dmabuf;
 
-	binding->attachment = dma_buf_attach(binding->dmabuf, dev->dev.parent);
+	binding->attachment = dma_buf_attach(binding->dmabuf, dma_dev);
 	if (IS_ERR(binding->attachment)) {
 		err = PTR_ERR(binding->attachment);
 		NL_SET_ERR_MSG(extack, "Failed to bind dmabuf to device");
Signature · 1 line
2.50.0
#23 lore raw

RFC 4/4 net/mlx5e: Enable HDS zerocopy flows for SFs

In reply to Dragos Tatulea

An SF has an auxiliary device as a parent. This type of device can't be used for zerocopy DMA mapping operations. A PCI device is required.

Use the new netdev dma_dev functionality to expose the actual PCI device to be used for DMA. Always set it to keep things generic.

---
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 3 +++
 1 file changed, 3 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +3 -0
index e8e5b347f9b2..c4e45205fba4 100644
@@ -5841,6 +5841,9 @@ static int mlx5e_nic_init(struct mlx5_core_dev *mdev,
 	/* update XDP supported features */
 	mlx5e_set_xdp_feature(netdev);
 
+	/* Set pci device for dma. Useful for SFs. */
+	netdev->dma_dev = &mdev->pdev->dev;
+
 	if (take_rtnl)
 		rtnl_unlock();
 
Signature · 1 line
2.50.0