Skip to contentSkip to conversation

mlx5: support SubFunction

RFC In review Xueming Li 7 participants 108 messages May 27, 2021 – Jul 22, 2021 Page 2 of 2

Subsystems (5)
lore
Share & tools
lore
#99 lore raw
In reply to Thomas Monjalon

Hi Thomas,

Quoted 20 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, June 10, 2021 9:23 PM
To: Yigit, Ferruh <[email protected]>
Cc: Xueming Li <[email protected]>; Viacheslav Ovsiienko <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

10/06/2021 12:33, Ferruh Yigit:

Quoted 2 lines · Xueming Li
Quoted 1 line

SubFunction [1] is a portion of the PCI device, a SF netdev has its

own

Quoted 8 lines
Quoted 7 lines

dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.

This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.

Quoted 7 lines
Quoted 1 line

There is already an mdev patch, originated from long ago. Aren't

subfunctions

Quoted 1 line

presented as mdev device? If so can't we use mdev for it?

No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.

Just out of curiosity:

Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊

[1] https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
[2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices

#100 lore raw

11/06/2021 07:14, Xia, Chenbo:

Quoted 22 lines

From: Thomas Monjalon <[email protected]>

Quoted 12 lines

10/06/2021 12:33, Ferruh Yigit:

Quoted 2 lines · Xueming Li
Quoted 1 line

SubFunction [1] is a portion of the PCI device, a SF netdev has its

own

Quoted 8 lines
Quoted 7 lines

dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.

This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.

Quoted 6 lines
Quoted 1 line

There is already an mdev patch, originated from long ago. Aren't

subfunctions

Quoted 1 line

presented as mdev device? If so can't we use mdev for it?

No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.

Just out of curiosity:

Quoted 7 lines

Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊

[1] https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
[2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices

Kernel maintainers rejected the use of mdev for this purpose and suggested to use a real bus.
You can follow the discussion here:
https://lore.kernel.org/netdev/[email protected]/

Does Intel plan to use mdev for SF?

For the sake of follow-up discussion, this is the official mdev doc:
https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt

#101 lore raw

Hi Thomas,

Quoted 36 lines

-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM
To: Yigit, Ferruh <[email protected]>; Xia, Chenbo <[email protected]>
Cc: Xueming Li <[email protected]>; Viacheslav Ovsiienko <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

11/06/2021 07:14, Xia, Chenbo:

Quoted 27 lines

From: Thomas Monjalon <[email protected]>

Quoted 12 lines

10/06/2021 12:33, Ferruh Yigit:

Quoted 2 lines · Xueming Li
Quoted 1 line

SubFunction [1] is a portion of the PCI device, a SF netdev has its

own

Quoted 8 lines
Quoted 7 lines

dedicated queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.

This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.

Quoted 6 lines
Quoted 1 line

There is already an mdev patch, originated from long ago. Aren't

subfunctions

Quoted 1 line

presented as mdev device? If so can't we use mdev for it?

No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.

Just out of curiosity:

Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊

Quoted 8 lines
Quoted 1 line

[1] https://patchwork.ozlabs.org/project/netdev/cover/20191107160448.20962-

[email protected]/

Quoted 1 line

[2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices

Kernel maintainers rejected the use of mdev for this purpose and suggested to use a real bus.
You can follow the discussion here:
https://lore.kernel.org/netdev/[email protected]/

OK. Thanks for the info.

Quoted 2 lines

Does Intel plan to use mdev for SF?

Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start to realize there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/... But ADI only focuses on the virtualization of the devices and splitting devices to logic parts and providing huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).

And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub-system (like for net, wrap it as netdev).

Thanks,
Chenbo

Quoted 5 lines

For the sake of follow-up discussion, this is the official mdev doc:
https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt

#102 lore raw

Hi Chenbo,

Quoted 7 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AM

Hi Thomas,

Quoted 2 lines

From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM

[..]

Quoted 8 lines

Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start to realize there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/...

Yes. net, rdma, vdpa, regex ++.
And eventually vfio_device to map to VM too.

Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.

Quoted 6 lines

But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts and providing huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).

SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a vfio_device) to map to a VM.

Quoted 3 lines

And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub- system (like for net, wrap it as netdev).

Correct.

#103 lore raw

Hi Parav,

Quoted 26 lines

-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

Hi Chenbo,

Quoted 7 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AM

Hi Thomas,

Quoted 2 lines

From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM

[..]

Quoted 6 lines

Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start to

Quoted 8 lines

realize

Quoted 2 lines

there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/...

Yes. net, rdma, vdpa, regex ++.
And eventually vfio_device to map to VM too.

Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.

OK. Got it.

Quoted 10 lines
Quoted 3 lines

But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts and

providing

Quoted 4 lines

huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).

SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a vfio_device) to map to a VM.

So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree?

Just FYI:

We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/[email protected]/

Thanks,
Chenbo

Quoted 5 lines
Quoted 4 lines

And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub- system (like for net, wrap it as netdev).

Correct.

#104 lore raw
Quoted 56 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 11:03 AM

Hi Parav,

Quoted 29 lines

-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

Hi Chenbo,

Quoted 7 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AM

Hi Thomas,

Quoted 2 lines

From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM

[..]

Quoted 8 lines

Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV
(https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start to

Quoted 8 lines

realize

Quoted 2 lines

there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/...

Yes. net, rdma, vdpa, regex ++.
And eventually vfio_device to map to VM too.

Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.

OK. Got it.

Quoted 9 lines
Quoted 4 lines

But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts and

providing

Quoted 4 lines

huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).

Quoted 6 lines
Quoted 2 lines

SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a vfio_device) to map to a VM.

So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree?

mlx5 driver doesn't support vfio_device for SFs.
Kernel plumbing for PASID assignment to SF is WIP currently kernel community. We do not have any out-of-tree kernel module.

Quoted 6 lines

Just FYI:

We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
[email protected]/

I am yet to read about it. But I am not sure what value does it add.
A user can open a vfio device using vfio subsystem and operate on it.
A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to vfio_foo driver. There is kernel work in progress to use vfio core as library.
So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF.

For Intel, ADI will never has any netdevs or rdma dev?

#105 lore raw

Hi Parav,

Quoted 78 lines

-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 1:43 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

Quoted 37 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 11:03 AM

Hi Parav,

Quoted 32 lines

-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction

Hi Chenbo,

Quoted 7 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AM

Hi Thomas,

Quoted 2 lines

From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM

[..]

Quoted 8 lines

Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV
(https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)

And vfio-mdev is chosen to be the software framework for it. I start to

realize

Quoted 2 lines

there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/...

Quoted 15 lines
Quoted 5 lines

Yes. net, rdma, vdpa, regex ++.
And eventually vfio_device to map to VM too.

Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.

OK. Got it.

Quoted 7 lines
Quoted 4 lines

But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts and

providing

Quoted 2 lines

huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your

term?

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

Correct me if I'm wrong).

SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a vfio_device) to map to a VM.

So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree?

mlx5 driver doesn't support vfio_device for SFs.
Kernel plumbing for PASID assignment to SF is WIP currently kernel community. We do not have any out-of-tree kernel module.

Quoted 2 lines

Just FYI:

Quoted 8 lines
Quoted 4 lines

We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
[email protected]/

I am yet to read about it. But I am not sure what value does it add. A user can open a vfio device using vfio subsystem and operate on it. A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to vfio_foo driver.

Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev to parent device and echo uuid to create a virtual device. VFIO APP like DPDK, as you said, should work similar with VFIO UAPI for vfio-pci devices or mdev-based devices. But currently DPDK only cares about vfio-pci devices and does not care things for other cases like mdev-based pci devices. For example, it does not scan /sys/bus/mdev and it always uses pci bdf as device address, which mdev-based pci devices do not have. Therefore I sent that patchset.

Quoted 1 line

There is kernel work in progress to use vfio core as library.

OK. Could you share me some link to it? Much appreciated.

Quoted 2 lines

So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF.

OK. For now, we are following the vfio-mdev standard, using UUID to create vfio devices.

Quoted 2 lines

For Intel, ADI will never has any netdevs or rdma dev?

I think technically it could have. But for some devices like our dma devices, it's just using mdev:

https://www.spinics.net/lists/kvm/msg244417.html

Thanks,
Chenbo

#106 lore raw
Quoted 17 lines

From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 4:49 PM

Quoted 11 lines
Quoted 7 lines

Just FYI:

We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
[email protected]/

I am yet to read about it. But I am not sure what value does it add. A user can open a vfio device using vfio subsystem and operate on it. A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to vfio_foo driver.

Quoted 8 lines

Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev to parent device and echo uuid to create a virtual device. VFIO APP like DPDK, as you said, should work similar with VFIO UAPI for vfio-pci devices or mdev-based devices. But currently DPDK only cares about vfio-pci devices and does not care things for other cases like mdev-based pci devices. For example, it does not scan /sys/bus/mdev and it always uses pci bdf as device address, which mdev-based pci devices do not have. Therefore I sent that patchset.

mdev device reside on mdev bus. So dpdk should identify the mdev object by specifying bus type = mdev, and device id = uuid. There should not be any attachment to pci as Thomas said.

Quoted 5 lines
Quoted 2 lines

There is kernel work in progress to use vfio core as library.

OK. Could you share me some link to it? Much appreciated.

[1] https://lore.kernel.org/kvm/[email protected]/

Quoted 6 lines
Quoted 2 lines

So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF.

OK. For now, we are following the vfio-mdev standard, using UUID to create vfio devices.

If this layer is going to work on top of VFIO devices, does it really care that is it mdev? Can it identify the vfio device by vfio device and its UAPI in uniform way? such as open("/dev/vfio/98" ..);

Quoted 4 lines
Quoted 2 lines

For Intel, ADI will never has any netdevs or rdma dev?

I think technically it could have.

Unlikely. As I explained in previous email that creating netdev, rdma dev using mdev bus was already rejected in my previous patches. And we step forward with auxiliary bus.

Quoted 4 lines

But for some devices like our dma devices,
it's just using mdev:

https://www.spinics.net/lists/kvm/msg244417.html

Possibly yes. Some devices might live on mdev bus. You should wait for kernel patches to be merged as Jason said.

I still think that identifying vfio device by its /dev/vfio/<id> will go long way regardless of its bus type.

#107 lore raw
Quoted 6 lines · Parav Pandit
Quoted 4 lines

But for some devices like our dma devices,
it's just using mdev:

https://www.spinics.net/lists/kvm/msg244417.html

Possibly yes. Some devices might live on mdev bus. You should wait for kernel patches to be merged as Jason said.

Also I would not expect dpdk to consume IDXD via vfio, but instead via the normal multi-process host IOCTL interface it has.

Jason