mlx5: support SubFunction
Subsystems (5)
Hi Thomas,
Quoted 20 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Thursday, June 10, 2021 9:23 PM
To: Yigit, Ferruh <[email protected]>
Cc: Xueming Li <[email protected]>; Viacheslav Ovsiienko <[email protected]>; [email protected]; Xia, Chenbo <[email protected]>
Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction10/06/2021 12:33, Ferruh Yigit:
Quoted 2 lines · Xueming Li
Quoted 1 line
SubFunction [1] is a portion of the PCI device, a SF netdev has its
own
Quoted 8 lines
Quoted 7 lines
dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.
Quoted 7 lines
Quoted 1 line
There is already an mdev patch, originated from long ago. Aren't
subfunctions
Quoted 1 line
presented as mdev device? If so can't we use mdev for it?
No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.
Just out of curiosity:
Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊
[1] https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
[2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices
11/06/2021 07:14, Xia, Chenbo:
Quoted 22 lines
From: Thomas Monjalon <[email protected]>
Quoted 12 lines
10/06/2021 12:33, Ferruh Yigit:
Quoted 2 lines · Xueming Li
Quoted 1 line
SubFunction [1] is a portion of the PCI device, a SF netdev has its
own
Quoted 8 lines
Quoted 7 lines
dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.
Quoted 6 lines
Quoted 1 line
There is already an mdev patch, originated from long ago. Aren't
subfunctions
Quoted 1 line
presented as mdev device? If so can't we use mdev for it?
No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.Just out of curiosity:
Quoted 7 lines
Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊
[1] https://patchwork.ozlabs.org/project/netdev/cover/[email protected]/
[2] https://docs.mellanox.com/display/BlueFieldSWv25011176/Mediated+Devices
Kernel maintainers rejected the use of mdev for this purpose and suggested to use a real bus.
You can follow the discussion here:
https://lore.kernel.org/netdev/[email protected]/
Does Intel plan to use mdev for SF?
For the sake of follow-up discussion, this is the official mdev doc:
https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt
Hi Thomas,
Quoted 36 lines
-----Original Message-----
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM
To: Yigit, Ferruh <[email protected]>; Xia, Chenbo <[email protected]>
Cc: Xueming Li <[email protected]>; Viacheslav Ovsiienko <[email protected]>; [email protected]; [email protected]; [email protected]
Subject: Re: [dpdk-dev] [RFC 00/14] mlx5: support SubFunction11/06/2021 07:14, Xia, Chenbo:
Quoted 27 lines
From: Thomas Monjalon <[email protected]>
Quoted 12 lines
10/06/2021 12:33, Ferruh Yigit:
Quoted 2 lines · Xueming Li
Quoted 1 line
SubFunction [1] is a portion of the PCI device, a SF netdev has its
own
Quoted 8 lines
Quoted 7 lines
dedicated
queues(txq, rxq). A SF shares PCI level resources with other SFs and/or with its parent PCI function. Auxiliary bus is the fundamental of SF.This patch set introduces SubFunction support for mlx5 PMD driver including class net, regex, vdpa and compress.
Quoted 6 lines
Quoted 1 line
There is already an mdev patch, originated from long ago. Aren't
subfunctions
Quoted 1 line
presented as mdev device? If so can't we use mdev for it?
No unfortunately that's different.
mlx5 SF is based on top of auxiliary bus in the kernel/sysfs.Just out of curiosity:
Does SF use mdev before aux bus is introduced in kernel. I see some history of it but am not sure: [1] seems SF was base on mdev. [2] seems BlueField software v2.5 is using mdev for SF. I saw it yesterday and try to figure out the history. Since you are here, guess you know something 😊
Quoted 8 lines
Kernel maintainers rejected the use of mdev for this purpose and suggested to use a real bus.
You can follow the discussion here:
https://lore.kernel.org/netdev/[email protected]/
OK. Thanks for the info.
Quoted 2 lines
Does Intel plan to use mdev for SF?
Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-scalable-i/o-virtualization-linux)
And vfio-mdev is chosen to be the software framework for it. I start to realize there is difference between SF and ADI: SF considers multi-function devices which may include net/regex/vdpa/... But ADI only focuses on the virtualization of the devices and splitting devices to logic parts and providing huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).
And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub-system (like for net, wrap it as netdev).
Thanks,
Chenbo
Quoted 5 lines
For the sake of follow-up discussion, this is the official mdev doc:
https://www.kernel.org/doc/Documentation/vfio-mediated-device.txt
Hi Chenbo,
Quoted 7 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AMHi Thomas,
Quoted 2 lines
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM
[..]
Quoted 8 lines
Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)And vfio-mdev is chosen to be the software framework for it. I start to realize there is difference between SF and ADI: SF considers multi-function devices which may include
net/regex/vdpa/...
Yes. net, rdma, vdpa, regex ++.
And eventually vfio_device to map to VM too.
Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.
Quoted 6 lines
But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts and providing huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).
SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a vfio_device) to map to a VM.
Quoted 3 lines
And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub- system (like for net, wrap it as netdev).
Correct.
Hi Parav,
Quoted 26 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunctionHi Chenbo,
Quoted 7 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AMHi Thomas,
Quoted 2 lines
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM[..]
Quoted 6 lines
Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV (https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)And vfio-mdev is chosen to be the software framework for it. I start to
Quoted 8 lines
realize
Quoted 2 lines
there is difference between SF and ADI: SF considers multi-function devices which may include
net/regex/vdpa/...Yes. net, rdma, vdpa, regex ++.
And eventuallyvfio_deviceto map to VM too.Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.
OK. Got it.
Quoted 10 lines
Quoted 3 lines
But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts andproviding
Quoted 4 lines
huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).
SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a
vfio_device) to map to a VM.
So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree?
Just FYI:
We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/[email protected]/
Thanks,
Chenbo
Quoted 5 lines
Quoted 4 lines
And I also noticed that the mdev-based interface can only be used in userspace but aux-based interface can also be used by other kernel sub- system (like for net, wrap it as netdev).
Correct.
Quoted 56 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 11:03 AMHi Parav,
Quoted 29 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunctionHi Chenbo,
Quoted 7 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AMHi Thomas,
Quoted 2 lines
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM[..]
Quoted 8 lines
Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV
(https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)And vfio-mdev is chosen to be the software framework for it. I start to
Quoted 8 lines
realize
Quoted 2 lines
there is difference between SF and ADI: SF considers multi-function devices which may include
net/regex/vdpa/...Yes. net, rdma, vdpa, regex ++.
And eventuallyvfio_deviceto map to VM too.Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.
OK. Got it.
Quoted 9 lines
Quoted 4 lines
But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts andproviding
Quoted 4 lines
huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your term? Correct me if I'm wrong).
Quoted 6 lines
Quoted 2 lines
SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a
vfio_device) to map to a VM.So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree?
mlx5 driver doesn't support vfio_device for SFs.
Kernel plumbing for PASID assignment to SF is WIP currently kernel community. We do not have any out-of-tree kernel module.
Quoted 6 lines
Just FYI:
We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
[email protected]/
I am yet to read about it. But I am not sure what value does it add.
A user can open a vfio device using vfio subsystem and operate on it.
A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to vfio_foo driver. There is kernel work in progress to use vfio core as library.
So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF.
For Intel, ADI will never has any netdevs or rdma dev?
Hi Parav,
Quoted 78 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 1:43 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunctionQuoted 37 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 11:03 AMHi Parav,
Quoted 32 lines
-----Original Message-----
From: Parav Pandit <[email protected]>
Sent: Tuesday, June 15, 2021 12:05 PM
To: Xia, Chenbo <[email protected]>; NBU-Contact-Thomas Monjalon <[email protected]>; Yigit, Ferruh <[email protected]>
Cc: Xueming(Steven) Li <[email protected]>; Slava Ovsiienko <[email protected]>; [email protected]; Jason Gunthorpe <[email protected]>
Subject: RE: [dpdk-dev] [RFC 00/14] mlx5: support SubFunctionHi Chenbo,
Quoted 7 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 7:41 AMHi Thomas,
Quoted 2 lines
From: Thomas Monjalon <[email protected]>
Sent: Friday, June 11, 2021 3:54 PM[..]
Quoted 8 lines
Yes. In our term it's called Assignable Device Interface (ADI) introduced in Intel Scalable IOV
(https://01.org/blogs/2019/assignable-interfaces-intel-
scalable-i/o-virtualization-linux)And vfio-mdev is chosen to be the software framework for it. I start to
realize
Quoted 2 lines
there is difference between SF and ADI: SF considers multi-function devices which may include
net/regex/vdpa/...Quoted 15 lines
Quoted 5 lines
Yes. net, rdma, vdpa, regex ++.
And eventuallyvfio_deviceto map to VM too.Non mdev framework is chosen so that all the use cases of kernel only, or user only or mix modes can be supported.
OK. Got it.
Quoted 7 lines
Quoted 4 lines
But ADI only focuses on the
virtualization of the devices and splitting devices to logic parts andproviding
Quoted 2 lines
huge number of interfaces to host APP. I think SF also considers this but is mainly used for multi-function devices (like DPU in your
term?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Correct me if I'm wrong).
SF also supports DPU mode too but it is in addition to above use cases. SF will expose mdev (or a
vfio_device) to map to a VM.So your SW actually supports vfio-mdev? I suppose the device-specific mdev Kernel module is out-of-tree?
mlx5 driver doesn't support
vfio_devicefor SFs.
Kernel plumbing for PASID assignment to SF is WIP currently kernel community. We do not have any out-of-tree kernel module.Quoted 2 lines
Just FYI:
Quoted 8 lines
Quoted 4 lines
We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
[email protected]/I am yet to read about it. But I am not sure what value does it add. A user can open a vfio device using vfio subsystem and operate on it. A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to
vfio_foodriver.
Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev to parent device and echo uuid to create a virtual device. VFIO APP like DPDK, as you said, should work similar with VFIO UAPI for vfio-pci devices or mdev-based devices. But currently DPDK only cares about vfio-pci devices and does not care things for other cases like mdev-based pci devices. For example, it does not scan /sys/bus/mdev and it always uses pci bdf as device address, which mdev-based pci devices do not have. Therefore I sent that patchset.
Quoted 1 line
There is kernel work in progress to use vfio core as library.
OK. Could you share me some link to it? Much appreciated.
Quoted 2 lines
So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF.
OK. For now, we are following the vfio-mdev standard, using UUID to create vfio devices.
Quoted 2 lines
For Intel, ADI will never has any netdevs or rdma dev?
I think technically it could have. But for some devices like our dma devices, it's just using mdev:
https://www.spinics.net/lists/kvm/msg244417.html
Thanks,
Chenbo
Quoted 17 lines
From: Xia, Chenbo <[email protected]>
Sent: Tuesday, June 15, 2021 4:49 PMQuoted 11 lines
Quoted 7 lines
Just FYI:
We are introducing a new mdev bus for DPDK:
http://patchwork.dpdk.org/project/dpdk/cover/20210601030644.3318-1-
[email protected]/I am yet to read about it. But I am not sure what value does it add. A user can open a vfio device using vfio subsystem and operate on it. A vfio device can be a create as a result of binding PCI VF/PF to vfio-pci driver or a SF by binding SF to
vfio_foodriver.
Quoted 8 lines
Yes, in general it is the way. For vfio-mdev, it works as binding the vfio-mdev to parent device and echo uuid to create a virtual device. VFIO APP like DPDK, as you said, should work similar with VFIO UAPI for vfio-pci devices or mdev-based devices. But currently DPDK only cares about vfio-pci devices and does not care things for other cases like mdev-based pci devices. For example, it does not scan
/sys/bus/mdevand it always uses pci bdf as device address, which mdev-based pci devices do not have. Therefore I sent that patchset.
mdev device reside on mdev bus. So dpdk should identify the mdev object by specifying bus type = mdev, and device id = uuid. There should not be any attachment to pci as Thomas said.
Quoted 5 lines
Quoted 2 lines
There is kernel work in progress to use vfio core as library.
OK. Could you share me some link to it? Much appreciated.
[1] https://lore.kernel.org/kvm/[email protected]/
Quoted 6 lines
Quoted 2 lines
So we do not anticipate to use add mdev layer and uuid to create a vfio device for a SF.
OK. For now, we are following the vfio-mdev standard, using UUID to create vfio devices.
If this layer is going to work on top of VFIO devices, does it really care that is it mdev? Can it identify the vfio device by vfio device and its UAPI in uniform way? such as open("/dev/vfio/98" ..);
Quoted 4 lines
Quoted 2 lines
For Intel, ADI will never has any netdevs or rdma dev?
I think technically it could have.
Unlikely. As I explained in previous email that creating netdev, rdma dev using mdev bus was already rejected in my previous patches. And we step forward with auxiliary bus.
Quoted 4 lines
But for some devices like our dma devices,
it's just using mdev:
Possibly yes. Some devices might live on mdev bus. You should wait for kernel patches to be merged as Jason said.
I still think that identifying vfio device by its /dev/vfio/<id> will go long way regardless of its bus type.
Quoted 6 lines · Parav Pandit
Quoted 4 lines
But for some devices like our dma devices,
it's just using mdev:Possibly yes. Some devices might live on mdev bus. You should wait for kernel patches to be merged as Jason said.
Also I would not expect dpdk to consume IDXD via vfio, but instead via the normal multi-process host IOCTL interface it has.
Jason