Dynamically assign MSI-X vectors count
Subsystems (6)
v8 cover Dynamically assign MSI-X vectors count
---------------------------------------------------------------------------------
Changelog
v8:
* Added "physical/virtual function" words near PF and VF acronyms.
v7: https://lore.kernel.org/linux-pci/[email protected]
* Rebase on top v5.12-rc1
* More english fixes
* Returned to static sysfs creation model as was implemented in v0/v1.
v6: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* English fixes
* Moved pci_vf_set_msix_vec_count() from msi.c to iov.c
* Embedded pci_vf_set_msix_vec_count() into sriov_vf_msix_count_store
* Deleted sriov_vf_msix_count_show
* Deleted vfs_overlay folder
* Renamed functions *_vfs_overlay_* to be *_vf_overlay_*
* Deleted is_supported and attribute_group because it confused people more than it gave advantage.
* Changed vf_total_msix to be callback
* Patch 3:
* Fixed english as suggested by Bjorn
* Added more explanations to the commit message
* Patch 4:
* Protected enable/disable with capability check
v5: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Added forgotten "inline" keyword when declaring empty functions.
v4: https://lore.kernel.org/linux-pci/[email protected]
* Used sysfs_emit() instead of sprintf() in new sysfs entries.
* Changed EXPORT_SYMBOL to be EXPORT_SYMBOL_GPL for pci_iov_virtfn_devfn().
* Rewrote sysfs registration code to be driven by PF that wants to enable VF overlay instead of creating to all SR-IOV devices.
* Grouped all such functionality under new "vfs_overlay" folder.
* Combined two PCI patches into one.
v3: https://lore.kernel.org/linux-pci/[email protected]
* Renamed pci_set_msix_vec_count to be pci_vf_set_msix_vec_count.
* Added VF msix_cap check to hide sysfs entry if device doesn't support msix.
* Changed "-" to be ":" in the mlx5 patch to silence CI warnings about missing kdoc description.
* Split differently error print in mlx5 driver to avoid checkpatch warning.
v2: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Renamed vf_msix_vec sysfs knob to be sriov_vf_msix_count
* Added PF and VF device locks during set MSI-X call to protect from parallel driver bind/unbind operations.
* Removed extra checks when reading sriov_vf_msix, because users will be able to distinguish between supported/not supported by looking on sriov_vf_total_msix count.
* Changed all occurrences of "numb" to be "count"
* Changed returned error from EOPNOTSUPP to be EBUSY if user tries to set MSI-X count after driver already bound to the VF.
* Added extra comment in pci_set_msix_vec_count() to emphasize that driver should not be bound.
* Patch 2:
* Changed vf_total_msix from int to be u32 and updated function signatures accordingly.
* Improved patch title
v1: https://lore.kernel.org/linux-pci/[email protected]
* Improved wording and commit messages of first PCI patch
* Added extra PCI patch to provide total number of MSI-X vectors
* Prohibited read of vf_msix_vec sysfs file if driver doesn't support write
* Removed extra function definition in pci.h
v0: https://lore.kernel.org/linux-pci/[email protected]
--------------------------------------------------------------------
Hi,
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Before this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-
Configuration script:
1. Start fresh
echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs modprobe -q -r mlx5_ib mlx5_core
2. Ensure that driver doesn't run and it is safe to change MSI-X echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe
3. Load driver for the PF
modprobe mlx5_core
4. Configure one of the VFs with new number
echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/sriov_vf_msix_count
After this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=21 Masked-
Thanks
Leon Romanovsky (4):
PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs net/mlx5: Add dynamic MSI-X capabilities bits
net/mlx5: Dynamically assign MSI-X vectors count net/mlx5: Implement sriov_get_vf_total_msix/count() callbacks
Documentation/ABI/testing/sysfs-bus-pci | 29 +++++ .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 12 +++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 73 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 48 ++++++++- drivers/pci/iov.c | 102 ++++++++++++++++-- drivers/pci/pci-sysfs.c | 3 +- drivers/pci/pci.h | 3 +- include/linux/mlx5/mlx5_ifc.h | 11 +- include/linux/pci.h | 8 ++ 10 files changed, 284 insertions(+), 11 deletions(-)
--
2.30.2
PATCH v8 1/4 PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a customer requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way to change the number of MSI-X vectors supported by a VF.
Some Mellanox ConnectX devices support dynamic assignment of MSI-X vectors to SR-IOV VFs. This can be done by the PF driver after VFs are enabled, and it can be done without affecting VFs that are already in use. The hardware supports a limited pool of MSI-X vectors that can be assigned to the PF or to individual VFs. This is device-specific behavior that requires support in the PF driver.
Add a read-only "sriov_vf_total_msix" sysfs file for the PF and a writable "sriov_vf_msix_count" file for each VF. Management software may use these to learn how many MSI-X vectors are available and to dynamically assign them to VFs before the VFs are passed through to a VM.
If the PF driver implements the ->sriov_get_vf_total_msix() callback, "sriov_vf_total_msix" contains the total number of MSI-X vectors available for distribution among VFs.
If no driver is bound to the VF, writing "N" to "sriov_vf_msix_count" uses the PF driver ->sriov_set_msix_vec_count() callback to assign "N" MSI-X vectors to the VF. When a VF driver subsequently reads the MSI-X Message Control register, it will see the new Table Size "N".
--- Documentation/ABI/testing/sysfs-bus-pci | 29 +++++++ drivers/pci/iov.c | 102 ++++++++++++++++++++++-- drivers/pci/pci-sysfs.c | 3 +- drivers/pci/pci.h | 3 +- include/linux/pci.h | 8 ++ 5 files changed, 137 insertions(+), 8 deletions(-)
Documentation/ABI/testing/sysfs-bus-pci +29 -0
@@ -375,3 +375,32 @@ Description: The value comes from the PCI kernel device state and can be one of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold". The file is read only. + +What: /sys/bus/pci/devices/.../sriov_vf_total_msix +Date: January 2021 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with a SR-IOV physical function (PF). + It contains the total number of MSI-X vectors available for + assignment to all virtual functions (VFs) associated with PF. + The value will be zero if the device doesn't support this + functionality. For supported devices, the value will be + constant and won't be changed after MSI-X vectors assignment. + +What: /sys/bus/pci/devices/.../sriov_vf_msix_count +Date: January 2021 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with a SR-IOV virtual function (VF). + It allows configuration of the number of MSI-X vectors for + the VF. This allows devices that have a global pool of MSI-X + vectors to optimally divide them between VFs based on VF usage. + + The values accepted are: + * > 0 - this number will be reported as the Table Size in the + VF's MSI-X capability + * < 0 - not valid + * = 0 - will reset to the device default value + + The file is writable if the PF is bound to a driver that + implements ->sriov_set_msix_vec_count().
drivers/pci/iov.c +96 -6
135 lines not on this page · raw message
drivers/pci/pci-sysfs.c +2 -1
@@ -1567,7 +1567,8 @@ static const struct attribute_group *pci_dev_attr_groups[] = { &pci_dev_attr_group, &pci_dev_hp_attr_group, #ifdef CONFIG_PCI_IOV - &sriov_dev_attr_group, + &sriov_pf_dev_attr_group, + &sriov_vf_dev_attr_group, #endif &pci_bridge_attr_group, &pcie_dev_attr_group,
drivers/pci/pci.h +2 -1
@@ -501,7 +501,8 @@ void pci_iov_update_resource(struct pci_dev *dev, int resno);resource_size_t pci_sriov_resource_alignment (struct pci_dev *dev,int resno);void pci_restore_iov_state (struct pci_dev *dev);int pci_iov_bus_range (struct pci_bus *bus); -extern const struct attribute_group sriov_dev_attr_group; +extern const struct attribute_group sriov_pf_dev_attr_group; +extern const struct attribute_group sriov_vf_dev_attr_group;#else static inline int pci_iov_init (struct pci_dev *dev) {
include/linux/pci.h +8 -0
@@ -856,6 +856,12 @@ struct module;* e.g. drivers/net/e100.c. * @sriov_configure: Optional driver callback to allow configuration of * number of VFs to enable via sysfs "sriov_numvfs" file. +* @sriov_set_msix_vec_count: PF Driver callback to change number of MSI-X +* vectors on a VF. Triggered via sysfs "sriov_vf_msix_count". +* This will change MSI-X Table Size in the VF Message Control +* registers. +* @sriov_get_vf_total_msix: PF driver callback to get the total number of +* MSI-X vectors available for distribution to the VFs. * @err_handler: See Documentation/PCI/pci-error-recovery.rst * @groups: Sysfs attribute groups. * @driver: Driver model structure.
@@ -871,6 +877,8 @@ struct pci_driver {int (*resume)(struct pci_dev *dev);/* Device woken up */ void (*shutdown)(struct pci_dev *dev);int (*sriov_configure)(struct pci_dev *dev,int num_vfs);/* On PF */ +int (*sriov_set_msix_vec_count)(struct pci_dev *vf,int msix_vec_count);/* On PF */ +u32 (*sriov_get_vf_total_msix)(struct pci_dev *pf);const struct pci_error_handlers *err_handler;const struct attribute_group **groups;struct device_driver driver;
--
2.30.2
Re: [PATCH mlx5-next v8 1/4] PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs
Possible subject, since this adds *two* files, not just "a file":
PCI/IOV: Add sysfs MSI-X vector assignment interface
Quoted 57 lines · Leon Romanovsky
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a customer requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way to change the number of MSI-X vectors supported by a VF.
...drivers/pci/iov.c +50
-0+#ifdef CONFIG_PCI_MSI +static ssize_t sriov_vf_msix_count_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct pci_dev *vf_dev =to_pci_dev (dev); +struct pci_dev *pdev =pci_physfn (vf_dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + +if (val <0 ) +return -EINVAL ; + +device_lock (&pdev->dev); +if (!pdev->driver || !pdev->driver->sriov_set_msix_vec_count) { + ret = -EOPNOTSUPP ; + goto err_pdev; + } + +device_lock (&vf_dev->dev); +if (vf_dev->driver) { +/* +* A driver is already attached to this VF and has configured +* itself based on the current MSI-X vector count. Changing +* the vector size could mess up the driver, so block it. +*/ + ret = -EBUSY ; + goto err_dev; + } + + ret = pdev->driver->sriov_set_msix_vec_count (vf_dev, val); + +err_dev: +device_unlock (&vf_dev->dev); +err_pdev: +device_unlock (&pdev->dev); +return ret ? : count; +} +static DEVICE_ATTR_WO (sriov_vf_msix_count); + +static ssize_t sriov_vf_total_msix_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +u32 vf_total_msix =0 ; +
drivers/pci/iov.c int pci_iov_sysfs_link(struct pci_dev *dev, · line 220
Quoted 10 lines
drivers/pci/iov.c +10
-0+device_lock (dev); +if (!pdev->driver || !pdev->driver->sriov_get_vf_total_msix) + goto unlock; + + vf_total_msix = pdev->driver->sriov_get_vf_total_msix (pdev); +unlock: +device_unlock (dev); +return sysfs_emit (buf,"%u\n" , vf_total_msix); +} +static DEVICE_ATTR_RO (sriov_vf_total_msix);
Can you reverse the order of sriov_vf_total_msix_show() and sriov_vf_msix_count_store()? Currently we have:
VF stuff (msix_count_store)
PF stuff (total_msix)
more VF stuff related to the above (vf_dev_attrs, are_visible)
so the total_msix bit is mixed in the middle.
Quoted 30 lines
drivers/pci/iov.c +26
-0+#endif + +static struct attribute *sriov_vf_dev_attrs[] = { +#ifdef CONFIG_PCI_MSI + &dev_attr_sriov_vf_msix_count.attr, +#endif +NULL , +}; + +static umode_t sriov_vf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct device *dev =kobj_to_dev (kobj); +struct pci_dev *pdev =to_pci_dev (dev); + +if (!pdev->is_virtfn) +return 0 ; + +return a->mode; +} + +const struct attribute_group sriov_vf_dev_attr_group = { + .attrs = sriov_vf_dev_attrs, + .is_visible = sriov_vf_attrs_are_visible, +}; +int pci_iov_add_virtfn (struct pci_dev *dev,int id) {int i;
drivers/pci/iov.c static DEVICE_ATTR_RO(sriov_stride); · line 490
Quoted 5 lines
drivers/pci/iov.c +1
-1@@ -400,18 +487,21 @@ static DEVICE_ATTR_RO(sriov_stride);static DEVICE_ATTR_RO (sriov_vf_device);static DEVICE_ATTR_RW (sriov_drivers_autoprobe); -static struct attribute *sriov_dev_attrs[] = { +static struct attribute *sriov_pf_dev_attrs[] = {
This and the related sriov_pf_attrs_are_visible change below are nice. Would you mind splitting them to a preliminary patch, since they really aren't related to the concept of *this* patch?
Quoted 30 lines
drivers/pci/iov.c +8
-5&dev_attr_sriov_totalvfs.attr, &dev_attr_sriov_numvfs.attr, &dev_attr_sriov_offset.attr, &dev_attr_sriov_stride.attr, &dev_attr_sriov_vf_device.attr, &dev_attr_sriov_drivers_autoprobe.attr, +#ifdef CONFIG_PCI_MSI + &dev_attr_sriov_vf_total_msix.attr, +#endif NULL, }; -static umode_t sriov_attrs_are_visible (struct kobject *kobj, -struct attribute *a,int n) +static umode_t sriov_pf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) {struct device *dev =kobj_to_dev (kobj);@@ -421,9 +511,9 @@ static umode_t sriov_attrs_are_visible(struct kobject *kobj,return a->mode; } -const struct attribute_group sriov_dev_attr_group = { - .attrs = sriov_dev_attrs, - .is_visible = sriov_attrs_are_visible, +const struct attribute_group sriov_pf_dev_attr_group = { + .attrs = sriov_pf_dev_attrs, + .is_visible = sriov_pf_attrs_are_visible, };
Re: [PATCH mlx5-next v8 1/4] PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs
Quoted 3 lines · Bjorn Helgaas
Possible subject, since this adds *two* files, not just "a file":
PCI/IOV: Add sysfs MSI-X vector assignment interface
Sure
Quoted 73 lines · Leon Romanovsky
Quoted 57 lines
A typical cloud provider SR-IOV use case is to create many VFs for use by guest VMs. The VFs may not be assigned to a VM until a customer requests a VM of a certain size, e.g., number of CPUs. A VF may need MSI-X vectors proportional to the number of CPUs in the VM, but there is no standard way to change the number of MSI-X vectors supported by a VF.
...drivers/pci/iov.c +50
-0+#ifdef CONFIG_PCI_MSI +static ssize_t sriov_vf_msix_count_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct pci_dev *vf_dev =to_pci_dev (dev); +struct pci_dev *pdev =pci_physfn (vf_dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + +if (val <0 ) +return -EINVAL ; + +device_lock (&pdev->dev); +if (!pdev->driver || !pdev->driver->sriov_set_msix_vec_count) { + ret = -EOPNOTSUPP ; + goto err_pdev; + } + +device_lock (&vf_dev->dev); +if (vf_dev->driver) { +/* +* A driver is already attached to this VF and has configured +* itself based on the current MSI-X vector count. Changing +* the vector size could mess up the driver, so block it. +*/ + ret = -EBUSY ; + goto err_dev; + } + + ret = pdev->driver->sriov_set_msix_vec_count (vf_dev, val); + +err_dev: +device_unlock (&vf_dev->dev); +err_pdev: +device_unlock (&pdev->dev); +return ret ? : count; +} +static DEVICE_ATTR_WO (sriov_vf_msix_count); + +static ssize_t sriov_vf_total_msix_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +u32 vf_total_msix =0 ; +drivers/pci/iov.c int pci_iov_sysfs_link(struct pci_dev *dev, · line 220
Quoted 10 lines
drivers/pci/iov.c +10
-0+device_lock (dev); +if (!pdev->driver || !pdev->driver->sriov_get_vf_total_msix) + goto unlock; + + vf_total_msix = pdev->driver->sriov_get_vf_total_msix (pdev); +unlock: +device_unlock (dev); +return sysfs_emit (buf,"%u\n" , vf_total_msix); +} +static DEVICE_ATTR_RO (sriov_vf_total_msix);Can you reverse the order of
sriov_vf_total_msix_show()andsriov_vf_msix_count_store()? Currently we have:
Quoted 5 lines
VF stuff (
msix_count_store)
PF stuff (total_msix)
more VF stuff related to the above (vf_dev_attrs,are_visible)so the
total_msixbit is mixed in the middle.
No problem, I'll do.
Quoted 34 lines
Quoted 34 lines
drivers/pci/iov.c +26
-0+#endif + +static struct attribute *sriov_vf_dev_attrs[] = { +#ifdef CONFIG_PCI_MSI + &dev_attr_sriov_vf_msix_count.attr, +#endif +NULL , +}; + +static umode_t sriov_vf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct device *dev =kobj_to_dev (kobj); +struct pci_dev *pdev =to_pci_dev (dev); + +if (!pdev->is_virtfn) +return 0 ; + +return a->mode; +} + +const struct attribute_group sriov_vf_dev_attr_group = { + .attrs = sriov_vf_dev_attrs, + .is_visible = sriov_vf_attrs_are_visible, +}; +int pci_iov_add_virtfn (struct pci_dev *dev,int id) {int i;@@ -400,18 +487,21 @@ static DEVICE_ATTR_RO(sriov_stride);static DEVICE_ATTR_RO (sriov_vf_device);static DEVICE_ATTR_RW (sriov_drivers_autoprobe);
Quoted 6 lines
drivers/pci/iov.c static DEVICE_ATTR_RO(sriov_stride); · line 490
Quoted 2 lines
drivers/pci/iov.c +1
-1-static struct attribute *sriov_dev_attrs[] = { +static struct attribute *sriov_pf_dev_attrs[] = {This and the related
sriov_pf_attrs_are_visiblechange below are nice. Would you mind splitting them to a preliminary patch, since they really aren't related to the concept of *this* patch?
I don't think so, that prepatch will have only two lines of renames from sriov_dev_attrs to be sriov_pf_dev_attrs. It is not worth the hassle.
Thanks
Quoted 31 lines
Quoted 31 lines
drivers/pci/iov.c +8
-5&dev_attr_sriov_totalvfs.attr, &dev_attr_sriov_numvfs.attr, &dev_attr_sriov_offset.attr, &dev_attr_sriov_stride.attr, &dev_attr_sriov_vf_device.attr, &dev_attr_sriov_drivers_autoprobe.attr, +#ifdef CONFIG_PCI_MSI + &dev_attr_sriov_vf_total_msix.attr, +#endif NULL, }; -static umode_t sriov_attrs_are_visible (struct kobject *kobj, -struct attribute *a,int n) +static umode_t sriov_pf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) {struct device *dev =kobj_to_dev (kobj);@@ -421,9 +511,9 @@ static umode_t sriov_attrs_are_visible(struct kobject *kobj,return a->mode; } -const struct attribute_group sriov_dev_attr_group = { - .attrs = sriov_dev_attrs, - .is_visible = sriov_attrs_are_visible, +const struct attribute_group sriov_pf_dev_attr_group = { + .attrs = sriov_pf_dev_attrs, + .is_visible = sriov_pf_attrs_are_visible, };
PATCH v8 2/4 net/mlx5: Add dynamic MSI-X capabilities bits
In reply to Leon RomanovskyThese new fields declare the number of MSI-X vectors that is possible to allocate on the VF through PF configuration.
Value must be in range defined by min_dynamic_vf_msix_table_size and max_dynamic_vf_msix_table_size.
The driver should continue to query its MSI-X table through PCI configuration header.
--- include/linux/mlx5/mlx5_ifc.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
include/linux/mlx5/mlx5_ifc.h +10 -1
@@ -1680,7 +1680,16 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 reserved_at_6e0[0x10 ];u8 sf_base_id[0x10 ]; -u8 reserved_at_700[0x80 ]; +u8 reserved_at_700[0x8 ]; +u8 num_total_dynamic_vf_msix[0x18 ]; +u8 reserved_at_720[0x14 ]; +u8 dynamic_msix_table_size[0xc ]; +u8 reserved_at_740[0xc ]; +u8 min_dynamic_vf_msix_table_size[0x4 ]; +u8 reserved_at_750[0x4 ]; +u8 max_dynamic_vf_msix_table_size[0xc ]; + +u8 reserved_at_760[0x20 ];u8 vhca_tunnel_commands[0x40 ];u8 reserved_at_7c0[0x40 ]; };
--
2.30.2
PATCH v8 3/4 net/mlx5: Dynamically assign MSI-X vectors count
In reply to Leon RomanovskyThe number of MSI-X vectors is a PCI property visible through lspci. The field is read-only and configured by the device. The mlx5 devices work in a static or dynamic assignment mode.
Static assignment means that all newly created VFs have a preset number of MSI-X vectors determined by device configuration parameters. This can result in some VFs having too many or too few MSI-X vectors. Till now this has been the only means of fine-tuning the MSI-X vector count and it was acceptable for small numbers of VFs.
With dynamic assignment the inefficiency of having a fixed number of MSI-X vectors can be avoided with each VF having exactly the required vectors. Userspace will provide this information while provisioning the VF for use, based on the intended use. For instance if being used with a VM, the MSI-X vector count might be matched to the CPU count of the VM.
For compatibility mlx5 continues to start up with MSI-X vector assignment, but the kernel can now access a larger dynamic vector pool and assign more vectors to created VFs.
--- .../net/ethernet/mellanox/mlx5/core/main.c | 4 + .../ethernet/mellanox/mlx5/core/mlx5_core.h | 5 ++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 73 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 13 +++- 4 files changed, 93 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +4 -0
@@ -571,6 +571,10 @@ static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)mlx5_vhca_state_cap_handle (dev, set_hca_cap); +if (MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix)) +MLX5_SET (cmd_hca_cap, set_hca_cap, num_total_dynamic_vf_msix, +MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix)); +return set_caps (dev, set_ctx,MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); }
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +5 -0
@@ -174,6 +174,11 @@ int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx, struct notifier_block *nb);int mlx5_irq_detach_nb (struct mlx5_irq_table *irq_table,int vecidx,struct notifier_block *nb); + +int mlx5_set_msix_vec_count (struct mlx5_core_dev *dev,int devfn, +int msix_vec_count); +int mlx5_get_default_msix_vec_count (struct mlx5_core_dev *dev,int num_vfs); +struct cpumask *mlx5_irq_get_affinity_mask (struct mlx5_irq_table *irq_table,int vecidx);struct cpu_rmap *mlx5_irq_get_rmap (struct mlx5_irq_table *table);
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +73 -0
79 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +11 -2
@@ -71,8 +71,7 @@ static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf)static int mlx5_device_enable_sriov (struct mlx5_core_dev *dev,int num_vfs) {struct mlx5_core_sriov *sriov = &dev->priv.sriov; -int err; -int vf; +int err, vf, num_msix_count;if (!MLX5_ESWITCH_MANAGER (dev)) goto enable_vfs_hca;
@@ -85,12 +84,22 @@ static int mlx5_device_enable_sriov(struct mlx5_core_dev *dev, int num_vfs) } enable_vfs_hca: + num_msix_count =mlx5_get_default_msix_vec_count (dev, num_vfs);for (vf =0 ; vf < num_vfs; vf++) { err =mlx5_core_enable_hca (dev, vf +1 );if (err) {mlx5_core_warn (dev,"failed to enable VF %d (%d)\n" , vf, err);continue ; } + + err =mlx5_set_msix_vec_count (dev, vf +1 , num_msix_count); +if (err) { +mlx5_core_warn (dev, +"failed to set MSI-X vector counts VF %d, err %d\n" , + vf, err); +continue ; + } + sriov->vfs_ctx[vf].enabled =1 ;if (MLX5_CAP_GEN (dev, port_type) ==MLX5_CAP_PORT_TYPE_IB ) { err =sriov_restore_guids (dev, vf);
--
2.30.2
PATCH v8 4/4 net/mlx5: Implement sriov_get_vf_total_msix/count() callbacks
In reply to Leon RomanovskyThe mlx5 implementation executes a firmware command on the PF to change the configuration of the selected VF.
--- .../net/ethernet/mellanox/mlx5/core/main.c | 2 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 7 ++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 35 +++++++++++++++++++ 3 files changed, 44 insertions(+)
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -0
@@ -1700,6 +1700,8 @@ static struct pci_driver mlx5_core_driver = { .shutdown = shutdown, .err_handler = &mlx5_err_handler, .sriov_configure = mlx5_core_sriov_configure, + .sriov_get_vf_total_msix = mlx5_sriov_get_vf_total_msix, + .sriov_set_msix_vec_count = mlx5_core_sriov_set_msix_vec_count, };static void mlx5_core_verify_params (void )
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +7 -0
@@ -140,6 +140,7 @@ void mlx5_sriov_cleanup(struct mlx5_core_dev *dev);int mlx5_sriov_attach (struct mlx5_core_dev *dev);void mlx5_sriov_detach (struct mlx5_core_dev *dev);int mlx5_core_sriov_configure (struct pci_dev *dev,int num_vfs); +int mlx5_core_sriov_set_msix_vec_count (struct pci_dev *vf,int msix_vec_count);int mlx5_core_enable_hca (struct mlx5_core_dev *dev,u16 func_id);int mlx5_core_disable_hca (struct mlx5_core_dev *dev,u16 func_id); int mlx5_create_scheduling_element_cmd(struct mlx5_core_dev *dev, u8 hierarchy,
@@ -278,4 +279,10 @@ int mlx5_load_one(struct mlx5_core_dev *dev, bool boot);int mlx5_vport_get_other_func_cap (struct mlx5_core_dev *dev,u16 function_id,void *out);void mlx5_events_work_enqueue (struct mlx5_core_dev *dev,struct work_struct *work); +static inline u32 mlx5_sriov_get_vf_total_msix (struct pci_dev *pdev) +{ +struct mlx5_core_dev *dev =pci_get_drvdata (pdev); + +return MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix); +}#endif /* __MLX5_CORE_H__ */
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +35 -0
@@ -187,6 +187,41 @@ int mlx5_core_sriov_configure(struct pci_dev *pdev, int num_vfs)return err ? err : num_vfs; } +int mlx5_core_sriov_set_msix_vec_count (struct pci_dev *vf,int msix_vec_count) +{ +struct pci_dev *pf =pci_physfn (vf); +struct mlx5_core_sriov *sriov; +struct mlx5_core_dev *dev; +int num_vf_msix, id; + + dev =pci_get_drvdata (pf); + num_vf_msix =MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix); +if (!num_vf_msix) +return -EOPNOTSUPP ; + +if (!msix_vec_count) + msix_vec_count = +mlx5_get_default_msix_vec_count (dev,pci_num_vf (pf)); + + sriov = &dev->priv.sriov; + +/* Reversed translation of PCI VF function number to the internal +* function_id, which exists in the name of virtfn symlink. +*/ +for (id =0 ; id <pci_num_vf (pf); id++) { +if (!sriov->vfs_ctx[id].enabled) +continue ; + +if (vf->devfn ==pci_iov_virtfn_devfn (pf, id)) +break ; + } + +if (id ==pci_num_vf (pf) || !sriov->vfs_ctx[id].enabled) +return -EINVAL ; + +return mlx5_set_msix_vec_count (dev, id +1 , msix_vec_count); +} +int mlx5_sriov_attach (struct mlx5_core_dev *dev) {if (!mlx5_core_is_pf (dev) || !pci_num_vf (dev->pdev))
--
2.30.2
Gentle reminder
Thanks
Quoted 128 lines · Leon Romanovsky
---------------------------------------------------------------------------------
Changelog
v8:
* Added "physical/virtual function" words near PF and VF acronyms.
v7: https://lore.kernel.org/linux-pci/[email protected]
* Rebase on top v5.12-rc1
* More english fixes
* Returned to static sysfs creation model as was implemented in v0/v1.
v6: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* English fixes
* Movedpci_vf_set_msix_vec_count()frommsi.ctoiov.c
* Embeddedpci_vf_set_msix_vec_count()intosriov_vf_msix_count_store
* Deletedsriov_vf_msix_count_show
* Deletedvfs_overlayfolder
* Renamed functions *_vfs_overlay_* to be *_vf_overlay_*
* Deletedis_supportedandattribute_groupbecause it confused people more than it gave advantage.
* Changedvf_total_msixto be callback
* Patch 3:
* Fixed english as suggested by Bjorn
* Added more explanations to the commit message
* Patch 4:
* Protected enable/disable with capability check
v5: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Added forgotten "inline" keyword when declaring empty functions.
v4: https://lore.kernel.org/linux-pci/[email protected]
* Usedsysfs_emit()instead ofsprintf()in new sysfs entries.
* ChangedEXPORT_SYMBOLto beEXPORT_SYMBOL_GPLforpci_iov_virtfn_devfn().
* Rewrote sysfs registration code to be driven by PF that wants to enable VF overlay instead of creating to all SR-IOV devices.
* Grouped all such functionality under new "vfs_overlay" folder.
* Combined two PCI patches into one.
v3: https://lore.kernel.org/linux-pci/[email protected]
* Renamedpci_set_msix_vec_countto bepci_vf_set_msix_vec_count.
* Added VFmsix_capcheck to hide sysfs entry if device doesn't support msix.
* Changed "-" to be ":" in the mlx5 patch to silence CI warnings about missing kdoc description.
* Split differently error print in mlx5 driver to avoid checkpatch warning.
v2: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Renamedvf_msix_vecsysfs knob to besriov_vf_msix_count
* Added PF and VF device locks during set MSI-X call to protect from parallel driver bind/unbind operations.
* Removed extra checks when readingsriov_vf_msix, because users will be able to distinguish between supported/not supported by looking onsriov_vf_total_msixcount.
* Changed all occurrences of "numb" to be "count"
* Changed returned error from EOPNOTSUPP to be EBUSY if user tries to set MSI-X count after driver already bound to the VF.
* Added extra comment inpci_set_msix_vec_count()to emphasize that driver should not be bound.
* Patch 2:
* Changedvf_total_msixfrom int to be u32 and updated function signatures accordingly.
* Improved patch title
v1: https://lore.kernel.org/linux-pci/[email protected]
* Improved wording and commit messages of first PCI patch
* Added extra PCI patch to provide total number of MSI-X vectors
* Prohibited read ofvf_msix_vecsysfs file if driver doesn't support write
* Removed extra function definition inpci.h
v0: https://lore.kernel.org/linux-pci/[email protected]--------------------------------------------------------------------
Hi,The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Before this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-Configuration script:
1. Start fresh
echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs modprobe -q -rmlx5_ibmlx5_core
2. Ensure that driver doesn't run and it is safe to change MSI-X echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe
3. Load driver for the PF
modprobemlx5_core
4. Configure one of the VFs with new number
echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/sriov_vf_msix_countAfter this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=21 Masked-Thanks
Leon Romanovsky (4):
PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs net/mlx5: Add dynamic MSI-X capabilities bits
net/mlx5: Dynamically assign MSI-X vectors count net/mlx5: Implement sriov_get_vf_total_msix/count() callbacksDocumentation/ABI/testing/sysfs-bus-pci | 29 +++++ .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 12 +++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 73 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 48 ++++++++- drivers/pci/iov.c | 102 ++++++++++++++++-- drivers/pci/pci-sysfs.c | 3 +- drivers/pci/pci.h | 3 +- include/linux/mlx5/mlx5_ifc.h | 11 +- include/linux/pci.h | 8 ++ 10 files changed, 284 insertions(+), 11 deletions(-)--
2.30.2
Bjorn,
Can you please help us progress with the series?
It needs to go from mlx5-next to net-next and maybe to rdma-next too and we need enough time to keep the series for automatic testing in all those branches.
The code was posted on -rc1 [1] while we are in -rc4 already.
[1] https://lore.kernel.org/linux-pci/[email protected]/
Thanks
Quoted 133 lines · Leon Romanovsky
Gentle reminder
Thanks
Quoted 128 lines · Leon Romanovsky
---------------------------------------------------------------------------------
Changelog
v8:
* Added "physical/virtual function" words near PF and VF acronyms.
v7: https://lore.kernel.org/linux-pci/[email protected]
* Rebase on top v5.12-rc1
* More english fixes
* Returned to static sysfs creation model as was implemented in v0/v1.
v6: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* English fixes
* Movedpci_vf_set_msix_vec_count()frommsi.ctoiov.c
* Embeddedpci_vf_set_msix_vec_count()intosriov_vf_msix_count_store
* Deletedsriov_vf_msix_count_show
* Deletedvfs_overlayfolder
* Renamed functions *_vfs_overlay_* to be *_vf_overlay_*
* Deletedis_supportedandattribute_groupbecause it confused people more than it gave advantage.
* Changedvf_total_msixto be callback
* Patch 3:
* Fixed english as suggested by Bjorn
* Added more explanations to the commit message
* Patch 4:
* Protected enable/disable with capability check
v5: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Added forgotten "inline" keyword when declaring empty functions.
v4: https://lore.kernel.org/linux-pci/[email protected]
* Usedsysfs_emit()instead ofsprintf()in new sysfs entries.
* ChangedEXPORT_SYMBOLto beEXPORT_SYMBOL_GPLforpci_iov_virtfn_devfn().
* Rewrote sysfs registration code to be driven by PF that wants to enable VF overlay instead of creating to all SR-IOV devices.
* Grouped all such functionality under new "vfs_overlay" folder.
* Combined two PCI patches into one.
v3: https://lore.kernel.org/linux-pci/[email protected]
* Renamedpci_set_msix_vec_countto bepci_vf_set_msix_vec_count.
* Added VFmsix_capcheck to hide sysfs entry if device doesn't support msix.
* Changed "-" to be ":" in the mlx5 patch to silence CI warnings about missing kdoc description.
* Split differently error print in mlx5 driver to avoid checkpatch warning.
v2: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Renamedvf_msix_vecsysfs knob to besriov_vf_msix_count
* Added PF and VF device locks during set MSI-X call to protect from parallel driver bind/unbind operations.
* Removed extra checks when readingsriov_vf_msix, because users will be able to distinguish between supported/not supported by looking onsriov_vf_total_msixcount.
* Changed all occurrences of "numb" to be "count"
* Changed returned error from EOPNOTSUPP to be EBUSY if user tries to set MSI-X count after driver already bound to the VF.
* Added extra comment inpci_set_msix_vec_count()to emphasize that driver should not be bound.
* Patch 2:
* Changedvf_total_msixfrom int to be u32 and updated function signatures accordingly.
* Improved patch title
v1: https://lore.kernel.org/linux-pci/[email protected]
* Improved wording and commit messages of first PCI patch
* Added extra PCI patch to provide total number of MSI-X vectors
* Prohibited read ofvf_msix_vecsysfs file if driver doesn't support write
* Removed extra function definition inpci.h
v0: https://lore.kernel.org/linux-pci/[email protected]--------------------------------------------------------------------
Hi,The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Before this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-Configuration script:
1. Start fresh
echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs modprobe -q -rmlx5_ibmlx5_core
2. Ensure that driver doesn't run and it is safe to change MSI-X echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe
3. Load driver for the PF
modprobemlx5_core
4. Configure one of the VFs with new number
echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/sriov_vf_msix_countAfter this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=21 Masked-Thanks
Leon Romanovsky (4):
PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs net/mlx5: Add dynamic MSI-X capabilities bits
net/mlx5: Dynamically assign MSI-X vectors count net/mlx5: Implement sriov_get_vf_total_msix/count() callbacksDocumentation/ABI/testing/sysfs-bus-pci | 29 +++++ .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 12 +++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 73 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 48 ++++++++- drivers/pci/iov.c | 102 ++++++++++++++++-- drivers/pci/pci-sysfs.c | 3 +- drivers/pci/pci.h | 3 +- include/linux/mlx5/mlx5_ifc.h | 11 +- include/linux/pci.h | 8 ++ 10 files changed, 284 insertions(+), 11 deletions(-)--
2.30.2
Quoted 113 lines · Leon Romanovsky
---------------------------------------------------------------------------------
Changelog
v8:
* Added "physical/virtual function" words near PF and VF acronyms.
v7: https://lore.kernel.org/linux-pci/[email protected]
* Rebase on top v5.12-rc1
* More english fixes
* Returned to static sysfs creation model as was implemented in v0/v1.
v6: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* English fixes
* Movedpci_vf_set_msix_vec_count()frommsi.ctoiov.c
* Embeddedpci_vf_set_msix_vec_count()intosriov_vf_msix_count_store
* Deletedsriov_vf_msix_count_show
* Deletedvfs_overlayfolder
* Renamed functions *_vfs_overlay_* to be *_vf_overlay_*
* Deletedis_supportedandattribute_groupbecause it confused people more than it gave advantage.
* Changedvf_total_msixto be callback
* Patch 3:
* Fixed english as suggested by Bjorn
* Added more explanations to the commit message
* Patch 4:
* Protected enable/disable with capability check
v5: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Added forgotten "inline" keyword when declaring empty functions.
v4: https://lore.kernel.org/linux-pci/[email protected]
* Usedsysfs_emit()instead ofsprintf()in new sysfs entries.
* ChangedEXPORT_SYMBOLto beEXPORT_SYMBOL_GPLforpci_iov_virtfn_devfn().
* Rewrote sysfs registration code to be driven by PF that wants to enable VF overlay instead of creating to all SR-IOV devices.
* Grouped all such functionality under new "vfs_overlay" folder.
* Combined two PCI patches into one.
v3: https://lore.kernel.org/linux-pci/[email protected]
* Renamedpci_set_msix_vec_countto bepci_vf_set_msix_vec_count.
* Added VFmsix_capcheck to hide sysfs entry if device doesn't support msix.
* Changed "-" to be ":" in the mlx5 patch to silence CI warnings about missing kdoc description.
* Split differently error print in mlx5 driver to avoid checkpatch warning.
v2: https://lore.kernel.org/linux-pci/[email protected]
* Patch 1:
* Renamedvf_msix_vecsysfs knob to besriov_vf_msix_count
* Added PF and VF device locks during set MSI-X call to protect from parallel driver bind/unbind operations.
* Removed extra checks when readingsriov_vf_msix, because users will be able to distinguish between supported/not supported by looking onsriov_vf_total_msixcount.
* Changed all occurrences of "numb" to be "count"
* Changed returned error from EOPNOTSUPP to be EBUSY if user tries to set MSI-X count after driver already bound to the VF.
* Added extra comment inpci_set_msix_vec_count()to emphasize that driver should not be bound.
* Patch 2:
* Changedvf_total_msixfrom int to be u32 and updated function signatures accordingly.
* Improved patch title
v1: https://lore.kernel.org/linux-pci/[email protected]
* Improved wording and commit messages of first PCI patch
* Added extra PCI patch to provide total number of MSI-X vectors
* Prohibited read ofvf_msix_vecsysfs file if driver doesn't support write
* Removed extra function definition inpci.h
v0: https://lore.kernel.org/linux-pci/[email protected]--------------------------------------------------------------------
Hi,The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Before this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-Configuration script:
1. Start fresh
echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs modprobe -q -rmlx5_ibmlx5_core
2. Ensure that driver doesn't run and it is safe to change MSI-X echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe
3. Load driver for the PF
modprobemlx5_core
4. Configure one of the VFs with new number
echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/sriov_vf_msix_countAfter this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=21 Masked-Thanks
Leon Romanovsky (4):
PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs net/mlx5: Add dynamic MSI-X capabilities bits
net/mlx5: Dynamically assign MSI-X vectors count net/mlx5: Implement sriov_get_vf_total_msix/count() callbacks
Quoted 11 lines
Documentation/ABI/testing/sysfs-bus-pci | 29 +++++ .../net/ethernet/mellanox/mlx5/core/main.c | 6 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 12 +++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 73 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 48 ++++++++- drivers/pci/iov.c | 102 ++++++++++++++++-- drivers/pci/pci-sysfs.c | 3 +- drivers/pci/pci.h | 3 +- include/linux/mlx5/mlx5_ifc.h | 11 +- include/linux/pci.h | 8 ++ 10 files changed, 284 insertions(+), 11 deletions(-)
Looks good to me, thanks for persevering with this.
Minor comments on 1/4, not critical.
Leon Romanovsky
AppliedQuoted 8 lines · Leon Romanovsky
---------------------------------------------------------------------------------
Changelog
v8:
* Added "physical/virtual function" words near PF and VF acronyms.
v7: https://lore.kernel.org/linux-pci/[email protected]
* Rebase on top v5.12-rc1
* More english fixes
* Returned to static sysfs creation model as was implemented in v0/v1.
<...>
Quoted 5 lines
Leon Romanovsky (4):
PCI: Add a sysfs file to change the MSI-X table size of SR-IOV VFs net/mlx5: Add dynamic MSI-X capabilities bits
net/mlx5: Dynamically assign MSI-X vectors count net/mlx5: Implement sriov_get_vf_total_msix/count() callbacks
applied to mlx5-next with changes asked by Bjorn.
e71b75f73763 net/mlx5: Implement sriov_get_vf_total_msix/count() callbacks 604774add516 net/mlx5: Dynamically assign MSI-X vectors count 0b989c1e3705 net/mlx5: Add dynamic MSI-X capabilities bits c3d5c2d96d69 PCI/IOV: Add sysfs MSI-X vector assignment interface
Thanks