Dynamically assign MSI-X vectors count
Subsystems (6)
v2 cover Dynamically assign MSI-X vectors count
From: Leon Romanovsky <[email protected]>
Changelog
v2:
* Patch 1:
* Renamed vf_msix_vec sysfs knob to be sriov_vf_msix_count
* Added PF and VF device locks during set MSI-X call to protect from parallel driver bind/unbind operations.
* Removed extra checks when reading sriov_vf_msix, because users will be able to distinguish between supported/not supported by looking on sriov_vf_total_msix count.
* Changed all occurrences of "numb" to be "count"
* Changed returned error from EOPNOTSUPP to be EBUSY if user tries to set MSI-X count after driver already bound to the VF.
* Added extra comment in pci_set_msix_vec_count() to emphasize that driver should not be bound.
* Patch 2:
* Chaged vf_total_msix from int to be u32 and updated function signatures accordingly.
* Improved patch title
v1: https://lore.kernel.org/linux-pci/[email protected]
* Improved wording and commit messages of first PCI patch
* Added extra PCI patch to provide total number of MSI-X vectors
* Prohibited read of vf_msix_vec sysfs file if driver doesn't support write
* Removed extra function definition in pci.h
v0: https://lore.kernel.org/linux-pci/[email protected]
--------------------------------------------------------------------
Hi,
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Before this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-
Configuration script:
1. Start fresh
echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs modprobe -q -r mlx5_ib mlx5_core
2. Ensure that driver doesn't run and it is safe to change MSI-X echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe
3. Load driver for the PF
modprobe mlx5_core
4. Configure one of the VFs with new number
echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/sriov_vf_msix_count
After this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=21 Masked-
Thanks
Leon Romanovsky (5):
PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
PCI: Add SR-IOV sysfs entry to read total number of dynamic MSI-X
vectors
net/mlx5: Add dynamic MSI-X capabilities bits
net/mlx5: Dynamically assign MSI-X vectors count
net/mlx5: Allow to the users to configure number of MSI-X vectors
Documentation/ABI/testing/sysfs-bus-pci | 34 +++++++ .../net/ethernet/mellanox/mlx5/core/main.c | 5 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 6 ++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 62 +++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 52 ++++++++++- drivers/pci/iov.c | 89 +++++++++++++++++++ drivers/pci/msi.c | 47 ++++++++++ drivers/pci/pci-sysfs.c | 1 + drivers/pci/pci.h | 5 ++ include/linux/mlx5/mlx5_ifc.h | 11 ++- include/linux/pci.h | 5 ++ 11 files changed, 314 insertions(+), 3 deletions(-)
--
2.29.2
PATCH v2 1/5 PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
From: Leon Romanovsky <[email protected]>
Extend PCI sysfs interface with a new callback that allows configure the number of MSI-X vectors for specific SR-IO VF. This is needed to optimize the performance of newly bound devices by allocating the number of vectors based on the administrator knowledge of targeted VM.
This function is applicable for SR-IOV VF because such devices allocate their MSI-X table before they will run on the VMs and HW can't guess the right number of vectors, so the HW allocates them statically and equally.
The newly added /sys/bus/pci/devices/.../sriov_vf_msix_count file will be seen for the VFs and it is writable as long as a driver is not bounded to the VF.
The values accepted are:
* > 0 - this will be number reported by the VF's MSI-X capability
* < 0 - not valid
* = 0 - will reset to the device default value
--- Documentation/ABI/testing/sysfs-bus-pci | 20 +++++++++ drivers/pci/iov.c | 58 +++++++++++++++++++++++++ drivers/pci/msi.c | 47 ++++++++++++++++++++ drivers/pci/pci-sysfs.c | 1 + drivers/pci/pci.h | 2 + include/linux/pci.h | 3 ++ 6 files changed, 131 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +20 -0
@@ -375,3 +375,23 @@ Description: The value comes from the PCI kernel device state and can be one of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold". The file is read only. + +What: /sys/bus/pci/devices/.../sriov_vf_msix_count +Date: December 2020 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with the SR-IOV VFs. + It allows configuration of the number of MSI-X vectors for + the VF. This is needed to optimize performance of newly bound + devices by allocating the number of vectors based on the + administrator knowledge of targeted VM. + + The values accepted are: + * > 0 - this will be number reported by the VF's MSI-X + capability + * < 0 - not valid + * = 0 - will reset to the device default value + + The file is writable if the PF is bound to a driver that + set sriov_vf_total_msix > 0 and there is no driver bound + to the VF.
drivers/pci/iov.c +58 -0
70 lines not on this page · raw message
drivers/pci/msi.c +47 -0
53 lines not on this page · raw message
drivers/pci/pci-sysfs.c +1 -0
@@ -1557,6 +1557,7 @@ static const struct attribute_group *pci_dev_attr_groups[] = { &pci_dev_hp_attr_group, #ifdef CONFIG_PCI_IOV &sriov_dev_attr_group, + &sriov_vf_dev_attr_group, #endif &pci_bridge_attr_group, &pcie_dev_attr_group,
drivers/pci/pci.h +2 -0
@@ -183,6 +183,7 @@ extern unsigned int pci_pm_d3hot_delay;#ifdef CONFIG_PCI_MSI void pci_no_msi (void ); +int pci_set_msix_vec_count (struct pci_dev *dev,int count);#else static inline void pci_no_msi (void ) { }#endif
@@ -502,6 +503,7 @@ resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno);void pci_restore_iov_state (struct pci_dev *dev);int pci_iov_bus_range (struct pci_bus *bus);extern const struct attribute_group sriov_dev_attr_group; +extern const struct attribute_group sriov_vf_dev_attr_group;#else static inline int pci_iov_init (struct pci_dev *dev) {
include/linux/pci.h +3 -0
@@ -856,6 +856,8 @@ struct module;* e.g. drivers/net/e100.c. * @sriov_configure: Optional driver callback to allow configuration of * number of VFs to enable via sysfs "sriov_numvfs" file. +* @sriov_set_msix_vec_count: Driver callback to change number of MSI-X vectors +* exposed by the sysfs "vf_msix_vec" entry. * @err_handler: See Documentation/PCI/pci-error-recovery.rst * @groups: Sysfs attribute groups. * @driver: Driver model structure.
@@ -871,6 +873,7 @@ struct pci_driver {int (*resume)(struct pci_dev *dev);/* Device woken up */ void (*shutdown)(struct pci_dev *dev);int (*sriov_configure)(struct pci_dev *dev,int num_vfs);/* On PF */ +int (*sriov_set_msix_vec_count)(struct pci_dev *vf,int msix_vec_count);/* On PF */ const struct pci_error_handlers *err_handler;const struct attribute_group **groups;struct device_driver driver;
--
2.29.2
Re: [PATCH mlx5-next v2 1/5] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
Quoted 111 lines · On Thu, 14 Jan 2021 12:31:36 +0200 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Extend PCI sysfs interface with a new callback that allows configure the number of MSI-X vectors for specific SR-IO VF. This is needed to optimize the performance of newly bound devices by allocating the number of vectors based on the administrator knowledge of targeted VM.
This function is applicable for SR-IOV VF because such devices allocate their MSI-X table before they will run on the VMs and HW can't guess the right number of vectors, so the HW allocates them statically and equally.
The newly added
/sys/bus/pci/devices/.../sriov_vf_msix_countfile will be seen for the VFs and it is writable as long as a driver is not bounded to the VF.The values accepted are:
* > 0 - this will be number reported by the VF's MSI-X capability
* < 0 - not valid
* = 0 - will reset to the device default valueSigned-off-by Leon Romanovsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 20 +++++++++ drivers/pci/iov.c | 58 +++++++++++++++++++++++++ drivers/pci/msi.c | 47 ++++++++++++++++++++ drivers/pci/pci-sysfs.c | 1 + drivers/pci/pci.h | 2 + include/linux/pci.h | 3 ++ 6 files changed, 131 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +20
-0@@ -375,3 +375,23 @@ Description: The value comes from the PCI kernel device state and can be one of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold". The file is read only. + +What: /sys/bus/pci/devices/.../sriov_vf_msix_count +Date: December 2020 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with the SR-IOV VFs. + It allows configuration of the number of MSI-X vectors for + the VF. This is needed to optimize performance of newly bound + devices by allocating the number of vectors based on the + administrator knowledge of targeted VM. + + The values accepted are: + * > 0 - this will be number reported by the VF's MSI-X + capability + * < 0 - not valid + * = 0 - will reset to the device default value + + The file is writable if the PF is bound to a driver that + set sriov_vf_total_msix > 0 and there is no driver bound + to the VF.drivers/pci/iov.c +39
-0@@ -31,6 +31,7 @@ int pci_iov_virtfn_devfn(struct pci_dev *dev, int vf_id)return (dev->devfn + dev->sriov->offset + dev->sriov->stride * vf_id) &0xff ; } +EXPORT_SYMBOL (pci_iov_virtfn_devfn);/* * Per SR-IOV spec sec 3.3.10 and 3.3.11, First VF Offset and VF Stride may @@ -426,6 +427,63 @@ const struct attribute_group sriov_dev_attr_group = { .is_visible = sriov_attrs_are_visible, }; +#ifdef CONFIG_PCI_MSI +static ssize_t sriov_vf_msix_count_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +int count =pci_msix_vec_count (pdev); + +if (count <0 ) +return count; + +return sprintf (buf,"%d\n" , count); +} + +static ssize_t sriov_vf_msix_count_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct pci_dev *vf_dev =to_pci_dev (dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + + ret =pci_set_msix_vec_count (vf_dev, val); +if (ret) +return ret; + +return count; +} +static DEVICE_ATTR_RW (sriov_vf_msix_count); +#endif + +static struct attribute *sriov_vf_dev_attrs[] = { +#ifdef CONFIG_PCI_MSI + &dev_attr_sriov_vf_msix_count.attr, +#endif
drivers/pci/iov.c const struct attribute_group sriov_dev_attr_group = { · line 477
Quoted 10 lines
drivers/pci/iov.c +10
-0+ NULL, +}; + +static umode_t sriov_vf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct device *dev =kobj_to_dev (kobj); + +if (dev_is_pf (dev)) +return 0 ;
Wouldn't it be cleaner to also hide this on VFs where pci_msix_vec_count() returns an error or where the PF driver doesn't implement .sriov_set_msix_vec_count()? IOW, expose it only where it could actually work.
Quoted 17 lines
drivers/pci/iov.c +9
-0+ +return a->mode; +} + +const struct attribute_group sriov_vf_dev_attr_group = { + .attrs = sriov_vf_dev_attrs, + .is_visible = sriov_vf_attrs_are_visible, +}; +int __weak pcibios_sriov_enable (struct pci_dev *pdev,u16 num_vfs) {return 0 ;drivers/pci/msi.c +0
-0
drivers/pci/msi.c int pci_msix_vec_count(struct pci_dev *dev) · line 998
Quoted 8 lines
drivers/pci/msi.c +5
-0@@ -991,6 +991,53 @@ int pci_msix_vec_count(struct pci_dev *dev) }EXPORT_SYMBOL (pci_msix_vec_count); +/** +* pci_set_msix_vec_count - change the reported number of MSI-X vectors +* This function is applicable for SR-IOV VF because such devices allocate +* their MSI-X table before they will run on the VMs and HW can't guess the +* right number of vectors, so the HW allocates them statically and equally.
Nit, this is an assumption of the VF usage and conjecture of the implementation.
drivers/pci/msi.c int pci_msix_vec_count(struct pci_dev *dev) · line 1002
Quoted 4 lines
drivers/pci/msi.c +4
-0+* @dev: VF device that is going to be changed +* @count amount of MSI-X vectors +**/ +int pci_set_msix_vec_count (struct pci_dev *dev,int count)
pci_vf_set_msix_vec_count()? Long, I know, but if it's limited to VFs name it accordingly. Thanks,
Alex
Quoted 97 lines
drivers/pci/msi.c +38
-0+{ +struct pci_dev *pdev =pci_physfn (dev); +int ret; + +if (!dev->msix_cap || !pdev->msix_cap || count <0 ) +/* +* We don't support negative numbers for now, +* but maybe in the future it will make sense. +*/ +return -EINVAL ; + +device_lock (&pdev->dev); +if (!pdev->driver || !pdev->driver->sriov_set_msix_vec_count) { + ret = -EOPNOTSUPP ; + goto err_pdev; + } + +device_lock (&dev->dev); +if (dev->driver) { +/* +* Driver already probed this VF and configured itself +* based on previously configured (or default) MSI-X vector +* count. It is too late to change this field for this +* specific VF. +*/ + ret = -EBUSY ; + goto err_dev; + } + + ret = pdev->driver->sriov_set_msix_vec_count (dev, count); + +err_dev: +device_unlock (&dev->dev); +err_pdev: +device_unlock (&pdev->dev); +return ret; +} +static int __pci_enable_msix (struct pci_dev *dev,struct msix_entry *entries,int nvec,struct irq_affinity *affd,int flags) {drivers/pci/pci-sysfs.c +1
-0@@ -1557,6 +1557,7 @@ static const struct attribute_group *pci_dev_attr_groups[] = { &pci_dev_hp_attr_group, #ifdef CONFIG_PCI_IOV &sriov_dev_attr_group, + &sriov_vf_dev_attr_group, #endif &pci_bridge_attr_group, &pcie_dev_attr_group,drivers/pci/pci.h +2
-0@@ -183,6 +183,7 @@ extern unsigned int pci_pm_d3hot_delay;#ifdef CONFIG_PCI_MSI void pci_no_msi (void ); +int pci_set_msix_vec_count (struct pci_dev *dev,int count);#else static inline void pci_no_msi (void ) { }#endif @@ -502,6 +503,7 @@ resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno);void pci_restore_iov_state (struct pci_dev *dev);int pci_iov_bus_range (struct pci_bus *bus);extern const struct attribute_group sriov_dev_attr_group; +extern const struct attribute_group sriov_vf_dev_attr_group;#else static inline int pci_iov_init (struct pci_dev *dev) {include/linux/pci.h +3
-1@@ -856,6 +856,8 @@ struct module;* e.g. drivers/net/e100.c. * @sriov_configure: Optional driver callback to allow configuration of * number of VFs to enable via sysfs "sriov_numvfs" file. +* @sriov_set_msix_vec_count: Driver callback to change number of MSI-X vectors +* exposed by the sysfs "vf_msix_vec" entry. * @err_handler: See Documentation/PCI/pci-error-recovery.rst * @groups: Sysfs attribute groups. * @driver: Driver model structure. @@ -871,6 +873,7 @@ struct pci_driver {int (*resume)(struct pci_dev *dev);/* Device woken up */ void (*shutdown)(struct pci_dev *dev);int (*sriov_configure)(struct pci_dev *dev,int num_vfs);/* On PF */ +int (*sriov_set_msix_vec_count)(struct pci_dev *vf,int msix_vec_count);/* On PF */ const struct pci_error_handlers *err_handler;const struct attribute_group **groups;struct device_driver driver; --2.29.2
Re: [PATCH mlx5-next v2 1/5] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
Quoted 121 lines · Alex Williamson
Quoted 118 lines · On Thu, 14 Jan 2021 12:31:36 +0200 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Extend PCI sysfs interface with a new callback that allows configure the number of MSI-X vectors for specific SR-IO VF. This is needed to optimize the performance of newly bound devices by allocating the number of vectors based on the administrator knowledge of targeted VM.
This function is applicable for SR-IOV VF because such devices allocate their MSI-X table before they will run on the VMs and HW can't guess the right number of vectors, so the HW allocates them statically and equally.
The newly added
/sys/bus/pci/devices/.../sriov_vf_msix_countfile will be seen for the VFs and it is writable as long as a driver is not bounded to the VF.The values accepted are:
* > 0 - this will be number reported by the VF's MSI-X capability
* < 0 - not valid
* = 0 - will reset to the device default valueSigned-off-by Leon Romanovsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 20 +++++++++ drivers/pci/iov.c | 58 +++++++++++++++++++++++++ drivers/pci/msi.c | 47 ++++++++++++++++++++ drivers/pci/pci-sysfs.c | 1 + drivers/pci/pci.h | 2 + include/linux/pci.h | 3 ++ 6 files changed, 131 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +20
-0@@ -375,3 +375,23 @@ Description: The value comes from the PCI kernel device state and can be one of: "unknown", "error", "D0", D1", "D2", "D3hot", "D3cold". The file is read only. + +What: /sys/bus/pci/devices/.../sriov_vf_msix_count +Date: December 2020 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with the SR-IOV VFs. + It allows configuration of the number of MSI-X vectors for + the VF. This is needed to optimize performance of newly bound + devices by allocating the number of vectors based on the + administrator knowledge of targeted VM. + + The values accepted are: + * > 0 - this will be number reported by the VF's MSI-X + capability + * < 0 - not valid + * = 0 - will reset to the device default value + + The file is writable if the PF is bound to a driver that + set sriov_vf_total_msix > 0 and there is no driver bound + to the VF.drivers/pci/iov.c +46
-0@@ -31,6 +31,7 @@ int pci_iov_virtfn_devfn(struct pci_dev *dev, int vf_id)return (dev->devfn + dev->sriov->offset + dev->sriov->stride * vf_id) &0xff ; } +EXPORT_SYMBOL (pci_iov_virtfn_devfn);/* * Per SR-IOV spec sec 3.3.10 and 3.3.11, First VF Offset and VF Stride may @@ -426,6 +427,63 @@ const struct attribute_group sriov_dev_attr_group = { .is_visible = sriov_attrs_are_visible, }; +#ifdef CONFIG_PCI_MSI +static ssize_t sriov_vf_msix_count_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); +int count =pci_msix_vec_count (pdev); + +if (count <0 ) +return count; + +return sprintf (buf,"%d\n" , count); +} + +static ssize_t sriov_vf_msix_count_store (struct device *dev, +struct device_attribute *attr, +const char *buf,size_t count) +{ +struct pci_dev *vf_dev =to_pci_dev (dev); +int val, ret; + + ret =kstrtoint (buf,0 , &val); +if (ret) +return ret; + + ret =pci_set_msix_vec_count (vf_dev, val); +if (ret) +return ret; + +return count; +} +static DEVICE_ATTR_RW (sriov_vf_msix_count); +#endif + +static struct attribute *sriov_vf_dev_attrs[] = { +#ifdef CONFIG_PCI_MSI + &dev_attr_sriov_vf_msix_count.attr, +#endif +NULL , +}; + +static umode_t sriov_vf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct device *dev =kobj_to_dev (kobj);
Quoted 8 lines
drivers/pci/iov.c const struct attribute_group sriov_dev_attr_group = { · line 477
Quoted 3 lines
drivers/pci/iov.c +3
-0+ +if (dev_is_pf (dev)) +return 0 ;Wouldn't it be cleaner to also hide this on VFs where
pci_msix_vec_count()returns an error or where the PF driver doesn't implement .sriov_set_msix_vec_count()? IOW, expose it only where it could actually work.
I wasn't sure about the policy in PCI/core, but sure will change.
Quoted 21 lines
Quoted 21 lines
drivers/pci/iov.c +9
-0+ +return a->mode; +} + +const struct attribute_group sriov_vf_dev_attr_group = { + .attrs = sriov_vf_dev_attrs, + .is_visible = sriov_vf_attrs_are_visible, +}; +int __weak pcibios_sriov_enable (struct pci_dev *pdev,u16 num_vfs) {return 0 ;drivers/pci/msi.c +0
-0@@ -991,6 +991,53 @@ int pci_msix_vec_count(struct pci_dev *dev) }EXPORT_SYMBOL (pci_msix_vec_count);
Quoted 8 lines
drivers/pci/msi.c int pci_msix_vec_count(struct pci_dev *dev) · line 998
Quoted 5 lines
drivers/pci/msi.c +5
-0+/** +* pci_set_msix_vec_count - change the reported number of MSI-X vectors +* This function is applicable for SR-IOV VF because such devices allocate +* their MSI-X table before they will run on the VMs and HW can't guess the +* right number of vectors, so the HW allocates them statically and equally. Nit, this is an assumption of the VF usage and conjecture of the implementation.
This is one of the possible implementations.
Quoted 8 lines
drivers/pci/msi.c int pci_msix_vec_count(struct pci_dev *dev) · line 1002
Quoted 5 lines
drivers/pci/msi.c +4
-0+* @dev: VF device that is going to be changed +* @count amount of MSI-X vectors +**/ +int pci_set_msix_vec_count (struct pci_dev *dev,int count)
pci_vf_set_msix_vec_count()? Long, I know, but if it's limited to VFs name it accordingly. Thanks,
I'll do.
Quoted 101 lines
Alex
Quoted 97 lines
drivers/pci/msi.c +38
-0+{ +struct pci_dev *pdev =pci_physfn (dev); +int ret; + +if (!dev->msix_cap || !pdev->msix_cap || count <0 ) +/* +* We don't support negative numbers for now, +* but maybe in the future it will make sense. +*/ +return -EINVAL ; + +device_lock (&pdev->dev); +if (!pdev->driver || !pdev->driver->sriov_set_msix_vec_count) { + ret = -EOPNOTSUPP ; + goto err_pdev; + } + +device_lock (&dev->dev); +if (dev->driver) { +/* +* Driver already probed this VF and configured itself +* based on previously configured (or default) MSI-X vector +* count. It is too late to change this field for this +* specific VF. +*/ + ret = -EBUSY ; + goto err_dev; + } + + ret = pdev->driver->sriov_set_msix_vec_count (dev, count); + +err_dev: +device_unlock (&dev->dev); +err_pdev: +device_unlock (&pdev->dev); +return ret; +} +static int __pci_enable_msix (struct pci_dev *dev,struct msix_entry *entries,int nvec,struct irq_affinity *affd,int flags) {drivers/pci/pci-sysfs.c +1
-0@@ -1557,6 +1557,7 @@ static const struct attribute_group *pci_dev_attr_groups[] = { &pci_dev_hp_attr_group, #ifdef CONFIG_PCI_IOV &sriov_dev_attr_group, + &sriov_vf_dev_attr_group, #endif &pci_bridge_attr_group, &pcie_dev_attr_group,drivers/pci/pci.h +2
-0@@ -183,6 +183,7 @@ extern unsigned int pci_pm_d3hot_delay;#ifdef CONFIG_PCI_MSI void pci_no_msi (void ); +int pci_set_msix_vec_count (struct pci_dev *dev,int count);#else static inline void pci_no_msi (void ) { }#endif @@ -502,6 +503,7 @@ resource_size_t pci_sriov_resource_alignment(struct pci_dev *dev, int resno);void pci_restore_iov_state (struct pci_dev *dev);int pci_iov_bus_range (struct pci_bus *bus);extern const struct attribute_group sriov_dev_attr_group; +extern const struct attribute_group sriov_vf_dev_attr_group;#else static inline int pci_iov_init (struct pci_dev *dev) {include/linux/pci.h +3
-1@@ -856,6 +856,8 @@ struct module;* e.g. drivers/net/e100.c. * @sriov_configure: Optional driver callback to allow configuration of * number of VFs to enable via sysfs "sriov_numvfs" file. +* @sriov_set_msix_vec_count: Driver callback to change number of MSI-X vectors +* exposed by the sysfs "vf_msix_vec" entry. * @err_handler: See Documentation/PCI/pci-error-recovery.rst * @groups: Sysfs attribute groups. * @driver: Driver model structure. @@ -871,6 +873,7 @@ struct pci_driver {int (*resume)(struct pci_dev *dev);/* Device woken up */ void (*shutdown)(struct pci_dev *dev);int (*sriov_configure)(struct pci_dev *dev,int num_vfs);/* On PF */ +int (*sriov_set_msix_vec_count)(struct pci_dev *vf,int msix_vec_count);/* On PF */ const struct pci_error_handlers *err_handler;const struct attribute_group **groups;struct device_driver driver; --2.29.2
Re: [PATCH mlx5-next v2 1/5] PCI: Add sysfs callback to allow MSI-X table size change of SR-IOV VFs
Quoted 32 lines · Leon Romanovsky
Quoted 31 lines · Alex Williamson
Quoted 28 lines · On Thu, 14 Jan 2021 12:31:36 +0200 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Extend PCI sysfs interface with a new callback that allows configure the number of MSI-X vectors for specific SR-IO VF. This is needed to optimize the performance of newly bound devices by allocating the number of vectors based on the administrator knowledge of targeted VM.
This function is applicable for SR-IOV VF because such devices allocate their MSI-X table before they will run on the VMs and HW can't guess the right number of vectors, so the HW allocates them statically and equally.
The newly added
/sys/bus/pci/devices/.../sriov_vf_msix_countfile will be seen for the VFs and it is writable as long as a driver is not bounded to the VF.The values accepted are:
* > 0 - this will be number reported by the VF's MSI-X capability
* < 0 - not valid
* = 0 - will reset to the device default valueSigned-off-by Leon Romanovsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 20 +++++++++ drivers/pci/iov.c | 58 +++++++++++++++++++++++++ drivers/pci/msi.c | 47 ++++++++++++++++++++ drivers/pci/pci-sysfs.c | 1 + drivers/pci/pci.h | 2 + include/linux/pci.h | 3 ++ 6 files changed, 131 insertions(+)
<...>
Quoted 13 lines
Quoted 4 lines
Quoted 4 lines
drivers/pci/iov.c +4
-0+static umode_t sriov_vf_attrs_are_visible (struct kobject *kobj, +struct attribute *a,int n) +{ +struct device *dev =kobj_to_dev (kobj);Quoted 8 lines
drivers/pci/iov.c const struct attribute_group sriov_dev_attr_group = { · line 477
Quoted 3 lines
drivers/pci/iov.c +3
-0+ +if (dev_is_pf (dev)) +return 0 ;Wouldn't it be cleaner to also hide this on VFs where
pci_msix_vec_count()returns an error or where the PF driver doesn't implement .sriov_set_msix_vec_count()? IOW, expose it only where it could actually work.
Quoted 1 line
I wasn't sure about the policy in PCI/core, but sure will change.
I ended adding checks of msix_cap, but can't check .sriov_set_msix_vec_count. The latter will require to hold device_lock on PF that can disappear later, it is too racy.
Thanks
PATCH v2 2/5 PCI: Add SR-IOV sysfs entry to read total number of dynamic MSI-X vectors
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Some SR-IOV capable devices provide an ability to configure specific number of MSI-X vectors on their VF prior driver is probed on that VF.
In order to make management easy, provide new read-only sysfs file that returns a total number of possible to configure MSI-X vectors.
cat /sys/bus/pci/devices/.../sriov_vf_total_msix
= 0 - feature is not supported
Quoted 1 line
0 - total number of MSI-X vectors to consume by the VFs
--- Documentation/ABI/testing/sysfs-bus-pci | 14 +++++++++++ drivers/pci/iov.c | 31 +++++++++++++++++++++++++ drivers/pci/pci.h | 3 +++ include/linux/pci.h | 2 ++ 4 files changed, 50 insertions(+)
Documentation/ABI/testing/sysfs-bus-pci +14 -0
@@ -395,3 +395,17 @@ Description: The file is writable if the PF is bound to a driver that set sriov_vf_total_msix > 0 and there is no driver bound to the VF. + +What: /sys/bus/pci/devices/.../sriov_vf_total_msix +Date: January 2021 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with the SR-IOV PFs. + It returns a total number of possible to configure MSI-X + vectors on the enabled VFs. + + The values returned are: + * > 0 - this will be total number possible to consume by VFs, + * = 0 - feature is not supported + + If no SR-IOV VFs are enabled, this value will return 0.
drivers/pci/iov.c +31 -0
@@ -394,12 +394,22 @@ static ssize_t sriov_drivers_autoprobe_store(struct device *dev,return count; } +static ssize_t sriov_vf_total_msix_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); + +return sprintf (buf,"%u\n" , pdev->sriov->vf_total_msix); +} +static DEVICE_ATTR_RO (sriov_totalvfs);static DEVICE_ATTR_RW (sriov_numvfs);static DEVICE_ATTR_RO (sriov_offset);static DEVICE_ATTR_RO (sriov_stride);static DEVICE_ATTR_RO (sriov_vf_device);static DEVICE_ATTR_RW (sriov_drivers_autoprobe); +static DEVICE_ATTR_RO (sriov_vf_total_msix);static struct attribute *sriov_dev_attrs[] = { &dev_attr_sriov_totalvfs.attr,
@@ -408,6 +418,7 @@ static struct attribute *sriov_dev_attrs[] = { &dev_attr_sriov_stride.attr, &dev_attr_sriov_vf_device.attr, &dev_attr_sriov_drivers_autoprobe.attr, + &dev_attr_sriov_vf_total_msix.attr, NULL, };
@@ -654,6 +665,7 @@ static void sriov_disable(struct pci_dev *dev)sysfs_remove_link (&dev->dev.kobj,"dep_link" ); iov->num_VFs =0 ; + iov->vf_total_msix =0 ;pci_iov_set_numvfs (dev,0 ); }
@@ -1112,6 +1124,25 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pci_sriov_get_totalvfs); +/** +* pci_sriov_set_vf_total_msix - set total number of MSI-X vectors for the VFs +* @dev: the PCI PF device +* @count: the total number of MSI-X vector to consume by the VFs +* +* Sets the number of MSI-X vectors that is possible to consume by the VFs. +* This interface is complimentary part of the pci_set_msix_vec_count() +* that will be used to configure the required number on the VF. +*/ +void pci_sriov_set_vf_total_msix (struct pci_dev *dev,u32 count) +{ +if (!dev->is_physfn || !dev->driver || + !dev->driver->sriov_set_msix_vec_count) +return ; + + dev->sriov->vf_total_msix = count; +} +EXPORT_SYMBOL_GPL (pci_sriov_set_vf_total_msix); +/** * pci_sriov_configure_simple - helper to configure SR-IOV * @dev: the PCI device
drivers/pci/pci.h +3 -0
@@ -327,6 +327,9 @@ struct pci_sriov {u16 subsystem_device;/* VF subsystem device */ resource_size_t barsz[PCI_SRIOV_NUM_BARS ];/* VF BAR size */ bool drivers_autoprobe;/* Auto probing of VFs by driver */ +u32 vf_total_msix;/* Total number of MSI-X vectors the VFs +* can consume +*/ };/**
include/linux/pci.h +2 -0
@@ -2075,6 +2075,7 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev);int pci_sriov_configure_simple (struct pci_dev *dev,int nr_virtfn);resource_size_t pci_iov_resource_size (struct pci_dev *dev,int resno);void pci_vf_drivers_autoprobe (struct pci_dev *dev,bool probe); +void pci_sriov_set_vf_total_msix (struct pci_dev *dev,u32 count);/* Arch may override these (weak) */ int pcibios_sriov_enable (struct pci_dev *pdev,u16 num_vfs);
@@ -2115,6 +2116,7 @@ static inline int pci_sriov_get_totalvfs(struct pci_dev *dev)static inline resource_size_t pci_iov_resource_size (struct pci_dev *dev,int resno) {return 0 ; }static inline void pci_vf_drivers_autoprobe (struct pci_dev *dev,bool probe) { } +static inline void pci_sriov_set_vf_total_msix (struct pci_dev *dev,u32 count) {}#endif #if defined(CONFIG_HOTPLUG_PCI ) || defined(CONFIG_HOTPLUG_PCI_MODULE )
--
2.29.2
Re: [PATCH mlx5-next v2 2/5] PCI: Add SR-IOV sysfs entry to read total number of dynamic MSI-X vectors
Quoted 30 lines · On Thu, 14 Jan 2021 12:31:37 +0200 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Some SR-IOV capable devices provide an ability to configure specific number of MSI-X vectors on their VF prior driver is probed on that VF.
In order to make management easy, provide new read-only sysfs file that returns a total number of possible to configure MSI-X vectors.
cat
/sys/bus/pci/devices/.../sriov_vf_total_msix
= 0 - feature is not supportedQuoted 1 line
0 - total number of MSI-X vectors to consume by the VFs
Signed-off-by Leon Romanovsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 14 +++++++++++ drivers/pci/iov.c | 31 +++++++++++++++++++++++++ drivers/pci/pci.h | 3 +++ include/linux/pci.h | 2 ++ 4 files changed, 50 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +2
-0@@ -395,3 +395,17 @@ Description: The file is writable if the PF is bound to a driver that set sriov_vf_total_msix > 0 and there is no driver bound to the VF. + +What: /sys/bus/pci/devices/.../sriov_vf_total_msix
Documentation/ABI/testing/sysfs-bus-pci Description: · line 409
Quoted 10 lines
Documentation/ABI/testing/sysfs-bus-pci +10
-0+Date: January 2021 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with the SR-IOV PFs. + It returns a total number of possible to configure MSI-X + vectors on the enabled VFs. + + The values returned are: + * > 0 - this will be total number possible to consume by VFs, + * = 0 - feature is not supported
As with previous, why expose it if not supported?
This seems pretty challenging for userspace to use; aiui they would need to iterate all the VFs to learn how many vectors are already allocated, subtract that number from this value, all while hoping they aren't racing someone else doing the same. Would it be more useful if this reported the number of surplus vectors available?
How would a per VF limit be exposed? Do we expect users to know the absolutely MSI-X vector limit or the device specific limit? Thanks,
Alex
Quoted 108 lines
Documentation/ABI/testing/sysfs-bus-pci +2
-0+ + If no SR-IOV VFs are enabled, this value will return 0.drivers/pci/iov.c +31
-0@@ -394,12 +394,22 @@ static ssize_t sriov_drivers_autoprobe_store(struct device *dev,return count; } +static ssize_t sriov_vf_total_msix_show (struct device *dev, +struct device_attribute *attr, +char *buf) +{ +struct pci_dev *pdev =to_pci_dev (dev); + +return sprintf (buf,"%u\n" , pdev->sriov->vf_total_msix); +} +static DEVICE_ATTR_RO (sriov_totalvfs);static DEVICE_ATTR_RW (sriov_numvfs);static DEVICE_ATTR_RO (sriov_offset);static DEVICE_ATTR_RO (sriov_stride);static DEVICE_ATTR_RO (sriov_vf_device);static DEVICE_ATTR_RW (sriov_drivers_autoprobe); +static DEVICE_ATTR_RO (sriov_vf_total_msix);static struct attribute *sriov_dev_attrs[] = { &dev_attr_sriov_totalvfs.attr,@@ -408,6 +418,7 @@ static struct attribute *sriov_dev_attrs[] = { &dev_attr_sriov_stride.attr, &dev_attr_sriov_vf_device.attr, &dev_attr_sriov_drivers_autoprobe.attr, + &dev_attr_sriov_vf_total_msix.attr, NULL, };@@ -654,6 +665,7 @@ static void sriov_disable(struct pci_dev *dev)sysfs_remove_link (&dev->dev.kobj,"dep_link" ); iov->num_VFs =0 ; + iov->vf_total_msix =0 ;pci_iov_set_numvfs (dev,0 ); }@@ -1112,6 +1124,25 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev) }EXPORT_SYMBOL_GPL (pci_sriov_get_totalvfs); +/** +* pci_sriov_set_vf_total_msix - set total number of MSI-X vectors for the VFs +* @dev: the PCI PF device +* @count: the total number of MSI-X vector to consume by the VFs +* +* Sets the number of MSI-X vectors that is possible to consume by the VFs. +* This interface is complimentary part of the pci_set_msix_vec_count() +* that will be used to configure the required number on the VF. +*/ +void pci_sriov_set_vf_total_msix (struct pci_dev *dev,u32 count) +{ +if (!dev->is_physfn || !dev->driver || + !dev->driver->sriov_set_msix_vec_count) +return ; + + dev->sriov->vf_total_msix = count; +} +EXPORT_SYMBOL_GPL (pci_sriov_set_vf_total_msix); +/** * pci_sriov_configure_simple - helper to configure SR-IOV * @dev: the PCI device drivers/pci/pci.h +3
-0@@ -327,6 +327,9 @@ struct pci_sriov {u16 subsystem_device;/* VF subsystem device */ resource_size_t barsz[PCI_SRIOV_NUM_BARS ];/* VF BAR size */ bool drivers_autoprobe;/* Auto probing of VFs by driver */ +u32 vf_total_msix;/* Total number of MSI-X vectors the VFs +* can consume +*/ };/** include/linux/pci.h +2
-1@@ -2075,6 +2075,7 @@ int pci_sriov_get_totalvfs(struct pci_dev *dev);int pci_sriov_configure_simple (struct pci_dev *dev,int nr_virtfn);resource_size_t pci_iov_resource_size (struct pci_dev *dev,int resno);void pci_vf_drivers_autoprobe (struct pci_dev *dev,bool probe); +void pci_sriov_set_vf_total_msix (struct pci_dev *dev,u32 count);/* Arch may override these (weak) */ int pcibios_sriov_enable (struct pci_dev *pdev,u16 num_vfs);@@ -2115,6 +2116,7 @@ static inline int pci_sriov_get_totalvfs(struct pci_dev *dev)static inline resource_size_t pci_iov_resource_size (struct pci_dev *dev,int resno) {return 0 ; }static inline void pci_vf_drivers_autoprobe (struct pci_dev *dev,bool probe) { } +static inline void pci_sriov_set_vf_total_msix (struct pci_dev *dev,u32 count) {}#endif #if defined(CONFIG_HOTPLUG_PCI ) || defined(CONFIG_HOTPLUG_PCI_MODULE ) --2.29.2
Re: [PATCH mlx5-next v2 2/5] PCI: Add SR-IOV sysfs entry to read total number of dynamic MSI-X vectors
Quoted 37 lines · Alex Williamson
Quoted 34 lines · On Thu, 14 Jan 2021 12:31:37 +0200 Leon Romanovsky <[email protected]> wrote:
From: Leon Romanovsky <[email protected]>
Some SR-IOV capable devices provide an ability to configure specific number of MSI-X vectors on their VF prior driver is probed on that VF.
In order to make management easy, provide new read-only sysfs file that returns a total number of possible to configure MSI-X vectors.
cat
/sys/bus/pci/devices/.../sriov_vf_total_msix
= 0 - feature is not supportedQuoted 1 line
0 - total number of MSI-X vectors to consume by the VFs
Signed-off-by Leon Romanovsky <[email protected]>--- Documentation/ABI/testing/sysfs-bus-pci | 14 +++++++++++ drivers/pci/iov.c | 31 +++++++++++++++++++++++++ drivers/pci/pci.h | 3 +++ include/linux/pci.h | 2 ++ 4 files changed, 50 insertions(+)Documentation/ABI/testing/sysfs-bus-pci +6
-0@@ -395,3 +395,17 @@ Description: The file is writable if the PF is bound to a driver that set sriov_vf_total_msix > 0 and there is no driver bound to the VF. + +What: /sys/bus/pci/devices/.../sriov_vf_total_msix +Date: January 2021 +Contact: Leon Romanovsky <[email protected]> +Description: + This file is associated with the SR-IOV PFs.
Quoted 8 lines
Documentation/ABI/testing/sysfs-bus-pci Description: · line 409
Quoted 6 lines
Documentation/ABI/testing/sysfs-bus-pci +6
-0+ It returns a total number of possible to configure MSI-X + vectors on the enabled VFs. + + The values returned are: + * > 0 - this will be total number possible to consume by VFs, + * = 0 - feature is not supportedAs with previous, why expose it if not supported?
It is much simpler to the users implement logic that operates accordingly to this value instead of relying on exist/not-exist and anyway handle 0 to be on the safe side.
Quoted 6 lines
This seems pretty challenging for userspace to use; aiui they would need to iterate all the VFs to learn how many vectors are already allocated, subtract that number from this value, all while hoping they aren't racing someone else doing the same. Would it be more useful if this reported the number of surplus vectors available?
Only privileged users are allowed to do it, so it is unlikely that we will have more than one entity which manages PFs/VFs assignments.
Users already count number of CPUs they give to the VMs, so counting resources is not new to them.
I didn't count in the kernel because it will require from users to understand and treat "0" differently to understand that the pool is depleted. So they will need to count max size of the pool anyway.
Unless we want to have two knobs, one of max and another for current, they will count. The thing is that users will count anyway and won't use the current value. It gives nothing.
Quoted 3 lines
How would a per VF limit be exposed? Do we expect users to know the absolutely MSI-X vector limit or the device specific limit? Thanks,
At this stage yes, we can discuss it later when the need will arise.
Thanks
PATCH v2 3/5 net/mlx5: Add dynamic MSI-X capabilities bits
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
These new fields declare the number of MSI-X vectors that is possible to allocate on the VF through PF configuration.
Value must be in range defined by min_dynamic_vf_msix_table_size and max_dynamic_vf_msix_table_size.
The driver should continue to query its MSI-X table through PCI configuration header.
--- include/linux/mlx5/mlx5_ifc.h | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-)
include/linux/mlx5/mlx5_ifc.h +10 -1
@@ -1656,7 +1656,16 @@ struct mlx5_ifc_cmd_hca_cap_bits {u8 reserved_at_6e0[0x10 ];u8 sf_base_id[0x10 ]; -u8 reserved_at_700[0x80 ]; +u8 reserved_at_700[0x8 ]; +u8 num_total_dynamic_vf_msix[0x18 ]; +u8 reserved_at_720[0x14 ]; +u8 dynamic_msix_table_size[0xc ]; +u8 reserved_at_740[0xc ]; +u8 min_dynamic_vf_msix_table_size[0x4 ]; +u8 reserved_at_750[0x4 ]; +u8 max_dynamic_vf_msix_table_size[0xc ]; + +u8 reserved_at_760[0x20 ];u8 vhca_tunnel_commands[0x40 ];u8 reserved_at_7c0[0x40 ]; };
Signature · 1 line
2.29.2
PATCH v2 4/5 net/mlx5: Dynamically assign MSI-X vectors count
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device. The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors.
Such change immediately increases the amount of MSI-X vectors for the system with @ VFs from 12 vectors per-VF, to be 32 vectors per-VF.
Before this patch:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-
After this patch:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=32 Masked-
--- .../net/ethernet/mellanox/mlx5/core/main.c | 4 ++ .../ethernet/mellanox/mlx5/core/mlx5_core.h | 5 ++ .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 62 +++++++++++++++++++ .../net/ethernet/mellanox/mlx5/core/sriov.c | 14 ++++- 4 files changed, 83 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +4 -0
@@ -567,6 +567,10 @@ static int handle_hca_cap(struct mlx5_core_dev *dev, void *set_ctx)if (MLX5_CAP_GEN_MAX (dev, mkey_by_name))MLX5_SET (cmd_hca_cap, set_hca_cap, mkey_by_name,1 ); +if (MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix)) +MLX5_SET (cmd_hca_cap, set_hca_cap, num_total_dynamic_vf_msix, +MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix)); +return set_caps (dev, set_ctx,MLX5_SET_HCA_CAP_OP_MOD_GENERAL_DEVICE ); }
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +5 -0
@@ -172,6 +172,11 @@ int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx, struct notifier_block *nb);int mlx5_irq_detach_nb (struct mlx5_irq_table *irq_table,int vecidx,struct notifier_block *nb); + +int mlx5_set_msix_vec_count (struct mlx5_core_dev *dev,int devfn, +int msix_vec_count); +int mlx5_get_default_msix_vec_count (struct mlx5_core_dev *dev,int num_vfs); +struct cpumask *mlx5_irq_get_affinity_mask (struct mlx5_irq_table *irq_table,int vecidx);struct cpu_rmap *mlx5_irq_get_rmap (struct mlx5_irq_table *table);
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +62 -0
68 lines not on this page · raw message
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +12 -2
@@ -71,8 +71,7 @@ static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf)static int mlx5_device_enable_sriov (struct mlx5_core_dev *dev,int num_vfs) {struct mlx5_core_sriov *sriov = &dev->priv.sriov; -int err; -int vf; +int err, vf, num_msix_count;if (!MLX5_ESWITCH_MANAGER (dev)) goto enable_vfs_hca;
@@ -85,12 +84,23 @@ static int mlx5_device_enable_sriov(struct mlx5_core_dev *dev, int num_vfs) } enable_vfs_hca: + num_msix_count =mlx5_get_default_msix_vec_count (dev, num_vfs);for (vf =0 ; vf < num_vfs; vf++) { err =mlx5_core_enable_hca (dev, vf +1 );if (err) {mlx5_core_warn (dev,"failed to enable VF %d (%d)\n" , vf, err);continue ; } + + err =mlx5_set_msix_vec_count (dev, vf +1 , num_msix_count); +if (err) { +mlx5_core_warn ( + dev, +"failed to set MSI-X vector counts VF %d, err %d\n" , + vf, err); +continue ; + } + sriov->vfs_ctx[vf].enabled =1 ;if (MLX5_CAP_GEN (dev, port_type) ==MLX5_CAP_PORT_TYPE_IB ) { err =sriov_restore_guids (dev, vf);
Signature · 1 line
2.29.2
PATCH v2 5/5 net/mlx5: Allow to the users to configure number of MSI-X vectors
In reply to Leon RomanovskyFrom: Leon Romanovsky <[email protected]>
Implement ability to configure MSI-X for the SR-IOV VFs.
--- .../net/ethernet/mellanox/mlx5/core/main.c | 1 + .../ethernet/mellanox/mlx5/core/mlx5_core.h | 1 + .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 4 +- .../net/ethernet/mellanox/mlx5/core/sriov.c | 38 +++++++++++++++++++ 4 files changed, 42 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -0
@@ -1647,6 +1647,7 @@ static struct pci_driver mlx5_core_driver = { .shutdown = shutdown, .err_handler = &mlx5_err_handler, .sriov_configure = mlx5_core_sriov_configure, + .sriov_set_msix_vec_count = mlx5_core_sriov_set_msix_vec_count, };static void mlx5_core_verify_params (void )
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +1 -0
@@ -138,6 +138,7 @@ void mlx5_sriov_cleanup(struct mlx5_core_dev *dev);int mlx5_sriov_attach (struct mlx5_core_dev *dev);void mlx5_sriov_detach (struct mlx5_core_dev *dev);int mlx5_core_sriov_configure (struct pci_dev *dev,int num_vfs); +int mlx5_core_sriov_set_msix_vec_count (struct pci_dev *vf,int msix_vec_count);int mlx5_core_enable_hca (struct mlx5_core_dev *dev,u16 func_id);int mlx5_core_disable_hca (struct mlx5_core_dev *dev,u16 func_id); int mlx5_create_scheduling_element_cmd(struct mlx5_core_dev *dev, u8 hierarchy,
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +2 -2
@@ -59,7 +59,7 @@ int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs) {int num_vf_msix, min_msix, max_msix; - num_vf_msix =MLX5_CAP_GEN (dev, num_total_dynamic_vf_msix); + num_vf_msix =MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix);if (!num_vf_msix)return 0 ;
@@ -83,7 +83,7 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,void *hca_cap, *cap;int ret; - num_vf_msix =MLX5_CAP_GEN (dev, num_total_dynamic_vf_msix); + num_vf_msix =MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix);if (!num_vf_msix)return 0 ;
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +38 -0
@@ -145,6 +145,7 @@ mlx5_device_disable_sriov(struct mlx5_core_dev *dev, int num_vfs, bool clear_vf)static int mlx5_sriov_enable (struct pci_dev *pdev,int num_vfs) {struct mlx5_core_dev *dev =pci_get_drvdata (pdev); +u32 num_vf_msix;int err; err =mlx5_device_enable_sriov (dev, num_vfs);
@@ -153,6 +154,8 @@ static int mlx5_sriov_enable(struct pci_dev *pdev, int num_vfs)return err; } + num_vf_msix =MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix); +pci_sriov_set_vf_total_msix (pdev, num_vf_msix); err =pci_enable_sriov (pdev, num_vfs);if (err) {mlx5_core_warn (dev,"pci_enable_sriov failed : %d\n" , err);
@@ -188,6 +191,41 @@ int mlx5_core_sriov_configure(struct pci_dev *pdev, int num_vfs)return err ? err : num_vfs; } +int mlx5_core_sriov_set_msix_vec_count (struct pci_dev *vf,int msix_vec_count) +{ +struct pci_dev *pf =pci_physfn (vf); +struct mlx5_core_sriov *sriov; +struct mlx5_core_dev *dev; +int num_vf_msix, id; + + dev =pci_get_drvdata (pf); + num_vf_msix =MLX5_CAP_GEN_MAX (dev, num_total_dynamic_vf_msix); +if (!num_vf_msix) +return -EOPNOTSUPP ; + +if (!msix_vec_count) + msix_vec_count = +mlx5_get_default_msix_vec_count (dev,pci_num_vf (pf)); + + sriov = &dev->priv.sriov; + +/* Reversed translation of PCI VF function number to the internal +* function_id, which exists in the name of virtfn symlink. +*/ +for (id =0 ; id <pci_num_vf (pf); id++) { +if (!sriov->vfs_ctx[id].enabled) +continue ; + +if (vf->devfn ==pci_iov_virtfn_devfn (pf, id)) +break ; + } + +if (id ==pci_num_vf (pf) || !sriov->vfs_ctx[id].enabled) +return -EINVAL ; + +return mlx5_set_msix_vec_count (dev, id +1 , msix_vec_count); +} +int mlx5_sriov_attach (struct mlx5_core_dev *dev) {if (!mlx5_core_is_pf (dev) || !pci_num_vf (dev->pdev))
Signature · 1 line
2.29.2
Jakub Kicinski
maintainerQuoted 7 lines · On Thu, 14 Jan 2021 12:31:35 +0200 Leon Romanovsky wrote:
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
Quoted 6 lines
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Hi Leon!
Looks like you got some missing kdoc here, check out the test in patchwork so we don't need to worry about this later:
https://patchwork.kernel.org/project/netdevbpf/list/?series=414497
Quoted 16 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 14 Jan 2021 12:31:35 +0200 Leon Romanovsky wrote:
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
Quoted 6 lines
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Quoted 6 lines
Hi Leon!
Looks like you got some missing kdoc here, check out the test in patchwork so we don't need to worry about this later:
https://patchwork.kernel.org/project/netdevbpf/list/?series=414497
Thanks Jakub,
I'll add kdocs to internal mlx5 functions.
IMHO, they are useless.
Thanks
Quoted 22 lines · Leon Romanovsky
Quoted 21 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 14 Jan 2021 12:31:35 +0200 Leon Romanovsky wrote:
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
Quoted 6 lines
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Hi Leon!
Looks like you got some missing kdoc here, check out the test in patchwork so we don't need to worry about this later:
Quoted 6 lines
Thanks Jakub,
I'll add kdocs to internal mlx5 functions.
IMHO, they are useless.
At the end, it looks like CI false alarm.
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'dev' not described in 'mlx5_set_msix_vec_count' drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'function_id' not described in 'mlx5_set_msix_vec_count' drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'msix_vec_count' not described in 'mlx5_set_msix_vec_count' New warnings added
The function mlx5_set_msix_vec_count() is documented.
+/**
+ * mlx5_set_msix_vec_count() - Set dynamically allocated MSI-X to the VF
+ * @dev - PF to work on
+ * @function_id - internal PCI VF function id
+ * @msix_vec_count - Number of MSI-X to set
+ **/
+int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
+ int msix_vec_count)
https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
Thanks
Quoted 2 lines
Thanks
Jakub Kicinski
maintainerQuoted 23 lines · On Sun, 17 Jan 2021 09:24:41 +0200 Leon Romanovsky wrote:
Quoted 22 lines · Leon Romanovsky
Quoted 21 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 14 Jan 2021 12:31:35 +0200 Leon Romanovsky wrote:
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
Quoted 6 lines
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Hi Leon!
Looks like you got some missing kdoc here, check out the test in patchwork so we don't need to worry about this later:
Quoted 6 lines
Quoted 6 lines
Thanks Jakub,
I'll add kdocs to internal mlx5 functions.
IMHO, they are useless.
It's just scripts/kernel-doc, and it's checking if the kdoc is _valid_, your call if you want to add kdoc, just a comment, or nothing at all.
Quoted 16 lines
At the end, it looks like CI false alarm.
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'dev' not described in 'mlx5_set_msix_vec_count'drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'function_id' not described in 'mlx5_set_msix_vec_count'drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'msix_vec_count' not described in 'mlx5_set_msix_vec_count' New warnings addedThe function
mlx5_set_msix_vec_count()is documented.(quoted diff) +8
-0+/** + * mlx5_set_msix_vec_count() - Set dynamically allocated MSI-X to the VF + * @dev - PF to work on + * @function_id - internal PCI VF function id + * @msix_vec_count - Number of MSI-X to set + **/ +int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id, + int msix_vec_count)
AFAIU that's not valid kdoc, I _think_ you need to replace ' -' with ':' for arguments (not my rules).
Quoted 26 lines · Jakub Kicinski
Quoted 25 lines · On Sun, 17 Jan 2021 09:24:41 +0200 Leon Romanovsky wrote:
Quoted 24 lines · Leon Romanovsky
Quoted 22 lines · Jakub Kicinski
Quoted 7 lines · On Thu, 14 Jan 2021 12:31:35 +0200 Leon Romanovsky wrote:
The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.
The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.
Quoted 6 lines
The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).
To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.
Hi Leon!
Looks like you got some missing kdoc here, check out the test in patchwork so we don't need to worry about this later:
https://patchwork.kernel.org/project/netdevbpf/list/?series=414497
Quoted 7 lines
Quoted 4 lines
Quoted 4 lines
Thanks Jakub,
I'll add kdocs to internal mlx5 functions.
IMHO, they are useless.It's just
scripts/kernel-doc, and it's checking if the kdoc is _valid_, your call if you want to add kdoc, just a comment, or nothing at all.
I prefer clean CI, so will add.
Quoted 8 lines
Quoted 8 lines
At the end, it looks like CI false alarm.
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'dev' not described in 'mlx5_set_msix_vec_count'drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'function_id' not described in 'mlx5_set_msix_vec_count'drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c:81: warning: Function parameter or member 'msix_vec_count' not described in 'mlx5_set_msix_vec_count' New warnings added
Quoted 13 lines
Quoted 10 lines
The function
mlx5_set_msix_vec_count()is documented.(quoted diff) +8
-0+/** + * mlx5_set_msix_vec_count() - Set dynamically allocated MSI-X to the VF + * @dev - PF to work on + * @function_id - internal PCI VF function id + * @msix_vec_count - Number of MSI-X to set + **/ +int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id, + int msix_vec_count)https://patchwork.kernel.org/project/netdevbpf/patch/[email protected]/
AFAIU that's not valid kdoc, I _think_ you need to replace ' -' with ':' for arguments (not my rules).
Right, I figured it when submitted v3.
Thanks