Skip to content

Dynamically assign MSI-X vectors count

Leon Romanovsky linux-pci linux-rdma netdev 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest4 patches 12 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/4PCI: Configure number of MSI-X vectors for SR-IOV VFsDocumentation/ABI/testing/sysfs-bus-pci drivers/pci/iov.c drivers/pci/msi.c +3 more+113 -011 replies
2/4net/mlx5: Add dynamic MSI-X capabilities bitsinclude/linux/mlx5/mlx5_ifc.h+10 -10 replies
3/4net/mlx5: Dynamically assign MSI-X vectors countdrivers/net/ethernet/mellanox/mlx5/core/main.c drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +1 more+83 -20 replies
4/4net/mlx5: Allow to the users to configure number of MSI-X vectorsdrivers/net/ethernet/mellanox/mlx5/core/main.c drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +1 more+39 -20 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH mlx5-next 0/4] Dynamically assign MSI-X vectors count

From: Leon Romanovsky <[email protected]>

Hi,

The number of MSI-X vectors is PCI property visible through lspci, that field is read-only and configured by the device.

The static assignment of an amount of MSI-X vectors doesn't allow utilize the newly created VF because it is not known to the device the future load and configuration where that VF will be used.

The VFs are created on the hypervisor and forwarded to the VMs that have different properties (for example number of CPUs).

To overcome the inefficiency in the spread of such MSI-X vectors, we allow the kernel to instruct the device with the needed number of such vectors, before VF is initialized and bounded to the driver.

Before this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=12 Masked-

Configuration script:
1. Start fresh
echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs modprobe -q -r mlx5_ib mlx5_core
2. Ensure that driver doesn't run and it is safe to change MSI-X echo 0 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_drivers_autoprobe
3. Load driver for the PF
modprobe mlx5_core
4. Configure one of the VFs with new number
echo 2 > /sys/bus/pci/devices/0000\:08\:00.0/sriov_numvfs echo 21 > /sys/bus/pci/devices/0000\:08\:00.2/vf_msix_vec

After this series:
[root@server ~]# lspci -vs 0000:08:00.2
08:00.2 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5 Virtual Function]
....
Capabilities: [9c] MSI-X: Enable- Count=21 Masked-

Thanks

Leon Romanovsky (4):
PCI: Configure number of MSI-X vectors for SR-IOV VFs net/mlx5: Add dynamic MSI-X capabilities bits net/mlx5: Dynamically assign MSI-X vectors count net/mlx5: Allow to the users to configure number of MSI-X vectors

 Documentation/ABI/testing/sysfs-bus-pci       | 16 +++++
 .../net/ethernet/mellanox/mlx5/core/main.c    |  5 ++
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |  6 ++
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 62 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/sriov.c   | 49 ++++++++++++++-
 drivers/pci/iov.c                             | 57 +++++++++++++++++
 drivers/pci/msi.c                             | 30 +++++++++
 drivers/pci/pci-sysfs.c                       |  1 +
 drivers/pci/pci.h                             |  1 +
 include/linux/mlx5/mlx5_ifc.h                 | 11 +++-
 include/linux/pci.h                           |  8 +++
 11 files changed, 243 insertions(+), 3 deletions(-)

--
2.29.2