Skip to contentSkip to conversation

mlx5 updates 2021-06-14

GIT PULL Applied Saeed Mahameed 2 participants 17 messages Jun 15, 2021

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

From: Saeed Mahameed <[email protected]>

Hi Dave, Jakub,

This series provides an IRQ allocation scheme update for Sub-Functions (SFs) Scalability.
For more information please see tag log below.

Please pull and let me know if there is any problem.

Thanks,
Saeed.

---
The following changes since commit 89212e160b81e778f829b89743570665810e3b13:

net: wwan: Fix WWAN config symbols (2021-06-14 13:17:10 -0700)

are available in the Git repository at:

git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git tags/mlx5-updates-2021-06-14

for you to fetch changes up to c36326d38d933199014aba5a17d384cf52e4b558:

net/mlx5: Round-Robin EQs over IRQs (2021-06-14 20:58:00 -0700)

----------------------------------------------------------------
mlx5-updates-2021-06-14

1) Trivial Lag refactroing in preparation for upcomming Single FDB lag feature
- First 3 patches

2) Scalable IRQ distriburion for Sub-functions

A subfunction (SF) is a lightweight function that has a parent PCI function (PF) on which it is deployed.

Currently, mlx5 subfunction is sharing the IRQs (MSI-X) with their parent PCI function.

Before this series the PF allocates enough IRQs to cover
all the cores in a system, Newly created SFs will re-use all the IRQs that the PF has allocated for itself.
Hence, the more SFs are created, there are more EQs per IRQs. Therefore, whenever we handle an interrupt, we need to pull all SFs EQs and PF EQs instead of PF EQs without SFs on the system. This leads to a hard impact on the performance of SFs and PF.

For example, on machine with:
Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz with 56 cores.
PCI Express 3 with BW of 126 Gb/s.
ConnectX-5 Ex; EDR IB (100Gb/s) and 100GbE; dual-port QSFP28; PCIe4.0 x16.

test case: iperf TX BW single CPU, affinity of app and IRQ are the same. PF only: no SFs on the system, 56 IRQs.
SF (before), 250 SFs Sharing the same 56 IRQs .
SF (now), 250 SFs + 255 avaiable IRQs for the NIC. (please see IRQ spread scheme below).

application SF-IRQ channel BW(Gb/sec) interrupts/sec
iperf TX affinity

PF only     cpu={0}     cpu={0} cpu={0}   79                 8200
SF (before) cpu={0}     cpu={0} cpu={0}   51.3 (-35%)        9500
SF (now)    cpu={0}     cpu={0} cpu={0}   78 (-2%)           8200

command:

$ taskset -c 0 iperf -c 11.1.1.1 -P 3 -i 6 -t 30 | grep SUM

The different between the SF examples is that before this series we allocate num_cpus (56) IRQs, and all of them were shared among the PF and the SFs. And after this series, we allocate 255 IRQs, and we spread the SFs among the above IRQs. This have significantly decreased the load on each IRQ and the number of EQs per IRQ is down by 95% (251->11).

In this patchset the solution proposed is to have a dedicated IRQ pool for SFs to use. the pool will allocate a large number of IRQs for SFs to grab from in order to minimize irq sharing between the different SFs.
IRQs will not be requested from the OS until they are 1st requested by an SF consumer, and will be eventually released when the last SF consumer releases them.

For the detailed IRQ spread and allocation scheme please see last patch:
("net/mlx5: Round-Robin EQs over IRQs")

----------------------------------------------------------------
Leon Romanovsky (3):
net/mlx5: Delay IRQ destruction till all users are gone net/mlx5: Provide cpumask at EQ creation phase net/mlx5: Clean license text in eq.[c|h] files

Mark Bloch (3):
net/mlx5: Lag, refactor disable flow net/mlx5: Lag, Don't rescan if the device is going down net/mlx5: Change ownership model for lag

Shay Drory (9):
net/mlx5: Introduce API for request and release IRQs net/mlx5: Removing rmap per IRQ
net/mlx5: Extend mlx5_irq_request to request IRQ from the kernel net/mlx5: Moving rmap logic to EQs
net/mlx5: Change IRQ storage logic from static to dynamic net/mlx5: Allocating a pool of MSI-X vectors for SFs net/mlx5: Enlarge interrupt field in CREATE_EQ net/mlx5: Separate between public and private API of sf.h net/mlx5: Round-Robin EQs over IRQs

 drivers/infiniband/hw/mlx5/odp.c                   |   8 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_main.c  |   4 +-
 drivers/net/ethernet/mellanox/mlx5/core/en_rep.c   |   4 +-
 drivers/net/ethernet/mellanox/mlx5/core/eq.c       | 179 ++++--
 drivers/net/ethernet/mellanox/mlx5/core/lag.c      | 267 +++++----
 drivers/net/ethernet/mellanox/mlx5/core/lag.h      |   3 +-
 drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c   |   2 +-
 drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h   |   3 +-
 drivers/net/ethernet/mellanox/mlx5/core/lib/sf.h   |  45 ++
 drivers/net/ethernet/mellanox/mlx5/core/main.c     |   3 +
 .../net/ethernet/mellanox/mlx5/core/mlx5_core.h    |  25 +-
 drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h |  35 ++
 drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c  | 608 ++++++++++++++-------
 drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h    |  37 +-
 drivers/net/ethernet/mellanox/mlx5/core/sriov.c    |   1 +
 include/linux/mlx5/eq.h                            |   1 +
 include/linux/mlx5/mlx5_ifc.h                      |   4 +-
 17 files changed, 794 insertions(+), 435 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/sf.h
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h
#1 lore raw

PATCH 1/15 net/mlx5: Lag, refactor disable flow

From: Mark Bloch <[email protected]>

When a net device is removed (can happen if the PCI function is unbound from the system) it's not enough to destroy the hardware lag. The system should recreate the original devices that were present before the lag. As the same flow is done when a net device is removed from the bond refactor and reuse the code.

---
 drivers/net/ethernet/mellanox/mlx5/core/lag.c | 40 ++++++++++++-------
 1 file changed, 25 insertions(+), 15 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag.c +25 -15
index 1fb70524d067..6642ff0115f8 100644
@@ -276,6 +276,29 @@ static void mlx5_lag_remove_devices(struct mlx5_lag *ldev)
 	}
 }
 
+static void mlx5_disable_lag(struct mlx5_lag *ldev)
+{
+	struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1].dev;
+	struct mlx5_core_dev *dev1 = ldev->pf[MLX5_LAG_P2].dev;
+	bool roce_lag;
+	int err;
+
+	roce_lag = __mlx5_lag_is_roce(ldev);
+
+	if (roce_lag) {
+		dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
+		mlx5_rescan_drivers_locked(dev0);
+		mlx5_nic_vport_disable_roce(dev1);
+	}
+
+	err = mlx5_deactivate_lag(ldev);
+	if (err)
+		return;
+
+	if (roce_lag)
+		mlx5_lag_add_devices(ldev);
+}
+
 static void mlx5_do_bond(struct mlx5_lag *ldev)
 {
 	struct mlx5_core_dev *dev0 = ldev->pf[MLX5_LAG_P1].dev;
@@ -322,20 +345,7 @@ static void mlx5_do_bond(struct mlx5_lag *ldev)
 	} else if (do_bond && __mlx5_lag_is_active(ldev)) {
 		mlx5_modify_lag(ldev, &tracker);
 	} else if (!do_bond && __mlx5_lag_is_active(ldev)) {
-		roce_lag = __mlx5_lag_is_roce(ldev);
-
-		if (roce_lag) {
-			dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
-			mlx5_rescan_drivers_locked(dev0);
-			mlx5_nic_vport_disable_roce(dev1);
-		}
-
-		err = mlx5_deactivate_lag(ldev);
-		if (err)
-			return;
-
-		if (roce_lag)
-			mlx5_lag_add_devices(ldev);
+		mlx5_disable_lag(ldev);
 	}
 }
 
@@ -620,7 +630,7 @@ void mlx5_lag_remove(struct mlx5_core_dev *dev)
 		return;
 
 	if (__mlx5_lag_is_active(ldev))
-		mlx5_deactivate_lag(ldev);
+		mlx5_disable_lag(ldev);
 
 	mlx5_lag_dev_remove_pf(ldev, dev);
 
Signature · 1 line
2.31.1
#3 lore raw

PATCH 2/15 net/mlx5: Lag, Don't rescan if the device is going down

In reply to Saeed Mahameed

From: Mark Bloch <[email protected]>

If MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV is set it means the device is going down and mlx5_rescan_drivers_locked() shouldn't be called.
With this patch and the previous one in the series, unbinding a PCI function when its netdev is part of a bond works and leaves the system in a working state.

---
 drivers/net/ethernet/mellanox/mlx5/core/lag.c | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/lag.c +8 -2
index 6642ff0115f8..4a4e9b228ba0 100644
@@ -258,6 +258,10 @@ static void mlx5_lag_add_devices(struct mlx5_lag *ldev)
 		if (!ldev->pf[i].dev)
 			continue;
 
+		if (ldev->pf[i].dev->priv.flags &
+		    MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV)
+			continue;
+
 		ldev->pf[i].dev->priv.flags &= ~MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
 		mlx5_rescan_drivers_locked(ldev->pf[i].dev);
 	}
@@ -286,8 +290,10 @@ static void mlx5_disable_lag(struct mlx5_lag *ldev)
 	roce_lag = __mlx5_lag_is_roce(ldev);
 
 	if (roce_lag) {
-		dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
-		mlx5_rescan_drivers_locked(dev0);
+		if (!(dev0->priv.flags & MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV)) {
+			dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
+			mlx5_rescan_drivers_locked(dev0);
+		}
 		mlx5_nic_vport_disable_roce(dev1);
 	}
 
Signature · 1 line
2.31.1
#5 lore raw

PATCH 4/15 net/mlx5: Delay IRQ destruction till all users are gone

In reply to Saeed Mahameed

From: Leon Romanovsky <[email protected]>

Shared IRQ are consumed by multiple EQ users and in order to properly initialize and later release such IRQs, we add kref counting of IRQ structure.

---
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 55 ++++++++++++-------
 1 file changed, 36 insertions(+), 19 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +36 -19
index c3373fb1cd7f..0e65ac3301c5 100644
@@ -16,6 +16,8 @@ struct mlx5_irq {
 	struct atomic_notifier_head nh;
 	cpumask_var_t mask;
 	char name[MLX5_MAX_IRQ_NAME];
+	struct kref kref;
+	int irqn;
 };
 
 struct mlx5_irq_table {
@@ -146,13 +148,35 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
 	return ret;
 }
 
+static void irq_release(struct kref *kref)
+{
+	struct mlx5_irq *irq = container_of(kref, struct mlx5_irq, kref);
+
+	free_irq(irq->irqn, &irq->nh);
+}
+
+static void irq_put(struct mlx5_irq *irq)
+{
+	kref_put(&irq->kref, irq_release);
+}
+
 int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx,
 		       struct notifier_block *nb)
 {
 	struct mlx5_irq *irq;
+	int err;
 
 	irq = &irq_table->irq[vecidx];
-	return atomic_notifier_chain_register(&irq->nh, nb);
+	err = kref_get_unless_zero(&irq->kref);
+	if (WARN_ON_ONCE(!err))
+		/* Something very bad happens here, we are enabling EQ
+		 * on non-existing IRQ.
+		 */
+		return -ENOENT;
+	err = atomic_notifier_chain_register(&irq->nh, nb);
+	if (err)
+		irq_put(irq);
+	return err;
 }
 
 int mlx5_irq_detach_nb(struct mlx5_irq_table *irq_table, int vecidx,
@@ -161,6 +185,7 @@ int mlx5_irq_detach_nb(struct mlx5_irq_table *irq_table, int vecidx,
 	struct mlx5_irq *irq;
 
 	irq = &irq_table->irq[vecidx];
+	irq_put(irq);
 	return atomic_notifier_chain_unregister(&irq->nh, nb);
 }
 
@@ -189,28 +214,26 @@ static int request_irqs(struct mlx5_core_dev *dev, int nvec)
 
 	for (i = 0; i < nvec; i++) {
 		struct mlx5_irq *irq = mlx5_irq_get(dev, i);
-		int irqn = pci_irq_vector(dev->pdev, i);
 
+		irq->irqn = pci_irq_vector(dev->pdev, i);
 		irq_set_name(name, i);
 		ATOMIC_INIT_NOTIFIER_HEAD(&irq->nh);
 		snprintf(irq->name, MLX5_MAX_IRQ_NAME,
 			 "%s@pci:%s", name, pci_name(dev->pdev));
-		err = request_irq(irqn, mlx5_irq_int_handler, 0, irq->name,
+		err = request_irq(irq->irqn, mlx5_irq_int_handler, 0, irq->name,
 				  &irq->nh);
 		if (err) {
 			mlx5_core_err(dev, "Failed to request irq\n");
 			goto err_request_irq;
 		}
+		kref_init(&irq->kref);
 	}
 	return 0;
 
 err_request_irq:
-	while (i--) {
-		struct mlx5_irq *irq = mlx5_irq_get(dev, i);
-		int irqn = pci_irq_vector(dev->pdev, i);
+	while (i--)
+		irq_put(mlx5_irq_get(dev, i));
 
-		free_irq(irqn, &irq->nh);
-	}
 	return  err;
 }
 
@@ -264,10 +287,8 @@ static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
 {
 	int vecidx = MLX5_IRQ_VEC_COMP_BASE + i;
 	struct mlx5_irq *irq;
-	int irqn;
 
 	irq = mlx5_irq_get(mdev, vecidx);
-	irqn = pci_irq_vector(mdev->pdev, vecidx);
 	if (!zalloc_cpumask_var(&irq->mask, GFP_KERNEL)) {
 		mlx5_core_warn(mdev, "zalloc_cpumask_var failed");
 		return -ENOMEM;
@@ -276,9 +297,9 @@ static int set_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
 	cpumask_set_cpu(cpumask_local_spread(i, mdev->priv.numa_node),
 			irq->mask);
 	if (IS_ENABLED(CONFIG_SMP) &&
-	    irq_set_affinity_hint(irqn, irq->mask))
+	    irq_set_affinity_hint(irq->irqn, irq->mask))
 		mlx5_core_warn(mdev, "irq_set_affinity_hint failed, irq 0x%.4x",
-			       irqn);
+			       irq->irqn);
 
 	return 0;
 }
@@ -287,11 +308,9 @@ static void clear_comp_irq_affinity_hint(struct mlx5_core_dev *mdev, int i)
 {
 	int vecidx = MLX5_IRQ_VEC_COMP_BASE + i;
 	struct mlx5_irq *irq;
-	int irqn;
 
 	irq = mlx5_irq_get(mdev, vecidx);
-	irqn = pci_irq_vector(mdev->pdev, vecidx);
-	irq_set_affinity_hint(irqn, NULL);
+	irq_set_affinity_hint(irq->irqn, NULL);
 	free_cpumask_var(irq->mask);
 }
 
@@ -344,8 +363,7 @@ static void unrequest_irqs(struct mlx5_core_dev *dev)
 	int i;
 
 	for (i = 0; i < table->nvec; i++)
-		free_irq(pci_irq_vector(dev->pdev, i),
-			 &mlx5_irq_get(dev, i)->nh);
+		irq_put(mlx5_irq_get(dev, i));
 }
 
 int mlx5_irq_table_create(struct mlx5_core_dev *dev)
@@ -422,8 +440,7 @@ void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)
 	irq_clear_rmap(dev);
 	clear_comp_irqs_affinity_hints(dev);
 	for (i = 0; i < table->nvec; i++)
-		free_irq(pci_irq_vector(dev->pdev, i),
-			 &mlx5_irq_get(dev, i)->nh);
+		irq_release(&mlx5_irq_get(dev, i)->kref);
 	pci_free_irq_vectors(dev->pdev);
 	kfree(table->irq);
 }
Signature · 1 line
2.31.1
#4 lore raw

PATCH 3/15 net/mlx5: Change ownership model for lag

In reply to Saeed Mahameed

From: Mark Bloch <[email protected]>

Lag is used to combine two PCI functions of the same HCA into a single logical unit. This is a core functionality and as such should be managed by the core driver. Currently this isn't the case. While we store the lag software structure inside the lower device, its lifetime (creation / destruction) is dictated by the mlx5e part. Change the ownership model so lag is tied to the lifetime of the lower level driver instead to the mlx5e part.

---
 .../net/ethernet/mellanox/mlx5/core/en_main.c |   4 +-
 .../net/ethernet/mellanox/mlx5/core/en_rep.c  |   4 +-
 drivers/net/ethernet/mellanox/mlx5/core/lag.c | 221 +++++++++++-------
 drivers/net/ethernet/mellanox/mlx5/core/lag.h |   3 +-
 .../net/ethernet/mellanox/mlx5/core/lag_mp.c  |   2 +-
 .../net/ethernet/mellanox/mlx5/core/main.c    |   2 +
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   |   6 +-
 7 files changed, 154 insertions(+), 88 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/en_main.c +2 -2
index 59ee28156603..930b225dfe77 100644
@@ -5114,7 +5114,7 @@ static void mlx5e_nic_enable(struct mlx5e_priv *priv)
 	mlx5e_set_netdev_mtu_boundaries(priv);
 	mlx5e_set_dev_port_mtu(priv);
 
-	mlx5_lag_add(mdev, netdev);
+	mlx5_lag_add_netdev(mdev, netdev);
 
 	mlx5e_enable_async_events(priv);
 	mlx5e_enable_blocking_events(priv);
@@ -5162,7 +5162,7 @@ static void mlx5e_nic_disable(struct mlx5e_priv *priv)
 		priv->en_trap = NULL;
 	}
 	mlx5e_disable_async_events(priv);
-	mlx5_lag_remove(mdev);
+	mlx5_lag_remove_netdev(mdev, priv->netdev);
 	mlx5_vxlan_reset_to_default(mdev->vxlan);
 }
 
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +2 -2
index 8290e0086178..2d2cc5f3b03f 100644
@@ -976,7 +976,7 @@ static void mlx5e_uplink_rep_enable(struct mlx5e_priv *priv)
 	if (MLX5_CAP_GEN(mdev, uplink_follow))
 		mlx5_modify_vport_admin_state(mdev, MLX5_VPORT_STATE_OP_MOD_UPLINK,
 					      0, 0, MLX5_VPORT_ADMIN_STATE_AUTO);
-	mlx5_lag_add(mdev, netdev);
+	mlx5_lag_add_netdev(mdev, netdev);
 	priv->events_nb.notifier_call = uplink_rep_async_event;
 	mlx5_notifier_register(mdev, &priv->events_nb);
 	mlx5e_dcbnl_initialize(priv);
@@ -1009,7 +1009,7 @@ static void mlx5e_uplink_rep_disable(struct mlx5e_priv *priv)
 	mlx5e_dcbnl_delete_app(priv);
 	mlx5_notifier_unregister(mdev, &priv->events_nb);
 	mlx5e_rep_tc_disable(priv);
-	mlx5_lag_remove(mdev);
+	mlx5_lag_remove_netdev(mdev, priv->netdev);
 }
 
 static MLX5E_DEFINE_STATS_GRP(sw_rep, 0);
drivers/net/ethernet/mellanox/mlx5/core/lag.c +141 -80

337 lines not on this page · raw message

drivers/net/ethernet/mellanox/mlx5/core/lag.h +2 -1
index 8d8cf2d0bc6d..191392c37558 100644
@@ -40,6 +40,7 @@ struct lag_tracker {
 struct mlx5_lag {
 	u8                        flags;
 	u8                        v2p_map[MLX5_MAX_PORTS];
+	struct kref               ref;
 	struct lag_func           pf[MLX5_MAX_PORTS];
 	struct lag_tracker        tracker;
 	struct workqueue_struct   *wq;
@@ -49,7 +50,7 @@ struct mlx5_lag {
 };
 
 static inline struct mlx5_lag *
-mlx5_lag_dev_get(struct mlx5_core_dev *dev)
+mlx5_lag_dev(struct mlx5_core_dev *dev)
 {
 	return dev->priv.lag;
 }
drivers/net/ethernet/mellanox/mlx5/core/lag_mp.c +1 -1
index fd6196b5e163..c4bf8b679541 100644
@@ -28,7 +28,7 @@ bool mlx5_lag_is_multipath(struct mlx5_core_dev *dev)
 	struct mlx5_lag *ldev;
 	bool res;
 
-	ldev = mlx5_lag_dev_get(dev);
+	ldev = mlx5_lag_dev(dev);
 	res  = ldev && __mlx5_lag_is_multipath(ldev);
 
 	return res;
drivers/net/ethernet/mellanox/mlx5/core/main.c +2 -0
index a1d67bd7fb43..310518fabf77 100644
@@ -1185,6 +1185,7 @@ static int mlx5_load(struct mlx5_core_dev *dev)
 	}
 
 	mlx5_sf_dev_table_create(dev);
+	mlx5_lag_add_mdev(dev);
 
 	return 0;
 
@@ -1219,6 +1220,7 @@ static int mlx5_load(struct mlx5_core_dev *dev)
 
 static void mlx5_unload(struct mlx5_core_dev *dev)
 {
+	mlx5_lag_remove_mdev(dev);
 	mlx5_sf_dev_table_destroy(dev);
 	mlx5_sriov_detach(dev);
 	mlx5_ec_cleanup(dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +4 -2
index a22b706eebd3..dd95aa6eb2f8 100644
@@ -164,8 +164,10 @@ int mlx5_query_mcam_reg(struct mlx5_core_dev *dev, u32 *mcap, u8 feature_group,
 int mlx5_query_qcam_reg(struct mlx5_core_dev *mdev, u32 *qcam,
 			u8 feature_group, u8 access_reg_group);
 
-void mlx5_lag_add(struct mlx5_core_dev *dev, struct net_device *netdev);
-void mlx5_lag_remove(struct mlx5_core_dev *dev);
+void mlx5_lag_add_netdev(struct mlx5_core_dev *dev, struct net_device *netdev);
+void mlx5_lag_remove_netdev(struct mlx5_core_dev *dev, struct net_device *netdev);
+void mlx5_lag_add_mdev(struct mlx5_core_dev *dev);
+void mlx5_lag_remove_mdev(struct mlx5_core_dev *dev);
 
 int mlx5_irq_table_init(struct mlx5_core_dev *dev);
 void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
Signature · 1 line
2.31.1
#6 lore raw

PATCH 5/15 net/mlx5: Introduce API for request and release IRQs

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

Introduce new API that will allow IRQs users to hold a pointer to mlx5_irq.
In the end of this series, IRQs will be allocated on demand. Hence, this will allow us to properly manage and use IRQs.

---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  | 22 +++++++------
 .../net/ethernet/mellanox/mlx5/core/lib/eq.h  |  1 +
 .../net/ethernet/mellanox/mlx5/core/main.c    |  1 +
 .../ethernet/mellanox/mlx5/core/mlx5_core.h   | 19 ------------
 .../ethernet/mellanox/mlx5/core/mlx5_irq.h    | 30 ++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 31 +++++++++++++------
 .../net/ethernet/mellanox/mlx5/core/sriov.c   |  1 +
 7 files changed, 68 insertions(+), 37 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h
drivers/net/ethernet/mellanox/mlx5/core/eq.c +13 -9
index 77c0ca655975..7e7bbed3763d 100644
@@ -45,6 +45,7 @@
 #include "eswitch.h"
 #include "lib/clock.h"
 #include "diag/fw_tracer.h"
+#include "mlx5_irq.h"
 
 enum {
 	MLX5_EQE_OWNER_INIT_VAL	= 0x1,
@@ -309,13 +310,19 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 	mlx5_init_fbc(eq->frag_buf.frags, log_eq_stride, log_eq_size, &eq->fbc);
 	init_eq_buf(eq);
 
+	eq->irq = mlx5_irq_request(dev, vecidx);
+	if (IS_ERR(eq->irq)) {
+		err = PTR_ERR(eq->irq);
+		goto err_buf;
+	}
+
 	inlen = MLX5_ST_SZ_BYTES(create_eq_in) +
 		MLX5_FLD_SZ_BYTES(create_eq_in, pas[0]) * eq->frag_buf.npages;
 
 	in = kvzalloc(inlen, GFP_KERNEL);
 	if (!in) {
 		err = -ENOMEM;
-		goto err_buf;
+		goto err_irq;
 	}
 
 	pas = (__be64 *)MLX5_ADDR_OF(create_eq_in, in, pas);
@@ -359,6 +366,8 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 err_in:
 	kvfree(in);
 
+err_irq:
+	mlx5_irq_release(eq->irq);
 err_buf:
 	mlx5_frag_buf_free(dev, &eq->frag_buf);
 	return err;
@@ -377,10 +386,9 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 int mlx5_eq_enable(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 		   struct notifier_block *nb)
 {
-	struct mlx5_eq_table *eq_table = dev->priv.eq_table;
 	int err;
 
-	err = mlx5_irq_attach_nb(eq_table->irq_table, eq->vecidx, nb);
+	err = mlx5_irq_attach_nb(eq->irq, nb);
 	if (!err)
 		eq_update_ci(eq, 1);
 
@@ -399,9 +407,7 @@ EXPORT_SYMBOL(mlx5_eq_enable);
 void mlx5_eq_disable(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 		     struct notifier_block *nb)
 {
-	struct mlx5_eq_table *eq_table = dev->priv.eq_table;
-
-	mlx5_irq_detach_nb(eq_table->irq_table, eq->vecidx, nb);
+	mlx5_irq_detach_nb(eq->irq, nb);
 }
 EXPORT_SYMBOL(mlx5_eq_disable);
 
@@ -415,10 +421,9 @@ static int destroy_unmap_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq)
 	if (err)
 		mlx5_core_warn(dev, "failed to destroy a previously created eq: eqn %d\n",
 			       eq->eqn);
-	synchronize_irq(eq->irqn);
+	mlx5_irq_release(eq->irq);
 
 	mlx5_frag_buf_free(dev, &eq->frag_buf);
-
 	return err;
 }
 
@@ -863,7 +868,6 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
 	}
 
 	return 0;
-
 clean:
 	destroy_comp_eqs(dev);
 	return err;
drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h +1 -0
index f607a3858ef5..f618cf95e030 100644
@@ -32,6 +32,7 @@ struct mlx5_eq {
 	unsigned int            irqn;
 	u8                      eqn;
 	struct mlx5_rsc_debug   *dbg;
+	struct mlx5_irq         *irq;
 };
 
 struct mlx5_eq_async {
drivers/net/ethernet/mellanox/mlx5/core/main.c +1 -0
index 310518fabf77..390b1d3a6fde 100644
@@ -76,6 +76,7 @@
 #include "sf/vhca_event.h"
 #include "sf/dev/dev.h"
 #include "sf/sf.h"
+#include "mlx5_irq.h"
 
 MODULE_AUTHOR("Eli Cohen <[email protected]>");
 MODULE_DESCRIPTION("Mellanox 5th generation network adapters (ConnectX series) core driver");
drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h +0 -19
index dd95aa6eb2f8..343807ac2036 100644
@@ -169,25 +169,6 @@ void mlx5_lag_remove_netdev(struct mlx5_core_dev *dev, struct net_device *netdev
 void mlx5_lag_add_mdev(struct mlx5_core_dev *dev);
 void mlx5_lag_remove_mdev(struct mlx5_core_dev *dev);
 
-int mlx5_irq_table_init(struct mlx5_core_dev *dev);
-void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
-int mlx5_irq_table_create(struct mlx5_core_dev *dev);
-void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
-int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx,
-		       struct notifier_block *nb);
-int mlx5_irq_detach_nb(struct mlx5_irq_table *irq_table, int vecidx,
-		       struct notifier_block *nb);
-
-int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
-			    int msix_vec_count);
-int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs);
-
-struct cpumask *
-mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx);
-struct cpu_rmap *mlx5_irq_get_rmap(struct mlx5_irq_table *table);
-int mlx5_irq_get_num_comp(struct mlx5_irq_table *table);
-struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
-
 int mlx5_events_init(struct mlx5_core_dev *dev);
 void mlx5_events_cleanup(struct mlx5_core_dev *dev);
 void mlx5_events_start(struct mlx5_core_dev *dev);
drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h new file +30 -0
new file mode 100644
index 000000000000..dd138b38bf36
@@ -0,0 +1,30 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies. */
+
+#ifndef __MLX5_IRQ_H__
+#define __MLX5_IRQ_H__
+
+#include <linux/mlx5/driver.h>
+
+struct mlx5_irq;
+
+int mlx5_irq_table_init(struct mlx5_core_dev *dev);
+void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
+int mlx5_irq_table_create(struct mlx5_core_dev *dev);
+void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
+struct cpu_rmap *mlx5_irq_get_rmap(struct mlx5_irq_table *table);
+int mlx5_irq_get_num_comp(struct mlx5_irq_table *table);
+struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
+
+int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
+			    int msix_vec_count);
+int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs);
+
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx);
+void mlx5_irq_release(struct mlx5_irq *irq);
+int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
+int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
+struct cpumask *
+mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx);
+
+#endif /* __MLX5_IRQ_H__ */
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +22 -9
index 0e65ac3301c5..ecace7ca4a01 100644
@@ -6,6 +6,7 @@
 #include <linux/module.h>
 #include <linux/mlx5/driver.h>
 #include "mlx5_core.h"
+#include "mlx5_irq.h"
 #ifdef CONFIG_RFS_ACCEL
 #include <linux/cpu_rmap.h>
 #endif
@@ -160,13 +161,10 @@ static void irq_put(struct mlx5_irq *irq)
 	kref_put(&irq->kref, irq_release);
 }
 
-int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx,
-		       struct notifier_block *nb)
+int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb)
 {
-	struct mlx5_irq *irq;
 	int err;
 
-	irq = &irq_table->irq[vecidx];
 	err = kref_get_unless_zero(&irq->kref);
 	if (WARN_ON_ONCE(!err))
 		/* Something very bad happens here, we are enabling EQ
@@ -179,16 +177,31 @@ int mlx5_irq_attach_nb(struct mlx5_irq_table *irq_table, int vecidx,
 	return err;
 }
 
-int mlx5_irq_detach_nb(struct mlx5_irq_table *irq_table, int vecidx,
-		       struct notifier_block *nb)
+int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb)
 {
-	struct mlx5_irq *irq;
-
-	irq = &irq_table->irq[vecidx];
 	irq_put(irq);
 	return atomic_notifier_chain_unregister(&irq->nh, nb);
 }
 
+void mlx5_irq_release(struct mlx5_irq *irq)
+{
+	synchronize_irq(irq->irqn);
+	irq_put(irq);
+}
+
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx)
+{
+	struct mlx5_irq_table *table = mlx5_irq_table_get(dev);
+	struct mlx5_irq *irq = &table->irq[vecidx];
+	int err;
+
+	err = kref_get_unless_zero(&irq->kref);
+	if (!err)
+		return ERR_PTR(-ENOENT);
+
+	return irq;
+}
+
 static irqreturn_t mlx5_irq_int_handler(int irq, void *nh)
 {
 	atomic_notifier_call_chain(nh, 0, NULL);
drivers/net/ethernet/mellanox/mlx5/core/sriov.c +1 -0
index 2338989d4403..e8185b69ac6c 100644
@@ -34,6 +34,7 @@
 #include <linux/mlx5/driver.h>
 #include <linux/mlx5/vport.h>
 #include "mlx5_core.h"
+#include "mlx5_irq.h"
 #include "eswitch.h"
 
 static int sriov_restore_guids(struct mlx5_core_dev *dev, int vf)
Signature · 1 line
2.31.1
#7 lore raw

PATCH 6/15 net/mlx5: Provide cpumask at EQ creation phase

In reply to Saeed Mahameed

From: Leon Romanovsky <[email protected]>

The users of EQ are running their code on different CPUs and with various affinity patterns. Move the cpumask setting close to their actual usage.

---
 drivers/infiniband/hw/mlx5/odp.c              |   5 +
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  |  27 +++--
 .../ethernet/mellanox/mlx5/core/mlx5_irq.h    |   3 +-
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 103 ++++--------------
 include/linux/mlx5/eq.h                       |   1 +
 5 files changed, 49 insertions(+), 90 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +5 -0
index 782b2af8f211..8f88b044ccbc 100644
@@ -1564,7 +1564,12 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq)
 		.nent = MLX5_IB_NUM_PF_EQE,
 	};
 	param.mask[0] = 1ull << MLX5_EVENT_TYPE_PAGE_FAULT;
+	if (!zalloc_cpumask_var(&param.affinity, GFP_KERNEL)) {
+		err = -ENOMEM;
+		goto err_wq;
+	}
 	eq->core = mlx5_eq_create_generic(dev->mdev, &param);
+	free_cpumask_var(param.affinity);
 	if (IS_ERR(eq->core)) {
 		err = PTR_ERR(eq->core);
 		goto err_wq;
drivers/net/ethernet/mellanox/mlx5/core/eq.c +20 -7
index 7e7bbed3763d..5a88887c1a58 100644
@@ -310,7 +310,7 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 	mlx5_init_fbc(eq->frag_buf.frags, log_eq_stride, log_eq_size, &eq->fbc);
 	init_eq_buf(eq);
 
-	eq->irq = mlx5_irq_request(dev, vecidx);
+	eq->irq = mlx5_irq_request(dev, vecidx, param->affinity);
 	if (IS_ERR(eq->irq)) {
 		err = PTR_ERR(eq->irq);
 		goto err_buf;
@@ -621,8 +621,11 @@ setup_async_eq(struct mlx5_core_dev *dev, struct mlx5_eq_async *eq,
 
 	eq->irq_nb.notifier_call = mlx5_eq_async_int;
 	spin_lock_init(&eq->lock);
+	if (!zalloc_cpumask_var(&param->affinity, GFP_KERNEL))
+		return -ENOMEM;
 
 	err = create_async_eq(dev, &eq->core, param);
+	free_cpumask_var(param->affinity);
 	if (err) {
 		mlx5_core_warn(dev, "failed to create %s EQ %d\n", name, err);
 		return err;
@@ -740,6 +743,9 @@ mlx5_eq_create_generic(struct mlx5_core_dev *dev,
 	struct mlx5_eq *eq = kvzalloc(sizeof(*eq), GFP_KERNEL);
 	int err;
 
+	if (!param->affinity)
+		return ERR_PTR(-EINVAL);
+
 	if (!eq)
 		return ERR_PTR(-ENOMEM);
 
@@ -850,16 +856,21 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
 			.irq_index = vecidx,
 			.nent = nent,
 		};
-		err = create_map_eq(dev, &eq->core, &param);
-		if (err) {
-			kfree(eq);
-			goto clean;
+
+		if (!zalloc_cpumask_var(&param.affinity, GFP_KERNEL)) {
+			err = -ENOMEM;
+			goto clean_eq;
 		}
+		cpumask_set_cpu(cpumask_local_spread(i, dev->priv.numa_node),
+				param.affinity);
+		err = create_map_eq(dev, &eq->core, &param);
+		free_cpumask_var(param.affinity);
+		if (err)
+			goto clean_eq;
 		err = mlx5_eq_enable(dev, &eq->core, &eq->irq_nb);
 		if (err) {
 			destroy_unmap_eq(dev, &eq->core);
-			kfree(eq);
-			goto clean;
+			goto clean_eq;
 		}
 
 		mlx5_core_dbg(dev, "allocated completion EQN %d\n", eq->core.eqn);
@@ -868,6 +879,8 @@ static int create_comp_eqs(struct mlx5_core_dev *dev)
 	}
 
 	return 0;
+clean_eq:
+	kfree(eq);
 clean:
 	destroy_comp_eqs(dev);
 	return err;
drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h +2 -1
index dd138b38bf36..81bfb5f0d332 100644
@@ -20,7 +20,8 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
 			    int msix_vec_count);
 int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs);
 
-struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx);
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx,
+				  struct cpumask *affinity);
 void mlx5_irq_release(struct mlx5_irq *irq);
 int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
 int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +21 -82

159 lines not on this page · raw message

include/linux/mlx5/eq.h +1 -0
index e49d8c0d4f26..cea6ecb4b73e 100644
@@ -16,6 +16,7 @@ struct mlx5_eq_param {
 	u8             irq_index;
 	int            nent;
 	u64            mask[4];
+	cpumask_var_t  affinity;
 };
 
 struct mlx5_eq *
Signature · 1 line
2.31.1
#8 lore raw

PATCH 7/15 net/mlx5: Clean license text in eq.[c|h] files

In reply to Saeed Mahameed

From: Leon Romanovsky <[email protected]>

The eq.[c|h] files are under major rewrite. so use this opportunity and update their copyright and license texts.

---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  | 31 ++-----------------
 .../net/ethernet/mellanox/mlx5/core/lib/eq.h  |  2 +-
 2 files changed, 3 insertions(+), 30 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eq.c +2 -29
index 5a88887c1a58..ef0fe499eaed 100644
@@ -1,33 +1,6 @@
+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
 /*
- * Copyright (c) 2013-2015, Mellanox Technologies. All rights reserved.
- *
- * This software is available to you under a choice of one of two
- * licenses.  You may choose to be licensed under the terms of the GNU
- * General Public License (GPL) Version 2, available from the file
- * COPYING in the main directory of this source tree, or the
- * OpenIB.org BSD license below:
- *
- *     Redistribution and use in source and binary forms, with or
- *     without modification, are permitted provided that the following
- *     conditions are met:
- *
- *      - Redistributions of source code must retain the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer.
- *
- *      - Redistributions in binary form must reproduce the above
- *        copyright notice, this list of conditions and the following
- *        disclaimer in the documentation and/or other materials
- *        provided with the distribution.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
+ * Copyright (c) 2013-2021, Mellanox Technologies inc.  All rights reserved.
  */
 
 #include <linux/interrupt.h>
drivers/net/ethernet/mellanox/mlx5/core/lib/eq.h +1 -1
index f618cf95e030..624cedebb510 100644
@@ -1,5 +1,5 @@
 /* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
-/* Copyright (c) 2018 Mellanox Technologies */
+/* Copyright (c) 2018-2021, Mellanox Technologies inc.  All rights reserved. */
 
 #ifndef __LIB_MLX5_EQ_H__
 #define __LIB_MLX5_EQ_H__
Signature · 1 line
2.31.1
#9 lore raw

PATCH 8/15 net/mlx5: Removing rmap per IRQ

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

In next patches, IRQs will be requested according to demand, instead of statically on driver boot.
Also, currently, rmap is managed by the IRQ layer. rmap management will move out from the IRQ layer in future patches.

Therefore, we want to remove the IRQ from the rmap, when IRQ is destroyed, instead of removing all the IRQs from the rmap when irq_table is destroyed.

---
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c   | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +12 -5
index 81b06b5693cd..6a5a6ec0ddbf 100644
@@ -154,8 +154,14 @@ static void irq_release(struct kref *kref)
 {
 	struct mlx5_irq *irq = container_of(kref, struct mlx5_irq, kref);
 
+	/* free_irq requires that affinity and rmap will be cleared
+	 * before calling it. This is why there is asymmetry with set_rmap
+	 * which should be called after alloc_irq but before request_irq.
+	 */
 	irq_set_affinity_hint(irq->irqn, NULL);
 	free_cpumask_var(irq->mask);
+	/* this line is releasing this irq from the rmap */
+	irq_set_affinity_notifier(irq->irqn, NULL);
 	free_irq(irq->irqn, &irq->nh);
 }
 
@@ -378,6 +384,11 @@ int mlx5_irq_table_create(struct mlx5_core_dev *dev)
 	return err;
 }
 
+static void irq_table_clear_rmap(struct mlx5_irq_table *table)
+{
+	cpu_rmap_put(table->rmap);
+}
+
 void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)
 {
 	struct mlx5_irq_table *table = dev->priv.irq_table;
@@ -386,11 +397,7 @@ void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)
 	if (mlx5_core_is_sf(dev))
 		return;
 
-	/* free_irq requires that affinity and rmap will be cleared
-	 * before calling it. This is why there is asymmetry with set_rmap
-	 * which should be called after alloc_irq but before request_irq.
-	 */
-	irq_clear_rmap(dev);
+	irq_table_clear_rmap(table);
 	for (i = 0; i < table->nvec; i++)
 		irq_release(&mlx5_irq_get(dev, i)->kref);
 	pci_free_irq_vectors(dev->pdev);
Signature · 1 line
2.31.1
#10 lore raw

PATCH 9/15 net/mlx5: Extend mlx5_irq_request to request IRQ from the kernel

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

Extend mlx5_irq_request so that IRQs will be requested upon EQ creation, and not on driver boot.

---
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 128 ++++++++----------
 1 file changed, 57 insertions(+), 71 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +57 -71
index 6a5a6ec0ddbf..7d6ca2581532 100644
@@ -17,7 +17,6 @@ struct mlx5_irq {
 	struct atomic_notifier_head nh;
 	cpumask_var_t mask;
 	char name[MLX5_MAX_IRQ_NAME];
-	spinlock_t lock; /* protects affinity assignment */
 	struct kref kref;
 	int irqn;
 };
@@ -60,7 +59,7 @@ int mlx5_irq_get_num_comp(struct mlx5_irq_table *table)
 
 static struct mlx5_irq *mlx5_irq_get(struct mlx5_core_dev *dev, int vecidx)
 {
-	struct mlx5_irq_table *irq_table = dev->priv.irq_table;
+	struct mlx5_irq_table *irq_table = mlx5_irq_table_get(dev);
 
 	return &irq_table->irq[vecidx];
 }
@@ -192,37 +191,7 @@ int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb)
 	return atomic_notifier_chain_unregister(&irq->nh, nb);
 }
 
-void mlx5_irq_release(struct mlx5_irq *irq)
-{
-	synchronize_irq(irq->irqn);
-	irq_put(irq);
-}
-
-struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx,
-				  struct cpumask *affinity)
-{
-	struct mlx5_irq_table *table = mlx5_irq_table_get(dev);
-	struct mlx5_irq *irq = &table->irq[vecidx];
-	int err;
-
-	err = kref_get_unless_zero(&irq->kref);
-	if (!err)
-		return ERR_PTR(-ENOENT);
-
-	spin_lock(&irq->lock);
-	if (!cpumask_empty(irq->mask)) {
-		/* already configured */
-		spin_unlock(&irq->lock);
-		return irq;
-	}
-
-	cpumask_copy(irq->mask, affinity);
-	irq_set_affinity_hint(irq->irqn, irq->mask);
-	spin_unlock(&irq->lock);
-	return irq;
-}
-
-static irqreturn_t mlx5_irq_int_handler(int irq, void *nh)
+static irqreturn_t irq_int_handler(int irq, void *nh)
 {
 	atomic_notifier_call_chain(nh, 0, NULL);
 	return IRQ_HANDLED;
@@ -230,7 +199,7 @@ static irqreturn_t mlx5_irq_int_handler(int irq, void *nh)
 
 static void irq_set_name(char *name, int vecidx)
 {
-	if (vecidx == 0) {
+	if (!vecidx) {
 		snprintf(name, MLX5_MAX_IRQ_NAME, "mlx5_async");
 		return;
 	}
@@ -239,41 +208,67 @@ static void irq_set_name(char *name, int vecidx)
 		 vecidx - MLX5_IRQ_VEC_COMP_BASE);
 }
 
-static int request_irqs(struct mlx5_core_dev *dev, int nvec)
+static int irq_request(struct mlx5_core_dev *dev, int i)
 {
+	struct mlx5_irq *irq = mlx5_irq_get(dev, i);
 	char name[MLX5_MAX_IRQ_NAME];
 	int err;
-	int i;
-
-	for (i = 0; i < nvec; i++) {
-		struct mlx5_irq *irq = mlx5_irq_get(dev, i);
-
-		irq->irqn = pci_irq_vector(dev->pdev, i);
-		irq_set_name(name, i);
-		ATOMIC_INIT_NOTIFIER_HEAD(&irq->nh);
-		snprintf(irq->name, MLX5_MAX_IRQ_NAME,
-			 "%s@pci:%s", name, pci_name(dev->pdev));
-		err = request_irq(irq->irqn, mlx5_irq_int_handler, 0, irq->name,
-				  &irq->nh);
-		if (err) {
-			mlx5_core_err(dev, "Failed to request irq\n");
-			goto err_request_irq;
-		}
-		if (!zalloc_cpumask_var(&irq->mask, GFP_KERNEL)) {
-			mlx5_core_warn(dev, "zalloc_cpumask_var failed\n");
-			err = -ENOMEM;
-			goto err_request_irq;
-		}
-		spin_lock_init(&irq->lock);
-		kref_init(&irq->kref);
+
+	irq->irqn = pci_irq_vector(dev->pdev, i);
+	irq_set_name(name, i);
+	ATOMIC_INIT_NOTIFIER_HEAD(&irq->nh);
+	snprintf(irq->name, MLX5_MAX_IRQ_NAME,
+		 "%s@pci:%s", name, pci_name(dev->pdev));
+	err = request_irq(irq->irqn, irq_int_handler, 0, irq->name,
+			  &irq->nh);
+	if (err) {
+		mlx5_core_err(dev, "Failed to request irq. err = %d\n", err);
+		return err;
 	}
+	if (!zalloc_cpumask_var(&irq->mask, GFP_KERNEL)) {
+		mlx5_core_warn(dev, "zalloc_cpumask_var failed\n");
+		free_irq(irq->irqn, &irq->nh);
+		return -ENOMEM;
+	}
+	kref_init(&irq->kref);
 	return 0;
+}
 
-err_request_irq:
-	while (i--)
-		irq_put(mlx5_irq_get(dev, i));
+/**
+ * mlx5_irq_release - release an IRQ back to the system.
+ * @irq: irq to be released.
+ */
+void mlx5_irq_release(struct mlx5_irq *irq)
+{
+	synchronize_irq(irq->irqn);
+	irq_put(irq);
+}
 
-	return  err;
+/**
+ * mlx5_irq_request - request an IRQ for mlx5 device.
+ * @dev: mlx5 device that requesting the IRQ.
+ * @vecidx: vector index of the IRQ. This argument is ignore if affinity is
+ * provided.
+ * @affinity: cpumask requested for this IRQ.
+ *
+ * This function returns a pointer to IRQ, or ERR_PTR in case of error.
+ */
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx,
+				  struct cpumask *affinity)
+{
+	struct mlx5_irq_table *table = mlx5_irq_table_get(dev);
+	struct mlx5_irq *irq = &table->irq[vecidx];
+	int ret;
+
+	ret = kref_get_unless_zero(&irq->kref);
+	if (ret)
+		return irq;
+	ret = irq_request(dev, vecidx);
+	if (ret)
+		return ERR_PTR(ret);
+	cpumask_copy(irq->mask, affinity);
+	irq_set_affinity_hint(irq->irqn, irq->mask);
+	return irq;
 }
 
 static void irq_clear_rmap(struct mlx5_core_dev *dev)
@@ -369,14 +364,8 @@ int mlx5_irq_table_create(struct mlx5_core_dev *dev)
 	if (err)
 		goto err_set_rmap;
 
-	err = request_irqs(dev, nvec);
-	if (err)
-		goto err_request_irqs;
-
 	return 0;
 
-err_request_irqs:
-	irq_clear_rmap(dev);
 err_set_rmap:
 	pci_free_irq_vectors(dev->pdev);
 err_free_irq:
@@ -392,14 +381,11 @@ static void irq_table_clear_rmap(struct mlx5_irq_table *table)
 void mlx5_irq_table_destroy(struct mlx5_core_dev *dev)
 {
 	struct mlx5_irq_table *table = dev->priv.irq_table;
-	int i;
 
 	if (mlx5_core_is_sf(dev))
 		return;
 
 	irq_table_clear_rmap(table);
-	for (i = 0; i < table->nvec; i++)
-		irq_release(&mlx5_irq_get(dev, i)->kref);
 	pci_free_irq_vectors(dev->pdev);
 	kfree(table->irq);
 }
Signature · 1 line
2.31.1
#11 lore raw

PATCH 10/15 net/mlx5: Moving rmap logic to EQs

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

IRQs are being simplified in order to ease their sharing and any feature specific object will be moved to upper layer.
Hence we move rmap object into eq_table.

---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  | 67 ++++++++++++++-
 .../ethernet/mellanox/mlx5/core/mlx5_irq.h    |  1 -
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 84 +++----------------
 3 files changed, 78 insertions(+), 74 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eq.c +66 -1
index ef0fe499eaed..898ae3d47f20 100644
@@ -58,6 +58,9 @@ struct mlx5_eq_table {
 	struct mutex            lock; /* sync async eqs creations */
 	int			num_comp_eqs;
 	struct mlx5_irq_table	*irq_table;
+#ifdef CONFIG_RFS_ACCEL
+	struct cpu_rmap		*rmap;
+#endif
 };
 
 #define MLX5_ASYNC_EVENT_MASK ((1ull << MLX5_EVENT_TYPE_PATH_MIG)	    | \
@@ -899,7 +902,7 @@ EXPORT_SYMBOL(mlx5_comp_irq_get_affinity_mask);
 #ifdef CONFIG_RFS_ACCEL
 struct cpu_rmap *mlx5_eq_table_get_rmap(struct mlx5_core_dev *dev)
 {
-	return mlx5_irq_get_rmap(dev->priv.eq_table->irq_table);
+	return dev->priv.eq_table->rmap;
 }
 #endif
 
@@ -916,12 +919,57 @@ struct mlx5_eq_comp *mlx5_eqn2comp_eq(struct mlx5_core_dev *dev, int eqn)
 	return ERR_PTR(-ENOENT);
 }
 
+static void clear_rmap(struct mlx5_core_dev *dev)
+{
+#ifdef CONFIG_RFS_ACCEL
+	struct mlx5_eq_table *eq_table = dev->priv.eq_table;
+
+	free_irq_cpu_rmap(eq_table->rmap);
+#endif
+}
+
+static int set_rmap(struct mlx5_core_dev *mdev)
+{
+	int err = 0;
+#ifdef CONFIG_RFS_ACCEL
+	struct mlx5_eq_table *eq_table = mdev->priv.eq_table;
+	int vecidx;
+
+	eq_table->rmap = alloc_irq_cpu_rmap(eq_table->num_comp_eqs);
+	if (!eq_table->rmap) {
+		err = -ENOMEM;
+		mlx5_core_err(mdev, "Failed to allocate cpu_rmap. err %d", err);
+		goto err_out;
+	}
+
+	vecidx = MLX5_IRQ_VEC_COMP_BASE;
+	for (; vecidx < eq_table->num_comp_eqs + MLX5_IRQ_VEC_COMP_BASE;
+	     vecidx++) {
+		err = irq_cpu_rmap_add(eq_table->rmap,
+				       pci_irq_vector(mdev->pdev, vecidx));
+		if (err) {
+			mlx5_core_err(mdev, "irq_cpu_rmap_add failed. err %d",
+				      err);
+			goto err_irq_cpu_rmap_add;
+		}
+	}
+	return 0;
+
+err_irq_cpu_rmap_add:
+	clear_rmap(mdev);
+err_out:
+#endif
+	return err;
+}
+
 /* This function should only be called after mlx5_cmd_force_teardown_hca */
 void mlx5_core_eq_free_irqs(struct mlx5_core_dev *dev)
 {
 	struct mlx5_eq_table *table = dev->priv.eq_table;
 
 	mutex_lock(&table->lock); /* sync with create/destroy_async_eq */
+	if (!mlx5_core_is_sf(dev))
+		clear_rmap(dev);
 	mlx5_irq_table_destroy(dev);
 	mutex_unlock(&table->lock);
 }
@@ -951,6 +999,18 @@ int mlx5_eq_table_create(struct mlx5_core_dev *dev)
 		goto err_async_eqs;
 	}
 
+	if (!mlx5_core_is_sf(dev)) {
+		/* rmap is a mapping between irq number and queue number.
+		 * each irq can be assign only to a single rmap.
+		 * since SFs share IRQs, rmap mapping cannot function correctly
+		 * for irqs that are shared for different core/netdev RX rings.
+		 * Hence we don't allow netdev rmap for SFs
+		 */
+		err = set_rmap(dev);
+		if (err)
+			goto err_rmap;
+	}
+
 	err = create_comp_eqs(dev);
 	if (err) {
 		mlx5_core_err(dev, "Failed to create completion EQs\n");
@@ -959,6 +1019,9 @@ int mlx5_eq_table_create(struct mlx5_core_dev *dev)
 
 	return 0;
 err_comp_eqs:
+	if (!mlx5_core_is_sf(dev))
+		clear_rmap(dev);
+err_rmap:
 	destroy_async_eqs(dev);
 err_async_eqs:
 	return err;
@@ -966,6 +1029,8 @@ int mlx5_eq_table_create(struct mlx5_core_dev *dev)
 
 void mlx5_eq_table_destroy(struct mlx5_core_dev *dev)
 {
+	if (!mlx5_core_is_sf(dev))
+		clear_rmap(dev);
 	destroy_comp_eqs(dev);
 	destroy_async_eqs(dev);
 }
drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h +0 -1
index 81bfb5f0d332..d4be79884cb4 100644
@@ -12,7 +12,6 @@ int mlx5_irq_table_init(struct mlx5_core_dev *dev);
 void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
 int mlx5_irq_table_create(struct mlx5_core_dev *dev);
 void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
-struct cpu_rmap *mlx5_irq_get_rmap(struct mlx5_irq_table *table);
 int mlx5_irq_get_num_comp(struct mlx5_irq_table *table);
 struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
 
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +12 -72

148 lines not on this page · raw message

Signature · 1 line
2.31.1
#13 lore raw

PATCH 12/15 net/mlx5: Allocating a pool of MSI-X vectors for SFs

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

SFs (Sub Functions) currently use IRQs from the global IRQ table their parent Physical Function have. In order to better scale, we need to allocate more IRQs and share them between different SFs.

Driver will maintain 3 separated irq pools:
1. A pool that serve the PF consumer (PF's netdev, rdma stacks), similar to what the driver had before this patch. i.e, this pool will share irqs between rdma and netev, and will keep the irq indexes and allocation order. The last is important for PF netdev rmap (aRFS).

2. A pool of control IRQs for SFs. The size of this pool is the number of SFs that can be created divided by SFS_PER_IRQ. This pool will serve the control path EQs of the SFs.

3. A pool of completion data path IRQs for SFs transport queues. The size of this pool is:
num_irqs_allocated - pf_pool_size - sf_ctrl_pool_size. This pool will served netdev and rdma stacks. Moreover, rmap is not supported on SFs.

Sharing methodology of the SFs pools is explained in the next patch.

Important note: rmap is not supported on SFs because rmap mapping cannot function correctly for IRQs that are shared for different core/netdev RX rings.

---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  |  12 +-
 .../ethernet/mellanox/mlx5/core/mlx5_irq.h    |   6 +-
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 292 ++++++++++++------
 3 files changed, 209 insertions(+), 101 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eq.c +4 -8
index 96649dbcef39..b8ac9f58d2b5 100644
@@ -471,14 +471,7 @@ static int create_async_eq(struct mlx5_core_dev *dev,
 	int err;
 
 	mutex_lock(&eq_table->lock);
-	/* Async EQs must share irq index 0 */
-	if (param->irq_index != 0) {
-		err = -EINVAL;
-		goto unlock;
-	}
-
 	err = create_map_eq(dev, eq, param);
-unlock:
 	mutex_unlock(&eq_table->lock);
 	return err;
 }
@@ -996,8 +989,11 @@ int mlx5_eq_table_create(struct mlx5_core_dev *dev)
 
 	eq_table->num_comp_eqs =
 		min_t(int,
-		      mlx5_irq_get_num_comp(eq_table->irq_table),
+		      mlx5_irq_table_get_num_comp(eq_table->irq_table),
 		      num_eqs - MLX5_MAX_ASYNC_EQS);
+	if (mlx5_core_is_sf(dev))
+		eq_table->num_comp_eqs = min_t(int, eq_table->num_comp_eqs,
+					       MLX5_COMP_EQS_PER_SF);
 
 	err = create_async_eqs(dev);
 	if (err) {
drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h +5 -1
index 63b33cd37f7c..48656e8624a9 100644
@@ -6,13 +6,17 @@
 
 #include <linux/mlx5/driver.h>
 
+#define MLX5_COMP_EQS_PER_SF 8
+
+#define MLX5_IRQ_EQ_CTRL (0)
+
 struct mlx5_irq;
 
 int mlx5_irq_table_init(struct mlx5_core_dev *dev);
 void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
 int mlx5_irq_table_create(struct mlx5_core_dev *dev);
 void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
-int mlx5_irq_get_num_comp(struct mlx5_irq_table *table);
+int mlx5_irq_table_get_num_comp(struct mlx5_irq_table *table);
 struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
 
 int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +200 -92

416 lines not on this page · raw message

Signature · 1 line
2.31.1
#12 lore raw

PATCH 11/15 net/mlx5: Change IRQ storage logic from static to dynamic

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

Store newly created IRQs in the xarray DB instead of a static array, so we will be able to store only IRQs which are being used.

---
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  | 12 ++-
 .../ethernet/mellanox/mlx5/core/mlx5_irq.h    |  3 +-
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 79 +++++++++++--------
 3 files changed, 58 insertions(+), 36 deletions(-)
drivers/net/ethernet/mellanox/mlx5/core/eq.c +9 -3
index 898ae3d47f20..96649dbcef39 100644
@@ -892,10 +892,16 @@ EXPORT_SYMBOL(mlx5_comp_vectors_count);
 struct cpumask *
 mlx5_comp_irq_get_affinity_mask(struct mlx5_core_dev *dev, int vector)
 {
-	int vecidx = vector + MLX5_IRQ_VEC_COMP_BASE;
+	struct mlx5_eq_table *table = dev->priv.eq_table;
+	struct mlx5_eq_comp *eq, *n;
+	int i = 0;
+
+	list_for_each_entry_safe(eq, n, &table->comp_eqs_list, list) {
+		if (i++ == vector)
+			break;
+	}
 
-	return mlx5_irq_get_affinity_mask(dev->priv.eq_table->irq_table,
-					  vecidx);
+	return mlx5_irq_get_affinity_mask(eq->core.irq);
 }
 EXPORT_SYMBOL(mlx5_comp_irq_get_affinity_mask);
 
drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h +1 -2
index d4be79884cb4..63b33cd37f7c 100644
@@ -24,7 +24,6 @@ struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx,
 void mlx5_irq_release(struct mlx5_irq *irq);
 int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
 int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
-struct cpumask *
-mlx5_irq_get_affinity_mask(struct mlx5_irq_table *irq_table, int vecidx);
+struct cpumask *mlx5_irq_get_affinity_mask(struct mlx5_irq *irq);
 
 #endif /* __MLX5_IRQ_H__ */
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +48 -31

170 lines not on this page · raw message

Signature · 1 line
2.31.1
#15 lore raw

PATCH 14/15 net/mlx5: Separate between public and private API of sf.h

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

Move mlx5_sf_max_functions() and friends from the privete sf/sf.h to the public lib/sf.h. This is done in order to have one direction include paths.

---
 .../net/ethernet/mellanox/mlx5/core/lib/sf.h  | 45 +++++++++++++++++++
 .../net/ethernet/mellanox/mlx5/core/sf/sf.h   | 37 +--------------
 2 files changed, 46 insertions(+), 36 deletions(-)
 create mode 100644 drivers/net/ethernet/mellanox/mlx5/core/lib/sf.h
drivers/net/ethernet/mellanox/mlx5/core/lib/sf.h new file +45 -0
new file mode 100644
index 000000000000..84e5683861be
@@ -0,0 +1,45 @@
+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
+/* Copyright (c) 2021 Mellanox Technologies Ltd */
+
+#ifndef __LIB_MLX5_SF_H__
+#define __LIB_MLX5_SF_H__
+
+#include <linux/mlx5/driver.h>
+
+static inline u16 mlx5_sf_start_function_id(const struct mlx5_core_dev *dev)
+{
+	return MLX5_CAP_GEN(dev, sf_base_id);
+}
+
+#ifdef CONFIG_MLX5_SF
+
+static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
+{
+	return MLX5_CAP_GEN(dev, sf);
+}
+
+static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
+{
+	if (!mlx5_sf_supported(dev))
+		return 0;
+	if (MLX5_CAP_GEN(dev, max_num_sf))
+		return MLX5_CAP_GEN(dev, max_num_sf);
+	else
+		return 1 << MLX5_CAP_GEN(dev, log_max_sf);
+}
+
+#else
+
+static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
+{
+	return false;
+}
+
+static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
+{
+	return 0;
+}
+
+#endif
+
+#endif
drivers/net/ethernet/mellanox/mlx5/core/sf/sf.h +1 -36
index 0b6aea1e6a94..81ce13b19ee8 100644
@@ -5,42 +5,7 @@
 #define __MLX5_SF_H__
 
 #include <linux/mlx5/driver.h>
-
-static inline u16 mlx5_sf_start_function_id(const struct mlx5_core_dev *dev)
-{
-	return MLX5_CAP_GEN(dev, sf_base_id);
-}
-
-#ifdef CONFIG_MLX5_SF
-
-static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
-{
-	return MLX5_CAP_GEN(dev, sf);
-}
-
-static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
-{
-	if (!mlx5_sf_supported(dev))
-		return 0;
-	if (MLX5_CAP_GEN(dev, max_num_sf))
-		return MLX5_CAP_GEN(dev, max_num_sf);
-	else
-		return 1 << MLX5_CAP_GEN(dev, log_max_sf);
-}
-
-#else
-
-static inline bool mlx5_sf_supported(const struct mlx5_core_dev *dev)
-{
-	return false;
-}
-
-static inline u16 mlx5_sf_max_functions(const struct mlx5_core_dev *dev)
-{
-	return 0;
-}
-
-#endif
+#include "lib/sf.h"
 
 #ifdef CONFIG_MLX5_SF_MANAGER
 
Signature · 1 line
2.31.1
#14 lore raw

PATCH 13/15 net/mlx5: Enlarge interrupt field in CREATE_EQ

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

FW is now supporting more than 256 MSI-X per PF (up to 2K). Hence, enlarge interrupt field in CREATE_EQ to make use of the new MSI-X's.

---
 include/linux/mlx5/mlx5_ifc.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
include/linux/mlx5/mlx5_ifc.h +2 -2
index 057db0eaf195..2d1ed78289ff 100644
@@ -3806,8 +3806,8 @@ struct mlx5_ifc_eqc_bits {
 
 	u8         reserved_at_80[0x20];
 
-	u8         reserved_at_a0[0x18];
-	u8         intr[0x8];
+	u8         reserved_at_a0[0x14];
+	u8         intr[0xc];
 
 	u8         reserved_at_c0[0x3];
 	u8         log_page_size[0x5];
Signature · 1 line
2.31.1
#16 lore raw

PATCH 15/15 net/mlx5: Round-Robin EQs over IRQs

In reply to Saeed Mahameed

From: Shay Drory <[email protected]>

Whenever users provided affinity for an EQ creation request, map the EQ to a matching IRQ.
Matching IRQ=IRQ with the same affinity and type (completion/control) of the EQ created.

This mapping is being done in agressive dedicated IRQ allocation scheme, which described bellow.

First, we check whether there is a matching IRQ that his min threshold is not exhausted.
- min_eqs_threshold = 3 for control EQ.
- min_eqs_threshold = 1 for completion EQ.
In case no matching IRQ was found, try to request a new IRQ. In case we can't request a new IRQ, reuse least-used matching IRQ.

---
 drivers/infiniband/hw/mlx5/odp.c              |   3 +-
 drivers/net/ethernet/mellanox/mlx5/core/eq.c  |  14 +-
 .../ethernet/mellanox/mlx5/core/mlx5_irq.h    |   4 +-
 .../net/ethernet/mellanox/mlx5/core/pci_irq.c | 197 ++++++++++++++++--
 4 files changed, 189 insertions(+), 29 deletions(-)
drivers/infiniband/hw/mlx5/odp.c +1 -2
index 8f88b044ccbc..1338c11fd121 100644
@@ -1559,8 +1559,7 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq)
 	}
 
 	eq->irq_nb.notifier_call = mlx5_ib_eq_pf_int;
-	param = (struct mlx5_eq_param){
-		.irq_index = 0,
+	param = (struct mlx5_eq_param) {
 		.nent = MLX5_IB_NUM_PF_EQE,
 	};
 	param.mask[0] = 1ull << MLX5_EVENT_TYPE_PAGE_FAULT;
drivers/net/ethernet/mellanox/mlx5/core/eq.c +8 -6
index b8ac9f58d2b5..7e5b3826eae5 100644
@@ -263,7 +263,7 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 	u32 out[MLX5_ST_SZ_DW(create_eq_out)] = {0};
 	u8 log_eq_stride = ilog2(MLX5_EQE_SIZE);
 	struct mlx5_priv *priv = &dev->priv;
-	u8 vecidx = param->irq_index;
+	u16 vecidx = param->irq_index;
 	__be64 *pas;
 	void *eqc;
 	int inlen;
@@ -292,6 +292,7 @@ create_map_eq(struct mlx5_core_dev *dev, struct mlx5_eq *eq,
 		goto err_buf;
 	}
 
+	vecidx = mlx5_irq_get_index(eq->irq);
 	inlen = MLX5_ST_SZ_BYTES(create_eq_in) +
 		MLX5_FLD_SZ_BYTES(create_eq_in, pas[0]) * eq->frag_buf.npages;
 
@@ -629,7 +630,6 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
 	mlx5_eq_notifier_register(dev, &table->cq_err_nb);
 
 	param = (struct mlx5_eq_param) {
-		.irq_index = 0,
 		.nent = MLX5_NUM_CMD_EQE,
 		.mask[0] = 1ull << MLX5_EVENT_TYPE_CMD,
 	};
@@ -642,7 +642,6 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
 	mlx5_cmd_allowed_opcode(dev, CMD_ALLOWED_OPCODE_ALL);
 
 	param = (struct mlx5_eq_param) {
-		.irq_index = 0,
 		.nent = MLX5_NUM_ASYNC_EQE,
 	};
 
@@ -652,7 +651,6 @@ static int create_async_eqs(struct mlx5_core_dev *dev)
 		goto err2;
 
 	param = (struct mlx5_eq_param) {
-		.irq_index = 0,
 		.nent = /* TODO: sriov max_vf + */ 1,
 		.mask[0] = 1ull << MLX5_EVENT_TYPE_PAGE_REQUEST,
 	};
@@ -985,15 +983,19 @@ int mlx5_eq_table_create(struct mlx5_core_dev *dev)
 	int num_eqs = MLX5_CAP_GEN(dev, max_num_eqs) ?
 		      MLX5_CAP_GEN(dev, max_num_eqs) :
 		      1 << MLX5_CAP_GEN(dev, log_max_eq);
+	int max_eqs_sf;
 	int err;
 
 	eq_table->num_comp_eqs =
 		min_t(int,
 		      mlx5_irq_table_get_num_comp(eq_table->irq_table),
 		      num_eqs - MLX5_MAX_ASYNC_EQS);
-	if (mlx5_core_is_sf(dev))
+	if (mlx5_core_is_sf(dev)) {
+		max_eqs_sf = min_t(int, MLX5_COMP_EQS_PER_SF,
+				   mlx5_irq_table_get_sfs_vec(eq_table->irq_table));
 		eq_table->num_comp_eqs = min_t(int, eq_table->num_comp_eqs,
-					       MLX5_COMP_EQS_PER_SF);
+					       max_eqs_sf);
+	}
 
 	err = create_async_eqs(dev);
 	if (err) {
drivers/net/ethernet/mellanox/mlx5/core/mlx5_irq.h +3 -1
index 48656e8624a9..abd024173c42 100644
@@ -17,17 +17,19 @@ void mlx5_irq_table_cleanup(struct mlx5_core_dev *dev);
 int mlx5_irq_table_create(struct mlx5_core_dev *dev);
 void mlx5_irq_table_destroy(struct mlx5_core_dev *dev);
 int mlx5_irq_table_get_num_comp(struct mlx5_irq_table *table);
+int mlx5_irq_table_get_sfs_vec(struct mlx5_irq_table *table);
 struct mlx5_irq_table *mlx5_irq_table_get(struct mlx5_core_dev *dev);
 
 int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int devfn,
 			    int msix_vec_count);
 int mlx5_get_default_msix_vec_count(struct mlx5_core_dev *dev, int num_vfs);
 
-struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, int vecidx,
+struct mlx5_irq *mlx5_irq_request(struct mlx5_core_dev *dev, u16 vecidx,
 				  struct cpumask *affinity);
 void mlx5_irq_release(struct mlx5_irq *irq);
 int mlx5_irq_attach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
 int mlx5_irq_detach_nb(struct mlx5_irq *irq, struct notifier_block *nb);
 struct cpumask *mlx5_irq_get_affinity_mask(struct mlx5_irq *irq);
+int mlx5_irq_get_index(struct mlx5_irq *irq);
 
 #endif /* __MLX5_IRQ_H__ */
drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c +177 -20

290 lines not on this page · raw message

Signature · 1 line
2.31.1
#2 lore raw

Re: [net-next 01/15] net/mlx5: Lag, refactor disable flow

In reply to Saeed Mahameed
This series was applied to netdev/net-next.git (refs/heads/master):

Hello:

This series was applied to netdev/net-next.git (refs/heads/master):

Quoted 9 lines · On Mon, 14 Jun 2021 21:01:09 -0700 you wrote:

From: Mark Bloch <[email protected]>

When a net device is removed (can happen if the PCI function is unbound from the system) it's not enough to destroy the hardware lag. The system should recreate the original devices that were present before the lag. As the same flow is done when a net device is removed from the bond refactor and reuse the code.

[...]

Here is the summary with links:
- [net-next,01/15] net/mlx5: Lag, refactor disable flow

    https://git.kernel.org/netdev/net-next/c/8c22ad36eefa
  - [net-next,02/15] net/mlx5: Lag, Don't rescan if the device is going down
    https://git.kernel.org/netdev/net-next/c/8ed19471fdaa
  - [net-next,03/15] net/mlx5: Change ownership model for lag
    https://git.kernel.org/netdev/net-next/c/8a66e4585979
  - [net-next,04/15] net/mlx5: Delay IRQ destruction till all users are gone
    https://git.kernel.org/netdev/net-next/c/c38421abcf21
  - [net-next,05/15] net/mlx5: Introduce API for request and release IRQs
    https://git.kernel.org/netdev/net-next/c/3b43190b2f25
  - [net-next,06/15] net/mlx5: Provide cpumask at EQ creation phase
    https://git.kernel.org/netdev/net-next/c/e4e3f24b822f
  - [net-next,07/15] net/mlx5: Clean license text in eq.[c|h] files
    https://git.kernel.org/netdev/net-next/c/652e3581f248
  - [net-next,08/15] net/mlx5: Removing rmap per IRQ
    https://git.kernel.org/netdev/net-next/c/2de61538377c
  - [net-next,09/15] net/mlx5: Extend mlx5_irq_request to request IRQ from the kernel
    https://git.kernel.org/netdev/net-next/c/e8abebb3a48e
  - [net-next,10/15] net/mlx5: Moving rmap logic to EQs
    https://git.kernel.org/netdev/net-next/c/2d74524c0106
  - [net-next,11/15] net/mlx5: Change IRQ storage logic from static to dynamic
    https://git.kernel.org/netdev/net-next/c/fc63dd2a85be
  - [net-next,12/15] net/mlx5: Allocating a pool of MSI-X vectors for SFs
    https://git.kernel.org/netdev/net-next/c/71e084e26414
  - [net-next,13/15] net/mlx5: Enlarge interrupt field in CREATE_EQ
    https://git.kernel.org/netdev/net-next/c/3af26495a247
  - [net-next,14/15] net/mlx5: Separate between public and private API of sf.h
    https://git.kernel.org/netdev/net-next/c/c8ea212bfdff
  - [net-next,15/15] net/mlx5: Round-Robin EQs over IRQs
    https://git.kernel.org/netdev/net-next/c/c36326d38d93

You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html