Skip to contentSkip to conversation

mlx4_core: module param to limit msix vec allocation

PATCH/RFC Arthur Kepner 3 participants 5 messages Jun 10, 2010 – Jun 17, 2010 2 messages not in the archive

Subsystems (2)
Network drivers InfiniBand
lore
Share & tools
lore

Arthur Kepner

author
#0 lore raw

The mlx4_core driver allocates 'nreq' msix vectors (and irqs), where:

nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,

               num_possible_cpus() + 1);

ConnectX HCAs support 512 event queues (4 reserved). On a system with enough processors, we get:

mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again

Further attempts (by other drivers) to allocate interrupts fail, because mlx4_core got 'em all.

How about this?

Signed-off-by Arthur Kepner <akepner-sJ/[email protected]>

---

 main.c |    8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
drivers/net/mlx4/main.c +7 -1
index e3e0d54..0a316d0 100644
@@ -68,6 +68,10 @@ static int msi_x = 1;
 module_param(msi_x, int, 0444);
 MODULE_PARM_DESC(msi_x, "attempt to use MSI-X if nonzero");
 
+static int max_msi_x_vec = 64;
+module_param(max_msi_x_vec, int, 0444);
+MODULE_PARM_DESC(max_msi_x_vec, "max MSI-X vectors we'll attempt to allocate");
+
 #else /* CONFIG_PCI_MSI */
 
 #define msi_x (0)

@@ -968,8 +972,10 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
 	int i;
 
 	if (msi_x) {
+		nreq = min_t(int, num_possible_cpus() + 1, max_msi_x_vec);
 		nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
-			     num_possible_cpus() + 1);
+			     nreq);
+
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
 			goto no_msi;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20100610165921.GF22247-sJ/[email protected]

#2 lore raw
Quoted 10 lines · Arthur Kepner

The mlx4_core driver allocates 'nreq' msix vectors (and irqs), where:

nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,

               num_possible_cpus() + 1);

ConnectX HCAs support 512 event queues (4 reserved). On a system with enough processors, we get:

Quoted 7 lines

mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again

Further attempts (by other drivers) to allocate interrupts fail, because mlx4_core got 'em all.

How about this?

Hi Arthur,

how many CPU cores are in your system? What kernel version did you use?
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

author
#3 lore raw
Quoted 3 lines · Eli Cohen

....
how many CPU cores are in your system? What kernel version did you use?

I'm almost certain that it was a 2048 core system (it's not available right now for me to verify).

We used 2.6.32.12 (sles11 sp1).

Signature · 5 lines
Arthur
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#4 lore raw
In reply to a message not in the archive
Quoted 8 lines

The mlx4_core driver allocates 'nreq' msix vectors (and irqs), where:

nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,

              num_possible_cpus() + 1);
Quoted 7 lines

ConnectX HCAs support 512 event queues (4 reserved). On a system with enough processors, we get:

mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again

Further attempts (by other drivers) to allocate interrupts fail, because mlx4_core got 'em all.

How about this?

Hi,
I think that this patch would do the job,
Anyway we are thinking of ways to change our interrupt allocation scheme.

--Yevgeny--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): E113D394D7C5DB4F8FF691FA7EE9DB4439A77CCD66-WQlSmcKwN8Te+A/[email protected]

author
#6 lore raw
Quoted 1 line · Yevgeny Petrilin

I think that this patch would do the job,

(Is that an ack?)

Quoted 2 lines

Anyway we are thinking of ways to change our interrupt allocation scheme.

Would be interested to know what you've got in mind.

Signature · 5 lines
Arthur
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html