Skip to contentSkip to conversation

mlx4_core: request MSIX vectors as much as there CPU cores

PATCH In review Eli Cohen 5 participants 7 messages May 5, 2010 – May 6, 2010 6 messages not in the archive

Subsystems (2)
lore
Share & tools
lore

Eli Cohen

author
#0 lore raw

PATCH 1/1 mlx4_core: request MSIX vectors as much as there CPU cores

The current code requires num_possible_cpus() + 1 MSIX vectors. However, num_possible_cpus() stands for the max number of supported CPUs by the kernel. We should use num_online_cpus() which is the number of available CPUs for the system.

---
 drivers/net/mlx4/main.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/mlx4/main.c +1 -1
index e3e0d54..0559df4 100644
@@ -969,7 +969,7 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)
 
 	if (msi_x) {
 		nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
-			     num_possible_cpus() + 1);
+			     num_online_cpus() + 1);
 		entries = kcalloc(nreq, sizeof *entries, GFP_KERNEL);
 		if (!entries)
 			goto no_msi;
Signature · 6 lines
1.7.1

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20100505113047.GA12242-8YAHvHwT2UEvbXDkjdHOrw/[email protected]

#2 lore raw
Quoted 2 lines

The current code requires num_possible_cpus() + 1 MSIX vectors. However, num_possible_cpus() stands for the max number of supported CPUs by the kernel.

I believe this is wrong -- num_possible_cpus() is the maximum number of CPUs for the running system, including hotplug.

Quoted 2 lines

We should use num_online_cpus() which is the number of available CPUs for the system.

But then we don't have vectors for hotplugged CPUs.

- R.

Signature · 6 lines
Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adahbmmpfep.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#4 lore raw
Quoted 5 lines · Roland Dreier
Quoted 2 lines

The current code requires num_possible_cpus() + 1 MSIX vectors. However, num_possible_cpus() stands for the max number of supported CPUs by the kernel.

I believe this is wrong -- num_possible_cpus() is the maximum number of CPUs for the running system, including hotplug.

FWIW, I've seen some systems running RH kernels that blow up here because they run out of interrupt numbers if you have two IB cards in them.

Mainline kernels have a reworked IRQ number allocator and don't have a problem, so maybe this is an ofed only patch?

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 20100505163252.GG15969-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org

#6 lore raw
Quoted 8 lines · Jason Gunthorpe
Quoted 6 lines · Roland Dreier
Quoted 2 lines

The current code requires num_possible_cpus() + 1 MSIX vectors. However, num_possible_cpus() stands for the max number of supported CPUs by the kernel.

I believe this is wrong -- num_possible_cpus() is the maximum number of CPUs for the running system, including hotplug.

Quoted 7 lines

FWIW, I've seen some systems running RH kernels that blow up here because they run out of interrupt numbers if you have two IB cards in them.

Mainline kernels have a reworked IRQ number allocator and don't have a problem, so maybe this is an ofed only patch?

This is not related to OFED
We found it in performance work of our EN (10G) driver

Tziporet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): 4BE1C82F.4000009-VPRAkNaXOzVS1MOuV/[email protected]

#8 lore raw
Quoted 1 line

We found it in performance work of our EN (10G) driver

What does this have to do with performance?

Do you have a system where num_possible_cpus() differs significantly from num_online_cpus()? What kernel is that with? As far as I can tell, for x86 they should only be different if there genuinely are CPUs that are available for hotplug.

- R.

Signature · 6 lines
Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): adaaasejfew.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#10 lore raw
Quoted 1 line

We found it in performance work of our EN (10G) driver

By the way, it would certainly make sense for the ethernet driver to use a number of queues that matches num_online_cpus() at the time the interface is brought up. Since we can't change the # of MSI-X vectors very easily I think we need to allow for the possible CPUs, but bouncing a net interface seems lighter weight to me.

Although perhaps reloading a driver on CPU hotplug is OK too?

- R.

Signature · 2 lines
Roland Dreier <[email protected]> || For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/index.html

A message that is not in the archive (sent off-list or not yet received): ada6332jf79.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org

#12 lore raw
Quoted 10 lines · Roland Dreier
Quoted 1 line

We found it in performance work of our EN (10G) driver

By the way, it would certainly make sense for the ethernet driver to use a number of queues that matches num_online_cpus() at the time the interface is brought up. Since we can't change the # of MSI-X vectors very easily I think we need to allow for the possible CPUs, but bouncing a net interface seems lighter weight to me.

Although perhaps reloading a driver on CPU hotplug is OK too?

Yes, we have a system where num_possible_cpus is 32 and num_online_cpus is 16. It's a RH5.4 and the kernel has no problem allocating 33 MSI-X vectors. The point is that using more than one EQ per CPU core does not buy us anything; in fact it can contiribute to a higher rate of interrupts since the same EQ serves less CQs and the chances for coalescing EQEs are lower.

So what do you think about the following patch to mlx4_en:

drivers/net/mlx4/en_cq.c +2 -1
index 21786ad..07c0779 100644
@@ -49,11 +49,12 @@ int mlx4_en_create_cq(struct mlx4_en_priv *priv,
 {
 	struct mlx4_en_dev *mdev = priv->mdev;
 	int err;
+	int num_active_vectors = min_t(int, num_online_cpus(), mdev->dev->caps.num_comp_vectors);
 
 	cq->size = entries;
 	if (mode == RX) {
 		cq->buf_size = cq->size * sizeof(struct mlx4_cqe);
-		cq->vector   = ring % mdev->dev->caps.num_comp_vectors;
+		cq->vector   = ring % num_active_vectors;
 	} else {
 		cq->buf_size = sizeof(struct mlx4_cqe);
 		cq->vector   = 0;

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html