mlx4_core: request MSIX vectors as much as there CPU cores
Subsystems (2)
PATCH 1/1 mlx4_core: request MSIX vectors as much as there CPU cores
The current code requires num_possible_cpus() + 1 MSIX vectors. However, num_possible_cpus() stands for the max number of supported CPUs by the kernel. We should use num_online_cpus() which is the number of available CPUs for the system.
--- drivers/net/mlx4/main.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
drivers/net/mlx4/main.c +1 -1
@@ -969,7 +969,7 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)if (msi_x) { nreq =min_t (int, dev->caps.num_eqs - dev->caps.reserved_eqs, -num_possible_cpus () +1 ); +num_online_cpus () +1 ); entries =kcalloc (nreq,sizeof *entries,GFP_KERNEL );if (!entries) goto no_msi;
Signature · 6 lines
1.7.1 -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20100505113047.GA12242-8YAHvHwT2UEvbXDkjdHOrw/[email protected]
Quoted 2 lines
The current code requires
num_possible_cpus()+ 1 MSIX vectors. However,num_possible_cpus()stands for the max number of supported CPUs by the kernel.
I believe this is wrong -- num_possible_cpus() is the maximum number of CPUs for the running system, including hotplug.
Quoted 2 lines
We should use
num_online_cpus()which is the number of available CPUs for the system.
But then we don't have vectors for hotplugged CPUs.
- R.
Signature · 6 lines
Roland Dreier <[email protected]> || For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): adahbmmpfep.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 5 lines · Roland Dreier
Quoted 2 lines
The current code requires
num_possible_cpus()+ 1 MSIX vectors. However,num_possible_cpus()stands for the max number of supported CPUs by the kernel.I believe this is wrong --
num_possible_cpus()is the maximum number of CPUs for the running system, including hotplug.
FWIW, I've seen some systems running RH kernels that blow up here because they run out of interrupt numbers if you have two IB cards in them.
Mainline kernels have a reworked IRQ number allocator and don't have a problem, so maybe this is an ofed only patch?
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 20100505163252.GG15969-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org
Quoted 8 lines · Jason Gunthorpe
Quoted 6 lines · Roland Dreier
Quoted 2 lines
The current code requires
num_possible_cpus()+ 1 MSIX vectors. However,num_possible_cpus()stands for the max number of supported CPUs by the kernel.I believe this is wrong --
num_possible_cpus()is the maximum number of CPUs for the running system, including hotplug.
Quoted 7 lines
FWIW, I've seen some systems running RH kernels that blow up here because they run out of interrupt numbers if you have two IB cards in them.
Mainline kernels have a reworked IRQ number allocator and don't have a problem, so maybe this is an ofed only patch?
This is not related to OFED
We found it in performance work of our EN (10G) driver
Tziporet
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): 4BE1C82F.4000009-VPRAkNaXOzVS1MOuV/[email protected]
Quoted 1 line
We found it in performance work of our EN (10G) driver
What does this have to do with performance?
Do you have a system where num_possible_cpus() differs significantly from num_online_cpus()? What kernel is that with? As far as I can tell, for x86 they should only be different if there genuinely are CPUs that are available for hotplug.
- R.
Signature · 6 lines
Roland Dreier <[email protected]> || For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
A message that is not in the archive (sent off-list or not yet received): adaaasejfew.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 1 line
We found it in performance work of our EN (10G) driver
By the way, it would certainly make sense for the ethernet driver to use a number of queues that matches num_online_cpus() at the time the interface is brought up. Since we can't change the # of MSI-X vectors very easily I think we need to allow for the possible CPUs, but bouncing a net interface seems lighter weight to me.
Although perhaps reloading a driver on CPU hotplug is OK too?
- R.
Signature · 2 lines
Roland Dreier <[email protected]> || For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/index.html
A message that is not in the archive (sent off-list or not yet received): ada6332jf79.fsf-BjVyx320WGW9gfZ95n9DRSW4+XlvGpQz@public.gmane.org
Quoted 10 lines · Roland Dreier
Quoted 1 line
We found it in performance work of our EN (10G) driver
By the way, it would certainly make sense for the ethernet driver to use a number of queues that matches
num_online_cpus()at the time the interface is brought up. Since we can't change the # of MSI-X vectors very easily I think we need to allow for the possible CPUs, but bouncing a net interface seems lighter weight to me.Although perhaps reloading a driver on CPU hotplug is OK too?
Yes, we have a system where num_possible_cpus is 32 and num_online_cpus is 16. It's a RH5.4 and the kernel has no problem allocating 33 MSI-X vectors. The point is that using more than one EQ per CPU core does not buy us anything; in fact it can contiribute to a higher rate of interrupts since the same EQ serves less CQs and the chances for coalescing EQEs are lower.
So what do you think about the following patch to mlx4_en:
drivers/net/mlx4/en_cq.c +2 -1
@@ -49,11 +49,12 @@ int mlx4_en_create_cq(struct mlx4_en_priv *priv, {struct mlx4_en_dev *mdev = priv->mdev;int err; +int num_active_vectors =min_t (int,num_online_cpus (), mdev->dev->caps.num_comp_vectors); cq->size = entries;if (mode ==RX ) { cq->buf_size = cq->size *sizeof (struct mlx4_cqe ); - cq->vector = ring % mdev->dev->caps.num_comp_vectors; + cq->vector = ring % num_active_vectors; }else { cq->buf_size =sizeof (struct mlx4_cqe ); cq->vector =0 ;
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html