mlx4_core: module param to limit msix vec allocation
Subsystems (2)
PATCH 1/1 mlx4_core: module param to limit msix vec allocation
(Resending to netdev. Previously sent to linux-rdma)
The mlx4_core driver allocates 'nreq' msix vectors (and irqs), where:
nreq = min_t(int, dev->caps.num_eqs - dev->caps.reserved_eqs,
num_possible_cpus() + 1);
ConnectX HCAs support 512 event queues (4 reserved). On a system with enough processors, we get:
mlx4_core 0006:01:00.0: Requested 508 vectors, but only 256 MSI-X vectors available, trying again
Further attempts (by other drivers) to allocate interrupts fail, because mlx4_core got 'em all.
Use a module parameter to limit the number of MSI-X vectors that the mlx4_core driver will attempt to allocate.
---
drivers/net/mlx4/main.c +7 -1
@@ -68,6 +68,10 @@ static int msi_x = 1;module_param (msi_x, int,0444 );MODULE_PARM_DESC (msi_x,"attempt to use MSI-X if nonzero" ); +static int max_msi_x_vec =64 ; +module_param (max_msi_x_vec, int,0444 ); +MODULE_PARM_DESC (max_msi_x_vec,"max MSI-X vectors we'll attempt to allocate" ); +#else /* CONFIG_PCI_MSI */ #define msi_x (0)
@@ -968,8 +972,10 @@ static void mlx4_enable_msi_x(struct mlx4_dev *dev)int i;if (msi_x) { + nreq =min_t (int,num_possible_cpus () +1 , max_msi_x_vec); nreq =min_t (int, dev->caps.num_eqs - dev->caps.reserved_eqs, -num_possible_cpus () +1 ); + nreq); + entries =kcalloc (nreq,sizeof *entries,GFP_KERNEL );if (!entries) goto no_msi;
David Miller
maintainerFrom: Arthur Kepner <[email protected]>
Date: Thu, 26 Aug 2010 10:06:46 -0700
Quoted 4 lines
Use a module parameter to limit the number of MSI-X vectors that the
mlx4_coredriver will attempt to allocate.Signed-off-by Arthur Kepner <[email protected]>
And what the heck happens if every single device driver tries to handle this problem in the same way? It makes no sense to make the user specify N special module parameters just to get a working system using MSI-X.
This must be fixed in a centralized way that allows coordination between devices which compete for MSI-X vector resources, rather than in a way that is specific to each and every device type.
I am not applying this patch, sorry.
Quoted 10 lines · David Miller
....
And what the heck happens if every single device driver tries to handle this problem in the same way? It makes no sense to make the user specify N special module parameters just to get a working system using MSI-X.This must be fixed in a centralized way that allows coordination between devices which compete for MSI-X vector resources, rather than in a way that is specific to each and every device type.
OK, how about this. Each time the kernel assigns a default affinity for an interrupt, it generates a netlink message, and a user-level interrupt balancer can decide whether and how to reassign the interrupt?
I have that running on a (very small) system and it seems to work OK. Will send out the patch in just a minute.
Signature · 1 line
Arthur
David Miller
maintainerFrom: Arthur Kepner <[email protected]>
Date: Fri, 3 Sep 2010 13:30:44 -0700
Quoted 2 lines
I have that running on a (very small) system and it seems to work OK. Will send out the patch in just a minute.
Failure vs. non-failure of allocation of a kernel managed physical resource can't be determined by a userland process which may or may not be running.
In fact, if I build all of my drivers built-in and use NFS root which will activate and bring up network devices, userland won't even be present when the interrupts are requested.
No, you really can't do it this way, IRQ allocation management has to be in the kernel.
Quoted 9 lines · David Miller
From: Arthur Kepner <[email protected]>
Date: Fri, 3 Sep 2010 13:30:44 -0700Quoted 2 lines
I have that running on a (very small) system and it seems to work OK. Will send out the patch in just a minute.
Failure vs. non-failure of allocation of a kernel managed physical resource can't be determined by a userland process which may or may not be running.
But even if there's no user process running, it's no worse than what we've got now.
Quoted 4 lines
In fact, if I build all of my drivers built-in and use NFS root which will activate and bring up network devices, userland won't even be present when the interrupts are requested.
Yep. This only works once the user-level irq balancer is available.
Quoted 3 lines
No, you really can't do it this way, IRQ allocation management has to be in the kernel.
Really? I was specifically trying to avoid that, and let the policy about interrupt assignment be done in a user process.
Do you have any specific ideas about how that'd look?
Signature · 1 line
Arthur
David Miller
maintainerFrom: Arthur Kepner <[email protected]>
Date: Fri, 3 Sep 2010 15:13:03 -0700
Quoted 8 lines · David Miller
Quoted 2 lines
No, you really can't do it this way, IRQ allocation management has to be in the kernel.
Really? I was specifically trying to avoid that, and let the policy about interrupt assignment be done in a user process.
Do you have any specific ideas about how that'd look?
Likely there would be a set of policies just like the cpu power management layer we have. And there would be a default, and userland could override the default.
This also means there would have to be notifications sent to drivers when the distribution and allocation policy of MSI-X interrupts is changed. This is so that, for example, network drivers can reconfigure their network queues for the new set of MSI-X vectors available to them after the policy change.
Quoted 4 lines · David Miller
...
Likely there would be a set of policies just like the cpu power management layer we have. And there would be a default, and userland could override the default.
Thanks. I had thought this over again, and come up with something like what you describe. Testing now, and plan to send a work-in-progress patch later today for comments.
Quoted 6 lines
This also means there would have to be notifications sent to drivers when the distribution and allocation policy of MSI-X interrupts is changed. This is so that, for example, network drivers can reconfigure their network queues for the new set of MSI-X vectors available to them after the policy change.
Hadn't considered this. Will have to add it.
Signature · 1 line
Arthur
David Miller
maintainerFrom: Arthur Kepner <[email protected]>
Date: Mon, 6 Sep 2010 10:42:45 -0700
Quoted 8 lines · David Miller
Quoted 5 lines
This also means there would have to be notifications sent to drivers when the distribution and allocation policy of MSI-X interrupts is changed. This is so that, for example, network drivers can reconfigure their network queues for the new set of MSI-X vectors available to them after the policy change.
Hadn't considered this. Will have to add it.
BTW, reading this again made me consider another case that needs to invoke a notifier.
Let's say that a policy specifies an even distribution of MSI-X allocation amongst all present devices behind a PCI domain. Well, if a PCI-E device hot-plugs you'll have to do notifications for that too because a card might now be able to get more of the MSI-X vectors it wants.