be2net: Adding an option to use INTx instead of MSI-X

PATCH Applied Somnath Kotur 6 participants 14 messages Oct 25, 2010 – Nov 3, 2010

Subsystems (2)

Networking Network drivers

lore

Share & tools

lore

Somnath Kotur

author

2010-10-25 11:12 UTC #0 lore raw

PATCH 1/2 be2net: Adding an option to use INTx instead of MSI-X

By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.

Signed-off-by Somnath Kotur <[email protected]>

---
 drivers/net/benet/be_main.c |   12 +++++++++++-
 1 files changed, 11 insertions(+), 1 deletions(-)

drivers/net/benet/be_main.c +11 -1

index 45b1f66..1262292 100644
@@ -27,10 +27,13 @@ MODULE_LICENSE("GPL");
 
 static unsigned int rx_frag_size = 2048;
 static unsigned int num_vfs;
+static unsigned int msix = 1;
 module_param(rx_frag_size, uint, S_IRUGO);
 module_param(num_vfs, uint, S_IRUGO);
+module_param(msix, uint, S_IRUGO);
 MODULE_PARM_DESC(rx_frag_size, "Size of a fragment that holds rcvd data.");
 MODULE_PARM_DESC(num_vfs, "Number of PCI VFs to initialize");
+MODULE_PARM_DESC(msix, "Enable/Disable the MSIx (MSIx enabled by default)");
 
 static bool multi_rxq = true;
 module_param(multi_rxq, bool, S_IRUGO | S_IWUSR);

@@ -2856,7 +2859,8 @@ static int __devinit be_probe(struct pci_dev *pdev,
 	if (status)
 		goto stats_clean;
 
-	be_msix_enable(adapter);
+	if (msix)
+		be_msix_enable(adapter);
 
 	INIT_DELAYED_WORK(&adapter->work, be_worker);

@@ -3082,6 +3086,12 @@ static int __init be_init_module(void)
 		num_vfs = 32;
 	}
 
+	if (!msix && num_vfs > 0) {
+		printk(KERN_WARNING DRV_NAME
+			" : MSIx required for num_vfs > 0. Ignoring msix=0\n");
+		msix = 1;
+	}
+
 	return pci_register_driver(&be_driver);
 }
 module_init(be_init_module);

Signature · 1 line

1.5.6.1

David Miller

maintainer

2010-10-25 19:09 UTC #1 lore raw

From: Somnath Kotur <[email protected]>
Date: Mon, 25 Oct 2010 16:42:35 +0530

Quoted 4 lines

By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.
Signed-off-by Somnath Kotur <[email protected]>

Either add a new ethtool flag, or use the PCI subsystem facilities for tweaking things to implement this.

Do not use a module option, otherwise every other networking driver author will get the same "cool" idea, give the module option different names, and the resulting user experience is terrible.

Ben Hutchings

2010-10-25 22:38 UTC #2 lore raw

Quoted 6 lines · David Miller

From: Somnath Kotur <[email protected]>
Date: Mon, 25 Oct 2010 16:42:35 +0530
Quoted 3 lines
By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.

Quoted 8 lines

Quoted 1 line
Signed-off-by Somnath Kotur <[email protected]>
Either add a new ethtool flag, or use the PCI subsystem facilities for tweaking things to implement this.
Do not use a module option, otherwise every other networking driver author will get the same "cool" idea, give the module option different names, and the resulting user experience is terrible.

This has already happened, sadly. So far as I can see it's mostly done to allow users to work around systems with broken MSIs; I'm not aware of any other reason to prefer legacy interrupts. However, the PCI subsystem already implements a blacklist and a kernel parameter for disabling MSIs on these systems.

Ben.

Signature · 3 lines

Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

David Miller

maintainer

2010-10-25 23:25 UTC #3 lore raw

From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100

Quoted 16 lines

Quoted 6 lines · David Miller
From: Somnath Kotur <[email protected]>
Date: Mon, 25 Oct 2010 16:42:35 +0530
Quoted 3 lines
By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.
Quoted 8 lines
Quoted 1 line
Signed-off-by Somnath Kotur <[email protected]>
Either add a new ethtool flag, or use the PCI subsystem facilities for tweaking things to implement this.
Do not use a module option, otherwise every other networking driver author will get the same "cool" idea, give the module option different names, and the resulting user experience is terrible.

Quoted 5 lines

This has already happened, sadly. So far as I can see it's mostly done to allow users to work around systems with broken MSIs; I'm not aware of any other reason to prefer legacy interrupts. However, the PCI subsystem already implements a blacklist and a kernel parameter for disabling MSIs on these systems.

The PCI subsystem bits I'm totally fine with.

But in the drivers themselves, that's what I don't want.

[email protected]

author

2010-10-26 05:27 UTC #4 lore raw

Dave,
Could you pls ignore this patch and apply the next patch in the series ?
( PATCH 2/2] be2net: Schedule/Destroy worker thread in probe()/remove() rather than open()/close()

Thanks
Som
________________________________________
From: David Miller [[email protected]]
Sent: Tuesday, October 26, 2010 4:55 AM
To: [email protected]
Cc: Kotur, Somnath; [email protected]; [email protected]
Subject: Re: [PATCH net-next-2.6 1/2] be2net: Adding an option to use INTx instead of MSI-X

From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100

Quoted 16 lines

Quoted 6 lines · David Miller
From: Somnath Kotur <[email protected]>
Date: Mon, 25 Oct 2010 16:42:35 +0530
Quoted 3 lines
By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.
Quoted 8 lines
Quoted 1 line
Signed-off-by Somnath Kotur <[email protected]>
Either add a new ethtool flag, or use the PCI subsystem facilities for tweaking things to implement this.
Do not use a module option, otherwise every other networking driver author will get the same "cool" idea, give the module option different names, and the resulting user experience is terrible.

Quoted 5 lines

This has already happened, sadly. So far as I can see it's mostly done to allow users to work around systems with broken MSIs; I'm not aware of any other reason to prefer legacy interrupts. However, the PCI subsystem already implements a blacklist and a kernel parameter for disabling MSIs on these systems.

The PCI subsystem bits I'm totally fine with.

But in the drivers themselves, that's what I don't want.

Michael Ellerman

2010-10-26 06:52 UTC #5 lore raw

In reply to David Miller

Quoted 25 lines · David Miller

From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100
Quoted 16 lines
Quoted 6 lines · David Miller
From: Somnath Kotur <[email protected]>
Date: Mon, 25 Oct 2010 16:42:35 +0530
Quoted 3 lines
By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.
Quoted 8 lines
Quoted 1 line
Signed-off-by Somnath Kotur <[email protected]>
Either add a new ethtool flag, or use the PCI subsystem facilities for tweaking things to implement this.
Do not use a module option, otherwise every other networking driver author will get the same "cool" idea, give the module option different names, and the resulting user experience is terrible.
Quoted 5 lines
This has already happened, sadly. So far as I can see it's mostly done to allow users to work around systems with broken MSIs; I'm not aware of any other reason to prefer legacy interrupts. However, the PCI subsystem already implements a blacklist and a kernel parameter for disabling MSIs on these systems.

Quoted 3 lines

The PCI subsystem bits I'm totally fine with.
But in the drivers themselves, that's what I don't want.

That horse has really really bolted, it's gawn.

I count 26 drivers with "disable MSI/X" parameters. Some even have more than one.

11 of them are network drivers, 9 scsi, 3 ata.

I agree it's a mess for users, but it's probably preferable to a non-working driver.

Ethtool would be nice, but only for network drivers. Is there a generic solution, quirks are obviously not keeping people happy.

cheers

MSI:

Param "msi", 10 users:
drivers/ata/sata_mv.c:MODULE_PARM_DESC(msi, "Enable use of PCI MSI (0=off, 1=on)"); drivers/ata/sata_nv.c:MODULE_PARM_DESC(msi, "Enable use of MSI (Default: false)"); drivers/ata/sata_sil24.c:MODULE_PARM_DESC(msi, "Enable MSI (Default: false)"); drivers/net/cxgb3/cxgb3_main.c:MODULE_PARM_DESC(msi, "whether to use MSI or MSI-X"); drivers/net/cxgb4/cxgb4_main.c:MODULE_PARM_DESC(msi, "whether to use INTx (0), MSI (1) or MSI-X (2)"); drivers/net/cxgb4vf/cxgb4vf_main.c:MODULE_PARM_DESC(msi, "whether to use MSI-X or MSI"); drivers/net/forcedeth.c:MODULE_PARM_DESC(msi, "MSI interrupts are enabled by setting to 1 and disabled by setting to 0."); drivers/net/qla3xxx.c:MODULE_PARM_DESC(msi, "Turn on Message Signaled Interrupts."); drivers/scsi/aacraid/aachba.c:MODULE_PARM_DESC(msi, "IRQ handling." drivers/scsi/stex.c:MODULE_PARM_DESC(msi, "Enable Message Signaled Interrupts(0=off, 1=on)");

Param "use_msi", 4 users:
drivers/net/qlcnic/qlcnic_main.c:MODULE_PARM_DESC(use_msi, "MSI interrupt (0=disabled, 1=enabled"); drivers/scsi/3w-9xxx.c:MODULE_PARM_DESC(use_msi, "Use Message Signaled Interrupts. Default: 0"); drivers/scsi/3w-sas.c:MODULE_PARM_DESC(use_msi, "Use Message Signaled Interrupts. Default: 0"); drivers/scsi/aic94xx/aic94xx_init.c:MODULE_PARM_DESC(use_msi, "\n"

Param "disable_msi", 4 users:
drivers/scsi/vmw_pvscsi.c:MODULE_PARM_DESC(disable_msi, "Disable MSI use in driver - (default=0)"); drivers/net/bnx2.c:MODULE_PARM_DESC(disable_msi, "Disable Message Signaled Interrupt (MSI)"); drivers/net/chelsio/cxgb2.c:MODULE_PARM_DESC(disable_msi, "Disable Message Signaled Interrupt (MSI)"); drivers/net/sky2.c:MODULE_PARM_DESC(disable_msi, "Disable Message Signaled Interrupt (MSI)");

Misc:
sound/pci/hda/hda_intel.c:MODULE_PARM_DESC(enable_msi, "Enable Message Signaled Interrupt (MSI)"); drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_spi, " Enable MSI Support for SPI \
drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_fc, " Enable MSI Support for FC \
drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_sas, " Enable MSI Support for SAS \
drivers/net/myri10ge/myri10ge.c:MODULE_PARM_DESC(myri10ge_msi, "Enable Message Signalled Interrupts");

MSI-X:

Param "msi_x", 2 users:
drivers/infiniband/hw/mthca/mthca_main.c:MODULE_PARM_DESC(msi_x, "attempt to use MSI-X if nonzero"); drivers/net/mlx4/main.c:MODULE_PARM_DESC(msi_x, "attempt to use MSI-X if nonzero");

Misc:
drivers/net/forcedeth.c:MODULE_PARM_DESC(msix, "MSIX interrupts are enabled by setting to 1 and disabled by setting to 0."); drivers/net/qlcnic/qlcnic_main.c:MODULE_PARM_DESC(use_msi_x, "MSI-X interrupt (0=disabled, 1=enabled"); drivers/scsi/bfa/bfad_intr.c:MODULE_PARM_DESC(msix_disable_cb, "Disable MSIX for Brocade-415/425/815/825" drivers/scsi/bfa/bfad_intr.c:MODULE_PARM_DESC(msix_disable_ct, "Disable MSIX for Brocade-1010/1020/804" drivers/scsi/mpt2sas/mpt2sas_base.c:MODULE_PARM_DESC(msix_disable, " disable msix routed interrupts (default=0)"); drivers/scsi/qla4xxx/ql4_os.c:MODULE_PARM_DESC(ql4xenablemsix, drivers/scsi/vmw_pvscsi.c:MODULE_PARM_DESC(disable_msix, "Disable MSI-X use in driver - (default=0)");

Ben Hutchings

2010-10-26 13:32 UTC #6 lore raw

Quoted 35 lines · Michael Ellerman

Quoted 25 lines · David Miller
From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100
Quoted 16 lines
Quoted 6 lines · David Miller
From: Somnath Kotur <[email protected]>
Date: Mon, 25 Oct 2010 16:42:35 +0530
Quoted 3 lines
By default, be2net uses MSIx wherever possible.
Adding a module parameter to use INTx for users who do not want to use MSIx.
Quoted 8 lines
Quoted 1 line
Signed-off-by Somnath Kotur <[email protected]>
Either add a new ethtool flag, or use the PCI subsystem facilities for tweaking things to implement this.
Do not use a module option, otherwise every other networking driver author will get the same "cool" idea, give the module option different names, and the resulting user experience is terrible.
Quoted 5 lines
This has already happened, sadly. So far as I can see it's mostly done to allow users to work around systems with broken MSIs; I'm not aware of any other reason to prefer legacy interrupts. However, the PCI subsystem already implements a blacklist and a kernel parameter for disabling MSIs on these systems.
Quoted 3 lines
The PCI subsystem bits I'm totally fine with.
But in the drivers themselves, that's what I don't want.
That horse has really really bolted, it's gawn.
I count 26 drivers with "disable MSI/X" parameters. Some even have more than one.

Quoted 7 lines

11 of them are network drivers, 9 scsi, 3 ata.
I agree it's a mess for users, but it's probably preferable to a non-working driver.
Ethtool would be nice, but only for network drivers. Is there a generic solution, quirks are obviously not keeping people happy.

Since this is (normally) a property of the system, pci=nomsi is the generic solution.

[...]

Quoted 6 lines

Misc:
sound/pci/hda/hda_intel.c:MODULE_PARM_DESC(enable_msi, "Enable Message Signaled Interrupt (MSI)"); drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_spi, " Enable MSI Support for SPI \
drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_fc, " Enable MSI Support for FC \
drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_sas, " Enable MSI Support for SAS \
drivers/net/myri10ge/myri10ge.c:MODULE_PARM_DESC(myri10ge_msi, "Enable Message Signalled Interrupts");

[...]

  drivers/net/sfc/efx.c:MODULE_PARM_DESC(interrupt_mode,
  drivers/net/sfc/efx.c-                 "Interrupt mode (0=>MSIX 1=>MSI 2=>legacy)");

Ben.

Signature · 3 lines

Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Michael Ellerman

2010-10-26 23:20 UTC #7 lore raw

Quoted 10 lines · Ben Hutchings

Quoted 6 lines · Michael Ellerman
Quoted 2 lines · David Miller
From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100
Ethtool would be nice, but only for network drivers. Is there a generic solution, quirks are obviously not keeping people happy.
Since this is (normally) a property of the system, pci=nomsi is the generic solution.

Sort of, it's a big hammer. Did all these driver writers not know about pci=nomsi or did they prefer to add a parameter to their driver for some reason?

Quoted 11 lines

[...]
Quoted 6 lines
Misc:
sound/pci/hda/hda_intel.c:MODULE_PARM_DESC(enable_msi, "Enable Message Signaled Interrupt (MSI)"); drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_spi, " Enable MSI Support for SPI \
drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_fc, " Enable MSI Support for FC \
drivers/message/fusion/mptbase.c:MODULE_PARM_DESC(mpt_msi_enable_sas, " Enable MSI Support for SAS \
drivers/net/myri10ge/myri10ge.c:MODULE_PARM_DESC(myri10ge_msi, "Enable Message Signalled Interrupts");
[...]
  drivers/net/sfc/efx.c:MODULE_PARM_DESC(interrupt_mode,
  drivers/net/sfc/efx.c-                 "Interrupt mode (0=>MSIX 1=>MSI 2=>legacy)");

Well spotted.

cheers

David Miller

maintainer

2010-10-27 15:46 UTC #8 lore raw

From: Michael Ellerman <[email protected]>
Date: Wed, 27 Oct 2010 10:20:35 +1100

Quoted 9 lines · Ben Hutchings

Quoted 8 lines
Quoted 6 lines · Michael Ellerman
Quoted 2 lines · David Miller
From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100
Ethtool would be nice, but only for network drivers. Is there a generic solution, quirks are obviously not keeping people happy.

Quoted 6 lines

Quoted 2 lines
Since this is (normally) a property of the system, pci=nomsi is the generic solution.
Sort of, it's a big hammer. Did all these driver writers not know about pci=nomsi or did they prefer to add a parameter to their driver for some reason?

Every time I've actually done the work to try and track down the true issue, it always turned out to be a PCI chipset problem rather than a device specific issue.

Grant Grundler

2010-10-28 05:37 UTC #9 lore raw

Quoted 15 lines · David Miller

From: Michael Ellerman <[email protected]>
Date: Wed, 27 Oct 2010 10:20:35 +1100
Quoted 12 lines · Ben Hutchings
Quoted 10 lines
Quoted 6 lines · Michael Ellerman
Quoted 2 lines · David Miller
From: Ben Hutchings <[email protected]>
Date: Mon, 25 Oct 2010 23:38:53 +0100
Ethtool would be nice, but only for network drivers. Is there a generic solution, quirks are obviously not keeping people happy.
Since this is (normally) a property of the system, pci=nomsi is the generic solution.

Quoted 7 lines

Quoted 3 lines
Sort of, it's a big hammer. Did all these driver writers not know about pci=nomsi or did they prefer to add a parameter to their driver for some reason?
Every time I've actually done the work to try and track down the true issue, it always turned out to be a PCI chipset problem rather than a device specific issue.

I agree with your generalization. I can think of only one exception:
ISTR pre-5705 tg3 chips would send both MSI and assert IRQ line at the same time.

My guess is driver writers just want knob to "work around" any issues *their* driver might see with chipset. Disabling MSI for all drivers doesn't leave opportunity for experimenting with individual drivers.

hth,
grant

Quoted 4 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-pci" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

David Miller

maintainer

2010-10-27 15:45 UTC #10 lore raw

In reply to Michael Ellerman

From: Michael Ellerman <[email protected]>
Date: Tue, 26 Oct 2010 17:52:08 +1100

Quoted 9 lines

That horse has really really bolted, it's gawn.
I count 26 drivers with "disable MSI/X" parameters. Some even have more than one.
11 of them are network drivers, 9 scsi, 3 ata.
I agree it's a mess for users, but it's probably preferable to a non-working driver.

Stupid inappropriate things being in the tree doesn't mean I need to accept more of them.

Matthew Wilcox

2010-10-30 23:21 UTC #11 lore raw

In reply to Michael Ellerman

Quoted 4 lines · Michael Ellerman

That horse has really really bolted, it's gawn.
I count 26 drivers with "disable MSI/X" parameters. Some even have more than one.

That's 26 patches someone needs to write, then. You can put my Acked-by on all of them.

Quoted 2 lines

I agree it's a mess for users, but it's probably preferable to a non-working driver.

What more drivers need is an automatic detection of a non-working interrupt situation, great big warning messages, and fallback to an alternate interrupt mechanism. Doing it for one driver, then generalising as much of it into the core as possible would be nice.

Signature · 4 lines

Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

Michael Ellerman

2010-11-03 12:45 UTC #12 lore raw

Quoted 8 lines · Matthew Wilcox

Quoted 4 lines · Michael Ellerman
That horse has really really bolted, it's gawn.
I count 26 drivers with "disable MSI/X" parameters. Some even have more than one.
That's 26 patches someone needs to write, then. You can put my Acked-by on all of them.

Bah, come on it's hardly the most horrendous sin committed by driver writers. And removing them risks breaking someone's system, even if they are clueless, should RTFM etc.

Quoted 7 lines

Quoted 2 lines
I agree it's a mess for users, but it's probably preferable to a non-working driver.
What more drivers need is an automatic detection of a non-working interrupt situation, great big warning messages, and fallback to an alternate interrupt mechanism. Doing it for one driver, then generalising as much of it into the core as possible would be nice.

More detection would be good.

I don't see much potential for generalising it though. Looking at e1000e and tg3 there is really not much in common except the very basic idea.

cheers

David Miller

maintainer

2010-11-03 15:28 UTC #13 lore raw

From: Michael Ellerman <[email protected]>
Date: Wed, 03 Nov 2010 23:45:57 +1100

Quoted 12 lines · Matthew Wilcox

Quoted 8 lines · Michael Ellerman
Quoted 4 lines
That horse has really really bolted, it's gawn.
I count 26 drivers with "disable MSI/X" parameters. Some even have more than one.
That's 26 patches someone needs to write, then. You can put my Acked-by on all of them.
Bah, come on it's hardly the most horrendous sin committed by driver writers.

Yes but it's pretty high on the list because it deteriorates the user experience.

It's a real selfish move to try and "fix" things in this way and we've established that this isn't acceptable in the past.