Skip to contentSkip to conversation

e1000e: test MSI interrupts

PATCH In review Auke Kok 5 participants 16 messages Mar 26, 2008 – Mar 28, 2008

Subsystems (2)
lore
Share & tools
lore

Auke Kok

author
#0 lore raw

PATCH 1/1 e1000e: test MSI interrupts

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

---

 drivers/net/e1000e/defines.h |    1 
 drivers/net/e1000e/e1000.h   |    1 
 drivers/net/e1000e/netdev.c  |  161 ++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 155 insertions(+), 8 deletions(-)
drivers/net/e1000e/defines.h +1 -0
index a4f511f..c3593f3 100644
@@ -385,6 +385,7 @@
 
 /* Interrupt Cause Set */
 #define E1000_ICS_LSC       E1000_ICR_LSC       /* Link Status Change */
+#define E1000_ICS_RXSEQ     E1000_ICR_RXSEQ     /* rx sequence error */
 #define E1000_ICS_RXDMT0    E1000_ICR_RXDMT0    /* rx desc min. threshold */
 
 /* Transmit Descriptor Control */
drivers/net/e1000e/e1000.h +1 -0
index 4bf0c6c..556b258 100644
@@ -305,6 +305,7 @@ struct e1000_info {
 #define FLAG_MSI_ENABLED                  (1 << 27)
 #define FLAG_RX_CSUM_ENABLED              (1 << 28)
 #define FLAG_TSO_FORCE                    (1 << 29)
+#define FLAG_MSI_TEST_FAILED              (1 << 30)
 
 #define E1000_RX_DESC_PS(R, i)	    \
 	(&(((union e1000_rx_desc_packet_split *)((R).desc))[i]))
drivers/net/e1000e/netdev.c +153 -8

190 lines not on this page · raw message

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

#1 lore raw
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 8 lines

---

 drivers/net/e1000e/defines.h |    1 
 drivers/net/e1000e/e1000.h   |    1 
 drivers/net/e1000e/netdev.c  |  161 ++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 155 insertions(+), 8 deletions(-)

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

The platforms with MSI problems should be discovered, made public, and worked around.

Otherwise you hide the same problem, just for someone else to discover with another component of the machine.

Jeff

author
#2 lore raw
Quoted 20 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 8 lines

---

 drivers/net/e1000e/defines.h |    1 
 drivers/net/e1000e/e1000.h   |    1 
 drivers/net/e1000e/netdev.c  |  161 ++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 155 insertions(+), 8 deletions(-)

Ah, the perennial add-same-test-to-every-driver conundrum.

Quoted 8 lines

I think we are far enough along with MSI to _not_ do this anymore in drivers.

The platforms with MSI problems should be discovered, made public, and worked around.

Otherwise you hide the same problem, just for someone else to discover with another component of the machine.

ok, that's reasonable as I have not recently seen any "new" reports against the current trees.

We have to keep this patch in our out-of-tree version around though, since a lot of distros still ship kernels without all the new MSI quirks. But that's not for this forum.

Thanks,

Auke

#3 lore raw
Quoted 10 lines · Auke
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

Actually, I'm hoping you'll allow this Jeff, we have a production system (see below) we know about that doesn't like the way 82571 formats MSI interrupt messages. All other systems seem to be okay with this format of MSI messages, but this system implemented a stricter interpretation of the spec, and so even though that system doesn't need a quirk for MSI because MSI works in general, we still MUST test the MSI vector to make sure it works *for us* In this case it comes down to being an errata workaround.

Since there is no way to "test" generation of an interrupt from any specific hardware device without internal knowledge of said device, there isn't a way for us to help the kernel by writing a generic "test MSI" routine.

I would prefer this "generic test" code be in the driver rather than having to identify all the chipsets that fail and have the driver do *specific chipset* detection ala bnx2.c's 8132 bridge workaround.

Quoted 2 lines
Quoted 2 lines

The platforms with MSI problems should be discovered, made public, and worked around.

This is our workaround, it is our fault, the incompatible chipset is in an x3850 IBM system.

Quoted 2 lines
Quoted 2 lines

Otherwise you hide the same problem, just for someone else to discover with another component of the machine.

We had avoided this test in the past based on your recommendation but in this case I don't see a way around it, do you? I'm open to other suggestions but we need to solve the problem somehow in an *automated* way. IBM requested we do it this way.

Jesse

#4 lore raw
Quoted 28 lines · Jesse
Quoted 10 lines · Auke
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

Actually, I'm hoping you'll allow this Jeff, we have a production system (see below) we know about that doesn't like the way 82571 formats MSI interrupt messages. All other systems seem to be okay with this format of MSI messages, but this system implemented a stricter interpretation of the spec, and so even though that system doesn't need a quirk for MSI because MSI works in general, we still MUST test the MSI vector to make sure it works *for us* In this case it comes down to being an errata workaround.

Quoted 8 lines

Since there is no way to "test" generation of an interrupt from any specific hardware device without internal knowledge of said device, there isn't a way for us to help the kernel by writing a generic "test MSI" routine.

I would prefer this "generic test" code be in the driver rather than having to identify all the chipsets that fail and have the driver do *specific chipset* detection ala bnx2.c's 8132 bridge workaround.

Well if it's a problem with the networking chip rather than the platform, then absolutely stick it in the driver (unless its so severe it needs to be in pci/quirks.c before the driver even loads).

But that seems like a quick id test, with no need for all that generic MSI test code.

Certainly the work is scoped based on where the problem lies, either platform, device, or platform+device.

Jeff

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

maintainer
#11 lore raw
In reply to Brandeburg, Jesse

From: "Brandeburg, Jesse" <[email protected]>
Date: Thu, 27 Mar 2008 10:53:43 -0700

Quoted 5 lines
Quoted 2 lines
Quoted 2 lines

The platforms with MSI problems should be discovered, made public, and worked around.

This is our workaround, it is our fault, the incompatible chipset is in an x3850 IBM system.

Then you look for that system chipset variant in the e1000e driver and not use MSI when it is found.

A generic test really is not warranted.

#12 lore raw
In reply to Brandeburg, Jesse
Quoted 37 lines · Jesse
Quoted 10 lines · Auke
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 7 lines
Quoted 7 lines
Quoted 2 lines

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

Actually, I'm hoping you'll allow this Jeff, we have a production system (see below) we know about that doesn't like the way 82571 formats MSI interrupt messages. All other systems seem to be okay with this format of MSI messages, but this system implemented a stricter interpretation of the spec, and so even though that system doesn't need a quirk for MSI because MSI works in general, we still MUST test the MSI vector to make sure it works *for us* In this case it comes down to being an errata workaround.

Since there is no way to "test" generation of an interrupt from any specific hardware device without internal knowledge of said device, there isn't a way for us to help the kernel by writing a generic "test MSI" routine.

I would prefer this "generic test" code be in the driver rather than having to identify all the chipsets that fail and have the driver do *specific chipset* detection ala bnx2.c's 8132 bridge workaround.

Quoted 6 lines
Quoted 2 lines
Quoted 2 lines

The platforms with MSI problems should be discovered, made public, and worked around.

This is our workaround, it is our fault, the incompatible chipset is in an x3850 IBM system.

I've seem similar problems on other systems, so this would be a nice bit of code to have somewhere. I can see Jeff's argument for having this outside of drivers, but to make my life easier I'd like to see this in e1000e (and e1000!). :-)

I suppose an more general alternative would be to make this code run as an ethtool test (current or new one) and then have a module option to disable/enable MSI so all could be done in userspace, but something automatic like this sure would be great.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

maintainer
#5 lore raw
In reply to Jeff Garzik

From: Jeff Garzik <[email protected]>
Date: Thu, 27 Mar 2008 15:43:42 -0400

Quoted 2 lines

But that seems like a quick id test, with no need for all that generic MSI test code.

Yes, that is the way to do it.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

author
#13 lore raw
In reply to Andy Gospodarek
Quoted 38 lines · Andy Gospodarek
Quoted 37 lines · Jesse
Quoted 10 lines · Auke
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

Actually, I'm hoping you'll allow this Jeff, we have a production system (see below) we know about that doesn't like the way 82571 formats MSI interrupt messages. All other systems seem to be okay with this format of MSI messages, but this system implemented a stricter interpretation of the spec, and so even though that system doesn't need a quirk for MSI because MSI works in general, we still MUST test the MSI vector to make sure it works *for us* In this case it comes down to being an errata workaround.

Since there is no way to "test" generation of an interrupt from any specific hardware device without internal knowledge of said device, there isn't a way for us to help the kernel by writing a generic "test MSI" routine.

I would prefer this "generic test" code be in the driver rather than having to identify all the chipsets that fail and have the driver do *specific chipset* detection ala bnx2.c's 8132 bridge workaround.

Quoted 2 lines
Quoted 2 lines

The platforms with MSI problems should be discovered, made public, and worked around.

Quoted 8 lines
Quoted 3 lines

This is our workaround, it is our fault, the incompatible chipset is in an x3850 IBM system.

I've seem similar problems on other systems, so this would be a nice bit of code to have somewhere. I can see Jeff's argument for having this outside of drivers, but to make my life easier I'd like to see this in e1000e (and e1000!). :-)

e1000 will soon no longer support PCI Express devices, so this is unneeded.

Quoted 4 lines

I suppose an more general alternative would be to make this code run as an ethtool test (current or new one) and then have a module option to disable/enable MSI so all could be done in userspace, but something automatic like this sure would be great.

a lot of work ...

#14 lore raw
Quoted 43 lines · Auke
Quoted 42 lines · Andy Gospodarek
Quoted 40 lines · Jesse
Quoted 10 lines · Auke
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

Actually, I'm hoping you'll allow this Jeff, we have a production system (see below) we know about that doesn't like the way 82571 formats MSI interrupt messages. All other systems seem to be okay with this format of MSI messages, but this system implemented a stricter interpretation of the spec, and so even though that system doesn't need a quirk for MSI because MSI works in general, we still MUST test the MSI vector to make sure it works *for us* In this case it comes down to being an errata workaround.

Since there is no way to "test" generation of an interrupt from any specific hardware device without internal knowledge of said device, there isn't a way for us to help the kernel by writing a generic "test MSI" routine.

I would prefer this "generic test" code be in the driver rather than having to identify all the chipsets that fail and have the driver do *specific chipset* detection ala bnx2.c's 8132 bridge workaround.

Quoted 2 lines
Quoted 2 lines

The platforms with MSI problems should be discovered, made public, and worked around.

This is our workaround, it is our fault, the incompatible chipset is in an x3850 IBM system.

Quoted 7 lines
Quoted 4 lines

I've seem similar problems on other systems, so this would be a nice bit of code to have somewhere. I can see Jeff's argument for having this outside of drivers, but to make my life easier I'd like to see this in e1000e (and e1000!). :-)

e1000 will soon no longer support PCI Express devices, so this is unneeded.

So you plan to drop the bits around the device additions that are related to e1000e config sometime soon?

author
#15 lore raw
Quoted 47 lines · Andy Gospodarek
Quoted 46 lines · Auke
Quoted 45 lines · Andy Gospodarek
Quoted 31 lines · Jesse
Quoted 10 lines · Auke
Quoted 9 lines · Jeff Garzik
Quoted 8 lines · Auke Kok

From: Jesse Brandeburg <[email protected]>

Test the MSI interrupt physically once before assuming that it actually works. Several platforms have already come across that have non-functional MSI interrupts and this code will attempt to detect those safely. Once the test succeeds MSI interrupts will be enabled.

Quoted 6 lines
Quoted 6 lines
Quoted 2 lines

Ah, the perennial add-same-test-to-every-driver conundrum.

I think we are far enough along with MSI to _not_ do this anymore in drivers.

Actually, I'm hoping you'll allow this Jeff, we have a production system (see below) we know about that doesn't like the way 82571 formats MSI interrupt messages. All other systems seem to be okay with this format of MSI messages, but this system implemented a stricter interpretation of the spec, and so even though that system doesn't need a quirk for MSI because MSI works in general, we still MUST test the MSI vector to make sure it works *for us* In this case it comes down to being an errata workaround.

Since there is no way to "test" generation of an interrupt from any specific hardware device without internal knowledge of said device, there isn't a way for us to help the kernel by writing a generic "test MSI" routine.

Quoted 9 lines

I would prefer this "generic test" code be in the driver rather than having to identify all the chipsets that fail and have the driver do *specific chipset* detection ala bnx2.c's 8132 bridge workaround.

Quoted 2 lines
Quoted 2 lines

The platforms with MSI problems should be discovered, made public, and worked around.

This is our workaround, it is our fault, the incompatible chipset is in an x3850 IBM system.

I've seem similar problems on other systems, so this would be a nice bit of code to have somewhere. I can see Jeff's argument for having this outside of drivers, but to make my life easier I'd like to see this in e1000e (and e1000!). :-)

Quoted 5 lines
Quoted 2 lines

e1000 will soon no longer support PCI Express devices, so this is unneeded.

So you plan to drop the bits around the device additions that are related to e1000e config sometime soon?

yes I was hoping to do this for 2.6.26 already, if not 2.6.27.

I definately don't want to have that linger.

Auke

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

#6 lore raw
In reply to David Miller
Quoted 7 lines · David Miller

From: Jeff Garzik <[email protected]>
Date: Thu, 27 Mar 2008 15:43:42 -0400

Quoted 2 lines

But that seems like a quick id test, with no need for all that generic MSI test code.

Yes, that is the way to do it.

I get your point, but this seems a maintainance problem due to not being able to "future proof" the solution. I know what is (IDs) available now, but I don't know how many systems in the future IBM will release with a similar bridge but a different device ID that causes the same issue. Should we take on the maintenance of continually having to add every new bridge device that has this issue to our driver? Users just want this stuff to work when they plug it in.

tg3.c has exactly this kind of test and workaround (in fact its where I got the code) to work around the same kind of issues.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

maintainer
#7 lore raw

From: "Brandeburg, Jesse" <[email protected]>
Date: Thu, 27 Mar 2008 15:55:34 -0700

Quoted 9 lines
Quoted 7 lines · David Miller

From: Jeff Garzik <[email protected]>
Date: Thu, 27 Mar 2008 15:43:42 -0400

Quoted 2 lines

But that seems like a quick id test, with no need for all that generic MSI test code.

Yes, that is the way to do it.

Quoted 7 lines

I get your point, but this seems a maintainance problem due to not being able to "future proof" the solution. I know what is (IDs) available now, but I don't know how many systems in the future IBM will release with a similar bridge but a different device ID that causes the same issue. Should we take on the maintenance of continually having to add every new bridge device that has this issue to our driver? Users just want this stuff to work when they plug it in.

This is exactly what the PCI quirks list is and it works just fine.

I know the truth is that Intel as a vendor frowns upon putting into it's driver a list of another vendor's PCI IDs as errata items because it looks bad.

So just be honest about that.

It is, however, the correct way to address this problem.

Quoted 2 lines

tg3.c has exactly this kind of test and workaround (in fact its where I got the code) to work around the same kind of issues.

That is from an era when the situation was much different.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

#8 lore raw
In reply to Brandeburg, Jesse
Quoted 5 lines · Jesse

I get your point, but this seems a maintainance problem due to not being able to "future proof" the solution. I know what is (IDs) available now, but I don't know how many systems in the future IBM will release with a similar bridge but a different device ID that causes the same issue.

Future-proofing in that way is a pipe dream.

You hope to predict what _errata_, what out-of-spec behavior future hardware will have. Trying to code for such N^M possible futures will lead to code bloat, depression, and eventually madness.

Quoted 3 lines

Should we take on the maintenance of continually having to add every new bridge device that has this issue to our driver? Users just want this stuff to work when they plug it in.

As David noted, we touch quirks.c all the time for various platform eccentricities. Adding a new id is easy and takes two seconds. The same ease of change applies to any driver-local list of ids, too.

Anyway, I think a better question to ask is: should we bloat up every driver testing for platform quirks found on a minority of platforms?

Moreover, "it doesn't work" type errata is typically fixed in future chip generations -- making any such generic test /less/ valuable, because of the lower likelihood that IBM will continue to release this buggy hardware for decades.

We have an existing "this bridge and MSI don't get along" list. Adding an id is a one-line patch.

Jeff

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

author
#9 lore raw
Quoted 25 lines · Jeff Garzik
Quoted 5 lines · Jesse

I get your point, but this seems a maintainance problem due to not being able to "future proof" the solution. I know what is (IDs) available now, but I don't know how many systems in the future IBM will release with a similar bridge but a different device ID that causes the same issue.

Future-proofing in that way is a pipe dream.

You hope to predict what _errata_, what out-of-spec behavior future hardware will have. Trying to code for such N^M possible futures will lead to code bloat, depression, and eventually madness.

Quoted 3 lines

Should we take on the maintenance of continually having to add every new bridge device that has this issue to our driver? Users just want this stuff to work when they plug it in.

As David noted, we touch quirks.c all the time for various platform eccentricities. Adding a new id is easy and takes two seconds. The same ease of change applies to any driver-local list of ids, too.

Anyway, I think a better question to ask is: should we bloat up every driver testing for platform quirks found on a minority of platforms?

Quoted 7 lines

Moreover, "it doesn't work" type errata is typically fixed in future chip generations -- making any such generic test /less/ valuable, because of the lower likelihood that IBM will continue to release this buggy hardware for decades.

We have an existing "this bridge and MSI don't get along" list. Adding an id is a one-line patch.

well it's just slightly more complex than just that... the bridge works OK with MSI for every device except 82571

so 82572, 82573 e1000's work fine as well as all the newer pci-e e1000's, or any of the non-e1000 hardware that uses MSI interrupts.

so disabling MSI entirely for that chipset by quirking it is not what we want to do in the first place.

hence the test :)

the only alternative I see is that we have a quirk in the e1000 driver that acts when the combination of an 82571 and this particular chipset is found, but I have no information ATM on the chipset id's yet and on how to identify this configuration.

Auke

maintainer
#10 lore raw

From: "Kok, Auke" <[email protected]>
Date: Thu, 27 Mar 2008 16:53:55 -0700

Quoted 4 lines

the only alternative I see is that we have a quirk in the e1000 driver that acts when the combination of an 82571 and this particular chipset is found, but I have no information ATM on the chipset id's yet and on how to identify this configuration.

And this is how you will fix this bug, once you know these IDs.

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace. It's the best place to buy or sell services for just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace