Skip to contentSkip to conversation

Driver SFC: Possible bug in LM87 temperature XFP detection code

PATCH In review Jesper Dangaard Brouer 5 participants 11 messages Apr 28, 2009 – Apr 30, 2009

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Ben,

I have borrowed some SMC10GPCIe-XFP NICs directly from SMC for evaluation. The NICs uses a Solarflare Chip and the SFC driver.

If unpluging the fiber cable I start getting these errors:

+--------
sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 30:00) INTERNAL EXTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY

sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 30:00) INTERNAL EXTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY

sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 10:00) INTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY
+---------

Reading through the driver code (drivers/net/sfc/boards.c), this problem is related to temperature.

The real issues is that I cannot get the device up and running again after lowering the temperature. Only if I unload and load the sfc driver, then I can get the device running again.

I'm thinking perhaps there is missing a PHY power up again, after the temperature alarm has gone?

I'm using kernel 2.6.30-rc1-net-next-00664-gd93fe1a.

To Ben; do you have anything you want me to try. Do you want to fix this you self, or can you give me some code hints or patches to try out?

I'm wondering what chip the SMC NIC is using? From lspci is says SFC4000, but does that corrospond to EFX_BOARD_SFE4001 or EFX_BOARD_SFE4002 ?

(Additional tech info below signature)

Signature · 80 lines
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer



* XFP modules used
------------------

 I have tested with two different XFP modules:
  (1) Finisar FTLX1412D3BCL 10GBASE-LR/LW
  (2) Sumitomo SXP3101LX

* lspci info
------------

 lspci -xvs 12:00.0

+---------
12:00.0 Ethernet controller: Solarflare Communications SFC4000 rev B
[Solarstorm] (rev 02)
        Subsystem: Standard Microsystems Corp [SMC] SMC10GPCIe-XFP (A1)
[TigerCard]
        Flags: bus master, fast devsel, latency 0, IRQ 19
        I/O ports at e800 [size=256]
        Memory at f8000000 (64-bit, non-prefetchable) [size=64M]
        Expansion ROM at feaf0000 [disabled] [size=64K]
        Capabilities: [40] Power Management version 3
        Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
Queue=0/3 Enable-
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [90] MSI-X: Enable+ Mask- TabSize=64
        Capabilities: [b0] Vital Product Data <?>
        Capabilities: [100] Advanced Error Reporting <?>
        Capabilities: [140] Device Serial Number 00-0f-53-ff-ff-2c-b2-1a
        Kernel driver in use: sfc
        Kernel modules: sfc
00: 24 19 10 07 07 05 10 00 02 00 00 02 10 00 00 00
10: 01 e8 00 00 00 00 00 00 04 00 00 f8 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 b8 10 01 02
30: 00 00 af fe 40 00 00 00 00 00 00 00 0a 01 00 00
+-------

* Relevant kernel CONFIG_xxx
----------------------------

 CONFIG_SFC_MTD not set
 CONFIG_SENSORS_LM87=m
 CONFIG_SFC=m


* ethtool status
----------------
 ethtool eth88

+-------
Settings for eth88:
        Supported ports: [ FIBRE ]
        Supported link modes:   
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 2
        Transceiver: internal
        Auto-negotiation: off
        Link detected: no
+-------

* SMC links
-----------

 SMC10GPCIe-XFP TigerCard™ 10G
 http://www.smc.com/index.cfm?event=viewProduct&cid=9&scid=51&pid=1648
#1 lore raw
Quoted 14 lines · Jesper Dangaard Brouer

Hi Ben,

I have borrowed some SMC10GPCIe-XFP NICs directly from SMC for evaluation. The NICs uses a Solarflare Chip and the SFC driver.

If unpluging the fiber cable I start getting these errors:

+--------
sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 30:00) INTERNAL EXTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY

sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 30:00) INTERNAL EXTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY

Quoted 6 lines

sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 10:00) INTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY
+---------

Reading through the driver code (drivers/net/sfc/boards.c), this problem is related to temperature.

Right. And the sensors are not polled while the link is up, on the assumption that a temperature or voltage fault will cause the link to go down, and because bit-banged I2C will reduce throughput slightly.

Quoted 6 lines

The real issues is that I cannot get the device up and running again after lowering the temperature. Only if I unload and load the sfc driver, then I can get the device running again.

I'm thinking perhaps there is missing a PHY power up again, after the temperature alarm has gone?

We considered it most important to shut down the board to prevent or mitigate damage, and did not implement any recovery beyond that.

Quoted 5 lines

I'm using kernel 2.6.30-rc1-net-next-00664-gd93fe1a.

To Ben; do you have anything you want me to try. Do you want to fix this you self, or can you give me some code hints or patches to try out?

I don't intend to fix this myself. If you want to try implementing this then you should start by looking at efx_monitor() in efx.c. However, I think your time might be better spent in fixing the air flow in the computer before the board is permanently damaged.

Quoted 3 lines

I'm wondering what chip the SMC NIC is using? From lspci is says SFC4000, but does that corrospond to EFX_BOARD_SFE4001 or EFX_BOARD_SFE4002 ?

The SMC10GPCIe-XFP is based on SFE4002.

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#2 lore raw
Quoted 19 lines · Ben Hutchings
Quoted 18 lines · Jesper Dangaard Brouer

Hi Ben,

I have borrowed some SMC10GPCIe-XFP NICs directly from SMC for evaluation. The NICs uses a Solarflare Chip and the SFC driver.

If unpluging the fiber cable I start getting these errors:

+--------
sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 30:00) INTERNAL EXTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY

sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 30:00) INTERNAL EXTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY

sfc 0000:12:00.0: ERR: eth88 LM87 detected a hardware failure (status 10:00) INTERNAL sfc 0000:12:00.0: ERR: eth88 Board sensor reported fault; shutting down PHY
+---------

Quoted 6 lines
Quoted 2 lines

Reading through the driver code (drivers/net/sfc/boards.c), this problem is related to temperature.

Right. And the sensors are not polled while the link is up, on the assumption that a temperature or voltage fault will cause the link to go down, and because bit-banged I2C will reduce throughput slightly.

In my situation the link does not go down due the temperature issue.

Quoted 9 lines
Quoted 6 lines

The real issues is that I cannot get the device up and running again after lowering the temperature. Only if I unload and load the sfc driver, then I can get the device running again.

I'm thinking perhaps there is missing a PHY power up again, after the temperature alarm has gone?

We considered it most important to shut down the board to prevent or mitigate damage, and did not implement any recovery beyond that.

Im my case putting the PHY in PHY_MODE_LOW_POWER, does not help lowering the temperature. The errors are continous, until I apply "manual" airflow ;-)

Quoted 10 lines
Quoted 5 lines

I'm using kernel 2.6.30-rc1-net-next-00664-gd93fe1a.

To Ben; do you have anything you want me to try. Do you want to fix this you self, or can you give me some code hints or patches to try out?

I don't intend to fix this myself. If you want to try implementing this then you should start by looking at efx_monitor() in efx.c. However, I think your time might be better spent in fixing the air flow in the computer before the board is permanently damaged.

I see you point, I don't want to damage the board... not sure I want to fix it then... Although in a production environment, I think the driver should support exchanging a failed XFP without rebooting the server.

Then I also think that we should make the error message a bit more explicit, in order to warn people before the board is permanently damaged. I'll post a patch proposal as reply to this message...

Signature · 7 lines
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer
#3 lore raw

PATCH 1/1 sfc: Make temperature warnings/alarms more explicit.

The sfc driver can detect different hardware failures via the LM87 system. One of the failures I have experienced is the temperature alarm, but the error message didn't reveal that this error was temperature related. I had to read the code to discover that.

I think that the temperature error should be more explicit, in order to warn people before the board is permanently damaged.

---

 drivers/net/sfc/boards.c |    4 +++-
 1 files changed, 3 insertions(+), 1 deletions(-)
drivers/net/sfc/boards.c +3 -1
index 4a4c74c..b1822fe 100644
@@ -121,8 +121,10 @@ static int efx_check_lm87(struct efx_nic *efx, unsigned mask)
 	if (alarms1 || alarms2) {
 		EFX_ERR(efx,
 			"LM87 detected a hardware failure (status %02x:%02x)"
-			"%s%s\n",
+			"%s%s%s\n",
 			alarms1, alarms2,
+			(alarms1 & (LM87_ALARM_TEMP_INT|LM87_ALARM_TEMP_EXT1))
+			 ? " high temperature" : "",
 			(alarms1 & LM87_ALARM_TEMP_INT) ? " INTERNAL" : "",
 			(alarms1 & LM87_ALARM_TEMP_EXT1) ? " EXTERNAL" : "");
 		return -ERANGE;
maintainer
#4 lore raw

Re: [PATCH] sfc: Make temperature warnings/alarms more explicit.

From: Jesper Dangaard Brouer <[email protected]>
Date: Tue, 28 Apr 2009 16:48:04 +0200

Quoted 11 lines

The sfc driver can detect different hardware failures via the LM87 system. One of the failures I have experienced is the temperature alarm, but the error message didn't reveal that this error was temperature related. I had to read the code to discover that.

I think that the temperature error should be more explicit, in order to warn people before the board is permanently damaged.

Ben, ACK or something?

#5 lore raw

Re: [PATCH] sfc: Make temperature warnings/alarms more explicit.

In reply to Jesper Dangaard Brouer
Quoted 8 lines · Jesper Dangaard Brouer

The sfc driver can detect different hardware failures via the LM87 system. One of the failures I have experienced is the temperature alarm, but the error message didn't reveal that this error was temperature related. I had to read the code to discover that.

I think that the temperature error should be more explicit, in order to warn people before the board is permanently damaged.

You are right, but...

Quoted 6 lines
drivers/net/sfc/boards.c +0 -0
index 4a4c74c..b1822fe 100644
@@ -121,8 +121,10 @@ static int efx_check_lm87(struct efx_nic *efx, unsigned mask)
 	if (alarms1 || alarms2) {

drivers/net/sfc/boards.c static int efx_check_lm87(struct efx_nic *efx, unsigned mask) · line 130

Quoted 10 lines
drivers/net/sfc/boards.c +3 -1
 		EFX_ERR(efx,
 			"LM87 detected a hardware failure (status %02x:%02x)"
-			"%s%s\n",
+			"%s%s%s\n",
 			alarms1, alarms2,
+			(alarms1 & (LM87_ALARM_TEMP_INT|LM87_ALARM_TEMP_EXT1))
+			 ? " high temperature" : "",
 			(alarms1 & LM87_ALARM_TEMP_INT) ? " INTERNAL" : "",
 			(alarms1 & LM87_ALARM_TEMP_EXT1) ? " EXTERNAL" : "");
 		return -ERANGE;

We could be more explicit still. How about:

		EFX_ERR(efx,
			"%s out of range (LM87 status %02x:%02x)\n",
			(alarms1 & LM87_ALARM_TEMP_INT) ? "Board temperature" :
			(alarms1 & LM87_ALARM_TEMP_EXT1) ? "Controller temperature :
			"Voltage",
			alarms1, alarms2);

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#6 lore raw

Re: [PATCH] sfc: Make temperature warnings/alarms more explicit.

Quoted 22 lines · Ben Hutchings
Quoted 8 lines · Jesper Dangaard Brouer

The sfc driver can detect different hardware failures via the LM87 system. One of the failures I have experienced is the temperature alarm, but the error message didn't reveal that this error was temperature related. I had to read the code to discover that.

I think that the temperature error should be more explicit, in order to warn people before the board is permanently damaged.

You are right, but...

Quoted 10 lines
drivers/net/sfc/boards.c +1 -1
index 4a4c74c..b1822fe 100644
@@ -121,8 +121,10 @@ static int efx_check_lm87(struct efx_nic *efx, unsigned mask)
 	if (alarms1 || alarms2) {
 		EFX_ERR(efx,
 			"LM87 detected a hardware failure (status %02x:%02x)"
-			"%s%s\n",
+			"%s%s%s\n",
Quoted 8 lines

drivers/net/sfc/boards.c static int efx_check_lm87(struct efx_nic *efx, unsigned mask) · line 130

Quoted 6 lines
drivers/net/sfc/boards.c +2 -0
 			alarms1, alarms2,
+			(alarms1 & (LM87_ALARM_TEMP_INT|LM87_ALARM_TEMP_EXT1))
+			 ? " high temperature" : "",
 			(alarms1 & LM87_ALARM_TEMP_INT) ? " INTERNAL" : "",
 			(alarms1 & LM87_ALARM_TEMP_EXT1) ? " EXTERNAL" : "");
 		return -ERANGE;

We could be more explicit still. How about:

Yes, the "INTERNAL" and "EXTERNAL" is not very descriptive.

Quoted 6 lines
		EFX_ERR(efx,
			"%s out of range (LM87 status %02x:%02x)\n",
			(alarms1 & LM87_ALARM_TEMP_INT) ? "Board temperature" :
			(alarms1 & LM87_ALARM_TEMP_EXT1) ? "Controller temperature :
			"Voltage",

Notice that both LM87_ALARM_TEMP_INT and LM87_ALARM_TEMP_EXT1 can be set at the same time. In that case we only print "Board temperature", is that good enough? (I thinks its okay, as the status will provide developers enough info for debugging the event)

Quoted 1 line
			alarms1, alarms2);

I would like to emphesize that its a hardware alarm, by adding "HW alarm:" see revised patch...

Signature · 25 lines
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer


[PATCH] sfc: Make temperature warnings/alarms more explicit.

The sfc driver can detect different hardware failures via the
LM87 system.  One of the failures I have experienced is the
temperature alarm, but the error message didn't reveal that this
error was temperature related.  I had to read the code to
discover that.

I think that the temperature error should be more explicit, in
order to warn people before the board is permanently damaged.

Signed-off-by: Jesper Dangaard Brouer <[email protected]>
---

 drivers/net/sfc/boards.c |   10 +++++-----
 1 files changed, 5 insertions(+), 5 deletions(-)
drivers/net/sfc/boards.c +5 -5
index 4a4c74c..223866c 100644
@@ -120,11 +120,11 @@ static int efx_check_lm87(struct efx_nic *efx, unsigned mask)
 	alarms2 &= mask >> 8;
 	if (alarms1 || alarms2) {
 		EFX_ERR(efx,
-			"LM87 detected a hardware failure (status %02x:%02x)"
-			"%s%s\n",
-			alarms1, alarms2,
-			(alarms1 & LM87_ALARM_TEMP_INT) ? " INTERNAL" : "",
-			(alarms1 & LM87_ALARM_TEMP_EXT1) ? " EXTERNAL" : "");
+			"HW alarm: %s out of range (LM87 status %02x:%02x)\n",
+			(alarms1 & LM87_ALARM_TEMP_INT) ? "Board temperature" :
+			(alarms1 & LM87_ALARM_TEMP_EXT1) ? "Controller temperature" :
+			 "Voltage",
+			alarms1, alarms2);
 		return -ERANGE;
 	}
 
#7 lore raw
In reply to Ben Hutchings
Quoted 3 lines · I wrote:

However, I
think your time might be better spent in fixing the air flow in the computer before the board is permanently damaged.

It may also be worth checking that the sensors are actually reading correctly. You can read them using the "sensors" command from the lm_sensors package or by running:

    grep . /sys/class/net/eth88/device/i2c-adapter:*/*-002e/temp*_input

Divide the numbers by 1000 to get temperatures in degrees celsius.

The "internal" temperature sensor (temp1_input) is in the LM87, which is placed at the corner of the board away from the bracket and the edge connector.

The "external" temperature sensor (temp2_input) is in the SFC4000.

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.
#8 lore raw
Quoted 5 lines · Ben Hutchings
Quoted 3 lines · I wrote:

However, I
think your time might be better spent in fixing the air flow in the computer before the board is permanently damaged.

Quoted 7 lines

It may also be worth checking that the sensors are actually reading correctly. You can read them using the "sensors" command from the lm_sensors package or by running:

   grep . /sys/class/net/eth88/device/i2c-adapter:*/*-002e/temp*_input

Divide the numbers by 1000 to get temperatures in degrees celsius.

/sys/class/net/eth88/device/i2c-adapter:i2c-1/1-002e/temp1_input:61000 /sys/class/net/eth88/device/i2c-adapter:i2c-1/1-002e/temp2_input:70000

Guess the temperature is fairly high... I can touch it with a finger without getting burned. I also have a 10GbE Sun Neptune (niu) in the machine, if I touch the heatsink on that I burn my finger, so that is even hotter...

Quoted 5 lines

The "internal" temperature sensor (temp1_input) is in the LM87, which is placed at the corner of the board away from the bracket and the edge connector.

The "external" temperature sensor (temp2_input) is in the SFC4000.

I though this was read from the XFP. I was hoping the NIC supported reading i2c stuff from the XFP. Does it support that? (Robert Olsson wanted to play with this stuff)

Cheers,
Jesper Brouer

--
-------------------------------------------------------------------
MSc. Master of Computer Science
Dept. of Computer Science, University of Copenhagen Author of http://www.adsl-optimizer.dk
-------------------------------------------------------------------

#9 lore raw
Quoted 13 lines · Jesper Dangaard Brouer
Quoted 7 lines · Ben Hutchings

It may also be worth checking that the sensors are actually reading correctly. You can read them using the "sensors" command from the lm_sensors package or by running:

   grep . /sys/class/net/eth88/device/i2c-adapter:*/*-002e/temp*_input

Divide the numbers by 1000 to get temperatures in degrees celsius.

/sys/class/net/eth88/device/i2c-adapter:i2c-1/1-002e/temp1_input:61000 /sys/class/net/eth88/device/i2c-adapter:i2c-1/1-002e/temp2_input:70000

Quoted 6 lines

Guess the temperature is fairly high... I can touch it with a finger without getting burned.

I also have a 10GbE Sun Neptune (niu) in the
machine, if I touch the heatsink on that I burn my finger, so that is even hotter...

I got my hands on a InfraRed Thermometer.
The Sun NIC was 98 degrees celsius!!!

Quoted 3 lines
Quoted 3 lines

The "internal" temperature sensor (temp1_input) is in the LM87, which is placed at the corner of the board away from the bracket and the edge connector.

Measuring that point gives the correct measurement using the external infrared thermometer.

Quoted 1 line
Quoted 1 line

The "external" temperature sensor (temp2_input) is in the SFC4000.

It was harder to measure the temperature on the NIC CPU because of its blank surface. I can get a reading of 66 Celcius where the sensor says 68-69 Celcius.

The XFP it self of "only" 55 Celcius. Measured by quickly pulling it out and measuring (and taking care of measuring on a non blank surface point).

Quoted 3 lines

I though this was read from the XFP. I was hoping the NIC supported reading i2c stuff from the XFP. Does it support that? (Robert Olsson wanted to play with this stuff)

Signature · 7 lines
Med venlig hilsen / Best regards
  Jesper Brouer
  ComX Networks A/S
  Linux Network developer
  Cand. Scient Datalog / MSc.
  Author of http://adsl-optimizer.dk
  LinkedIn: http://www.linkedin.com/in/brouer
#10 lore raw
In reply to Jesper Dangaard Brouer

On Wed, 2009-04-29 at 10:52 +0200, Jesper Dangaard Brouer wrote:
[...]

Quoted 7 lines
Quoted 5 lines

The "internal" temperature sensor (temp1_input) is in the LM87, which is placed at the corner of the board away from the bracket and the edge connector.

The "external" temperature sensor (temp2_input) is in the SFC4000.

I though this was read from the XFP.

No.

Quoted 3 lines

I was hoping the NIC supported
reading i2c stuff from the XFP. Does it support that? (Robert Olsson wanted to play with this stuff)

That's connected to the PHY and not the controller. According to the datasheet for the QT2022, you can read out the EEPROM from MDIO registers 1.32775-1.33030 (ignoring bits 8-15). Use the MII ioctl interface and set phy_id to 0x441 (the formula is 0x400 | (prtad << 5) | devad).

Ben.

Signature · 3 lines
Ben Hutchings, Senior Software Engineer, Solarflare Communications
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.